FOSSA: Control Your Software Supply Chain

FOSSA provides a comprehensive solution for open-source license compliance, SBOM (software bill of materials), and vulnerability management, helping companies manage their open-source software usage effectively.  Founded in 2015, the FOSSA SaaS platform runs across multiple services on Kubernetes. Their current observability solution is Datadog which is also used as their logging system. 

Hmmmm, Observability is Expensive!

Observability is a must have for developers building and operating the FOSSA platform. The DevOps team at FOSSA has been happy with their choice of Datadog. The team has built 100+ custom logging based dashboards and about 30 log based alerts, not to mention tracing and metrics based alerts that have been structured. But as the FOSSA business has grown, so has the size of their platform, and thus the amount of logs generated and flowing into Datadog. Since Datadog charges by the amount of data that is ingested and stored on their cloud, the cost of using Datadog caught the attention of management. 

According to Dave Bortz, VP Engineering at FOSSA, “We originally chose Datadog as our logging platform, as we wanted to ensure that something as essential as production logs live on a reliable platform. But as our platform and business scaled, the volume of logs we were sending grew significantly, along with our Datadog bills. We first tried to reduce our bills by being more selective about what we logged, but that's a slow moving process that requires a lot of engineering time, and ultimately limits how much visibility we have into our system.”

What Were the Options?

The platform team at FOSSA started to investigate their options on how to bring the costs of their observability system into line with the value it was delivering to their organization. They had made significant investments in operationalizing Datadog. However, the sheer volume of logs was leading to substantial costs and limiting their ability to retain logs for longer periods, which is crucial for historical analysis and compliance.

Dave continued saying, “That’s when we found Grepr.  We point all logs to them, which moves our primary log storage to a much more affordable S3 bucket, and then they take care of downsampling before continuing to forward the most important data to Datadog. This unlocked huge savings for us while still allowing us to take advantage of Datadog's visibility and tooling.  Implementation was very simple and the initial deployment took less than an hour.” 

The FOSSA DevOps team knew how to use Datadog to help them when they needed to access log data, and they were actively monitoring operations with many custom dashboards - it would be painful to change tools! FOSSA had built over 100 custom logging-based dashboards and approximately 30 log-based alerts in Datadog. Their primary goal was to reduce these expenses without compromising their ability to monitor and troubleshoot their application systems.

But what could be done to attack the costs? They concluded that the optimal solution would be to reduce their ingested data set with changing tooling. Without Grepr, the only way to accomplish this would be to go back and scrutinize/change the logging strategy of every log source by hand. That would take a lot of engineering effort and divert those engineers from building new application features. Dave concluded with, “Since we deployed Grepr, we’re seeing a 95% reduction in log volume and didn’t have to change a thing in our app. I'd recommend Grepr to any team that's experiencing rising costs from an expensive logging platform.”

Conclusion

FOSSA has now rolled out Grepr to 30% of their log ingestion and are continuing to expand to the remainder. The data is quite clear, with Grepr acting as the initial ingestion destination for logs, FOSSA is experiencing a continuous reduction of 94% in the logging volume stored in Datadog. This directly translates to similar percentages in cost reduction.

FOSSA's experience demonstrates Grepr's effectiveness as a powerful tool for optimizing observability costs without sacrificing critical insights or demanding large changes to operational processes.  Learn more at Grepr.ai.

‍