Grepr Blog

Grepr is an intelligent observability data engine that uses pipelines to process log data from sources like Splunk, Datadog, and New Relic. It stores data in low-cost S3 buckets, extracts key information into a standard format, and then uses a series of advanced processing steps like masking, tokenizing, and machine learning-based clustering to reduce the volume of logs by up to 90%. Users can tune the engine's performance with a variety of settings, including a configurable aggregation time window and a logarithmic sampling strategy, to ensure that important troubleshooting information is preserved while noisy, repetitive logs are filtered out.

Is your observability bill growing faster than your engineering team can say "log volume"? You're not alone. FOSSA, a leader in software supply chain management, faced a similar challenge. Their reliance on Datadog, while providing essential visibility, was becoming a significant financial burden as their platform scaled. Instead of a painful, time-consuming overhaul of their entire logging strategy, FOSSA found a smarter way. They discovered a solution that allowed them to dramatically reduce their Datadog costs without sacrificing the crucial insights they needed to monitor and troubleshoot their systems. Want to know how FOSSA achieved a whopping 95% reduction in log volume and kept their observability costs in check? Click to read the full story and discover their secret!

Observability tools are vital for troubleshooting, but their high operational cost, driven by data volume, creates a tension between DevOps teams needing more data and businesses seeking lower bills. This dilemma stems from platforms treating all data as equally important, leading to an "impossible situation." Grepr breaks this conundrum by acting as a shim between log shippers and backends, using semantic machine learning to summarize frequent, noisy messages while passing critical, unique ones straight through. This innovative approach reduces log volume by 90-98% for significant cost savings, yet all data remains accessible in low-cost storage via the Grepr dashboard, REST API, and familiar query syntaxes (Splunk, Datadog, New Relic). This ensures that while you pay only for the 2-10% of data actively used, the rest is available on demand for queries or backfilling during incident investigations, solving the operational versus cost challenge and allowing you to pay only for the data you truly need, when you need it.

Grepr is your ultimate solution for tackling high log management costs without sacrificing crucial insights. Our semantic machine learning technology intelligently sifts through your log data, automatically identifying and summarizing common, noisy messages while ensuring unique, critical events pass straight through. This means you can reduce your log volume sent to backend platforms like Splunk, Datadog, or New Relic by up to 90%, drastically cutting your observability expenses. Plus, with all data retained in low-cost storage and accessible via your preferred query syntax, you maintain 100% troubleshooting capability. Optimize your logs, cut your costs, and keep all your valuable data with Grepr—it's a win-win for your operations and your budget.

This blog post introduces a revolutionary approach to observability, addressing the long-standing "AI-in-a-Haystack" problem in log analysis. Traditional methods struggle with the sheer volume and lack of context in modern telemetry data, making AI analysis financially and technically unfeasible. Grepr offers a unique solution built on three core principles: intelligent telemetry reduction, which de-noises log volumes by over 99% before storage; a stateful stream processing engine, providing AI with the necessary memory and context to understand data trends; and dynamic pipeline control, enabling the AI to reconfigure data streams on the fly to "zoom in" on specific issues. These capabilities transform monitoring from a reactive chore into a proactive, conversational partnership, allowing AI to intelligently flag issues, suggest causes, and dynamically adjust its focus, ultimately leading to faster incident resolution and more efficient operations.

Revolutionary SQL Operators turn your log data into malleable clay—reshape streaming logs in real-time into custom metrics, business-specific traces, security alerts, and compliance reports using familiar SQL syntax. Unlike traditional APM and SIEM tools that force you into predefined structures, SQL Operators with Apache Flink give you unlimited flexibility to create exactly the observability insights your business needs. Perfect for gaming companies tracking multiplayer sessions, security teams detecting advanced threats, and any organization wanting observability that matches their actual business logic. AI-assisted query generation makes complex stream processing accessible to all skill levels.

Discover how Grepr acts as an intelligent pipeline, leveraging AI to detect patterns, summarize noisy data, and directly pass unique messages, which can lead to up to a 90% reduction in data volume and substantial savings on Datadog platform costs. The article walks you through the setup process, from creating necessary integrations and configuring data lakes for low-cost storage, to building pipelines that process and route your data. It also covers how to adjust your Datadog Agent configurations and addresses strategies to mitigate potential data skewing from summarization, ensuring you maintain full insight into your applications without discarding any valuable data.

Grepr is an intelligent observability pipeline that optimizes, analyzes, and routes data in real time, sitting between your agents and observability platform. Utilizing machine learning and a rules engine, it efficiently detects data patterns, filters out repetitive information, and forwards only essential summaries or unique messages. This seamless integration helps organizations significantly cut observability costs by up to 90%, enable long-term data retention, and make valuable insights available for business reporting and AI, all with minimal configuration changes.

Grepr uses unsupervised machine learning to reduce log volume by over 90% while preserving important data through smart, configurable aggregation. It passes low-frequency messages through unmodified, allows engineers to retain specific parameters like user IDs, and supports backfilling logs via API triggers when deeper detail is needed—such as during support tickets. For added flexibility, trace sampling can capture full logs for a subset of users, and all original logs are archived in a searchable data lake. This gives teams control, reduces noise, and enables cost-effective observability without sacrificing access to critical information.

With apologies to George Orwell. Not all Observability data is salient all the time, some data is required all the time but most data is only germane when investigating an issue.

Vector and Grepr both function as observability data pipelines, but they differ sharply in complexity and automation. Vector, an open-source tool sponsored by Datadog, is powerful and flexible but requires extensive manual configuration, domain-specific scripting (VRL), and careful infrastructure planning. In contrast, Grepr is a fully automated, AI-driven observability platform that dynamically manages thousands of data transformations without requiring custom coding. It reduces observability costs by up to 90%, stores all data in queryable formats like Apache Iceberg on AWS S3, and integrates seamlessly with tools like Datadog and Splunk. With Grepr, organizations can deploy in minutes instead of days—without the operational overhead.

Modern web applications are rich, dynamic, and heavily reliant on frontend frameworks like React and Vue, which makes browser-side logging essential for understanding both code execution and user behavior. The Datadog browser logs SDK allows developers to collect this data, but with high traffic, logging can become expensive due to Datadog’s volume-based pricing. Grepr solves this by acting as an intelligent intermediary: it receives all logs, stores them cost-effectively, and uses AI-powered filtering to reduce the volume sent to Datadog by 90%—without dropping any data. It aggregates and summarizes repetitive logs, maintains full fidelity through semantic understanding, and even retains query access to all original data via a dashboard using the same syntax as Datadog. This approach allows developers to maintain 100% insight with only 10% of the data volume and cost, enabling full visibility into user behavior and app performance without budget concerns.