Grepr Blog

This blog post introduces a revolutionary approach to observability, addressing the long-standing "AI-in-a-Haystack" problem in log analysis. Traditional methods struggle with the sheer volume and lack of context in modern telemetry data, making AI analysis financially and technically unfeasible. Grepr offers a unique solution built on three core principles: intelligent telemetry reduction, which de-noises log volumes by over 99% before storage; a stateful stream processing engine, providing AI with the necessary memory and context to understand data trends; and dynamic pipeline control, enabling the AI to reconfigure data streams on the fly to "zoom in" on specific issues. These capabilities transform monitoring from a reactive chore into a proactive, conversational partnership, allowing AI to intelligently flag issues, suggest causes, and dynamically adjust its focus, ultimately leading to faster incident resolution and more efficient operations.

Revolutionary SQL Operators turn your log data into malleable clay—reshape streaming logs in real-time into custom metrics, business-specific traces, security alerts, and compliance reports using familiar SQL syntax. Unlike traditional APM and SIEM tools that force you into predefined structures, SQL Operators with Apache Flink give you unlimited flexibility to create exactly the observability insights your business needs. Perfect for gaming companies tracking multiplayer sessions, security teams detecting advanced threats, and any organization wanting observability that matches their actual business logic. AI-assisted query generation makes complex stream processing accessible to all skill levels.

Discover how Grepr acts as an intelligent pipeline, leveraging AI to detect patterns, summarize noisy data, and directly pass unique messages, which can lead to up to a 90% reduction in data volume and substantial savings on Datadog platform costs. The article walks you through the setup process, from creating necessary integrations and configuring data lakes for low-cost storage, to building pipelines that process and route your data. It also covers how to adjust your Datadog Agent configurations and addresses strategies to mitigate potential data skewing from summarization, ensuring you maintain full insight into your applications without discarding any valuable data.

Grepr is an intelligent observability pipeline that optimizes, analyzes, and routes data in real time, sitting between your agents and observability platform. Utilizing machine learning and a rules engine, it efficiently detects data patterns, filters out repetitive information, and forwards only essential summaries or unique messages. This seamless integration helps organizations significantly cut observability costs by up to 90%, enable long-term data retention, and make valuable insights available for business reporting and AI, all with minimal configuration changes.

Grepr uses unsupervised machine learning to reduce log volume by over 90% while preserving important data through smart, configurable aggregation. It passes low-frequency messages through unmodified, allows engineers to retain specific parameters like user IDs, and supports backfilling logs via API triggers when deeper detail is needed—such as during support tickets. For added flexibility, trace sampling can capture full logs for a subset of users, and all original logs are archived in a searchable data lake. This gives teams control, reduces noise, and enables cost-effective observability without sacrificing access to critical information.

With apologies to George Orwell. Not all Observability data is salient all the time, some data is required all the time but most data is only germane when investigating an issue.

Vector and Grepr both function as observability data pipelines, but they differ sharply in complexity and automation. Vector, an open-source tool sponsored by Datadog, is powerful and flexible but requires extensive manual configuration, domain-specific scripting (VRL), and careful infrastructure planning. In contrast, Grepr is a fully automated, AI-driven observability platform that dynamically manages thousands of data transformations without requiring custom coding. It reduces observability costs by up to 90%, stores all data in queryable formats like Apache Iceberg on AWS S3, and integrates seamlessly with tools like Datadog and Splunk. With Grepr, organizations can deploy in minutes instead of days—without the operational overhead.

Modern web applications are rich, dynamic, and heavily reliant on frontend frameworks like React and Vue, which makes browser-side logging essential for understanding both code execution and user behavior. The Datadog browser logs SDK allows developers to collect this data, but with high traffic, logging can become expensive due to Datadog’s volume-based pricing. Grepr solves this by acting as an intelligent intermediary: it receives all logs, stores them cost-effectively, and uses AI-powered filtering to reduce the volume sent to Datadog by 90%—without dropping any data. It aggregates and summarizes repetitive logs, maintains full fidelity through semantic understanding, and even retains query access to all original data via a dashboard using the same syntax as Datadog. This approach allows developers to maintain 100% insight with only 10% of the data volume and cost, enabling full visibility into user behavior and app performance without budget concerns.

This blog post shows how to reduce log volume by up to 90% by integrating New Relic with Grepr. Using a simple Docker-based microservices demo, we walk through configuring Fluent Bit to ship logs to New Relic, then show how easily Grepr can be inserted into the pipeline to intelligently filter out noise. The result is cleaner, more actionable log data, reduced observability spend, and no disruption to existing workflows. All raw data is retained in low-cost storage and can be backfilled on demand—helping teams stay in control of both their visibility and their budget.

Goldsky, a Web3 realtime data platform, partnered with Grepr to significantly reduce their log management costs while maintaining observability performance. Initially facing misalignment between logging spend and value, Goldsky deployed Grepr, led by Lead Engineer Paymahn Moghadasian, who quickly integrated it using Terraform and Datadog's dual-shipping feature. Over four weeks, they successfully filtered noisy logs and transitioned their production environment with zero disruption. The result: a 96% reduction in indexed logs, 93% less data ingested, and over 85% savings in Datadog costs—without any negative impact on Mean Time to Resolution (MTTR). Additional benefits included improved log readability, faster searches without rehydration, and white-glove support from Grepr.

Grepr and Cribl both offer data pipelines for observability, but they differ in complexity and approach. Cribl is a powerful, flexible platform requiring significant setup, ongoing management, and learning its custom query language. Grepr is the newer, simpler option, using AI to automate data filtering and reduce manual configuration by 90%. While Cribl offers more integrations, Grepr supports common sources, uses familiar query languages, and enables faster, lower-maintenance deployment. Cribl suits large enterprises with dedicated teams, while Grepr is ideal for organizations seeking a faster, more automated solution.

Grepr reduces observability costs by storing all data in low-cost storage and using machine learning to forward only unique or summarized insights to platforms like DataDog, Splunk, or New Relic. Engineers can query retained data, generate reports, power AI, or trigger dynamic backfill during incidents—automatically via webhooks or manually through the Grepr interface. To learn more or request a demo, visit grepr.ai.