Grepr Blog

Cloudflare generates numerous logs of different types, including HTTP request logs, firewall events, access logs, DNS query logs, etc. These logs contain plenty of helpful information that can provide insight into the health and performance of web applications. However, the profusion of data presents a challenge in extracting the useful signals from all the noise. The Grepr Intelligent Observability Data Engine can suppress the noise and provide a clear signal.

Kubernetes audit logs are extremely useful for tracking interactions with the API Server for debugging and providing insight into workloads. By default the audit logs are retained in etcd for only one hour. With the low cost storage of Grepr, much longer retention periods are possible for minimal cost and greater insight.

This blog post provides a comprehensive, step-by-step guide on how to seamlessly integrate the Grepr Intelligent Observability Data Engine with Splunk. It explains that with a few simple configuration changes, you can reroute your logs to Grepr, which uses machine learning to automatically detect and summarize frequent log patterns. This process can reduce your Splunk log volume and associated cloud costs by up to 90%, all without discarding any data. The post walks you through the entire setup, from configuring integrations for Splunk S2S or HEC to creating pipelines and datasets, ultimately demonstrating how to achieve significant cost savings while maintaining full diagnostic visibility.

In modern, complex software environments, unstructured logs can create chaos and make it difficult to gain insights. This blog post explains why structured logging, which captures log data in a consistent, machine-readable format like JSON, is an essential practice. By standardizing your logs, you can dramatically improve observability, ensure consistency across teams, and future-proof your systems. The post details how this approach facilitates faster troubleshooting, enables powerful automation, and turns your log data into a valuable source for metrics and analytics, ultimately transforming logs from simple text files into a critical source of truth for your applications.

Many IT teams face a difficult trade-off: managing the high costs of observability data while still maintaining full visibility into increasingly complex systems. This blog post introduces a solution to this problem, explaining how to achieve 100% visibility with just 10% of the data. It breaks down observability data into two tiers—essential "heartbeat" data and voluminous "diagnostic" data—and demonstrates how the Grepr Intelligent Observability Data Engine uses machine learning to summarize diagnostic logs, retaining all of the raw data in low-cost storage. This approach allows teams to dramatically reduce their ingestion costs, while still having the ability to backfill all of the relevant diagnostic data for troubleshooting incidents, ensuring no critical information is lost.

In the fast-paced world of data pipelines, making a mistake can have serious consequences. This blog introduces Grepr's new Live Edit feature, which allows you to safely test changes to your production pipelines. By creating a temporary, risk-free clone of your pipeline, you can add new parsers, exceptions, or other modifications and see the results in real time. This ensures you can validate changes and their impact on your data stream before committing, preventing errors and giving you the confidence to maintain your pipelines with ease.

Data backfilling is a powerful tool for troubleshooting, but doing it manually can slow you down when you're racing to resolve an issue. This blog explores how to automate the backfill process using the Grepr Intelligent Observability Data Engine. By configuring webhooks with popular monitoring tools like Splunk, Datadog, and New Relic, or by using Grepr’s built-in rule engine, you can automatically trigger a backfill job when an alert is fired. This provides a complete, unabridged dataset for the time period of an incident, giving you the full context you need to debug without manually running queries—saving you time and making your workflows more efficient.

Grepr is an intelligent observability data engine that uses pipelines to process log data from sources like Splunk, Datadog, and New Relic. It stores data in low-cost S3 buckets, extracts key information into a standard format, and then uses a series of advanced processing steps like masking, tokenizing, and machine learning-based clustering to reduce the volume of logs by up to 90%. Users can tune the engine's performance with a variety of settings, including a configurable aggregation time window and a logarithmic sampling strategy, to ensure that important troubleshooting information is preserved while noisy, repetitive logs are filtered out.

Is your observability bill growing faster than your engineering team can say "log volume"? You're not alone. FOSSA, a leader in software supply chain management, faced a similar challenge. Their reliance on Datadog, while providing essential visibility, was becoming a significant financial burden as their platform scaled. Instead of a painful, time-consuming overhaul of their entire logging strategy, FOSSA found a smarter way. They discovered a solution that allowed them to dramatically reduce their Datadog costs without sacrificing the crucial insights they needed to monitor and troubleshoot their systems. Want to know how FOSSA achieved a whopping 95% reduction in log volume and kept their observability costs in check? Click to read the full story and discover their secret!

Observability tools are vital for troubleshooting, but their high operational cost, driven by data volume, creates a tension between DevOps teams needing more data and businesses seeking lower bills. This dilemma stems from platforms treating all data as equally important, leading to an "impossible situation." Grepr breaks this conundrum by acting as a shim between log shippers and backends, using semantic machine learning to summarize frequent, noisy messages while passing critical, unique ones straight through. This innovative approach reduces log volume by 90-98% for significant cost savings, yet all data remains accessible in low-cost storage via the Grepr dashboard, REST API, and familiar query syntaxes (Splunk, Datadog, New Relic). This ensures that while you pay only for the 2-10% of data actively used, the rest is available on demand for queries or backfilling during incident investigations, solving the operational versus cost challenge and allowing you to pay only for the data you truly need, when you need it.

Grepr is your ultimate solution for tackling high log management costs without sacrificing crucial insights. Our semantic machine learning technology intelligently sifts through your log data, automatically identifying and summarizing common, noisy messages while ensuring unique, critical events pass straight through. This means you can reduce your log volume sent to backend platforms like Splunk, Datadog, or New Relic by up to 90%, drastically cutting your observability expenses. Plus, with all data retained in low-cost storage and accessible via your preferred query syntax, you maintain 100% troubleshooting capability. Optimize your logs, cut your costs, and keep all your valuable data with Grepr—it's a win-win for your operations and your budget.

This blog post introduces a revolutionary approach to observability, addressing the long-standing "AI-in-a-Haystack" problem in log analysis. Traditional methods struggle with the sheer volume and lack of context in modern telemetry data, making AI analysis financially and technically unfeasible. Grepr offers a unique solution built on three core principles: intelligent telemetry reduction, which de-noises log volumes by over 99% before storage; a stateful stream processing engine, providing AI with the necessary memory and context to understand data trends; and dynamic pipeline control, enabling the AI to reconfigure data streams on the fly to "zoom in" on specific issues. These capabilities transform monitoring from a reactive chore into a proactive, conversational partnership, allowing AI to intelligently flag issues, suggest causes, and dynamically adjust its focus, ultimately leading to faster incident resolution and more efficient operations.