Logging Paradox

The information contained in application service logs fulfills two roles: it provides a heartbeat to notify that the service is alive and transacting, and when an incident occurs, it provides essential diagnostics to aid in restoring service. Most of the time, most of the data is not required. Only a very small percentage of the time is all the data required. However, organizations spend a significant part of their IT budget on ingesting and indexing all the data all the time, even though they know that most of it will not be needed. The crux of the problem is that there is no way of knowing which data will be needed when, so the Fear Of Missing Out (FOMO) takes hold and everything is indexed until the financial pain is too great.

Stuck Between A Rock And A Hard Place

This is where most organizations find themselves stuck. The financial burden of ingesting and indexing a massive volume of log data pulls against the cost and effort of re-tooling and/or manually managing log volume reduction. This results in many organizations being stuck with the devil they know and continuing to spend a significant portion of their IT budget on log management.

Time Is The Enemy

It’s not just the volume of log data; most organizations also have regulatory requirements to retain data for a set time period. Holding log data long term in high-cost storage just continues to pile on the financial pressure.

Murphy’s Law

The Fear Of Missing Out is real. Murphy’s Law would tell us that the information organizations choose to drop as part of their log reduction strategy is exactly the information required to resolve a serious incident and maintain service-level objectives.

Grepr Is The Silver Bullet

The Grepr Intelligent Observability Data Engine uses machine learning to continuously identify repetition patterns in the log data stream. Frequently occurring messages are summarized, while unique messages are passed straight through. To put it another way, the frequent heartbeat messages are summarized, and the unique error messages are passed through. This approach results in 90% or greater reduction in log volume with the associated reduction in costs.

‍

Implementation is simple, requiring only a small configuration change to redirect the log shippers output to Grepr, where the messages are reduced before being forwarded to the original platform. There is no change to existing workflows and dashboards; DevOps teams continue to work as they did before. Retaining 100% visibility with 10% of data.

‍

No data is dropped. All data sent to Grepr is retained in low-cost storage, significantly reducing the cost of retaining data to meet regulatory requirements.

‍

All the data retained in the low-cost storage is accessible in the event of an incident; Murphy’s Law does not apply here. The low-cost storage can be queried manually via the Grepr web dashboard and/or automatically backfilled to the logging platform when triggered by an alert.

Give It A Spin

Take control of your logs with minimal effort.

“20 minutes to set up, immediately saw a 95% drop” - Goldsky

The Grepr platform is SOC Type 2 and HIPAA compliant, as well as being recognized by Gartner as a Cool Vendor.

Sign up for a free trial today.

‍