Application logs provide vital information to assist with finding the root cause of an issue when an incident occurs. Under these circumstances there is no such thing as too much logging; the more information the better. However, the greater the volume of log information the greater the cost to process and store it whether using your own servers or those from one of the many SaaS providers.

Some application frameworks support changing the log level without restarting, allowing for increased detail when an issue is detected. While this is better than nothing in most circumstances the vital piece of information was logged before the issue was detected and has already been lost.

Dynamic Detail When You Need It

‍

‍

Grepr slips in like a shim between the log shippers and the log processing and storage servers. All logs messages are retained in low cost storage then using machine learning and a rules engine only less frequent unique messages and summaries of the more frequent noisy messages are sent through. Using these advanced techniques results in a typical reduction of 90% to the volume of messages being processed and stored by the existing logging backend.

During normal operation, this level of information provides just the right level of detail for users: low-frequency messages that are usually important like errors or misbehaviors are passed straight through, so they’re searchable and easy to find while reading through a log stream. Noisy messages that repeat often are summarized so they don’t clog the log stream and make it harder to read.

Remember no log messages are dropped, all messages are retained in low cost storage. When an incident occurs any log messages pertinent to the incident can be selectively and quickly backfilled into the logging backend. Thus providing the engineers all the detailed information they need in the tools they are familiar with.

Grepr dynamic backfill is just like going back in time to increase log level detail before the issue happened, ensuring that all the information is captured for diagnosis.

Reduce Cost Without Reducing Logging

‍

‍

Utilising Grepr the majority of log messages are retained in low cost storage, significantly reducing the cost of processing and storing them. The compromise between the level of detail captured in log messages and the cost now swings firmly in favour of capturing more detailed information all the time. More detail available in log messages enables engineers to diagnose issues and resolve incidents quicker, freeing them to work on new features and fixes.

Give Time Travel A Try

Give Grepr a spin and see how easy it is to start saving 90% on your logging services cost with zero interruption to your existing workflows. Stop worrying about achieving that fine balance between logging visibility and cost. With Grepr dynamic backfill you can use detailed logging at low cost and debug with your current tools.

‍