Webinar Recap: Custom Ink cut their CloudWatch bill 85% without dropping a single log

Custom Ink had a CloudWatch bill that kept climbing and a hard rule about how they could bring it down: they would not drop or sample a single log. With SOC 2 and PCI requirements, and engineers who needed those logs to investigate production issues, losing data to save money was not an option.

Custom Ink cut their CloudWatch costs by about 85% anyway, from roughly $100K to $15K a year. In a recent webinar, Grepr's CEO Jad Naous sat down with Will Spencer, DevOps Manager at Custom Ink, and Vít Šesták, a DevOps engineer on his team, to walk through how they did it. A link to the full recording is at the bottom.

The one thing they refused to do

Custom Ink had already done the obvious things. They trimmed log formats and moved from big JSON documents to smaller lines, the usual minimization, and it kept growing. What they would not do was drop or sample logs, and Will was clear about why. "If you start dropping 25% of your logs to try and reduce that cost, if the thing that's happening that you're concerned about is a part of that 25%, well then you never see it." With SOC 2 and PCI requirements on top of that, the logs had to stay.

Jad framed the bigger pattern earlier in the session. Over the last decade, microservices multiplied, every layer of every service started emitting telemetry, and the volume of telemetry data has exploded. Meanwhile the value of any single log line kept falling. Most of it, as Jad put it, "ends up kind of being like insurance that maybe you're going to need at some point in time." Teams spend more and more on observability without getting more reliability for it.

The skeptic and the rollout

Will has sat through a lot of sales pitches. "I had a healthy sense of skepticism, because it's rare that you find situations where people are delivering pretty much exactly what they say they're gonna deliver." Vít, who first ran into Grepr at KubeCon in Europe, had a similar first reaction: “is it really going to be that good?”

The rollout answered the question quickly. Custom Ink pointed Fluent Bit at Grepr instead of CloudWatch, started in their dev environment, and moved to staging. Rolling out to their production environment took only a couple more days, where they saw an 80% log reduction out of the box. After some fine-tuning within Grepr, their noise reduction climbed into the 85% to 90% range. Vít was caught off guard by how much came for free: "Without any configuration on our side, I was surprised that we were able to achieve such a large reduction of logs out of the box."

What the developers noticed

In the best possible way, the Grepr rollout didn’t even register with Custom Ink’s software engineers. "They didn't see any change whatsoever," Will said, and when Jad asked whether anyone had complained, Will didn't have much to report: "I would like to tell you some horror stories, but unfortunately I don't have any."

Eventually, engineers did provide one piece of positive feedback: When the team showed their engineering managers how they search for logs using the Grepr UI, the developers emphasized that they preferred Grepr’s search interface over that of CloudWatch. Vít said the interface felt more user-friendly, and the query syntax was a welcome change.

And Will was happy that Grepr provides Okta SSO integration out of the box, simplifying the process of onboarding and offboarding engineers as part of the standard workflow rather than being a separate chore.

Raw telemetry data stays in their own S3 bucket

Like most companies, maintaining compliance is a non-negotiable for Custom Ink. The fact that Grepr sends Custom Ink’s raw telemetry data to their own S3 bucket made adopting Grepr easy.. They get to make their own choices around retention, tiering, and archive policies. The full-fidelity data is there for an audit or an investigation, and if they ever need it back in CloudWatch, they can backfill it into their observability platform on demand, either manually or automatically.

Jad’s walkthrough of Grepr: how the reduction worked

Jad's demo showed how the reduction works. Grepr assigns a pattern to every message, passes through a few samples of each pattern, and summarizes the noisy repetition, which he describes as “passing through the needles and compressing the hay.” Because the raw data is always in a customer’s own data lake, nothing is lost. That said, Jad noted that most customers only care to analyze the interesting signals that Grepr sends to their observability platform, and they almost never query raw data from within their S3 bucket. But the option is always there.

The advice that keeps coming up

When Jad asked Will what advice he would give someone who’s considering adopting Grepr, he said, "Start involving the people that deal with your contracts early enough, because that's probably gonna take longer than it will take to get Grepr through a POC." Jad pointed out it was nearly word for word what Envoy’s Engineering Director, Ben Ede, had said in a previous webinar: the proof of concept finishes faster than you expect, so procurement ends up being the long pole.

Watch the full session

Will and Vít go deeper on their architecture, the internal buy-in, and the audience Q&A, and Jad demos log summarization, trace signature sampling, backfill from the data lake, and the new agent sessions capability. Watch the full recording to see it end to end.

‍

Frequently Asked Questions

How much did Custom Ink reduce their CloudWatch costs?

‍Custom Ink cut their AWS CloudWatch costs by about 85%, from roughly $100K to $15K a year by using Grepr. They saw around 80% log reduction on day one before any tuning, and reached the 85 to 90% range in production after fine-tuning their Grepr pipelines.

Did Custom Ink have to drop or sample logs to cut the bill?

‍No. Dropping or sampling logs was off the table, because losing 25% of logs could mean losing the exact log they needed during an investigation, and they had SOC 2 and PCI requirements to meet. Grepr reduces the volume by summarizing repetitive log patterns while keeping every raw message, so nothing is thrown away.

How does Custom Ink stay compliant with the logs reduced?

‍Grepr writes every raw log message to Custom Ink's own S3 bucket, which they own and control. They set their own retention and storage tiers, keep full-fidelity data for audit and investigation, and can backfill it into CloudWatch on demand. Grepr does not charge for that storage.

How is this different from CloudWatch subscription or metric filters?

‍Filters are static rules that drop data once configured, and once data is dropped it is gone. Log patterns also change constantly, so filters need continual rework. Grepr operates on millions of patterns dynamically, passes through low-volume and unique messages, and only compresses high-frequency noise, so you keep the rare logs that matter.