9 Days from Kickoff to Production: How Envoy Cut Log Volume by 90%

Envoy: Redefining Workplace Experience

Envoy builds workplace technology that helps companies compliantly and securely manage visitors, contractors, deliveries, and hybrid work. The Envoy platform serves as an operational layer and ecosystem that keeps physical spaces — from manufacturing sites and data centers to life sciences labs to corporate headquarters —  running smoothly. Like the modern SaaS platforms it serves, Envoy's engineering infrastructure spans multiple services, generating the kind of telemetry volume that comes with running software at scale. And like most growing engineering organizations, Envoy eventually had to reckon with what that volume was costing them.

The Problem: Costs That Doubled Without Delivering More Value

When Ben Ede joined Envoy as Engineering Director, one of the first problems on his desk was observability spend that had nearly doubled. His team, including infrastructure engineers Raymond Hardy and Owen Cummings, dug into the data to understand where the money was going.

The answer was logs. And the deeper finding was uncomfortable: approximately 98% of the logs being ingested were never queried, never surfaced in a dashboard, and never used in a troubleshooting investigation. The team was paying full Datadog prices for data that was delivering almost no operational value.

The obvious solution, manually writing exclusion rules and sampling configurations to cut volume, had always been deferred. Not because it wasn't the right idea, but because the team couldn't answer a fundamental question with confidence: if 98% of logs were going unused, which 2% actually mattered? Without a reliable way to make that distinction, any manual filtering effort carried real risk. Write the wrong rule, and you might quietly eliminate the one data source that surfaces a production issue three months later.

So the costs kept climbing.

Searching For a Better Path

Envoy needed a way to reduce costs without pulling engineers off product work, without changing how the team used Datadog, and without creating a new category of risk around data loss. The self-service filtering approach wasn't viable. What they needed was something that could make the right decisions about which data mattered, automatically. “We knew the waste was there. What we didn't have was a safe way to act on it. We needed something that could make the right calls about which logs mattered without putting our engineers in the position of guessing,” said Ben Ede.

How Grepr Works for Envoy

Grepr sits between Envoy's telemetry sources and Datadog, processing log streams in real time. Rather than requiring engineers to define rules for what to keep or drop, Grepr's signal processing engine identifies log patterns dynamically, separating noisy logs from the low-volume, novel signals that actually matter: errors, anomalous application behavior, and unique events.

And Grepr helped Envoy avoid engineering toil by automatically parsing Envoy's existing Datadog dashboards and alerts, adding those data sources as pipeline exceptions. So any log feeding a live dashboard or alert continued flowing into Datadog without modification. No manual work required to protect existing workflows.

High-noise logs aren't dropped. They're summarized for Datadog and simultaneously written in full fidelity to Envoy's own S3-based data lake. Everything is retained, in the event of an investigation, but the volume sent to Datadog is reduced significantly.

When an anomaly is detected, Grepr can backfill the associated raw logs from S3 into Datadog automatically, pulling the hour before and after an event into full view for investigation, exactly when engineers need them, without paying to store everything in Datadog permanently.

Rollout and Results

Envoy moved from an initial engineering meeting on June 10th to a full production cutover on June 22nd, a span of 9 business days. The implementation followed three phases across that window: dev environment rollout and tuning, team onboarding, and production cutover. The pace was possible because Grepr required no changes to Envoy's existing Datadog configuration, no instrumentation updates, and no dashboard rebuilds. The engineering lift was low enough that the team could move fast without taking on meaningful risk.

And the operational model changed. What had previously been framed as a massive, risky engineering project became a straightforward configuration process. Engineers didn't notice a change in how they worked. Alerts still fired. Dashboards still populated. When teams needed to go deep on an investigation, the raw data was in S3.

As Ben noted, “Engineers didn't change how they work at all. Dashboards and alerts still worked as expected. We just stopped paying for 90% of our log volume that was never doing anything for us.”

"Nine days from kickoff to production is not a typical enterprise software story," Ben added. "We expected this to take months. The fact that it didn't is as much a part of the value as the cost reduction itself."

The team went from paying for data nobody used to paying only for data that matters.

What's Next

Envoy is exploring Grepr's trace reduction capabilities, which have already delivered 97% span reduction for other customers. They’re also planning to join the early access program for Grepr’s Proactive AI SRE Agent, so they can build agents that detect, prevent, and avert incidents in real-time.

Want to see what Grepr can do for your observability bill? Get started free and see results in under 30 minutes.

‍