Back
Author
John Withers
Head of Marketing
LAST UPDATED
November 19, 2026
Description
Point an AI SRE agent at your production logs and it burns its budget reading health checks long before it finds the thing that broke.
Engineering Guides

Pre-processing Telemetry for AI SRE Agents

Blog post featured image
IN THIS ARTICLE
SHARE
Keep up with Grepr
Subscribe now for best practices, research reports, and more..

Every AI SRE demo looks the same. An alert fires, the agent pulls the logs, and 40 seconds later it names the root cause. Then you point it at your own production environment, and it spends ten minutes reading health checks before giving up.

The model isn't the problem. The input is. Production telemetry was built for dashboards and grep, not for an LLM that pays per token and believes everything it reads. If you're putting an AI SRE agent on call, the most important engineering work isn't the prompt. It's what happens to your logs and traces before the agent sees them.

That pre-processing is the difference between an agent that triages like a senior engineer, and one that burns budget like an intern with tail -f.

The firehose problem 

Raw telemetry fails agents because it fundamentally doesn't fit. A busy service writes millions of log lines an hour. Even with a massive context window, an agent can only hold a few minutes of that data. It ends up sampling—and sampling inevitably picks up the 99% of your logs that are just routine health checks, retries, and noise.

Worse, feeding raw logs to an agent is expensive and dangerous. Every line costs tokens, and every unredacted string is an open channel for prompt injection or leaking PII (like emails or session tokens) directly to a model provider.

A human SRE handles all of this by instinct: they skim past the noise, and they know not to paste a customer's password into ChatGPT. An agent needs that judgment applied for it, upstream.

Collapsing noise into signal 

You can't just feed an agent raw strings; you have to feed it patterns.

Pre-processing needs to happen in the telemetry pipeline, in-stream, before the data is stored or served. First, you mask PII and rewrite sensitive attributes to fixed values so they never reach the prompt. Then, you parse the free-text lines into filterable fields and group them.

Instead of sending an agent 12,000 individual copies of the same timeout error, you collapse them into a single pattern with a count and a novelty flag. The agent shouldn't start by reading logs; it should start by looking at what has changed against the baseline.

Instead of overwhelming the context window, a pre-processing pipeline splits the data stream. The agent only reads compressed novelty patterns, while raw events are kept in a low-cost data lake for drill-down.

Before and After: What the agent actually reads 

Here are five minutes of a checkout service. In the raw version, the real signals are buried under thousands of health checks and expose customer emails.


Raw:

14:02:11 INFO  GET /health 200 2ms
14:02:11 INFO  GET /health 200 3ms
14:02:12 WARN  payment timeout user=jane.doe@example.com order=88123 after 5000ms
14:02:12 INFO  GET /health 200 2ms
14:02:13 WARN  payment timeout user=r.patel@example.com order=88124 after 5000ms
... 40,000 more lines


Pre-processed JSON payload for the Agent:

[
  {
    "pattern_id": "chk_882a",
    "template": "payment timeout user=<redacted> order=<*> after 5000ms",
    "occurrences": 1912,
    "first_seen": "14:02:12",
    "status": "novel_spike"
  },
  {
    "pattern_id": "db_019b",
    "template": "db pool exhausted service=payments",
    "occurrences": 340,
    "first_seen": "14:04:01",
    "status": "novel_spike"
  }
]


The pre-processed version puts the new problems at the top, removes the PII, and keeps the raw evidence safely stored in a low-cost data lake, just one query away if the agent needs to drill down.

Proof from our own Prod environment

At Grepr, our Autonomous Telemetry Pipeline handles this in-stream reduction automatically. But we wanted to see how much of a difference this made for an AI, so we ran the Grepr Agent on our own production cluster for a week.

Instead of reading a raw firehose of 10+ TBs of data, the agent only reasoned over the compressed novelty patterns. Because of this, it successfully:

  • Caught a leaked license key
  • Diagnosed a crashlooping CI runner
  • Traced query parsing bugs to specific lines
  • Correctly filtered 110 timeouts as benign

Because we didn't waste tokens on health checks and noise, the agent cost us about $300 in API usage for the entire week and saved roughly 40 engineering hours of manual triage.

Zero vendor lock-in, 90% cost reduction

The best part is that solving this data problem not only allows you to keep your existing observability stack in place, but Grepr also reduces your observability noise and observability platform cost by 90%. By placing Grepr in front of Datadog, Splunk, New Relic, and other observability platforms, you only send the roughly 10% signal to those tools while simultaneously giving your AI agents exactly what they need to identify problems at scale.

The bottom line

‍An AI SRE agent can't be smarter than its input. Feed it raw telemetry and it reads health checks, leaks data, and runs up a massive bill. Feed it redacted, structured, pattern-level telemetry with novelty flags, and it starts exactly where a senior engineer would: at the thing that changed.

Pre-processing isn't just extra plumbing for the AI era. It's the prerequisite that makes the agent worth running.

Ready to give your agents eyes on production, safely?
Get started free or schedule a demo to see how Grepr works.

Ready to reduce your observability TCO by 75%?
SHARE
Keep up with Grepr
Subscribe now for best practices, research reports, and more..