Pre-processing Telemetry for AI SRE Agents


Every AI SRE demo looks the same. An alert fires, the agent pulls the logs, and 40 seconds later it names the root cause. Then you point it at your own production environment, and it spends ten minutes reading health checks before giving up.
The model isn't the problem. The input is. Production telemetry was built for dashboards and grep, not for an LLM that pays per token and believes everything it reads. If you're putting an AI SRE agent on call, the most important engineering work isn't the prompt. It's what happens to your logs and traces before the agent sees them.
That pre-processing is the difference between an agent that triages like a senior engineer, and one that burns budget like an intern with tail -f.
Raw telemetry fails agents because it fundamentally doesn't fit. A busy service writes millions of log lines an hour. Even with a massive context window, an agent can only hold a few minutes of that data. It ends up sampling—and sampling inevitably picks up the 99% of your logs that are just routine health checks, retries, and noise.
Worse, feeding raw logs to an agent is expensive and dangerous. Every line costs tokens, and every unredacted string is an open channel for prompt injection or leaking PII (like emails or session tokens) directly to a model provider.
A human SRE handles all of this by instinct: they skim past the noise, and they know not to paste a customer's password into ChatGPT. An agent needs that judgment applied for it, upstream.
You can't just feed an agent raw strings; you have to feed it patterns.
Pre-processing needs to happen in the telemetry pipeline, in-stream, before the data is stored or served. First, you mask PII and rewrite sensitive attributes to fixed values so they never reach the prompt. Then, you parse the free-text lines into filterable fields and group them.
Instead of sending an agent 12,000 individual copies of the same timeout error, you collapse them into a single pattern with a count and a novelty flag. The agent shouldn't start by reading logs; it should start by looking at what has changed against the baseline.

Here are five minutes of a checkout service. In the raw version, the real signals are buried under thousands of health checks and expose customer emails.
Raw:
14:02:11 INFO GET /health 200 2ms
14:02:11 INFO GET /health 200 3ms
14:02:12 WARN payment timeout user=jane.doe@example.com order=88123 after 5000ms
14:02:12 INFO GET /health 200 2ms
14:02:13 WARN payment timeout user=r.patel@example.com order=88124 after 5000ms
... 40,000 more lines
Pre-processed JSON payload for the Agent:
[
{
"pattern_id": "chk_882a",
"template": "payment timeout user=<redacted> order=<*> after 5000ms",
"occurrences": 1912,
"first_seen": "14:02:12",
"status": "novel_spike"
},
{
"pattern_id": "db_019b",
"template": "db pool exhausted service=payments",
"occurrences": 340,
"first_seen": "14:04:01",
"status": "novel_spike"
}
]
The pre-processed version puts the new problems at the top, removes the PII, and keeps the raw evidence safely stored in a low-cost data lake, just one query away if the agent needs to drill down.
At Grepr, our Autonomous Telemetry Pipeline handles this in-stream reduction automatically. But we wanted to see how much of a difference this made for an AI, so we ran the Grepr Agent on our own production cluster for a week.
Instead of reading a raw firehose of 10+ TBs of data, the agent only reasoned over the compressed novelty patterns. Because of this, it successfully:
Because we didn't waste tokens on health checks and noise, the agent cost us about $300 in API usage for the entire week and saved roughly 40 engineering hours of manual triage.
The best part is that solving this data problem not only allows you to keep your existing observability stack in place, but Grepr also reduces your observability noise and observability platform cost by 90%. By placing Grepr in front of Datadog, Splunk, New Relic, and other observability platforms, you only send the roughly 10% signal to those tools while simultaneously giving your AI agents exactly what they need to identify problems at scale.
An AI SRE agent can't be smarter than its input. Feed it raw telemetry and it reads health checks, leaks data, and runs up a massive bill. Feed it redacted, structured, pattern-level telemetry with novelty flags, and it starts exactly where a senior engineer would: at the thing that changed.
Pre-processing isn't just extra plumbing for the AI era. It's the prerequisite that makes the agent worth running.
Ready to give your agents eyes on production, safely? Get started free or schedule a demo to see how Grepr works.