We Pointed the Grepr Agent at Our Own Cluster. Here's What It Found.

We recently turned on our new Grepr Agent in a live cluster (one our own engineering team ships to every day), connected it to the signals coming off our telemetry pipeline, and went back to work. And almost immediately, the Grepr Agent began sending us relevant findings, helping us avoid incidents and customer escalations.

For the unacquainted: the Grepr Agent is a semantic anomaly detection capability (and much more!) within Grepr's Autonomous Operations Platform. It reads the signals Grepr's Autonomous Telemetry Pipeline surfaces, performs semantic analysis on the interesting ones (instead of relying on static rules), and decides what to do next. It has access to provided tools over MCP and integrations like Slack and PagerDuty for escalation. We've written about the thinking behind it in From Reactive Alerts to Proactive Prevention.

The loop is simple: a new pattern becomes a signal, the signal wakes the agent, the agent reads the offending logs in context, works out what's actually wrong, checks its own reasoning against the evidence, and posts the triage to Slack. Notice what's missing: we didn't write a single alert rule, tune a threshold, or carry a pager. 

Every AI SRE agent on the market is only as good as its trigger. Most start from a human-built alert, which means they only work on the failures someone already predicted. The Grepr Agent's trigger is its own Autonomous Telemetry Pipeline, watching every signal in the stream all the time. That's how it caught a string of problems we could never have written alerts for, because we didn't know to look for them in the first place: the unknown unknowns.

In our case, reports started arriving almost immediately, ranging from ops incidents to code-level bugs. Here's a sample of what it filed over the course of a week.

Ops saves

• A leaked license key, caught in minutes. Leaded credentials got echoed into pod stdout from an interactive shell session in a scratch pod. Not a known error signature: a brand-new pattern that shouldn't have existed at all. The agent flagged it and prescribed the fix before anyone went looking.
• A query bug traced to the line. It traced a set of failing Lucene queries to an unsupported range search: not "something is erroring," but the class, the method, and the line.
• A scheduler failure root-caused to the exact fix. A class hadn't implemented the job interface its store required. The agent identified the root cause and the exact orphaned trigger row that needed to be cleaned up.
• One failing sync job, two unrelated root causes. It untangled a single failing integration into two distinct problems: a vendor API key missing monitors_read permissions, and a second integration with no exception parser bound. It shipped a fix for each.
• 502s isolated to a single failed handshake. It traced a run of mysterious 502s to a failed MCP handshake with an external endpoint, after first ruling out our other integrations by confirming they were returning clean 200s. The blast radius arrived pre-scoped.
• Query translation failures. It identified an error in how we were parsing certain queries, then opened a PR to fix them before customers saw them.

Infrastructure saves

The cluster problems nobody was watching for.

• It caught our self-hosted CI runners crashlooping every ~25 seconds on a GitHub 404 and correctly called it as a revoked credential, before anyone noticed CI was down.
• It flagged hundreds of errors from the EBS volume snapshotter, diagnosed the missing VolumeSnapshot CRDs behind them, and offered a clean fix: install the CRDs, or drop the sidecar.

Knowing when to say nothing

An agent that escalates everything is just a louder alert pipeline. Judgment (knowing what not to page on) is what makes the rest of this trustworthy.

• It reviewed 110 handler-timeout errors from the component our autoscaling depends on, independently verified that every real metrics query was still succeeding in roughly 15 milliseconds, and closed the whole batch as noise. No incident, no page, no false alarm.
• It kept filtering all week: certificate-rotation chatter mislabeled as errors, cached API discovery responses, log-tailer cleanup from short-lived pods. Each one triaged, labeled benign, with the reasoning attached, and kept off our plate.

What we learned

1. Pattern novelty is a remarkably good trigger. Every investigation above started from a log pattern the pipeline hadn't seen before, not a threshold, not a rule someone remembered to write. Zero alert rules configured for any of it.
2. A trigger is only as good as its false-positive rate. An agent nobody trusts gets muted within a week. What made this run different is that every triage came with the reasoning attached, so we could check its work, not just its conclusion.
3. Restraint builds trust. The value here wasn't just what the agent flagged. It was that "benign, no action required" showed up as often as it did, with the evidence to back it up. That's the difference between an agent making the case for a conclusion and an agent asking us to trust a black box.
4. It is efficient. While not free, it might as well have been. Running the Grepr Agent cost us about $300/week, but it saved us approximately 40 engineering-hours and eliminated the potential of incidents and customer escalations. I’d call that excellent value.

We run the Grepr Agent in our own cluster today, on the same Autonomous Operations Platform we ship to customers: the Autonomous Telemetry Pipeline that separates signal from noise, and the agent that reasons over the interesting signals. If you want to see what it finds in yours, talk to us.

‍
FAQ
‍
‍

‍What is the Grepr Agent?
The Grepr Agent is a semantic anomaly detection capability within Grepr's Autonomous Operations Platform. It reads the signals surfaced by Grepr's Autonomous Telemetry Pipeline, performs semantic analysis on the interesting ones instead of matching static rules, and decides what to do next, with access to tools over MCP and integrations like Slack and PagerDuty for escalation.

How is the Grepr Agent different from other AI SRE agents?
Most AI SRE agents start from a human-built alert, so they only catch failures someone already predicted. The Grepr Agent is triggered by its own telemetry pipeline watching every signal in the stream, so it catches the unknown unknowns, problems no one wrote an alert for.

Do I have to write alert rules or tune thresholds to use it?
No. In this run, zero alert rules were configured. The trigger is pattern novelty: when the pipeline sees a log pattern it has not seen before, that wakes the agent to investigate.

What kinds of problems did it actually catch?
In one week on our own cluster it flagged a leaked license key echoed into pod stdout, traced failing Lucene queries to an unsupported range search, root-caused a scheduler failure, split one failing sync job into two unrelated causes, isolated a run of 502s to a failed MCP handshake, caught self-hosted CI runners crashlooping on a revoked credential, and diagnosed missing VolumeSnapshot CRDs behind hundreds of EBS snapshotter errors.

Does it escalate everything?
No, and that is the point. It closed 110 handler-timeout errors as noise after verifying the real queries were still succeeding, and it filtered a week of benign chatter like certificate rotation and log-tailer cleanup. Each triage came with its reasoning attached, so you can check its work rather than trust a black box.

How much does it cost to run?
In our cluster the Grepr Agent cost about $300 a week and saved roughly 40 engineering-hours, on top of the incidents and customer escalations it helped prevent.

How does the Grepr Agent get its signals?
It sits on top of Grepr's Autonomous Telemetry Pipeline, which separates signal from noise in your telemetry stream. The agent reasons over the interesting signals the pipeline surfaces, so detection is driven by what is actually happening in your systems, not by predefined rules.

‍

‍