What we learned at KubeCon: AI SREs are boring


We spent last week at KubeCon + CloudNativeCon North America in Salt Lake City. Between booth conversations, talks, and walking the floor, a few patterns stood out. None of them is surprising on its own. Together they say a lot about where operations tooling is right now.
This KubeCon was quieter. According to event organizers, attendance dropped from ~9.5K last year to ~8K this year. The people who stopped at our booth tended to arrive with a specific problem and stay to talk it through. Fewer badge scans, longer conversations.
Less hype on the floor, more questions about what works.
Cost and telemetry noise came up in nearly every conversation, usually first. Teams told us the same story in different words. Telemetry volume grows faster than the business, and most of it is just noise. The bill grows with it. Teams have been forced to sample, drop, or shorten retention. But that creates new problems: The data these teams are forced to cut in order to save money is the data they need when something new breaks. It is also the data any AI system would need when trying to determine what caused an incident. So teams are forced to decide what to keep before they know what matters.
AI SRE vendors were everywhere, and their pitches were close to interchangeable. An alert fires. An agent pulls logs, metrics, traces, and recent deploys. It posts a root cause hypothesis in Slack. Some add a suggested fix or a runbook step.
The demos are good, but they’re all very similar. It’ll be interesting to see how many of these solutions are around next year.
Making matters more difficult for AI SRE vendors, many of the platform and SRE teams we talked to are building this themselves: an LLM, a connection to their observability tools, their runbooks, and Slack. These homegrown solutions likely aren’t as good as the off-the-shelf AI SREs, but for many we spoke with, they’re good enough.
What every one of these systems shares is where it starts. Nothing happens until an alert fires.
Alerts are written by people, for failures someone anticipated. That’s useful. But known failure modes rarely cause outages anymore, because teams have already written rules for them. It also means an alert is a lagging indicator. By the time it fires, the problem is in production and someone is usually already affected.
The incidents that hurt are the ones nobody predicted: a new failure that never throws an error, a field renamed upstream that quietly breaks dashboards, a token written to an access log. No rule exists for those, so no alert fires, so the AI SRE never starts.
When the alert does fire, the agent has to work backward from a symptom through a chaotic system. As we wrote in Why AI SRE is harder than AI coding, backtracking to a root cause is very difficult, and worse, didn’t prevent the incident in the first place.
Most incidents trace back to a change: a deploy, a config update, a dependency, a new code path. Changes show up in telemetry well before they show up as an alert. A new pattern appears. An existing one shifts. Most of these are harmless. A few are the first signs of an incident, or of a security problem.
Watching for those changes gives you a leading indicator. It doesn't require anyone to predict the failure. It does require two things the AI SRE pitches mostly skip. You have to keep the telemetry instead of dropping it to save money. And you need a scalable and affordable way to tell which changes deserve an agent's attention.
That's why the two biggest themes amongst the conversations we had with attendees, cost and reliability, are one problem. If telemetry is too expensive to keep, there is nothing to watch. If every event needs an LLM call, watching is too expensive too.
This is the approach we've taken with Grepr AI. The Signal Processor separates signal from noise in the live stream, sending raw telemetry to a low-cost data lake that makes collecting everything affordable. And agents run against the few, interesting signals vs. the entire telemetry stream. There's more detail in Introducing Grepr AI.
If you stopped by booth 291, thanks for the conversations. If you want to see what's changing in your own telemetry, you can try Grepr free.