AI Agent Telemetry: Seven Questions to Ask of Every Agent Run

‍

Your agents get fixed when someone notices that they are broken.

Auser complains and an issue gets assigned. Someone prompts an agent to go look at the errors. A cost dashboard looks wrong. An eval fails on a case that someone wrote after the last incident. All of them work, and none of them scale. Each one finds only what a person thought to look for, and each one ends when the fix ships. Nothing checks whether the fix held, and nothing finds the problems that nobody reported: the tool that fails one call in ten, the run that sat in a queue for an hour, the alert that started the same run forty times.

So what's missing? A loop: the same questions, asked of every run, on a schedule, with each fix measured by the question that found it. Your agents already emit almost all the input the loop needs: their telemetry.

Agents are services

A run is one execution of an agent, from the event that triggers it to its result. In between are model calls, tool calls, and their results. Each one carries a timestamp and a status. Your services emit the same kind of telemetry, and you already know what to do with it: measure in production, fix what the numbers show, measure again.

Agents need the loop even more than other services do. The same input produces a different trace on every run, so one good run after a fix proves little. Every run costs money, whether or not anyone needed it. And agents fail quietly. A run that gives up early ends the same way as a run that looked hard and found nothing. An agent with a broken tool works around it and still finishes, only slower and at a higher cost. Nothing crashes, so nobody files an issue.

What to collect

Record one event for each thing that happened: the trigger, every model call with its token counts, every tool call and its result, the end of the run, and what happened to the result afterwards.

Five details matter most:

1. Record when the run was queued separately from when it started.
2. Put an explicit error flag on every tool result.
3. Count cached input tokens separately from uncached ones.
4. Record the outcome: whether the result was accepted, corrected, retried, or rejected, even if that signal arrives days later.
5. Flag each event that contains a secret, a credential, or personal data, and record which kind.

If you already use OpenTelemetry, the GenAI semantic conventions cover most of this. A tool call event might look like this:

{  "timestamp": "2026-11-10T14:03:22Z",  "run.id": "run_8f2c",  "run.trigger": "new_log_pattern",  "gen_ai.operation.name": "execute_tool",  "gen_ai.tool.name": "search_tickets",  "tool.error": true,  "error.type": "schema_validation"}

Keep every event. The answers are in the tail, and sampling drops the tail. You don't need a hot index to do it. Every question below is a group-by that you ask once a day, and a data lake can run a group-by over everything you kept for far less than an index costs. That's why Grepr keeps full-fidelity telemetry in your own storage, in open formats, instead of sampling it away.

Keep the events with the rest of your telemetry, too, not in a separate agent tool. The cause of a bad run is often outside the agent: the tool failed because a service deployed, or the run waited because a queue filled. You only see that when both sides are in one place.

Seven questions to ask

1. What fails? Group failed runs by their final error, and note when each group first and last appeared. The top of the list tends to be a few causes outside the model: a provider error treated as final, a deploy window, a null field. Those get fixed in the code around the model, not in the prompt.

2. What is slow? Split each run into queue wait, model time, and tool time. The largest part tells you where to work. If it is wait, no model change will help: you need more capacity, or fewer runs in the queue. If it is tool time, find the tool that blocks.

3. What should not have started? Count runs by trigger, and ask whether each bucket deserved a run. An agent that fires on events inherits the noise in those events, and the cheapest run is the one that never starts.

4. What does it cost? Sum the tokens per run from the model calls, and price cached input at its own rate. An agent that takes 20 steps with a 10,000-token prompt reads that prompt 20 times: 200,000 input tokens on the prompt alone, before the conversation history that grows with every step. Caching lowers the price of those reads, not the count. A shorter prompt and fewer wasted runs are the two levers.

5. Which tools break? Compute the error rate per tool, then read the most common error. A tool that fails one call in ten with a schema error has a bug, and the agent pays tokens to work around it on every run. In Grepr, this is a SQL job over the dataset that holds the events, and the job sets the time window:

SELECT  MAX(receivedtimestamp) AS `receivedTimestamp`,  MAX(eventtimestamp) AS `eventTimestamp`,  PARSE_JSON(JSON_OBJECT(    'tool' VALUE VARIANT_VALUE(attributes, '$.gen_ai.tool.name', 'STRING'),    'calls' VALUE COUNT(*),    'error_rate' VALUE AVG(CASE WHEN VARIANT_VALUE(attributes, '$.tool.error', 'BOOLEAN') THEN 1.0 ELSE 0.0 END)  )) AS `data`FROM logsWHERE VARIANT_VALUE(attributes, '$.gen_ai.operation.name', 'STRING') = 'execute_tool'GROUP BY VARIANT_VALUE(attributes, '$.gen_ai.tool.name', 'STRING')

6. Was the result used? The first five questions can't see a run that finished on time with a wrong answer. The outcome signal can. A trigger with many runs and few accepted results is the best target you have.

7. What did it leak? Agents read tool output and write it back into model calls, transcripts, and tickets. Count secrets, credentials, and personal data in agent events by tool and by trigger. One API key in a tool result can travel into every system the transcript reaches, and nobody files an issue for that either.

Where the telemetry lies

The loop only works if the numbers are right, and agent telemetry is wrong in predictable ways.

• Keyword counts over-report. Searching tool output for "error" finds every line the agent read that contains the word. Count the error flag instead.
• Finished is not succeeded. A workflow engine or job queue reports that its own work completed, even when the agent inside it failed. Record the run's own status yourself.
• Created-at is queue time. A duration measured from it includes the wait, which on a busy day can be most of the total.
• "Input tokens" depends on the provider. Some providers include cached reads in the count and some report them separately, so a single token field can be off by ten times on a cache-heavy agent. If an agent reads 90% of its input from cache, a field that leaves out cached reads shows a tenth of the real count.
• Billed output with nothing recorded. If a model call reports output tokens and no message follows, the harness lost the answer. No transcript will show it.

Fix one thing, then ask again

Take the top line of one question, change one thing, and run the same query after the next deploy. The query is the test.

Fix upstream first. A bad trigger becomes a filter, a flaky provider becomes a retry policy, a broken tool gets repaired. Each one removes work from the agent instead of asking it to cope. Touch the prompt last, because a prompt change moves every run a little and is the hardest to measure.

Let an agent run the loop

You may already prompt an agent to go look at errors. The loop is that same agent with three changes: it runs on a schedule, it asks the same seven questions every time, and it compares the answers with the last pass.

Give it read-only access to the events and permission to open a ticket. Each day it runs the seven questions, picks one line that is new or growing, reads a few of those runs to find the cause, and proposes a fix along with the query that will prove it worked. A person approves the change. After the next deploy, the same query shows whether it worked.

Each pass makes the next one easier. Fewer bad triggers mean fewer runs to pay for and fewer traces to question. A clean failure list means a new failure stands out on the day it appears.

Collect. Ask. Fix. Repeat.

Your agents already emit almost everything you need to improve them. Send it to the same place as the rest of your telemetry; any OpenTelemetry pipeline works. Grepr keeps every event at a cost that makes keeping everything practical, and lets you build agents that watch the stream for you. Ask the seven questions, fix the top answer, then ask again tomorrow.

Try Grepr free

FAQ

Does this replace evals?

No. Evals stop a known failure from coming back, but they only test the cases someone thought to write. Telemetry shows what production does, and the rejected and corrected runs are the eval cases worth writing.

Do I need a dedicated agent observability tool?

Not for this. A dedicated tool is good at showing one run, so keep it for debugging single runs. The questions here are aggregates across all runs, and the causes are often in other systems, so agent events are most useful next to your service logs, traces, and metrics.

‍