7 Things I’ve Learned Building a Threat Hunter Agent Using Grepr’s Harness

Building an autonomous, proactive threat hunter sounds straightforward on paper: stream raw system and application logs in real-time, look for bad behavior, and alert the SOC when there's an active threat like a brute force login or data exfiltration.

In practice, applying AI to security telemetry in a live production environment is exceptionally messy.

Over the last few months, I’ve been using Grepr to build a proactive threat hunter agent that actually works at scale, without burying our team in noise. If you are trying to combine LLMs with observability data to detect active kill chains, here are 7 hard lessons I’ve learned along the way.

1. You can’t just "send it all to the LLM"

This is the most common trap. As I wrote in a previous blog post, you simply cannot pipe raw, voluminous telemetry data directly into an LLM. Not only will the token costs bankrupt you, but LLMs suffer from severe "lost in the middle" degradation. When you stuff a massive context window full of system logs, the model becomes incoherent. We realized early on that the LLM cannot be the first line of defense, it has to be the last.

2. Anomaly detection (UEBA) has to do the heavy lifting

Instead of using AI to read every log, you need a deterministic way to find the needle in the haystack. To find the signals that trigger an investigation, we rely on Grepr’s Autonomous Telemetry Pipeline’s pattern recognition.

By comparing real-time data against existing baselines for different cohorts (hosts, containers, processes, users), we can identify new log patterns, rate anomalies, and feature anomalies (like an unusual IP address or UUID). This provides a robust User Entity Behavior Analysis (UEBA) foundation without needing a single prompt or without spending months normalizing all log data.

3. Raw anomalies alone will drown you in false positives

While UEBA anomalies are well-qualified mathematically, relying on them as direct alerts is a mistake. An unexpected container restart or a misconfigured cron job will trigger an anomaly, but it isn't malicious. If you treat every statistical anomaly as evidence in a threat kill chain, your SOC will quickly succumb to alert fatigue. Anomalies are just signals; they are not inherently useful as evidence until they are qualified.

4. Agents need data lake access to contextualize the noise

To bridge the gap between "statistical anomaly" and "security event," you need contextualization. We take these raw signals and send them to Grepr AI, which has direct query access to a given environment’s log data lake.

By looking at the logs surrounding the anomaly, the agent can qualify what actually happened. For example, it can determine that a rate anomaly isn't just a traffic spike, but a targeted brute-force login attempt. Now, we finally have a qualified starting point for a threat hunt.

5. Threat-hunting agents will happily hallucinate evidence

Once we have a qualified anomaly, the next logical step is to pass it to a secondary threat-hunter agent and ask: Is this malicious, and is it part of a larger kill chain?

This is where agentic performance varies wildly. Left to its own devices, the LLM will often hallucinate timestamps, invent causal links between entities, or weave completely unrelated evidence (like those routine cron jobs) into a compelling, but entirely fictional, kill chain narrative.

6. Log analysis lacks the "compiler feedback loop"

Why do coding agents work so much better than log-analysis agents? Because coding agents have a compiler. If they write bad code, they run tests, see the error, and correct themselves.

LLMs analyzing voluminous log data in production have no objective system to check themselves against. If you give an agent open-ended tooling to query raw logs and ask it to present a conclusion, it has no way to verify if the narrative it just constructed actually reflects reality.

7. Constrain the agent with causal graphs and deterministic checkers

The biggest breakthrough came when we stopped giving the agent tools to query raw logs and started giving it graphs.

Instead of searching raw text, we provide the agent with an entity graph (hosts, containers, users) and a temporal causal graph (e.g., User A attempted to log into Host B at 10:00 AM). The signals point out the relevant entities and edges. The LLM’s job is simply to traverse this graph and build a proposed kill chain.

Crucially, we don't trust the LLM's output. The proposed kill chain is submitted to a deterministic checker that validates the entities, verifies the timestamps, and scores the chain based on the number of proposed causal links we actually have hard evidence for.

The takeaway

By combining deterministic anomaly detection with LLM graph traversal and rigorous validation scoring, we vastly improved our kill chain recall. More importantly, this architecture explicitly highlights the parts of the kill chain where the agent has low confidence, allowing human SOC analysts to step in exactly where they are needed.

AI is an incredible tool for connecting the dots, but it's terrible at finding them or verifying them. Keep your pipeline deterministic at the edges, and let the agent operate only in the middle.

Try it out: Design partnership is open

We’re actively seeking design partners to provide feedback on what we’re building. In exchange, you’ll get to run Grepr AI for free. If you’d like to join the program, reach out to us.

FAQ

Why can’t you just send raw logs to an LLM for threat hunting?

The token costs are unsustainable, and LLMs suffer badly from “lost in the middle” degradation. Stuff a context window full of system logs and the model becomes incoherent and misses the subtle signals you’re paying it to find. The LLM can’t be the first line of defense, it has to be the last.

If statistical anomalies aren’t alerts, how do you avoid drowning the SOC in false positives?

Anomalies are signals, not evidence. An unexpected container restart or a misconfigured cron job will trip an anomaly without being malicious. Grepr’s pipeline surfaces the anomalies through pattern recognition and baselines, then the agent queries the surrounding log data to qualify what actually happened, so only a genuinely suspicious event becomes the start of a threat hunt.

How do you stop the agent from hallucinating a kill chain?

Constrain it. Instead of letting the agent query raw text, you give it an entity graph and a temporal causal graph to traverse, and its job is just to propose a chain. That proposal then goes to a deterministic checker that validates the entities, verifies the timestamps, and scores the chain by how many links have hard evidence behind them. It also flags the low-confidence parts so a human analyst knows exactly where to step in.