Why AI Is Insufficient for Threat Hunting

AI coding agents are writing our code, and AI operators are trying to manage our clusters. So it’s no surprise that security and platform teams are asking: "Why can’t we just stream our logs to Claude and have it hunt for threats?"

AI might sound like the ultimate SOC automation: Skip writing complex detection rules, point a frontier LLM at your observability data lake, and ask it to find the bad actors.

But reality paints a very different picture. Treating an LLM as a raw log analysis engine breaks at scale. Whether you’re using Claude or any other model, below is why LLMs fall short for infrastructure threat hunting, and what you should be doing instead.

1. The token math doesn't work

Threat hunting requires sifting through an immense volume of telemetry - applications, firewalls, containers, and orchestration layers. In observability, we already talk about the crippling cost of indexing all this data in traditional platforms. Now imagine paying a per-token tax on that same firehose.

Log data is notoriously voluminous and repetitive. Sending raw infrastructure telemetry to an LLM API at scale isn't just inefficient, it's financially prohibitive. The ROI of finding a single anomaly is entirely wiped out by the compute cost of processing the 99.99% of logs that are benign.

2. Context collapse and the baseline problem

To get around the cost of the firehose, the logical next step is partitioning. Instead of sending everything, what if we partition the logs by thread - grouping by user, host, container, or process ID - and feed those discrete chunks to the LLM?

This approach still fails for two key reasons:

• The Context Window: Even partitioned threads in a modern microservices architecture generate enough volume to easily blow past standard context limits. When an LLM gets overwhelmed with data, its attention mechanism degrades. It loses coherence, "forgets" earlier events in the prompt, and misses the exact subtle anomalies you're paying it to find.
• The Baseline Blindspot: Threat hunting relies heavily on anomaly detection, which requires an implicit understanding of normalcy. An LLM doesn't have a historical baseline for what usual behavior looks like for your specific threads, hosts, or CI runners. Without that baseline, it can't accurately judge if a sudden spike in network traffic is a lateral movement attempt or just a scheduled Tuesday backup.

3. The summarization and external memory trap

At this point, you might argue that Claude doesn't need to read everything at once. It can use targeted queries, external memory, and compacted summaries instead of trying to keep everything in one massive prompt. Those are legitimate architectural solutions. But they introduce their own fatal flaws for threat hunting.

When you compact raw logs into summaries, you inevitably drop details. In security investigations, a seemingly irrelevant artifact, like an obscure port number or a generic user-agent string, often has an importance that only becomes apparent days later when a new piece of evidence emerges. If that detail was summarized away, the trail goes cold.

Furthermore, in an enterprise environment, those summaries themselves become so voluminous that searching across them just recreates the original big data problem. Ultimately, the LLM isn't actually doing the hunting. The surrounding system still has to do all the heavy lifting: maintaining the historical timeline, retrieving precise evidence, and preserving complex relationships across thousands of entities.

4. Hallucinated kill chains and misattribution

Let’s assume you completely solve the cost and context problems. You filter the logs down to a sufficiently small, highly suspicious dataset, and you ask the LLM to connect the dots and determine the kill chain. This is where relying solely on LLMs becomes actively dangerous to an investigation.

When an LLM attempts to reconstruct a complex kill chain, it fundamentally struggles to understand the definitive causal links between discrete log events. Because these models are designed to generate plausible narratives rather than strictly factual chronologies, they often fill logical gaps in their reasoning by hallucinating evidence. They will confidently fabricate connections or treat hypothesized events as absolute facts just to make the timeline fit together.

Instead of an accurate attribution of the attack, your team ends up chasing ghost IPs, non-existent payload executions, and entirely manufactured attack vectors.

LLMs are for summarization, not hunting

Does this mean Claude and other LLMs have no place in security and observability? No. It just means we’re putting them at the wrong end of the pipeline. LLMs are fundamentally built for linguistic synthesis, not high-volume data filtering.

To effectively hunt for threats, you still need a dedicated telemetry pipeline to parse, reduce, and analyze log volume at the source. You need deterministic rules and statistical machine learning to catch actual anomalies, establish historical baselines, and drastically reduce the noise.

How Grepr uses LLMs to hunt for threats

Grepr is built on the principle this blog argues for: filter first, then reason. The platform has two components that work in sequence.

The Autonomous Telemetry Pipeline drinks from the firehose so the LLM never has to. It sits between your telemetry sources and your existing observability tools, with no proprietary agents, reinstrumentation, or dashboard migration. As data streams through, Grepr detects millions of log patterns, trace signatures, and metric trends dynamically, learning what normal looks like for your specific environment. That learned baseline is exactly what an LLM lacks on its own. Repetitive, benign data is reduced, and the signals that deviate from normal rise to the surface. Critically, nothing gets summarized away: the pipeline writes full-fidelity raw data to your own S3 bucket, so the obscure port number or odd user-agent string is still there days later, when it suddenly matters.

Grepr AI is the reasoning layer. Instead of reading raw telemetry, it performs semantic analysis only on the filtered signals the pipeline surfaces. That keeps token costs proportional to what's actually interesting, keeps the context window small and coherent, and gives the model a constrained, high-signal dataset to work from. From there, Grepr AI explains what's happening in plain language for a human operator and can plan and execute user-defined remediation workflows, moving your team from detection to response faster. And because the complete raw record is retained, any conclusion it draws can be checked against the original events rather than taken on faith.

Together, the two components put each technology where it belongs: pattern recognition handles volume and baselines, and the LLM handles synthesis.

Try it out: Design partnership is open

We’re actively seeking design partners to provide feedback on what we’re building. In exchange, you’ll get to run Grepr AI for free. If you’d like to join the program, reach out to us.

FAQ

Why can’t I just stream my logs to an LLM and have it hunt for threats?

The token math doesn’t work. Threat hunting means sifting through an enormous, repetitive volume of telemetry, and paying a per-token fee on that firehose is financially prohibitive. The cost of processing the 99.99% of logs that are benign wipes out the value of finding the one anomaly.

Why are LLMs risky for reconstructing a kill chain?

These models generate plausible narratives, not strictly factual chronologies, so when they hit a gap in the evidence they tend to fill it by hallucinating connections or treating a hypothesis as fact. Instead of accurate attribution, your team ends up chasing ghost IPs and attack vectors that never happened.

How does Grepr use LLMs to hunt threats?

Filter first, then reason. The Autonomous Telemetry Pipeline drinks from the firehose, learns what’s normal for your environment, surfaces the signals that deviate, and writes full-fidelity raw data to your own S3 so nothing is summarized away. Grepr AI then runs semantic analysis only on those filtered signals, which keeps token costs and the context window small, and every conclusion can be checked against the original events.

‍