Grepr support for traces is now GA

Following up on our limited release last year, we’re excited to announce the general availability of processing trace data in Grepr. GA reflects months of production use with enterprise customers, whose feedback shaped the capabilities that make this a complete, enterprise-ready offering.

A lot has changed since the limited release. Here is what is new for GA:

• Proven in production. Enterprise customers like FOSSA run Grepr Traces in production at scale, cutting ingested span volume (trace noise) by 90%.
• Span head sampling. Ability to add sampling rules for the traces you care about, so known exceptions (traces) are retained directly in your observability platform.
• SQL transformations. Use SQL transformations inside the pipeline to generate metrics directly from your spans or mask and drop sensitive attributes.
• Automatic backfill for correlated logs. Automatically backfill the logs tied to the traces you care about.
• Automatic critical path analysis. Grepr shows you the exact paths to focus on when optimizing your applications.
• Scale and reliability hardening. The pipeline is hardened for enterprise trace volumes and production reliability.
• Automatically drop or merge duplicate spans within a trace to reduce trace sizes

Why you shouldn't sample traces at your collectors

Most teams running microservices at scale already know that trace data grows faster than log data. Datadog’s own analysis across its customer base found that unsampled trace volumes are roughly five times larger than log volumes.

The standard response is to sample aggressively at the collector, which reduces what reaches your observability platform, but it also means the majority of traces are gone before anyone decides they need them.

The benefits of Grepr’s path-based tail sampling

• Path-based analytics: Grepr summarizes traces that belong to the same path together so you can see aggregate behavior for a particular application path and how a particular trace within the aggregate compares.
• Automatic critical path analysis. Grepr identifies the critical path through each trace, so your developers know exactly which parts of the codebase to optimize.
• Centralized policy engine. Instead of chasing down individual teams to change rules, you can control trace policies across all your services and agents in one place, from sampling decisions to attribute mutation such as masking PII.
• Cost reduction. By forwarding only the traces or spans worth indexing to your observability platform while keeping the raw data in low-cost storage, Grepr significantly reduces how much you pay your observability vendor.

How traces flow through Grepr

Adding traces to Grepr takes a single configuration change in the collector you already run, the same way you sent logs to Grepr. Once that change is live, three things happen:

1. Ingest. Grepr ingests your spans from your existing collectors.
2. Reduce and forward. Grepr processes the spans through a Grepr pipeline and forwards only the traces worth indexing to your observability platform, using path-based tail sampling.
3. Retain everything. Grepr writes the complete, unmodified trace data to your Grepr data lake, where you can search it using familiar query syntaxes or backfill into your observability platform during an incident.

How path-based tail sampling works

Sampling exists because trace ingestion and indexing are expensive, so most teams reach for one of two standard techniques:

• Head-based sampling: Simple to configure, but it drops traces before their outcome is known; the risk is discarding traces with errors or slow requests before anyone sees it. 
• Tail-based sampling: Waits until a trace completes, which allows for filtering on latency or error status. But most tail-sampling techniques are coarse-grained, hiding different execution paths within the same bucket, thereby missing performance issues that are unique per path.

Both techniques force a trade-off between cost and ensuring that the exact trace you need survives.

 Grepr identifies the most interesting and slowest traces for every signature, and runs critical path analysis and other checks on them, so you get complete coverage of each path through your system instead of a random slice of your traffic. Those interesting traces are sent to your observability platform while the low-value traces are sent to low-cost storage.

Under the hood, path-based tail sampling runs as a multi-stage process that turns millions of spans into a representative, full-fidelity set of traces:

• Assemble. Grepr buffers incoming spans and partitions them by trace ID, so every span belonging to a request is processed together and assembled into a complete end-to-end trace.
• Map. Grepr traverses the full span hierarchy and serializes it into a structural signature, a canonical record of the trace’s exact execution path. Loop collapsing keeps repeated operations from inflating that signature.
• Bucket. Grepr groups traces by structural signature, a far finer split than the root span name, so every distinct path through your system is tracked on its own.
• Analyze. Grepr compares each trace only against others that took the same path, then samples by uniqueness and performance, keeping the rare paths and anomalies that coarse bucketing buries.
• Route. Selected (interesting) traces are forwarded to your observability platform. Separately, all traces are sent to low-cost storage and available for analysis or backfill during an investigation.

How path-based tail sampling reduces costs

Grepr eliminates the tradeoff between having enough trace data and spending too much on observability. Because Grepr sends only the traces worth indexing to your observability platform (and raw traces to low-cost storage), you spend significantly less. During an incident, you can always backfill specific traces within seconds into your observability platform and investigate with your existing workflows, dashboards, and service maps.

So you can stop sampling at your collectors and dropping traces before anyone knows if they matter or not.

How FOSSA reduced distributed trace volume by 90% in less than hour

The Problem: Unsustainable APM costs

As FOSSA’s microservices architecture scaled, distributed trace volume grew with it, and Datadog’s host-based APM billing became hard to predict as clusters scaled during peak loads. FOSSA needed to keep all of their traces, since they could not know in advance which ones would matter, and per-service sampling was error-prone and tedious. Engineers ended up spending time on cost reviews and quota discussions instead of building product.

The Solution: Extending Grepr to distributed traces

FOSSA had already seen what Grepr’s signal processing engine did for logs, so in less than an hour, they set up a pipeline for traces and pointed their collectors to Grepr. Their goal was the same as with logs: dramatically reduce what reaches Datadog while ensuring nothing operationally important gets lost. Grepr filters out redundant, low-value traces and sends high-value signals to Datadog, unmodified, prioritizing errors and high-latency outliers.

The results

FOSSA reduced distributed trace spans by 90% without losing important trace data or impacting operations. Because Datadog was billing them for APM overages, that reduction took their overage charges to zero, with no changes to how engineers instrument code or troubleshoot incidents. Dashboards kept populating, alerts kept firing, and any specific trace remained available in S3.

As Dave Bortz, VP Engineering at FOSSA, put it, “We wanted the same outcome we got with logs, and we got it. Fast implementation, no code changes, no dashboard rebuilds, no workflow disruptions. Our engineers kept working exactly the way they always had, and the costs came down.”

Read the full case study.

How to send traces to Grepr

If you already run a Grepr pipeline for logs, adding traces is a single configuration change. If you are new to Grepr, the steps are the same: you create a pipeline and point your collector at it.

Here is the full walkthrough:

1. Create an OpenTelemetry or Datadog source in your existing Grepr pipeline.

2. Point your agent at the Grepr ingestion endpoint.

3. Confirm spans are arriving in Grepr and that path-based tail sampling is forwarding traces to your observability platform.

4. Verify full trace data is landing in your data lake, queryable with your existing syntax.

Your existing log pipelines keep running exactly as before. Traces and logs are separate signals, each flowing through its own pipeline, and both land in the same Grepr data lake.

Logs Documentation

Trace Pipeline Documentation


‍

FAQ

How does Grepr reduce trace volume without losing the traces I need? Grepr uses path-based tail sampling. It identifies every unique execution path in your application and samples each path independently, keeping the most interesting and slowest traces for every path. Out of thousands of traces, usually less than a hundred represent distinct paths, so you get complete coverage of your system instead of a random slice of your traffic.

What is path-based tail sampling? It is a multi-stage process that turns millions of spans into a representative, full-fidelity set of traces. Grepr assembles complete traces by trace ID, maps each one to a structural signature of its exact execution path, buckets traces by that signature, analyzes each trace against others that took the same path, and routes the selected traces to your observability platform.

Do I need to change my instrumentation or collector? No. You continue to use your existing agents or collectors with a simple configuration change to use the Grepr ingestion endpoint. 

Where does my full trace data go? Grepr writes the complete, unmodified trace data to your data lake, so nothing is permanently lost. During an incident, you backfill the specific traces you need into your observability platform and investigate with your existing workflows.

How much can I save? FOSSA cut distributed trace volume by 90% and saw Datadog APM costs drop sharply, with no changes to how engineers instrument code or troubleshoot incidents. Your results will depend on your traffic patterns and current sampling setup.

Does it work with my observability platform? Yes. Grepr forwards the selected traces to your existing platform and preserves the full dataset in low-cost storage, so your day-to-day workflows, dashboards, and service maps stay intact.

‍