A first look at Tracing in Seq 2024.1

TL;DR: Seq 2024.1 will fully support distributed tracing, including OpenTelemetry trace ingestion, trace indexing, and hierarchical trace visualization. The datalust/seq:preview container images and 2024.1.x-pre MSIs are ready to download and try in non-mission-critical environments. We'd love for you to check these out and share your experiences and feedback!

We've integrated tracing directly into Seq's Events screen:

• In the event search results, click on the little timing bar to the right of a span to show the trace to which it belongs.
• Click on spans in the trace view itself to search for events collected during the span.

Keep in mind that the list of search results is always filtered using the current date range, selected signals, and search expression, so you may need to relax or remove these filters in order to see all events that occurred during a span.

What does Tracing add?

Tracing reveals how systems and components work together in a complex operation. This comes in handy when:

• Understanding the performance costs of different system functions,
• Figuring out what chain of events led to a failure or interesting outcome, or
• Discovering the relationship between components in an unfamiliar system.

These are the core scenarios we're targeting with tracing in Seq 2024.1.

OpenTelemetry clients

Seq's HTTP/protobuf OTLP endpoint is /ingest/otlp/v1/traces. This works with both HTTP and HTTPS, on any port that Seq is listening on.

If you're using HTTPS and the Kestrel web server (the default on Docker/Linux, or optionally enabled on Windows), then traces can be sent using the gRPC flavor of OTLP, on the same ports.

Here's an example showing the .NET OpenTelemetry SDK configured in a minimal dotnet new web ASP.NET Core app:

using System.Diagnostics;
using OpenTelemetry.Exporter;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .ConfigureResource(resource => resource.AddService("example"))
        .AddAspNetCoreInstrumentation()
        .SetSampler(new AlwaysOnSampler())
        .AddOtlpExporter(exporter =>
        { 
            exporter.Endpoint = new Uri("http://localhost:5341/ingest/otlp/v1/traces");
            exporter.Protocol = OtlpExportProtocol.HttpProtobuf;
            exporter.Headers = "X-Seq-ApiKey=Your-Seq-API-Key";
        })
    );

var app = builder.Build();

app.MapGet("/", () =>
{
    Activity.Current?.SetTag("Random", Random.Shared.Next());
    return "Hello World!";
});

app.Run();

Each request to the application results in a simple single-span trace.

Other clients

Seq also supports trace ingestion using plain HTTP/JSON requests. We're currently investigating how this can be used to "light up" alternative client libraries such as Serilog, and we'll post more about this in the coming weeks.

Behind the scenes

The most extensive back-end update in this version is the addition of high-cardinality trace indexes. Seq 2024.1 achieves subsecond response times for trace id searches even within very large data sets.

We do this using on-disk segmented hash indexes, which provide great performance for a very small additional storage cost (less than 1% on disk, in our test environment).

The second substantial back-end update is support for deep sparse materialization support in our vectorized JSON parser stack.

This means that to serve the query http.status_code > 299, the query engine will ignore other parts of the http object when pulling out the value of status_code.

Because of this, you'll notice an improvement in query throughput anywhere you might already be searching inside nested structures: counting records matching a particular OpenTelemetry resource attribute sees a 56% speed-up using the new version.

The path to RTM

We hope you'll like Seq 2024.1 even in its preview form: tracing support is already very versatile, and improving rapidly.

Our plan is to ship this release as early as possible in the new year, so we need your feedback on today's preview 🙏 either here in the comments, on the Seq issue tracker, or through our support email address (find this under Support in Seq itself).