Analyze syslog messages with Seq

Using Seq.Input.Syslog, Seq is able to ingest syslog messages — both RFC3164 and RFC5424 formats — as structured logs.

Contents

What is syslog?
Syslog message formats
    RFC 3164
    RFC 5424
How to ingest syslog messages into Seq
    Method 1: (Windows, Docker) installing Seq.Input.Syslog directly in Seq
    Method 2: (Docker) running a separate seq-input-syslog "sidecar" container
Example usage: analysing NGINX logs with Seq

What is syslog?

Syslog (System Logging) is a logging format and protocol created by Eric Allman as part of Sendmail in the 1980s, and has since gained popularity in *nix based systems — including BSD (Berkeley Software Distrubution) Unix, Linux, and macOS — as well as network devices, such as printers and routers.

Syslog was first standardized by the IETF (Internet Engineering Task Force) in 2001, when the team published a Request for Comments titled "The BSD Syslog Protocol" (RFC 3164). "The Syslog Protocol" (RFC 5424), a more modern syslog standard, was later published in 2009, and obsoleted RFC 3164.

Seq.Input.Syslog is able to parse message formats described in both RFC 3164 and RFC 5424, with a few important things to note.

Firstly, Seq.Input.Syslog currently only supports receiving syslog messages over UDP. Secondly, the MSG component of syslog messages sent to Seq via Seq.Input.Syslog are not currently parsed — even if they contain structured elements, they are sent to Seq as free text.

Finally, we do not recommend using Seq.Input.Syslog as a first choice for most use-cases, as there are more convenient log formats that also add structure to the MSG portion of the log. We hope syslog support does help those who maintain and support systems that rely on syslog.

Syslog message formats

Even though RFC 3164 has been obsoleted by RFC 5424, the older log format is still supported in many applications. Seq.Input.Syslog supports structured events for both versions.

Here is a handy reference for both log formats.

RFC 3164

Take the following RFC 3164-formatted syslog message

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

This message is made up of several important "parts".

Below is our simplified explanation of Section 4.1 syslog Message Parts in RFC 3164.

• PRI — or "priority", is a number calculated from Facility (what kind of message) code and Severity (how urgent is the message) code: PRI = Facility * 8 + Severity
  • Reference tables for Facility and Severity in RFC 3164
• TIMESTAMP — format is Mmm dd hh:mm:ss
• HOSTNAME — must contain the hostname, IPv4 or IPv6 address of the message sender
• MSG is made up of two parts:
  • TAG — the name of the program or process that generated the message. Usually followed by a : or a [pid]: (the beginning of the MSG CONTENT)
  • CONTENT — the details of the message

These parts are parsed into structured log messages in Seq using the Seq syslog input. Here's what the above message looks like in Seq:

RFC 5424

Here is an example RFC 5424-formatted syslog message

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

Again, this message is made up of important "parts". These parts are explained in the next section.

RFC 5424 messages contain more parts than RFC 3164, probably due to no longer being limited to maximum 1024 byte message size.

This is our simplified explanation of Section 6. Syslog Message Format in RFC 5424.

• HEADER
  • PRI — or "priority", is a number calculated from Facility (what kind of message) code and Severity (how urgent is the message) code: PRI = Facility * 8 + Severity
    • Reference tables for Facility and Severity in RFC 5424
  • VERSION — version is always "1" for RFC 5424
  • TIMESTAMP — valid timestamp examples (must follow ISO 8601 format with uppercase "T" and "Z")
    • 1985-04-12T23:20:50.52Z
    • 2003-08-24T05:14:15.000003-07:00
    • - ("nil" value) if time not available
  • HOSTNAME — using FQDN (fully qualified domain name) is recommended, e.g. mymachine.example.com
  • APP-NAME — usually the name of the device or application that provided the message
  • PROCID — often used to provide the process name or process ID (is - "nil" in the example)
  • MSGID — should identify the type of message, more detail in RFC 5424 Section 6.2.7. MSGID
• STRUCTURED-DATA — named lists of key-value pairs for easy parsing and searching, more detail in RFC 5424 Section 6.3. STRUCTURED-DATA
• MSG — details about the event
  • if the MSG is encoded in UTF-8, the string must start with the Unicode byte order mask (BOM), more detail in RFC 5424 Section 6.4. MSG

And here's what the same RFC 5424 formatted message looks like in Seq:

How to ingest syslog messages into Seq

There are two ways to send syslog messages to Seq:

• by installing Seq.Input.Syslog directly in Seq (Windows and Docker), or
• by running a separate seq-input-syslog (Docker only).

Below is a convenient shell command to test if your setup is working, while you're getting set up with Seq.Input.Syslog, you can use this test syslog message to check everything is configured correctly (if your system has netcat):

$ echo '<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] -' | nc -w0 -u localhost 514

Method 1: (Windows, Docker) installing Seq.Input.Syslog directly in Seq

The simplest method is

1. installing Seq.Input.Syslog directly in Seq via Settings > Apps, and then
2. set up an instance (i.e. syslog receiver) using the Add Instance button in Apps

If you are running Seq as a Windows service, you must first check that your chosen syslog listener port is allowed through Windows firewall (UDP port 514 is the default, but you can pick a different UDP port).

If you are running Seq in Docker, you must first expose your chosen syslog listener port via your docker run command or docker-compose file. Remember, this is only required if you are installing Seq.Input.Syslog in Settings > Apps. Here's a docker run datalust/seq command with the correct ports exposed:

$ docker run \
    --name seq \
    -e ACCEPT_EULA=Y \
    -p 5341:80 \
    -p 514:514/udp \
    datalust/seq

If you are running Seq in Docker, we recommend you choose the seq-input-syslog sidecar container method, where you do not need to expose any extra ports on the seq container, and will also save you the extra app installation steps.

Method 2: (Docker) running a separate seq-input-syslog "sidecar" container

For Seq to ingest syslog messages, you can deploy datalust/seq-input-syslog as a Docker container alongside a separate Seq instance.

The seq-input-syslog container receives syslog messages (via UDP on port 514 by default), and forwards them to the Seq ingestion endpoint specified in the SEQ_ADDRESS environment variable.

Here's a docker run datalust/seq-input-syslog command with the default UDP listener port exposed, and sends logs to a Seq ingestion endpoint at https://seq.example.com:5341:

$ docker run \
    --name seq-input-syslog \
    -p 514:514/udp \
    -e SEQ_ADDRESS=https://seq.example.com:5341 \
    datalust/seq-input-syslog

Example: analysing NGINX logs with Seq

Here is an example docker-compose.yml which uses Docker's syslog log driver to forward NGINX docker container logs to Seq (i.e. whatever you see in stdout when you run docker logs -f <container-name>).

Important note: This is not our recommended way to get NGINX logs into Seq. Instead, use Seq.Input.GELF.

version: "3.8"
services:
  web:
    image: nginx
    volumes:
      - ./html:/usr/share/nginx/html:ro
    ports:
      - "8888:80"
    environment:
      - NGINX_HOST=example.org
      - NGINX_PORT=80
    logging:
      driver: "syslog"
      options:
        syslog-address: "udp://localhost:514" # read below for why this is "localhost"
        syslog-format: "rfc5424"
        tag: "nginx/syslog"

  seq-input-syslog:
    image: datalust/seq-input-syslog:latest
    depends_on:
      - seq
    ports:
      - "514:514/udp"
    environment:
      - SEQ_ADDRESS=http://seq:5341

  seq:
    image: datalust/seq:latest
    ports:
      - "5341:80"
    environment:
      - ACCEPT_EULA=Y

Why is localhost allowed for the logging-driver syslog-address? This is because the logging driver daemon is actually on the host machine, and localhost is resolved outside of the docker container.

Here is what an NGINX log looks like in Seq, after accessing localhost:8888:

That's it! Hope you're up and running in minutes with Seq as your new centralized syslog server :)

If you have any feedback or would like to contribute, please create a GitHub issue for Seq.Input.Syslog.

Happy logging! ❤️