Seq is a log server designed for modern structured logging and application monitoring. It's used in all kinds of organizations, with a wide variety of development technologies.
Back in March, we published the first Seq 5 preview for Windows and Docker/Linux. Although that release successfully demonstrated Seq running natively on both platforms, our vision for Seq 5 is broader. With the 5.0.1822-pre preview build published to the Seq downloads page (Windows) and Docker Hub today, the second major pillar of Seq 5, Native Storage, is ready to try.
Seq Native Storage is a new, cross-platform, log-data-specific storage engine built using the Rust programming language. By eliminating our dependency on third-party and platform-specific libraries for on-disk storage, we've opened up the possibility of significantly improving the performance and manageability of the Seq event store, with some substantial benefits already realized in Seq 5.
Where does Native Storage fit in?
Seq can be loosely separated into two major architectural components:
- the Seq UI and web API, which exposes most of the recognizable functionality of Seq, like dashboards, users, alerts, plug-in apps, and API keys, and
- the Seq log database, called Flare, which includes the SQL query engine and log-specific JSON event store.
The Flare log database has a pluggable storage model that allows different storage engines to provide on-disk persistence. Historically, most Seq instances will be using the Windows-only ESENT storage engine for this.
Our initial Seq 5 preview used LMDB as a cross-platform storage engine so that Seq could run on Docker and Linux. From today's preview onwards, Native Storage will replace both of these alternatives and become the default storage engine on both Windows and Linux.
What benefits can we expect from Native Storage in Seq 5?
Seq uses RAM caching extensively to provide fast queries over recent log events. On large systems, however, queries do often fall back to slow disk-backed searches, and it's in this situation that the new storage engine will provide the most benefits.
Native Storage implements transparent signal indexing to accelerate (sometimes dramatically) the time required for searching within a subset of the log stream. This means that, when searching the Seq disk archive, the time required for a search in "Errors" will be less than the time required for searching the unbounded stream, in proportion to the volume of events that match the "Errors" filter. We'll write more about signal indexing in the coming months.
In addition to this new capability, Native Storage is modestly faster for raw ingestion and retrieval, and lets us manage memory usage and other aspects of performance more predictably.
Status and schedule
Native Storage is the biggest overhaul of Seq's event store that we've ever embarked upon. Today's build is not ready for mission-critical deployment, but is suitable for development use and testing. We want to gather as much feedback and experience as possible over the next few months, leading up to a production release later this year.
One important note: Seq uses a rolling 7-day extent mechanism to switch between storage engines. If your Seq server is currently using ESENT, upgrading to the new build won't result in a storage switch until the end of the current 7-day period, and benefits of the new storage engine won't become visible until the older ESENT data ages out (existing data won't be moved automatically to the new engine).
Known issues with the current build include:
- Some Docker hosts run the container but fail to ingest any events (#703); this may be either kernel or volume-specific, we're still collecting reports and attempting to narrow this down .
- Disk space may be exhausted during retention policy processing, particularly on systems with low free space (#704); we believe the underlying bug has been fixed here, but are waiting for verification.
While we complete work on Native Storage, we'll be actively investing in Seq 5 features including enhancements to alerts and dashboarding, and various other quality-of-life improvements alongside those we've already implemented in the current Seq 5 preview.
A huge thanks is due to everyone who has tried and provided feedback on the variety of Seq 5 builds so far. We're increasingly feeling that Seq 5 is going to be something special, and we're deeply grateful for all of your help in making it a reality.
If you are able to try the new previews please get in touch and let us know how you're going! Bug reports and feedback are most welcome, either to our issue tracker or via support@getseq.net.