Moving metadata to Seq's native storage engine

In Seq 2022.1 we've hit a bit of an internal milestone. It's the first release of Seq ever with no dependency on any external storage engine. On Windows that means ESENT is no longer used for local metadata (document) storage, and on Linux that means no more LMDB. All traces of these other engines have been hoisted out-of-tree and all local storage now runs through Flare, the storage engine initially built for Seq's event store.

We made the move to migrate event storage into Flare some time ago, but at the time we left metadata storage in the hands of third-party engines:

Seq 2021 storage architecture; the Flare storage engine is used by the event store and metadata store; the metadata store also uses ESENT, LMDB, and has MSSQL and PostgreSQL options for external storage.

When we made ARM support generally available in 2021.4, we made Flare the default engine for metadata storage on that architecture, but left the other platforms alone. Now we're moving all document storage into Flare:

Seq 2022 storage architecture; ESENT and LMDB no longer appear as local metadata storage options.

This doesn't have much direct impact on end-users, but helps us keep the Seq codebase fresh, focused, and malleable. It makes supporting Seq simpler because we've got fewer complex moving parts to understand. It makes building Seq simpler because components are now common to all platforms. It makes developing Seq simpler because effort we invest in features and tooling for event storage is naturally shared by document storage too.

This is an exciting milestone for us, but we're extremely grateful to the many people who've contributed to ESENT and LMDB. We couldn't have gotten to where we are today without them.

Seq 2022.1 will be ready to download or docker pull later this month.

Ashley Mannix

Read more posts by this author.