Our journey from nightly to stable Rust

Ashley Mannix -

When we shipped Seq 5.0 back in November, our new storage engine was compiled against Rust's unstable nightly channel. As of Seq 5.1, we can instead use the supported stable channel. That feels like a bit of a milestone so I'd like to share a few details about our journey from nightly to stable, and celebrate the progress the community has made on the language, libraries, and tooling over the last twelve months that made that journey painless for us.

How Rust is shipped

The Rust programming language is released through a few channels with different levels of stability:

  • stable: the currently supported version. A new minor version is cut every six weeks. Point releases are uncommon but are made from time to time.
  • beta: candidate for the next supported version.
  • nightly: the current head of the master branch where most development takes place, and enables access to unstable features.

The trade-off between the stable and nightly channels is early access to new features but less stability in the compiler, libraries, and the availability of associated tools like the language server.

Starting on nightly

When we started working on our Rust codebase in early 2018 there were some very compelling unstable features that pulled us towards the nightly channel. The main one was non-lexical lifetimes, and the previews of the 2018 edition as they became available. Through the course of the year we both took on new unstable features and shed a few as they stabilized. We were careful not to depend on unstable features that were difficult to shim or didn't have a clear path to stabilization in the near future though, one example being specialization.

I dug back through the history of our Rust codebase to find some of the unstable features we ended up using. Very early into development the list was already pretty large:

// For the custom `reason` attribute used by unsafe macros
#![feature(custom_attribute)]
// For supporting visibility on the `unsafe_fn` macro
#![feature(macro_vis_matcher)]
// For attributes on blocks in `unsafe_block` macro expansion
#![feature(stmt_expr_attributes)]
// For using the `RangeArgument` trait
#![feature(collections_range)]
// For non-zero unique timestamps
#![feature(nonzero)]
// For hardware accelerated crc32c
#![feature(stdsimd, target_feature, cfg_target_feature)]
// For converting Rust results into FFI results
#![feature(try_trait)]
// For sanity
#![feature(nll)]

Turns out at this point we almost had more unstable features than implemented methods:

impl IngestBuf {
    /**
    Create an ingestion buffer.

    The file must not already exist. A new file will be created at `path`.
    */
    pub fn create(path: impl AsRef<Path>) -> Result<Self, Error> {
        unimplemented!()
    }
}

Aspirational-Doc-Comment-Driven Development at its best.

A year later that set of unstable features looked like this:

// For converting Rust results into FFI results
#![feature(try_trait)]
// Easier to create safe synchronization primitives
#![feature(optin_builtin_traits)]
// For using the `RangeBounds` trait
#![feature(range_contains)]
// For using the `SliceIndex` trait
#![feature(slice_index_methods)]

By this time the two big unstable features we needed; nll and target_feature; had both stabilized so we were left with just a few things to work around.

Moving from nightly to stable

In a previous post we described our std_ext module where various utilities and extensions to the standard library live. To support the remaining unstable features in some form on the stable channel we ended up adding a new unstable module under std_ext specifically for shimming:

src
└── std_ext
   ├── unstable
   │   ├── marker.rs
   │   ├── mod.rs
   │   ├── ops.rs
   │   └── range.rs
   ├── fs.rs
   ├── io.rs
   ├── iter.rs
   ├── mod.rs
   ├── num.rs
   ├── ops.rs
   ├── option.rs
   ├── range.rs
   ├── rc.rs
   ├── result.rs
   ├── slice.rs
   ├── sync.rs
   └── vec.rs

Collecting these shims together lets us track them as tech-debt that we can pay down once the original feature stabilizes. Let's look at how we shimmed our way around each of the remaining unstable features we depended on.

try_trait

The Try trait lets you hook into the ? operator. We use it to convert Rust's Result<T, E> into a flat, FFI-safe FlareResult in our C bindings. We ended up copying the implementation of the Try trait itself so we didn't need to change any impl blocks on our FlareResult type:

mod ops {
    // #![feature(try_trait)]
    pub(crate) trait Try {
        type Ok;
        type Error;

        fn into_result(self) -> Result<Self::Ok, Self::Error>;

        fn from_error(v: Self::Error) -> Self;

        fn from_ok(v: Self::Ok) -> Self;
    }
}

The ?s themselves were replaced with a flare_try! macro (reminiscent of ye olde try! macro) that called from_ok or from_err to carry the result like ? would.

Once try_trait stabilizes we can remove our version and replace flare_try with ? again.

range_contains

Our storage engine has a lot of range code and we used RangeBounds::contains extensively. Thankfully its implementation was fairly straightforward (we've got some truly crazy range code) so we made it an extension trait. We had to change the name of the method from contains to contains_item though so it didn't clash with the unstable one:

mod range {
    use std::ops::{
        Bound,
        RangeBounds,
    };

    // #![feature(range_contains)]
    pub(crate) trait RangeContainsExt<T> {
        fn contains_item<U>(&self, item: &U) -> bool
        where
            T: PartialOrd<U>,
            U: ?Sized + PartialOrd<T>;
    }

    impl<T, R> RangeContainsExt<T> for R
    where
        R: RangeBounds<T>,
    {
        fn contains_item<U>(&self, item: &U) -> bool
        where
            T: PartialOrd<U>,
            U: ?Sized + PartialOrd<T>,
        {
            (match self.start_bound() {
                Bound::Included(ref start) => *start <= item,
                Bound::Excluded(ref start) => *start < item,
                Bound::Unbounded => true,
            }) && (match self.end_bound() {
                Bound::Included(ref end) => item <= *end,
                Bound::Excluded(ref end) => item < *end,
                Bound::Unbounded => true,
            })
        }
    }
}

Once range_contains stabilizes we can remove our implementation.

slice_index_methods

Turns out we were only using the SliceIndex::get_unchecked method in a single place, so we inlined its implementation there.

optin_builtin_traits

There's an unstable feature that lets you opt-out of opt-in-built-in-traits (OIBITs) like Send and Sync using special syntax. If you find yourself needing this you're usually either building some synchronization primitive or a container that lies about what's in it. We have some primitive types that are explicitly !Sync, so instead of impl !Sync for Foo, we add a zero-sized field to these types that is !Send + !Sync:

mod marker {
    use std::marker::PhantomData;

    // #![feature(optin_builtin_traits)]
    #[derive(Default)]
    pub(crate) struct ClobberOibits(PhantomData<*mut fn()>);
}

Once optin_builtin_traits stabilizes we can remove ClobberOibits and restore the explicit opt-out impl blocks.

test

We did have a few internal micro-benchmarks that used the #[bench] attribute for checking things like the difference between our SSE4.2-optimized CRC checksum and its fall-back implementation. These benchmarks were useful in initial development, but not really worthwhile anymore so we just removed them. Our regular benchmark suite was already using a simple, custom framework that we have more control over so we didn't lose much.

Going forwards on stable

There's an ongoing conversation in the Rust community right now about minimal supported rustc versions (MSRVs). How hard should libraries work to support older versions of the Rust compiler? As a data-point, for our codebase we expect to bump the compiler version to the latest stable whenever we update our Cargo.lock (which is checked in to source control). So the MSRV we anticipate from our dependencies is effectively the latest stable.

Now that we can build our Rust codebase on the stable channel we'll be very hesitant to move back to nightly. There are some upcoming features I'd be interested to spike out though so that we can get a sense of how effective they are and provide feedback. The first one that comes to mind is impl Trait on type aliases, which could reduce some of the clumsy type boilerplate we have around iterators.

It's been exciting to see our list of reasons to depend on the unstable nightly channel grow smaller and smaller since we started as important features have been polished and stabilized! It's a solid platform to build on for the future.