· 4 min read

Sentence Embeddings in Rust on the CPU, without ONNX Runtime or libtorch

Run all-MiniLM-L6-v2 and other sentence-transformers models from Rust with one crate: no ONNX export, no libtorch, no Python. The same scores as sentence-transformers, a 25 MB binary, and 1,000 sentences embedded in about a second.

The usual ways to get sentence embeddings in Rust each bring something along. rust-bert links libtorch. ort and the crates built on it run an ONNX export of the model, inside ONNX Runtime's C++ library. candle is pure Rust, but it is a framework: tokenizing, pooling and normalizing are yours to get right.

kjarni is a Rust crate that loads the model's safetensors from Hugging Face and does the whole pipeline, with output that matches sentence-transformers:

use kjarni::Embedder;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let embedder = Embedder::builder("minilm-l6-v2").quiet(true).build().await?;

    let docs = [
        "How do I reset my password?",
        "Where can I track my order?",
        "What is your refund policy?",
        "How do I update my billing address?",
        "Do you ship internationally?",
    ];
    let query = "I need to change my login credentials";

    for (i, score) in embedder.rank_by_similarity(query, &docs).await? {
        println!("{score:7.4}  {}", docs[i]);
    }

    let vectors = embedder.embed_batch(&docs).await?;
    println!("{} vectors of {} dimensions", vectors.len(), embedder.dimension());
    Ok(())
}
 0.5981  How do I reset my password?
 0.4067  How do I update my billing address?
 0.0767  Where can I track my order?
-0.0027  What is your refund policy?
-0.0451  Do you ship internationally?
5 vectors of 384 dimensions

Those are the scores Python's sentence-transformers gives for the same sentences, to four decimals.

Try it

cargo new embed-demo
cd embed-demo
cargo add [email protected] anyhow
cargo add tokio --features rt-multi-thread,macros

Put the code above in src/main.rs, then:

cargo run --release

The first run downloads all-MiniLM-L6-v2 (90 MB) into a local cache. Build in release mode: the matrix kernels are slow in a debug build.

What it costs

Measured on an Intel i7-13700 (24 threads), Rust 1.91, Linux x86-64:

First cargo build --release, all dependencies2 min 14 s
Rebuild after changing main.rs3 s
Release binary25 MB
Load the model from the cache60 to 70 ms
Embed 1,000 sentences in one batch1.15 s

The binary links against nothing but the system C library:

$ ldd target/release/embed-demo
	linux-vdso.so.1
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
	/lib64/ld-linux-x86-64.so.2

There is no shared library to ship next to it and no LD_LIBRARY_PATH to set. The matrix kernels use AVX2 and FMA on x86-64 and NEON on ARM, picked at runtime.

Builder options

let embedder = Embedder::builder("minilm-l6-v2")
    .cpu()                      // or .gpu(), or .auto_device()
    .offline()                  // never download; fail if the model is not cached
    .cache_dir("/var/lib/myapp/models")
    .normalize(true)            // L2-normalize, so a dot product is the cosine
    .max_length(256)
    .quiet(true)
    .build()
    .await?;

.offline() is the one to use in production. With it, a missing model is an error at start-up rather than a download in the middle of a request:

Model 'mpnet-base-v2' not downloaded and download policy is set to Never

Download the model when you build the image or install the service, for example with the Kjarni CLI: kjarni model download minilm-l6-v2.

Other models

Pass another name to Embedder::builder:

NameDimensionsUse
minilm-l6-v2384Fast English default
mpnet-base-v2768Better English quality, slower
multilingual-e5-small384Many languages, small
bge-m31024Over 100 languages, long inputs
nomic-embed-text768Long English inputs

kjarni::embedder::available_models() lists every encoder the crate can load by name. Embedder::from_path loads a directory of safetensors you already have, for one of the supported architectures.

Async, and the runtime

The API is async because loading can download files. Encoding itself is CPU work spread over a thread pool. In an async web server, a large batch inside a request handler keeps that request busy for the whole batch, so embed big collections in a background task at start-up or on a schedule, and keep request handlers to the query.

What it does not do

  • No training or fine-tuning. It runs models.
  • Inputs are cut at the model's limit, 256 tokens for all-MiniLM-L6-v2, the same as sentence-transformers. See all-MiniLM-L6-v2 Max Sequence Length.
  • No vector database. You get Vec<f32>s. For a few thousand documents a scan is fast; the crate also has an Indexer and Searcher for an on-disk index with keyword, semantic and hybrid search.

FAQ

How do I generate sentence embeddings in Rust?

Add the kjarni crate, build an Embedder with Embedder::builder("minilm-l6-v2").build().await?, and call embed or embed_batch. It downloads the model on first use and runs it on the CPU.

Does it need ONNX Runtime, libtorch or Python?

No. The inference engine is Rust, and the release binary links only against the system C library.

Does it match sentence-transformers?

Yes. Similarity scores agree to four decimal places on the same model, and the crate's tests compare outputs against PyTorch.

Does it need a GPU?

No. .gpu() uses one through wgpu (Vulkan, Metal or DirectX 12) when present.

Getting it

Crate:   cargo add kjarni
Docs:    https://docs.rs/kjarni
GitHub:  https://github.com/olafurjohannsson/kjarni
Example: https://github.com/kjarni-ai/kjarni-examples/tree/main/rust-semantic-search

Kjarni runs the same engine on every platform: