Sentence Embeddings in Rust on the CPU, without ONNX Runtime or libtorch
Run all-MiniLM-L6-v2 and other sentence-transformers models from Rust with one crate: no ONNX export, no libtorch, no Python. The same scores as sentence-transformers, a 25 MB binary, and 1,000 sentences embedded in about a second.
The usual ways to get sentence embeddings in Rust each bring something along. rust-bert links libtorch. ort and the crates built on it run an ONNX export of the model, inside ONNX Runtime's C++ library. candle is pure Rust, but it is a framework: tokenizing, pooling and normalizing are yours to get right.
kjarni is a Rust crate that loads the model's safetensors from Hugging Face and does the whole pipeline, with output that matches sentence-transformers:
use kjarni::Embedder;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let embedder = Embedder::builder("minilm-l6-v2").quiet(true).build().await?;
let docs = [
"How do I reset my password?",
"Where can I track my order?",
"What is your refund policy?",
"How do I update my billing address?",
"Do you ship internationally?",
];
let query = "I need to change my login credentials";
for (i, score) in embedder.rank_by_similarity(query, &docs).await? {
println!("{score:7.4} {}", docs[i]);
}
let vectors = embedder.embed_batch(&docs).await?;
println!("{} vectors of {} dimensions", vectors.len(), embedder.dimension());
Ok(())
}
0.5981 How do I reset my password?
0.4067 How do I update my billing address?
0.0767 Where can I track my order?
-0.0027 What is your refund policy?
-0.0451 Do you ship internationally?
5 vectors of 384 dimensions
Those are the scores Python's sentence-transformers gives for the same sentences, to four decimals.
Try it
cargo new embed-demo
cd embed-demo
cargo add [email protected] anyhow
cargo add tokio --features rt-multi-thread,macros
Put the code above in src/main.rs, then:
cargo run --release
The first run downloads all-MiniLM-L6-v2 (90 MB) into a local cache. Build in release mode: the matrix kernels are slow in a debug build.
What it costs
Measured on an Intel i7-13700 (24 threads), Rust 1.91, Linux x86-64:
First cargo build --release, all dependencies | 2 min 14 s |
Rebuild after changing main.rs | 3 s |
| Release binary | 25 MB |
| Load the model from the cache | 60 to 70 ms |
| Embed 1,000 sentences in one batch | 1.15 s |
The binary links against nothing but the system C library:
$ ldd target/release/embed-demo
linux-vdso.so.1
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
/lib64/ld-linux-x86-64.so.2
There is no shared library to ship next to it and no LD_LIBRARY_PATH to set. The
matrix kernels use AVX2 and FMA on x86-64 and NEON on ARM, picked at runtime.
Builder options
let embedder = Embedder::builder("minilm-l6-v2")
.cpu() // or .gpu(), or .auto_device()
.offline() // never download; fail if the model is not cached
.cache_dir("/var/lib/myapp/models")
.normalize(true) // L2-normalize, so a dot product is the cosine
.max_length(256)
.quiet(true)
.build()
.await?;
.offline() is the one to use in production. With it, a missing model is an error at
start-up rather than a download in the middle of a request:
Model 'mpnet-base-v2' not downloaded and download policy is set to Never
Download the model when you build the image or install the service, for example with
the Kjarni CLI: kjarni model download minilm-l6-v2.
Other models
Pass another name to Embedder::builder:
| Name | Dimensions | Use |
|---|---|---|
minilm-l6-v2 | 384 | Fast English default |
mpnet-base-v2 | 768 | Better English quality, slower |
multilingual-e5-small | 384 | Many languages, small |
bge-m3 | 1024 | Over 100 languages, long inputs |
nomic-embed-text | 768 | Long English inputs |
kjarni::embedder::available_models() lists every encoder the crate can load by name.
Embedder::from_path loads a directory of safetensors you already have, for one of the
supported architectures.
Async, and the runtime
The API is async because loading can download files. Encoding itself is CPU work spread over a thread pool. In an async web server, a large batch inside a request handler keeps that request busy for the whole batch, so embed big collections in a background task at start-up or on a schedule, and keep request handlers to the query.
What it does not do
- No training or fine-tuning. It runs models.
- Inputs are cut at the model's limit, 256 tokens for all-MiniLM-L6-v2, the same as sentence-transformers. See all-MiniLM-L6-v2 Max Sequence Length.
- No vector database. You get
Vec<f32>s. For a few thousand documents a scan is fast; the crate also has anIndexerandSearcherfor an on-disk index with keyword, semantic and hybrid search.
FAQ
How do I generate sentence embeddings in Rust?
Add the kjarni crate, build an Embedder with Embedder::builder("minilm-l6-v2").build().await?,
and call embed or embed_batch. It downloads the model on first use and runs it on the
CPU.
Does it need ONNX Runtime, libtorch or Python?
No. The inference engine is Rust, and the release binary links only against the system C library.
Does it match sentence-transformers?
Yes. Similarity scores agree to four decimal places on the same model, and the crate's tests compare outputs against PyTorch.
Does it need a GPU?
No. .gpu() uses one through wgpu (Vulkan, Metal or DirectX 12) when present.
Getting it
Crate: cargo add kjarni
Docs: https://docs.rs/kjarni
GitHub: https://github.com/olafurjohannsson/kjarni
Example: https://github.com/kjarni-ai/kjarni-examples/tree/main/rust-semantic-search
Related
Kjarni runs the same engine on every platform: