# Sentence Embeddings in Rust on the CPU, without ONNX Runtime or libtorch

> Run all-MiniLM-L6-v2 and other sentence-transformers models from Rust with one crate: no ONNX export, no libtorch, no Python. The same scores as sentence-transformers, a 25 MB binary, and 1,000 sentences embedded in about a second.

Source: https://kjarni.ai/blog/sentence-embeddings-rust/ (published 2026-11-12)

The usual ways to get sentence embeddings in Rust each bring something along.
[rust-bert](https://github.com/guillaume-be/rust-bert) links libtorch.
[ort](https://github.com/pykeio/ort) and the crates built on it run an ONNX export of the
model, inside ONNX Runtime's C++ library. [candle](https://github.com/huggingface/candle)
is pure Rust, but it is a framework: tokenizing, pooling and normalizing are yours to get
right.

[kjarni](https://crates.io/crates/kjarni) is a Rust crate that loads the model's
safetensors from Hugging Face and does the whole pipeline, with output that matches
sentence-transformers:

```rust
use kjarni::Embedder;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let embedder = Embedder::builder("minilm-l6-v2").quiet(true).build().await?;

    let docs = [
        "How do I reset my password?",
        "Where can I track my order?",
        "What is your refund policy?",
        "How do I update my billing address?",
        "Do you ship internationally?",
    ];
    let query = "I need to change my login credentials";

    for (i, score) in embedder.rank_by_similarity(query, &docs).await? {
        println!("{score:7.4}  {}", docs[i]);
    }

    let vectors = embedder.embed_batch(&docs).await?;
    println!("{} vectors of {} dimensions", vectors.len(), embedder.dimension());
    Ok(())
}
```

```
 0.5981  How do I reset my password?
 0.4067  How do I update my billing address?
 0.0767  Where can I track my order?
-0.0027  What is your refund policy?
-0.0451  Do you ship internationally?
5 vectors of 384 dimensions
```

Those are the scores Python's sentence-transformers gives for the same sentences, to four
decimals.

## Try it

```bash
cargo new embed-demo
cd embed-demo
cargo add kjarni@0.4.2 anyhow
cargo add tokio --features rt-multi-thread,macros
```

Put the code above in `src/main.rs`, then:

```bash
cargo run --release
```

The first run downloads all-MiniLM-L6-v2 (90 MB) into a local cache. Build in release
mode: the matrix kernels are slow in a debug build.

## What it costs

Measured on an Intel i7-13700 (24 threads), Rust 1.91, Linux x86-64:

| | |
|---|---|
| First `cargo build --release`, all dependencies | 2 min 14 s |
| Rebuild after changing `main.rs` | 3 s |
| Release binary | 25 MB |
| Load the model from the cache | 60 to 70 ms |
| Embed 1,000 sentences in one batch | 1.15 s |

The binary links against nothing but the system C library:

```
$ ldd target/release/embed-demo
	linux-vdso.so.1
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
	/lib64/ld-linux-x86-64.so.2
```

There is no shared library to ship next to it and no `LD_LIBRARY_PATH` to set. The
matrix kernels use AVX2 and FMA on x86-64 and NEON on ARM, picked at runtime.

## Builder options

```rust
let embedder = Embedder::builder("minilm-l6-v2")
    .cpu()                      // or .gpu(), or .auto_device()
    .offline()                  // never download; fail if the model is not cached
    .cache_dir("/var/lib/myapp/models")
    .normalize(true)            // L2-normalize, so a dot product is the cosine
    .max_length(256)
    .quiet(true)
    .build()
    .await?;
```

`.offline()` is the one to use in production. With it, a missing model is an error at
start-up rather than a download in the middle of a request:

```
Model 'mpnet-base-v2' not downloaded and download policy is set to Never
```

Download the model when you build the image or install the service, for example with
the [Kjarni CLI](/blog/cli/): `kjarni model download minilm-l6-v2`.

## Other models

Pass another name to `Embedder::builder`:

| Name | Dimensions | Use |
|---|---|---|
| `minilm-l6-v2` | 384 | Fast English default |
| `mpnet-base-v2` | 768 | Better English quality, slower |
| `multilingual-e5-small` | 384 | Many languages, small |
| `bge-m3` | 1024 | Over 100 languages, long inputs |
| `nomic-embed-text` | 768 | Long English inputs |

`kjarni::embedder::available_models()` lists every encoder the crate can load by name.
`Embedder::from_path` loads a directory of safetensors you already have, for one of the
supported architectures.

## Async, and the runtime

The API is async because loading can download files. Encoding itself is CPU work spread
over a thread pool. In an async web server, a large batch inside a request handler keeps
that request busy for the whole batch, so embed big collections in a background task at
start-up or on a schedule, and keep request handlers to the query.

## What it does not do

- **No training or fine-tuning.** It runs models.
- **Inputs are cut at the model's limit**, 256 tokens for all-MiniLM-L6-v2, the same as
  sentence-transformers. See [all-MiniLM-L6-v2 Max Sequence Length](/blog/embedding-truncation/).
- **No vector database.** You get `Vec<f32>`s. For a few thousand documents a scan is
  fast; the crate also has an `Indexer` and `Searcher` for an on-disk index with keyword,
  semantic and hybrid search.

## FAQ

### How do I generate sentence embeddings in Rust?

Add the `kjarni` crate, build an `Embedder` with `Embedder::builder("minilm-l6-v2").build().await?`,
and call `embed` or `embed_batch`. It downloads the model on first use and runs it on the
CPU.

### Does it need ONNX Runtime, libtorch or Python?

No. The inference engine is Rust, and the release binary links only against the system
C library.

### Does it match sentence-transformers?

Yes. Similarity scores agree to four decimal places on the same model, and the crate's
tests compare outputs against PyTorch.

### Does it need a GPU?

No. `.gpu()` uses one through wgpu (Vulkan, Metal or DirectX 12) when present.

## Getting it

```
Crate:   cargo add kjarni
Docs:    https://docs.rs/kjarni
GitHub:  https://github.com/olafurjohannsson/kjarni
Example: https://github.com/kjarni-ai/kjarni-examples/tree/main/rust-semantic-search
```

## Related

- [Rust API reference](/reference/rust/)
- [Why I Built a Native ML Inference Engine in Rust](/blog/nativeinference/)
- [Sentence Embeddings in Python without PyTorch](/blog/sentence-transformers-without-pytorch/)
