Why I Built a Native ML Inference Engine in Rust

I needed semantic search in a Rust application. Users would search for "doctor" and the system needed to find documents mentioning "physician" or "medical practitioner." The standard approach is sentence embeddings, which turn text into vectors that capture meaning, then compare them.

The problem wasn't the algorithm. The problem was getting it to run.

The Python Tax

In Python, generating embeddings is straightforward:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
vector = model.encode("hello world")

Three lines. It works. But installing sentence-transformers pulls in PyTorch, tokenizers, NumPy, and everything they depend on. A fresh virtual environment balloons to 6.8 GB, mostly PyTorch alone.

For a Python application that already lives in that ecosystem, this is fine. For anything else, whether a C# backend, a Go microservice, a Rust CLI tool or a desktop application, it means shipping a Python runtime alongside your application, managing virtual environments in production, and accepting a multi-gigabyte deployment for what should be a single function call.

The ONNX Detour

ONNX Runtime seemed like the answer. Export the model to ONNX format, load it from Rust using the ort crate. No Python at runtime.

pub fn new(model_path: &str, tokenizer_path: &str) -> Result<Self> {
    let environment = ort::environment::init()
        .with_execution_providers([CUDAExecutionProvider::default().build()])
        .commit()?;
}

It worked. But ort pulled in 80+ crates, expanded the release binary to 350 MB, and relied on system libraries: libstdc++, libpthread, libm, libc. Then came the OpenSSL problem:

error: OpenSSL 3.3 required
$ openssl version
OpenSSL 1.1.1k  # Can't upgrade: would break RHEL dependencies

The C++ dependencies wanted OpenSSL 3.3. The production server ran RHEL with OpenSSL 1.1. Upgrading would break half the system. The binary worked on my machine and nowhere else.

This is the dependency trap. You trade Python's runtime dependency for C++'s build-time dependency, and end up with a different set of problems that are harder to debug.

Building It From Scratch

The core operations of a transformer model aren't complicated. Matrix multiplication, layer normalization, attention, softmax. The model architecture is well-documented. The hard part isn't the math. It's getting the infrastructure (tokenizer, weight loading, memory layout, SIMD dispatch) right without pulling in an existing runtime.

So I built one. Pure Rust. No ONNX. No external C++ libraries. No system dependencies beyond libc.

The result is Kjarni, a native ML inference engine that compiles to a single shared library. It runs transformer models for embeddings, classification, reranking, search, and text generation.

What it looks like in practice

Rust:

use kjarni::Embedder;

let embedder = Embedder::new("minilm-l6-v2").await?;
let vector = embedder.embed("hello world").await?;
let similarity = embedder.similarity("doctor", "physician").await?;
// 0.859813

C# / .NET:

using Kjarni;

using var embedder = new Embedder("minilm-l6-v2");
Console.WriteLine(embedder.Similarity("doctor", "physician"));
// 0.859813

CLI:

$ kjarni similarity "doctor" "physician" -q
0.859813

$ kjarni classify "I love this product!"
  ✓       POSITIVE  ████████████████████  100.0%
          NEGATIVE  ░░░░░░░░░░░░░░░░░░░░    0.0%

$ echo "Terrible quality" | kjarni classify --model toxic-bert
  Input "Terrible quality"

  ✓          toxic  ░░░░░░░░░░░░░░░░░░░░    0.6%
            insult  ░░░░░░░░░░░░░░░░░░░░    0.0%
           obscene  ░░░░░░░░░░░░░░░░░░░░    0.0%

Models download on first use and cache locally. That is the entire setup.

What's Under the Hood

Kjarni doesn't wrap ONNX, LibTorch, or any external inference engine. The runtime is written in Rust from the ground up:

• Hand-tuned SIMD kernels: AVX2/FMA on x86, NEON on ARM. Matrix multiplication, layer norm, and softmax are the hot paths; these are written to use the full width of the vector registers.
• GPU compute via WebGPU: custom WGSL shaders for GPU inference, using whichever adapter the platform provides. Vulkan on Linux and Windows is the path that gets exercised; Metal is reached through the same abstraction on macOS.
• Zero-copy model loading: models are memory-mapped (mmap) so the OS manages paging. Loading a 90MB model doesn't allocate 90MB of heap.
• BF16 compute path: bfloat16 arithmetic where the hardware supports it, cutting memory bandwidth in half on the attention layers.
• Quantization: Q4, Q6, Q8 for reduced model size and faster inference on CPU.

The entire engine, including the tokenizer, compiles to a single .so (Linux) or .dll (Windows). The only system dependency is glibc 2.17, which means it runs on anything from a modern Ubuntu back to RHEL 7.

Supported Models

Kjarni currently supports:

Encoder, decoder, and seq2seq architectures are all supported, and generation is exposed in every binding: Chat in C#, IChatClient for Microsoft.Extensions.AI, chat and generate on the command line, and the equivalent in Go.

Language Bindings

The Rust core exposes a C ABI through kjarni-ffi, which makes it straightforward to call from any language with a C FFI:

• C# / .NET: available on NuGet. dotnet add package Kjarni.
• Go: a module that loads the library at runtime rather than through cgo, so CGO_ENABLED=0 still works and cross compiling stays simple.
• WebAssembly: kjarni-wasm on npm, the same engine compiled for the browser.
• C and C++: kjarni.h and kjarni.hpp ship with the library, since the C ABI is what every other binding is built on.
• Python: kjarni on PyPI, currently an alpha.
• CLI: a standalone binary that reads from stdin, writes JSON, and pipes like any UNIX tool.

Adding a new language binding means writing a thin wrapper around the C API. The inference logic stays in Rust.

The Dependency Comparison

Try It

C# / .NET:

dotnet add package Kjarni

CLI:

curl -fsSL https://kjarni.ai/install.sh | sh

Rust / Build from source:

git clone https://github.com/olafurjohannsson/kjarni
cargo build --release -p kjarni-cli

Where it went

• Semantic Search in C# and Sentiment Analysis in C#: the two things people reach for first
• Build a Document Search Engine in C#: hybrid keyword and semantic retrieval over your own files
• ML From the Command Line: the same engine as a UNIX tool that reads stdin and writes JSON
• Your MiniLM Embeddings Are Probably Truncating at 256 Tokens: a bug this engine had, and why the parity tests missed it

The source is on GitHub. Licensed MIT or Apache-2.0.