Kjarni vs PyTorch: Where a Dependency-Free Engine Wins and Loses

Kjarni is a Rust inference engine that ships as a single binary with no Python, no ONNX runtime, and no BLAS installation. PyTorch ships with Intel MKL. This post is the measurement of what that trade costs and what it buys.

Every number below is from one machine on one afternoon, and the methodology section says exactly what was and was not included. Where PyTorch wins, it wins by a lot, and that is in the table too.

The setup

An Intel i7-13700: eight performance cores, eight efficiency cores, 24 threads. PyTorch 2.13.0+cpu, which uses 16 threads by default. Kjarni on CPU, f32, using all 24. Both load the same weights from the same local cache, and both mean-pool and L2-normalise the output, so the comparison is like for like.

Timings are the mean of five runs after a warm-up call, on an otherwise idle machine. Kjarni's numbers include tokenisation; PyTorch's do too.

Encoding

Four models, both encode (one string) and encode_batch, with documents of one sentence (about 18 tokens) and twelve (about 216).

Across 36 comparable measurements PyTorch is faster in 33 and Kjarni in 3.

The shape matters more than the count. PyTorch's lead is largest on small inputs, 1.5x to 1.9x, and shrinks as the work grows until Kjarni draws level and, on the largest batches of the two largest models, passes it. The worst case for Kjarni is a single long document, where it is 2.4x behind: there is one large matrix multiply to do and nothing to amortise the per-layer overhead against.

Decode

Generation is the other direction entirely.

Kjarni is about 1.6x faster at generating tokens. Decoding runs one row at a time, so it is bound by how fast weights stream from memory rather than by matrix multiply throughput, and per-token overhead dominates. HuggingFace's generate() carries a meaningful amount of Python per step. A native loop does not.

Prompt processing is the reverse: prefilling a 480-token prompt takes Kjarni 2500 ms against PyTorch's 712 ms, because prefill is one big matrix multiply and that is the same regime as encoding.

Why

The profile is unambiguous. Sampling a whole encode of MiniLM:

54.50%  gemm_f32::microkernel::fma::f32::x2x6   faer's GEMM kernel
 9.89%  gemm_common::simd::vectorize            faer's packing
 8.86%  libm::math::erff::erff                  GELU
 4.56%  crossbeam_epoch::with_handle            rayon bookkeeping

Sixty-four percent of the time is in matrix multiplication, which is where it should be. Doing the same for PyTorch:

45.44%  mkl_blas_avx2_sgemm_kernel_0    MKL's GEMM
 7.32%  mkl_blas_avx2_sgemm_scopy       MKL's packing
22%     blas_thread_server              idle spinning
15%     gomp barrier waits              idle

PyTorch spends 37% of its runtime waiting on thread barriers and idle BLAS spin servers. It has more overhead than we do. It wins anyway, because the workload is 19 GFLOP of matrix multiply per forward pass and MKL's kernel achieves roughly 856 GFLOPS on this machine, about 78% of its AVX2 peak. Ours achieves closer to 321.

That is the whole story. Not architecture, not memory layout, not language. One hand-tuned kernel that Intel has been refining for two decades.

The trade

MKL is not a Rust crate. It is a large C library, x86_64 only, under Intel's own license. Linking it statically takes a binary from 18 MB to hundreds of MB, and it would not cover the aarch64 Linux and macOS builds we ship, nor WebAssembly, where the same engine runs in a browser tab.

So we use faer, a Rust GEMM, and hand-written AVX2 kernels for the decode path where a single-row matrix multiply is a different problem. The cost is measured above. What it buys is a tar.gz with a shared library and two headers that runs anywhere, and a cargo add with nothing behind it.

If you want MKL's numbers, use PyTorch. If you want an inference engine you can drop into a C++ desktop app, an ASP.NET service, a Go binary and a browser tab without asking anyone to install a runtime, the difference is 1.0x to 1.9x on encoding and 1.6x in our favour on generation.

Correctness

Speed claims are worthless without them. Every model here is checked against PyTorch's own output on every release:

Four of five agree to f32 rounding. mpnet has a residual we have not yet explained, and it is listed here rather than omitted.

One model in our registry, nomic-embed-text, could not be benchmarked against PyTorch at all: loading it there needs trust_remote_code=True and fetching a custom architecture from the Hub. Kjarni runs it natively.

Caveats

One machine, one afternoon, f32, CPU only. Repeated runs of the same configuration varied by 12% to 22% on the largest batches, so any two rows within about 1.2x of each other should be read as a tie. Rows are single measurements, not distributions.

The benchmark harness is in the repository. If your numbers differ, we would like to know.