# Local Semantic Search in C++ without libtorch or ONNX Runtime

> Run embeddings, classification and reranking from C++ with no libtorch and no ONNX Runtime: one header and one shared library that links only against libc. Works with C++11, C++14, C++17 and C++23, and takes four commands from nothing to output.

Source: https://kjarni.ai/blog/cppinference/ (published 2026-09-05)

Ask how to run a transformer model from C++ and you get two answers: link libtorch, or
convert the model and link ONNX Runtime. Both work, and both are a second inference engine
inside your process with a toolchain of its own.

There is a third answer, and it takes four commands. Kjarni runs transformer models in a
C++ process through one shared library and a C header, and it works from C++11 upward: only
the optional `kjarni.hpp` convenience wrapper needs C++23, and only for `std::expected`.

```bash
mkdir kjarni-quickstart && cd kjarni-quickstart
curl -sL https://github.com/olafurjohannsson/kjarni/releases/latest/download/kjarni-x86_64-linux.tar.gz | tar xz
curl -sO https://raw.githubusercontent.com/olafurjohannsson/kjarni/main/crates/kjarni-ffi/examples/cpp/hello.cpp
g++ -std=c++23 hello.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello && ./hello
```

```
related:   0.5510
unrelated: -0.0630
```

That is a transformer model, downloaded, loaded and run, from an empty directory. No
package manager and no model conversion step. The archive holds the shared library,
`kjarni.h` (the C ABI) and `kjarni.hpp` (a header-only C++23 wrapper). macOS and Windows
builds are on the same
[releases page](https://github.com/olafurjohannsson/kjarni/releases).

Here is what `hello.cpp` contains:

```cpp
#include "kjarni.hpp"
#include <print>

int main() {
    // Downloaded once and cached under ~/.cache/kjarni, then loaded from disk.
    auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
    if (!embedder) {
        std::println("{}", embedder.error().message());
        return 1;
    }

    auto question = embedder->encode("How do I get my money back?");
    auto related = embedder->encode("What is your refund policy?");
    auto unrelated = embedder->encode("The weather in Reykjavik is unpredictable.");

    // No shared words with the question, but the same meaning.
    std::println("related:   {:.4f}", kjarni::cosine(*question, *related));
    std::println("unrelated: {:.4f}", kjarni::cosine(*question, *unrelated));
}
```

"How do I get my money back?" and "What is your refund policy?" share no words at all, and
score 0.55. The sentence about the weather scores below zero. That gap is the entire idea
behind semantic search.

## How semantic search works

An embedding model reads text and returns a vector, an array of floats, 384 numbers for the
model above. Text with similar meaning produces vectors that point in similar directions.

```
"refund policy"  -> [0.12, -0.34, 0.56, ...]  (384 numbers)
"get money back" -> [0.11, -0.33, 0.55, ...]  (384 numbers)  <- close
"weather today"  -> [-0.45, 0.23, -0.12, ...] (384 numbers)  <- far
```

You compare two vectors with cosine similarity, which measures the angle between them and
ignores their length. It runs from 1 for identical direction to -1 for opposite.

## What you actually link against

This is the part that decides whether the approach is worth anything, so it is worth
checking rather than believing. Ask the linker:

```
$ ldd hello
    linux-vdso.so.1
    libkjarni_ffi.so => /home/you/kjarni-quickstart/libkjarni_ffi.so
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    /lib64/ld-linux-x86-64.so.2
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
```

Kjarni, the C++ runtime, and the parts of glibc every program already uses: libc, libm,
libgcc, libpthread and libdl. That is the whole list: no libtorch, no `onnxruntime`, no
CUDA runtime, and nothing installed on the machine to make it work. The binary is 283 KB and the library is 19.4 MB, which includes
the tokenizer, the model loaders and every kernel.

The `-Wl,-rpath,'$ORIGIN'` in the build line is what makes that first entry resolve to the
library sitting next to your binary rather than something in `/usr/local/lib`. Keep it, and
the directory you built in is a directory you can copy somewhere else and run.

A dependency you cannot see in `ldd` is a dependency that cannot break you on a machine that
is not yours.

## Errors are values

Every fallible call returns `std::expected<T, kjarni::Error>`. Nothing in the header throws
except `std::bad_alloc`.

```cpp
auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
if (!embedder) {
    std::println(stderr, "could not load model: {}", embedder.error().message());
    return 1;
}
```

Whether a missing model file is exceptional depends on the program. A batch job should die;
a desktop application should show a message and carry on. Returning the failure lets the
caller decide, and puts it in the signature where it cannot be missed.

The options are a designated-initialiser aggregate, so a call names only what it changes:
`Embedder::create({.model = "mpnet-base-v2", .gpu = true})`.

## Searching a corpus

Encode the documents once, encode the query at search time, sort by similarity.

```cpp
#include "kjarni.hpp"

#include <algorithm>
#include <print>
#include <ranges>
#include <string_view>
#include <vector>

int main() {
    auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
    if (!embedder) {
        std::println(stderr, "could not load model: {}", embedder.error().message());
        return 1;
    }

    constexpr std::array docs = {
        std::string_view{"How do I reset my password?"},
        std::string_view{"What is your refund policy?"},
        std::string_view{"Do you ship internationally?"},
        std::string_view{"How do I update my billing address?"},
        std::string_view{"Where can I track my order?"},
    };

    std::vector<std::vector<float>> corpus;
    corpus.reserve(docs.size());
    for (std::string_view d : docs) {
        auto v = embedder->encode(d);
        if (!v) {
            std::println(stderr, "encode failed: {}", v.error().message());
            return 1;
        }
        corpus.push_back(v->to_vector());
    }

    constexpr std::string_view query = "I need to change my login credentials";
    auto q = embedder->encode(query);
    if (!q) {
        std::println(stderr, "encode failed: {}", q.error().message());
        return 1;
    }

    std::vector<std::pair<float, std::string_view>> scored;
    for (auto [i, doc] : std::views::enumerate(docs))
        scored.emplace_back(kjarni::cosine(q->values(), corpus[i]), doc);

    std::ranges::sort(scored, std::ranges::greater{},
                      &std::pair<float, std::string_view>::first);

    std::println("query: \"{}\"", query);
    for (auto [score, doc] : scored)
        std::println("  {:.4f}  {}", score, doc);
}
```

```
query: "I need to change my login credentials"
  0.5981  How do I reset my password?
  0.4067  How do I update my billing address?
  0.0767  Where can I track my order?
  -0.0027  What is your refund policy?
  -0.0451  Do you ship internationally?
```

"Change my login credentials" matches "reset my password" at 0.60 while sharing no words
with it, and "update my billing address" comes second because changing account details is a
related idea. That is what you cannot get from keyword matching.

`encode` returns a `kjarni::Embedding`, which owns the array the C API returned and frees it
in its destructor. It hands out a `std::span<const float>` through `values()`, so it drops
straight into ranges, and `to_vector()` copies when the data has to outlive the object.
There is no raw pointer to forget.

## Classification and reranking

The other models follow the same shape.

```cpp
auto clf = kjarni::Classifier::create({.model = "roberta-sentiment"});
if (!clf) { std::println(stderr, "{}", clf.error().message()); return 1; }

for (std::string_view t : {"I love this product!",
                           "Terrible quality, broke after one day."}) {
    auto top = clf->top(t);
    if (!top) { std::println(stderr, "{}", top.error().message()); return 1; }
    if (*top) std::println("  {:<9} {:5.1f}%  \"{}\"", (*top)->name, (*top)->score * 100, t);
}
```

```
  positive   98.5%  "I love this product!"
  negative   94.1%  "Terrible quality, broke after one day."
```

`top()` returns `Result<std::optional<Label>>`, which puts two separate questions in the
type: did the call work, and did it produce a label. A model with nothing above threshold is
not a failure.

Reranking uses a cross-encoder, which reads the query and each document together instead of
comparing two independently produced vectors. It is slower and much more precise, so it runs
as a second pass over whatever the embeddings retrieved:

```cpp
auto rr = kjarni::Reranker::create();
if (!rr) { std::println(stderr, "{}", rr.error().message()); return 1; }

const std::vector<std::string> docs = {
    "Machine learning is a subset of artificial intelligence.",
    "Deep learning uses neural networks with many layers.",
    "The weather today is sunny.",
};

auto ranked = rr->rerank("What is machine learning?", docs);
if (!ranked) { std::println(stderr, "{}", ranked.error().message()); return 1; }

for (const auto& r : *ranked)
    std::println("  {:>9.4f}  {}", r.score, docs[r.index]);
```

```
    10.5139  Machine learning is a subset of artificial intelligence.
    -5.5301  Deep learning uses neural networks with many layers.
   -11.1001  The weather today is sunny.
```

The scores are logits, not probabilities. What matters is the ordering and the size of the
gap between one document and the next. `Ranked` gives back an index into your input rather
than a copy of the text, so whatever IDs, URLs and permissions came with your documents stay
attached to them.

One signature detail: `rerank` takes `std::span<const std::string>` rather than
`string_view`, because the C call underneath needs an array of null terminated pointers and
a `string_view` does not promise one.

## The same numbers in every language

Those figures are not specific to C++. The reranker scores 10.5139, -5.5301 and -11.1001 are
the same values the [C# document search post](/blog/documentsearchengine/) prints, and the
similarity scores are the ones in [Semantic Search in C#](/blog/semanticsearch/).

That is the reason a C ABI is the right shape here. There is one engine and one set of
kernels, and the C++ header is a wrapper over the same entry points the C#, Go and Python
packages call. There is no second implementation to drift from the first.

## Choosing a model

| Model | Dimensions | Input limit | Notes |
|-------|-----------|-------------|-------|
| `minilm-l6-v2` | 384 | 256 tokens | Default. Fast, good quality per byte |
| `mpnet-base-v2` | 768 | 384 tokens | Higher quality, slower |
| `nomic-embed-text` | 768 | 8192 tokens | Long documents, though trained at 2048 |
| `bge-m3` | 1024 | 8192 tokens | Large, multilingual |

Mind the input limit column. `minilm-l6-v2` reads 256 tokens, roughly 900 characters, and
silently drops the rest: no error, no warning, just a vector computed from the part it saw.
The cross-encoder has its own limit, reading query and document as one sequence capped at
512 tokens. If your documents are longer than that, chunk them. There is a measurement of
what the truncation costs in
[Your MiniLM Embeddings Are Probably Truncating at 256 Tokens](/blog/embedding-truncation/).

## Practical notes

**Threading.** The handles are not individually thread safe. Give each thread its own, or
serialise calls. The engine already parallelises across cores inside a single call, so one
embedder will use the machine.

**C++23 is only needed for `std::expected`.** Everything else in `kjarni.hpp` is C++20, and
GCC 13 or Clang 17 and newer will build it.

**There is no package manager integration yet.** No Conan recipe, no vcpkg port. The four
commands above are the install story on Linux, and the equivalent archives for macOS and
Windows are on the releases page.

## Does it work with C++11 or C++17?

Yes. Kjarni works with C++11, C++14, C++17 and C++23.

Only `kjarni.hpp` requires C++23, and only because it returns `std::expected`. The engine
itself is reached through `kjarni.h`, which is plain C, is the interface every other
language binding in the project is built on, and compiles from C++11 upward. Plenty of
codebases cannot move, and a header that demands C++23 is not much use to them.

The whole of the C++23 convenience is one RAII wrapper and a copy:

```cpp
#include "kjarni.h"

#include <cstdio>
#include <memory>
#include <string>
#include <vector>

namespace {

struct EmbedderDeleter {
    void operator()(KjarniEmbedder* p) const { kjarni_embedder_free(p); }
};
using EmbedderPtr = std::unique_ptr<KjarniEmbedder, EmbedderDeleter>;

std::vector<float> encode(KjarniEmbedder* emb, const char* text) {
    KjarniFloatArray arr{};
    if (kjarni_embedder_encode(emb, text, &arr) != KJARNI_ERROR_CODE_OK) {
        std::fprintf(stderr, "encode failed: %s\n", kjarni_last_error_message());
        return {};
    }
    std::vector<float> out(arr.data, arr.data + arr.len);
    kjarni_float_array_free(arr);   // copied out, so release the engine's buffer
    return out;
}

} // namespace

int main() {
    KjarniEmbedderConfig cfg = kjarni_embedder_config_default();
    cfg.model_name = "minilm-l6-v2";
    cfg.quiet = 1;

    KjarniEmbedder* raw = nullptr;
    if (kjarni_embedder_new(&cfg, &raw) != KJARNI_ERROR_CODE_OK) {
        std::fprintf(stderr, "could not load model: %s\n", kjarni_last_error_message());
        return 1;
    }
    EmbedderPtr embedder(raw);

    const std::vector<float> question  = encode(embedder.get(), "How do I get my money back?");
    const std::vector<float> related   = encode(embedder.get(), "What is your refund policy?");
    const std::vector<float> unrelated = encode(embedder.get(), "The weather in Reykjavik is unpredictable.");
    if (question.empty() || related.empty() || unrelated.empty()) return 1;

    std::printf("related:   %.4f\n",
                kjarni_cosine_similarity(question.data(), related.data(), question.size()));
    std::printf("unrelated: %.4f\n",
                kjarni_cosine_similarity(question.data(), unrelated.data(), question.size()));
}
```

That file ships alongside `hello.cpp`, so the C++11 route is two commands as well:

```bash
curl -sO https://raw.githubusercontent.com/olafurjohannsson/kjarni/main/crates/kjarni-ffi/examples/cpp/hello_c_api.cpp
g++ -std=c++11 hello_c_api.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello_c_api && ./hello_c_api
```

```
related:   0.5510
unrelated: -0.0630
```

The same numbers as the C++23 version, because it is the same engine underneath. It builds
unchanged under `-std=c++11`, `c++14`, `c++17` and `c++23`.

Two rules cover the manual memory. Any `KjarniFloatArray` you receive is freed with
`kjarni_float_array_free` once you have copied what you need out of it, and any handle is
freed with its matching `_free` function. Wrapping the handle in a `unique_ptr` with a
custom deleter, as above, means the second rule takes care of itself on every return path.

Error text comes from `kjarni_last_error_message()`, and it reports the most recent failure
process wide, so read it immediately after the call that failed rather than saving it up.

## How the three compare

| | libtorch | ONNX Runtime | Kjarni |
|---|---|---|---|
| Install | download SDK, match ABI | package plus model conversion | one archive |
| Model format | TorchScript | `.onnx`, converted | safetensors and GGUF directly |
| Entries in `ldd` | many | its own stack | none beyond libc |
| Tokenizer | bring your own | bring your own | included |
| GPU | CUDA toolkit | CUDA or DirectML | Vulkan, Metal, DirectX 12 |

The scope is deliberate, and worth being plain about. Kjarni implements particular model
families rather than an arbitrary computation graph: BERT-style encoders, cross-encoders,
Llama-family decoders, T5, BART and Whisper. If you need to run a novel research
architecture, that is the wrong shape and you should reach for a general runtime. If you
need embeddings, classification, reranking or chat inside a program that has to ship
somewhere, it is why the whole thing fits in one archive with no toolchain.

## FAQ

### Does Kjarni work with C++11 or C++17?

Yes. The engine is reached through `kjarni.h`, which is plain C and compiles from C++11
upward. Only the optional `kjarni.hpp` wrapper needs C++23, and only for `std::expected`.
`hello_c_api.cpp` in the repository is the same program written against the C header, and
it builds unchanged under `-std=c++11`, `c++14`, `c++17` and `c++23`.

### Does it need libtorch or ONNX Runtime?

No. Kjarni is a single shared library with its own inference engine, tokenizer and model
loaders. `ldd` on a built binary shows Kjarni, the C++ runtime and glibc, and nothing else.
There is no model conversion step either: it reads HuggingFace safetensors and GGUF files
as they are.

### Does it need Python installed?

No. The C++ library needs no Python toolchain to build or to run, and no `pip` step in your
build. Kjarni also ships on PyPI, if you want the same engine from Python elsewhere in your
stack, but the C++ library is independent of it.

### Does it need a GPU or a CUDA toolkit?

No. Everything on this page runs on CPU. GPU support goes through WebGPU rather than CUDA,
so there is no CUDA toolkit to install even when you use it.

### What models can it run?

BERT-style encoders for embeddings, cross-encoders for reranking, sequence classifiers for
sentiment and toxicity, Llama-family decoders for chat, plus T5, BART and Whisper. Models
download on first use and cache under `~/.cache/kjarni`.

### Can I ship the result without the user installing anything?

Yes. Build with `-Wl,-rpath,'$ORIGIN'` and keep `libkjarni_ffi.so` next to the binary. The
directory you built in is a directory you can copy to another machine and run, provided the
model cache travels with it or the machine can reach the network once.

### Is it thread safe?

The handles are not individually thread safe. Give each thread its own, or serialise calls.
A single handle already parallelises across cores inside one call.

## Getting it

```
Releases:          https://github.com/olafurjohannsson/kjarni/releases
GitHub:            https://github.com/olafurjohannsson/kjarni
NuGet (C#):        https://www.nuget.org/packages/Kjarni
npm (WebAssembly): https://www.npmjs.com/package/kjarni-wasm
Go module:         https://pkg.go.dev/github.com/olafurjohannsson/kjarni-go
```

## Related

- [Semantic Search in C#](/blog/semanticsearch/): the same engine and the same vectors, from .NET
- [Build a Document Search Engine in C#](/blog/documentsearchengine/): keyword and semantic retrieval combined, with reranking
- [Why I Built a Native ML Inference Engine in Rust](/blog/nativeinference/): what is underneath all of this
- [ML from the Command Line](/blog/cli/): the same models as a UNIX tool
