Semantic Search in C++ without Python, libtorch or ONNX Runtime

Ask how to run a transformer model from C++ and you get two answers: link libtorch, or convert the model and link ONNX Runtime. Both work. Both are large, both want a toolchain of their own, and both put a second inference engine inside your process.

There is a third answer, and it takes four commands. Kjarni runs transformer models in a C++ process through one shared library and a C header, and it works from C++11 upward: only the optional kjarni.hpp convenience wrapper needs C++23, and only for std::expected.

mkdir kjarni-quickstart && cd kjarni-quickstart
curl -sL https://github.com/olafurjohannsson/kjarni/releases/latest/download/kjarni-x86_64-linux.tar.gz | tar xz
curl -sO https://raw.githubusercontent.com/olafurjohannsson/kjarni/main/crates/kjarni-ffi/examples/cpp/hello.cpp
g++ -std=c++23 hello.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello && ./hello

related:   0.5510
unrelated: -0.0630

That is a transformer model, downloaded, loaded and run, from an empty directory. No package manager, no Python, no model conversion step. The archive holds the shared library, kjarni.h (the C ABI) and kjarni.hpp (a header-only C++23 wrapper). macOS and Windows builds are on the same releases page.

Here is what hello.cpp contains:

#include "kjarni.hpp"
#include <print>

int main() {
    // Downloaded once and cached under ~/.cache/kjarni, then loaded from disk.
    auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
    if (!embedder) {
        std::println("{}", embedder.error().message());
        return 1;
    }

    auto question = embedder->encode("How do I get my money back?");
    auto related = embedder->encode("What is your refund policy?");
    auto unrelated = embedder->encode("The weather in Reykjavik is unpredictable.");

    // No shared words with the question, but the same meaning.
    std::println("related:   {:.4f}", kjarni::cosine(*question, *related));
    std::println("unrelated: {:.4f}", kjarni::cosine(*question, *unrelated));
}

"How do I get my money back?" and "What is your refund policy?" share no words at all, and score 0.55. The sentence about the weather scores below zero. That gap is the entire idea behind semantic search.

How semantic search works

An embedding model reads text and returns a vector, an array of floats, 384 numbers for the model above. Text with similar meaning produces vectors that point in similar directions.

"refund policy"  -> [0.12, -0.34, 0.56, ...]  (384 numbers)
"get money back" -> [0.11, -0.33, 0.55, ...]  (384 numbers)  <- close
"weather today"  -> [-0.45, 0.23, -0.12, ...] (384 numbers)  <- far

You compare two vectors with cosine similarity, which measures the angle between them and ignores their length. It runs from 1 for identical direction to -1 for opposite.

What you actually link against

This is the part that decides whether the approach is worth anything, so it is worth checking rather than believing. Ask the linker:

$ ldd hello
    linux-vdso.so.1
    libkjarni_ffi.so => /home/you/kjarni-quickstart/libkjarni_ffi.so
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    /lib64/ld-linux-x86-64.so.2
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2

Kjarni, the C++ runtime, and the parts of glibc every program already uses: libc, libm, libgcc, libpthread and libdl. That is the whole list. No libtorch, no onnxruntime, no Python, no CUDA runtime. The binary is 283 KB and the library is 19.4 MB, which includes the tokenizer, the model loaders and every kernel.

The -Wl,-rpath,'$ORIGIN' in the build line is what makes that first entry resolve to the library sitting next to your binary rather than something in /usr/local/lib. Keep it, and the directory you built in is a directory you can copy somewhere else and run.

A dependency you cannot see in ldd is a dependency that cannot break you on a machine that is not yours.

Errors are values

Every fallible call returns std::expected<T, kjarni::Error>. Nothing in the header throws except std::bad_alloc.

auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
if (!embedder) {
    std::println(stderr, "could not load model: {}", embedder.error().message());
    return 1;
}

Whether a missing model file is exceptional depends on the program. A batch job should die; a desktop application should show a message and carry on. Returning the failure lets the caller decide, and puts it in the signature where it cannot be missed.

The options are a designated-initialiser aggregate, so a call names only what it changes: Embedder::create({.model = "mpnet-base-v2", .gpu = true}).

Searching a corpus

Encode the documents once, encode the query at search time, sort by similarity.

#include "kjarni.hpp"

#include <algorithm>
#include <print>
#include <ranges>
#include <string_view>
#include <vector>

int main() {
    auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"});
    if (!embedder) {
        std::println(stderr, "could not load model: {}", embedder.error().message());
        return 1;
    }

    constexpr std::array docs = {
        std::string_view{"How do I reset my password?"},
        std::string_view{"What is your refund policy?"},
        std::string_view{"Do you ship internationally?"},
        std::string_view{"How do I update my billing address?"},
        std::string_view{"Where can I track my order?"},
    };

    std::vector<std::vector<float>> corpus;
    corpus.reserve(docs.size());
    for (std::string_view d : docs) {
        auto v = embedder->encode(d);
        if (!v) {
            std::println(stderr, "encode failed: {}", v.error().message());
            return 1;
        }
        corpus.push_back(v->to_vector());
    }

    constexpr std::string_view query = "I need to change my login credentials";
    auto q = embedder->encode(query);
    if (!q) {
        std::println(stderr, "encode failed: {}", q.error().message());
        return 1;
    }

    std::vector<std::pair<float, std::string_view>> scored;
    for (auto [i, doc] : std::views::enumerate(docs))
        scored.emplace_back(kjarni::cosine(q->values(), corpus[i]), doc);

    std::ranges::sort(scored, std::ranges::greater{},
                      &std::pair<float, std::string_view>::first);

    std::println("query: \"{}\"", query);
    for (auto [score, doc] : scored)
        std::println("  {:.4f}  {}", score, doc);
}

query: "I need to change my login credentials"
  0.5981  How do I reset my password?
  0.4067  How do I update my billing address?
  0.0767  Where can I track my order?
  -0.0027  What is your refund policy?
  -0.0451  Do you ship internationally?

"Change my login credentials" matches "reset my password" at 0.60 while sharing no words with it, and "update my billing address" comes second because changing account details is a related idea. That is what you cannot get from keyword matching.

encode returns a kjarni::Embedding, which owns the array the C API returned and frees it in its destructor. It hands out a std::span<const float> through values(), so it drops straight into ranges, and to_vector() copies when the data has to outlive the object. There is no raw pointer to forget.

Classification and reranking

The other models follow the same shape.

auto clf = kjarni::Classifier::create({.model = "roberta-sentiment"});
if (!clf) { std::println(stderr, "{}", clf.error().message()); return 1; }

for (std::string_view t : {"I love this product!",
                           "Terrible quality, broke after one day."}) {
    auto top = clf->top(t);
    if (!top) { std::println(stderr, "{}", top.error().message()); return 1; }
    if (*top) std::println("  {:<9} {:5.1f}%  \"{}\"", (*top)->name, (*top)->score * 100, t);
}

  positive   98.5%  "I love this product!"
  negative   94.1%  "Terrible quality, broke after one day."

top() returns Result<std::optional<Label>>, which puts two separate questions in the type: did the call work, and did it produce a label. A model with nothing above threshold is not a failure.

Reranking uses a cross-encoder, which reads the query and each document together instead of comparing two independently produced vectors. It is slower and much more precise, so it runs as a second pass over whatever the embeddings retrieved:

auto rr = kjarni::Reranker::create();
if (!rr) { std::println(stderr, "{}", rr.error().message()); return 1; }

const std::vector<std::string> docs = {
    "Machine learning is a subset of artificial intelligence.",
    "Deep learning uses neural networks with many layers.",
    "The weather today is sunny.",
};

auto ranked = rr->rerank("What is machine learning?", docs);
if (!ranked) { std::println(stderr, "{}", ranked.error().message()); return 1; }

for (const auto& r : *ranked)
    std::println("  {:>9.4f}  {}", r.score, docs[r.index]);

    10.5139  Machine learning is a subset of artificial intelligence.
    -5.5301  Deep learning uses neural networks with many layers.
   -11.1001  The weather today is sunny.

The scores are logits, not probabilities. What matters is the ordering and the size of the gap between one document and the next. Ranked gives back an index into your input rather than a copy of the text, so whatever IDs, URLs and permissions came with your documents stay attached to them.

One signature detail: rerank takes std::span<const std::string> rather than string_view, because the C call underneath needs an array of null terminated pointers and a string_view does not promise one.

The same numbers in every language

Those figures are not specific to C++. The reranker scores 10.5139, -5.5301 and -11.1001 are the same values the C# document search post prints, and the similarity scores are the ones in Semantic Search in C#.

That is the reason a C ABI is the right shape here. There is one engine and one set of kernels, and the C++ header is a wrapper over the same entry points the C#, Go and Python packages call. There is no second implementation to drift from the first.

Choosing a model

Mind the input limit column. minilm-l6-v2 reads 256 tokens, roughly 900 characters, and silently drops the rest: no error, no warning, just a vector computed from the part it saw. The cross-encoder has its own limit, reading query and document as one sequence capped at 512 tokens. If your documents are longer than that, chunk them. There is a measurement of what the truncation costs in Your MiniLM Embeddings Are Probably Truncating at 256 Tokens.

Practical notes

Threading. The handles are not individually thread safe. Give each thread its own, or serialise calls. The engine already parallelises across cores inside a single call, so one embedder will use the machine.

C++23 is only needed for std::expected. Everything else in kjarni.hpp is C++20, and GCC 13 or Clang 17 and newer will build it.

There is no package manager integration yet. No Conan recipe, no vcpkg port. The four commands above are the install story on Linux, and the equivalent archives for macOS and Windows are on the releases page.

Does it work with C++11 or C++17?

Yes. Kjarni works with C++11, C++14, C++17 and C++23.

Only kjarni.hpp requires C++23, and only because it returns std::expected. The engine itself is reached through kjarni.h, which is plain C, is the interface every other language binding in the project is built on, and compiles from C++11 upward. Plenty of codebases cannot move, and a header that demands C++23 is not much use to them.

The whole of the C++23 convenience is one RAII wrapper and a copy:

#include "kjarni.h"

#include <cstdio>
#include <memory>
#include <string>
#include <vector>

namespace {

struct EmbedderDeleter {
    void operator()(KjarniEmbedder* p) const { kjarni_embedder_free(p); }
};
using EmbedderPtr = std::unique_ptr<KjarniEmbedder, EmbedderDeleter>;

std::vector<float> encode(KjarniEmbedder* emb, const char* text) {
    KjarniFloatArray arr{};
    if (kjarni_embedder_encode(emb, text, &arr) != KJARNI_ERROR_CODE_OK) {
        std::fprintf(stderr, "encode failed: %s\n", kjarni_last_error_message());
        return {};
    }
    std::vector<float> out(arr.data, arr.data + arr.len);
    kjarni_float_array_free(arr);   // copied out, so release the engine's buffer
    return out;
}

} // namespace

int main() {
    KjarniEmbedderConfig cfg = kjarni_embedder_config_default();
    cfg.model_name = "minilm-l6-v2";
    cfg.quiet = 1;

    KjarniEmbedder* raw = nullptr;
    if (kjarni_embedder_new(&cfg, &raw) != KJARNI_ERROR_CODE_OK) {
        std::fprintf(stderr, "could not load model: %s\n", kjarni_last_error_message());
        return 1;
    }
    EmbedderPtr embedder(raw);

    const std::vector<float> question  = encode(embedder.get(), "How do I get my money back?");
    const std::vector<float> related   = encode(embedder.get(), "What is your refund policy?");
    const std::vector<float> unrelated = encode(embedder.get(), "The weather in Reykjavik is unpredictable.");
    if (question.empty() || related.empty() || unrelated.empty()) return 1;

    std::printf("related:   %.4f\n",
                kjarni_cosine_similarity(question.data(), related.data(), question.size()));
    std::printf("unrelated: %.4f\n",
                kjarni_cosine_similarity(question.data(), unrelated.data(), question.size()));
}

That file ships alongside hello.cpp, so the C++11 route is two commands as well:

curl -sO https://raw.githubusercontent.com/olafurjohannsson/kjarni/main/crates/kjarni-ffi/examples/cpp/hello_c_api.cpp
g++ -std=c++11 hello_c_api.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello_c_api && ./hello_c_api

related:   0.5510
unrelated: -0.0630

The same numbers as the C++23 version, because it is the same engine underneath. It builds unchanged under -std=c++11, c++14, c++17 and c++23.

Two rules cover the manual memory. Any KjarniFloatArray you receive is freed with kjarni_float_array_free once you have copied what you need out of it, and any handle is freed with its matching _free function. Wrapping the handle in a unique_ptr with a custom deleter, as above, means the second rule takes care of itself on every return path.

Error text comes from kjarni_last_error_message(), and it reports the most recent failure process wide, so read it immediately after the call that failed rather than saving it up.

Compared to the alternatives

The trade is scope. libtorch runs anything expressible in TorchScript. Kjarni runs the model families it implements: BERT-style encoders, cross-encoders, Llama-family decoders, T5, BART and Whisper. For an arbitrary research model, convert it and use ONNX Runtime. For embeddings, classification, reranking or chat inside a C++ program that has to ship somewhere, one archive with no toolchain is a smaller problem than either.

FAQ

Does Kjarni work with C++11 or C++17?

Yes. The engine is reached through kjarni.h, which is plain C and compiles from C++11 upward. Only the optional kjarni.hpp wrapper needs C++23, and only for std::expected. hello_c_api.cpp in the repository is the same program written against the C header, and it builds unchanged under -std=c++11, c++14, c++17 and c++23.

Does it need libtorch or ONNX Runtime?

No. Kjarni is a single shared library with its own inference engine, tokenizer and model loaders. ldd on a built binary shows Kjarni, the C++ runtime and glibc, and nothing else. There is no model conversion step either: it reads HuggingFace safetensors and GGUF files as they are.

Does it need Python installed?

No. There is no Python at build time or run time, and no pip dependency anywhere in the chain.

Does it need a GPU or a CUDA toolkit?

No. Everything on this page runs on CPU. GPU support goes through WebGPU rather than CUDA, so there is no CUDA toolkit to install even when you use it.

What models can it run?

BERT-style encoders for embeddings, cross-encoders for reranking, sequence classifiers for sentiment and toxicity, Llama-family decoders for chat, plus T5, BART and Whisper. Models download on first use and cache under ~/.cache/kjarni.

Can I ship the result without the user installing anything?

Yes. Build with -Wl,-rpath,'$ORIGIN' and keep libkjarni_ffi.so next to the binary. The directory you built in is a directory you can copy to another machine and run, provided the model cache travels with it or the machine can reach the network once.

Is it thread safe?

The handles are not individually thread safe. Give each thread its own, or serialise calls. A single handle already parallelises across cores inside one call.

Getting it

Releases:          https://github.com/olafurjohannsson/kjarni/releases
GitHub:            https://github.com/olafurjohannsson/kjarni
NuGet (C#):        https://www.nuget.org/packages/Kjarni
npm (WebAssembly): https://www.npmjs.com/package/kjarni-wasm
Go module:         https://pkg.go.dev/github.com/olafurjohannsson/kjarni-go

Related

• Semantic Search in C#: the same engine and the same vectors, from .NET
• Build a Document Search Engine in C#: keyword and semantic retrieval combined, with reranking
• Why I Built a Native ML Inference Engine in Rust: what is underneath all of this
• ML from the Command Line: the same models as a UNIX tool