Running a Local LLM from C++

A C++ program that wants to generate text usually ends up talking to something else: an HTTP call to a hosted model, or a local server on a port that has to be running before your process starts and stay running while it does. Either way the language model lives outside your binary, and your program's job becomes moving JSON.

There is a smaller arrangement. Kjarni loads the model into your own process and answers from there. One header, one shared library, and the weights are memory in your address space rather than a service you depend on.

#include "kjarni.hpp"

auto chat = kjarni::Chat::create({
    .model = "llama3.2-1b-instruct",
    .system_prompt = "You are terse. Answer in one short sentence.",
});

auto reply = chat->send("What is the capital of Iceland?",
                        kjarni::Generation::greedy(48));
std::println("{}", *reply);

Reykjavik.

That is the whole thing. No client, no port, no process to supervise.

Getting there

The release archive holds the shared library, kjarni.h (the C ABI) and kjarni.hpp (a header-only C++23 wrapper). Point CMake at the directory and link:

cmake_minimum_required(VERSION 3.16)
project(chat CXX)

set(CMAKE_CXX_STANDARD 23)   # std::expected, used by kjarni.hpp
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(chat main.cpp)
target_include_directories(chat PRIVATE ${KJARNI_DIR}/include)
target_link_libraries(chat PRIVATE ${KJARNI_DIR}/lib/libkjarni_ffi.so)

The C++23 requirement is the wrapper's, not the engine's. kjarni.hpp returns std::expected so failures are values rather than exceptions. If you are on an older standard, kjarni.h is a plain C ABI and works from C++11 onward.

The model downloads on first use and is cached afterwards. llama3.2-1b-instruct is around a gigabyte; the library itself is 19.4 MB, which includes the tokenizer, the model loaders and every kernel.

Errors are values

create returns kjarni::Result<Chat>, so a missing model or an unreadable cache is something you handle rather than something that unwinds:

auto chat = kjarni::Chat::create({.model = "llama3.2-1b-instruct"});
if (!chat) {
    std::println("could not start: {}", chat.error().message());
    return 1;
}
std::println("context window: {} tokens", chat->context_size());

context window: 131072 tokens

That number comes from the model's own configuration, not a constant in the library.

Streaming

A blocking send is fine for a one-shot answer. For anything a person waits on, take the tokens as they arrive. stream accepts any callable, and returning false stops generation early:

int tokens = 0;
auto streamed = chat->stream(
    "Count from one to five.", kjarni::Generation::greedy(32),
    [&tokens](std::string_view token) {
        std::print("{}", token);
        return ++tokens < 40;      // false ends generation
    });

One, two, three, four, five.

The callback runs on the generating thread, so keep it cheap: append to a buffer, post to your UI queue, write to a socket. The early-stop return is not just a convenience. It is how you implement a stop button without tearing down the model.

Sampling

Generation::greedy(n) always takes the highest-probability token, which makes runs reproducible and is what you want for extraction or classification-shaped prompts. For prose, sample instead:

kjarni::Generation creative;
creative.temperature = 0.7f;
creative.top_p = 0.95f;
creative.max_new_tokens = 64;

auto reply = chat->send("Describe Reykjavik in one sentence.", creative);

Reykjavik is the vibrant, eclectic capital of Iceland, known for its colorful
buildings, lively nightlife, and unparalleled natural beauty.

Generation holds std::optional fields, so anything you leave alone keeps the model's own default rather than a value the library invented. greedy() is the one shortcut: it sets temperature to zero and nothing else.

Greedy decoding on the same prompt gives the same answer every time. That property is worth more than it sounds when you are writing tests.

What ends up linked

ldd on the built binary shows Kjarni, the C++ runtime, and the parts of glibc every program already uses: libc, libm, libgcc, libpthread and libdl. That is the whole list. Nothing is installed on the machine to make it work, and nothing runs beside your process.

For embeddings, classification and reranking from the same library, see Local Semantic Search in C++. The same archive covers both: one Embedder for vectors, one Chat for generation.

FAQ

Which models can it load?

Llama-family instruction models, which includes Llama 3.x, Mistral, Qwen2.5 and Phi. They are read from HuggingFace safetensors or GGUF as they are, with no conversion step.

Does it need a GPU?

No. The example above runs on CPU. The GPU path uses Vulkan, Metal or DirectX 12, so AMD, Intel and Apple Silicon work as they are.

Can I run more than one conversation?

Yes. Each Chat keeps its own history and its own KV cache. Two instances are two independent conversations, and a single instance keeps its turns across calls to send.

Is the callback thread-safe?

It runs on the thread that called stream, one token at a time, so no locking is needed inside the callback itself. Anything you hand the tokens to needs the usual care.

How large is the context window?

Whatever the model declares. llama3.2-1b-instruct reports 131,072 tokens, and context_size() reads it from the loaded model rather than assuming.