# Running a Local LLM from C++

> Load Llama, Mistral, Qwen or Phi inside a C++ program and generate text in-process. One header and one shared library, blocking replies and token streaming, with the model running on your own machine.

Source: https://kjarni.ai/blog/cppllm/ (published 2026-09-10)

A C++ program that wants to generate text usually ends up talking to something else: an
HTTP call to a hosted model, or a local server on a port that has to be running before your
process starts and stay running while it does. Either way the language model lives outside
your binary, and your program's job becomes moving JSON.

There is a smaller arrangement. Kjarni loads the model into your own process and answers
from there. One header, one shared library, and the weights are memory in your address
space rather than a service you depend on.

```cpp
#include "kjarni.hpp"

auto chat = kjarni::Chat::create({
    .model = "llama3.2-1b-instruct",
    .system_prompt = "You are terse. Answer in one short sentence.",
});

auto reply = chat->send("What is the capital of Iceland?",
                        kjarni::Generation::greedy(48));
std::println("{}", *reply);
```

```
Reykjavik.
```

That is the whole thing. No client, no port, no process to supervise.

## Getting there

The release archive holds the shared library, `kjarni.h` (the C ABI) and `kjarni.hpp` (a
header-only C++23 wrapper). Point CMake at the directory and link:

```cmake
cmake_minimum_required(VERSION 3.16)
project(chat CXX)

set(CMAKE_CXX_STANDARD 23)   # std::expected, used by kjarni.hpp
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(chat main.cpp)
target_include_directories(chat PRIVATE ${KJARNI_DIR}/include)
target_link_libraries(chat PRIVATE ${KJARNI_DIR}/lib/libkjarni_ffi.so)
```

The C++23 requirement is the wrapper's, not the engine's. `kjarni.hpp` returns
`std::expected` so failures are values rather than exceptions. If you are on an older
standard, `kjarni.h` is a plain C ABI and works from C++11 onward.

The model downloads on first use and is cached afterwards. `llama3.2-1b-instruct` is around
a gigabyte; the library itself is 19.4 MB, which includes the tokenizer, the model loaders
and every kernel.

## Errors are values

`create` returns `kjarni::Result<Chat>`, so a missing model or an unreadable cache is
something you handle rather than something that unwinds:

```cpp
auto chat = kjarni::Chat::create({.model = "llama3.2-1b-instruct"});
if (!chat) {
    std::println("could not start: {}", chat.error().message());
    return 1;
}
std::println("context window: {} tokens", chat->context_size());
```

```
context window: 131072 tokens
```

That number comes from the model's own configuration, not a constant in the library.

## Streaming

A blocking `send` is fine for a one-shot answer. For anything a person waits on, take the
tokens as they arrive. `stream` accepts any callable, and returning `false` stops
generation early:

```cpp
int tokens = 0;
auto streamed = chat->stream(
    "Count from one to five.", kjarni::Generation::greedy(32),
    [&tokens](std::string_view token) {
        std::print("{}", token);
        return ++tokens < 40;      // false ends generation
    });
```

```
One, two, three, four, five.
```

The callback runs on the generating thread, so keep it cheap: append to a buffer, post to
your UI queue, write to a socket. The early-stop return is not just a convenience. It is
how you implement a stop button without tearing down the model.

## Sampling

`Generation::greedy(n)` always takes the highest-probability token, which makes runs
reproducible and is what you want for extraction or classification-shaped prompts. For
prose, sample instead:

```cpp
kjarni::Generation creative;
creative.temperature = 0.7f;
creative.top_p = 0.95f;
creative.max_new_tokens = 64;

auto reply = chat->send("Describe Reykjavik in one sentence.", creative);
```

```
Reykjavik is the vibrant, eclectic capital of Iceland, known for its colorful
buildings, lively nightlife, and unparalleled natural beauty.
```

`Generation` holds `std::optional` fields, so anything you leave alone keeps the model's own
default rather than a value the library invented. `greedy()` is the one shortcut: it sets
temperature to zero and nothing else.

Greedy decoding on the same prompt gives the same answer every time. That property is worth
more than it sounds when you are writing tests.

## What ends up linked

`ldd` on the built binary shows Kjarni, the C++ runtime, and the parts of glibc every
program already uses: libc, libm, libgcc, libpthread and libdl. That is the whole list.
Nothing is installed on the machine to make it work, and nothing runs beside your process.

For embeddings, classification and reranking from the same library, see
[Local Semantic Search in C++](/blog/cppinference). The same archive covers both: one
`Embedder` for vectors, one `Chat` for generation.

## FAQ

### Which models can it load?

Llama-family instruction models, which includes Llama 3.x, Mistral, Qwen2.5 and Phi. They
are read from HuggingFace safetensors or GGUF as they are, with no conversion step.

### Does it need a GPU?

No. The example above runs on CPU. The GPU path uses Vulkan, Metal or DirectX 12, so AMD,
Intel and Apple Silicon work as they are.

### Can I run more than one conversation?

Yes. Each `Chat` keeps its own history and its own KV cache. Two instances are two
independent conversations, and a single instance keeps its turns across calls to `send`.

### Is the callback thread-safe?

It runs on the thread that called `stream`, one token at a time, so no locking is needed
inside the callback itself. Anything you hand the tokens to needs the usual care.

### How large is the context window?

Whatever the model declares. `llama3.2-1b-instruct` reports 131,072 tokens, and
`context_size()` reads it from the loaded model rather than assuming.