Local LLM Chat in C# without Ollama or an API Key

Running a language model from .NET usually means one of two things: an HTTP call to someone else's GPU, or installing Ollama and talking to a daemon on localhost:11434.

Kjarni is a NuGet package. The model runs inside your process.

using Kjarni;

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);
Console.WriteLine(chat.Send("Explain retrieval-augmented generation in one sentence."));

The weights download once to a local cache, and everything after that is offline.

Install

dotnet add package Kjarni

Native binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the package and are picked automatically by RID. The only runtime dependency is glibc.

Streaming

Waiting for a complete response feels broken. Stream tokens as they are produced:

chat.Stream("Write a haiku about Reykjavík.", token =>
{
    Console.Write(token);
    return true;      // return false to stop generating
});

The callback returns a bool, which is also your cancellation mechanism. Return false and generation stops immediately rather than running to completion and throwing the result away.

Multi-turn conversations

A Chat is stateless between calls. For a conversation that remembers, ask it for one:

var convo = chat.Conversation();

convo.Send("My name is Ólafur.");
Console.WriteLine(convo.Send("What is my name?"));
// Ólafur.

Console.WriteLine(convo.Length);   // 4: two user messages, two replies
convo.Clear();                     // keeps the system prompt

ChatConversation also streams, via the same Stream(message, onToken) shape, and appends both sides to the history for you.

System prompts and modes

using var chat = new Chat(
    "llama3.2-3b-instruct",
    systemPrompt: "You are a terse assistant. Answer in one sentence.",
    mode: ChatMode.Default,      // or Creative, Reasoning
    quiet: true);

Controlling the output

var config = GenerationConfig.Default() with
{
    Temperature = 0.2f,
    MaxNewTokens = 512,
};

chat.Send("Summarise this changelog.", config);

GenerationConfig exposes Temperature, TopK, TopP, MinP, RepetitionPenalty, MaxNewTokens and DoSample, with three presets:

Reach for Greedy() when you are doing extraction or classification with an LLM and want the same answer every time.

Which model?

Run kjarni model list for the full registry. The practical range:

Real numbers for llama3.2-3b-instruct, on a desktop i7-13700 with an RTX A2000. Decode only, so model load and prompt processing are excluded, best of several interleaved runs on an otherwise idle machine:

Bold marks the fastest where there is something to compare against. The gaps marked n/a are not omissions: PyTorch does not read GGUF files, llama.cpp does not read bfloat16 safetensors, and the PyTorch build here is CPU-only. Every engine within a row reads the identical file, so the rows are comparisons and the columns are not.

Model load is a few seconds from a warm page cache, and the context window is 131,072 tokens.

Quantisation is the single biggest lever, and it is worth understanding why. Generating one token reads every weight in the model exactly once, so decode speed is set by how fast weights stream out of memory rather than by arithmetic. Four-bit weights are a quarter the size of bfloat16, so they arrive four times faster.

Kjarni is about 1.4x PyTorch on the same weights, which is the number that matters if you are moving off a Python service. Against llama.cpp on the same quantised file it is within about 8% on CPU. On GPU it is not close, and that is the honest state of things. llama.cpp has years of hand-tuned kernels behind it and kjarni's GPU path is newer. If raw GPU throughput is what you are optimising for, use llama.cpp.

If you want to check any of this yourself:

kjarni generate "The capital of Iceland is" \
  --model llama3.2-3b-instruct --model-path model.gguf \
  --max-tokens 100 --temperature 0

That 128K context is not a typo. It is the model's native window, and you are not paying per token to use it.

What this is not

The limits, plainly.

No tool calling yet. The engine supports it in principle; the C# surface does not expose it. If you need function calling, this is not your library today.

The GPU path is newer than the CPU path. Rather than ask you to take that on trust, here is the same prompt, the same Q4_K_M file, greedy decoding, on both devices and against llama.cpp reading the identical file:

kjarni CPU  Reykjavik, which is also the largest city in the country. It has a
            population of around 120,
kjarni GPU  Reykjavik, which is also the largest city in the country. It has a
            population of around 120,
llama.cpp   Reykjavik, which is also the largest city in the country. Reykjavik
            is a popular tourist destination

Kjarni's two devices are character-identical, which is the property you actually want: switching to the GPU does not change the answer. Against llama.cpp the first sentence matches exactly and then the two part ways.

That parting is normal and worth understanding, because it will happen to you with any two engines. Greedy decoding picks the single highest-scoring token, so when two candidates score within a rounding error of each other, a difference of one part in a million decides the word, and every token afterwards is conditioned on that choice. Independent implementations sum in different orders and diverge sooner or later. Matching text token for token is not the right test of an engine. Producing an equally sensible continuation is.

It is not going to beat llama.cpp on raw throughput. That is not the trade. The trade is that chat, embeddings, classification, reranking and search all come from one dependency-free package you already know how to install.

FAQ

Does this need Ollama installed?

No. There is no daemon, no server, and no localhost port. The model runs in your process.

Does it need Python or PyTorch?

No. Kjarni is a Rust engine compiled to a native library. No Python, no PyTorch, no ONNX Runtime, no CUDA toolkit.

Does it need a GPU?

No. Every code sample above runs on CPU, and the CPU numbers in the table are the ones to plan around. A GPU roughly triples decode speed if you have one, but nothing here requires it and there is no CUDA toolkit to install either way.

Where do the model weights go?

A local cache on first use, reused afterwards. Pre-download with kjarni model download llama3.2-3b-instruct to avoid a long pause on the first call.

Note that Llama models live in gated Hugging Face repositories, so downloading them needs an accepted licence and an HF_TOKEN in your environment. Qwen and Phi are openly downloadable and need no token.

Can I use it in a Docker container?

Yes, with no Python layer, no CUDA base image and no sidecar model server.

Does it implement IChatClient from Microsoft.Extensions.AI?

Yes, in Kjarni.Extensions.AI:

using IChatClient client = new KjarniChatClient("llama3.2-3b-instruct");
var response = await client.GetResponseAsync("Explain RAG in one sentence.");

That gives you streaming, multi-turn history and dependency injection through the standard abstraction, so Kjarni can stand in for any other IChatClient, including inside Semantic Kernel. The Chat API above is Kjarni's own, and is the lower-level option if you don't want the abstraction.

What does it cost?

Nothing. MIT licensed, no API key, no per-token billing.

Try it

dotnet add package Kjarni

Related

• Local IChatClient for Microsoft.Extensions.AI: the same models behind the standard .NET abstraction, so Semantic Kernel and anything else targeting IChatClient works unchanged
• RAG in C# Without a Vector Database: give the model your own documents to answer from
• Local Embeddings for Microsoft.Extensions.AI: the retrieval half of that pipeline

Source and issues: github.com/olafurjohannsson/kjarni