Local LLM Chat in C#

Running a language model from .NET usually means one of two things: an HTTP call to someone else's GPU, or installing Ollama and talking to a daemon on localhost:11434.

Kjarni is a NuGet package. The model runs inside your process.

using Kjarni;

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);
Console.WriteLine(chat.Send("Explain retrieval-augmented generation in one sentence."));

No API key. No daemon. No pip install. No CUDA toolkit. The weights download once to a local cache, and everything after that is offline.

Install

dotnet add package Kjarni

Native binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the package and are picked automatically by RID. The only runtime dependency is glibc.

Streaming

Waiting for a complete response feels broken. Stream tokens as they are produced:

chat.Stream("Write a haiku about Reykjavík.", token =>
{
    Console.Write(token);
    return true;      // return false to stop generating
});

The callback returns a bool, which is also your cancellation mechanism. Return false and generation stops immediately rather than running to completion and throwing the result away.

Multi-turn conversations

A Chat is stateless between calls. For a conversation that remembers, ask it for one:

var convo = chat.Conversation();

convo.Send("My name is Ólafur.");
Console.WriteLine(convo.Send("What is my name?"));
// Ólafur.

Console.WriteLine(convo.Length);   // 4: two user messages, two replies
convo.Clear();                     // keeps the system prompt

ChatConversation also streams, via the same Stream(message, onToken) shape, and appends both sides to the history for you.

System prompts and modes

using var chat = new Chat(
    "llama3.2-3b-instruct",
    systemPrompt: "You are a terse assistant. Answer in one sentence.",
    mode: ChatMode.Default,      // or Creative, Reasoning
    quiet: true);

Controlling the output

var config = GenerationConfig.Default() with
{
    Temperature = 0.2f,
    MaxNewTokens = 512,
};

chat.Send("Summarise this changelog.", config);

GenerationConfig exposes Temperature, TopK, TopP, MinP, RepetitionPenalty, MaxNewTokens and DoSample, with three presets:

Reach for Greedy() when you are doing extraction or classification with an LLM and want the same answer every time.

Which model?

Run kjarni model list for the full registry. The practical range:

Real numbers from a desktop CPU with llama3.2-3b-instruct:

model load        2,615 ms
short response    5,607 ms
context window  131,072 tokens

That 128K context is not a typo. It is the model's native window, and you are not paying per token to use it.

What this is not

Worth being straight about the limits.

No tool calling yet. The engine supports it in principle; the C# surface does not expose it. If you need function calling, this is not your library today.

CPU is the tested path. There is a device: "gpu" option and it works well for embeddings and classification, but decoder models on GPU are less well covered, and one architecture currently produces incorrect output. Use CPU for chat unless you have verified your specific model.

It is not going to beat llama.cpp on raw throughput. That is not the trade. The trade is that chat, embeddings, classification, reranking and search all come from one dependency-free package you already know how to install.

FAQ

Does this need Ollama installed?

No. There is no daemon, no server, and no localhost port. The model runs in your process.

Does it need Python or PyTorch?

No. Kjarni is a Rust engine compiled to a native library. No Python, no PyTorch, no ONNX Runtime, no CUDA toolkit.

Does it need a GPU?

No. Everything above ran on CPU.

Where do the model weights go?

A local cache on first use, reused afterwards. Pre-download with kjarni model download llama3.2-3b-instruct to avoid a long pause on the first call.

Note that Llama models live in gated Hugging Face repositories, so downloading them needs an accepted licence and an HF_TOKEN in your environment. Qwen and Phi are openly downloadable and need no token.

Can I use it in a Docker container?

Yes, with no Python layer, no CUDA base image and no sidecar model server.

Does it implement IChatClient from Microsoft.Extensions.AI?

Yes, in Kjarni.Extensions.AI 0.2.0:

using IChatClient client = new KjarniChatClient("llama3.2-3b-instruct");
var response = await client.GetResponseAsync("Explain RAG in one sentence.");

That gives you streaming, multi-turn history and dependency injection through the standard abstraction, so Kjarni can stand in for any other IChatClient, including inside Semantic Kernel. The Chat API above is Kjarni's own, and is the lower-level option if you don't want the abstraction.

What does it cost?

Nothing. MIT licensed, no API key, no per-token billing.

Try it

dotnet add package Kjarni

Source and issues: github.com/olafurjohannsson/kjarni