# Local LLM Chat in C# without Ollama or an API Key

> Run Llama, Mistral, Qwen or Phi locally from C#. One NuGet package, loaded in-process and answering offline. Streaming, multi-turn conversations and sampling control in .NET.

Source: https://kjarni.ai/blog/localchat/ (published 2026-08-26)

Running a language model from .NET usually means one of two things: an HTTP call to
someone else's GPU, or installing Ollama and talking to a daemon on
`localhost:11434`.

Kjarni is a NuGet package. The model runs inside your process.

```cs
using Kjarni;

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);
Console.WriteLine(chat.Send("Explain retrieval-augmented generation in one sentence."));
```

The weights download
once to a local cache, and everything after that is offline.

## Install

```bash
dotnet add package Kjarni
```

Native binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the
package and are picked automatically by RID. The only runtime dependency is glibc.

## Streaming

Waiting for a complete response feels broken. Stream tokens as they are produced:

```cs
chat.Stream("Write a haiku about Reykjavík.", token =>
{
    Console.Write(token);
    return true;      // return false to stop generating
});
```

The callback returns a `bool`, which is also your cancellation mechanism. Return
`false` and generation stops immediately rather than running to completion and
throwing the result away.

## Multi-turn conversations

A `Chat` is stateless between calls. For a conversation that remembers, ask it for
one:

```cs
var convo = chat.Conversation();

convo.Send("My name is Ólafur.");
Console.WriteLine(convo.Send("What is my name?"));
// Your name is Ólafur.

Console.WriteLine(convo.Length);   // 4: two user messages, two replies
convo.Clear();                     // keeps the system prompt
```

`ChatConversation` also streams, via the same `Stream(message, onToken)` shape, and
appends both sides to the history for you.

## System prompts and modes

```cs
using var chat = new Chat(
    "llama3.2-3b-instruct",
    systemPrompt: "You are a terse assistant. Answer in one sentence.",
    mode: ChatMode.Default,      // or Creative, Reasoning
    quiet: true);
```

## Controlling the output

```cs
var config = GenerationConfig.Default() with
{
    Temperature = 0.2f,
    MaxNewTokens = 512,
};

chat.Send("Summarise this changelog.", config);
```

`GenerationConfig` exposes `Temperature`, `TopK`, `TopP`, `MinP`,
`RepetitionPenalty`, `MaxNewTokens` and `DoSample`, with three presets:

| Preset | Use it for |
|---|---|
| `GenerationConfig.Default()` | model defaults |
| `GenerationConfig.Greedy()` | deterministic output, `Temperature = 0` |
| `GenerationConfig.Creative()` | higher temperature, more variation |

Reach for `Greedy()` when you are doing extraction or classification with an LLM and
want the same answer every time.

## Which model?

Run `kjarni model list` for the full registry. The practical range:

| Model | Good for |
|---|---|
| `qwen2.5-0.5b-instruct` | smallest; simple structured tasks |
| `llama3.2-1b-instruct` | fast on modest hardware |
| **`llama3.2-3b-instruct`** | the sweet spot on CPU |
| `phi3.5-mini` | reasoning-leaning, 3.8B |
| `mistral-7b` | stronger, needs more RAM |
| `deepseek-r1-8b` | reasoning, largest |

Real numbers for `llama3.2-3b-instruct`, on a desktop i7-13700 with an RTX A2000.
Decode only, so model load and prompt processing are excluded, best of several
interleaved runs on an otherwise idle machine:

| Weights | Device | kjarni | PyTorch | llama.cpp |
|---|---|---|---|---|
| bfloat16 | CPU | **3.8** | 2.4 | n/a |
| bfloat16 | GPU | 25.4 | n/a | n/a |
| Q4_K_M | CPU | 6.8 | n/a | **7.9** |
| Q4_K_M | GPU | 46.4 | n/a | **68.5** |

Bold marks the fastest where there is something to compare against.

The bfloat16 CPU row was measured in October 2026 and the rest in November. The gaps marked n/a
are not omissions: PyTorch does not read GGUF files, llama.cpp does not read bfloat16
safetensors, and the PyTorch build here is CPU-only. Every engine within a row reads the
identical file, so the rows are comparisons and the columns are not.

Model load is a few seconds from a warm page cache, and the context window is 131,072
tokens.

Quantisation is the single biggest lever, and it is worth understanding why. Generating
one token reads every weight in the model exactly once, so decode speed is set by how
fast weights stream out of memory rather than by arithmetic. Four-bit weights are a
quarter the size of bfloat16, so they arrive four times faster.

Kjarni is about 1.6x PyTorch on the same weights, which is the number that matters if you
are moving off a Python service. Against llama.cpp on the same quantised file it is about 14% behind on CPU. On GPU
llama.cpp is about 1.5x faster over Vulkan, and further ahead with CUDA. It has years of
hand-tuned kernels behind it and kjarni's GPU path is newer. If raw throughput is what
you are optimising for, use llama.cpp.

If you want to check any of this yourself:

```bash
kjarni generate "The capital of Iceland is" \
  --model llama3.2-3b-instruct --model-path model.gguf \
  --max-tokens 100 --temperature 0 --repetition-penalty 1.0
```

The command line applies a repetition penalty of 1.1 unless told otherwise, so
`--repetition-penalty 1.0` is what makes this plain greedy decoding, the same as
llama.cpp's defaults.

That 128K context is not a typo. It is the model's native window, and you are not
paying per token to use it.

## What this is not

The limits, plainly.

**No tool calling yet.** The engine supports it in principle; the C# surface does
not expose it. If you need function calling, this is not your library today.

**The GPU path is newer than the CPU path.** Rather than ask you to take that on
trust, here is the same prompt, the same Q4_K_M file, greedy decoding, on both devices
and against llama.cpp reading the identical file:

```
kjarni CPU  Reykjavik, which is also the largest city in the country. Reykjavik is a
            popular tourist destination known for its vibrant cultural scene, stunning
kjarni GPU  Reykjavik, which is also the largest city in the country. Reykjavik is a
            popular tourist destination known for its vibrant cultural scene, historic
llama.cpp   Reykjavik, which is also the largest city in the country. Reykjavik is a
            popular tourist destination known for its vibrant cultural scene, historic
```

All three agree for the first twenty-odd tokens. Kjarni's GPU output then stays with
llama.cpp for another sentence and a half before the two part ways.

That parting is normal for quantised weights, and it will happen with any two engines.
Each one dequantises the four-bit blocks and sums the products in its own order, so the
scores differ in the last few bits. Greedy decoding picks the single highest-scoring
token, and when two candidates score within that rounding of each other, the difference
decides the word and every token afterwards is conditioned on it. Between quantised
engines, matching text token for token is not the right test. Producing an equally
sensible continuation is.

At full precision the test is fair, and Kjarni passes it. On the original bfloat16
weights its greedy output matches Hugging Face transformers token for token: on this
prompt for all 100 tokens, and on every chat prompt we test with across Qwen 2.5 0.5B
and 1.5B, Llama 3.2 1B and 3B, Phi-3.5, Mistral 7B and DeepSeek R1 8B.

**It is not going to beat llama.cpp on GPU throughput.** That is not the trade.
The trade is that chat, embeddings, classification, reranking and search all come
from one dependency-free package you already know how to install.

## FAQ

### Does this need Ollama installed?

No. There is no daemon, no server, and no localhost port. The model runs in your
process.

### Does it need Python or PyTorch?

No. The NuGet package ships a native library. It runs inside your process, and the GPU
path uses Vulkan, Metal or DirectX 12, so AMD, Intel and Apple Silicon work as they are.

### Does it need a GPU?

No. Every code sample above runs on CPU, and the CPU numbers in the table are the
ones to plan around. A GPU makes decoding nearly five times faster on bfloat16 weights
and two and a half times faster on Q4_K_M, but nothing here requires it and there is no CUDA toolkit
to install either way.

### Where do the model weights go?

A local cache on first use, reused afterwards. Pre-download with
`kjarni model download llama3.2-3b-instruct` to avoid a long pause on the first call.

Note that Llama models live in gated Hugging Face repositories, so downloading them
needs an accepted licence and an `HF_TOKEN` in your environment. Qwen and Phi are
openly downloadable and need no token.

### Can I use it in a Docker container?

Yes. A plain .NET runtime image is enough: the library links only against glibc, and there
is no separate model server to run beside it.

### Does it implement IChatClient from Microsoft.Extensions.AI?

Yes, in [Kjarni.Extensions.AI](https://www.nuget.org/packages/Kjarni.Extensions.AI):

```cs
using IChatClient client = new KjarniChatClient("llama3.2-3b-instruct");
var response = await client.GetResponseAsync("Explain RAG (retrieval-augmented generation) in one sentence.");
```

That gives you streaming, multi-turn history and dependency injection through the
standard abstraction, so Kjarni can stand in for any other `IChatClient`, including
inside Semantic Kernel. The `Chat` API above is Kjarni's own, and is the lower-level
option if you don't want the abstraction.

### What does it cost?

Nothing. MIT or Apache-2.0 licensed, no API key, no per-token billing.

## Try it

```bash
dotnet add package Kjarni
```

## Related

- [Local IChatClient for Microsoft.Extensions.AI](/blog/ichatclient/): the same models behind the standard .NET abstraction, so Semantic Kernel and anything else targeting `IChatClient` works unchanged
- [RAG in C# Without a Vector Database](/blog/ragincsharp/): give the model your own documents to answer from
- [Local Embeddings for Microsoft.Extensions.AI](/blog/embeddinggenerator/): the retrieval half of that pipeline

Source and issues: [github.com/olafurjohannsson/kjarni](https://github.com/olafurjohannsson/kjarni)
