Local IChatClient for Microsoft.Extensions.AI

Microsoft.Extensions.AI gave .NET one abstraction for language models. Write against IChatClient, swap providers, leave your application code alone.

Then you look for a provider that runs on your own machine and the list gets short. OpenAI, Azure OpenAI and Anthropic are HTTP calls to somebody else's GPU. Ollama and LM Studio are "local" in the sense that the GPU is yours, but you install a daemon, keep it running, and talk to it over http://localhost:11434. Your app still makes a network call, and your deployment still has a second moving part.

Kjarni is a NuGet package. The model runs inside your process.

using Kjarni.Extensions.AI;
using Microsoft.Extensions.AI;

using IChatClient client = new KjarniChatClient("llama3.2-3b-instruct");

var response = await client.GetResponseAsync("Explain retrieval-augmented generation in one sentence.");
Console.WriteLine(response.Text);

No API key. No daemon. No pip install. No CUDA toolkit. The weights download once to a local cache; everything after that is offline.

Install

dotnet add package Kjarni.Extensions.AI

That pulls in Kjarni (the engine and its native library) along with Microsoft.Extensions.AI.Abstractions. Binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the package and are picked by RID. There is no second install step.

Dependency injection

using Microsoft.Extensions.AI;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddKjarniChatClient("llama3.2-3b-instruct");

var app = builder.Build();

Then inject the standard interface:

app.MapPost("/ask", async (string question, IChatClient client) =>
{
    var response = await client.GetResponseAsync(question);
    return Results.Ok(response.Text);
});

Look at what that handler depends on: IChatClient. Nothing about Kjarni appears in the signature. Point it at Azure OpenAI in production and the handler doesn't change.

AddKjarniChatClient registers a singleton, deliberately: weights load once and are expensive to re-initialize. See Concurrency below for what that means under load, because it is genuinely different from a hosted provider.

Multi-turn conversations

IChatClient is stateless by design: you own the transcript and replay it each call. That is exactly how a web application wants it, since there's no server-side session to keep alive.

List<ChatMessage> conversation =
[
    new(ChatRole.System, "You are a terse assistant. Answer in one sentence."),
    new(ChatRole.User,   "My name is Olafur and I work on inference engines."),
];

var first = await client.GetResponseAsync(conversation);
conversation.Add(new ChatMessage(ChatRole.Assistant, first.Text));

conversation.Add(new ChatMessage(ChatRole.User, "What do I work on?"));
var second = await client.GetResponseAsync(conversation);

Console.WriteLine(second.Text);   // Inference engines.

Append the assistant's reply to the list yourself. Kjarni keeps nothing between calls.

Streaming

Waiting for a whole response feels broken. Stream it:

await foreach (var update in client.GetStreamingResponseAsync("Write a haiku about Reykjavik."))
{
    Console.Write(update.Text);
}

Each ChatResponseUpdate carries a fragment as the model produces it. Cancellation works the way you'd expect. Pass a CancellationToken and generation stops at the next token rather than running to completion and discarding the result:

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));

try
{
    await foreach (var update in client.GetStreamingResponseAsync(conversation, null, cts.Token))
        Console.Write(update.Text);
}
catch (OperationCanceledException)
{
    Console.WriteLine("\n[stopped]");
}

Semantic Kernel

Semantic Kernel builds on these abstractions, so registration is the same:

var builder = Kernel.CreateBuilder();
builder.Services.AddKjarniChatClient("llama3.2-3b-instruct");

var kernel = builder.Build();

Any SK component resolving IChatClient now runs against a local model. Combined with local embeddings, you can run a Semantic Kernel application with no cloud dependency at all.

Options that map, and options that don't

ChatOptions is honoured where Kjarni's sampler has an equivalent:

var options = new ChatOptions
{
    Temperature = 0f,        // 0 selects greedy decoding
    TopP = 0.95f,
    TopK = 40,
    MaxOutputTokens = 512,
};

var response = await client.GetResponseAsync("Extract the company name: ...", options);

Temperature = 0 is worth knowing about: it switches to greedy decoding, so the same prompt returns the same answer every time. That's what you want for extraction and classification work, where a different answer on each call is a bug rather than variety.

Everything else is deliberately explicit rather than quietly dropped:

Throwing on Tools is a choice. Accepting the option and ignoring it would leave you believing your functions had been offered to the model when they never were, a bug you'd find in production rather than at the call site. If you need function calling, this is not your library today.

FrequencyPenalty is ignored rather than mapped onto Kjarni's repetition penalty, because the two are different things and silently substituting one would change your output in ways you didn't ask for.

Concurrency

This is the part that differs most from a hosted provider, and it's worth being straight about.

Generation is compute-bound, and the native model is not re-entrant, so calls are serialized per instance. Two simultaneous requests queue; they do not overlap. A request waiting on the gate is waiting for a real CPU, not for a socket.

That has a concrete consequence: a single instance is not a throughput story. It is excellent for a desktop application, a CLI, a background worker, a batch job, or an internal tool with a handful of users. For a high-traffic public endpoint, you want either a queue in front of it or a hosted provider. Because you coded against IChatClient, switching is a registration change.

Which model?

Run kjarni model list for the registry. The practical range:

builder.Services.AddKjarniChatClient(
    model: "llama3.2-3b-instruct",
    systemPrompt: "You are a terse assistant.",
    device: "cpu",     // or "gpu"
    quiet: true);

Wrapping an existing Chat

If you already hold a Kjarni Chat, because you also use the lower-level API, wrap it instead of loading the weights twice:

using Kjarni;
using Microsoft.Extensions.AI;

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);
var client = chat.AsChatClient();

AsChatClient and AddKjarniChatClient live in the Microsoft.Extensions.AI namespace, so they appear alongside the abstractions themselves with no third using to guess at.

By default the client borrows the chat: disposing the client leaves your Chat alive, since you created it. Pass ownsChat: true to hand over ownership.

Going underneath the abstraction

GetService unwraps, when you need something IChatClient doesn't model:

var chat = client.GetService(typeof(Chat)) as Chat;
Console.WriteLine(chat?.ContextSize);   // 131072 for llama3.2-3b-instruct

var metadata = client.GetService(typeof(ChatClientMetadata)) as ChatClientMetadata;
Console.WriteLine(metadata?.ProviderName);   // kjarni

That 128K context window is the model's native size, and you aren't paying per token to use it.

FAQ

Does this need Ollama installed?

No. There is no daemon, no server, and no localhost port. The model runs in your process.

Does it need Python or PyTorch?

No. Kjarni is a Rust engine compiled to a native library. No Python, no PyTorch, no ONNX Runtime, no CUDA toolkit.

Does it need a GPU?

No. Everything above runs on CPU. There is a device: "gpu" option using WebGPU (Vulkan on Linux, DX12 or Vulkan on Windows, Metal on macOS), but CPU is the well-tested path for decoder models.

Does it support tool calling / function calling?

Not yet. ChatOptions.Tools throws rather than silently ignoring your functions. The engine supports it in principle; the surface doesn't expose it.

Does it implement IEmbeddingGenerator too?

Yes. See Local Embeddings for Microsoft.Extensions.AI. The same package provides both, so a Semantic Kernel RAG pipeline can run entirely locally.

Can I use it in a Docker container?

Yes, with no Python layer, no CUDA base image and no sidecar model server. The native binary is in the NuGet package.

What does it cost?

Nothing. MIT licensed, no API key, no per-token billing.

Try it

dotnet add package Kjarni.Extensions.AI

Source and issues: github.com/olafurjohannsson/kjarni