A local IChatClient for Microsoft.Extensions.AI

Microsoft.Extensions.AI gave .NET one abstraction for language models, but the providers that implement it are mostly HTTP calls to somebody else's GPU. Ollama and LM Studio are local in the sense that the GPU is yours, but you still install a daemon, keep it running, and talk to it over http://localhost:11434.

This one loads the weights into your process:

using Kjarni.Extensions.AI;
using Microsoft.Extensions.AI;

using IChatClient client = new KjarniChatClient("llama3.2-3b-instruct");

var response = await client.GetResponseAsync("Explain RAG in one sentence.");
Console.WriteLine(response.Text);

dotnet add package Kjarni.Extensions.AI

That pulls in the engine and its native library along with Microsoft.Extensions.AI.Abstractions. Binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the package and are picked by RID, so there is no second install step. Weights download once to a local cache and everything after that is offline.

Dependency injection

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddKjarniChatClient("llama3.2-3b-instruct");
var app = builder.Build();

app.MapPost("/ask", async (string question, IChatClient client) =>
{
    var response = await client.GetResponseAsync(question);
    return Results.Ok(response.Text);
});

Nothing about Kjarni appears in that handler's signature. Point it at Azure OpenAI in production and the handler does not change.

AddKjarniChatClient registers a singleton, deliberately: weights load once and are expensive to re-initialise. That has consequences under load, covered below.

Multi-turn

IChatClient is stateless by design, so you own the transcript and replay it each call. Kjarni keeps nothing between calls.

List<ChatMessage> conversation =
[
    new(ChatRole.System, "You are a terse assistant. Answer in one sentence."),
    new(ChatRole.User,   "My name is Olafur and I work on inference engines."),
];

var first = await client.GetResponseAsync(conversation);
conversation.Add(new ChatMessage(ChatRole.Assistant, first.Text));

conversation.Add(new ChatMessage(ChatRole.User, "What do I work on?"));
var second = await client.GetResponseAsync(conversation);

Console.WriteLine(second.Text);   // Inference engines.

Streaming and cancellation

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));

await foreach (var update in client.GetStreamingResponseAsync(conversation, null, cts.Token))
    Console.Write(update.Text);

Each ChatResponseUpdate carries a fragment as the model produces it. Cancellation stops generation at the next token rather than running to completion and discarding the result.

Semantic Kernel

Registration is the same, since SK builds on the same abstractions:

var builder = Kernel.CreateBuilder();
builder.Services.AddKjarniChatClient("llama3.2-3b-instruct");
var kernel = builder.Build();

Any SK component resolving IChatClient now runs against a local model. Combined with local embeddings the whole pipeline stays on your hardware.

Options that map, and options that don't

var options = new ChatOptions
{
    Temperature = 0f,        // greedy decoding
    TopP = 0.95f,
    TopK = 40,
    MaxOutputTokens = 512,
};

Temperature = 0 switches to greedy decoding, so the same prompt returns the same answer every time. That is what you want for extraction and classification, where a different answer per call is a bug rather than variety.

The rest is explicit rather than quietly dropped:

Throwing on Tools is a choice. Accepting the option and ignoring it would leave you believing your functions had been offered to the model when they never were, which is a bug you find in production rather than at the call site. If you need function calling, this is not your library today.

FrequencyPenalty is ignored rather than mapped onto the repetition penalty, because the two are different things and substituting one for the other would change your output in ways you did not ask for.

Concurrency

Generation is compute bound and the native model is not re-entrant, so calls are serialised per instance. Two simultaneous requests queue rather than overlap, and a request waiting on the gate is waiting for a real CPU, not a socket.

So a single instance is not a throughput story. It is good for a desktop app, a CLI, a background worker, a batch job, or an internal tool with a handful of users. For a high traffic public endpoint you want a queue in front of it or a hosted provider, and because you coded against IChatClient that is a registration change.

Wrapping a Chat you already have

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);
var client = chat.AsChatClient();

AsChatClient and AddKjarniChatClient live in the Microsoft.Extensions.AI namespace, so they appear alongside the abstractions with no third using to guess at. The client borrows the chat by default, so disposing it leaves your Chat alive. Pass ownsChat: true to hand over ownership.

Going underneath the abstraction

var chat = client.GetService(typeof(Chat)) as Chat;
Console.WriteLine(chat?.ContextSize);        // 131072

var metadata = client.GetService(typeof(ChatClientMetadata)) as ChatClientMetadata;
Console.WriteLine(metadata?.ProviderName);   // kjarni

Two questions specific to this package

Does it support tool calling? Not yet. ChatOptions.Tools throws rather than silently ignoring your functions. The engine supports it in principle, the surface does not expose it.

Does it implement IEmbeddingGenerator too? Yes, from the same package, so a Semantic Kernel RAG pipeline runs entirely locally. See Local Embeddings for Microsoft.Extensions.AI.

For model choice, hardware expectations and how this compares with running a daemon, see Local LLM chat in C#.

dotnet add package Kjarni.Extensions.AI

Source and issues: github.com/olafurjohannsson/kjarni