RAG in C# Without a Vector Database

The standard .NET recipe for retrieval-augmented generation has four bills attached to it: a vector database to host, an embedding API to call, a chat API to call, and usually a Python service somewhere in the middle doing the parts .NET has no library for.

Every one of those is a network hop, a key to rotate, and a thing that can be down at 3am.

Here is the whole pipeline with one NuGet package and no network:

using Kjarni;

// 1. Index a folder of documents.
using var indexer = new Indexer(model: "minilm-l6-v2", quiet: true);
indexer.Create("docs.idx", ["./documentation"]);

// 2. Retrieve what's relevant.
using var searcher = new Searcher(model: "minilm-l6-v2", quiet: true);
var hits = searcher.Search("docs.idx", "How do I rotate the signing key?", topK: 4);

// 3. Answer from what you retrieved.
using var chat = new Chat("llama3.2-3b-instruct", quiet: true);

var context = string.Join("\n\n", hits.Select(h => h.Text));
var answer = chat.Send($"""
    Answer the question using only the context below.
    If the context does not contain the answer, say so.

    Context:
    {context}

    Question: How do I rotate the signing key?
    """);

Console.WriteLine(answer);

That is a complete RAG system. No Postgres with pgvector, no Pinecone, no Qdrant, no OpenAI key, no Ollama daemon, no Python.

Install

dotnet add package Kjarni

Native binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the package. Models download on first use and cache locally. After that the whole pipeline runs offline.

Step 1: Indexing

Indexer walks a directory, chunks what it finds, embeds each chunk, and writes an index file:

using var indexer = new Indexer(
    model: "minilm-l6-v2",
    chunkSize: 512,
    chunkOverlap: 50,
    quiet: true);

var stats = indexer.Create("docs.idx", ["./documentation"], force: true);

Console.WriteLine($"{stats.DocumentsIndexed} documents, {stats.ChunksCreated} chunks");
Console.WriteLine($"{stats.SizeBytes / 1024}KB in {stats.ElapsedMs}ms");

The index is a file. You can commit it, ship it inside a container image, or drop it on a network share. There's no server to run and nothing to keep in sync.

Chunking is the parameter that actually matters for answer quality. chunkSize: 512 with chunkOverlap: 50 is a reasonable default: large enough to hold a coherent thought, with enough overlap that a sentence spanning a boundary still appears intact in one chunk.

For a long-running index job, report progress and allow cancellation:

using var cancel = new CancelToken();

var stats = indexer.Create("docs.idx", ["./documentation"],
    onProgress: p => Console.WriteLine($"{p.Current}/{p.Total}  {p.Message}"),
    cancelToken: cancel);

Incremental updates don't need a full rebuild:

int added = indexer.Add("docs.idx", ["./documentation/new-page.md"]);

Step 2: Retrieval

Pure vector search is the default choice in most RAG tutorials and it is often the wrong one. Embeddings are good at meaning and bad at exact tokens: error codes, function names, SKUs, surnames. Ask about ERR_SIGNING_KEY_EXPIRED and a semantic index will cheerfully return five chunks about authentication in general.

Kjarni indexes for both and defaults to hybrid:

using var searcher = new Searcher(model: "minilm-l6-v2", quiet: true);

var results = searcher.Search("docs.idx", "rotate the signing key",
    mode: SearchMode.Hybrid,   // Keyword | Semantic | Hybrid
    topK: 8);

foreach (var r in results)
    Console.WriteLine($"{r.Score:F3}  {r.Text[..Math.Min(80, r.Text.Length)]}");

Use Hybrid unless you have a specific reason not to. It is the default for this reason.

Step 3: Reranking

Retrieval optimises for recall: get the right chunk somewhere in the top 20. Generation needs precision: the right chunk in the top 3, because that's all you can fit in the prompt.

A cross-encoder closes that gap. Bi-encoders embed the query and document separately and compare vectors, which is fast and lossy. A cross-encoder reads query and document together and scores the pair directly. Far more accurate, and far too slow to run over a whole corpus. So you use each for what it's good at: retrieve broadly, then rerank narrowly.

using var searcher = new Searcher(
    model: "minilm-l6-v2",
    rerankerModel: "minilm-l6-v2-cross-encoder",
    quiet: true);

var results = searcher.Search("docs.idx", "rotate the signing key",
    topK: 4,
    rerank: true);

That retrieves a wider candidate set, rescores it with the cross-encoder, and returns the best four. In a RAG pipeline this is usually the single highest-leverage change you can make to answer quality, because the model can only reason about what you put in front of it.

Step 4: Generation

Now hand the retrieved context to a local model. Two details matter more than the rest.

Ground the model explicitly. Tell it to answer from the context and to admit when the context doesn't cover the question. This is most of your defence against confident invention:

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);

var context = string.Join("\n\n---\n\n", results.Select(r => r.Text));

var prompt = $"""
    Answer the question using only the context below.
    If the context does not contain the answer, say "I don't know based on the provided documents."

    Context:
    {context}

    Question: {question}
    """;

var answer = chat.Send(prompt, GenerationConfig.Greedy());

Use Greedy(). RAG is an extraction task, not a creative one. The same question against the same documents should produce the same answer; a temperature above zero turns that into a dice roll.

Stream it if a human is waiting:

chat.Stream(prompt, GenerationConfig.Greedy(), token =>
{
    Console.Write(token);
    return true;
});

Citations

Answers without sources are hard to trust and impossible to audit. The search results carry metadata, so you can show where each answer came from:

foreach (var r in results)
{
    var source = r.Metadata.TryGetValue("source", out var s) ? s : "unknown";
    Console.WriteLine($"  [{r.Score:F3}] {source}");
}

Threading identifiers into the prompt and asking the model to cite them works, but verifying the citation against the retrieved set afterwards is what actually stops fabricated sources.

Putting it together

A small, honest RAG service:

using Kjarni;

public sealed class DocumentQa : IDisposable
{
    private readonly Searcher _searcher;
    private readonly Chat _chat;
    private readonly string _index;

    public DocumentQa(string index)
    {
        _index = index;
        _searcher = new Searcher(
            model: "minilm-l6-v2",
            rerankerModel: "minilm-l6-v2-cross-encoder",
            quiet: true);
        _chat = new Chat("llama3.2-3b-instruct", quiet: true);
    }

    public (string Answer, IReadOnlyList<SearchResult> Sources) Ask(string question)
    {
        var hits = _searcher.Search(_index, question, topK: 4, rerank: true);

        if (hits.Count == 0)
            return ("No relevant documents found.", hits);

        var context = string.Join("\n\n---\n\n", hits.Select(h => h.Text));
        var answer = _chat.Send($"""
            Answer using only the context below. If it does not contain the
            answer, say you don't know.

            Context:
            {context}

            Question: {question}
            """, GenerationConfig.Greedy());

        return (answer, hits);
    }

    public void Dispose()
    {
        _searcher.Dispose();
        _chat.Dispose();
    }
}

Both models load once and are reused. Register it as a singleton, but read the next section before you put it behind a public endpoint.

What this is not

It is not a throughput story. Generation is compute-bound and the model is not re-entrant, so calls serialize. This shape is excellent for desktop apps, CLIs, background workers, batch jobs and internal tools. For a high-traffic public endpoint you want a queue in front of it, or a hosted model for the generation step while keeping retrieval local.

It is not going to beat a dedicated vector database at scale. A file-backed index is the right tool for thousands to low millions of chunks. At a hundred million, you want the database.

Retrieval quality is still your problem. No engine saves you from badly chunked documents. If answers are poor, look at what Search returned before you look at the model. In practice the retrieval step is wrong far more often than the generation step.

Why local RAG at all

The pitch isn't that local is cheaper, though it is. It's that some data cannot leave the building: legal, healthcare, defence, finance, anything under a data residency clause. For those, the hosted-API path isn't expensive, it's closed. A pipeline that runs entirely inside your process is the difference between shipping the feature and not shipping it.

And there is the ordinary case: it works on a plane, in CI, in an air-gapped network, and in a container with no egress.

FAQ

Do I need a vector database?

No. The index is a file on disk. For thousands to low millions of chunks that is faster to operate and faster to query than a database you have to host.

Do I need an OpenAI API key?

No. Embeddings, reranking and generation all run locally. There is no API key anywhere in this pipeline.

Does this need Python?

No. Kjarni is a Rust engine shipped as a native library inside the NuGet package.

Does it work with Semantic Kernel?

Yes. Kjarni.Extensions.AI implements both IEmbeddingGenerator and IChatClient, so a Semantic Kernel pipeline can run against local inference end to end.

How large an index can it handle?

Comfortably into the millions of chunks. Beyond that, a dedicated vector database earns its keep.

Can I update the index without rebuilding it?

Yes. indexer.Add(indexPath, paths) adds documents to an existing index.

What does it cost?

Nothing. MIT licensed, no API key, no per-token billing, no database to host.

Try it

dotnet add package Kjarni

Source and issues: github.com/olafurjohannsson/kjarni