Reranking in C#

Your search works. Someone types a question, twenty results come back, and the one they wanted is at position eleven.

That's not a broken index. It's the gap between recall and precision. Retrieval is built for recall: get the right document somewhere in the top twenty, fast, across millions of candidates. But people read the top three, and a RAG prompt only has room for the top three. Precision at the very top is a different problem, and it needs a different tool.

That tool is a cross-encoder.

using Kjarni;

using var reranker = new Reranker(quiet: true);

var results = reranker.Rerank(
    "how do I cancel my subscription",
    [
        "Our office is open until 5pm on weekdays.",
        "To end your plan, go to Settings and choose Close account.",
        "Cancellation of orders is handled by the warehouse team.",
        "You can update your billing address at any time.",
    ]);

foreach (var r in results)
    Console.WriteLine($"{r.Score:F3}  {r.Document}");

The winner is the sentence that never says "cancel" or "subscription": "To end your plan, go to Settings and choose Close account." The decoy that repeats both words, "Cancellation of orders is handled by the warehouse team", ranks below it. Keyword search gets this exactly backwards.

Install

dotnet add package Kjarni

The cross-encoder is about 88MB and downloads on first use. Everything after that runs on your CPU, offline.

Bi-encoders and cross-encoders

Worth understanding, because it explains both why reranking works and why you can't just use it for everything.

A bi-encoder, which is what ordinary embedding search uses, encodes the query and each document separately into vectors, then compares them. The document vectors are computed once, ahead of time, so querying a million documents is a million cheap vector comparisons. Fast. But query and document never actually meet: the model compresses each into a fixed vector before any comparison happens, and nuance is lost in that compression.

A cross-encoder feeds the query and the document through the model together, as a pair, and outputs a single relevance score. The model sees both texts at once and can weigh the specific words in the query against the specific words in the document. Much more accurate, and much slower, because nothing can be precomputed. Every pair is a fresh forward pass.

So you use both, each where it's strong:

Retrieve broadly, rerank narrowly. Pull 50 candidates with whatever search you already have, then rerank those 50 and keep the best 5.

Reranking what you already have

The most useful thing about a standalone reranker is that it doesn't care where the candidates came from. Elasticsearch, Postgres full-text, a LIKE query, a vector database, a hand-written filter. If you can produce candidate strings, you can rerank them:

// However you already search:
List<Document> candidates = await _elastic.Search(query, size: 50);

using var reranker = new Reranker(quiet: true);

var ranked = reranker.RerankTopK(
    query,
    candidates.Select(c => c.Body).ToArray(),
    k: 5);

// RerankResult.Index points back into the array you passed in,
// so you can recover the original object with its metadata.
var best = ranked.Select(r => candidates[r.Index]).ToList();

RerankTopK is the one to reach for: it scores everything and returns only the best k, which is what you almost always want.

That Index field matters. Reranking gives you back a reordering, not new documents, so you map the indices onto your own objects and keep IDs, URLs, permissions and timestamps intact.

Scoring a single pair

Sometimes you don't want a ranking, you want a number:

using var reranker = new Reranker(quiet: true);

float score = reranker.Score(
    "capital of Iceland",
    "Reykjavik is the capital and largest city of Iceland.");

Console.WriteLine(score);   // 8.55

These are raw logits, not probabilities. They are unbounded and frequently negative, which surprises people the first time they see it. A score of -5 does not mean "no match". Only the ordering is meaningful, and the absolute value only means something relative to other scores from the same model.

Here is the actual range, measured with minilm-l6-v2-cross-encoder:

Two things worth noticing. A direct factual hit scores far above everything else, and total irrelevance clusters around −11, so there is a usable signal for thresholding. But look at the refund pair: "Returns are accepted within 30 days of delivery" is a reasonable answer to "what is the refund window" and still scores −10.92, because the model does not treat returns and refunds as the same thing. A threshold tuned on the first example would discard the second.

That's the lesson: thresholding works, but calibrate it on your own queries and your own documents. No threshold anyone can quote you will transfer to your corpus. Run fifty real queries, look at where the good and bad answers actually fall, and put the cutoff between them.

If you'd rather work in 0 to 1, apply a sigmoid, 1f / (1f + MathF.Exp(-score)), but that is a presentation change, not a calibration. It moves the same ordering onto a friendlier scale.

Built into search

If you're using Kjarni's Searcher, reranking is a constructor argument and a flag. The two-stage retrieval happens for you:

using var searcher = new Searcher(
    model: "minilm-l6-v2",
    rerankerModel: "minilm-l6-v2-cross-encoder",
    quiet: true);

var results = searcher.Search("docs.idx", "rotate the signing key",
    topK: 5,
    rerank: true);

It retrieves a wider candidate set internally, rescores with the cross-encoder, and returns the top 5. See RAG in C# for where this sits in a full pipeline.

What it costs

Reranking is a forward pass per candidate, so cost scales linearly with how many you rerank. That's the entire performance model, and it leads to one rule: rerank tens, not thousands.

Retrieve 50 and rerank 50. Retrieving 1,000 and reranking all of them abandons the reason the two-stage design exists.

If latency is tight, rerank fewer candidates before you reach for a smaller model. Going from 100 to 30 candidates is a much larger saving than any model swap, and usually costs less accuracy.

When it isn't worth it

Reranking is not free and not always the answer.

• Your retrieval returns nothing relevant. Reranking reorders candidates; it cannot invent one. If the right document isn't in the top 50, fix retrieval first: widen the candidate set, or switch to hybrid search so exact identifiers match.
• Your corpus is tiny. With 20 documents total, embed and compare them all directly.
• Your queries are exact lookups. Order numbers and SKUs want an index, not a language model.

The clearest signal that reranking will help: the right answer is reliably in your results, just not at the top. That's precisely the gap it closes.

FAQ

What is a cross-encoder?

A model that takes a query and a document together and outputs a relevance score. Unlike embedding search, which compares two separately-computed vectors, a cross-encoder sees both texts at once. More accurate, and too slow to run over a whole corpus.

Do I need to replace my existing search?

No. Reranking sits on top of whatever you already use: Elasticsearch, Postgres, a vector database, or plain SQL. You hand it the candidates and it reorders them.

How many results should I rerank?

Tens. Retrieve 50 and rerank 50 is a good default. Cost is linear in candidate count.

Does this need Python or an API key?

No. Kjarni is a native library in a NuGet package. No Python, no ONNX Runtime, no cloud call.

Does it work offline?

Yes. The model downloads once on first use and caches locally.

Can I use my own threshold to filter weak results?

Yes. Use Score(query, document) and calibrate a cutoff against your own data. Scores are relative to the model rather than absolute probabilities.

What does it cost?

Nothing. MIT licensed, no API key, no per-query billing.

Try it

dotnet add package Kjarni

Source and issues: github.com/olafurjohannsson/kjarni