Local Semantic Search in Python

Most Python search examples stop at cosine similarity over a list. That works until the list stops fitting in memory, or until someone searches for an exact product code and discovers that embeddings do not do exact.

Kjarni ships an index instead: BM25 and vectors in the same store, on disk, queryable three ways. The engine is Rust, the package is pip install kjarni, and it has no dependencies.

from kjarni import Embedder

e = Embedder("minilm-l6-v2")
print(e.dim)                                     # 384
print(e.similarity("capital of Iceland",
                   "Reykjavik is the capital of Iceland"))

384
0.8984

numpy is imported on demand by encode_batch and falls back to lists when it is absent, so the base install pulls in nothing at all.

Indexing a directory

from kjarni import Indexer

indexer = Indexer(model="minilm-l6-v2")
stats = indexer.create("my_index", ["docs/"])
print(stats)

IndexStats(documents_indexed=4, chunks_created=4, dimension=384,
           size_bytes=11602, files_processed=4, files_skipped=0, elapsed_ms=27)

The index is a directory on disk. Nothing is hosted, nothing listens on a port, and the next process to open it just reads it.

Files are discovered, split into chunks and embedded in one pass. chunk_size, chunk_overlap, extensions and exclude_patterns are constructor arguments, and long runs take an on_progress callback and a CancelToken so a user can stop an import.

Three ways to search

from kjarni import Searcher, SearchMode

s = Searcher(model="minilm-l6-v2")
hits = s.search("my_index", "Where do most Icelanders live?",
                mode=SearchMode.SEMANTIC, top_k=3)
for h in hits:
    print(f"{h.score:8.4f}  {h.text[:50]}")

The interesting part is what the three modes actually return for that query, over four short documents about Iceland, Norway, the Icelandic krona, and Python:

KEYWORD      1.0950  norway.txt
SEMANTIC     0.6064  iceland.txt     0.4241  currency.txt   0.3680  norway.txt
HYBRID       0.0323  norway.txt      0.0164  iceland.txt    0.0161  currency.txt

Three different scales, because they are three different things. BM25 scores term overlap. Semantic scores cosine distance between embeddings. Hybrid fuses the two rankings with reciprocal rank fusion, which produces small numbers by construction and is comparable only against itself.

Where hybrid goes wrong

Look at that table again. Semantic gets it right and hybrid gets it wrong.

BM25 returned exactly one document, and it was the wrong one. Remove the word "most" from the query and it returns nothing at all:

'Where do most Icelanders live?'   →  norway.txt 1.095
'Where do Icelanders live?'        →  (no matches)
'Iceland population'               →  iceland.txt 1.761, currency.txt 0.717

The only term BM25 matched was "most", which appears in "the most populous city in the country". "Icelanders" never matches "Iceland", because the tokenizer does not stem one to the other. BM25 is working correctly, and it is working correctly on a word that carries no meaning for this query.

Reciprocal rank fusion does not know that. It sees a document ranked first by one retriever and promotes it. One noisy match in the keyword channel is enough to outrank the document that actually answers the question.

This is worth knowing before you reach for hybrid search by default. It is usually better than either half, and it inherits the failure modes of both.

What fixes it

A cross-encoder reads the query and the document together rather than comparing two vectors computed separately, so it can tell that a document about Norway does not answer a question about Iceland:

s = Searcher(model="minilm-l6-v2",
             rerank_model="minilm-l6-v2-cross-encoder")

hits = s.search("my_index", "Where do most Icelanders live?",
                mode=SearchMode.HYBRID, top_k=3, rerank=True)

 3.6761  iceland.txt
-7.0353  currency.txt
-9.6460  norway.txt

Right answer first, and the gap is not subtle. Those are raw logits, the same numbers a cross-encoder produces under PyTorch: negative is normal, and only the ordering carries meaning. Pass them through a sigmoid if you need something between zero and one.

The cost is a second model pass over the candidates, which is why it runs on the top few rather than the whole index. Retrieve broadly, rerank narrowly.

Note that once a rerank_model is configured, reranking is on by default for every search. That is convenient and it also means the scores you see are the cross-encoder's, not the retriever's, in every mode. Pass rerank=False when you want to see what retrieval alone did.

Reranking on its own

The reranker is usable without an index, over any list of strings you already have:

from kjarni import Reranker

r = Reranker("minilm-l6-v2-cross-encoder")
for hit in r.rerank_top_k("What is the capital of Iceland?", documents, 3):
    print(f"{hit.score:8.3f}  {hit.document}")

   8.342  Reykjavik is the capital and largest city of Iceland.
  -5.096  Oslo is the capital of Norway.
 -11.136  Python is a high-level programming language.

RerankResult carries index, score and document, so you can map back to whatever the strings came from.

When something is wrong

Model names are checked against the registry, and a wrong one tells you so:

>>> Reranker("ms-marco-minilm-l6-v2")
KjarniException: Unknown model 'ms-marco-minilm-l6-v2'. Did you mean: minilm-l6-v2?

FAQ

What does pip install kjarni pull in?

Nothing. The package declares no dependencies. numpy is used if it is already present, and encode_batch returns lists of lists when it is not.

Does it need a GPU?

No. Everything above runs on CPU. Pass device="gpu" to Indexer or Searcher to use one; the GPU path is Vulkan, Metal or DirectX 12, so AMD, Intel and Apple Silicon work as they are.

Which mode should I use?

Semantic when queries are phrased as questions or paraphrases. Keyword when they contain identifiers, product codes or names that must match exactly. Hybrid when they are a mix, with a reranker on top if precision at the first position matters.

Can I search an index built by another language?

Yes. The index format is the engine's, not the binding's, so an index written by the CLI or from C# opens unchanged in Python.

How large can an index get?

The store is on disk and searched without loading everything into memory. Chunk counts in the tens of thousands are ordinary on a laptop.

Is generation included?

Not from Python yet. The Python package covers embeddings, classification, reranking, indexing and search. Chat and generation are available from the C#, C++ and Rust APIs today; see Running a Local LLM from C++.