Cross-Encoder Reranking in Python without PyTorch
Rerank search results with ms-marco-MiniLM-L-6-v2 from Python, without sentence-transformers or PyTorch. The same scores as CrossEncoder, a 23 MB install, and loaded in 0.08 seconds. With an example of what a reranker fixes that embeddings get wrong.
Embedding search finds text that looks like the query. That is not always text that answers it. Ask "how long does a refund take" over five support snippets, and all-MiniLM-L6-v2 ranks a sentence that only repeats the question first:
0.8055 How long does it take to get a refund? Customers often ask this.
0.7874 Refunds are issued to the original payment method within 5 business days of receiving the return.
0.4608 You can return any item within 30 days of delivery.
0.4105 Refund policy: we refund all orders.
0.3323 Shipping takes 3 to 7 business days.
A cross-encoder reads the query and each document together and scores how well the document answers it. Run over the same five, it puts the answer first, far ahead:
7.8444 Refunds are issued to the original payment method within 5 business days of receiving the return.
1.1465 How long does it take to get a refund? Customers often ask this.
-0.1466 You can return any item within 30 days of delivery.
-5.4474 Shipping takes 3 to 7 business days.
-6.8249 Refund policy: we refund all orders.
The usual Python route to that is sentence_transformers.CrossEncoder, which installs
PyTorch. This one does not:
pip install kjarni
from kjarni import Reranker
reranker = Reranker("minilm-l6-v2-cross-encoder")
query = "how long does a refund take"
docs = [
"Refunds are issued to the original payment method within 5 business days of receiving the return.",
"You can return any item within 30 days of delivery.",
"Refund policy: we refund all orders.",
"Shipping takes 3 to 7 business days.",
"How long does it take to get a refund? Customers often ask this.",
]
for hit in reranker.rerank(query, docs):
print(f"{hit.score:7.4f} {hit.document}")
The model is cross-encoder/ms-marco-MiniLM-L-6-v2 from Hugging Face, downloaded on the
first run (90 MB) and cached.
The same scores as CrossEncoder
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", device="cpu")
scores = model.predict([(query, d) for d in docs])
That gives the same five scores. Four agree to every printed decimal, and one differs in the fourth: -5.4473 against Kjarni's -5.4474. Measured in two fresh virtual environments on Python 3.13, Linux x86-64, with PyTorch from its CPU-only index and Hugging Face set to offline so neither side waits on the network:
| kjarni 0.4.2 | sentence-transformers 6.2 + torch 2.15 (CPU) | |
|---|---|---|
| Installed size | 23 MB | 1.4 GB |
| Import and load the model | 0.08 s | 2.6 s |
| Rerank 200 documents | 0.30 s | 0.26 s |
PyTorch scores a batch about 15% faster. Kjarni starts about 30 times faster and installs in a sixtieth of the space, which is what decides it in a web service, a serverless function or a Docker image.
Where it goes in a search
A cross-encoder is too slow to score a whole collection: it runs the model once per document, per query. The usual shape is two steps:
- Retrieve 50 to 200 candidates cheaply, with embeddings, BM25 or both.
- Rerank those candidates with the cross-encoder and keep the top few.
top = reranker.rerank_top_k(query, candidates, 5)
for hit in top:
print(hit.index, round(hit.score, 2), hit.document)
hit.index is the position in the list you passed, so you can map back to your own ids.
reranker.score(query, document) scores a single pair.
Kjarni's Searcher does both steps for an index on disk:
Searcher(rerank_model="minilm-l6-v2-cross-encoder"). See Local Semantic Search in Python.
Reading the scores
The scores are logits, not probabilities: higher is more relevant, and they are not limited to 0 to 1. Above 0 usually means the document is on topic, but the useful signal is the gap. In the example, 7.84 against 1.15 says the first answer is clearly better than anything else in the list.
A reranker is not always right. Asked "does python have a garbage collector", the same model ranked "Garbage collection day in Python County is Tuesday" second and a correct sentence about CPython's reference counting last. It learned from web search queries, and a document that repeats the query's words still pulls. Evaluate it on your own queries before trusting it.
What it does not do
- English. ms-marco-MiniLM-L-6-v2 was trained on English web search queries.
- 512 tokens per query and document pair. Longer documents are cut. Rerank chunks or passages, not whole files.
- No fine-tuning. Kjarni runs models; train with sentence-transformers.
FAQ
Can I use a cross-encoder without PyTorch?
Yes. pip install kjarni and Reranker("minilm-l6-v2-cross-encoder") run
ms-marco-MiniLM-L-6-v2 from a native library, with no PyTorch or transformers, and
give the same scores as sentence-transformers' CrossEncoder.
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder embeds the query and each document separately, so document vectors can be computed once and searched fast. A cross-encoder reads the query and a document together, which is more accurate and too slow for a whole collection. Use the first to retrieve and the second to rerank.
How many documents should I rerank?
Typically 50 to 200 from the first stage. Reranking 200 took 0.3 seconds on a desktop CPU.
Getting it
Install: pip install kjarni
PyPI: https://pypi.org/project/kjarni/
GitHub: https://github.com/olafurjohannsson/kjarni
Related
- Sentence Embeddings in Python without PyTorch: the first stage
- Local Semantic Search in Python: an index on disk with reranking built in
- Reranking in C# with a Cross-Encoder: the same model from .NET
Kjarni runs the same engine on every platform: