# Cross-Encoder Reranking in Python without PyTorch

> Rerank search results with ms-marco-MiniLM-L-6-v2 from Python, without sentence-transformers or PyTorch. The same scores as CrossEncoder, a 23 MB install, and loaded in 0.08 seconds. With an example of what a reranker fixes that embeddings get wrong.

Source: https://kjarni.ai/blog/cross-encoder-reranking-python/ (published 2026-11-19)

Embedding search finds text that looks like the query. That is not always text that
answers it. Ask "how long does a refund take" over five support snippets, and
all-MiniLM-L6-v2 ranks a sentence that only repeats the question first:

```
0.8055  How long does it take to get a refund? Customers often ask this.
0.7874  Refunds are issued to the original payment method within 5 business days of receiving the return.
0.4608  You can return any item within 30 days of delivery.
0.4105  Refund policy: we refund all orders.
0.3323  Shipping takes 3 to 7 business days.
```

A cross-encoder reads the query and each document together and scores how well the
document answers it. Run over the same five, it puts the answer first, far ahead:

```
 7.8444  Refunds are issued to the original payment method within 5 business days of receiving the return.
 1.1465  How long does it take to get a refund? Customers often ask this.
-0.1466  You can return any item within 30 days of delivery.
-5.4474  Shipping takes 3 to 7 business days.
-6.8249  Refund policy: we refund all orders.
```

The usual Python route to that is `sentence_transformers.CrossEncoder`, which installs
PyTorch. This one does not:

```bash
pip install kjarni
```

```python
from kjarni import Reranker

reranker = Reranker("minilm-l6-v2-cross-encoder")

query = "how long does a refund take"
docs = [
    "Refunds are issued to the original payment method within 5 business days of receiving the return.",
    "You can return any item within 30 days of delivery.",
    "Refund policy: we refund all orders.",
    "Shipping takes 3 to 7 business days.",
    "How long does it take to get a refund? Customers often ask this.",
]

for hit in reranker.rerank(query, docs):
    print(f"{hit.score:7.4f}  {hit.document}")
```

The model is `cross-encoder/ms-marco-MiniLM-L-6-v2` from Hugging Face, downloaded on the
first run (90 MB) and cached.

## The same scores as CrossEncoder

```python
from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", device="cpu")
scores = model.predict([(query, d) for d in docs])
```

That gives the same five scores. Four agree to every printed decimal, and one differs in
the fourth: -5.4473 against Kjarni's -5.4474. Measured in two fresh virtual environments on
Python 3.13, Linux x86-64, with PyTorch from its CPU-only index and Hugging Face set to
offline so neither side waits on the network:

| | kjarni 0.4.2 | sentence-transformers 6.2 + torch 2.15 (CPU) |
|---|---|---|
| Installed size | 23 MB | 1.4 GB |
| Import and load the model | 0.08 s | 2.6 s |
| Rerank 200 documents | 0.30 s | 0.26 s |

PyTorch scores a batch about 15% faster. Kjarni starts about 30 times faster and installs
in a sixtieth of the space, which is what decides it in a web service, a serverless
function or a Docker image.

## Where it goes in a search

A cross-encoder is too slow to score a whole collection: it runs the model once per
document, per query. The usual shape is two steps:

1. **Retrieve** 50 to 200 candidates cheaply, with embeddings, BM25 or both.
2. **Rerank** those candidates with the cross-encoder and keep the top few.

```python
top = reranker.rerank_top_k(query, candidates, 5)
for hit in top:
    print(hit.index, round(hit.score, 2), hit.document)
```

`hit.index` is the position in the list you passed, so you can map back to your own ids.
`reranker.score(query, document)` scores a single pair.

Kjarni's `Searcher` does both steps for an index on disk:
`Searcher(rerank_model="minilm-l6-v2-cross-encoder")`. See [Local Semantic Search in Python](/blog/semanticsearchpython/).

## Reading the scores

The scores are logits, not probabilities: higher is more relevant, and they are not
limited to 0 to 1. Above 0 usually means the document is on topic, but the useful
signal is the gap. In the example, 7.84 against 1.15 says the first answer is clearly
better than anything else in the list.

A reranker is not always right. Asked "does python have a garbage collector", the same
model ranked "Garbage collection day in Python County is Tuesday" second and a correct
sentence about CPython's reference counting last. It learned from web search queries,
and a document that repeats the query's words still pulls. Evaluate it on your own
queries before trusting it.

## What it does not do

- **English.** ms-marco-MiniLM-L-6-v2 was trained on English web search queries.
- **512 tokens per query and document pair.** Longer documents are cut. Rerank chunks or
  passages, not whole files.
- **No fine-tuning.** Kjarni runs models; train with sentence-transformers.

## FAQ

### Can I use a cross-encoder without PyTorch?

Yes. `pip install kjarni` and `Reranker("minilm-l6-v2-cross-encoder")` run
ms-marco-MiniLM-L-6-v2 from a native library, with no PyTorch or transformers, and
give the same scores as sentence-transformers' CrossEncoder.

### What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder embeds the query and each document separately, so document vectors can be
computed once and searched fast. A cross-encoder reads the query and a document
together, which is more accurate and too slow for a whole collection. Use the first to
retrieve and the second to rerank.

### How many documents should I rerank?

Typically 50 to 200 from the first stage. Reranking 200 took 0.3 seconds on a desktop
CPU.

## Getting it

```
Install: pip install kjarni
PyPI:    https://pypi.org/project/kjarni/
GitHub:  https://github.com/olafurjohannsson/kjarni
```

## Related

- [Sentence Embeddings in Python without PyTorch](/blog/sentence-transformers-without-pytorch/): the first stage
- [Local Semantic Search in Python](/blog/semanticsearchpython/): an index on disk with reranking built in
- [Reranking in C# with a Cross-Encoder](/blog/reranking/): the same model from .NET
