Your MiniLM Embeddings Are Probably Truncating at 256 Tokens

all-MiniLM-L6-v2 is the default embedding model almost everywhere. It is the default in sentence-transformers tutorials, in half the RAG walkthroughs on this site's competitors, and in a lot of production code that nobody has revisited since it started working.

Three files in that model's repository disagree about how long an input can be. If you have never picked one deliberately, you are using whichever your library picked for you, and the other two are silently wrong for your setup.

The three numbers

Download sentence-transformers/all-MiniLM-L6-v2 and look:

sentence-transformers reads the middle one. SentenceTransformer.encode passes max_length=self.max_seq_length on every call, so the tokenizer's own 128 is overridden and the model's 512 is never reached. The effective answer is 256.

For all-mpnet-base-v2 the same file says 384, which is one of the real differences between the two models and is rarely the reason people pick one.

None of this is hidden, exactly. It is just in a file most people have never opened, under a key that does not appear in any quickstart.

Why it matters more than it sounds

The number only bites when your input is longer than it. So the question is how often that happens, and the answer is: constantly, at the chunk sizes people actually use.

A very common default is 1000 characters per chunk. LangChain's RecursiveCharacterTextSplitter ships with chunk_size=1000. Plenty of homegrown splitters use the same round number.

I measured 1000-character chunks of ordinary technical prose through MiniLM's tokenizer, with truncation disabled so the real length shows:

chunk_size=1000 chars, 98 chunks
  median   286 tokens
  p90      323 tokens
  max      359 tokens
  over 256: 85/98  (87%)

At 2000 characters it is 100%, with a median of 565.

So on a default configuration, roughly nine chunks in ten are longer than the model will read, and everything past token 256 is discarded before the encoder ever sees it. No warning, no error, no truncation flag in the output. The embedding you get back is a perfectly ordinary 384-dimensional vector describing the first two thirds of your chunk.

The rough conversion is about 3.5 characters per token for English prose, so 256 tokens is around 900 characters. A 1000-character chunk sits just past the edge, which is the worst place for it to sit: far enough over to lose content, close enough that nothing looks obviously broken.

What it costs you

I built a document with a distinctive fact in the tail, past the 256-token mark, and queried it two ways. Once for something in the head, once for the fact at the end. Same model, same text, only the truncation length changed:

 tail query: 256 -> 0.1099   512 -> 0.2845
 head query: 256 -> 0.3850   512 -> 0.3286

Two things worth reading carefully there.

The tail query is the obvious one. At 256 the answer is not in the vector at all, and the score is close to noise. Extending to 512 nearly triples it.

The head query is the interesting one. Extending the window made that query worse. Mean pooling averages every token into one vector, so adding 100 more tokens of unrelated content pulls the centroid away from whatever the query was actually about. A longer window is not free. It buys recall on the tail and pays for it in precision everywhere else.

The fix is chunking, not a bigger window

The instinct on discovering this is to reach for a longer-context model. Nomic at 8192, or bge-m3, or anything that advertises a big number.

The data above argues against it. A chunk that fits under the limit gets the tail and keeps the focus. A chunk that needs an 8192-token window is a chunk whose pooled vector is an average of far too many things to match anything sharply.

Practically:

Chunk by tokens, not characters. Characters are a proxy that drifts with your content. Code, tables and non-English text all have very different ratios. If your splitter counts characters, it does not actually know how big your chunks are.

Target comfortably under the limit. For MiniLM at 256, something like 200 tokens with overlap leaves room and never truncates.

Score a document by its best chunk, not its average. If a document has to stay whole, embed its chunks separately and take the maximum similarity rather than the mean. This is the single change that most reliably improves retrieval on long documents, and it costs nothing at query time.

Check what your stack actually does. The number is in sentence_bert_config.json in the model repository. Confirm your library reads it rather than defaulting to max_position_embeddings.

The failure mode past 512

There is one more edge worth knowing, because it fails in a way that looks like success.

max_position_embeddings is 512 because there are exactly 512 learned position embeddings in the weights. Ask for more than that and a naive implementation has nothing to add for the extra positions. Depending on how the code is written, you either get an index error, or you get tokens that enter attention with no positional information at all.

The second outcome returns a normal-looking vector that is quietly wrong. If you are setting a truncation length by hand, 512 is a hard ceiling for this model family, not a soft one.

How I found it

Building Kjarni, a native inference engine, I had encoders reading max_position_embeddings and truncating at 512. Numerical parity against PyTorch passed, because every parity test used short sentences where the difference cannot appear. The longest string in the entire encoder test suite was 83 characters.

So the engine agreed with the reference implementation exactly, right up until an input crossed 256 tokens, at which point it silently diverged. Since the default chunk size put 87% of chunks above that line, the divergence covered most real usage.

The fix was to read sentence_bert_config.json when the model ships one, which is what sentence-transformers does. The lesson was that parity tests written on short inputs prove less than they appear to, and that a test which skips or passes vacuously is worse than no test, because it reports confidence it has not earned.

Checking your own setup

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

lengths = [len(tok(c, truncation=False)["input_ids"]) for c in your_chunks]
over = sum(l > 256 for l in lengths)
print(f"{over}/{len(lengths)} chunks exceed 256 tokens")

If that number is not close to zero, some of your corpus is not in your index, and nothing in your pipeline is going to tell you.

Related

• Local Embeddings for Microsoft.Extensions.AI: using these models from .NET
• Build a Document Search Engine in C#: where chunk size actually shows up
• Reranking in C#: a cross-encoder reads query and document together, so it is not subject to the pooling problem above

Kjarni is a native inference engine for embeddings, search, reranking and chat, with bindings for C#, the browser, Go, Python, C++ and the command line. The source is on GitHub, and there is a browser demo that needs no install.