The 256-token limit in all-MiniLM-L6-v2

Three files in the model repository disagree about how long an input can be, and the one that wins is not the one most people assume.

from sentence_transformers import SentenceTransformer
from huggingface_hub import hf_hub_download
import json

m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
tj = json.load(open(hf_hub_download("sentence-transformers/all-MiniLM-L6-v2",
                                    "tokenizer.json")))

print(tj["truncation"]["max_length"])                       # 128
print(m.max_seq_length)                                     # 256
print(m[0].auto_model.config.max_position_embeddings)       # 512

encode passes max_length=self.max_seq_length on every call, so the tokenizer's 128 gets overridden and the model's 512 is never reached. 256 wins. For all-mpnet-base-v2 it is 384.

It truncates without telling you

Same text, same model, only the window changes:

body = ("The retrieval service keeps an inverted index in memory and refreshes it "
        "nightly from the document store. Queries are served from that snapshot, so "
        "writes are not visible until the next refresh completes. ")
doc = body * 12 + "The rollback procedure is documented under runbook QX-4417."
print(len(m.tokenizer(doc, truncation=False)["input_ids"]))   # 509

m.max_seq_length = 256
a256 = m.encode(doc, normalize_embeddings=True)
m.max_seq_length = 512
a512 = m.encode(doc, normalize_embeddings=True)

print(a256 @ a512)                                            # 0.9359

Two different vectors for the same document. No warning, no error, no truncation flag in the output. At the default 1000-character chunk size this is the normal case, not an edge case:

lengths = [len(m.tokenizer(c, truncation=False)["input_ids"]) for c in chunks]
print(sum(l > 256 for l in lengths), "/", len(lengths))       # 85 / 98

LangChain's RecursiveCharacterTextSplitter defaults to chunk_size=1000, so on a stock setup nine chunks in ten are longer than the model reads. English runs about 3.5 characters per token, so 256 tokens is roughly 900 characters and a 1000-character chunk sits just past the edge.

A longer window is not the fix

The document above has its distinctive fact in the tail, past the 256 mark. Two queries, one for the tail and one for the head:

tail = "what is the rollback runbook number"
head = "how often is the inverted index refreshed"

# tail: 256 -> 0.1239   512 -> 0.2049
# head: 256 -> 0.5154   512 -> 0.4457

The tail improving is expected. The head getting worse is the part that matters: these models mean-pool, so 250 more tokens of unrelated content drag the centroid away from whatever the query was about. A longer window buys recall on the tail and pays for it in precision everywhere else.

Chunk instead, and score by the best chunk

import numpy as np

qv = m.encode(tail, normalize_embeddings=True)
cv = np.stack([m.encode(c, normalize_embeddings=True) for c in chunks])  # 3 chunks
best = (cv @ qv).max()

# tail: truncated 0.1239   best-chunk 0.4789
# head: truncated 0.5154   best-chunk 0.5058

The tail goes from noise to 0.48, more than double what the 512 window gave, and the head holds. That is the whole argument against reaching for nomic at 8192 or bge-m3: a chunk that fits under the limit gets the tail and keeps its focus, while a chunk that needs 8192 tokens averages too many things to match any of them sharply.

So chunk by tokens rather than characters, since code and tables and non-English text all have different ratios. Target well under the limit, 200 tokens with overlap never truncates. Then take the maximum similarity across a document's chunks instead of embedding it whole.

Checking your own corpus

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

lengths = [len(tok(c, truncation=False)["input_ids"]) for c in your_chunks]
over = sum(l > 256 for l in lengths)
print(f"{over}/{len(lengths)} chunks exceed 256 tokens")

If that is not close to zero, part of your corpus is not in your index and nothing in your pipeline will tell you.

512 is a hard ceiling

There are exactly 512 learned position embeddings in the weights. Ask for more and a naive implementation has nothing to add for the extra positions, so you get either an index error or tokens entering attention with no positional information, which returns a normal looking vector that is quietly wrong.

What Kjarni does about it

I hit this building Kjarni, a native inference engine. Its encoders read max_position_embeddings and truncated at 512. Parity against PyTorch passed, because every parity test used short sentences: the longest string in the entire encoder suite was 83 characters. So it matched the reference exactly right up until an input crossed 256 tokens, which the default chunk size put 87% of chunks above.

Kjarni now reads sentence_bert_config.json when the model ships one, the same file sentence-transformers reads, so the truncation point matches whatever your Python pipeline already does. Asking for more positions than the weights hold is an error rather than a silently wrong vector.

kjarni embed "your text" --model minilm-l6-v2

The wider lesson was about the tests, not the config. A parity suite whose longest input is 83 characters proves the model loads, and nothing about the case that actually broke.

Kjarni is a native inference engine for embeddings, search, reranking and chat, with bindings for C#, the browser, Go and Python. The source is on GitHub, and there is a browser demo that needs no install.

Related

• Local Embeddings for Microsoft.Extensions.AI: using these models from .NET
• Build a Document Search Engine in C#: where chunk size shows up
• Reranking in C#: a cross-encoder reads query and document together, so it avoids the pooling problem above