zerank-2-reranker-vllm

ZeroEntropy's zerank-2 reranker, ready for vllm serve. Same weights, same scores, about twice as fast as the official path.

Serving zerank-2 on vLLM is deceptively easy to get wrong: the server starts, every request returns 200, and the scores are quietly off. We converted the model, measured it against the official implementation, benchmarked it on an H100 and wrote down every trap we hit, so you can just serve it.

The official 0-1 score, sigmoid(logit / 5), straight from /rerank and /score. No --hf-overrides, no client-side prompt formatting.
Same scores as CrossEncoder("zeroentropy/zerank-2-reranker").predict(), within bf16 noise (see Parity).
Faster: on an H100, a rerank of 100 documents takes 0.62 s instead of 1.19 s, and throughput goes from 84 to 184 pairs/s (see Benchmark).

This is a format conversion of zeroentropy/zerank-2-reranker (formerly zeroentropy/zerank-2) into Qwen3ForSequenceClassification, not a new model: all credit for the model goes to ZeroEntropy. Converted from revision 5eae30d5, after ZeroEntropy relicensed the model under Apache 2.0 on 2026-07-24. Not affiliated with ZeroEntropy.

Validity. Validated in October 2026 with vLLM 0.29.0 and 0.31.0, sentence-transformers 6.1.0, on H100. The --chat-template requirement and the truncation caveats below are vLLM behaviours of those versions; re-run the parity script before relying on a newer vLLM.

Quick start

pip install "vllm>=0.29,<=0.31" huggingface_hub   # validated range, see Validity

TEMPLATE=$(hf download polaria-tech/zerank-2-reranker-vllm score_template.jinja --quiet)

vllm serve polaria-tech/zerank-2-reranker-vllm \
  --runner pooling \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --chat-template "$TEMPLATE"

--max-model-len 32768 matches zerank-2's advertised context. Without it, vLLM uses the Qwen3 config's 40,960 positions; scores stay faithful to the original model up to 38k tokens (tested), but ZeroEntropy only advertises 32k.

import requests

resp = requests.post("http://localhost:8000/rerank", json={
    "model": "polaria-tech/zerank-2-reranker-vllm",
    "query": "What is 2+2?",
    "documents": ["4", "The answer is definitely 1 million"],
    "max_tokens_per_doc": 8000,  # optional, the only safe way to truncate (see below)
}).json()

for r in resp["results"]:
    print(r["index"], round(r["relevance_score"], 4))
# 0 ≈ 0.75   (raw logit ≈ 5.4)
# 1 ≈ 0.29   (raw logit ≈ -4.5)

/score works the same way (text_1 = query, text_2 = documents). To get the raw logit that CrossEncoder.predict() returns: logit = 5 * log(s / (1 - s)).

Custom instructions

zerank-2 is an instruction-following reranker (see ZeroEntropy's prompting guide). Two equivalent ways to pass an instruction:

# 1. vLLM's `instruction` field (also works on /score and the Cohere-style /v2/rerank)
resp = requests.post("http://localhost:8000/rerank", json={
    "model": "polaria-tech/zerank-2-reranker-vllm",
    "query": "VAT limitation period",
    "instruction": "We are a French tax law firm. Prefer statutes and recent French case law "
                   "over blog posts and commentary.",
    "documents": documents,
}).json()

# 2. Inside the query string, in any readable format, as in ZeroEntropy's guide
query = (
    "<query>VAT limitation period</query>\n"
    "<instruction>We are a French tax law firm. Prefer statutes and recent French case law "
    "over blog posts and commentary.</instruction>"
)

The template turns the instruction field into exactly the second form. Without an instruction, the prompt is the official zerank-2 prompt, unchanged: no default instruction is added, since zerank-2 is meant to work from the query alone. (Templates that ignore the instruction variable, including zerank-2's original one, silently drop that field.)

Long instructions are cheap: with prefix caching (on by default), the query is computed once per rerank call and shared by all its documents. In the benchmark below, a ~300-token query costs 0.69 s per 100 documents with the cache, 1.11 s without.

⚠️ Three things that silently break the scores

None of these raise an error: the server answers, the scores are just wrong.

--chat-template is mandatory. vLLM (≤ 0.31) ignores the checkpoint's template for scoring and falls back to concatenating query and document. Without the flag: mean error of 11.4 logit and many ranking inversions.
Use the provided score_template.jinja, not the original zerank-2 chat template. vLLM renders score templates without add_generation_prompt, so the original template drops the final <|im_start|>assistant\n and the score is read at the wrong position (mean error 9.9 logit).
Truncate with max_tokens_per_doc, never with truncate_prompt_tokens. The latter cuts the end of the prompt (the template suffix, or the query with truncation_side=left), producing wrong scores. Keep query tokens + document tokens + 13 < --max-model-len.

Also note a vLLM scheduler bug (0.29 and 0.31): a prompt of exactly --max-model-len tokens never completes (the request hangs without error). One token less works. truncate_prompt_tokens=-1 always hits it. Fixed upstream by vllm-project/vllm#48039 (merged 2026-10-07), not yet in a release as of 0.31.1rc0.

Recommended settings (H100)

vllm serve polaria-tech/zerank-2-reranker-vllm \
  --runner pooling \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --chat-template "$TEMPLATE" \
  --max-num-batched-tokens 65536 \
  --gpu-memory-utilization 0.3

Keep prefix caching on (the default). The query sits at the head of every prompt of a rerank call, so it is computed once for all documents: −35% latency with ~300-token queries.
--max-num-batched-tokens 65536: +5% throughput over the default 8192.
--gpu-memory-utilization 0.3 (~24 GiB) performs the same as 0.92. Reranking is compute-bound, not memory-bound: weights are 7.8 GiB and the KV cache is short-lived. With 0.3, the KV cache holds ~77k tokens, enough for a full 32k (or 40k) prompt; below ~0.25 a 40k prompt no longer fits.

Benchmark

H100 80GB SXM, vLLM 0.31.0, bf16. One rerank call = 1 query + 100 documents of 300–500 tokens. "Short" queries ≈ 15 tokens, "long" queries ≈ 300 tokens. Every call has a unique query (no cross-call cache hits). 32+ measured calls per cell after warm-up.

	sentence-transformers	vLLM defaults	vLLM, no prefix cache	vLLM recommended
latency, 1 call, short query	1.193 s	0.645 s	0.656 s	0.622 s
latency, 1 call, long query	1.866 s	0.717 s	1.110 s	0.689 s
throughput at saturation, short query	84 pairs/s	175 pairs/s	171 pairs/s	184 pairs/s
throughput at saturation, long query	53 pairs/s	169 pairs/s	101 pairs/s	177 pairs/s

sentence-transformers = the official CrossEncoder.predict() path, bf16, best batch size (32 of 16–128). It processes one call at a time, so its saturation throughput is its single-call throughput.

Under load (recommended settings, short queries)

concurrent calls	p50 latency	p95 latency	pairs/s
1	0.622 s	0.639 s	161
8	4.30 s	4.32 s	184
32	17.3 s	18.1 s	183

The GPU saturates from ~8 concurrent calls, at ~79k prompt tokens/s. Beyond that, extra calls queue and latency grows linearly: size client concurrency to your latency budget. Cost scales with total tokens, roughly n_docs × (query + doc + 13).

Parity with the official model

Reference: CrossEncoder("zeroentropy/zerank-2-reranker").predict() (sentence-transformers 6.1.0), which returns the raw Yes logit; its 0-1 score is (predict() / 5).sigmoid(). On the vLLM side, the raw logit is recovered as 5 * log(s / (1 - s)), and both logits and 0-1 scores are compared.

Ground truth: the same official model in fp32. The official bf16 path is itself not exact, so its deviation from fp32 is used as the noise floor, and vLLM must stay within 1.5x of it.

max deviation from fp32 (logit / 0-1 score)	main set	long set
official `predict()`, bf16 (noise floor)	0.183 / 0.0053	0.121 / 0.0042
this model, vLLM 0.31.0	0.095–0.136 / 0.0040	0.066 / 0.0032
this model, vLLM 0.29.0	same range	0.057 / 0.0028

Main set: 40 pairs, 9 queries, multilingual, 19 → 6,051 tokens, mixed lengths in every batch. Long set: 7 pairs, 48 → 38,203 tokens.
Prompt token ids identical to the official path on 47/47 pairs.
On these 9 queries: top-1 identical to fp32 everywhere, and no ranking inversion between documents further apart than 2x the noise floor. The sample is sized to measure bf16 noise, not to certify ranking quality on your data.
No extra drift with length.

vLLM ends up closer to fp32 than the official bf16 path, because the score head runs in fp32.

How it was converted

zerank-2 is a Qwen3ForCausalLM (Qwen3-4B) whose score is the logit of the token Yes (id 9454) at the last position, with no negative token; the 0-1 score is sigmoid(logit / 5).

score.weight [1, 2560] = model.embed_tokens.weight[9454] / 5, stored in fp32 (zerank-2 has tied embeddings, so there is no lm_head.weight). vLLM applies a sigmoid for num_labels=1 and runs pooling heads in fp32, so it outputs sigmoid(logit / 5) with no extra rounding.
The backbone shards are copied unchanged.
config.json: Qwen3ForSequenceClassification, num_labels: 1, tie_word_embeddings: false, use_sep_token: false. None of the keys (method, classifier_from_token, is_original_qwen3_reranker) that make vLLM re-derive the head from lm_head at load time.
The sentence-transformers files are deliberately left out: their activation_fn: Identity would replace vLLM's sigmoid.
The Qwen3-Reranker recipe (no/yes pair, from_2_way_softmax) does not apply to zerank-2 and gives compressed, wrong scores.

The prompt is:

<|im_start|>system
{query}<|im_end|>
<|im_start|>user
{document}<|im_end|>
<|im_start|>assistant

Conversion script, reference and parity scripts, truncation tests and benchmark: github.com/polaria-tech/zerank-2-reranker-vllm.

Long documents

This is the original model's behaviour (identical in fp32), not a conversion artifact: relevance gets diluted in long documents, most when the relevant passage sits in the middle.

document (query about argparse subcommands)	tokens	score
argparse docs alone	6,228	0.68
argparse docs + unrelated text after	15,187	0.41
unrelated text + argparse docs at the end	23,207	0.52
argparse docs in the middle	31,561	0.22
unrelated text only	31,810	0.11

Ordering stays correct, but chunking long documents is advisable.

Limitations

Tested with vLLM 0.29.0 and 0.31.0 only, in bf16. fp16 untested.
Advertised context 32,768 tokens. Parity tested up to 38k tokens per pair, which shows the conversion is faithful there, not that the model is reliable beyond 32k.
Built for vLLM. Loading it with transformers or sentence-transformers is untested: for those, use zeroentropy/zerank-2-reranker directly.

License and attribution

Apache 2.0, like the original. Model by ZeroEntropy: zeroentropy/zerank-2-reranker (formerly zeroentropy/zerank-2; relicensed under Apache 2.0 on 2026-07-24, converted from revision 5eae30d5ee3c6b2df2ef6d723bde45172d761c4c), technical report arXiv:2509.12541. Conversion and evaluation by Polaria.