Spaces:

AxiomicLabs
/

Open_SLM_Leaderboard

Running

What it is: cagliostro-v3's final checkpoint (the v3 row already on the board, pretrained from scratch on 75B tokens) with one more training stage of 0.39B tokens. That stage is v3's own cooldown mixture with Cosmopedia v1's WikiHow-style tutorials (30%) and its OpenStax, Khan Academy and Stanford textbook subsets (15%) added, 4,000 steps from v3's last checkpoint with the learning rate warmed to a tenth of v3's peak and decayed to zero. Same architecture and tokenizer as v3, so I've given it its own row, the way base and post-trained variants are listed separately (Speck2-140M and Speck2-140M-Instruct, for example).

No benchmark items, benchmark training splits or benchmark generators were used. Every new document went through 13-gram decontamination against HellaSwag validation, ARC test, PIQA validation and ArithMark-3. Since most of HellaSwag comes from WikiHow, the card breaks the HellaSwag gain down by item source: Cosmopedia's WikiHow set shares article titles with HellaSwag validation and train at the same rate (19.3% and 18.5%), the gain is about the same on items whose title appears in the training data as on items whose title does not, and ActivityNet items barely moved.

The card lists all six post-training runs that were evaluated on the test sets, including the ones that lost (chat data scored 26.00, below v3), and the rule used to pick this one, which was fixed before the results: it had to clear 27.13 on two different data seeds. The other seed of this recipe scored 27.31. A WikiHow-only recipe passed once and failed its second seed, so it was dropped.

On the margin: 0.36 over SmolLM2-135M on the board's figures. Re-measured in the same harness, a paired bootstrap over benchmark items still puts the gap to SmolLM2-135M inside the noise, and the card says so.

Reproducing:

python -m lm_eval --model hf \
  --model_args pretrained=bench-labs/cagliostro-v3.5,dtype=float32,trust_remote_code=True \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 64 --device cuda:0

ArithMark-3, with the official script from AxiomicLabs/ArithMark-3.0. It defaults to bfloat16, so pass --dtype float32:

python bencharithmark-3.py --model bench-labs/cagliostro-v3.5 --data-path arithmark-3.jsonl --device cuda --dtype float32

Please evaluate in float32. The model uses a logit cap of 15. trust_remote_code=True is needed because the architecture ships in the model repo.

Rules: 146,352,000 parameters (RULE 01), pretrained from random initialization by Bench Labs with no distillation and no outside checkpoint, the only starting point being our own cagliostro-v3 (RULE 02), weights public and ungated, pointing at the final checkpoint of the training run (RULE 03), listed evaluations above (RULE 04).