Joakimpalm-Zen/Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered-GGUF

Qwen3.8-27B GSQ-RCO IQ3_S with distillation-recovered scales (GGUF)

Research artifact, not a general-purpose release.

• What was changed: only the fp16 d/dmin scale fields of every quantised block in 399 tensors, retrained by distillation against the BF16 parent; integer codes, format, byte length and offsets identical to the source file.
• Why it exists: to test whether scale-only recovery brings an existing low-bit file inside the fidelity bar without re-quantising it.
• Parent model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (IQ3_S), from Qwen/Qwen3.8-27B
• Useful for normal inference? yes, wherever the source IQ3_S file was: same loader path, and it tracks the parent closer (0.0450 / 97.80% vs the source's matched row on this card).
• Experiment: Model Surgery and Scale Recovery
• Evidence: scale-recovery evidence (dataset: run record, logs, fidelity and HellaSwag records, code), also copied under evidence/ here
• Runner compatibility: Runner v0.5.3 or later serves the whole file on CUDA (i-quant kernels on the tensor cores, no CPU fallback); Runner v0.5.6 or later serves it on Metal too (codebook i-quant kernels, byte-exact against the CPU on the fixture gate). Any engine that reads the source file reads this one.
• MTP draft head: Qwen3.8-27B-MTP-Q8_0.gguf in this repo, a speculative-decoding companion (llama.cpp on a GPU: about 1.9x decode; Runner 1.1.3 on CUDA: about 1.1x, 1.5x on its opt-in int8 route; same text); see the head section.

Want this for another model? The tools behind these files are open: requantization, expert pruning and sublayer removal in Xyntetik Runner, and the method for this file is on this card or in its evidence dataset.

Request a model · Report your result · Community results · All files

If this file is useful to you, a contribution funds the hardware time behind the next one.

ISTA-DASLab's GSQ-RCO IQ3_S file of Qwen3.8-27B, with every quantised block's fp16 scales retrained end to end against the BF16 parent by knowledge distillation on a half-prose corpus. Same integer codes, same format, same 11,771,546,784 bytes, same loader path as the source file. The only bytes that changed are the d/dmin scale fields.

Runner fidelity against the BF16 parent on 500 held-out prose positions: mean KLD 0.0450, margin-qualified top-1 97.80%, inside the house bar (≤ 0.05, ≥ 97%). The matched-position row for the unmodified source file is below, so the reader can see exactly what recovery changed.

The MTP draft head (added 2026-10-08)

Qwen3.8-27B-MTP-Q8_0.gguf is a speculative-decoding draft file for this trunk: with llama.cpp on a GPU that holds trunk and head it decodes about 1.9x faster with the same greedy text, and it is not a model of its own (it cannot generate without the trunk, it does not change what the trunk says, and nothing in it was trained here).

The file (3,164,006,784 bytes, sha256 89e31f13400bac27b888f10bab02ff4d8da1f8610fa078dc32f9f3f0d2b6f6c9) holds the publisher's own multi-token-prediction block: the 15 mtp.* tensors of Qwen/Qwen3.8-27B (revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, shard model-00018-of-00018), exported with llama.cpp's converter (convert_hf_to_gguf.py --mtp --outtype q8_0): norms F32, matrices Q8_0, with the token embedding, output norm and LM head of the same checkpoint that the converter writes beside the block. No other change.

Measured on one MIG 1g.24gb slice of an RTX PRO 6000 Blackwell (trunk and head fully on the device), 512-token and 4,096-token prompts, 128 greedy tokens:

Engine	plain decode tok/s	with the head	drafts accepted	text vs plain
llama.cpp b11433, -md ... --spec-type draft-mtp	29.5-29.9	56.4-59.3	77%	not compared (server rows)
Xyntetik Runner 1.1.3, --mtp-file, CUDA (512-token prompt, two interleaved rounds)	13.8-13.9	15.4 (k=2), 15.3 (k=3)	2.72 / 3.20 tokens per round	byte-identical
Xyntetik Runner 1.1.3, CUDA with RUNNER_CUDA_Q8X=1 (experimental int8 route, off by default)	14.8	22.1-22.3 (k=2), 22.0-22.1 (k=3)	same	byte-identical to the route's own plain decode; 3 argmax flips in 480 positions against the CPU, hence opt-in
Xyntetik Runner 1.1.2, CPU only, 16 cores	4.3-4.6	4.2-4.6	same	byte-identical

The head pays most where the engine's batched rows are nearly free, which is llama.cpp on a GPU. On Runner 1.1.3 it pays a little on CUDA (about 1.1x at --draft-k 2; the default of 4 is slower than plain decoding on this file) and is a correctness feature on the CPU: the drafted text is the plain text, byte for byte. Runner's plain decode on these i-quant formats is still about half of llama.cpp's on the same slice; that is an engine item being worked on, not a property of the head.

# llama.cpp b11433 or newer
llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf \
  -md Qwen3.8-27B-MTP-Q8_0.gguf --spec-type draft-mtp -ngl 99
# Xyntetik Runner 1.1.3 or newer (CUDA and CPU; 1.1.1 and 1.1.2 draft on the CPU only)
runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf --mtp-file Qwen3.8-27B-MTP-Q8_0.gguf --draft-k 2

The head is Qwen's weights under the same Apache-2.0 license as the trunk; the conversion is deterministic from the stated revision.

Serve with xyntetik-runner

runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf --serve --port 8080

Runner v0.5.3 or later serves every tensor in this file on CUDA, i-quants included, with prefill on the tensor cores; the whole 11.8 GB file is device-resident on a 16 GB card. Runner v0.5.6 or later serves it on Metal as well: the seven codebook i-quants have Metal kernels, measured 2.2 to 2.5x the CPU and gated byte-exact against the CPU encoders on the 29-fixture set, so on Apple silicon with enough unified memory the file runs on the GPU. Any engine that reads the source file reads this one: nothing in the header, the tensor list, or the offsets moved.

Quickstart

runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf -p "The capital of France is" -n 32

What was changed, exactly

• Changed: the fp16 d field of every block in 399 quantised tensors (103,567,360 blocks), plus the fp16 dmin field where the format carries one (Q2_K, Q4_K). Nothing else.
• Unchanged: every integer code, codebook index and sign; every F32, BF16 and IQ1_M tensor; the embedding and output-head scales; all metadata; all tensor offsets. The output has the source's exact byte length.
• Verified after writing: every patched tensor re-decoded with the reference gguf decoder equals the trained weights; every unpatched tensor is byte-identical to the source. Largest change to any single decoded weight: 0.0195.

RECOVERY.json records the source file's SHA-256, the parent revision, the training checkpoint hash, the output hash, the counts above and the measurements below.

Why scales

GSQ optimises the integer grid assignments with a gradient method and RCO allocates a type per tensor under a size budget; neither trains anything through the model. In a fixed GGUF the remaining continuous degree of freedom is the per-block scale, which is linear in the decoded weight for every block format, so it can be trained end to end against the parent's distribution on the student's own forward pass without reimplementing any codec; the second phase of EfficientQAT applied to an existing file. The distillation blend was half Wikitext-103 prose on purpose: prose is where every low-bit variant of this model separates from its parent, and the one published trained quant of it traded prose away.

Training: 300 steps × 4 sequences × 512 tokens (0.6 M tokens), lr 3e-4 cosine, top-64 forward-KL to the parent with the tail as one bucket, only scales trainable, one decoder block resident on a 24 GB slice at a time.

Measured envelope

Two protocols, kept apart. The first is the release bar. The second is the selection metric used during development and is not comparable to the first.

Runner fidelity, scripts/kld-compare-raw.py, against the BF16 parent, the same 500 held-out Wikitext-2 positions for both rows, tie band 0.5 nats:

vs BF16 parent	mean KLD	top-1	margin-qualified top-1	top-8 overlap
source GSQ-RCO IQ3_S, unmodified	0.0480	86.60%	97.60%	0.882
this file	0.0450	87.20%	97.80%	0.884

House bar: KLD ≤ 0.05 and margin-qualified top-1 ≥ 97%. This file: PASS. Against the unmodified source on the same positions, every reported metric moved in the recovered file's favour (KLD 0.0480 → 0.0450, 261 of 500 positions improved), but the paired 95% interval on the KLD difference is [−0.0098, +0.0038] and includes zero at this sample size, so "passes the bar" is the claim, and "better than the source" is a consistent direction with an unresolved magnitude. A larger-sample comparison is being added.

Selection protocol (development only, not a release claim): top-64 forward KL and top-1 agreement against the parent on the student's own forward, 32 mixed instruction examples and 32 Wikitext-2 prose examples of 512 tokens, measured before and after training.

dev set	top-64 KL before → after	top-1 before → after
mixed (SmolTalk test)	0.1939 → 0.1379	89.38% → 89.95%
prose (Wikitext-2 test)	0.0485 → 0.0437	89.74% → 90.64%

Benchmark annex (added 2026-09-14): HellaSwag validation, fixed random subsample of 1,000 items (seed 20260911), parent and candidate scored through one float32 reference forward, paired. Fidelity and benchmark equivalence are different quantities; see Limits.

	acc	acc_norm
BF16 parent	63.4%	84.6%
this file	61.8%	83.5%
paired Δ vs parent (points), 95% bootstrap	-1.60 [-2.80, -0.50]	-1.10 [-2.00, -0.20]
discordant items (wins / losses), exact binomial p	34 (9 / 25), p = 0.009	23 (6 / 17), p = 0.035
source GSQ-RCO IQ3_S, unmodified (added 2026-09-15)	61.7%	83.4%
source Δ vs parent	-1.70 [-2.80, -0.60]	-1.20 [-2.10, -0.30]
this file − source, paired	+0.10 [-0.30, +0.50] (3 / 2)	+0.10 [-0.40, +0.60] (4 / 3)

Both differences from the parent fall outside a ±0.5-point equivalence margin: the file is inside the fidelity bar and measurably below the parent on this task, the divergence documented in the Muse-Glimmer surgery report. The source-file row (added 2026-09-15) locates that cost: it is the source quantisation's, and scale recovery neither reduced nor added to it; the recovered and source files disagree on 5–7 of 1,000 items, net +0.1 point, interval through zero.

Limits, read before quoting

• Parent-agreement is not a capability benchmark. The fidelity rows measure how closely next-token distributions track the parent on held-out prose.
• "No quality loss" is not claimed and is not supported by this evidence.
• Two fidelity rows, two protocols. The Runner KLD is variant‖reference over the union of both sides' top-20 log-probabilities; the selection KL is teacher‖student over the top-64 with a tail bucket. They are not interchangeable, and neither is the full-vocabulary KL some cards report.
• Metal needs Runner v0.5.6. Runner serves the file on CUDA since v0.5.3 and on Metal since v0.5.6, when the codebook i-quant kernels landed; an older Runner on Apple silicon runs the whole file on the CPU.
• Selection data. The training corpus, both development sets and the harness corpus were used to build or select this file. Only the benchmark annex touches none of them.
• Thin margin on agreement. 97.80% against a 97% bar on 500 positions.
• Multimodal not evaluated. The parent includes vision; this is the text backbone and text-only measurements cannot speak to it.
• One model, one corpus blend, one shared 24 GB slice.

Reproduce

# 1. Decompose the source file: integer codes frozen, scales trainable (bit-exact by construction).
python build_unit_cache.py --gguf Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --output cache/
# 2. Teacher targets from the BF16 parent on the training corpus.
python teacher_topk.py --corpus evidence/training-corpus.json --output evidence/teacher-train
# 3. Train the scales.
python train_scales.py --gguf Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --cache cache/ \
    --corpus evidence/training-corpus.json --teacher evidence/teacher-train \
    --steps 300 --batch 4 --lr 3e-4 --warmup 25 --output run/
# 4. Write them back into a byte copy of the source, with verification.
python write_scales.py --source Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --cache cache/ \
    --checkpoint run/checkpoint-00300.pt --output Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf
# 5. Fidelity in the release protocol.
python scripts/kld-compare-raw.py --model-a Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf \
    --model-b Qwen3.8-27B-BF16-backbone.gguf --runner ./runner \
    --corpus harness-prose-heldout.txt --max-positions 500 --out fidelity.json

Code and the full evidence record (every anchor, every run, the negative results on the smaller IQ3_XXS file): the code/ folder of this repository (the study is not yet on GitHub) at commit 51cead8.

Provenance

• Parent: Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, Apache-2.0.
• Source quant: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf, SHA-256 64b53b64c7aa39f2…, Apache-2.0. GSQ: arXiv:2604.18556. RCO: arXiv:2605.00649. The hard part of this artifact, the codes, is theirs.
• This file: SHA-256 873df48dd3ecf205…, 11,771,546,784 bytes.
• Training checkpoint: SHA-256 8a97f066829ec798…, step 300.
• Training data: Wikitext-103 (train) and SmolTalk (train shards 1–8), 2,048,000 tokens, hashes in RECOVERY.json. No development, harness or benchmark text.