AI Product / PoC
Efficient Thinking fine-tuning
Validating "Efficient Thinking" on Qwen 3.5 9B / 3.6 27B: Self Distill selects only the good reasoning traces the model itself generates and distills from them, reducing reasoning tokens while preserving accuracy. A practical example of making SLMs viable by cutting inference cost and latency.
Key results
- Improved pass@1 from 63.9% to 86.5% (+22.6 pp)
- Reduced average reasoning tokens from 2,895 to 1,071 (−63%)
- On an unseen generalization test (5,407 questions the teacher Gemma got wrong): 3.5% → 33.9% (+30 pp)
- best-of-4 (pass@4): 96.7%
Background
Reasoning models such as the Qwen series consume large numbers of thinking tokens in exchange for accuracy. On Qwen 3.5 9B the reasoning portion can reach roughly 30,000 tokens, making inference cost and latency a real bottleneck in production. If thinking can be shortened without losing accuracy, SLMs become viable in far more real-world environments, including on-prem and edge.
Approach
Rather than relying entirely on a large external teacher model, the project runs a Self Distill (self-distillation) loop that trains on only the best reasoning the model itself generates. First, with Gemma 4 31B as the Round 0 teacher, I collect problems that can be solved within a token budget along with the teacher traces. Then Qwen 3.5 9B generates multiple reasoning traces per problem, from which only the ones that are both correct and short are selected. The selected traces become the training signal for LoRA updates (SFT / DPO), and repeating the generate → select → distill loop teaches the model to reach the same conclusions with shorter reasoning.
The goal is to compress reasoning that can reach roughly 30,000 tokens down by an order of magnitude (to about one tenth).
What I am validating
The focus is on quantitative evaluation of the accuracy-versus-reasoning-tokens trade-off, whether the model can adapt how much it thinks to task difficulty, and how the difference in model size between 9B and 27B affects tolerance to compression. The inference stack uses vLLM, and I am also validating throughput improvements in combination with NVFP4 quantization and MTP (speculative decoding).
Results
Measured results from Round 1 (Qwen 3.5 9B as student, distilled from Gemma 4 31B as teacher). On a mixed multi-domain evaluation set (math, science, medicine, Japanese, and more), accuracy (pass@1) improved while average reasoning tokens dropped by about 60%. Output length shrank by 60–85% across all nine domains, with no catastrophic forgetting in domains the model was already strong in.
Crucially, accuracy also rose sharply on problems the teacher Gemma could not solve (5,407 JMMLU questions excluded from the training data). This is not mere imitation of the teacher: it shows generalization that surpasses the teacher, where suppressing the habit of overthinking lets the model's true ability come through. With best-of-4 (pass@4) the model reaches 96.7%, confirming that sampling can lift accuracy further where the use case allows.
Measured results
| Metric | base | after SFT |
|---|---|---|
| pass@1 | 63.9% | 86.5% (+22.6 pp) |
| pass@4 | — | 96.7% |
| Avg. reasoning tokens | 2,895 | 1,071 (−63%) |
| Model | pass@1 | Avg. tokens |
|---|---|---|
| Qwen 3.5 9B base | 3.5% | 2,777 |
| Qwen 3.5 9B SFT | 33.9% | 1,068 |
Against the 25% random baseline for four-choice questions, base fell below it (dominated by format breakage and truncation) while SFT scored above it.
Demo & write-up
A demo video and write-up are in preparation and will be published on this page.