AI Product / PoC
Yumeuranai — AI dream reading
A testbed for how cheap and fast an LLM service can run on a small model with CPU-only inference: a 1.2B SLM distilled and DPO-tuned from a Gemma 31B teacher, served on AWS Lambda by a hand-built Rust CPU inference engine. The vehicle is a dream-reading web app that returns symbol meanings and a gentle reading. A live demo is available.
Key results
- Best-of-N teacher data and corrected LoRA targets lift the 1.2B distilled model's symbol coverage from 0.835 to 0.962
- Hand-built Rust CPU inference engine: +57% prefill and +21% decode versus llama.cpp (same GGUF, same machine)
- Verified on real AWS Lambda — 852 MB peak memory in a 3 GB configuration (366 MB of Q2_K QAT quantized weights)
The pipeline at a glance
Background
The cost of an LLM service is its inference cost. Build the quality into a large teacher model, hand execution to a distilled small model on a dedicated runtime — and how far can CPU-only inference actually go? This PoC builds the whole chain, model to serving, to find out. Dream reading is a fitting testbed: a real end-to-end service where both free-form generation quality and response time matter. The service itself is framed as gentle self-reflection rather than fortune-telling, with a non-assertive voice and a fixed safety footer.
How it works
The frontend is a three-screen Astro + React flow (input → analysis → result), with Cloudflare Workers serving static assets and /api. For the reading itself, the Worker calls an AWS Lambda function URL (Tokyo) that runs LFM2.5-1.2B-JP on a hand-built Rust CPU inference runtime. Symbol meanings are deliberately not written by the LLM: they come from a curated dream-symbol knowledge base (5,764 entries), retrieved with exactly the same logic used at training time — suppressing hallucination and keeping interpretations updatable. The prompt is a single contract shared between training-data generation and inference — changing it means retraining.
Key design points
On the model side, a Gemma 31B teacher (self-hosted vLLM) generates high-quality data distilled into LFM2.5-1.2B-JP. Fixing the LoRA target modules to match LFM2's convolution-hybrid architecture improved training loss from 0.99 to 0.69; best-of-N teacher sampling (k=16, judge-scored) raised symbol coverage from 0.835 to 0.948; and a final DPO stage — after automatic metrics had saturated — won 62% of randomized judged pairwise comparisons against the SFT-only model, lifting quality the metrics could not capture.
On the inference side, a dedicated Rust runtime for LFM2.5 — hand-written AVX2 kernels plus Q2_K QAT quantization (366 MB) — measures +57% prefill and +21% decode versus llama.cpp on the same GGUF and machine, about 73% of the memory-bandwidth-derived theoretical limit.
Improvement continues on the deployed target: the repetition loops found in the demo were fixed by implementing repetition penalty (the model card's recommended 1.05), while n-gram speculative decoding — +7–15% on the dev box — was disabled after an A/B on real Lambda showed it slower there. Optimizations are accepted or rejected by measuring on the actual target hardware.
Measured results
| Metric | Custom runtime | llama.cpp (AVX2) |
|---|---|---|
| Prefill (1,000 tokens) | ~1.55 s (~645 tok/s) | 2.43 s (412 tok/s) |
| Decode (at 1,000 context) | ~111 tok/s | 91.5 tok/s |
Table figures are measured on a dev box constrained to 6 cores with Lambda-like CPU/memory limits. The runtime is also smoke-tested on real AWS Lambda (3,008 MB configuration, 852 MB peak memory), with detailed on-Lambda benchmarks in progress.
Demo & write-up
A live demo is available — try it yourself. It is published for demonstration and validation purposes, not as a production service. A write-up is also in preparation.