Projects Case study
Training small LLMs to run on iPhone — measured, not vibes.
An end-to-end distillation pipeline that trained small open-source models (Qwen3 0.6B and 1.7B) to replace general-purpose LLMs as the on-device text generators inside two of my shipping iOS apps — an AI newscast generator and an interview-podcast generator built on Wikipedia articles.
The problem: general-purpose models fail focused apps
Both apps need one thing done exactly right, every time: a newscast generator that produces broadcast-ready segments in a strict format, and an interview-podcast generator that turns Wikipedia articles into structured host/guest dialogue. General-purpose instruct models — even good ones — fail these apps' own output-quality gates 60–100% of the time. The format is too specific. So instead of prompt-hoping at runtime, I trained the model to do the one job.
Teacher–student distillation, filtered by production gates
A large frontier LLM served as the teacher, generating ~1,900 training examples — byte-exact prompt/target pairs matching each app's runtime prompt format, including the chat template. Then the step that makes this pipeline measurable: every training target was filtered through the app's own output-validation gates, the same code that judges production output. Only examples the production code would accept became training data — a 99%+ pass rate.
LoRA training on Apple Silicon
Training ran locally with MLX (mlx-lm) on macOS/Metal — rank-8 LoRA adapters, roughly 1,000 iterations per run, best checkpoints selected by validation loss. On datasets this small, models overfit past ~200–800 iterations; I documented and handled that instead of pretending it away.
Quantization and deployment
Adapters were merged into the base model, converted to GGUF, and quantized to Q4_K_M — small enough to ship inside an app. Before each model went in, it was validated offline with llama.cpp under greedy decoding, matching the iOS runtime's chat template and thinking-mode seeding exactly. If it didn't pass there, it didn't ship.
Results
Held-out evaluation, scored by each app's production quality gates: baseline Qwen3 models passed 0–40% of the time; the trained models pass 70–100%. The standout: the 1.7B newscast model went from a 20% gate-pass rate to 100%.
Two lessons worth reusing
First, training prompts must byte-match inference prompts — chat template, anti-reasoning seeds, everything. Any drift between what you trained on and what the app sends is silent quality loss. Second, filtering training data through the production validators is what turns distillation from a feeling into a number: when the training targets are exactly what production accepts, gate-pass rate becomes a real, comparable metric.
The trained models ship inside Blue News Network and Podkipedia — both on this site, now running models trained for the job.