Base LLM → SFT → DPO → benchmarks
Mini-ChatGPT
The ChatGPT recipe end to end at a scale that fits one GPU: take a base model that can only continue text, teach it to be an assistant, align it to human preferences, and measure every stage against the model builder's own chat model.
- Supervised fine-tuning of Qwen2.5-0.5B on 160k conversations (127M tokens) with assistant-only loss masking and length bucketing.
- Direct Preference Optimization written from scratch (no TRL) on 12k UltraFeedback pairs: held-out preference accuracy 52% → 68%.
- Evaluated with EleutherAI lm-evaluation-harness, the tool behind the Open LLM Leaderboard.
- PyTorch
- Transformers
- SFT
- DPO
- lm-eval-harness
- A100
View as table
| Benchmark | Base | Ours | Official |
|---|---|---|---|
| IFEval (instruction) | 22.7 | 36.7 | 36.3 |
| IFEval (prompt) | 12.6 | 24.0 | 26.4 |
| ARC-Easy | 58.4 | 64.3 | 59.2 |
| ARC-Challenge | 32.3 | 33.8 | 33.2 |
| HellaSwag | 52.2 | 52.0 | 52.4 |
| MMLU | 47.5 | 45.9 | 45.8 |