This is a free ebook documenting a complete end-to-end pretraining run. The author, Raj Dandekar, is co-founder of Vizuara AI Labs and an MIT PhD. He spent $252.35 renting an H200 GPU, trained a 1.02B-parameter replica of Kimi K3 from scratch using 5B tokens, with 145M active parameters. The whole book has 30 chapters, 179 figures, and takes roughly 5 hours to read.

Don't expect a step-by-step tutorial. It is a work log, where failures are covered in far more detail than successes.

**Scaling Down the Model**

Kimi K3 is Moonshot's 2.8 trillion-parameter model. The author breaks down the official configuration line by line, then explains how to shrink the 2.8T model to fit on a single GPU without turning it into an entirely different architecture. The attention stack uses 9 KDA layers plus 3 MLA layers; the MoE (Mixture of Experts) uses top-6 routing, and parameter counting gets its own dedicated chapter.

**No Cutting Corners on Data**

The total corpus pool contains 104.8B tokens from six different sources. Before training, the author decontaminated 8 downstream benchmarks using 13-gram matching, and the shard format supports resuming training from interruptions. The tokenizer was not retrained from scratch—an existing off-the-shelf tokenizer was used directly.

**The Most Valuable Part: Failure Records**

There are 4 undocumented issues in the code released by Moonshot that will prevent you from running it successfully out of the box. The author provides a working training loop that circumvents all of them.

By the 20th training step, the MoE router had already killed 94.5% of all experts. Before you even finish warmup, most experts have already stopped learning. There is also another type of problem: loss doesn't diverge, but training has already quietly gone wrong. The book documents this silent collapse that wasn't caught by standard metrics.

Cross-GPU distributed training had 3 bugs that don't crash training, but produce models that *look normal but are actually wrong*. These bugs are a hundred times harder to debug than a full training crash.

**Speedup Experiments: 16 Attempts, 3 Successes**

The author documents all 16 attempts to improve MFU (Model FLOPS Utilization), including all 13 failed ones. Two counterintuitive findings:

- Performance was actually worse with FP8 precision, hitting only 0.19x of baseline MFU.

- Switching to a larger GPU also resulted in worse performance, again at 0.19x of baseline.

The book explains why these seemingly reasonable approaches failed. This kind of documentation is far more valuable than only listing what worked.

**The Final Result**

Loss dropped from 12.10 to 2.62. The author ran 24 benchmark evaluations throughout training, instead of only testing once at the end. The final chapter directly answers the question: what exactly do you get for $252?

This resource is perfect for anyone who wants to pretrain their own MoE model from scratch, or readers who want to understand architectures like K3 / DeepSeek. Read it for free online here: [Pretraining a Mini Kimi K3](https://books.vizuara.ai/book/pretraining-a-mini-k3)