Building and Scaling Machine Learning Systems

Models are the easy part. The system around them is where projects fail.

Most machine learning books stop at the model. This one starts there and builds the production system around it. You will learn to turn a business goal into concrete latency, cost, and quality budgets, then make the data decisions that actually determine accuracy: sampling, labelling, imbalance handling, and point-in-time correct features that do not leak. The book treats machine learning engineering as a design discipline, not a collection of algorithms.

From there, you will choose baselines before architectures, train at scale with data and model parallelism, and evaluate with slices, calibration, and behavioural tests rather than a single aggregate number. Serving chapters cover batch, online, and streaming inference, tail latency, quantisation, distillation, caching, and accelerator cost. Operations chapters cover drift detection, retraining triggers, shadow deployments, and experiment design, followed by retrieval-augmented language model systems and five worked design exercises. Each chapter connects mathematical foundations to the engineering decisions they drive, so you can reason about trade-offs instead of memorising recipes.

What you will learn:

• Translate business goals into system requirements, latency budgets, and cost constraints
• Apply the mathematics that drives sampling, labelling, imbalance, and evaluation decisions
• Build leak-free feature pipelines and feature stores with point-in-time correctness
• Select baselines before architectures and train at scale with data and model parallelism
• Evaluate models with slices, calibration, and behavioural tests that predict production behaviour
• Design batch, online, and streaming serving architectures with tail-latency control
• Optimise models and infrastructure using quantisation, distillation, caching, and accelerator cost analysis
• Monitor drift, trigger retraining, run shadow deployments, and design online experiments
• Build retrieval-augmented language model systems and work through five end-to-end design exercises

Written for machine learning engineers, platform engineers, and design interview candidates who need to build and scale production-ready AI systems. If you are moving from notebooks to reliable, cost-aware, and maintainable machine learning systems, this book gives you the foundations, the mathematics, and an iterative design process to do it.