دانلود کتاب سیستم‌های یادگیری ماشین در مقیاس بزرگ، جلد ۲

This book is for anyone who understands machine learning on a single machine and now faces the challenge of making it work at scale. The problems of scale are not the problems of a single node, only bigger. They are qualitatively different: networks partition, hardware fails as a statistical certainty, and the societal impact of every design decision is amplified by millions of users. The scope ranges from designing the physical substrate of an AI data center, to scaling training beyond the limits of a single accelerator, to reasoning about what happens when a production fleet serves a global user base. Throughout, the focus is on the physics of distribution—the invariants that govern every fleet-scale ML system regardless of the framework, the model, or the era. Why a Fleet-Level Textbook In 2012, training AlexNet took five to six days on two GPUs (Krizhevsky et al. 2012). By 2023, public GPT-4-class estimates placed frontier training at roughly 25,000 GPUs running for about three months (SemiAnalysis 2023); OpenAI did not disclose the actual hardware configuration (OpenAI et al. 2023). This is not the same engineering problem at a larger scale. It is a categorically different engineering problem. On a single machine, performance is governed by the memory wall—the gap between processor speed and memory bandwidth. In a distributed cluster, a new wall emerges: the bisection bandwidth wall, where the minimum network bandwidth that bisects the cluster caps total system throughput. Data no longer moves through local hierarchies alone; it traverses optical fabrics governed by the speed of light between racks and across continents. Hardware failures, rare enough on a single machine to be treated as exceptions, become routine statistical events at fleet scale. A training job that does not plan for failure will fail. These are not operational annoyances. They are physical constraints—as fundamental and as permanent as the memory wall, Amdahl’s Law, or the energy cost of data movement. They require their own engineering discipline, with its own invariants: • The Bisection Bandwidth Theorem: The performance of a distributed system is limited by the minimum bandwidth that bisects the network topology. • The Fleet Law: Every training step pays a communication tax that grows with cluster size while per-node computation shrinks. • The Young-Daly Checkpoint Law: The optimal checkpoint interval balances the cost of writing state against the expected cost of lost work due to failure. • The Serving Cost Dominance Law: For high-traffic production models, cumulative inference operational expenditure can exceed the one-time training cost, making serving efficiency a lifecycle optimization target. • The Fairness Impossibility Law: No system can simultaneously satisfy calibration, equalized odds, and demographic parity when base rates differ between groups. If single-machine ML systems engineering asks “what can one machine build?”, then fleet-scale ML systems engineering asks “what can a thousand machines build together, and at what cost to reliability, efficiency, and society?”