دانلود کتاب هوش مصنوعی پس از آموزش، راهنمای عملی برای تنظیم دقیق و یادگیری تقویتی

Post-training is the second training phase a modern language model goes through. The three subsections below separate it from pre-training, name the techniques that make up a typical post-training pipeline, and mark the limits of what post-training can change. Building a modern language model happens in two phases. Pre-training comes first. It consumes trillions of tokens of text (web pages, books, code repositories, scientific papers) and trains the model to predict the next token in a sequence. This is an unsupervised objective. The model is not told what good output looks like. Instead, it is told what the next word is across the entire corpus. The result is a model that has absorbed the statistical structure of language, along with a broad and noisy representation of world knowledge. Pre-training is expensive. Training a frontier model from scratch requires thousands of GPUs running for weeks or months, and datasets measured in the tens of terabytes. The investment buys something valuable but incomplete: a base model that can continue any text sequence in a plausible way, but cannot respond to a user’s instructions. Prompt a base model with a question and it may answer the question, or it may generate five more questions, or it may continue with something that looks like a textbook paragraph. It is completing text, not responding to a user’s instructions. Post-training is the second phase. It takes the base model and applies smaller, targeted training runs that shape the model’s behavior. If pretraining teaches the model what language looks like, post-training teaches the model how to use that knowledge. The training data is smaller (thousands to millions of examples rather than trillions of tokens), but it is curated, structured, and designed to produce specific behavioral outcomes. The two phases optimize different signals. Pre-training on a broad corpus builds representations that transfer across tasks. Post-training on curated data uses those representations to produce targeted behavior. Training a model from scratch on curated instruction data produces worse results because the model never develops the broad representations that make it flexible. Skipping post-training produces a model that has knowledge but no discipline in how it uses that knowledge.