Mastering Language Models: From Architecture to Optimization

Maya and Leo open the series with the map: seven stops from the Transformer blueprint to the machinery under massive models, anchored by a three-person startup building an insurance-claims assistant on eight GPUs. They lay out the mental models every LLM expert shares — trust curves, find the bottleneck, separate capability from behavior — then stage the field's cleanest fight on air: bigger models versus more data, from OpenAI's 2020 scaling curves to Chinchilla's flip to the serving-cost era that ran past both camps. Plus trailers for the live attention debate and the alignment fight to come.

45 episodes

1mo ago latest

4x a week cadence

English language

▶ Play latestRSS feed Website Technology Education

https://www.williamliu.ai/podcasts/mastering-language-models/feed.xml

Episodes 45

Series Overview — Mastering Language Models: From Architecture to Optimization
4mo ago · 11:21
Maya and Leo open the series with the map: seven stops from the Transformer blueprint to the machinery under massive models, anchored by a three-person startup building an insurance-claims assistant on eight GPUs. They lay out the mental models every LLM expert shares — trust curves, find the bottleneck, separate capability from behavior — then stage the field's cleanest fight on air: bigger models versus more data, from OpenAI's 2020 scaling curves to Chinchilla's flip to the serving-cost era that ran past both camps. Plus trailers for the live attention debate and the alignment fight to come
T1E0 · Foundations of Sequence Modeling: The Transformer Revolution
4mo ago · 13:44
Topic 1 opens the series at the foundation: how 'Attention Is All You Need' replaced step-by-step recurrence with self-attention — every token seeing every other token in one parallel hop — and why that single move made large-scale pre-training possible. Maya and Leo build the topic's shared mental models (attention as content-based lookup, the two bills of training versus serving, architectures as hardware bets), then stage the field's live fight on air: exact full attention versus Kimi Linear's expressive hybrid, with the KV cache and million-token contexts as the battleground. A running law
T1E1 · Attention Is All You Need
2mo ago · 11:33
The first deep dive of the series opens the 2017 Transformer paper itself. Maya walks the machine in three stations — the Matchmaker (query-key-value lookup), the Committee (eight specialist attention heads), and the Chord (wave-stamped word order) — while Leo brings the receipts: a two-point BLEU jump over every published system, base-model training in about twelve hours on eight GPUs, and an ablation table that proves the committee earns its seat. Then the skeptic's pass: what the paper never claimed, why 'foundation of modern AI' was the field's later inference, and how the authors wrote th
T1E2 · Kimi Linear: An Expressive, Efficient Attention Architecture
2mo ago · 13:48
Topic 1 closes with the 2017 paper's confession answered. Kimi Linear, from Moonshot AI's Kimi Team, claims a first: a mostly-linear attention architecture that beats full attention under matched training runs. Maya builds the machine in four landmarks — the Board (a fixed-size memory that never grows), the Eraser (the delta rule's overwrite-don't-pile update), the Knobs (a learned forgetting dial per feature channel, which doubles as the position signal), and the Floorplan (three linear layers for every one full-attention layer, with the full layers carrying no position encoding at all). Leo
T1E3 · Attention Residuals
1mo ago · 12:57
Standard PreNorm residual streams add every layer output with the same fixed weight, so hidden magnitude grows and each new update becomes relatively diluted. Maya and Leo follow the paper's proposed replacement: attention across depth, where a learned pseudo-query selects among RMS-normalized earlier representations. A law-firm contracts assistant grounds the move from an additive pile to a selector and then to block summaries. The hosts also unpack the cache-based pipeline and two-phase schedule that make Block AttnRes practical, debate whether the architectural gain justifies the systems bu
T2E0 · Scaling and Training Large Models Efficiently
4mo ago · 14:16
Topic 2 opens with the question the Transformer made urgent: once you can build big, how should one fixed training budget be split between model size, training tokens, and data quality? Maya and Leo stage the scale-first versus compute-optimal argument in its strongest forms, introduce the smooth-curve predictability of scaling laws, the four interacting knobs of scale, and the two-bills view of training versus inference cost — then map the three deep-dives: Kaplan's scaling laws, Chinchilla's budget correction, and the data-constrained regime where fresh text runs short. Sources: • Scaling La
T2E1 · Scaling Laws for Neural Language Models
2mo ago · 14:04
Deep dive into Kaplan et al.'s Scaling Laws for Neural Language Models (2020), the paper that made giant training runs forecastable. Maya and Leo walk the three landmarks: the ruler — loss falls along smooth power laws in parameters, data, and compute, so cheap pilot runs predict frontier runs; the early exit — larger models learn more per token, so a fixed budget should buy a huge model trained on modest data and stopped before convergence; and the edge of the map — loss is a proxy, curves are fitted to a measured range, and averages can hide brittle rare-task behavior. They stage the real ar
T2E2 · Training Compute-Optimal Large Language Models
2mo ago · 14:12
Deep dive into Hoffmann et al.'s Training Compute-Optimal Large Language Models (2022) — the Chinchilla paper that re-measured the parameters-versus-tokens trade-off and found a generation of large models undertrained. Maya and Leo walk the three landmarks: the rebalance — under a fixed compute budget, model size and training tokens should scale roughly together; the rematch — a seventy-billion-parameter model trained on far more data outperforming much larger models like Gopher at a comparable budget, while also being far cheaper to serve; and the fine print — tokens are not interchangeable,
T2E3 · Scaling Data-Constrained Language Models
2mo ago · 13:59
Deep dive into Muennighoff et al.'s Scaling Data-Constrained Language Models (2023) — the paper that asks what happens when the balanced scaling recipe demands more fresh, high-quality text than exists. Maya and Leo walk the usable shelf (why the responsibly trainable internet is far smaller than the internet), the second pass (epochs and repetition), and the repetition discount (a few passes are surprisingly close to fresh data before value decays — and excess parameters are discounted too). Then they argue out the workarounds the field is split over: relaxed quality filters, code data, synth
T3E0 · Advanced Distributed Training: Overcoming Bottlenecks
2mo ago · 12:35
Topic 3 opens with a map of the bottlenecks that decide whether a hundred-billion-parameter model can be trained at all: model-state memory, activation memory, GPU-to-GPU communication, pipeline bubbles, and the data movement inside a single chip. Maya and Leo stage the field's real argument — partition the model versus shard the redundant states — introduce the four levers (copy, slice, split, shard), and set up the diagnostic habit behind GPipe, Megatron-LM, ZeRO, FSDP, and FlashAttention: find the bottleneck first, then choose the technique. Sources: • GPipe: Efficient Training of Giant Neu
T3E1 · GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
2mo ago · 11:55
The first deep dive of Topic 3 takes on the bluntest bottleneck: the model does not fit on one device. Maya and Leo unpack GPipe's move — slice the layer stack into stages, stream microbatches through them like trays down a sandwich line, and re-materialize activations instead of storing them — then stage the field's real argument between pipeline and tensor parallelism: idle bubbles versus constant communication, reach across servers versus fully busy chips. Plus the trap of equal-layer splits, and the four measurements that tell you whether a pipeline is actually parallel or only looks that
T3E2 · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
2mo ago · 13:43
Episode two of Topic 3 goes inside the layer. When a single Transformer layer is too big for one chip, no pipeline schedule can save you — so Megatron-LM cuts the matrix multiplications themselves across GPUs, column-wise then row-wise, with an all-reduce 'huddle' only where partial results must meet. Maya and Leo walk the feed-forward and attention splits with their sixty-four-GPU team, then swap chairs and restage the tensor-versus-pipeline fight from the other side: no bubbles and graduate-student-readable code versus a conference call that never hangs up and a ceiling at the server chassis
T3E3 · ZeRO: Memory Optimization Towards Training Trillion Parameter Models
2mo ago · 13:07
Episode three of Topic 3 stops cutting the model. After two episodes of pipeline and tensor surgery, ZeRO — the Zero Redundancy Optimizer — asks the almost-rude question: what if the real waste was never the math, but the copies? Maya and Leo open in a town where eight libraries stock identical collections and ledgers fatter than the books, then follow the sharding ladder — optimizer states, gradients, finally the parameters themselves — as memory turns from possession into scheduling. They stage the split-the-model versus shard-the-states argument with concessions on both sides, land on the p
T3E4 · Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs
2mo ago · 11:34
Episode four of Topic 3 is the sequel to ZeRO's argument: what happens when sharding wins and moves into PyTorch as Fully Sharded Data Parallel. Maya and Leo open in a machine shop where no parts live at the bench — crates arrive exactly when a job needs them — and follow FSDP's loop of gathering full parameters for one wrapped block, computing locally, and letting the copy go. Then the four negotiations that decide whether the run gets faster or merely fits: crate size (wrapping policy), departure time (gather and prefetch), the recompute bargain (activation checkpointing), and the overflow a
T3E5 · Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing
2mo ago · 13:20
Episode five of Topic 3 steps back from single techniques to the whole system. Maya and Leo open at a container port at dawn — the cranes are the postcard, but the slowest gate decides when the ship leaves — and use a 2025 ACM survey to define a training architecture as a distributed system with machine-learning math inside it. They walk six harbor-named stops where real runs get caught: the Channels (topology), the Berth Plan (scheduling and placement), the Feeder Road (data supply), the Logbook Window (checkpointing), the Watchtower (monitoring), and the Recovery Drill (fault tolerance). A s
T3E6 · FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
2mo ago · 12:11
Episode six of Topic 3 stops spreading work across machines and crawls inside a single GPU. FlashAttention's accusation is that attention was slow for the wrong reason: not too much math, too much traffic — the quadratic score matrix hauled back and forth between high-bandwidth memory and the tiny on-chip SRAM beside the compute units. Maya and Leo open at a laundromat where the machines were never the problem, walk through tiling and the online-softmax running tally that keeps blockwise attention mathematically exact, then stage the field's real fight: approximate the asymptote or engineer th
T3E7 · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
2mo ago · 11:45
Episode seven of Topic 3 stays inside the GPU for the sequel. FlashAttention ended attention's memory commute, and the same author's audit found the chip still far from busy — so FlashAttention-2 re-divides the labor instead of the math: less slow non-matmul bookkeeping, a single attention head's work split across many thread blocks so long-sequence runs fill the machine, and warps exchanging less through shared memory. Maya maps the three findings onto a moving crew — the tape gun, the empty stairwells, the over-the-shoulder pass — while Leo reads the stopwatch: about twice the already-fast k
T3E8 · Embarrassingly Simple Self-Distillation Improves Code Generation (SSD)
2mo ago · 11:59
Episode eight of Topic 3 steps out of the machine room. After seven episodes of chips, memory, cables, and schedules, the bottleneck moves to the post-training bill — and the paper attacking it is almost suspiciously simple. SSD — Embarrassingly Simple Self-Distillation — samples code solutions from a model under chosen temperature and truncation settings, then fine-tunes the same model on those raw samples with standard supervised fine-tuning. No stronger teacher, no execution verifier, no reward model: three chairs that post-training pipelines normally keep filled, all empty. Maya builds the
T3E9 · Test-Time Scaling Makes Overtraining Compute-Optimal
2mo ago · 12:44
The final episode of Topic 3 closes the loop by picking a fight with Topic 2. Chinchilla's compute-optimal recipe balances model size against training tokens under a training budget — but deployed reasoning systems don't pay for one forward pass per task. They sample candidates and vote, run verifiers, search, retry. Today's paper, Test-Time Scaling Makes Overtraining Compute-Optimal, writes down Train-to-Test — T-squared — scaling laws: one end-to-end budget, three knobs turned jointly — model size, training tokens, and inference samples. Maya builds the accounting through a math olympiad tha
T4E0 · Fine-Tuning and Specialization: LoRA and Beyond
2mo ago · 14:54
Topic 4 opens with the move that defines modern model specialization: freeze the giant base model and train a tiny low-rank update beside it. Maya and Leo map the topic's landmarks — the Rank Knob, the Precision Floor, the What-Got-Worse Test, and the Long Haul — and stage the field's real arguments: PEFT-everywhere versus full fine-tuning's capacity case, and how many bits the frozen base can lose before quality quietly collapses. A hospital discharge-note summarizer anchors the whole topic, setting up deep dives on LoRA, QLoRA, LowRA, and continual learning. Sources: • LoRA: Low-Rank Adaptat
T4E1 · LoRA: Low-Rank Adaptation of Large Language Models
2mo ago · 13:38
The paper that made fine-tuning feel modular. Maya and Leo open up LoRA's central trick — freeze the pretrained weights and learn the update as the product of two thin matrices, around sixty-five thousand trainable numbers standing in for sixteen million — then follow it through the merge fork (flatten for zero-overhead serving, or keep adapters swappable on one frozen base), the rank and alpha dials, and why low intrinsic dimension makes the whole bet plausible. Two real practitioner arguments get staged on air: attention-only versus broad target modules, and whether LoRA's cheapness has made
T4E2 · QLoRA: Efficient Finetuning of Quantized LLMs
2mo ago · 14:11
The paper that put large-model fine-tuning on a single GPU. Maya and Leo open QLoRA's central rule — read the compressed thing, write somewhere else — and follow gradients through a frozen four-bit base into full-precision LoRA adapters. Along the way: NF4's bell-curve-shaped buckets, double quantization's compress-the-labels trick, paged optimizers as the relief valve that saves hour-nine runs, and the thousand-model study behind Guanaco that both topped the Vicuna benchmark and warned against trusting chatbot benchmarks. The staged debate takes on the choice QLoRA created: a bigger model at
T4E3 · LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
2mo ago · 13:51
The paper that asks how few bits an adapter can survive on. Maya opens with a mosaic master firing a portrait in four tile shades, and the analogy turns out to be the method: LowRA pushes LoRA fine-tuning below two bits per parameter through three deliberate decisions — the Palette (which values the codes stand for), the Cut Lines (where bucket boundaries sit), and the Bit Budget (where bits get spent) — plus the CUDA kernels that keep the memory win from leaking back out as runtime. The reported floor: accuracy down to about 1.15 bits, with up to fifty percent memory reduction. Then the stage
T4E4 · Continual Learning of Large Language Models: A Comprehensive Survey
2mo ago · 15:47
Topic 4 closes by stretching specialization across time. Maya opens with an interpreter in Lisbon losing words in her own first language — first-language attrition as the human face of catastrophic forgetting — and the survey behind the episode maps the machine version: erosion with good manners, where the model stays fluent while old domains, formats, and safety behaviors quietly slip. The hosts walk the survey's two axes (vertical apprenticeship versus horizontal keeping-up), its three training stages, and four named defense families — the Songbook, the Tether, the New Wing, and the Diet — t
T5E0 · Reinforcement Learning from Human Feedback (RLHF)
3mo ago · 14:02
Maya and Leo open Topic 5 by mapping reinforcement learning from human feedback as a preference-control loop — feedback bench, preference mirror, reward compass, drift fence — using a healthcare scheduling assistant as the running example. They stage the field's real fights in first person: Maya defends the reward-model-plus-PPO pipeline while Leo argues DPO's simplicity, then they swap heat over human versus AI feedback, before closing on reward hacking, robustness, and what evaluation must catch. Sources: • Proximal Policy Optimization Algorithms: https://arxiv.org/pdf/1707.06347 • Learning

Episodes

Reading the feed…

More Technology shows

Mastering Language Models: From Architecture to Optimization

Episodes 45

Series Overview — Mastering Language Models: From Architecture to Optimization

T1E0 · Foundations of Sequence Modeling: The Transformer Revolution

T1E1 · Attention Is All You Need

T1E2 · Kimi Linear: An Expressive, Efficient Attention Architecture

T1E3 · Attention Residuals

T2E0 · Scaling and Training Large Models Efficiently

T2E1 · Scaling Laws for Neural Language Models

T2E2 · Training Compute-Optimal Large Language Models

T2E3 · Scaling Data-Constrained Language Models

T3E0 · Advanced Distributed Training: Overcoming Bottlenecks

T3E1 · GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

T3E2 · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

T3E3 · ZeRO: Memory Optimization Towards Training Trillion Parameter Models

T3E4 · Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

T3E5 · Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing

T3E6 · FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

T3E7 · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

T3E8 · Embarrassingly Simple Self-Distillation Improves Code Generation (SSD)

T3E9 · Test-Time Scaling Makes Overtraining Compute-Optimal

T4E0 · Fine-Tuning and Specialization: LoRA and Beyond

T4E1 · LoRA: Low-Rank Adaptation of Large Language Models

T4E2 · QLoRA: Efficient Finetuning of Quantized LLMs

T4E3 · LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

T4E4 · Continual Learning of Large Language Models: A Comprehensive Survey

T5E0 · Reinforcement Learning from Human Feedback (RLHF)

Episodes

More Technology shows

AppleVis Podcast

Material

Área de Transferência

Super Feed

Don't Panic

HTML All The Things - Web Development, AI, and Developer Careers

Episodes 45

Series Overview — Mastering Language Models: From Architecture to Optimization

T1E0 · Foundations of Sequence Modeling: The Transformer Revolution

T1E1 · Attention Is All You Need

T1E2 · Kimi Linear: An Expressive, Efficient Attention Architecture

T1E3 · Attention Residuals

T2E0 · Scaling and Training Large Models Efficiently

T2E1 · Scaling Laws for Neural Language Models

T2E2 · Training Compute-Optimal Large Language Models

T2E3 · Scaling Data-Constrained Language Models

T3E0 · Advanced Distributed Training: Overcoming Bottlenecks

T3E1 · GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

T3E2 · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

T3E3 · ZeRO: Memory Optimization Towards Training Trillion Parameter Models

T3E4 · Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

T3E5 · Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing

T3E6 · FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

T3E7 · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

T3E8 · Embarrassingly Simple Self-Distillation Improves Code Generation (SSD)

T3E9 · Test-Time Scaling Makes Overtraining Compute-Optimal

T4E0 · Fine-Tuning and Specialization: LoRA and Beyond

T4E1 · LoRA: Low-Rank Adaptation of Large Language Models

T4E2 · QLoRA: Efficient Finetuning of Quantized LLMs

T4E3 · LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

T4E4 · Continual Learning of Large Language Models: A Comprehensive Survey

T5E0 · Reinforcement Learning from Human Feedback (RLHF)