Replace Week 7 GPT content with diffusion models (lectures 21-23, animations, notebook, cross-refs)

Author: jeremymanning

admin/syllabus.md

@@ -119,7 +119,7 @@ We strive to create an inclusive learning environment where all students feel su
 | 4 | Jan 26--30 | Word Embeddings, Contextual Embeddings, Dimensionality Reduction | [Assignment 2](https://contextlab.github.io/llm-course/assignments/assignment-2/) due (Jan 26) |
 | 5 | Feb 2--6 | Attention Mechanisms, Transformer Architecture | [Assignment 3](https://contextlab.github.io/llm-course/assignments/assignment-3/) due (Feb 6), [Assignment 4](https://contextlab.github.io/llm-course/assignments/assignment-4/) released (Feb 6) |
 | 6 | Feb 9--13 | BERT Deep Dive, Encoder Models | |
-| 7 | Feb 16--20 | GPT Architecture, Scaling Up, Implementation | [Assignment 4](https://contextlab.github.io/llm-course/assignments/assignment-4/) due (Feb 16), [Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) released (Feb 16), [Assignment 5](https://contextlab.github.io/llm-course/assignments/assignment-5/) available (optional/extra credit) |
+| 7 | Feb 16--20 | Diffusion Models, Extensions, Applications & Ethics | [Assignment 4](https://contextlab.github.io/llm-course/assignments/assignment-4/) due (Feb 16), [Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) released (Feb 16), [Assignment 5](https://contextlab.github.io/llm-course/assignments/assignment-5/) available (optional/extra credit) |
 | 8 | Feb 23--27 | **No classes** (instructor away) | Work on [Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) |
 | 9 | Mar 2--6 | RAG, Mixture of Experts, Ethics | |
 | 10 | Mar 9 | Final Project Presentations | [Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) due (Mar 9, 11:59 PM EST) |
@@ -180,13 +180,13 @@ We strive to create an inclusive learning environment where all students feel su
 | Wed Feb 11 | Lecture 19 | BERT Variants | [HTML](https://contextlab.github.io/llm-course/slides/week6/lecture19.html), [PDF](https://contextlab.github.io/llm-course/slides/week6/lecture19.pdf) | [Liu et al. (2019)](https://arxiv.org/abs/1907.11692), [Sanh et al. (2019)](https://arxiv.org/abs/1910.01108) |
 | Fri Feb 13 | Lecture 20 | Applications of Encoder Models | [HTML](https://contextlab.github.io/llm-course/slides/week6/lecture20.html), [PDF](https://contextlab.github.io/llm-course/slides/week6/lecture20.pdf) | |
 \needspace{3in}
-### Week 7: Decoder Models & GPT (February 16--20)
+### Week 7: Diffusion Models (February 16--20)
 
 | Day | Lecture | Topics | Slides | Materials |
 |-----|---------|--------|--------|-----------|
-| Mon Feb 16 | Lecture 21 | GPT Architecture | [HTML](https://contextlab.github.io/llm-course/slides/week7/lecture21.html), [PDF](https://contextlab.github.io/llm-course/slides/week7/lecture21.pdf) | [Radford et al. (2018)](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf), [Radford et al. (2019)](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), **[Assignment 4](https://contextlab.github.io/llm-course/assignments/assignment-4/) Due (Feb 16, 11:59 PM EST)**, **[Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) Released**, **[Assignment 5](https://contextlab.github.io/llm-course/assignments/assignment-5/) Available (Optional/Extra Credit)** |
-| Wed Feb 18 | Lecture 22 | Scaling Up to GPT-3 and Beyond | [HTML](https://contextlab.github.io/llm-course/slides/week7/lecture22.html), [PDF](https://contextlab.github.io/llm-course/slides/week7/lecture22.pdf) | [Brown et al. (2020)](https://arxiv.org/abs/2005.14165), [OpenAI (2023)](https://arxiv.org/abs/2303.08774) |
-| Fri Feb 20 | Lecture 23 | Implementing GPT from Scratch | [HTML](https://contextlab.github.io/llm-course/slides/week7/lecture23.html), [PDF](https://contextlab.github.io/llm-course/slides/week7/lecture23.pdf) | [Let's build GPT (Karpathy)](https://www.youtube.com/watch?v=kCc8FmEb1nY) |
+| Mon Feb 16 | Lecture 21 | Diffusion Models | [HTML](https://contextlab.github.io/llm-course/slides/week7/lecture21.html), [PDF](https://contextlab.github.io/llm-course/slides/week7/lecture21.pdf) | [Ho et al. (2020)](https://arxiv.org/abs/2006.11239), [Song & Ermon (2019)](https://arxiv.org/abs/1907.05600), **[Assignment 4](https://contextlab.github.io/llm-course/assignments/assignment-4/) Due (Feb 16, 11:59 PM EST)**, **[Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) Released**, **[Assignment 5](https://contextlab.github.io/llm-course/assignments/assignment-5/) Available (Optional/Extra Credit)** |
+| Wed Feb 18 | Lecture 22 | Diffusion Model Extensions | [HTML](https://contextlab.github.io/llm-course/slides/week7/lecture22.html), [PDF](https://contextlab.github.io/llm-course/slides/week7/lecture22.pdf) | [Rombach et al. (2022)](https://arxiv.org/abs/2112.10752), [Peebles & Xie (2023)](https://arxiv.org/abs/2212.09748) |
+| Fri Feb 20 | Lecture 23 | Diffusion Applications and Ethics | [HTML](https://contextlab.github.io/llm-course/slides/week7/lecture23.html), [PDF](https://contextlab.github.io/llm-course/slides/week7/lecture23.pdf) | [Ramesh et al. (2022)](https://arxiv.org/abs/2204.06125), [OpenAI (2024) - Sora](https://openai.com/research/video-generation-models-as-world-simulators) |
 ### Week 8: No Classes (February 23--27)
 
 Instructor away February 23--27. No classes this week. Use this time to work on your [final project](https://contextlab.github.io/llm-course/assignments/final-project/).

notes/2026-02-12-week7-diffusion-rewrite.md

@@ -0,0 +1,35 @@
+# Week 7 Diffusion Rewrite — Session Notes (2026-02-12)
+
+## Status
+Replacing Week 7 from "GPT/Decoder Models" to "Diffusion Models".
+
+## Completed (committed)
+- Cross-reference updates: lecture20.md, lecture27.md, README.md, syllabus.md (uncommitted changes from prior session)
+- Animation infrastructure: render_gifs.py, postprocess_gifs.py, requirements.txt, README.md (uncommitted new files)
+
+## In Progress (background tasks)
+All 4 content files are being generated by background agents:
+
+| Task ID | File | Agent | Session ID |
+|---------|------|-------|------------|
+| bg_55690b17 | lecture21.md (diffusion architecture) | deep | ses_3abe4e6ccffeIxuC41mJgn0fb9 |
+| bg_66ef8d46 | lecture22.md (extensions) | deep | ses_3abe480feffeANUy7UfoxJ9QLd |
+| bg_4ba2b2b4 | lecture23.md (applications/ethics) | deep | ses_3abe4148affeNSF9u0KPz5qPvA |
+| bg_45dcd847 | diffusion_scenes.py (10 Manim scenes) | ultrabrain | ses_3abe320d3ffepDE9jAj9uyQEh8 |
+
+## Remaining after content creation
+1. Create diffusion_demo.ipynb companion notebook
+2. Render Manim GIFs (requires manim venv)
+3. Compile all 3 lectures with compile.sh
+4. Verify via Playwright screenshots
+5. Verify citation URLs
+6. Run tests
+7. Commit and push
+
+## Key Constraints
+- Assignment announcements ONLY in lecture21 (not 22/23)
+- Companion notebook referenced ONLY in lecture23
+- Animation GIFs ONLY in lecture21
+- gpt_from_scratch_demo.ipynb stays as-is (Assignment 5 ref)
+- Sentence case capitalization
+- Use 'C' for embedding dimension (consistency with lecture 15)

slides/README.md

@@ -237,43 +237,45 @@ Explore concepts hands-on with our interactive web demos! Each demo runs directl
 - 📊 [Slides PDF](https://contextlab.github.io/llm-course/slides/week6/lecture20.pdf) | 🌐 [Slides HTML](https://contextlab.github.io/llm-course/slides/week6/lecture20.html)
 ---
 
-## Week 7: Decoder Models & GPT
-
-**Monday (Lecture 21):** GPT Architecture
-- Generative pre-training paradigm, fine-tuning, weight tying
-- The BooksCorpus controversy: training data ethics
-- Open-weight decoders: the LLaMA revolution
-- Test-time compute and inference scaling
-- Modern decoder innovations: RMSNorm, SwiGLU, RoPE, GQA
-- Multi-token prediction and hybrid architectures (Jamba)
-- Reading: [Radford et al. (2018)](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) - GPT-1
-- Reading: [Radford et al. (2019)](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) - GPT-2
-- Reading: [Touvron et al. (2023)](https://arxiv.org/abs/2302.13971) - LLaMA
-- Reading: [Gloeckle et al. (2024)](https://arxiv.org/abs/2404.19737) - Multi-token prediction
+## Week 7: Diffusion Models
+
+**Monday (Lecture 21):** Diffusion Models
+- The diffusion architecture: forward process, reverse process, noise schedules
+- U-Net for noise prediction with timestep conditioning
+- Training objective: predicting the noise (simplified DDPM loss)
+- Score matching intuition and sampling algorithms (DDPM, DDIM)
+- Animated walkthrough of the complete diffusion pipeline
+- Reading: [Ho, Jain & Abbeel (2020)](https://arxiv.org/abs/2006.11239) - Denoising Diffusion Probabilistic Models
+- Reading: [Sohl-Dickstein et al. (2015)](https://arxiv.org/abs/1503.03585) - Deep Unsupervised Learning using Nonequilibrium Thermodynamics
+- Reading: [Song & Ermon (2019)](https://arxiv.org/abs/1907.05600) - Generative Modeling by Estimating Gradients
+- Reading: [Nichol & Dhariwal (2021)](https://arxiv.org/abs/2102.09672) - Improved DDPM
 - **📝 Assignment 4 Due (Feb 16, 11:59 PM EST)**
 - **Final Project Released:** [Final Project](https://contextlab.github.io/llm-course/assignments/final-project/) (Due: Mar 9, 11:59 PM EST)
 - **Assignment 5 Available (Optional/Extra Credit):** [Build GPT](https://contextlab.github.io/llm-course/assignments/assignment-5/)
-- 🎮 **Try it:** [GPT Playground](https://contextlab.github.io/llm-course/demos/gpt-playground/)
 - 📊 [Slides PDF](https://contextlab.github.io/llm-course/slides/week7/lecture21.pdf) | 🌐 [Slides HTML](https://contextlab.github.io/llm-course/slides/week7/lecture21.html)
 
-**Wednesday (Lecture 22):** Scaling Up to GPT-3 and Beyond
-- From GPT-1 to GPT-4: the scaling journey
-- Beyond RLHF: DPO and GRPO alignment methods
-- Reasoning models (o1, o3, DeepSeek-R1) and emergent abilities debate
-- Reading: [Brown et al. (2020)](https://arxiv.org/abs/2005.14165) - GPT-3
-- Reading: [Rafailov et al. (2023)](https://arxiv.org/abs/2305.18290) - DPO
-- Reading: [Schaeffer et al. (2023)](https://arxiv.org/abs/2304.15004) - Are Emergent Abilities a Mirage?
+**Wednesday (Lecture 22):** Diffusion Model Extensions
+- Latent diffusion: VAE compression for efficient generation
+- Classifier-free guidance for conditional generation
+- Diffusion Transformer (DiT): replacing U-Net with transformers
+- Flow matching and rectified flow: simpler, faster diffusion
+- Stable Diffusion 3: MMDiT architecture
+- Reading: [Rombach et al. (2022)](https://arxiv.org/abs/2112.10752) - Latent Diffusion Models
+- Reading: [Ho & Salimans (2022)](https://arxiv.org/abs/2207.12598) - Classifier-Free Diffusion Guidance
+- Reading: [Peebles & Xie (2023)](https://arxiv.org/abs/2212.09748) - Scalable Diffusion Models with Transformers (DiT)
+- Reading: [Lipman et al. (2023)](https://arxiv.org/abs/2210.02747) - Flow Matching for Generative Modeling
 - 📊 [Slides PDF](https://contextlab.github.io/llm-course/slides/week7/lecture22.pdf) | 🌐 [Slides HTML](https://contextlab.github.io/llm-course/slides/week7/lecture22.html)
 
-**Friday (Lecture 23):** Implementing GPT from Scratch
-- Build a complete mini-GPT (~30M params) in PyTorch
-- Tokenization, embeddings, masked attention, transformer blocks, training loop
-- Weight tying, gradient accumulation, LR scheduling, mixed precision training
-- Text generation: greedy, temperature, top-k, nucleus sampling
-- KV caching, FlashAttention, and nanoGPT comparison
-- Tutorial: [Let's build GPT (Karpathy)](https://www.youtube.com/watch?v=kCc8FmEb1nY)
-- Reading: [Dao et al. (2022)](https://arxiv.org/abs/2205.14135) - FlashAttention
-- 📓 [Companion Notebook](https://colab.research.google.com/github/ContextLab/llm-course/blob/main/slides/week7/gpt_from_scratch_demo.ipynb)
+**Friday (Lecture 23):** Diffusion Applications and Ethics
+- Text-to-image: DALL-E 2, Stable Diffusion, Imagen
+- Text-to-video: Sora and spacetime patches
+- Text-to-audio: FluxMusic and spectral diffusion
+- Discrete diffusion for text generation (MDLM)
+- Ethics: deepfakes, consent, bias in generation, regulation
+- Reading: [Ramesh et al. (2022)](https://arxiv.org/abs/2204.06125) - DALL-E 2
+- Reading: [OpenAI (2024)](https://openai.com/research/video-generation-models-as-world-simulators) - Sora
+- Reading: [Sahoo et al. (2024)](https://arxiv.org/abs/2406.07524) - Masked Diffusion Language Models
+- 📓 [Companion Notebook](https://colab.research.google.com/github/ContextLab/llm-course/blob/main/slides/week7/diffusion_demo.ipynb)
 - 📊 [Slides PDF](https://contextlab.github.io/llm-course/slides/week7/lecture23.pdf) | 🌐 [Slides HTML](https://contextlab.github.io/llm-course/slides/week7/lecture23.html)
 ---
 

slides/week10/lecture27.md

@@ -108,7 +108,7 @@ Please be a supportive and engaged audience -- ask questions, offer feedback, an
 | 3--4 | Embeddings (LSA → Word2Vec → ELMo) | Meaning lives in vector space |
 | 5 | Transformer architecture | Attention is all you need |
 | 6 | BERT and encoder models | Bidirectional understanding |
-| 7 | GPT and decoder models | Autoregressive generation at scale |
+| 7 | Diffusion models | Iterative refinement and generation beyond text |
 | 8 | *(Break)* | |
 | 9 | Agents, MoE, ethics | Action, efficiency, responsibility |
 | 10 | **Your projects** | Putting it all together |

slides/week6/lecture20.md

@@ -392,6 +392,6 @@ Which framework you adopt determines your answer to every question on the next s
 
 <div class="tip-box" data-title="Up next...">
 
-Week 7: GPT and autoregressive generation — from decoder architecture to scaling laws
+Week 7: Diffusion models — from denoising to text-to-image generation
 
 </div>

slides/week7/animations/README.md

@@ -0,0 +1,79 @@
+# Diffusion Model Animation GIFs for Lecture 21
+
+This directory contains Python Manim scripts to generate animated GIFs for the diffusion model lecture slides.
+
+## Prerequisites
+
+1. Python 3.9+
+2. System dependencies for Manim:
+   - macOS: `brew install py3cairo ffmpeg pango`
+   - Ubuntu: `sudo apt install libcairo2-dev ffmpeg libpango1.0-dev`
+
+3. Avenir font (for macOS, this is pre-installed)
+
+## Setup
+
+```bash
+cd slides/week7/animations
+python -m venv venv
+source venv/bin/activate  # or `venv\Scripts\activate` on Windows
+pip install -r requirements.txt
+```
+
+## Rendering GIFs
+
+```bash
+python render_gifs.py
+```
+
+This will:
+1. Render each scene as an MP4 via Manim
+2. Convert to high-quality GIFs with transparent backgrounds
+3. Save GIFs to `./gifs/` directory
+
+## Output
+
+Generated GIFs:
+- `forwarddiffusion.gif` - Progressive noising from clean image to pure noise
+- `noiseschedule.gif` - Linear vs cosine noise schedules
+- `reverseprocess.gif` - Iterative denoising from noise to clean image
+- `unetarchitecture.gif` - U-Net encoder-decoder with skip connections
+- `timestepembedding.gif` - Sinusoidal timestep conditioning
+- `trainingobjective.gif` - Training loop visualization
+- `simplifiedloss.gif` - DDPM simplified loss equation breakdown
+- `scorematching.gif` - Score function vector field visualization
+- `samplingprocess.gif` - DDPM sampling algorithm step by step
+- `diffusionvstransformer.gif` - Autoregressive vs diffusion paradigm comparison
+
+## Rendering Individual Scenes
+
+```bash
+# Render a single scene
+manim -qh --format=mp4 -r 1920,1080 --fps 24 diffusion_scenes.py ForwardDiffusion
+
+# Preview (faster, lower quality)
+manim -ql --format=gif --transparent diffusion_scenes.py ForwardDiffusion
+```
+
+## Customization
+
+Edit `diffusion_scenes.py` to modify:
+- Colors (Dartmouth palette used throughout)
+- Font sizes
+- Animation timing
+- Grid sizes and values
+
+## Integration with Slides
+
+The generated GIFs are designed to have transparent backgrounds, allowing them to overlay seamlessly on the slide background. Reference them in lecture21.md like:
+
+```markdown
+![height:500](animations/gifs/forwarddiffusion.gif)
+```
+
+## Content Source
+
+Animation content based on:
+- [Ho, Jain & Abbeel (2020)](https://arxiv.org/abs/2006.11239) - DDPM
+- [Song & Ermon (2019)](https://arxiv.org/abs/1907.05600) - Score matching
+- [Ronneberger et al. (2015)](https://arxiv.org/abs/1505.04597) - U-Net architecture