Building makemore Part 4: Becoming a Backprop Ninja
We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd’s (): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks.
!!!!!!!!!!!!
I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here:
!!!!!!!!!!!!
Links:
- makemore on github:
- jupyter notebook I built in this video:
- collab notebook:
- my website:
- my twitter:
- our Discord channel:
Supplementary links:
- Yes you should understand backprop:
- BatchNorm paper:
- Bessel’s Correction:
- Bengio et al. 2003 MLP LM
Chapters:
00:00:00 intro: why you should care & fun history
00:07:26 starter code
00:13:01 exercise 1: backproping the atomic compute graph
01:05:17 brief digression: bessel’s correction in batchnorm
01:26:31 exercise 2: cross entropy loss backward pass
01:36:37 exercise 3: batch norm layer backward pass
01:50:02 exercise 4: putting it all together
01:54:24 outro
1 view
130
27
4 months ago 00:55:49 1
Inside the Life of Titus 🇺🇸
4 months ago 00:43:58 1
45 Min Best Full Body Workout With Dumbbells - No Repeat, Home Workout
4 months ago 00:40:04 1
40 MIN PILATES TONE FULL BODY: light weights, ankle weights, booty band
4 months ago 00:40:37 1
40 MIN PILATES X STRENGTH WORKOUT: dumbbells and Pilates total full body at home workout
4 months ago 00:02:48 1
WTC Simulation - World Trade Center Case Study - Blender Demolition (Demo 3)
4 months ago 00:07:08 1
Presale Crypto | Presale Crypto Tokens | Early Crypto Presale
4 months ago 00:23:59 1
1,200HP TWIN TURBO BMW E30 Wide Body Build | INSANE V8 Power with SEQUENTIAL Shift
4 months ago 04:24:15 1
30 Days Alone Building Wood And Stone Warm Shelter in The Woods, Smoked Eel, Catch and Cook
4 months ago 00:08:08 1
Tim Minchin “One day you’ll be as good as Elton John“ | Universal Comedy
4 months ago 00:10:28 1
Building $10 Million Offshore Wind Turbine in Middle of the Sea
4 months ago 00:29:31 1
I Made $114,127,975,963 by Exploiting the Public and Collapsing the Film Industry - Blockbuster Inc
4 months ago 00:10:48 1
Warren Buffett’s 19 Game-Changing Saving Tips
4 months ago 00:06:53 1
The Game-Changer in Action | Building a Custom PC with EK Nexe in Record Time!
4 months ago 00:00:10 1
This dog saved its owner from a falling branch 🤨 Real or fake
4 months ago 00:21:39 1
Hrithik Roshan & Vinod Rawat: How Anyone Can Be An Actor, Reinventing Yourself and More! | Pushtaini
4 months ago 00:10:02 1
Inside Abandoned Olympic Buildings Around The World
4 months ago 00:06:22 1
Building a better world with wood? ⏲️ 6 Minute English
4 months ago 00:06:41 1
Audi Films: An Avant Story
4 months ago 00:17:16 1
How to read ALL NUMBERS in English Big numbers Decimals/Ordinal/Fractions/Dates/Addresses/Phone
4 months ago 00:05:56 1
1969 Dodge Super Bee, Gateway Classic Cars Louisville #3093 LOU
4 months ago 00:18:06 1
🎵 Hans Zimmer - Interstellar Organ Suite (Salisbury Cathedral Organ)
4 months ago 00:11:15 1
Elden Ring Backhand Blade Build - How to Build a Knight Blade Guide (Shadow of the Erdtree Build)
4 months ago 02:12:46 1
Kis Kisko Pyaar Karoon | Hindi Full Movie | Kapil Sharma | Varun Sharma | Hindi Comedy Movies
4 months ago 00:16:29 5
catapult travel system, to ashlands from home: valheim