Let's build GPT: from scratch, in code, spelled out.
4.7 میلیون بار بازدید -
2 سال پیش
-
We build a Generatively Pretrained
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.
Links:
- Google colab for the video: https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing
- GitHub repo for the video: https://github.com/karpathy/ng-video-lecture
- Playlist of the whole Zero to Hero series so far: https://www.seevid.ir/fa/w/VMj-3S1tku0
- nanoGPT repo: https://github.com/karpathy/nanoGPT
- my website: https://karpathy.ai/
- my twitter: https://twitter.com/karpathy
- our Discord channel: https://discord.gg/3zy8kqD9Cp
Supplementary links:
- Attention is All You Need paper: https://arxiv.org/abs/1706.03762
- OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165
- OpenAI ChatGPT blog post: https://openai.com/blog/chatgpt/
- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com/ . If you prefer to work in notebooks, I think the easiest path today is Google Colab.
Suggested exercises:
- EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).
- EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.)
- EX3: Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?
- EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?
Chapters:
https://www.seevid.ir/fa/w/kCc8FmEb1nY intro: ChatGPT, Transformers, nanoGPT, Shakespeare
baseline language modeling, code setup
https://www.seevid.ir/fa/w/kCc8FmEb1nY reading and exploring the data
https://www.seevid.ir/fa/w/kCc8FmEb1nY tokenization, train/val split
https://www.seevid.ir/fa/w/kCc8FmEb1nY data loader: batches of chunks of data
https://www.seevid.ir/fa/w/kCc8FmEb1nY simplest baseline: bigram language model, loss, generation
https://www.seevid.ir/fa/w/kCc8FmEb1nY training the bigram model
https://www.seevid.ir/fa/w/kCc8FmEb1nY port our code to a script
Building the "self-attention"
https://www.seevid.ir/fa/w/kCc8FmEb1nY version 1: averaging past context with for loops, the weakest form of aggregation
https://www.seevid.ir/fa/w/kCc8FmEb1nY the trick in self-attention: matrix multiply as weighted aggregation
https://www.seevid.ir/fa/w/kCc8FmEb1nY version 2: using matrix multiply
https://www.seevid.ir/fa/w/kCc8FmEb1nY version 3: adding softmax
https://www.seevid.ir/fa/w/kCc8FmEb1nY minor code cleanup
https://www.seevid.ir/fa/w/kCc8FmEb1nY positional encoding
https://www.seevid.ir/fa/w/kCc8FmEb1nY THE CRUX OF THE VIDEO: version 4: self-attention
https://www.seevid.ir/fa/w/kCc8FmEb1nY note 1: attention as communication
https://www.seevid.ir/fa/w/kCc8FmEb1nY note 2: attention has no notion of space, operates over sets
https://www.seevid.ir/fa/w/kCc8FmEb1nY note 3: there is no communication across batch dimension
https://www.seevid.ir/fa/w/kCc8FmEb1nY note 4: encoder blocks vs. decoder blocks
https://www.seevid.ir/fa/w/kCc8FmEb1nY note 5: attention vs. self-attention vs. cross-attention
https://www.seevid.ir/fa/w/kCc8FmEb1nY note 6: "scaled" self-attention. why divide by sqrt(head_size)
Building the Transformer
https://www.seevid.ir/fa/w/kCc8FmEb1nY inserting a single self-attention block to our network
https://www.seevid.ir/fa/w/kCc8FmEb1nY multi-headed self-attention
https://www.seevid.ir/fa/w/kCc8FmEb1nY feedforward layers of transformer block
https://www.seevid.ir/fa/w/kCc8FmEb1nY residual connections
https://www.seevid.ir/fa/w/kCc8FmEb1nY layernorm (and its relationship to our previous batchnorm)
https://www.seevid.ir/fa/w/kCc8FmEb1nY scaling up the model! creating a few variables. adding dropout
Notes on Transformer
https://www.seevid.ir/fa/w/kCc8FmEb1nY encoder vs. decoder vs. both (?) Transformers
https://www.seevid.ir/fa/w/kCc8FmEb1nY super quick walkthrough of nanoGPT, batched multi-headed self-attention
https://www.seevid.ir/fa/w/kCc8FmEb1nY back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF
https://www.seevid.ir/fa/w/kCc8FmEb1nY conclusions
Corrections:
https://www.seevid.ir/fa/w/kCc8FmEb1nY Oops "tokens from the future cannot communicate", not "past". Sorry! :)
https://www.seevid.ir/fa/w/kCc8FmEb1nY Oops I should be using the head_size for the normalization, not C
2 سال پیش
در تاریخ 1401/10/27 منتشر شده
است.
4,706,218
بـار بازدید شده