Welcome :)
Wi-Fi
Name: FAC
Pass:
StochasticParrots27
Materials
Discord
Starting at 18:10
Attention is all you need
How to build a transformer from scratch
Everything behind ChatGPT fits in one architecture, the transformer: the T in GPT. Tonight, following Andrej Karpathy's brilliant tutorial "Let's build GPT", we build one from nothing and watch it learn to write Shakespeare.
Next-character prediction
The text-generation loop
Every GPT does exactly one thing: read the text so far, output a probability for every possible next character, sample one, go again. All the text it will ever write comes out of this loop.
Tokenisation
A line of text and its integer encoding
Our whole world is one long file of text, over a million characters. Each unique character gets a number, and that is the entire tokeniser. Be clear what these integers are not: they are not vectors and they carry no meaning yet. They are just numbers, the raw numerical substrate, because everything inside a transformer has to be a number. All this slide does is turn text into numbers. (Real models tokenise word fragments instead: tens of thousands of fragments rather than tens of characters, trading a bigger vocabulary for a context window that covers far more text. Character level keeps our build small and inspectable.)
The bigram model
The bigram table: current, next, probability
This model trains and generates. What can no amount of training fix?
This is step one, and it is deliberately primitive: predict the next character from the current one alone, by reading a probability straight off a table. It is only slightly better than random. Train it, run it, and you get islands of recognisable text, fragmented and incoherent, because every guess sees exactly one character.
Answer: Each prediction sees exactly one character. Everything useful sits earlier in the line, and the table cannot reach it. More training cannot help; only more context can.
Embeddings
Each character becomes a learned vector
The integers carried no meaning. So where does meaning come from?
Until now a character was just an integer, an address with no meaning of its own. Here we give each character a vector: a short list of numbers looked up from a learned table, one row per character. They begin as random numbers, and as the model trains to predict the next character those numbers drift into whatever pattern predicts best. That is where meaning enters, not put in by us, but squeezed in by the task. From here on everything the model does, averaging, comparing, attending, is arithmetic on these vectors, never on the bare integers.
Answer: From learning, not from us. Each character's vector starts as random numbers and is nudged over training towards values that help predict the next character. The integer was only an address; the vector is what the model actually works with, and the task is what fills it with meaning.
Bag of characters
Predicting the next character from a bag
Reframe the whole thing: take the characters so far, h, a, l, l, average them into one vector, then predict the next character from that vector. This is a bag of characters, Karpathy's bag of words (just averaging) moved onto characters. The vector is a learned numerical summary, not human meaning: the model starts from random and learns whatever summary predicts best. And here is the word-game intuition: given a hand of letters like h, a, l, l, only a few continuations are plausible, so even a flat average already pushes probability towards `o`.
The attention table
How much to attend to each earlier character
Here every earlier character counts equally. When is that the wrong thing to do?
This is our first attention table, and attention just means how much significance to give each earlier character when predicting the next. Read it by row: each position spreads its attention over itself and the positions before it. Multiply this flat table by the character vectors and you get exactly the bag of characters from the slide before, now drawn as a weight grid. Two different things are going on, and it pays to keep them apart. First, a rule we will never break: the blank upper triangle is causality, a position cannot look at the future, and that stays fixed for the rest of the talk. Second, a starting guess we fully intend to throw away: the weights are equal because nothing has been learned yet, the most neutral thing the model can say, "I do not yet know which earlier character matters most". Notice too that each row is a different length, position 1 sees only itself, position 4 sees four, so the triangle is really the model predicting at every context length at once. Everything from here replaces that flat guess with learned relevance, while the causal mask stays put. Spoken aside, intuition and not a rule: once trained, the weights tend to concentrate towards the lower right, because within a word the recent characters constrain the next one more than the distant ones.
Answer: When some earlier characters matter far more than others. A vowel hunting for the consonant before it does not want a flat average; it wants to weight that one position heavily. That is what the next few slides make possible.
Similarity vs relevance
Not how alike, but how well they fit
Why isn't similarity what we want here?
You already know similarity search: the dot product of two vectors tells you how alike they are, and search leans on it all the time. So why not use it here, to decide which earlier characters a position should draw on? Because the thing we want is not similarity. The most useful earlier character is rarely the one most like the current one. Think of it as fit rather than likeness: two puzzle pieces join because they are complementary, not because they look the same.
Answer: Because the most useful earlier character is rarely the one most like the current one. Relevance is a fit between what a position needs and what another offers, not a resemblance. And a fit takes two different vectors, not one compared with itself, which is exactly queries and keys.
Queries and keys
Every query, weighed against the keys before it
A query and a key are two different vectors, not one compared to itself. What is their dot product measuring?
Now the weights come from the characters themselves, but not by similarity. The query and the key are the tab and socket from the previous slide, made concrete: each position emits a query, what it is looking for in order to predict what comes next, and each position also emits a key, what it offers. The query and the key answer each other; they are not a like-for-like comparison. Each character has one fixed query and one fixed key: the same query meets every key, and different keys give different dot products, so different weights. And relevance is gathered across the whole context window, not from isolated pairs, because a position's query is itself shaped by the characters before it.
Answer: How relevant one position is to another: the querying position's "what I'm looking for" dotted with an earlier position's "what I offer". A high score means that earlier character matters for the prediction, whether or not the two characters are alike.
Values
A token's output: a weighted blend of values
The key and the value are two different vectors. Why separate what a token advertises from what it hands over?
One more vector per token, built exactly like the query and key, but with a different job, and it matters that it is new: until now we mixed the tokens' own vectors, but from here what flows is a separate learned payload. Queries and keys only set the weights; the value is what a token hands over. What a token advertises (its key) and what it delivers (its value) are decoupled, and the flow is one-way, into the output of whoever attended.
Answer: Because a token can be a strong match for your query while the information it should pass you is something else. Splitting the key (how findable I am) from the value (what I contribute) lets the model tune those independently, and it means what flows is a learned payload, not the token's raw content.
Self-attention
Every position attends at once, its slot baked in
Attention sees the tokens as a set, with no built-in order. So what stops the model reading `hallo` the same as any anagram of it?
Put the pieces together and you have self-attention: the operation the 2017 paper is named after, now run for every position at once rather than one. The one genuinely new idea here is position. Attention treats its input as a set, so on its own it cannot tell order at all; we fix that by adding a position vector to each token's embedding, so the two l's arrive as different inputs. One aside for the curious: the scores get divided by the square root of the vector size, so attention starts spread out instead of fixating before it has learned anything.
Answer: Nothing in attention itself, it is order-blind by construction. That is why each token also carries a position vector, added to its embedding, so "the l in slot 3" and "the l in slot 4" arrive as different inputs. Position is what breaks the tie.
Multiple heads
Several heads, each hunting something different
One head can only chase one kind of relationship at a time. How does the model track several at once?
One head is one point of view: it finds one kind of relationship, say each vowel reaching for the consonant before it. Run several in parallel, each with its own query, key and value, and each becomes a separate channel, one on adjacent letters, another on longer-range structure. Combine their outputs and the token has gathered several kinds of information at once. That is what the word head is for: it presumes there are others alongside it.
Answer: Run several heads in parallel, each with its own query, key and value, so each can specialise; their outputs are then combined. That parallelism is exactly why the single unit is called a head.
The transformer block
Attention communicates, feed-forward computes
Attention let each token gather information from all the others. What still hasn't happened to what it gathered?
Attention only gathers. A small perceptron network (the machine this workshop series started with) then processes what each token collected, position by position. Communicate, then compute: that pair is the block, and the transformer is just this block repeated. Two supports keep deep stacks trainable: a residual path that adds each sublayer's output back onto the stream, and a normalise step that keeps each token's numbers well-scaled. Karpathy applies the normalise before each sublayer, a small departure from the paper, which applies it after.
Answer: Nothing has been done with it yet. Attention only moves information between tokens; it does not process it. That is the feed-forward sublayer's job: a small per-token network (the perceptron this series began with) that computes on each token's gathered vector on its own. Communicate, then compute.
Emergent structure
Output samples at loss 4.4, 2.5, and 1.5
Nobody programmed speaker names, line breaks, or iambic rhythm. Where did they come from?
Train it (guess, measure the surprise, nudge, repeat) and structure condenses out: names in capitals, dialogue in lines, word-shaped early-modern English. Ten million parameters gets you unmistakable Shakespeare.
Answer: Predicting the next character well *requires* internalising them. Every regularity in the data gets compressed into the weights; the objective never changed.
Scaling up to ChatGPT
The same architecture at 0.2M, 124M, and 175B
This is the part to take home: ChatGPT is this same decoder-only architecture with the internet instead of Shakespeare and billions instead of millions, plus a round of fine-tuning to turn a document-completer into an assistant. Scale and polish, not a different idea. You now know what the T stands for, from the inside.
Whiteboard
For h, a, l, l, draw each character's integer looking up its row in the embedding table to become a vector, then draw a position vector added onto each
Draw the triangular attention table over positions h, a, l, l: each row spreads weight over itself and the positions before it, upper triangle empty (no future); then say what multiplying this flat table by the character vectors does
Give each of h, a, l, l a query and a key; draw the causal links from each query to the keys it may see, making a couple thicker to show stronger relevance
Add a value to each token; for the last l, blend the four values in proportion to its link strengths to build its output vector
Draw one transformer block: a residual spine with two sublayers, attention (communication, cross-token) then feed-forward (computation, per-token), each normalised on the way in and added back; label which is which
Every exercise redraws a figure they have already seen, nothing to invent. Mapping: 1 → embeddings and position, 2 → the attention table, 3 → queries and keys, 4 → values, 5 → the transformer block. Exercises 3 and 4 together are one full head of self-attention, the non-negotiable crux; if time is short, do those two.
Build it yourself
The full build, code included, on the platform
The module walks the whole build with runnable code: two to three minutes of training on your laptop, every step explained. And the source for all of it is Andrej Karpathy's lecture "Let's build GPT": two hours, freely available, fully credited in the module. Go and watch it.
Let's Build
Machine Learning Apprenticeship
52 in-person workshops format
Fully funded funding
Peer-led discussion every session cohort
Projects tied to real work outcomes
Eligibility
Employed or own company employed
Resident for 3+ years residency
