Watch on YouTubeWatch on Instagram
About this video
Change one word and AI flips from cat to mat. Here is why.
Attention, the T in GPT, explained with real numbers.
Episode 3 of Learn AI in Under 3 Minutes.
Measured for this video: Qwen3.5-0.8B (open weights, Feb 2026). It picks "cat" for "lazy" (97%) and "mat" for "dirty" (99.5%), scored by comparing the sentence with "it" replaced by each noun. The arcs are the raw weights of one attention head (layer 19, head 1) from the last word back to the nouns. One head and one sentence are an example, not a benchmark; other heads and sentences give other numbers.
gpt-oss figures are from OpenAI's published config: 36 layers, 64 attention heads, every other layer looking only at the last 128 tokens. Pair counts: n words make n(n+1)/2 pairs (1,000 tokens: 500,500). Caching: OpenAI's prompt-caching docs (cached input discounted up to 95% on the newest models; cache writes cost extra).
0:00 Attention
0:47 How attention works
1:13 Real attention
1:30 Many heads
1:48 What it costs
2:16 Money tip
Sources:
- Transformer Explainer (Georgia Tech Polo Club), the free tool shown in the video: https://poloclub.github.io/transformer-explainer/
- OpenAI prompt caching guide: static content first, cached input discounted up to 95%: https://developers.openai.com/api/docs/guides/prompt-caching
- What Does BERT Look At? An Analysis of BERT's Attention (Clark et al., 2019): heads for syntax, coreference and the previous word: https://arxiv.org/abs/1906.04341
- Attention Is All You Need (Vaswani et al., Google, 2017): https://arxiv.org/abs/1706.03762
- gpt-oss-120b config: 36 layers, 64 attention heads, sliding window 128 on alternating layers: https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json
- Measured for this video: Qwen3.5-0.8B attention weights and sentence scores (research/study/attention_data.py): https://huggingface.co/Qwen/Qwen3.5-0.8B
Music: Special Spotlight by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Special_Spotlight_(ISRC_USUAN1600067).mp3)
Follow @PranavInnovates for the tech story that matters, every day.
What the video says
Change one word, and this AI flips: ninety-seven percent cat, to ninety-nine percent mat. Read it yourself. The cat didn't sit on the mat because it was lazy. Now: dirty. You switched without thinking. But when an AI reads the word "it", what does it know? The cat, the mat, or can't it know yet? Lock it in. It can't know yet. Models like GPT read left to right, and up to "it", both sentences are identical. The answer arrives one word later. That one-word trick is attention, the heart of the T in GPT: the transformer. Last episode, every token got a row of numbers from a giant table. But that table has one row for "it". Same row, every sentence. Attention fixes that. Each new word looks back at every word before it and asks: which of you matter to me? The new word sends a question. Dirty asks: what here can get dirty? Each earlier word holds up a label. Mat says: I'm a thing you sit on. Good matches score high, and dirty soaks up a blend of them, mostly the mat. Question, label, blend: query, key, value. This is real: in one head of Qwen 3.5, an open model from this year, dirty sends forty percent of its attention to mat, twenty to cat. Swap in lazy, and the same head flips: thirty percent to cat, nine to mat. That's the flip you saw at the start. One head is one way of looking. Others track grammar, or the word before. gpt-oss, OpenAI's open model, runs sixty-four heads in each of thirty-six layers. Stack those layers and you have the transformer, from Google's 2017 paper: Attention Is All You Need. Builder's question. Double the length of your prompt. Does attention's work grow two times, four times, or eight? Lock it in. Four. Every word looks back at every earlier word, so twice the words means about four times the pairs. A thousand tokens: half a million pairs. A million tokens: half a trillion. So models cut corners: gpt-oss lets half its layers see just the last hundred and twenty-eight tokens. Money tip: words only look back, so the start of your prompt never changes. It can be saved and reused. So put what never changes first: instructions, documents, then the question. On OpenAI's API, a cached start costs up to ninety-five percent less. Want to see it? Transformer Explainer is free in your browser. Type a sentence and watch the attention light up. If you picked "can't know yet", you think like a transformer. "It" isn't looked up; it's worked out. Next: how a model like this is made.