Watch on YouTubeWatch on Instagram
About this video
Prompt tricks: 0 of 14 right. One page: 13. Here's why.
Why magic prompts faded, what context engineering is, why an extra page makes answers worse, how coding agents manage their own context, and the expert advice that failed my test.
Episode 7 of Learn AI in Under 3 Minutes, the first of level 2: using models well.
Measured for this video on Qwen3.5-0.8B (open weights, February 2026), run on a 2020 MacBook Air (greedy decoding, thinking off). Zephyra Air is made up, so no model could know its rules. 14 traveller questions, each naming the airline, asked with: the question only (0/14); an expert persona, "take a deep breath", a $1,000 tip and "this is very important to my career" (0/14); this year's policy page (13/14); the tricks plus the page (13/14); this year's page followed by last year's page, labelled 2025, with different numbers (8/14, 5 answers quoted last year's number; with last year's page first: 11/14). An answer counts only if this year's number is the only number in it. Answer format (name the section, like [S3]): told by a rule in words, 9/13; shown by one worked example, 4/13 (the rule won on this model; a bigger model may differ). One small model and one made-up policy are an example, not a benchmark.
Sources: Meincke, Mollick, Mollick and Shapiro, "Prompting Science Report 3: I'll pay you or I'll kill you - but will you care?" (Wharton, August 2025, arXiv:2508.00614). Basil et al., "Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy" (Wharton, December 2025, arXiv:2512.05858). Andrej Karpathy on context engineering (X, June 2025). Anthropic, "Effective context engineering for AI agents" (September 29, 2025): "the smallest possible set of high-signal tokens", examples as pictures worth a thousand words, Claude Code's compaction and just-in-time retrieval. OpenAI, "Unrolling the Codex agent loop": Codex compacts the conversation automatically past a limit.
0:00 Magic words vs the page
0:30 Why tricks fail
0:53 Context engineering
1:09 The extra page
1:35 Test it
1:55 Agents
2:11 My take
Music: Special Spotlight by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Special_Spotlight_(ISRC_USUAN1600067).mp3)
Follow @PranavInnovates for the tech story that matters, every day.
What the video says
I asked an AI about an airline's baggage rules. Twenty-three kilos, it said. Confident. Wrong. So I threw every famous prompt trick at it. You're a world-class expert. Take a deep breath. I'll tip you a thousand dollars. Fourteen questions. Zero right, with or without the tricks. Then I did one boring thing, and it got thirteen. That's the shift from prompting to context engineering. What it is, the extra page that made it worse, and expert advice that failed my test. Zephyra Air is made up, so no model could know its rules. A model knows two things: its training, and what's on its desk right now. Magic words change how it talks, not what it knows. They can't put a fact on the desk. The page can. And not just my small model. Wharton researchers tested tips, threats and expert personas on big models. No reliable gain. Andrej Karpathy's name for the real job: context engineering. The art and science of filling the window with just the right information. Instructions, examples, documents, tools, memory, history. Everything on that desk is yours to choose. So more is better? I added last year's policy page too, as background. Did it get more right, or fewer? Fewer. Down from thirteen to eight. Five times, it quoted last year's numbers, though that page says twenty twenty-five. Anthropic's advice: the smallest set of high-signal tokens. Every extra page is something the model can grab instead. Even good advice needs a test. Anthropic says examples are worth a thousand words. I asked for the section in brackets. A written rule: nine out of thirteen. One example: four. On a bigger model, it might flip. That's the point: don't trust tips, even mine. Test them on your own questions. Episode ten. In agents, the desk fills itself: files, searches, tool results. So Claude Code and Codex compact. Claude Code keeps decisions and open bugs, and drops old tool output. And it pulls in files just in time, with search, not everything up front. My take: when an answer is wrong, I don't reword the prompt. I ask what was on the desk. Missing, stale, or noise. It's almost always that. Try it this week. Open your longest prompt. Cut what this step doesn't need, remove anything out of date, and add the source page. But how do you find the right page among millions, automatically? That's RAG. Episode eight.