Watch on YouTubeWatch on Instagram
About this video
I let a small AI think before every answer. It got no smarter, and it never stopped. Here's what reasoning models really do, and how much thinking to pay for.
Episode 9 of Learn AI in Under 3 Minutes, level 2: using models well.
Measured on a 2020 MacBook Air: 24 maths word problems written for this video (8 easy, 8 medium, 8 hard), each checked by a program. Qwen3.5-0.8B (open weights, Feb 2026) answered with thinking off (greedy) and with thinking on (model card's precise settings), its thinking cut at 64, 256, 1,024 and 4,096 tokens, then made to answer (budget forcing). Right: off 14/24; 64 tokens 13/24, 256 tokens 12/24, 1024 tokens 13/24, 4096 tokens 13/24. No budget solved a problem thinking off missed; the model never ended its thinking itself. Tokens per answer: 349 off, 4,380 at 4,096 (12.6x). One run per problem on one small model: an example, not a benchmark. Bigger reasoning models are trained to think and do gain from it.
Sources: Muennighoff et al., "s1: Simple test-time scaling" (arXiv:2501.19393, 2025). DeepSeek-AI, "DeepSeek-R1" (arXiv:2501.12948; Nature 645, 2025): rule-based rewards; R1-Zero's AIME 2024 pass@1 rose from 15.6% to 77.9% during RL; "Wait, wait. Wait. That's an aha moment I can flag here." OpenAI reasoning guide: reasoning tokens are hidden and billed as output; effort from none to max, by model. Sun et al., "What Effort Level Should I Choose?" (Johns Hopkins, Together AI, Cornell; arXiv:2610.34491, Oct 2026, CC BY 4.0): on tasks every level solved, top effort cost up to 42x the lowest; higher effort often failed tasks lower effort solved; start low, escalate after a failed check matched or beat fixed top effort.
0:00 Does thinking help?
0:28 Inside the scratchpad
0:51 Budgets
1:35 How models learn to think
2:04 The effort dial
2:29 My take
Music: Special Spotlight by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Special_Spotlight_(ISRC_USUAN1600067).mp3)
Follow @PranavInnovates for the tech story that matters, every day.
What the video says
I let a small AI think before answering twenty-four maths problems. Thinking off: fourteen right. Thinking for four thousand tokens first: thirteen. And on all twenty-four, it never once decided to stop thinking. That's a reasoning model's scratchpad. What thinking is, why big models gain from it, and how much to buy. Last episode, the model needed the right page. Here everything is in the question: seventeen croissants at three euros, paid with a hundred. With thinking on, it first writes notes to itself. Fifty-one euros. Change, forty-nine. Right, by token three hundred and twenty-four. Then: wait, is there any trick? I am confident. Maybe the answer is forty-nine. Still asking at token four thousand. Those notes are thinking tokens. On OpenAI's API you don't even see them, but you pay for them as output. To give thinking a budget, a 20 25 paper called S1 simply cuts it off and forces the answer. Budget forcing. I cut it at sixty-four, two fifty-six, a thousand and four thousand tokens. Did more thinking get more right? No. Not at any budget. Fourteen with thinking off. Then thirteen, twelve, thirteen, thirteen. It gained zero problems, and lost a few it had right. Like the Madrid train: two hundred and thirty-two without thinking. After a thousand tokens, four hundred and sixty-four. And it isn't free: three hundred and fifty tokens per answer became four thousand four hundred. Five minutes a question, not half a minute. So why do big models get better when they think? They're trained to: reinforcement learning with verifiable rewards. A checker, like my grader: right final number, reward one. No human grades the reasoning, only the answer. DeepSeek trained R1 this way. Its maths score rose from sixteen percent to seventy-eight, and its thinking got longer on its own. It even wrote: wait, wait, wait, that's an aha moment. Mine says wait too, but was never trained to make it pay. Today it's a dial on the big models. OpenAI's goes from none to max. So, max for everything? An October paper tested eight models. On easy tasks every setting solved, max cost up to forty-two times more. And it lost answers lower effort found. Their fix: start low, check the answer, and climb only when the check fails. It matched or beat always-max, usually for less. My take: thinking is something you buy per question. I start low, and pay for more only when a check fails. Try it this week: run your ten hardest prompts at low and high effort. Count what high fixed, and what it cost. Next: hallucinations, and how to know when a model is right. Episode ten.