Watch on YouTubeWatch on Instagram
About this video
Rounded to 4 bits, a 9B AI fits on a 16 GB laptop and gets 48 of 48 facts. At 2 bits, models break.
At full size that 9B model is 18 GB. How far can you round a model before it breaks, why a bigger rounded model beats a smaller exact one, and why rounding makes it faster.
Episode 11 of Learn AI in Under 3 Minutes, level 2: using models well.
Measured on a 2020 MacBook Air (M1, 16 GB) with llama.cpp and Qwen3.5 (open weights, Feb 2026, thinking off, greedy), GGUF files from unsloth. The 2B model at 16-bit (3.8 GB), 8-bit (Q8_0, 2.0 GB), 4-bit (Q4_K_M, 1.3 GB), 3-bit (Q3_K_M, 1.1 GB) and 2-bit (Q2_K, 1.0 GB, made from the 16-bit file with llama-quantize) answered the same 48 short real facts, graded by program: 46, 46, 46, 40 and 1 right. An importance-aware 2-bit file (UD-IQ2_XXS, 0.8 GB) got 18. Next-token agreement with the 16-bit model on 3,072 tokens of The Adventures of Sherlock Holmes (llama-perplexity KL mode): 98%, 92%, 83%, 33%. Same memory: Qwen3.5-0.8B at 16-bit (1.5 GB) got 35; the 2B at 4-bit (1.3 GB) 46; Qwen3.5-4B at 2-bit (UD-IQ2_XXS, 1.5 GB) 28; Qwen3.5-4B at 4-bit (2.7 GB) 46. Qwen3.5-9B at 4-bit (Q4_K_M, 5.7 GB) ran on the same laptop and got 48 of the 48 facts. Speed (llama-bench, 64 generated tokens, Metal GPU): 11.6 tokens/s at 16-bit, 20.5 at 8-bit, 26.5 at 4-bit. File sizes are whole files: "4-bit" files keep some layers at higher precision. The 9B 16-bit size is the published Qwen3.5-9B BF16 file (17.9 GB). The rounding picture uses 32 real weights from layer 3 of Qwen3.5-2B, rounded to evenly spaced levels. One machine and 48 questions: an example, not a benchmark.
Sources: Apple, "Apple Intelligence Foundation Language Models: Tech Report 2025" (arXiv:2507.13575): an on-device model of about 3B parameters with 2-bit quantization-aware training. llama.cpp (ggml-org) quantization types. Qwen3.5 model cards (Alibaba Qwen team).
0:00 It won't fit
0:25 What a model weighs
0:58 How far can you go
1:34 Big and rounded, or small and exact
1:58 Faster too
2:11 Why run it yourself
2:29 My take
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Follow @PranavInnovates for the tech story that matters, every day.
What the video says
This laptop has sixteen gigabytes of memory. Qwen three point five, nine B, at full size, needs eighteen. It won't fit. Rounded to four bits, it's five point seven gigabytes, with room to spare. It runs here, and gets all forty-eight of my test facts right. But round too far, and a model tells you Bhutan's capital is Kathmandar. So how far can you round before it breaks? I measured it. A model is billions of numbers, called weights, usually sixteen bits each: two bytes. Nine billion times two bytes: eighteen gigabytes. Here are thirty-two real weights from Qwen three point five, two B. At sixteen bits, each can sit on sixty-five thousand levels. Quantizing rounds each weight to fewer levels. Eight bits: two hundred fifty-six. Four bits: sixteen, and the bars barely move. Two bits leaves four levels, and most weights land somewhere wrong. The trick that helps: each block of thirty-two gets its own scale. So I rounded the two B model again and again, and asked each copy forty-eight facts. Sixteen bits: forty-six right. Eight bits: forty-six. And at four bits: still forty-six, at a third of the size. Three bits: forty. Two bits: one. At two bits it just loops. Ask for Bhutan's capital, and it repeats the question, again and again. A finer test, on a Sherlock Holmes story: how often each copy picks the same next word as the full model. Two bits: a third. Three: eighty-three percent. Four: ninety-two. Eight: ninety-eight. Quick quiz. You get one and a half gigabytes. Which knows more facts: a small model at full sixteen bits, the two B at four bits, or a four B at two bits? The two B at four bits: forty-six. The small one: thirty-five. The four B: twenty-eight. Bigger and rounded wins, until the rounding goes too far. Rounding also makes it faster. Each token, the chip reads every weight from memory, so fewer bytes, less to read. On this laptop: almost twelve tokens a second at sixteen bits, over twenty-six at four. Why run one yourself? Your files never leave the machine, each question is free, and it works offline. Phones do this too. Apple's 20 25 report: its iPhone model has about three billion weights, trained to work at two bits. Rounding during training keeps two bits from breaking. My take: start at four bits. Go lower only when your own test still passes. This week: install Ollama or LM Studio, run a four-bit model, and ask it ten questions from your work. Next: what an agent is, the loop behind AI that acts. Episode twelve.