AI Daily Grind // AI news, freshly ground daily

← All lessons

🧠 Models and how they work · ⏱️ 2 min Latte

What is Quantization?

Illustration: Quantization
Simply put, like I'm 5 years oldIt is like a recipe that says 'about one cup of flour' instead of '236.6 grams'. The recipe card gets much shorter, and the cake tastes almost the same.

📖 The slightly longer version

An AI model is mostly a giant list of numbers called parameters, or weights. They hold what the model learned during training. Normally each number is stored very precisely, with 16 or 32 bits. A bit is the tiny on/off piece a computer counts with. A big model can need hundreds of gigabytes of memory.

Quantization stores each number with fewer bits, for example 8 or 4. It is a bit like rounding 3.14159 to 3.1. The model file gets much smaller. So it can fit on a home computer or a cheaper graphics card, and it often runs faster too. The price: some precision is lost, so answers can get a little worse. When people share a 'compressed version' of an open model, it is very often a quantized one.

How a big model shrinks to fit at home

  1. Full modelBillions of numbers, each stored with 16 or 32 bits.
  2. Round the numbersEach number is stored with fewer bits, like 8 or 4.
  3. Smaller fileThe model now needs a fraction of the memory.
  4. Runs at homeIt fits on a normal computer or a cheaper graphics card.
  5. Check the qualityTest it, because answers can get a little worse.
Why it matters

Quantization is a big reason why open models now run at home, and that keeps your data on your own machine. Today's stories show both sides. A user ran a compressed Qwen model on llama.cpp and saw it fetch web addresses nobody asked for. Llama-modchip claims a 284-billion-parameter model on 64GB of RAM. When you download a 'small' version of a model, you should know it is not exactly the same model the company tested.

⚠️ What to watch out for

How much quality you lose is not fixed. It depends on the model, the method and how many bits you keep, and very low bit versions can make strange mistakes. We do not know if compression played any role in the Qwen reports, so do not assume it. The Llama-modchip claim is still just a claim: simple math says 4-bit numbers alone would not fit 284 billion of them in 64GB, so other tricks must be involved, and nobody has checked it independently yet. Many quantized files are also made by strangers online, so ask where yours comes from.

📰 Where it showed up

This notion came up in Who is checking? Nvidia's board, a cautious bank and a quiet Gemini privacy change. More lessons: Learn.

🎯 Keep learning

New 🧠 Models and how they work · 2 min Latte What is Small model? A small model is like a little kitchen knife next to a big chef's knife. It cannot do every job, but for cutting a lot of small things fast, it is lighter, quicker and cheaper to keep. Learn it → 🧠 Models and how they work · 2 min Latte What is Chain-of-thought? It is like a cook who says each step out loud while making soup: "now the onions, now the salt." If you listen, you can stop them before they add something bad. Learn it → 🧠 Models and how they work · 2 min Latte What is Token? A token is like one small piece of a chopped carrot. The AI does not eat a whole sentence at once: it cuts the words into little pieces and reads them one by one. Learn it →

Sources

Written on 2026-11-09. This explanation was accurate on that day. AI changes fast, so check the sources and the date before you rely on it.

Get The AI Daily Grind by email, every morning, in simple words.