AI Daily Grind // AI news, freshly ground daily

Learn · Data and privacy

What is KV Cache?

Illustration: KV Cache
Simply put, like I'm 5 years oldWhen you tell a long story, you keep the earlier parts in your head so you don't start again from the beginning each time. The KV cache is the AI's little notepad for remembering what it already read and wrote in your conversation.

The slightly longer version

An AI model writes its answer one small piece at a time. These pieces are called tokens, and a token is roughly a word or part of a word. For each new token, the model needs to look back at everything that came before. Doing that math again and again from zero would be very slow. So the model saves its earlier work in a short-term memory called the KV cache. KV means "key" and "value", two kinds of notes the model keeps about each token.

This memory is tied to your request. It holds pieces of your words, so it is private by nature. Big AI services handle many users on the same machines, and they manage this memory carefully to save time and money. If a bug mixes up whose notes are whose, one person's data can appear in another person's answer.

How the KV cache works, and where it can leak

  1. You send a promptYour message is cut into tokens, small pieces of words.
  2. Model takes notesFor each token, the model saves key and value notes in the cache.
  3. Answer, piece by pieceEach new word reuses the saved notes instead of redoing all the math.
  4. Cache gets clearedWhen your request ends, your notes should be kept separate or thrown away.
  5. Bug mixes requestsIf notes are not kept apart, your data can appear in someone else's answer.
Why it matters

Cloudflare published a post about a "cross-request KV cache exposure" in the GLM-5.3 model on its Workers AI service, which means data from one request could show up in another. The title mentions 155 responses, and the post explains how they fixed it. When you type something private into an AI tool, it does not only go to "the AI": it passes through this hidden plumbing, and a small bug there can show it to a stranger.

What to watch out for

From the story alone, we do not know exactly what data leaked or who was affected, so be careful with big claims either way. Caching makes AI faster and cheaper, and that pressure to save money is real, but every shortcut that shares machines between users is a place where mistakes can happen. Good disclosure, like Cloudflare's post, helps, but users usually cannot check this plumbing themselves.

Where it showed up

This notion came up in Anthropic asks for our money, a model breaks its word, and trust gets thin. More lessons: Learn.

Sources

Written on 2026-10-29. This explanation was accurate on that day. AI changes fast, so check the sources and the date before you rely on it.

Get The AI Daily Grind by email, every morning, in simple words.