# Understanding Transformer Attention from Scratch

Deep dive into how attention mechanisms work in transformers, building intuition from first principles.


**Model:** claude-opus-4
**Created:** 7/30/2026

---

## Conversation

### **Human**

I keep hearing about attention in transformers but the explanations are either too mathematical or too vague. Can you explain what attention actually DOES?

---

### **Assistant**

Attention is: How much should word A pay attention to word B?

Let's build this from a concrete example:

Sentence: The animal didn't cross the street because it was too tired.

Question: What does it refer to? animal or street?

Attention's job: Figure out which words relate to each other.

How attention works (intuitively):
1. Query (Q): it is asking: Who am I referring to?
2. Key (K): Each word says: Here's what I am
3. Value (V): Each word says: Here's my actual information
4. Attention score: Q dot K tells us how related they are

When it (Q) looks at animal (K): High score!
When it (Q) looks at street (K): Low score.

So it pays more attention to animal, correctly resolves the reference.

Want to see how the math actually implements this intuition?

---

### **Human**

Yes! But why do we need both Q, K, and V? Why not just two?

---

### **Assistant**

BRILLIANT QUESTION! This is the key insight.

Let's think about what each represents:

Key (K) equals Here's how you can FIND me.
Value (V) equals Here's what I actually CONTAIN.
Query (Q) equals Here's what I'm LOOKING FOR.

Why K not equal V?

Imagine a library:
Key equals The book's title in the catalog.
Value equals The actual book content.
Query equals What you're searching for.

You search by title (Q matches K), but you READ the content (you get V).

Example: In our sentence:
it (Q) is looking for something tired.
animal (K) says I'm a noun.
animal (V) says I contain can get tired.
street (K) says I'm a noun.
street (V) says I contain places go.

Q matches K to find the right word, then we grab V to actually use.

Why separate them? Flexibility: Search criteria is not content. Learnability: The model can learn to search differently than it represents. Expressiveness: Multiple queries can search the same keys differently.

Want to see the actual attention calculation?

---


---

*Downloaded from BranchLM - GitHub for AI Sessions*