INTERACTIVE TOY TRANSFORMER · 2 HEADS · D_MODEL=4
Transformer Generation, Step by Step
Trace each token through attention, hidden states, KV cache, and vocabulary logits.
Input [3][4]Q/K/V [2][3][2]
Tokens
Model flow
Block n is shown · other blocks are omittedInput
C
→… earlier blocks …→
Block n
Q
ABC
K
ABC
V
ABC
Out
ABC
→… later blocks …→
LM Head
C→
A
B
C
D
E
F
EOS
→
OutputD
Compute this passUsed by selected QReused from cache
Block n · Attention Head 1
Selected token · CQuery & Keys
Weighted Values
Head output
Live values
| Token | K | Score | Weight | V | Weight × V |
|---|---|---|---|---|---|
| A | [-0.20, +0.90] | -0.53 | +0.25 | [-0.72, +1.44] | [-0.18, +0.36] |
| B | [-1.00, -0.30] | -0.28 | +0.32 | [+1.26, +0.72] | [+0.41, +0.23] |
| C | [+0.70, +0.60] | +0.00 | +0.43 | [+0.54, -1.62] | [+0.23, -0.69] |
| Head output | [+0.46, -0.10] | ||||
Query override
Projected QBlock n · Attention Head 2
Selected token · CQuery & Keys
Weighted Values
Head output
Live values
| Token | K | Score | Weight | V | Weight × V |
|---|---|---|---|---|---|
| A | [+0.80, +0.40] | -0.08 | +0.25 | [+1.62, +0.36] | [+0.41, +0.09] |
| B | [+0.40, -0.70] | +0.53 | +0.47 | [-0.54, +1.80] | [-0.25, +0.85] |
| C | [-0.90, -0.30] | +0.00 | +0.28 | [+1.08, -1.26] | [+0.30, -0.35] |
| Head output | [+0.46, +0.59] | ||||