Mini-lesson 02 / toward the global-workspace paper · prev: what a Jacobian is
From the math to the instrument
The Jacobian was a matrix that turns input-nudges into output-nudges. Point it from the middle of a transformer to its exit, and it becomes a lens: a readout of what a hidden thought is disposed to make the model say.
A transformer processes text in a tall stack of layers. Running up the middle of that stack is the residual stream — a running vector, one per token position, that each layer reads from and writes back to. Call the vector at layer ℓ the hidden activation hℓ.
That vector is the model’s state of mind at that depth: everything it has figured out so far about the current token. The trouble is it’s written in the model’s own internal code — a list of a few thousand numbers with no labels. We can’t read it directly.
We want to ask a hidden activation a single question: if the model had to speak right now, what words are you pushing it toward?
At the very top of the stack the model does something we can read: it takes the final activation hfinal and multiplies it by the unembedding matrix, turning that internal vector into a score for every word in the vocabulary. High score → likely next token. That’s the model’s built-in translator from internal-vector language into vocabulary.
But the unembedding is calibrated for the final layer only. Feed it a mid-stack activation directly and you get garbage — it’s the wrong dialect. A layer-6 vector doesn’t live in the same basis as a layer-32 vector, even though both are “the residual stream.” So we need a step in between:
carry the mid-stack vector up to where the translator works — into the final-layer basis — and the thing that carries it is a Jacobian.
Recall lesson 1: the Jacobian of a function is the matrix that best approximates it locally — nudge the input, and Δout ≈ J · Δin. Now choose the function to be “the rest of the network”: the map from the activation at layer ℓ all the way to the final activation.
The Jacobian of “layer ℓ → final layer”
Jℓ = E ∂hfinal∂hℓ
Read it exactly as a Jacobian: how does the final activation move when you nudge the layer-ℓ activation? That matrix is precisely the linear map that carries a layer-ℓ vector into the final-layer basis. The E (an average over many prompts) we’ll unpack in a moment — it’s what turns a fragile per-input derivative into a stable, reusable lens.
Two moves, in order: transport the activation up with Jℓ, then translate it to words with the model’s own unembed. Compose them and you have the whole instrument:
The Jacobian lens
lensℓ(h) = unembed(Jℓ · h)
So the output of lensℓ(h) is not a prediction and not an echo of the input — it’s a ranked list of words naming what that buried activation is leaning toward. Point the lens at different depths and you watch a thought resolve into language.
Prompt: “The currency used in the country shaped like a boot is”. Click a layer to move the lens down the stack and read what the activation there is disposed to say.
lensℓ(h) — top tokens
Real output. These are genuine Jacobian-lens readouts — the pre-fitted neuronpedia lens (qwen-n1000) run on Qwen3.5-4B for this exact prompt, read at the final position, top tokens by probability. The lens is noise through the first two-thirds of the stack; then at layer 21 the currency category snaps in (74%), by layer 26 it is homing on the euro, and by layer 30 it resolves to “Euro” — the model’s thought climbing into words. Layers picked from a full 0–30 sweep. (Raw JSON in the repo.)
A Jacobian is a local approximation — strictly, it’s different for every input. If we recomputed it fresh for each prompt, the “lens” would wobble with every sentence and tell us nothing general. The fix is the expectation E: average the Jacobian over hundreds of prompts and positions in ordinary text.
the move that makes it a lens, not a derivative
Averaging turns a per-sentence tangent into one fixed matrix for layer ℓ — a stable, reusable instrument you fit once and then aim at any activation. That’s the difference between “the derivative here, right now” and “a lens you can pick up and look through.”
Two questions to lock in the shape of the idea.
Check 1 — why not use unembed directly
Why can’t we just feed a mid-stack activation hℓ straight into unembed and skip the Jacobian?
Because unembed is calibrated for the final-layer basis only. A layer-ℓ vector is written in a different internal dialect, so decoding it directly gives noise. The Jacobian Jℓ is exactly the linear map that transports the vector into the basis where the translator works. Transport first, then translate.
Check 2 — what the readout means
The lens outputs a ranked list of vocabulary tokens. Is that the model’s next-token prediction? If not, what is it?
No. It’s what that particular activation is disposed to make the model say — the words this internal vector is leaning toward, read out in isolation. The true next-token prediction only emerges at the final layer after every activation combines. Aim the lens mid-stack and you catch a concept mid-formation, before it’s the answer. That’s the whole point: it exposes the internal “words” the model reasons with — the raw material of the global workspace.