Building up from what you know

What a Jacobian actually is

It’s the ordinary derivative — the same “nudge the input, watch the output move” idea — grown up to handle vectors going in and vectors coming out.

1 Start with one variable

You already know the ordinary derivative. If f(x) = x², then f′(x) = 2x. That derivative answers exactly one question:

If I nudge the input a tiny bit, how much does the output move — and in which direction?

At x = 3, we get f′(3) = 6. Read that as: near x = 3, push the input up by a tiny amount ε and the output rises by about 6ε. The derivative is a local multiplier — the best straight-line stand-in for a curved function, right at one point.

That’s the whole seed. Everything below is this one idea, generalized. Drag the nudge and watch it:

The derivative is a local multiplier

The curve is f(x) = x². The straight line is its derivative at the dot — the function’s best linear impersonation there. Move the input nudge; the output response is the multiplier times the nudge.

point x 1.40

slope f′(x) = 2x = 2.80 nudge input Δx = +0.30 → output moves Δy ≈ +0.84

2 Let the input and output be vectors

An ordinary function takes one number in and gives one number out. But most interesting functions — including a layer of a neural network — take a whole vector in and hand back a vector.

Say a function F takes a 3-dimensional input (x₁, x₂, x₃) and produces a 2-dimensional output (y₁, y₂).

Now “the derivative” is trickier, because there are many directions to nudge and many outputs to watch. So the honest question splits into many copies of the same question:

If I nudge input component x_j, how much does output component y_i move?

There’s one such number for every pairing of an output with an input. Here that’s 2 outputs × 3 inputs = 6 numbers. Each one is a partial derivative, written ∂y_i / ∂x_j — “how y_i responds to x_j, holding the others still.”

3 The Jacobian is just those numbers in a grid

Arrange the six numbers in a matrix. Row = which output. Column = which input. That grid is the Jacobian:

Jacobian of F, evaluated at a point

J = ∂y₁∂x₁ ∂y₁∂x₂ ∂y₁∂x₃ ∂y₂∂x₁ ∂y₂∂x₂ ∂y₂∂x₃

rows → 2 outputs • columns → 3 inputs • shape = 2 × 3

That’s the entire definition. A Jacobian is the derivative when both the input and the output are vectors: a matrix of “how does each output respond to each input,” evaluated at one point. The top-left entry, for instance, is how y₁ responds to x₁.

4 The one property that makes it worth having

Same punchline as the 1-D derivative. For a small input nudge Δx (now itself a vector), the output moves by:

Best local linear approximation

Δy ≈ J Δx

Zoom in close enough to a point and the messy nonlinear F simply acts like multiplication by the matrix J. So the Jacobian tells you precisely how the function stretches, rotates, and squishes input-space in the neighborhood of that point. One matrix, standing in for the whole tangled function — locally.

Quick checks

The whole paper rides on this, so let’s make sure it landed. Answer in your head, then reveal.

Check 1 — shape

If F maps a 5-dimensional input to a 3-dimensional output, what is the shape (rows × columns) of its Jacobian?

3 × 5 — three rows (one per output component), five columns (one per input component). The rule is always outputs × inputs, in that order. So it holds 15 partial derivatives.

Check 2 — meaning of one entry

In that same 3×5 Jacobian, what does the entry in row 2, column 4 tell you?

It is ∂y₂ / ∂x₄: how much the 2nd output component moves when you nudge the 4th input component (holding the other inputs fixed). Row picks the output; column picks the input.