Tensors and PyTorch: The Data Structure That Runs Deep Learning

A transformer, a convolutional net, and a diffusion model look wildly different on a slide. Underneath, they run the same two things over and over: multiply a matrix, then take a derivative. Every activation, attention score, and weight update is one of those. Learn the data structure they operate on — the tensor — and the shape rules that govern it, and the rest of deep learning stops being magic and starts being bookkeeping.

This is a tour of exactly that toolkit: tensors as n-dimensional arrays, a matrix as a transformation, the dot product as similarity, and PyTorch's autograd computing every gradient from a single .backward() call. If you can read a shape, you can read a model.

What is a tensor, really?#

A tensor is just an array with a rank — the number of axes it has. Start at zero and climb one dimension at a time. Nothing new appears at each rung; you only add an index.

The .shape is the single most useful thing you can print. It tells you how many dimensions exist and how big each one is, which is enough to catch most bugs before they crash. A [8, 3, 768] tensor is a batch of 8 sentences, each 3 tokens long, each token a 768-number vector — read left to right, outermost axis first. When two operations disagree, they almost always disagree on shape, not on values.

A matrix isn't storage — it's a transformation#

The mental upgrade that unlocks linear algebra: a matrix M is a function that moves vectors. The trick to reading any 2×2 matrix is that its columns are the images of the basis vectors. The unit x-axis vector î = [1, 0] lands exactly on the first column; the unit y-axis vector ĵ = [0, 1] lands on the second.

Take the rotation matrix M = [[0, -1], [1, 0]]. Its first column is [0, 1], so î rotates onto the y-axis. Its second column is [-1, 0], so ĵ rotates onto the negative x-axis. That is a 90° counter-clockwise rotation, and you read it straight off the columns without plugging in a single number.

To apply M to any vector v = [x, y], scale each basis image by v's components and add:

The determinant det = m00·m11 − m01·m10 measures how much M scales area. A determinant of 2 doubles every region; a determinant of 1 (like the rotation above) preserves area; a negative determinant flips orientation. And det = 0 is the danger sign — the matrix collapses 2D space onto a line, information is destroyed, and it can no longer be inverted. That singular case is the same failure you hit when a layer's weight matrix loses rank.

Matrix multiplication and the dot product#

Matmul is where the compute goes, so it pays to see it as thousands of independent dot products. Multiply A of shape (R×K) by B of shape (K×Cn) and you get C of shape (R×Cn). The shared inner dimension K must match — that is the whole shape rule — and every output cell is one dot product of a row of A with a column of B:

With A = [[1, 2, 3], [4, 5, 6]] (2×3) and B = [[7, 8], [9, 10], [11, 12]] (3×2), the top-left cell C[0][0] is the dot product of row [1, 2, 3] and column [7, 9, 11]:

The dot product also measures alignment. Its cosine form, dot / (‖row‖ · ‖col‖), is +1 when two vectors point the same way and 0 when they are orthogonal — which is exactly why attention scores two token vectors by dotting their query and key. Every C[r][c] is independent of every other, so a GPU computes thousands of these cells in parallel. That embarrassing parallelism is the entire reason GPUs, not CPUs, train models.

Reshaping and broadcasting without copies#

Two shape operations show up constantly, and they behave differently under the hood.

Broadcasting lets you combine tensors of different shapes without writing loops. Right-align the two shapes and walk axis by axis: if the sizes are equal, keep it; if one of them is 1, stretch that operand to match; otherwise, error. Adding a [3, 1] tensor to a [1, 4] tensor stretches the first across 4 columns and the second across 3 rows, producing a [3, 4] result — no data is actually duplicated in memory, the size-1 axis is just read repeatedly.

Reshaping rearranges the same elements. .view() is a zero-copy reinterpretation: arange(6) laid out as [2, 3] becomes [3, 2] by re-reading the same flat buffer 0,1,2,3,4,5 in row-major order — the bytes never move. .permute(1, 0) is different: it swaps axes, so the values move to new positions (a transpose), not just the index math. The catch on .view() is that prod(from) must equal prod(to); you cannot reshape 6 elements into a [3, 3] grid of 9. When you get a "view size is not compatible" error, this is why.

Walkthrough: autograd computes x.grad by itself#

The payoff of PyTorch is autograd. When you set requires_grad=True, every operation is recorded onto a tape — a computation graph — as the forward pass runs. One .backward() call then walks that graph in reverse and fills in every gradient using the chain rule. No hand calculus.

Trace y = x * x at x = 2. Written out, that expression is three nodes: two leaves both reading x, and one multiply node combining them. Watch the backward pass accumulate:

The multiply node applies the product rule — d(a·b) sends the other operand down each branch — so both copies of x receive a gradient of 2. Because x appears twice, its final gradient is the sum of the two paths: 2 + 2 = 4. That matches the analytic answer, d/dx[x²] = 2x = 4 at x = 2, and a numeric central-difference check confirms it.

That accumulation (+=, not =) is the single most important detail in the whole engine. When a value feeds several downstream places, its gradient is the sum of all the paths back to it. It is also why gradients pile up across training steps if you forget to call zero_grad() between them. The same reverse-accumulation scales unchanged from these 3 nodes to billions of parameters.

Complexity: where the cost actually is#

The headline number is matmul's O(R·K·Cn). Because each output cell is independent, all R·Cn dot products run in parallel on a GPU, so wall-clock time is dominated by memory bandwidth and how well the shapes tile — not by the raw operation count. Autograd adds only a constant-factor overhead: the backward pass visits each node once, so it is roughly the same cost as the forward pass that built the tape.

What to remember#

• A tensor is an array plus a rank. Print .shape first — most bugs are shape bugs.
• A matrix is a transformation; its columns are where the basis vectors land, and its determinant is the area scale (0 = collapsed and non-invertible).
• Matmul is a grid of independent dot products; the shared inner dimension must match, and that independence is why GPUs win.
• The dot product doubles as a similarity score — the engine behind attention.
• .view() reinterprets the same bytes for free; .permute() moves values; broadcasting stretches size-1 axes without copying.
• requires_grad=True records a tape; one .backward() reverse-accumulates every gradient, summing across paths where a value is reused.

Keep going#

You now have the vocabulary the rest of the stack is written in. Next, see these tensors become a working model:

• Transformers, Part 1: Tokenization, Embeddings & Positional Encoding — where the [tokens, dims] matrix comes from.
• Transformers, Part 2: Self-Attention & Multi-Head Attention — the dot product turned into attention scores.
• What Changed Since 2017: RMSNorm, SwiGLU, RoPE, KV Cache & GQA — how those matmuls got cheaper and faster.
• Train Your First Language Model: From Raw Text to Generation — autograd driving a real training loop.

You can also explore this interactively on YeetCode — climb the dimension ladder, transform a vector with your own 2×2 matrix, accumulate a dot product cell-by-cell, and type any f(x) to watch the tape compute x.grad and check it against the numeric derivative.

FAQ#

What is the difference between a tensor and a matrix?#

A matrix is a rank-2 tensor — exactly two axes, rows and columns. "Tensor" is the general term for an array of any rank: a scalar is rank 0, a vector rank 1, a matrix rank 2, and anything higher (a batch of matrices, an attention tensor) is rank 3 or more. In PyTorch every one of these is the same torch.Tensor object; only the .shape differs. So a matrix is a tensor, but most tensors in deep learning have three or four axes because they carry a batch dimension and often a heads dimension on top of the 2D data.

Why does matrix multiplication need the inner dimensions to match?#

Because each output cell is a dot product between a row of the left matrix and a column of the right one, and a dot product only exists between two vectors of the same length. If A is (R×K), its rows have length K; for the columns of B to also have length K, B must be (K×Cn). The shared K is consumed by the summation and disappears from the result, leaving C as (R×Cn). When shapes don't line up on that inner dimension, there is no valid pairing to sum over, which is exactly the "mat1 and mat2 shapes cannot be multiplied" error.

What does requires_grad=True actually do?#

It tells PyTorch to record every operation involving that tensor onto a computation graph — the "tape" — as the forward pass runs. Each op stores the input values it will need later (the product rule, for instance, needs both operands). When you call .backward() on the output, PyTorch walks that recorded graph in reverse, applies each node's local derivative, and accumulates the results into every leaf's .grad. Without requires_grad=True no tape is recorded, so .backward() has nothing to traverse and the tensor's .grad stays None.

Why do I have to call zero_grad() between training steps?#

Because gradients accumulate with +=, not =. That accumulation is intentional and correct: when a value is used in several places, its true gradient is the sum of all the paths back to it — which is why x * x gives x.grad = 4, from two paths of 2 each. But that same rule means a second .backward() adds its gradients on top of the first instead of replacing them. If you don't reset .grad to zero between steps, your update uses a stale sum of several batches' gradients, and training destabilizes. optimizer.zero_grad() clears the slate before each backward pass.