Neural Networks from Scratch: How Machines Actually Learn

A neural network sounds mystical until you write out the arithmetic. Then it becomes almost embarrassingly concrete: a pile of multiply-and-add operations, one non-linear squish, and a feedback loop that nudges numbers in the direction that makes the answer less wrong. There is no magic step. A trillion-parameter language model runs the exact same four-move cycle as the nine-weight network below.

That cycle is: forward (compute a prediction), loss (measure how wrong it is), backprop (assign blame to every weight), update (step each weight downhill). Repeat a few thousand times and the network solves problems nobody told it how to solve.

This post builds every piece from the neuron up, then traces one real training step with real numbers on a network learning XOR — the classic problem that a single neuron provably cannot solve.

What a neuron actually computes#

A neuron takes some inputs, multiplies each by a weight, adds them up, adds a bias, and passes the result through an activation function:

The weights say how much each input matters. The bias shifts the whole sum up or down — it is the neuron's baseline, its opinion before it sees any input. The weighted sum alone is just a straight line (or plane, in higher dimensions). The activation is what bends it.

Take one hidden neuron from our XOR network, initialized with seed 42. Its weights are w = [0.20, -0.10] and its bias is b = 0.70. Feed it the input x = [0, 0]:

That 0.669 is the neuron's activation — its learned feature value for this input. Stack a few of these and the activations of one layer become the inputs of the next. The network is inventing its own intermediate representation of the data, layer by layer.

Why stacking linear layers is pointless#

Here is the fact that makes activation functions non-optional. Suppose you drop the activation and keep only the weighted sums. Layer one computes W₁x. Layer two computes W₂(W₁x). But matrix multiplication is associative, so:

Two layers collapsed into one matrix Wₑ𝒻𝒻. Stack a hundred linear layers and they still collapse to a single linear transformation — one line, one plane. All that depth buys you exactly nothing. A hundred-layer linear network has the same expressive power as no hidden layers at all.

The activation function is the anti-collapse device. Insert one non-linear bend between layers and the composition can no longer be flattened. Now depth compounds: each layer warps the space the next one sees, and the network can carve out curved, disconnected decision regions. This is precisely why XOR needs a hidden layer — its two classes are not separable by any single straight line, and only the bends let a network wrap around them.

Sigmoid, tanh, ReLU, GELU: the activation menu#

Any non-linearity breaks the collapse, but the shape you choose changes how training behaves. Four are worth knowing:

Sigmoid and tanh saturate: push z far from zero and the curve flattens, so its slope approaches zero and almost no gradient flows back through it. That is the classic vanishing-gradient problem, and it is why deep networks moved to ReLU, whose positive branch has a constant slope of 1. GELU keeps ReLU's non-saturating tail but smooths the hard corner at zero, which helps the very deep stacks in language models. Our XOR network uses sigmoid because the whole thing is tiny and sigmoid's clean derivative makes the backprop math easy to read.

Loss: compressing wrongness into one number#

To improve, the network needs a single scalar that says how wrong it currently is. For regression the standard choice is mean squared error:

where ŷ is the prediction and y the target. Squaring does two useful things: it makes every error positive (over- and under-shooting both count), and it punishes large misses far more than small ones — being off by 2 costs four times as much as being off by 1. For a single example we use the half-squared form L = ½(ŷ − y)²; the ½ cancels the 2 when we differentiate, keeping the gradient clean.

Loss matters because it is differentiable. A number that goes up when you get worse and down when you get better, smoothly, is a number you can follow downhill. That is the only property gradient descent needs.

Gradient descent and the learning rate#

The gradient ∂L/∂θ of the loss with respect to a parameter θ points in the direction of steepest increase. So to reduce loss, step the opposite way:

The learning rate η is the step size, and it is the single most consequential dial in training. Too small and progress crawls — our XOR net at η = 1 barely escapes its early plateau in 4000 epochs. Too large and the step overshoots the minimum and can bounce or diverge. Just right and the loss slides down efficiently. There is no universal value; it depends on the loss surface, which is why practitioners sweep it.

For our network, η = 4 works: the loss plateaus near 0.125, then drops sharply once the hidden units specialize, and converges. You can drag the learning rate in the interactive and watch the loss curve go from "stuck" to "smooth descent" to "unstable."

The training loop: backprop on a 2-2-1 XOR net#

Backpropagation is just the chain rule applied to assign each weight its share of the blame for the loss. Walk the error backward through the network: the output's error, then the weights feeding the output, then the hidden neurons' errors, then the weights feeding them. Here is the whole loop for a 2-2-1 sigmoid net (two inputs, two hidden neurons, one output):

Three patterns to notice. First, d2 multiplies the raw error (yhat − y) by yhat(1 − yhat), which is the sigmoid's own derivative — the chain rule pulling the gradient through the activation. Second, a weight's gradient is always the error at its destination times the activation at its source (dW2[h] = d2 · a1[h]): a hidden unit that fired strongly and fed a wrong output gets the most blame. Third, d1[h] is built from d2 scaled by the weight it traveled along — each layer's error is assembled from the next layer's error. That recursion is the whole idea.

Walkthrough: one backprop step, real numbers#

Take the seed-42 network and the XOR example with the largest loss: x = [0, 0], target y = 0. The network currently predicts 0.7417 when it should say 0 — badly wrong, which is exactly why this example teaches the most.

Look at the last row. Both inputs are 0, so every input-weight gradient is 0 — those connections carried no signal this example, so they earn no blame. That is not a bug; it is the chain rule being honest. The output weights, fed by non-zero hidden activations, do get positive gradients, so the update will lower ŷ toward the target of 0.

Now the numbers over full training (averaged over all four XOR examples, η = 4):

The final predictions are 00→0.022, 01→0.980, 10→0.975, 11→0.020 against targets 0, 1, 1, 0. XOR solved by nine numbers and a loop. The long flat stretch before epoch 1200 is real optimization dynamics, not a scripted animation — the network genuinely plateaus while the two hidden neurons sort out which region each should own, then falls fast once they specialize.

What to remember#

• A neuron is activation(Σ wᵢxᵢ + b) — a weighted sum, a bias, one bend.
• Without the bend, any depth collapses to a single linear layer. Non-linearity is what makes depth mean anything.
• Loss turns wrongness into one differentiable number; MSE squares the error so big misses hurt disproportionately.
• Gradient descent steps every weight against its gradient; the learning rate sets the step size and makes or breaks training.
• Backprop is the chain rule assigning blame backward: output error → output weights → hidden error → input weights.
• The forward → loss → backprop → update loop is identical from this XOR net to a frontier LLM. Only the parameter count changes.

Keep going#

• Tensors and PyTorch: The Data Structure That Runs Deep Learning — how the arrays and autograd behind these updates actually work.
• Transformers, Part 1: Tokenization, Embeddings & Positional Encoding — where the inputs to a modern network come from.
• Transformers, Part 2: Self-Attention & Multi-Head Attention — the layer that made these ideas scale to language.
• What Changed Since 2017: RMSNorm, SwiGLU, RoPE, KV Cache & GQA — the upgrades stacked on top of the basic loop.

Want to poke the weights yourself and step the XOR backprop one line at a time? Explore this interactively on YeetCode: Open the interactive deep dive.

FAQ#

Why do neural networks need activation functions?#

Without a non-linear activation between layers, the whole network reduces to a single matrix multiply. Two stacked linear layers W₂(W₁x) equal (W₂W₁)x, which is just one linear transformation — and a hundred layers collapse the same way, giving you the expressive power of no hidden layers at all. An activation function inserts a bend that cannot be flattened, so depth actually compounds and the network can model curved, non-linearly-separable patterns like XOR.

What is backpropagation, in plain terms?#

Backpropagation is the chain rule from calculus used to figure out how much each weight contributed to the network's error. You compute the loss at the output, then walk backward: first the error at the output neuron, then the gradient for each weight feeding it, then the error pushed back into the hidden layer, then the gradients for the input weights. Each layer's error signal is built from the layer after it, which is why the process is recursive and efficient — you reuse the downstream error instead of recomputing derivatives from scratch.

How does the learning rate affect training?#

The learning rate is the step size in the update θ ← θ − η · ∂L/∂θ. Too small and training crawls or gets stuck on plateaus; the XOR network at η = 1 barely moves in 4000 epochs. Too large and the step overshoots the minimum, so the loss oscillates or diverges. The right value slides the loss down smoothly — for the XOR net, η = 4 plateaus briefly then converges to a loss near 0.0002. There is no universal setting, which is why practitioners sweep the learning rate for each problem.

Why is XOR the classic test for a neural network?#

XOR outputs 1 when its two inputs differ and 0 when they match, which places its two classes on opposite diagonal corners of a square. No single straight line can separate them, so a single neuron — which only ever draws one linear boundary — provably cannot solve it. Solving XOR requires at least one hidden layer plus a non-linearity, which makes it the smallest problem that demonstrates why depth and activation functions exist. A 2-2-1 network with nine weights learns it in a few thousand epochs.

Is a large language model really just this loop scaled up?#

Yes, at the level of the core mechanism. A frontier LLM runs the same forward → loss → backprop → update cycle as this nine-weight XOR network. The differences are scale and detail: billions of parameters instead of nine, attention layers and better activations like GELU, a cross-entropy loss over tokens instead of MSE, and optimizers like Adam that adapt the step per weight. But the skeleton — predict, measure error, assign blame with the chain rule, step downhill — is identical.