Gradient descent, step by step

01 — The mechanism

The loop is the whole idea.

Imagine standing on a foggy hillside. You cannot see the bottom of the valley, but you can feel the slope under your feet. A sensible move is to step in the downhill direction, check the slope again, and repeat.

Gradient descent is that strategy written as arithmetic. A parameter x starts somewhere. A loss function f(x) scores how wrong the current setting is. The derivative f'(x) gives the slope. Then:

x_new = x − η · f'(x)

η is the learning rate. It scales every step.

The minus sign is the important part. The derivative points uphill; subtracting it points downhill. If the slope is zero, you are at a flat spot and the update is zero. That is how the loop knows to stop.

1
Measure. Compute the loss at the current position.
2
Feel the slope. Compute the gradient at that same position.
3
Step. Subtract the gradient times the learning rate. Repeat.

02 — The chart

The ball is not rolling through time.

The horizontal axis is the parameter value x. The vertical axis is the loss at that value. The curve is the entire landscape, calculated for every position. The orange marker is where the parameter currently sits.

As the animation runs, the marker moves left or right along the curve. The dashed trail is the sequence of positions visited. A vertical drop line connects the marker to the axis so you can read its position, and the statistics panel gives the exact loss, slope, and update at the current step.

Most loss landscapes in real machine learning have thousands or billions of dimensions, so we cannot draw them. This one has a single parameter, which is exactly why we can see every step.

03 — Your turn

Three runs that make the lesson stick.

Try A

The patient walk

Choose Smooth bowl, set the rate to 0.08, and start at −2.5. Run it.

The point keeps moving in the right direction but runs out of steps. A small rate is safe, not efficient.

Try B

The overshoot

Keep the bowl, but set the rate to 4.20. Run it.

Each step lands farther from the bottom than the last. The algorithm is still following the slope; the stride is simply too large for this valley.

Try C

The other valley

Switch to Twin valleys at rate 0.30. Start at 1.0, then at −2.0.

The same algorithm finds two different flat spots. One is deeper. Gradient descent is local: it follows the slope in front of it, not a map of the whole terrain.

Can you find a starting point where the update is immediately zero? Try the exact bottom of the smooth bowl at x = 1.5. The run should converge at step zero.

04 — The knob

Learning rate is a tension, not a setting to maximize.

A small learning rate takes reliable steps but needs many of them. A large rate covers ground quickly, until it overshoots the bottom and climbs the other side. Push it further and the updates grow instead of shrink: the loss diverges.

The safe range depends on the landscape. A steep curve needs a smaller step than a gentle one. That is why real training often uses schedules that start with a larger rate and reduce it later, or adaptive methods such as Adam that adjust each parameter separately.

The shape of the curve still sets the rules. On the smooth bowl, the gradient gets smaller as you approach the bottom, so the steps naturally shorten. On the twin valleys, the gradient can point toward a nearby dip and never reveal the deeper one.

05 — Scope

What this tiny model leaves out.

The computation in the lab is real: every point is produced by the update rule above. The toy is the setting, not the math.

One parameter. Neural networks optimize millions or billions of parameters at once.
A known, smooth function. Real losses are noisy, non-convex, and usually estimated from a minibatch of data.
No stochasticity. Every run here is deterministic. Real training samples different batches, which adds noise to the slope.
No momentum or adaptive rates. Modern optimizers add memory and per-parameter scaling on top of this basic loop.

Keep this

Gradient descent is a feedback loop: measure the slope, take a step, measure again. The learning rate controls how aggressively that loop responds. Most of the drama in training comes from that one tension.

Gradient descent, step by step.

Find the bottom