What Is Gradient Descent? How Models Learn

Gradient descent is the optimization algorithm that trains most neural networks, including large language models: it repeatedly nudges a model’s parameters in the direction that reduces error the fastest, using the gradient — the multi-dimensional slope — of a loss function. It’s the mechanism underneath the word “training” in “training a model”: every weight update during training is one step of gradient descent.

The loss function: a measure of wrongness

Before a model can be trained, it needs a way to measure how wrong its current predictions are. A loss function takes the model’s output and the correct answer and returns a single number — higher for worse predictions, lower for better ones. Training a model means finding the parameter values that make this number as small as possible across the training data. With millions or billions of parameters, there’s no way to solve for the minimum directly; instead, the model starts at some (usually random) point and iteratively moves toward lower loss.

Following the gradient downhill

The gradient of the loss function, with respect to the model’s parameters, is a vector pointing in the direction of steepest increase in loss. Gradient descent does the opposite: it moves each parameter a small step in the negative gradient direction — downhill, toward lower loss. Imagine standing on a foggy hillside with no view of the valley below, only able to feel the slope under your feet. Gradient descent is the strategy of always stepping in the steepest downhill direction you can currently feel, one step at a time, until the ground stops sloping.

The update rule for a single parameter looks like:

new_weight = old_weight - learning_rate * gradient

The learning rate controls how large each step is. Too small, and training crawls, taking an enormous number of steps to make progress. Too large, and updates overshoot the minimum, bouncing around or diverging entirely instead of settling. Tuning the learning rate — often via a schedule that starts higher and decreases over training — is one of the most consequential decisions in training any model.

Computing the gradient: backpropagation

For a neural network with many stacked layers, computing how the loss depends on every parameter — including ones buried deep in early layers, far from the final output — is handled by backpropagation: an efficient application of the chain rule from calculus that computes gradients layer by layer, working backward from the output. Backpropagation is how gradient descent becomes computationally tractable for a network with billions of parameters; without it, computing each parameter’s gradient independently would be prohibitively expensive.

Batch, stochastic, and mini-batch gradient descent

Computing the exact gradient requires evaluating the loss across the entire training dataset, which is far too slow for datasets with millions or billions of examples. In practice, training uses one of these variants:

• Batch gradient descent computes the gradient over the full dataset before each update. Accurate, but prohibitively slow at scale — rarely used for large models.
• Stochastic gradient descent (SGD) computes the gradient from a single example at a time. Fast per step, but noisy — each step is based on a rough estimate, so progress toward the minimum is erratic rather than smooth.
• Mini-batch gradient descent splits the difference, computing the gradient over a small batch (commonly tens to thousands of examples) at a time. This is what almost all real training uses: batches are large enough to give a reasonably stable gradient estimate, small enough to fit in GPU memory, and parallelize well across hardware.

Optimizers built on top of gradient descent

Plain gradient descent with a fixed learning rate is rarely used as-is in modern training. Optimizers like Adam, AdamW, and RMSprop build on the same core idea — step in the direction that reduces loss — but adapt the effective step size per parameter based on the history of recent gradients, which tends to converge faster and more reliably than a fixed learning rate applied uniformly across every parameter. These optimizers are still, fundamentally, doing gradient descent; they’re refinements of how big a step to take and in what adjusted direction, not a departure from the underlying approach.

Where this fits in training an LLM

Gradient descent (via backpropagation and an optimizer like AdamW) is the mechanism behind every phase of training a large language model: the initial pretraining pass over a large text corpus, and later stages like fine-tuning or LoRA fine-tuning on a narrower dataset. What changes between these phases is the data, the loss function, and how many parameters are being updated — not the underlying optimization algorithm. It’s also the reason models can suffer catastrophic forgetting: the same gradient steps that push weights toward the new task can push them away from patterns the model previously learned, since the algorithm has no separate notion of “protect old knowledge” — it only ever moves downhill on whatever loss it’s currently being shown.

Inside a transformer, gradient descent updates every trainable component — attention weights, feed-forward layers, embeddings — simultaneously, all driven by the same backpropagated gradient computed from a single loss value at the output.

The takeaway

Gradient descent trains a model by repeatedly measuring how wrong its predictions are, computing which direction reduces that error fastest, and nudging every parameter a small step that way. Backpropagation makes computing that direction tractable for networks with billions of parameters; mini-batching makes each step affordable to compute; and the choice of learning rate and optimizer determines whether those steps converge smoothly or waste computation overshooting the target. Nearly everything described as “training” a neural network, at any scale, is this loop running for a very large number of iterations.