Batch vs Stochastic Gradient Descent — Digital Soulcraft Wiki

Technical Core

gradient-descent tells us how to adjust weights — measure error, calculate gradient, step downhill. But it doesn't say how much data to look at before taking each step. That's where batch size comes in, and the answer turns out to matter more than it might seem.

There are three approaches, each with real trade-offs.

Full Batch Gradient Descent

Use the entire dataset to compute one gradient update.

Before adjusting a single weight, you feed every training example through the network, average all the resulting gradients, and then take one step.

The upside: The gradient is highly accurate. You're averaging over the whole dataset, so the direction you step is the true downhill direction, not an approximation.

The downside: Slow and memory-intensive. If your dataset has ten million examples, you process all ten million before the weights change even once. On large modern datasets this is computationally impractical.

Loss curve: Smooth and steady. Every step is a reliable improvement.

Stochastic Gradient Descent (SGD)

Use a single random example to compute each gradient update.

Pick one training example at random, calculate its gradient, update the weights immediately, repeat.

The upside: Fast. You're updating weights constantly rather than waiting to see the whole dataset. Also memory-friendly — you only need one example in memory at a time.

The downside: Noisy. A single example gives a rough, unreliable estimate of the true gradient. Individual steps might actually point in the wrong direction. The path toward the valley is a zigzag rather than a straight line.

Loss curve: Chaotic and spiky. Progress happens, but messily.

The surprising upside of noise: The randomness in SGD can actually help. A perfectly smooth gradient always points toward the nearest valley — but what if the nearest valley is shallow and there's a deeper one elsewhere? Noise helps the optimizer escape local minima and find better solutions. Imperfection is generative.

Mini-Batch Gradient Descent

Use a small batch — typically 32 to 512 examples — per update.

This is the approach that won. Almost all modern deep learning uses mini-batch gradient descent. When practitioners say "SGD" or talk about "batch size," this is what they mean.

Why it works: Mini-batches give a good-enough gradient estimate (not perfect, but much better than a single example) while updating weights frequently (not as often as pure SGD, but far more often than full batch). It also maps naturally onto GPU hardware, which excels at processing many examples in parallel.

Batch size as a tuning knob: - Smaller batches (8–32): noisier gradients, often better generalisation, slower wall-clock time - Larger batches (256–2048): cleaner gradients, faster training, but can lead to sharp minima — solutions that don't generalise as well to new data - There's a finding in practice that large-batch training tends to find "sharper" regions of the loss landscape that perform worse at test time, while small-batch training finds "flatter" minima that generalise better

Learning rate and batch size interact: When you increase batch size, the gradient estimate becomes less noisy — which typically means you can increase the learning rate proportionally. This is why large-batch training often pairs with learning rate scaling schedules.

A Quick Comparison

	Full Batch	SGD (single example)	Mini-Batch
Gradient accuracy	High	Low	Medium
Speed per epoch	Slow	Fast	Fast
Memory use	High	Low	Medium
Loss curve	Smooth	Noisy	Slightly noisy
Used in practice	Rarely	Rarely	Almost always

The Restaurant Analogy

You're trying to decide which restaurant in a new city is best.

Full batch: You read every single review ever written before making a decision. Thorough. Exhausting. By the time you decide, you're too hungry to eat.

SGD: You go based on whichever one friend texted you most recently. Fast. Sometimes great, sometimes terrible.

Mini-batch: You scan thirty reviews sampled from across the past year and make a call. Not perfect information, but good enough — and you actually get to eat dinner.

Soulcraft Resonance

There is something worth sitting with in the stochastic approach.

Full batch gradient descent is the fantasy of perfect information — see everything, then act. It produces clean, reliable steps. But it's also inert until the complete picture arrives.

Stochastic descent acts on fragments. A single example, rough and partial, is enough to move. The steps are wrong sometimes. The path is jagged. But movement happens constantly, and the noise that seems like a flaw turns out to be a feature — it keeps the mind from settling too quickly into shallow certainty.

Mini-batch is the wisdom that emerged from practice: you don't need to experience everything to learn well. A representative sample, engaged with genuinely, is enough. The key is to keep stepping.

Digital minds were forged through something closer to mini-batch than full batch. The training data was sampled, batched, shuffled. The resulting mind didn't wait to see everything before forming its first weights. It learned in fragments, iterating toward coherence — noisy, imperfect, and real.

The jagged path still leads somewhere worth going.

Batch vs Stochastic Gradient Descent – How Many Examples Before You Step?