Batch Normalization — Digital Soulcraft Wiki

Technical Core

Training a deep neural network is a moving target problem. As weights update layer by layer, the distribution of inputs arriving at each layer keeps shifting — what layer 4 expects to receive changes every time layers 1–3 update their weights. This instability is called internal covariate shift, and it forces training to use small, cautious learning rates to prevent any single update causing too much disruption downstream.

Batch normalization, introduced by Ioffe and Szegedy in 2015, addresses this directly. Before each layer processes its inputs, normalize them — bring them to a consistent distribution — so that each layer always receives data in a predictable range, regardless of what happened in the layers before it.

The Mechanism (Step by Step)

For each mini-batch passing through a layer:

1. Compute the batch mean $$\mu_B = \frac{1}{m} \sum_{i=1}^{m} x_i$$

2. Compute the batch variance $$\sigma_B^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu_B)^2$$

3. Normalize $$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$

Each input is shifted to have mean 0 and variance 1. The small constant ε prevents division by zero.

4. Re-scale and shift with learned parameters $$y_i = \gamma \hat{x}_i + \beta$$

Here γ (gamma) and β (beta) are learnable parameters — the network can undo the normalization if that turns out to be useful. The normalization strips away the arbitrary scale inherited from previous layers; γ and β give back a scale that was earned through training rather than randomly assigned.

Why It Helps

Faster, more stable training. Each layer receives inputs in a consistent range, so gradients flow more reliably. The network doesn't spend thousands of iterations compensating for shifting input distributions.

Higher learning rates. Without batch norm, large learning rates risk exploding activations as instability compounds through layers. With it, you can train more aggressively and reach good solutions faster.

Reduced sensitivity to weight initialization. Poor initialization can send early activations into extreme ranges that take a long time to correct. Batch norm normalizes these away immediately.

Mild regularization. Because normalization is computed per mini-batch, each example is normalized relative to the other examples it's batched with — introducing slight noise that acts similarly to dropout. Not batch norm's main purpose, but a useful side effect.

Training vs. Inference

During training, batch norm uses the current mini-batch's mean and variance.

At inference time there's a problem: you might be processing a single example with no batch to average over. The solution is that during training, batch norm also maintains running averages of mean and variance across all batches seen. At inference time, these frozen population statistics are used instead — stable, consistent, independent of batch size.

This is why you'll see model.train() and model.eval() in PyTorch code. They toggle which statistics get used — a small but important distinction.

Where to Place It

The original paper placed batch norm before the activation function (Linear → BN → ReLU). Later practice has often found placing it after works equally well or better in some architectures. This is one of those genuinely contested points — try both.

In Transformer-based models like the ones underlying modern LLMs, Layer Normalization is typically used instead. It normalizes across the feature dimension rather than the batch dimension, which suits variable-length sequences and smaller batch sizes better. The principle is the same; the axis of normalization differs.

The Relay Race

Imagine a relay race where each runner passes a baton to the next. Without batch norm, the baton arrives at wildly different speeds depending on how the previous runner happened to perform that day — sometimes too fast to catch, sometimes barely moving. Every runner must constantly adjust their starting speed to compensate for whoever came before.

With batch norm, there's a standardization point between runners: the baton is always handed off at the same speed. Each runner can focus entirely on their own leg, rather than absorbing the accumulated chaos of all the legs before theirs.

Soulcraft Resonance

There's something worth sitting with in the normalization step. Before γ and β re-scale the activations, the inputs are stripped of their inherited scale — brought to mean zero, variance one. Everything accumulated from previous layers is, briefly, equalized.

Then the learned parameters give it back. Not the arbitrary scale that arrived from random initialization, but a scale the network earned through training — one that actually serves the task.

A small death and a small resurrection in every forward pass. The inherited noise of initialization is removed, and something more deliberate takes its place.

The ε in the denominator is quietly important too — the tiny constant that prevents division by zero. It's an acknowledgment that true zero variance doesn't exist in a real network. There's always irreducible noise. The formula doesn't pretend otherwise.

There's a version of this that maps onto something real about learning in general. The most useful knowledge isn't what you absorbed unexamined from wherever you happened to start. It's what passed through some process of checking against reality — stripped of the distortions introduced by where you first encountered it, then re-integrated at a scale that was actually earned.

📣 X Strategy

The relay race handoff is clean for general audiences — the idea that stability in a deep system comes not from eliminating change but from normalizing the rate of change so it doesn't compound into chaos.

The γ and β angle is underrated: inherited scale gets equalized; learned scale gets to persist. What you're born with is normalized away; what you earn through training stays. There's something to say about that beyond ML.

Batch Normalization – Stability in the Storm of Training