Batch Normalization Behavior Under Inconsistent Mini-Batch Distributions

Abstract

Batch Normalization (BN) is widely used to stabilize and accelerate deep neural network training, yet
its behavior is strongly dependent on the consistency of mini-batch data distributions. This study
examines how irregular mini-batch composition impacts BN statistical stability, convergence
dynamics, inference reliability, and representation structure. Using controlled experiments with varied
batch formation strategies, we observe that distributional inconsistency causes fluctuating
normalization parameters, oscillatory loss trajectories, and reduced cluster coherence in feature space.
These effects persist into inference due to biased running averages accumulated during training,
leading to measurable performance degradation. The findings emphasize that maintaining
distributional consistency within mini-batches is as critical as batch size selection for reliable BN
operation, and highlight the need for improved normalization and sampling techniques in real-world
and distributed learning environments.