Lie Brackets Quantify How Training Example Order Shapes Neural Network Learning

Ideal machine learning models would treat training data as an unordered set, where updating on one example then another yields the same result as the reverse order. From a Bayesian perspective, the dataset is exchangeable, so gradient steps should commute. However, neural networks trained via gradient descent violate this principle—the sequence in which examples are presented influences the final parameters. This order dependence isn't just a theoretical curiosity; it affects model behavior in practice, potentially encoding biases or inefficiencies in the learning process.

To analyze this phenomenon, researchers turned to differential geometry. Each training example can be viewed as a vector field on the parameter space, where the vector at any point indicates the direction of steepest descent for that example's loss. Specifically, for parameters 𝜃 and example 𝑥, the vector field is 𝑣(𝑥)(𝜃) = −∇_𝜃 𝐿(𝑥). A single gradient step then moves parameters along this field: 𝜃' = 𝜃 + 𝜖 𝑣(𝑥)(𝜃), with 𝜖 as the learning rate.

The key insight lies in the Lie bracket of two such vector fields, [𝑣(𝑥), 𝑣(𝑦)] = (𝑣(𝑥)⋅∇_𝜃)𝑣(𝑦) − (𝑣(𝑦)⋅∇_𝜃)𝑣(𝑥). This operation measures the failure of the flows to commute. Through a second-order Taylor expansion in the learning rate 𝜖, the difference in parameters after updating 𝑥 then 𝑦 versus 𝑦 then 𝑥 is precisely 𝜖² times the Lie bracket evaluated at 𝜃. Thus, the Lie bracket directly quantifies how swapping two examples perturbs the optimization trajectory at a per-parameter level.

To make this concrete, the team experimented with an MXResNet-like architecture (minus attention layers) trained on the CelebA dataset for 5,000 steps with a batch size of 32, using Adam (lr=5e-3, betas=(0.8,0.999)). CelebA presents 40 binary facial attributes, requiring the network to predict each independently—a setup where the loss implicitly assumes feature independence. They saved weight checkpoints periodically and computed Lie brackets between the first six test examples at each checkpoint, focusing on how swapping examples affected logits across the batch.

Results showed that Lie bracket magnitudes varied widely across parameters but exhibited a striking correlation with gradient magnitudes when plotted on a log-log scale. For each parameter tensor, the root-mean-square (RMS) of the Lie bracket scaled linearly with the RMS gradient, suggesting a constant of proportionality tied to the intrinsic non-commutativity of the example pair and training progress. This implies that the bracket's size is largely driven by the vector field 𝑣(𝑥) interacting with the derivative of 𝑣(𝑦), rather than parameter-specific nuances.

Notably, past step 600, the Black_Hair and Brown_Hair logits consistently showed large perturbations under most Lie brackets. Since these attributes are mutually exclusive in the dataset (no image has both), yet the model predicts them independently, the loss function penalizes correlated errors as if they were independent events. When uncertain, the network might assign 50% probability to each, which the loss interprets as a 25% chance for the (True,True) combination—an impossibility. The hypothesis is that this mismatch between the model's desired prediction (a 50:50 split between (True,False) and (False,True)) and the loss function's assumptions amplifies order sensitivity for these features, making Lie brackets a potential diagnostic tool for identifying such loss-function inadequacies.

The work bridges abstract geometry with practical training dynamics, offering a lens to scrutinize how example sequencing shapes learned representations. While not proposing a new training algorithm, it provides a framework to detect and understand order-dependent effects—knowledge that could inform future optimizers, curriculum design, or bias mitigation strategies in deep learning.