Adam and Eve

I used Adam for years before noticing something embarrassingly basic: Adam almost never takes the step that the gradient asks it to take.

The gradient has a direction and a magnitude. Adam smooths the direction, divides each coordinate by a different estimate of its recent scale, corrects two startup biases, adds a numerical floor, and only then moves the parameters. If the raw gradient points north-east, Adam may walk almost east. If one coordinate's gradient is a thousand times larger than another, their first Adam steps can still be nearly equal.

That sounds like a betrayal of gradient descent. It is actually the point.

Adam was introduced by Diederik Kingma and Jimmy Ba in December 2014 as a first-order method for stochastic optimization.[1] Its name means adaptive moment estimation. The name is unusually honest: the optimizer keeps an exponential estimate of the gradient's first raw moment and another of its second raw moment, then uses their ratio to build a new coordinate system for learning.

This essay is about what that sentence hides. We will arrive at Adam from inside its update, not by marching through a museum of optimizers. Momentum, AdaGrad, and RMSProp will appear exactly where Adam needs them. Then we will take the method apart mathematically, see why AdamW and AMSGrad repair different weaknesses, and put Adam beside Newton, BFGS, L-BFGS, Shampoo, Sophia, Lion, Adafactor, and Muon. The final question is not which optimizer has the grandest equation. It is what each optimizer knows, what that knowledge costs, and whether “better” survives contact with an actual training run.

One gradient, one learning rate, one immediate problem

Let the parameters be \(\theta\in\mathbb{R}^d\), and let the mini-batch loss at step \(t\) be \(L_t(\theta)\). Plain stochastic gradient descent computes

The equation is almost offensively clean. Every coordinate shares the same learning rate \(\alpha\), and the optimizer remembers nothing. Yet a neural network rarely offers coordinates of comparable scale. One parameter can receive large, erratic gradients; another can receive tiny but consistently signed ones. A single \(\alpha\) must be small enough not to explode the first coordinate and large enough not to freeze the second.

On a narrow quadratic valley, the gradient makes the mismatch visible. It points in the direction of steepest local increase, not necessarily toward the minimum. A large component across the valley causes oscillation; a small component along the valley makes progress slow. This is the first crack that Adam tries to seal, but it borrows two different pieces of machinery to do it.

Momentum gives the gradient a memory

Suppose the gradient changes because each mini-batch is a noisy view of the same underlying direction. Instead of trusting the current batch, keep a moving average:

Repeated substitution reveals what \(m_t\) really is:

assuming \(m_0=0\). Recent gradients receive more weight; old ones fade exponentially. When gradients keep the same sign, their evidence accumulates. When they alternate across a steep ravine, they cancel. This is momentum interpreted as a low-pass filter.

But momentum solves only the directional part of the problem. If one coordinate is naturally a thousand times larger, its moving average is also roughly a thousand times larger. The ball has memory, but every dimension still shares one speed limit.

AdaGrad gives every coordinate its own clock

AdaGrad takes a different route. It accumulates the squared gradients coordinate by coordinate:

All products, square roots, and divisions here are element-wise. A coordinate that repeatedly receives large gradients grows a large denominator and slows down. A rarely active feature retains a relatively large effective step. This was precisely the “needle in a haystack” geometry highlighted by Duchi, Hazan, and Singer: sparse but predictive features should not be drowned by frequently updated ones.[2]

AdaGrad's gift is also its failure mode. Since \(r_t\) only grows, the effective learning rate

only shrinks. On a long non-stationary training run, ancient gradients retain permanent voting rights. Eventually a coordinate can become cautious not because it is currently unstable, but because it was active a million updates ago.

RMSProp, presented in Geoffrey Hinton's 2012 neural-net lectures and attributed there to Tijmen Tieleman, changes one symbol and changes the behavior:[3]

The sum becomes an exponential moving average. RMSProp keeps AdaGrad's per-coordinate scale, but it forgets. Old curvature-like evidence fades, so the learning rate can recover when the landscape changes.

Adam is the splice

Adam joins momentum's first-moment memory to RMSProp's second-moment scaling and adds one correction that both moving averages need at startup:

The original defaults were \(\alpha=10^{-3}\), \(\beta_1=0.9\), \(\beta_2=0.999\), and \(\varepsilon=10^{-8}\). They became familiar enough to look like constants of nature. They are not. They describe two memory horizons, a global update scale, and the point at which numerical stability begins to dominate the adaptive denominator.

m = 0; v = 0
for t = 1, 2, ...:
    g = gradient(loss, parameters)
    m = beta1 * m + (1 - beta1) * g
    v = beta2 * v + (1 - beta2) * g * g
    m_hat = m / (1 - beta1 ** t)
    v_hat = v / (1 - beta2 ** t)
    parameters -= learning_rate * m_hat / (sqrt(v_hat) + epsilon)

The two memories run on different clocks

The defaults \(\beta_1=0.9\) and \(\beta_2=0.999\) are not merely “close to one.” They imply dramatically different horizons. The contribution of an observation \(k\) steps old is proportional to \(\beta^k\). Its half-life is therefore

For \(\beta_1=0.9\), the half-life is about \(6.6\) steps. For \(\beta_2=0.999\), it is about \(693\) steps. Adam's numerator changes its mind quickly; its denominator carries a much longer memory of how large gradients have recently been.

This asymmetry is useful. Direction should respond when the loss changes; scale should be stable enough not to let one strange mini-batch rewrite every effective learning rate. It also creates a subtle lag: after a regime change, \(m_t\) may adapt while \(v_t\) still describes the old world.

Bias correction is not cosmetic

Both moving averages start at zero. If gradients are drawn from a stationary distribution with mean \(\mu\), then

So \(m_t\) is pulled toward zero by exactly the factor \(1-\beta_1^t\). If \(q=\mathbb{E}[g_t^2]\) coordinate-wise, then similarly

Dividing by those missing masses gives \(\widehat m_t\) and \(\widehat v_t\). The correction matters especially for \(v_t\): with \(\beta_2=0.999\), its uncorrected first value contains only one-thousandth of \(g_1^2\). Without correction, the denominator starts much too small and the first updates can be much too large.

The cleanest way to see Adam's early behavior is to give it a constant non-zero gradient. At the first step, bias correction yields

and therefore

Unless a gradient coordinate is comparable to \(\varepsilon\), its magnitude almost disappears. A gradient of \(10^{-3}\) and one of \(10^2\) can both produce an update near \(\alpha\). This is why Adam can move parameters with very different raw gradient scales without requiring a separate hand-tuned learning rate for every layer.

The second moment is not curvature

The notation \(v_t\) has misled generations of readers into calling it a variance. It is an exponential estimate of the uncentered second moment, because Adam stores \(g_t^2\), not \((g_t-m_t)^2\). For a scalar gradient,

In a roughly stationary region, write \(\mu=\mathbb{E}[g]\) and \(\sigma^2=\operatorname{Var}(g)\). If the moving averages are well settled, Adam's normalized direction is approximately

This makes Adam look less like “divide by curvature” and more like a softened sign method whose magnitude is controlled by signal-to-noise ratio. A coordinate with a consistent sign approaches a unit-sized normalized update. A coordinate whose gradients are mostly noise is damped. Balles and Hennig developed this interpretation in detail: Adam's numerator largely determines the sign, while relative stochastic variance strongly shapes the magnitude.[5]

Adam changes the coordinate system

Define the diagonal preconditioner

Then the update is simply

The word preconditioner is doing real work. Adam does not merely choose a smaller or larger global step; it stretches parameter space along coordinate axes. Directions with a history of large gradients are compressed. Directions with small gradients are expanded. Ignoring \(\varepsilon\), this makes the update invariant to positive diagonal rescaling of the gradients, one of the properties emphasized in the original paper.[1]

But diagonal is the limitation. Rotate the parameterization, and Adam generally traces a different path. It can fix an axis-aligned ellipse; it cannot represent the off-diagonal coupling of a tilted valley. Newton and BFGS can, at a price.

The convergence problem was not “Adam never works”

In 2018, Reddi, Kale, and Kumar showed a simple convex construction on which Adam can fail to converge.[6] The problem is easiest to describe through Adam's coordinate-wise effective learning rate:

Because \(v_t\) forgets, its value can decrease. That means an effective learning rate can increase later, even when the global schedule \(\alpha_t\) does not. In the counterexample, rare large gradients and frequent small gradients interact with that forgetting so that Adam keeps assigning the wrong relative importance to updates.

AMSGrad changes one line:

and divides by \(\sqrt{\widetilde v_t}\). The denominator can no longer shrink, restoring the long-term memory that RMSProp removed from AdaGrad. Notice the irony: RMSProp solved AdaGrad's overlong memory by forgetting; AMSGrad repairs a theoretical Adam failure by remembering the largest scale forever.

This does not make every ordinary Adam run suspect. Later analyses proved convergence under smoothness and bounded-gradient assumptions with suitable schedules.[7] Constant learning-rate Adam does not generally converge exactly to a stationary point, but neither does constant learning-rate SGD; both can settle into a noisy neighborhood. The useful conclusion is narrower: Adam's excellent empirical behavior is not a universal mathematical guarantee, and the moving denominator is part of the optimization dynamics, not harmless bookkeeping.

AdamW fixes a different mistake

For plain SGD, adding an \(L_2\) penalty to the loss is equivalent, after rescaling, to multiplying weights by a decay factor. With an adaptive preconditioner, that equivalence breaks.

If we add \(\frac{\lambda}{2}\|\theta\|^2\) to the loss, the gradient becomes \(g_t+\lambda\theta_{t-1}\). Adam then preconditions the penalty too:

Every coordinate now decays at a different rate because \(P_t\) is diagonal and history-dependent. A parameter with a large second-moment estimate receives less regularization. Calling this ordinary “weight decay” hides what the optimizer is doing.

AdamW decouples the operations:[8]

The loss gradient is adapted; the parameters are decayed directly. AdamW is therefore not “a newer Adam with better momentum.” It corrects the interaction between regularization and Adam's geometry. That distinction is why modern recipes so often mean AdamW when they casually say Adam.

Newton sees curvature; Adam sees history

Near a point \(\theta\), approximate a smooth objective by a quadratic:

where \(H=\nabla^2L(\theta)\) is the Hessian. Differentiating this local model with respect to \(s\) gives

If the model is exactly quadratic and \(H\) is positive definite, Newton's step lands at the minimum in one move. More generally it rescales and rotates the gradient using local curvature. This is the information Adam does not have.

But an explicit Hessian has \(d^2\) entries. At a billion parameters, the problem is not that the matrix is large. The problem is that the phrase “store the matrix” has stopped being physically serious.

BFGS: reconstruct the Hessian from footprints

In 1970, Charles Broyden, Roger Fletcher, Donald Goldfarb, and David Shanno independently arrived at closely related rank-two variable-metric updates.[10][11][12][13] Their initials became BFGS.

The method does not evaluate the Hessian directly. It observes a step and the resulting change in gradient:

For a quadratic, \(y_k=Hs_k\). So a Hessian approximation \(B_{k+1}\) should at least satisfy the secant equation

BFGS chooses a symmetric rank-two update that satisfies this constraint while changing the previous metric as little as possible in a particular weighted sense. In inverse form, with \(H_k\) now denoting an approximation to the inverse Hessian and \(\rho_k=(y_k^\top s_k)^{-1}\),

The search direction is \(p_k=-H_kg_k\), usually followed by a line search. If \(H_k\) is positive definite and \(y_k^\top s_k>0\), the update preserves positive definiteness. On smooth deterministic problems, good line searches make BFGS remarkably powerful; near a well-behaved solution it can converge superlinearly without ever forming the true Hessian.

Full BFGS still stores a dense \(d\times d\) matrix. L-BFGS, introduced by Jorge Nocedal, avoids that by storing only the most recent \(m\) pairs \((s_k,y_k)\) and applying the implied inverse Hessian through a two-loop recursion.[14] Its memory is \(O(md)\), not \(O(d^2)\).

That sounds as though L-BFGS should replace Adam. It usually does not, for three connected reasons.

First, Adam stores two state vectors, about \(2d\) scalars. L-BFGS with a history of ten stores roughly \(20d\) scalars for ten step-gradient pairs, before temporary workspaces. Limited-memory is relative to a dense Hessian, not necessarily relative to Adam.

Second, mini-batch noise corrupts the curvature observation. If

then \(y_k\) mixes the effect of moving \(\theta\) with the effect of changing the sampled batch \(\xi\). The secant pair can describe data noise as if it were curvature. Using the same or very large batches helps, but costs more computation.

Third, line searches and curvature updates create synchronization and extra function evaluations. Adam's cheap streaming update fits accelerator-heavy training: one backward pass yields the gradient, and the optimizer performs element-wise operations. A quasi-Newton method asks more of both the mathematics and the system.

This is not a verdict against BFGS. For small or medium smooth deterministic objectives, full-batch fine-tuning, and settings where accurate local convergence matters more than cheap noisy steps, BFGS or L-BFGS can be decisively better. For networks with tens of millions of variables, even specialized quasi-Newton research uses block-diagonal, Kronecker-factored, and damped approximations because naive BFGS is impractical.[15] “Better” changes when the budget changes.

So are the optimizers after Adam actually better?

There is no single scalar called optimizer quality. Better can mean fewer steps, lower wall-clock time, less optimizer memory, fewer total FLOPs, less hyperparameter search, lower training loss, or better downstream generalization. Those objectives disagree often enough that a leaderboard without a budget is mostly typography.

Still, the methods that followed Adam are intellectually revealing because each attacks a specific bill Adam leaves unpaid.

Adafactor notices that Adam's second-moment tensor costs one scalar per parameter. For a matrix, it stores row and column statistics and reconstructs a factored approximation, reducing auxiliary memory to sublinear in the number of matrix entries. In the original Transformer experiment, this recovered performance comparable to the published Adam regime while using much less optimizer state.[16] It is better when memory is the binding constraint; factorization is also an approximation, not free accuracy.

Shampoo attacks Adam's inability to rotate. It maintains preconditioning matrices along each tensor dimension, retaining some cross-coordinate structure without storing a full \(d\times d\) matrix.[17] This can improve conditioning and step efficiency. The bill arrives as matrix inverse roots, implementation complexity, and distributed systems work.

Lion goes in the opposite direction. It discards the second-moment buffer and uses the sign of a momentum-like quantity, keeping only one state vector. Its authors reported gains on several vision and diffusion workloads, similar or better results on some language tasks, and also cases where improvements were small or not statistically significant.[18] Lion can be leaner than Adam, but its sign update has a different norm and generally needs a smaller learning rate and a retuned recipe.

Sophia finally puts actual curvature into a scalable language-model optimizer. It divides momentum by a periodically estimated diagonal Hessian and clips the result to control non-convex or rapidly changing curvature. The original experiments reported roughly a twofold speed-up over Adam on GPT models from 125 million to 1.5 billion parameters.[19] That is substantial evidence for those settings, not a proof that diagonal Hessian estimation dominates every model, scale, and infrastructure stack.

Muon treats a weight matrix as a matrix rather than a bag of unrelated scalars. It orthogonalizes a momentum update, approximately replacing its singular values by ones, commonly through a few Newton-Schulz iterations. Scaling work reported about twice the computational efficiency of AdamW under the paper's compute-optimal language-model experiments.[20] Yet current Muon recipes still use AdamW for embeddings and other non-matrix parameters. Even the proposed successor does not entirely dismiss Adam; it specializes where matrix geometry is useful.

A useful 2025 comparison makes the ambiguity concrete. Under its pre-training setup, Sophia achieved the lowest training and validation loss, Lion used the fewest GPU hours, and AdamW produced the best downstream evaluations.[21] Three winners appeared because three meanings of “best” were measured.

The generalization argument is real, but not universal

Adaptive optimizers do more than reach a solution quickly; their preconditioners choose a path through parameter space. In overparameterized models, many parameter vectors can fit the training data, and different paths can end at different solutions. Wilson and colleagues constructed settings and reported experiments where adaptive methods reached worse test performance than SGD even when they optimized the training objective well.[9]

It is tempting to compress this into “SGD generalizes, Adam overfits.” That slogan is too strong. Architecture, normalization, weight decay, schedule, batch size, data regime, and training duration all matter. AdamW specifically repaired one regularization mismatch that had made older comparisons less clean. In large language models, AdamW became a default not because generalization stopped mattering, but because stable optimization across huge, heterogeneous parameter groups became extraordinarily valuable.

What I would choose, and what I would measure

If I had to begin a modern Transformer fine-tune tomorrow, I would start with AdamW. Not because it is theoretically supreme, but because its behavior, schedules, and failure modes are well mapped, and because it tolerates heterogeneous gradient scales with little ceremony.

If optimizer memory were the bottleneck, I would test Adafactor or another deliberately memory-efficient variant. If I were training a vision model with a mature SGD recipe and cared about the final fraction of a percent, I would not abandon momentum SGD merely because Adam reaches low training loss sooner. If the objective were smooth, deterministic, and modest in dimension, I would try L-BFGS early rather than forcing a stochastic deep-learning default onto a numerical optimization problem.

And if a training run were expensive enough that a ten-percent optimizer improvement mattered financially, I would benchmark Shampoo, Sophia, Muon, or Lion on a smaller but faithful proxy. I would hold the compute budget fixed, tune each method fairly, and report at least four things: final validation quality, wall time, total FLOPs, and peak memory. Step count alone rewards optimizers that do more work per step; wall time alone can reward one implementation; training loss alone may not survive evaluation.

The practical hierarchy is less dramatic than the paper titles. AdamW is the robust baseline. Specialized methods are hypotheses about a particular bottleneck. BFGS is brilliant when gradients are clean and curvature is worth learning. Muon is promising when parameters are large matrices and the system can exploit that structure. No method is better before the problem defines the bill.

What this dissection does not say

It does not say that \(v_t\) is a Hessian. It does not say that Adam is literally a sign optimizer after the first step. It does not say that AMSGrad always beats Adam, that AdamW cures every generalization gap, or that a twofold improvement reported on one family of language-model runs transfers unchanged to another architecture.

It also does not say optimizer choice can be separated from the rest of the training recipe. Learning-rate warmup, decay schedules, gradient clipping, parameter groups, normalization, batch size, mixed precision, distributed communication, and weight decay can alter the result as much as the named update rule. “Adam” in a codebase is often an ecosystem, not six equations.

The gradient is only the beginning of the step

Return to the first observation: Adam does not follow the gradient.

It listens to the gradient through two memories. One asks, “Which direction has been consistent lately?” The other asks, “How large has this coordinate usually been?” Bias correction stops both memories from confusing youth with evidence. The ratio turns magnitude into something closer to confidence, and the diagonal preconditioner quietly replaces the geometry of the parameter space.

That is why Adam works so often, why it can fail in ways SGD cannot, why AdamW matters, and why BFGS is not simply Adam with more mathematics. Every optimizer is an argument about which parts of the past deserve to shape the next step. Adam's argument is cheap, local, diagonal, and extraordinarily useful. Its successors are better only when they know something the problem is willing to pay for.