I was looking for the most pointless neural network I could build.
Take an input, pass it through one hundred fully connected layers, and remove every activation function. No ReLU. No sigmoid. No normalization. Nothing between the matrices except more matrices. It looks deep in the architecture diagram, consumes memory like a deep model, and gives back:
\[ f(x)=W_{100}W_{99}\cdots W_2W_1x. \]
But the product of matrices is another matrix. If I define
\[ W:=W_{100}W_{99}\cdots W_2W_1, \]
then the entire network becomes \(f(x)=Wx\). One hundred layers collapse into one. It is linear regression wearing a deep learning costume.
That should end the story. It does not.
The network is linear as a function of its input, but training it is a nonlinear dynamical system. It can sit on long plateaus and then learn suddenly. It can prefer low-rank solutions even when no rank constraint appears in the objective. Make the factorization deeper and that preference can become stronger, despite the set of end-to-end matrices remaining unchanged. The architecture has gained no new functions, yet gradient descent can choose a different answer.
This is the strange point of deep linear networks. They separate two ideas that are usually fused together in deep learning: what a model can represent, and what optimization makes it learn.
The function is shallow, but the parameterization is deep
Consider a depth-\(N\) linear network:
\[ h_0=x, \qquad h_j=W_jh_{j-1}, \qquad f_\theta(x)=W_NW_{N-1}\cdots W_1x. \]
The parameters are \(\theta=(W_1,\ldots,W_N)\), while the end-to-end predictor depends only on their product:
\[ W(\theta)=W_NW_{N-1}\cdots W_1. \]
If the hidden layers are wide enough and introduce no rank bottleneck, every end-to-end matrix available to a one-layer model is also available to the deep model. The two models therefore have the same function class. This caveat matters. A narrow hidden layer would force \(\operatorname{rank}(W)\) to be at most its width and would genuinely change what the network can represent. I am interested in the full-width case, where depth adds parameters without adding functions.
Even here, the parameter spaces are radically different. The shallow model assigns one point \(W\) to one predictor. The deep model assigns an enormous family of points to the same predictor. For any invertible matrix \(R\), the two factorizations
\[ W_2W_1 \quad\text{and}\quad (W_2R^{-1})(RW_1) \]
produce exactly the same end-to-end matrix. Predictions cannot tell these parameter settings apart, but gradient descent moves through them differently. The redundancy is invisible in function space and very real in parameter space.
Figure 1. A full-width deep linear network and a single linear layer can represent the same end-to-end matrix. What changes is not the function class but the coordinates through which gradient descent reaches that matrix.
The exact place where equivalence breaks
Suppose the training examples are stored as columns of \(X\), the targets as columns of \(Y\), and we use squared loss:
\[ \ell(W)=\frac{1}{2}\lVert WX-Y\rVert_F^2. \]
As a function of the single matrix \(W\), this is a convex problem. Its gradient is
\[ \nabla_W\ell(W)=(WX-Y)X^\top. \]
Direct gradient flow is therefore as plain as it looks:
\[ \dot W=-\nabla_W\ell(W). \]
Now write the same matrix as \(W=W_N\cdots W_1\). Let
\[ A_j=W_N\cdots W_{j+1}, \qquad B_j=W_{j-1}\cdots W_1, \]
so that \(W=A_jW_jB_j\). The chain rule gives
\[ \frac{\partial \ell}{\partial W_j} =A_j^\top\nabla_W\ell(W)B_j^\top. \]
Every layer sees the same end-to-end gradient, but it sees that gradient filtered through all the layers before and after it. If each factor follows gradient flow, \(\dot W_j=-\partial\ell/\partial W_j\), then the product evolves as
\[ \dot W =-\sum_{j=1}^{N}A_jA_j^\top\,\nabla_W\ell(W)\,B_j^\top B_j. \]
This is not ordinary gradient flow on \(W\). The matrices on the left and right of the gradient change throughout training because they are built from the factors themselves. Depth has converted a fixed Euclidean update into a state-dependent, direction-dependent transformation of the gradient. In modern language, the factorization behaves like an adaptive preconditioner that the model creates for itself.
The predictions may depend only on \(W\), but the velocity of \(W\) remembers how it was factorized. That is the whole trick.
A one-number network is already nonlinear
The easiest way to see the strange dynamics is to throw away the matrices.
Take a scalar depth-\(N\) network whose end-to-end weight is
\[ p=w_Nw_{N-1}\cdots w_1, \]
and ask it to match a positive target \(y\) using
\[ \ell(p)=\frac{1}{2}(p-y)^2. \]
Assume the factors start positive and balanced, so \(w_1=\cdots=w_N=a\) and \(p=a^N\). The gradient-flow equation for each factor is
\[ \dot a=-(p-y)a^{N-1}. \]
Differentiate \(p=a^N\):
\[ \dot p =Na^{N-1}\dot a =-Na^{2N-2}(p-y) =Np^{2-2/N}(y-p). \]
There it is. The predictor is one scalar multiplication, but its learning rule is nonlinear and depends explicitly on depth.
For a one-layer model, \(\dot p=y-p\). For depth two, \(\dot p=2p(y-p)\). For depth four, \(\dot p=4p^{3/2}(y-p)\). If \(p\) begins extremely close to zero, the deeper model barely moves because the gradient is multiplied by a power of \(p\). Once \(p\) grows, the same multiplier makes movement much faster. Long silence, then a sudden transition.
Saxe, McClelland, and Ganguli found precisely this kind of behaviour in their exact analysis of deep linear learning dynamics: plateaus followed by rapid drops in error, even though the network's input-output map remained linear. These are usually described as nonlinear learning phenomena. Here they emerge without a nonlinear activation anywhere in sight.
Figure 2. Schematic trajectories for balanced positive scalar factorizations initialized near zero. Depth does not change the target function, but the multiplier \(Np^{2-2/N}\) can create an increasingly long plateau before rapid learning. The curves are illustrative rather than a numerical reproduction of one fixed experiment.
A non-convex problem without bad local minima
There is another reason deep linear networks became a useful theoretical laboratory. They are non-convex for the most boring possible reason: matrix multiplication.
The loss \(\ell(W)\) may be convex in the product matrix, while
\[ \phi(W_1,\ldots,W_N)=\ell(W_N\cdots W_1) \]
is non-convex in the individual factors. If one factor changes, the effect of that change depends multiplicatively on all the others. Yet the landscape is much kinder than the word non-convex suggests. For squared loss, classical and later results show that every local minimum of a deep linear network is global under the standard setup. Critical points that are not global minima are saddles.
That does not mean optimization is trivial. Kawaguchi showed that sufficiently deep linear networks can contain non-strict or "bad" saddle points whose Hessian has no negative eigenvalue. The network can also suffer vanishing or exploding gradients because every layer's update contains products of other layers. No poor local minima does not mean no slow regions, no delicate initialization, or guaranteed success for every finite-step optimizer.
This is a useful correction to a common instinct. Looking only at the locations of local minima can miss the real difficulty. The path can be hard even when the destination map looks friendly.
When fitting the data does not determine the answer
If a regression problem has a unique optimal end-to-end matrix, shallow and deep parameterizations may eventually agree on that matrix. Their journeys differ, but their destination is fixed. The more interesting case is overparameterized or underdetermined learning, where many matrices fit the observed data perfectly.
Then optimization must choose.
This choice is implicit regularization. There is no explicit penalty in the loss saying "prefer this solution." The preference is produced by the parameterization, initialization, learning rate, and optimization dynamics. Ordinary gradient descent on an underdetermined linear regression initialized at zero famously selects a minimum Euclidean-norm solution. Factorize the same predictor into several matrices and the preference changes.
Matrix completion is the cleanest place to watch this happen. Let \(M^\star\in\mathbb{R}^{d\times d}\) be an unknown matrix, and suppose we observe only entries indexed by a set \(\Omega\). We train by minimizing
\[ \ell(W) =\frac{1}{2}\sum_{(i,j)\in\Omega} \left(W_{ij}-M^\star_{ij}\right)^2 =\frac{1}{2}\left\lVert P_\Omega(W-M^\star)\right\rVert_F^2, \]
where \(P_\Omega\) keeps observed entries and sets the rest to zero. Any matrix matching the visible entries has zero training loss. Most such matrices say nonsense about the hidden entries. If the true matrix is low rank, however, choosing a low-rank completion can recover the missing structure.
Now parameterize \(W\) as
\[ W=W_NW_{N-1}\cdots W_1 \]
with full-dimensional factors. Nothing in this parameterization forces the product to be low rank. Every factor can be \(d\times d\), so the product can have rank \(d\). Nothing in the loss penalizes rank either. Still, gradient descent from a small initialization tends to discover low-rank products. More surprisingly, making the factorization deeper can intensify that tendency.
Figure 3. Matrix completion turns generalization into a visible choice. Many matrices interpolate the observed entries, but only some recover the unobserved structure. Full-dimensional factorization does not impose a hard rank constraint, yet its gradient dynamics can prefer low-rank completions.
Depth changes the clock of each singular value
This is where the scalar toy model returns in matrix form.
Write the end-to-end matrix using a singular value decomposition:
\[ W(t)=U(t)\Sigma(t)V(t)^\top, \qquad \Sigma(t)=\operatorname{diag}\!\left(\sigma_1(t),\ldots,\sigma_r(t)\right). \]
Under gradient flow and a balanced initialization, Arora, Cohen, Hu, and Luo derived the evolution of each singular value:
\[ \dot\sigma_r(t) =-N\,\sigma_r(t)^{\,2-2/N} \left\langle \nabla\ell(W(t)), u_r(t)v_r(t)^\top \right\rangle. \]
The inner product measures how strongly the loss gradient wants to move along the rank-one direction \(u_rv_r^\top\). The striking term is the multiplier
\[ N\sigma_r^{,2-2/N}. \]
With no factorization, \(N=1\), the multiplier is one. Every singular direction follows the projected gradient without depending on its current size. At depth two, the multiplier is \(2\sigma_r\). At depth three, it is \(3\sigma_r^{4/3}\). As depth grows, the exponent approaches two.
Suppose two singular directions receive comparable projected gradients and \(\sigma_a>\sigma_b>0\). Their relative update multipliers are
\[ \frac{\sigma_a^{,2-2/N}}{\sigma_b^{,2-2/N}} =\left(\frac{\sigma_a}{\sigma_b}\right)^{2-2/N}. \]
Large singular values move faster. Small singular values are held near zero. Increasing depth raises the exponent and widens the difference between their clocks. The model does not explicitly delete small modes. It simply gives already-growing modes a larger effective learning rate and makes weaker modes wait.
This is a form of rich-get-richer dynamics in the spectrum. A few singular directions emerge early, while the rest remain dormant. Stop training after the observed entries are fitted and the product matrix can look much lower rank than its full-dimensional factors ever required.
Figure 4. Conceptual view of the singular-value dynamics. The update multiplier \(\sigma_r^{2-2/N}\) favors modes that are already large and suppresses small ones. A larger depth increases the exponent and strengthens this spectral separation. The bars illustrate the mechanism rather than reproduce a particular run.
The experiment where extra layers help without adding capacity
The NeurIPS 2019 study tested this on matrix completion with full-dimensional factors. The targets were random \(100\times100\) matrices of rank 5 or rank 10. The authors trained factorizations of depths 2, 3, and 4 using small learning rates and near-zero random initialization. Because every hidden dimension was 100, the deeper models had no tighter rank constraint than the shallow one.
When many entries were visible, all depths recovered the target well. There was enough information for the problem itself to determine the right answer. When few entries were visible, depth mattered. The depth-3 and depth-4 factorizations produced substantially lower reconstruction error than depth 2. The advantage appeared precisely where many zero-training-loss completions remained possible and the optimizer's preference had room to matter.
The important comparison was not merely test error. The deeper factorizations returned products with lower effective rank. If
\[ p_r=\frac{\sigma_r}{\sum_j\sigma_j}, \]
then one common effective-rank measure is
\[ \operatorname{erank}(W) =\exp\!\left(-\sum_r p_r\log p_r\right). \]
It is the exponential of the entropy of the normalized singular-value spectrum. If one singular value dominates, the effective rank approaches one. If \(k\) singular values are equal and the rest are zero, it equals \(k\). The experiments showed deeper factorizations concentrating their spectrum more aggressively.
This is the clean answer to the original provocation. A hundred linear layers do not create a richer predictor. They create a different route through the same predictors, and in an underdetermined problem the route decides which predictor is selected.
The tempting explanation that turns out to be too neat
At this point there is an obvious thing to say: deep factorization must secretly minimize some low-rank norm.
The rank of a matrix counts its non-zero singular values:
\[ \operatorname{rank}(W)=\sum_r\mathbf{1}\{\sigma_r(W)>0\}. \]
Because rank is discontinuous and difficult to optimize, a standard convex surrogate is the nuclear norm:
\[ \lVert W\rVert_*=\sum_r\sigma_r(W). \]
Gunasekar and collaborators studied depth-2 matrix factorization and provided theoretical and empirical evidence connecting gradient descent from infinitesimal initialization to minimum nuclear-norm solutions, with a proof under restricted commuting measurements and a broader conjecture beyond that case. It was natural to imagine a ladder: depth two minimizes the nuclear norm, while greater depth minimizes increasingly rank-like Schatten quasi-norms,
\[ \lVert W\rVert_{S_p}^{p}=\sum_r\sigma_r(W)^p, \qquad 0<p<1. \]
As \(p\) approaches zero, this expression behaves more like a count of non-zero singular values. The particularly seductive choice is \(p=2/N\), which also appears when explicit squared penalties on the individual factors are collapsed into an end-to-end variational penalty.
Seductive is not the same as correct.
Arora and collaborators showed that the existing nuclear-norm theory in the restricted commuting setting extends to arbitrary depth. In other words, that theory cannot explain why depth three behaves differently from depth two. In the data-poor matrix-completion experiments, the factorized solutions also did not generally minimize nuclear norm. Deeper models sometimes accepted a higher nuclear norm while achieving lower effective rank and better recovery.
Razin and Cohen later sharpened the objection. They constructed natural matrix-factorization problems in which gradient-based optimization moves toward low-rank completion while every norm and quasi-norm of the product diverges. No fixed norm-minimization story can describe such a trajectory.
The better explanation is dynamical. Depth does not merely add a hidden penalty to the final objective. It changes how different spectral modes move throughout training. The preference lives in the trajectory, not necessarily in a scalar regularizer waiting at the endpoint.
Depth also changes which minimum looks flat
There is a second complication. Even after the end-to-end predictor is fixed, it has infinitely many factorizations, and those factorizations do not have the same curvature.
For the simplest two-factor case, scaling one factor up and the other down leaves the product untouched:
\[ (cW_2)(c^{-1}W_1)=W_2W_1. \]
The function and loss are identical, but the Hessian with respect to \((W_1,W_2)\) can change dramatically. Parameter-space flatness is therefore not a property of the predictor alone. It depends on which representative of the factorization the optimizer chooses.
A 2024 analysis by Marion and Chizat studied deep linear networks for univariate regression and proved an implicit preference toward relatively flat global minima under gradient flow. The smallest achievable sharpness itself grows with depth, so "flat" must be judged relative to the geometry created by that depth. For small-scale initialization, they also showed that learned factors become approximately rank one and their singular vectors align.
Once again, the product map hides the story. Two models can implement the same linear function while living in parameter-space neighborhoods with different curvature, different gradient scales, and different sensitivity to finite learning rates.
What deep linear networks are actually good for
Deep linear networks are obviously not replacements for nonlinear models. Collapse all their layers and they still cannot represent XOR, curved decision boundaries, or hierarchical nonlinear features. Their value is diagnostic.
They strip away nonlinear expressivity while retaining several difficulties associated with deep learning: non-convex parameterization, saddle points, vanishing and exploding gradients, sensitivity to initialization, balancedness across layers, stage-like learning, and implicit regularization. If one of these phenomena survives after every activation is removed, nonlinearity cannot be its full explanation.
They also make one distinction impossible to ignore:
\[ \text{function class} \neq \text{optimization bias}. \]
An architecture is not specified completely by the set of functions it can express. The coordinates used to express those functions decide how gradients move. The optimizer, parameterization, initialization, and stopping time together define a preference over solutions. In an underdetermined problem, that preference can be the only reason one interpolating solution generalizes and another does not.
What the result does not say
There are several ways to overstate this story, so I want to be precise.
First, depth does not always help. The strongest low-rank results depend on the task, loss, initialization, and optimization regime. Near-zero or balanced initialization and sufficiently small learning rates are not decorative assumptions. Deep products can train painfully slowly near zero, and badly scaled factors can make gradients vanish or explode.
Second, full-width deep linear networks have the same end-to-end function class only when no hidden layer forms a bottleneck. A width-\(k\) bottleneck explicitly enforces \(\operatorname{rank}(W)\le k\), which is a different and much simpler source of low rank.
Third, the singular-value equation above is a gradient-flow result under balancedness conditions. Finite-step gradient descent often tracks this picture when the learning rate is small, but the equation should not be quoted as an exact description of arbitrary optimizers, large learning rates, or arbitrary initialization.
Finally, a deep nonlinear network is not a deep linear network with a few harmless decorations. ReLU and attention alter the function class and create new geometry. The linear case is a controlled demonstration that parameterization alone can generate rich optimization bias. It is a clue about deep learning, not a complete theory of it.
Back to the useless hundred-layer network
So was the network pointless?
As a function approximator, yes. If I already know the trained product \(W\), replacing one hundred layers with that single matrix preserves every prediction. Inference has no reason to keep the factorization.
But training is not only about the final algebraic expression. Before the weights are known, the factorization determines the route by which they are found. A shallow matrix follows the gradient directly. A deep product wraps that gradient in its own evolving geometry. Large singular modes are accelerated, small ones are delayed, and among many perfect fits the network can select a spectrally simpler completion.
The model is linear in \(x\), nonlinear in time, and biased by depth.
That is the sentence I was missing when I first collapsed all the matrices and decided there was nothing left to study. I had proved that the architectures compute the same functions. I had not proved that gradient descent sees those functions in the same way.
One hundred layers can equal one matrix after training. During training, they are an entirely different machine.
References and links
- Baldi, P., & Hornik, K. (1989). Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima. Neural Networks, 2(1), 53-58. Paper link.
- Saxe, A. M., McClelland, J. L., & Ganguli, S. (2014). Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks. International Conference on Learning Representations. Paper link.
- Kawaguchi, K. (2016). Deep Learning Without Poor Local Minima. Advances in Neural Information Processing Systems. Paper link.
- Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., & Srebro, N. (2017). Implicit Regularization in Matrix Factorization. Advances in Neural Information Processing Systems. Paper link.
- Arora, S., Cohen, N., Golowich, N., & Hu, W. (2019). A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. International Conference on Learning Representations. Paper link.
- Arora, S., Cohen, N., Hu, W., & Luo, Y. (2019). Implicit Regularization in Deep Matrix Factorization. Advances in Neural Information Processing Systems. Paper link.
- Razin, N., & Cohen, N. (2020). Implicit Regularization in Deep Learning May Not Be Explainable by Norms. Advances in Neural Information Processing Systems. Paper link.
- Bah, B., Rauhut, H., Terstiege, U., & Westdickenberg, M. (2022). Learning Deep Linear Neural Networks: Riemannian Gradient Flows and Convergence to Global Minimizers. Information and Inference: A Journal of the IMA, 11(1), 307-353. Paper link.
- Marion, P., & Chizat, L. (2024). Deep Linear Networks for Regression Are Implicitly Regularized Towards Flat Minima. Advances in Neural Information Processing Systems. Paper link.