The Tiny Functions Eating Your Job

I got inspired from Vivek's tweet and thought, lemme write this.

It is a very funny amount of blame to place on functions that are usually one line of code. A neuron collects some numbers, multiplies them by weights, adds them, and then sends the result through something like \(\max(0,z)\) or \(z\Phi(z)\). The matrix multiplication may contain millions of learned numbers. The activation function contains none. Yet remove that tiny function and a hundred layers can collapse into one linear map. Change its shape and the same architecture can become easy to train, slow to train, sparse, smooth, unstable, or unexpectedly good at preserving a signal.

I have already written about why a network needs nonlinearity. This essay asks the question underneath that answer: once we agree to bend the line, which bend should we choose? ReLU chops the negative half away. Sigmoid compresses the whole real line into a probability-shaped interval. GELU and SiLU let a small negative signal leak through, then become almost linear on the positive side. A sine never settles down at all.

The differences look cosmetic on a plot. They are not. An activation has an anatomy, and every part of that anatomy enters training.

First, where does an activation function live?

Take one neuron. It receives a vector \(x\), keeps a vector of learned weights \(w\), and owns a learned bias \(b\). It first computes

The first number, \(z\), is the pre-activation: the weighted evidence collected before the neuron decides what to pass on. The second number, \(a\), is the activation. The symbol \(\phi\) is the rule between them. If \(\phi(z)=z\), the neuron simply forwards its weighted sum. If \(\phi(z)=\max(0,z)\), negative evidence becomes exactly zero. If \(\phi(z)=1/(1+e^{-z})\), every possible input is squeezed between zero and one.

That sounds like a scalar detail because it is applied one number at a time. Across a hidden layer, however, the same rule acts on every coordinate:

The matrix \(W_{\ell}\) mixes information. The activation \(\phi\) decides which parts of that mixture survive and in what form. The next matrix mixes the edited result. A deep network is this alternation repeated: mix, bend, mix, bend.

“Nonlinear” only means the graph is not one straight line. It does not tell us whether the graph is bounded, whether it has a sharp corner, whether it kills negative values, whether it is smooth enough to differentiate twice, or whether it makes large inputs indistinguishable. Those details are the skeleton.

The skeleton of an activation function

I wanted a way to inspect any activation before learning its name, so I gave it seven bones. This is an actual rotatable skeleton because the metaphor became too literal to stop. Drag it, use the arrow keys, and click a labelled part. Then switch on the brain.

The shoulders are the domain. The domain is the set of inputs the function accepts. Most hidden activations accept every real number, written \(\mathbb R\). A pre-activation of \(-10^6\), \(0\), or \(3.7\) is all legal. The interesting question is what the function does with each one.

The skull is the range. The range is the set of outputs that can leave. Sigmoid has range \((0,1)\); it can approach zero or one forever but never reach them at a finite input. Tanh has range \((-1,1)\). ReLU has range \([0,\infty)\), so it can produce an exact zero and arbitrarily large positive values. Range is not decoration. It determines whether a neuron can represent a negative signal, whether outputs are bounded, and whether the next layer receives a permanent positive offset.

The spine is the curve. This is \(\phi(z)\) itself: the forward rule. A straight spine gives a linear activation. A kink, an S-curve, a soft gate, or a periodic wave changes the function class of the whole network.

The elbows are the slope. The derivative \(\phi'(z)\) measures how much the output changes when the input moves a tiny amount.A derivative, without ceremony. If increasing \(z\) by \(0.001\) increases \(\phi(z)\) by roughly \(0.0007\), the local slope is about \(0.7\). Backpropagation repeatedly multiplies by these local slopes. A slope of one passes a small change through unchanged. A slope near zero makes the neuron locally deaf. A negative slope reverses the direction of the change.

The knees are curvature. The second derivative \(\phi''(z)\) measures how the slope itself changes. Zero curvature means locally straight. Positive curvature bends upward; negative curvature bends downward. Curvature matters because it changes how abruptly learning dynamics vary as a neuron moves through its input space. ReLU has zero second derivative away from its single kink. GELU spreads curvature across a whole neighbourhood.

The feet are the limits. We ask what happens as \(z\to\infty\) and \(z\to-\infty\). If the curve approaches a flat ceiling or floor, it saturates.Saturation. A function saturates when increasingly extreme inputs produce almost the same output. The slope then approaches zero, so the forward signal is bounded but the backward signal weakens. Sigmoid saturates at both ends. ReLU saturates on its entire negative half but stays linear on the positive half. SiLU approaches zero from below on the left and behaves like the identity on the right.

The hands are the parameters. Some activations contain knobs. Leaky ReLU has a negative slope \(\alpha\). PReLU learns that slope. Softplus has a sharpness \(\beta\). Swish can use \(z\sigma(\beta z)\). A parameter lets the curve adapt, but it also adds another quantity whose initialization and gradients need care.

Now add the brain

The skeleton describes one function at one input. The brain is what happens when millions of copies communicate through a network. I define that brain through four questions.

What information moves forward? The value \(\phi(z)\) becomes data for the next layer. ReLU makes the representation sparse because every negative input becomes exactly zero. Tanh preserves sign but compresses magnitude. GELU reduces small negative values rather than amputating them.

What credit moves backward? If a loss \(L\) depends on \(a=\phi(z)\), the chain rule gives

The gradient arriving from later layers is multiplied by the activation's slope. Through \(D\) scalar layers, the activation contribution looks like \(\prod_{\ell=1}^{D}\phi'(z_\ell)\); the weight matrices contribute their own factors too. If twenty sigmoid units all sit at zero and the intervening scalar weights equal one, the product is \((1/4)^{20}\approx9.1\times10^{-13}\). That is not a metaphor for a small gradient. It is nearly a trillionth of the original signal.

What statistics survive? A layer receives a distribution of inputs, not one number. The activation changes its mean and spread. Feed a symmetric bell-shaped distribution through ReLU and half the values become zero while the other half remain positive. The result is no longer centred at zero. Initialization schemes compensate for this transformation so that signals do not shrink or explode with depth.

What geometry does a population create? Two similar inputs produce correlated pre-activations across neurons. The activation changes that correlation. In a sufficiently wide random layer, the choice of \(\phi\) becomes an exact rule for how angles and similarities are warped. We will return to that after meeting the individual functions.

The first activation was a switch

The cleanest ancestor is the binary step:

A weighted sum crosses a threshold and the unit fires. This is the logic behind the perceptron associated with Rosenblatt's early neural model.[2] Its range contains one bit. Its decision boundary is the plane \(w^{\top}x+b=0\). On one side the answer is zero; on the other it is one.

The step has a fatal relationship with ordinary backpropagation. Its derivative is zero everywhere except at the jump, where the derivative does not exist. Move the weights a tiny amount without crossing the threshold and the output does not change at all. A gradient therefore sees no direction to move. The historical perceptron used its own mistake-driven update rule; it did not train a deep stack by differentiating through the step.

This creates the first design tension. A hard decision is useful in the forward pass, but a smooth response is useful in the backward pass. Sigmoid was the soft switch.

Sigmoid: a probability-shaped gate

The logistic sigmoid is

It turns the entire real line into \((0,1)\). At \(z=0\), the output is \(1/2\). Negating the input complements the output:

This is why sigmoid feels like a soft yes-or-no gate. Large positive evidence approaches one, large negative evidence approaches zero, and zero evidence lands exactly in the middle. If \(z\) is a log-odds value, \(\sigma(z)\) converts it into a probability.Odds and log-odds. A probability \(p\) has odds \(p/(1-p)\). Taking the logarithm gives the log-odds. Solving \(z=\log(p/(1-p))\) for \(p\) produces \(p=\sigma(z)\).

Its derivative contains the function itself:

Let \(p=\sigma(z)\). Since \(0<p<1\), the derivative is \(p(1-p)\), a downward-facing quadratic with its maximum at \(p=1/2\). Therefore

The best possible slope is only \(0.25\), and it occurs at the origin. At \(z=5\), the derivative is about \(0.00665\). At \(z=10\), it is about \(0.0000454\). A confident sigmoid is also a stubborn sigmoid: once it sits near zero or one, a gradient has little power to change the evidence that put it there.

There is a second problem. If the input distribution is symmetric around zero, then \(\mathbb E[\sigma(Z)]=1/2\). The next layer receives positive-centred values. Here \(\mathbb E\) means the average we would obtain across repeated draws. This offset can push later sigmoids toward saturation; Glorot and Bengio connected that behaviour to the plateaus seen in deep sigmoid networks.[3]

Sigmoid did not become useless. It moved to jobs where \((0,1)\) is the point: binary output probabilities and learned gates in LSTMs, GRUs, and gated blocks. A hidden feature may need to preserve magnitude; a gate only needs to decide how much of another feature should pass.

Tanh: sigmoid, recentered and sharpened

The hyperbolic tangent looks like sigmoid moved down and stretched:

That identity is more than visual trivia. It proves that tanh has range \((-1,1)\), crosses the origin, and changes sign when its input changes sign: \(\tanh(-z)=-\tanh(z)\). A function with that last property is called odd. For any input distribution symmetric around zero, an odd activation has zero mean. Tanh therefore removes sigmoid's built-in \(1/2\) offset.

Its derivative is

The maximum slope is one at the origin, four times sigmoid's maximum. Near zero, tanh behaves almost like the identity: \(\tanh(z)\approx z\). That gives small signals a clean path. Large signals still saturate at \(\pm 1\), so the vanishing-gradient problem remains in both tails.

Tanh is still natural when sign and bounded state matter. Recurrent networks traditionally used it for candidate states because a memory update can be positive or negative without becoming unbounded. It is also a useful output when the target itself lives in a known interval. The important distinction is between choosing a bounded range because the task needs one and imposing it on every hidden layer merely because the curve looks tidy.

ReLU: half a line changed deep learning

The rectified linear unit is almost aggressively simple:

Its derivative is zero for \(z<0\) and one for \(z>0\). At exactly zero the two slopes disagree, so the derivative is undefined. Software chooses a conventional value, commonly zero, and training continues. A continuous input distribution hits exactly zero with probability zero, so the single ambiguous point is rarely the practical problem people imagine.“Almost everywhere.” Mathematicians use this phrase when exceptions occupy zero volume. ReLU is differentiable almost everywhere because one exceptional point has no length on the real line.

ReLU's positive half does not saturate. A positive signal sees slope one no matter how large it becomes. That created a much cleaner gradient path than sigmoid or tanh and helped deep supervised networks train without layer-wise pretraining.[4]

ReLU is also positively homogeneous:

This means positive scaling commutes with the activation. Multiply the incoming weights of a ReLU neuron by \(c\) and divide its outgoing weights by \(c\); the whole network can compute exactly the same function. That symmetry helps explain why parameter size and function size are not the same thing in ReLU networks.

Its second derivative is zero everywhere except the kink. Yet a ReLU network is globally nonlinear. Each neuron introduces a boundary \(w^{\top}x+b=0\). On any region where the active and inactive neurons stay fixed, the network is affine. Cross a boundary and a different affine map takes over. The nonlinearity lives in how many linear regions are stitched together, not in continuous curvature inside each region.

A Gaussian goes in; half a Gaussian comes out

Let \(Z\sim\mathcal N(0,1)\), meaning \(Z\) follows a standard bell curve with mean zero and variance one. Set \(Y=\operatorname{ReLU}(Z)\). Symmetry immediately tells us that half the probability becomes exactly zero. The surviving positive half gives

The second moment \(\mathbb E[Y^2]\) is the average squared size. It halves after ReLU. That fact leads to He initialization. If a pre-activation sums \(n\) independent inputs of equal second moment, choosing weight variance roughly \(2/n\) compensates for the half that ReLU removes.[5]Second moment versus variance. Variance subtracts the squared mean: \(\operatorname{Var}(Y)=\mathbb E[Y^2]-\mathbb E[Y]^2\). Initialization derivations often track the second moment because weighted sums with zero-mean weights become especially simple.

The same hard zero creates ReLU's famous failure mode. If a neuron's pre-activation becomes negative for every training example, its output and local derivative are always zero. No gradient reaches its incoming weights through that route. The unit is “dead.” A large learning rate or badly placed bias can push a neuron into this state.

The negative half fights back

Many ReLU variants are answers to one question: must negative evidence become exactly zero?

SELU's “self-normalizing” name needs its assumptions attached.A fixed point. A transformation has a fixed point when applying it leaves a value unchanged. Here the transformation takes one layer's mean and variance to the next layer's mean and variance. Stability means nearby values move back toward the fixed point. The proof uses a particular initialization, independent activations, sufficiently wide feed-forward layers, and alpha dropout rather than ordinary dropout. SELU inserted casually into an architecture with normalization, residual paths, or another initialization does not inherit a magical guarantee.

Softplus: a ReLU with no corner

Softplus replaces the hard maximum with a smooth one:

The parameter \(\beta>0\) controls sharpness. As \(\beta\to\infty\), Softplus approaches ReLU. The approximation error is exact and pleasantly small:

The largest gap occurs at the origin. Softplus always sits slightly above ReLU and never outputs an exact zero. Its first two derivatives are

Sigmoid, which struggled as a hidden activation, returns as Softplus's derivative. This is a lovely recurring pattern: an old activation can remain useful as the gate controlling the slope of a newer one. Softplus is also the two-number log-sum-exp, a smooth approximation to \(\max(0,z)\).

GELU, SiLU, and Mish: the soft gates

ReLU makes a deterministic decision: keep \(z\) if it is positive, otherwise replace it with zero. Modern smooth activations make that decision gradual. Small negative inputs are reduced rather than erased; large positive inputs pass almost unchanged.

GELU: weight by a Gaussian probability

The Gaussian error linear unit is

where \(\Phi(z)\) is the probability that a standard normal random variable is at most \(z\). At \(z=0\), the gate \(\Phi(z)\) is \(1/2\). For a large positive input it approaches one; for a large negative input it approaches zero. GELU weights the input by how probable it is under a Gaussian threshold instead of using ReLU's hard sign test.[8]

If \(\varphi(z)=e^{-z^2/2}/\sqrt{2\pi}\) is the standard normal density, then

The curvature changes sign at \(z=\pm\sqrt2\). GELU is also slightly non-monotonic: around \(z\approx-0.752\), it reaches a small minimum of about \(-0.170\), then rises toward zero as \(z\to-\infty\). More negative input can therefore produce a less negative output in the far left tail. That sounds strange only if an activation is expected to preserve ordering. Neural networks do not require that.

SiLU or Swish: let sigmoid gate its own input

The sigmoid linear unit is

The name Swish is often used for the wider family \(z\sigma(\beta z)\), which was rediscovered through an automated search over activation formulas.[9] Its derivative is

SiLU reaches its minimum near \(z=-1.278\), with value about \(-0.278\). Like GELU, it can have a negative local slope and a positive slope slightly above one. The two curves are remarkably close because the logistic CDF \(\sigma(1.702z)\) approximates the Gaussian CDF \(\Phi(z)\). This gives the useful approximation

They arrived through different stories—Gaussian weighting and searched self-gating—but much of their shape is the same story told with two probability distributions.

Mish: gate with tanh of Softplus

Mish nests two familiar smooth functions:

Let \(u=\operatorname{softplus}(z)\) and \(t=\tanh(u)\). Then the derivative is still readable:

Mish approaches zero from below on the far left and the identity on the far right. Its minimum is about \(-0.309\). The construction is smooth everywhere, but smoothness alone does not prove better optimization or generalization. Mish, GELU, and SiLU make empirical bets about whether a gentle negative basin and a soft transition are useful for a particular architecture.[10]

One activation becomes a geometry in a crowd

So far we have followed one scalar. A wide layer reveals a stranger fact: the activation decides how the network measures similarity.

Take two unit-length input vectors \(x\) and \(x'\), separated by an angle \(\theta\). Draw a random Gaussian weight vector \(w\), and form two pre-activations \(u=w^{\top}x\) and \(v=w^{\top}x'\). Their correlation is \(\cos\theta\). After ReLU, the expected product has a closed form:

This is the first-order arc-cosine kernel.[13] A kernel is simply a rule that returns a similarity between two inputs. An infinitely wide random ReLU layer computes this particular rule. The one-dimensional kink has become high-dimensional geometry.

Stacking random layers repeatedly applies a correlation map of this kind. Depending on the weight scale, bias scale, and activation, two distinct signals can become indistinguishable, remain usefully separated, or diverge chaotically. Mean-field analyses call the narrow trainable boundary the “edge of chaos”: correlations propagate without collapsing too quickly and gradients avoid immediate explosion or disappearance.[14] This is the brain at population scale. The shape of one neuron decides how information travels through depth.

Other activations build other geometries. A sine activation, \(\phi(z)=\sin(\omega z)\), is periodic and has derivative \(\omega\cos(\omega z)\). It can express fine spatial frequencies that ReLU networks learn less naturally, which is why SIREN networks use it for continuous representations of images, audio, and physical fields.[15] There is no universally best curve because different curves make different structures cheap.

Some “activations” act on more than one number

Most functions above are pointwise: each coordinate is transformed independently. Two important modern cases break that simple picture.

Softmax is a conversation among coordinates

For a vector \(z\in\mathbb R^K\), softmax produces

Every output is positive and the outputs sum to one, so \(p\) lies on the probability simplex. Increasing one logit changes every probability because they share a denominator. Its derivative is therefore a matrix, called a Jacobian:

A Jacobian is the vector version of a derivative: entry \((i,j)\) tells us how output \(i\) responds to input \(j\). The matrix obeys \(J\mathbf1=0\). Adding the same constant to every logit changes nothing, because the shared factor cancels. Softmax deliberately forgets one direction: the all-coordinates-together direction.

SwiGLU gives the activation learned context

A gated linear unit forms two learned projections. One carries content; the other decides how much content passes. A common Transformer variant is

followed by another projection. The symbol \(\odot\) means coordinate-by-coordinate multiplication. Unlike \(\operatorname{SiLU}(z)\), which lets a scalar gate itself, SwiGLU learns a separate gate from the whole input vector. The activation has grown a small decision-making circuit. GLU variants have improved Transformer feed-forward blocks in controlled comparisons and now appear across large language models.[11][12]

So which activation should I use?

The output layer should follow the meaning of the prediction. Binary probability suggests a sigmoid, usually fused numerically with binary cross-entropy. Mutually exclusive classes suggest softmax, usually fused with multiclass cross-entropy. An unrestricted regression target suggests the identity. A strictly positive quantity can use Softplus. A bounded physical control may justify tanh.

For hidden layers, the architecture matters more than a universal ranking. ReLU remains a strong, cheap baseline for convolutional networks and ordinary MLPs. Leaky ReLU is a simple response when dead units are visible. GELU is conventional in many Transformers. SiLU is common in modern vision and language systems. SwiGLU is more than a pointwise replacement and changes parameter counts, so comparisons should adjust hidden width rather than dropping it into a ReLU block unchanged.

Three checks matter more than memorising that table. First, inspect the distribution of pre-activations: a beautiful central slope is irrelevant if every neuron lives in a saturated tail. Second, inspect activation and gradient statistics by layer: means, second moments, dead fractions, and gradient norms reveal whether the chosen function and initialization cooperate. Third, compare under a fixed compute and parameter budget. A slightly better curve in one paper is evidence from one training system, not a law of nature.

What this dissection does not say

It does not say smooth functions are automatically better. ReLU's corner is compatible with gradient training, and its piecewise-linear structure is often a computational gift. Smoothness becomes essential only when the application needs smooth higher derivatives, as some physics-informed or implicit-representation problems do.

It does not say sigmoid always causes vanishing gradients or ReLU always prevents them. The full Jacobian contains weight matrices, normalization, residual connections, attention, and the loss. A ReLU network with badly scaled weights can explode or die; a carefully designed gated recurrent network can use sigmoids successfully for thousands of steps.

It does not say universal approximation chooses the winner. Under standard assumptions, a broad class of non-polynomial activations can approximate continuous functions with enough width.[16] Existence is a weak promise. It does not tell us how many neurons are required, whether gradient descent can find them, how stable the computation is, or how well the result generalizes.

And it does not say the activation works alone. Initialization is chosen partly around the activation. Normalization changes the inputs it sees. Residual connections create alternative gradient paths. Gating changes the block. Hardware changes the cost. The right unit of comparison is usually a training recipe, not a curve floating by itself.

The smallest part was shaping the whole machine

I began with Vivek blaming activation functions for eating jobs. The serious version of the joke is surprisingly defensible.

A neural network stores most of its learned information in matrices, but the activation decides what those matrices are allowed to build together. Its range controls what can be represented in one coordinate. Its slope controls what can learn. Its curvature controls how that learning changes across input space. Its tails decide whether extreme evidence is preserved, clipped, or forgotten. Repeated across width, it defines a similarity geometry. Repeated across depth, it helps decide whether signals remain distinct and whether gradients survive the trip home.

The historical movement now looks less like a list of fashionable names. The step function made a decision but offered calculus nothing. Sigmoid softened the decision but compressed gradients everywhere. Tanh recentered the signal. ReLU removed the positive ceiling and accepted a hard kink. Leaky ReLU, ELU, and SELU renegotiated the negative half. Softplus rounded the kink. GELU and SiLU turned the hard gate into a probability-shaped weighting. SwiGLU gave the gate its own learned view of the input.

Every new function noticed one habit of the previous one and changed the anatomy.

So the activation is not the decorative squiggle between two large matrix multiplications. It is the joint that lets the skeleton move and the nervous system that tells the earlier layers what hurt. The matrices contain the strength. The activation decides how the body uses it.