AIM-02-002-P · Primer — The Multilayer Perceptron

The idea in one sentence

A multilayer perceptron (MLP) is a stack of dense layers — each one a linear weighted sum plus bias, followed by a nonlinearity — that transforms raw input features into progressively more useful representations, ending in a prediction.

Why it exists

A single perceptron can only draw one straight line (or hyperplane) through feature space, so it can only correctly separate data that is linearly separable — it fails on something as simple as XOR, where the two classes form an X pattern no single line can split. The fix is to stack layers: each dense layer bends and re-projects the space, and a nonlinear activation between layers is what actually allows this bending (without it, stacked linear layers collapse back into one linear layer, algebraically). Think of it like sorting mail with a single straight divider in a room versus being handed a sequence of moveable partitions — after a few strategic folds of the room, two piles that looked hopelessly tangled from the doorway become two piles as if separated by a straight line at the end. That's what an MLP's hidden layers do to data: fold the space, layer by layer, until the classes become separable.

How it works

The equation that matters for every layer \(l\) is:

\[a^{(l)} = f\left(W^{(l)} a^{(l-1)} + b^{(l)}\right)\]

where \(a^{(0)} = x\) (the raw input), \(W^{(l)}\) is that layer's weight matrix (rows = output units, columns = inputs from the previous layer), \(b^{(l)}\) is its bias vector, and \(f\) is a nonlinear activation (commonly ReLU: \(f(z) = \max(0, z)\)). The term inside \(f\) — the pre-activation \(z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}\) — is pure linear algebra, computed before any nonlinearity is applied.

Concretely: say \(x = [1.0,\ 0.5]\) feeds a hidden layer with 2 units, \(W_1 = \begin{bmatrix} 0.3 & -0.2 \\ 0.1 & 0.4 \end{bmatrix}\) and \(b_1 = [0.1,\ -0.05]\). Unit 0's pre-activation is \(z^{(1)}_0 = 0.3(1.0) + (-0.2)(0.5) + 0.1 = 0.3 - 0.1 + 0.1 = 0.3\). Unit 1's is \(z^{(1)}_1 = 0.1(1.0) + 0.4(0.5) - 0.05 = 0.1 + 0.2 - 0.05 = 0.25\). Applying ReLU to both (they're already positive) gives \(a_1 = [0.3,\ 0.25]\) — the hidden representation. That vector then feeds the output layer: with \(W_2 = [0.5,\ -0.6]\) and \(b_2 = 0.2\), \(z_2 = 0.5(0.3) + (-0.6)(0.25) + 0.2 = 0.15 - 0.15 + 0.2 = 0.2\), which is the network's final output (or, for binary classification, gets passed through a sigmoid: \(\sigma(0.2) \approx 0.5498\)).

Every dense layer, no matter how deep the network, repeats this exact two-step recipe — linear combination (dot products + bias) then a nonlinear squash — with each layer's output becoming the next layer's input. In matrix form for a full layer at once, \(z = Wx + b\) computes every unit's dot product simultaneously; the lab below implements exactly this step, one row (one output unit) at a time.

See it — Forward pass through an MLP: each dense layer's linear step feeds a nonlinearity, whose output becomes the next layer's input.

flowchart TD
    X["Input x<br/>(features)"] --> L1["Dense Layer 1<br/>z = W1x + b1"]
    L1 --> A1["ReLU<br/>a = max(0, z)"]
    A1 --> L2["Dense Layer 2<br/>z = W2a + b2"]
    L2 --> A2["ReLU<br/>a = max(0, z)"]
    A2 --> L3["Output Layer<br/>z = W3a + b3"]
    L3 --> Y["Prediction y"]

The one gotcha

The near-universal mistake: stacking dense layers without a nonlinearity in between. If you compute \(a_2 = W_2(W_1 x + b_1) + b_2\) with no activation applied to the hidden layer, that whole expression is still just \(Ax + c\) for some combined matrix \(A\) and vector \(c\) — a two-layer network with no nonlinearity is mathematically identical to a single linear layer, no matter how many layers or units you add. The nonlinearity (ReLU, sigmoid, tanh) applied to every hidden pre-activation is what actually buys you extra representational power; it's not optional plumbing, it's the entire point of "deep."

The one-line takeaway

Depth without nonlinearity collapses to one linear layer.

Where you'll meet it

PyTorch (torch.nn.Linear layers chained in nn.Sequential, each followed by nn.ReLU); TensorFlow/Keras (tf.keras.layers.Dense stacked in a Sequential or Functional model); scikit-learn's MLPClassifier/MLPRegressor, which run this exact forward pass internally.

Named concepts: mlp forward