Build your own GPT-2
The MLP block·8 min

gelu_new

0 / 1
# grade this stage (from your course directory)
npx theinference test --stage gelu --watch

The MLP's nonlinearity is gelu_new. Build it first — it's elementwise, so the same tiny function works on a [4, 64] test fixture and on the [T, 3072] scratchpad in the real MLP.

GPT-2 gelu_new (tanh approximation)
gelu_new(x) = 0.5 · x · (1 + tanh( √(2/π) · (x + 0.044715 · x³) ))
gelu_new at a few values (check your implementation against these)
x        gelu_new(x)
-3.0     -0.00364
-1.0     -0.15881
 0.0      0.00000
 0.5      0.34571
 1.0      0.84119
 2.0      1.95460

Notice GELU lets small negatives leak through — gelu_new(−1) ≈ −0.159, not 0 like ReLU — and it's smooth everywhere. That gentle negative dip (minimum ≈ −0.17 near x ≈ −0.75) is exactly what ReLU's hard hinge can't do.

What to build

Your task

Implement runGelu

Implement runGelu(x): apply gelu_new to every element of x (any shape in, same shape out). Use the tanh approximation above — not the exact erf-based GELU.

tests/adapters.ts
export function runGelu(x: number[][]): number[][] {
  // elementwise gelu_new; [r, c] -> [r, c]
}
tanh approximation, exact constants — not the erf GELU

Use these constants exactly: √(2/π) = 0.7978845608…, 0.044715, and x³ computed as x*x*x.

The erf-based exact GELU (x · Φ(x)) looks like the same function but differs by up to ~1e-3. That's enough to fail this step's atol 1e-5, and it compounds through 12 layers to blow the accumulated atol 1e-3 at the final logits.

This isn't an approximation you're free to "improve": GPT-2 was trained with gelu_new, so the tanh form is the ground truth here.

It really is just a map

gelu_new is elementwise, so the whole thing is x.map(row => row.map(geluNew)). No shape logic, no special-casing — it works on [4, 64] (the fixture) and [T, 3072] (the MLP) without changes.

How it's graded

Your output (same shape as the input) is compared with allclose, atol 1e-5.

The feed-forward networkGELU vs ReLU