one neuron
A hidden neuron is wired to 96 fixed pixels S. Its sum is the usual weighted sum:
z = Σ wᵢ·xᵢ + b, over i in S
The page doesn't send z. It sends Jev the list of products wᵢxᵢ and b, and asks: do these add up to
more than zero? Jev's probability of yes, p, is the neuron's output. The page also works out z itself, so it
can tell when Jev gets it wrong: Jev says yes (p ≥ 0.5) when z isn't above zero, or no when it is. Those
errors are the pink tiles.
the network
784 pixels feed 64 hidden neurons, 96 pixels each, which feed 10 outputs, one per digit. That's 74 Jev calls a
digit: the 64 hidden ones at once, then the 10 outputs. The highest output wins:
ŷ = argmaxₖ pₖ
The other network in the menu is dense: every hidden neuron sees all 784 pixels (784 → 32 → 10, 42 calls).
training
Jev has no gradient, so training needs a stand-in for how p changes with z. We measured one: we asked Jev
thousands of sums whose true totals we knew and recorded how often it said yes. Call that curve f, so
f(z) is roughly the chance Jev says yes to a list that adds up to z. Training runs Jev's real answer
p forward and the slope of f backward (a straight-through estimator):
∂L/∂z ≈ ∂L/∂p · f′(z)
L is the training loss. f′(z) is large when z is near zero, where Jev's answer is still
deciding, and small far from it, so the weights get pushed hardest on the sums Jev is least sure about.
the comparison
The perfect-math twin starts from the same weights but trains on an exact step neuron, so it never sees Jev's
mistakes. Run on Jev, over 1,000 test digits:
- trained through Jev
- 85.3%
- trained with perfect math
- 65.8%
The 19.5-point gap is what the first network's weights learned about Jev. They're tuned to its mistakes and
biases, like its habit of saying yes to sums that should be no. The twin's weights assume a neuron that adds
perfectly, and Jev isn't one.