The perceptron

The perceptron (1958) is the simplest possible «neuron»:

1. Compute z = w·x + b.
2. Answer 1 if z > 0, otherwise 0.

The learning rule is just as simple: for every example, if the answer was wrong, adjust the weights in the right direction:

w ← w + η(y − ŷ)x, b ← b + η(y − ŷ)

Did it guess 0 when the answer was 1? Increase the weights where x was large. Did it guess 1 when the answer was 0? Decrease them.

The perceptron convergence theorem: if the data can be separated by a straight line the algorithm is guaranteed to find such a line in a finite number of steps.

import numpy as np

def perceptron(X, y, eta=0.1, epochs=20):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(epochs):
        errors = 0
        for xi, yi in zip(X, y):
            pred = 1 if xi @ w + b > 0 else 0
            if pred != yi:
                w += eta * (yi - pred) * xi
                b += eta * (yi - pred)
                errors += 1
        if errors == 0:
            break
    return w, b, errors

X = np.array([[0,0],[0,1],[1,0],[1,1]])
print(perceptron(X, np.array([0,0,0,1]))[2])   # AND  → 0 errors, solved
print(perceptron(X, np.array([0,1,1,1]))[2])   # OR   → 0 errors, solved
print(perceptron(X, np.array([0,1,1,0]))[2])   # XOR  → errors remain, never solved

Why XOR does not work: draw the points. (0,0) and (1,1) should give 0; (0,1) and (1,0) should give 1. They lie diagonally — no straight line can separate them.

Minsky and Papert showed this in 1969, and interest in neural networks collapsed for more than a decade. The solution — a hidden layer — lets the network bend the decision boundary. With two neurons in a hidden layer and a non-linear activation XOR is solved immediately. That is the whole motive for the «deep» in deep learning.