11. Separating Classes with Dividing Lines
By Bernd Klein. Last modified: 20 Aug 2026.
In this chapter, we will develop a perceptron – one of the simplest building blocks of neural networks. It will distinguish between two classes whose points can be separated by a straight line in a two-dimensional feature space.
Linear Separation
Before we program the perceptron, let us first look at a simple geometric concept. We want to find straight lines that can separate two points, or more generally the points belonging to two different classes, in the plane.
At first, we will only consider lines that pass through the origin of the coordinate system. We will deal with the general case, that is, arbitrary lines, later in the tutorial.
Imagine that we have two attributes describing properties of an edible object such as a fruit, for example sweetness and sourness.
We can represent these attributes as points in a two-dimensional space. The x-axis represents sweetness and the y-axis sourness. Suppose we have two fruits: an orange at $(3.5,1.8)$ and a lemon at $(1.1,3.9)$.
We can now define dividing lines that separate points that are more lemon-like from points that are more orange-like.
The following diagram shows one lemon and one orange. The green line separates the two points. For the moment, we assume that all other lemons lie above this line and all oranges below it.
The green line is defined by
where:
- $m$ is the slope of the line, and
- $x$ is the independent variable.
Let $P=(p_1,p_2)$ be a point on this line. Then
For $p_1 \neq 0$, we obtain
An arbitrary point $P'=(p'_1,p'_2)$ therefore lies on the line exactly when
or equivalently
The following Python program draws a diagram of the situation described above:
import matplotlib.pyplot as plt
import numpy as np
X = np.arange(0, 7)
fig, ax = plt.subplots()
orange = (3.5, 1.8)
lemon = (1.1, 3.9)
ax.plot(*orange, "o", color="darkorange", markersize=15)
ax.plot(*lemon, "o", color="gold", markersize=15)
point_on_line = (4, 4.5)
m = point_on_line[1] / point_on_line[0]
ax.plot(X, m * X, "g-", linewidth=3)
plt.show()
It is clear that a point $A=(a_1,a_2)$ does not lie on the line if
But we want to know more: we want to determine whether an arbitrary point lies above or below the line.
If a point $B=(b_1,b_2)$ lies below the line, there must be a $\delta_B>0$ such that the point
lies on the line. Therefore
and hence
The reasoning for a point above the line is analogous. If $A=(a_1,a_2)$ lies above the line, there must be a $\delta_A>0$ such that $(a_1,a_2-\delta_A)$ lies on the line. Thus
which gives
In summary, a point $P=(p_1,p_2)$ lies
- below the line if $m\cdot p_1-p_2>0$,
- on the line if $m\cdot p_1-p_2=0$,
- above the line if $m\cdot p_1-p_2<0$.
We can now verify this for our fruits. The lemon has the coordinates $(1.1,3.9)$ and the orange $(3.5,1.8)$. The point on the line used to define our dividing line has the coordinates $(4,4.5)$. Therefore
A positive result means that the point lies below the line; a negative result means that it lies above the line.
lemon = (1.1, 3.9)
orange = (3.5, 1.8)
m = 4.5 / 4
# Orange: a positive value is expected.
print(orange[0] * m - orange[1])
# Lemon: a negative value is expected.
print(lemon[0] * m - lemon[1])
OUTPUT:
2.1375 -2.6624999999999996
We did not determine the green line by means of a mathematical procedure. We simply chose it by looking at the diagram. We could just as well have used many other lines.
The following Python program draws a collection of lines through the origin. The red lines are unsuitable for separating the two fruits because the lemon and the orange lie on the same side. The green lines separate the two points.
However, even the green lines might not be particularly useful once we have many fruits. Some lemons may be relatively sweet and some oranges may be rather sour. We therefore need a systematic method for finding a suitable dividing line.
import numpy as np
import matplotlib.pyplot as plt
orange = (3.5, 1.8)
lemon = (1.1, 3.9)
fig, ax = plt.subplots()
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
x_min, x_max = -1, 7
y_min, y_max = -1, 8
ax.set_xlim(x_min, x_max)
ax.set_ylim(y_min, y_max)
X = np.arange(x_min, x_max, 0.1)
for m in np.arange(0.1, 4.1, 0.15):
Y = m * X
orange_position = m * orange[0] - orange[1]
lemon_position = m * lemon[0] - lemon[1]
# Different signs mean that the fruits are on different sides.
if orange_position * lemon_position < 0:
ax.plot(X, Y, "g-", linewidth=0.8, alpha=0.9)
else:
ax.plot(X, Y, "r-", linewidth=0.8, alpha=0.9)
ax.plot(*orange, "o", color="darkorange", markersize=10)
ax.plot(*lemon, "o", color="gold", markersize=10)
plt.show()
In principle, we have already performed a classification with our dividing lines, even if one would hardly use that term for such a simple example.
It is easy to imagine that instead of one orange and one lemon we have many fruits with different sweetness and sourness values. We then have two classes: an orange class and a lemon class.
Let us generate such data with make_blobs from sklearn.datasets. The function creates clusters around specified centres; cluster_std controls their spread and random_state makes the example reproducible.
We use the convention 0 = orange and 1 = lemon throughout the rest of this tutorial.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
# 0 = orange, 1 = lemon
fruit_data, fruit_labels = make_blobs(
n_samples=200,
centers=[(5, 2), (2, 5)],
cluster_std=[0.6, 0.6],
random_state=42
)
oranges = fruit_data[fruit_labels == 0]
lemons = fruit_data[fruit_labels == 1]
oranges_x, oranges_y = oranges[:, 0], oranges[:, 1]
lemons_x, lemons_y = lemons[:, 0], lemons[:, 1]
X = np.linspace(0, 8, 100)
fig, ax = plt.subplots()
ax.scatter(oranges_x, oranges_y,
color="darkorange", label="oranges")
ax.scatter(lemons_x, lemons_y,
color="gold", label="lemons")
# A dividing line chosen by eye
ax.plot(X, 0.9 * X, "g-", linewidth=2)
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.legend()
ax.grid()
plt.show()
Live Python training
Enjoying this page? We offer live Python training courses covering the content of this site.
See our Python training courses
Automatically Finding the Dividing Line
The dividing line in the previous diagram was again chosen by visual inspection. How can we determine such a line systematically?
Before applying an algorithm to many fruits, let us preserve the very simple idea from the original example and look at one incorrect classification. We still restrict ourselves to lines through the origin, so the only parameter to change is the slope $m$.
import matplotlib.pyplot as plt
import numpy as np
def plot_fruits(p1, p2, point_on_line=(5, 1)):
X = np.arange(0, 7)
fig, ax = plt.subplots()
ax.plot(p1[0], p1[1], "o",
color="darkorange", markersize=15)
ax.annotate("orange", xy=p1,
xytext=(p1[0] + 0.5, p1[1] + 0.5),
arrowprops=dict(facecolor="darkorange", shrink=0.05))
ax.plot(p2[0], p2[1], "o",
color="gold", markersize=15)
ax.annotate("lemon", xy=p2,
xytext=(p2[0] - 0.5, p2[1] - 0.5),
arrowprops=dict(facecolor="goldenrod", shrink=0.05))
ax.plot(*point_on_line, "x",
color="black", markersize=12,
label="point on line")
m = point_on_line[1] / point_on_line[0]
ax.plot(X, m * X, "g-", linewidth=3,
label=f"m = {m:.3f}")
ax.set_xlim(0, 7)
ax.set_ylim(0, 5)
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.grid(True)
ax.legend()
plt.show()
orange = (4, 2)
lemon = (1, 3)
point = (5, 1)
plot_fruits(orange, lemon, point)
This line is not suitable because both fruits lie above it. For the orange we can verify this directly with the criterion developed above.
m = point[1] / point[0]
orange_position = m * orange[0] - orange[1]
print("orange position:", orange_position)
OUTPUT:
orange position: -1.2
The result is negative, so the orange lies above the line although an orange should lie below it.
One direct correction is to move the line just above this orange. Let $\delta>0$ be a small vertical safety margin. A line through
has the slope
With $\delta=0.3$, this places the orange below the corrected line.
delta = 0.3
plot_fruits(orange, lemon, point_on_line=(4, 2 + delta))
new_slope = (2 + delta) / 4
print("new slope:", new_slope)
print("orange position after correction:",
new_slope * orange[0] - orange[1])
OUTPUT:
new slope: 0.575 orange position after correction: 0.2999999999999998
The correction can also be expressed as an error in the slope. If $m_{\mathrm{initial}}$ is the old slope and $m_{\mathrm{target}}$ the desired one, then
A complete correction would therefore set
This elementary idea – calculate an error and use it to change a parameter – will reappear later when we train the perceptron.
targeted_slope = new_slope
initial_slope = point[1] / point[0]
error = targeted_slope - initial_slope
print("initial slope:", initial_slope)
print("targeted slope:", targeted_slope)
print("error:", error)
print("initial slope + error:", initial_slope + error)
OUTPUT:
initial slope: 0.2 targeted slope: 0.575 error: 0.37499999999999994 initial slope + error: 0.575
Applying Complete Corrections to More Fruits
Let us now apply this correction principle to more than two fruits. We again use make_blobs, but this time create only nine points so that the individual corrections remain easy to follow.
from sklearn.datasets import make_blobs
# 0 = orange, 1 = lemon
demo_data, demo_labels = make_blobs(
n_samples=9,
centers=[(1.5, 1.0), (1.0, 1.5)],
cluster_std=0.12,
random_state=42
)
fig, ax = plt.subplots()
for i, ((x, y), label) in enumerate(zip(demo_data, demo_labels)):
color = "darkorange" if label == 0 else "gold"
ax.scatter(x, y, color=color, edgecolor="black")
ax.annotate(str(i), (x + 0.02, y + 0.02))
ax.set_xlim(0.5, 2.0)
ax.set_ylim(0.5, 2.0)
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.grid()
plt.show()
For every misclassified point we can calculate a slope that puts that point just on the desired side of the line. The following functions implement this complete correction.
For an orange that lies above the line we move the line above the point,
whereas for a lemon that lies below the line we move the line below the point,
def is_misclassified(slope, x, y, label):
position = slope * x - y
# label 0 = orange: should be below the line -> position > 0
# label 1 = lemon: should be above the line -> position < 0
return ((label == 0 and position <= 0) or
(label == 1 and position >= 0))
def target_slope(x, y, label, delta=0.05):
if label == 0:
return (y + delta) / x
else:
return (y - delta) / x
def adjust_completely(data, labels, slope=0.3, delta=0.05, ax=None):
X_demo = np.linspace(0, 2.2, 100)
counter = 0
for (x, y), label in zip(data, labels):
if is_misclassified(slope, x, y, label):
slope = target_slope(x, y, label, delta)
counter += 1
if ax is not None:
ax.plot(X_demo, slope * X_demo,
linewidth=1, alpha=0.6,
label=f"step {counter}")
return slope
X_demo = np.linspace(0, 2.2, 100)
fig, ax = plt.subplots()
for i, ((x, y), label) in enumerate(zip(demo_data, demo_labels)):
color = "darkorange" if label == 0 else "gold"
ax.scatter(x, y, color=color, edgecolor="black")
ax.annotate(str(i), (x + 0.02, y + 0.02))
start_slope = 0.3
ax.plot(X_demo, start_slope * X_demo,
linewidth=2, label="start")
final_slope = adjust_completely(
demo_data, demo_labels,
slope=start_slope, delta=0.05, ax=ax
)
ax.plot(X_demo, final_slope * X_demo,
"g-", linewidth=3, label="final")
ax.set_xlim(0.5, 2.0)
ax.set_ylim(0.5, 2.0)
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.grid()
ax.legend()
plt.show()
print("final slope:", final_slope)
OUTPUT:
final slope: 0.7813563446257347
Live Python training
Enjoying this page? We offer live Python training courses covering the content of this site.
Upcoming online Courses
See our Python training courses
Why a Complete Correction Can Be Problematic
The strategy above reacts fully to every error. To see why this can be undesirable, we deliberately add an orange inside the lemon region. Such a point could represent an unusual fruit, a measurement error, or an incorrect label.
# Add an orange inside the lemon cluster.
outlier = np.array([[1.05, 1.55]])
demo_data_outlier = np.vstack([demo_data, outlier])
demo_labels_outlier = np.append(demo_labels, 0)
fig, ax = plt.subplots()
for i, ((x, y), label) in enumerate(zip(demo_data_outlier, demo_labels_outlier)):
color = "darkorange" if label == 0 else "gold"
ax.scatter(x, y, color=color, edgecolor="black")
ax.annotate(str(i), (x + 0.02, y + 0.02))
ax.plot(X_demo, start_slope * X_demo,
linewidth=2, label="start")
outlier_slope = adjust_completely(
demo_data_outlier, demo_labels_outlier,
slope=start_slope, delta=0.05, ax=ax
)
ax.plot(X_demo, outlier_slope * X_demo,
"g-", linewidth=3, label="final")
ax.set_xlim(0.5, 2.0)
ax.set_ylim(0.5, 2.0)
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.grid()
ax.legend()
plt.show()
print("final slope with outlier:", outlier_slope)
OUTPUT:
final slope with outlier: 1.5238095238095237
The additional orange reveals the weakness of a complete correction: one unusual point can cause a large change in the line and can make previously correct points wrong.
Instead of jumping all the way to the target slope, we can move only part of the way. This fraction is controlled by the learning rate $\eta$:
For $\eta=1$, this is the complete correction above. For a smaller value, such as $\eta=0.2$, each individual point changes the slope more cautiously.
This does not solve the problem of non-separable data. It merely limits the influence of a single correction step.
def adjust_gradually(data, labels,
slope=0.3,
learning_rate=0.2,
delta=0.05,
max_epochs=1,
shuffle=False,
random_state=42):
rng = np.random.default_rng(random_state)
history = []
error_history = []
for epoch in range(max_epochs):
errors = 0
indices = (rng.permutation(len(data)) if shuffle
else np.arange(len(data)))
for index in indices:
x, y = data[index]
label = labels[index]
if is_misclassified(slope, x, y, label):
wanted = target_slope(x, y, label, delta)
slope += learning_rate * (wanted - slope)
history.append(slope)
errors += 1
error_history.append(errors)
if errors == 0:
return slope, epoch + 1, history, error_history, True
return slope, max_epochs, history, error_history, False
def misclassified_indices(slope, data, labels):
return [
i
for i, ((x, y), label) in enumerate(zip(data, labels))
if is_misclassified(slope, x, y, label)
]
# Exactly the same points, order and start slope as above.
gradual_outlier_slope, _, gradual_history, _, _ = adjust_gradually(
demo_data_outlier, demo_labels_outlier,
slope=start_slope, learning_rate=0.2, delta=0.05,
max_epochs=1, shuffle=False
)
fig, ax = plt.subplots()
for i, ((x, y), label) in enumerate(zip(demo_data_outlier, demo_labels_outlier)):
color = "darkorange" if label == 0 else "gold"
ax.scatter(x, y, color=color, edgecolor="black")
ax.annotate(str(i), (x + 0.02, y + 0.02))
ax.plot(X_demo, start_slope * X_demo,
linewidth=2, label="start")
for slope_value in gradual_history:
ax.plot(X_demo, slope_value * X_demo,
linewidth=1, alpha=0.35)
ax.plot(X_demo, gradual_outlier_slope * X_demo,
"g-", linewidth=3, label="final")
ax.set_xlim(0.5, 2.0)
ax.set_ylim(0.5, 2.0)
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.grid()
ax.legend()
plt.show()
print("complete adjustment slope:", outlier_slope)
print("misclassified after complete adjustment:",
misclassified_indices(outlier_slope, demo_data_outlier, demo_labels_outlier))
print("gradual adjustment slope:", gradual_outlier_slope)
print("misclassified after gradual adjustment:",
misclassified_indices(gradual_outlier_slope, demo_data_outlier, demo_labels_outlier))
OUTPUT:
complete adjustment slope: 1.5238095238095237 misclassified after complete adjustment: [6] gradual adjustment slope: 0.7633503234225044 misclassified after gradual adjustment: [9]
With the same data and the same starting slope, the difference is visible immediately. The complete correction reacts strongly to the unusual orange. The gradual correction preserves the overall pattern better after this single pass, although the unusual point itself remains wrong.
The learning rate therefore controls how strongly a single error is allowed to change the current model. Smaller learning rates usually require more learning steps.
Multiple Passes: Epochs
For the next step, we return to our deliberately linearly separable data set without the additional problematic point. We now allow several complete passes through the data. Such a pass is called an epoch.
Training stops when an epoch contains no misclassification or when a maximum number of epochs is reached.
learned_slope, epochs, history, error_history, converged = adjust_gradually(
fruit_data, fruit_labels,
slope=0.3, learning_rate=0.3,
max_epochs=50, shuffle=True
)
X = np.linspace(0, 8, 100)
fig, ax = plt.subplots()
ax.scatter(oranges_x, oranges_y,
color="darkorange", label="oranges")
ax.scatter(lemons_x, lemons_y,
color="gold", label="lemons")
for slope_value in history[:12]:
ax.plot(X, slope_value * X, linewidth=1, alpha=0.25)
ax.plot(X, learned_slope * X,
"g-", linewidth=3, label="learned dividing line")
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.grid()
ax.legend()
plt.show()
print("learned slope:", learned_slope)
print("epochs:", epochs)
print("converged:", converged)
print("misclassifications per epoch:", error_history)
OUTPUT:
learned slope: 0.7520377022992157 epochs: 7 converged: True misclassifications per epoch: [14, 3, 2, 1, 1, 1, 0]
Now we repeat exactly the same learning procedure after adding one deliberately problematic orange at $(2,5)$, inside the lemon region. This controlled comparison explains why max_epochs is necessary: if the data cannot be separated by a line through the origin, the algorithm may continue correcting indefinitely.
problem_orange = np.array([[2.0, 5.0]])
problem_fruit_data = np.vstack([fruit_data, problem_orange])
problem_fruit_labels = np.append(fruit_labels, 0)
problem_slope, problem_epochs, problem_history, problem_errors, problem_converged = adjust_gradually(
problem_fruit_data, problem_fruit_labels,
slope=0.3, learning_rate=0.3,
max_epochs=50, shuffle=True, random_state=42
)
print("epochs:", problem_epochs)
print("converged:", problem_converged)
print("errors in the last 10 epochs:", problem_errors[-10:])
fig, ax = plt.subplots()
ax.plot(range(1, len(problem_errors) + 1), problem_errors)
ax.set_xlabel("epoch")
ax.set_ylabel("misclassifications")
ax.grid()
plt.show()
OUTPUT:
epochs: 50 converged: False errors in the last 10 epochs: [5, 8, 5, 7, 5, 6, 5, 9, 7, 6]
The difference is fundamental. With the separable data, the algorithm eventually reaches an epoch with zero misclassifications. With the additional problematic point, it does not converge within the maximum number of epochs.
This does not mean that the learning rate is useless. It controls the size of individual corrections. But no learning rate can create a perfect straight-line solution when no such solution exists.
We are now ready to connect this simple slope-learning idea to a perceptron.
Live Python training
Enjoying this page? We offer live Python training courses covering the content of this site.
See our Python training courses
From the Dividing-Line Algorithm to the Perceptron
So far, we have changed only one parameter: the slope $m$ of a line through the origin. A perceptron expresses the same geometric idea with weights.
For two input values $x_1$ and $x_2$, consider the weighted sum
The decision boundary is where this sum is zero:
For $w_2\neq0$ we can solve for $x_2$:
Comparing this with $y=mx$, we obtain
Thus, changing the weights changes the slope of the decision boundary. The perceptron will learn the weights instead of changing $m$ directly.
A Simple Perceptron
Our perceptron has two inputs: $x_1$ for sweetness and $x_2$ for sourness. Each input has a corresponding weight.
To preserve the simple step from the original tutorial, we first choose the weights by hand: $w_1=-0.45$ and $w_2=0.5$. This is not yet learning. We choose these values because
which is the slope of the dividing line we have already used.
import numpy as np
class Perceptron:
def __init__(self, weights):
self.weights = np.array(weights, dtype=float)
def __call__(self, in_data):
weighted_input = self.weights * np.asarray(in_data)
return weighted_input.sum()
p = Perceptron(weights=[-0.45, 0.5])
An instance of this class is callable like a function. Let us first use a single point, as in the original tutorial:
p([2.9, 4])
OUTPUT:
np.float64(0.6950000000000001)
We can now call the same perceptron with several oranges and lemons:
print("oranges:")
for point in zip(oranges_x[:10], oranges_y[:10]):
print(f"{p(point):.3f}", end=" ")
print("\nlemons:")
for point in zip(lemons_x[:10], lemons_y[:10]):
print(f"{p(point):.3f}", end=" ")
OUTPUT:
oranges: -2.049 -1.108 -1.713 -1.682 -1.399 -1.609 -1.063 -0.817 -1.176 -1.077 lemons: 1.265 1.862 2.034 1.913 1.832 2.004 0.438 1.466 1.621 1.304
For oranges we obtain negative values; for lemons positive values. The sign of the weighted sum can therefore be used directly for classification.
Let us verify this for the complete data set:
from collections import Counter
evaluation = Counter()
for point in zip(oranges_x, oranges_y):
if p(point) < 0:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
for point in zip(lemons_x, lemons_y):
if p(point) >= 0:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
print(evaluation)
OUTPUT:
Counter({'correct': 200})
The line on which the weighted sum is exactly zero is the decision boundary:
As derived above,
For our manually chosen weights, the slope is therefore
We can draw this decision boundary together with the data:
X = np.linspace(0, 8, 100)
fig, ax = plt.subplots()
ax.scatter(oranges_x, oranges_y,
color="darkorange", label="oranges")
ax.scatter(lemons_x, lemons_y,
color="gold", label="lemons")
w1, w2 = p.weights
slope = -w1 / w2
ax.plot(X, slope * X,
"g-", linewidth=2, label="decision boundary")
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.grid()
ax.legend()
plt.show()
print("slope:", slope)
OUTPUT:
slope: 0.9
Live Python training
Enjoying this page? We offer live Python training courses covering the content of this site.
See our Python training courses
Training the Perceptron
So far, we have chosen the weights ourselves. A learning model should determine suitable weights from labelled examples.
Before training, we split the original data into training and test data. To make the task more instructive, we add a few deliberately difficult but still linearly separable points close to the decision boundary to the training set only.
# Difficult cases close to the decision boundary
# 0 = orange, 1 = lemon
difficult_oranges = np.array([
[3.8, 3.6],
[4.0, 3.85]
])
difficult_lemons = np.array([
[3.6, 3.8],
[3.85, 4.0]
])
fig, ax = plt.subplots()
ax.scatter(oranges_x, oranges_y,
color="darkorange", alpha=0.35, label="oranges")
ax.scatter(lemons_x, lemons_y,
color="gold", alpha=0.35, label="lemons")
ax.scatter(difficult_oranges[:, 0], difficult_oranges[:, 1],
color="darkorange", edgecolor="black", s=100,
label="difficult oranges")
ax.scatter(difficult_lemons[:, 0], difficult_lemons[:, 1],
color="gold", edgecolor="black", s=100,
label="difficult lemons")
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.grid()
ax.legend()
plt.show()
from sklearn.model_selection import train_test_split
clean_train_data, test_data, clean_train_labels, test_labels = train_test_split(
fruit_data,
fruit_labels,
test_size=0.2,
random_state=42,
stratify=fruit_labels
)
train_data = np.vstack([
clean_train_data,
difficult_oranges,
difficult_lemons
])
train_labels = np.concatenate([
clean_train_labels,
np.zeros(len(difficult_oranges), dtype=int),
np.ones(len(difficult_lemons), dtype=int)
])
rng = np.random.default_rng(1)
order = rng.permutation(len(train_data))
train_data = train_data[order]
train_labels = train_labels[order]
print("clean training samples:", len(clean_train_data))
print("training samples including difficult cases:", len(train_data))
print("test samples:", len(test_data))
OUTPUT:
clean training samples: 160 training samples including difficult cases: 164 test samples: 40
How the Weight Correction Works
We start with arbitrary weights, so some predictions will be wrong. For every training sample we compare the predicted class with the target class. The error is
If $e=0$, the prediction is correct and no change is required. Otherwise we update each weight:
Why should the input value $x_i$ take part in the correction? Suppose, for example, that $x_1=0$ while $x_2>0$. Then $w_1x_1$ contributes nothing to the weighted sum. Changing $w_1$ cannot correct the output for this sample; only the contribution involving $w_2$ matters. This suggests scaling each correction by its corresponding input value.
The direction of the change comes from the sign of the error. We therefore use
where $\eta$ is the learning rate. In vector form,
This is the same principle we encountered earlier: calculate an error and use it to change the current model parameters, but now the parameters are the weights rather than a single slope.
A small learning rate makes cautious updates and usually requires more steps. A very large learning rate reacts more strongly to individual samples and can lead to unstable behaviour.
We now add the learning rule and repeated epochs to our Perceptron class.
import numpy as np
from collections import Counter
class Perceptron:
def __init__(self, weights, learning_rate=0.1):
self.weights = np.array(weights, dtype=float)
self.learning_rate = learning_rate
self.error_history = []
self.converged_ = False
@staticmethod
def unit_step_function(x):
return 0 if x < 0 else 1
def __call__(self, in_data):
weighted_sum = np.dot(self.weights, in_data)
return Perceptron.unit_step_function(weighted_sum)
def adjust(self, target_result, calculated_result, in_data):
error = target_result - calculated_result
if error != 0:
correction = (error * np.asarray(in_data) *
self.learning_rate)
self.weights += correction
def train(self, data, labels, max_epochs=100):
self.error_history = []
self.converged_ = False
for epoch in range(max_epochs):
errors = 0
for in_data, target in zip(data, labels):
calculated = self(in_data)
if calculated != target:
self.adjust(target, calculated, in_data)
errors += 1
self.error_history.append(errors)
if errors == 0:
self.converged_ = True
return epoch + 1
return max_epochs
def evaluate(self, data, labels):
evaluation = Counter()
for in_data, target in zip(data, labels):
if self(in_data) == target:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
return evaluation
# First, train on the easier data for comparison.
p_clean = Perceptron(weights=[0.1, 0.1], learning_rate=0.1)
clean_epochs = p_clean.train(clean_train_data, clean_train_labels,
max_epochs=500)
# Then train on the set containing the difficult borderline cases.
p = Perceptron(weights=[0.1, 0.1], learning_rate=0.1)
epochs = p.train(train_data, train_labels, max_epochs=500)
print("epochs with clean data:", clean_epochs)
print("epochs with difficult cases:", epochs)
print("converged:", p.converged_)
print("training:", p.evaluate(train_data, train_labels))
print("test:", p.evaluate(test_data, test_labels))
print("weights:", p.weights)
OUTPUT:
epochs with clean data: 2
epochs with difficult cases: 51
converged: True
training: Counter({'correct': 164})
test: Counter({'correct': 40})
weights: [-2.70452452 2.80934343]
The difficult points make the learning process longer, but they do not change the fundamental solvability of the task: the data are still linearly separable and the perceptron eventually converges.
We can visualize the learned decision boundary:
X = np.linspace(0, 8, 100)
fig, ax = plt.subplots()
train_oranges = train_data[train_labels == 0]
train_lemons = train_data[train_labels == 1]
ax.scatter(train_oranges[:, 0], train_oranges[:, 1],
color="darkorange", alpha=0.55, label="oranges")
ax.scatter(train_lemons[:, 0], train_lemons[:, 1],
color="gold", alpha=0.55, label="lemons")
ax.scatter(difficult_oranges[:, 0], difficult_oranges[:, 1],
color="darkorange", edgecolor="black", s=100)
ax.scatter(difficult_lemons[:, 0], difficult_lemons[:, 1],
color="gold", edgecolor="black", s=100)
w1, w2 = p.weights
slope = -w1 / w2
ax.plot(X, slope * X,
"g-", linewidth=3, label="learned decision boundary")
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.grid()
ax.legend()
plt.show()
print("weights:", p.weights)
print("slope:", slope)
OUTPUT:
weights: [-2.70452452 2.80934343] slope: 0.9626891793855403
Live Python training
Enjoying this page? We offer live Python training courses covering the content of this site.
See our Python training courses
A Real Problem Case: No Perfect Dividing Line
We now return to the same kind of problematic case that we already investigated with the simple slope adjustment. We deliberately place an orange at
inside the lemon region.
For an orange we would need
so $m>2.5$. At the same time, the difficult lemon $(3.6,3.8)$ requires
or approximately $m<1.056$.
Both conditions cannot be satisfied simultaneously. For our model of a straight line through the origin, no slope can classify every point correctly.
problem_orange = np.array([[2.0, 5.0]])
problem_train_data = np.vstack([train_data, problem_orange])
problem_train_labels = np.concatenate([train_labels, [0]])
rng = np.random.default_rng(7)
order = rng.permutation(len(problem_train_data))
problem_train_data = problem_train_data[order]
problem_train_labels = problem_train_labels[order]
p_problem = Perceptron(weights=[0.1, 0.1], learning_rate=0.1)
problem_epochs = p_problem.train(
problem_train_data, problem_train_labels, max_epochs=100
)
print("epochs:", problem_epochs)
print("converged:", p_problem.converged_)
print("training:",
p_problem.evaluate(problem_train_data, problem_train_labels))
print("errors in the last 10 epochs:",
p_problem.error_history[-10:])
fig, ax = plt.subplots()
problem_oranges = problem_train_data[problem_train_labels == 0]
problem_lemons = problem_train_data[problem_train_labels == 1]
ax.scatter(problem_oranges[:, 0], problem_oranges[:, 1],
color="darkorange", alpha=0.4, label="oranges")
ax.scatter(problem_lemons[:, 0], problem_lemons[:, 1],
color="gold", alpha=0.4, label="lemons")
ax.scatter(problem_orange[:, 0], problem_orange[:, 1],
color="darkorange", edgecolor="black",
marker="X", s=160, label="problem case")
w1, w2 = p_problem.weights
if w2 != 0:
problem_slope = -w1 / w2
ax.plot(X, problem_slope * X,
"g-", linewidth=3, label="boundary after 100 epochs")
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.grid()
ax.legend()
plt.show()
fig, ax = plt.subplots()
ax.plot(range(1, len(p_problem.error_history) + 1),
p_problem.error_history)
ax.set_xlabel("epoch")
ax.set_ylabel("misclassifications")
ax.grid()
plt.show()
OUTPUT:
epochs: 100
converged: False
training: Counter({'correct': 162, 'wrong': 3})
errors in the last 10 epochs: [6, 2, 6, 3, 7, 6, 2, 2, 7, 7]
Here the perceptron behaves fundamentally differently from the merely difficult borderline case. After max_epochs=100, converged is still False; the number of errors does not settle at zero.
This is the perceptron version of the phenomenon we already observed with our simpler slope-learning algorithm: if the classes are not linearly separable by the model we have chosen, repeated correction cannot produce a perfect boundary.
Finally, let us look at the learning algorithm in motion. Each line below corresponds to a state of the weights after a necessary correction.
import matplotlib.pyplot as plt
import matplotlib.cm as cm
p_anim = Perceptron(weights=[0.1, 0.1], learning_rate=0.1)
changes = []
for epoch in range(100):
errors = 0
for index, (in_data, target) in enumerate(zip(train_data, train_labels)):
calculated = p_anim(in_data)
if calculated != target:
p_anim.adjust(target, calculated, in_data)
changes.append((index, target, in_data.copy(),
p_anim.weights.copy()))
errors += 1
if errors == 0:
break
fig, ax = plt.subplots()
ax.scatter(oranges_x, oranges_y,
color="darkorange", alpha=0.18)
ax.scatter(lemons_x, lemons_y,
color="gold", alpha=0.18)
ax.scatter(difficult_oranges[:, 0], difficult_oranges[:, 1],
color="darkorange", edgecolor="black", s=80)
ax.scatter(difficult_lemons[:, 0], difficult_lemons[:, 1],
color="gold", edgecolor="black", s=80)
colors = cm.rainbow(np.linspace(0, 1, max(1, len(changes))))
if len(changes) <= 20:
selected = list(range(len(changes)))
else:
selected = sorted(set(np.linspace(0, len(changes) - 1,
20, dtype=int)))
for counter in selected:
index, target, point, weights = changes[counter]
color = "darkorange" if target == 0 else "gold"
ax.scatter(point[0], point[1], color=color, edgecolor="black")
ax.annotate(str(counter), (point[0], point[1]))
w1, w2 = weights
if w2 != 0:
current_slope = -w1 / w2
ax.plot(X, current_slope * X,
color=colors[counter], alpha=0.65,
label=str(counter))
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
ax.set_xlim(0, 8)
ax.set_ylim(0, 8)
ax.grid()
ax.legend(title="weight update")
plt.show()
print("number of weight updates:", len(changes))
print("epochs:", epoch + 1)
OUTPUT:
number of weight updates: 112 epochs: 51
Each numbered point triggered a change in the weights. The corresponding lines show how the decision boundary changes step by step.
We have now encountered three increasingly realistic situations:
- clearly linearly separable classes, which are easy to learn;
- difficult points close to the decision boundary, which require more corrections but remain separable;
- contradictory or non-separable data, for which a perceptron with a straight decision boundary cannot converge to a perfect solution.
The simple geometric idea with which we started – determining on which side of a line a point lies – has therefore led directly to the basic learning mechanism of a perceptron.
Live Python training
Enjoying this page? We offer live Python training courses covering the content of this site.
Upcoming online Courses
See our Python training courses