## Training a Neural Network with Python

### Introduction

In the chapter "Running Neural Networks", we programmed a class in Python code called 'NeuralNetwork'. The instances of this class are networks with three layers. When we instantiate an ANN of this class, the weight matrices between the layers are automatically and randomly chosen. It is even possible to run such a ANN on some input, but naturally it doesn't make a lot of sense exept for testing purposes. Such an ANN cannot provide correct classification results. In fact, the classification results are in no way adapted to the expected results. The values of the weight matrices have to be set according the the classification task. We need to improve the weight values, which means that we have to train our network. To train it we have to implement backpropagation in the train method. If you don't understand backpropagation and want to understand it, we recommend to go back to the chapter Backpropagation in Neural Networks.

After knowing und hopefully understanding backpropagation, you are ready to fully understand the train method.

The train method is called with an input vector and a target vector. The shape of the vectors can be one-dimensional, but they will be automatically turned into the correct two-dimensional shape, i.e. reshape(input_vector.size, 1) and reshape(target_vector.size, 1). After this we call the run method to get the result of the network output_vector_network = self.run(input_vector). This output may differ from the target_vector. We calculate the output_error by subtracting the output of the network output_vector_network from the target_vector.

import numpy as np

@np.vectorize
def sigmoid(x):
return 1 / (1 + np.e ** -x)
activation_function = sigmoid

from scipy.stats import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):
return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neural network"""
self.weights_in_hidden = X.rvs((self.no_of_hidden_nodes,
self.no_of_in_nodes))
self.weights_hidden_out = X.rvs((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train(self, input_vector, target_vector):
"""
input_vector and target_vector can be tuples, lists or ndarrays
"""
# make sure that the vectors have the right shape
input_vector = np.array(input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
target_vector = np.array(target_vector).reshape(target_vector.size, 1)

output_vector_hidden = activation_function(self.weights_in_hidden @ input_vector)
output_vector_network = activation_function(self.weights_hidden_out @ output_vector_hidden)

output_error = target_vector - output_vector_network
tmp = output_error * output_vector_network * (1.0 - output_vector_network)
self.weights_hidden_out += self.learning_rate  * (tmp @ output_vector_hidden.T)

# calculate hidden errors:
hidden_errors = self.weights_hidden_out.T @ output_error
# update the weights:
tmp = hidden_errors * output_vector_hidden * (1.0 - output_vector_hidden)
self.weights_in_hidden += self.learning_rate * (tmp @ input_vector.T)

def run(self, input_vector):
"""
running the network with an input vector 'input_vector'.
'input_vector' can be tuple, list or ndarray
"""
# make sure that input_vector is a column vector:
input_vector = np.array(input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
input4hidden = activation_function(self.weights_in_hidden @ input_vector)
output_vector_network = activation_function(self.weights_hidden_out @ input4hidden)
return output_vector_network



We assume that you save the previous code in a file called neural_networks1.py. We will use it under this name in the coming examples.

To test this neural network class we need train and test data. We create the data with make_blobs from sklearn.datasets.

from sklearn.datasets import make_blobs

n_samples = 300
samples, labels = make_blobs(n_samples=n_samples,
centers=([2, 6], [6, 2]),
random_state=0)


Let us visualize the previously created data:

import matplotlib.pyplot as plt

colours = ('green', 'red', 'blue', 'magenta', 'yellow', 'cyan')
fig, ax = plt.subplots()

for n_class in range(2):
ax.scatter(samples[labels==n_class][:, 0], samples[labels==n_class][:, 1],
c=colours[n_class], s=40, label=str(n_class))


We are going to create a train and a test data set:

size_of_learn_sample = int(n_samples * 0.8)
learn_data = samples[:size_of_learn_sample]
test_data = samples[-size_of_learn_sample:]


We create a neural network with two input nodes, two hidden nodes and one output node:

from neural_networks1 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2,
no_of_out_nodes=1,
no_of_hidden_nodes=5,
learning_rate=0.3)


The next step consists in training our network with the samples from our training samples:

for i in range(size_of_learn_sample):
simple_network.train(learn_data[i], labels[i])


We now have to check how well our network has learned. The network has only one output neuron. This means that the values will be between 0 and 1. If the output values were ideal, which they cannot be, the output values would be one for class 1 and 0 for class 0. Due to the sigmoid function 0 and and will not be even a possible result. So we have to assign result to the values between 0 and 1. We use 0.5 as a threshold. Everything greater or equal than 0.5 is considered to be 1 and everything smaller is taken as a 0. Now we are capable of comparing the results with the labels:

from collections import Counter

evaluation = Counter()
for i in range(size_of_learn_sample):
point, label = learn_data[i], labels[i]
res = simple_network.run(point)
if label == 1:
if res >= 0.5:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
elif label == 0:
if res <= 0.5:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
print(evaluation)

Counter({'correct': 239, 'wrong': 1})


The flaw in the design above is this: If we have a value of 0.5 or close to this, the classifier is rather undecided. The result is in the middle between two possible results.

from collections import Counter

def evaluate(data, labels, threshold=0.5):
evaluation = Counter()
for i in range(len(data)):
point, label = data[i], labels[i]
res = simple_network.run(point)
if threshold < res < 1 - threshold:
evaluation["undecided"] += 1
elif label == 1:
if res >= 1 - threshold:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
elif label == 0:
if res <= threshold:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
return evaluation

res = evaluate(learn_data, labels)
res

Output::
Counter({'correct': 239, 'wrong': 1})

### Neural Network with Bias Nodes

We already introduced the basic idea and necessity of bias node in the chapter "Simple Neural Network", in which we focussed on very simple linearly separable data sets. We learned that a bias node is a node that is always returning the same output. In other words: It is a node which is not depending on some input and it does not have any input. The value of a bias node is often set to one, but it can be set to other values as well. Except for zero, which makes no sense for obvious reasons. If a neural network does not have a bias node in a given layer, it will not be able to produce output in the next layer that differs from 0 when the feature values are 0. Generally speaking, we can say that bias nodes are used to increase the flexibility of the network to fit the data. Usually, there will be not more than one bias node per layer. The only exception is the output layer, because it makes no sense to add a bias node to this layer.

The following diagram shows the first two layers of our previously used three-layered neural network:

We can see from this diagram that our weight matrix needs one additional column and the bias value has to be added to the input vector:

Again, the situation for the weight matrix between the hidden and the output layer is similar:

The same is true for the corresponding matrix:

The following is a complete Python class implementing our network with bias nodes:

import numpy as np
from scipy.stats import truncnorm

@np.vectorize
def sigmoid(x):
return 1 / (1 + np.e ** -x)
activation_function = sigmoid

def truncated_normal(mean=0, sd=1, low=0, upp=10):
return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate,
bias=None):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.no_of_out_nodes = no_of_out_nodes
self.learning_rate = learning_rate
self.bias = bias
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neural
network with optional bias nodes"""
bias_node = 1 if self.bias else 0
rad = 1 / np.sqrt(self.no_of_in_nodes + bias_node)
self.weights_in_hidden = X.rvs((self.no_of_hidden_nodes,
self.no_of_in_nodes + bias_node))
rad = 1 / np.sqrt(self.no_of_hidden_nodes + bias_node)
self.weights_hidden_out = X.rvs((self.no_of_out_nodes,
self.no_of_hidden_nodes + bias_node))

def train(self, input_vector, target_vector):
""" input_vector and target_vector can be tuple, list or ndarray """

# make sure that the vectors have the right shap
input_vector = np.array(input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
if self.bias:
# adding bias node to the end of the input_vector
input_vector = np.concatenate( (input_vector, [[self.bias]]) )
target_vector = np.array(target_vector).reshape(target_vector.size, 1)

output_vector_hidden = activation_function(self.weights_in_hidden @ input_vector)
if self.bias:
output_vector_hidden = np.concatenate( (output_vector_hidden, [[self.bias]]) )
output_vector_network = activation_function(self.weights_hidden_out @ output_vector_hidden)

output_error = target_vector - output_vector_network
# update the weights:
tmp = output_error * output_vector_network * (1.0 - output_vector_network)
self.weights_hidden_out += self.learning_rate  * (tmp @ output_vector_hidden.T)

# calculate hidden errors:
hidden_errors = self.weights_hidden_out.T @ output_error
# update the weights:
tmp = hidden_errors * output_vector_hidden * (1.0 - output_vector_hidden)
if self.bias:
x = (tmp @input_vector.T)[:-1,:]     # last row cut off,
else:
x = tmp @ input_vector.T
self.weights_in_hidden += self.learning_rate *  x

def run(self, input_vector):
"""
running the network with an input vector 'input_vector'.
'input_vector' can be tuple, list or ndarray
"""
# make sure that input_vector is a column vector:
input_vector = np.array(input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
if self.bias:
# adding bias node to the end of the inpuy_vector
input_vector = np.concatenate( (input_vector, [[1]]) )
input4hidden = activation_function(self.weights_in_hidden @ input_vector)
if self.bias:
input4hidden = np.concatenate( (input4hidden, [[1]]) )
output_vector_network = activation_function(self.weights_hidden_out @ input4hidden)
return output_vector_network

def evaluate(self, data, labels):
corrects, wrongs = 0, 0
for i in range(len(data)):
res = self.run(data[i])
res_max = res.argmax()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt

data, labels = make_blobs(n_samples=250,
centers=([2.5, 3], [6.7, 7.9]),
random_state=0)

data, labels = make_blobs(n_samples=250,
centers=([2, 7.9], [8, 3]),
random_state=0)

colours = ('green', 'blue', 'red', 'magenta', 'yellow', 'cyan')
fig, ax = plt.subplots()

for n_class in range(2):
ax.scatter(data[labels==n_class][:, 0], data[labels==n_class][:, 1],
c=colours[n_class], s=40, label=str(n_class))

from neural_networks2 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2,
no_of_out_nodes=2,
no_of_hidden_nodes=10,
learning_rate=0.1,
bias=1)

simple_network.__dict__

Output::
{'no_of_in_nodes': 2,
'no_of_hidden_nodes': 10,
'no_of_out_nodes': 2,
'learning_rate': 0.1,
'bias': 1,
'weights_in_hidden': array([[-0.38709697,  0.34216142, -0.53558474],
[-0.38265064,  0.07216054, -0.19346557],
[ 0.06640281, -0.22866835,  0.33205037],
[ 0.06057826,  0.49377468,  0.2012993 ],
[ 0.37061155, -0.12533196, -0.31096331],
[-0.26451103, -0.18483328,  0.34717534],
[-0.13736405, -0.11134647,  0.49381627],
[-0.29948127, -0.09301347, -0.33229527],
[ 0.52635705, -0.43268611,  0.36468496],
[-0.02565338,  0.19563129,  0.02560361]]),
'weights_hidden_out': array([[-0.2630956 , -0.28078972, -0.27659632,  0.28614888, -0.21546324,
-0.11180555, -0.04695778,  0.27741456, -0.04321395,  0.24967042,
-0.03368758],
[-0.05675981, -0.28455935,  0.12732364,  0.07407335, -0.12645493,
0.07372685, -0.23219729, -0.02740009, -0.03702381, -0.23802179,
-0.28229358]])}
import numpy as np
labels_one_hot = (np.arange(2) == labels.reshape(labels.size, 1))
labels_one_hot = labels_one_hot.astype(np.float)

for i in range(len(data)):
simple_network.train(data[i], labels_one_hot[i])

simple_network.evaluate(data, labels)

Output::
(250, 0)
data

Output::
array([[ 1.61267318,  7.59769725],
[ 3.13940068,  6.66517418],
[ 6.98195812,  2.92214524],
[ 3.53277921,  9.36935877],
[ 5.34082776,  3.60631952],
[ 2.61407937,  8.82220667],
[ 3.49407907,  7.69484174],
[ 1.18663574,  6.43357567],
[ 8.46313033,  3.27909576],
[ 8.52106488,  2.42421203],
[ 1.044055  ,  7.55401822],
[ 2.14404357,  9.35427351],
[ 7.43068795,  3.26990435],
[ 7.34759142,  2.60904662],
[ 7.91406923,  3.19429294],
[ 1.13877431,  9.81006495],
[ 6.06372019,  3.1887786 ],
[ 6.30189418,  3.38728048],
[ 7.60055097,  3.37005589],
[ 7.95429604,  3.22050766],
[ 1.56484645,  9.74926373],
[ 0.38610215,  7.68725972],
[ 7.4971833 ,  5.41245368],
[ 7.28039561,  2.18700701],
[ 3.17877957,  7.72007516],
[ 0.82687659,  9.84362119],
[ 1.40268393,  7.66207827],
[ 8.14195316,  2.68067158],
[ 2.29823817,  9.2263859 ],
[ 7.95782855,  2.71311281],
[ 8.27451636,  2.10908492],
[ 1.48919486,  6.71936782],
[ 8.82158571,  3.00529265],
[ 1.30543214,  7.75036546],
[ 7.27440262,  1.61663604],
[ 3.76405235,  8.30015721],
[ 8.77325298,  1.81611936],
[ 2.3130677 ,  7.04590426],
[ 7.01448926,  1.52816499],
[ 7.48248096,  2.02117014],
[ 8.81451982,  3.05892559],
[ 6.95474663,  4.21114529],
[ 2.20827498,  8.87663904],
[ 2.39600671,  6.80693849],
[ 4.25930895,  7.85774285],
[ 0.74720464,  8.67749036],
[ 1.32753955,  7.54044684],
[ 2.40234164,  7.21518991],
[ 9.12859406,  3.49600095],
[ 7.63530646,  3.15670386],
[ 6.87317419,  2.26932225],
[ 7.03949562,  2.20688264],
[ 2.01050002,  9.68587049],
[ 8.33890413,  5.02104356],
[ 1.68844747,  7.95616534],
[ 3.18802979,  8.21694261],
[ 2.72909056,  8.02898291],
[ 2.85683061,  7.24897441],
[ 3.48825219,  9.79588918],
[ 2.17742614,  7.49821906],
[ 1.58638102,  7.15254519],
[ 0.68409259,  7.4384154 ],
[ 7.79309632,  3.88017891],
[ 3.92294203,  9.38051479],
[ 9.64813493,  3.16422776],
[ 7.9383736 ,  2.89269472],
[ 1.09270164,  7.9519454 ],
[ 1.36256297,  7.50272819],
[ 7.53315445,  1.58309389],
[ 8.04949498,  3.49383678],
[ 7.86989305,  3.09395323],
[ 8.28634369,  3.60884383],
[ 1.65208785,  8.05634897],
[ 2.3563664 ,  8.60657317],
[ 1.32566734,  7.93183056],
[ 2.46566244,  6.36375631],
[ 9.31913688,  2.11758118],
[ 1.3563816 ,  5.67659685],
[ 2.12691209,  8.30198936],
[ 7.72432947,  2.29027203],
[ 6.15693045,  2.522026  ],
[ 1.89678115,  8.3105985 ],
[ 7.55907737,  2.7196445 ],
[ 9.10028434,  4.29802197],
[ 1.93175839,  9.61334272],
[ 8.68981816,  4.30184623],
[ 0.96575716,  8.58159452],
[ 0.92924738,  8.95445173],
[ 7.73226646,  1.87198867],
[ 7.56081048,  3.18133843],
[ 0.95144703,  6.47998206],
[ 2.62523145,  6.29794234],
[ 6.69347315,  4.65813068],
[ 3.23029068,  9.10237985],
[ 2.04575852,  7.71281615],
[ 8.87583276,  2.88489253],
[ 4.16323595,  9.23652795],
[ 9.51999486,  4.71958931],
[ 7.68911383,  3.09740017],
[ 8.64331447,  1.42937659],
[ 1.97181777,  8.32833187],
[ 2.1666735 ,  8.53503144],
[ 8.66638308,  2.53928021],
[ 2.2799246 ,  7.80184961],
[ 8.69153875,  3.69474914],
[ 1.49034782,  7.4619257 ],
[ 1.73199663,  8.7024564 ],
[ 1.98297959,  8.27915174],
[ 9.36453185,  2.31055082],
[ 1.05555374,  7.48995031],
[ 1.96071718,  6.7319065 ],
[ 3.8831507 ,  6.55224094],
[ 7.88945934,  4.02017271],
[ 7.09123675,  2.57568238],
[ 2.42625873,  8.57690804],
[ 1.36415392,  8.57643329],
[ 1.260437  ,  9.4430146 ],
[ 9.73887268,  3.99439439],
[ 0.50874241,  8.3393917 ],
[ 1.50196755,  9.82953205],
[ 2.91017891,  8.21721822],
[ 1.45713852,  8.31605005],
[-0.55298982,  8.5536186 ],
[ 8.86896349,  3.27687191],
[ 4.26975462,  6.44563433],
[ 8.76666318,  3.35629282],
[ 5.90539693,  3.12372191],
[ 7.86629844,  4.07774381],
[ 6.23146155,  3.35548179],
[ 0.85253135,  7.46217996],
[ 1.18685372,  6.1737174 ],
[ 2.67229476,  8.30746184],
[ 7.98431789,  3.16092817],
[ 7.34144703,  2.48576603],
[ 6.66574153,  1.65328249],
[ 4.38314477,  8.84447949],
[ 0.83485016,  8.80082649],
[ 8.49374178,  2.88389606],
[ 0.36980165,  8.36278226],
[ 6.24410942,  3.45093446],
[ 2.97873798, 10.1408932 ],
[ 7.29529972,  3.94326072],
[ 9.73272119,  3.68450111],
[ 7.70816264,  2.23850779],
[ 7.69088556,  2.76653334],
[ 8.8005648 ,  3.07826018],
[ 7.3159891 ,  4.6595508 ],
[ 1.25524518,  7.07356146],
[ 1.23008393,  8.43924919],
[ 6.94537154,  3.82024784],
[ 7.31216239,  1.7859226 ],
[ 2.94725197,  7.74498991],
[ 6.84264474,  2.68770775],
[ 2.78632796,  7.4335809 ],
[10.69622405,  2.92607533],
[ 7.40134606,  1.88410301],
[ 8.86259601,  0.34438091],
[ 8.93184837,  3.33996498],
[ 7.3477064 ,  2.47881069],
[ 9.95591231,  3.39009332],
[ 1.08717777,  9.01701629],
[ 2.92085882,  8.21872765],
[ 2.06651722,  8.2024719 ],
[10.3039167 ,  1.93998418],
[ 6.4170616 ,  3.61037938],
[ 8.56729028,  2.7773249 ],
[ 3.15233156,  8.97961859],
[ 2.37642553,  6.80059921],
[ 8.92950511,  3.58222459],
[ 5.98359337,  2.46054537],
[ 7.60477102,  1.84057948],
[ 8.77140595,  4.02943883],
[ 1.36567791,  7.53725883],
[ 8.39904635,  0.22740724],
[ 8.74718833,  1.81105504],
[ 8.370825  ,  3.14206181],
[ 0.77456448,  8.74436298],
[ 8.84163126,  2.75054142],
[ 5.74443577,  1.97749316],
[ 0.89561666,  7.95216508],
[ 0.57593909,  7.40668012],
[ 0.84381757,  8.6811981 ],
[ 2.57659082,  7.69170124],
[ 8.1993002 ,  2.94939646],
[ 3.86755896,  8.80604466],
[ 6.5534653 ,  3.80029795],
[ 5.96931553,  5.06449286],
[ 2.15650654,  8.13218104],
[ 9.51332808,  3.55313206],
[ 7.81494633,  2.19235151],
[ 8.03863055,  1.3432849 ],
[ 2.64013153,  6.28304396],
[ 7.52034419,  3.6203583 ],
[ 2.44386323,  8.23367433],
[ 3.49448454,  5.83001497],
[ 1.97567388,  7.16196909],
[ 8.69845715,  3.00377089],
[ 7.84233298,  5.2567235 ],
[ 3.12663592,  6.82006849],
[ 2.15494743,  8.27816252],
[ 9.46657872,  3.85255194],
[ 1.86711942,  7.60220912],
[ 8.52389102,  3.08842209],
[ 3.86755799,  6.92272212],
[ 7.61512019,  3.09435159],
[ 5.71137996,  3.25148442],
[ 8.5785215 ,  3.34965446],
[ 1.64600609,  6.52504871],
[ 2.95008842,  7.74864279],
[ 7.88183595,  2.3198218 ],
[ 1.11221425,  5.91920353],
[ 0.729515  ,  8.86939671],
[ 1.12920285,  7.32115034],
[ 1.19659034,  7.21045022],
[ 1.5444675 ,  7.91747916],
[ 2.76103773,  8.02167502],
[ 1.59682305,  9.12244507],
[ 0.70714309,  8.16705087],
[ 8.45741561,  2.03538799],
[ 1.10453344,  8.2869025 ],
[ 7.21737084,  2.8896107 ],
[ 0.45920299,  7.96326199],
[ 7.8640503 ,  4.13689136],
[ 2.52327666,  7.72845367],
[ 6.97006472,  2.65005664],
[ 1.53640403,  8.38148147],
[ 7.64656825,  1.38352581],
[ 1.63081816,  7.66062082],
[ 7.80934651,  2.60515049],
[ 8.94304609,  0.26032283],
[ 7.02889543,  3.3148172 ],
[ 3.0996596 ,  8.55526373],
[ 7.37191244,  2.51897288],
[ 8.28044171,  2.00687639],
[ 1.69098703,  6.22399619],
[ 9.0685094 ,  2.5466142 ],
[ 8.09772497,  3.58295368],
[ 0.29372981,  9.8507754 ],
[ 2.77179055,  8.72350415],
[ 7.23585608,  1.56220853],
[ 8.85792392,  4.14110187],
[ 8.69377315,  2.84042656],
[ 2.94942081,  7.98755124],
[ 7.53113581,  0.79855871],
[ 7.30795015,  4.53637705],
[ 2.8644362 ,  7.15783498],
[ 1.90154748,  7.23652171],
[ 7.40368596,  2.9474327 ],
[ 0.99978465,  6.3552289 ],
[ 6.81114074,  2.49318365]])
In [ ]: