Training a Neural Network with Python



Introduction

Teacher explaining neural network

In the chapter "Running Neural Networks", we programmed a class in Python code called 'NeuralNetwork'. The instances of this class are networks with three layers. When we instantiate an ANN of this class, the weight matrices between the layers are automatically and randomly chosen. It is even possible to run such a ANN on some input, but naturally it doesn't make a lot of sense exept for testing purposes. Such an ANN cannot provide correct classification results. In fact, the classification results are in no way adapted to the expected results. The values of the weight matrices have to be set according the the classification task. We need to improve the weight values, which means that we have to train our network. To train it we have to implement backpropagation in the train method. If you don't understand backpropagation and want to understand it, we recommend to go back to the chapter Backpropagation in Neural Networks.

After knowing und hopefully understanding backpropagation, you are ready to fully understand the train method.


The train method is called with an input vector and a target vector. The shape of the vectors can be one-dimensional, but they will be automatically turned into the correct two-dimensional shape, i.e. reshape(input_vector.size, 1) and reshape(target_vector.size, 1). After this we call the run method to get the result of the network output_vector_network = self.run(input_vector). This output may differ from the target_vector. We calculate the output_error by subtracting the output of the network output_vector_network from the target_vector.

import numpy as np

@np.vectorize
def sigmoid(x):
    return 1 / (1 + np.e ** -x)
activation_function = sigmoid

from scipy.stats import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):
    return truncnorm(
        (low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:
    
    def __init__(self, 
                 no_of_in_nodes, 
                 no_of_out_nodes, 
                 no_of_hidden_nodes,
                 learning_rate):
        self.no_of_in_nodes = no_of_in_nodes
        self.no_of_out_nodes = no_of_out_nodes
        self.no_of_hidden_nodes = no_of_hidden_nodes
        self.learning_rate = learning_rate 
        self.create_weight_matrices()
        
    def create_weight_matrices(self):
        """ A method to initialize the weight matrices of the neural network"""
        rad = 1 / np.sqrt(self.no_of_in_nodes)
        X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
        self.weights_in_hidden = X.rvs((self.no_of_hidden_nodes, 
                                       self.no_of_in_nodes))
        rad = 1 / np.sqrt(self.no_of_hidden_nodes)
        X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
        self.weights_hidden_out = X.rvs((self.no_of_out_nodes, 
                                        self.no_of_hidden_nodes))
        
    
    def train(self, input_vector, target_vector):
        """
        input_vector and target_vector can be tuples, lists or ndarrays
        """
        # make sure that the vectors have the right shape
        input_vector = np.array(input_vector)
        input_vector = input_vector.reshape(input_vector.size, 1)
        target_vector = np.array(target_vector).reshape(target_vector.size, 1)

        output_vector_hidden = activation_function(self.weights_in_hidden @ input_vector)
        output_vector_network = activation_function(self.weights_hidden_out @ output_vector_hidden)
        
        output_error = target_vector - output_vector_network
        tmp = output_error * output_vector_network * (1.0 - output_vector_network)    
        self.weights_hidden_out += self.learning_rate  * (tmp @ output_vector_hidden.T)

        # calculate hidden errors:
        hidden_errors = self.weights_hidden_out.T @ output_error
        # update the weights:
        tmp = hidden_errors * output_vector_hidden * (1.0 - output_vector_hidden)
        self.weights_in_hidden += self.learning_rate * (tmp @ input_vector.T)  
    
    def run(self, input_vector):
        """
        running the network with an input vector 'input_vector'. 
        'input_vector' can be tuple, list or ndarray
        """
        # make sure that input_vector is a column vector:
        input_vector = np.array(input_vector)
        input_vector = input_vector.reshape(input_vector.size, 1)
        input4hidden = activation_function(self.weights_in_hidden @ input_vector)
        output_vector_network = activation_function(self.weights_hidden_out @ input4hidden)
        return output_vector_network
            
            

We assume that you save the previous code in a file called neural_networks1.py. We will use it under this name in the coming examples.

To test this neural network class we need train and test data. We create the data with make_blobs from sklearn.datasets.

from sklearn.datasets import make_blobs

n_samples = 300
samples, labels = make_blobs(n_samples=n_samples, 
                             centers=([2, 6], [6, 2]), 
                             random_state=0)

Let us visualize the previously created data:

import matplotlib.pyplot as plt


colours = ('green', 'red', 'blue', 'magenta', 'yellow', 'cyan')
fig, ax = plt.subplots()


for n_class in range(2):
    ax.scatter(samples[labels==n_class][:, 0], samples[labels==n_class][:, 1], 
               c=colours[n_class], s=40, label=str(n_class))

We are going to create a train and a test data set:

size_of_learn_sample = int(n_samples * 0.8)
learn_data = samples[:size_of_learn_sample]
test_data = samples[-size_of_learn_sample:]

We create a neural network with two input nodes, two hidden nodes and one output node:

from neural_networks1 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2, 
                               no_of_out_nodes=1, 
                               no_of_hidden_nodes=5,
                               learning_rate=0.3)

The next step consists in training our network with the samples from our training samples:

for i in range(size_of_learn_sample):
    simple_network.train(learn_data[i], labels[i])

We now have to check how well our network has learned. The network has only one output neuron. This means that the values will be between 0 and 1. If the output values were ideal, which they cannot be, the output values would be one for class 1 and 0 for class 0. Due to the sigmoid function 0 and and will not be even a possible result. So we have to assign result to the values between 0 and 1. We use 0.5 as a threshold. Everything greater or equal than 0.5 is considered to be 1 and everything smaller is taken as a 0. Now we are capable of comparing the results with the labels:

from collections import Counter

evaluation = Counter()
for i in range(size_of_learn_sample):
    point, label = learn_data[i], labels[i]
    res = simple_network.run(point)
    if label == 1:
        if res >= 0.5:
            evaluation["correct"] += 1
        else:
            evaluation["wrong"] += 1
    elif label == 0:
        if res <= 0.5:
            evaluation["correct"] += 1
        else:
            evaluation["wrong"] += 1
print(evaluation)
Counter({'correct': 239, 'wrong': 1})

The flaw in the design above is this: If we have a value of 0.5 or close to this, the classifier is rather undecided. The result is in the middle between two possible results.

from collections import Counter

def evaluate(data, labels, threshold=0.5):
    evaluation = Counter()
    for i in range(len(data)):
        point, label = data[i], labels[i]
        res = simple_network.run(point)
        if threshold < res < 1 - threshold:
            evaluation["undecided"] += 1
        elif label == 1:
            if res >= 1 - threshold:
                evaluation["correct"] += 1
            else:
                evaluation["wrong"] += 1
        elif label == 0:
            if res <= threshold:
                evaluation["correct"] += 1
            else:
                evaluation["wrong"] += 1
    return evaluation

                
res = evaluate(learn_data, labels)
res
Output::
Counter({'correct': 239, 'wrong': 1})

Neural Network with Bias Nodes

We already introduced the basic idea and necessity of bias node in the chapter "Simple Neural Network", in which we focussed on very simple linearly separable data sets. We learned that a bias node is a node that is always returning the same output. In other words: It is a node which is not depending on some input and it does not have any input. The value of a bias node is often set to one, but it can be set to other values as well. Except for zero, which makes no sense for obvious reasons. If a neural network does not have a bias node in a given layer, it will not be able to produce output in the next layer that differs from 0 when the feature values are 0. Generally speaking, we can say that bias nodes are used to increase the flexibility of the network to fit the data. Usually, there will be not more than one bias node per layer. The only exception is the output layer, because it makes no sense to add a bias node to this layer.

The following diagram shows the first two layers of our previously used three-layered neural network:

Weights Array from input to hidden layer with bias

We can see from this diagram that our weight matrix needs one additional column and the bias value has to be added to the input vector:

Weights Array from hidden to output layer with bias, matrix multiplication

Again, the situation for the weight matrix between the hidden and the output layer is similar:

Weights Array from hidden  to output layer with bias

The same is true for the corresponding matrix:

Weights Array from hidden to output layer with bias, matrix multiplication

The following is a complete Python class implementing our network with bias nodes:

import numpy as np
from scipy.stats import truncnorm

@np.vectorize
def sigmoid(x):
    return 1 / (1 + np.e ** -x)
activation_function = sigmoid

def truncated_normal(mean=0, sd=1, low=0, upp=10):
    return truncnorm(
        (low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)



class NeuralNetwork:
            
    def __init__(self, 
                 no_of_in_nodes, 
                 no_of_out_nodes, 
                 no_of_hidden_nodes,
                 learning_rate,
                 bias=None):  
        self.no_of_in_nodes = no_of_in_nodes
        self.no_of_hidden_nodes = no_of_hidden_nodes
        self.no_of_out_nodes = no_of_out_nodes
        self.learning_rate = learning_rate 
        self.bias = bias
        self.create_weight_matrices()
    
        
    def create_weight_matrices(self):
        """ A method to initialize the weight matrices of the neural 
        network with optional bias nodes"""   
        bias_node = 1 if self.bias else 0 
        rad = 1 / np.sqrt(self.no_of_in_nodes + bias_node)
        X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
        self.weights_in_hidden = X.rvs((self.no_of_hidden_nodes, 
                                        self.no_of_in_nodes + bias_node))
        rad = 1 / np.sqrt(self.no_of_hidden_nodes + bias_node)
        X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
        self.weights_hidden_out = X.rvs((self.no_of_out_nodes, 
                                         self.no_of_hidden_nodes + bias_node))

        
    def train(self, input_vector, target_vector):
        """ input_vector and target_vector can be tuple, list or ndarray """

        # make sure that the vectors have the right shap
        input_vector = np.array(input_vector)
        input_vector = input_vector.reshape(input_vector.size, 1)        
        if self.bias:
            # adding bias node to the end of the input_vector
            input_vector = np.concatenate( (input_vector, [[self.bias]]) )
        target_vector = np.array(target_vector).reshape(target_vector.size, 1)

        output_vector_hidden = activation_function(self.weights_in_hidden @ input_vector)
        if self.bias:
            output_vector_hidden = np.concatenate( (output_vector_hidden, [[self.bias]]) ) 
        output_vector_network = activation_function(self.weights_hidden_out @ output_vector_hidden)
        
        output_error = target_vector - output_vector_network  
        # update the weights:
        tmp = output_error * output_vector_network * (1.0 - output_vector_network)     
        self.weights_hidden_out += self.learning_rate  * (tmp @ output_vector_hidden.T)

        # calculate hidden errors:
        hidden_errors = self.weights_hidden_out.T @ output_error
        # update the weights:
        tmp = hidden_errors * output_vector_hidden * (1.0 - output_vector_hidden)
        if self.bias:
            x = (tmp @input_vector.T)[:-1,:]     # last row cut off,
        else:
            x = tmp @ input_vector.T
        self.weights_in_hidden += self.learning_rate *  x

           
    def run(self, input_vector):
        """
        running the network with an input vector 'input_vector'. 
        'input_vector' can be tuple, list or ndarray
        """
        # make sure that input_vector is a column vector:
        input_vector = np.array(input_vector)
        input_vector = input_vector.reshape(input_vector.size, 1)
        if self.bias:
            # adding bias node to the end of the inpuy_vector
            input_vector = np.concatenate( (input_vector, [[1]]) )
        input4hidden = activation_function(self.weights_in_hidden @ input_vector)
        if self.bias:
            input4hidden = np.concatenate( (input4hidden, [[1]]) )
        output_vector_network = activation_function(self.weights_hidden_out @ input4hidden)
        return output_vector_network
            
    def evaluate(self, data, labels):
        corrects, wrongs = 0, 0
        for i in range(len(data)):
            res = self.run(data[i])
            res_max = res.argmax()
            if res_max == labels[i]:
                corrects += 1
            else:
                wrongs += 1
        return corrects, wrongs
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt

data, labels = make_blobs(n_samples=250, 
                             centers=([2.5, 3], [6.7, 7.9]), 
                             random_state=0)

data, labels = make_blobs(n_samples=250, 
                             centers=([2, 7.9], [8, 3]), 
                             random_state=0)

colours = ('green', 'blue', 'red', 'magenta', 'yellow', 'cyan')
fig, ax = plt.subplots()


for n_class in range(2):
    ax.scatter(data[labels==n_class][:, 0], data[labels==n_class][:, 1], 
               c=colours[n_class], s=40, label=str(n_class))
from neural_networks2 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2, 
                               no_of_out_nodes=2, 
                               no_of_hidden_nodes=10,
                               learning_rate=0.1,
                               bias=1)
    
simple_network.__dict__
Output::
{'no_of_in_nodes': 2,
 'no_of_hidden_nodes': 10,
 'no_of_out_nodes': 2,
 'learning_rate': 0.1,
 'bias': 1,
 'weights_in_hidden': array([[-0.38709697,  0.34216142, -0.53558474],
        [-0.38265064,  0.07216054, -0.19346557],
        [ 0.06640281, -0.22866835,  0.33205037],
        [ 0.06057826,  0.49377468,  0.2012993 ],
        [ 0.37061155, -0.12533196, -0.31096331],
        [-0.26451103, -0.18483328,  0.34717534],
        [-0.13736405, -0.11134647,  0.49381627],
        [-0.29948127, -0.09301347, -0.33229527],
        [ 0.52635705, -0.43268611,  0.36468496],
        [-0.02565338,  0.19563129,  0.02560361]]),
 'weights_hidden_out': array([[-0.2630956 , -0.28078972, -0.27659632,  0.28614888, -0.21546324,
         -0.11180555, -0.04695778,  0.27741456, -0.04321395,  0.24967042,
         -0.03368758],
        [-0.05675981, -0.28455935,  0.12732364,  0.07407335, -0.12645493,
          0.07372685, -0.23219729, -0.02740009, -0.03702381, -0.23802179,
         -0.28229358]])}
import numpy as np
labels_one_hot = (np.arange(2) == labels.reshape(labels.size, 1))
labels_one_hot = labels_one_hot.astype(np.float)

for i in range(len(data)):
    simple_network.train(data[i], labels_one_hot[i])

    
simple_network.evaluate(data, labels)
Output::
(250, 0)
data
Output::
array([[ 1.61267318,  7.59769725],
       [ 3.13940068,  6.66517418],
       [ 6.98195812,  2.92214524],
       [ 3.53277921,  9.36935877],
       [ 5.34082776,  3.60631952],
       [ 2.61407937,  8.82220667],
       [ 3.49407907,  7.69484174],
       [ 1.18663574,  6.43357567],
       [ 8.46313033,  3.27909576],
       [ 8.52106488,  2.42421203],
       [ 1.044055  ,  7.55401822],
       [ 2.14404357,  9.35427351],
       [ 7.43068795,  3.26990435],
       [ 7.34759142,  2.60904662],
       [ 7.91406923,  3.19429294],
       [ 1.13877431,  9.81006495],
       [ 6.06372019,  3.1887786 ],
       [ 6.30189418,  3.38728048],
       [ 7.60055097,  3.37005589],
       [ 7.95429604,  3.22050766],
       [ 1.56484645,  9.74926373],
       [ 0.38610215,  7.68725972],
       [ 7.4971833 ,  5.41245368],
       [ 7.28039561,  2.18700701],
       [ 3.17877957,  7.72007516],
       [ 0.82687659,  9.84362119],
       [ 1.40268393,  7.66207827],
       [ 8.14195316,  2.68067158],
       [ 2.29823817,  9.2263859 ],
       [ 7.95782855,  2.71311281],
       [ 8.27451636,  2.10908492],
       [ 1.48919486,  6.71936782],
       [ 8.82158571,  3.00529265],
       [ 1.30543214,  7.75036546],
       [ 7.27440262,  1.61663604],
       [ 3.76405235,  8.30015721],
       [ 8.77325298,  1.81611936],
       [ 2.3130677 ,  7.04590426],
       [ 7.01448926,  1.52816499],
       [ 7.48248096,  2.02117014],
       [ 8.81451982,  3.05892559],
       [ 6.95474663,  4.21114529],
       [ 2.20827498,  8.87663904],
       [ 2.39600671,  6.80693849],
       [ 4.25930895,  7.85774285],
       [ 0.74720464,  8.67749036],
       [ 1.32753955,  7.54044684],
       [ 2.40234164,  7.21518991],
       [ 9.12859406,  3.49600095],
       [ 7.63530646,  3.15670386],
       [ 6.87317419,  2.26932225],
       [ 7.03949562,  2.20688264],
       [ 2.01050002,  9.68587049],
       [ 8.33890413,  5.02104356],
       [ 1.68844747,  7.95616534],
       [ 3.18802979,  8.21694261],
       [ 2.72909056,  8.02898291],
       [ 2.85683061,  7.24897441],
       [ 3.48825219,  9.79588918],
       [ 2.17742614,  7.49821906],
       [ 1.58638102,  7.15254519],
       [ 0.68409259,  7.4384154 ],
       [ 7.79309632,  3.88017891],
       [ 3.92294203,  9.38051479],
       [ 9.64813493,  3.16422776],
       [ 7.9383736 ,  2.89269472],
       [ 1.09270164,  7.9519454 ],
       [ 1.36256297,  7.50272819],
       [ 7.53315445,  1.58309389],
       [ 8.04949498,  3.49383678],
       [ 7.86989305,  3.09395323],
       [ 8.28634369,  3.60884383],
       [ 1.65208785,  8.05634897],
       [ 2.3563664 ,  8.60657317],
       [ 1.32566734,  7.93183056],
       [ 2.46566244,  6.36375631],
       [ 9.31913688,  2.11758118],
       [ 1.3563816 ,  5.67659685],
       [ 2.12691209,  8.30198936],
       [ 7.72432947,  2.29027203],
       [ 6.15693045,  2.522026  ],
       [ 1.89678115,  8.3105985 ],
       [ 7.55907737,  2.7196445 ],
       [ 9.10028434,  4.29802197],
       [ 1.93175839,  9.61334272],
       [ 8.68981816,  4.30184623],
       [ 0.96575716,  8.58159452],
       [ 0.92924738,  8.95445173],
       [ 7.73226646,  1.87198867],
       [ 7.56081048,  3.18133843],
       [ 0.95144703,  6.47998206],
       [ 2.62523145,  6.29794234],
       [ 6.69347315,  4.65813068],
       [ 3.23029068,  9.10237985],
       [ 2.04575852,  7.71281615],
       [ 8.87583276,  2.88489253],
       [ 4.16323595,  9.23652795],
       [ 9.51999486,  4.71958931],
       [ 7.68911383,  3.09740017],
       [ 8.64331447,  1.42937659],
       [ 1.97181777,  8.32833187],
       [ 2.1666735 ,  8.53503144],
       [ 8.66638308,  2.53928021],
       [ 2.2799246 ,  7.80184961],
       [ 8.69153875,  3.69474914],
       [ 1.49034782,  7.4619257 ],
       [ 1.73199663,  8.7024564 ],
       [ 1.98297959,  8.27915174],
       [ 9.36453185,  2.31055082],
       [ 1.05555374,  7.48995031],
       [ 1.96071718,  6.7319065 ],
       [ 3.8831507 ,  6.55224094],
       [ 7.88945934,  4.02017271],
       [ 7.09123675,  2.57568238],
       [ 2.42625873,  8.57690804],
       [ 1.36415392,  8.57643329],
       [ 1.260437  ,  9.4430146 ],
       [ 9.73887268,  3.99439439],
       [ 0.50874241,  8.3393917 ],
       [ 1.50196755,  9.82953205],
       [ 2.91017891,  8.21721822],
       [ 1.45713852,  8.31605005],
       [-0.55298982,  8.5536186 ],
       [ 8.86896349,  3.27687191],
       [ 4.26975462,  6.44563433],
       [ 8.76666318,  3.35629282],
       [ 5.90539693,  3.12372191],
       [ 7.86629844,  4.07774381],
       [ 6.23146155,  3.35548179],
       [ 0.85253135,  7.46217996],
       [ 1.18685372,  6.1737174 ],
       [ 2.67229476,  8.30746184],
       [ 7.98431789,  3.16092817],
       [ 7.34144703,  2.48576603],
       [ 6.66574153,  1.65328249],
       [ 4.38314477,  8.84447949],
       [ 0.83485016,  8.80082649],
       [ 8.49374178,  2.88389606],
       [ 0.36980165,  8.36278226],
       [ 6.24410942,  3.45093446],
       [ 2.97873798, 10.1408932 ],
       [ 7.29529972,  3.94326072],
       [ 9.73272119,  3.68450111],
       [ 7.70816264,  2.23850779],
       [ 7.69088556,  2.76653334],
       [ 8.8005648 ,  3.07826018],
       [ 7.3159891 ,  4.6595508 ],
       [ 1.25524518,  7.07356146],
       [ 1.23008393,  8.43924919],
       [ 6.94537154,  3.82024784],
       [ 7.31216239,  1.7859226 ],
       [ 2.94725197,  7.74498991],
       [ 6.84264474,  2.68770775],
       [ 2.78632796,  7.4335809 ],
       [10.69622405,  2.92607533],
       [ 7.40134606,  1.88410301],
       [ 8.86259601,  0.34438091],
       [ 8.93184837,  3.33996498],
       [ 7.3477064 ,  2.47881069],
       [ 9.95591231,  3.39009332],
       [ 1.08717777,  9.01701629],
       [ 2.92085882,  8.21872765],
       [ 2.06651722,  8.2024719 ],
       [10.3039167 ,  1.93998418],
       [ 6.4170616 ,  3.61037938],
       [ 8.56729028,  2.7773249 ],
       [ 3.15233156,  8.97961859],
       [ 2.37642553,  6.80059921],
       [ 8.92950511,  3.58222459],
       [ 5.98359337,  2.46054537],
       [ 7.60477102,  1.84057948],
       [ 8.77140595,  4.02943883],
       [ 1.36567791,  7.53725883],
       [ 8.39904635,  0.22740724],
       [ 8.74718833,  1.81105504],
       [ 8.370825  ,  3.14206181],
       [ 0.77456448,  8.74436298],
       [ 8.84163126,  2.75054142],
       [ 5.74443577,  1.97749316],
       [ 0.89561666,  7.95216508],
       [ 0.57593909,  7.40668012],
       [ 0.84381757,  8.6811981 ],
       [ 2.57659082,  7.69170124],
       [ 8.1993002 ,  2.94939646],
       [ 3.86755896,  8.80604466],
       [ 6.5534653 ,  3.80029795],
       [ 5.96931553,  5.06449286],
       [ 2.15650654,  8.13218104],
       [ 9.51332808,  3.55313206],
       [ 7.81494633,  2.19235151],
       [ 8.03863055,  1.3432849 ],
       [ 2.64013153,  6.28304396],
       [ 7.52034419,  3.6203583 ],
       [ 2.44386323,  8.23367433],
       [ 3.49448454,  5.83001497],
       [ 1.97567388,  7.16196909],
       [ 8.69845715,  3.00377089],
       [ 7.84233298,  5.2567235 ],
       [ 3.12663592,  6.82006849],
       [ 2.15494743,  8.27816252],
       [ 9.46657872,  3.85255194],
       [ 1.86711942,  7.60220912],
       [ 8.52389102,  3.08842209],
       [ 3.86755799,  6.92272212],
       [ 7.61512019,  3.09435159],
       [ 5.71137996,  3.25148442],
       [ 8.5785215 ,  3.34965446],
       [ 1.64600609,  6.52504871],
       [ 2.95008842,  7.74864279],
       [ 7.88183595,  2.3198218 ],
       [ 1.11221425,  5.91920353],
       [ 0.729515  ,  8.86939671],
       [ 1.12920285,  7.32115034],
       [ 1.19659034,  7.21045022],
       [ 1.5444675 ,  7.91747916],
       [ 2.76103773,  8.02167502],
       [ 1.59682305,  9.12244507],
       [ 0.70714309,  8.16705087],
       [ 8.45741561,  2.03538799],
       [ 1.10453344,  8.2869025 ],
       [ 7.21737084,  2.8896107 ],
       [ 0.45920299,  7.96326199],
       [ 7.8640503 ,  4.13689136],
       [ 2.52327666,  7.72845367],
       [ 6.97006472,  2.65005664],
       [ 1.53640403,  8.38148147],
       [ 7.64656825,  1.38352581],
       [ 1.63081816,  7.66062082],
       [ 7.80934651,  2.60515049],
       [ 8.94304609,  0.26032283],
       [ 7.02889543,  3.3148172 ],
       [ 3.0996596 ,  8.55526373],
       [ 7.37191244,  2.51897288],
       [ 8.28044171,  2.00687639],
       [ 1.69098703,  6.22399619],
       [ 9.0685094 ,  2.5466142 ],
       [ 8.09772497,  3.58295368],
       [ 0.29372981,  9.8507754 ],
       [ 2.77179055,  8.72350415],
       [ 7.23585608,  1.56220853],
       [ 8.85792392,  4.14110187],
       [ 8.69377315,  2.84042656],
       [ 2.94942081,  7.98755124],
       [ 7.53113581,  0.79855871],
       [ 7.30795015,  4.53637705],
       [ 2.8644362 ,  7.15783498],
       [ 1.90154748,  7.23652171],
       [ 7.40368596,  2.9474327 ],
       [ 0.99978465,  6.3552289 ],
       [ 6.81114074,  2.49318365]])
In [ ]: