UNDERSTANDING ARTIFICIAL NEURAL NETWORKS

NATURAL NEURAL NETWORKS

A natural neural network is composed of connections between neurons. The human brain has approximately 86 billion neurons, a chimpanzee has approximately 28 billion, a honey bee has 960 thousand neurons, and a nematode, a living creature 1 millimeter long and 65 micrometers thick, has 320 neurons.

The structure of a typical neuron is shown in the figure.

Axon terminals are connected to the dendrites of other neurons. The connection points are called synapses, and the entire set of connections is called a synaptic connection. The structure of a synaptic connection is shown in the figure below.

There are electrochemical interactions between axons and dendrites. When the change in ion levels exceeds a certain threshold value, an electron flow occurs from one side to the other. Electrons, or electric current, create our movements, thoughts, and all our actions. When we raise our arm, electrons cause muscle cells to contract, when we dream, flows occur from some neuron groups to others, hormones are released, and so on.

When scientists observed the structures of neurons that we have summarized above, they thought that it could be modeled mathematically. After a system is modeled mathematically, it can be imitated in different artificial structures. This is how the foundation of artificial neural networks was laid.

The mathematical model above means the following:The information received from the inputs (dendrites) is multiplied by certain weight ratios called W (chemical levels in the dendrite channels), summed (enters the neuron body), and then enters an activation function called f (chemical structure of the axon), and then the output.
In the natural structure, the amount of electrons produced at the axon output corresponds to the concept of output in our model.

Now we will solve a problem that is a product of our mind, that is, a product of natural intelligence, using an artificial neural network that we have modeled as above.

IMPLEMENTATION OF XOR LOGIC GATE WITH ARTIFICIAL NEURAL NETWORK

The input (x1,x2) and output (y) connections for the XOR logic gate are as follows:

We need to design an artificial neural network with two inputs and one output. For example, something like this:

Here, X1 and X2 are the input neurons, A, B, and C are the intermediate layers (also called hidden layers), and O is the output neuron. w1-w9 are the weight multipliers. In an artificial neural network, the inputs and weight multipliers are multiplied and summed to create the output values. There is no other trick to artificial neural networks. In fact, the basic working structure of the brain is not much different. However, when this structure consists of thousands and millions of elements, it creates amazing things.

Our main goal is to find such mathematical multipliers that the system will give the appropriate "XOR output" value for each "XOR input" value applied to the input.

In the first stage, we take the weight multipliers randomly because we do not know what they are. Our goal will be to find their values.

The value of x1*w1 enters neuron A, and the value of x2*w2 also enters neuron A. In other words, the product of the input and the weight is the input of the corresponding neuron. For example, according to our first input value, neuron A:

A = x1*w1 + x2*w2 = 0 * 0.1 + 0 * 0.2 = 0

Each neuron other than X1 and X2 has an activation function. The input value is the input of this activation function, and the output of the function is reflected in the output of the neuron. Different activation functions can be used. One of the most common is the sigmoid function.

The sigmoid function is 1 / (1+e^(-x)) and its curve is as follows;

 

Now let's see the output value produced by each neuron for the first input value, that is, for (x1,x2) = (0,0)

For the output neuron O, the input value is:

O-input = sigmoid(A)*w7 + sigmoid(B)*w8 + sigmoid(C)*w9
= 0.5*0.7 + 0.5*0.8 + 0.5*0.9 = 1.2

Since the output of neuron O is the sigmoid value of the input value,
Output = sigmoid(O) = 0.7685248

To summarize what we have done so far, we found an output value of 0.7685248 for the input value (x1,x2) = (0,0) and the randomly determined weight values w1-9.

Similarly, let's write the output values we found for each input pair and the differences between our findings and the expected values, i.e. the errors:

 

We need to find new w values that will minimize the error at the output, but how will we find them? Different algorithms can be used for this task. In the method we will use, we will take the derivative of the activation function. The derivative will give us the rate and direction of change. By multiplying the output error with this derivative, we find in which direction and by what amount we can increase or decrease the w weights. By doing this repeatedly, we find the most suitable weight values. This method is called backpropagation. Let's see step by step how this is calculated for the output neuron:

Input value of the output neuron = w7*Aout + w8*Bout + w9*Cout

Output value of the output neuron = sigmoid(Input value of the output neuron)

Error = Output value of the output neuron - desired output value

Delta output = SigmoidDerivative(Input value of the output neuron) * error

We can also call this "global error".

We need to find new weight multipliers based on the error. For example:

New w7 value = Old w7 value + Delta output * Aout

w1-w6 weights can be calculated with the same logic:

New w1 value = Old w1 value + x1*(SigmoidDerivative(Ain)*Delta output * old w7)

In this way, we find the new weight multipliers w1-w9. For each w, we do the same calculations with the corresponding input and output neurons. If we apply the new multipliers to the system, we will see that the output values approach our desired outputs.

As you can see, the outputs are always evolving towards the desired value. If we do this process 10000 times...

 

After 10,000 corrections, the values seem to be very close to the desired values. After 10,000 corrections, the w1-w9 weight multipliers calculated for the values (x1,x2) = (1,1) are as follows:

(1.251617 1.351617 0.7538233 0.8538233 7.1306686 7.2306686 -2.1630681 -2.2256228 -1.628696 )

Initially, we set the weight multipliers randomly as w = [0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9]. We performed error correction 10 thousand times and found new weight multipliers. When we replace the newly found weight multipliers in our system as seen in the figure below, it gives an output value of 0.0040834 for the input values (1,1). It should have been 0, but we approached this value with a very small margin of error.

This is how artificial neural networks basically work.