How do Neural Networks Learn?
We have already seen
- Each neuron can accept inputs and provide output through synapses.
- Each synapse has some weight
- Neuron calculates the weighted sum of all inputs and associated weights and then activation function decides its output.
In this process, data is only flowing in forward direction i.e. inputs applied to neuron, neuron performs certain calculation, then activation function triggers the output. However, once output is done, there is no way to improve its accuracy. We can improve it once neural network learns the optimal way for predictions. The question still remains “How do neural networks learn?”.
The Cost Function:
A commonly used cost function is
C = ½(y’-y)2
- y’ is predicted value for input variables
- y is actual value from test dataset
- C is the cost function value.
Now, again inputs provided to neuron, it calculates the weighted sum. Activation function is applied to neuron and it predicts output value. This prediction is compared with actual value and cost function is calculated. Still we want to know “How do neural networks learn?”.
Now let us use the cost function we calculated.
- Cost function is fed to neuron and weights are adjusted.
- Again, neuron calculates weighted sum with updated weights.
- Activation function is applied to neuron and it makes another prediction for same data points.
- Then again, cost function is calculated from new prediction and actual value and it is fed back to neuron.
- The process goes on until optimum cost function i.e. having value closest to zero is obtained.
A whole neural network can learn just like a neuron learns. It decides optimum cost function based on gradient descent or stochastic gradient descent. However, we will learn about them in different article.