Neural Network From Scratch with C++ — Part 2

In the last part, we defined two classes: Neuron and Axon, as well as two activation functions: sigmoid and sigmodPrime. Now, we need to implement two core methods of the Neuron class: feedForward and backpropagation.

Feed Forward

“Forward propagation is the process in a neural network where the input data is passed through the network’s layers to generate an output” — geeksforgeeks.org

So basically, we need to calculate the output based on the input nodes using predefined weights and biases, then subtract this from the actual output values in the dataset to determine the error slope. This error slope is then used in backpropagation to adjust the weights and biases.

First Layer:
We multiply the input nodes by the weights and then perform a column-wise sum with bias to produce the output of the first layer.

The number of columns in the weight matrix must match the number of input nodes, and the number of rows in the weight matrix determines the number of nodes we want in the first layer.

This will be the result matrix:

$$\begin{bmatrix} Oa1 & Ob1 & Oc1 & Od1 & Oe1 \\ Oa2 & Ob2 & Oc2 & Od2 & Oe2 \\ Oa3 & Ob3 & Oc3 & Od3 & Oe3 \\ Oa4 & Ob4 & Oc4 & Od4 & Oe4 \\ Oa5 & Ob5 & Oc5 & Od5 & Oe5 \\ ... \end{bmatrix}$$

However, before we pass this result to the next layer, we need to activate it by applying the sigmoid function to each element.

Output Layer:
Since we want three nodes as the result setosa, virginica, and versicolor the number of columns in the output should match the number of columns in the first layer’s matrix, and the row should have 3 row.

Then, after obtaining the result, we should activate it again using the sigmoid function. Finally, we just need to implement the code based on the calculations above.

void Neuron::feedForward(Node inputNode, std::vector<Cell> &isolatedCell) {
    Node node                                           = (inputNode * isolatedCell.at(0).weight) 
                                                                         + isolatedCell.at(0).bias.replicate(inputNode.rows(), 1);
    isolatedCell.at(0).node                     = node.array().unaryExpr(&sigmoid);

    for (int cellIndex = 1; cellIndex < isolatedCell.size(); cellIndex++) {
        node                                                    = (isolatedCell.at(cellIndex - 1).node * isolatedCell.at(cellIndex).weight)
                                                                            + isolatedCell.at(cellIndex).bias.replicate(isolatedCell.at(cellIndex - 1).node.rows(), 1);
        isolatedCell.at(cellIndex).node     = node.array().unaryExpr(&sigmoid);
    }
}

After we have all the results (Hidden Nodes, Weights, and Biases), we need to subtract our output results from the actual output nodes in the dataset to obtain the error slope. Finally, we adjust the weights and biases using backpropagation.

Backpropagation

Now that we’ve implemented the feedforward process, backpropagation is where the real learning happens. By evaluating the errors between predicted and actual outcomes, backpropagation adjusts the weights and biases to improve the model’s performance.

Let’s explore how backpropagation optimizes the learning process and enhances the effectiveness of our neural network.

Output Layer:
We start from the output layer and work our way to the front. First, we need to count the error slope by substracting output node from dataset with output node from feedForward.

After that, we apply sigmoidPrime to the output node from the feedforward, then perform a coefficient-wise product with the error slope. This results in a new delta, which will later be used to calculate the error slope in the first hidden layer.

First Layer:
To get the error slope in this layer, we multiply the delta by the weight from the output layer. Then, just like before, we apply sigmoidPrime to this node and perform a coefficient-wise product with the error slope. This gives us the delta for this layer.

This is what the code would look like:

void Neuron::backProp(std::vector<Cell> &isolatedCell) {
    int cellIndex                                       = isolatedCell.size() - 1;
    Link errorSlope                                  = this->learningAxon->getOutputNode() - isolatedCell.at(cellIndex).node;
    Link delta                                            = isolatedCell.at(cellIndex).node.array().unaryExpr(&sigmoidPrime);
    isolatedCell.at(cellIndex).delta     = errorSlope.cwiseProduct(delta);

    cellIndex--;

    while (-1 < cellIndex) {
        errorSlope                                          = isolatedCell.at(cellIndex + 1).delta * isolatedCell.at(cellIndex + 1).weight.transpose();
        delta                                                   = isolatedCell.at(cellIndex).node.array().unaryExpr(&sigmoidPrime);
        isolatedCell.at(cellIndex).delta    = errorSlope.cwiseProduct(delta);

        cellIndex--;
    }
}

And that’s it—the implementation of feedForward and backpropagation. We still need to implement one more function, which is linkAdjustment, but I’ll cover that in the next part.