Lab 5: Neural Networks
Due Wednesday, October 7th by 11:59pm

Learning goals

• How artificial neurons compute their activation value.
• The process of forward propagation of activation in a neural network.
• How error is computed at the output layer of a neural network.
• The process of backward propagation of error in a neural network.
• How training proceeds in a series of epochs until convergence is reached or a max number of epochs has been completed.
• How a learned set of weights and biases solve the XOR problem.

Starting point code

You should have a github repo ready to go by the time lab starts. Please let the instructor know if you have any issues accessing it.

Introduction

There are many good sofware packages that you can use to experiment with neural networks, such as keras/tensorflow, PyTorch, skikit-learn or aitk. All of these packages use a form of backpropagation to train their neural network models. It is important for you to understand how backpropagation works at a detailed level. Therefore, in this lab you will implement your own neural network from scratch.

Your neural network will be made up of a number of classes, which have been defined in the scaffolding:

• Network
  A network stores a list of layers.
  A layer stores a list of nodes.
• Node
  A node stores:
  • list of incoming edges
  • list of outgoing edges
  • activation value
  • delta
  There are several specialized types of nodes:
  • InputNode - activation set from input example
  • BiasNode - activation always 1
  • SigmoidNode - activation computed using logistic sigmoid
• Edge
  An edge stores:
  • source node
  • destination node
  • weight
  • old_weight

  Neural Network Implementation

  As you are working on your implementation, you should frequently try the provided unit tests:

    python3 nn_test.py
  

  There are 16 unit tests in this file. At the very top of the output that is produced, are 16 characters summarizing your progress. An E indicates a runtime error, a F indicates a failure, and a . indicates a success.

  Below is a short summary of the tasks you need to complete. Note that much more detail about all of these methods can be found in the comments within the file neural_net.py. Higher-level descriptions and explanations can be found in the lecture notes.

  1. Initially you will focus on implementing the SigmoidNode class methods:

     • compute_activation: weighted sum of inputs passed through the logistic sigmoid function, g(z) = 1 / (1 + e^-z)
     • compute_output_deltas: (output)(1 - output)(target - output)
     • compute_hidden_deltas: (output)(1 - output)(weighted sum of next-layer deltas)
     • update_weights: weight += (learning_rate)(delta)(input activation)

     The first three functions store their results in the node's self.activation and self.delta fields. update_weights changes the weights of the edges stored in self.in_edges.

  2. Next you will work on completing the Network class. It is already set up to initialize a neural network with input, bias, and sigmoid nodes. You must implement the following functions for the neural network:

     • predict: computes the network's output for a given input, by calling compute activation on each successive layer (this implements the forward propagation of activation)
     • backpropagation: updates the network's weights using the error between the most recent output and the target
     • test: computes the fraction of inputs for which the output matches the target
     • train: calls predict then backpropagation on each example in the (shuffled) training set until convergence or max epochs (see lecture notes for pseudo-code)

  The main function sets up a network with a single two-neuron hidden layer and trains it to represent the function XOR.

  Once you can successfully learn XOR, move on to the next section.

  Explaining your XOR network

  In the latex document provided (called explanation.tex), fill in the diagram of the XOR network with the specific weights that were found by your program. Then explain how your network has solved the XOR problem. In particular, you need to clearly articulate:

  • when each hidden node will be active
  • how the activity of the hidden nodes combined with the weights from the hidden nodes to the output nodes produces the correct output

  It is important to note that this should not just be an exhaustive list of what input will produce what output, but an explanation of how the particular weights that were learned relate to human-understandable concepts such as logic operators.

  To compile your latex document into a pdf, run the following command:

  pdflatex explanation.tex
  

  Or if you prefer, a Makefile has also been provided. Once compiled, you can then open the pdf from the command line with the command:

  evince explanation.pdf
  

  Note that evince will only work if you are logged in physically; if you are working remotely, you may want to use the PDF-viewer plugin for vscode. You can also set up X-forwarding to get evince to work over an ssh connection.

  Be sure to include your name(s) in the title of the document.

  Submitting

  As usual, use git to add, commit, and push your work.