Lab 7: Convolutional Neural Networks

Lab 7: Convolutional Neural Networks
Due March 31 by 11:59pm

Starting point code

Use Teammaker to form your team. You can log in to that site to indicate your partner preference. Once you and your partner have specified each other, a GitHub repository will be created for your team.

Introduction

In this lab you will be using Keras and aitk to explore convolutional neural networks. You will start by experimenting with an already constructed convNet to learn the MNIST dataset. Once you understand how to use Keras and aitk to construct, train, and test a convNet, you will create your own convNet to classify fashion images of clothing.

Relative to all of the previous labs in this class, much of the design of the lab is left up to you. This is intended to start to prepare you for your upcoming AI project. You will likely be using some type of publicly available software, like Keras, to complete your project. You should get used to consulting the documentation when you have questions. I am also willing to help, of course.

Using Keras and aitk

Before using Keras and aitk, you need to activate a virtual environment that contains all of the software packages that you'll need:

source /usr/swat/bin/CS63env

To leave the virtual environment, just type:

deactivate

We will be using Jupyter Lab to enable lots of interactive experiments with the deep networks. Once you have activated the virtual environment, you should cd to the directory containing your repo for this lab and then you can start up jupyter like this:

jupyter-lab

To leave jupyter shut down the kernels and close out the browser window. You may also need to press CTRL-C in the terminal window where you started jupyter.

Remember, we used Jupyter for the first two labs, and there are more detailed instructions (including remote access) on the lab 0 page. You can also look at the Notebook for lab 0 for reminders about the Notebook format.

Classifying handwritten digits

Within jupyter, open up the file digits.ipynb and read through the notebook. Doing deep learning will typically require the steps listed below. Take your time and try to understand how each of these steps is being handled for learning the MNIST dataset.

Prepare the data
This may include loading, reshaping, normalizing, and dividing the data into separate training and testing sets.
Construct the model
We will be using Sequential() models. Add the layers in order from input to output.
Compile the model
Specify the optimizer, loss, and metric that will be used.
Train the model
Determine the number of epochs to complete.
Test the model
After training is complete, analyze the results.

Typically the network's performance will not be as successful as you would like. To improve the performance you will need to iterate over these steps multiple times. Here are some ways to consider adjusting the deep learning network:

If loss is still dropping, you may just need to train the model for more epochs.
Tweak the model architecture, by adding more layers, or changing the parameters within existing layers, such as the number and size of filters.

For convolutional layers, be sure to use odd and symmetric filter sizes, e.g. 3x3, 5x5, 7x7
Compare what happens as you increase/decrease filter size, or number of filters
Compare what happens if you add extra convolutional layers
Compare what happens if you add extra pooling layers
Compare what happens if you add extra fully-connected layers, and/or change the number of nodes in them
As you make modifications, notice how the total count of 'trainable parameters' changes

Try a different optimizer method (e.g. SGD for the base version we used last week, or Nadam for a different version of momentum).
Try modifying the hyperparameters of the optimizer (e.g. learning_rate, etc.)
Try a different activation function (e.g. sigmoid instead of ReLU) for one (or more) hidden layers (though remember that for deeper networks, this may make things worse rather than better).
Consider adding dropout layers between hidden layers to improve generalization and reduce overfitting; experiment with different percentages of 'likelihood of dropping a node'.
You may also consider adding a 'bottleneck' to combat overfitting; this means having a layer with a smaller number of nodes, to force the network to learn more general patterns. This is most commonly done just before the output layer, but you can try doing it elsewhere to see what happens.

NOTE:It is NOT recommended that you change the activation functions or loss functions being used at the output layer.

For more details about the various Keras functions and their options, see the documentation:
https://keras.io/api/
(or just google for 'keras [function-name]', and the API docs for that function usually show up near the top).

Designing neural network architectures and choosing hyperparameters is a difficult task, and is often described as being as much 'art' as 'science.' This is due to the fact that not only are there a lot of choices, but also that the impacts of those choices are not independent. In other words, the 'best' choice of optimizer might depend on the number of hidden nodes, so you can't just tune each thing separately.

That being said, there are still often general trends that you can discover by experimentation. The best approach for a lab like this is to start off with 'play,' meaning just trying stuff out to discover what happens. Then, once you've started to get a sense for what the options are and what they do, you can start being more careful and strategic with your experiments to try to find a configuration that produces a result you're happy with.

There are some methods for 'automatic' hyperparameter tuning, such as grid search or Bayesian hyperparameter search, but those tend to be extremely time consuming, since they involve running very large numbers of experiments. We won't be using them for this lab, but you could try exploring one as part of a final project if you like.

Once you feel that you understand what is happening with this network to learn to classify handwritten digits you can move on to classifying images of fashion apparel.

Classifying fashion

This data set is very similar in format to the MNIST data set. It contains the same number of patterns and is in exactly the same format. However, it is more challenging because there is greater variation within each category, and the boundaries are not as clear. The categories are as follows:

0: t-shirt
1: trouser
2: pullover
3: dress
4: coat
5: sandal
6: shirt
7: sneaker
8: bag
9: ankle boot

In the collection of images below, for each category, there are three rows of example images.

Using jupyter, open up the notebook fashion.ipynb. Experiment with this data and try to find the best network you can to learn this data. Remember that your goal is to generalize as well as possible on the test data, though you should also keep in mind that you may not perform equally well on all categories. Note that this problem is harder than MNIST, and you will not be able to achieve as high an overall accuracy; do the best you can. Once you are satisfied with your network, write up what you did.

Writeup

In the file writeup.tex:

Describe the neural network you built that showed the best validation performance on the test data. Use screen shots of both the network summary and the training graphs to support your explanation.
Analyze how the network has used its convolution and/or pooling layers to find helpful features for classifying these images. Use screen shots of the feature maps to support your points.
Report on which categories within the fashion data proved to be the easiest and hardest to learn. Speculate about why this was the case.
Be sure to describe and analyze both quantitative results (e.g. accurary, confusion matrix, etc.) and qualitative observations (e.g. what the filters seem to have learned, your observations of examples the network is getting wrong, etc.).

You can edit the writeup.tex file with any text editor. There's lots of great help available online for latex; just google "latex topic" to find lots of tutorials. To compile your latex document into a pdf, run the following command:

pdflatex writeup.tex

You can then open the pdf from the command line with the command evince. Feel free to use services like sharelatex to edit your latex file. Be sure to include the final PDF version of your writeup in your repo.

Submitting

You should use git to add, commit, and push any files that you modify.