Computational Models of Language
Spring 2001
Homework 1: Experimenting with Pattern Associators
Due: Saturday, February 3 by midnight

GETTING STARTED

Read chapter 3 in the modeling book and try the exercise described there.

DESCRIPTION

For this homework we will build a connectionst network that associates the image of a digit represented on a 6x6 grid with a label. For example, here is the image we will use for the digit two:

0 1 1 1 0 0
1 0 0 0 1 0
0 0 0 1 0 0
0 0 1 0 0 0
0 1 0 0 0 0
1 1 1 1 1 1

0 1 0 0 0 0 0

INSTRUCTIONS

We have created the following files to help you get started: digits.cf, digits.data, and digits.teach. Copy these files into the Chapter 3 folder for the tlearn software and then open the digits project.

Under the "Network" menu, check that the "Training Options" are set to 1000 training sweeps, a learning rate of 0.1, a momentum of 0, and seeding randomly. Then train the network. Under the "Displays" menu, select "Node Activities" and then push the "Next Data Pattern" button repeatedly to see if the network has learned the correct associations. The output nodes should each be on in order starting from the left.

1. Examining Connection Weights
   Now let's look at how the network managed to accomplish this task. Under the "Displays" menu, select "Connection Weights". There are 36x7 weights in this network and they will all appear here as a matrix of rectangles. The size of a rectangle reflects the magnitude of a weight while the color reflects the sign--black means negative and white means positive. Look carefully at some of the weights with the greatest magnitude. Try to figure out why they are important by looking back at the digits.data file. Which places in the 6x6 input grid has the network paid particular attention to? Explain why.

2. Lesioning the Network
   Now let's test the fault tolerance of the solution by gradually removing connections from the network. Under the "Special" menu, choose "Lesioning". Then select the weight file that resulted from the earlier training. It should be digits-1000.wts. Check the "Connections" box and choose 10% removal. Then delete anything that appears in the "Location" box. Finally push the "Lesion" button. A "Save file as" box will appear; call it digits-10%-lesion.wts. In order to test how this loss of connections has affected the network, under the "Network" menu choose "Testing options". For the weights file, use the one we just created. Now bring up the "Node Activities" display again. Does the network still recognize each input pattern?

   Continue lesioning the network by 10% more until you see a marked decrease in performance. Give each new weight file a unique name based on the amount of lesioning done. To show your results, create a table with two columns: one for amount of lesioning, and the other for the number of correct patterns.

3. Noisy Input
   Now let's see whether the network can generalize. Duplicate the file digits.data and call the copy noisy.data. Edit this new file. For each input pattern, randomly change three of the 0's to 1's and three of the 1's to 0's. This will add about 17% noise to each pattern. Under the "Network" menu, choose "Testing Options". The weight file should be set to digits-1000.wts and the testing set should be set to noisy.data. Now bring up the "Node Activities" display again. Does the network still recognize each input pattern?

   Create a second copy of the original data file. Based on what you saw when you looked at the connection weights, try flipping only the bits that seem to be most significant to the network's solution for each digit. Describe the results and compare to the results obtained with random noise.

TURNING IN YOUR ANSWERS