CS63 Spring 2004
Lab 4: Using Conx to create neural networks
Due: Friday, February 20 by 11am

CONTENTS

• Introduction
• Lab Instructions
• What to hand in

1. Conx is one module of the Pyro project. Pyro is an open source project, so you are free to go look at the implmentation of Conx at any time. It is available on our system at:

   /usr/local/pyro/brain/conx.py
   

2. Before using Conx you'll need to set the PYTHONPATH variable in your shell. For bash shell, add the following to your .bashrc file:

   export PYTHONPATH=/usr/local:/usr/local/pyro
   

   Then you'll need to execute your updated .bashrc by doing:

   source .bashrc
   

   Any new shell you open will now include this environment variable.

3. There is a collection of materials available online to help you learn Conx.
   • Start with this brief introduction to Conx.
   • You may want to review the material on the basics of neural nets.
   • Focus more closely on how to build networks in Conx. Do all of the exercises in this section except #2.
   • Try the auto-associative and recurrent network examples. These two examples are very similar to the experiments you'll be reproducing for this lab.
   • You may want to refer to the summary of Conx methods.

4. Using the auto-associative network example as a model, reproduce the compression experiment described on pages 106-7 of Mitchell's book. You'll need to create a network with 8 inputs, 3 hiddens, and 8 outputs. The goal of this network is to output the same pattern that appears in the input. Test various network parameter settings.
   • Once you have settled on some reasonable settings, run the experiment three times and describe the results.
   • Does each network come up with the same hidden layer representations?
   • Are the hidden layer representations similar to the result described by Mitchell?
   • Can this task be learned with only 2 hidden nodes? Try this and describe the results.

5. Using the simple recurrent network example as a model, reproduce the letter sequence experiment described on pages 7-9 of Elman's article. You'll need to create a recurrent network with 6 inputs, 20 hiddens, and 6 outputs. The goal of this network is to predict the next input based on the current input and its context memory of the recent past.
   • Create a training set that contains 1000 different sub-sequences as Elman did.
   • Using an epsilon of 0.25 and a momentum of 0.1, train your network on this data for 20 epochs. Elman states that he used 200 epochs, but after only 20 epochs the network should be performing pretty well. Save the weights to a file, so that as you work on the testing phase you don't have to re-run the learning phase each time.
   • Create another program to run the testing phase. It should re-create the network, load the saved weights, set up prediction, set the inputs to a short test sequence, set learning off, and set interactive on. You'll also need to setLearnDuringSequence on, even though we aren't doing learning, because this ensures that the hidden activations are copied to the context layer as the sequence is processed by the nework. Finally you can call sweep.
   • Now we'd like to create graphs like those in Figures 4 and 5 of Elman's article. To do this we'll need to calculate the overall error for each pattern as well as the individual error on particular output units. Unfortunately the interactive mode with the sweep method does not report error. So let's create our own SRN class and re-write the sweep method.
   • In your testing program, create a new class, as shown below, which is a specialization of the SRN class.

     class MySRN(SRN):
         def sweep(self):
     

     Copy the sweep method from the SRN class definition in the conx.py file. Make sure you don't get the sweep method from the Network class by accident.
   • Add the following lines to the sweep method after the print statement with the message "After propagation................"

     unitErrors = map(abs, self.getLayer('output').error)
     sumError = 0
     for index in range(len(unitErrors)):
         sumError += unitErrors[index]
         print "Unit", index, "error:", unitErrors[index]
     print "Total error", sumError
     

   • Now when you run your test program you can see that the network has learned which of the individual units are predictable (0, 1, 2, and 5) and which are not predictable(3 and 4).
   • You can save these values into individual files to make them easier to import into your favorite graphing tool. You can label the points by hand if you'd like.