There were a bunch of terms that I found to be weakly defined or over-burdened by some historical meaning in the cog sci community. These included tabula rasa, nativism, associationism, behaviorism, etc. There have to be some good stories behind the way that the authors attack and cringe from those terms.
I was most impressed with the chapter on development and the incredible amount of flexibility and compression that the human genome has accomplished. I never realized that nature IS nurture to such a great extent. The way we just set up brain structures and then depend on our surroundings to generate the necessity of using them makes me wonder just how adaptable we are. Its impressive that we are the result of algorithms (genes) which design neural networks extensively depending on a highly chaotic universe to develope and tweak the system.
In both chapters the question of modules interested me. Mostly both chapters just raised questions about them, but even that was exciting. The quote 'parsing is a reflex' along with the statement that some basic principles of language are innate, even under the strictest definition, would seem to indicate that at least some modules are set up and raring to go as soon as the child is born. (A side note; have any studies been done of the parsing abilities of children who are deaf, ie do not receive normal stimulus, but then learn to write or sign later in life?) Modules are also very important because they seem to be used by the brain to perform very basic, yet difficult tasks very quickly, which is pretty much What we are trying to do with our little neural nets. Get a bunch of good modules together, add a little insanity, and you've got yourself a squirrel brain.
A final question. Which would benefit the attempt to model cognition more: a perfect understanding and model of human neurons, or a technological breakthrough which allowed us to use millions or billions of neurodes in our networks?
It seems remarkable that a simple recurrent network, when trained to predict the next word in a sentence, can develop representations of words, making such grammatical and semantic distinctions as noun/verb, human/nonhuman, etc. This result seems to point to the ever more apparent significance of temporal arrangement in mental processes.
How would such a network perform if implemented in an asynchronous fashion where neural processes did not occur in lockstep? Would it make still finer categorical distinctions between its input words?
Another interesting idea covered to some degree in the reading was that of plasticity. In particular, the relation of plasticity of a neural network to the nonlinear(sigmoid) activation function. The convention that the initial weights of the network are small and centered around 0.0 turns out to not be completely arbitrary after all. In fact, these small initial values cause neurodes to have similarly small inputs near the beginning of training, which in turn causes them to have more of a nonlinear response towards the beginning of the training as compared to their response later in training, when the values tend to be all-or-nothing. This aspect of neural network behavior was a striking analogue to the loss of plasticity with age in the biological nervous system.
I was really excited by the way this reading started off. The "Advances in Neuroscience" that they listed were very close to the things that I had thought of as most interesting about neurobiology, athough for some of the points, I wish they had given citations. The same goes for the points about connectionist networks: I thought, "Just what I wanted to hear!" and so I was interested to read what exactly connectionist networks did. I liked their angle, which seemed to be more grounded and respectful of biological orgins than the Anderson reading. Do they come from different schools of neural network thought?
One thing that was persuasive about the constraints that they use for their thinking about neural networks is that the same kind of constraints operate in overall developmental biology as well. Humans can not evolve wings, or a hand that begins at the elbow, because there are no bone support structures that would allow them to grow there. This is one of the main tenets of evolutionary theory. These structural restrictions do not inhibit significant change over many generations, but they do imply an "innate" limitation to change. Thus, it seems reasonable that there would be similar classes of constraints operating on the brain and therefore, in a truly connectionist network.
I wondered, while I was reading, whether connectionist theory would work for simpler animals than humans. Invertebrates, for instance, while they may have extrodinarily complex behaviors, tend to have specific neurons that initiate a specific complex behavoir. Thus a particularly stereotyped respose that needs to be quick (say the escape response of a lobster) will be initiated by just one neuron. The homologous behavior in humans (most akin, I guess, to the flight response/adrenaline rush) requires the interaction of many neurons. Could this one neuron = complex behavior be modeled in a connectionist model? In addition, such upper-level complexities as context senstive shading would be more or less meaningless in a simple animal. To me, it seems that if a network models brain activity in humans, simple modifications in number and NOT overall functional changes should imply equally accurate function for less complex organisms. And vice versa.
The thing about this reading was that it was suspiciously convincing - the writers are really well organized and have all the answers to all the questions laid out. I would say that even though I was pretty well sold on the idea of connectionism even before I knew the definition (in this reading), Im more convinced that this is the way to look at the modeling of neurosystems.
One of the more interesting points in the use of neural networks to represent data analysis is the representation of hidden units as tools of abstraction. For example, the recurrent example of this use of hidden units is in encryption, where a multidimensional input pattern is forced to be reduced to a lesser-dimensional hidden unit input pattern, and then converted back to the input dimension in the output layer. This forces the hidden units to crunch the input pattern into some sort of abstraction in order to still be able to convert it back to its original pattern in the output. What it means, however, is that a complex input can be reduced to a lesser size, using an analog system - in much the same way as perhaps the brain has to function in order to contain the vast amount of information it contains, and to be able to retrieve it so quickly and efficiently.
This abstraction is maybe better exemplified in another example in the book - that of the Elman experiment in which a network was trained to guess the next word in a given sentence, when trained on a number of grammatically correct sentences. The inputs were represented by (I believe) orthogonal vectors represented by a 32-digit binary number. The hidden units, in order to generalize the grammatical structure and to find the possible ways to use those words in a sentence, abstracted the input words into the hidden unit space, organizing them in clusters representing actual English grammatical atoms - nouns, verbs, then animate nouns and inanimate nouns, etc... This shows the power of abstraction in the hidden units - and gives us a glimpse into our own abstractions. One of the things about the mind that fascinates me is our ability to categorize things based on information we know - for example, if we see enough dogs of different breeds, at some point we realize that they are fundamentally the same animal (dog), with different instantiations - but with this information, if we see an elephant, although we know its not a dog, we know probably how the elephant locomotes, how it moves its head, that its not attached to that tree it happens to be resting against, etc...
Another thing, on an entirely different subject, that I found really interesting and really enlightening, because, you know, i guess I was asleep in high school bio when we covered highly advanced mutogenetical neuroscience, was the possibility of really small changes in one gene having large consequences in phenotype. Because of the redundancy of many genes ,if one small gene works in concert with other genes, in many different parts of the body, then a change in that small gene may cause a huge change in how it works with other genes in those many other parts of the body. This has really interesting consequences in neural modeling - suggesting that if we have many small KINDS of neurodes, that work really simply and very efficiently together, if one small part of one of those neurodes in changed (with the change in function happening in every place that neurode occurs in the system), then we may get a completely different result. This is part of the connectionist approach, obviously, with its emphasis on local knowledge having global consequences, but I think it is really interesting to compare to Genetic Programming - taking really simple algorithms, and combining them in genetically modelled ways, to form a working computer program. Then if one of those algorithms changed, so that every time it occurred in the program it executed in the changed fashion, the function of the program would change - maybe it would crash (the species dies), or maybe it becomes fitter - and I think these two fields can and should work more closely to discover exactly how they can benefit each other.
Once again, high school biology basically covered the introduction, this time of genetics. Genetics is chaos theory at it's best, tiny changes have enormous effects, and while we may create models that do similar looking things, or even slowly and painstakingly map out effects and changes (as in the case of those poor fruitflies), most of the time we can't figure out how it works.
What does normal environment mean? This could be difficult to study in humans, since cultural isolation is disappearing, it may soon be difficult to discern cultural environmental factors and environmental factors for which we are genetically pre-disposed.
Inateness as a system of constraints is an interesting idea. Can something which does not need to be learned necessarily not be learned? "Triggered" knowledge definitely seems to fall in a fuzzy area here. How much knowledge about the social customs of chimps (or whatever) do we have locked away in our heads but are never triggered? Or what if genetic "constrains" can be entirely overridden by experience? The option that a particular way of solving a problem can be the "natural" way, as opposed to genetically predisposed is an interesting one. Hard to prove, but interesting.
Intro to connectionist nets is good summary of Lisa's class last year, but not terribly fascinating for anyone in the course.
The lack of a homunculus is fairly counter intuitive, but makes perfect sense. Any conscious being probably has trouble accepting that their consciousness is the effect of the culmination of local phenomena and not a central entity at all. On the other hand, many complex systems follow such a model. For instance, the failure of communism mostly proves that attempting to apply a central strategy to an economy is usually disastrous.
As cool as the concept of reverse engineering nature, as they put it, is, the idea that even nature would design intelligence differently given another go at it makes it difficult to apply information about innate knowledge. For instance, if we have a model in which some parts of language are innate, it does not mean that it must be so or that it works that way in us. Likewise, if we identify a piece of knowledge in us which is innate, it does not mean that a model must make it likewise.
The first stuff was rather boring (the dna stuff) because I've grown up with DNA in the house. However, the thought that regulatory systems need a period of interdepent development and that this may slow development down was rather novel. The discussion of how representations are in the brain in terms of the mechanical setup was rather intresting I don't happen to agree with the partial knowledge stuff
well, whether or not i agree with it as an approach, connectionism is fascinating stuff. i found the characterization of theories about what things can be innate useful and well done, as with the summary of methods of learning in simple neural networks.
i don't really understand the significance of the repeated statement that "the relationship between the genome and the phenotype is nonlinear", which seems basically just to be saying that a smaller number of additional genomes than one would expect is responsible for increased complexity of function in higher organisms. if humans and chimps share 86% of their genomes, then maybe the basic behavioural and internal functions that both species share are complex enough in themselves that they require all of this material, and the extra "more complex" functions that humans have are relatively easy for a system to come up with, given that it already has enough complexity to keep itself alive.
they claim later that what makes the response of artificial neural networks interesting is the sigmoid (specifically "nonlinear") nature of the function used. is this really the case? my understanding of ann's is that their operation was fairly independent of the specific functions used, that is, that a straight-line if x < 0, f(x) = 0; if x > 1, f(x) = 1; else f(x) = x function would work just as well, and the sigmoid function's only advantage is that it's differentiable. anyone?
The first chapter may have seemed really cryptic at first, but I thought it did a really good job of providing a framework for making connections between the models and the actual biological constraints, observations and knowledge of biological systems that we currently have. In an area without true absolutes (as the non-mosaic regulatory development) the authors have provided a way of finding rules and organizing principles that are from the biological but apply equally well to connectionist computational models.
I especially refer to the constraints section here, as this was a section that truly peaked (sp?) my interest. I think that these constraints (other than the local constraints, which I think have been somewhat well explored) provide excellent paths for further research. For example, I'd like to see more systems taking advantage of more global architectural variation (possibly by interconnecting large sub-networks trained to specific tasks under an arbitrator network in order to allow for more complex behavior). Also this chapter again brings up the question of time . . .
The second chapter seems to be much more of a review, but it certainly summed up many of the best arguments for nn's as biollogical models.
I was pleasantly struck during the reading with the idea that despite the authors' obvious love affair with connectionism (not to mention that bizarre dialogue at the beginning of Chapter 2) they have a decently grounded view of the constraints they have to deal with. They also do a good job of trying to refute the argument that connectionism too thoroughly tries to model biology, and that the biological plausibility of connectionism is sketchy at best. (There is no known biological system which implements backprop, as they point out.)
They make the good argument that the connectionist approach provides a parallelism, nonlinearity, unpredictability, and interactivity that they think (and I think I agree) provides *a* basis for computer learning that follows some of the most basic things about human learning, even if we can't model down to the neuron or totally diverge from the bio model because we think we've got a more useful one. They've convinced me that connectionism is a good model to work with, if only because it has potential to grow into something much more than it is.
This reading was much more interesting to me than last weeks. I really enjoyed the explanation of connectionism and the careful sitting on the fence that was done by these particular connectionists, making sure that they kept the important features of all of behaviorism, innateness, biology, time concerns, etc..
I was highly interested with the difficulty of being "trapped" in weight space and have a possible solution. The solution depends on being able to find the global maximum. I would hope this to be easier than finding the global minimum. If the global maximum can be found, in a very time consuming (and perhaps the time consumption makes this almost pointless) way, the minimum of the space should be found. I refer to the question the other week in Claudil about getting stuck in a local minimum. One could not get stuck in a local minimum if they started from the global maximum (under the assumption of no friction), from a purely physics related standpoint. You can always make it back up to the "height" you started from. Therefore, if you start at the global max, you should be able to reach the global minimum. I am curious if this idea could be employed in searching the weight space. I'm not certain exactly how it would work though.
One I the key points I feel is raised by this reading is the balance between innateness and plasticity. It's possible -- and has been shown to be the case -- that an organism have many possible paths of development, but that, in the absence of factors which directly prevent it, a given final organization is preferred over others (in the sense of occurring more often). This is a point which often gets lost in louder arguments about innateness. It's easy to ask whether a given trait is "innate" in the sense (rejected by the authors) of "predetermined and fully fixed," or "not innate" in the sense of "unprededetermined and freely variable." Yet, most traits of real organisms, and of complexly functional systems in general, are variable, but not entirely freely.
The authors do a good job of pulling the very idea of "innateness" out of its black box and elucidating what it might mean. I found the section on the different possible levels of innateness very interesting, and I agree with the observation that even the early connectionists, with their claims of having escaped innateness, made certain ideas implicitly innate in their models by their choice of architecture. I'd be very interested in looking at the models were investigating in terms of what ideas are, in fact, innate in them as the result of their architectures.
One of the concepts and themes presented throughout chapter one which I particullarly enjoyed reading about was the authors application of the concept of nonlinearity to the entirety of human development. One of the things they said was that we traditionally percieve human development as a linear path, with all steps on it predefined, and I think that I fell into that category, as well. It is much harder to concieve of development as having multitude of various bits and pieces all of which sort of vaguely interact to produce a human being than it is to imagine a gene for blue eyes and mean disposition, and on the surface the linear approach is much simpler and thus appeals to Occams razor. However, even the brief overview of the molecular aspect of human development seems to make much more sense to me now than it did before. (Admittedly, I didn't know all that much about it, and still don't.)
The other thing that appeals to me about their developmental discussion is that it appears to explain the brain, behavior, and the rest of the body, all in one go. The authors claim it is the complex interaction of slight genetic "guidance" in concert with the input from the environment which defines physical development, but that is EXACTLY the same thinking which is done to describe learning in a neural network; i.e., slight structural influence with much input from the environment. It seems to me the fact that one understandable theory that can explain most of our development is likely to have some bearing on the reality of the situation.
the book says elman on the cover and whatever else i have read by that fella was very informative, organized, and interesting. this generalization also applies for this text, which sprung from a collaboration of elman with various others from the field of connectionism. sometimes, reading texts does leave the reader with questions, but elman et al. seemed to have mastered the art no to do so, at least in those two chapters, at least for me. then again, i could ask many questions, but these would not necessarily be about the text but more about other topics that were stimulated by the text in my connectionist brain.
I was yet once again amazed at the close parallels between connectionism and nature, which was especially emphasized when the advances in connectionism were described. obviously, many theories and half-theories exist which attempt to describe the brain's operation, but connectionism is so simple, beautiful, logical in a mathematical sense, practical, versatile, and real, and unlike the other theories, which have been majorly overthrown by 'does-not-work-statements,' connectionism, although moving on slowly, has been able to withstand such attacks. it was very interesting to read about genes. most of what was included in the text was not completely new to me, but in general, the text enhance the clarity i had on those issues (a bit). that chimps fo resemble humans closely was obvious, but that 98.4% of our genes are in common is incredible, in the very semantic meaning of the word; how does one go about to measure this? can we trust this figure?
the second section on neural nets was probably well written too, but i skimmed most of it. the essence has been covered in previous cognitive science and psych classes, or in lisa's ai class. and i did some external reading on those issues too. elman's word prediction experiment is fascinating every time i read about it, although dear mr. durgin took great care that we would have studied this one as inside-out as possible. the reading, aside being very long, provided some additional info required to fill in some gaps in my collection of connectionism. and it emphasized even more that, naive as i am on this field--knowing only a slice of the whole thing--i love connectionism! lastly, it was a reading, that should have been interesting to a wider group of people than last week's--right martine?