I found the reading pretty intresting, esp the stuff about the two systems that are involved in the chicks. The idea that the two networks are there, and that the initial one primes the later one without any direct interaction was pretty cool. Its a rather elegant solution to have one system get initilized by simply watching the behvior of the chick while the first system is active.
The stuff about why development takes time is also intresting. It seems to indicate that learning as infants is different than learning as adults because the brain is physically different and that infants can't learn some connections because their brain can't make the physical changes. It may explain the child prodigies, just that their genes are a little out of sync so the timing of the brain development is earlier.
Have any of the studies mentioned been done on child prodigies? it might be intresting to see what thier brains look like, if they have more child-like or more adult-like brains.
I found this reading very interesting because while I strongly questioned some of the assumptions that they made in Chapter 6, I mostly agreed with their conclusions, as delineated in Chapter 7.
One of the things I was thinking about while reading this chapter was how one could incorporate natural selection into a neural network. The authors are fond of the idea that nature doesn't "specify" everything, but rather that external, architectural constrains end up forcing things to happen in a way that looks very complicated. Ie., the geometrical shape of the honey cells in a bee hive is a result of a bunch of circles being tightly packed. I support this concept in general -- I think that there are many things in nature that look highly adaptive but are, in fact, more or less results of pre-existing structure. However, I don't think that the way this happens is really analagous to the way that a neural network might end up arriving at a solution. In essence, while a neural network may reasonably inact ontogenetic development, I don't think that they have made a case for the generation of phylogenetic development, and, at the same time, they aren't making enough of a distinction between the two. I am uncomfortable with the mostly seemless mixing of discussions of evolution and discussions of individual development. It is interesting, however, in light of their hierarchy of the evolution of ontogenesis, to think about where individualism develops. We know that every human is unique because we all experience differences in our environmental-organismal interactions. But what if we talk about organisms who differ only based on systems-level or cellular-level interactions? Can we call them individuals? Take 2 nematodes. We don't expect them to have a highly developed "personality." Yet, if you put them in identical situations, wouldn't you be surprised if they behaved the exact same way? What is that difference in behavior due too? How much of that is genetic and innate? Maybe all of it. But if you take identical human twins, even raised in different environments, popular thought suggests that they will have a lot of the same behaviors, even though their organismal leval interactions have been very different. How much of that is genetic? Thus, how individual should a network be to be useful? Would there be any way to "breed" networks of different so-called genetic backgrounds? Because I think that it is only by simulating natural selection in the phylogenetic sense as well as ontogenetic development that we will be able to create networks that behave in a way that is useful in terms of really describing organismal behavior in anything more than a strictly general sense.
This reaction is kind of long already, but I also had some thoughts on language learning and starting small. I appreciate their evidence for biological constraints on language, I just would like to know how they are constructed. Also, I would be more ready to believe the idea that children learn languages more easily because they can only process short sentences if I knew what happened when adults were just placed in a foreign culture with no knowledge of that language. Can they learn the language just by being immersed without prior knowledge? If they can, how? And why is it (supposedly) easier to learn a language by immersion if you already know a few words? A third area of questioning has to do with the oft-repeated dogma that children learn languages better than adults. Perhaps they do learn better. But do they learn quicker? It takes a child 3-6 years to become fluent in their native languages. Adults can become fluent in less time than that, can't they? If their hypothesis is true and the way you learn different languages when you are older is completely different, than how come there are people who are bilingual as adults? Is it all just really fast computational? And if this hypothesis IS true, how would it be possible to use this knowledge to "our" advantage? How could we change the way we, as adults are taught languages in order to increase attention to morphology and decrease emphasis on rules and computation?
Something that struck me about this reading is that at the beginning of chapter 6, the authors finally took a stand on what they "really mean" by saying that something is 'innate'. The meaning of an innate behavior or form is that "given normal developmental experiences, it is a highly probable outcome." This seems like a truly oversimplified view of innateness that stands in contrast to the complex view of innateness revealed throughout the rest of the book. Now, it seems, innateness is simply another way of specifying probability. Immediately after making this statement, to their credit, they return to examining in detail the complex interactions between internal constraints and properties of the organism's environment.
Later in this chapter, the statement (attributed to Gold, 1967) was made
that there are certain structural features of a language (e.g. relative
clauses) that may render the language 'unlearnable'. This statement is
absurd. It is clear that this statement was made in support of the
Chomskyan idea of the Universal Grammar. However, even with a UG, the
specific language must itself be learned by the speaker. Elman's training
of a recurrent network seemed to be directly aimed to support this
opinion. The omission of words like "the", "a", etc., was mentioned in a
way that made it seem that these words are unimportant in the learning of
a language. Would it be easier for a network to learn sentences like
4) Boy who dogs chase feeds cat.
or more realistic sentences like
4') The boy who(m?) dogs chase feeds the cat. ?
It would seem that articles appear in the English language for a reason.
If they were not essential in the learning of complex sentences, would
they not be omitted from the language? The omission of such words in the
input of "tarzan"-type 3-word sentences to a network is justifiable. But
their omission in the learning of complex sentences would seem to make a
difference in the network's ability to learn them, especially if we are
making outrageous claims about the unlearnability of certain language
structures.
It was quite refreshing to read a sceptical review of Rethinking Innateness, or for that matter any sort of viewpoint that challenges the dogma of connectionism. This review helped in some way to bring the book into the context of psychology in general. It made some good points concerning RI's outright rejection of content nativism and questioned the basis of this rejection.
Chapter six is an excellent example of what was wrong with chapter four. The chick imprinting example made lot of sense and explained the learning process very well, as well as exploring all sorts of side paths and similarities between the model and the chicks which reassured me as to the appropriateness of the model. Its too bad they didn't manage to pull a couple of graphs and equations together, though...
I'm very interested in the bit where children were surmised to filter out most of their input so that they could "start small" and learn gradually. This combination of noise and annealing sounds like a very effective method for maintaining plasticity while searching a large state space. And the fact that the neural net automatically developed modules for AND and OR and then for XOR as the wave of plasticity swept up the net really seems great, but I wish we could use the same explanation for later modularization in the brain during learning. Perhaps there are just neurons which refuse to make up their minds until very late, when everyone else has already become specialized.
I was very surprised that most linguists seem to adhere to the theory that language is 'an organ' or some innate module in the brain. The lesion studies provide some very strong evidence to the contrary. Perhaps it could be said that language learning functions very much like a innate organ, and that it is up to the connectionists to prove that such behavior can emerge from a loosely planned neural net. Its another confusion of definitions of innate.
I likopponentser 6. I think it had a good balance of examples and theory, and I like the theory it presented. I was especially interested in the proposed method of language learning in children. Language learning has been sort of a weak spot in the connectionist armor for some time now (as we see in chapter 7), and any simulations which appear to defend language learning as anything other than a pre-installed language subsystem sounds good to me. It is also interesting to think that development could play a bigger part in learning than I might have thought. If there is really some good point to the years we spend learning, then maybe I'm willing to not feel that upset about having to endure grade school.
I also liked the example of the "Wave of plasticity." While I don't really see a concrete neurological basis behind the construction, I think that self organization is a very important concept to strive for, and this network has very interesting self-determined properties. It is highly multi-layered, which can mean that not much generalization is going on. But it is good proof that self-organization is a possibility for neural networks.
Chapter 7, however, was just more of the same: random examples and counter examples, with vague attempts to prove their point. I suppose that it is good that they are rebutting the opponents arguments, but they could do it in a more readable way.
the phenomenon that they call chick imprinting is in fact one that has astonished me all the time. i was always told to stay away from just-born chicks because they would take me as their mom or dad or whatever, but i never believed it, just like i failed to believe the santa claus story. however, it is very interesting to see that a chick will actually follow the first thing it sees, be it a human or a ball. and it is a perfect example of how the brain/mind loses plasticity during evolution.
i liked the discussion about the blueprint (bauplan) of a human body, but i sort of missed further discussion on exactly this topic. yes, the chapter deals with interactions and the evolution of the initial cell and its offspring, but i never found an acceptable answer how all this info is represented in the first cell. and noone said that this is not known. maybe i did not read well enough...
fodor's article is classical. i like how this single entity steps up against six well-known personalities and criticizes their book to be incoherent in places, paradox in other, and plainly false here and there. i mean, jerry does not seem to have a problem with this - this is not his first such critique - and in some ways he is right. _rethinking innateness_ surely has a lot to say, but as the authors put it nicely, it is not a theory, but something that may one day evolve into a theory. innateness is very undefined - there exist not even parties which support different aspects as such (compare to connectionism vs. standard cognitive approaches) - and i believe it to be too high of a goal for people to attack this subject in an attempt to coin its definition into a book. little do we know about the relation of cognitive models and the brain, and putting out a book on the topic is not necessarily that helpful. it certainly was an interesting book - although i missed a lot in it - and i probably do not have the right to give this criticism, but i believe it is too much trying to impose theories and thoughts on the reader, something that fodor obviously disliked a lot. and retrospectively, it is true that the book is not a connectionist perspective on development, as the subtitle states, but an empirical approach to connectionism. it is hard to isolate an argument in there, which is not based or backed up by empirical study. however, the book does offer a broad (maybe too broad) perspective on the whole field and i am glad to have read it. but jerry, way to go!
Wow! I really enjoyed this week's reading. As I mentioned in an earlier reaction, I agreed with the authors' point that any network architecture implicitly involves many "innate" architectural constraints, and wondered out loud whether any work had been done involving the explicit exploration of these constraints and the outcomes resulting from them. Chapter 6 made my day, at least as far as those speculations were concerned. I especially liked the particular kinds of constraints that were involved, since a moving wave of neuronal plasticity or a gradually increasing attention span are constraints that could occur in a real system like a human brain. I think it's really neat that patterns of organization can arise from such simple architectural features, and it's definitely an avenue of research worth going into.
The most striking point that was made in the last two chapters was that of developmental learning - the example was that of a NNet unable to learn a complex body of grammatical input, but able to learn that same body of input under two conditions: either with the context units parsed in time (allowing only short-term memory), or by varying the input from simple to more complex. This is easily one of the most convincing arguments for development, and the need for a maturational perios. My only question would then be, "is this how the brain always works (if we assume it follows the first strategy and allows short term memory to help in boostrapping the solution space), or does this strategy only apply to the maturational period of the human?"
I would argue that this paradigm of learning, if it is in fact how the child learns complex grammar, is transposable to the adult learning mechanism. For example, if you were to throw fully-fledged calculus at an adult never exposed to it before, and even explained every term, the chances are that s/he would be unable to understand it immediately. However, it is totally possible to teach an adult calculus if you work on a traditional teaching regimen, parsing the knowledge into discrete chunks. The teaching regimen is in fact a possible case of the emergent architectural form the book talks about so much - becasue it is the best way to learn, teaching strategies have focused around teching simplest concepts first and then dealing with the more advanced later.
This can also be seen in foreign language learning. Expose an adult to a new language, with no training, abd that adult will be able to learn that language agonizingly slowly, but only with a maturational period in which s/he first learns the simplest forms (i want, hello, excuse me), then progresses to more complex. In the case of two adult native speakers of two different languages, without a translator, trying to communicate, the two will first form an intermediary language called a pidgin, based on the simplest forms, then to a creole, which begins to use a grammar, and then to fluency in an intermediate language.
I would argue that this process of parsing the input space into discrete and simple forms, like the way a short term memory would, is an architectural constraint of the system of human learning, and is not entirely limited to how childrem think. While this raises questions as to the reasons for maturational time, I don't think that it jeapordizes it (especially as i have no formal proofs of my arguments, and no studies to cite), beacuse of the amazing plasticity of the immature human mind. It has been proven again and again that the mind of the child is able to deal with input is a very subtle way, leading to a fine-grained representation of its contents. SImply the discrepancy between a native learner of a language and an adult learner should be enough to argue for the use of a maturational period, if we accept that the system of learning outlined above is accurate.
The reading went fine for me until I reached the connectionist proposal for chick imprinting, at which point I got thoroughly lost. By the end, all I could be sure of was that our six authors clearly thought that chick imprinting, like so many other supposedly instinctive behaviors, could be represented as learning done by a network.
On the other hand, I was sort of pleased to find a simulation of animal behavior rather than human. It strikes me that we have a tendency to refer to animal behavior more opaquely as "instinctual" and innate than we do human activity. Since these animals' cognition level is supposedly below ours, and their brains simpler, in theory we should be able to simulate, say, schooling fish and birds flying south for the winter much more easily than, say, human acquisition of language. Trouble is, just judging from the chick imprinting example versus most human examples in the book, we can't. That is to say, our networks for language simulation seem much easier than our network for chick imprinting, although I suppose we could explain this with the fact that we have a greater understanding of our own minds than of some other species. But it's still troubling. Oh, well.
Anywy, that last chapter is about the only thing that prevented me from agreeing with Fodor that Elman et al. make a generally incomplete and not well-grounded argument for connectionism, because it played up the idea that connectionism is here only a suggested tool for (not to be too cheesy) rethinking innateness. My response to Fodor is just that: claiming that Elman et al. don't completely prove that Connectionism Is Right and other theories of development are Wrong denies the basic project of the book, which is just to "raise questions" on development and point out how connectionism can lead to perhaps interesting preliminary ideas about human cognition that are very different from older theories. And while the "it's the best we've got" argument is not so strong either, it seems at the moment like it's the best we've got. To that end, I wouldn't mind reading not only anti-connectionist stuff like Fodor's review, but also recent pro-nativist articles so that we could see an opposing point of view that's not just being oppositional. Perhaps Spelke, or Leslie? Or even Chomsky, to get some sort of handle on the foundations of this nativist algorithmic look at linguistics?
Watching a philosopher point out all the philosophical flaws with connectionism is sort of fun. However, he neatly uses the following two arguements on connectionists, first, that there is no way that we know of to draw paralelles between neuronic activity and thoughts, i.e., no one knows how we think. Second, that correllated experinces alone cannot account for how we think. Um, what was that first point again? It is still impossible to defend connectionism to a philosopher, since all the alternitives are far more intuitive and more rationally sound, not relying on emergence, etc, to explain sentience. However, I think even after buying all his philisophical points, I think connectionist still have a leg up on everyone else when striving for biological plausibility, and this is the goal and focus which connectionism must cling tightly to to maintain creadability. Without it, neural nets are simply methods of searching a huge space of multidimentional, nonlinear, dynamic equasions to find ones which we consider particually interesting. Lorenz attractors are simmilar equasions which were found trying to model weather systems, and while we still find them nifty and cool, the _only_ thing they prove is that you can't use them to predict the weather. Connectionism should be very careful no not sucumb to the same fate.
On to the book: Chick imprinting is cool, however, I think a big part of it is that chicks are attracted to moveing objects, and it's not quite clear how this is an archetectual bais. So when they make it one in the model, I'm not quite convinced. Also, Gold's proof is not quite convincing either. First of all, does correction of words or sentences that young children perform count as negative evidence? Also, if a child has not yet mastered simple sentences, they will probably not be able to understand statements made about a sentence itself, so it is obvious that there must be some other way that this information is aquired, it's just not obvious that it's inate. Third, infants are not taught to speak by being read shakespeare, most sentances directed at them are beyond simple, Elman just seems to be simulating that process and claiming that it proves something about neural structures. Lastly, I don't think I stopped forming gramatically incorrect sentences until after 9th grade english. (in fact, I still do occasionally) The last chapter seemed to entirely be a recapulation of previous ideas. The flavor of the entire thing reminds me of en Einstein quote, "If we knew what we were doing, it wouldn't be called research." Besides that, I think they're fairly convinceing that there is some sort of relationship between what's going on in the brain and thier models, but if it's the same sort of relationship that lorenz attractors have to the weather remains to be seen.
i think the last chapter did a pretty good job of summarizing the arguments made in the book, and, again, a pretty good job of making it sound like the nativists don't have a leg to stand on - i'd really like to read some opposing viewpoints. the "wave of plasticity" example was fascinating - both as a practical description of how to implement time-dependence, and in the fact that it seemed to work so well. hmm; my comments are very short today. i'll just end by making sure that everyone caught the really amusing typo on p. 364.