this reading was rather boring,b/c I've already had most of this stuff in either psyc 1, bio or my electrial engineering courses. However, I was suprised to learn that the dendrite tree adjusts the size of the signal by the length of the branches. I wasn't expecting that bio neural networks do the adjustment by such a simple mechanisim. I also was supriesed to learn about the holes inthe mylin sheathes that speed up transmission of the action potential spikes. Is the spacing of the holes related to the frequency of the pulse, ie like some kind of resonance chamber?
Is there any way to include the holes in the silicon neural nets, so that pulses with a particular threshold will move much faster?
At first, reading the Anderson was extremely boring. There is a good deal of REALLY PRECISE biology that all of us remember from high school about neurons, that I didnt want to connect with computational modelling except in the basic sense. For example, showing electron microscope pictures of a neuron synapse (which I couldnt really decipher from the picture anyway) probably will not be too helpful to me in understanding a threshold logic unit.
Then it got more interesting.
For example, I knew very little about the slow response theory of neuron communication. The importance of the action potential traveling down the axon is what I was always taught, and the complexities of the interaction from multiple dedrites to the soma, processed to be sent down the axon as a frequency, is really interesting, and answers a few questions I had about the modelling of the general neurode. It still seems, however, that the general neurode is not as accurate as it could be in its modelling of the neuron. It could be an accurate model of a dendrite, taking inputs and comparing them to a threshold, then sending the impulse on. The soma would them have to be some sort of processing mechanism, that would take the impulses from multiple sources that are 1 or 0 (an action potential is still either +10mV or -60mV, whether in a general neurode or in a McCulloch-Pitts neurode), and would process these into a slow response, which would be sent to the outputs of the neurode. This is obviously more than is needed, (the general neurode deals with the difference in response between multiple dendrites) unless the model neurode wants to implement the slow response. If the input across a model synapse were taken as a function of the firing frequency of the pre-synaptic neurode, then maybe the input would be a more accurate model of biology. With this more accurate model, perhaps things like the slow response could be explored more in depth than possible with biological sampling.
Another issue that Anderson brought up that interested me was morphology. He stated that the only difference between the human DNA and the chimp DNA (which are 99% identical) is in the stuctural genes - the chains that code for size and shape of molecule structures. Its obvious to biologists that morphology can cause HUGE changes in function - just look at how particular the active site of an enzyme is - to destroy its function, you denature it, as simple as heating it so the shape of the site changes. But morphology also matters to computational models - what if the input isnt being given the right way, or the numbers need to go through one more manipulation before ending up somewhere else. Its really interesting to me that the change necessary to go from chimp to human, so to speak, may be a change in shape and structure, and not in logic. Another biological example of morphology: the rats spine that Anderson talks about -he writes that it is so fascinating because it is forced into a small space - with only a little change in size, it can change function drastically. This example bears directly on neural networks: the more hidden units (or is it layers?) you have, the more hyperplanes you can use to cut up the input space into discrete input patterns. This is a REALLY simple morphological issue thats not about how you dot product the vectors, its about how you assemble the really simple modules into the greater work. (And maybe Im biased in that Im really bad at math, but hey, what can I say? The morphological issue is really interesting).
Why would the brain be the system that works best? It is a system which works, but its operation is grossly limited by other factors other than intelligence and facility of function. Human heads have to fit through human birth canals.
What is the relation between hardware and software in the brain? Is the brain running a program, or is the program hard-coded into a medium which itself is maleable (since no one can deny that the system changes)? On a computer-based NN, how much can we understand about the software that the net has learned? Does it resemble more classical parallel programming?
How much of the physical developement of the human brain occurs while the child is learning? Has the necessity of flexibility required that nature make such important parts of the brain as intelligence and sexuality learnable to some extent?
The brain would have to have enough bootstrap to start learning, but beyond that it would have to be as flexible as possible to accomodate the most specific learning. The brain is probably disconnected to a point near the line beyond which a system will never self-organize. Or is there such a line?
The fact that the brain trims itself down based on the stimulus that the newborn recieves has to have some limit; a baby cannot possible be exposed to all the stimulus that an adult will have to cope with. Does this mean that the systems which are trimmed are just perception sub systems which can be used interchangeably by the higher systems?
Are neurons in the brain the only ones which form new connections? To grow an axon all the way from the small of your back to your toe seems to be too much work to happen haphazardly.
I'm curious about electrical synapses. Especially since the author says that theyre only used in invertibrates and that chemical synapses are much better, and then goes on to say that were going to ignore many intricacies of the latter because its too hard to model.
What has been done with neural nets to simulate the voltage-to-frequency converter function of neurons? What complexities are introduced by the necesity of timing and other temporal considerations?
Penrose, or whoever it was who said that there was just something about the mind which made it impossible to simulate, may have a point. One gets the impression of a seething mound of organic fibers, interactions between which are immeasurably inter-related.
It seems that the neural nets I've seen so far are more related to directly connected slow potential neurons than action potential neurons.
I found this a refreshingly objective portrait of the neuron and of various attempts to model it, relative to, say, Paul Churchland. It is somewhat sobering to remember how complicated a system a single real neuron is, and how much detail is being set aside in the computational neurodes we're working with. In response to the suggestion that we have so many neurons because we need them all, I would say that this is probably the case largely because individual neurons are so slow -- an individual element (transistor) in a conventional digital computer is much faster. I also agree wholeheartedly with the observation that no animal brain, even ours, is a seriously general computing device. The brain evolved subject to the constraints of the body and the world it occupies. The hunk of silicon on any of our desks is much more "general" in function. From some of the readings for Psych 28, I know that there is a tendency among some of the more radical connectionists to see neural networks as magic (e.g. Churchland, as mentioned), and it's nice to see a balanced viewpoint.
The thing that struck me most out of this week's readings was the introduction to the slow potential theory of the neuron. In my experience, it has been the case that when discussing the neuron in detail, the emphasis has been focused on the action potential. The slow potential theory shows that these 'spikes' and the frequency with which they occur may not be very significant in and of themselves after all and that the most significant factor in unraveling the behavior of the neuron is the changes in the neuron's 'slow potential'. The 'slow potential' of a neuron is the slowly varying potential level that is usually translated into frequency of action potentials.
In fact, this theory states that the action potential itself is merely an adaptation of neuronal cells to produce signals that can overcome the relatively long distances travelled along the length of their axons. Since voltage signals cannot be propogated through the axons without avoiding leakage through the membane of the cell, these voltage levels are translated into frequencies of action potentials, these frequencies varying with the waxing and waning of voltage levels in the cell body of the neuron. It turns out that certain neurons which have only to communicate with neurons a short distance away do not have action potentials at all, but only communicate values of their slow potential with surrounding neurons.
With the spotlight taken away from the action potential itself, we can start to examine such phenomena as spatiotemporal interactions of postsynaptic potentials (PSP). The investigation of the sophisticated signal processing that goes on between the synapses and dendrites of nervous systems seems to hold much more potential for understanding what is going on in these systems than the simple focus on action potentials themselves.
As interesting as these details of neuronal functioning may be, it is presently necessary for us to discard some or most of this detail when creating artificial neural network models. A few questions then arise: What aspects of the neurons should we glaze over and leave out of our simulations? Also, what kind of abstraction should we use to integrate our present knowledge of neural systems into a working neural network model?
The answers to these questions, as we shall probably see, lie in whether our aim in creating an artificial neural network is in order to discover more about the processing that occurs in biological neural networks, or if we are interested in trying to harness the power of subsymbolic processing for any of a number of possible applications.
If we are interested in finding out more about the nervous system and its functioning, it is clear that we are nowhere near a full understanding of how these systems function. In this case, applications that attempt to simulate biological neurons in all of their electrochemical complexity may provide new opportunities to approach the study of the nervous system.
On the other hand, if we do not particularly care to find out any more about biological intricacies of the nervous system's functioning, but rather to take advantage of the way that these systems store and process sensory information, we can use any of a number of neural network abstractions. These systems range from simple, all-or-none neurons like the McCulloch-Pitts neuron, that can be viewed as a way of representing complex neural interactions in terms of boolean logic, to more complex systems that use nonlinear sigmoid activation functions and backpropogation.
really, chapter 1 of anderson's text was, well...interesting...i understand that it is important to understand exactly what a neuron does before we try to model it and use it in computational models, but there is no need to appreciate neuron operations to the smallest detail. okay, i was probably also very overwhelmed by the information presented--i happen to despise biology and thus never had a class since 7th grade or so--but most of the info laid out in the text was lectured wonderfully by the allan schneider himself during psych 1. so except for numerous formulas and actual maths behind the facts--the stuff that you can look up--the chapters presented very little new information to me, also some details were very interesting, slow-response neurons for instance, or the fact that up to half of a human's brain cells die during birth, or that axons can be as long as humans are tall.
one thing that i did not grasp while reading the text was why action potentials face no transportation problem the way that continuous currents do. the reasoning behind the phone line problem is perfectly clear, but action potentials are short bursts of energy and i don't see why they should not diminish with distance. action potentials have more of a digital than analog nature ("all-or-none"), so this may account for the fact that diminishing signals do not change the information--i could not extract this info from the text.
also, what did anderson mean by the "nonzero spontaneous firing rate" and how does it justify having negative output values?
furthermore, what is so great about the bistate neurons that received so much attention by computer scientists? after all, all they do is mimic logical AND and OR gates, with the introduction of a delay (ie. non-spontaneous, but allowing for the inputs of time t to affect the output at time t+1).
I found it very interesting, to read a computer scientest's take on neurobiology, but I was a little disappointed that I didn't really glean any new insights into brain or network function. I guess it might be a little early for that. One thing that bothered me was the way that s/he dealt with synapses. Its pretty important that the synapse is not an actual structure, but a space, and it seemed like the article almost neglected that, perhaps to make the conceptual model of the brain fit better with the neural network model. Another weird thing was all the TEM pictures. They were neat but completely unnessesary, it seemed to me. I was interested in the idea that synapses vary in strength -- something that is obviously important to a neural network, but I'm not sure is as accepted as the article implies. As far as I remember, it may be more the way the neurons are arranged and the chemicals released at the synapse than any one synapse being more "powerful" than another. As I remember, a strong action potential (high frequency) will result in an increased concentration of a given neurotransmitter in the synapse and hence, a stronger reaction from the post-synaptic terminal. However, I guess that is more or less the same as a weighted dynapse. Except that the weight is not a static thing, with one synapse always being stronger than some other synapse.
I was intrigued by the passing mention, in the conclusion, of the "quite different computer" one could build with more realistic neurons. I'm not sure I quite understand exactly how the new neuronal model is constructed, but it seems to me that there are many important considerations to make when building a more realistic neuron (and hence, a novel computer), and not all of them have to do with integration and inputs. For instance, there is increasing evidence that a great deal of function, even in "higher" organisms, is a direct result of unique morphologies. Thus, some charicteristic actions (sidestepping when the ground is uneven, for instance), have been theorized to be direct results of the way that a given body is constructed, and not controled by neuronal function at all. So it seems to me that computer scientists need to also take into consideration the overall structure of the problem and the "organism" that they are trying to model.
Well, I was trying to gauge what I wanted to write for this reaction, so I peeked at the reactions already posted. I must say I agree very much with Simon. The review of psycho-bio concepts was at first tedious, but then illuminating.
One of the key "draws" of artificial neural network research is the claim that neural network architectures, as they currently stand, model to some degree the mechanisms of biological brains. This not only generates excitement and interest (and funding) in neural network research, but it also supposedly gives legitimacy to the eventual flexibility and power of the model once fully developed. Obviously, a more careful consideration of the facts, along with a provocative (however brief) conversation with Jon Shlens, has caused me to be disabused somewhat of the legitimacy of neural networks as a model for the brain. I think there are an infinite number of complexities of actual neurons that most likely (due to the highly essential adaptive value of neuron architecture) contribute a great deal to the actual power and "usefulness" (the quality of conferring fitness advantages) of the biological brain.
Of course this doesn't mean I'm going to hustle to the registrar before add/drop is over because I've lost all faith in the research that's been done up to this point. I have great respect for it, and I'm excited to learn about it (especially that third article in the packet, it looks really good ;) ), but certainly, those who make claims of a true biological basis for current feed-forward, back-prop nets are exaggerating at best. Current neural networks, though biologically inspired and clearly very capable of solving a wide variety of tasks, are clearly based on an oversimplified model of neurons. Whether an architecture based on a more complex oversimplification (because we sure as heck don't know enough to model all neurons precisely) would be more powerful overall than current models is the question I'm left with, and I would be curious to read about research on that topic. Who knows, maybe it turns out that for _virtual_ neurons, a simple mathematical model makes them more powerful. But I doubt that we would see such specialization and complexity among our own biological brains if that did not confer large advantages.
As to the clear differences between connectionist models of neurons and what we know of the biological neuron, the one that sprung out at me (with a little provocation from the discussion with Jon I mentioned earlier) was the way that time factors are ignored in what I know of current neural models. Time is a critical factor in the "shape" of impulses, the distribution of firings of synapses (a Poisson distribution was mentioned for one instance), the refractory period, and most importantly (IMHO) in the very nature of the transfer function, which is some time-independent function of the sum of the inputs for most artificial neural networks and an explicitly time-dependent voltage-to-frequency conversion for biological "neural networks." The implications of this last difference alone are that the ability of ANN's to model BNN's true behavior is severly limited without time dependence. I have no idea how it would work, but I'd love to see an ANN that tried to incorporate that idea.
In addition to time-related issues, I also see the oversimplification of terminal button/dendrite transmission model, the lack of some global state approximating neuro-transmitter levels, and oversimplified morphology as all presenting powerful avenues of extending artificial neural networks to more accurately model the biological structures they were inspired by and, for a time, claimed to model. I have a ton of questions, but really, they all can be summed up in two questions: "What has been done in artificial neural network research that has attempted to more closely model biological neural networks?" and "How well did it work in comparison to other ANN models?"
I found the reading to be rather boring and tedious. Praticularly at the beginning. Being uncertain of the intent of the authors and coming at this section of reading from the perspective of someone who is interested in creating objects that behave intelligently, a "true" understanding of how things "really work" is not necessary. However, that did not seem to be the point of these chapters.
This was apparent immediately from the second page which said "the brain is not a general-purpose computer." The goal seemed to be to understand how an actual neuron worked. I was not originally prepared for such a thing and so the majority of the reading was approached in drudgery.
There were some points made that seemed less plausible than others, and I was not certain of the "correctness" of them, however, my knowledge of the subject matter is not that great. For instance, is it true that "Adaptation and learning...are kept under tight biological control"?
I was also amused (as a physics student who has seen the amazing abilities of approximations and how useful and almost completely correct they can be) how the book constantly referred to approximations. Things "theoreticians are fond of." What is wrong with approximations? Particularly when the results they give match the real world?
I think the key point of the whole thing when discussing good models was "Most people like models that are dimple enough to understand, but rich enough to give behavior that is surprising, interesting, and significant." I think I would have to add that it matches reality. Again, this is a physics perspective. Let's face it, we determine an equation is "correct" if it correctly predicts the results of interactions. We do not try to determine the why of the "force." I guess this mentality pervades me, and I can not see the necessity of understanding the human mind entirely in order to create something that can respond like the human mind does. However, I do not deny that a fuller understanding is very helpful in obtaining such a goal.
First of all, I read the Anderson nervously simply because I have always been nervous about computer scientists being restricted by a slavish adherence to our knowledge of the biological model of neurons.
Apparently, though, so is Anderson. Just to take an example, he says that the disparity between real and simulated networks "should cause uneasiness," which is a cute way of saying it's an unnerving thing to consider that perhaps a lot of our problems could be solved if only we had the capacity to produce a network of 10 billion neurodes. On the other hand, our problem isn't really scope, it's depth. That is, it's not like we can succesfully model a cockroach, but when we try for a human brain we miss. It's that we can't even do a cockroach. So perhaps scale will become an issue later? Or is there any way it can affect things now?
Twenty-five pages later, our dear author remarks that the dendrites receiving synaptic potentials are performing the work of a complicated analog computer that "filters, delays, attentuates, and synchronizes" its input potentials, but that it's been hard to analyze this stuff because of its complexity. I know this is recent writing, and I don't want to sound too "This is the future, where are the flying cars?" about this, but why ISN'T more work being done on this sort of analysis?
Which brings me to my main conclusion from this reading: the connectionists are confused. The field doesn't seem to be sure whether it wants to *really* accurately model the biological system, or whether it instead wants to squeeze the biological system into a purely digital system and hope it sticks.
The problem with the first idea is primarily that it is really hard. Penrose or somebody like him theorized that the mind was impossible to accurately simulate, and even with the level of precision we have now, it doesn't seem to be getting so much easier. In fact, the farther we can look into the workings of a neuron, the more we seem to see that a digital computer is not particularly a good way to model one. Right? So it seems to me that connectionists are going to get tripped up insofar as they try to perfectly model biology. It is a system that works but it is not necessarily the ONLY system that could work; it is not so hard to believe that a nervous system for a digital silicon computer should be far different from a nervous system for an analog carbon one.
Oh, and one little piece of confusion: Are all backprop networks slow-potential based? Are any neural networks action-potential based anymore?
Efficiency: Anderson attempts to use the birth and death rates of neurons to show that the human brain is efficient as it can possibly be and that any neural network must be at least as large as it to accomplish an intelligence of our level. However, he makes two unwarranted assumptions, that selective pressures exist to reduce the energy consumption of neurons and to kill off any unused neurons. While both these situations seem plausible, they do not seemed to be backed up by convincing evidence. Also, since natural selection operates on first, the ability to survive or not, and second, relative reproductive advantages, other, equally intuitive counter-situations are equally as plausible. For instance, a genetic trait that dramatically increases energy consumption but slightly increases thought speed would be much less efficient, but even slight increases in reaction time and generally thinking faster could lead to huge advantages in resource acquisition and therefore reproductive attractiveness, as well as providing for more than enough extra energy for the new variety of neuron. Also, a genetic disposition to keep a lot of extra, unused neurons around could theoretically lead to a higher ability to learn later in life, which could prove to be a larger reproductive advantage than the lower energy usage. He also assumes that both AI and human minds must solve any problem in the most efficient way possible. Aside from the fact that neural networks may not always produce the most efficient of all solutions, he provides no real evidence to lead us to believe that the human mind solves any problem in the most efficient way possible. At best, the biological brain, as the only working example of intelligence, provides only an upper bound on the most efficient possibility and a lower bound on the least efficient. Since we do not understand how the brain accomplishes it's task, establishing further limits is impossible.
Time: The main area where neural nets seem to be woefully inadequate representations of neurons is in the timing of the impulses. Anderson goes into great detail in describing the timing of the Electro-chemical retains, such as the voltage-frequency conversion properties of neurons. Also, given his discussion of the speed of impulses, it is doubtful than the brain as a whole operates on synchronized clock cycles as does a computer, in addition, the timing of impulses are better thought of as an analog rather than digital property. However, all current neural network implementations treat the net as a whole as a static evaluation of a equation rather than the continuous behavior of a system. This inadequacy should probably be one of the most pressing issues for AI research to examine. The "integrate and fire" model has much potential for addressing this problem, but getting the multi-dimensional activation function accurate may be slightly problematic. The only other option is to once again abandon the accuracy of the parallel between biological and simulated neurons, in the hope that the biological method is not the only way to create intellegence.
the introductory part of the reading was basically all i understood well enough to comment on it. a lot of the bio descriptions were interesting, but i got bogged down reading the descriptions of how a neuron works, which were not written at all from a computer science perspective. i'd like to go over this stuff in class, at least a little bit. a couple of quick comments:
the author seems at one point to be implying that all thinking systems work alike. it strikes me as silly to suggest that a computer thinks most efficiently the way humans think, even if we are trying to replicate human-like thinking in computers. computers have an ability to number-crunch and evaluate situations by brute force that is superior to that of humans. does it really make sense to suggest that humans and animals have achieved an arrangement and style of thought that is optimal for both natural and artificial brains? it may, but i'm not convinced.
i'm also a little unsure that computers will not be able to think like humans until we can give them as many neurons as a human has. while this is probably strictly speaking true, (assuming that the human brain uses all of its neurons, which i can accept), many of a human's neurons must be used for survival functions of the body, physical reflexes and so on. so, unless we intend to create a physical robot as complex as the human body (which i highly doubt is going to happen any time soon), we probably don't need that many neurons for a robot to perform purely mental functions.
One of the most interesting things which occurred to me as I was reading this was my reaction to the talk of myelin sheaths surrounding our axons. That they exist is all well and good, because they allow us to think the way we do. But there is a disease (I forget the name) which involves the deterioration of these sheaths, and the subsequent interruption of muscle control and cognition which occurs as the transmision speeds in the brain slow down. As a complete alternate to simulation, I think much could be learned about processing and transmision by observing the details of this disease, both at the cellular level and macroscopically. I'm sure work has been done in this field, but it might be fun to try and apply it directly to cognative science.
Another topic of discussion in the reading was the blase statement that much of the low level detail would be left out in simulations, due to the complexity needed to run better simulations and because it had traditionally been done before. They (the authors) do mention that new simulators are coming out which can will be able to more accurately portray neuron, but they also seem satisfied with what has occurred so far in the field. I think that if we want to make any process at all in understanding cognition, we need highly accurate neuronal models, especially because they systems we are modeling are so nonlinear. The authors state that learning probably occurs in the synapses, and then in the same chapter state that most models ignore the synapse as it actually is completely. With they state of computers currently, I think it needs to be a priority that close work between neurobiologists and computer scientists occus, so we can gain insight into what the human brain does, instead of what we can make computers do.