Neural correlates of perception (what's wrong with them)

Broadly speaking, neural correlates of a percept, e.g. seeing a face, are what happens with neurons when we see a face. For example, a bunch of neurons would fire when we see Jennifer Aniston. What do neural correlates teach us about perception, or more generally about the mind-body problem?

The interest in neural correlates of perception is largely subtended by the implicit belief that there must be a mapping between perception and brain state seen as a physical system. That is, the percept of seeing Jennifer Aniston's face corresponds to a particular brain state; the percept of a sound being at a particular spatial position corresponds to another one. There are considerable conceptual difficulties with this belief. Consider these two thoughts experiments.

1) Imagine we can instantaneously freeze the brain so that its components, ions, tissues, etc are preserved in the same state (it's a thought experiment!). Does the brain still experience seeing Jennifer Aniston's face?

2) Imagine we record the precise spatiotemporal activity of all neurons in response to Jennifer Aniston's face. Then we inactivate all synapses, and we replay the activity pattern optogenetically. Would the brain still experience seeing Jennifer Aniston's face?

Intuitively, our answer to both questions is negative. If you answered the second question positively, then consider this one:

2b) Imagine we record the precise spatiotemporal activity of all neurons in response to Jennifer Aniston's face. Then we replay the activity pattern on a set of light diodes. Would the diodes experience seeing Jennifer Aniston's face?

If our intuition is correct, then brain states, even understood more broadly as “firing patterns” are not constitutive of percepts. It appears that whatever a percept is, it must involve not just the state or even the activity of neurons, but the interaction between neurons. Therefore when we describe neural correlates of perception in terms of neural “activity”, we appear to be missing a crucial ingredient, which has to do with interactional properties of neurons. To be honest, I must admit here that “interactional properties of neurons” is a loosely defined concept, but apparently there seems to be a need for a concept that goes beyond the concept of activity “pattern”, a concept to be clarified (see afterthought below).

Underlying the problematic concept of neural correlates of perception is the representational view of perception; the idea that whatever we perceive must somehow be “represented” in the brain, like neural paintings of the world. I have pointed out the deep problems with the representational view on this blog (for example here and there) – and obviously I am not the first one to do so (see e.g. Gibson, Brooks, Merleau-Ponty, O'Regan etc). Let us simply reflect on the following one. When we look at Jennifer Anniston's face, we experience the percept of seeing her face at different moments. It seems as if at any instant, we are experiencing the same percept (along with others, of course). Possibly this is an illusion and experience is actually discrete in time, but in any case the perceptual “grain of time” is no more than a few tens of ms. Therefore when looking for neural correlates of the percept, then we cannot be happy to rely on average activity, over time or over trials. We do not experience percepts “on average”, but at any instant (this is related to my points on the rate vs. spike debate). What we should be looking for is something in the interactional properties of neurons that is invariant through the entire time during which the percept is experienced. The concept is quite different from the more traditional “neural paintings” concept.

So, in the current state of research, what neural correlates of perception tell us about perception, more specifically about the mind-body problem, is disappointingly: not so much.

 

Afterthought: an interesting analogy is the concept of temperature in physics. Since temperature corresponds to the movement of particles, you cannot really define the temperature of a physical object at any given time. Temperature corresponds to the activity, not the position or nature of the particles. What's more, the concept of temperature makes no sense except when considering the interaction between agitated particles. Temperature is perhaps an example of “interactional property” of a set of particles.

Why do neurons spike?

Why do neurons produce those all-or-none electrical events named action potentials?

One theory, based on the coding paradigm, is that the production of action potentials is like analog-to-digital conversion, which is necessary if a cell wants to communicate to a distant cell. It would not be necessary if neurons were only communicating with their neighbors. For example, in the retina, most neurons do not spike but interact through graded potentials, and only retinal ganglion cells produce spikes, which travel over long distances (note that there is actually some evidence of spikes in bipolar cells). In converting graded signals into discrete events, some information is lost, but that is the price to pay in order to transmit any signal at all over a long distance. There is some theoretical work on this trade-off by Manwani and Koch (1999).

Incidentally, this theory is sometimes (wrongly) used to argue that spike timing does not matter because spikes are only used as a proxy for an analog signal, which is reflected by the firing rate. This theory is probably not correct, or at least incomplete.

First, neurons start spiking before they make any synaptic contact, and that activity is important for normal development (Pineda and Ribera, 2009). Apparently, normal morphology and mature properties of ionic channels depend on the production of spikes. In many neuron types, those early spikes are long calcium spikes.

A more convincing argument to me is the fact that a number of unicellular organisms produce spikes. For example, in paramecium, calcium spikes are triggered in response to various sensory stimuli and trigger an avoidance reaction, where the cell swims backward (reverting the beating direction of cilia). An interesting point here is that those sensory stimuli produce graded depolarizations in the cell, so from a pure coding perspective, the conversion of that signal to an all-or-none spike in the same cell seems very weird, since it reduces information about the stimuli. Clearly, coding is the wrong perspective here (as I have tried to argue in my recent review on the spike vs. rate debate). The spike should not be seen as a code for the stimulus, but rather as a decision or action, in this case to reverse the beating direction. This argues for another theory, that action potentials mediate decisions, which are by definition all-or-none.

Action potentials are also found in plants. For example, mimosa pudica produces spikes in response to various stimuli, for example if it is touched, and those spikes mediate an avoidance reaction where the leaves fold. Those are long spikes, mostly mediated by chloride (which is outward instead of inward). Again the spike mediates a timed action. It also propagates along the plant. Here spike propagation allows organism-wide coordination of responses.

It is also interesting to take an evolutionary perspective. I have read two related propositions that I found quite interesting (and neither is about coding). Andrew Goldsworthy proposed that spikes started as an aid to repair a damaged membrane. There is a lot of calcium in the extracellular space, and so when the membrane is ruptured, calcium ions rush into the cell, and they are toxic. Goldsworthy argues that the flow of ions can be reduced by depolarizing the cell, while repair takes place. We can immediately make two objections: 1) if depolarization is mediated by calcium then this obviously has little interest; 2) to stop calcium ions from flowing in, one needs to raise the potential to the reversal potential of calcium, which is very high (above 100 mV). I can think of two possible solutions. One is to trigger a sodium spike, but it doesn't really solve problem #2. Another might be to consider evenly distributed calcium channels on the membrane, perhaps together with calcium buffers/stores near them. When the membrane is ruptured, lots of calcium ions enter through the hole, and the concentration increases locally by a large amount, which probably immediately starts damaging the cell and invading it. But if the depolarization quickly triggers the opening of calcium channels all over the membrane, then the membrane potential would increase quickly with relatively small changes in concentration, distributed over the membrane. The electrical field then reduces the ion flow through the hole. It's an idea, but I'm not sure the mechanism would be so efficient in protecting the cell.

Another related idea was proposed in a recent review. When the cell is ruptured, cellular events are triggered to repair the membrane. Brunet and Arendt propose that calcium channels sensitive to stretch have evolved to anticipate damage: when the membrane is stretched, calcium enters through the channels to trigger the repair mechanisms before the damage actually happens. In this theory, it is the high toxicity of calcium that makes it a universal cellular signal. The theory doesn't directly explain why the response should be all-or-none, however. An important aspect, maybe, is cell-wide coordination: the opening of local channels must trigger a strong enough depolarization so as to make other calcium channels open all over the membrane of the cell (or at least around the stretched point). If the stretch is very local, then this requires an active amplification of the signal, which at a distance is only electrical. In other words, fast coordination at the cell-wide level requires a positive electrical feedback, aka an action potential. Channels must also close (inactivate) once the cellular response has taken place, since calcium ions are toxic.

Why would there be sodium channels? It's actually obvious: sodium ions are not as toxic as calcium and therefore it is advantageous to use sodium rather than calcium. However, this is not an entirely convincing response since in the end, calcium is in the intracellular signal. But a possible theory is the following: sodium channels appear whenever amplification is necessary but no cellular response is required at that cellular location. In other words, sodium channels are useful for quickly propagating signals across the cell. It is interesting to note that developing neurons generally produce calcium spikes, which are then converted to sodium spikes when the neurons start to grow axons and make synaptic contacts.

These ideas lead us to the following view: the primary function of action potentials is cell-wide coordination of timed cellular decisions, which is more general than fast intercellular communication.

4 key questions about consciousness and the mind-body problem

It is fair to say that we have little idea about how neural activity gives rise to consciousness, and about the relationship between neural activity and conscious states (i.e., what you are experiencing). This is the mind-body problem. In my opinion, there has been relatively little fundamental progress on this question because it has been addressed mainly within the computationalist framework (ie in terms of information processing), which is very inappropriate for this question (this is partly Chalmers' criticism). So below I am listing a number of unanswered questions on this matter, which I believe requires a very different kind of approach. First of all, let me remark that because being conscious is always being conscious of something, understanding consciousness is largely about understanding perception at the phenomenal level (perception in the broadest sense, e.g., perceiving your thoughts).

1) How can perception be stable?

Why is it that a pure tone feels like a stable percept when 1) the acoustic wave is time-varying, 2) the activity of neurons everywhere in the brain is dynamic? The same can be said of all senses; in vision, the eyes move at high frequency even when fixating an object, and there is no visual percept if they are forced to be still. More generally: if there is a mapping between states of the brain and percepts, then why is it that percepts are not changing all the time?

A thought experiment. Imagine the state of the brain is held fixed. Someone scratches her nose and time is stopped. Would you still experience something? Any conscious experience seems to require a change, not just a state. This suggests that the relevant mapping is actually not from brain states to percepts, but from brain activity to percepts. This immediately raises a problem, because a conscious state can be defined at any point in time, but it is not immediate that brain activity can (as this would reduce activity to state). This is not a fatal problem, though, for there is a precedent in physics: a gas is composed of individual particles, but the pressure of a gas at a given instant cannot be defined as a function of the state of the particles at that moment, because pressure corresponds to the force exerted by the particles impacting a surface. It might be that the relation between neural activity and conscious states is of a similar kind as the relation between mechanics and thermodynamics.

Two more thoughts experiments. 1) Record the firing of all neurons in the brain, then play them on a set of unconnected light diodes, does that set feel the same experience? 2) (adapted from Chalmers) Replace randomly every other neuron in the brain by an artificial neuron that interacts with other neurons in exactly same way as the neuron it replaces, would there be a conscious experience? My personal answers would be: (1) no and (2) yes, and this suggests to me that the right substrate to look at is not neural activity as a state (e.g. firing rates of all neurons) but neural activity as an interaction between neurons.

 

2) What is time for a conscious brain?

A fundamental property of consciousness is its unity: a single conscious entity sees, hears and thinks. If visual and auditory areas where independent and, say, control speech, then one conscious entity would report visual experience and another conscious entity would report auditory experience. It could not be a single conscious entity since the two relevant parts are physically disconnected. Thus the unity of consciousness requires an interdependence between all the elements that compose it. This is, as I understand it, the issue that is addressed by a number of biological theories of consciousness, for example Edelman's “reentrant loops” or Tononi's integrated information theory.

However, as far as I know, there is another crucial aspect to this problem, which is the unity of consciousness, or lack of it, in time. There is no general unity of consciousness across time: two things that happen at, say, 1 minute of interval produce distinct percepts, not a single one. Clearly, consciousness is dynamic. But the big question is: how can there be a unique conscious state at any given moment in time when all the elements of the conscious network interact with some delay (since they are physical elements), typically of a few milliseconds? And what is time for such a network? Imagine there is a (physical) visual event at time t1 and an auditory event at time t2. At what time do they occur for the network, as they are sensed at different times by all its elements?Why is it that electricity changes on a millisecond timescale in the brain but conscious states seem to change at a much slower rate?

 

3) How can there be an intrinsic relation between neural activity and percepts?

Why is it that a particular pattern of neural activity produces the experience of redness? Most biological explanations are of this kind: I experience redness because when some red object is presented, neurons fire in that specific way. This is the coding perspective. The problem in the coding perspective is of course: who decodes the code? Ultimately, this kind of explanation is strongly dualist: it is implicitly assumed that, at some point, neural activity is transformed into the redness experience by some undetermined process that must be of a very different nature.

I would like to point out that proposals in which perception lies in the interaction between the organism and the environment (e.g. the sensorimotor theory) do not solve this problem either. I can close my eyes and imagine something red. It could be that redness corresponds to a particular way in which visual inputs change when I move my eyes or the surface, which I am anticipating or imagining, but this does not explain what is intrinsically red about the pattern of neural activity now. If we cannot explain it without referring to what happened before, then we are denying that the pattern of neural activity itself determines experience, and again this is a strong dualist view.

An experiment of thought. Consider two salamanders, and each of them has only one neuron, which is both a sensory neuron and motor neuron; say, its firing produces a particular movement. The salamanders are very similar, but their visual receptors are tuned to different wavelengths. In the first salamander, the neuron reacts to red stimuli; in the second salamander, the neuron reacts to blue stimuli. What might happen in terms of visual experience when the neuron fires? Does the first salamander see red and the other see blue? If we think that neural activity alone determines experience, then in fact the two salamanders should experience exactly the same thing – and this is also independent of the sensorimotor contingencies in this case.

 

4) What is the relationship between the structure of experience and the structure of neural activity?

Subjective experience is highly structured. There might be some dispute about how rich it actually is, but it is at least as rich as what you can describe with words. A striking fact about language is that the meaning of sentences is not only implied by the words but also by the relations between them, i.e., the syntax. For example, a visual scene is composed of objects with spatial relations between them, and with attributes (a red car in front of a small house). In fact, there must be more to it than syntax, there must also be semantics: if neural activity completely determines subjective experience, it must not only specify that there is a car, but also what a car is. A useful notion in psychology of perception is the concept of “affordance” introduced by James Gibson: the affordance of an object is what it allows you to do (e.g. a car affords driving). Affordances are potentialities of interaction, and they gives some meaning (rather than labels) to perceptual objects. This brings an inferential structure to experience (if I did that, this would happen).

This stands in sharp contrast with the central perceptual concept in neuroscience, the notion that “cell assemblies” represent particular percepts. A cell assembly is simply a set of neurons, and their co-activation represents a particular percept (say, a particular face). Let us say that one neuron represents “red”, another represents “car”, then the assembly of the two neurons represents the red car. The problem with this concept is that it is very poorly structured. It cannot represent relations between objects, for example. This type of representation is known as the “bag-of-words” model in language processing: a text is represented by its set of words, without any syntactic relationship; clearly, the meaning of the text is quite degraded. The concept of cell assembly is simply too unstructured to represent experience.

If we are looking for a mapping between neural activity and percepts, then 1) we must find a way to define some structure on neural activity, and 2) the mapping must preserve that structure (in mathematical terms, we are looking for a morphism, not a simple mapping).

I can summarize this discussion by pointing out that to make progress on the mind-body problem, there are two crucial steps: 1) to understand the articulation between physical time and the time of consciousness, 2) to understand the articulation between the structure of neural activity and the structure of phenomenal experience.

The phenomenal content of neural activity

This post is about the mind-body problem. Specifically, what is the relationship between the activity of the brain and the phenomenal content of conscious experience? It is generally thought that experience is somehow produced by the electrical activity of neurons. The caricatural example of this idea is the concept of the “grandmother cell”: a neuron lights up when you think of your grandmother, or conversely the activation of that neuron triggers the experience of, say, the vision of your grandmother's face. The less caricatural version is the concept of cell assemblies, where the single cell is replaced by a set of neurons. There are variations around this theme, but basically, the idea is that subjective experience is produced by the electrical activity of neurons. There actually is some experimental evidence for this idea, coming from the electrical stimulation of the brain of epileptic patients (read any book by Oliver Sacks). Electrical stimulation is used to locate the epileptic focus in those patients, and depending on where the electrode is in the brain, electrical stimulation can trigger various types of subjective experiences. Epileptic seizures themselves can produce such experiences, for example auditory experiences of hearing specific musics. Migraines can also trigger perceptual experiences (called “aura”), in particular visual hallucinations. So there is some support for the idea of a causal relationship between neural activity and subjective experience.

The obvious question, of course, is: why? At this moment, I have no idea why neural activity should produce any conscious experience at all. We do not believe that the activity of the stomach causes any subjective experience for the stomach, or the activity of any set of cells, including cardiac cells, which also have an electrical activity (but of course, maybe we are wrong to hold this belief).

I propose to start with a slighly more specific question: why does neural activity cause subjective experience of a particular quality? Any conscious experience is an experience of something (a property called intentionality in philosophy), for example the vision of your grandmother's face. Why is it that a particular spatio-temporal pattern of activity in a neural network produces, for that neural network, the experience of seeing a face? One type of answer is to say that this particular pattern has been associated with the actual visual stimulus of the face, ie, it “encodes” the face, and so the meaning of those neurons lighting up is the presence of that visual stimulus. This is essentially the “neural coding” perspective. But there is a big logical problem here. What if the visual stimulus is not present, but the neurons that “encode” the face light up either naturally (memory, dream) or by electrode stimulation? Why would that produce a visual experience rather than anything else? If experience is produced by neural activity alone, then it should not matter what external stimulus might cause those neurons to fire, or what happened in the past to those neurons, or even what world the neurons live in, but only which neurons fire now. Which neurons fire now should entirely determine, by itself, the content of subjective experience. Again the problem with the neural coding perspective is that it is essentially dualist: at some stage, there is some other undefined process that “reads the code” and produces subjective experience. The problem we face here is that the firing of neurons itself must intrinsically specify the experience of seeing a face, independent of the existence of an outside world.

I will try to be more specific, with a very simple example. Imagine there is just one neuron, and two stimuli in the world, A and B. Now suppose, by conditioning or even simply by anatomical assumption, that stimulus A makes the neuron fire. A neural coder would say: this neuron codes for stimulus A, and therefore this neuron's firing causes the experience of A. But you could also assume a different situation, maybe a different organism or the same organism conditioned in a different way, where stimulus B, and not A, makes the neuron fire. If neural activity is what causes subjective experience, then this neuron's firing should produce exactly the same experience in both cases, even though different stimuli cause them to fire. This example can be vastly generalized, and the implication is that any two patterns of neural activity that are identical up to a permutation of neurons should produce the same subjective experience for that set of neurons.

As if all this were not puzzling enough, I will now end on a disturbing experiment of thought. Imagine we measure the entire pattern of neural activity of someone experiencing the vision of his grandmother. Then we build a set of blinking red lights, one for each neuron, programmed so as to light up at the same time as the neurons did. The red lights don't even need to be connected to each other. The electrical activity of this set of lights is thus the same as the activity of the neural network. Therefore, by the postulate that electrical activity is what causes subjective experience, the set of lights should experience the sight of the grandmother, with the impression of being the grandson. Would it?

Why does a constant stimulus feel constant? (I)

What is the relationship between neural activity and perception? That is, how does the quality of experience (qualia) relate with neural activity? For any scientist working in sensory neuroscience, this should be a central question – perhaps THE question. Unfortunately the obsession of the community for the question of “neural coding”, which is about relating neural activity with externally defined properties of sensory stimuli, does not help much in this regard. In fact, very naïve philosophical assumptions about this question seem to pervade the field. A popular one is that the perceived intensity of a stimulus corresponds to the firing rate of the neurons that are sensitive to it, in particular sensory receptor neurons. This was indeed an idea proposed by Lord Adrian in the 1920s, and the basic argument is that the firing rate of sensory neurons generally increases when the strength of the stimulus is increased. Clearly the argument is very weak (only an observed correlation), but I will try to refute it explicitly, because it triggers some interesting remarks. To refute it, I will turn the perspective around: why is it that a constant stimulus feels constant? In fact, what is a constant stimulus? This is a very basic question about qualia, but it turns out that it is a surprisingly deep one.

Let us start by listing a few sensory stimuli that feel constant, in terms of perceptual experience. A pure tone (e.g. the sound produced by a diapason or a phone) feels constant. In particular its intensity and pitch seem to be constant. Another example could be a clear blue sky, or any object that you fixate. In the tactile modality, a constant pressure on a finger. For these stimuli, there is a perceived constancy in their qualities, ie, the color of the sky does not seem to change, the frequency of the tone does not seem to change. Attention to the stimulus might fade, but one does not perceive that the stimulus changes. In contrast, neural activity is not constant at all. For a start, neurons fire spikes, and that means that their membrane potential always changes, but we do not feel this change. Secondly, in sensory receptor neurons but also in most sensory neurons in general, the frequency of those spikes changes in response to a constant stimulus: it tends to decrease (“adapt”). But again, a blue sky still feels the same blue (and not darker) and the pitch and intensity of a pure tone do not decrease. There appears to be no such simple connection between neural activity and the perceived intensity of a stimulus. Why is it that the intensity of a pure tone feels constant when the firing rate of every auditory nerve fiber decreases?

In response to this question, one might be tempted to propose a homunculus-type argument: the brain analyzes the information in the responses of the sensory neurons and “reconstructs” the true stimulus, which is constant. In other words, it feels constant because the brain represents the outside world and so it can observe that the stimulus is constant. As I noted a number of times in this blog, there is a big conceptual problem with this kind of argument (which is vastly used in “neural coding” approaches), and that is circular logic: since the output of the reconstruction process is in the external world (the stimulus), how can the brain know what that output might be, as it is precisely the aim of the reconstruction process to discover it? But in fact in this case, the fallacy of this argument is particularly obvious, for two reasons: 1) whereever the representation is supposed to be in the brain, neural activity is still not constant (in particular, made of spikes); 2) even more importantly, in fact what I called “constant stimulus” is physically not constant at all.

Physically, a pure tone is certainly not constant: air vibrates at a relatively high rate, the acoustic pressure at the ear fluctuates. The firing of auditory nerve fibers actually follows those fluctuations, at least if the frequency is low enough (a phenomenon called phase locking), but it certainly doesn't feel this way. Visual stimuli are also never constant, because of eye movements – even when one fixates an object (these are called fixational eye movements). In fact, if the retinal image is stabilized, visual perception fades away quickly. In summary: the sensory stimulus is not constant, neurons adapt, and generally neural activity is dynamic for any stimulus. So the question one should ask is not: how does the brain know that the stimulus is constant, but rather: what is it that make those dynamic stimuli feel perceptually constant?

The challenge of retrograde amnesia to theories of memory

I am reading Oliver Sacks' “The man who mistook his wife for a hat”. On chapter 2, he describes a case of retrograde amnesia. Around 1970, the patient went into an episode of alcoholism, which resulted in the loss of 25 years of his most recent memories (declarative memory). As a result, the patient thought he was 19 and lived in 1945, as if time had stopped. So not only could he not transfer short-term memory to long-term memory, but a large part of his previously stored memory got erased. In addition, it is not a random fraction of his memories that were erased: it was exactly the most recent ones. He seemed to perfectly remember old memories, and have absolutely no memory of the more recent events.

This is quite a challenge for current neural theories of memory. The main theoretical concept about memory is the notion of neural assembly supporting associative memory. Imagine a memory is made of a number of elements that are associated together, then the substrate of this memory is a connected network of neurons that “code” for those elements, in some structure in the brain, with connections to relevant parts of the brain (say, the visual cortex for visual features, etc). This conceptual framework can be extended with sequential activation of neurons. Now in this framework, how do you erase the most recent memories? Note that by “most recent”, I am talking of 25 years, not of short-term memory.

One trivial possibility would be that each memory has a timestamp, encoded as part of the neural assembly supporting that memory. Then some mechanism erases all memories that have a timestamp more recent than a particular date. Why and how this would happen is mysterious. In addition, “reading the timestamp” would entail activating those memories (all of them), which would then need to exist at that time, and then erasing them. It simply sounds absurd. A more plausible explanation is that, for some reason, recent memories are more fragile than old ones. But why is that?

This is a very interesting point, because in current neural theories of memory, it is the old memories that are more fragile than the recent ones. The reason is that memories are imprinted by modifications of synaptic connections according to a Hebbian mechanism (neurons that are co-activated strengthen their connections), and then these connections get degraded over time because of the activation of the same neurons in other contexts, by ongoing activity. So in current theories of memory, memory traces decay over time. But what retrograde amnesia implies is exactly the opposite: memory traces should strengthen over time. How is it possible that memories strengthen over time?

One possibility is that memories are replayed. If you recall a memory, the neurons supporting that memory activate and so the corresponding connections strengthen. But conscious recollection will probably not do the trick, because then there would not be a strict temporal cut-off: i.e., some recent memories might be recalled more often than some older ones. So what seems to be necessary is a continuous subconscious replay of memories, independent of emotional or attentional states. Clearly, this is quite a departure from current neural theories of memory.

Rate vs. timing (XXI) Rate coding in motor control

Motor control is sometimes presented as the prototypical example of rate coding. That is, muscle contraction is determined by the firing rate of motoneurons, so ultimately the “output” of the nervous system follows a rate code. This is a very interesting example, precisely because it is actually not an example of coding, which I previously argued is a problematic concept.

I will briefly recapitulate what “neural coding” means and why it is a problematic concept. “Coding” means presenting some property of things in the world (the orientation of a bar, or an image) in another form (spikes, rates). That a neuron “codes” for something means nothing more than its activity co-varies with that thing. For example, pupillary diameter encodes the amount of light captured by the retina (because of the pupillary contraction reflex). Or blood flow in the primary visual cortex encodes local visual orientation (this is what is actually measured by intrinsic optical imaging). So coding is really about observations made by an external observer, it does not tell much about how the system works. It is a common source of confusion because when one speaks of neural coding, there is generally the implicit assumption that the nervous system “decodes it” somehow. But presumably the brain does not “read-out” blood flow to infer local visual orientation. The coding perspective leaves the interesting part (what is the “representation” for?) largely unspecified, which is the essence of the homunculus fallacy.

The control of muscles by motoneurons does not fit this framework, because each spike produced by a motoneuron has a causal impact on muscle contraction: its activity does not simply co-vary with muscle contraction, it causes it. So first of all, motor control is not an example of rate coding because it is not really an example of coding. But still, we might consider that it conforms to rate-based theories of neural computation. I examine this statement now.

I will now summarize a few facts about muscle control by motoneurons, which are found in neuroscience textbooks. First of all, a motoneuron controls a number of muscle fibers and one fiber is contacted by a single motoneuron (I will only discuss α motoneurons here). There is indeed a clear correlation between muscle force and firing rate of the motoneurons. In fact, each single action potential produces a “muscle twitch”, i.e., the force increases for some time. There is also some amount of temporal summation, in the same way as temporal summation of postsynaptic potentials, so there is a direct relationship between the number of spikes produced by the motoneurons and muscle force.

Up to this point, it seems fair to say that firing rate is what determines muscle force. But what do we mean by that exactly? If we look at muscle tension as a function of a time, resulting from a spike train produced by a motoneuron, what we see is a time-varying function that is determined by the timing of every spike. The rate-based view would be that the precise timing of spikes does not make a significant difference to that function. But it does make a difference, although perhaps small: for example, the variability of muscle tension is not the same if the spike train is regular (small variability) or if it is random, e.g. Poisson (larger variability). Now this gets interesting: during stationary muscle contraction (no movement), those motoneurons generate constant muscle tension and they fire regularly, unlike cortical neurons (for example). Two remarks: 1) this does not at all conform to standard rate-based theory where rate is the intensity of a Poisson process (little stochasticity); 2) regularly firing is exactly what motoneurons should be doing to minimize variability in muscle tension. This latter remark is particularly significant. It means that, beyond the average firing rate, spikes occur at a precise timing that minimizes tension variability, and so spikes do matter. Thus motor control rather seems to support spike-based theories.

Subjective physics

I just finished writing a text about "subjective physics": a term I made up to designate the description of the laws that govern sensory signals and their relationships with actions. It is relevant to systems computational neuroscience, embodiment theories and psychological theories of perception (in particular Gibson's ecological theory and the sensorimotor theory). Here is the abstract:

Imagine a naive organism who does not know anything about the world. It can capture signals through its sensors and it can make actions. What kind of knowledge about the world is accessible to the organism? This situation is analog to that of a physicist trying to understand the world through observations and experiments. In the same way as physics describes the laws of the world obtained in this way by the scientist, I propose to name subjective physics the description of the laws that govern sensory signals and their relationships with actions, as observed from the perspective of the perceptual system of the organism. In this text, I present the main concepts of subjective physics, illustrated with concrete examples.

What is computational neuroscience? (XIX) Does the brain process information?

A general phrase that one reads very often about the brain in the context of perception is that it “processes information”. I have already discussed the term “information”, which is ambiguous and misleading. But here I want to discuss the term “process”. Is it true that the brain is in the business of “information processing”?

“Processing” refers to a procedure that takes something and turns it into something else by a sequence of operations, for example trees into paper. So the sentence implies that what the brain is doing is transforming things into other things. For example, it transforms the image of a face into the identity of the face. The coding paradigm, and more generally the information-processing paradigm, relies on this view.

I will take a concrete example. Animals can localize sounds, based on some auditory cues such as the level difference between the two ears. In the information processing view, what sound localization means is a process that takes a pair of acoustic signals and turns it into a value representing the direction of the sound source. However, this not literally what an animal does.

Let us take a cat. The cat lives and, most of the time, does nothing. Through its ears, it receives a continuous acoustic flow. This flow is transduced into electrical currents, which triggers some activity in the brain, that is, electrical events happening. At some moment in time, a mouse scratches the ground for a second, and the cat turns its eyes towards the source, or perhaps crawls to the mouse. During an extended period of time, the mouse is there in the world, and its location exists as a stable property. What the cat “produces”, on the other hand, is a discrete movement with properties that one can relate to the location of the mouse. Thus, sound localization behavior is characterized by discrete events that occur in a continuous sensory flow. Behavior is not adequately described as a transformation of things into things, because behavior is an event, not a thing: it happens.

The same remark applies to neurons. While a neuron is a thing that exists, a spike is an event that happens. It is a transient change in electrical properties that triggers changes in other neurons. As the terms “neural activity” clearly suggest, a spike is not a “thing” but an event, an action on other neurons or muscles. But the notion of information processing implies that neural activity is actually the end result of a process rather than the process itself. There is a confusion between things and events. In a plant that turns trees into paper, trees and papers are the things that are transformed; the action of cutting trees is not one of these things that are transformed. Yet this is what the information processing metaphor says about neural activity.

There are important practical implications for neural models. Traditionally, these models follow the information-processing paradigm. There is an input to the model, for example a pair of acoustical signals, and there is an output, for example an estimate of sound location (I have worked on this kind model myself, see e.g. Goodman & Brette, PLoS Comp Biol 2010). The estimate is generally calculated from the activity of the neurons over the course of the simulation, which corresponds to the time of the sound. For example, one could select the neuron with the maximum firing rate and map its index to location; or one could compute estimate based on population averages, etc. In any case, there is a well-defined input corresponding to a single sound event, and a single output value corresponding to the estimated location.

Now try to embed this kind of model into a more realistic scenario. There is a continuous acoustic flow. Sounds are presented at various locations in sequence, with silent gaps between them. The model must estimate the locations of these sounds. We have a first problem, which is that the model produces estimates based on total activity over time, and this is clearly not going to work here since there is a sequence of sounds. The model could either produce a continuous estimate of source location (the equivalent of continuously pointing to the source), or it could produce an estimate of source location at specific times (the equivalent of making a discrete movement to the source), for example when the sounds stop. In either case, what is the basis for the estimate, since it cannot be the total activity any more? If it is a continuous estimate, how can it be a stable value if neurons have transient activities? More generally, how can the continuous flow of neural activity produce a discrete movement to a target position?

Thus, sound localization behavior is more than a mapping between pairs of signals and direction estimates. Describing perception as “information processing” entails the following steps: a particular time interval of sensory flow is selected and considered as a thing (rather than a flow of events); a particular set of movements is considered and some of its properties are extracted (e.g. direction); what the brain does is described as the transformation of the first thing into the second thing. Thus, it is an abstract construction by an external observer.

Let me summarize this post and the previous one. What is wrong about “information processing”? Two things are wrong. First (previous post), the view that perception is the transformation of information of some kind into information of another kind is self-contradictory, because a signal can only be considered “information” with respect to a perceptual system. This view of perception therefore proposes that there are things to be perceived by something else than the perceptual system. Second (this post), “processing” is the wrong term because actions produced by the brain are not things but events: it is true at the scale of the organism (behavior) and it is true at the scale of neurons (spikes). Both behavior and causes of behavior are constituted by events, not things. It is also true of the mind (phenomenal consciousness). A thing can be transformed into another thing; an event happens.

What is computational neuroscience? (XVIII) Representational approaches in computational neuroscience

Computational neuroscience is the science of how the brain “computes”: how it recognizes faces or identifies words in speech. In computational neuroscience, standard approaches to perception are representational: they describe how neural networks represent in their firing some aspect of the external world. This means that a particular pattern of activity is associated to a particular face. But who makes this association? In the representational approach, it is the external observer. The approach only describes a mapping between patterns of pixels (say) and patterns of neural activity. The key step, of relating the pattern of neural activity to a particular face (which is in the world, not in the brain), is done by the external observer. How then is this about perception?

This is an intrinsic weakness of the concept of a “representation”: a representation is something (a painting, etc) that has a meaning for some observer, it is not about how this meaning is formed. Ultimately, it does not say much about perception, because it simply replaces the problem of how patterns of photoreceptor activity lead to perception by the problem of how patterns of neural activity lead to perception.

A simple example is the neural representation of auditory space. There are neurons in the auditory brainstem whose firing is sensitive to the direction of a sound source. One theory proposes that the sound's direction is signaled by the identity of the most active neuron (the one that is “tuned” to that direction). Another one proposes that it is the total firing rate of the population, which covaries with direction, that indicates sound direction. Some other theory considers that sound direction is computed as a “population vector”: each neuron codes for direction, and is associated a vector oriented in that direction, with a magnitude equal to its firing rate; the population vector is sum of all vectors.

Implicit in these representational theories is the idea that some other part of the brain “decodes” the neural representation into sound's direction, which ultimately leads to perception and behavior. However, this part is left unspecified in the model: neural models stop at the representational level, and the decoding is done by the external observer (using some formula). But the postulate of a subsequent neural decoder is problematic. Let us assume there is one. It takes the “neural representation” and transforms it into the target quantity, which is sound direction. But the output of a neuron is not a direction, it is a firing pattern or rate that can perhaps be interpreted as a direction. So how is sound direction represented in the output of the neural decoder? It appears that the decoder faces the same conceptual problem, which is that the relationship between output neural activity and the actual quantity in the world (sound direction) has to be interpreted by the external observer. In other words, the output is still a representation. The representational approach leads to an infinite regress.

Since neurons are in the brain and things (sound sources) are in the world, the only way to avoid an external “decoding” stage that relates the two is to include both the world and the brain in the perceptual model. In the example above, this means that, to understand how neurons estimate the direction of a sound source, one would not look for the “neural representation” of sound sources but for neural mechanisms that, embedded in an environment, lead to some appropriate orienting behavior. In other words, neural models of perception are not complete without an interaction with the world (i.e., without action). In this new framework, “neural representations” become a minor issue, one for the external observer looking at neurons.