In two companion papers in Neural Computation (followed by a related paper on working memory), Sophie Denève developed a spike-based theory of Bayesian inference. It can be categorized as a representational spike-based theory, in the sense that spikes collectively represent some objective variable of the world, for which there is some uncertainty. It follows a typical Marr-ian approach, in which the function of the neural network (level 1) is first postulated, in terms of external properties of the world, and then the properties of the network (dynamics, connectivity) are derived. But unlike Marr’s approach, the algorithmic and physical levels are not considered independent, that is, the algorithm is defined directly at the level of spikes. In the first paper, it is assumed that a neuron codes for a hidden variable, corresponding to an external property of the world. The neuron must infer that variable from a set of observations, which are independent Poisson inputs whose rates depend on the binary value. The neuron codes for other neurons (as opposed to the external observer), that is, it is postulated that the log odds of the hidden variable are estimated from the spike train produced by the train as a sum of PSPs in a target neuron. Thus, the decoding process is fixed, and the dynamics of the neuron can then be deduced from an optimization principle, that is, so that the decoded quantity is as close as possible to the true quantity.

One can write a differential-spike equation that describes how the log-odds evolve with the inputs. At any time, the estimated log-odds can be estimated by the neuron from its output spike train. A spike is then produced if it brings the estimation closer to the true value of the log-odds, calculated from the inputs.

In this proposition, the spiking process is deterministic and spiking is seen as a timed decision. This is completely different from rate-based theory, in which spikes are random and instantiate a time-varying quantity. Although this is a rather abstract sensory scenario, the idea that spikes could be corrective signals is powerful. It connects to the point I made in the previous post, that the key point in spike-based theories is not temporal precision or reproducibility, as is sometimes wrongly claimed, it is the fact that spikes between different neurons are coordinated. When this theory is extended to a population of neurons, an immediate consequence is that spiking decisions are dependent on the decisions made by other neurons, and it follows that although spiking is deterministic at the cellular level and precise at the functional level (estimation of the hidden variable), it may not be reproducible between trials. In fact, even the relative timing of neurons may not be reproducible – for exactly the same reason as in sparse coding theory.

Spike-based theories are sometimes discarded on the basis that spike timing is not reproducible in vivo, in response to the same stimulus. I already argued that, in addition to the fact that this is a controversial statement (because for example this could be due to a lack of control of independent variables such as attentional state), this is not a case for rate-based theories but for stochastic theories.

But I think it also reveals a misunderstanding of the nature of spike-based theories, because in fact, even deterministic spike-based theories may predict irreproducible spike timing. Underlying the argument of noise is the assumption that spikes are produced by applying some operation on the stimulus and then producing the spikes (with some decision threshold). If the timing of these spikes is not reproducible between trials, so the argument goes, then there must be noise inserted at some point in the operation. However, spike-based theories, at least some of them, do not fit this picture. Rather, the hypothesis is that spikes produced by different neurons are coordinated so as to produce some function. But then there is no reason why spikes need to be produced at the same time by the same neurons in all trials in order to produce the same global result. What matters is that spikes are precisely coordinated, which means that the firing of one neuron depends on the previous firing of other neurons. So if one neuron misses a spike, for example, then it will affect the firing of other neurons, precisely so as to make the computation more reliable. In other words, it is implied by the hypothesis of precise spike-based coordination that the firing of a spike by a single neuron should impact the firing of all other neurons, which makes individual firing non-reproducible.

The theory of sparse coding is line with this idea. In this theory, it is postulated that the stimulus can be reconstructed from the firing of neurons. That is, each spike contributes a “kernel” to the reconstruction, at the time of the spike, and all such contributions are added together so that the reconstruction is as close as possible to the original stimulus. Note how this principle is in some way the converse of the previously described principle: the spikes are not described as the result of a function applied to the stimulus, but rather the stimulus is described as a function of the spikes. So spike encoding is defined as an inverse problem. This theory has been rather successful in explaining receptive fields in the visual (Olshausen) and auditory (Lewicki) systems. It is also meant to make sense from the point of view of minimizing energy consumption, as it minimizes the number of spikes required to encode the stimulus with a given precision. There are two interesting points here, regarding our present discussion. First, it appears that spikes are coordinated in the way I just described above: if one spike is missed, then the other spikes should be produced so as to compensate for this loss, which means there is a precise spike-based coordination between neurons. Second, the pattern of spikes is seen as a solution to an inverse problem. This implies that if the problem is degenerate, then there are several solutions that equally good in terms of reconstruction error. Imagine for example that two neurons contribute exactly the same kernel to the reconstruction – which is not useless if one considers the fact that firing rate is limited by the refractory period. Then on one given trial, either of these two neurons may spike. From the observer point of view, this represents a lack of reproducibility. However, this lack of reproducibility is precisely due to the fact that there is a precise spike-based coordination between neurons: to minimize the reconstruction error, just one of the two neurons should be active, and the timing should be precise too.

Sparse coding with spikes also implies that reproducibility should depend on the stimulus. That is, a stimulus that is highly redundant such as a sinusoidal grating makes a degenerate inverse problem, leading to lack of reproducibility of spikes, precisely because of the coordination between spikes; a stimulus that is highly informative such a movie of a natural scene should lead to higher reproducibility of spikes. Therefore, in the sparse coding framework, the spike-based coordination hypothesis predicts, contrary to rate-based theories, that spike time reproducibility should depend on the information content of the stimulus – in the sense that a more predictable stimulus leads more irreproducible spiking. But even when spiking is not reproducible, it is still precise.

Following on the previous post, with the analog-digital analogy often comes the idea that the relation between rates and spikes is that of an analog-digital conversion. Or spikes are seen as an analog-digital conversion from the membrane potential. I believe this comes from the evolutionary argument that it seems that spikes appeared for fast propagation of information on long distances, and not because there is anything special about them in terms of computation. It is quite possible that this was indeed the evolutionary constraint that led to the appearance of action potentials (although this is pure speculation), but even if this is true, the reasoning is wrong: for example, the ability of humans to make tools might have developed because they stood up. Yet standing up does not explain tool-making at all. So standing up allows new possibilities, but these possibilities follow a distinct logic. Spikes might have appeared primarily to transmit information at long distances, but once they are there, they have properties that are used, possibly for other purposes, in new ways. In addition, that they appeared to transmit information and the information was analog does not mean information is now used in the same way. Consider: to transmit information over long distances, one uses Morse code on the telegraph. Do you speak to the telegraph? No, you change the code and use a discrete code that has little connection with the actual sound wave. Finally, even if all this makes sense, it still is not an argument in favor rate-based theories, because rate is an abstract quantity that is derived from spikes. So if we wanted to make the case that spikes are only there to carry a truly analog value, the membrane potential, then it would lead us to discard spikes as a relevant descriptive quantity, and a fortiori to discard rates as well. From a purely informational viewpoint (in the sense of Shannon), spikes produced by a neuron carry less information than its membrane potential, but rate carries even less information, since it is abstracted from spikes.

It is sometimes stated that rate-based computing is like analog computing while spike-based computing is like digital computing. The analogy comes from the fact, of course, that spikes are discrete whereas rates are continuous. But as any analogy, it has its limits. First of all, spikes are not discrete in the way digital numbers are discrete. In digital computing, the input is a stream of binary digits, coming one after another in a cadenced sequence. The digits are gathered by blocks, say of 16 or 32, to form words that stand for instructions or numbers. Let us examine these two facts with respect to spikes. Spikes do not arrive in a cadenced sequence. Spikes arrive at irregular times, and time is continuous, not digital. What was meant by digital is presumably that there can be a spike or there can be no spike, but there is nothing in between. However, given that there is also a continuous timing associated to the occurrence of a spike, a spike is better described as a timed event rather than as a binary digit. But of course one could decide to divide the time axis into small time bins, and associate a digit 0 when there is no spike and 1 when there is a spike. This is certainly true, but as one performs this process as finely as possible to approximate the real spike train, it appears that there are very few 1s drowned in a sea of 0s. This is what is meant by “event”: information is carried by the occurrence of 1s at specific times rather than by the specific combinations of 0s and 1s, as in digital computing. So in this sense, spike-based computing is not very similar to digital computing.

The second aspect of the analogy is that digits are gathered in words (of say 32 digits), and these words are assigned a meaning in terms of either an instruction or a number. Transposed to spikes, these “words” could be the temporal pattern of spikes of a single neuron, or perhaps more meaningfully a pattern of spikes across neurons, as in synchrony-based schemes, or across neurons and time, as in polychronization. Now there are two ways of understanding the analogy. Either a spike pattern stands for a number, and in this case the analogy is not very interesting, since this is pretty much saying that spikes implement an underlying continuous value, in other words this is the rate-based view of neural computation. Or a spike pattern stands for a symbol. This case is more interesting, and it may apply to some proposed spike-based schemes (like polychronization). It emphasizes the idea that unlike rate-based theories, spike-based theories are not (necessarily) related to usual mathematical notions of calculus (e.g. adding numbers), but possibly to more symbolic manipulations.

However, this does not apply to all spike-based theories. For example, in Sophie Denève’s spike-based theory of inference (which I will describe in a future post), spike-based computation actually implements some form of calculus. But in her theory, analog signals are reconstructed from spikes, in the same way as the membrane potential results from the action of incoming spikes, rather than the other way around as in rate-based theories (i.e., a rate description is postulated, then spikes are randomly produced to implement that description). So in this case the theory describes some form of calculus, but based on timed events.

This brings me to the fact that neurons do not always interact with spikes. For example, in the retina, there are many neurons that do not spike. There are also gap junctions, in which the membrane potentials of several neurons directly interact. There are also ephaptic interactions (through the extracellular field potential). There is also evidence that the shape of action potentials can influence downstream synapses (see a recent review by Dominique Debanne). In these cases, we may speak of analog computation. But this does not bring us closer to rate-based theories. In fact, quite the opposite: rates are abstracted from spikes, and stereotypical spikes are an approximation of what really goes on, which may involve other physical quantities. The point here is that firing rate is not a physical quantity as the membrane potential, for example. It is an abstract variable. In this sense, spike-based theories, because they are based on actual biophysical quantities in neurons, might be closer to what we might call “analog descriptions” of computation than rate-based theories.

I started with an overview of spike-based theories based on synchrony. I would like to stress that synchrony-based theories should not be mistaken for theories that predict widespread synchrony in neural networks. In fact, quite the opposite: as synchrony is considered as a meaningful event, it is implied that it is a rare event (since otherwise it would not be informative). But there are also theories that do not assign any particular role to synchrony, which I will discuss now.

One popular theory based on asynchrony is rank order coding or “first spike” theories. It was popularized in particular by Simon Thorpe, who showed that humans can categorize faces in such a small time that any neuron involved in the processing chain could fire little more than one spike. This observation discards theories based on temporal averages, both rate-based theories and interval-based theories. Instead, Simon Thorpe and colleagues proposed that the information is carried by the order in which spikes are fired. Indeed, receptors that are more excited (receiving more light) generally fire earlier, so that the order in which receptors are excited carries information that is isomorphic to the pattern of light on the retina. However, by itself, the speed of processing does not discard processing schemes that do not require temporal averages, for example synfire chains or rate-based schemes based on spatial averages. Indeed it is known that the speed at which the instantaneous firing rate of a population of noisy neurons can track a time-varying input is very fast and is not limited by the membrane time constant – an interesting point is that this fact is consistent with integrate-and-fire models and not with isopotential Hodgkin-Huxley models, but this is another story. However, one argument against rate-based schemes is that they are much less energetically efficient, since information is used only after averaging. To be more precise, a quantity can theoretically be estimated from the firing of N neurons with precision of order 1/N if their responses are coordinated, but of order 1/√N if the neurons are independent. In other words, the same level of precision requires N² neurons in a rate-based scheme, vs. N neurons in a spike-based scheme.

Computationally, first spike codes are not fundamentally different, at a conceptual level, from standard rate-based codes, because first spike latency is monotonically related to input intensity. However, one interesting difference is that if only the rank order, and not the exact timing, is taken into account, then this code becomes invariant to monotonous transformations of input intensity, for example global changes in contrast or luminance. However it is not invariant to more complex transformations.

Rank order codes are also different at a physiological level. Indeed an interesting aspect of this theory is that it acknowledges a physiological fact that is ignored by both rate-based theories and synchrony-based theories, namely the asymmetry between excitation and inhibition in neurons. How can a neuron be sensitive to the temporal order of its inputs? In synchrony-based theories, which rely on excitation, neurons are sensitive to the relative timing of their inputs rather than to their temporal order. Indeed temporal order is discontinuous with respect to relative timing: it abruptly switches at time lag 0. Such a discontinuity is provided by inhibition: excitation followed by inhibition is more likely to trigger a spike than inhibition followed by excitation. The asymmetry is due to the fact that spikes are produced when the potential exceeds a positive threshold (i.e., the trajectory crosses the threshold from below).

One criticism of rank order coding is that it requires a time reference. Indeed, when comparing two spike trains, any spike is both followed and preceded by a spike from the other train, unless only the “first spike” is considered. Such a time reference, which defines the notion of “first spike”, could be the occurrence of an ocular saccade, or the start of an oscillation period if there is a global oscillation in the network that can provide a common reference.

I finish this overview of synchrony-based theories with my recent proposal (PLoS Comp Biol 2012). In the next posts, I will discuss theories based on asynchrony. In the theories I have described so far, the starting point is a code based on spike timing, in general a spatiotemporal pattern of spikes assumed to represent some sensory input. But the connection between the sensory input and the spike pattern is not addressed, or at least not considered as a central issue. My proposition connects spike-based computation with the psychological theory of James Gibson, specifically the notion of structural invariant. Gibson starts his book “An ecological approach to visual perception” by criticizing the idea that perception is the process of inferring the objective properties of the world from ambiguous patterns of sensory data, as is often postulated. Indeed, since perception is the source of all knowledge, it is inconsistent to view the objective properties of the world as preexisting to perception. But how then can one know anything about the world?

I will rephrase Gibson’s thinking in a different way by using the dictionary analogy. Inferring the objective world from an image or some sensory data is like looking in a dictionary for the translation of a word in one’s native language. In fact, this is precisely what is generally meant by the “neural coding” metaphor. But this cannot be used to understand a new word in one’s own native language. Instead one uses a different kind of dictionary, in which the word is defined in relationship with other words. Thus the definition of objects in the world is relational, not inferential. Inference can only be secondary, since one must first know what is to be inferred.

How does this relate to perception? Gibson argues that information about the world is present in the invariant structure of sensory inputs, that is, in properties of sensory inputs (relationships) that persist through time, which is to say: the laws that sensory inputs follow. More precisely, invariant structure is a relationship that is invariant with respect to some change. The notion has been extended to sensorimotor relationships by Kevin O’Regan. Two simple examples in hearing are in pitch perception and sound localization. Sounds that evoke a pitch are generally periodic. Periodicity is a relationship on the sensory input, i.e., S(t+T)=S(t) for all times (T is the period and S(t) is the acoustical pressure), and it is precisely this relationship, rather than the spectrum of the sound, which defines the pitch (for example pitch is unchanged if the fundamental frequency is missing). This relationship is not spatial because it is unaffected by movements. In the same way, a sound source produces two acoustical waves at the two ears that have specific relationships, for example (if sound diffraction is neglected) the wave at the contralateral ear is a delayed version of the wave at the ipsilateral ear. This relationship is spatial because it is affected by one’s movements. Besides, there is a systematic relationship between interaural delay and head position that is isomorphic to the source direction. Therefore this relationship can be identified to the source direction, without the need for an externally defined notion of physical angle. When a sound is presented in a noisy environment, the direction has to be inferred since it is ambiguous, but what is inferred is the relationship that defines the direction.

How does this relate to synchrony? Simply put, synchrony is a relationship defined through time, so it qualifies as invariant structure. In my paper, I show how this relationship between spike timings can correspond to a relationship between sensory inputs by introducing the concept of “synchrony receptive field” (SRF). The SRF of a given pair of neurons is the set of sensory signals that elicit synchronous spiking in the two neurons (it can be extended to a group of neurons). Suppose the two neurons receive different versions of the sensory signal S: F(S) and G(S). Then assuming a deterministic mapping from signals to spikes, synchrony reflects the relationship F(S) = G(S), a relationship defined on the sensory inputs. Therefore, across the neural population, synchrony patterns reflect the set of relationships on the sensory inputs, and neurons that respond to coincidences signal these relationships.

This mechanism can be used practically to recognize relationships, for example: detecting an odor defined by ratios of receptor affinities, estimating the pitch of a sound, estimating the location of a sound source (Goodman and Brette, PLoS CB 2010), estimating binocular disparities. The key computational benefit is that it solves the difficult problem of invariance, that is, the fact that objects of perception are invariant under many different perspectives. For example a face can be seen under different angles, or the same sound source can produce different sounds at the same location. To be more precise, the problem is dissolved by this approach rather than solved. Indeed, the key insight is that invariance is only a problem for an inferential process. When the objects to be perceived are defined instead by relationships, then there is no invariance problem, since a relationship is itself an invariant. For example, periodicity is a relationship that is invariant to the spectrum of a sound.

The theory connects to the other spike-based theories I mentioned previously. Indeed sensory relationships are reflected by synchrony between neurons (or relative spike timing, considering conduction delays), and it addresses the problem of binding in the same way as synfire chains: sensory signals that are not temporally coherent, and therefore not originating from the same object, cannot produce synchronous firing. It also connects with the polychronous theory of working memory: the spike pattern that is stored in that theory corresponds here to a sensory relationship. This makes it is possible to store sensory relationships in the form of spike timing relationships, without the need for an explicit conversion to a “rate code”.

On the empirical side, the theory relies on the fact that neurons operate in a fluctuation-driven regime, in which excitation and inhibition are approximately balanced (or inhibition dominant), as empirically observed. But shouldn’t this theory predict widespread synchrony in neural populations, unlike what is observed in the brain? In fact it should not. First of all, synchrony is only informative if it is a rare event. This is precisely what is captured by the concept of synchrony receptive field: synchrony occurs only for specific sensory signals (or more precisely, sensory relationships). Even though I did not include it in the paper, it would actually make sense that correlations that are not stimulus-specific (i.e., those that can be predicted) are minimized as much as possible. This would support the idea that recurrent inhibition is tuned to cancel excitatory correlations (see my previous post), which would produce weak correlations on average.