In a synfire chain, activity propagates synchronously from one layer to the next. Transmission delays are identical for all synaptic connections between two layers. Bienenstock proposed the notion of synfire braid for the case when transmission delays are heterogeneous (Bienenstock (1994), A model of neocortex, see Appendix A). The idea was expanded by Izhikevich with the terminology “polychronization” (Izhikevich, Neural Comp 2006; Szatmary and Izhikevich, PLoS CB 2010), and related to Edelman’s theory of neural darwinism (see his 1988 book).

Polychronization relies on the same propagation mechanism as synfire chains, but a polychronous group differs from a synfire chain in that there are no layers per se. When a set of neurons fire spikes at times such that, added to the transmission delays, they arrive simultaneously at a common target neuron, this neuron may fire. A spatio-temporal pattern of activity congruent with synaptic delays may then propagate along the “polychronous group”. Polychronization is a natural generalization of synfire chains, but there are interesting new properties. First, in a recurrent neural network, there are potentially many more polychronous groups than neurons, unlike synfire chains, and each neuron may participate in many such groups. In fact, in theory, the exact same set of neurons can participate in two different groups, but in a different activation order. In a recurrent network, polychronous groups spontaneously ignite at random times and for short durations. This means in particular that repeating spatiotemporal patterns would be very difficult to observe in spontaneous activity. It is important to emphasize that “polychronization” is not a specific mechanism, in that it does not rely on particular anatomical or physiological mechanisms other than what is currently generally accepted. It occurs in balanced excitatory-inhibitory networks with irregular activity and standard spike-timing-dependent plasticity, and in such conditions neurons are known to be highly sensitive to coincidences. In other words, it is a particular perspective on the dynamics of such networks.

The interesting application of polychronization is working memory (Szatmary and Izhikevich 2010). As I mentioned, the general theoretical context is Edelman’s theory of neural darwinism. Edelman got the Nobel prize in the 1970s for his work on the immune system, and he then moved to neuroscience where he developed a theory that relies on an analogy with the immune system. In that system, there is a preexisting repertoire of antibodies, which are more or less random. When a foreign antigen is introduced, it binds to some of these antibodies, and the response is then amplified with clonal multiplication – much like in Darwinian evolution theory. Here a memory item is presented under the form of a specific spatiotemporal pattern of activation. This pattern may be congruent with some of the connection delays of the network, i.e., it may correspond to a polychronous group (or part of one): this corresponds to the binding of antibodies to an antigen. It is then hypothesized that these connections are amplified through quick associative plasticity, i.e., STDP acting on a short time scale, fading over about 10 seconds (one hypothesis relies on NMDA spikes). Note that this is very similar in spirit to von der Malsburg’s “dynamic link matching”. This step corresponds to clonal amplification in the immune system. Because of the reinforcement of the connections, the polychronous group is spontaneously reactivated at random times, until it ultimately fades out. In this way the spatiotemporal pattern is replayed for a few tens of seconds.

The articulation of this theory with empirical evidence is rather interesting in the context of the rate vs. timing debate. First, it builds on generally accepted empirical evidence, including irregular firing statistics and excitatory-inhibitory balance. The theory is based on precise spike timing, but spike timing is not reproducible. Indeed spike timing is not locked to the stimulus, and only relative spike timing matters. What is more, in different trials, different polychronous groups may be selected and therefore even relative spike timing may not be reproducible across trials (but probably within a trial). Another interesting observation is that it predicts that the firing rate of some neurons should show an increasing (ramping) firing rate after stimulus presentation. This is not because these neurons “encode duration” with their rate, but because as the polychronous group is spontaneously reactivated, its length progressively increases, which means that neurons in the group’s tail fire more and more often through the course of reactivation.

Without judging the validity of the polychronization theory, I note that it provides a concrete example of a spike-based theory that appears consistent with many aspects of neural statistics (irregularity, lack of reproducibility, etc). This fact demonstrates once more that these aspects cannot be used as arguments in support of rate-based theories.

Before I discuss polychronization, I would like to complement the previous post with a brief discussion of the binding problem, and the idea of binding by synchrony. Synfire chains address this issue (see in particular Bienenstock (1994), “A model of neocortex”), but they are not the only such proposition based on spike timing. In the 1980s, Christoph von der Malsburg developed a theory of neural correlations as a binding mechanism (1981, “The correlation theory of brain function”). He wrote a very clear review in a special issue of Neuron on the binding problem (1999).

In classical neural network theory, objects are represented by specific assemblies of neurons firing together. A problem arises when two objects are simultaneously presented, the “superposition catastrophe”: now two assemblies are simultaneously active, and it may be ambiguous whether the entire set of neurons corresponds to one big assembly or to two smaller assemblies, and if there are two assemblies it may be unclear which neurons belong together. There is a specific example due to Rosenblatt (1961, “Principles of neurodynamics”): suppose there are two neurons coding for shape, one for squares and another for triangles, and two neurons coding for location, one for top and another for bottom. This type of coding scheme, which von der Malsburg calls “coarse coding”, is efficient when there are many possible dimensions, because the number of possible combinations increases exponentially with the number of dimensions. But in the classical neural network framework, it fails when a square and a triangle are simultaneously presented: indeed all four neurons are activated, and it is impossible to tell whether the square is above the triangle.

The proposed solution to the superposition catastrophe is to use spike timing as a signature of objects. In this example, neurons coding for the same object would fire at the same time, so that the two neural assemblies can be distinguished unambiguously. von der Malsburg mentions in this review that the source of synchrony can be external (as in my paper on computing with synchrony), or internal – one proposition being that binding is mediated by transient gamma oscillations (see Wolf Singer’s work). But he also warns that it may be difficult to find experimental evidence of such a mechanism, because in his mind synchrony should be transient since it has to be dynamic, and should involve many neurons so as to immediately impact postsynaptic neurons (this is related to my post on the difference between synchrony and correlation). Thus to observe such transient distributed synchrony requires to simultaneously record a large number of neurons, and perhaps to know which of these neurons are relevant.

He postulates that binding requires a fast synaptic plasticity mechanism, “dynamic link matching”, the ability to quickly form connections between neurons coding for different features. The idea applies to the binding problem, i.e., representing several objects at the same time, but also to the compositionality problem, which is slightly different and perhaps more general. A good example, perhaps, is working memory for melodies. A melody is a sequence of musical notes with specific pitch and duration. One can hear a melody for the first time and repeat it (e.g. sing it). If the melody is really novel, this ability cannot rely on “combination cells”, it really is the particular sequence of notes that must be kept in memory. This example instantiates both the binding problem and the compositionality problem. There is a binding problem because each note is defined by two properties, pitch and duration, that have to be bound together. There is a compositionality problem because the notes must be organized in the correct order, and not just as a set of notes. So what needs to be represented is not just a set of features, but the organization (the links) between these features, the knowledge of which note goes after each note. In mathematical terms, one needs to represent an oriented graph between the individual notes. For this problem, classical connectionism seems insufficient – for example neural models of working memory based on attractors.

To summarize this post and the previous post, in these approaches, spike-based theories were introduced to address two shortcomings of standard rate-based neural network theories: the binding problem and the compositionality problem. Therefore, if contenders of these theories still need to find empirical evidence, contenders of rate-based theories also still need to respond to these criticisms – compositionality being probably the most difficult problem.

I will now give a brief overview of different spike-based theories. This is not meant to be an exhaustive review – although I would be very glad to have some feedback on what other theories would deserve to be mentioned. My aim is rather to highlight a common theme in these theories, which distinguishes them from rate-based theories. At the same time, I also want to emphasize the diversity of these theories.

“Synfire chains” were introduced by Moshe Abeles in the 1980s (see his 1991 book: Corticonics: Neural Circuits of the Cerebral Cortex), although in fact the concept can be traced back to 1963 (Griffith, Biophysical Journal 1963). It is based on the observation that neurons are extremely sensitive to coincidences in their inputs (in the fluctuation-driven regime) - I commented on this fact in a previous post. So if a small number of input spikes are received synchronously in one neuron (on the order of ten), then the neuron spikes. Now if these presynaptic neurons also have postsynaptic neurons in common, then these postsynaptic neurons will also fire synchronously. By this mechanism, synchronous activity propagates from one group of neurons to another group. This is a feedforward mode of synchronous propagation along a chain of layers (hence the terms “synfire chains”), but note that this feedforward mode of propagation may be anatomically embedded in a recurrent network of neurons. It is important to note that this propagation mode 1) can only be stable in the fluctuation-driven regime (as opposed to the mean-driven regime), 2) is never stable for perfect integrators (without the leak current). A simple explanation of the phenomenon was presented by Diesmann et al. (Nature 1999), in terms of dynamical systems theory (synchronous propagation is a stable fixed point of the dynamics). In that paper, neurons are modeled with background noise – it is not a deterministic model. Much earlier, in 1963, Griffith presented the deterministic theory of synfire chains (without the name) in discrete time and with binary neurons (although he did consider continuous time at the end of the paper). He also considered the inclusion of inhibitory neurons in the chain, and showed that it could yield stable propagation modes with only a fraction of neurons active in each layer. Izhikevich extended the theory of synfire chains to synchronous propagation with heterogeneous delays, which he termed “polychronization” (I will discuss it later).

As I have described them so far, synfire chains are postulated to result from the dynamics of neural networks, given what we know of the physiology of neurons. This raises two questions: 1) do they actually exist in the brain? and 2) are they important? I do not think there is a definite empirical answer to the first question (especially considering possible variations of the theory), but it is interesting to consider its rejection. If one concludes, on the basis of empirical evidence, that synfire chains do not exist, then another question pops up: why are they not observed, given that they seem to be implied by what we know of neural physiology? Possibly then, there could be specific mechanisms to avoid the propagation of synchronous activity (e.g. cancellation of expected correlations with inhibition, which I mentioned in a previous post). I am not taking side here, but I would simply like to point out that, because the existence of synfire chains is deduced from generally accepted knowledge, the burden of proof should in fact be shared between the tenants of synfire chains and their opponents: either synfire chains exist, or there is a mechanism to prevent them from existing (or at least their non-existence deserves an explanation). Before someone objects that models of recurrent spiking neural networks do not generally display synfire activity, I would like to point out that these are generally sparse random networks (i.e., with no short cycle in the connectivity graph) in which the existence of the irregular state is artificially maintained by external noisy input (e.g. Brunel (2000)), or by a suprathreshold intrinsic current (e.g. Vogels and Abbott 2005).

The second question is functional: what might be the computational advantage of synfire chains? As I understand it, they were not introduced for computational reasons. However, a computational motivation was later proposed in reference to the binding problem, or more generally compositionality by Bienenstock (1996, in Brain Theory). A similar proposition was also made by Christof von der Marlsburg and Wolf Singer, although in the context of oscillatory synchrony, not synfire chains. A scene is composed of objects that have relationships to each other. Processing such a scene requires identifying the properties of objects, but also identifying the relationships between these objects, e.g. that different features belong to the same object. In a rate-based framework, the state of the network at a given time is given a vector of rates (one scalar value per neuron). If the activation of each neuron or set of neurons represents a given property, then the network can only represent an unstructured set of properties. The temporal organization of neural activity may then provide the required structure. Neurons firing in synchrony (at a given timescale) may then represent properties of the same object. The proposition makes sense physiologically because presynaptic neurons can only influence a postsynaptic target if they fire together within the temporal integration window of the postsynaptic neuron (where synchrony is seen from the postsynaptic viewpoint, i.e., after axonal propagation delays). In my recent paper on computing with neural synchrony (which is not based on synfire chains but on stimulus-specific synchrony), I show on an olfactory example how properties of the same object can indeed be bound together by synchrony. I also note that it also provides a way to filter out irrelevant signals (noise), because these are not temporally coherent.

Apart from compositionality, the type of information processing performed by synfire chains is very similar to those of feedforward networks in classical neural network theory. That is, the activation (probability of firing) of a given unit is essentially a sigmoidal function of a weighted sum of activations in the preceding layer (if heterogeneous synaptic weights are included) – neglecting the temporal dispersion of spikes. But there may be another potential computational interest of synfire chains, compared to traditional rate-based feedforward models, which is processing speed. Indeed, in synfire chains, the propagation speed is limited by axonal conduction delays, not by neural integration time.

In the next post, I will comment on polychronization, an extension of synfire chains that includes heterogeneous delays.

According to the rate-based hypothesis, 1) neural activity can be entirely described by the dynamics of underlying firing rates and 2) firing is independent between neurons, conditionally to these rates. This hypothesis can be investigated in models of spiking neural networks by a self-consistency strategy. If all inputs to a neuron are independent Poisson processes, then the output firing rate can be calculated as a function of input rates. Rates in the network are then solutions of a fixed point equation. This has been investigated in random networks in particular by Nicolas Brunel. In a statistically homogeneous network, theory gives the stationary firing rate, which can be compared to numerical simulations. The approach has also been applied to calculate self-sustained oscillations (time-varying firing rates) in such networks. In general, theory works nicely for sparse random networks, in which a pair of neurons is connected with a low probability. Sparseness implies that there are no short cycles in the connectivity graph, so that the fact that the inputs to a neuron and its output are strongly dependent has little impact on the dynamics. Results of simulations diverge from theory when the connection probability increases. This means that the rate-based hypothesis is not true in general. On the contrary, it relies on specific hypotheses.

Real neural networks do not look like random sparse networks, for example they can be strongly connected locally, neurons can be bidirectionally connected or form clusters. Recently, there have been a number of nice theoretical papers on densely connected balanced networks (Renart et al., Science 2010; Litwin-Kumar and Doiron, Nat Neurosci 2012), which a number of people have interpreted as supporting rate-based theories. In such networks, when inhibition precisely counteracts excitation, excitatory correlations (due to shared inputs) are cancelled by the coordination between inhibition and excitation. As a result, there are very weak pairwise correlations between neurons. I hope it is now clear from my previous posts that this is not an argument in favor of rate-based theories. The fact that correlations are small says nothing about whether dynamics can be faithfully described by underlying time-varying rates.

In fact, in such networks, neurons are in a fluctuation-driven regime, meaning that they are highly sensitive to coincidences. What inhibition does is to cancel the correlations due to shared inputs, i.e., the meaningless correlations. But this is precisely what one would want the network to do in spike-based schemes based on stimulus-specific synchrony (detecting coincidences that are unlikely to occur by chance) or on predictive coding (firing when there is a discrepancy between input and prediction). In summary, these studies do not support the idea that rates are an adequate basis for describing network dynamics. They show how it is possible to cancel expected correlations, a useful mechanism in both rate-based and spike-based theories.

Update. These observations highlight the difference between correlation and synchrony. Correlations are meant as temporal averages, for example pairwise cross-correlation. But on a timescale relevant to behavior, temporal averages are irrelevant. What might be relevant are spatial averages. Thus, synchrony is generally meant as the fact that a number of neurons fire at the same time, or a number of spikes arrive at the same time at a postsynaptic neuron. This is a transient event, which may not be repeated. A single event is meaningful if such synchrony (possibly involving many neurons) is unlikely to occur by chance. The terms “by chance” refer to what could be expected given the past history of spiking events. This is precisely what coordinated inhibition may correspond to in the scheme described above: the predicted level of input correlations. In this sense, inhibition can be tuned to cancel the expected correlations, but by definition it cannot cancel coincidences that are not expected. Thus, the effect of such an excitation-inhibition coordination is precisely to enhance the salience of unexpected synchrony.

In the 1990s, there was a famous published exchange about the rate vs. timing debate, between Softky and Koch on one side, and Shadlen and Newsome on the other side. Softky and Koch argued that if spike trains were random, as they seemed to be in single unit recordings, and cortical neurons sum many inputs, then by the law of large numbers their output should be regular, since the total input would be approximately constant. Therefore, so they argued, there is an inconsistency in the two hypotheses (independence of inputs and integration). They proposed to resolve it by postulating that neurons do not sum their inputs but rather detect coincidences at a millisecond timescale, using dendritic nonlinearities. Shadlen and Newsome demonstrated that the two hypotheses are in fact not contradictory, if one postulates that the total mean input is subthreshold, so that spikes only occur when the total input fluctuates above its average. This is called the “fluctuation-driven regime”, and it is a fairly well accepted hypothesis nowadays. When there are many inputs, this can happen when excitation is balanced by inhibition, hence the other standard name “balanced regime” (note that balanced implies fluctuation-driven, but not the other way round). An electrophysiological signature of this regime is a distribution of membrane potential that peaks well below threshold (instead of monotonically increasing towards threshold).

In the fluctuation-driven regime, output spikes occur irregularly, because the neuron only spikes when there is a fluctuation of the summed input. Thus the two hypotheses are not contradictory: it is completely possible that a neuron receives independent Poisson inputs, integrates them, and fires in a quasi-Poisson way. This argument indeed makes the submillisecond coincidence detection hypothesis unnecessary. However, Softky then correctly argued that even then, output spikes are still determined by input spikes, so they cannot be seen as random. To be more precise: input spike trains are independent Poisson processes, the output spike train is (approximately) a Poisson process, but inputs and outputs are not independent. In their reply, Shadlen and Newsome miss this argument. They show that if they replay the same pattern of spikes to the neuron that led to a spike, but with a different history of inputs, then the neuron may not spike. This happened in their model for two reasons: 1) they used a variation of the perfect integrator, a very particular kind of model that is known to be unreliable, contrary to almost every other spiking neuron model, and to actual neurons (Brette & Guigon 2003), 2) they considered a pattern of input spikes restricted to a window much shorter than the integration time constant of the neuron. If they had played a pattern covering one integration time window to a standard integrate-and-fire model (or any other model), then they would have seen output spikes. But perhaps more importantly, even if the input pattern is restricted either temporally or to a subset of synapses, the probability that the neuron fires is much higher than chance. In other words, the output spike train is not independent of any of the input spike trains. This would appear in a cross-correlogram between any input and the output, as an extra firing probability at positive lags, on the timescale of the integration time constant, with a correlation of order 1/N (since there is 1 output spike for N input spikes, assuming identical rates).

Note that this is a trivial mathematical fact, if the output depends deterministically on the inputs. Yet, it is a critical point in the debate. Consider: here is an elementary example in which all inputs are independent Poisson processes with the same constant firing rate, and the output is also a (quasi-) Poisson process with constant rate. But the fact that one input neuron spikes is informative about whether the output neuron will spike shortly after, conditionally to the knowledge of the rate of the first neuron. In other words, rates do not fully describe the (joint) activity of the network. This is a direct contradiction of the rate-based postulate.

Even though this means that the rate-based hypothesis is mathematically wrong (at least in this case), it may still be that it is a good enough approximation. If one input spike is known, one gets a little bit of extra information about whether the output neuron spikes, compared to the sole knowledge of the rates. Maybe this is a slight discrepancy. But consider: if all input spikes are known, one gets full information about the output spikes, since the process is deterministic and reliable. This is a very strong discrepancy with the rate-based hypothesis. One may ask the question: if I observe p input spikes occurring together, how much can I predict about output spiking? This is the question we tried to answer in Rossant et al. (2011), and it follows an argument proposed by Abeles in the 1980s. In a fluctuation-driven regime, if one observes just one input spike, chances are that the membrane potential is far from threshold, and the neuron is very unlikely to fire. But if, say, 10 spikes are observed, each producing a 1 mV depolarization and the threshold is about 10 mV about the mean potential, then there is 50% chance of observing an output spike. Abeles called the ratio between the extra firing produced by 10 independent spikes and by 10 coincident spikes the “coincidence advantage”, and it is a huge number. Consider again: if you only know the input rates, then there is a 5% chance of observing a spike in a 10 ms window, for an output neuron firing at 5 Hz; if you additionally know that 10 spikes have been fired, then there is a 50% chance of observing an output spike. This is a huge change, involving the observation of just 0.1% of all synapses (assuming 10,000 synapses).

Thus, it is difficult to argue here that rates entirely determine the activity of the network. Simply put, the fact that the input-output function of neurons is essentially deterministic introduces strong correlations between input and output spike trains. It is a simple fact, and it is well known in the theoretical literature about neural network dynamics. For example, one line of research, initiated mainly by Nicolas Brunel, tries to determine the firing rates (average and time-varying) of networks of spiking models, using a self-consistent analysis. It is notably difficult to do this in general in the fluctuation-driven regime, because of the correlations introduced by the spiking process. To solve it, the standard hypothesis is to consider sparse networks with a random connectivity. This ensures that there is no short cycle in the connectivity graph, and therefore that inputs to a given neuron are approximately independent. But the theoretical predictions break when this hypothesis is not satisfied. It is in fact a challenge in theoretical neuroscience to extend this type of analysis to networks with realistic connectivity – i.e., with short cycles and non-random connectivity.

It is interesting to note that the concept of the balanced or fluctuation-driven regime was proposed in the 1990s as a way to support rate-based theories. In fact, analysis shows that it is specifically in this regime, and not in the mean-driven regime, that 1) neurons are essentially deterministic, 2) neurons are highly sensitive to the relative timing of input spikes, 3) there is a strong coordination between input and output spikes. The rate-based postulate is not valid at all in this regime.

I have identified the rate-based hypothesis as a methodological postulate, according to which neural activity can entirely be described by underlying rates, which are abstract variables. In general, individual spikes are then seen as instantiations of a random point process with the given rates. It can also be seen as a hypothesis of independence between the algorithmic and physical levels, in David Marr’s terminology. On the contrary, spike-based theories consider that algorithms are defined at the spike level.

What is the empirical evidence? I will start by showing that most arguments that have been used in this debate do not actually help us distinguish between the two alternatives.

Variability. Perhaps the most used argument against spike-based theories is the fact that spike trains in vivo are variable both temporally and over trials, and yet this might well be the least relevant argument. I addressed this point in detail in my third post, so I will only briefly summarize. Within one recording, inter-spike intervals (ISIs) are highly variable. This has been used as a sign that spike trains are instantiations of random point processes. But variability of ISIs is also a requirement of spike-based theories, because it increases the amount of information available in spike timing (mathematically, entropy in the ISI distribution is maximal for Poisson processes). More generally, temporal or spatial variability can never be a way to distinguish between random and deterministic schemes, because the entropy of a distribution reflects either the information content (if it is used for the code) or the amount of randomness (if it cannot be used). This brings us to the argument of variability across trials, that is, the lack of reproducibility of neural responses to the same stimulus. But this is a category error: these observations tell us that responses are stochastic, not that the activity can be fully described by rates. Therefore, it is an argument in the stochastic vs. deterministic debate, not in the rate vs. spike debate. In addition, it is a weak argument because only the stimulus is controlled. The state of the brain (e.g. due to attention, or any other aspect of the network dynamics) is not. In some cases, the sensory inputs themselves are not fully controlled (e.g. eye movements in awake animals). Therefore, the lack of reproducibility may represent either true stochasticity or simply reflect the uncertainty about uncontrolled variables, which may still be accessible to the nervous system. The lack of reproducibility itself is also contentious, at least in subcortical and primary cortical areas. But since I pointed out that this is not a very relevant argument anyway, I will not comment on this evidence (although I should note that strong reproducibility at spike level would be an argument against rate-based theories).

Chaos. Related to the variability arguments is the chaos argument (see my related post). It has been claimed that neural networks are chaotic. This is an interesting point, because it has been used to argue in favor of rate-based theories when really, it is an argument against them. What chaos implies is an absence of reproducibility of neural responses to a given stimulus. As I argued in the previous paragraph, by itself the argument has no value in the rate vs. spike debate. But if it is true that the lack of reproducibility is due to chaotic dynamics, then this goes against the rate-based hypothesis. Indeed, chaotic systems are deterministic, they cannot be described as random processes. In particular, the variables are not independent, and trajectories live in lower-dimensional spaces (attractors). I am not convinced that network dynamics are truly chaotic (although they might be), but if they are, then defenders of rate-based theories should rather be worried.

Selectivity curves. The concept of the selectivity curve or tuning curve has been used very much in research on sensory systems (e.g. the visual cortex). It has been found for example that many cells in the primary visual cortex fire more in response to a moving bar or grating with a specific orientation. This observation is often reported as the statement that these neurons code for orientation. Implicitly, this means that the firing rate of these neurons contains information about orientation, and that this is the information used by the rest of the system. However, this is not what these experiments tell us. They only tell us that the firing rate covaries with stimulus orientation, nothing more. This cannot be an argument for rate-based theories, because in spike-based theories, the firing rate also varies with stimuli (see my specific post). Indeed, to process stimuli with spikes requires producing spikes, and so stimulus-dependent variations in firing rate are a necessary correlate of spike-based computation. It is useful to interpret spike counts in terms of energy consumption, and with this notion in mind, what orientation selectivity curves tell us is not that the cells code for orientation, but rather they care about orientation (or about a specific orientation). This is still quite an informative statement, but it does not tell us anything about whether the firing rate is the right quantity to look at.

Fast processing. To be fair, I will now critically examine an argument that has been used to contradict rate-based theories. It has been shown with psychophysical experiments that complex visual tasks can be performed by humans in very little time, so little time that any neuron along the processing chain may only fire once or not at all. This observation contradicts any scheme based on counting spikes over time, but it does not contradict views based on rate as a firing probability or as a spatial average – however, it does impose constraints on these views. It also rules out schemes based on interspike intervals. In other words, it discards computing schemes based on information obtained within single neurons (interspike interval or spike count) rather than across neurons (relative timing or population firing rate).

High correlations. A number of studies claim that there are significant correlations between neural responses, in some cases. For example, neurons of the LGN that share a presynaptic retinal ganglion cell tend to fire synchronously, at a short timescale. This contradicts one of the claims of rate-based theories, that firing between neurons is independent, conditionally to the underlying rates. Other studies have shown oscillations that organize spiking at different timescales (e.g. in the visual cortex, and in the hippocampus). These observations may be seen as contradicting rate-based theories (especially the former), but it could be opposed that 1) these correlations may still not have a big impact on neural dynamics, and 2) even if they do, it is a minor modification to rate-based theory if they do not depend systematically on the stimulus. For example, opponents of oscillation-based theories would argue that oscillations are a by-product of the fact that networks are recurrent, and as feedback systems they can develop oscillations, which bear no functional significance. In the same way, fine scale correlations between neighboring LGN neurons may result from anatomical factors, but it may only contribute to amplify the thalamic input to the cortex – not a fundamental change in rate-based theory. But there are now a number of studies, in the cortex (e.g. from Singer’s lab) and in the hippocampus (e.g. from Buzsaki’s lab) that show a systematic relationship between functional aspects and oscillatory properties. Fine scale correlations have not been studied so extensively in relationship to stimulus properties, but recently there was a study showing that the correlation between two neighboring LGN neurons in response to oriented gratings is tuned to orientation (Stanley et al. 2012). These cells project to cortical neurons in V1, whose firing rate is tuned to orientation. Thus, there is pretty clear evidence that correlations can be stimulus-dependent. The main question, then, is whether these correlations actually make a difference. That is, does the firing rate of a neuron depend mainly on the underlying rates of the presynaptic neurons, or can fine scale correlations (or, say, a few individual spikes) make a difference? I will come back to this question in more detail below.

Low correlations. Before I discuss the impact of correlations on neural firing, I will also comment on the opposite line of arguments. A few recent studies have actually claimed that there are weak correlations between cortical neurons. First of all, the term “weak” is generally vague, i.e., weak compared to what? Is 0.1 a weak or a strong correlation? Such unqualified statements are subjective. One would intuitively think that 0.01 is a very weak correlation, in the sense that it is probably as if it were zero. But this is mere speculation. Another statement might be that correlations are not statistically significant. This statement is objective, but not conclusive. It only means that positive correlations could not be observed given the duration of the recordings, which amounts to saying that correlations are smaller than the maximum amount that can be measured. This is not more informative than saying that there is (say) an 0.1 correlation – it is even less informative, if this maximum amount is not stated. So is 0.1 a weak or a strong pairwise correlation? The answer is, in general, that it is a huge correlation. As argued in Rossant et al. (2011), correlations make a huge difference to postsynaptic firing unless they are negligible compared to 1/N, where N is the number of synapses of the cell. So for a typical cortical neuron, this would mean negligible compared to 0.0001. The argument is very simple: independent inputs contribute to the membrane potential variance as N, but correlated inputs as c.N², where c is the pairwise correlation. The question, in fact, is rather how to deal with such huge correlations (more on this below).

Below I will discuss a little more the impact of correlations on postsynaptic firing, but before that, I would first like to stress two important facts: 1) the presence of pairwise correlations is not critical to all spike-based theories, 2) in other spike-based theories, relative spike timing is stimulus-dependent and possibly transient. Indeed there are prominent spike-theories based on asynchrony rather than synchrony. For example, the rank-order theory (e.g. from Thorpe’s lab) proposes that information is encoded in the relative activation order of neurons, and there is no particular role for synchrony. The theory does not predict a high amount of correlations. However, this rank order information may still manifest itself in cross-correlograms, as a stimulus-dependent asymmetry. Another example is the predictive spike coding theory defended by Sophie Denève, in which neurons fire when a specific criterion is fulfilled, so as to minimize an error. This predicts that neurons fire asynchronously - in fact in a slightly anti-correlated way. Finally, even in those theories based on synchrony, such as the one I presented recently (Brette 2012), neurons are not correlated in general. In the theory I proposed, synchrony is an unlikely event, which is detected by neurons. It is precisely because it is unlikely that it is meaningful – in this case, it signals some structure that is unlikely to be observed by chance. I have to recognize, however, that when a structured stimulus is presented, specific neuron groups fire in synchrony, throughout the duration of the stimulus. I actually do not think that it should necessarily be the case (except for the binaural system). Pushing the theory further, I would argue that once the stimulus structure is established and recognized, it is not unlikely anymore, and therefore only the onset of the synchrony event is meaningful and required by the theory. Therefore, the prediction of the theory is rather that there are transient synchrony events, associated to specific properties of stimuli, which have an impact on target neurons. To summarize, spike-based theories do not generally predict strong correlations, and none of these theories predict correlations in spontaneous activity.

This post is already long, so I will finish with a brief discussion of the impact of correlations on postsynaptic firing – a longer one in the next post. As I mentioned above, very small pairwise correlations have a huge impact on postsynaptic firing. To be negligible, they should be small compared to 1/N, where N is the number of synapses of the postsynaptic neuron. Another way to look at it, which is discussed in detail in Rossant et al. (2011), is that changing the timing of a few spikes (on the order of 10 synapses, out of 10,000) has a dramatic effect on postsynaptic firing (i.e., from silent to strongly firing). This point was already made in the 1980s by Abeles. The phenomenon occurs specifically in the fluctuation-driven regime, so in the next post I will describe this regime and what it means for the debate.

In summary, the debate of rate vs. timing is not about the description timescale, but about the notion that neural activity and computation may be entirely and consistently defined by the time-varying rates r(t) in the network. In fact, it is interesting to cast this debate in the analysis framework proposed by David Marr. I have discussed this framework in other posts, but it is worth explaining it here again. Marr proposed that information processing systems can be analyzed at three levels:

1) The computational level: what does the system do? (for example: estimating the location of a sound source)

2) The algorithmic/representational level: how does it do it? (for example: by calculating the maximum of cross-correlation between the two monaural signals)

3) The physical level: how is it physically realized? (for example: with axonal delay lines and coincidence detectors)

Rate-based theories postulate that algorithms and representations can be defined independently of spikes that instantiate them. Indeed, as I argued in previous posts, the instantaneous firing rate is not a physical quantity but an abstraction (in general a probability of firing), and it is postulated that all algorithms can be defined at the level of rates, without loss. The conversion between spikes and rates is seen as independent from that level. In other words, the rate-based hypothesis is the postulate that the algorithmic and the physical levels are independent. In contrast, spike-based theories consider that these levels are not independent, i.e., that algorithms are defined at the spike level.

In the example of sound localization I used in the description of three levels, the binaural neuron implements the cross-correlation between two monaural signals. This is possible if one assumes that monaural signals are transduced to spikes, through a Poisson process with rate equal to these signals, the binaural neuron responds to coincidences and the result is the spike count of the binaural neuron. This is rate-based theory (even though based on coincidence detection). Alternatively, in Goodman & Brette (2010), signals are transduced to spikes through an essentially deterministic process, and the binaural neuron spikes to signal the similarity between the transduced signals (note that a single spike is meaningful here, see my post on the difference between correlation and synchrony). This is spike-based theory. It also makes a functional difference in the example I just described, because in the spike-based version, the neuron is also sensitive to interaural intensity differences.

When expressed as the independence between the algorithmic and spike level, the rate-based hypothesis seems like an ad hoc postulate. Why would evolution make it such that it is possible to describe neural algorithms in terms of rates, what is the advantage from the organism’s point of view? This is why I see the rate-based hypothesis as a methodological postulate, rather than a true scientific hypothesis. That is, it is a postulate that makes it simpler for us, external observers, to describe and understand what neurons are doing. This is so because most of our calculus is based on operations on analog signals rather on discrete entities (spikes). It is then hoped that this level of description is adequate, but there is no strong biological reason why it should be so. It just seems adequate enough to defenders of rate-based theories, and they endorse it because it is methodologically convenient.

This reminds me of discussions I have had with strong advocates of rate-based theories, who are also reasonable scientists. When faced with evidence and arguments that strongly suggest that rates cannot fully describe neural activity, they may agree. But they remain unconvinced, because they do not see why they should abandon a seemingly working theory (rate-based calculus) for a hypothesis that does not help them understand the system, even though it is more empirically valid (neural computation is based on spikes, but how exactly?). In other words, why bother with the extra complication of spikes? This is what I mean by a “methodological postulate”: it is not that, for empirical reasons, neurons are more likely to discard any information about spike timing, but rather that it seems conceptually more convenient to think in terms of analog quantities rather than spikes.

This means that this debate will not be resolved by accumulating empirical evidence for or against either alternative. For defenders of spike-based theories, it can only be resolved by providing a convincing theory of spike-based computation that could replace rate-based calculus. For defenders of rate-based theories, the challenge is rather to find mechanisms by which neural activity can truly be reduced to calculus with analog signals – a difficult task, as I will show in the next posts.

How much intrinsic noise is there in a neuron? This question would deserve a longer post, but here I will just make a few remarks. In vitro, when the membrane potential is recorded in current-clamp, little noise is seen. There could be hidden noise in the spike generating process (i.e., in the sodium channels), but when a time-varying current in injected somatically into a cortical neuron, the spike trains are also highly reproducible (Mainen & Sejnowski, 1995). This means that the main source of intrinsic noise in vivo is synaptic unreliability.

Transmission at a given synapse is unreliable, in general. That is, there is a high probability of transmission failure, in which there is a presynaptic spike but no postsynaptic potential. However, an axon generally contacts a postsynaptic neuron at multiple release sites, which we may consider independent. If there are N sites with a transmission probability p, then the variance of the noise represents a fraction x=(1-p)/(pN) of the variance of the signal (expected PSP size). We can pick some numbers from Branco & Staras (2009). There seems to be quite different numbers depending on studies, but it gives an order of magnitude. For cat and rat L2/3 pyramidal cells, we have for example N=4 and p=0.5 (ref. 148). This gives x=0.25. Another reference (ref. 149) gives x=0.07 for the same cells.

These numbers are not that big. But it is possible that transmission probability is lower in vivo. So we have to recognize that synaptic noise might be substantial. However, even if it is true, it is an argument in favor of the stochasticity of neural computation, not in favor of rate-based computation. In addition, I would like to add that synaptic unreliability has little impact on theories based on synchrony and coincidence detection. Indeed, a volley of synchronous presynaptic spikes arriving at a postsynaptic neuron has an essentially deterministic effect, by law of large numbers. That is, synchronous input spikes are equivalent to multiple release sites. If there are m synchronous spikes, then the variance of the noise represents a fraction x=(1-p)/(pmN) of the variance of the signal (compound PSP). Taking the same numbers as above, if there are 10 synchronous spikes then we get x=0.025 (ref. 148) and x=0.007 (ref. 149), i.e., an essentially deterministic compound PSP. And we have shown that neurons are very sensitive to fast depolarizations in a background of noise (Rossant et al. 2011). The theory of synfire chains is also about the propagation of synchronous activity in a background of noise, i.e., taking into account synaptic unreliability.

In summary, the main source of intrinsic noise in neurons is synaptic noise. Experimental figures from the literature indicate that it is not extremely large but possibly substantial. However, as I noted in previous posts, the presence of large intrinsic noise does invalidate spiked-based theories but deterministic theories. In addition, synaptic noise has no impact on synchronous events, and therefore it is essentially irrelevant for synchrony-based theories.

Misconception #4: “A stochastic spike-based theory is nothing else than a rate-based theory, only at a finer timescale”.

It is sometimes claimed or implied that there is no conceptual difference between the two kinds of theories, the only difference being the timescale of the description (short timescale for spike-based theories, long timescale for rate-based theories). This is a more subtle misconception, which stems from a confusion between coding and computation. If one only considers the response of a neuron to a stimulus and how much information there is in that response about the stimulus, then yes, this statement makes sense.

But rate-based and spike-based theories are not simply theories of coding, they are also theories of computation, that is, of how responses of neurons depend on the responses of other neurons. The key assumption of rate-based theories is that it is possible and meaningful to reduce this transformation to a transformation between analog variables r(t), the underlying time-varying rates of the neurons. These are hidden variables, since only the spike trains are observable. The state of the network is then entirely defined by the set of time-varying rates. Therefore there are two underlying assumptions: 1) that the output spike train can be derived from its rate r(t) alone, 2) that a sufficiently accurate approximation of the presynaptic rates can be derived from the presynaptic spike trains, so that the output rate can be calculated.

Since spike trains are considered as stochastic with (expected) instantaneous rate r(t), assumption #1 means that spike trains are stochastic point processes defined from and consistent with the time-varying rate r(t) – they could be Poisson processes, but not necessarily. The key point here is that the spiking process is only based on the quantity r(t). This means in particular that the source of noise is independent between neurons.

The second assumption means that the operation performed on input spike trains is essentially independent of the specific realizations of the random processes. There are two possible cases. One alternative is that the law of large numbers can be applied, so that integrating inputs produces a deterministic value that depends on the presynaptic rates. But then the source of noise, which produces stochastic spike trains from a deterministic quantity, must be entirely intrinsic to the neuron. Given what we know from experiments in vitro (Mainen and Sejnowski, 1995), this is a fairly strong assumption. The other alternative is that the output rate depends on higher statistical orders of the total input (e.g. variance) and not only on the mean (e.g. through the central limit theorem). But in this case, the inputs must be independent, for otherwise it would not be possible to describe the output rate r(t) as a single quantity, since the transformation would also depend on higher-order quantities (correlations).

In other words, the assumptions of rate-based theories mean that spike trains are realizations of independent random processes, with a source of stochasticity entirely intrinsic to the neuron. This is a strong assumption that has little to do with the description timescale.

This assumption is also known to be inconsistent in general in spiking neural network theory. Indeed it is possible to derive self-consistent equations that describe the transformation between the input rates of independent spike trains and the output rate of an integrate-and-fire model (Brunel 2001), but these equations fail unless one postulates that connections between neurons are sparse and random. This postulate means that there are no short cycles in the connectivity graph, so that inputs to a neuron are effectively independent. Otherwise, the assumption of independent outputs is inconsistent with overlaps in inputs between neurons. Unfortunately, neural networks in the brain are known to be non-random and with short cycles (Song et al. 2005).

To be fair, it is still possible that neurons that share inputs have weakly correlated outputs, if inhibition precisely tracks excitation (Renart et al. 2010). But it should be stressed that it is the assumptions of rate-based theories that require a specific non-trivial mechanism, rather than those of spike-based theories. It is ironic that spike-based theories are sometimes depicted as exotic by tenants of rate-based theories, while the burden of proof should in fact reside on the latter.

To summarize this post: the debate of rate vs. timing is not about the description timescale, but about the notion that neural activity and computation may be entirely and consistently defined by the time-varying rates r(t) in the network. This boils down to whether neurons spike in a stochastic independent manner, conditionally to the input rates. It is worth noting that this is a very strong assumption, with currently very little evidence in favor, and a lot of evidence against.

Misconception #3: “Neural codes can only be based on rates because neural networks are chaotic”. Whether this claim is true or not (and I will comment on it below), chaos does not imply that spike timing is irrelevant. To draw this conclusion is to commit the same category error as I discussed in the previous post, i.e., confusing rate vs. timing and stochastic vs. deterministic.

In a chaotic system, nearby trajectories quickly diverge. This means that it is not possible to predict the future state from the present state, because any uncertainty in estimating the present state will result in large changes in future state. For this reason, the state of the system at a distant time in the future can be seen as stochastic, even though the system itself is deterministic.

Specifically, in vitro experiments suggest that individual neurons are essentially deterministic devices (Mainen and Sejnowski 1995) – at least the variability seen in in vitro recordings is often orders of magnitude lower than in vivo. But a system composed of interacting neurons can be chaotic, and therefore for all practical aspects their state can be seen as random, so the chaos argument goes.

The fallacy of this argument can be seen by considering the prototypical chaotic system, climate. It is well known that the weather cannot be predicted more than 15 days in the future, because even tiny uncertainties in measurements make the climate models diverge very quickly. But this does not mean that all you can do is pick a random temperature according to the seasonal distribution. It is still possible to make short term predictions, for example. It also does not mean that climate dynamics can be meaningfully described only in terms of mean temperatures (and other mean parameters). For example, there are very strong correlations between weather events occurring at nearby geographical locations. Chaos implies that it is not possible to make accurate predictions in the distant future. It does not imply that temperatures are random.

In the same way, the notion that neural networks are chaotic only implies that one cannot predict the state of the network in the distant future. This has nothing to do with the distinction between rate and spike timing. Rate (as mean seasonal temperature) may still be inadequate to describe the dynamics of the system, and firing may still be correlated across neurons.

In fact the chaos argument is an argument against rate-based theories, precisely because a chaotic system is not a random system. In particular, in a chaotic system, there are lawful relationships between the different variables. Taking the example of climate again, the solutions of the Lorenz equations (a model of atmostpheric convection) live in a low-dimensional manifold with a butterfly shape known as the Lorenz attractor. Even though one cannot predict the values of the variables in the distant future, these variables evolve in a very coordinated way. It would be a mistake to replace them by their average values. Therefore, if it is true than neural networks are chaotic, then it is probably not true that their dynamics can be described in terms of rates only.

I will end this post by commenting on the notion that neural networks are chaotic. I very much doubt that chaos is an adequate concept to describe spiking dynamics. There are different definitions of a chaotic system, but essentially they state that a chaotic system is a system that is very sensitive to initial conditions, in the sense that two trajectories that are initially very close can be very far apart after a relatively short time. Now take a neuron and inject a constant current: it will fire regularly. In the second trial, inject the exact same current but 1 ms later. Initially the state of the neuron is almost identical in both trials. But when the neuron fires in the first trial, its membrane potential diverges very quickly from the trajectory of the second trial. Is this chaos? Of course not, because the trajectories meet again about 1 ms later. In fact, I showed in a study of spike time reliability in spiking models (Brette and Guigon, 2003) that even if the trajectories diverge between spikes (such as with the model dv/dt=v/tau), spike timing can still be reliable in the long run in response to fluctuating inputs. This counter-intuitive property can be seen as nonlinear entrainment.

In summary, 1) chaos does not support rate-based theories, it rather invalidates them, and 2) chaos is probably not a very meaningful concept to describe spiking dynamics.