Musical notes have a particular perceptual quality called “pitch”. Pitch is the percept corresponding to how low or high a musical note is. Vowels also have a pitch. To a large extent, the pitch of a periodic sound corresponds to its repetition rate. The important point is that what matters in pitch is more the periodicity than the frequency content. For example, a periodic sound with repetition rate f0 has frequency components at multiples of f0 (n.f0), which are called harmonics. A pure tone of frequency f0 and a complex tone with all harmonics except the first one, i.e., which does not contain the frequency component f0, will evoke the same pitch. It is in fact a little more complex than that, there are many subtleties, but I will not enter into these details in this post. Here I simply want to describe the kind of sensory or sensorimotor structure there is in pitch. It turns out that pitch has a surprisingly rich structure.

The most obvious type of structure is periodicity. Pitch-evoking sounds have this very specific property that the acoustical wave is unchanged when temporally shifted by some delay. This delay is characteristic of the sound’s pitch (i.e., same period means same pitch). This is the type of structure that is emphasized in temporal theories of pitch. This is what I call the “similarity structure” of the acoustical signal, and this notion can in fact be extended and accounts for a number of interesting phenomena related to pitch. But this is work in progress, so I will discuss it further at a later time.

Another way to see periodic sounds is to realize that a periodic sound is predictable. That is, after a couple periods, one can predict the future acoustical wave. Compared to most other sounds, periodic sounds have a very high degree of predictability. Perhaps the perceptual strength of pitch (which depends on a number of factors) is related to the degree of predictability of the sound.

There is another type of structure that is in some sense orthogonal to the similarity structure I just described, which one might call the “dissimilarity structure”. Natural sounds (apart from vocalizations) tend to have a smooth spectrum. Periodic sounds, on the other hand, have a discrete spectrum. Thus, in some sense, periodic sounds have a “surprisingly discontinuous” spectrum. Suppose for example that two auditory receptors respond to different but overlapping parts of the spectrum (e.g., two nearby points on the basilar membrane). Then one can usually predict the sensory input to the second receptor given the sensory input to the first receptor, because natural sounds tend to have a continuous spectrum. But this prediction would fail with a periodic sound. Periodic sounds are maximally surprising in this sense. The interesting thing about the dissimilarity structure of pitch is that it accounts for binaural pitch phenomena such as Huggins’ pitch: noise with flat spectrum is presented on both ears, and the interaural phase difference changes abruptly at a given frequency; a tone is perceived, with the pitch corresponding to that frequency.

Thus, pitch-evoking sounds simultaneously have two types of structure that distinguish them from other types of sounds: the similarity structure, which consists of different views of the acoustical signal that are unusually similar, and the dissimilarity structure, which consists of different views of the acoustical signal that are unusually dissimilar. This first type of structure corresponds to what I examined in my paper on computing with neural synchrony. It is important to notice that these two types of structure have a different nature. The similarity structure corresponds to a law that the sensory signals follow. Here the percept is associated to the specific law that these signals follow. The dissimilarity structure corresponds to the breaking of a law that sensory signals usually follow. Here the percept is associated to a law that is specific not of the presented sensory signals, but of the usual sensory signals. Thus we might relate the similarity structure to the notion of discovery, and the dissimilarity structure to the notion of surprise (and perhaps the term “structure” is not appropriate for the latter).

So far, I have only considered the structure of the acoustical signal, but one may also consider the sensorimotor structure of pitch. As I mentioned in another post, periodic sounds are generally produced by living beings, so it makes sense to examine these sounds from the viewpoint of their production. When one produces a pitch-evoking sound (for example a vowel, or when one sings), there is a very rich structure that goes beyond the acoustical structure. First, there is proprioceptive information about vocal muscles and tactile information about the vibrations of the larynx, and both are directly related to the period of sounds. There is also the efferent copy, i.e., the motor commands issued to make the vocal folds vibrate in the desired way. For a person who can produce sounds, pitch is then associated to a rich and meaningful sensorimotor structure. In fact, the sensorimotor theory of pitch perception would be that to perceive the pitch of a sound is, perhaps, to perceive the movements that would be required to produce such acoustical structure. An interesting aspect of this view is that it provides some meaning to the notion of how low or high a pitch-evoking sound is, by associating it with the state of the different elements involved in sound production. For example, to produce a high sound requires to increase the tension of the vocal cords, and to move the larynx up (higher!). One question then is whether congenitally mute people have a different perception of pitch.

Observe that, as for binaural hearing , the sensorimotor structure of pitch should not be understood as the relationship between motor commands and auditory signals, but rather as the relationship between motor commands and the structure of auditory signals (e.g. the periodicity). In this sense, it is higher-order structure.

What kinds of sounds are there in the world? This is essentially the question William Gaver addresses in a very interesting paper (Gaver, 1993), in which he describes an ontology of sounds, categorized by the type of interaction. There are three categories: sounds made by solids, liquids and gases. An example of a sound made by liquid is dripping. There are also hybrid sounds, such as the rain falling on a solid surface. It makes sense to categorize sounds based on the nature of the objects because the mechanical events are physically very different. For example, in sounds involving solids (e.g. a footstep), energy is transmitted at the interface between two solids, which is a surface, and the volumes are put in motion (i.e., they are deformed). This is completely different for sounds involving gases, e.g. wind. In mechanical events involving solids, the shape is essentially unchanged (only transiently deformed). This is a sort of structural invariance that ought to leave a specific signature on the sounds (more on this in another post). Sounds made by gases, on the other hand, correspond to irreversible changes.

These three categories correspond to the physical nature of the sound producing substances. There are subcategories that correspond to the nature of the mechanical interaction. For example, a solid object can be hit or it can be scraped. The same object vibrates but there is a difference in the way it is made to vibrate. This also ought to produce some common structure in the auditory signals, as is explained in Gaver's companion article. For example, a vibrating solid object has modes of vibration that are determined by its shape (more on this in another post). These modes do not depend on the type of interaction with the object.

Interactions that are localized in time are impact sounds, while continuous interactions produce auditory textures. These are two very distinct types of sounds. Both have a structure, but auditory textures, it seems, only have a structure in a statistical sense (see McDermott & Simoncelli, 2011). Another kind of auditory texture is the type of sounds produced by a river, for example. These sounds also have a structure in a statistical sense. An interesting aspect, in this case, is that these sounds are not spatially localized: they do have an auditory size (see my post on spatial hearing ).

The examples I have described correspond to what Gaver calls "basic level events", elementary sounds produced by a single mechanical interaction. There are also complex events, which are composed of simple events. For example, a breaking sound is composed of a series of impact sounds. A bouncing sound is also composed of a series of impact sounds, but the temporal patterning is different, because it is lawful (predictable) in the case of a bouncing sound. Walking is yet another example of a series of impact sounds, which is also lawful, but it differs in the temporal patterning: it is approximately periodic.

Gaver only describes sounds made by non-living elements of the environment (except perhaps for walking). But there are also sounds produced by animals. I will describe them now. First, some animals can produce vocalizations. In Gaver's terminology, vocalizations are a sort of hybrid gas-solid mechanical event: periodic pulses of air make the vocal folds vibrate. The sound then resonates in the vocal tract, which shapes the spectrum of the sound (in a similar way as the shape of an object determines the resonating modes of impact sounds). One special type of structure in these sounds is the periodicity of the sound wave. The fact that a sound is periodic is highly meaningful, because it means that energy is continuously provided, and therefore that a living being is most likely producing it. There are also many other interesting aspects that I will describe in a later post.

Animals also produce sounds by interacting with the environment. These are the same kinds of sounds as described by Gaver, but I believe there is a distinction. How can you tell that a sound has been produced by a living being? Apart from identifying specific sounds, I have two possible answers to provide. First, in natural non-living sounds, energy typically decays. This distinguishes walking sounds from bouncing sounds, for example. In a bouncing sound, the energy decreases at each impact. This means both the intensity of the sound and the interval between successive impacts decay. This is simply because a bouncing ball starts its movement with a potential energy, that can only decay. In a walking sound, roughly the same energy is brought at each impact, so it cannot be produced by the collision of two solids. Therefore, sounds contain a signature of whether it is produced by continuous source of energy. But a river is also a continuous source of energy (and the same would apply to all auditory textures). Another specificity is that sounds produced by the non-living environment are governed by the laws of physics, and therefore they are lawful in a sense, i.e., they are predictable. A composed sound with a non-predictable pattern (even in a statistical sense) is most likely produced by a living being. In a sense, non-predictability is a signature of decision making. This remark is not specific to hearing.

These are specificities of sounds produced by living beings, as heard by another observer. But one can also hear self-produced sounds. There are two new specificities about these types of sounds. First, they also make the body vibrate, for example, a foot hits the ground. This produces sound waves with a specific structure. But more importantly, self-produced sounds have a sensorimotor structure. Scraping corresponds to a particular way in which one interacts with an object. The time of impact corresponds to the onset of the sound. The intensity of the sound is directly related to the energy with which an object is hit. Finally, the periodicity of vocalizations (i.e., the pitch), corresponds to the periodicity of self-generated air pulses through the vocal folds, and the formant frequencies correspond to the shape of the vocal tract. Self-generated sounds also have a multimodal structure. For example, they produce vibrations in the body than can be perceived by tactile receptors. In the next post, I will look at the structure of pitch.