James Gibson defended the idea that what we perceive of the environment is affordances, that is, the possibilities of interactions they allow. For example, a knob affords twisting, or the ground affords support. The concept of affordance makes a lot of sense, but Gibson also insisted that we directly perceive these affordances. It has never been very clear to me what he meant by that. But following recent discussions, I have thought of a way in which this statement might make sense - although I have no idea whether this is what Gibson meant.

The way sensory systems work is traditionally defined as an early extraction of “features”, like edges, which are then combined through a hierarchical architecture into more and more complex things, until one gets “an object”. In this view, affordances are obtained at the end of the chain, and so it is not direct at all. In robotics, another kind of architecture was proposed by Rodney Brooks in the 1980s, the “subsumption architecture”. It was meant as a way to make robots in an incremental way, by progressively adding layers of complexities. In his example, the first layer of the robot would be a simple control system by which external motor commands produce movement, and there is a sonar that computes a repulsive force in case there is a wall in front of it, and the force is sent to the motor module. Then there is a second layer that makes the robot wander. It basically randomly chooses a direction at regular intervals, and it combines it with the force computed by the sonar in the first layer. The second layer is said to “subsume” the first one, i.e., it takes over. Then there is another level on top of it. The idea in this architecture is that the set of levels below any level is functional, it can do something on its own. This is quite different from standard hierarchical sensory systems, in which the only purpose of each level is to send information to the next level. Here we get to Gibson’s affordances: if the most elementary level must be functional, then what it senses is not simple features, but rather simple affordances, simple ways to interact with its environment. So in this view, what is elementary in perception is affordances, rather than elementary sensations.

I think it makes a lot of sense from an evolutionary point of view that sensory systems should look like subsumption architectures rather than standard hierarchical perception systems. If each new structure (say, the cortex) is added on top of an existing set of structures, then the old set of structures has a function by itself, independently of the new structure. Somehow the old set is “subsumed” by the new structure, and the information this new structure gets must then already have a functional meaning. This would mean that affordances are the basis, and not the end of result, of the sensory system. In this sense, perhaps, one might say that affordances are “directly” perceived.

When thinking about what it means for consciousness, I like to refer to the man and horse analogy. The horse is perfectly functional by itself. It can run, it can see, etc. Now the man on top of it can “subsume” the horse. He sends commands to it so as to move it where he wants, and also gets signals from the horse. The man is conscious, but he has no idea of what the horse feels, for example how the horse feels the ground. All the sensations that underlie the man-horse’s ability to move around are inaccessible to the conscious man, but it is not a problem at all for the man to go where he wants to.

Now imagine that the man is blind. If there is an obstacle in front of the horse, the horse might stop, perhaps get nervous, things that the man can feel. The man cannot feel the wall in terms of “raw sensations”, but he can perceive that there is something that blocks the way. In other words, he can perceive the affordance of the wall – something that affords blocking, without seeing the wall.

So in this sense, it does not seem crazy anymore that what we directly perceive (we = our conscious self) is made of affordances rather than raw sensations.