Key to All Optical Illusions Discovered | LiveScience

Researcher Mark Changizi of Rensselaer Polytechnic Institute in New York says it starts with a neural lag that most everyone experiences while awake. When light hits your retina, about one-tenth of a second goes by before the brain translates the signal into a visual perception of the world. Scientists already knew about the lag, yet they have debated over exactly how we compensate, with one school of thought proposing our motor system somehow modifies our movements to offset the delay. Changizi now says it’s our visual system that has evolved to compensate for neural delays, generating images of what will occur one-tenth of a second into the future. That foresight keeps our view of the world in the present. It gives you enough heads up to catch a fly ball (instead of getting socked in the face) and maneuver smoothly through a crowd. His research on this topic is detailed in the May/June issue of the journal Cognitive Science.

It makes perfect sense for those of us who’ve done work compensating for lag in networked 3D worlds. To counter the lag, the client usually predicts the motion of key objects a small way into the future. Those predictions can be wrong, in which case you have to correct or suffer the consequences. Even in video compression codecs, we exploit these properties to reduce file size.

In our visual system, those consequences may be the misperception of relative size of objects or of motion in perfectly static images. In a virtual world, it could be an avatar being in the wrong place or popping. In video codecs, guessing wrong can be more easily corrected during encoding, but at the cost of more bits to store the same information .

But it’s amazingly cool, assuming it’s true, that our visual systems can predict the future position of every "pixel" we see and do it well enough that we rarely if ever notice. That may indeed be an important cue for machine vision systems to adopt — predict and notice when the algorithmic prediction is wrong.

Of course, as some people will point out, we’re not actually "seeing" the future. It’s only our own image of what is likely to happen next, which can be wrong (in the case of these illusions). But it’s equally true to say that we don’t actually "see" the present either — only our internally processed and highly synthetic memory of our environment, which in some ways, can be far more incorrect than any low-level extrapolation.