Modeling the use of gaze in everyday behavior
by Dana Ballard
Gaze changes and their resultant fixations that orchestrate the sequential acquisition of information from the visual environment are the central feature of primate vision. How are we to understand their function? For the most part, theories of fixation targets have been image based: the hypothesis being that the eye is drawn to places in the scene that contain discontinuities in image features such as motion, color and texture. But are these features the cause of the fixations or merely the result of the fixations that have been planned to serve some visual function? We first and review evidence from various image-based and task-based sources. Our conclusion is that the evidence is overwhelmingly in favor of fixation control being essentially task-based. We next describe a new model of gaze control posits that gaze changes are entirely reward-based. Normal vision is a multi-tasking environment and the information needs of the different tasks must be served by gaze changes. Thus such changes can be described by a competition between active tasks. This competition can be described in terms of a reinforcement learning setting wherein uncertainties in different state spaces are reduced by gaze changes. The advantage of the setting is that it allows the value of a gaze change to be quantified. We illustrate these points with examples of modeling human performance on everyday tasks in natural and virtual reality environments.