In recent years, attempts have been made to explain complex phenomena such as common knowledge and linguistic reference in terms of perceptual joint attention. The notion of joint attention is itself complex, and there are many forms of joint attention, some that rely on simple mechanisms, others that require the intentional and consciously aware interaction between the subjects. Because joint attention is not a primitive, it is not possible to in any obvious way reduce complex social interaction to it. The aim of the present talk is to examine whether verbal referential communication depends on any particular form of joint attention, and thus whether it is possible to identify a particular element in specifically human forms of joint attention that might enable phenomena such as common knowledge and verbal reference. This will involve an analysis of the ways in which joint attention may contribute to verbal communication.

I will present a series of recent studies from our lab showing that already in one-year-old infants, joint attention is much more complex than just coordinating visual attention to objects with others. I will also show how sharing experiences with others can help – but also in some instances hurt – infants, for example, when interpreting others’ communication and reading others’ minds. I will end with a discussion of the broader significance and human uniqueness of joint attention and other types of shared experience.

Explanations of the phenomenon of joint attention in babies have led to various suggestions, by philosophers and psychologists, of the kind of access we have in general to others’ mental states. Some of these have radical implications for the kinds of account we should give of both phenomenal consciousness and of self consciousness. The paper will lay out and explore the plausibility and consequences of some of these accounts.

Early in development infants are sensitive to social cues such as eye gaze direction or emotional expressions. Recent evidence suggests that infants are sensitive to subtle disruptions of complex triadic person-object-infant interactions even in the first months of life. Moreover, by 3 months infants use the eye gaze direction of others to guide their attention towards objects in the environment. These objects are consequently more familiar than objects that had not been cued by eye gaze. I will present new data showing that infants’ neural processing of novel objects is both affected by eye gaze and by emotional expression. An object paired with a fearful face gazing toward it will consequently elicit an enhanced neural response compared to one that was presented with a neutral face or not gaze cued.

I will discuss possible functions of these early capacities and their relations to later developing more sophisticated social cognitive skills.

Merleau-Ponty spoke of the discovery of vision coming from the experience of ‘a gaze at grips with a visible world’. In this talk I would like to argue that if there were not a gaze at grips with me, i.e., the experienced self, a gaze at grips with a visible world would mean something rather different. Studies of emotional responses to others’ attention and emotional directing of others’ attention in typically developing infants and in children with developmental disabilities show that feeling and perceiving emotional response to others’ attention is crucial for an ‘appropriate’ development of the awareness of attention; that is, for a participant rather than bystander awareness of attending.

The ability and motivation to share experiences is a unique aspect of human cognition, but its neural basis remains elusive. To investigate the neural correlates of joint attention we developed a paradigm in which participants’ fixations influenced a virtual character’s looking behavior in real time. Convinced that the character was controlled by a real person outside the scanner, participants interacted with the virtual other while undergoing neuroimaging. Results demonstrate that looking at the same object as the other resulted in recruitment of medial prefrontal cortex (MPFC) previously related to the ‘meeting of minds’. Being able to direct someone’s gaze activated reward-related neurocircuitry and – in light of ratings obtained from participants – appears to underlie the hedonic aspects of initiating the sharing of experiences oneself.

During the second half of their first year of life infants show new competencies in interacting with other people: they start to follow and direct others’ attention towards entities outside the dyad, thereby engaging in episodes of joint attention.

Referential gestures such as the pointing gesture or an ostensive change in gaze direction are the most common and basic means to establish a joint focus of attention.

However, the developmental precursors and conceptual status of these early joint attentional behaviors are the topic of much debate: Are these behaviors manifestations of a radically new underlying concept of intentionality in human action or do they reflect a more step-wise development building on earlier dyadic developments? Are early interactive competencies indicative of an understanding of human action as driven by underlying intentions attributed to self and others or are rather of social responsiveness?

In order to approach these issues we will present a series of cross-sectional and longitudinal studies combining interactive settings and non-interactive looking time paradigms tapping into how infants and toddlers use behavioural cues associated with goal-directedness (such as reaching) and reference (such as gazing and pointing) in  order to make sense of other people’s behavior even when they themselves are not part of the ongoing action.

Social dysfunction is a core feature of many psychiatric disorders like alcoholism, depression and, above all, schizophrenia. Adequate social functioning strongly relies upon the ability to engage in complex forms of joint attention enabling the individual to infer other peoples’ mental states and thus, to form “a theory of mind”. It has been shown that fronto-subcortical networks, which are characterized by pronounced structural and functional changes in psychiatric disorders, represent the most important neuroanatomical substrate of social cognition. Furthermore, there is accumulating evidence that social cognition both draws upon and contributes to other higher-order cognitive processes, like for instance executive control functions, which are also mediated by fronto-subcortical circuitry.

One cognitive domain which strongly requires the ability to infer other peoples’ intentions, often communicated in an indirect manner, is the use and understanding of pragmatic language. The talk will focus on data regarding two instances of pragmatic language comprehension - the understanding of verbal humour and proverbs - in alcoholism, depression and schizophrenia. Additionally, the contribution of deficient executive control mechanisms to impaired pragmatic language comprehension and the consequences of these deficits for the social life of patients suffering from psychiatric disorders will be elucidated.

A foundational skill in human social interaction, joint attention is also receiving increased interest in human-robot interaction and in virtual humans research. Joint attention can be defined as simultaneously allocating attention (i.e. intentionally directed perception) to a target object as a consequence of attending to each other’s attentional states. While existing computational models mostly deal with surface behaviors like simultaneous looking or perceptual attention, aspects of aligning the attentional foci of the interactants are not covered.

We (joint work with Nadine Pfeiffer-Leßmann) investigate joint attention in a cooperative interaction scenario with the virtual human Max. The human interlocutor meets the human-sized embodied agent face-to-face in 3D virtual reality. The human’s body and gaze are picked up by Max by use of an infrared camera system and an eye-tracker; e.g., Max can follow the human’s gaze as a basic manifestation of joint attention. The agent’s mental state is modeled in the BDI (Belief-Desire-Intention) paradigm and serves as the origin of attention mechanisms. For establishing joint attention, three main aspects are considered. Firstly, the human interlocutor’s focus of attention has to be inferred from the interlocutor’s overt behaviors. Secondly, the situational context is taken into account by activation processes marking relevant objects as salient. Thirdly, the agent itself needs to display appropriate overt behaviors to accentuate its focus of attention and manipulate the interlocutor’s mental state.

As an indicator of the human interlocutor’s focus of attention, the human’s gaze is evaluated; e.g., an object is detected as being in human’s attentional focus when it has been focused at least for a total of 400 ms in a 600 ms time frame. In addition, pointing gestures may serve as intentional cues. For the agent to ascribe a desire to establish joint attention to its interlocutor, the following heuristic is used: An object has to be focused by the human interlocutor several times, with additional glances addressing the agent in between (triadic intentional relation).

When the activation of an object passes a threshold and the interlocutor has shown interactive glances, the agent asserts a belief about the interlocutor’s intention and responds, e.g., by gazing at the same object to achieve joint attention. While attention detection can be seen as a prerequisite for establishing joint attention, the agent also employs pro-active mechanisms to manipulate the interlocutor’s focus of attention, e.g., by intentional gaze or pointing gestures.

Pfeiffer-Lessmann, N., & Wachsmuth, I. (2008). Toward alignment with a virtual human – achieving joint attention. In Dengel, A.R., Berns, K., & Breuel, T.M. (Eds.), KI 2008: Advances in Artificial Intelligence. Berlin: Springer, to appear.

Wachsmuth, I. (2008). 'I, Max' – Communicating with an artificial agent. In Wachsmuth, I. & Knoblich, G. (Eds.), Modeling Communication with Robots and Virtual Humans (pp. 279-295). Berlin: Springer (LNAI 4930).