Multimodal, interactive media and the illusion of reality Elena Pasquinelli CRAL - EHESS [email protected]

Cecilia is sitting in a movie theatre and seeing “The purple rose of Cairo” once again. Suddenly, Tom Baxter, the main character, comes out from the screen and invites Cecilia for a romance. Tom, a character, has become alive and now walks with her in the real world, while Cecilia, the real girl, can have a walk in the shining, fictional world of “The purple rose of Cairo” (The purple rose of Cairo, 1985). To Cecilia Tom is no more a representation, a moving and sounding image that excites her imagination to make “as if” he was real. Cecilia can now believe in the reality of Tom. Do we believe in the reality of fictional characters, places and worlds as Cecilia believes in the reality of Tom? Can we forget the presence of the medium or the fact that what we perceive is not “filmed” reality? Can we feel as if being there, in the fictional world, or as if fictional characters were here in our real world? It is a fact that we are strongly caught by the story, that we are moved by fictional characters and worlds. But are we fooled as to believe that what we perceive is real?

Illusion of reality in traditional media

It is meaningful that one of the seminal myths of cinema consists in the image of terrified spectators running away form the theatre during the first projections of Arrival of a train at La Ciotat (1896)

1

[Loiperdinger & Elzer, 2004]. This myth well describes the fact that cinema, since its birth, candidates to the position of the most “illusionistic” of arts. Movement of images is at first the very ingredient of this achievement, in contrast with the stillness of photography. Later, the introduction of sound will reinforce this image by reinforcing realism. But illusionism of cinema will not be bound to the reproduction of reality, and to Lumière’s documentary style. Since the very beginning of cinema history, authors like Georges Méliès will exploit the new medium for proposing their audiences with fantasy worlds, trips to the Moon, uncanny monsters. And creative authors like René Clair will accuse sound of spoiling cinema of cinema the capacity of letting spectators forget reality and enter the fictional world [Clair (1929), 1985]. In 1817, Coleridge the poet had described this mental state as of “suspension of disbelief” [Coleridge, 1817]. Others have referred to it as to an “illusion of life” or “illusion of reality” [Thomas & Johnston, 1984]: the illusion that fictional characters, including animated characters, are real and alive. Philosophers have long discussed about the nature of fiction and of the public’s engagement in fiction. An exemplary debate was raised in 1975 by a paper from philosopher Colin Radford who introduced the “paradox of fiction” [Radford, 1975]. In short, Radford asked: how can we be moved by the fate of Anna Karenina? In fact we can only be moved if we believe in the existence of Anna Karenina (if we hold existence beliefs). But we know that Anna Karenina is not real (we do not hold existence beliefs). And in spite of this, we are moved, irrationally moved. The subsequent debate shows a substantial accord on the fact that existence beliefs are not sufficient and not necessary in order to explain the public’s engagement and emotions produced by fiction. It is a fact that reactions to fiction are similar but not identical to reactions to reality and beliefs [Walton, 2001; Carroll, 1990; Currie, 1995]. First of all, behavioural reactions are mostly inhibited. Spectators might close their eyes, startle, cry, shout, but they do not go further as if they did really believe in the reality of what they see. They do not call the police or fly away from horror movies. And if we accepted the myth that Lumière’s spectators did leave their seats because they did believe in the existence of

2

the filmed train, we should nevertheless acknowledge that this phenomenon has not often reproduced. Second, emotions and beliefs generated by cinema and fiction in general have a short duration: they disappear more or less with the end of the movie. In short, it is true that we are moved by the fate of Anna Karenina, thrilled by the menace of green slimes and horrified by blood-sucking entities, but we do not come to believe that our world is populated by monsters or grieve for long. This difference can be explained if we assume that adults that endorse fictional experiences just pretend to believe in the existence of the green monster, and for consequence pretend to feel (or really feel) afraid [Walton, 1978, 2001]. Like mud and sand in children’s games of make-believe, images, sounds and words are the props that excite, sustain and constrain a mental activity which does not consist in forming beliefs, but imaginings. Imagination and belief must nevertheless be related if we want to account for the similarity - and not just for the difference between responses to the real world and responses to fictional ones. Because, during the green monster movie we are afraid or quasi-afraid. This is possible if we consider (as [Currie, 1995]) that the activity of imagination which responds to fiction is a form of simulation in which beliefs are run off-line. This means that while beliefs are cut-off from their normal causes (the existence of some entity or event) and from their behavioural consequences (actions and full emotions), the connections that normally hold between beliefs (a monster is a dangerous creature) and between beliefs and emotions (a monster is frightening) are preserved [Currie, 1995]. Fictional worlds that are capable of inducing this belief-like mental state of imagination can be defined as believable worlds. Since theories based on existence belief can explain only the similarity between responses to real and fictional worlds, theories based on imagination have a greater explanatory power, because they can account also for the differences.

Illusion of reality in new media

3

The debate about the illusion of reality has recently been re-actualized by the development of “virtual reality” applications of various kinds. Virtual reality applications include new form arts, such as virtual drama, and in general interactive forms of creative and/or aesthetic experience. The term “virtual reality” ambiguously refers to a form of representation and to the technologies that mediate representations that are ideally multimodal and interactive. Another ambiguity is introduced by the notion of “immersion” in the virtual world. Immersion is obtained thank to multimodal stimulation or immersive systems, like headmounted displays or gigantic sceens that cover at least three of the walls of the virtual reality room. It is not uncontroversial whether the term “virtual reality” refers to immersive systems only, or can be applied to more diffused systems like video-games and on-line games [Biocca, Kim, Levy, 1995; Kim & Biocca, 1997; Lessiter, et al., 2001; Lombard & Ditton, 1997, 2000; Loomis, 1992; Riva, et al., 2003; Steuer, 1995; Vorderer, et al. 2004]. For these reasons, I prefer to illustrate my use of the term “virtual reality” through some examples. I will start with LifePlus [Papagiannakis, et al., 2002], an interactive drama set up in the real environment of Pompeii ruins (for this reason called “mixed reality system”). The application consists in the simulation of a day in ancient Pompeii life. Five realistic virtual humans interact with each other and manipulate objects around a tavern following a script. Users wear a head-mounted display and witness the representation while moving through real ruins. Hair and cloth animation, skin rendering, behavioural animation, facial emotion expression, intentionality, desires, beliefs, actions are accurately represented [Papagiannakis, et al., 2005]. Virtual characters of the virtual world do not have however any “awareness” of the real users: they do not react to their presence, but continue to play their script. Additionally, real users simultaneously exploring the scene cannot interact between each others. It is as if Cecilia could jump in the world of the Purple rose of Cairo without being noticed by Tom Baxter and the other characters.

4

Other experiences of virtual drama have been conceived to allow users to interact with virtual agents embedded into the virtual environment, and to actively modify the eventual original script of the play. It is the case of Edge of Intention [Reilly, 1996] and Animated puppets [Doyle, 2002; Hayes-Roth, 2001]: the fictional world is inhabited by four ball-like creatures called Woogles. One of them is controlled by the user, while the others are controlled by the computer. There is no set story, but the Woogles have personalities, emotions, and engage in social behaviours. Spectators become actors in a not-completely pre-defined world. The world is nevertheless structured by the definition of the environment, personality and possible reactions of the computer-guided characters. There’s no interactivity between human users.

Interactivity between users is on the contrary an essential attribute of Second Life, a synthetic on-line world where agents are digital representations of users (called ‘avatars’). In addition to human-to-human social interaction, Second Life users are enabled to create virtual objects of art, pieces of furniture, architectural buildings. This is done by combining and modifying elementary forms (a sort of virtual and

5

flexible Lego building set). Complex objects are subsequently associated with colours and textures, but also sounds in order to create multimodal objects, including art objects.

It is not difficult to imagine a further evolution in which users will be able to feel and not just to see the texture of the virtual objects of art [Frisoli, et al., 2006]. This is the scope of so-called haptic devices, a crucial feature of virtual reality systems. Force feedback or haptic devices exploit one specific component of active touch: the fact that material objects resist (oppose a contrary force) to the force which is impressed by the touching agent. This force is perceived through deep touch receptors that are located in the muscles and joints rather than in the skin. It conveys sensations that are identified by users as specific textures, materials or as specific shapes. A “haptic shape” can then be represented by producing a characteristic pattern of resistance and by applying it – thank to a material, suitable device - to the fingers, hand or arm of the user. Haptic devices are exemplary of the multimodal possibilities and of the interactive nature of virtual reality systems. Perceptual experience depends in fact, as in natural perception of the real world, on the specific perceiver’s actions and exploratory movements (other devices exist that passively stimulate the skin through vibrations and provide tactile, superficial sensation, or even temperature perception). Haptic devices are actually in use for producing virtual museums of Pure Form in which users can visualize 3D graphic reproduction of distant works of art (sculptures), and also explore them with their hands and feel their shape from the contact [Carrozzino, et al.,

6

2005]. In a creative perspective, virtual sculptures could also be modified and exploration could become active creation of new works of haptic or multimodal art.

It is a fact, then, that thanks to virtual reality systems, new forms of aesthetic, social and creative experiences are at reach that enlarge the field of human possibilities. But the question is: are virtual reality systems responsible for a qualitative shift from the realm of traditional engagement through imagination to the realm of illusion of reality? Does virtual reality makes a real, qualitative difference in the perception and cognition of users by inducing existence beliefs? An optimistic answer is largely diffused in the domain of virtual reality [Dihn, et al., 1999; Witmer & Singer, 1997; Witmer & Singer, 1998; Lombard & Ditton, 1997; Slater & Steed, 2000]. Applications like Second Life, LifePlus, or the Museum of Pure Form would then be better candidates than traditional cinema to produce a compelling illusion of reality. Multimodality and interactivity are identified as the responsible factors for this qualitative shift.

Optimism about the capacity of new media of producing full illusions of reality encounters nonetheless a major conceptual difficulty: the notion of illusion of reality has not received an uncontroversial definition yet. How, then, can we measure a qualitative shift towards an objective that we are not able to precisely identify? In the literature about virtual reality we encounter in fact at least three different definitions of “illusion of reality”: illusion of non-mediation, fidelity, illusion of being there. In what follows I will show

7

that each of these uses of the term “illusion of reality” corresponds to specific cognitive and perceptual attitudes. Furthermore, each of them entails different possibilities of success, different enhancing conditions and different measures of achievement. My claim is thus that designers of creative and artistic experiences with virtual reality systems should rather carefully choose their objective rather than pursuing an underspecified idea of illusion of reality.

1. Illusion of reality as illusion of non-mediation

The first definition of the illusion of reality is: illusion of non-mediation. The user fails to acknowledge (cognitively and perceptually) the presence of the technological medium which is responsible for the experience of the artificial world: “an illusion of nonmediation in which users of any technology overlook or misconstrue the technology's role in their experience.” [ISPR, 2000]. The failure to acknowledge the presence of the medium would produce the “hallucination of an apparently concrete and persisting threedimensional object in the real world...” [Dennett, 1991, p. 7] The illusion of non-mediation thus reminds of the famous mental experiment of the brain in a vat [Putnam, 1982]. A mad scientist has removed the brain from the body of a person. The brain is now kept alive in a vat, thank to a special liquid. Wires connect the brain to a computer which stimulates the brain so as to produce perceptual sensations. These sensations include sensations of movement, and all the sensations that a human being can receive in response to his actions in a real world. Only, the world is a synthetic, digital simulation produced by the computer. The mental experiment of the brain in a vat has been used in order to illustrate the sceptical argument that we cannot know whether there is a world outside or if we are just brains in vats or…lost in a virtual reality environment. Science fiction’s expert can recognize the basic plot of TheMatrix (1999), the direct brain recordings of Strange days (1995), the basic features of cyberpunk literature. As people immersed in the

8

matrix or brains immersed in a vat, deluded users of virtual reality systems are supposed to fail perceiving the medium (perceptual component) and to forget their knowledge of the medium (cognitive component). The illusion of non-mediation is double: perceptual and cognitive.

This definition attracts an objection on the side of motor theories of perceptual functioning [O’Regan & Noe, 2001; Stoffregen & Bardy, 2001]. According to these theories, perception is not the simple recording of states of the world: the content of perception varies both along factual conditions (how the world is) and along other conditions, such as the way the perceiving organism moves. Let us see an example. A visual representation, for instance a statue of the Museum of Pure Form, is transmitted through a head-mounted display, while an exoskeleton mediates haptic representations of the contours of the same statue. The user perceives the presence of both media through his tactile system. The medium is then perceptually acknowledged as part of the factual content of perception. When the visual stimulus is presented through large screens the cover the virtual reality room by transforming it in what is called a “cave” the visual medium is no more experienced in this direct fashion. Nonetheless, the presence of the medium could still affect “the perceiver’s relation to how things are” [Noë, 2003, p. 94]: seeing a baseball game on TV, for instance, structures the optical flow in a way which is different from seeing the same baseball game live, and the movements of the perceiver have a different effect on the perceptual result in the two situations [Stoffregen, 1997]; immersion in a flight simulator will leave behind the vestibular components that are typical of the same, multimodal experience in the real world [Stoffregen, et al., 2003]. The presence of a medium can thus have an effect upon perspectival or motor contents of perception, even in the case in which the medium would not affect the factual contents. These effects make the difference between mediated and non-mediated conditions, and perceptually inform the user about the mediated nature of its experience.

9

In addition to non-interpretative, natural capacities other interpretative, learnt capacities are used in order to make sense of experiences with virtual and fictional environments. A long apprenticeship with make-believe games paves the way to adults’ capacity of dealing with representations displayed by representations (such as theater) that are not displayed though a particular perceptual medium. Can knowledge of wearing material devices (even if they are not perceived) and awareness that the experience is of an artificial world be evicted from the mind of the user?

Weaker versions of this definition exist however, that do not entail similar objections. It has been proposed, for instance, to consider the “illusion of non-mediation” as an attentional state [Witmer & Singer, 1998; Slater & Steed, 2000]. Attention can alternatively be focused on the virtual world (lime the world represented in Second Life), the mixed world (the real Pompeii ruins with virtual agents moving around), or on the real room and devices. Shifts towards the second condition can be provoked by sounds (produced by other people in the room, by the functioning of the devices), by uncomfortable conditions (a heavy, constraining haptic or visual devices, nausea), or other perceptual agents of distraction (odours from the kitchen announcing lunch time). These shifts can be qualitatively and quantitatively measured; they will quantify the compelling power of the virtual world in engaging the user’s attention to focus on a selected group of stimuli. However, shifting the focus of attention cannot be considered as an illusion, because there is no error. The weaker version of the illusion of non-mediation is then not an illusion at all. It is now an empirical matter whether this compelling focalisation of attention is enhanced by new media in comparison with traditional ones, and which media characteristics are more effective. Multisensoriality and immersion in the virtual environment seem to be rather effective [Dihn, et al., 1999; Witmer & Singer, 1997; Slater & Steed, 2000]. In fact, they both isolate the user from the stimuli that come from the real world and require his attention in order to cope with multiple stimuli and to correctly act.

10

2. Illusion of reality as fidelity

The second definition of the term “illusion of reality” is: illusion of fidelity to reality. Users are aware that their experience is the effect of a medium, but believe that the medium reproduces or presents reality. Now, this is not true, since the medium represents events that do not occur in reality. As for the illusion of non-mediation, the fact that the user is mistaken is a crucial condition for ranging the experience between illusions. Cannibal holocaust (1980), The Blair witch project (1999), fake snuff movies, all try to establish this form of illusion or hoax: that a fiction film is in fact a documentary that represents actual reality. Let us see the most famous case of hoax. in 1938 a series of newsflashes interrupted a music program of CBS: it was Orson Welles announcing and giving voice to Martian landing close to a US farm. Some audiences took the fake newsflashes for real, and run into panic. Audiences knew that the sounds of destruction came from a radio and were recorded in a distant place from them, but some of them also thought they were listening at the sound produced by real events. The smuggling of fictional contents in place of real ones is the object of strict rules in several ethical codes of communication targeting hoax and fraud [Steele & Black, 1999]. At the time of Welles’ broadcast of the War of the worlds, CBS escaped punishment because the fictional nature of the performance was reminded several times during the broadcast. Nonetheless, CBS was formerly invited to avoid the “we interrupt this program" device for fiction. Since then, US TV broadcasts featuring realistic news bulletins post messages to inform audiences of the fictional nature of the spectacle. It is plausible to require that new media, such as virtual reality, do not escape this rule of ethical communication.

The form of fidelity to reality which is invoked in the domain of virtual reality is of a special type: it concerns the perceptual and eventually motor aspects of the experience and is called subjective fidelity

11

[Riccio, 1995]. It consists in the “extent of subjectively experiences similarity between simulator and the real-life situation (Korteling and Sluimer, 1999), hence between the simulator and the simulated (Stoffregen et al., 2003)” [Morice, et al., 2008]. This it is grounded in a comparison between reality and simulation. The user of a flight simulator that finds sensations produced by the simulation very similar to sensations experienced during natural experiences thus continues to acknowledge the perceptual the medium (the simulator), or the artificial nature of the experience. Again, mystification and misperception of the medium are not necessary conditions for undergoing the experience of subjective fidelity. However, other forms of illusion can be useful for producing subjective fidelity. Subjective fidelity is in fact significantly different from realism as the simulation of the features of the world. As we have seen before, for instance, force-feedback devices cannot reproduce all the variety of tactile, kinesthetic, thermal stimulation that is produced by the manipulation of real objects. The sensation of touching a virtual representation of a statue through a state-of-the-art haptic device will then be far from the complexity of natural sensations generated by the touching of a real statue with no medium. However, the fidelity of the perceptual experience obtained through these systems can be enhanced or suitably modified by appropriate associations with visual stimuli. Multimodal devices can thus exploit multimodal illusions in order to produce fidelity through “unrealistic” stimulations. Let us see some examples of classic visuo-haptic illusions, as they have been described in literature, and as they can be exploited by multimodal displays. [Gibson, 1933] describes the following illusion: a subject moves his hand along a straight surface while looking through a prism that causes the surface to look curved; he feels it to be curved as well. [Nielsen, 1963] introduces the use of the mirror. The subjects follow a straight line with their hand in full sight; on some of the trials, a mirror is introduced, unbeknownst to the subjects, so that they see another person’s hand (that they believe to be their own): the subjects continue to have the sensation that the seen hand is their own, but they also feel as if they had lost control over its movement. In both cases discrepant visual and haptic/proprioceptive

12

information produce a vivid haptic/proprioceptive experience which does not correspond to the haptic/proprioceptive stimulation, but to the visual one. Other cognitive components can play a role in the aspect of the final percept, hence on subjective fidelity. [Lederman & Abbott, 1981] have presented the subjects of their experiments with two different abrasive surfaces, one to be examined by vision and the other by touch; the subjects were lead to believe that they were exploring one and the same surface (the experimenters induced an assumption of unity). When asked to match the perceived surface with one from a set of surfaces, the results showed that, touch is more influential when the subject is requested to evaluate the roughness of a surface, while vision is more influential when the subject is requested to evaluate the spatial distribution of the dots for the same surface: the kind of solution adopted thus varies with the verbal instruction assigned to the subjects [Lederman, Thorne & Jones, 1986]. This fact indicates that narratives, stories and even simple verbal instructions can influence what the user will feel and thus have an effect on subjective fidelity. Perceptual fidelity can be evaluated through subjective measures, like free reports in which the user evaluates the degree of similarity between sensations in the artificial set and sensations experienced during comparable, natural situations [Stoffregen, et al., 2003]. Rather than asking “how real does this seem”, a better question might be “how closely does this resemble the real situation?” [Stoffregen, et al., 2003] propose the concept of action fidelity as a measure of the perception of fidelity for interactive systems (like flight simulators): when actions in the virtual world are comparable to actions in the real one, then the system shows a great fidelity. This measure makes sense only in the case of perceptual experiences that moreover aim at reproducing real experiences. This can be the case for the Pure Form Museum. The user will in fact be able to compare how the simulated experience resembles to previous experiences in real museums. But this measure will be unsuitable for virtual theatre with caricatured, purely fictional characters as the Woogles. In

13

this case, the concept of fidelity will be rather applied to the complexity of the character’s personality, the reactions of computer-guided characters to the user-guided character’s actions, and their relative coherence with natural human beings (a form of social rather than a perceptual or motor fidelity). In the same way, multimodality will loose much of its power as an enhancing factor of fidelity, while the nature of the social interactions will take the most importance.

3. Illusion of reality as illusion of transportation. The third definition of the term “illusion of reality” is: illusion of being there in the virtual environment [Riva, et al., 1999; Slater & Steed, 2000]. Can users experience an illusion of being there without being mistaken about its fictional nature and without loosing the awareness of mediation? In the literature about virtual reality, the illusion of non-mediation and the illusion of being there are confounded into the definition of presence: “The International Society for Presence Research (ISPR) supports academic research related to the concept of (tele)presence, commonly referred to as a sense of 'being there' in a virtual environment and more broadly defined as an illusion of nonmediation in which users of any technology overlook or misconstrue the technology's role in their experience.” [ISPR, 2000]. This is not necessarily so. I will consider that the illusion of being there is the label assigned to a group of illusions of position and movement (proprioceptive illusions) in the artificial environment, and of identification of parts of the body with artificial representations. Now, the experience of local perceptual illusions like these (rather than of a massive illusion of reality, or cognitive illusion) is compatible with the knowledge of the mediated nature of the experience. Perceptual illusions present in fact a special feature described as “resilience to knowledge”: they persist in spite of the fact that the subject of perception knows that what she is experiencing is an illusion.

14

Let us go back to one of our examples. Second Life users “enter” the virtual world after having created an avatar of themselves, a virtual representation of their self. They see their avatar flying, assisting to virtual expositions of art and performances, creating objects, manipulating them. The illusion of being there in the virtual environment is then equivalent to the illusion of moving when the avatars is seen moving and to the illusion that, at some extent, the hands of the user are there where the hands of the avatar are manipulating virtual objects. We have then three main components in the “illusion of being there”: the first component is a general visual (or haptic, or acoustic) bias upon proprioception. The second component is an experience of spatial identification with some artificial entity, associated with an out-of-body experience induced by vision (a proprioceptive illusion biased by vision). The third one is an illusion of movement. In fact users are often not allowed to move around while exploring the virtual world (they can do it in mixed reality like LifePlus or in immersive systems), and for sure they cannot fly. This component includes specific illusions of movement. As an example of visual bias, in the previous section I have illustrated a proprioceptive illusion induced by vision in which the subject feels as if the hand of someone else was his very one [Nielsen, 1963]. There are many other examples of visual bias over other perceptual modalities [Welch & Warren, 1981; Rock & Victor, 1964; Holmes, 2004; Hay, et al., 1963; Mon-Williams, et al., 1997]. More recently, some experiments on out-of-body experience have used virtual reality in order to propose experimental subjects with discrepant visual and proprioceptive information. These studies have independently confirmed the possibility, given certain conditions, of visual bias upon proprioception when the two are discrepant even for the whole body (and no only for limited parts such as the hand) [Ehrsson, 2007; Lenggenhager, et al., , 2007; Miller, 2007]. Proprioceptive illusions so produced are currently exploited for the treatment of pathologies such as phantom limbs, stroke, hemiplegy and spatial neglect (the success with pathologies other than phantom limbs is however still uncertain [Holmes, 2004; Altschuler, et al., 1999; Sathian, et al., 2000;

15

Ramachandran, et al., 1995]). Vision has for long been considered the dominant perceptual modality in case of mismatch between information presented to the visual perceptual modality and information presented to the auditory, proprioceptive or haptic modalities. The model of visual dominance has been contested, and evidence shows that all the sensory modalities differently contribute to the aspect of the final percept; however, the more precise, less variant information biases the less reliant one [Ernst, et al., 2004]. However, it is justified to consider that vision normally dominates other modalities in spatial tasks, because vision is usually more precise and reliable for spatial location. So, vision can be very helpful in broadening the limits of non-immersive virtual reality systems. Also the identification of a part of the body or self with artefacts (like avatars) seems to be possible, in certain conditions. In experimental conditions, for instance, a subject can feel a rubber hand placed in face of him as his own hand. The illusion is known under the name of “fake hand illusion” or “rubber hand illusion” [Tastevin, 1937, Botvinik & Cohen, 1998; Pavani, et al., 2000; Austen, et al., 2001]: a subject watches a rubber hand; the hand is stroked synchronously with his own hidden hand (positioned congruously with the rubber hand). Subsequently, the subject undergoes an illusion of identification and an illusion of dislocation: she identifies her hand with the rubber hand and she feels her hand to be “there”, where the rubber hand is. . It is then possible to “quantify” the illusion by measuring how close the felt position is to the real position of the rubber hand. It is worth noting that this illusion depends on both visual and tactile stimulation of the real hand [Schaefer, et al., 2005]. A variance of the rubber hand illusion can in fact be obtained without vision (with tactile and proprioceptive modalities) [Ehrsson, et al., 2005]. In order to produce the same phenomenon with a digital rather than a rubber hand it would then be necessary to dispose of a multimodal virtual reality set, that include tactile or haptic devices. Another important aspect of the illusion is the level of “realism” or fidelity which is required in order to generate the illusion. Can we undergo this kind of illusion with avatars that look like animals (as many Second Life avatars) or like

16

Woogles? It is controversial whether the “sense of ownership” and the displacement of the sensed position of the hand can be obtained with objects that do not resemble to the human body (as suggested by [Armel & Ramachandran, 2003] but invalidated by [Tsakiris & Haggard, 2005]). Evidence from the incorporation of tools seems to confirm that realism is not a necessary condition for the extension of the peripersonal space to hand-held tools [Holmes et al., 2004]. Experiments in mediated conditions are required in order to validate the hypothesis that intermodal bias upon proprioception occurs during perceptual experiences with virtual agents and objects, and that users locate their body or parts of it in the place where their avatar is. For the time being, [Ijsselstejn, at al., 2006] have reproduced rubber hand experiments in virtual (projected rubber hand and stimulus) and mixed reality (projected rubber hand and real stimulus) conditions, showing the presence of a weaker illusion in the two synthetic conditions than in the non-mediated one. Different factors can explain the difference. In all the ‘natural’ cases of visual bias upon proprioception, for instance, active movement and the activation of another perceptual modality (touch or audition) seem to be necessary conditions for the illusion to take place. The induction of proprioceptive illusions could hence depend on multimodality and interactivity. The role of multimodality in the enhancement of proprioceptive illusions is confirmed by the third component of the illusion of “being there”: illusions of self-motion or “vection”. Vection illusions are illusions of movement that can be induced by vision, but also by audition and are enhanced by the combination of visual and auditory cues and of visual cues with vibration. Top-down processes seem to enhance the illusion as well: whether the source of the auditory stimulation can be consistently interpreted with self-movement, chances of undergoing illusions of vection produced are increased [Riecke, et al., 2005]. The role of multimodality in the production of illusions of position and movement confirms the idea that virtual reality systems are the best candidates for producing effects of presence and movement in the synthetic world [Witmer & Singer, 1998; Lombard & Ditton, 1997]. Virtual reality systems are in fact

17

normally multimodal and haptic, and tactile and vibratory devices are more and more associated to visual and auditory displays. On the other hand, [Holmes, et al., 2004] alert to the fact that the presence of discrepancies between proprioception and the visual position of the hand have disruptive effects on reaching tasks (measured in terms of lowered speed and of accuracy, that is, of the distance between the finger tip and the target to be reached). This fact confirms a general consideration about the importance of coherence for successful behaviour [Stein & Meredith, 1993], and is pragmatically significant for virtual reality contexts: the bringing about of illusions has not necessarily positive effects, and while it enhances the sense of presence it can disrupt performance. For this reason, the production of illusions of presence and transportation cannot be considered as a universal desideratum for virtual reality applications.

Believability and imagination We have seen that the correct manipulation of multimodal stimuli can give rise to local illusions of movement and of presence in the setting of virtual dramas, virtual museums or artistic landscapes and in the use of virtual creative tools. These illusory phenomena are far from equating the massive form of delusion which is described as illusion of non-mediation. They can be associated (but are not necessarily associated) with focalised attention. They can also be associated with judgments of fidelity to the perceptual experience of the parts of the real world or to social and motor behaviour of inhabitants of the real world. Focalised attention and the judgment of fidelity to a certain perceptual experience can never be considered as an illusion, even if the second can be sustained by the play of multimodal stimulation. Proprioceptive illusions, attention, and subjective fidelity are then three important mental components of the experience of virtual or fictional worlds. Each of them can be differently and separately measured and enhanced, in order to produce a more vivid, more engaging experience of virtual forms of art, performative, visual or multimodal.

18

Subjective fidelity and multisensoriality can be especially important in virtual reality training applications, such as flight simulators, in tele-operation, such as at-distance surgery, but also in cultural heritage settings, like museums composed of representations of existing objects of art. At the opposite, in the domain of virtual arts, realism and also multisensoriality could present undesired drawbacks; while unrealistic experiences seem to be very effective in stimulating a rich and engaging activity of imagination. We have said that visual, acoustic, and haptic representations, work as props that awake and constrain imagination. We have also seen that this particular form of imagination can be identified as the running offline of beliefs. Works of art or of fiction that are capable of exciting this kind of activity are “believable”. Believable fictional worlds can be inhabited by green monsters, ancient Pompeiians, and Woogles. Believability is thus not the same than subjective fidelity, because believability does not require that the fictional world resembles to the real one. It just requires that the representation succeeds in exciting and constraining imagination towards the imaginings that are appropriate to that particular representation. Believability can thus be defined as the condition that enables users, audiences and readers to imagine the specific world which is represented by the work of fiction, and of running the relevant beliefs off-line. For these reasons, even abstract representations can be considered as believable [Walton, 2001]. If the representation is supposed to guide audiences in imagining colours, shapes, sounds, bodily sensations in a certain way, and audiences do; then the representation is believable as the representation of a perceptual world where colours, shapes, sounds and bodily sensations occur in a certain way. But it is possible that a work of art fails in exciting the relevant beliefs and imaginings that belong to the world that the work of art intended to represent. The work of art is not believable, then. Which are the conditions that enhance or threat believability? Believability seems to depend much on expectations. When expectations raised by the experience are unfulfilled or coherence between the elements of the experience is

19

violated one is not disposed to form beliefs, and judges the experience as unbelievable. This is showed in particular by experiences of conflict, like perceptual conflicts between perceptual modalities: prosthetic hands that look very realistic but feel hard and cold can raise a sense of uncanny that immediately lowers believability [Mori, 1970]. In this case, believability concerns the belief that the hand is a real one. In the domain of fiction, spectators and users can be asked to imagine that a certain fictional world is inhabited by agents that fully look like to real human beings. It is the case for recent animation movies such as The polar express, or Final Fantasy. At this scope, motion capture systems are employed to provide agents with realistic biologic motor behaviours (obtained through captors that are placed at the level of the principal joints of real human beings and that serve to represent the movement of the artificial agents). However, facial expressions are not as realistic as bodily movements. The result is again an uncanny effect of unbelievability. . Realism can then represent a risk for the believability of virtual agents, since it exposes the public to conflicts and to the frustration of expectations [Garau, et al., 2003; Goetz, et al. 2003]. If this is proved, a recipe for obtaining believable agents could be: lower the user’s expectations, display odd animals and quirk aliens rather than simulations of human beings, follow the example of the successful virtual pets called Tamagotchi. And eventually provide broad, if shallow, characters with a specific personality, tics, social and goal directed behaviours, reactivity to the environment [Bates, 1994; Reilly, 1995; Papagiannakins, et al., 2005]. In any case, an important consequence of the dependence of believability on the fulfilment of expectations is that believability is enhanced rather than bothered by the awareness of mediation and virtuality. In fact, the awareness that virtual agents and objects are not real lowers the users’ expectations and makes their fulfilment easier, thus reducing the risk of a drop of believability.

20

Naturally, artistic applications can choose to play with the frustration of the expectations they raise. The Pebble box is a multimodal mixed object composed of a real box filled with pebbles and of a library of sounds that are activated when the users manipulates the pebbles, and stop when haptic exploration ends [Essl, et al., 2005].The sound library contains synthetic sounds of pebbles, sounds of pebbles in water, sounds of coins, sounds of crunching apples, sounds of birds. While the first three sounds appear believable when associated with the haptic sensation produced by the pebbles, crunched apples and birds appear weird and the proposed world is unbelievable as a world of pebbles that crunch or chirp. Users do not imagine themselves touching pebbles that crunch or chirp, but just touching pebbles and hearing sounds that do not correspond to how pebbles should sound.. The example of the Pebble box, as the ones of LifePlus, of virtual dramas featuring the Woogles, of virtual museums of Pure Form and of on-line fictions like those one can invent in Second Life show that very different virtual reality devices represent a wonderful tool for exploring how far artists can go in manipulating stimuli for different perceptual modalities. For measuring the limits of believability and exploring the effects of unbelievability. For creating worlds never experienced and make imagination live them… though the senses.

21

References

Altschuler, E. L., Wisdom, S. B., Stone, L., Foster, C., Galasko, D., Llewellyn, D. M., & Ramachandran, V. S. (1999). Rehabilitation of hemiparesis after stroke with a mirror. Lancet, 353, 2035-2036. Armel, & Ramachandran, V. S. (2003). Projecting sensations to external objects: evidence from the skin conductance response. Proceedings of the Royal Society of London: Biological(270), 1499-1506. Austen, E. L., Soto-Faraco, S., Pinel, J. P. J., & Kingstone, A. F. (2001). Virtual body effect: Factors influencing visual–tactile integration. Abstracts of the Psychonomic Society, 6, 2. Bates, J., Loyall, A. B., & Reilly, W. S. (1991). Broad Agents. Paper presented at the Proceedings of the AAAI Spring Symposium on Integrated Intelligent Architectures, Stanford University. Botvinick, M., & Cohen, J. (1998). Rubber hands “feel” touch that eyes see. Nature Reviews Neuroscience, 391, 756. Carroll, N. (1990). The Philosophy of Horror; or, Paradoxes of the Heart. New York: Routledge. Carrozzino, M., Frisoli, A., Rossi, F., Tecchia, F., Ruffaldi, E., Bergamasco, M. (2005). The Museum of Pure Form. Proceedings of MIDECH 2005. Clair, R. ((1929) 1985). The art of sound. In E. Weis & J. Belton (Eds.), Film sound. Theory and practice: University of Columbia Press. Coleridge, W. W. (1817). Biographia Literaria. Currie, G. (1995). Image and Mind: Film, Philosophy, and Cognitive Science. Cambridge: Cambridge University Press. Currie, G. (2005). Imagination and make-believe. In B. Gaut & D. M. Lopes (Eds.), The Routledge companion to Aesthetics (pp. 335-346). London New York: Routledge. Dennett, D. C. (1991). Consciousness explained. Boston: Little, Brown & Company.

22

Dinh, H. Q., Walker, N., Song, C., Kobayashi, A., & L.F., H. (1999). Evaluating the importance of multisensory input on memory and the sense of presence in virtual environments. Paper presented at the IEEE Virtual Reality. Doyle, P. (2002). Believability through context: Using "knowledge in the world" to create intelligent characters. In Proceedings of the International Joint Conference on Agents and Multi-Agent Systems (AAMAS 2002), 342-349, ACM Press, Bologna, Italy. Ehrrson, H. H., Holmes, N. P., & Passingham, R. E. (2005). Touching a Rubber Hand: Feeling of Body Ownership Is Associated with Activity in Multisensory Brain Areas. The Journal of Neuroscience, 25(45), 10564-10573. Ehrsson, H.H. (2007). The Experimental Induction of Out-of-Body Experiences. Science, 317(5841): 10481048. Ernst, M., O., & Buelthoff, H. H. (2004). Merging the senses into a robust percept. Trends in cognitive sciences, 8(4). Essl, G., Magnusson, C., Eriksson, J., O’Modhrain, S. (2005). Towards evaluation if performance, control, and preference in physical and virtual sensorimotor integration, Proccedings Enactive05. Frisoli, A., Ruffaldi, E., Gottlieb, C., Tecchia, F., Bergamasco, M. (2006). A Haptic toolkit for the development of immersive and Web enabled games. Proceedings of ACM Symposium on Virtual Reality Software and Technology VRTS2006. Gibson, J. J. (1933). Adaptation, after-effect, and contrast in the perception of curved lines. Journal of experimental psychology, 16, 1-31. Hay, J. C., Pick, H. L., Jr., & Ikeda, K. (1963). Visual capture produced by prism spectacles. Psychonomic Science, 2(215-216). B. Hayes-Roth (2001), "What is a Virtual Person -- Not Just a Pretty Face!", SIGGRAPH2001.

23

Holmes, n. P., Crozier, G., & Spence, C. (2004). When mirrors lie: “Visual capture” of arm position impairs reaching performance. Cognitive, affective & behavioural neuroscience, 4(2), 193-200. IJsselsteijn, W. A., de Kort, Y. A. W., & Haans, A. (2006). Is this my hand I see before me? The rubber hand illusion in reality, virtual reality, and mixed reality. Presence: Teleoperators and Virtual Environments, 15, 455-464. ISPR (2000). The Concept of Presence: Explication Statement. Retrieved <03/2007> from http: ispr info/ 11 Lederman, S., & Abbott, S. G. (1981). Texture perception: studies of intersensory organisation using a discrepancy paradigm and visual versus tactual psychophysics. Journal of experimental psychology: human perception & performance, 7(4), 902-915. Lenggenhager, B., Tadi, T., Metzinger, T., Blanke, O. (2007). Video ergo sum: manipulating bodily selfconsciousness. Science, 24;317(5841):1096-9. Loiperdinger, M., & Elzer, B. (2004). Lumiere's Arrival of the Train: Cinema's Founding Myth. The Moving Image 4:1, 89-118 Lombard, M., & Ditton, T. (1997). At the hearth of it all : the concept of Presence. Journal of computermediated communication. Loyall, A. B. (1997). Believable Agents: Building Interactive Personalities (CMUCS-97-123). Pittsburgh, PA: School of Computer Science, Computer Science Department, Carnegie Mellon University. Miller, G. (2007). Out of body experiences enter the laboratory. Sciencei. 317 (5841), 1020-1021. Mon-Williams, M., Wann, J. P., Jenkinson, M., & Rushton, K. (1997). Synaesthesia in the normal limb. Proceedings of the Royal Society of London, Series B(264), 1007-1010. Mori, M. (1970). The Uncanny Valley. Energy, 7, 33-35. Morice, A.H.P., Siegler, I. A., Bardy, B. (2008). Action-perception patterns in virtual ball bouncing: Combating system latency and tracking functional validity. Journal of neuroscience methods 169(1):255-66

24

Nielsen, T. I. (1963). Volition: a new experimental approach. Scandinavian journal of psychology, 4, 225230. O'Regan, K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 939-1011. Papagiannakis, G., Ponder, M., Molet, M., Kshirsagar, S., Cordier, F., Magnenat Thalmann, N., & Thalmann, D. (2002). LIFEPLUS:Revival of life in ancient Pompeii. Paper presented at the Virtual Systems and Multimedia,VSMM2002. Papagiannakis, G., Schertenleib, S., O'Kennedy, B., Arevalo-Poizat, M., Magnenat Thalmann, N., Stoddart, A., & Thalmann, D. (2005). Mixing Virtual and Real scenes in the site of ancient Pompeii. Computer Animation and Virtual Worlds, 16(1), 11-24. Pavani, F., Spence, C., & Driver, J. (2000). Visual capture of touch: Out-of-the-body experiences with rubber gloves. Psychological Science, 11, 353-359. Putnam, H. (1982). Reason, truth and history. Cambridge: Cambridge University Press. Radford, C. (1975). How Can We Be Moved by the Fate of Anna Karenina? Proceedings of the Aristotelian Society, 49, 67-80. Ramachandran, V. S., Rogers-Ramachandran, D., & Cobb, S. (1995). Touching the phantom limb. Nature Reviews Neuroscience, 377, 489-490. Reilly, W. S. (1996). Believable Social and Emotional Agents (CMU-CS-96-138). Pittsburgh, PA: School of Computer Science, Carnegie Mellon University. Riccio, G. E. (1995). Coordination of postural control and vehicular control: Implications for multimodal perception and simulation of self-motion. In Hancock & Flach & Caird & Vicente (Eds.), Local applications of the ecologial approach to human-machine systems (pp. 122-181). Hillsdale, NJ: Lawrence Erlbaum Associates.

25

Riecke, B. E., Schulte-Pelkum, J., Caniard, F., & Bulthoff, H. H. (2005). Spatialized auditory cues enhance the visually- induced self-motion illusion (circular vection) in Virtual Reality. Tubingen: Max Plank Institute. Rock, I., & Victor, J. (1964). Vision and touch; an experimentally created conflict between the senses. Science, 143, 594-596. Sathian, K., Greenspan, A. I., & Wolf, S. L. (2000). Doing it with mirrors: A case study of a novel approach to neurorehabilitation. Neurorehabilitation & Neural Repair, 14, 73-76. Schaefer. M. (2006). Fooling your feelings: artificially induced referred sensations are linked to a modulation of the primary somatosensory cortex. Neuroimage, 29(1), 67-73. Slater, M., & Steed, A. (2000). A Virtual Presence Counter. Presence: Teleoperators and Virtual Environments, 9(5), 413-434. Steele, R., & Black, J. (1999). Media ethics codes and beyond. American Society of Newspapers editors. Stein, B. E., & Meredith, M. E. (1993). The merging of the senses. Cambridge, Mass.: MIT Press. Stoffregen, T. A. (1997). Filming the world. An essay review of Anderson's The Reality of Illusion. Ecological Psychology, 9, 161-167. Stoffregen, T. A., & Bardy, B. G. (2001). On Specification and the Senses. Behavioral and Brain Sciences, 24(1). Stoffregen, T. A., Bardy, B. G., Smart, L. J., & Pagulayan, R. J. (2003). On the nature and evaluation of fidelity in virtual environments. In L. J. Hettinger & M. W. Haas (Eds.), Virtual & Adaptive Environments: Applications, Implications, and Human Performance Issues (pp. 111-128). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Tastevin, J. (1937). En partant de l'expérience d'Aristote. Encéphale, 32, 57-84. Thomas, & Johnston. (1984). Disney: The illusion of life. New York: Abbeville.

26

Tsakiris, M., & Haggard, P. (2005). The Rubber Hand Illusion Revisited: Visuotactile Integration and SelfAttribution. Journal of Experimental Psychology: Human Perception and Performance, 31(1), 80–91. Walton, K. (1978). Fearing Fictions. Journal of Philosophy, 75(1), 5-27. Walton, K. (2001). Mimesis as Make-Believe: On the Foundations of the Representational Arts: Harvard University Press. Welch, R. B., Warren, D. H. (1981). Immediate perceptual response to intersensory discordance. Psychological bulletin, 88, 638-667. Witmer, B. G., & Singer, M. J. (1998). Measuring Presence in Virtual Environments: a Presence Questionnaire. Presence, 7(3), 225-240.

27