Bridging the gap: from cognitive sciences and traditional media to new media Elena Pasquinelli Institut Jean Nicod (EHESS - DEC, ENS – CNRS) e-mail : [email protected] Abstract The present contribution aims at favoring a transfer of knowledge from the domains of traditional media theory (in particular cinema cognitive theory) and from the psychology of perception to new media theory. The notion of illusion of reality is here taken as exemplary of the possibilities offered by this transfer in terms of conceptual analysis, individuation of cognitive bases of users’ behaviors and ethical considerations.

1. Introduction. The illusion of reality The notion of illusion of reality dominates the field of new media (in particular virtual and mixed reality) through the notions of presence and illusion of non-mediation. The “official concept of Presence” [10] essentially makes reference to the possibility of deceiving the users as for what concerns the origin of their experience: the users fail to recognize the role of technology. Multisensoriality and interactivity are widely accepted as arguments which account for the qualitative shift from the realm of traditional representation to the realm of a full illusion of reality [6], [7], [15], thus making of enactive interfaces a privileged field for discussing the conceptual, cognitive and ethical aspects of the illusion of reality as illusion of non-mediation. First, I will attack myself to the illusion of non-mediation and I will present evidence that traditional media audiences are aware of the fictional, mediated nature of their experience. Second, I will illustrate perceptual studies suggesting that this awareness is not contingent to traditional media, but affects mediated experiences in general in virtue of perceptual information about the medium.

Third, I will present ethical considerations that are operational in traditional media codes of ethics and that suggest that the obliteration of the difference between mediated and non-mediated, fictional and non-fictional experiences should not be taken as a goal for the new generation of virtual reality and enactive media. Fourth, I will discuss some possibilities of naturalization of the aspect of the illusion of reality, which is termed ‘illusion of transportation’ or ‘illusion of presence’.

2. The paradox of fiction: cognitive awareness of fiction Philosophers of art and fiction have long discussed about the possibility of audiences and readers of being massively fooled by works of fiction of different kinds. An exemplary case of this discussion is represented by the debate about the so-called paradox of fiction [12], in which deception is equated to the fact of holding existence beliefs (beliefs about the real existence) about the characters represented on screen or described in books. The paradox can be so expressed: how can we be moved by the fate of Anna Karenina without believing in the existence of Anna Karenina? The debate shows a substantial accord on the fact that audiences and readers do not hold existence beliefs about fictional contents, and this accord is based on the observation that responses to fictional contents are similar but never identical to responses to corresponding non-fictional contents [3]. Two strategies are then adopted in order to explain how we can be moved by the fate of Anna Karenina. So-called Thought theories propose that it is sufficient that certain events are mentally represented, or entertained in thought, in order to raise emotional reactions, eventually by evoking thoughts about real people and events. Existence beliefs are hence not required for experiencing full emotional states. Pretend theories advance that emotions and other

reactions that are raised by fiction are not the same than reactions evoked by real world entities and events. They are just quasi-emotions [20] or full emotions with inhibited behavioural responses [4]. In any case, these quasi-emotions and other responses are the effect of the reader’s or spectator’s activity of imagination. Imagination is a cognitive activity which shares many similarities with believing, but which is essentially different because it runs off-line as other forms of simulation. However, the same relations that hold among beliefs hold among imaginings [4]. Through this particular relationship that holds between believing and imagining it is possible to explain both the similarity and the difference between reactions to fictional and to non-fictional experiences. For this reason, the idea that audiences and readers accomplish off-line simulations has a greater explanatory power than the idea that they hold existence beliefs (equivalent to the idea of illusion of reality). Can this consideration be extended so as to include new media such as virtual reality and enactive interfaces? Considerations about the existence of different reactions towards experiences with simulators (including sickness) and experiences with the real world have been expressed for new media as well [17], [19]. As in the case of traditional fictional media, these divergences as well as similarities must then be explained. A model can be proposed according to which new media users’ responses (motor, perceptual and cognitive) are appropriate both to the contents and to the context of the experience (double appropriateness hypothesis). Thus, users will appropriately try to move and navigate in a virtual environment, but they will comply to the constraints offered by the device (a mouse, a joystick or a treadmill). Exactly in the same way in which horror movies’ spectators will feel fear but will not try to catch the killer’s hand. This model presupposes that new and traditional media users are aware (at the cognitive level) of the fictional nature of the experience and informed (at the perceptual level) of its mediated character. It presupposes as well that users perceive the characteristics of the medium and act in accord with them.

In the debate concerning the paradox of fiction focus is on the audiences’ cognitive awareness of the fictional nature of the experience (no existence beliefs, but activation of imaginative activity with internal simulation), and little consideration is devoted to perceptual information about mediation. However, in addition to the fact of activating specific cognitive attitudes, perceptual media (such as cinema, virtual reality devices and enactive interfaces) present specific perceptual features as well. As simulated things, material devices for simulation are in fact real things that fall under the activity of the perceptual senses. The global perceptual stimulation (termed ambient array by [18]) obtained through a medium is hence different from the global perceptual stimulus obtained through direct contact with reality, and includes the perceptual characteristics of the medium itself. In other words, both the contents and the context (the medium and its characteristics) of the experience contribute to specifying the global perceptual stimulation. Additional perceptual information about the presence and nature of a medium can be provided by the perspectival aspects of the perceptual content. Perspectival contents of perception are those properties of the perceptual content that depend on the position of the perceiver and on her actions [11]. An insect seen through magnifying lenses or represented in a cinematic or static image, can eventually leave the factual content unmodified (the insect), but it will necessarily change the perspectival content of non-mediated seeing, in a way that is typical of the fact of using magnifying, static or dynamic images, with related possibilities of perception and effects of action. Photographs for instance, can represent reality with high fidelity, but cannot provide the same egocentric information provided by ordinary seeing (information about the relation of the depicted objects towards the body of the perceiver) [4]. Contents perceived through movies do not vary with the actions of spectators. And enactive interfaces and virtual reality devices are not capable, at least for the moment, of providing perceptual feedback in real time to all the possible actions users can accomplish [5].

3. The perspectival nature of perception: perceptual awareness of mediation

4. Ethical issues raised by the illusion of reality as illusion of non-mediation Illusions of reality can have complex ethical consequences. A world-famous attempt to

produce an illusion of reality by the mean of a traditional medium is constituted by the broadcast “War of the Worlds” (from E. G. Wells), ideated and played by Orson Welles and diffused in the framework of the series Mercury Theater on the Air in 1938 by CBS. At some extent the attempt worked (part of the audience was fooled): a certain number of American people resulted convinced of being under Martian attack, and reacted with panic. CBS escaped punishment because the fictional nature of the performance was reminded through all the broadcast (even if with large holes, for instance, between minute 12 and minute 40 of the broadcast). However, CBS was formerly invited to avoid the “we interrupt this program" device in fictional contexts. Since then, US TV broadcasts featuring realistic news bulletins post messages to inform audiences of the fictional nature of the spectacle. The risks and side-effects of illusion of reality are hence well known in the domain of traditional documentary media, where ethical guidelines are put in place in order to avoid gullibility and deception, not only fraud [14]. It is consistent with this concern that narrative media as well tend to frame in order to distinguish it from non-fiction [1]. It is a duty for those who propose a different attitude towards new media such as virtual reality or enactive interfaces to demonstrate that the case of new media is different from the case of traditional media and how.

5. Psychological explanations for the illusion of reality as illusion of transportation The notion of illusion of non-mediation does not exhaust the idea of illusion of reality as it is developed in the context of new media. Another aspect of the notion of illusion of reality consists in the possibility of producing a sense of transportation in the virtual world or sense of presence. This rather obscure notion can be naturalized through the reference to a number of proprioceptive illusions that are documented in psychological literature, such as: proprioceptive illusions induced by mirrors [9], proprioceptive illusions induced by prisms [8] and proprioceptive illusions induced by artificial limbs [2]. In all these cases, thank to the appropriate manipulation of visual stimuli, the subjects feel that some of their body parts are displaced in comparison with their real position.

Movement or vection illusions can be induced as well, and even through the proposition of appropriate acoustic stimuli [13]. It is a common characteristics of each of these crossmodal illusions or crossmodal biases that the presence of multiple sensory modalities (touch for the artificial limb illusions and audition plus vision or vibration for vection) and the possibility for the perceiver of accomplishing some movements (especially in the case of mirror and prism illusions) enhances the chances of experiencing the illusion and its strength. This fact would justify in a certain measure the assertion that the multimodal and interactive character of new media produces a qualitative shift in relation to traditional, non-interactive, more limitedly multimodal media. However, evidence produced by experiments with reaching tasks shows that proprioceptive illusions have a negative effect on reaching performances [9]. This fact confirms more general considerations about the disruptive effect violations of coherence and intersensory discrepancies [16]. The aim of producing illusions of transportation should hence be attentively evaluated against the general objectives of the application. Illusion of reality might not be, even when it can be realized, a desirable effect.

6. Conclusions The general indication that can be extracted from the debate of the paradox of fiction, the considerations about the nature of perception, and the ethical rules adopted for traditional media is the following: the production of an illusion of reality as illusion of non-mediation cannot be an objective for new media, both for epistemic and for ethical reasons. On the contrary, in addition to the sub-personal perception of the difference between mediated and non-mediated experiences, the awareness of the fictional nature of mediated experiences should be favoured. However, aspects included in the notion of illusion of reality, such as the illusion of transportation or illusion of presence can find empirical justifications in the existence of local, specific proprioceptive illusions.

References [1] Bordwell, D. (1985). Narration in the Fiction Film. Madison: University of Wisconsin Press. 18

[2] Botvinick, M., & Cohen, J. (1998). Rubber hands “feel” touch that eyes see. Nature Reviews Neuroscience, 391, 756. 21 [3] Carroll, N. (1990). The Philosophy of Horror; or, Paradoxes of the Heart. New York: Routledge 6 7 [4] Currie, G. (1995). Image and Mind: Film, Philosophy, and Cognitive Science. Cambridge: Cambridge University Press. 9 10 15 [5] Dennett, D. C. (1991). Consciousness explained. Boston: Little, Brown & Company. 16 [6] Dinh, H. Q., Walker, N., Song, C., Kobayashi, A., & L.F., H. (1999). Evaluating the importance of multisensory input on memory and the sense of presence in virtual environments. Paper presented at the IEEE Virtual Reality. [7] Durlach, P. J., Fowlkes, J., & Metevier, C. J. (2005). Effect of variations in sensory feedback on performance in a virtual reaching task. Presence: Teleoperators and Virtual Environments, 14(4), 450 462. [8] Hay, J. C., Pick, H. L., Jr., & Ikeda, K. (1963). Visual capture produced by prism spectacles. Psychonomic Science, 2(215-216). 20 [9] Holmes, N. P., Crozier, G., & Spence, C. (2004). When mirrors lie: “Visual capture” of arm position impairs reaching performance. Cognitive, affective & behavioral neuroscience, 4(2), 193-200. 19 23 [10] International Society for Presence Research. (2000). The Concept of Presence: Explication Statement. Retrieved <03/2007> from http://ispr.info/. 1 [11] Noe, A. (2003). Causation and perception. Analysis, 63.2, 93-100. 14 [12] Radford, C. (1975). How Can We Be Moved by the Fate of Anna Karenina? Proceedings of the Aristotelian Society, 49, 67-80. 5 [13] Riecke, B. E., Schulte-Pelkum, J., Caniard, F., & Bulthoff, H. H. (2005). Spatialized auditory cues enhance the visually- induced self-motion illusion (circular vection) in Virtual Reality. Tubingen: Max Plank Institute. 22 [14] RTNDA, (2000). Code of ethics and professional conduct. http://www.rtnda.org/ethics/coe.shtml. 17 [15] Schubert, T., Friedmann, F., & Regenbrecht, H. (1999). Decomposing the sense of presence: Factor analytic insights. Paper presented at the 2nd International Workshop on Presence, University of Essex, Colchester, UK. 2 [16] Stein, B. E., & Meredith, M. E. (1993). The merging of the senses. Cambridge, Mass.: MIT Press. 24 [17] Stoffregen, T. A. (1997). Filming the world. An essay review of Anderson's The Reality of Illusion. Ecological Psychology, 9, 12, 161-167. 12 [18] Stoffregen, T. A., & Bardy, B. G. (2001). On Specification and the Senses. Behavioral and Brain Sciences, 24(1). 13 [19] Stoffregen, T. A., Bardy, B. G., Smart, L. J., & Pagulayan, R. J. (2003). On the nature and evaluation of fidelity in virtual environments. In L. J. Hettinger & M. W. Haas (Eds.), Virtual & Adaptive Environments: Applications, Implications, and Human Performance

Issues (pp. 111-128). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. 11 [20] Walton, K. (1978). Fearing Fictions. Journal of Philosophy, 75(1), 5-27.