An Embodied Approach to Digital Tangible Musical Interfaces Alexander Müller TU Berlin – Deutsche Telekom Laboratories – Design Research Lab Ernst-Reuter-Platz 7, 10587 Berlin, Germany [email protected] ABSTRACT

This work aims to contribute to the theory and practice of the design and performance of digital musical interfaces. It presents an attempt to combine different phenomenological, sociological and design research approaches to point at issues in the tangible interface design of digital musical instruments. Considerations towards gesture acquisition and skill acquisition are emphasized and the integration of methods of practice-theory transfers for performance and musical interaction systems are illustrated. Author Keywords

Embodied Interaction, Digital Musical Interfaces, Design Methods ACM Classification Keywords

H5.2 [Information interfaces and presentation]: User Interfaces – Input devices and strategies INTRODUCTION

Today’s mobility and availability of computers in conjunction with the progress of digital sound synthesis enables the development and design of more unprecedented performances and musical expressions. Furthermore a greater number of interested users are granted access to play music. In the design of musical interfaces as a part of HCI research, developers and designers have explored interfaces for musical expression for decades [10]. However, computer scientists and design researchers still agonise over a scientific gap: How can meaningful and embodied digital interfaces for tangible interactions can be designed, if in computational systems due to their symbolic nature input and output are decoupled, whereas in physical and analogue artefacts they are not [9,3]?

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

The main issue of this research work is to disclose new perspectives on the design process of digital musical instruments that aim to enhance the richness, subtlety and coordination of embodied musical expression. My work bases on previous research, which has been done mainly from a phenomenological approach towards the embodiment of digital instruments, e.g. by Magnusson [9], Armstrong [1], Essl and O’Modhrain [5]. Beyond that I aim to extend existing methods in order to reveal knowledge that helps find possible answers to the aforementioned question. From that starting point it is important to conceive the notion of Embodiment by taking into account gesture and skill acquisition: THE EMBODIMENT OF INTERFACES Gesture acquisition

When speaking of musical instruments controlled by humans Claude Cadoz defined the term instrumental gesture [2]. His theory defines three functions concerning the concept of physical determinants of gesture space for musical instruments. His criteria for gestures to be instrumental gestures are: •

Ergotic: Gestures include actions on the physical system, thus they modify and transform the environment.

•

Epistemic: They cause the perception of the environment and context

•

Semiotic: They transfer information environment (and audience).

to

the

Despite his tight classification of instrumental gestures this definition is helpful for the analysis of and reflection on new designed prototypes. It provides an initial frame for further investigation into my research. Skill Acquisition

The theory of Enactivism by Varela, Thompson and Rosch that is highly influenced by phenomenological thinking, does not distinguish between perception and action. Such close coupling – where action demands perception and

perception effects and informs action – is derived from “[…] a world and a mind on the basis of a history of the variety of actions that a being in the world performs” [16]. The phenomenon of bodily incorporation of instruments (and tools) through practicing is described by the transformation from vorhanden (present-at-hand) to zuhanden (ready-at-hand) [7]. As Dreyfus describes, Merleau-Ponty’s Phenomenology of Perception forms a more evolved view on gradual acquisition of practical skills [4]. The improvement thereof occurs less through interpretation and understanding of symbolic structures than embodied action.

PROJECT-GROUNDED RESEARCH AS METHODOLOGICAL FRAMEWORK

Drawing on current available theoretical knowledge in the field of HCI research, design research, phenomenological philosophy and to some extend social sciences this work is positioned within the methodological framework of Research through Design. Findeli calls this epistemological approach Project-grounded Research [6] in which concrete design questions and possible answers are embedded into the broader context of a research question. In my work one essential step towards answering these questions is using methods of hands-on prototyping and participatory design.

DESIGN ANSWERS: DIGITAL MUSICAL CONTROLLERS

To support the theoretical base of my work I am investigating tangible musical instruments by the means of physical prototypes. The following projects focus on haptic feedback for orientation in virtual space as well as enhancing the performer’s skill acquisition through stimulating haptic perception. Vibetone

With the musical input device Vibetone (Fig. 1), tactile feedback is investigated. To enhance emotional, experimental and dynamic expression with a performing nature, the instrument allows users to create music not only via haptic gestures with extremities but also through semiotic gestures, integrating the motion of the user’s body. We chose the tactile channel due to its non-intrusive character and because it is a visceral channel for feedback. Through the utilization of sensible vibrotactile information signals the musician’s movements were guided through the performance [11].

(Co-) Designing Prototypes

One of the expected outcomes of prototyping is the identification of new interaction principles. Through prototyping “ideal” and “visionary” artefacts come into being. They contain information about functional, haptic and aesthetic characteristics that inform the design of tangible user interfaces and gestural mapping. Thus, knowledge is emerging through the human activity of “making”, commonly known as tacit knowledge [13]. By analyzing performances with these prototypes and by providing connections from aspects of theory to a specific design question unexpected outcomes can be discovered. As a means for analysing the design process of instruments depending on the intended performance and social context I propose the use of Activity Theory on the basis of Kuutti’s work within HCI [8]. Participatory design methods help gain a diverse perspective towards social context. Sanders states that there is a noticeable shift from user-centered design to Participatory Design and co-creation in the field of HCI [14]. Moreover, Participatory Design is a fruitful extension of traditional qualitative research tools. Musicians, experts as well as amateurs, shall be integrated into a collaborative research and design process. Within this setting prototypes play the role of boundary objects that support the dialogue between researching and performing actors. On the one hand they allow for an inspiring interpretation, on the other hand they comprise content that depicts a “global identity” [15].

Figure 1. The Vibetone prototype in use

Reflective Haptics: The GUI///RO

In the Reflective Haptics project we proposed a controller (Fig. 2) that aimed to support rhythmical musical performance. The system builds on resistive force feedback, which is achieved through a brake-augmented ball pen stylus on a sticky touch-sensitive surface. By the design we intended to provide an easily accessible way for the performer’s sound control, considering the gradual specification of movements [12].

Embodiment, Honi Haber and Gail Weiss (eds.), Routledge, New York and London, 2002. 5. Essl, G. and O'Modhrain, S., An enactive approach to the design of new tangible musical instruments, in Organised Sound, vol. 11, 2006, 285 - 296. 6. Findeli, A., Searching for Design Research Questions, Keynote Presentation, Question&Hypotheses Conference, Berlin, 2008. 7. Heidegger, M., Sein und Zeit, 1927. 8. Kuutti, K., Activity Theory as a potential framework for human- computer interaction research, Context and Consciousness: Activity Theory and Human Computer Interaction, B. Nardi (ed.), MIT Press, 1995, pp. 17-44. Figure 2. The GUI///RO prototype OUTLOOK

This is a rough outline of my research work. The prototypes that were developed during the last year allow for a rich embodied interaction. Musical performances can successfully be supported through the utilization of sensible haptic signals, rather than solely visual or auditive cues. Besides the prototyped objects of investigation this work suggests a multifaceted theoretical approach for the design process of digital musical interfaces. ACKNOWLEDGMENTS

The author would like to thank Gesche Joost, Georg Essl and Fabian Hemmert for their support and numerous discussions and Jochen Fuchs for his technical assistance.

REFERENCES

1. Armstrong, N, An Enactive Approach to Digital Musical Instrument Design, PhD Thesis, Princeton University, 2006.

9. Magnusson, T., Of Epistemic Tools: musical instruments as cognitive extensions, in Organised Sound, vol. 14, 2009, pp. 168-176. 10. Miranda, E.R. and Wanderly, M.M., New Digital Musical Instruments: Beyond the Keyboard, A-R Editions, Middleton, Wisconsin, 2006. 11. Müller, A. and Essl, G., Utilizing Tactile Feedback to Guide Movements Between Sounds, in Proceedings of the Ninth International Conference on New Interfaces for Musical Expression, Pittsburgh, 2009, pp. 33-34. 12. Müller, A., Hemmert, F., Wintergerst, G. and Jagodzinski, R., Reflective Haptics: Resistive Force Feedback for Musical Performances with StylusControlled Instruments, in Proceedings of the tenth Conference on New Interfaces for Musical Expression, Sydney, 2010, pp. 477-478. 13. Polanyi, M., The Tacit Dimension, Doubleday & Company, Garden City, NY, 1966. 14. Sanders, E., From User-Centered to Participatory Design Approaches, in Design and the Social Sciences, Frascara, J. (ed.), 2002.

2. Cadoz, C. and Wanderley, M.M., Gesture - Music, Trends in Gestural Control of Music, M.M. Wanderley and M. Battier, 2000, 71-94.

15. Star, S.L. and Griesemer, J.R., Institutional Ecology, Translations’ and Boundary Objects: Amatuers and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39, in Social Studies of Science, 19, No 4, 1989, 187-420.

3. Dourish, P., Where the Action is: The Foundations of Embodied Interaction, MIT Press, 2001.

16. Varela, F., Thompson, E. and Rosch, E., The Embodied Mind, MIT Press, 1991.

4. Dreyfus, H., The Current Relevance of Merleau-Ponty‘s Phenomenology of Embodiment, in Perspectives on