SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Design and Evaluation at a crossroad between Art, Science, and Engineering: the case of Sonic Interaction Design
Davide Rocchesso
Università IUAV di Venezia
[email protected] Davide Rocchesso 2011
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
COST Action IC0601 !
Design the sonic appearance of products;
!
Achieve soundscapes that are informative, pleasant, and non obtrusive;
!
Exploit the technologies by designing relevant sonic interactions;
!
Link psychology, computer science, acoustics, design, art and music;
!
Dialogue with communication, sociology, and cultural studies.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Research Directions !
Quality assessment of sound in interaction;
!
Creative exploration of contexts and opportunities;
!
Interactive sonification;
!
Cognitive and perceptual evaluation;
!
Synergies between sound arts and sonification;
!
Technologies.
Scientists, designers, and artists conceive, test and share sonic interactions. The disciplinary boundaries and assumptions are continuously re-discussed.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Articulation of Interaction Esthetics
[Löwgren, 2009] !
Pliability (information appears malleable)
!
Rhythm (the beat of interaction)
!
Dramaturgical structure (from exposition to resolution)
!
Fluency (the graceful dance among media streams)
Abstracting
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Tuning
Abstracting
Design Processes and Pedagogics
Research through Design !
Basic design methods
!
Sensitizing to sonic interactions
!
Sketching (and faking) sonic interactions
!
Prototyping with interactive sound
!
Tweaking and tuning
!
Abstracting for understanding
SID
Basic
Sensitizing
Sketching
Prototyping
Basic Design !
Exercises with well-defined objectives and constraints (Moholy-Nagy, Albers, etc.)
!
Demonstrations and intersubjectivity
!
Ex.: Antiprimadonna [Maldonado and Anceschi at Ulm, 1962]
!
In design for multisensory continuous interaction, the controllable dimensions are many
!
Sensors and actuators considered together in the action–perception loop
!
Look for fundamental interaction gestalts [Svanaes, 2000]: abstraction of actual interactions.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Experimental Phenomenology
SID
!
[P. Bozzi, 1978 - L’interosservazione come metodo per la fenomenologia sperimentale.]
!
“Against” psychophysics: when the experiments start from or lead to discoveries, they make things complicated for theory, and theory lives of complications;
!
Any discovery must be supported by inter-observation;
!
Investigating the “pillars” of experience: out of the lab, in noisy everyday environments;
!
Negotiation between observers removes bias.
Basic
Sensitizing
Sketching
Prototyping
Tuning
Exercises in Basic SID
Multi-stage yet continuous coupling
Counter-inter-actions Cyclic interactions Acoustic Antiprimadonna A notebook
pliability, rhythm, fluency
Abstracting
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
A theme for basic SID in continuous interaction Temporal dynamics of connections [Kloeckl 2008] (*%#34"/
%&''$%!$(
12//.*
!"#$ $%&'()%**+(, -*./ (&567 ,*%"0 !"#
(")%&''$%!$(
torque measurement tool 1 x beep = torque too low, 2 x beeps = OK, 3 x beeps = torque too high
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Basic SID exercise: closing the moka [Rocchesso, Polotti, and Delle Monache, IJD 2009] Theme Continuous feedback for mechanical connections. Objective Design the feedback for a screw connection, such as found in the moka, in such a way that the right degree of tightness in coupling can be easily reached. Constraints The feedback should be continuous, non-symbolic, immediate to catch (or pre-attentional), and yet divisible into three clear stages. ! !
Symbolic signals (beeps, flashes) → disembodied feedback Continuous auditory feedback → engagement
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
The Schelleng diagram:
Relative force
Solving the exercise: The violin as inspiration Max
imum bo
1
w forc
e
RAUCOUS Brilliant
0.1 Min
Sul ponticello
imu
m
0.01
bow
NORMAL for
ce
Sul tasto
HIGHER MODES
0.001 0.01
0.02
0.04
0.06
0.1
0.2
Relative position of bow, ß
SID
!
By exploring the space of control parameters one can gradually step into different oscillatory regimes
!
Listeners catch the quality of physical actions immediately
!
Three distinct areas of effort (too low, ok, too high) are readily associated with the three tone-quality areas.
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Exercise solved: Sonifying the moka
!
Apply a force sensor between B and C, and use force signal to control an elasto-plastic friction model [Avanzini et al., 2005]
!
Several parameters of the model are covaried
!
Sparse use of short friction bursts
!
MaxMSP patch
Timbre: gradually from “glassy” (loose) to “rubbing” (tight). When coupling is “too tight”, it sounds like a squeaking hinge.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Solution evaluated: Experiencing the moka Shared impressions: !
Sound enhances the screw connection and makes the task more engaging
!
The experience of acting directly on the moka is different from simply listening
!
The auditory feedback blends with kinesthetic perception (pseudohaptics)
!
Users tend to push the boundaries of meaningful interaction. At the end of session, the force sensor was broken.
!
When an acoustic actuator was directly attached to the moka the sound became really part of the object (schizophonia vanished).
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
How to become sensitive to sonic interactions !
In the Bauhaus, the grundkurs taught by J. Itten at the Bauhaus was a sort of sensory training
!
A soundwalk [Schafer] is an empirical method to identify keynotes, signals, and soundmarks of a soundscape
!
Exploration of objects (tactile, proprioceptive, and auditory, extension of [Lederman-Klatzky, 1990]).
Analysis and imitation of sonic interactions (gesture + sound) !
A sound drama is a collection of non-speech sounds combined in a soundtrack. It allows to isolate “abstracted narrative themes and attributes of artefacts and interactions”, or metatopics [Hug, 2010]
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Sonic Interaction Design as Theater !
The relation between sound and action can be distorted and even subverted in theatrical performance [Pauletto et al 2009]
!
Anny in the Kitchen becomes Mark in the Bathroom: Stephen Barrass sound design
rhythm, drama, fluency
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Vocal Sketching
Vocal sketching involves the use of the voice, along with the body, to demonstrate the relationship between action and sonic feedback. It is !
Intuitive, as in children playing with toys
!
Available to anybody. Yet virtuosism is possible
!
Communicative, as it facilitates discussion
!
Group-friendly, transcending a solo voice
!
Enactive, as when combined with bodystorming
It can also be embarrassing, evanescent, limited.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
A Vocal Sketching Workshop Holon, Israel - november 2009 !
How do we sketch when designing sonic interactions?
!
Can the voice for sound designers be like hand+pencil for the graphic designer?
!
Professional designers, engineers, and conductor;
!
Exercises:
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Video Sketching In Human-Computer Interaction, video prototyping and video brainstorming are commonly used (e.g., Spence’s bifocal display as filmed cardboard model, 1980). Sonic interactions can be faked with vocal or synthetic Sonic Overlays, which are: Foley artistry ! Sound-centered !
Diverse
!
Communicative
but also !
Non-enactive
!
Non-realtime
Abstracting
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
A Training School on Product Sound Design Design Factory, Aalto University - august 2010
Exercise
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Gamelunch: a sonic dining table !
Sonic augmentation of pouring, cutting, mixing, etc.
!
Physically-based sound models and continuous interaction
!
Contradictory feedback by manipulation of transformational or structural invariants
!
Auditory manipulation of cognitive expectation and emotional impact
Documentation
Video
Abstracting
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
A Gamelunch instrument: the sangria bowl
Action: mixing, pouring. !
Grasping the ladle;
!
Rotating and lifting the spoons;
!
Twisting the wrist.
Sound: Hard crumpling, bumping, and stretching Model: Crumpling Sonic percept: Sense of effort, change in nature of the pieces.
SID
Basic
Sensitizing
Sketching
Prototyping
How do we access parameters in the world? (1) Perceptual spaces to arrange model parameters Material (rubber, wood, glass, steel) in a decay-time / pitch space. Saturation ≡ inter-subject agreement.
Manipulation of points in space, direct production of semantic distances (e.g., Fernström’s Sonic Browser)
Tuning
Abstracting
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
How do we access parameters in the world? (2) Example [Ottaviani and Rocchesso, 2004]: Object size is !
perceptually related to pitch,
!
physically related to frequency (and intensity)
!
geometrically measured by length, area, or volume.
Given an object shape, →Psychophysics is used to scale the size perceptually.
What can we do across shapes? When are cubic and spherical cavities perceived to be the same size? → Psychophysics looks for the PSE (Point of Subj. Equiv.).
sphere (solid) vs. cube freq. resp.
SID
Basic
Sensitizing
subjective size: sphere vs. 13 cubes, fixed and randomized order
Sketching
Prototyping
Tuning
Abstracting
Ad-hoc psychophysics Example: auditory saltation [Rocchesso and Delle Monache, ongoing research] !
Inspiring Problem: Non-visual highlight of a hot spot:
wireless charging by
!
Research: Spatio-temporal unfolding of sonic sequences:
manual reproduction of sonic gestures (x-coordinate in time)
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
Abstract interactive objects
!
Evaluation of a sonically augmented interactive object requires to measure how sound affects user behavior;
!
The quality of the design becomes indexed by the users’ performance, and by the speed of their learning;
!
Designers do not have to forget that sounds create strong aesthetical and emotional reactions in users;
!
Experimental, abstract objects designed to study one specific interaction gestalt.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Embodied Interaction with Rolling the Ballancer [Rath and Rocchesso, IEEE MM 2003]
!
Simple control metaphor: Balancing a ball on a tiltable track
!
Familiar principle from everyday experience, no explanation/exploration, training only for performance
!
Parallel mechanical realization straightforward: complete testability
Abstracting
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
The Spinotron [Lemaitre et al, IJHCS 2009] !
Task: pumping to reach and keep a precise and constant pace;
!
Indep. Var.: auditory (virtual spinning top) or visual feedback;
!
Observation: only auditory feedback affords performance improvement across trials; Users are not aware of this, and even find the sound irritating.
SID
Basic
Sensitizing
Sketching
Prototyping
Tuning
Abstracting
The Flops [Lemaitre et al, SMC 2009] !
Study the emotions felt by users manipulating a computationally and acoustically augmented artifact;
!
Task: exactly pour a predetermined number of items;
!
Indep. Var.: sound design and dynamics of interaction; Modulate valence and dominance dimensions of users’ feeling.
!
Observation: quality of sounds systematically influences valence.
