The Flote: An Instrument for People with Limited Mobility Amal Dar Aziz
Chris Warren
Department of Computer Science Stanford University Stanford, California, USA
CCRMA Department of Music Stanford University Stanford, California, USA
[email protected]
[email protected] Hayden Bursk
Sean Follmer
CCRMA Department of Music Stanford University Stanford, California, USA
HCI Group Department of Computer Science Stanford University Stanford, California, USA
[email protected]
[email protected]
ABSTRACT The Flote1 is a wind instrument designed for people with limited mobility. Past work in this area has been unable to simultaneously address both the musical expressiveness expected of an instrument and the low-cost requirements for wide adoption. Using only head movement and breath control (calibrated to match the player's abilities), the Flote is both an avenue for creative expression and an enjoyable form of physical therapy. The software is available as a free download and the hardware can be easily built by anyone with minimal electronics familiarity.
(IRED) mounted to the boom arm allows the webcam to track the player's movements. As the player tilts his or her head up and down, the pitch rises and falls. Software written using Max/MSP 5 processes the motion and breath tracking and provides visual feedback to the player.
Categories and Subject Descriptors H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems; H.5.2 [Information Interfaces and Presentation]: User Interfaces; H.5.2 [Information Interfaces and Presentation]: Sound and Music Computing
General Terms Design, Experimentation, Human Factors
Keywords Musical instrument, expressiveness, DIY, HCI, limited mobility,
1. INTRODUCTION The main hardware component of the Flote [fig. 1] is a headset with a boom microphone. Notes are produced when the player blows directly into the microphone. An infrared emitting diode Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 1 The complete documentation and the software for the Flote is available at http://www.theflote.com
Figure 1. An example of the adapted headset, with LEDs. One of our main goals was to create an accessible instrument, in terms of playability and cost. To accomplish this, we needed to design an instrument easily fabricated from commonly available parts. DIY-style instructions and online documentation make the instrument easy to build for those without in-depth experience with electronics.
2. PREVIOUS WORK Projects like Hyperscore [1] have allowed people with disabilities to easily compose and listen to their own music, allowing for creative expression. Though an enjoyable experience, creative expression is limited; the interaction is not similar to the realtime experience of playing an instrument. Work done by the Adaptive Use Group at the Deep Listening Institute [2] used a web cam to track distinct points on the user's face, mapping them directly to a piano keyboard. Moving one's head left to right triggers different notes. Because any head movement immediately triggers notes, the interaction has limited musical potential. The Magic Flute [3] attempts to address this, adding
breath control to increase potential expressiveness. However, The Magic Flute requires expensive proprietary hardware, whereas the Adaptive Use project is free for all to use, simply requiring players to download software and have access to a webcam.
output, but the smaller indicator will track the change in movement. This allows the player to move to the position of the next note while still playing the current note.
Our early attempts used a Nintendo Wiimote for IR tracking, an air pressure sensor for breath control, and a micro-controller connected to the computer to process the information. While this was successful as an expressive instrument, it required quite a bit of specialized skills to build. With the Flote, we attempted to simplify the process of creating an instrument with maximum expressiveness.
3. THE INTERFACE 3.1 Breath Input The Flote works like any traditional wind instrument: the harder the player blows, the louder the note produced. A calibration process, performed each time the software is loaded, adjusts the sensitivity of the breath control to a range that is comfortable for the individual player. The audio input from the microphone is immediately fed into a low-pass filter set at 5 Hz. Since the breath pressure on the microphone is extremely broadband, it is possible to filter out the entire audible range while still retaining the breath pressure information. This eliminates both false triggering by external sound sources and feedback from the instrument itself.
3.2 Motion Tracking Using a webcam and a simple infrared filter for motion tracking made the Flote inexpensive and easy to assemble. The IR filter itself is a sheet of developed, unexposed slide film. This blocks most visible light, leaving just the IRED as the single point to be tracked. In order to ensure that anyone can use the Flote, the tracking system is customized on a per-user basis. Initial calibration of the instrument allows users to define their comfortable range of movement, either on a vertical or horizontal axis. The range of linear movement is then mapped to the scale of playable notes so regardless of the user's range of movement, each individual is capable of playing the same notes.
Figure 2. The interface indicating the tracking is out-of-range. The default key for the Flote is C major (no sharps or flats). When the user wants to change the key, this can be done by opening the menu and clicking on the "set root" button. At this point all notes become available to the player. The first note played becomes the new root. The majority of the players who helped us with user testing did not have musical backgrounds. They initially approached the instrument by moving their heads and gauging the sounds this produced. However, as they continued to play music,they remembered and responded to the written musical notes. Some users even indicated an interest in composing music. The Flote could potentially be used as a stepping stone for people to learn to play sheet music.
3.4 Electronics Besides the headset, the main electronic component of the Flote is a simple circuit consisting of an IRED bulb, a power source (battery or USB), a single resistor and a switch. The IRED is mounted on the mic boom arm of the headset for maximum visibility by the webcam. Optionally, a cluster of three IREDs can be used to ensure consistent tracking.
4. FUTURE WORK 3.3 User Interface The goal of the interface is to provide a simple, straightforward visualizations and feedback regarding the head tracking, playing modes, and calibration (see Figure 2). With prolonged use it may also help players to learn how to read sheet music. The focus of the interface is the staff and the note indicator. The staff itself reflects the current note, responding directly to the player's head movement. A green border around the note name becomes darker as the player blows harder. Visualizations of both the absolute and relative tracking positions based on the IRED tracking interface can be viewed to the right of the music staff. A bright yellow border around the absolute tracking figure immediately indicates when the player has moved out of the webcam's view. The Flote has two performance modes. In the default mode, the player's head movements directly change the pitch of the instrument. In "locked" mode, once a note has been begun, the player is free to move without changing the pitch. The larger note indicator will remain static, reflecting the current pitch
Future versions of the Flote would include the ability to output MIDI data to an external MIDI synthesizer will give the player access to a much broader tonal palette. One possible implementation of this would be to include a VST host. Automatic calibration would also allow the player to be more independent.
5. REFERENCES [1] Hyperscore.com: You Can Compose Music. Retrieved June 22, 2008, from Web site: http://web.media.mit.edu/~mary/hyperscore.html [2] Deep Listening Institute. Adaptive Use Musical Instruments. Retrieved June 22, 2008, from Web site: http://deeplistening.org/site/adaptiveuse [3] The Magic Flute. mybreathmymusic.com. Retrieved June 22, 2008, from Web site: http://mybreathmymusic.googlepages.com/magicflute
