Clash of the Brainwaves: A Cyber Musical Duet Between a Plant and a Human Being Bugra Karabey METU Informatics Institute Inonu Bulvari 06531 Ankara,Turkey [email protected]

Abstract Brain-computer music interfaces have been investigated and utilized in numerous studies. Also the synthesis of music by a plant has been envisioned and performed within a few works including our previous work “Cry of Nature”. In this previous study the bioelectric potential differences on a plant have been used to generate/compose music. Within “Clash of the Brain Waves”, a joint effort between a plant (via its bioelectric potential changes) and a human being (through the brainwaves) will be utilized in generating/composing music in an interactive format as a duet or a jam session between the plant and the mankind. Computer domain and the relevant software will act as the catalyst and a melting pot for this interaction. Keywords: Bio-art, plant-computer interaction, brain computer musical interface, plant-human-computer art

1. Introduction In previous Brain Computer Musical Interface (BCMI) applications, different means were utilized to map the Electroencephalogram (EEG)’s with the music synthesis process. Not only the EEG’s but also electromyogram (EMG)’s (muscle tension), blood pressure measurements, skin conductivity, heart rate, temperature and respiration rates have been embedded into the process. In most of these cases customized sensors have been used. However recently there emerged commercially off the shelf available tools for brainwave analysis that offers the readings of neuronal discharges in the brain, that is alpha and beta brainwaves, electrooculogram components (the positional differential between the front of the retina and the retinal pigmented epithelium which changes relative to the eye orientation) and electromyograms (the neuromuscular signals along with the electrical discharges resulting from the depolarization of the muscle cells). So using such an interface for the human part of the duet and utilizing our already available plant-computer interface from our previous works we will come up with a musical jam session environment between the plant and the human being. In the “Clash of the Brainwaves” the inputs from

the plant and the human being will be utilized within a generative music platform based upon probabilistic music generation platform Noatikl and Max/MSP patches, to come up with novel melodies by this peculiar pair of beings.

2. Related work 2.1 Bioelectric potential activities within plants, their measurement and prior usage for musical activities Bioelectric potential differences within plants are the result of the ion concentration differences between inside and outside plant cells. Analysis of the plant bioelectric potentials for communication with human beings have been proposed and in such studies the effect of temperature, humidity, light or human touch to the plant’s electrical properties have been documented [1], Even the existence of a human being in the vicinity of a plant effects this potential change [2]. Versatile probe designs exist in some studies and using such tools bioelectric potential changes in the range of tens of millivolts have been measured (for a jute plant between minus 18 mV to +40 mV) [3]. Also within [4] extracellular electrodes have been suggested with the note that although the piercing electrodes inflict damage (thus short term potential activity just due to this reason) these effects are temporary and these probes can be used for days and even weeks (which we opted to use for our work). Although there is no nerve structure within plants [5], within vascular plants the xylem and phloem tissues may be considered as transmission mechanisms for ion’s and thus result in potential changes and act like nervelike structures. In the past there exists studies on robotic interfaces driven by a plant [6], and also plant based data has been utilized as a basis for generating music, however in an offline (not realtime) manner by using the pre-recorded data [7]. Some other artists utilized plant bioelectricity for visual interactive installations [8]. Also as was mentioned in [9] Mamoru Fujieda wired up plants using a Plantron interface and further mapped the measurements into MIDI and finally to musical scores. Afterwards these scores (of offline plant bioelectric measurements) were performed by

ensembles and recorded and published as CD’s. John Lifton also wired six large plants using the legacy Altair 8800 computer for the installation “Green Music”. Further this setup was used by Richard Lowenberg for a short film in which human and plant biosignals create music simultaneously but within a disjoint context. We believe our approach presents a novel musical pursuit based upon the realtime interaction between the plant and the environment, and most importantly within a computer/software environment that act as the melting pot and/or the middleware for these two different species. 2.2 Brain computer musical interfaces Human brainwaves were first measured in 1924 by Hans Berger [10]. Although these results were not utilized for nearly a decade, in 1934 a paper backed up by Berger’s results was published in the Brain Journal [11]. First usage of brainwaves within a musical context was by Alvin Lucier in 1965 with his composition “Music for the Solo Performer”. Richard Teitelbaum a member of the Italian electronic music group Musica Elettronica Viva used both EEG (electroencephalogram) and EKG (electrocardiogram) signals to control electronic music synthesis. David Rosenboom was also one of the pioneers of brain musical interfaces and within his Laboratory of Experimental Aesthetics he has devised various methods to utilize brainwaves for musical purposes and has written papers and even books [12], and produced an album called Brainwave Music. Work of Jacques Vidal (though not within a musical context) was also influential in brain computer interface systems [13]. After the 1970’s numerous artists utilized brainwaves for musical purposes; a Montreal group made up of Pierre Droste, Andrew Culver and Charles de Mestral, Atau Tanaka who used the Biomuse that integrates the muscle movements (EMG) and eye movement (EOG) to the process Brouse and Miranda with their numerous works (BCMI Piano and others) [14] [15] and also Janez Janša who developed Brainscore and led the project Brainloop. Even an underwater brain concert has been performed during the ICMC 2007 using an electroencephalophone.

of the ubiquity of these interfaces in the medical field, a BCI (brain computer interface) that cost a huge amount in the past, nowadays can be accessed via commercially off the shelf available alternatives at a fraction of this price. Such solutions even exist for the gaming community and although their electrical precision may not be up to par with the medical/professional alternatives, we believe they are adequate for our purpose. Also within our work “Cry of Nature” we have devised a platform that enables the usage of plant bioelectric signals in a way that the plant acts as a “plant theremin” or a “bio theremin”. So our intention within the “Clash of the Brainwaves” is to offer a platform in the computer/software domain that enables joint effort of these two previously isolated environments. Our platform consists of the system architecture outlined in Figure 1. Plant biosignals are measured using extracellular piercing electrodes. We have observed that after the insertion at least an hour is required for the signal level to stabilize. These signals are further amplified with a custom designed/built analog amplifier with very high impedance and further converted to MIDI via the Eobody unit from Eowave. In the mankind domain the alpha and beta brainwaves and in parallel to these the electrooculogram (eye movements) and the electromyogram (face muscle movements) are measured using the Neuro Impulse Actuator platform. This is a cost effective solution that also offers ease of use and as was mentioned before as our aim is not medical analysis the precision of the measurements were adequate for our purpose. This product is also beginning to be used by the gaming community.

3. Clash of the brainwaves Within our work Clash of the Brainwaves, we have envisioned to come up with a platform that enables a plant (or plants) to combine their musical efforts with a human being and namely his brainwaves. Our platform will act as a melting pot and will catalyse the plant initiated bioelectrical signals with the mankind generated brainwaves. 3.1 Methodology As was mentioned above the brain computer interfacing field has reached a certain maturity mostly due to the needs in the medical arena. Numerous artists have utilized these means within their artistic work. As a direct consequence

Figure 1. Block schematic for the “Clash of the Brainwaves”

3.2 Musical outcome and findings As we have successfully utilized our plant-computer interface with our work “Cry of Nature”, our observations for this initial work were also valid here. Namely the reaction of the plant to environmental factors like heat,

light, humidity was of a slowly moving nature. So these changes most successfully effect the “drone like” qualities within the musical composition. However the reactions for a human in the vicinity or the touch by a human were very fast responses and they were useful for sound parameters that are responsive to such abrupt changes. As an example the modulation index of an FM synthesizer or triggering of a percussive sample. In the brain interfacing side the NIA (Neural Impulse Actuator) works by reading neuronal discharges in the brain, that is alpha and beta brainwaves and also electrooculogram components (the positional differential between the front of the retina and the retinal pigmented epithelium which changes relative to the eye orientation) and electromyograms (the neuro-muscular signals along with the electrical discharges resulting from the depolarization of the muscle cells). We have observed that the electrooculogram and electromyogram signals were really easy to use and can be used to trigger samples or to change the parameters of a synthesizer and/or filter easily. However the alpha and beta brainwaves necessitate a long training process to hone one’s skills in manipulating these. As an example the alpha waves (8-14 Hz) were more active when the eyes are closed and the brain was in a relaxed mode, whereas the beta waves (higher frequencies) necessitate alertness. We suggest to use these parameters in triggering samples (or switches within the DAW) as their analog or continuous level usage is really hard to implement. As a result we have utilized the plant signals to modify the parameters of Max/Msp synthesis patches (as theses signals are of a continuous nature) and also the tempo of the piece in Noatikl. Whereas the brainwaves and EOG and EMG signals were utilized to trigger percussive samples and also switches (filters etc) within the DAW. They can also be used to make changes within the Noatikl generative music platform for scale rule, harmony rule or rhythm rule selections (as they are suitable for triggering). Our experience resulted in a jam like atmosphere with the composition moving in a collaborative manner due to the human players mind activity and mostly to his physical interaction with the plant. Also it is worth noting that the electrical grounding of the human player and the electromagnetic noise level within the environment (screens, mobile phones etc) were critical success factors.

Figure 2. Plant based player (Cry of nature) setup

Figure 1 shows the setup for the plant based player (our former work Cry of Nature) and in Figure 2 the human player connected with the EEG, EOG, EMG interface jams with the plant within the Clash of the Brainwaves setup.

Figure 3. Man-Machine-Botany Interfaces

4. Conclusion and future work Within this study we have devised a system that enables a plant and a human being to create a real time composition by utilizing the brainwaves of the human and the bioelectric signals of the plant. As was discussed above using our setup the plant player was more successful in tweaking the knobs and parameters of filters/synthesizers whereas the human player was better at triggering samples or making harmony/rhythm choices within the generative music platform. Training process to manipulate the alpha and beta waves for the human player was cumbersome wheras the electrooculogram and electromyogram signals were easier to manipulate. We believe that “Clash of the Brainwaves” enabled the mutual cooperation of two different species within the context/domain of music. Although it is definitely debatable if the plant has any clue of the process or it has any positive/negative affiliations with the process it was still interesting to listen to the outcome of this collaboration that most of the time offered unpredictable musical results. We believe our work presents a novel approach to the plant-computer-human (or man-machine-botany) interfaces and although there exists studies on plant computer interaction and brain computer interaction, our work offers a pioneering twist both in the form of a trio made up of man-machine-plant and also within the context of a joint musical effort between plant and mankind. Within this trio the computer/software domain act as the middleware and/or the melting pot that enables the cooperation of these two distinct species. As a future work we intend to connect more than one plant and potentially one man to come up with a “Band of Brainwaves” or “Brainwaves Ensemble” in which each plant or human player is responsible for different musical parameters (analog to playing different instruments) and come up with a more colorful jam environment.

References [1]

T. Shimbo, T. Oyabu, S. Hirobayashi, and K. Takenaka, “Statistical Analysis of Plant Bioelectric Potential for Communication with Humans,” Transactions of IEE of Japan, vol. 124, no. 12, pp. 470-475, 2004.

[2]

[3]

[4] [5] [6] [7] [8] [9]

[10] [11] [12] [13]

[14]

[15]

S. Hirobayashi, Y. Tamura, T. Yamabuchi, and T. Yoshizawa, “Verification of Individual Identification Method Using Bioelectric Potential of Plant during Human Walking,” Jpn. J. Appl. Phys., vol. 46, pp. 1768-1773, 2007. R. Hamilton, P. Datta and P. Palit, “Relationship Between Environmental Factors and Diurnal Variation of Bioelectric Potentials of an Intact Jute Plant,” J. Current Sci., vol. 87, no. 5, pp. 680-683, 2004. E. Davies, "Electrical signals in plants: Facts and hypotheses," in Plant Electrophysiology, A. G. Volkov, Ed. Heidelberg: Springer-Verlag, 2006, pp. 407-422. R. Stahlberg, "Historical introduction to plant electrophysiology," in Plant Electrophysiology, A. G. Volkov, Ed. Heidelberg: Springer-Verlag, 2006, pp. 3-14. (2009, Mar.). Guto Nobrega: Artworks [Online]. Available: http://www.gutonobrega.co.uk (2009, Mar.). John Gibson: Slowlife Installation [Online]. Available: http://www.john-gibson.com C. Sommerer and L.Mignonneau, Interactive Art Research Vienna: Springer-Verlag, 2009. D. Toop, "The generation game: experimental music and digital culture," in Audio Culture: Readings in Modern Music, A. C. Cox, D. Warner Ed. New York: Continuum, 2004, pp. 239-247. H. Berger, “Über Das Elekenkephalogramm Des Menschen,” Archiv für Psychiatrie und Nervenkrankheiten, pp. 527-570. 1929. E. D. Adrian and B.H.C. Matthews, “The Berger Rhythm: Potential changes from the occipital lobes in man,” Brain., part 4, vol. 57, pp. 355-383, 1934. D. Rosenboom, “Method of producing sounds or light flashes with alpha brain waves for artistic purposes,” Leonardo, vol. 5, no. 1, pp. 141-145, 1972. J.J. Vidal, “Toward direct brain computer communication,” in Annual Review of Biophysics and Bioengineering, L.J. Mullins Ed. Palo Alto:Annual Reviews , 1973, pp.157-180. E. Miranda and A.Brouse, “Toward Direct Brain Computer Musical Interfaces,” in Proc. of the Conf. on New Instruments for Musical Expression (NIME), 2005, pp. 216219. E. Miranda and A.Brouse, “Interfacing the brain directly with musical systems,” Leonardo, vol. 38, no. 4, pp. 331336, 2005.