Slavko Zagorac May 2016

Towards An Integrated Environment for Networked Music Composition and Performance Introduction The purpose of this paper is to outline the current status of my PhD research which focuses on the music composition and performance for mixed networked ensembles. The paper contains a brief review of the existing real-time notation software, the system requirements for an ideal composition and performance system I hope to build and utilise in the future and my current composition workflows including the software development undertaken so far. Finally, some of the conclusions and plans for the next steps will be laid out.

Existing software A number of software packages able to dynamically render music scores distributed over the network already exist. The most advanced in this respect are: InScore [8], MaxScore [2] and related Quintet.net [12] and Bach composer [6]. Despite their high level similarities, there are some major differences in the implementation and functionality of these packages as explained further below. System Dependencies Both Bach composer and MaxScore are implemented on top of the Max MSP platform and rely on its scheduling, GUI rendering and networking implementation. This dependency limits their deployment only to the devices and operating systems supported by Max MSP which currently excludes tablets and other mobile devices. Additionally, MaxScore requires Java libraries which are not available by default on more recent Mac OS X systems and it also requires a purchase of the commercial license for the underlying JMSL library (Java Music Specification Language). InScore is an open-source library written in C++ and can be deployed on multiple platforms which include standard Mac and Window computers, portable iOS (iPad) and Android devices as well as computers running the Linux operating system. InScore utilises standard Qt user interface library and GuidoLib library for symbolic score rendering. Score Creation Out of all mentioned packages InScore is the only one which does not provide any built-in symbolic notation creation and editing functionality. It was designed primarily as a lightweight network 1

Slavko Zagorac: Annual Review May 2016

client which can render multiple data formats. It supports some interactive features but any complex notation authoring needs to be done in a separate software. InScore supports the rendering of GUIDO symbolic notation format as well as the common graphical and scalable vector graphics (SVG) formats. MaxScore and Bach composer deployment packages contain custom symbolic notation editors by default. Max Score is more advanced in this respect as it offers a larger library of note heads, symbols and the ability to create custom graphics with Picster tool. In addition to basic symbolic notation, Bach composer allows for a definition of generic data maps which can be allocated to a note event and can contain any data (control, instructions …). This data can contain instructions for extended notation is rendered on the notation user interface as text. Despite these features it is fairly hard to generate more complex extended contemporary notation in either of the packages. Internally, Bach uses hierarchical Lisp-like linked lists (llll) to store score data while MaxScore uses proprietary object oriented JMSL data definitions. All three packages support rendering of MusicXML files, although with varying degrees of accuracy.Quintet.net allows for score creation and synchronisation of up to five participants (instruments) and mapping between graphical space and time in Timeline GUI. Score Distribution All the tools mentioned above allow for the distribution of score data over a network via Open Sound Control (OSC) protocol. Bach and MaxScore rely on Max libraries for network functionality while InScore supports OSC natively as its entire API is based on OSC protocol. The common problem for all tools is the efficiency of score data serialisation and transport. Large score data structures need time to be processed and transmitted resulting in increased latencies which in turn make precise time synchronisation over the network very difficult to achieve. For example, sending Bach score underlying linked lists via Max object udpsend causes intermittent errors, most likely caused by data structures larger than maximum allowed UDP packet size (64 kB). Unsurprisingly, most of MaxScore pieces written so far do not require tight synchronisation. Quintet.net, which uses MaxScore, provides the most sophisticated network distribution mechanism. It uses both UDP and TCP transport protocols for increased reliability, but is currently limited to only five clients. Score Display All mentioned packages support symbolic and graphic notation to a certain degree, but as a general rule, more ambitions contemporary notation is challenging in all cases. The most flexible tool in this respect is InScore which supports multiple notation formats including symbolic notation via GUIDO library, common graphic file formats (png, jpg ...) and vector graphics (SVG). Current score position can be visualised in all tools with different types of indicators on client interfaces. Control, placing and synchronisation of multiple parts is available by default only in Quntet.net. Other tools need additional software to author, distribute and synchronise multiple parts.

2

Slavko Zagorac: Annual Review May 2016

The next release of OpenMusic will provide the ability to arrange and synchronise multiple parts in updated Maquette interface. This has the potential to become the main driving tool for real-time score execution over a network, however, it remains to be seen whether this potential will be realised in the next version.

System Requirements Based on the analysis of currently available software, I drew a set of functional and non-functional requirements for the future, ideal system which would allow for the composition and performance of contemporary, complex extended technique scores written for heterogenous instrumentation including acoustic and digital sound sources, distributed over a computer network. These functional requirements are as follows: •

The system should transparently integrate processes of music composition and performance by sharing a common data model and using as few user interfaces as possible.

•

The composition data and control events should be distributed in digital format to all participants over the computer network, either on-demand or at pre-scheduled times.

•

The data model should be common for all compositional elements, including acoustic and digital parts, multi-media content, audience portable devices, etc. System processes should be as similar as possible for any combination of heterogenous parts.

•

Real-time composition data generation and control should be built-in, allowing for predetermined time schedules and ad-hoc events. The system should support pre-composed and improvised composition elements.

•

The system should be event driven. Composition control or data events can be trigged by any participant at any time during a performance. Event flow can be unidirectional, bidirectional or multidirectional.

•

It should be possible to execute both linear and non-linear time parts, allowing for bottomup and top-down hierarchical composition structuring.

•

Authoring and visualisation of complex contemporary composition notation, including any future innovations in extended playing techniques and well as the traditional symbolic music notation should be supported.

Including flowing non-functional requirements:

3

•

Overall end-to-end system latency should be humanly imperceptible (sub 5 milliseconds). This includes latency of the server processing, network transport, GUI rendering and sound production.

•

The composition data model should be common for all elements and transparently shared between processes of composition and performance. The data format needs to be portable and suitable for network transport including efficient serialisation and deserialisation.

Slavko Zagorac: Annual Review May 2016

•

It should be possible to integrate other systems via common API, data format (MusicXML) or a protocol (OSC).

System Design Based on the requirements above I created a diagram of the future system architecture (Figure 1) which outlines the main system components. The next step was to decide whether any of the existing software tools could be used in place of the outlined system components and which existing apis and protocols can be reused. From the analysis and conversations with other developers I concluded that the most flexible standardised data format, which can be used as a starting point for music notation definition and rendering, is Scalable Vector Graphics (SVG). Although it lacks musical context which, for example, MusicXML provides, its advantages as a standardised, scalable, extensible and scriptable format supported by all major web browsers outweigh its disadvantages. While fairly verbose, it is suitable for compression and slicing to allow for efficient transport. In an ideal scenario the only front end required for notation editing/viewing would be a standard Internet Browser with additional SVG authoring and rendering javascript library. Creation of such a library would require significant amount of work, so to start with, I had a look at other widely used SVG authoring tools such as Photoshop Illustrator and open-source Inkscape. A number of composers I spoke to already use Illustrator to create graphical style scores so I decided to use it as a starting point for notation generation. InScore supports SVG format, it has built-in network communication protocol, it allows for realtime notation distribution and can be run on the wide range of platforms. For these reasons, I decided to use it as the main notation visualisation software for acoustic instruments. Over the last few years OSC network protocol became the standard for musical data distribution over the network. Although it is based on UDP, which by definition is not reliable and can result in data loss, I decided to use it as the starting point. It is also natively supported by InScore. Future system versions should incorporate a network layer with a reliable multicast implementation. I initially planned to use new OpenMusic version as the Controller component but, unfortunately, it was not ready by the time of writing. The other alternative was to use Conductor component of Quintet.net system but I haven't managed to make it work with other system components. Eventually, I decided to develop a new Controller component in Java which would provide the required network communications and scheduling.

4

Slavko Zagorac: Annual Review May 2016

Figure 1: System components of an integrated music composition and performance environment

5

Slavko Zagorac: Annual Review May 2016

Current Implementation In order to provide initial composition material, I started working on a new piece called Ukodus for Clarinet, Cello and two digital instruments as counterparts to acoustic sound sources (Electro Clarinet and Electro Cello). The piece was inspired by multi-directional symmetrical contours of the Sudoku puzzle. After few iterations of experimentation with the selected technologies it became apparent that a significant amount of development is required to create a good enough environment for reasonably efficient composition and performance workflows. This realisation resulted in the development of a number of components as explained below. Score Creation Adobe Illustrator was selected as the notation tool for the reasons explained above. In order to provide a musical context, a generic hierarchical layer structure was created (Figure 2) based on ideas from SVG Score Project (Gottfried [13]).

Figure 2: Illustrator Score Hierarchical Layer Structure Next, I developed new javascript Illustrator plugin Score Tools (Figure 3) which enables more efficient composition workflows. Current plugin functionality allows for automated creation, persistence and editing of layer structures, score pages, bars and full score and parts file export.

6

Slavko Zagorac: Annual Review May 2016

Figure 3: New Illustrator “Score Tools“ Javascript Plugin The Layer button above opens the Layer Tools window (Figure 4) which allows for the import and export of generic XML formatted layer hierarchies, copying and positioning of layers within a single document or between different documents and deletion of layers in any open document.

Figure 4: Layer Tools Illustrator Javascript Plugin The Page button on the Score Tools window opens the new Page Tools window (Figure 5). In this window, score pages can be created, copied, positioned or deleted. If a page is created by copying, layer structure of the existing page is copied into a new page. 7

Slavko Zagorac: Annual Review May 2016

Figure 5: Page Tools Illustrator Javascript Plugin

A click on the Bar button in Score Tools opens new Bar Tools window (Figure 6). This tool speeds up the creation of bars on any score page. Bars can be created from scratch at a specified position with a time signature and tempo change, copied from the existing bar with or without the notation or be deleted. All bar actions are applied to all staves, therefore avoiding the need to edit individual instruments. The Score Tools Export button opens the Export Tools window (Figure 7). From this window it is possible to export the full score and parts in SVG and PNG graphical format. In order to provide accurate and automated space/time mapping required by user interfaces such as InScore, the concept of Beat Lines (BL) was introduced. Beat Lines mark the position of beat onset within the bar and can be automatically generated by the plugin if specified. Export process also creates necessary mapping files for InScore in the required format: ( [Xstart, Xend[ [Ystart, Yend[ ) ( [BBstart, BBend[ ) where X and Y are space coordinates in pixels and BB is a number of Base Beats since the beginning of the section (BB explained below). These values define a two dimensional rectangle between two Beat Lines and the corresponding start/end time.

8

Slavko Zagorac: Annual Review May 2016

Additionally, the export process automatically collates score meta data required by the score scheduling engine. This information about time signatures, tempo changes, other score events, Beat Lines positions and related Base Beat values are stored in a csv (comma separated values) file. The Base Beat value indicates the lowest time resolution available for event scheduling. The current Base Beat resolution is 1/8 (quaver) but it can be changed to any other duration in the plugin if required. The easiest way to explain the relationship between a bar beat and a Base Beat is by example. In a 4/4 bar each bar beat consists of two Base Beats (if Base Beat value is 1/8). In 5/8 bar with the beat division of 3+2/8, the first bar beat consists of 3 Base Beats and the second bar beat would last 2 Base Beats.

Figure 6: Bar Tools Illustrator Javascript plugin

Figure 7: Export Tools Illustrator Javascript plugin

9

Slavko Zagorac: Annual Review May 2016

Symbolic notation drawing in Illustrator can be extremely time consuming, particularly for a novice user. To accelerate symbolic notation generation a set of custom symbols based on open-source LilyPond notation font were imported into Illustrator (Figure 8). For a variable length continuous lines such as crescendo and decrescendo markings, a set of brushes were created and imported into Illustrator (Figure 9).

Figure 8: Symbolic notation custom Illustrator symbols

Figure 9: Symbolic notation Illustrator brushes 10

Slavko Zagorac: Annual Review May 2016

With a view to optimise composition workflows a set of stave templates with fully defined hierarchical layer structures were created (Figures 10 and Figure 12). As each stave was implemented in a separate layer it was possible to use Score Tools to copy and position stave templates on score pages where necessary. In the template examples below, Beat Lines are visible as greyed out lines within the bar. Once copied to a score page Beat Lines can be moved horizontally to any position if additional notation space is required. The shape and size of graphical staves were created to fit into the score visualisation strategy as explained in the Score Display chapter below.

Figure 10: Symbolic notation stave templates

11

Slavko Zagorac: Annual Review May 2016

Figure 11: Graphic notation stave templates

12

Slavko Zagorac: Annual Review May 2016

To verify graphical and symbolic notation creation with the Score Tools plugin, a sample score with common features of both notation styles was created. Figure 12 illustrates graphic score staves while Figure 13 shows symbolic notation staves. Beat lines were set to be visible in both full score and the part score example.

Figure 12: Example of graphic notation created in Illustrator

13

Slavko Zagorac: Annual Review May 2016

Figure 13: Example of symbolic notation created in Illustrator An example of an exported part stave is presented in Figure 14. The shape and size of the part staves were designed to comply with the Two Stave Configuration explained below.

14

Slavko Zagorac: Annual Review May 2016

Figure 14: Example of score part export Scheduling Engine In order to enable the synchronisation and scheduling of score data and events over a network, I developed a new Scheduling Engine (SE) in Java programming language. I used Java because of my familiarity with its syntax and its strong support for network communications. The downside of Java is its indeterministic thread scheduling and indirect memory management, however, I believe that it is possible to develop a good enough solutions for this purpose. At the moment, SE can only be run as a back-end server (Figure 15), however, I am hoping to develop a graphical user interface which would simplify management of SE features in the near future. SE can import and parse csv score file exported from the Score Tools Illustrator plugin. Based on the analysis of imported score meta data it creates and schedules necessary events, runs transport based timers, executes scheduled events at specified time and sends OSC events to registered clients over a network. The concept of a transport is widely used in other music software (such as Max MSP). It translates system time into a musical time defined by metre and tempo. SE allows for multiple transport object and therefore multiple independent metre and tempo streams. This in turn can simplify composition and performance of various polymetric and polyrhythmic structures. Currently, the SE timer resolution is 1 millisecond. Maximum scheduling deviation measured in local testing is 1 millisecond, which is acceptable. The aim is to keep cumulative latencies, including network jitter and transport latency below 5 milliseconds. The lowest event scheduling resolution is defined by the value of a Base Beat (1/8 at present). Events can be pre-scheduled from the score data or added to SE in real-time. The engine is agnostic regarding the type of events and their source/destination, so any notation type and event flow direction should be possible. Present implementation of SE is fairly basic and there are many areas marked for improvement as discussed below.

15

Slavko Zagorac: Annual Review May 2016

Figure 15: Java Scheduling Engine source code Score Display For real-time score rendering duties, I selected the InScore software due to its networking capabilities, multiple graphical format support and built-in interactive processing and scripting engine. Unfortunately, it was discovered that InScore SVG support, which relies on underlying Qt 5 library, is not fully compliant with the latest specification. The positioning of notation symbols referenced via xlink:href attribute was incorrect, which made exported scores unusable. The workaround was to export all score parts in PNG graphic raster format which worked reasonably well. After initial experimentation, I came to the conclusion that it is enough to have only two staves visible at any time in the part score (Two Stave Configuration). In this configuration one of the staves shows notation that needs to be played currently (Active Stave), and the other stave shows upcoming notation (Passive Stave) (Figure 16). The active stave in marked with red frame while the passive stave frame is greyed out. When the Active Stave is completed it becomes passive. while at the same time, previously Passive Stave becomes active, and so on. This Two Stave Configuration leaves musicians enough time for preparation and is flexible enough to cover different kinds of music material. Several features have been added to enhance perception of time and help with preparation of a musician playing from the real-time score. Current position is visualised with two indicators in the score: vertical line showing the current score position (which only moves in horizontal (X) direction), and the circular indicator indicating the beat and tempo (which moves in both X and Y 16

Slavko Zagorac: Annual Review May 2016

direction). The position indicator movement always starts from the upbeat position relative to the current beat. Another invented helper feature is the Semaphore which is visualised in the top left corner of the score. It draws the attention of a musician just before the score is about to be played and effectively acts as the countdown clock with the starting tempo indication. Figure 17 shows the same part as the previous figure, but after a period of time has passed and the current position has moved to bar 7, beat 3. Several features can be observed here: •

The bottom stave has become active while the top stave is now inactive.

•

The top stave notation has been updated with the upcoming notation. This happens automatically after the first bar of the active stave is completed.

•

Position indicators are on beat 3 in bar 7.

•

The semaphore to the left of the title is now green, indicating score in progress..

Figure 16: Example of the score rendered in InScore software

17

Slavko Zagorac: Annual Review May 2016

Figure 17: Score example with position indicator where bottom stave is active Time dependent features are best observed in live performance mode or recorded video format.

Conclusions and Future Work The work undertaken so far produced just a skeleton of the intended integrated system, however, the entire process chain from composition to performance over the local network has been covered, which was the initial aim. It is now apparent that the development of a comprehensive, generic networked system for contemporary notation composition and performance requires a significant development effort, big enough to keep a team of dedicated developers occupied full-time for several years. However, a number of milestones are achievable with incremental steps leading to this fully comprehensive solution.

18

Slavko Zagorac: Annual Review May 2016

The first step is to make the current system robust enough to be used by musicians in a live performance environment. The feedback from musicians would be invaluable in my efforts to make this system as user friendly and as workable as possible. I am hoping to organise the first live performance at the beginning of the next academic year (Oct 2016). After the initial performance I intend to concentrate on compositional aspects and the potential for innovation that this system promises. This includes exploration of polymetric and polyrhythmic streams, multidirectional interactive event triggering between musicians, automation of svg paths to Max MSP control data, blending of heterogenous sound sources, synchronised multimedia, live algorithmic and real-time notation, etc. The current system relies on several external dependencies. InScore developers are hoping that the new Qt release will have full support for SVG 1.1 specification which would make InScore SVG rendering more accurate. They are also working on javascript port which should work in all major browsers. This is good news as all major browsers already support SVG xlink:href attribute correctly so the exported scores would automatically be rendered as intended. Another positive consequence of running InScore inside a browser is the possibility of using the same user interface for notation authoring and score rendering, which is one of the long term aims. In the end, I hope that the outcome of this research will help myself, and others, invent unique compositional voices.

19

Slavko Zagorac: Annual Review May 2016

References [1] Hajdu G., (2012) “Composing for networks”, Proceedings of the Symposium for Laptop Ensembles and Orchestras [2] Hajdu G. and Didkovsky N., (2012) “MaxScore - Current State of the Art”. Proceedings of the International Computer Music Conference [3] Hajdu G., (2008) “Real-time Composition and Notation in Network Music Environments, Proceedings of the International Computer Music Conference [4] Hoadley R., (2012) “Calder’s Violin: real-time notation and performance through musically expressive algorithms” , Digital Performance Laboratory Anglia Ruskin University Cambridge UK [5] Hoadley R., (2014) “Dynamic music notation in Quantum Canticorum”, Goldsmiths University of London, Live Algorithms for Music Symposium [6] Agostini A., Ghisi D., (2012) Bach: An environment for computer-aided composition in Max, Proceedings of International Computer Music Conference [7] Fober D., Orlarey Y., Letz., (2014) “Augmented Interactive Scores for Music Creation”, Proceedings of Korean Electro-Acoustic Music Society's 2014 Annual Conference (KEAMSAC2014) Seoul, Korea, [8] Fober D., Orlarey Y., Letz., (2013) “Programming Interactive Music Scores with INScore”, Proceedings of the Sound and Music Computing conference – SMC’13 [9] Fober D., Orlarey Y., Letz., (2012) “Scores level composition based on the GUIDO music notation”, Proceedings of the International Computer Music Conference [10] Renaud A., Rebelo P. (2006) “Network Performance: Strategies and Applications”, SARC, NIME 2006 [11] Bhagwati S, “Alien Lands / monochrom / Nil Nisi Nive”, Festival de Montréal Nouvelles Musiques performance, https://vimeo.com/21464366 [12] Hajdu G., (2005) “Quintet.net: An Environment for Composing and Performing Music on the Internet”, Leonardo Music Journal, Feb 2005, Vol 38, No. 1, Pages 23-30 [13] Gottfried R.. (2015) “SVG to OSC Transcoding: Towords A Platform for Notational Praxis and Electronic Performance”, 2015 TENOR International Conference on Technologies for Music Notation and Representation (Ircam/Sorbonne).

20