ENSEEIHT 2 rue Charles Camichel BP 7122 31071 Toulouse Cedex 7 Télécommunications Spatiales et Aéronautiques (TéSA) 17 bis, rue Paul Riquet F-31000 Toulouse FRANCE

Automatic Music Transcription using Autoregressive Frequency Estimation

14. June 2001

Fredrik Hekland

NTNU, Norwegian University of Science and Technology [email protected]

Under the direction of: Corinne Mailhes (ENSEEIHT) David Bonacci (ENSEEIHT) 1 of 38

Preface Where and why With help from the Erasmus student exchange program, I had the opportunity to do my fourth year as an engineer student abroad. I had chosen France as country since I wanted to learn to speak French and see parts of Europe I hadn't seen before. I found ENSEEIHT in Toulouse which offered Signal Processing, and luckily they accepted my application. Since it was only the fifth year at this school who was offering enough courses within signal processing, I followed the option "Traitement du Signal et des Images" even though I was one year short. This also meant that a "Stage", a four months final project, was ahead of me. I was kindly given chance to do the Stage at TéSA, a research laboratory at the school. Some possible subjects were presented to me, and after some counsels from my responsible professor at NTNU I chose the subject regarding music transcription. TéSA (Télécommunications Spatiales et Aéronautiques) is a newly created research laboratory as a collaboration between several schools and enterprises. The lab is well equipped with both hardware and software, and a good library which made my literature review easy. The work In the beginning I had a two weeks period doing a literature review, trying to find previous work in the field and all the necessary background information. A lot of articles was found with help from google.com and citeseer.nj.nec.com, while the library contained most of the IEEE publications. After having gained an overview of the problems, a longer period of testing was conducted. Both synthetic signals and samples from real instruments were used. The different frequency estimators and model order criteria were explored, and it was decided to use the Modified Covariance Method coupled with a simple AIC/MDL order selection criterion for the transcriber. A working monophonic transcriber was built, and some of the findings in the project can permit an extension to polyphonic operation. Software Matlab 5.2 and 5.3 was used for all coding purposes and most of the analysis work, and Spectrogram 6.0.4 was helping to analysis the instrument samples. Yamaha's free wave-editor was indispensable to mix and manipulate the necessary wav-files. The Matlab files referred to in the text is not rugged enough to be used in any serious transcribing. For that reason, the code is not available to the public. Personal outcome Even though my work is not exactly ground-breaking, I have gained much personally. Especially concerning the process of doing a research work and writing a report. I now know more how to proceed and what to do underway, and certainly some of the pitfalls to avoid. Both within Matlab programming and Signal Processing I seen a great development, and within parametric modelling I have gained a greater understanding. Finally, it has been interesting to see how the life is in a laboratory and to observe the cultural differences and similarities between Norway and France. At last, it must be mentioned that I have learned a lot French, starting at ground-zero before my arrival in France now being able to communicate without too much problem at a basic level. I hope I'll be able to maintain and improve the language in the future.

2 of 38

Acknowledgements I would like to thank the Director of the laboratory Prof. Francis Castanié, and the responsible for "Traitement du Signal et des Images" Dr. Corinne Mailhes for giving me the opportunity to do this work in the labs. Mdm. Mailhes also being the responsible for my Stage, and David Bonacci was the person working on a subject closest to mine and being the most important advisor giving good ideas and tips. Both deserves a thank. The guys at "Bureau treize" for having received me well, and accepting me even though my French is at best confusing, and at times incomprehensible. C' est dommage que je ne pourrais pas participer dans vos conneries. J' ai passé un bon moment chez vous quand même. Thanks to all the other persons at the lab helping me out and being nice to me. Lots of moral support and positive words from Tonje helped me when I needed it most. I love you.

3 of 38

Abstract The project studies the use of Principle Component AR Frequency Estimation in automatic music transcription and discusses some of the problems arising when using AR models, among them model order selection. Some comparisons to classical Fourier method is done. A well-functioning monophonic transcriber using the Modified Covariance Method as pitch estimator is implemented in Matlab and some suggestions of further work is given.

4 of 38

Table of Contents Preface.......................................................................................................................2 Acknowledgements...................................................................................................3 Abstract.....................................................................................................................4 Chapter 1 – Initial theoretical studies ....................................................................6 Introduction.................................................................................................................................6 Presentation of the problem......................................................................................................6 The goal of this project.............................................................................................................7 Literature review.........................................................................................................................8 Papers dealing on musical transcription....................................................................................8 Commercial or free transcription programs available...............................................................9 A closer look at the challenges..................................................................................................10 Common features of instruments............................................................................................10 Problems encountered when estimation pitch.........................................................................11 Other problems related to transcription...................................................................................12 MIDI file format......................................................................................................................12 Pitch estimators..........................................................................................................................13 Cross-correlation between signal and sinusoids......................................................................13 Filter banks.............................................................................................................................13 Fourier-based methods............................................................................................................14 Autoregressive methods..........................................................................................................15 Model order selection..............................................................................................................16

Estimation of the Prediction Error Power.........................................................................................................16 Order estimations methods based on Singular Values or noise subspace estimation.......................................17

Chapter 2 – Implementation of a music-transcriber............................................18 Initial testing on real and synthetic musical signals...............................................................18 Fourier methods, real instruments...........................................................................................18 Yule-Walker, synthetic signals................................................................................................20 Modified covariance method, synthetic signals......................................................................22 Implementation of the transcriber...........................................................................................25 The structure of the transcriber...............................................................................................25 Transcribing real music...........................................................................................................26

Flute5.wav – a simple flute solo........................................................................................................................26 Flute00.wav – A quicker flute solo...................................................................................................................29 Some different sound files.................................................................................................................................30

Some final words........................................................................................................................31 Limitations of the transcriber..................................................................................................31 Improvement for the transcriber and ideas for future work.....................................................32 Conclusion..............................................................................................................................33

A – Converting between Note, MIDI and Frequency...........................................34 B – Matlab code for the transcriber......................................................................35 References...............................................................................................................44

5 of 38

Chapter 1 – Initial theoretical studies Introduction Presentation of the problem Music can be represented in many different ways, ranging from simple monophonic PCM files with low sample rates to highly symbolical multitrack representations containing control information for various musical parameters. Every abstraction level has its use and being able to easily switch to another representation would certainly be useful. The most obvious application is to aid musicians or composers write music by playing the actual instrument rather than writing notes or playing on a keyboard. This could also extend to analysis of existing music where nothing but the sound recordings exists. An application that could be of greater commercial interest is the possibility to search for music on the Internet by simply whistling the tune, so called content-based retrieval or query by audio [McNab],[Vercoe97]. One could also think of a scheme that tracked all radio stations for a particular music style. Areas which certainly would appreciate perfect transcription are the coming standards MPEG-4 for structured audio [Vercoe97], and MPEG-7 for description of audio-visual content. While the process of converting music from a symbolic format to a waveform representation (synthesis) has evolved over the years, and now gives a fairly realistic sound for a reasonable prize, the opposite process (analysis or transcription) is far from being ready for commercialisation. There exists several monophonic solutions that are reported to work well in real time, but at soon as we want to analyse polyphonic music our options are effectively reduced to zero. A quick search on the Internet found some shareware programs claiming to perform transcription (see table 1), but trials showed these programs to perform rather poorly even after substantial parameter regulations. Speech recognition is very similar to the problem of musical transcription, but while the former has experienced much interest and successful applications during the last three decades, research on music recognition has mainly been done by a few individuals with special interests in the subject. One reason for this is the apparent lack of commercial applications. Another is the complexity of the problem compared to speech recognition. While speech is limited to frequencies between 50Hz to 4kHz and the sources all have similar characteristics, musical frequencies ranges from 20Hz to 20kHz and there are many different instrument models. However, the main problem is the fact that western music is constructed upon harmonic relations (i.e. Different instruments playing frequencies that are in fractional relation) which gives rise to spectral overlapping and possibly complete masking of certain notes. When we think of a symphonic orchestra with many musicians playing simultaneously, the task of separating and recognising each one of them from a simple twochannel recording seems (and might prove to be) impossible. Currently most systems works in an bottom-up fashion, that is, all decisions are based upon the frequency- and segmentation-information one obtains from the music recording. This works satisfactory for monophonic music where the notes are well separated in time and frequency. But these systems are unable to correct even the most obvious errors, since they don' t possess any knowledge of compositional styles (rules). This problem is addressed by putting a top-down along with the normal bottom-up recognising engine, and for example letting the "knowledge-loaded" top-down engine monitor the transcription process and intervene when it disagrees to the estimations found. The common way to implement such systems are so called Blackboard system whose name stem from the idea that several ' experts' , each having knowledge of a certain parameter, are ' standing in front of a blackboard'and together solving the given problem. These systems are very flexible since it is easy to add experts and the system can be driven in a bottom-up correcting fashion or as a top-down predicting fashion.

6 of 38

The goal of this project Given the restricted time and lack of experience, the goal of this project was to explore the problems related to transcription and review the existing solutions, and thereby try to implement a monophonic transcriber that works better than the existing affordable programs. Instruments that are non-harmonic in nature, such as percussion instruments are left out, while any kind of harmonic instrument is targeted. In fact an independence of instrument was wanted, and additionally a possible later extension to polyphonic recognition was desired. It was therefor decided to try other ways to do frequency estimation and possibly obtain higher resolutions and more precise estimates than what is realisable using standard Fourier methods. A natural choice was to investigate the different parametric methods available, their advantages and disadvantages, and the well-known problem of model order selection. A very limited top-down based correcting system is implemented, as to improve the transcription process in lack of a segmentation system. As output format the standard MIDI file format was chosen. This because of it widespread use and its relative simplicity. All the programs was to be built in Matlab since it provides most of the needed routines and enables rapid development.

7 of 38

Literature review Papers dealing on musical transcription Only one article on the use of parametric methods in music recognition was found [Schro00]. Here, three algorithms are discussed and analysed with respect to relative frequency precision. It was concluded that the Modified Covariance Method (MODCOVAR) was superior to both standard Maximum Entropy Method (Yule-Walker) and Prony Spectral Line Estimation. Additionally, the two former methods give us the relative size of the spectral peaks for free. The "Modcovar" method was applied to a short piano recording using a fixed order of 20, showing a promising result. No order estimation was discussed. Another and more elaborate work is the master thesis of Anssi Klapuri [Klapuri98] where some of the problems related to automatic transcription are discussed, and a system trying to resolve the problem of harmonic overlapping is described. The shortcomings of the purely bottom-up approach when it comes to polyphonic music and the necessity to employ a top-down "knowledge system" are discussed. The thesis discusses very general problems and its possible solutions, and different techniques for extracting information are presented. Klapuri has also released several papers related to this thesis, more directed to a specific implementation; [Klapuri01A], [Klapuri01B]. Even more can be found at and Top-down systems are probably the most promising approach for polyphonic music recognition and despite the fact that such systems will need greater understanding of human musical perception and a huge amount of musical knowledge at its hand, some systems with limited knowledge have been implemented and shows improvements over the usual bottom-up systems. Examples of an ' expert'in a top-down system are probability of transition between different chords and rules for which notes can be played to which chord. See [Kashino98] and [Martin96]. Two other areas of importance, and yet untreated in this project, are segmentation and instrument recognition. Segmentation of events (i.e. Notes and pauses) gives us the possibility to respect the duration of each note, and to avoid including two notes in one frequency analysis frame to obtain better frequency estimates. The master thesis of T. Jehan deals with this problem [Jehan97], and proposes methods both in the time domain and in the frequency domain. Especially the third chapter using changes in AR model as basis for segmentation could be an interesting future addition to this project. Recognition of instruments gives us the opportunity to improve the process even more by using instrument models in the frequency analysis, and by automatically setting correct instruments in the output file. Methods seen tested are some kind of neural network working on cepstral coefficients or features from log-lag correlograms, See [Brown99] and [Martin98].

8 of 38

Commercial or free transcription programs available Name and Url

Technology

WIDI Recognition System 2.6 http://www.midi.ru/w2m

FFT based Polyphonic

AudioToMidi 1.01 http://www.midi.ru/AudioToMidi/

Unsure. Possibly cross-correlation with synthetic sinusoids. Polyphonic.

AmazingMIDI 1.60 http://www.pluto.dti.ne.jp/~araki/amazingmidi/

Unsure Single instrument, polyphonic

WAV2MID 1.5a http://www.audioworks.com

Unknown Monophonic

DigitalEar 3.0 http://www.digital-ear.com/index2.html

Unknown Monophonic

Table 1

As said in the introduction, none of these programs offers fully automatic operation and independence of instrument and even after many parameter adjustments, none of the polyphonic ones give an impressive result even on monophonic music. A more comprehensive list of existing programs can be found at: The two monophonic transcribers performs quite well. Digital Ear was difficult to test, because the demo version was very restricted. From what could be tested, it seemed a bit less powerful than both the project transcriber and the AudioWorks'transcriber. The AudioWorks transcriber performed nearly perfect with a minimum of parameter setting and excellent speed. Testing the program with the same wav-files as in the project showed a similar performance to what was obtained in this project with less parameters. A small bug in the program made the instrument in the MIDI file to always be a grand piano.

9 of 38

A closer look at the challenges Common features of instruments Apart from percussion instruments, the creation of sound in most instruments is based on generating standing waves on strings or in hollow tubes of some geometry. Since the length of the string and tube normally is fixed for a given note, the possible wavelengths are constrained to fulfil the wave equation for the given length, geometry and speed of sound: 2

 



 

2



u x,t 1 u x,t (1) 0 x² v² t² Usually strings are connected in both ends, while tubes can be open in one or both ends with either circular or conical inner geometry. These constraints only allows certain modes to operate in the resonator, and for strings and circular open tubes possible solutions are of the form:

   k   v ,k

Ak sin 2

(2)

eventually as a sum of cosines if the tube is circular half open. As for the conical case, the solution consists of spherical harmonics. A more elaborate explanation can be found at the website [WolfeA] This gives rise to a harmonic spectrum, with all over-harmonics being an integral number of the fundamental frequency. Figure 1 shows a typical frequency spectrum, with the fundamental frequency at 392Hz and all the over-harmonics being 2 to12 times the fundamental frequency. Another typical feature is that most of the energy is located in the lower harmonics, leading to weaker overharmonics. For example in piano sounds, more than 90% of the energy is contained in the fundamental [Schro00]. In western musical notation, it' s the frequency of the fundamental in the harmonic series, also called the pitch1 , that names the note. The Figure 1 relation between the notes is given in (3), with 440Hz being the so-called chamber note (A4), and k ranging from -48 to 39 for a standard piano. To convert between frequency, note name and Midi number, see table 4, appendix A. A G4 note from a flute

0

−10

Magnitude [dB]

−20

−30

−40

−50

−60

−70

−80

f



0

1000

2000

3000 Frequence [Hz]

4000

5000

   

440Hz 2

k 12

Hz

,k

1The term pitch is not well defined, and at least three different meanings exists; a) Production pitch: The rate of which the excitation device opens and closes (e.g. Glottal closure rate in human speech). b) The mathematical pitch: The fundamental of the harmonic series c) Perception pitch: The actual frequency perceived by the listener. In this project, the definition in b) is used, since it is directly connected to the searched notes.

10 of 38

(3)

Problems encountered when estimation pitch It is clear that a reliable pitch estimation is essential to successfully do the transcription, and the relative frequency error must not exceed 2.5% to avoid picking the note a semitone away from the real note. Further we want to be able to resolve several instruments playing simultaneously so we must find as many harmonics as possible, where the harmonics can be closely spaced or even overlapping. Instruments without pitch that have noise-like spectres (percussion/drums) can obviously not be identified by tracking the pitch, and are not treated in this project either. Another problem is irregularities in the harmonic spectrum. A classic example is the clarinet where only the odd harmonics are present, because of the constraints the circular half-open resonator impose on the wave equation. Guitars can sometimes play without the first harmonic present, or at least very weak. This leads to the phenomenon virtual pitch, where the human brain deduces the missing fundamental from the distance between the harmonics [Terhardt] (It is this concept that is used to in some headphones to obtain a lower frequency response than what is possible with the given membrane radius). Another problem related to string instruments is inharmonicities in the spectrums. This is due to the fact that the string isn' t infinitely thin, but has a certain mass, and when this mass is unevenly distributed some of the harmonics can get displaced from its ' correct'position [WolfeB]. Even if these exceptions cause problems for the transcription process, they can be countered fairly easily and the most concerning issue is still estimating pitch of multiple instruments where the harmonic series are overlapping. This might seem like a special case which only happens from time to time, but the fact is that this is the rule rather than the exception. The reason for composing the music in this fashion is rooted in the way the brain experiences the music; entities related harmonically are put together in a bigger entity which is experienced as a whole, that is, one cannot distinguish the different notes. This helps reducing the complexity of the listening process, and makes it easier to listen to the music. The more the elements in the music are in harmonic relation, the more the human brain is grouping the elements together, making it more pleasing to listen to. So what makes the listening easier, makes the transcription process harder. Many different approaches exists for estimating the pitch, where many seems to be adapted from methods originally developed for speech processing. Still, most of these methods are made for monophonic music and will perform poorly when applied to polyphonic music. The most used is the spectrogram, based on the Short-Time Fourier Transform (STFT). Other examples are correlogram, filterbanks, Constant Q-Transform, cepstrum, auto-correlation, cross-correlation, zero-crossing detection, wavelet transform, sinusoidal modelling, AR- or ARMA methods, Prony modelling. Some efforts are made in trying to improve multipitch estimation by subtracting the spectrum associated with an estimated note, and thereby continue searching for notes [Klapuri01B]. Some of the methods mentioned above are discussed further down, other can be found in [Klapuri98], [Schro00], [Fitch00]. Even though it' s not directly applicable to music, the tutorial paper on signal modelling in speech recognition by Picone is worth a read to get an overview over available techniques within speech processing [Picone93].

11 of 38

Other problems related to transcription It is a known fact that it is the attack portion of the note (the transient state before harmonics in steady-state take over) dictates the way we experience the timbre, and it is probably this region that can provide the necessary information needed to identify the instrument. It is therefor important to determine the location of the on- and off-sets of the notes. This will also help avoiding placing an pitch analysis frame over two different notes and thereby getting more accurate estimates. More importantly when the orders is rising, segmentation enables us to adjust the analysis windows to cover the whole note, giving us as many datapoints as possible. Many schemes for segmenting music exists, see [Jehan97], [Klapuri98]. Successfully identifying the instruments would enable us to apply instrument models to the pitch estimation process. This could help detecting notes with overlapping harmonics since we would to some extent know the expected amplitude signatures. Also, it would provide automatic selection of instruments in the output file. Processing cost and storage requirements are not often discussed in connection with automatic transcription. It might not be an issue with laboratory tests or professional applications not in need of realtime applications, but if used for music search on the Internet, the transcriber would most probably be implemented as a Java applet that is downloaded from a server. It is clear that as long as not everybody is blessed with a high-speed Internet connection and the fastest processors, only a limited amount of knowledge and calculations can be put into the applet. Thus, a system requiring as little resources as possible while still being able to do the job would be desired.

MIDI file format The MIDI file format is relative simple, and much documentation can be found on the Internet. It is in binary format something that helps making the files small. Both single track and multi track files are supported. In brief, the file is organised in chunks. Every chunk start with a four bytes ID field telling what type of chunk it is and a four bytes length field telling the number of bytes of data following this header. All MIDI files begin with the MThd chunk containing tempo information, and are followed by one or more MTrk containing meta-data and the actual music. All events in the track-chunk such as note-on and note-off are equipped with a delta-time field, indicating the duration (in MIDI clocks) between the event and the preceding event. This delta-time is stored in a variable-length format enabling shorter events to be represented with fewer bytes than with a fixed format. Three types of files exists. Type 0, 1 and 2. Type 0 contains only one track and is thereby monophonic. Type 1 contains one or more tracks where all tracks are played simultaneously. Type 2 contains one or more tracks, but the tracks are sequentially independent. Type 1 was the natural choice for this project, since an extension to polyphony is accomplished simply by adding MTrk chucks. A good overview over the event codes in the specification can be fount at , while a more textual introduction can be found at

12 of 38

Pitch estimators Cross-correlation between signal and sinusoids Since we assume that the signal consists of sinusoid with several harmonics (also sinusoids), we could find these sinusoids by taking the cross-correlation between the signal block and test sinusoids with frequencies given by (3). The frequencies found are those who yield the highest cross-correlation values. These test sinusoids could be synthetic ones or samples of real instruments. This method will of course be a bit costly in terms of computation, since every possible note must be tested. Further, we can' t reveal notes where all harmonics are masked by a lower note. To be able to to that, we must apply instrument models to compare the correlation value given from (4), with the expected value from the used model; a deviation from the expected value could indicate a hidden note.

            N

cxy k

Fig.2 shows an example of the G4, 392 Hz (same as in Fig.1) , where the cross-correlation between the signal and 8 sinusoids has been calculated.

x l y l k

,k 0,1,

l 0

(4)

k max

Xcorr between G4 from a flute, and 8 sinusoids

15

10

xy

5

c

The calculation cost is rather expensive since we have to do N multiplications for every possible note (128 for a piano), and the lower the note is, the bigger kmax gets to account for the longer sinus-period.

1

0

−5

−10

−15

392Hz 784Hz 1176Hz 1568Hz 0.67of the first 4 0

5

10

15 sample

20

25

30

Figure 2

Filter banks Some of the earliest attempts to estimate pitch was done with filterbanks. One simply separates the spectrum by filtering the signal with several bandpass filters, and then measures the energy present in the different filters. A better, and more promising approach is to use a dyadic filterbank in conjunction with wavelets, see [Karaj99] and [Fitch00].

13 of 38

Fourier-based methods As mentioned earlier, the spectrogram has been the most popular way to obtain a pitch estimate. To find the frequencies of the sinusoids, one simply picks the peaks in the spectrum. The spectrogram based on the Short-Time Fourier Transform (5), is an extension of the normal Fourier transform where the small analysis window is moved along the time axis and successive transforms are taken. Sx u,f

' !#"%& $ ' )! ( * ! & + !#"-, .! , f t

w t u e

i2

f

dt

(5)

The spectrogram gives the energy density similar to the periodogram, and is given by (6) P S f u,f

Sf u,f

2

(6)

Two problems are associated with the spectrogram. The first problem is the windowing function. This windowing introduces side-lobes which can mask weaker signals, and trying to reduce the sidelobes eventually leads to a wider main lobe. Further, the resolution in time and frequency is limited by Heisenberg' s uncertainty theorem, which states that the area of the rectangle defined by the windowing function has a minimum given by (7)

/ 0/ 1

1 (7) 2 This windowing of the data clearly limits the frequency resolution obtainable, and we are given the choice between high frequency resolution or high time resolution, but not both. Additionally, we have got some given minimal time resolution to respect if we want to avoid mixing several notes in the same analysis frame, and to know where the note is actually starting and stopping. For example, a musical piece playing at 200BPM (beats per minute or quarter notes per minute) sampled at 44.1kHz gives us 60sec/(200*4)=75msec per sixteenth note, which again gives us 44100*75msec=3309 samples per sixteenth note. Lower sample rate means fewer datapoints, and if we want to capture ornamental tones2, we' ll have even fewer sample points. f

t

Then to the second problem, namely the linear spaced frequency bins of the Fourier transform. Musical tones are ordered in a logarithmic fashion similar to the sensitivity of human hearing, so the distance between two neighbouring notes in the higher end of the musical scale is greater than in the lower end. That allows for higher time resolution by lowering the frequency resolution for the higher end of the spectrum. The Constant Q Transform obtains a such constant ratio between centre frequency and frequency resolution for each frequency partition by combining several bins of an FFT. This however fails to give better time resolution for the higher frequencies, and we haven' t won much.

2 Ornamental notes are very short notes not written on the music sheet, but added by the musician.

14 of 38

Autoregressive methods Since the windowing function is a problem for the resolution, we want to avoid windowing our data. Parametric methods like AR models and eigenvector methods make this possible. We assume our signal to be composed by a number of sinusoids in white noise, something that enables us to make use of an eigendecomposition of the estimated auto-correlation matrix. This is possible since the matrix is composed of a signal auto-correlation matrix and a noise auto-correlation matrix, where the eigenvalues associated with noise are generally small compared to those of the signal. Since we cannot find theoretical Maximum Likelihood Estimator (MLE) for more than one sinusoid analytically [Kay88], we must use a sub-optimal method. The most promising estimator that is not too computationally expensive is the Principal Component AR frequency estimator.

2 354 2 6 2

R xx1 r xx

a

(8)

Eq. (8) is the standard AR parameter estimation, with the auto-correlation matrix being positive definite hermitian. This allows for a eigendecomposition with real, positive eigenvalues and orthonormal eigenvectors. These eigenvectors span the entire auto-correlation space, so we can write the auto-correlation vector as a linear combination of the eigenvectors, which inserted into (8) gives:

2 35487 : 2 2 2 7 ; 2 9 9 2 3 4<7 ;: 2 2 9 M

a

1

i 1

vi v

H i

i

M

j 1

j

vj

(9)

Simplifying and discarding the (M-p) smallest eigenvalues, we obtain: p

a PC

i

i 1

i

vi

(10)

Finally, the frequency estimates are obtained by picking the peaks in the spectral estimator in (10) which is done by solving the roots of (10), and take the angles of these poles.

= >#3

P pc f

?2

2 pc

@| 7 9 2 = > 6 A |

2

p

1

k

1

a pc k e

j2

fk

(11)

In practice, we can estimate the AR coefficients with the Modified Covariance Method (MODCOV), pick the p poles closest to the unity circle and again we obtain the frequency estimations from the angles of these poles. The modified covariance method has proven to be insensitive to initial phases of the sinusoids and spectral peak shifting due to noise is low. In fact, in absence of noise the true frequencies are found. Tests show that the Principal Component Method reaches the Cramer-Rao bound for SNR higher than 10dB. Since the MODCOV method is Least-squares optimised without constraints, the poles can end up outside the unit circle leading to unstable models. This is not a problem since we are only interested in the angles of the poles. Further, the AR spectrum can be calculated from (11).

15 of 38

Model order selection The choice of model order is crucial in order to make the AR methods perform acceptably, but unluckily no perfect methods exists. This problems gets even harder as the number of available sample points are reduced. As soon as the possible order is greater than 0.1 times the available datapoints, we are dealing with a finite sample and the criteria for larger samples are no longer valid and need corrections. Most of the methods are based on the maximum likelihood estimator of some parameter with some bias/variance correcting factor. Estimation of the Prediction Error Power

Most of the methods that exists tries to fit several model structures and orders to the data and the order is chosen by minimising a cost function. This cost function is usually based on the Maximum Likelihood Estimate of the prediction error power (PEP) from the model, but since the PEP is on average decreasing for increasing model order, we have to add a certain penalty to the cost function. This because higher order leads to higher variance in the prediction coefficient estimates, and thereby higher variance in the PEP. The information-based criteria requires two passes over the data; one for calculating the cost-function and another for choosing the right order. PEP:

C B D EGF H IKJLB H IMF KN D B H IOQP H ISB H I R 2 pep

E x n

2

x n

p

r xx 0

k

1

a k r xx k

(12)

The a' s in the PEP is calculated using the actual type (YW, ModCov (also called FB or ForwardBackward) etc.) and PEPFB indicates that the Modified covariance method is used. The best known criterion (for AR models) is Akaike' s Information Criterion (AIC):

HB ID

AIC k

T

arg

[ H C B H IUI#O V ]

(13)

[ H C B H IUI#O V ]

(14)

min

k 0,1,..,p max

2 pep

ln

k 2 N

k

This criterion is however not consistent and tends to overestimate the model order. The minimum description length (MDL) criterion fixes this problem by having a higher penalty factor, and is asymptotically consistent:

HB ID

MDL k

arg

T

min

k 0,1,..,p max

ln

2 pep

k ln N N

k

These two criteria impose equal penalty to all unknown parameters (amplitudes, phases, frequencies etc.). Better performance can be obtained with the maximum a posteriori (MAP) criterion, where different penalties can be attributed to the different unknown parameters. For example for AR models the MAP is equal to MDL, and for sinusoids with unknown frequencies + white noise the penalty is 5k/N ln(N). For a development of the MAP, see [Djuric96] and [Djuric98]. A problem arises when the number of datapoints is less than ten times the model order, as we' re then dealing with finite samples and the criteria given above no longer works. This problem is especially present in small samples where they can even fail to give a minimum. In larger samples they tend to choose too high orders as they don' t account for the increased variance in the modelling error. Finite Sample Information Criterion (FSIC) [Broer00] tries to handle this overestimation by changing the penalty factor to better suit the variance from the given AR estimation method and model order.

HB ID

FSIC k

arg

T

min

[

k 0,1,..,p max

ln

H C B H I IO ( W R OJ J 2 pep

k

k

i 0

)]

1 vi 1 1 vi

(15)

Where the vi depend on the AR method, and for MODCOV it is given as vi =(N+1.5-1.5i)-1. The combined information criterion uses the maximum between the one in (15) and 3⋅Σvi, see [Broer00] eq.13. 16 of 38

Order estimations methods based on Singular Values or noise subspace estimation

Another group of methods for estimating orders are the ones based on determining singular values of the signal auto-correlation- or the covariance matrix. These methods are more specialised as they are usually based upon the assumption that the signal consists of sinusoids in white noise. This makes it possible to decompose the auto-correlation matrix in a signal matrix and a noise matrix because the singular values associated with the signal are generally much higher than those of the noise. Additionally, these two subspaces are orthogonal. The drawbacks with these methods based on eigen-calculation are that they need O(p3) operations, and that the true order is not always the best order (At least for autoregressive models which are AR(∞) when corrupted with white noise). The AIC and MDL criteria based on eigenvalues are given below, and the development can be found in [Wax85].

c d Y ( a` b ) f _ _ _ ( ) ] ^ c _ ae b Y

[{ } ] 1 p k

p

XZY [#\

AIC svd k

arg

min

k 0,1

p max

l k

ln

l

1

1 p kl

X)Y [#\

arg

k

1

2p k p k

min

k 0,1

[{ }

p max

l k

1 p k

ln

1

1 p kl

l

p

k

1

l

(16)

l

c d Y ( a` b ) f X _ _ ( X g ] ^ c _ ae b Y p

MDL svd k

k N

p

k N

]

[_ X [ ) [

2p k ln N 2 p k

(17)

where k is the estimated order; p is the order of the covariance matrix, N is the number of datapoints used to calculate the covariance matrix, and the λ' s are the eigenvalues of the covariance matrix ordered as λ1 > λ2 > .. > λp. The expression in the braces are simply the ratio of the geometric mean to the arithmetic mean of the p-k smallest eigenvalues. Similar to the PEP-based criteria, the AIC tends to overestimate the order, while the MDL has shown to be consistent. It should be possible to merge the order estimation and the parameter estimation in order to reduce the computational load. Another possible method is to continuously estimate the noise subspace by means of a QRfactorisation of the covariance matrix or even the data matrix itself. The idea is to decompose a matrix X into a square matrix Q with orthonormal columns and a upper-triangular matrix R,

h

XE QR where E is a permutation matrix that orders the diagonal of R in descending order. R

\(

)

R 11 R 12 0 R 22

(18)

(19)

The factorisation in (18) is called rank-revealing QR factorisation if R22 has a small norm. The dimension of R22 is equal to the rank-deficiency of X, or equivalently equal to the dimension of the noise subspace. The dimension of R11 is then equal to that of the signal subspace. An effective implementation requiring on average O(p²) operations can be found in [Bisc92]. Of course, one can always decompose the signal- or covariance matrix, and determine the the rank of the noise subspace by finding the smallest singular values. However, this proves to be difficult when the order of the matrix is small.

17 of 38

Chapter 2 – Implementation of a music-transcriber Initial testing on real and synthetic musical signals Unless otherwise cited, all sound files are sampled at 11025 Hz with 16 bits in one channel. Some single note sound files was found in the Internet. A series of tones from piano, plus some single clarinet and saxophone examples was also found. The most useful was a series of flute tones found at from different flutes, with impedance and spectral measurements available. These sound clips was chosen as the basis the project. The synthetic signals were created from the script synthsign.m and the signals are analysed with the script process.m. Description of the different program modules is found in the appendix B.

Fourier methods, real instruments In the beginning some testing was done on single notes from real instruments. This to better see the difference between parametric and non-parametric methods, and to get a picture of how instrument spectrums look like. Additionally, some modules for file-handling and note/frequency decision was built that also was needed for the real transcriber. Synthetic signals was not tested, since it was assumed that as long as the peaks are well separated the correct frequencies are found, and that the resolution is limited by the number of datapoints N in the analysis windows. A crude periodogram was first implemented, and a simple peak-picking method was used to estimate the frequencies. The peak-picking worked by searching for the maximum value and then deleting a certain number of samples around this maximum. An example from an A4B is shown in Fig.3 with the Matlab output found in Table 2. A problem is that we have to use a threshold and thereby miss some weaker peaks. Instead of a fixed-value threshold, a more adaptive method that estimates the noisefloor to use as threshold [Durne98] could be used.

a4b.wav 10

0

−10

Magnitude [dB]

−20

−30

−40

−50

−60

−70

−80

0

1000

2000

3000 Frequence [Hz]

4000

5000

Figure 3 Peak 1 is at freq 0.000 Hz

Difference from 0 Hz:

Peak 2 is at freq 425.954 Hz

Difference from previous peak: 425.954 Hz

Peak 3 is at freq 438.066 Hz

Difference from previous peak: 12.112 Hz

Peak 4 is at freq 864.693 Hz

Difference from previous peak: 426.627 Hz

Peak 5 is at freq 877.142 Hz

Difference from previous peak: 12.449 Hz

Peak 6 is at freq 888.245 Hz

Difference from previous peak: 11.103 Hz

Peak 7 is at freq 1303.432 Hz

Difference from previous peak: 415.187 Hz

Peak 8 is at freq 1314.871 Hz

Difference from previous peak: 11.440 Hz

Peak 9 is at freq 1325.301 Hz

Difference from previous peak: 10.430 Hz

Peak 10 is at freq 1753.274 Hz

Difference from previous peak: 427.972 Hz

Peak 11 is at freq 2200.088 Hz

Difference from previous peak: 446.814 Hz

Table 2

18 of 38

0.000 Hz

It is clear that the Fourier Transform will work fine for monophonic music, but a certain smoothing is necessary before the peaks are searched for. A different search routine would be desired, since one never knows how many points to delete around the peaks.

An averaged periodogram with overlapping windows 0 would be a better alternative, as it would −10 reduce both the amplitude−20 and frequency variance. Matlab provides this as a −30 Welch-periodogram. The same sound is tested in −40 Fig.4, and we see a clear improvement over the −50 periodogram: a more precise frequency estimate −60 is obtained, and the −70 spurious peaks present in table 2 are gone. This is −80 file has 18743 samples (1.7 0 sec) and is of course a bit longer than the average Figure 4 note duration. Magnitude [dB]

A4 − Welch, 1024 points FFT, 512 points window, 25% overlap

1000

2000

3000 Frequence [Hz]

4000

5000

Peak 1 is at freq 441.431 Hz

Difference from 0 Hz:

441.431 Hz

Peak 2 is at freq 872.095 Hz

Difference from previous peak: 430.664 Hz

Peak 3 is at freq 1313.525 Hz

Difference from previous peak: 441.431 Hz

Peak 4 is at freq 1754.956 Hz

Difference from previous peak: 441.431 Hz

Table 3

With such a short window we have a possible resolution of (300-1)*11025Hz = 36.75 Hz, and a smoothed periodogram makes it even worse. For example, an A1 (55Hz) would prove to be difficult to resolve, and overlapping harmonics would be unresolvable.

A4 − Welch, 1024 points FFT, 512 points window, 25% overlap 0

−10

−20 Magnitude [dB]

In music, we can expect the signal to be stationary for about 25 ms, so a window size of that duration is realistic. Fig.5 shows a 28 ms clip (308 points) of the previous A4, and the the same frequencies as in table.3 are found.

−30

−40

−50

−60

0

1000

Figure 5

19 of 38

2000

3000 Frequence [Hz]

4000

5000

Yule-Walker, synthetic signals Even though the Yule-Walker method is reported to perform poorly as a frequency estimator in noisy signals [Kay88], it was implemented partly because of its computational simplicity and partly because it was the only available method to estimate the AR coefficients in Matlab 5.2. A 25ms synthetic noise-free signal corresponding to an A4 with four overharmonics was created and the frequencies was estimated from rooting the AR coefficient polynom. The resulting spectra given from (11) are seen in Fig.6, and the frequencies found are 436.639, 879.691, 1321.294, 1763.093, 2207.368. We see Figure 6 that even though the signal is noise free, the true frequencies are not found. This is due to the ' zeroing'of the auto-correlation values outside the auto-correlation matrix, something that smoothes and displaces the peaks. Welch periodogram of signal, and estimated AR−spectrum

2

10

Original AR(27)

1

10

0

10

−1

10

−2

10

−3

10

−4

10

−5

10

−6

10

This is probably due to the fact that without noise, the auto-correlation matrix is singular. When the signal is corrupted with noise, the order stays the same, but the poles modelling the noise get even closer to the unit circle.

1000

2000

3000 Frequency [Hz]

4000

0 Real Part

0.5

5000

6000

1 0.8 0.6 0.4

Imaginary Part

Figure 7 shows the pole-zero plot of the noise-free A4. We see that the poles not modelling the sinusoids are fairly close to the unit circle. This will lead to problems selecting the threshold for which poles to accept. The reason for the relative high order selected compared to the true order (10) is the same as above: the zeroing of the auto-correlation values. To minimise the modellation error a higher order is necessary. The AIC function (with penalty k set to 1) is shown in figure 8. The same value is obtained for the MDL and FPE, while the eigenvalue-based criteria chooses 55 which is way too high.

0

0.2 0 −0.2 −0.4 −0.6 −0.8 −1 −1

−0.5

1

Figure 7

aic

−3.1

−3.2

−3.3

−3.4

−3.5

−3.6

−3.7

−3.8

−3.9

−4

0

Figure 8

20 of 38

10

20

30

40

50 model order

60

70

80

90

100

One of the reasons for using AR models instead of Fourier was to obtain higher resolution, and thereby to be able to resolve harmonics a semitone apart, or maybe even resolve overlapping harmonics. A 25ms synthetic signal consisting of a G4# (415.4 Hz) and an A4 (440) in white-noise with variance 0.16 was created. At a sampling rate 11025kHz we have 276 datapoints which gives us the Fourier-resolution of (1/276)*11025=39.95 Hz. This means that a standard periodogram should not be able to resolve the first harmonic (24.7 Hz apart), while the over-harmonics could be found (>49 Hz apart). Still using the YW-method to find the prediction coefficients, we analyse the signal, using a standard AIC with penalty 1 to estimate the order. Fig. 9 shows the result, and as predicted we see that the Welch-periodogram is unable to resolve the first harmonics. While not visible in this figure, the two peaks are found in the AR model, see fig.10. However, one of the poles are too far from the unit circle, and is considered as noise. This limited capability to separate signal and noise probably makes the the Yule-Walker method for estimating the poles less attractive and other methods should be Figure 9 tested.

Welch periodogram of signal, and estimated AR−spectrum

2

10

Original AR(51)

1

10

0

10

−1

10

−2

10

−3

10

−4

10

−5

10

One could also consider Burg' s algorithm since it has the same computational cost as YW but performs on general better. The problem is the phenomenon of line splitting when the order augments.

1000

2000

3000 Frequency [Hz]

4000

5000

415.30Hz + 440Hz, 25ms/276 points 1 0.8 0.6 0.4

Imaginary Part

On the other side, one could resolve all of the frequencies in the model, and from that search for possible harmonic series. However, care must be taken to avoid creation of non-existing notes, since we see from fig.10 and 7 that the poles not associated with the signal are not exactly randomly distributed.

0

0.2 0 −0.2 −0.4 −0.6 −0.8 −1 −1

Figure 10

21 of 38

−0.5

0 Real Part

0.5

1

6000

Modified covariance method, synthetic signals As reported in [Schro00], Marple' s modified covariance method holds promise of better frequency estimates than the traditional Yule-Walker method. And it was claimed that in the absence of noise, the true frequencies were found.

AIC,k=1, PEP from Modcov

0 −1 −2 −3 −4

Matlab 5.2 does not include the algorithm, but 5.3 does, and it' s called armcov.m

−5 −6 −7

The same noiseless A4 as used on the YW is tested with the ModCov algorithm. We see in fig.11 that the correct model order is found with the AIC when using the ModCov to find the PEP. The problem is Figure 11 that some of these poles are displaced more than 0.035 from the unit circle. −8 −9

−10

10

20

30

40

50 model order

60

70

80

90

A4, no noise, ModCov 1 0.8 0.6 0.4

Imaginary Part

We obtain 444.87, 899.84, 1332.10, 1764.21 and 2200.45 HZ which is even worse than YW. This could be due to round-off errors when forming the covariance matrix or inaccuracies when inverting it.

0

0.2

To improve the estimations, one could try to increase the order of the model. By doubling the order, we see that the poles are on the unit circle and give the exact frequencies. Fig.13 shows the result of adding two poles, in fact, just increasing the order to 12 gives a better result with maximum 0.12 Hz deviations, see fig.13 Figure 12 and 14. 0

−0.2 −0.4 −0.6 −0.8 −1

22 of 38

−1

−0.5

0 Real Part

0.5

1

100

Welch periodogram of signal, and estimated AR−spectrum

6

10

Original AR(12)

1 0.8

4

10

0.6 0.4

2

Imaginary Part

10

0

10

0.2 0 −0.2

−2

10

−0.4 −0.6

−4

10

−0.8 −1

−6

10

0

1000

2000

3000 Frequency [Hz]

4000

5000

6000

−1

Figure 13

−0.5

0 Real Part

0.5

1

Figure 14

Again, as with the Yule-Walker the signal with notes a semitone apart is examined (page 21). While using the ModCov to estimate the PEP, the MDL, FPE, FSIC and AIC with penalty 2 all gives order 21 which is approximately the correct order (20) for the noiseless case, but too low to model all the sinusoids in noise. Using penalty k=1 for AIC using PEPFB gives order 76, something that successfully finds all the sinusoids with less than 5 Hz error. This overestimation is typical for the AIC. On the other hand, using order 76 gives us five extra sinusoids, not in harmonic relation to those really existing. This shows that letting the order become Figure 15 too high quickly gives rise to spurious peaks which have to be ' filtered'away by some means. It is clear that this method is not optimal. If we look at fig.9 again, we see that order 51 was chosen for exactly the same signal using AICk=1 and PEPYW. In fact, this combination seem to work fine for selecting order to use with the ModCov, and was most often used when doing the transcription. Much because the calculation using the Levinson algorithm is faster than the ModCov algorithm, and thus speeds up the analysis. Welch periodogram of signal, and estimated AR−spectrum

2

10

Original AR(76)

1

10

0

10

−1

10

−2

10

−3

10

−4

10

−5

10

0

23 of 38

1000

2000

3000 Frequency [Hz]

4000

5000

6000

Synthetic signals are useful since we will have complete knowledge of the signal, but real instruments don' t create perfect sinusoids so it is interesting to do testing of order estimation on real instrument samples.

Welch periodogram of signal, and estimated AR−spectrum

4

10

Original AR(38)

2

10

0

10

Again an A4 was chosen, but this time from a flute. The number of datapoints was 308, which is about 25ms at 11kHz sample rate and this should be short enough to catch most of the notes. The Fig.17 shows the MDLk=2 using PEPFB, and looking at the periodogram in fig.16, we see that the correct order is found again (or more precisely: the correct number of sinusoids Figure 16 compared with the number of peaks). The same holds for CIC and FSIC, while AIC, FPE and the eigenvalue-based method in (16) and (17) overestimate the order. −2

10

−4

10

−6

10

−8

10

0

1000

2000

3000 Frequency [Hz]

4000

5000

6000

A4 flute − 308 points, PEPFB, MDLk=2

−550

−600

−650

The order chosen in the model is twice the order estimated. This seem to be a good balance between the correct order which is too smooth and too high order that gives spurious peaks. We see that one extra peak appears, something that is tolerable. Experiments showed that model orders of 1.5 to 2 times the number of sinusoids gave good results.

−700

−750

−800

−850

−900

0

10

20

30

40

50 model order

60

70

80

90

Figure 17 A4 flute − 308 points, ModCov, Order=38 1 0.8 0.6

Imaginary Part

0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 −1

Figure 18

24 of 38

−0.5

0 Real Part

0.5

1

100

Implementation of the transcriber The structure of the transcriber The transcriber WavToMid is completely implemented in Matlab, and takes a wav-file from the subdirectory ".\wav\" as input and gives a midi-file in ".\mid\" as output. Instrument type in the midi-file must be specified manually. For the moment the program first finds all midi notes, and then does the post-processing and writing to a MIDI file. The processing is done block-wise with each block being fixed to 250 samples (25ms). This is done to simplify the program, but it is obvious that better results can be obtained if a proper segmentation is implemented. The size of the block was chosen from the fact that at 200 BPM a sixteenth-note is 75ms long, and a 25ms window should then be able to capture most of the changes in the music. Each block is tested for silence before it is passed to frequency estimation. After the preliminary testing, it was clear that for the frequency estimation the Modified Covariance Method was the way to go. The frequencies are found by using Matlab' s tf2zp.m and calculating the angles of the poles returned. Finding the roots of the polynoms implies some eigenvalue-calculation. Whether this is possible to avoid is unknown. Model order selection was most of the time done by AICYW with penalty k=1 because of the speed advantages, but using MDLFB with penalty 2 gives us a closer estimate of the number of sinusoids and can in some cases give better results. After the frequency estimation, harmonic series are searched for in order to find potential fundamental frequencies, and afterwards converted to the corresponding midi-number. At this moment only monophonic music is supported, but an extension to polyphony should be straightforward. One problem is to decide which note belongs to which instrument. When the notes are determined some ' top-down processing'is done. Pauses and notes that are too short are removed. Reverb is removed by trying to detect whether a note is continuing to play while another note is present. Finally the midi numbers are converted to binary Midi 1.1 format and written to disk. The relations between the different tempo parameters used in the MIDI specification was not completely understood, so changing the analysis block size is not possible without some manual regulations in these parameters. The different modules in the transcriber are shown in fig.19. The shaded boxes are shared with process.m used in the initial testing, while the hatched box is non-essential for the program. It is just an early pitch-estimator calculating the most occurring distance between the frequencies found in the Fourier spectrum. It is used in the time-frequency plot of the data. The numbers in the boxes indicate in which order the functions are called. All parameters not set interactively can be found in the main script.

WavToMid. m

loadfile.m 1

ar_cov.m 3

orderselect. m 2

5

freq2mid.m

fixed2var.m

4 most_freq. m Figure 19

25 of 38

midiwrite.m

coeff2freq. m

Transcribing real music Since flute samples was used throughout the preliminary testing, a short flute solo (flute5.wav) was chosen as the reference transcribing clip. It is recorded in 11025Hz with 8 bits resolution, so the quality is average with not too many partitials present, and a small amount of reverb is present. The tempo is modest, with the shortest notes being around 260ms. To test the ability to transcribe quicker passages, some seconds of Bach' s "Badinerie" was used (flute00.wav). Here the shortest note is about 100ms, and there is almost no reverb present. Another quick passage with a lot of reverb was tested (bachfast11.wav). This is Bach' s "Partita in A Minor". A number of other small clips with different instruments was also tested to see how sensitive the transcriber was to the instrument used. Flute5.wav – a simple flute solo

Figure 20

Looking at the spectrogram in fig.20 we could expect to find three to five harmonics, and a visual comparison between this spectrogram and the time-frequency plot of the music clip was used as a benchmark when testing the order selection criteria. Additionally, when a MIDI file was made, the wav-file and the midi-file was compared by listening. In all of the following time-frequency plots, the blue circles indicate the frequencies found from the angle of the poles, while the red crosses are estimations of the pitch calculated with most_freq.m and are not used in the written MIDI-file.

26 of 38

Testing the different order selection criteria showed the same as the one-note testing: When we use PEPYW and directly using the order found, we have to use AICk=1 to avoid underestimation. In fact, this seems to be the most practical setting as it performs very well for different types of instruments and tempos. In fig.21 we see the time-frequency plot with this setting used, and it is not too far from the real spectrum, and the conversion to MIDI format is perfect. We see however that there are some spurious poles where the frequencies are changing. This effect could be reduced if the analysis windows Figure 21 are dynamically adjusted.

flute5.wav − AIC k=1, PEP

, order= 1 x estimated

YW

5000

4000

Hz

3000

2000

1000

0

50

100

150 Blocknr

200

250

flute5.wav − AIC k=1, PEPYW, order= 1 x estimated smoothed

5000

4000

3000

Hz

A more simple approach is to ' smooth'the order selection curve, taking the average between the estimated order and some of the last orders found. In fig.22 we see the result of taking the average with the two preceding orders, and the non-continuous areas are better estimated. However, this solution is probably not a good choice in polyphonic music since the orders will change more rapidly. A smoothing will then lead to instruments not detected at once because the order is too low . In this project the method seems to work fine, and was used most of the time

0

2000

1000

0

0

50

100

150 Blocknr

200

250

Figure 22

Looking at the spectrum in fig.1 again, we see that the over-harmonics are getting weaker and weaker, and are often modelled too far from the unit-circle to be chosen. This phenomenon is also present in speech, and sometimes a ' pre-flattening' filter that emphasises the higher frequencies is used [Picone93]. This filter is most often a one-tap FIR filter with a [-1,-0.4]. The problem with this filter is that the noise is boosted as well. In fig.23 we see just a minor improvement using a=-0.85. The gains might be higher when dealing with string-based instruments where the over-harmonics tend to die out Figure 23 quickly.

flute5.wav − AIC k=1, PEPYW, order= 1 x estim/smoo, flatten

5000

4000

i

Hz

3000

2000

1000

0

0

27 of 38

50

100

150 Blocknr

200

250

Using the method that seemed successful in the one-note case on page 24 ( with four times the number of sinusoids) misses many of the over-harmonics. That is simply because the signal to noise ratio (SNR) is worse in this case, and number of poles has to be further augmented in order to cope with the noise. The SNR will without doubt be important when adding even more instruments and thereby the order. Probably using a higher sampling rate (and possibly lowpass filtering to obtain the same signal bandwith as before) will help since we then will have more datapoints on which Figure 24 to base our modelling.

flute5.wav − MDL k=2, PEP , order= 2 x estimated FB

5000

4000

Hz

3000

2000

1000

0

0

50

100

150 Blocknr

200

250

This need to adjust the order according to the noise level is not good for our goal of creating a automatic transcriber. Perhaps an estimate of the SNR could be calculated, and from that select a multiplication factor to the number of sinusoids giving the best order to use. This might need the PEPFB in order to have a precise estimate of the number of sinusoids. If we knew the average number of sinusoid and the SNR, we could of course use a fixed order with some success. Eventually, the order could be estimated for bigger blocks, speeding up the analysis.

Fixed order − AR(32)

5000

4000

Figure 26 shows the output of a modified version of the transcriber, using the correlation based pitch tracker described on page 13. Even though the calculations are optimised to use as short test-sinuses as possible, the analysis is rather lengthy and using segmentation to reduce the number of blocks would help also here. The resulting MIDI file was perfect. Figure 25 Hz

3000

2000

1000

0

0

50

This method will not be able to distinguish harmonics in the same frequency band. The only hope is to apply signal models as to compare the correlation values found with those expected, assuming overlapping harmonics where the value found is higher than expected.

Figure 26

28 of 38

100

150 Blocknr

200

250

Flute00.wav – A quicker flute solo

This clip is quicker than the foregoing. No reverb and more harmonics. However, it proved to be a bit difficult for the transcriber. We see that the time-frequency plot is not too far away from the spectrogram, and the conversion to MIDI format is not too bad either. The problems that arise when the tempo is increased appear to be more related to the lack of segmentation and to the post-processing of the notes found. Both using PEPYW with AICk=1, and PEPFB with MDL or CIC works fine. Figure 27 flute00.wav − AIC k=1, PEPYW, order= 1 x estimated

5000

4000

Hz

3000

2000

1000

0

0

Figure 28

29 of 38

20

40

60

80

100 Blocknr

120

140

160

180

Some different sound files clarinet example.wav − AIC k=1, PEPYW, order= 1 x estim/smoo

5000

4000

3000

Hz

Fig.29 shows an example file from AudioWorks this time with a clarinet. Many more harmonics are present, and the chosen order is between 21 and 67. In such a file a fixed order would not work well. A spectrogram of this clip shows weak even harmonics which is typical for a clarinet. The transcription done is on par with the result from AudioWorks'own transcriber.

2000

Using PEPFB in this case is painfully slow, since the maximum allowable order must be set to 70. It is clear that an analysis window matched to each note must be used, since that would reduce the number Figure 29 of order estimations from about 470 to 35 in this example. 1000

0

0

The clip bachfast.wav is another flute solo with a lot of reverb. Energy from up to four notes can observed simultaneously. This leads to high model orders, and higher demand on the postprocessing to eliminate the echo. Both the spectrogram and the time-frequency plot is cluttered, but the result from the transcription is not too bad and is still on par with the AudioWorks transcriber. Such heavy reverb is problem for the autoregressive methods since the number of sinusoids exploses. If such files are expected to be converted successfully, some sort of echo-cancellation should be Figure 30 employed.

100

200

300 Blocknr

400

500

bachfast.wav − CIC k=3, PEPYW, order= 1.5 x estimated

5000

4000

Hz

3000

2000

1000

0

0

Oboe.wav, being of modest tempo and order, was converted easily.

Figure 31

30 of 38

100

200

300 Blocknr

400

500

Some final words Limitations of the transcriber All of the music clips tested share some common characteristics: 1. They are all created from tube resonators which means that all harmonics ' live'equally long. 2. They have a minimum frequency not too low which mean a limited number of harmonics. The reason for omitting piano and guitar music is twofold. Most importantly, these instruments are seldom monophonic. Another aspect is lower (possible) fundamental frequencies. Looking at fig.32 which is an A0 from a grand piano, we see that there are a lot of harmonics and the higher harmonics die out quickly.

Figure 32

The dying harmonics are not a problem since the transcriber needs only one harmonic to decide a note. The high number of harmonics is worse, especially if we are dealing with polyphony. This means that it is harder to find the best order, and the frequency estimations will be less accurate. To remedy this, we have to increase the number of datapoints by increasing the sampling rate and/or segmenting the music to form bigger analysis blocks. The program is using a fixed key-press (loudness) for all notes. This is seldom the case for real music. The duration of the notes are not rigourously respected since the analysis is done with fixed blocks. Additionally, the relations between the timing/tempo parameters in the MIDI specification was not completely understood, so some files experiences incorrect conversion with the standard setup. Some manual adjustments of the parameters fixes the problems. The speed is also a problem. Reducing the number of calculations by reducing the number of analysis blocks is desired. In other words, segmentation is needed. Of course, critical parts could be coded in C. Different noise levels are not accounted for. Since more noise implies higher AR models, some automatic adjustment of the model order according to noise level should be employed if less handadjusting is desired.

31 of 38

Improvement for the transcriber and ideas for future work No further work is planned in this project, at least not on a professional basis. However, some suggestions for further work are given. Without doubt, the most important modification of the transcriber is to do segmentation allowing the analysis window to cover the whole note. This has several advantages: 1. More datapoints available in the analysis window leading to better frequency- and orderestimations. 2. Avoid mixing two consecutive notes in the same analysis window. 3. The note duration will be respected in the MIDI representation. 4. The number of calculations can be reduced, since fewer order estimations is needed. 5. The attack of the note can be analysed to identify the instrument. 6. The relative loudness of each note can easily be determined. Further, implementing pkt.6 above to respect each note's loudness, and making a GUI for easier testing of different important parameters. Some ideas that requires a bit more research before integration into the transcriber are: AR modelling in sub-bands. If D. Bonacci' s research is successful we would be able to do AR modelling in sub-bands. This would enable us to use many low-order models in place of one high order model, possibly making the estimations more reliable (and faster). Adaptive sequential algorithms. Many adaptive algorithms for spectrum estimation exists, updating the spectrum for every data point arriving, and requiring only O(9m) multiplications instead of the usual O(p2) multiplications [Kalou87]. These algorithms could be used for frequency estimations in real-time, possibly adding the benefits of detecting note changes and avoiding order estimations. Directionality. A stereo signal usually contains information of spatial placement, and using for example MUSIC or Capon' s method could use this information to suppress all but one instrument, thus improving polyphonic transcription. Frequency estimations could be done simultaneously. Phase information. No work regarding the phase information in a music signal was found, not even excluding the possibility. The idea is that if there are any relation between the phases of the harmonics created in the instrument, one could be able to detect if a harmonic is mixed with a harmonic from another instrument (having a different phase).

32 of 38

Conclusion It has been demonstrated that AR frequency estimation is a plausible solution when transcribing music. The AR methods are able to provide higher resolutions than the Fourier counterparts, something that is important when the number of simultaneous notes increases. The frequency estimates from Principal Component Frequency Estimation using the Modified Covariance Method are reliable even in the presence of noise. Even if some of the harmonics are completely masked. Fig.33 shows two flutes played simultaneously, where the C5 is completely masked. Having 1000 datapoints and using MDL with PEPFB gives almost the correct number of sinusoids, and using order 4 times the number of sinusoids actually makes it possible to find the hidden C5. This is a promising result when considering polyphonic transcription. C4+C5, 91ms, MDLk=2, PEPFB, 2x 1 0.8 0.6

Imaginary Part

0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 −1

−0.5

0 Real Part

0.5

1

Figure 33

Order estimation is crucial for optimal performance. Best order is not equal to the 2*(number of real sinusoids), but somewhat higher depending on the amount of noise. Using the PEPFB together with the information criteria MDL or CIC seems to give best estimate of the number of sinusoids. Then using an estimate of the noise level to find a multiplicator for the order found to get the best model order seems to be a way to go in order to cope with different SNR. However, calculating all the orders up to the maximum allowable order for every block using the modified covariance method is slow. A faster and ' less correct' , but well performing method is utilised in the project, namely using the order found from AICk=1 with PEPYW. This works remarkably well even for different types of instruments. A monophonic transcriber has been built taking a wav-file as input and giving a MIDI-file as output. The program is performing on par with the commercial available monophonic transcribers. The program has been build with a possible extension to polyphonic operation in mind.

33 of 38

A – Converting between Note, MIDI and Frequency The A with MIDI number 21 is called A0, while the next is called A1 etc. Similar for all the other notes. The table is created from equation 3, where the MIDI number equals k+69. Note name C

MIDI Nr.

MIDI Freq.[Hz] Nr. 0

8.176

12

Freq.[Hz] 16.352

MIDI Nr. 24

MIDI Freq.[Hz] Nr.

MIDI Freq.[Hz] Nr.

32.703

65.406

36

48

MIDI Freq.[Hz] Nr. 130.813

60

Freq.[Hz] 261.626

Db

1

8.662

13

17.324

25

34.648

37

69.296

49

138.591

61

277.183

D

2

9.177

14

18.354

26

36.708

38

73.416

50

146.832

62

293.665

Eb

3

9.723

15

19.445

27

38.891

39

77.782

51

155.563

63

311.127

E

4

10.301

16

20.602

28

41.203

40

82.407

52

164.814

64

329.628

F

5

10.913

17

21.827

29

43.654

41

87.307

53

174.614

65

349.228

Gb

6

11.562

18

23.125

30

46.249

42

92.499

54

184.997

66

369.994

G

7

12.250

19

24.500

31

48.999

43

97.999

55

195.998

67

391.995

Ab

8

12.978

20

25.957

32

51.913

44

103.826

56

207.652

68

415.305

A

9

13.750

21

27.500

33

55.000

45

110.000

57

220.000

69

440.000

Bb

10

14.568

22

29.135

34

58.270

46

116.541

58

233.082

70

466.164

B

11

15.434

23

30.868

35

61.735

47

123.471

59

246.942

71

493.883

C

72

523.251

84

1046.502

96

2093.005

108

4186.009

120

8372.018

Db

73

554.365

85

1108.731

97

2217.461

109

4434.922

121

8869.844

D

74

587.330

86

1174.659

98

2349.318

110

4698.636

122

9397.273

Eb

75

622.254

87

1244.508

99

2489.016

111

4978.032

123

9956.063

E

76

659.255

88

1318.510

100

2637.020

112

5274.041

124

10548.082

F

77

698.456

89

1396.913

101

2793.826

113

5587.652

125

11175.303

Gb

78

739.989

90

1479.978

102

2959.955

114

5919.911

126

11839.822

127

12543.854

G

79

783.991

91

1567.982

103

3135.963

115

6271.927

Ab

80

830.609

92

1661.219

104

3322.438

116

6644.875

A

81

880.000

93

1760.000

105

3520.000

117

7040.000

Bb

82

932.328

94

1864.655

106

3729.310

118

7458.620

B

83

987.767

95

1975.533

107

3951.066

119

7902.133

Table 4

34 of 38

B – Matlab code for the transcriber

35 of 38

References REF

NAME

TITLE

PUBL

YEAR

[Bisc92]

C.H.Bischof, M.Shroff

"On Updating Signal Subspaces"

[Broer00]

P.M.T.Broersen

"Finite Sample IEEE 2000 Criteria for Trans.Sig.Proc,Vol.48 Autoregressive Order ,No.12 Selection

[Brown99]

J. Brown

"Computer identification of musical instruments using pattern recognition with cepstral coefficients as features"

MIT Media Labs

1999

[Dick94]

J.R.Dickie, A.K.Nandi

"On the Performance of AR Model Order Selection Methods"

Signal Processing VII; Theories and Applications

1994

[Djuric96]

P.M.Djuric

"A Model Selection Rule for Sinusoids in White Gaussian Noise"

1996 IEEE Trans.Sig.Proc,Vol.44 ,No.7

[Djuric98]

P.M.Djuric

"Asymptotic MAP Criteria for Model Selection"

1998 IEEE Trans.Sig.Proc,Vol.46 ,No.10

[Durne98]

M.Durnerin

Operation ASPECT

[Feldman]

J. Feldman

"Derivation of the Wave Equation"

http://www.math.ubc. ca/~feldman/apps/wa ve.pdf

[Fitch00]

J.Fitch, W.Shabana

"A Wavelet-based Pitch Detector For Musical Signals"

University of Bath, UK

[Fuchs88]

J.J.Fuchs

"Estimating the Number of Sinusoids in Additive White Noise"

1988 IEEE Trans.ASSP,Vol.36,N o.12

[Jehan97]

T. Jehan

"Musical Signal Parameter Estimation"

http://www.cnmat.ber 1997 keley.edu/~tristan/Th esis/thesis.html

[Kalou87]

N.Kalouptsidis, S.Theodoridis

"Fast Adaptive L-S IEEE 1987 Algorithms for Power Trans.ASSP,Vol.35,p Spectral Estimation" p.95-108 May

[Karaj99]

M.Karjalainen, T.Tolonen

"Multi-Pitch and Periodicity Analysis Model for Sound Separation and Auditory Scene Analysis"

http://citeseer.nj.nec.c 1999 om/411704.html

[Kashino95]

Kashino, Nakadai, Kinoshita, Tanaka.

"Organization of Hierarchical Perceptual Sounds"

http://citeseer.nj.nec.c 1995 om/27731.html

36 of 38

IEEE 1992 Trans.Sig.Proc,Vol.40 ,No.1

1998

REF

NAME

TITLE

PUBL

YEAR

[Kashino98]

Kashino, Nakadai, Kinoshita, Tanaka.

"Application of Bayesian Probability Network to Musical Scene Analysis"

[Kay88]

S.M.Kay

"Modern Spectral Prentice Hall Estimation, Theory & Application"

1988

[Klapuri01A]

A. Klapuri

"Means of Integrating Audio Content Analysis Algorithms"

2001

[Klapuri01B]

A. Klapuri

"Multipitch Estimation And Sound Separation By The Spectral Smoothness Principle"

2001

[Klapuri98]

A. Klapuri

"Automatic transcription of music"

Http://www.cs.tut.fi/s 1998 gn/arg/music/klapthes .pdf.zip

[Lee92]

H.B.Lee

"Eigenvalues and Eigenvectors of Covariance Matrices for Signals Closely Spaced in Frequency

1992 IEEE Trans.Sig.Proc,Vol.40 ,No.10

[Mallat99]

S.Mallat

"A Wavelet Tour of Signal Processing"

Academic Press

[Martin96]

K. Martin

"Automatic Transcription of Simple Polyphonic Music: Robust Front End Processing"

1996 Third Joint Meeting of the Acoustical Societies of America and Japan, ftp://sound.media.mit. edu/pub/Papers/kdmTR399.ps.gz

[Martin98]

K. Martin

"Musical instrument identification: A pattern-recognition approach"

136 th meeting of the Acoustical Society of America,

1998

[McNab]

R. J. McNab , L. A. Smith, I. H. Witten, C. L. Henderson ,S. Jo Cunningham

"Towards the Digital Music Library: Tune Retrieval from Acoustic Input"

University of Waikato, Hamilton, New Zealand.

1996

[Picone93]

J.W.Picone

"Signal Modeling Proc. of IEEE, Sept. Techniques in Speech 1993, p1215-1247 Recognition"

1993

[Proakis]

J.G.Proakis, D.G.Manolakis

Prentice Hall "Digital Signal Processing, principles, algorithms, and applications"

1996

[Schro00]

T. von Schroeter

"Auto-regressive spectral line analysis of piano tones"

2000

[Terhardt]

E. Terhardt

"Psychoacoustics related to musical perception"

37 of 38

Http://citeseer.nj.nec. 1998 com/kashino98applica tion.html

http://www.mmk.ei.tu m.de/persons/ter.html

1999

REF

NAME

TITLE

PUBL

YEAR

[Vercoe97]

B.L.Vercoe, W.G.Gardner, E.D.Schreirer

"Structured Audio: Creation, transmission, and Rendering of Parametric Sound Representations"

Proc. of IEEE, May 1998, p922-940

[Wax85]

M.Wax, T.Kailath

"Detection of Signals by Information Theoretic Criteria"

IEEE 1985 Trans.ASSP,Vol.33,N o.2

[WolfeA]

J. Wolfe

"The University of New South Wales, Australia – Music acoustics group"

http://www.phys.unsw .edu.au/music/

[WolfeB]

J. Wolfe

"How harmonic are harmonics"

http://www.phys.unsw .edu.au/~jw/harmonic s.html

38 of 38

1998

Automatic Music Transcription using Autoregressive ...

Jun 14, 2001 - indispensable to mix and manipulate the necessary wav-files. The Matlab ..... the problems related to automatic transcription are discussed, and a system trying to resolve the ..... and then deleting a certain number of samples.

3MB Sizes 1 Downloads 257 Views

Recommend Documents

Polyphonic music transcription using note event modeling - CiteSeerX
Oct 16, 2005 - Joint Conference on Artificial Intelligence (IJCAI), vol. 1,. Aug. 1995, pp. ... [9] L. R. Rabiner, “A tutorial on hidden markov models and selected ...

Polyphonic music transcription using note event modeling - CiteSeerX
Oct 16, 2005 - music databases, and interactive music systems. The automatic transcription of real-world music performances is an extremely challenging task ...

Automatic transcription of Broadcast News
recognition system for telephone speech. The var- ..... Table 4. Comparison of the BIC approach with the thresholding approach on the 1997 evaluation subset.

Using Fractional Autoregressive Integrated Moving Average (FARIMA ...
Using Fractional Autoregressive Integrated Moving Averag ... arture (Interior) in Sulaimani International Airport.pdf. Using Fractional Autoregressive Integrated ...

Automatic Score Alignment of Recorded Music - GitHub
Bachelor of Software Engineering. November 2010 .... The latter attempts at finding the database entries that best mach the musical or symbolic .... However, the results of several alignment experiments have been made available online. The.

Automatic Campus Network Management using GPS.pdf ...
Automatic Campus Network Management using GPS.pdf. Automatic Campus Network Management using GPS.pdf. Open. Extract. Open with. Sign In.

AUTOMATIC PITCH ACCENT DETECTION USING ...
CRF model has the advantages of modeling the relations of the sequential labels and is able to retain the long distance dependency informa- tion. Although ..... ECS-95-001,. Bonston University, SRI International, MIT, 1995. [8] R.-E. Fan, P.-H. Chen,

Automatic speaker recognition using dynamic Bayesian network ...
This paper presents a novel approach to automatic speaker recognition using dynamic Bayesian network (DBN). DBNs have a precise and well-understand ...

Time-Space Ensemble Strategies for Automatic Music ...
decision. In our approach we employ audio signal segmentation in time ... multi-class problems – producing complex decision surfaces – like decision trees, .... The employed classification framework is based on the Weka Datamining Tool.

a feature selection approach for automatic music genre ...
format [14]. The ID3 tags are a section of the compressed MP3 audio file that con- ..... 30-second long, which is equivalent to 1,153 frames in the MP3 file format. We argue that ...... in machine learning, Artificial Intelligence 97 (1997) 245–271

A Machine Learning Approach to Automatic Music ... - Semantic Scholar
by an analogous-to-digital converter into a sequence of numeric values in a ...... Proceedings of the 18th. Brazilian Symposium on Artificial Intelligence,.

Using Music to Teach Microeconomics
discrimination is indeed profitable to business, which is why firms engage in the ... Couldn't buy the seed to plant this spring and the Farmers Bank foreclosed ... illegal drug sales was so great in Los Angeles that no one could make a profit.

Mixtures of Sparse Autoregressive Networks
Given training examples x. (n) we would ... be learned very efficiently from relatively few training examples. .... ≈-86.34 SpARN (4×50 comp, auto) -87.40. DARN.

Multivariate Portmanteau Test For Autoregressive ...
Mar 24, 2012 - The paper provides a method of correction for heteroskedastic errors ... by linear combination of past p values Xt−1,..., Xt−p and an error ϵt .

Automatic Problem Decomposition using Co-evolution ...
Problem Decomposition. •. Interdependencies between subcomponents. •. Credit Assignment. •. Maintenance of diversity. •. Adding subcomponents ...

Using Human Perception for Automatic Accent ...
Using Human Perception for Automatic Accent Assessment. Freddy William, Abhijeet Sangwan, and John H. L. Hansen. 1. Center for Robust Speech Systems (CRSS), Eric Jonsson School of Engineering,. University of Texas at Dallas, Richardson, Texas, U.S.A.

ATGen: Automatic Test Data Generation using Constraint Logic ...
ATGen: Automatic Test Data Generation using Constraint Logic Programming and Symbolic Execution.pdf. ATGen: Automatic Test Data Generation using ...