A Different Approach to Independent Component Analysis to Separate Incomplete Wave Signals I. A. Ismail, S.I. Zaki, A. A. Soliman, M. G. Awad One of the constraints of using ICA in BSS is that the number of observed linear mixtures must be at least as large as the number of independent components [4,5,6,7]. In this paper, a new technique is introduced to separte a mixture of signals to their sources when the number of mixture signals is less than there original sources , i.e. when we have a lack of data.

Abstract - The objective of this research is to demix a number of input data vectors or streams that are mixed and demix them back to their original dimension. In this paper the independent component analysis (ICA) was adopted with the classical Gram-Schmidt orthogonalization process. Index terms - Independent Component Analysis, wave separation, Gram-Schmidt orthogonalization process

2.

Independent Component Analysis A central problem in signal and image processing, is finding a suitable representation of the data, by means of a suitable transformation [4]. Let us denote by X an m-dimensional random variable; the problem is then to find some transform f so that the n-dimensionally transformed T S = (s 1 , s 2 , ..., s n ) given by: S=f(X) (1) has some desirable properties. In most cases, the representation is thought of as a linear transform of the observed variables, i.e., S=WX (2) where W is a matrix to be determined. Linear transformation makes the problem computationally and conceptually simpler, and facilitates the interpretation of the results [4]. Several principles and methods have been developed to find a suitable linear transformation. This comprises principal component analysis (PCA). The optimality of any of these transformations may be defined in the sense of optimal dimension reduction, statistical of the resulting components si, simplicity of the transformation W [4]. Recently, the Independent Component Analysis (ICA) method has gained wide spread attention. As the name implies, the basic goal is to find a transformation in which the components si are statistically as independent from each other as possible [4, 5,7]. 2.1 Definition of ICA ICA is a statistical and computational technique for revealing hidden components that underlie sets of random variables, measurements, or signals. ICA defines a generative model for the observed multivariate data, which is typically given as a large database of samples. In this model, the data variables are assumed to be linear or nonlinear mixtures of some unknown hidden variables, and the mixing system is also unknown. The latent variables are assumed non-Gaussian and mutually independent and they are called the independent components of the observed data. These independent components, also called sources or factors, can be found by ICA [4, 5].

1. INTRODUCTION Recovering sources from a mixture of several others is an important topic in signal processing today. For example when some people try to communicate while at a loud party. In this example, the voice of one person is mixed with many other sounds of the party. Human ear and brain can easily extract the wanted voice, but it is much more difficult for machines. The solution to this problem is called Blind Source Separation (BSS). Blind source separation (BSS) is a technique for estimating original source signals using only sensor observations that are mixtures of the original signals [1]. The word “blind” is used because we have no prior knowledge about the statistics of the source in general [2]. There are many applications of blind signal processing in science and technology especially in wireless communication, noninvasive medical diagnosis, geophysical exploration, and image enhancement and recognition. Acoustic examples include the signals from several microphones in a sound field that is produced by several speakers (the so-called “cocktail-party” problem) or the signals from several acoustic transducers in an underwater sound field from the engine noises of several ships (the sonar problem). Other applications are in the area of noninvasive medical diagnosis and biomedical signal analysis, such as EEG, MEG, and ECG [3]. Independent component analysis (ICA) is one useful BSS methods for extracting signal components from linear mixed signal sources without a priori knowledge [2]. I. A. Ismail is the dean of Faculty of Computers and Informatics, El-Zagazig University, El-Zagazig, Egypt. S. I. Zaki is the head of Mathematics Department, Faculty of Science, Suez Canal University, Ismailia. Egypt. A. A. Soliman is with Mathematics Department, Faculty of Education, Suez Canal University, Arish. Egypt. M. G. Awad is with Mathematics Department, Faculty of Education, Suez Canal University, Arish. Egypt. E-mail: [email protected], phone: +20 121 737 710

67

ICA is an extension to principal component analysis PCA but it is much more powerful technique than the PCA itself [9]. It is capable of finding the underlying factors or sources where these classic methods fail completely. In many cases, the measurements are given as a set of parallel signals or time series; the term blind source separation is used to characterize this problem. Typical examples are mixtures of simultaneous speech signals that are picked up by several microphones, brain waves recorded by multiple sensors, interfering radio signals arriving at a mobile phone, or mixed time series that have been observed from some source. To define ICA we assume to observe n linear mixtures x1,...,xn of n independent components (3) x j = a j1 s 1 + a j2 s 2 + ... + a jn s n ∀ j

2.2 ICA learning algorithm The computations involved in the algorithm are simple and they may be summarized as follows: 1. Initialize the synaptic weights of the networkW (0) , as small random values at time n = 0 . Assign small positive value to the learning rate parameter η . 2. For n = 1 compute Y (n ) = W (n ) X (n ) W (n ) = W (n − 1) + μ[I − Φ (Y )Y ')W (n − 1) where X is the input matrix , Y is the output matrix, Φ is the activation function of the algorithm. 3. Increment n by 1 , go to step 2 , and continue until the synaptic weights matrix W (n ) reaches its steady-state values [4, 5, 8, and 9].

In the ICA model, we assume that each mixture xj as well as each independent component sk is a variate. Let X be the random matrix whose rows are the mixtures of x1, ..., xn, and likewise S be the random matrix with rows s1, ... , sn which are the original vectors, i.e., before mixing. Let the mixing matrix be denoted by A with elements aij. All vectors are understood to be column vectors. The above mixing model is written as X=AS (4) If we denote the columns of matrix A by aj the model can be written as

3. Gram-Schmidt process The Gram-Schmidt process is method of orthogonalizing a set of vectors in an inner product space, most commonly the Euclidean space Rn [10]. Orthogonalization in this context means the following: Starting with given vectors v1,...,vk which are linearly independent and we want to get mutually orthogonal vectors u1,...,uk which generate the same subspace as the vectors v1,...,vk [10]. 4. Applying Gram-Schmidt to incomplete data We can use the gram Schmidt process to produce a new set of data from an existing one. We thought that, if we have for example two vectors, we can use the Gram-Schmidt process to produce another two vectors. The new vectors have properties similar to the original vectors. In practical if we have four microphones (sensors) which record some wave signals in a party (cocktail party problem) after recording the signals we discovered that one of the microphone is broken, we can use the Gram-Schmidt process to produce three vectors from the three already existing ones and treat each new vector as a recorded signal from a different microphone. In practice using one ICA network to find the separated signals does not give a satisfactory results, so using recurrent ICA network to find the solution is suggested (i.e. we use certain number of iterations to do the separation with the ICA network then we use the output of the network as the input of the network again, and so on repetitively until we reach the steady state). The following figure shows the suggested technique used in this paper

n

X =

∑a s i

i

(5)

i=1

The Infomax approach [5] to ICA is employed here. The unknown mutually time independent source signals S are linearly mixed by the unknown mixing matrix A to produce the signals as demixing X . Our aim is to find the demixing matrix W = A -1 which demixes the X matrix to produce output matrix Y → S, and obtain the independent component simply by: Y=WX (6) We now try to obtain the original source signals S knowing that they are not identifiable in the statistical sense (i.e. we do not know the original signals) [5, 8]. Except for some permutation of indices we can obtain the components cisi, where the constants ci, i=1,2,…,m are definite and do not simultaneously vanish, the source signals are identifiable in this sense. In other words, it is possible to find a demixing matrix W whose individual rows are a rescaling and permutation of those of the mixing matrix A. that is the solution may be expressed in the form: Y = W X = W AS → DPS (7) where D is a nonsingular diagonal matrix and P is a permutation matrix [5]. So our goal is to find the matrix y such that it coincides with permutations of S except for a scale factor. The solution W is the matrix which gives independent components in the outputs.

Fig. 1: The suggested technique used to separate incomplete wave signals

68

These signals are mixed using a mixing matrix. Random mixing matrix is used to mix these signals. The 8 mixed signals are shown in the following figure

4. Computer Experiments In our computer experiments 8 sound source signals were used; each signal consists of 5x105 points. These signals are shown in the following figure

Fig. 3: The mixed signals In the first experiment 4 mixed vectors have been used only (vectors 1-4) from this data set (some data is lost or corrupted). Gram-Schmidt method is used to produce other four vectors with the aid of the existing ones. The 8 vectors (the four mixed and the four GramSchmidt) is used as the input vectors to the ICA

Fig. 2: The source signals used to perform computer experiments

69

network. The eight vectors are shown in the following figure

The separated signals are shown in the following figure

Fig. 5: The separated signals from the first experiment In the second experiment six mixed signals (vectors from 1-6) out of the given eight is used and the GramSchmidt method is used to produce 6 new vectors using these ones. The first two new vectors are selected (or randomly any other two vectors from this new set). These two signals are added to the six mixed vectors and the new data set (the eight vectors) is used as the input of the

Fig. 4: The input signals to the first experiment

70

The separated signals are shown in the following figure

ICA algorithm. The input vectors are shown in the following figure

Fig. 7: The separated signals from the second experiment In the third experiment seven mixed sources have been used and the Gram-Schmidt method is used to produce another seven vectors one of the vectors is randomly selected from the resulting Gram-Schmidt

Fig. 6: The input signals to the second experiment

71

set and then added to the seven vectors. The new set of eight vectors to is used as input to the ICA network. The input vectors are shown in the following figure

Fig. 9: The separated signals from the third experiment

Fig. 8: The input signals to the third experiment

The following table shows a correlation test results between the separated signals and the original signals from each experiment.

The separated signals are shown in the following figure

72

y1 y2 y3 y4 y5 y6 y7 y8

y1 y2 y3 y4 y5 y6 y7 y8

y1 y2 y3 y4 y5 y6 y7 y8

The first experiment s1 s2 s3 s4 s5 s6 s7 0.60 0.01 0.13 0.17 0.13 0.70 0.24 0.46 0.65 0.03 0.01 0.23 0.47 0.16 0.10 0.02 0.95 0.09 0.12 0.05 0.19 0.18 0.01 0.03 0.95 0.00 0.13 0.03 0.67 0.00 0.01 0.09 0.05 0.72 0.04 0.02 0.01 0.44 0.07 0.17 0.28 0.85 0.82 0.00 0.01 0.08 0.04 0.02 0.05 0.01 0.99 0.01 0.02 0.00 0.05 0.01 The second experiment s1 s2 s3 s4 s5 s6 s7 0.00 0.95 0.01 0.01 0.00 0.31 0.02 0.01 0.00 0.00 0.97 0.22 0.00 0.00 0.95 0.00 0.01 0.02 0.00 0.18 0.01 0.50 0.02 0.01 0.00 0.00 0.88 0.19 0.00 0.00 0.99 0.12 0.42 0.01 0.61 0.00 0.04 0.01 0.04 0.02 0.00 0.99 0.45 0.00 0.02 0.03 0.14 0.89 0.00 0.20 0.00 0.95 0.07 0.03 0.00 0.01 The third experiment s1 s2 s3 s4 s5 s6 s7 0.09 0.44 0.96 0.01 0.28 0.03 0.02 0.55 0.02 0.03 0.85 0.01 0.06 0.17 0.05 0.16 0.05 0.00 0.83 0.08 0.07 0.91 0.01 0.02 0.95 0.02 0.14 0.01 0.04 0.83 0.01 0.01 0.09 0.02 0.00 0.02 0.01 0.56 0.06 0.01 0.15 0.89 0.52 0.03 0.14 0.05 0.05 0.02 0.07 0.13 0.29 0.03 0.01 0.23 0.82 0.04

[4] A. Hyvärinen, "Survey on independent component analysis". Neural Computing Surveys, 2:94–128, 1999. [5] S. Haykin, "Neural networks: a comprehensive fundation." (2nd edition) Upper Saddle Rever, New Jersey: Prentice Hall 1999. [6] T.-W. Lee, M. Girolami, A.J. Bell, and T.J. Sejnowski. "A unifying information-theoretic framework for independent component analysis". International Journal on Mathematical and Computer Models, 1999 [7] A. Hyvärinen. Survey on Independent Component Analysis. Neural Computing Surveys 2:94--128, 1999. [8] S.amari , C. Cichocki , and H.H. Yang. "A new learning algorithm for blind signal separation". In: Advances in Neural Information Processing systems 8, Editors D. Touretzky, M. Mozer, and m. Hassclmo, pp.757-763, MIT Press, Cambridge MA,1996. [9] P. Comon. "Independent component analysis - a new concept". Signal Processing, 36:287-314, 1994 [10] H. Taub, D. L. Schilling "Principals Of Communications Systems", (2nd edition) McGraw-Hill Book Company 1986.

s8 0.34 0.19 0.46 0.02 0.25 0.05 0.25 0.03 s8 0.01 0.00 0.41 0.41 0.00 0.01 0.00 0.02

Prof. I. A. Ismail is the dean of Computers and Informatics collage, Zagazig University, Egypt. He was born on March 7, 1946. He received the B.S. degree in pure mathematics / physics from Cairo University, Egypt in 1967, the M.Sc. Degree and the Ph.D. degree in Signal Processing from the Cairo University, Egypt in 1971 and 1976, respectively.

s8 0.05 0.05 0.04 0.25 0.03 0.07 0.94 0.03

Prof. S. I. ZAki was born in Cairo, Egypt in 1954. He obtained his BSc in Mathematics from Cairo University, Egypt. Also obtained his MSC in Computational Mathematics from Zagazig University, Egypt. Then obtained his PHD from UCNW, UK in Computational Science. He is currently Professor of Computational Science and head of Mathematics and Computer Science, Science Collage, Suez Canal University for more than eight years. His research interests are in the area of Computational Science mainly modelling and simulations and in the area of Computer Science, mainly Data Processing and information Systems. Dr. A. A. Soliman, is with the Department of Mathematics, Faculty of Education, Suez Canal University, Arish, Egypt. He was born on Sep. 20, 1964. He received the B.Sc. Degree in Pure Mathematics form Menoufia University, Egypt in 1988, MSc. And Ph.D. degree in pure mathematics from the Menoufia University, Egypt in 1994 and 1998 respectively.

5. Conclusions In this paper the Gram-Schmidt process was applied to the incomplete input data of the ICA network. This technique enables us to make up of the missing signals and continue the separation process. To avoid the divergence of using one ICA network, using recurrent ICA network was suggested to find the separated signals; this technique is used until the signals reach the steady state. Comparing the separated signals in each experiment to the source signals they were found that they acceptably agree. 6. References [1] S. Winter, H. Sawada, S. Makino "Geometrical Understanding of the PCA Subspace Method for Overdetermined Blind Source Separation". Conf. Rec. Intl. Workshop on Acoustic Echo and Noise Control (IWAENC), pp. 231-234, Kyoto, Japan, Sep. 2003 [2] A. Hyvärinen, J. Karhunen, and E. Oja, "Independent Component Analysis", John Wiley & Sons, 2001. [3] S. Amari and A. Cichocki, "Adaptive blind signal processing: Neural network approaches", Proceedings of the IEEE, vol. 86, no. 10, pp. 2026-2048, Oct. 1998.

M.G. Awad, is with the Department of Mathematics, Faculty of Education, Suez Canal University, Arish, Egypt. He was born on March 22 , 1977. He received the B.Sc. Degree in Scientific Computations From Suez Canal University in 1998, and the M.Sc. degree in Computer Science from Suez Canal University in 2005.

73