A Novel Type-2 Fuzzy Membership Function: Application to the Prediction of Noisy Data Mojtaba Ahmadieh Khanesar, Mohammad Teshnehlab, Erdal Kayacan and Okyay Kaynak Abstract— A novel, diamond-shaped type-2 fuzzy membership function is introduced in this study. The proposed type2 fuzzy membership function has certain values on 0 and 1, but it has some uncertainties for the other membership values. It has been shown that the type-2 fuzzy system using this type of membership function introduced in this study has some noise reduction property in the presence of noisy inputs. The appropriate parameter selection to be able to achieve noise reduction property is also considered. A hybrid method consisting of particle swarm optimization (PSO) and gradient descent (GD) algorithm is used to optimize the parameters of the proposed type-2 fuzzy system. PSO is a derivative-free optimizer, and the possibility of the entrapment of this optimizer in local minimums is less than the gradient descent method. The proposed type-2 fuzzy system and the hybrid parameter estimation method are then tested on the prediction of a noisy, chaotic dynamical system. The simulation results show that the type-2 fuzzy predictor with the proposed novel membership functions shows a superior performance when compared to the other existing type-2 fuzzy systems in the presence of noisy inputs.

I. INTRODUCTION The type-2 fuzzy sets (T2FSs) are the extension of the ordinary type-1 fuzzy sets which aim to model the uncertainty in the rule base of the fuzzy systems. These types of membership functions are themselves fuzzy, and they have more degrees of freedom. The additional degrees of freedom make it possible to directly model uncertainties. there are different sources of uncertainty in the rule base of fuzzy system: 1) The uncertainty caused by the precision of the sensors, 2) The meanings of the words that are used in the antecedents and consequents of rules can be uncertain [1], 3) Consequents may have a histogram of values associated with them, especially when knowledge is extracted from a group of experts who do not all agree [1], 4) The data that are used to tune the parameters of a fuzzy logic system may also be noisy. Mojtaba Ahmadieh Khanesar is with the Faculty of Electrical Engineering, Control Department, K. N. Toosi University of Tech., Tehran, Iran

[email protected] Mohammad Teshnehlab is with the Faculty of Electrical Engineering, Control Department, K. N. Toosi University of Tech., Tehran, Iran

[email protected] Erdal Kayacan is with the Department of Electrical and Electronics Engineering, Bogazici University, Istanbul, Turkey

[email protected] Okyay Kaynak is with the Department of Electrical and Electronics Engineering, Bogazici University, Istanbul, Turkey

[email protected]

978-1-4244-7230-7/10/$26.00 ©2010 IEEE

5) Uncertainty caused by some unvisited data which the fuzzy system does not have any predefined rules for that. The performance of the type-2 fuzzy sets in the presence of measurement noise has been considered in a number of different papers. In [2] the effects of measurement noise in type-1 and type-2 fuzzy logic controllers are simulated to perform a comparative analysis of the systems’ response in the presence of uncertainty. It is concluded that using a type2 fuzzy logic controller in real world applications which exhibit measurement noise can be a good option. In [3] type-2 fuzzy logic theory is applied to predict Mackey-Glass chaotic time-series with uniform noise. The comparison between type-1 and type-2 fuzzy systems shows the superiority of type-2 fuzzy systems in the presence of noise at the inputs. In [4] the type-2 fuzzy system is used with a sliding mode controller to control a nonlinear dynamical plant. The effect of measurement noise is considered in this paper, and type-2 fuzzy system outperforms than its type-1 counterpart. The parameter estimation of the premise parts of the fuzzy type-2 models is relatively complex. To date several methods are introduced to optimize the premise parts of the type-2 fuzzy models. Examples of such approaches are the Singular Values Decomposition (SVD)-QR [5], gradient descent [6] and online clustering method [7]. In this study, a novel type-2 membership function with certain values at both full-firing and non-firing parts and uncertain values in the other grades of the membership functions is introduced. The noise reduction property of this membership function is then investigated. The effect of noise in the rule base of the type2 fuzzy systems using this type of membership function is discussed. In addition, in order to optimize the parameters of the premise part of the type-2 fuzzy system, particle swarm optimization (PSO) algorithm is used. Since the premise part of the type-2 fuzzy system is not differentiable, the use of PSO for premise part in this case is recommended. A gradient based method is used to optimize the parameters of the consequent part of the type-2 fuzzy system. II. T HE S TRUCTURE OF T YPE -2 TSK F UZZY S YSTEM In this study a type-2 fuzzy membership function is considered for the premise part and crisp numbers for the parameters of the consequent part. This fuzzy model is known as A2-C0 TSK fuzzy model. The general structure of an A2-C0 TSK model is as follows [8]: Ri : I f x1 is F˜1i and x2 is F˜2i and · · · xn is F˜ni T hen yi = ai0 + ai1 x1 + ai2 x2 + ... + ain xn

(1)

where i = 1, ..., M, F˜ ji represents interval type-2 fuzzy set of the input state j in rule i, (x1 , ..., xn ) are the inputs, ai0 , ai1 , ai2 , ..., ain are the coefficients of the output polynomial for the rule i (and hence are crisp numbers), yi is the output of the ith rule, and M is the number of the rules. The rules above allow us to model the uncertainties encountered in the antecedents. The final output of this model is given as: ∫

∫

YT SK/A2−C0 =

1 f 1 ∈[ f 1 f ]

...

M f M ∈[ f M f ]

1/

i i ∑M i=1 f y M ∑i=1 f i

(2)

i

where f and f i are given by: f i = µ F˜ i (x1 ) ∗ ... ∗ µ F˜ i (xn )

(3)

n

1

i

f = µ F˜ i (x1 ) ∗ ... ∗ µ F˜ni (xn )

(4)

1

Each rule can be represented as an interval. The inference engine in this study is adopted from [8], where a novel inference mechanism which can be formulated in a closed form is introduced. The deffizifier are based on the weighted average of the each rule. Three different formulas are introduced in [8], the third formula used in that paper is in the following form: i

YT SK =

and al,i , ar,i , ul,i , ll,i , ur,i , lr,i are positive. To have a less number of parameters, it is also possible to use: ul,i = ur,i , ll,i = lr,i . In this way, the number of the parameters in this type of membership function can be decreased to four. The shape of the proposed membership function is shown in Fig. 1. In order to study the effect of noise in the rule base of the fuzzy system constructed using this membership functions, a type-2 fuzzy system with one input and two membership functions is considered. The values for the first membership functions are considered as: al,i = ar,i = 1, ϕ = 0 and the values of ul,i = ll,i , ur,i = lr,i are some variable in the interval of [0,1]. The second membership function is considered as:

∑M i=1 f yi M i ∑M i=1 f + ∑i=1 f

i

+

i ∑M i=1 f yi M i ∑M i=1 f + ∑i=1 f

i

(5)

The firing part of the ith rule of the fuzzy system can be written as: i fi + f (6) ri = i M i ∑M i=1 f + ∑i=1 f III. T HE N OVEL D IAMOND T YPE I NTERVAL F UZZY M EMBERSHIP F UNCTION The proposed novel membership function has certain values on 0 and 1, and the uncertainty in the intermediate values. This kind of membership function can be used when the full belonging and non belonging parts are exactly known but the intermediate values are not certainly known. The ith membership functions of the input x is considered as:  ϕ +ϕ  (al,i + ul,i ) (x − ϕi−1 ) ϕi−1 ≤ x ≤ i 2 i−1   ( )  ϕi +ϕi−1 1 + al,i − ul,i (x − ϕi ) ≤ x ≤ ϕi 2 µ¯ i = (7) ϕi +ϕi+1  ϕ ϕ 1 − (a − u ) (x − ) ≤ x ≤ i i r,i r,i  2   ϕi +ϕi+1 (ar,i + ur,i ) (x − ϕi+1 ) ≤ x ≤ ϕi+1 2  ϕ +ϕ  (al,i − ll,i ) (x − ϕi−1 ) ϕi−1 ≤ x ≤ i 2 i−1   ( )  ϕi +ϕi−1 1 + al,i + ll,i (x − ϕi ) ≤ x ≤ ϕi 2 µi =  1 − (ar,i + lr,i ) (x − ϕi ) ϕi ≤ x ≤ ϕi +2ϕi+1    ϕi +ϕi+1 (ar,i − lr,i ) (x − ϕi+1 ) ≤ x ≤ ϕi+1 2 where:

ϕi−1 = ϕi − 1/al,i ϕi+1 = ϕi + 1/ar,i

µ2 = 1 − µ1 µ2 = 1 − µ1

(9) (10)

If it is considered that ul,i = ll,i = ur,i = lr,i = 0, then the type 2 membership function will be simplified to a type-1 fuzzy membership function. It is also possible to cover other types of type-2 functions. For instance, if ul,i = ur,i = 0 , ll,i = lr,i = 1, then the resulting type-2 membership function is achieved as in Fig. 2. This type of type-2 membership functions has been previously used in a number of different papers [9]. The effect of noise in the rule base of a type-2 fuzzy system constructed by the proposed membership function is shown in Fig. 3. As can be seen from Fig. 3, in order to achieve the noise reduction properties, the parameters of type-2 membership functions should be selected as ul,i < ll,i and ur,i < lr,i . A type-2 fuzzy system with one input and two membership functions are considered in that figure. The mean square distortion of the rule base is calculated versus the parameters of the membership functions and Fig. 3 is obtained. As can be seen from Fig. 3 for some specific parameters in the proposed membership functions, the effect of the distortion can be minimized.

(8)

Fig. 1. The proposed novel diamond-shaped type-2 fuzzy membership function

Fig. 2. The proposed novel diamond-shaped type-2 fuzzy membership function with specific parameters

best position it has explored so far; and based on the global best position explored by swarm [13], [14] [15] and [16]. The PSO process then is iterated a fixed number of times or until a minimum error based on desired performance index is achieved. It has been shown that this simple model can deal with the difficult optimization problems efficiently. A detailed description of PSO algorithm is presented in [13], [14] and [16]. In this study, a short description of the PSO algorithm proposed by Kennedy and Eberhart is given. Assume that the search space is d-dimensional, and the ith particle of the swarm can be represented by a d-dimensional position vector Xi = (xi1 , xi2 , ..., xid ). The velocity of the particle is denoted by Vi = (v1i , v2i , ..., vdi ). In addition, let us consider the best visited position for the particle is Pi = (Pi1 , Pi2 , ..., Pid ) and also the best position explored so far is Pg = (Pg1 , Pg2 , ..., Pgd ). So the position of the particle and its velocity are being updated using following equations: vi (t + 1) = w.vi (t) + c1 .φ1 [Pi (t) − xi (t)] + c2 .φ2 [Pg (t) − xi (t)]

IV. H YBRID T RAINING OF T YPE -2 F UZZY S YSTEM USING THE P ROPOSED T YPE -2 F UZZY M EMBERSHIP F UNCTION In this section a hybrid PSO plus gradient descent (GD) method is discussed to train the proposed type-2 fuzzy system. The use of hybrid PSO plus GD method to optimize the structure of the fuzzy systems is considered in a number of different papers [10] ,[11]. In this approach, the usage of PSO and GD algorithms have been introduced for the parameter tuning of the premise part and the consequent part, respectively. This method has been shown to have some superiority in comparison with the classical gradient descent based methods, because in [10] and [11]: 1) The calculation of the derivatives of the output with respect to the premise part’s parameters is too complex and time consuming. These calculations are more complex in type-2 fuzzy systems, 2) The rule base of the fuzzy system is not differentiable in all the cases, 3) The possibility of entrapment of global optimization algorithms like PSO in local minima is less, 4) The classical approaches like GD may exhibit some instability phenomena with different learning rates. A. Particle swarm optimization PSO was originally designed and introduced by Eberhart and Kennedy in [12] and [13] in 1995. The PSO is a population-based search algorithm imitating the social behavior of birds, bees or a school of fishes. This algorithm originally intended to graphically simulate the graceful and unpredictable choreography of a bird folk. A vector in multidimensional search space represents each individual within the swarm. This vector has also one assigned vector, which determines the next movement of the particle and it is called the velocity vector. The PSO algorithm also determines how to update the velocity of a particle. Each particle updates its velocity based on current velocity and the

xi (t + 1) = xi (t) + vi (t + 1)

(11)

where both c1 and c2 are the two positive constants, φ1 and φ2 are the two uniformly distributed numbers between 0 and 1. In (11), w is the inertia weight which shows the effect of previous velocity vector on the new vector, and it plays the role of balancing the global and local searches. Its value may vary during the optimization process. Whereas a large inertia weight encourages a global search, a small value pursues a local search. An adaptive weighted PSO (AWPSO) algorithm is proposed in [17]. In this algorithm the velocity formula of PSO is modified as follows: vi (t + 1) = w.vi (t) + α [r1 (pi − xi (t)) + r2 (pg − xi (t))] (12) The second term in (12) can be viewed as an acceleration term, which depends on the distances between the current position xi , the personal best Pi , and the global best Pg . The acceleration factor α is defined as follows:

α = α0 + t/Nt

(13)

where Nt denotes the number of the iterations, t represents the current generation, and the suggested range for α0 is [0.5,1]. As can be seen from (13), the acceleration term will increase as the number of iterations increases, which will enhance the global search ability at the end of run and help the algorithm to jump out of the local optimum, especially in the case of multi-modal problems. Furthermore, instead of using a linearly decreasing inertia weight, a random number is used, which was proved in [18] to improve the performance of the PSO in some benchmark functions as: w = w0 + r(1 − w0 )

(14)

where w0 is in the interval of [0, 1], and r is a random number uniformly distributed in the same interval as w0 . The suggested range for w0 is [0,0.5], which makes the weight w randomly varying between 0 and 1. An upper bound is placed on the velocity in all dimensions. This limitation prevents the particle from moving too rapidly from one region in

search space to another. This value is usually initialized as a function of the range of the problem. For example, if the range of all xi j is in [-1, 1], then Vmax is proportional to 1. Pi for each particle is updated in each iteration when a better position for the particle or for the whole swarm is obtained. The feature that drives PSO is social interaction. Individuals (particles) within the swarm learn from each other. Based on the knowledge obtained from their own experience or from the experience of the other members of the swarm, they update their velocity vector. In this case each particle is attracted towards the best particle (best problem solution) found by any member of the entire swarm. Each particle therefore imitates the overall best particle. So the Pg is updated when a new best position within the whole swarm is found. B. Hybrid training of type-2 fuzzy system In this section a hybrid PSO plus GD learning algorithm for the training of type-2 fuzzy systems is discussed. PSO is a derivative-free optimization technique. Since the derivative of the cost function with respect to the antecedent part parameters is very complex, the use of PSO for the optimization of these parameters is suitable. Each membership function in the antecedent part of the type 2 fuzzy system has four parameters to be trained. The parameters of the consequent parts are trained using the GD learning algorithm. The components of each particle in PSO population are the parameters of the parameters of the type 2 fuzzy membership functions. The update rules of each population are as (11), (12), (13) and (14) which are the update rules for AWPSO. In each step of this algorithm, the PSO will update the fuzzy membership function parameters and then the GD optimizer will run once to update the consequent parameters using training data and the update rule for the consequent parts parameters. After the update of the parameters of both consequent part and the antecedent part, one epoch has been completely performed and the mean square of the error of the train data is calculated. This value would be the cost function of each particle which is to be minimized. This cost function is mathematically defined as: E=

)2 1 ( Target( j) − Out put( j) ∑ 2 j

(15)

For a better exploration of the search space, a mutation operator is defined. The previous vector of velocity in this way is reset to a random vector if the global best value does not change for some iterations. V. S IMULATION R ESULTS In this section the proposed type-2 fuzzy system with the hybrid training method is used to predict noisy chaotic Mackey-Glass time series. This chaotic system can be described by the following dynamic equation [19]: x(t) ˙ = 0.2

x(t − τ ) − 0.1x(t) 1 + x10 (t − τ )

(16)

Fig. 3. The distortion of fuzzy membership functions caused by uniform noise in premise part with respect to different parameters of type-2 membership function

The numerical values selected for this chaotic system are τ = 17, x(0) = 1.2 in this study. The following approximation is used: x(k + 1) − x(k) x(t) ˙ = (17) Ts where Ts = 1. The number of the training and the test data are considered as equal to 500. The predictor goal is to predict x(t + 1) using the inputs x(t − 2), x(t − 1) and x(t). For each input three membership functions are selected. In this way the number of rules of the system are 27. This experiment is similar to the simulation done in [3]. In order to study the effect of noise in the proposed system, two experiences are performed. The data are corrupted with noise with different levels of signal to noise ratio (SNR). Figures 4 and 5 show the corrupted Mackey-Glass signal by SNRs equal to 0dB and 10dB, respectively. The well-known formula for SNR is used as in [20]: ( 2) σs SNR = 10log10 (18) σn2 where σs2 is the variance of the signal and σn2 is the variance of the noise. Table I shows the results of the simulations of the proposed membership function and the optimization method. In each experiment the PSO algorithm is iterated for 10 epochs with 10 number of particles. In each epoch the rule base is constructed based on the obtained swarm. The consequent part is iterated for 50 epochs using the gradient based method. The obtained results are then compared with the results of previously done simulations in [3]. As can be seen from Table I, the obtained results are comparable with the results obtained in [3]. Figures 6 and 7 show the output of the proposed type-2 fuzzy system and the actual pure signal for SNR = 10dB and SNR = 0dB, respectively.

Fig. 4.

The actual and noisy data distorted by noise (SNR = 0dB)

Fig. 6. The output of the proposed type-2 fuzzy system and the actual pure signal (SNR = 10dB)

TABLE I RMSE VALUES FOR T YPE -1 SNR (dB) 0 10

Type-1 FLS 0.1517 0.0953

AND

T YPE -2 FLS F ORECASTERS

Type-2 FLS 0.1429 0.0838

The proposed Type-2 FLS 0.1422 0.0935

Figures 8 and 9 show the error between the output of the fuzzy model and the actual data for SNR = 0dB and SNR = 10dB, respectively. In these simulations we have two types of error, the error of the output of the fuzzy model with respect to the noisy data e and the result of the output of the fuzzy model with respect to actual data e1 . The latter is more important. It is seen that there are some particles for which e is a small value but e1 is a large one and vice versa. Since we select our choice based on e, it is possible to choose a particle with worse e1 and vice versa.

Fig. 5.

The actual and noisy data distorted by noise (SNR = 10dB)

Fig. 7. The output of the proposed type-2 fuzzy system and the actual pure signal (SNR = 0dB)

Fig. 8. The error of the output of the proposed type-2 fuzzy system (SNR = 10dB)

R EFERENCES

Fig. 9. The error of the output of the proposed type-2 fuzzy system (SNR = 0dB)

VI. C ONCLUSIONS AND F UTURE W ORKS A. Conclusions A novel diamond-shaped membership function for the type-2 fuzzy systems is introduced in this study. The proposed membership function has certain values on 0 and 1 and some uncertain values in the middle. It has been shown that it is possible to gain a better noise reduction property if the lower membership function is considered with lower grades. The proposed membership function is then used to construct a type-2 fuzzy predictor. The resulting type-2 fuzzy system is then optimized by a hybrid PSO-GD optimization method. Since PSO is a derivative-free global optimization method, it is possible to achieve better results without the need of calculating the difficult derivatives of the output of the fuzzy system with respect to the type-2 membership functions. The approach is then tested to predict a noisy chaotic dynamical system namely Mackey-Glass. Different levels of the noise are tested, and satisfactory results have been achieved. B. Future Works Encouraged by the simulations given in this study, in order to be able to achieve a differentiable rule base, a smooth type2 membership function will be studied. Although PSO-GD method gives some good results and has its advantages, it is a time consuming method. Using differentiable membership functions, it is possible to use pure gradient descent method to find the optimal parameters of the type-2 fuzzy system. ACKNOWLEDGMENT The authors would like to acknowledge the financial support of the Bogazici University Research Fund with the project number 09HA203D, the TUBITAK with the project number 107E284.

[1] J. M. Mendel and R. I. B. John, “Type-2 fuzzy sets made simple,” IEEE transactions on fuzzy systems, vol. 10, pp. 117–127, 2002. [2] R. Sepulveda, P. Melin, A. Rodriguez, A. Mancilla, and O. Montiel, “Analyzing the effects of the footprint of uncertainty in type-2 fuzzy logic controllers,” Engineering Letters, vol. 13, pp. 138–147, 2006. [3] J. M. Mendel, “Uncertainty, fuzzy logic, and signal processing,” Signal Processing, vol. 80, pp. 913 – 933, 2000. [4] M.-Y. Hsiao, T.-H. S. Li, J. Z. Lee, C. H. Chao, and S. H. Tsai, “Design of interval type-2 fuzzy sliding-mode controller,” Information Sciences: an International Journal, vol. 178, pp. 1696 – 1716, 2008. [5] Q. Liang and J. M. Mendel, “Designing interval type-2 fuzzy logic systems using an svd-qr method: Rule reduction,” International Journal of Intelligent Systems, vol. 15, pp. 939 – 957, 2000. [6] J. Mendel, “Computing derivatives in interval type-2 fuzzy logic systems,” Fuzzy Systems, IEEE Transactions on, vol. 12, no. 1, pp. 84 – 98, feb. 2004. [7] C.-F. Juang and Y.-W. Tsao, “A self-evolving interval type-2 fuzzy neural network with online structure and parameter learning,” IEEE transactions on fuzzy systems, vol. 16, pp. 1411–1424, 2008. [8] M. Begian, W. Melek, and J. Mendel, “Stability analysis of type-2 fuzzy systems,” in Fuzzy Systems, 2008. FUZZ-IEEE 2008. (IEEE World Congress on Computational Intelligence). IEEE International Conference on, 1-6 2008, pp. 947 –953. [9] J. Starczewski, “A triangular type-2 fuzzy logic system,” in Fuzzy Systems, 2006 IEEE International Conference on, 0-0 2006, pp. 1460 –1467. [10] M. A. Khanesar, M. A. Shoorehdeli, and M. Teshnehlab, “Hybrid training of recurrent fuzzy neural network model,” in Proceedings of the 2007 IEEE International Conference on Mechatronics and Automation, 2007, pp. 2598–2603. [11] M. A. Shoorehdeli, M. Teshnehlab, A. K. Sedigh, and M. A. Khanesar, “Identification using anfis with intelligent hybrid stable learning algorithm approaches and stability analysis of training methods,” Applied Soft Computing, vol. 9, pp. 833–850, 2009. [12] R. Eberhart and J. Kennedy, “A new optimizer using particles swarm theory,” in Sixth International Symposium on Micro Machine and Human Science, 1995, pp. 39–43. [13] J. Sadri and C. Y. Suen, “A genetic binary particle swarm optimization model,” in IEEE Congress on Evolutionary Computation, 2006, pp. 1942–1948. [14] M. A. Khanesar, M. Teshnehlab, and M. A. Shoorehdeli, “A novel binary particle swarm optimization,” in Control Automation, 2007. MED ’07. Mediterranean Conference on, 27-29 2007, pp. 1 –6. [15] A. P. Engelbrecht, Fundamentals of computational swarm intelligence. Wiley, 2005. [16] ——, Computational Intelligence. John Wiley and Sons, 2002. [17] M. Mahfouf, M.-Y. Chen, and D. A. Linkens, “Adaptive weighted particle swarm optimisation for multi-objective optimal design of alloy steels,” Lecture Notes in Computer Science, pp. 762 – 771, 2004. [18] Zhang and S. Hu, “A new approach to improve particle swarm optimization,” Lecture Notes in Computer Science, vol. 2723, pp. 134– 139, 2003. [19] M. Mackey and L. Glass, “Oscillation and chaos in physical control system,” Science, vol. 197, pp. 287–289, 1977. [20] J. M. Mendel, Uncertain Rule-Based Fuzzy Logic Systems: Introduction and New Directions. Prentice Hall, 2001.