A Metric and Multiscale Color Segmentation using the Color Monogenic Signal Guillaume Demarcq? , Laurent Mascarilla and Pierre Courtellemont {gdemar01, lmascari, pcourtel}@univ-lr.fr Laboratoire Math´ematiques, Images, Applications Universit´e de La Rochelle, France

Abstract. In this paper, we use the formalism of Clifford algebras to extend the so-called Monogenic Signal to color images. This extension consists in a function with values in the Clifford algebra R5,0 that encodes color as well as geometric structure information. Using geometric calculus, such a mathematical object can be used to extend classical concepts of signal processing (filtering, Fourier Transform...) to color images in a consistent manner. Regarding this paper, a local color phase is introduced, which generalizes the one for grayscale image. As an example of application, we provide a new method for color segmentation. Based on our phase definition and the multiscale aspect of the Color Monogenic Signal, we provide a metric approach using differential geometry which reveals relevant on the Berkeley Image Dataset. Key words: Monogenic signal, Clifford algebras, color segmentation, color image processing, differential geometry

1

Introduction

We propose in this paper a new framework for high dimensional signal processing based on Clifford algebras. The aim is to generalize in the context of color images the work of M. Felsberg [4] about the monogenic extension of the analytic signal to grayscale images. After some recalls on analytic and monogenic signals we first introduce the color monogenic signal of a color image as a scale-space signal using the Dirac operator and the Laplace equation. We show then how to define a color local phase that is parametrized by a vector of R5,0 containing color and geometric structures information. This color local phase can be used in many applications [1] [2]. We focus here on defining a new color segmentation method based on a metric and multiscale approach. Segmentation in a chosen color can be done and experiments show accurate results on images from the Berkeley dataset [7]. ?

This work has been partially founded by R´egion Poitou-Charente and ONR Grant N00014-09-1-0493

2

2

G. Demarcq, L. Mascarilla and P. Courtellemont

Dirac operator and Cauchy-Riemann equations

To a vector space E together with a quadratic form Q is associated a noncommutative algebra Cl(E, Q) called the Clifford algebra of the couple (E, Q). In what follows we deal with the Clifford algebra of the euclidean vector space Rn , usually denoted by Rn,0 . In this algebra, the product of two vectors a and b of Rn , embedded in Rn,0 , is given by: ab = a · b + a ∧ b

(1)

where a · b is the inner product and a ∧ b, the wedge product of a and b, is a bivector. This product is usually called the geometric product of a and b. One could refer to [3] for further details. 2.1

The Clifford algebra R3,0

As a vector space over R3 , it is of dimension 8. A base of R3,0 is given by: {1, e1 , e2 , e3 , e1 e2 , e1 e3 , e2 e3 , e1 e2 e3 } where {e1 , e2 , e3 } is an orthonormal basis {z } | {z } | {z } | vectors

bivectors

trivector

of R3 . Given two vectors u = ae1 + be2 + ce3 and v = a0 e1 + b0 e2 + c0 e3 of R3,0 , the geometric product uv is: uv = (aa0 + bb0 + cc0 ) {z } | scalar part

+ (ab0 − ba0 )e1 e2 + (ac0 − ca0 )e1 e3 + (bc0 − cb0 )e2 e3 {z } | bivector part

One can recognize immediately the combination of the usual dot product and cross product of R3 . In particular: ∀i, j ∈ {1, 2, 3}, ei ej + ej ei = 2δij where δij is the delta function. 2.2

Generalized Cauchy-Riemann equations

In Rn,0 , the Dirac operator is defined by D =

Pn

k=1 ek

∂ , where ∀i, j ∈ ∂xk

{1, .., n}, ei ej + ej ei = 2δij . Let f : R2 → R2,0 such that f (x, y) = f1 (x, y)e1 + f2 (x, y)e2 . Applying the Dirac operator to this function gives: Df (x, y) = D · f (x, y) + D ∧ f (x, y)   ∂f2 ∂f2 ∂f1 ∂f1 (x, y) + (x, y) + e12 (x, y) − (x, y) . = ∂x ∂y ∂x ∂y Then, solving the Dirac equation Df = 0 in R2,0 is equivalent to find solution f : R2 → R2 with f (x, y) = (f1 (x, y), f2 (x, y)) that satisfy the Cauchy-Riemann (CR) equations. Moreover, this can be extended to higher dimension and the CR equations are generalized by the Dirac equation.

A Metric and Multiscale Color Segmentation using the CMS

3

3

Analytic Signal and Monogenic Signal

3.1

Analytic Signal

Let s : R → R be a real-valued signal and f : R → R2,0 be a vector-valued signal such that f (x) = s(x)e2 . The purpose is to construct a function fulfilling the Dirac equations (i.e an holomorphic function) whose real part is the real-valued signal. It appears that it is equivalent to find the solution of a boundary value problem of the second kind (a Neumann problem):  ∂2u ∂2u   ∆u = + 2 = 0 if y > 0 ∂x2 ∂y (2)   e2 ∂u = f (x) if y = 0 ∂y ∂ ∂ + e2 and ∆ = D2 . ∂x ∂y The first equation is the 2D-Laplace equation restricted to the open domain y > 0. The second equation is called the boundary condition and the choice of the basis vector e2 is coherent with the embedding of complex functions as vector fields (the real part is embedded as the e2 -component). Using the fundamental solution of the 2D-Laplace equation, the solution of the problem leads to: with D = e1

fA (x, y) = hp ∗ f (x, y) + hp ∗ hH ∗ f (x, y)

(3)

e12 y (1D-Poisson kernel) and hH = (Hilbert kernel). π(x2 + y 2 ) πx The variable y is a scale parameter and setting it to zero1 , we obtain the classical analytic signal. where hp =

3.2

Monogenic Signal

Following the previous construction of the analytic signal, M. Felsberg has proposed an extension to 2D signals (such as grayscale images) [5]. Let s : R2 → R be a real-valued signal and f : R2 → R3,0 be a vector-valued signal such that f (x, y) = s(x, y)e3 . Generalizing 3.1 we are looking for a monogenic function (extension of holomorphic function) the e3 -component of which is the real-valued signal. The associated boundary value problem of the second kind is:  2 2 2   ∆u = ∂ u + ∂ u + ∂ u = 0 si z > 0 2 2 ∂x ∂y ∂z 2 (4) ∂u   e3 = f (x, y) si z = 0 ∂z ∂ ∂ ∂ + e2 + e3 and ∆ = D2 . ∂x ∂y ∂z The first equation is the 3D-Laplace equation restricted to the open half-space where D = e1 1

In the Fourier domain, the Poisson kernel has an exponential form and equal to one for y = 0.

4

G. Demarcq, L. Mascarilla and P. Courtellemont

z > 0 and the second equation is the boundary condition. Using the fundamental solution of the 3D-Laplace equation, the solution of this problem leads to: fM (x, y, z) = hp ∗ f (x, y) + hp ∗ hR ∗ f (x, y)

(5)

z xe1 + ye2 is a 2D-Poisson kernel and hR = 2 2 3/2 +y +z ) 2π(x2 + y 2 )3/2 is the Riesz kernel, extension in 2D of the Hilbert kernel. The variable z is a scale parameter and setting it to zero2 , we obtain the monogenic signal.

where hp =

4

2π(x2

The Color Monogenic Signal

4.1

Construction

The aim of this paper is to construct a scale-space signal for color images seen as vectors in R5,0 . Let s : R2 → R3 be a real-valued signal and f : R2 → R5,0 be a vector-valued signal such that f (x1 , x2 ) = f3 (x1 , x2 )e3 + f4 (x1 , x2 )e4 + f5 (x1 , x2 )e5 . Thus a color image is decomposed in the f3 f4 f5 space represented as the subspace spanned by {e3 , e4 , e5 }. Here, any orthonormal colorimetric system can be chosen for f3 f4 f5 such as RGB, CIE XYZ or CIE L*a*b*. According to the previous construction, we need to find a function which is monogenic and the e3 , e4 and e5 -component of which are the components f3 , f4 and f5 respectively.  ∂2u ∂2u ∂2u ∂2u ∂2u   ∆u = + + + + =0 ∂x21 ∂x22 ∂x23 ∂x24 ∂x25 (6)   e3 ∂u + e4 ∂u + e5 ∂u = f (x1 , x2 ) ∂x3 ∂x4 ∂x5 5 X

∂ and ∆ = D2 . ∂x i i=1 The choice of R5,0 is related to the construction of the monogenic signal. Indeed, we want to define a scale-space signal which have independent scales in each component (x3 for f3 , x4 for f4 and x5 for f5 ). We split the problem into three boundary value problems in R5,0 as follows:  ∂2u ∂2u ∂2u    2+ + = 0 if xi > 0 ∂x22 ∂x2i (i = 3, 4, 5) ∂x1 (7) ∂u   = fi (x1 , x2 )ei if xi = 0  ei ∂xi with D =

ei

Solving each system is achieved by adapting the results of section 3.2, however we will not explain the construction in details. Each solution of the previous systems (7) leads to monogenic functions S1 , S2 , S3 : they satisfy the Dirac equation in each subspace Ei = span{e1 , e2 , ei } (i = 3, 4, 5) and consequently the Dirac 2

Similar reason as footnote 1

A Metric and Multiscale Color Segmentation using the CMS

5

equation in R5,0 (DSi = 0). Let fc = S1 + S2 + S3 , then fc is still monogenic in R5,0 (i.e. Dfc = 0) and satisfies the boundary conditions in (6). We call fc the Scale-Space Color Monogenic Signal, it has the following form: fc =h3p ∗ f3 e3 + h4p ∗ f4 e4 + h5p ∗ f5 e5 + h3p ∗ hR ∗ f3 + h4p ∗ hR ∗ f4 + h5p ∗ hR ∗ f5 xi , (i 2 2π(x1 + x22 + x2i )3/2 x1 e1 + x2 e2 is the Riesz kernel. 2π(x21 + x22 )3/2

where hip =

4.2

(8)

= 3, 4, 5) is a 2D-Poisson kernel and hR =

Local Color Phase

Let us first introduce some notations. We denote: fc = Ae1 + Be2 + Ce3 + De4 + Ee5 where

(9)

 A = h3p ∗ hRx1 ∗ f3 + h4p ∗ hRx1 ∗ f4 + h5p ∗ hRx1 ∗ f5 ,      B = h3p ∗ hRx2 ∗ f3 + h4p ∗ hRx2 ∗ f4 + h5p ∗ hRx2 ∗ f5 , C = h3p ∗ f3 ,   D = h4p ∗ f4 ,    E = h5p ∗ f5

The Color Monogenic Signal contains two kinds of information: – The e1 and e2 components correspond to the smoothed vertical and horizontal structures with the meaning of Riesz transform. – The e3 , e4 and e5 correspond to the smoothed color represented in the f3 f4 f5 space spanned by {e3 , e4 , e5 }. If V = ue1 + ve2 + ae3 + be4 + ce5 ∈ R5,0 then the geometric product fc V in R5,0 is given by: fc V = hfc V i0 + hfc V i2 (10) where the 0-graded part hfc V i0 is the scalar part and the 2-graded part hfc V i2 is the bivector part. We can explain this result in the context of Clifford algebra as follows. If B is any normalized bivector, the subspace spanned by {1, B} is hfc V i2 isomorphic to C. Applying this to fc V = hfc V i0 + | hfc V i2 |, allows to | hfc V i2 | consider it as a complex number: fc V = hfc V i0 + i| hfc V i2 |

(11)

This precisely means that fc V is a spinor which acts as a rotation in the plane spanned by the bivector hfc V i2 . Then the local color phase is the angle of the rotation and is given by:   | hfc V i2 | ϕ = arg (fc V ) = arctan (12) hfc V i0

6

G. Demarcq, L. Mascarilla and P. Courtellemont

This phase describes the angular distance between fc and a given vector V in R5,0 , i.e it gives a correlation measure between a pixel fitted with color and structure information and a vector containing chosen color and structure.

5 5.1

A Metric and Multiscale Color Segmentation Differential Geometry

A usual method of edge detection using metric information given by the first fundamental form [8] is to consider a multidimensional image, of components (f1 , ..., fn ) defined on a domain D of R2 as a two-dimensional surface S parametrized by ψ : (x, y) → (x, y,f1 (x, y), ...,fn (x, y)) embedded into Rn+2 fitted with the λ1 0   10   .. metric g = ⊕ . This latter induces a metric on S called the . 01 0 λn first fundamental form of S, which takes the following form: dS 2 = dx2 + dy 2 + λ1 df12 + ... + λn dfn2 Then variations on the image are assimilated to tangent vectors of S and a measure of these variations is given by dS 2 . The rest of the method is devoted to select the strongest local variation, called edges. More precisely, let I(q) be the matrix representation of the metric dS 2 at q = ψ(p) (p ∈ R2 ) in the coordinates system given by (dp ψ(1, 0), dp ψ(0, 1)). Then I(q) has the following form:  I(q) =

E(q) F (q) F (q) G(q)



Let λ+ (q) and λ− (q), λ+ ≥ λ− , be the two eigenvalues of I(q) and θ+ (q), θ− (q) the corresponding eigenvectors. The edge measure is then given by: p w(q) = λ+ (q) − λ− (q) and we say that p ∈ D is an edge point if the function w has a local maximum at ψ(p) in the direction given by θ+ (ψ(p)). 5.2

A Multiscale Approach

Due to the construction of the Color Monogenic Signal, this latter inherits a multiscale character in the Poisson Scale-Space. As shown in [6], this linear scale-space satisfies the axiomatic of Iijima and then has the same properties than the Gaussian Scale-Space. We will not extend the whole linear scale-space theory in this paper, the reader may refer to [6] for further details.

A Metric and Multiscale Color Segmentation using the CMS

7

Fig. 1. From left to right: Lab image, segmentation at scale 0, 1.5 and 3.5 (hysteresis thresholds are the same for each scale.)

5.3

Segmentation Method

First of all, we need to choose a colorimetric system for our Color Monogenic Signal. In this application, we take the CIE L*a*b* space and considering each pixels of a color image as a vector in R5,0 is done by using the Color Monogenic Signal at a given scale. Then, geometric calculus allows us to consider the geometric product between each vectors and a reference (chosen) vector V = ue1 + ve2 + ae3 + be4 + ce5 . As in (4.2), a local color phase ϕ(x, y) is obtained and associated with the scalar part p(x, y) = hfc V (x, y)i0 , we obtain two functions which provide an angle and magnitude information respectively. We consider these two functions as a two-dimensional surface parametrized 4 by  ψ : (x,  y) →  (x, y, ϕ(x, y), p(x, y)) embedded into R with metric M = 10 λ0 ⊕ . Then variations in the local color phase and scalar part are 01 0µ assimilated to tangent vectors of S and a measure of these variations is given by dS 2 = dx2 + dy 2 + λdϕ2 + µdp2 . Calculating I(q) and using w(q) defined in (5.1), we are able to measure edges in a chosen color.

6

Experiments and Results

Some results obtained on well-known images and images from the Berkeley image dataset [7] are presented. We aim at showing the relevance of our approach on these highly regarded images. Firstly, we look at the Lab image (figure 1) and we study a yellow objects segmentation. Taking a vector carrying yellow in the L*a*b* space, we compute the local color phase and scalar part which are associated. Considering the method described above, we obtain images in figure 1. As the reader can see, when the scale parameter is taken to zero, we get not only yellow objects but all object with strong variations in this color. When the scale parameter increases, one can see in a first step that edges are smoothed and then that the blue block disappears. Finally we keep the desired yellow objects but also the green block, this is due to the strong variations in yellow between this object and the background. Next, we choose two challenging images from the Berkeley image dataset: the Plane and Garden images (see figure 2). For the first image (first row), we would like to get the red edges. Using the multiscale aspect and taking the metric λ = 2, µ = 0.5, we obtain results shown in figure 2. For the second image (second row), we would like to get green edges. Using

8

G. Demarcq, L. Mascarilla and P. Courtellemont

Fig. 2. From left to right. First row, the Plane image, the function w and the result at scale 3,5. Second row, the Garden image, the function w and the result at scale 3,5.

the multiscale aspect and taking the euclidean metric (λ = 1, µ = 1), results in figure 2 show an interesting segmentation compared with segmentation in [7].

7

Conclusion

In this paper, we introduce a new theoretical framework to represent color images as scale-space signal. We use Clifford algebra formalism to encode structure and color information in a single vector-valued function. By geometric calculus, we define a local color phase that may be used in a wide range of applications. We have treated in this paper a method of color segmentation which reveals relevant for finding edges relative to a chosen color. Future work will be devoted to the analysis of the scale-space in the Color Monogenic Signal framework.

References 1. Demarcq, G., Mascarilla, L., Courtellemont, P.: The Color Monogenic Signal: A New Framework for Color Image Processing. Application to Color Optical Flow. Submitted to ICIP 2009, Id 3074. 2. Demarcq, G., Mascarilla, L., Courtellemont, P.: A Color Extension of The Monogenic Signal. Definition of a Phase Based Block-Matching for Color Object Tracking. Submitted to ICCV 2009, Id 1490. 3. Sommer, G.: Geometric computing with Clifford Algebras. Theorical Foundations and Applications in Computer Vision and Robotics. Scale Space and Variational Methods. Springer-Verlag (2001). 4. Felsberg, M., Sommer, G.: The monogenic signal. IEEE Transaction on Signal Processing. 49(12), 3136–3144 (2001). 5. Felsberg, M.: Low-level image processing with the structure multivector. PhD thesis, Christian Albrechts University Kiel (2002). 6. Felsberg, M., Sommer, G.: The monogenic scale-space: A unifying approach to phase-based image processing in scale-space. JMIV, 21:5-26 (2004). 7. D. Martin, D.,Fowlkes, C., Tal, D., Malik, J.: A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. ICCV, 2:416–423 (2001). 8. Di Zenzo, S.: A note on the gradient of a multi-image. Comput. Vis. Graph. Image Process. 33(1):116–125 (1986).