U NIVERSIDAD DE C HILE D EPARTAMENTO DE I NGENIER´I A E L E´ CTRICA

3D A NALYSIS OF C ELLULAR AND S UPRACELLULAR M IGRATION IN D EVELOPMENTAL B IOLOGY

J OS E´ D ELPIANO

T HESIS P ROPOSAL D OCTORAL P ROGRAM IN E LECTRICAL E NGINEERING

T HESIS A DVISOR : JAVIER RUIZ - DEL -S OLAR ¨ T HESIS C O -A DVISOR : S TEFFEN H ARTEL

D ECEMBER , 2008

Contents 1

2

3

Introduction 1.1 In vivo confocal microscopy in experimental biology 1.2 Morphology, topology and migration of structures observed by confocal microscopy . . . . . . . . . . . 1.3 Thesis project . . . . . . . . . . . . . . . . . . . . . 1.4 Document structure . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

1 1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 2 4

State-of-the-art 2.1 Object tracking in confocal microscopy 2.2 Optical flow methods . . . . . . . . . . 2.3 Multigrid methods for optical flow . . . 2.4 Active surface models . . . . . . . . . . 2.5 Perspectives . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

5 5 8 9 10 11

Research Proposal 3.1 Hypotheses . . . . . . . . . . . . . . . 3.2 Project objectives . . . . . . . . . . . . 3.3 Research methodology . . . . . . . . . 3.4 Specific contribution to computer vision 3.5 Expected publications . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

12 12 13 14 15 15

4

Project schedule 17 4.1 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2 Gantt chart and critical paths . . . . . . . . . . . . . . . . . . . . . . . . . 18

5

Conclusions

19

Bibliography

20

2

Chapter 1 Introduction In the context of confocal microscopy applied to biological problems, morphology, topology and feature tracking has an increasing interest for researchers. This chapter describes the problem to be addressed in this Ph.D. thesis.

1.1

In vivo confocal microscopy in experimental biology

Developmental biology studies all processes that occur while a single-cell zygote becomes a full multicellular organism [1]. Development of model organisms like common fruit fly (Drosophila melanogaster), zebrafish (Danio rerio) and laboratory mice (Mus musculus) provide insight into the study of other organisms. The orderly formation of structures of a living organism is known as morphogenesis. This area of developmental biology studies the processes related to formation of structures and to their functional organization. To observe morphogenetic dynamics in spatial and temporal detail, common bright field microscopy is not sufficient. Its resolution is diffraction limited, and cannot cope with in vivo 3D imaging. Their resolution is also limited by the fact that the image acquired receives light passing trough a cone-shaped 3D region that includes a lot out of focus space, resulting in blur. This kind of microscopes is only suitable for applications in which the information to be obtained is located in a 2D plane. The diffraction limit is challenged by several microscopy technologies, such as Photo Activated Localization Microscopy (PALM) [2] and Stimulated Emission Depletion (STED) [3]. The preferred alternative for in vivo observation of cellular dynamics is spinning disk confocal microscopy (SDM), because it allows large in vivo observation intervals without damaging the biological sample. In vivo confocal microscopy combines the idea of excitation of fluorescent proteins

1

1.2 : Morphology, topology and migration of structures observed by confocal microscopy

2

with light, focusing a very small 3D region of the sample with the selection of transparent specimens to allow focusing on a point inside the sample. In this way, it is possible to scan the sample in three dimensions with the help of fluorophores, molecules that emit light of a specific wavelength in response to an excitation of shorter wavelength [4][5].

1.2

Morphology, topology and migration of structures observed by confocal microscopy

From a biological point of view, it is interesting to know which of the cells that form a tissue initially, then become a different, more specialized tissue, for example migrating to a new position, or changing their shape (morphology) and their relative position and orientation (what will be called topology in this proposal). Figure 1.1 shows an example of formation of structures in which relative position and orientation are important. Many of these questions cannot be answered properly without the fast sampling frequency allowed by SDM. The advantage of SDM is that it captures a full 2D slice of the sample in a short time (ms to s), then it scans only in the z-axis (so-called z-stacks). Conventional confocal microscopes must perform a scan process in three (xyz-)axes. In practice, this gives a speed up of about ten times. For example, a stack of 60 slices can take 12 to 15 minutes in a conventional confocal microscope and 50 to 70 seconds in a spinning disk confocal microscope. This time shortening results in less fluorescence depletion and opens the field for observation of much faster dynamic processes. Many biological questions require quantitative analysis, beyond systematic observations. The analysis of this huge amount of information requires automated tools. Moreover, confocal microscopy has one advantage over many computer vision applications: the technology intrinsically avoids occlusion. To see inside the sample, it is selected to be transparent. Then, only a small 3D point of the sample is excited with coherent light and focused to be imaged. The previous points motivate the development of a methodology for tracking morphology, topology and xyz-position of cellular and supracellular events during morphogenesis in model organisms such as the zebrafish.

1.3

Thesis project

There are published articles related to tracking in biological problems, but the subject is in a preliminary stage. There have been also efforts on 3D tracking of cell nuclei/membranes and 2D shape tracking of cells. To our knowledge, no research work has been published

1.3 : Thesis project

3

Figure 1.1: Increasing cell polarity and ’rosette formation’ during asymmetric migration of the parapineal organ during morphogenesis of transgenic zebrafish embryos (26, 32 and 38 hours post fertilization, hpf, unpublished data; Laboratory of Experimental Ontogenesis, LEO, and Laboratory for Scientific Image Analysis, SCIAN). The morpho-topological model points out ’rosette formation’ between 32-38 hpf which is defined by a concentration of polarized cell orientation (red colours mark apical cell surfaces pointing towards the center of gravity of the parapineal organ).

about 3D morphology, topology and position tracking during morphogenesis in developing fish embryos. This problem of 3D morphology, topology and position tracking can be described as based on finding discretizations of surface representations of cellular membranes. These discretizations are given by Njk points rijk = (xijk , yijk , zijk ), that define an object j at time k. Having enough points for each object and time frame, this discrete representation allows the analysis of single-cell shape, relative position and orientation between cells and trajectory of individual cells with respect to a reference point. This thesis project aims at understanding whether or not a method based on a cellular dynamical model and active surfaces (a model of shape) can track cell morphology, topology and 3D position during morphogenetic events. To assure the biological interest of the methodologies that will be designed and to get

1.4 : Document structure

4

real biological sequences of image-stacks, the thesis project will be carried out partially at Laboratory of Experimental Ontogenesis (LEO), Institute of Biomedical Sciences, U. of Chile. Working at the Computational Vision laboratory at the Electrical Engineering Department, U. of Chile, will allow taking advantage of experience with fast computer vision methods and, therefore, processing large amounts of data in reasonable times.

1.4

Document structure

Proposal chapters are organized as follows. Chapter 2 gives a review of relevant literature. It covers the fields of object tracking in experimental biology, optical flow, multigrid methods for obtention of optical flow, active surface models and future perspectives. Chapter 3 shows the research proposal hypotheses and objectives. Chapter 4 details the scheduled tasks. Finally, chapter 5 gives some concluding remarks.

Chapter 2 State-of-the-art Although object tracking has been explored widely in computer vision, only a few years ago it has been possible to apply it to microscopic biological problems. In this field, first applications of basic optical flow (OF) and active surface models have shown to be reliable techniques for cell identification and tracking. The following sections summarize recent literature, including the most important aspects of optical flow approaches using multigrid methods.

2.1

Object tracking in confocal microscopy

In living biological systems, cell tracking is difficult due to several reasons: the motion model is not always known and can be very different from one problem to another. The observed objects are nonrigid and can divide or die during observation intervals. Furthermore, tracking not only position, but shape, makes the problem even harder. Different tracking efforts related to similar applications have been made in the last decade. An interesting approach called Kalman snakes or velocity snakes was presented about ten years ago [6][7]. In this method, the author uses an extended Kalman filter-based active contour model for tracking nonrigid objects in a combined spatio-velocity space. The system measurements for parameter estimation are image gradient and OF along the contour. The results show a good behavior under occlusion for rigid and nonrigid objects. A very robust technique, the so-called condensation method, was introduced by the same years [8]. Condensation is based on randomly distributed hypotheses and recalls Monte Carlo and particle filter methods. Therefore, it can handle multi-modal probability density functions. Condensation method can be considered a particle filter in which the state is an affine transform of a model of the object. Then, it is not expected to handle tracking of objects with strong deformations, like those of many microscopic biological

5

2.1 : Object tracking in confocal microscopy

6

subjects. More recently, a combination of optical flow and active polygons [9] showed to be a reliable method for tracking of deformable objects, under occlusion and noise. Another recent approach integrated particle filters and level set active contours to track deformable objects [10]. Level set methods are variational methods that can perform contour and surface-based segmentation [11]. They are based in the optimization of a functional to obtain a solution function whose level sets give the desired contour or surface. Examples of applications of level sets are given in [12][13]. In [9], the authors proposed two stages for a 2D approach of interaction between active contours and OF. First, a prediction stage, s− j,k = sj,k−1 + Vj,k

(2.1) (2.2)

where s− j,k is the predicted position of vertex j of the active surface model (ASM) at time interval k, and Vj,k is the optical flow at vertex j or at a neighborhood of vertex j. Then, a correction stage. The correction stage for an active contour is given by the minimization of: Z E=

1

n o 2 2 0.5[α |C 0 (s)| + β |C 00 (s)| ] + Eext (C(s)) ds

(2.3)

0

where E is a functional related to the forces that act over the active contour, C(s) is a parameterization of the contour, Eext are external forces and α and β are force parameters. At each iteration, it will be necessary to “resample” the active surface vertices, following criteria related to minimum and maximum values for the area of the faces of the ASM. In the last five years, several researchers have addressed the problem of tracking biological objects. A recent approach is based on extended Kalman filters for artificial [14] and real [15] sequences. Their method applies the discrete wavelet transform for detection of spot-like biological objects and keeps track of 3D position using interacting multiple models (IMM), a methodology that will be discussed later in this section. In [16], four different variational OF techniques are applied to motion estimation in fluorescently labeled targets in living cells. A series of conference articles present a tracking approach with variational methods for image filtering, segmentation, and obtention of the optical flow field [17][18][19]. The proposed technique is applied to confocal microscopy images of zebrafish embryos.

2.1 : Object tracking in confocal microscopy

7

A position tracking approach was reported for phase contrast microscopy [20][13][21]. This kind of microscopes have a much lower resolution than confocal microscopes, especially in the z-axis. In [13], for example, the track space shown is a (2D) rectangle of sides 1mm and 2mm, far larger than in a common confocal microscopy observation (for example, 100µm). This fact should reduce considerably the problems related to high resolutions, like image drift. 1 The approach presented in this articles is also based in the IMM filter. The movement and observation model of IMM is given by

sk = Ask−1 + wk−1 ok = Csk + vk

(2.4) (2.5)

CM CM T where the state at time k is sk = (rkCM , rk−1 , rk−2 ) , with rkCM = (xk , yk , zk )T the center of mass of the enclosed volume or the center of mass of the active surface. The alternative of including object intensities in the state vector was also proposed. The observations ok correspond to the observed center of mass. In the article cited above, the errors vk and wk are supposed to be gaussian and with correlation matrixes

Q = E[wk wkT ]

(2.6)

R = E[vk vkT ]

(2.7)

There are three typical models of cell motion: Brownian motion, constant speed migration and constant acceleration migration. They determine the matrixes in eqs. 2.4 and 2.5, and are described by A1 , A2 and A3 , given below. The observation model is described by C below. 

   I 0 0 2I −I 0 0 0  A1 =  I 0 0  , A2 =  I 0 I 0 0 I 0   3I −3I I   0 0 ,C = I 0 0 A3 =  I 0 I 0 1

(2.8)

(2.9)

In confocal microscopy, a displacement of the whole biological sample with respect to the microscope is observed, due to sample movement and due to thermal dilation and compression of microscopic components. This drift does not allow the correct evaluation of motion inside the sample due to self motion.

2.2 : Optical flow methods

8

where I is a 3x3 identity matrix. As a Kalman-based method, IMM requires a prediction and a correction stage. Here, the prediction stage is given by: s− k = Ask−1 Pk− = APk−1 AT + Q

(2.10) (2.11)

where the minus sign notation (s− k ) indicates a preliminary estimate, that will be improved in the other stage for time k. Pk− is the estimated covariance at time k. The correction stage is given by: Kk = Pk− C T (CPk− C T + R)−1 − sk = s− k + Kk (ok − Csk ) Pk = (I − Kk C)Pk−

(2.12) (2.13) (2.14)

where Kk is the calculated Kalman gain, which is used to correct the state and covariance predictions. Cell division is preceded be changes in shape and motion model. Then, divisions can be detected analyzing morphological and dynamical parameters of cells over time.

2.2

Optical flow methods

Optical flow is a vector field that measures apparent motion in images. The optical flow constraint (eq. 2.15) has a strong analogy with mass conservation in fluid mechanics (eq. 2.16): as fluid mass, image intensity is often supposed to remain constant under deformation and motion. This idea has been explored further in the past [22]. However, the equations would only be equivalent exactly if ∇u = 0. This condition matches the smooth flow assumption, that is considered when regularizing the flow field. In eq. 2.15, f = f (x, y) is a 2D image and u is the OF. In eq. 2.16, u is the fluid speed and ρ is the fluid density. ∂f + u∇f = 0 ∂t ∂ρ + ∇(ρu) = 0 ∂t

(2.15) (2.16)

2.3 : Multigrid methods for optical flow

9

The optical flow constraint is particularly suitable for fluorescence microscopy, because the quantity of fluorophores is expected to remain constant. On the other hand, image intensity is not always constant in computer vision applications. The early work presented in [23] stated the need for an extra constraint to compute OF from an image sequence and proposed one ad-hoc constraint based on the assumption of flow smoothness. More than ten years later [24] gave a comparison of several methods, mainly with respect to their average angular error (AAE, defined in Section 3.3) when applied to some image sequences. Several researchers have tried to preserve the discontinuity of natural flow fields [25][26], overcoming the original assumption of flow smoothness in [23]. A novel approach is to use harmonic phase information to obtain OF [27][28]. The harmonic phase has the advantage of being a low level feature that can be used for additional objectives, like stereo vision and motion-in-depth [29][28]. Many applications have incorporated optical flow. Two particularly interesting application fields are robotics and medical images. In mobile robotics, OF divergence has been proved to estimate time-to-collision [30]. This idea has also been tested in indoor microflyer robots [31][32][33]. Recent medical applications include the visualization of 3D Lucas-Kanade OF to study soft tissue motion in the orbit during gaze [34] and motion extraction from cardiac sequences [35]. After the comparison in [24], there have been further attempts to compare different methods. For example, [36] shows a trade-off between computation time of an algorithm and angular error obtained, given a fixed density. The authors use that operation curve to compare OF algorithms. [37] gave a tool for generating artificial 2D sequences from 3D scenes and characterized OF methods using curves of percentage of flows under an error threshold. It is interesting to note a time comparison between OF algorithms given in [38], because it gives a more recent idea of computational load of OF algorithms. More recently, [39] presents a series of real image sequences with ground truth OF obtained by tracking hidden fluorescent textures. The authors also suggest a method for evaluation of optical flow algorithms. To give a summary about optical flow, currently it can be calculated accurately for common computer vision applications and the calculation of 3D OF has begun being explored.

2.3

Multigrid methods for optical flow

Partial differential equations have been extensively used in image processing [40]. A recently appeared family of methods for OF take advantage of a variational formulation of the OF PDE problem and solve it with fast numerical methods [41][42][26][43]. These

2.4 : Active surface models

10

variational OF methods have shown frame rates near video acquisition. They are based on a functional that includes terms related to the optical flow constraint and to regularization. The corresponding Euler-Lagrange equations are discretized using finite differences and solved with efficient multigrid numerical methods. The combined local global (CLG) approach, presented in [42], minimizes a functional to solve for 2D OF. The functional extended to 3D is as follows: Z E(u) =


uT Jρ (∇4 f )u + α(|∇u|2 + |∇v|2 + |∇w|2 ) dxdydz

(2.17)

Ω

where u is the OF vector (u, v, w), ∇4 f = (fx , fy , fz , ft ) is the derivative vector of the image f = f (x, y, z, t). Jρ (∇4 f ) is a function of the image derivatives, with smoothing of standard deviation ρ and corresponds to the optical flow constraint. The functional above gives the following Euler-Lagrange equations as optimum conditions. α∆u − (J11 u + J12 v + J13 w + J14 ) = 0 α∆v − (J21 u + J22 v + J23 w + J24 ) = 0 α∆w − (J31 u + J32 v + J33 w + J34 ) = 0

(2.18) (2.19) (2.20)

where the values Jij are the elements of the matrix Jρ (∇4 f ). The boundary conditions for the problem are:

∂n u = 0, ∂n v = 0, ∂n w = 0

(2.21)

on ∂Ω, where ∂n u is the normal derivative with respect to ∂Ω.

2.4

Active surface models

As mentioned before, different publications discuss object tracking with active contour models [6][7][9]. Active surface adjustment to a 3D shape can be guided by an imagestack-dependent force field (see eq. 2.3). A widely spread method for calculating force fields is gradient vector flow (GVF). In [44], the authors describe original active contours (snakes) and GVF snakes. Then, they propose generalized GVF (GGVF) snakes. The advantage of GVF and GGVF is to spread the influence of borders. GVF and GGVF were applied successfully to artificial and

2.5 : Perspectives

11

actual images from magnetic resonance acquisitions. In simple cases, they allow detection of “subjective” contours, those that are not explicitly present in the image. GGVF snakes have also been applied to microscopic biological subjects [45][46]. Active surfaces are a generalization of active contours to 3D [47]. ASM functional includes terms related to torsional forces, that are not considered in the 2D version [48]. ASM reparameterization must take into account face area instead of length of sides. An approach for detection of subjective surface models (SSM) was presented a few years ago and was shown to be succeessful when segmenting echographic images [49][50]. They allow detection of contours that are not completely explicit in an image, for example due to noise. SSM’s are based on level set methods and have been applied to 3D embryogenesis segmentation recently [51].

2.5

Perspectives

There have been works related to 3D tracking of cell nuclei [17] and 3D segmentation of cell nuclei/membrane using variational approaches [51] in zebrafish embryos. There have also been articles related to 2D shape tracking of cells [13]. However, no research has been published about 3D shape tracking of cell membranes during neural morphogenesis in developing fish embryos.

Chapter 3 Research Proposal 3.1

Hypotheses

Recent articles emphasize the importance of tracking of cell nuclei at early stages of tissue, organ, and organism development [52][17], showing the suitability of optical flow as a low level approach. However, no method has shown fast and reliable 3D tracking of cell morphology and topology for more complex problems like neural morphogenetic events imaged with spinning disk confocal microscopy. Based on this observation and in the existence of fast optical flow methods [26], the following hypothesis is proposed: Main hypothesis: • A method based on the joint use of a cellular dynamical model and active surfaces can track cell morphology and topology during morphogenetic events imaged with three dimensional spinning disk microscopy. Detection of subjective surfaces can improve tracking performance, when considering the percentage of valid tracks obtained. Specific hypotheses: 1. 3D OF approaches based on a fast 2D method like the CLG variational technique, can obtain results with high density and low error. 2. A 3D-CLG OF can get results in shorter execution times than a 3D Lucas-Kanade like that presented in [34]. 3. A technique based on optical flow or parameter estimation (Kalman filters, interacting multiple model filters [15][13]) can correctly track the trajectory and shape of a group of cells, outperforming the maximum overlap method.

12

3.2 : Project objectives

3.2

13

Project objectives

General objectives: • Design 3D tracking methods based on the joint use of a cellular dynamical model and a shape model. • Apply the methods developed to in-vivo microscopy acquisitions related to developmental biology problems and perform a comparative study of its results and those of another methods to test correctness quantitatively. Specific objectives: 1. Design a method for drift correction. The method can be developed on the basis of neglecting global displacements of the sample. 2. Generalize to 3D a fast technique for optical flow like the CLG approach. 3. Study comparatively the performance of the optical flow method that will be developed and the ground truth optical flow of artificial sequences, using a generalization of the commonly used AAE (see section 3.3). This study should include also the performance of the optical flow method that will be developed versus other methods. 4. Extend an existing active surface model to be able to start from given vertex points. The current implementation available at the Laboratory for Scientific Image Analysis, SCIAN only can start from a segmentation result, and not from a vertex set (see Figure 3.1, for examples of 3D reconstructions of cell groups with current implementation). However, this implementation already incorporates several morphology and topology descriptors, that will be the basis for the analysis of the tracking results. 5. Develop a 3D tracking technique for biological structures, combining 3D CLG OF with advanced active surface models. 6. Develop and compare quantitatively the proposed methods and alternative tracking methods. Example alternative methods are: • Manual tracking. • 3D tracking technique based on active surfaces and interacting multiple models (IMM) or another motion model. • Similar to the previous methodology, but adding optical flow as a new observed variable.

3.3 : Research methodology

14

• Using a combination of optical flow and active surfaces as the source for observing the position of the center of mass of individual cells. • Changing the parameter estimation method to more recent unscented Kalman filters or particle filters.

3.3

Research methodology

First, the performance of 2D and 3D optical flow techniques will be evaluated. For the 3D case, it is interesting to evaluate it when applied to general images and to confocal microscopy images related to developmental biology. For 2D methods, the focus will be on biological images. To evaluate optical flow methods, the following sets of images will be involved, each one related to one series of experiments: • 2D biological image sequences. • 3D artificial image sequences with known optical flow field to be used as ground truth for error evaluation. • 3D biological sequences of image stacks. On these series of experiments, the following measures will be evaluated to know the flow correctness and efficiency: • Average angular error (AAE), and a generalization of AAE to 3D, that is proposed to be: 

 ε1 =

X

ugt ue + vgt ve + wgt we + 1

arccos  q

2 2 (u2gt + vgt + wgt + 1)(u2e + ve2 + we2 + 1)



(3.1)

where (ugt , vgt , wgt ) and (ue , ve , we ) are the ground truth and estimated optical flow vectors. • End-point error, given by: ε2 =

Xq

(ugt − ue )2 + (vgt − ve )2 + (wgt − we )2

• Processing time and processing time per 2D/3D frame or per pixel.

(3.2)

3.4 : Specific contribution to computer vision

15

The following aspects of the performance of the tracking method will be evaluated: tracking quality, tracking reliability and tracking efficiency. To evaluate the tracking methodology with respect to tracking quality, it would be interesting to apply it first to short 3D sequences with few cell divisions and disappearings. Then the method should be evaluated with more complex sequences. To measure the tracking quality, reliability and efficiency, the following quantities can be observed: • Number of cells correctly tracked. • Total length of valid tracks. • Measures of position and shape track error. One possible shape error measure is the mean Hausdorff distance [19]. • Processing time and processing time per tracked object.

3.4

Specific contribution to computer vision

The results of this project will not be interesting to biologists only, but to computer vision researchers. 3D nonrigid tracking has been studied before, but the problem of realibility and efficiency of the methods has not been solved yet.

3.5

Expected publications

It is expected to present the results obtained from this thesis in two international conferences and two ISI journal articles.

3.5 : Expected publications

Figure 3.1: Principal axes alignment during parapineal morphogenesis. Left: alignment between the 1st principal axis (PA) of individual parapineal cells (circles) is compared to the alignment between the 1st PA of the individual cells with the 1st PA of the entire parapineal organ (squares). The parameter ’principal axis alignment (PAA)’ represents the mean values calculated from the respective angular distributions in three independent experiments. Error bars represent SD. Significant changes for the 1st PAA of the entire parapineal organ are detected by ANOVA between 28 and 30 hours post fertilization (hpf) and presented as a simplified block scheme. Center: representative surface reconstructions of the parapineal cells based on active contours in combination with the respective 1st PA of individual parapineal cells and of the entire parapineal organ at 26, 30, and 34 hpf. The data reveals a transition of the organization of parapineal cells from a predominantly parallel orientation towards predominantly perpendicular orientation [47].

16

Chapter 4 Project schedule 4.1

Tasks

1. Project definition. 2. Data acquisition/preparation: implementation of routines related to image opening and preprocessing. The high amount of images (a typical acquisition can have more than 100 times, 50 z-slices each) that are managed requires automatic methods for file opening, saving, applying naming conventions, etc. There is also the need for correcting the drift effect, already mentioned. Partially done. 3. 3D OF design and implementation: generalization and implementation of a fast 3D optical flow technique. Adaptation of existing 3D OF technique for comparison. Preliminary work done: solution of a simpler PDE, the Poisson equation, with Dirichlet boundary conditions using the multigrid method. 4. 3D OF debug and evaluation: comparison with ground truth optical flow. Comparison between OF methods. 5. Conference article preparation. 6. Active surface improvement: adaptation and improvements to existing code for a shape model. 7. 3D tracking development: implementation of an efficient method for 3D tracking of biological specimens using in-vivo microscopy images. 8. 3D tracking debug and test: test using simple biological sequences.

17

4.2 : Gantt chart and critical paths

18

9. 3D tracking evaluation: comparison with manual tracking. Comparison with other techniques. 10. Conference article preparation. 11. Journal article preparation. 12. Tracking experiments/analysis: application to other biological problems. 13. Thesis writing. 14. Journal article preparation.

4.2

Gantt chart and critical paths

Figure 4.1 shows the Gantt chart for the thesis project. It has one main path, then it is a critical path. Previous experience with optical flow and tasks already accomplished should make possible the timely ending of this thesis project.

Figure 4.1: Gantt chart for the thesis project.

Chapter 5 Conclusions The problem of neural morphogenesis is a challenging biomedical image processing application. Observing morphogenesis dynamics requires going to the resolution limit of current microscopes. Processing the huge amount of acquired data makes hardware and software speed a need. 3D position and shape tracking during neural morphogenesis is a novel subject. In the case of zebrafish, it has been almost not studied. Only for the first 24 hours of development of zebrafish, there have been some preliminary position tracking efforts. This thesis project will address the design of a novel method for cellular and supracellular morpho-topological tracking. A reliable motion measurement is 3D optical flow. The application of active surface models and a motion model based on fast 3D OF and parameter estimation techniques will open the access to efficient nonrigid tracking over time. The expected results of this project will be interesting not only to computer vision research but to several biological research areas. The project will also develop research subjects not well known locally.

19

Bibliography [1] Scott F. Gilbert, Susan R. Singer, Mary S. Tyler, and Ronald N. Kozlowski, Developmental Biology, Sinauer Associates, 2006. [2] Eric Betzig, George H. Patterson, Rachid Sougrat, O. Wolf Lindwasser, Scott Olenych, Juan S. Bonifacino, Michael W. Davidson, Jennifer Lippincott-Schwartz, and Harald F. Hess, “Imaging Intracellular Fluorescent Proteins at Nanometer Resolution,” Science, vol. 313, no. 5793, pp. 1642–1645, 2006. [3] Stefan W Hell, Marcus Dyba, and Stefan Jakobs, “Concepts for nanoscale resolution in fluorescence microscopy,” Current Opinion in Neurobiology, vol. 14, no. 5, pp. 599–609, 2004. [4] Charles Fink, Frank Morgan, and Leslie M. Loew, “Intracellular Fluorescent Probe Concentrations by Confocal Microscopy,” Biophys. J., vol. 75, no. 4, pp. 1648–1658, 1998. [5] C. Vonesch, F. Aguet, J.-L. Vonesch, and M. Unser, “The Colored Revolution of Bioimaging,” IEEE Signal Processing Magazine, vol. 23, no. 3, pp. 20–31, 2006. [6] N. Peterfreund, “The velocity snake,” in Proceedings of the IEEE Nonrigid and Articulated Motion Workshop, San Juan, 1997, pp. 70–79. [7] N. Peterfreund, “Robust tracking of position and velocity with Kalman snakes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 6, pp. 564– 569, 1999. [8] Michael Isard and Andrew Blake, “CONDENSATION—Conditional density propagation for visual tracking,” International Journal of Computer Vision, vol. 29, no. 1, pp. 5–28, 1998. [9] G. Unal, H. Krim, and A. Yezzi, “Fast incorporation of optical flow into active polygons,” Image Processing, IEEE Transactions on, vol. 14, no. 6, pp. 745–759, 2005. 20

BIBLIOGRAPHY

21

[10] Y. Rathi, N. Vaswani, A. Tannenbaum, and A. Yezzi, “Tracking Deforming Objects Using Particle Filtering for Geometric Active Contours,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 29, no. 8, pp. 1470–1475, 2007. [11] Stanley Osher, “Level set methods,” in Geometric Level Set Methods in Imaging, Vision, and Graphics, Nikos Paragios and Osher Stanley, Eds., pp. 3–20. Springer, 2003. [12] G. Slabaugh and G. Unal, “Active polyhedron: surface evolution theory applied to deformable meshes,” in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, 2005, vol. 2, pp. 84–91 vol. 2. [13] K. Li and T. Kanade, “Cell population tracking and lineage construction using multiple-model dynamics filters and spatiotemporal optimization,” in Proceedings of the 2nd International Workshop on Microscopic Image Analysis with Applications in Biology (MIAAB), Piscataway, NJ, USA, September 2007. [14] A. Genovesio, B. Zhang, and J.-C. Olivo-Marin, “Tracking of multiple fluorescent biological objects in three dimensional video microscopy,” in Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, 2003, vol. 1, pp. I–1105– 8 vol.1. [15] A. Genovesio, T. Liedl, V. Emiliani, W.J. Parak, M. Coppey-Moisan, and J.-C. OlivoMarin, “Multiple particle tracking in 3-D+t microscopy: method and application to the tracking of endocytosed quantum dots,” Image Processing, IEEE Transactions on, vol. 15, no. 5, pp. 1062–1070, 2006. [16] Jan Huben´y, Vladim´ır Ulman, and Pavel Matula, “Estimating Large Local Motion in Live-Cell Imaging Using Variational Optical Flow,” in VISAPP 2007 Second International Conference On Computer Vision Theory And Applications, Proceedings, 8.3.2007, Barcelona, Spain, pp. 542–548, Portugal : INSTICC PRESS. [17] C. Melani, N. Peyrieras, K. Mikula, C. Zanella, M. Campana, B. Rizzi, F. Veronesi, A. Sarti, B. Lombardot, and P. Bourgine, “Cells tracking in a live zebrafish embryo,” in Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, 2007, pp. 1631–1634. [18] C. Zanella, B. Rizzi, C. Melani, M. Campana, P. Bourgine, K. Mikula, N. Peyrieras, and A. Sarti, “Segmentation of Cells from 3-D Confocal Images of Live Zebrafish Embryo,” in Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, 2007, pp. 6027–6030.

BIBLIOGRAPHY

22

[19] B. Rizzi, M. Campana, C. Zanella, C. Melani, R. Cunderlik, Z. Kriva, P. Bourgine, K. Mikula, N. Peyrieras, and A. Sarti, “3-D Zebrafish Embryo Image Filtering by Nonlinear Partial Differential Equations,” in Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, 2007, pp. 6251–6254. [20] Kang Li, E.D. Miller, L.E. Weiss, P.G. Campbell, and T. Kanade, “Online Tracking of Migrating and Proliferating Cells Imaged with Phase-Contrast Microscopy,” in Computer Vision and Pattern Recognition Workshop, 2006 Conference on, 2006, p. 65. [21] Kang Li, Eric D. Miller, Mei Chen, Takeo Kanade, Lee E. Weiss, and Phil G. Campbell, “Cell population tracking and lineage construction with spatiotemporal context,” Medical Image Analysis, vol. 12, no. 5, pp. 546–566, 2008. [22] D. Bereziat, I. Herlin, and L. Younes, “A generalized optical flow constraint and its physical interpretation,” in Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, 2000, vol. 2, pp. 487–492 vol.2. [23] Berthold K.P. Horn and Brian G. Schunck, “Determining Optical Flow,” Artificial Intelligence, vol. 17, pp. 185–203, 1981. [24] J.L. Barron, D.J. Fleet, and S.S. Beauchemin, “Performance of optical flow techniques,” International Journal of Computer Vision, vol. 12, no. 1, pp. 43–77, 1994. [25] D.-G. Sim and R.-H. Park, “A two-stage algorithm for motion discontinuity preserving optical flow estimation,” Computer Vision and Image Understanding, vol. 65, no. 1, pp. 19–37, 1997. [26] A. Bruhn and J. Weickert, “A multigrid platform for real-time motion computation with discontinuity-preserving variational methods,” International Journal of Computer Vision, vol. 70, no. 3, pp. 257–277, 2006. [27] T. Gautama and Marc M. Van Hulle, “A Phase-Based Approach to the Estimation of the Optical Flow Field Using Spatial Filtering,” IEEE Trans. on Neural Networks, vol. 13, no. 5, pp. 1127–1136, 2002. [28] S.P. Sabatini, G. Gastaldi, F. Solari, K. Pauwels, M. van Hulle, J. Diaz, E. Ros, N. Pugeault, and N. Kruger, “Compact (and accurate) early vision processing in the harmonic space,” in International Conference on Computer Vision Theory and Applications (VISAPP) 2007, Barcelona, Espa˜na, 2007.

BIBLIOGRAPHY

23

[29] F. Solari, S.P. Sabatini, and G.M. Bisio, “Fast technique for phase-based disparity estimation with no explicit calculation of phase,” Electronics Letters, vol. 37, no. 23, pp. 1382–1383, 2001. [30] D. Coombs, M. Herman, Tsai-Hong Hong, and M. Nashman, “Real-time obstacle avoidance using central flow divergence, and peripheral flow,” Robotics and Automation, IEEE Transactions on, vol. 14, no. 1, pp. 49–59, 1998. [31] J.-C. Zufferey and D. Floreano, “Fly-inspired visual steering of an ultralight indoor aircraft,” Robotics, IEEE Transactions on, vol. 22, no. 1, pp. 137–146, 2006. [32] A. Beyeler, J.-C. Zufferey, and D. Floreano, “3D Vision-based Navigation for Indoor Microflyers,” in Robotics and Automation, 2007 IEEE International Conference on, 2007, pp. 1336–1341. [33] W.E. Green and P.Y. Oh, “Optic-Flow-Based Collision Avoidance,” Robotics & Automation Magazine, IEEE, vol. 15, no. 1, pp. 96–103, 2008. [34] M.D. Abramoff and M.A. Viergever, “Computation and visualization of threedimensional soft tissue motion in the orbit,” Medical Imaging, IEEE Transactions on, vol. 21, no. 4, pp. 296–304, 2002. [35] L. Florack, H. van Assen, and A. Suinesiaputra, “Dense Multiscale Motion Extraction from Cardiac Cine MR Tagging using HARP Technology,” in Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, 2007, pp. 1–8. [36] Hongche Liu, Tsai-Hong Hong, Martin Herman, Ted Camus, and Rama Chellappa, “Accuracy vs efficiency trade-offs in optical flow algorithms,” Comput. Vis. Image Underst., vol. 72, no. 3, pp. 271–286, 1998. [37] B. McCane, K. Novins, D. Crannitch, and B. Galvin, “On benchmarking optical flow,” Comput. Vis. Image Underst., vol. 84, no. 1, pp. 126–143, 2001. [38] David Gibson and Michael Spann, “Robust Optical Flow Estimation Based on a Sparse Motion Trajectory Set,” IEEE Transactions on Image Processing, vol. 12, no. 4, pp. 431–445, 2003. [39] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” in Proceedings of the 2007 International Conference on Computer Vision, 2007. [40] T. Chan, J. Shen, and L. Vese, “Variational PDE models in image processing,” Notices of the AMS, vol. 50, no. 1, pp. 14—26, 2003.

BIBLIOGRAPHY

24

[41] Andr´es Bruhn, Joachim Weickert, and Christoph Schn¨orr, “Combining the advantages of local and global optic flow methods,” in Lecture Notes in Computer Science, L. Van Gool, Ed., vol. 2449/2002, pp. 454–462. 2002. [42] A. Bruhn, J. Weickert, T. Feddern, C. amd Kohlberger, and C. Schn¨orr, “Variational optical flow computation in real time,” IEEE Transactions on Image Processing, vol. 14, no. 5, pp. 608–615, 2005. [43] Nils Papenberg, Andr´es Bruhn, Thomas Brox, Stephan Didas, and Joachim Weickert, “Highly Accurate Optic Flow Computation with Theoretically Justified Warping,” International Journal of Computer Vision, vol. 67, no. 2, pp. 141–158, 2006. [44] Chenyang Xu and Jerry L. Prince, “Generalized gradient vector flow external forces for active contours,” Signal Processing, vol. 71, no. 2, pp. 131–139, 1998. [45] Luisina De Tullio, Bruno Maggio, Steffen H¨artel, Jorge Jara, and Maria Fanani, “The initial surface composition and topography modulate sphingomyelinase-driven sphingomyelin to ceramide conversion in lipid monolayers,” Cell Biochemistry and Biophysics, vol. 47, no. 2, pp. 169–177, 2007. [46] Maria Laura Fanani, Luisina De Tullio, Steffen H¨artel, Jorge Jara, and Bruno Maggio, “Sphingomyelinase-induced domain shape relaxation driven by out-of-equilibrium changes of composition,” Biophys. J., in press. [47] H¨artel S., Jara J., Lemus C.G., and Concha M.L., “3D morpho-topological analysis of asymmetric neuronal morphogenesis in developing zebrafish,” in Computational Modelling of Objects Represented in Images. Fundamentals, Methods and Applications, Jo˜ao Manuel R.S. Tavares and Jorge R.M. Natal, Eds., pp. 215–220. Taylor and Francis, 2007. [48] J.F. Jara, “Contornos activos en tres dimensiones para la segmentaci´on y parametrizaci´on de estructuras biol´ogicas,” Tesis de grado para ingenier´ıa civil en inform´atica, Universidad Austral de Chile, Valdivia, Chile, 2006. [49] A. Sarti, R. Malladi, and J.A. Sethian, “Subjective Surfaces: A Method for Completing Missing Boundaries,” Proceedings of the National Academy of Sciences of USA, vol. 97, no. 12, pp. 6258—6263, 2000. [50] A. Sarti, R. Malladi, and J.A. Sethian, “Subjective Surfaces: A Geometric Model for Boundary Completion,” International Journal of Computer Vision, vol. 46, no. 3, pp. 201–221, 2002.

[51] K. Mikula, N. Peyrieras, M. Remesikova, and A. Sarti, “3D embryogenesis image segmentation by the generalized subjective surface method using the finite volume technique,” in Proceeding of FVCA5 - 5th International Symposium on Finite Volumes for Complex Applications, Aussois, France, June 2008, Hermes Publ. [52] Gang Li, Tianming Liu, Ashley Tarokh, Jingxin Nie, Lei Guo, Andrew Mara, Scott Holley, and Stephen Wong, “3D cell nuclei segmentation based on gradient flow tracking,” BMC Cell Biology, vol. 8, no. 40, pp. 1–10, 2007.

25