ucRL-IC-\24362 Preprint

Hypertools in Image and Volume Visualization

Pierre-LouisBossart

This paper was prepared for submittal to '96 4th Annual Tcl/Tk Workshop Monterey, CA I,rly 1,0-1.3,1996

|une 77, 7996

This is a preprintof a paperintendedfor publicationin a joumal or Sincechangesmay be madebeforepublication,thispreprintis madeavailable ttratit will not be ciædor reproducedwittroutthe with the understanding permissionof the author.

Hypertools in image and volume visualization Pierre-LouisBossart LawrenceLiuermoreNational Laboratory N on-D estructiaeEualuationSection L-416, 7000East Auenue Liuermore,CA 94550,USA Far: 510-/*22-78 19 E-mail: [email protected] Phone:510-423-9350

Abstract This paper describes our experience in image and volume visualization at the Lawrence Livermore National Laboratory. After an introduction on visualization issues, we present a new sofbware approach to the analysis and visualization of images and volumes. The efficiency of the visualization process is improved by letting the user combine small and reusable applications by the means of a machineindependent interpreted language such as Tcl/Tk. These hypertools can communicate with each other over a network, which has a direct impact on the design of graphical interfaces. We first describe the implementation of a flexible gray-scale image widget that can handle large data sets, provides complete control of the color palette and allows for manual and semi-interactive segmentation. This visualization tool can be embedded in a data-flow image processing environment to assessthe quality of acquisition, preprocessing and filtering of raw data. This approach combines the simplicity of visual programming with the power of a high-level interpreted language. We show how hypertools can be used in surface and volume rendering and how they increase the interaction efficiency by performing complex or tedious tasks automatically. One biomedical application is presented.

1

Introduction

Visualizing images or volumes helps extract qualitative information and quantitative measurements from raw data sets. Visualization software is thus becoming critical in virtually every domain of engineering. However, despite the number of packages available from commercial vendors and from the public domain, it is extremely difficult to find

a package that fulfills the needs of a research laboratory dealing with large data sets. Indeed, all the packagesare implemented based on the following scheme. o First, the file formats need to be decoded, and the raw data read in. The data can then be formatted or extracted. This step includes operation such as subsampling, interpolation or dimensional reduction, e.g when a slice is extracted from a 3D volume. o Next, the data are normalized, typically to 8or 16-bit integers, and displayed in a window after the colors have been allocated. IJsing predefined color look-up tables, stretching the histogram, reducing the image dynamic range and providing a colorbar help identify the relevant features of the data set. o l.{umerical values, extracted data or regions of interest can then be written back into a file. e At any point, the user may interact with a graphical user interface (GUI) or issues commands to an interpreter, e.g. to change the color look-up table, look at pixel values, etc. Practical experiments showed that none of the all command-line oriented (VIEW, IDL, PWave, Matlab, etc), data^flow-oriented(AVS, Explorer,..) or "self-contained" (Analyze, 3DVIEWNIX, etc) visualization packagescould handle large data sets whose size exceed both the size of the memory and swap space. The user is thus compelled to manually extract smaller pieces of the data sets, which can be time-consuming and inefficient. It is thus mandatory to tightly link the data extraction and data visualization steps, typically by reading and visualizing one slice at a time instead of loading a 3D volume in memory before visualizing its 2D slices.

The efficiency of interactive visualization is also timited by the GUI design, which can almost never be customized by the user. Even when the source code is available, adding or removing features is very difficult if not impossible, as the GUIs are implemented as monoliths of hundreds of thousands of lines. However, the user needs to control the way the colors €ue allocated, €.8., interactive thresholds, linear and non-linear colormaps. Similarly, extracting profiles, histograms or non-rectangular regions of interest make quantitative measurements possible. Off-line data transforms or extractions are in our opinion too cumbersome in a research environment. In addition, unsupervised automatic segmentation performs poorly when the data are noisy. The alternative, that is the use of manual and semi-automatic segmentation techniques, is however limited by the lack of flexibility of most graphical interfaces. The considerations above led us to implement a new visualization tool, geared to large data sets. Since we could not a^fford, nor had the experience required to write a self-contained application in XMotif, our approach was to divide the GUI into small reusable components by relying on Tcl/Tk. Tcl/Tk is now used by thousands of users in every domain of graphical and engineering applications. "glue" difIndeed, Tcl/Tk provides simple ril/aysto ferent modules together and it can be extended easily, in contrast to other GUI builders. However, the use of Tcl/Tk in the signal and image processing community is scarce. Two main reasons can explain this situation. First, efficient data management mechanisms and number-crunching capabilities in image processing are generally believed to require high-performance languages, in contrast to Tcl which only handles character strings. Next, the photo image widget was not designed for interactive visualization and its flexibility is very limited, mainly because image processing applications are not the main focus of Tcl/Tk developers. In this paper, we describe the implementation of a new gray-scale image widget. By focusing on data management and color allocation problems, we were able reach a level of performance which compares favorably with other image processing and visualization packages. A transparent overlay mechanism provides a link to manual a^ndsemi-interactive segmentation techniques. We will show how this widget helped us build VISU, a flexible VISUalization software described below. VISU is made up of several sta^ndalone applications which communicate with each other over the network. The power a,nd flexibility of these hypertools in image and volume visualization will be described with some examples.

A biomedical application will be presented.

2

The p'ict gray-scale image widget

As it can be guessed from the name, the pict grayscale image widget is based on the photo image widget. After few experiments, it became clear that the photo widget was not appropriate for our application. First, this widget can only handle 8- or 24-bit color images, and it does not provide any mechanism to visualize floating-point images. Irtrext, the colors are allocated statically and cannot be changed dynamically by the user. Thus, we decided to implement a new widget by focusing on data management and color allocation.

2.1

Data management

Since we wanted to support byte, short, integer or float-point types, we modified the data structure, in order to remove the fields related to color management and dithering. The raw data is allocated in a block of memory and can be accessed by using the data type information. The master structure also provides a pointer to a block of byte data, corre-' sponding to the normalized raw data. The contrast can be increased or decreasedby setting the dynamic range of the raw data. This feature proves most helpful when comparing two floating-point images whose dynamic range is different. The pixel values can be queried, and the resultant string contains the actual value, for example a floating-point value. The Khorosl and VIEW (local LLNL format) file formats are now supported. The GIF and PPM readers were modified in order to read only one color band. Readers/writers for the SUN Raster file format are provided for both the photo and the pict widget. Raw binary files can also be read from a file (or a channel if the code is linked against Tcl7.5) by specifying the dimensions, the data type and the number of bytes corresponding to the header. In all cases,only one slice is read at a time, which decreasesthe memory requirements dramatically. However, volume rotations and transpositions need to be done off-line. Support will be added in the neaxfuture for HDF. netCDF and ACR-NEMA file formats. Profiling the Photo source code showed that me' mory management was tairly inefficient, tr â lot of time is spent copying blocks between different addresses. The TkPictPutBlock and TkPictPutZov

medBlock routines were rewritten to make sure memory blocks are duplicated only when necessary.

2.2

Color allocation

The human visual system cannot see more than 60 shades of gray, which makes color management for gray-scale images much more simple than for color ones. Since we wanted to change the colors dynamically and use predefined colormaps, we chose to display the images using an 8-bit PseudoColor Display. This requirement is in our opinion fairly minimal. Besides, our experience proved that dithered images cannot be compared accurately. In order to allow for fast array transformations, the colors axe allocated from a contiguous set. For example, the palette can be inverted quickly by reversing the color indices. The images are displayed in false colors by choosing from a variety of prefined look-up tables. The histogram can be stretched or thresholded to produce a binary image. Furthermore, the colors c€m be allocated from shared, de. fault or private colormaps. Changing the colors of one shared colormap will affect all the images that share it. This feature allows the user to visualize the same image displayed with different colors, or to compare the result of two different thresholds. One additional benefit is that the color allocation can be used to display "semi-transparent" overlays. This is a feature that was found very useful in our interactive segmentation work. The user can for example "paint" on the image, draw polygons, Bezier or free-form curves, and yet guess the gray-level values. The overlays can be saved as a miÉk image. Let us point out that this feature enables the user to extract non-rectangular regions of interest. Alternatively, a mask image can be overlaid on top of the active image, in order to check and verify the accuracy of an off-line segmentation, remove spurious pixels or compare two data sets. The user can interactively combine overlays with logical operations (or, xor, and, etc..) by changing the Graphic Context. The pict widget provides an interface to advanced image processing routines [1, 2], and the overlays can also be used in semi-interactive segmentation 13],where a coarse initialization is specified; the segmented result can then be displayed. Examples in Figure t(a)-(d) show two cross-sectionsof a CT volume displayed with different look-up tables. The concept of transparent overlays is described in Figure 2(a)-(c). The use of overlays in semi-automatic segmentation is presented in Figure 2(d)-(f).

3

VISU: a volume VISUalization application

In this section, we describe how the pict widget is incorporated in VISU, a user-friendly volume visualization extension of wish. In addition to the lowlevel widget commands, we provide a set of default scripts and high-level commands which make the life of the average user easier. In order to remove any learning curve for LLNL users, the synta;c of these commands reproduces that of VIEW, a general signal processing package used in our group. For example, the following commands will read the first slice of volume "hand", display the image, and then display the tenth slice. ./ rdfile hand-0.sdt f % di sp t 7. rdslice f 10 The simplicity of these commands enables the user to write his/her own set of macros. In addition to the command line, VISU also provides a graphical interface to most of the high-level commands. In order to reduce the GIII design, we chose to provide only one Control Panel (see Figure 3) and a one-to-one mapping between images and windows. At a given time, one image is considered "active". An image becomes active when the user clicks on it, and its window title is changed. The mouse location and pixel values displayed in the Control Panel correspond to those of the active window. Similarly, moving the slice scale will result in another slice being displayed; the dynamic range of the images can be typed in two entry boxes. The Palette Menu (Figure 4) lets the user change the colors with the mouse. Four scales can be moved in order to choose the low and high thresholds. The pixels whose values are between the low and high threshold appear white, and the rest appear black. As soon as the scale is changed, the look-up table is updated. Visualizing the actual values of the thresholds instead of normalized values proved very helpful. For example, in our CT applications, the user can see where the attenuation value exceeds a threshold. Predefined look-up tables can be loaded by clicking on one of the radiobuttons. A color tool makes it possible to stretch the colormap in a nonlinear way. The graphical interface allows the user to change the intensity and each of the RGB channels independently to create their own look-up table. In the Overlays Menu (Figure 5), the active mask can be chosen and overlaid onto the active image. The user can choose how to combine the overlays by

setting the overlay Graphic Context with the mouse. For example, the intersection of two binary masks can be seen and saved into a new image. We also provide a graphical interface to our segmentation routines. The GUI can be customized within minutes without recompiling any code. For example, displaying several images side-by-side could be done by packing them in the stune canva.s, instead of difierent windows. This flexibility enables us to design the best GUI and to take into account the requirement of a specific imaging application. In most cases,the user will be able to configure the GUI himself. Releasing the source code on the Internetl helped test VISU on various platforms and operating systems we did not have access to. The configure tool generates Makefiles automatically. While it is still dependent on Xlib, VISU can be ported to Windows and Mac-OS without too much effort, to become a machine-independent visualization sofbware. About 500 anonymous ftps were logged on our server. Let us remark that this figure may appear small, but most of the VISU users have very specific needs and would probably not use Tcl/Tk at all if this grayscale image widget did not exist.

4

the dynamic range is likely to be different for each slice. Moreover, visualization tools may run on different systems and display the images on the s€une screen. In summary, choosing a correct dynamic range is a tedious time-consuming task. However, the send command makes things simple, as the following script demonstrates:

proc set-range {} { # initializations set 1 [winfo interps] set mi n { }; set nax t};

# l i st vi su appl i cati ons foreach k $1 { i f { [stri ng natch " vi su* " [lindex $k 0] I ] t lappend visu-list $k ) )

So far, the features of VISU we described are tairly standard, ffid can be found in other less flexible visualization packages. However, the Tk library makes it possible to use our widget to build visualization hypertools, defined in John Ousterhout's book [ ] * "stand-alone applications which can communicate with each other and be reused in ways not foreseen by their original designers". Indeed, it came as a surprise to us how easily the send command can be used by visualization tools based on the pict widget in peer-to-peer or master/slave relationships. In order to demonstrate the power of these hypertools, we take several examples.

# compute the global ninimun set fmin [lindex $nin 0] set fmax [lindex $nax 0] foreach k $nin { if{$t<<$tnin}t set fnin $k ) if{$t<>$tnax}t set fnax $k ) ) # set the new dynamic range foreach k $vi su-l i st { range) send $k t$curr-ing )

Image visualization

We mentioned several times in this paper the importance of normalizing floating-point images in the sarne way. Typically, this can be done by querying the minimum a^nd maximum 'ralue of each image, and by computing the absolute minimum and maximum of all the images being displayed. Setting the dynamic range has to be done each time a^nimage is updated; for exa.rrple, in volume visualization, 1ftp : redhook. llnl. gov/pub/visu //

{}

\

# query dynamic rânge of active images foreach k $vi su-Ii st { lappend min \ getnin}J [send $k {$curr-ing lappend nax \ getmax}J [send $t< {$curr-ing )

Hypertools

4.L

set vi su-l ist

arrd maximun

$fnin

$fnax

) In this script, the dynamic range is broadcast to all the visu applications. Other possibilities include showing the pixel values at the sarne mouse location in different images, so as to compaxe images on a pixel-by-pixel basis. Using the visu scripts, this option could be written as the following comma.nd.

f or eac h k $ v i s u -l i s t { send $k {set x $x} send gk {set y $y} send $k {set pix [$curr-img )

4.3 get $x $y]]

Instead of sending values, it is possible to send commands, and for example to visualize the same slice in different data sets by broadcasting the comA direct mand rdslice $curr-irng $curr-slice. application is the creation of simultaneous animations of volumes located in different file systems and accessedover the network.

4.2

Image profiles and histograms

Visualization of image profiles along an arbitrary line is an invaluable tool to evaluate the presence of artifacts in reconstructed images. For example, in X-ray CT, beam-hardening produces "cupping" artifacts that can easily be detected by extracting a profile. Similarly, the accuracy of the reconstruction techniques can be assessedby visualizing the profiles of sharp edges. Typically, low-pass filtering and noise elimination smooth and spread the transitions. Along the same lines, visualizing a histogram helps understand the statistics of an image and the distribution of its gray levels. In the case where the histogram is made up of several modes, the objects can typicaly be segmented out by applying different thresholdings to the data. Although most image visualization packagesprovide 1D signal viewers, our experience proved that they cannot be easily customized by the user and are not flexible enough. In our applications, the lD signals are extracted by issuing a command to the pict image widget and sent to a BLT graph. Tcl/Tk handles only character strings and relies on the X protocol to communicate between applications. In addition, the profiles are generally made of less than 1000 samples, so that the overhead introduced by the communications is minimal. This feature allows us to let the user configure the graphical interface. Figure 6 shows €ul example where a profile is extracted interactively and then sent to a 1D signal viewer. The main benefit of this approach is that the user can dynamically cre' ate and extract new profiles or histograms of images viewed across the network, combine and comp€Ire them without having to save these one'dimensional signals in files, and transfer them by ftp.

Data-flow environments

In our biomedical application, all the data prepro. cessing and segmentation is implemented in Khoros, a data-flow image processing environment. Khoros provides a network editor; the input data undergo a series of transformations each time a network node is executed by the scheduler (Figure 7). This data flow environment helps the user design an imaging application, as the output of each node can be visualized by connecting it to a visualization module (other packages such as AVS, Explorer, LabView use the same paradigm). However, dynamically setting the range of all the images in a data flow environment is next to impossible. Similarly, writing a macro or a loop in a command-line interface sometimes makes more sensethan editing a network of nodes. These two problems were solved by implementing a Khoros module that executes a VISU script. This approach results in several important benefits. First, it compensates for some of the drawbacks of VISU, e.g. by providing simple ways to extract arbitrary slices. Basically, we rely on modules provided in the Khoros distribution, or write our ou/n modules to filter and transform the raw data. Next, the power of both command-line and data-flow environment can be combined: a command-line interface is provided by a remote controller derived from the rmt example of the Tk distribution, which can communicate either with a specific application or send messagesto all of them. As a result, the user can rely on the power of visual programming, while being able to rely on a more traditional command-line interface. To our knowledge, this feature is not provided by any other visualization package. Finally, the embedded visualization modules can communicate with each other. For example, the same profile could be extracted in different images and displayed in the same lD signal viewer. Let us remark that the same approach could be easily implemented in other data-flow environments such as AVS or Explorer.

4.4

Volume visualization

As described before,VISU candisplayslicesof a 3D volume. Another way to visualize a volume is to extract isosurfaces,either by using the 3D Marching Cubes algorithm [5] or by reconstruction of a 3D surface from 2D contours [6]. Visualizing a surface helps understand the spatial distribution of objects in 3space, in contrast to visualizing 2D slices. Almost every visualization package provides these tools, but they are almost never combined or used simultane. ously.

Several other approaches make use of Tcl/Tk Schroeder, Lorensen and in volume visualization. Martin recently released wk2, a visualization toolkit which can be used either by writing C++ prograrns, or through Tcl/Tk scripts. Similarly, Lacroute [7] released VolPack 3, a volume rendering library coupled with a Tcl/Tk-based graphical interface. These two toolkits could be used in conjunction with VISU in order to build a volume visualization package by relying on the send command. This ap proach results in two main benetts. First, the different visualization modules can be combined easily, even if they are not executed on the same machine. Typically, it will be possible to highlight a point of the surface and to see the intersecting slice. Conversely, viewing a new slice will automatically change the coordinates of this highlighted point. The color tools provided with VISU will also be used in order to choose isosurface values, or the transfer functions in volume rendering. The sarne graphical interface can also be used to set the viewing pararneters in surface and volume rendering. It is our belief that the combination of these visualization techniques will help identify more efficiently the relevant features of volumetric data sets. Next, provided that the interface between hypertools does not change, the user does not need to know how the surface or volume rendering techniques are implemented. As a result, the best visualization modules can be chosen by the user. Flor example, vtk supports an abstract rendering engine and can be slower than the tkSM widget a, which provides easy irccess to the OpenGL and Mesa libraries. In the case where the software modules were not linked against Tcl/Tk, they can still be used and executed from the Tcl/Tk environment.

in the near-future to enhanceour image processing library and to port the code to Windows and Mac. The Tcl7.5 socketswill also be used to develop distributed visualization applications for the Internet.

5

[5] W.E. Lorensenand H.E. Cline. Marching Cubes: a high resolution surface extraction algorithm. ComputerGraphic,s,21(3):163-169,1987.

Conclusion

Although Tcl/Tk is widely distributed in engineering and graphical applications, its use in imaging and visualization is recent. In this paper, we de scribed a new gray-scale widget and its use in visualization of large data sets. Although it relies on a interpreter, its speed compares favorably with typical X-Motif self-contained applications, as we impl* mented an efficient management of data and colors. Desiguing high level commands and conbining hypertools helps perform tedious tasks automatically, thus increasing the interastion efficiency. We plan 2bttp: wttw.cs.rpi.edu:80/- martink // 3http: //www-graphics. stanford.edu/eofhrare/volpack/ ahttp: www. isr. umd.edu/ ihsu/tksm. html //

6

Acknowledgments

This research was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under contract No. \ry-740SENG-48 and LDRD grant No. 9GERI-003, with the support of H. Martz and K. Hollerbach. J. Pearlman, M. Abramowitz, E. Nicolas, J.P. Hebert, M. Cody and D. Garrett helped improve VISU by providing invaluable bug reports and suggestions.

References [1] L. Vincent and P. Soille. W'atershedsin digital spaces: An efficient algorithm based on immersion simulations. I.E.E.E. Tbonsoctiotts on Pottert Anolysis ond Machine Intelligenen, 13(6):583-598, June 1991. [2] L. Vincent. Morphological grayscalereconstruction in image analysis: Applications and efÊcient algorithm.s. I.E.E.E. Tlunsoctions on Imoge Processing, 2(2):L76-201,April 1993. [3] P-L. Bossart. Détection de contours réguliers dans des images bruitees et texturées : association des contours actifs et d'une approche multiechelle.Thèse,Institut National Polytechnique de Grenoble,Octobre 1994. Tcl and the Ik toolkit. [4] J.K. Ousterhout. Addison-Wesley,1994. ISBN G201-63:137-X.

t6l B. Geiger. Three-Dimensionol Modeling of Humon Oryatls ond its Applicotion to Diagnosisand SurgicalPlonning. PhD thesis, Ecole des Mines de Paris, 1993. [4 P.G. Lacroute. Fast Volume Rnndering Using a Sheor-Wol Foctorizatbn of the Viewing ?frz;nsfortnatbn" PhD thesis, Stanford University, 1995. also Technical Report: CSLTR-9S678.

Figures

'ct' Figure 1: Cross-sectionsof a thumb and index near joint: (a) corresponds to slice 308 displayed with a 'gray' private colormap; (b) is also slice 308 with a shared colormap; (c) corresponds to slice 314 with a private colormap; (d) is slice 314 with a shared colormap. In this example, the private colormaps ilre used a^sa reference while the colors are changed simultaneously in the two shared colormaps.

(')

(d /

(eJ

(f )

Figure 2: Cross-section of a hand: (a) contours from an off-line segmentation; (b) contours overlaid on the image; (c) Regions created from these contours; (d) Coarse manual initialization of overlays; (e) Interactive segmentation result after region-growing; (h) Resulting segmented mask

Figure 3: Control Panel

Figure 4: Palette options

Figure 5: Overlaysoptions

sqdFg $jrr pclËlti.:::::r,.,,,,ùt@e pçgc

]Ï;ilrr

landtcrpr ç ncrcrdfng

Figure 6: The profiles extracted in VISIJ are sent across the network to a graph viewer

Figure 7: Khoros data flow environment