Input scene restoration in pattern recognition correlator based on digital photo camera Sergey N. Starikov, Nikita N. Balan, Mikhail V. Konnik, Vladislav G. Rodin, Ivan V. Solyakin, Ekaterina A. Shapkarinaa a Moscow

Engineering Physics Institute (State University), Russia, Moscow ABSTRACT

Diffraction image correlator based on commercial digital SLR photo camera was reported earlier. The correlator was proposed for recognition of external scenes illuminated by quasimonochromatic spatially incoherent light. The correlator hardware consists of digital camera with plugged in optical correlation filter unit and control computer. The kinoform used as correlation filter is placed in a free space of the SLR camera body between the interchangeable camera lens and the swing mirror. On the other hand, this correlator can be considered as a hybrid optical-digital imaging system with wavefront coding. It allows not only to recognize objects in input scene but to restore, if needed, the whole image of input scene from correlation signals distribution registered by SLR camera sensor. Linear methods for image reconstruction in the correlator are discussed. The experimental setup of the correlator and experimental results on images recognition and input scenes restoration are presented. Keywords: diffraction correlator, imaging with wavefront coding, digital camera, pattern recognition, image restoration. Keywords: diffraction correlator, imaging with wavefront coding, digital camera, pattern recognition, image restoration

1. INTRODUCTION The direct digital processing of registered images essentially expands capabilities of optical systems. Opticaldigital information processing systems are now actively investigated.1, 2 In these systems the digital electronic post-processing of images is combined with processing performed directly in an optical channel. In our previous report3 diffraction image correlator based on commercial digital SLR photo camera was described. The correlator is intended for preliminary recognition of the two- and three-dimensional scenes at the step of their registration by the camera at quasimonochromatic spatially incoherent illumination. The principal optical scheme of the correlator is analogous to that of incoherent holographic correlator by Lohmann4, 5 used quasimonochromatic spatially incoherent light. The correlator hardware consists of a digital SLR photo camera with attached optical correlation filter unit and, if necessary, control computer. No modifications have been introduced in units of commercial digital SLR photo camera. Correlation filter unit is placed in a free space of the SLR camera body between the interchangeable camera lens and the swing mirror (see Fig.1). To achieve processing of whole area of the camera frame, the kinoforms are used as correlation filters6 instead of Fourier holograms. As the amplitude and phase Fourier holograms have more than one diffraction order, and the reference image is formed in the lateral order, the acquisition of the correlation of whole field of frame with the reference image is impossible. If kinoforms are used, only the axial diffraction order, containing the reference image, is present. Thus, the camera registers only signals of mutual correlation of input images with the reference image, which have been written on synthesized kinoform. Computer connected via camera interface provides control of camera shooting, post-processing, transmission and storing of detected correlation signals. The aim of the present paper is to extent capabilities of this kind of image correlators by giving them abilities of imaging systems. The described correlator can be interpreted as an optical-digital imaging system based on “wavefront coding” or “pupil-phase engineering” principle. The role of encoding phase mask is carried out by kinoform used as correlation filter for pattern recognition. Correlation signals captured by camera sensor can be regarded as an “intermediate“ image of input scene in front of the camera. Therefore one can try to digitally reconstruct “true“ image corresponding to that one captured by the camera without inserted correlation filter.

Figure 1. The scheme of digital camera based image correlator for pattern recognition.

This ability may be of practical interest when considerable reducing of dataflow for remote control of scene is demanded. Only snapshots or their fragments with significant correlation signals can be transmitted and stored. Image reconstruction of selected input scenes may be performed subsequently for various additional analyses if needed.

2. EXPERIMENTAL SETUP FOR CORRELATOR TESTING The scheme of the experimental setup for test objects recognition and input image reconstruction by the correlator is shown in Fig. 2. The setup consists of illumination assembly (units 1 - 11) and the diffraction image correlator (digital photo camera 12 and kinoform 13 with the recorded reference image). The computer 14 was used for transferring, postprocessing and storing images of correlation signals, and also for shooting control and camera parameters setting. The radiation of He-Ne laser 1, passed though attenuating polarizer 2, is divided into two beams by beam splitter 3. The first beam used for registration of point spread function (PSF) of the optical system of the camera with kinoform. For this purpose, the radiation reflected by mirrors 4 and 5 and passed though attenuating polarizer 6 is focused by a microscope objective 7 on the pinhole posed in the objects plane 8, registered by the camera 12. The diverging spherical wave forms in the plane of the camera CCD-sensor PSF of the optical system of the correlator. The second beam is necessary for illumination of shooting object by monochromatic spatially incoherent light. The radiation is extended by microscope objective with the cleaning diaphragm 10 and, passed through the rotating ground glass, illuminates objects 8 by monochromatic spatially incoherent light. In experiments the commercial digital SLR photo camera with following main parameters was used: - number of CCD pixels 3024×2016, - pixel size - 7.6×7.6 µm2 (matrix size - 23.0×15.5 mm2 ), - sensitivity - ISO 100÷1600, - analog-to-digital converter - 12 bit, - interchangeable lens - 2.8/100, interface - IEEE1394 for data storage and shooting control. The characteristics of kinoform are: diameter – 5 mm; the size of a recurring segment – 2 mm; the pixel size – 4 µm. The size of pulse response of the kinoform (the reference image) on a CCD-sensor of the photo camera was of 2.7×2.3 mm2 (350×300 pixels) at wavelength of 0.63 µm and the distance from kinoform up to CCD-sensor was equal to 50 mm. The intensity of zero-order peak exceeds intensity of other points of kinoform’s response in 2.5 times.

Figure 2. The scheme of the experimental setup for test objects recognition and input image reconstruction by the correlator: 1 - laser, 2, 6 - attenuators, 3 - beam splitter, 4, 5, 9 - mirrors, 7, 10 - microscope objectives, 8 - object, 11 rotating ground glass, 12 - digital SLR photo camera, 13 - kinoform, 14 - computer.

3. EXPERIMENTAL RESULTS The reference image of bicyclist recorded on the kinoform is shown in Fig. 3. Pulse response of the kinoform used in experiments as correlation filter for recognition of test objects is presented in Fig. 4. Plane images of test objects on lists of a paper were positioned before the photo camera. The input scene was illuminated by spatially incoherent radiation of the He-Ne laser. The recognition of input scene with multiple objects is illustrated in Fig. 5. The size of each object on a paper list, shown in Fig. 5a, was about 35 mm. That one on CCD-sensor was about 350 pixels. The image contrast of test objects was less than 8:1. The view of correlation signals of input objects with reference one registered by photo camera is shown in Fig. 5b. A local light spots indicate positions in the input scene of two objects similar to reference one. The signal-to-noise ratio of correlation peaks is more than 1.2. The view and intensity distribution of correlation signals after additional digital post-processing are shown in Fig. 5c and Fig. 5d. The signal-to-noise ratio of correlation peaks has increased up to value of 2.1 after the post-processing. The post-processing of correlation signals includes: median filtering over area of 3×3 pixels (elimination of “hot“ pixels) and subsequent subtraction of correlation signals locally averaged over area of 64×64 pixels (for sharpening correlation peaks). The results of recognition experiments confirm that with this correlator it is possible to determine the presence and the position in the input scene of objects similar to the reference one. The error of position determination was lesser then 0.02 sizes of object. Experiments on input scene image reconstruction are described in next section.

Figure 3. Reference image for test objects recognition.

Figure 4. Pulse response of the kinoform on the CCD-sensor of the photo camera.

a

b

c d Figure 5. Recognition of multiple objects: input objects in front of camera (a), view of correlation signals registered by camera CCD-sensor (b), view (c) and intensity distribution (d) of correlation signals after digital post-processing.

4. IMAGE RECONSTRUCTION FROM CORRELATION SIGNALS Because correlation signal is a convolution of original input image and point spread function of kinoform, deconvolution algorithm for image reconstruction is required. Restoration of an image from observed data is often an ill-posed inverse problem. The solution of such inverse problems can be achieved through regularization methods, which turn the problem into a well-posed one, and prevent the amplification of measurement noise during the reconstruction process. Among others, Tikhonov7 regularization method produces good results, computationally efficient and comparable easy to implement, that is significant for large correlation images. The version of Tikhonov deconvolution filter, used in this work, is described in frequency domain as

F =

|H|∗ · G, |H|2 + α · max|H|2

(1)

where F is desired Fourier spectrum of reconstructed image f, G is Fourier spectrum of correlation signal’s image g from camera, H Fourier spectrum of kinoform’s PSF, and is regularization parameter, which corresponds signal-tonoise ratio. Regularization parameter is connected with mean squared signal to noise ratio (SNR) as 1 SN R = √ α

(2)

Numerical experiments were performed with optically convolved images (correlation signal’s images), one of them is presented in Fig. 5b. Fig. 5a shows the original image to be reconstructed, and Fig. 4 presents kinoform’s PSF. According to preliminary numerical experiments, degree of regularization can be varied by parameter within the ambit of 10−1 to 10−9 . Algorithm is implemented as a MATLAB program, which can work with TIFF or PPM image file formats. All images of correlation signals from digital camera were stored in RAW format to prevent quantization errors and image degradation. For RAW format conversion, open-source GNU GPL licensed Dave Coffin’s DCRAW converter8 was used. Images from RAW format were converted in 16-bit PPM format with linear gamma to avoid degradation of convolved image. The quality of reconstructed images was estimated numerically by normalized root mean square (RMS) error between original f0 and deconvolved f images

E=

sP

|g(x, y) − f (x, y)|2 P |f (x, y)|2

(3)

Using aforementioned Tikhonov-based algorithm (Eq.1) for image deconvolution and varying regularization parameter, it is possible to reconstruct convolved image. After deconvolution, normalized RMS error from original image was calculated. Results on RMS measurements are presented in Fig. 6

Figure 6. RMS error of deconvolved images as a function of regularization.

The root mean square error minimum corresponds to regularization parameter α between 10−2 and 10−4 . More precise measurements were required to find exact minimum of RMS for this type of images. Results of more detailed measurements with increment of equal to 10−0.1 are presented in Fig. 7. Lowest root mean square error, as it can be see from Fig.7, is reached with α = 10−2.8 . But in visual observations, best deconvolved image in the group corresponds to α = 10−4 as shown in Fig.8. The original image, presented in Fig.5a, was convolved optically with PSF, shown in Fig.3. Result of convolution is presented

Figure 7. Detailed RMS error of deconvolved images as a function of regularization parameter.

in Fig.5b. Image in Fig.8b, which is optimal by means of root mean square error, seems to be blurred and noised in compare with that one, shown in Fig.8c, which is visually best in the group. Thus, RMS optimal deconvolved image may not correspond to visually best image in the group.

a

b

c

d

Figure 8. Images of correlator’s input scene reconstructed by Tikhonov-based algorithm with different regularization parameter : a) 10−1 - blurred, b) 10−2.8 - RMS optimal, c) 10−4 - visually best in the group, d) 10−6 - noised and degradated.

In Fig.8d presented image with deficient regularization parameter α, which is 10−6 for this image. Lower regularization corresponds to higher noise level, but excessive regularization produces blurred and unidentifiable image, as shown in Fig.8a. Obtained results agreed with estimations of signal to noise ratio of commercial digital photo camera used in the correlator. Measured SNR was equal approximately to 40, which gives (see Eq.3) RMS optimal value of regularization parameter equal to 10−3.2 . It is significant to show recovery PSF’s, which produced by deconvolution filter (Eq.1). Fig.9 and Fig.10 shows absolute value and phase of PSF’s respectively,

which corresponds to images, restored by Tikhonov-based deconvolution algorithm with different . Differences between original image of input scene of the correlator (see Fig.5b) and reconstructed images (see Fig.8) are characterized by summary PSF of optical-digital imaging system, which includes optical correlator and digital reconstruction procedure. Summary PSF’s for some values of regularization parameter are presented in Fig.11.

a

b

c

d

Figure 9. Absolute value of PSF’s, found by Tikhonov-based algorithm with different regularization parameter : a) 10−1 - excessive regularization, b) 10−2.8 - RMS optimal, c) 10−4 - visually best in the group, d) 10−6 - deficient regularization.

a

b

c

d

Figure 10. Phase of PSF’s, found by Tikhonov-based algorithm with different regularization parameter: a) 10−1 excessive regularization, b) 10−2.8 - RMS optimal, c) 10−4 - visually best in the group, d) 10−6 - deficient regularization. Black colour means phase equal to 0, and white colour corresponds to π.

a

b

c

d

Figure 11. Summary PSF’s of optical-digital imaging system, found by Tikhonov-based algorithm with different regularization parameter: a) 10−1 - excessive regularization, b) 10−2.8 - RMS optimal, c) 10−4 - visually best in the group, d) 10−6 - deficient regularization.

Resolution of reconstructed images could be estimated by summary PSF’s width on half-height. There were obtained that widths on half-height of summary PSF’s equals 24 pixels for α = 10−1 , 10 pixels for α = 10−2.8 , 6 pixels for α = 10−4 , and 3 pixels for α = 10−6 . Thus, image in Fig.8a, is blurred because summary PSF is too wide, optimal by RMS image is also slight blurred, whereas best image in group by visual observation is sharp enough. Summary PSF, presented in Fig.11d, corresponding α = 10−6 , offers resolution, which is nearly to that one of used photo camera, but the presence of noise leads to degradation of reconstructed image (see Fig.8d).

All aforementioned results of deconvolution retrieved from single correlation picture, which wasn’t averaged. To compare results of deconvolution from individual correlation image and averaged, 16 shots of the same scene were taken. Comparison of RMS for single and averaged picture deconvolution are presented in Fig.12.

Figure 12. Comparison of deconvolution’s RMS error for individual and averaged images.

One can see that there are almost no differences between averaged and non-averaged images, so that could result to higher performance of optical-digital correlator. It is became possible by using Dave Coffin’s DCRAW, converting to linear 16-bit PPM format and digital camera noise estimation.

5. CONCLUSIONS It is shown, that correlator based on commercial digital SLR photo camera can be interpreted as an opticaldigital imaging system based on “wavefront coding“ or “pupil-phase engineering“ principle. Correlation signals captured by camera sensor can be regarded as an “intermediate“ image of input scene in front of the camera. It allows not only to recognize objects in input scene but to restore, if needed, the whole image of input scene from correlation signals registered by camera sensor. This extents capabilities of this kind of correlators by giving them abilities of imaging systems. This ability may be of practical interest when considerable reducing of dataflow for remote control of scene is demanded. Only snapshots or their fragments with significant correlation signals can be transmitted and stored. Image reconstruction of selected input scenes may be performed subsequently for additional analyses if needed. The experimental setup for correlator testing and experimental results on recognition and restoration of images of input scenes are presented. Version of Tikhonov deconvolution filter was used for input images reconstruction from correlation signals. Among linear methods for image reconstruction, Tikhonov regularization method produces good results, computationally efficient and comparable easy to implement, that is significant for large correlation images. To find optimal deconvolution result, regularization parameter was varied. Comparison of reconstruction from individual snapshots and that one from images, averaged over 16 shots of a same scene, demonstrates that results are practically identical in both cases. Obtained RMS error estimations of reconstructed images from correlation signals agreed with estimations of signal to noise ratio of the photo camera used in the correlator. Best of reconstructed image have resolution, which is twice lesser than camera resolution, due to noise of digital photo camera sensor. Available static kinoforms were used as correlation filters in the experimental setup of image correlator based on digital photo camera. Considerably large functionalities of the correlator will be achieved with real time re-writable kinoform, implemented on electronically controlled spatial light modulator. It is possible to expect an appearance of modulators with parameters acceptable for these purposes.

ACKNOWLEDGMENTS This work was partially supported by the Ministry of education and science of Russian Federation (Program “The development of the scientific potential of High School“, project RNP.2.1.2.5657).

REFERENCES 1. W.T.Cathey and E.R.Dowski, “New paradigm for imaging systems,” Appl. Opt. 41, pp. 6080–6092, 2002. 2. S. Prasad, T. Torgersen, V. Pauca, R. Plemmons, and J. van der Gracht, “Engineering the pupil phase to improve image quality,” Proc. SPIE 5108, pp. 1–12, 2003. 3. S. Starikov, N. Balan, V. Rodin, I. Solyakin, and E. Shapkarina, “Pattern recognition correlator based on digital photo camera,” Proc.SPIE 6245, 62450C, 2006. 4. A. Lohmann, “Matched filtering with self-luminous objects,” Appl. Opt. 7(3), pp. 561–563, 1968. 5. Y. Bykovsky, A. Markilov, V. Rodin, and S. Starikov, “Optical information processing with transformation of the spatial coherence of light,” Quantum Electronics 25(10), pp. 1014–1019, 1995. 6. S. Starikov, V. Rodin, E. Shapkarina, I. Solyakin, and A. Chervonkin, “Incoherent acoustooptic image correlator with the kinoform,” Proc.SPIE 5437, pp. 301–308, 2004. 7. A. N. Tikhonov and V. Y. Arsenin, Solutions of Ill-posed Problems, V. H. Winston & Sons: Washington, DC, 1977.