Enhancing Dynamic Range of Optical-Digital Correlator Using Assorted Pixels Technique Mikhail V. Konnik*a , Sergey N. Starikova a Moscow Engineering Physics Institute (State University), Russia ABSTRACT High dynamic range (HDR) registration of correlation signals allows increasing recognition’s reliability. For such purpose, digital photo sensors with Bayer colour filters array can be utilized. Bayer’s mosaic is considered as an array of attenuating filters in the quasimonochromatic light. When main colour pixels related to wavelength of the light source are saturated, pixels covered by different colour filters (accessorial pixels) are generally not. Hence oversaturated regions of the image of correlation signals can be reconstructed using information from other colour channels. The registered image of the correlation signal is mapped to the linear HDR image in a simple way. The application of such technique called Assorted Pixels for obtaining linear HDR images of correlation signals from digital photo sensors with Bayer mosaic is presented. Experimental results on HDR images reconstruction of correlation signals are presented. 1 INTRODUCTION The photo registration with high dynamic range (HDR) is demanded in such practical applications as optical-digital correlators. Using conventional technical solid-state cameras, it is possible to register light distributions with relatively low dynamic range of 40-70 dB. As an example, many input scenes for optical-digital correlators are characterised by high dynamic range (HDR). Correlation signals of such HDR scenes are difficult to register properly on conventional solid-state photo sensors that lead to deterioration of correlators’ reliability. In order to overcome such problem, both hardware [1, 2] and software [3, 4] methods were proposed recently. In this paper we describe the application of the Assorted Pixels technique [3, 4] for enhancing dynamic range of the optical-digital correlator [5]. For such purpose we utilize Bayer colour filters array (CFA) of the digital camera’s sensor that is used for the correlator. Bayer CFA can be considered as an array of attenuating filters in the quasimonochromatic light. When pixels related to wavelength of the light source are saturated, pixels covered by other colour filters are far from saturation. Extracting data from pixels of different colour-filter type it is possible to recover an HDR image from a single registered oversaturated image. Experimental results on HDR images reconstruction of correlation signals from digital photo sensors with Bayer mosaic using Assorted Pixels technique are provided. Quantitative estimations of the dynamic range’s increase are presented. * [email protected],

[email protected]; phone +7 (495) 3247403; Sergey N. Starikov: [email protected]

2 LINEAR HDR IMAGING USING ASSORTED PIXELS TECHNIQUE The HDR registration of correlations signals is often necessary because of wide DR of input scenes for optical-digital correlators. Among others, the software approach such as Assorted Pixels technique [3, 4] can be used for enhancing the dynamic range of the optical-digital correlator. The correlator described in [5] uses the quasimonochromatic spatially incoherent light and the photo camera with Bayer’s CFA. These circumstances make it possible of application of Assorted Pixels technique. If the exposure value is long enough, some of pixels may be oversaturated but neighbour pixels under other light filters are generally not. Utilizing data from the neighbour pixels it is possible to restore oversaturated regions of the captured image. Knowledge of Bayer CFA’s relative transmittance to the desired light is essential for restoration of oversaturated regions of images. Values of relative transmittance are estimated from the direct measures. Let is denote the relative transmittance of red pixels as e1 ; transmittance coefficient for green pixels as e2 (e2 < e1 ); transmittance coefficients for blue pixels as e3 (e3 < e2 < e1 ). For restoration of oversaturated data it is necessary to evaluate the parameters of radiometric functions, which is camera’s response to illumination versus exposure value. Assume that the radiometric function for each colour channel can be approximated as Sn = an · E + bn , where Sn is the value of the registered signal, an and bn are a slope and bias of line approximation for n-th colour channel, and E is exposure value. Hence using approximation coefficients an and bn , it is possible to correct oversaturated value of the 2 main pixel S1 by the value S2 of green pixels S1 = a1 · S2a−b + b1 and by the value S3 of 2 S3 −b3 blue pixels S1 = a1 · a3 + b1 . For our case (Canon EOS 400D camera, He-Ne laser λ = 0.63µm), coefficients of line approximations were found as a1 = 83005 ± 2573, a2 = 12698 ± 267 and b1 = −4.4 ± 3.9, b2 = −2.1 ± 2.2 for red and green pixels respectively. We prolonged the exposure time for registration of input scene’s correlation signals in HDR registration mode; main pixels were partially oversaturated but neighbour pixels contained a distinguishable signal. 3 HDR RECONSTRUCTION OF CORRELATION SIGNALS The HDR input scene was used in order to provide the DR estimation of the HDR registration mode. The DR of radiance of input objects was higher than linear DR of the digital camera used in this work. The HDR input scene was formed by placing light attenuating filter over part of the input scene. To estimate the correlation signals’ quality, averaging by 16 frames was carried out (not required for normal correlator’s operation). Correlation signals of the HDR input scene were registered in normal DR mode. The top right signal was within the ambit of normal DR of 58 dB (as seen in Fig. 1a and in cross-sections of the correlation signals see Fig. 1b); left plot corresponds to bottom left correlation signal and right plot corresponds to top right one). The dynamic range of the HDR input scene and hence the correlation signals’ dynamic range exceeds camera’s DR (see Fig. 1a). That is why bottom left signal in normal DR mode cannot be detected and identified (see Fig. 1b). The top right signal can be easily identified (see Fig. 1a,b): peak

value of the top right signal is 3300 ± 300 DN. Level of bottom left signal (2.3 ± 0.5 DN) is comparable with camera’s noises of 2 DN and hence cannot be identified reliably. 25000 14

Signal (Digital Units)

12

20000

10 15000 8

6

10000

4 5000 2

0

0 10

15

20

25

30

90

100

Pixel’s number

a)

110

120

130

140

150

Pixel’s number

b)

Figure 1. Experimental results for registration of correlation signals in normal registration mode: image (a) and plot (b) of correlation signals.

Then correlation signals in HDR registration mode with prolonged exposure were registered. Data from accessorial green pixels were used in order to reconstruct the HDR image of correlation signals. The reconstructed HDR image of correlation signals is presented Fig. 2a. From the experimental data shown in Fig. 2b one can see that both correlation signals of HDR input scene can be identified only if the HDR registration mode is used. Peak value of the top right signal is 18600 ± 2300 DN, peak value of bottom left signal is 12.7 ± 0.7 DN, and the background level for the bottom left signals is 7.1 ± 0.5 DN when the corresponded object in the input scene was covered with opaque screen. Hence the use of HDR registration mode and Assorted Pixels technique provides more reliability of the registration of weak bottom left correlation signal. 25000 14

Signal (Digital Units)

12

20000

10 15000 8

6

10000

4 5000 2

0

0 10

15

20

Pixel’s number

a)

25

30

90

100

110

120

130

140

150

Pixel’s number

b)

Figure 2. Experimental results for registration of correlation signals in HDR mode: image (a) and plot (b) of correlation signals.

Thus in HDR mode the achieved dynamic range of registration is 73 dB that is 18600 DN divided on signal’s level of 4 DN corresponded to SNR=2. Hence the use of HDR registration mode has allowed to increase dynamic range of correlation signal’s registration on 15 dB from 58 dB in this experiment.

4 CONCLUSIONS The application of the Assorted Pixels technique for increasing dynamic range of the optical-digital correlator is described in this paper. Digital camera’s sensor with Bayer colour filter array is used for such purpose. Bayer colour filter array is considered as an array of attenuating filters in the quasimonochromatic light. When main colour pixels are saturated, pixels under other colour filters are generally not. The use of data from the accessorial pixels provides the HDR image of correlation signals from the single frame. Experiments of correlation signal registration in HDR mode were performed using the digital camera with Bayer colour filters array. Using Assorted Pixels technique we obtained the increase of dynamic range of correlation signal’s registration from 58 dB up to 73 dB. Improved DR allowed registering the inputs scene with high dynamic range more reliable than in normal DR. The dynamic range of registration can be increased further by utilization data from blue accessorial pixels. According to obtained experimental data we conclude that using Assorted Pixels reconstruction technique leads to the significant increase of the registered correlation signals’ dynamic range. ACKNOWLEDGMENTS This work was partially supported by the Ministry of education and science of the Russian Federation (Program “The development of the scientific potential of High School”, project RNP.2.1.2.1103). REFERENCES 1. O. Yadid-Pecht and E. Fossum. Wide intrascene dynamic range CMOS APS using digital sampling. IEEE Trans. Electron Devices, 44:1721–1723, Oct. 1997. 2. B. Fowler D. Yang, A. El Gamal and H. Tian. A 640 × 512 CMOS image sensor with ultrawide dynamic range floating-point pixel-level adc. IEEE J. Solid-State Circuits, 34:1821–1834, Dec. 1999. 3. S. K. Nayar and T. Mitsunaga. High dynamic range imaging: Spatially varying pixel exposures. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1:pp.472–479, June, 2000. 4. Srinivasa G. Narasimhan and Shree K. Nayar. Enhancing resolution along multiple imaging dimensions using assorted pixels. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, No. 4:pp. 518–530, April 2005. 5. Sergey N. Starikov and Mikhail V. Konnik. Using commercial photo camera’s rawbased images in optical-digital correlator for pattern recognition. In Proc. SPIE, Optical Pattern Recognition XIX , volume 6977:69770R, 2008.