Flexible Combination of Time-Domain Lapped Transforms with Various Downsampling Factors Yuichi Tanaka, Madoka Hasegawa, and Shigeo Kato Graduate School of Engineering, Utsunomiya University Utsunomiya, Tochigi, 321-8585 Japan Email: {tanaka, madoka, kato}@is.utsunomiya-u.ac.jp Tel: +81-28-689-6267 Abstract—In this paper, we present an idea to combine lapped transforms with various downsampling factors. The factor is changed depending on a local feature of a given signal, and it can be realized by using time-domain lapped transforms. In image coding application, our method gives more flexibility to image coding performance from very low to high bitrates than the lapped transforms with a fixed downsampling factor.

I. I NTRODUCTION Sampling rate is one of the crucial aspects in digital signal processing. Traditionally, Nyquist-Shannon sampling theorem [1], [2] has been widely considered, however, there are recent advances in sampling theorems; compressed sensing [3] and finite rate of innovation [4] are now studied by many researchers. In this paper, we focus on obtaining optimal downsampling factors of a given signal for a specific application; image coding. Image coding is a fundamental part of image/video processing and many standards have been released, e.g., JPEG [5], JPEG 2000 [6], and H.264/AVC [7]. Their core strategies include frequency transform, quantization, and entropy coding. As frequency transforms, they use (integer-)DCT or wavelets to reduce spatial redundancy in images. These transforms show good performances due to their energy compaction properties. However, there still exists a room to improve image coding performances against these conventional methods and further improvement is required since the number of digital images is incredibly increased. In traditional schemes shown above, critically-sampled transforms are always considered to transform image signals. A critically-sampled transform is that the transform does not change the length of the input signal. This indicates the number of channels of a filter bank is the same as the downsampling factor in the analysis bank. It is an intuitive way to implement a transform-based image coding: A filter bank transforms a given signal followed by quantization and entropy coding. The way is, however, based on an implicit assumption: the critically-sampled transforms work well for any compression ratios. This assumption also indicates any portions of a given input signal are equally important. Unfortunately, especially in the case of low bitrate coding, it is not reasonable due to large quantization errors occurred at a quantizer.

We demonstrate that the implicit assumption is not always appropriate and switching downsampling factors adaptively is effective for image coding in low and high bitrates. A combination of time-domain lapped transforms (TDLTs) with various downsampling factors [8]–[11] is utilized to yield an entire transformation matrix of an input image. An appropriate downsampling factor is determined automatically based on the theoretical coding gain of critically-sampled, undersampled, and oversampled TDLTs. The proposed method shows good transitions of these TDLTs, and interestingly, the combined one sometimes outperforms each of them. A. Notations Upper case bold-face letters indicate matrices. A subscript of a matrix represents the matrix size if there is no explicit expression of the subscript. For example, AS is a matrix A with the size S × S. Superscript ·T is the transpose of the matrix. I and J are the identity and reverse identity matrices, respectively. II. T IME -D OMAIN L APPED T RANSFORMS AND I TS A PPLICATION TO JPEG-BASED I MAGE C ODER A. TDLTs TDLT is a variant of (bi)orthogonal LTs and it can be implemented by the pre-/postfilterings of DCT-based block transforms [8], [9]. Its polyphase matrix has been re-factorized from the type-II fast lapped orthogonal transform [12] and the matrix is generalized to have the biorthogonal property. The analysis polyphase matrix E(z) and synthesis one R(z) are represented as follows: E(z) = CN ΛN (z)P

(1)

T TΛ−1 N (z)CN

(2)

R(z) =

where CN is the N × N type-II DCT matrix, ΛN (z) =  0N/2 IN/2 and zIN/2 0N/2   I 0 P = WN (3) WN 0 V   I 0 (4) T = WN WN , 0 V−1

687 Proceedings of the Second APSIPA Annual Summit and Conference, pages 687–691, Biopolis, Singapore, 14-17 December 2010.

DCT DCT N

N

N

N

P

N/2 N/2

P

N/2 N/2

P

DCT

N/2 N/2

P

DCT

DCT

N/2 N/2

DCT

N

M

P

M

N/2

DCT

N

N

N

M

P

M DCT

N

M

P

P

P

M DCT

N

N

(b)

DCT

N

DCT

N

DCT

N

DCT

N

N/2 N/2

P

N

N/2 N/2

P

DCT N/2 N/2

N/2 N/2

(a)

N

N/2 N/2

M

N

N/2

N/2 N/2 (c)

Fig. 1. TDLTs with various downsampling factors: (a) critically-sampled, (b) undersampled and (c) oversampled systems, respectively.

in which V is an N × N invertible matrix and   1 IN/2 JN/2 WN = √ . 2 JN/2 −IN/2

Prefiltering P

8-bit truncation

JPEG coder

Postfiltering T

Recovering truncation

JPEG decoder

(5)

The structure can be implemented by a pre-/postfiltering performed at a boundary of two N × N DCT matrices, which is shown in Fig. 1(a). Although the polyphase matrices mentioned above are the critically-sampled structure, they are easily extended to undersampled and oversampled filter banks by changing the sizes of pre-/postfilters, P and T [10], [11]. Hereafter, let M be the downsampling factor and N be the number of channels of the filter bank. The generalized structures of P and T are shown as follows:   U 0 P = WN (6) WM 0 V  +  0 U (7) T = WM WN 0 V+ where U and V are of the size N/2 × M/2 and U+ and V+ are (pseudo-)inverses of U and V, respectively. Intuitively, U and V are “fat” matrices when N < M (undersampled case), whereas they are “tall” ones when N > M (oversampled case). Note that there always exists reconstruction errors when N < M : the undersampled system is not a perfect reconstruction (PR) filter bank. The undersampled and oversampled systems are illustrated in Figs. 1(b) and 1(c), respectively. The undersampled system performs well in low bitrate image coding since the reconstruction error is not a main cause of image quality degradation in such bitrates. In the high bitrate case, the oversampled system outperforms the criticallysampled ones (and of course, the undersampled ones) due to its property to provide low truncation errors of transformed coefficients and high coding gain. B. TDLTs as Pre-/Postfiltering of JPEG-Based Image Coder TDLTs can be applied directly to an image and encoded by a sophisticated encoder, such as L-CEB coder [13]. However,

Fig. 2. JPEG-based image compression using a TDLT.

the advantage of TDLTs as pre-/postfiltering of the DCT matrix is well utilized by employing JPEG-based image compression method. In this approach, pre-/postfilters of TDLTs are completely outside of JPEG framework shown in Fig. 2 [9], [10]. First an image is transformed by the prefilter P for each row and column, and then the dynamic range of the transformed image is adjusted to have the integer value between [0, 255] since JPEG typically requires 8-bit pixel data. Then the truncated image is compressed by JPEG algorithm. The dynamic range of the decoded JPEG image is recovered with transmitted side information, then the postfiltering T is performed to obtain the final output image. Even in this simple truncation approach, TDLTs show significant coding gain against JPEG alone [9]. In this paper, we also demonstrate our coding performance based on this truncation approach. III. C OMBINED TDLT S FOR VARIOUS S AMPLING FACTORS In this section, we present how to implement the combined TDLTs with various sampling factors. First the fundamental structure is shown, and then the method to decide the downsampling factor is presented. A. Combined TDLTs Combined TDLTs for various sampling factors are very simple since all the TDLTs can be represented as pre-/postfilters of DCT matrices. The combined TDLTs are shown in Fig. 3. It is implemented by placing one of the prefilters P at the boundary of DCT matrices. The important thing is that the

688

M0

N

N

M1

P

N

DCT

N

DCT

N

where UMn and VMn are the components of Pn . If P0 and P1 have no DC leakage condition,  UM0 1M0 =d0 1M0 and UM1 1M1 = d1 1M1 . Finally, if UM0 (:, i) = UM1 (:, i), we have P1N = d1N . From this proposition, a diagonal scaling matrix is multiplied by each prefilter so that the combined TDLT does not produce DC leakage. B. Decision of Downsampling Factors with Theoretical Coding Gain

N/2 N/2

P

N

N/2 N/2

P

DCT

N/2 N/2

P

N

N/2 N/2

P

DCT

DCT

N

DCT

N

N/2 N/2

Fig. 3. Combined TDLT with various downsampling factors.

PR condition is always guaranteed even if the downsampling factors are switched as long as N ≥ M . The final filter bank at the boundary of two different prefilters has a different properties of the one which has the same two prefilters at the boundary. It has a different filter length from the normal TDLTs and thus, it does not have linear-phase filters at all. However, the fact is not important for image processing except for image boundaries. In contrast to these trivial properties, the DC leakage of filter banks at the boundary where the prefilters are switched is a not-so-trivial issue. Here, we present a proposition. Proposition 1. A TDLT whose two different prefilters P0 and DC P1 are connected to a DCT matrix does not produce  leakage of highpass subbands if and only if two U(:, i)’s in P0 and P1 , where U(:, i) is ith row of U, are the same. Proof: If a TDLT has no DC leakage, the fact is represented as follows [9]: P1N = d1N

(8)

where 1N is a column vector with length N whose all values are 1 and d is a constant factor. That is, the output of the prefilter should be a piecewise constant sequence. From Fig. 3, it is clear that the upper N/2 and lower N/2 signals output from the combined prefilter are, respectively, from the upper and lower prefilters P0 and P1 . Let M0 and M1 denote the downsampling factors of P0 and P1 , respectively. Consequently, P1N in (8) at the boundary of two different prefilters is shown as follows: P1N =  1 JN/2 UM0 − VM0 JM0 /2 × 2 UM1 + JN/2 VM1 JM1 /2

JN/2 UM0 JM0 /2 + VM0 UM1 JM1 /2 − JN/2 VM1



× 1M0 +M1   J U 1 = N/2 M0 M0 UM1 1M1

Although the switching downsampling factors is possible without a significant drawback, a guideline to decide downsampling factors is required. The theoretical coding gain for a target bitrate is utilized in this paper. We elaborate the details below. Malvar [14] presented theoretical coding gain for a critically-sampled subband coder as a function of bitrates by estimating an assigned bitrate Bi for ith subband. It is shown as follows: σ2 CCG = 10 log10 P CM (10) σn2 where σP2 CM = c2−2B σx2 σn2 =

c N

N −1 

2−2Bi σi2 ||fi ||2 ,

(11) (12)

i=0

 −1 in which N i=0 Bi = N B and c is a constant which depends on the statistics of the input signal x. If x conforms of Gaussian distribution1, c = 0.75. Furthermore, σi and fi are the variance of ith subband and the norm of the ith synthesis basis function, respectively. Undersampled and oversampled systems respectively spend more and less bitrates for each subband than the criticallysampled one. Furthermore, the undersampled one always yields reconstruction errors. The variance of reconstruction noise σr2 is changed based on downsampling factors [11]. Consequently, σn2 ’s for various downsampling factors are represented as follows [10], [11]: ⎧ N −1 ⎪ c  −2BiU S 2 ⎪ 2 ⎪ σ + 2 σi ||fi ||2 N ≤ M ⎪ ⎨ r N i=0 (13) σn2 = N −1  ⎪ c ⎪ −2BiOS 2 2 ⎪ ⎪ 2 σi ||fi || N ≥M ⎩N i=0

N −1 where i=0 BiUS = i=0 BiOS = M B. In this paper, Bi ’s are determined by the nonlinear optimization technique [14]. When N = M , σr2 = 0 and BiUS = BiOS = Bi and thus (13) is equal to (12). The theoretical coding gain does not consider the entropy coding process, however, it gives a good approximation of the compression performance of a given filter bank.

(9)

689

N −1

1 It

is a very typical assumption and this paper follows to it.

40

C. Truncation Variance

38

As previously mentioned, the JPEG-based image encoder with TDLTs performs 8-bit truncation of transformed coefficients after prefiltering. Therefore, it is not enough to use the theoretical coding gain alone to switch the downsampling factor. In this paper, we propose to include the truncation variance σt2 in the theoretical coding gain. It is clear that the truncation error is occurred just after prefiltering. The prefiltering matrix P is only considered for the truncation variance. Similar to the calculation of the variance of ith subband, the truncation variance σt2 is defined as follows by using the quantization error methodology [15]: =

c2

N

σo2

PSNR (dB)

34

28

JPEG alone LT-88 LT-810 LT-816 LT-87 comLT

22 20 0

0.2

0.4

0.6 bpp

0.8

1

1.2

42 40 38 36

(14)

where σo2 is the variance of the original signal, Bt is the truncation accuracy (typically Bt = 8 for 8-bit truncation), and Rxx is the autocorrelation matrix of the input signal x. It measures the truncation error variance according to the difference between the prefiltered signal and the original one. It is clear that σt2 = 0 if P = I, i.e., no prefiltering case. It should be included into the calculation of the coding gain represented as follows: σ ˆn2 = σn2 + αRσt2

30

24

  E PxxT PT − E xxT

  c2−2Bt 2

σo Trace PRxx PT − Trace (Rxx ) = N

32

26

PSNR (dB)

σt2

−2Bt

36

34 32 30 JPEG alone LT-88 LT-810 LT-816 LT-87 comLT

28 26 24 22 20 0

0.2

0.4

0.6 bpp

0.8

1

1.2

Fig. 4. R-D curve comparison for various transforms. (Top) Lena. (Bottom) Milk.

(15)

where α is an arbitrary constant and R is the target bitrate since the direct use of σt2 will be incorrect due to Huffman coding in JPEG. Furthermore, R is included because of reflecting a relative importance of the truncation error for various bitrates: for a given α and σt2 , αRσt2 is small if R is small which corresponds to the low bitrate encoding where the truncation error is not a main cause of image quality degradation, and vice versa. Finally, for the JPEG-based image compression using TDLTs, we switch the downsampling factor for various bitrates based on the modified coding gain as follows: σ2 . (16) CˆCG = 10 log10 P CM σ ˆn2 IV. A PPLICATION TO I MAGE C ODING In this section, TDLTs with combined downsampling factors are applied to JPEG-based image compression. First we indicate several setups to encode images, and then image coding performances are shown. A. Procedure Setups Two 512 × 512 8-bit grayscale images Lena and Milk were used for the experiment. We compare our proposed combined TDLT with TDLTs of fixed downsampling factors and JPEG alone. A TDLT is specified with a label LT-N M : For example, LT-88 and LT-810 refer to the TDLTs of the critically-sampled and undersampled (M = 10), respectively. Whereas our proposed TDLT is labeled as comLT. The downsampling factors M = 7, 8, 10, 16 are used and N is fixed to 8.

For the combined TDLTs, the downsampling factor is decided based on CˆCG for each L rows of an input image. Moreover, the downsampling factors along horizontal and vertical directions are independently determined. Therefore, the transmitted side information of the downsampling factor is 2512/Llog2 Fc  bits where Fc is the number of filter candidates. It can be generally negligible for the entire bit budget since it just requires 28 bits when L = 80 and Fc = 4 which is our experiment condition. The input image is assumed to be AR(1) process for the calculation of CˆCG . The autoregressive coefficient is calculated by using Matlab function of Yule-Walker method aryule for each L rows. A target bitrate is required to calculate CˆCG but the required bitrate is obtained after the JPEG compression since JPEG encodes an image based on QP (quality parameter). Therefore, we should match QPs to appropriate bitrates. It is a difficult issue, however, in our preliminary experiment, the relationship between QP and bitrate is highly linear from low to high QPs, e.g., QP = [0, 70]. Therefore, we calculate a target bitrate from a given QP with linear fitting function. In this paper, the QPbitrate curve of LT-88 for Lena is used to estimate the target bitrates of the comLT. We used Ezyfit toolbox2 to obtain the variables of the linear function.

690

2 http://www.fast.u-psud.fr/ezyfit/

Fig. 5. Reconstructed Milk image at 0.16 bpp. Top row: (left) JPEG alone, 30.04 dB; (middle) LT-88, 31.00 dB; (right) LT-810, 32.61 dB. Bottom row: (left) LT-816, 32.63 dB; (middle) LT-87, 25.78 dB; (right) comLT, 33.66 dB.

B. Experimental Results The R-D curves for test images are shown in Fig. 4. It is clear that LT-810 and LT-816 show good performances in low bitrates, whereas LT-87 slightly outperforms other transforms in high bitrates. The comLT seems to take the highest PSNRs in each bitrate. This is the positive effect of switching downsampling factors depending on image characteristics and target bitrates. In addition, interestingly, our comLT shows the highest PSNRs for Milk within the bitrate range 0.1–0.4 bpp. As a result, switching downsampling factors for TDLTs presents not only good transition from low to high bitrates but also the possibility to have the best performance. The reconstructed Milk images are shown in Fig. 5. Similar to the R-D curve comparison, the reconstructed image with the comLT is clearer than others. As the comparison with critically-sampled and oversampled transforms, it shows smoother backgrounds. Moreover, the undersampled TDLTs yield more ringing artifacts around edges of milk splash than the comLT. V. C ONCLUSIONS In this paper, we present a flexible combination of timedomain lapped transforms with various downsampling factors. To select the best downsampling factor for a given bitrate and signal characteristics, the theoretical coding gain is utilized. Furthermore, it is adjusted for JPEG-based image coder by using the truncation variance, which estimates the 8-bit truncation effect of JPEG compression using TDLTs. Our filter selection uniformly shows the best R-D curves compared to the TDLTs with fixed downsampling ratios.

R EFERENCES [1] C. E. Shannon, “Communication in the presence of noise,” Proc. Inst. Radio. Eng., vol. 37, no. 1, pp. 10–21, 1949. [2] H. Nyquist, “Certain topics in telegraph transmission theory,” Trans. Amer. Inst. Electr. Eng., vol. 47, no. 2, pp. 617–644, 1949. [3] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289–1306, 2006. [4] M. Vetterli, P. Marziliano, and T. Blu, “Sampling signals with finite rate of innovation,” IEEE Trans. Signal Process., vol. 50, no. 6, pp. 1417–1428, 2002. [5] W. B. Pennebaker and J. L. Mitchell, JPEG: Still Image Data Compression Standard. NY: Van Nostrand Reinhold, 1993. [6] D. S. Taubman and M. W. Marcellin, JPEG2000: Image Compression Fundamentals, Standards and Practice. Norwell, MA: Kluwer, 2001. [7] T. Wiegand, G. J. Sullivan, G. Bjntegaard, and A. Luthra, “Overview of the H. 264/AVC video coding standard,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 560–576, 2003. [8] T. D. Tran, J. Liang, and C. Tu, “Lapped transform via time-domain pre- and post-filtering,” IEEE Trans. Signal Process., vol. 51, no. 6, pp. 1557–1571, 2003. [9] W. Dai and T. D. Tran, “Regularity-constrained pre- and post-filtering for block DCT-based systems,” IEEE Trans. Signal Process., vol. 51, no. 10, pp. 2568–2581, 2003. [10] L. Gan and K.-K. Ma, “Time-domain oversampled lapped transforms: theory, structure, and application in image coding,” IEEE Trans. Signal Process., vol. 52, no. 10, pp. 2762–2775, 2004. [11] L. Gan, C. Tu, J. Liang, T. D. Tran, and K.-K. Ma, “Undersampled boundary pre-/postfilters for low bit-rate DCT-based block coders,” IEEE Trans. Image Process., vol. 16, no. 2, pp. 428–441, 2007. [12] H. S. Malvar, Signal processing with lapped transforms. Norwood, MA: Artech House, 1992. [13] C. Tu and T. D. Tran, “Context-based entropy coding of block transform coefficients for image compression,” IEEE Trans. Image Process., vol. 11, no. 11, pp. 1271–1283, 2002. [14] H. S. Malvar, “Biorthogonal and Nonuniform Lapped Transforms for Transform Coding with Reduced Blocking and Ringing Artifacts,” IEEE Trans. Signal Process., vol. 46, no. 4, p. 1043, 1998. [15] P. P. Vaidyanathan, Multirate Systems and Filter Banks. NJ: PrenticeHall, 1993.

ACKNOWLEDGMENTS This work was supported in part by KAKENHI 22760263 and The Nakajima Foundation.

691