TIME-VARIANT MODELING FOR GENERAL SURFACE APPEARANCE Yi-Lei Chen and Chiou-Ting Hsu Department of Computer Science, National Tsing Hua University, Taiwan ABSTRACT Describing time-variant appearance of object surface is still an open problem. With intricate environmental factors and different material characteristics over time, no researcher did tackle the principal problem: how to formulate timevariant change on general surface appearance? In this paper, we attempt to solve this challenging issue. Using multilinear algebra representation, we propose a novel appearance model and characterize the surface-specific and time-variant properties. When given an unknown sample, we propose a robust method to estimate its aging degree. In addition, we also propose an approach to synthesize its realistic appearance changes even when the material of this given sample does not exist in our database. Experimental results demonstrate the feasibility and effectiveness of our proposed approach. To the best of our knowledge, this challenging issue is first explored in image processing applications. Index Terms— Time-variant, multi-linear algebra, aging degree, appearance synthesis 1.

INTRODUCTION

Aging often changes the appearance of materials over time. The gradual appearance change usually depends on intricate environmental factors and different material characteristics. Existing research has tried to analyze this time-variant change using different methods such as computer graphic techniques, image editing, and optical model of surface properties. Computer graphic based techniques [1-3] usually synthesized the variation of surface appearance by modeling the physics of aging source or material. These methods require precise model parameters to synthesize realistic appearance and are restricted to the specific aging source. On the other hand, image editing based methods often modify the object appearance based on the information from one single input image. In [4], the authors proposed an interactive framework to edit the aged object. Similar editing tools have also been provided by popular image processing software. Although these image editing based methods are practical, no theoretical analysis on appearance model has been discussed. Other approaches [5-7] proposed to capture a sequence of images on materials over time to fit the optical model and then measure the time-variant surface parameters. For

example, in [7], the authors first use heat gun and chemical solution to obtain time-variant materials. Next, they measure bidirectional reflectance distribution function (BRDF) data and fit these data to their proposed appearance model. Nevertheless, since their model is material dependent and pixelwise, this method needs to re-train a specific model for each pixel in different captured data, even for data belonging to the same material. In addition, these approaches [5-7] are not feasible unless the sophisticated capturing device is available, and are thus impractical to general image processing problem. From the above discussion, the main difficulty of timevariant model depends on multiple unknown factors from environment and time. To better represent multi-factor data, some researchers use multi-linear algebra for data analysis. In [8], the authors first proposed a tensor face method to characterize facial images under different pose, expression, and illumination. Similar idea is extended to [9]. The proposed tensor-based AAM approach [9] accurately solves AAM fitting problem. Also, more work showed that multilinear analysis outperforms traditional approaches [10-12]. In this paper, instead of using existing techniques, we aim to formulate time-variant appearance change as a machine learning problem. To better represent multi-factor data, we first present a general time-variant appearance model via multi-linear analysis. Next, we extract a new middle-level feature to characterize surface-specific and time-variant properties. Using these features, we propose a robust aging degree estimation method and a synthesis approach for object appearance at arbitrary aging degree. The rest of this paper is organized as follows. Section 2 details our proposed time-variant appearance model. Section 3 shows the experimental results of aging degree estimation and appearance synthesis. Finally, section 4 gives the conclusion. 2. PROPOSED METHOD 2.1. Time-variant appearance database Acquisition of time-variant appearance for general surface is challenging. In this paper, we use the data released in [7], where the time-variant object surfaces are obtained by controlling various chemical processes under different lighting directions. Four process types, including burning, corrosion, decaying and drying, are conducted on different materials (such as wood and steel). Fig. 1 shows some of the released data. We use these data as our database

and analyze the four types independently. Since different time-variant sequences have different time duration and image resolution (from 180x180 to 512x512), in our experiments, we use bilinear interpolation to normalize each time-variant sequence into 25 time steps and resize to the same resolution.

If we use the global bases Vtime in equation (2) to project c time , although Vtime c time could characterize the time-variant

appearance change, the important surface-specific information (including material, lighting direction and color) is missing. Therefore, with c material , c light , and c color obtained in equation (3), we propose to construct surface-specific aging bases to characterize both the time-variant and surface-specific properties. Below we describe how we construct the surfacespecific aging bases. We first use the surface coefficients , c , and c to obtain a sub-tensor Aˆ c material

light

color

T T T Aˆ = A ×1 c material × 2 c light × 4 c color .

(4)

ˆ by SVD Then we unfold Aˆ and decompose the matrix A

(a) (b) (c) (d) Figure 1. Released data from [7]: (a) burning; (b) corrosion; (c) decaying; and (d) drying.

2.2. Time-variant appearance model via tensor representation Multi-linear algebra, also called tensor, indicates a generalization of vector or a multi-dimensional array. Using multi-linear analysis, we first construct a data tensor T to represent our database I ×I ×I ×I ×I (1) T ∈R , where I material and I light indicate the number of different material

light

time

color

pixel

materials and lighting directions (details please see [7]), and I time , I color and I pixel indicate the number of time steps, color channels and image pixels, respectively. In our experiment, I time = 25 , I color = 3 , and I pixel = 128 × 128 . In order to characterize the appearance variation, we construct another tensor, called the aging tensor A , by calculating the difference of the data appearance in T between the initial time step and all the other time steps. To further analyze T and A in different factors, we decompose the two tensors by high order singular value decomposition (HOSVD) and obtain T = X ×1 U material ×2 U light ×3 U time ×4 U color ×5 U pixel A = Y ×1 Vmaterial ×2 Vlight ×3 Vtime ×4 Vcolor ×5 V pixel

.

(2)

With the data tensor T , we could represent any sample t by T T T T t = T ×1 c material ×2 clight ×3 ctime ×4 ccolor . (3) In equation (3), we use least square error (LSE) method and rank-1 approximation algorithm [16] to estimate the unknown coefficients c material , clight , ctime and c color . The coefficients c material , c light , c color relate to surface properties, and the coefficient c time indicates the appearance similarity between input sample and the training data along time axis.

ˆ =U ˆ ΣV ˆT . (5) A Now the projection basis Uˆ only depends on the sub-

database, which possesses the same surface properties as the input sample. Using Uˆ , we obtain the time coefficients α time T T T ˆ T (t − unfold (T × c α time = U 0 1 material ×2 clight ×4 c color ))

,

(6)

where unfold (⋅) indicates a matrix unfolding operation to unfold a multi-dimensional array into a vector, and T0 is a sub-tensor of T with data appearance at the initial time step. 2.3. Aging degree estimation Given an unknown sample, once we obtain its surface coefficients c surface = [cTmaterial , cTlight , cTcolor ]T and the time coefficients α time by equations (3) and (6), an intuitive approach is to concatenate c surface and α time as low-level features and to learn a regression function along time axis. However, since the time-variant appearance change is very complicate, these features tend to distribute discontinuously in temporal domain. Therefore, instead of using regressionbased methods, in our approach, we model the aging degree estimation as a classification problem. Classification-based methods are generally more feasible for discontinuous distributions in lower dimensional space. In addition, since the class number (i.e. 25 time steps) is limited, our classification approach often involves no ambiguity in multi-class estimation. We use Fig. 2 to show that classification-based is more feasible than regression-based method in our problem. Fig. 2 shows the two estimation results, where the x axis is the data index and the y axis indicates the aging degree. Here we use the state-of-the-art techniques, support vector machine (SVM) and support vector regression (SVR), for demonstration. In Figure 2(a), SVM achieves better estimation than SVR in higher and lower aging degrees. Since SVR attempts to minimize the total estimation error, the estimated results usually close to average. Nevertheless, our other experiment (as shown in Fig. 3 (a)) shows that even SVM achieves poor estimation for intricate appearance

change such as corrosion. In other words, for a more complicate appearance change, when we concatenate c surface and α time without considering their dependency, the concatenated features become less discriminative for SVM to learn a robust classifier. Therefore, to better discriminate the appearance change, instead of using the concatenated features, we first define a matrix c c(i, j ) = c surface (i ) × α time ( j ) . (7) Then we propose to extract the middle-level features by unfolding c into 1-D vector followed by PCA for dimension reduction. In our experiments, only the first 100 features are preserved. As shown in Fig. 3 (b), the proposed middle-level features outperform the concatenated low-level features.

(a) (b) Figure 2. The ground truth (red) and the estimation results of aging degree (blue) for burning type: (a) SVM; and (b) SVR.

(a) (b) Figure 3. The ground truth (red) and the estimation results of aging degree (blue) for corrosion type using SVM: (a) low-level features; and (b) middle-level features.

2.4. Synthesis of time-variant appearance Once we estimate the aging degree d t of an unknown sample, we could synthesize this sample at arbitrary aging degree d k . We approximate the time-variant appearance change α from the aging tensor A by T

T

T

β = A ×1 c material × 2 c light × 3 (c k − c t ) T × 4 c color , ⎧1 , if i = d k . ⎧1 , if i = d t where c t (i ) = ⎨ and c k (i ) = ⎨ ⎩0 , otherwise ⎩0 , otherwise

(8)

In equation (8), c t (i ) = 1 indicates that we will only refer to the data with aging degree i , and β indicates the appearance change from aging degree d t to d k . Finally, we obtain the synthesized appearance by t + β . 3. EXPERIMENTAL RESULTS Since there is no existing method on estimating the aging degree or synthesizing arbitrary time-variant change for any

given sample, we found it very difficult to conduct comparison. Therefore, we focus on validating our proposed time-variant appearance model on general object surfaces. 3.1. Evaluation of aging degree estimation We use mean absolute error (MAE) to evaluate the proposed aging degree estimation. We first divide the database into two parts, 2/3 for training and 1/3 for testing. Then we extract the proposed middle-level features. The projection matrix of dimension reduction is obtained from training features. Finally we adopt SVM to learn the oneagainst-one classifier. Fig. 4 shows the average MAE of the estimated aging degree. As shown in Fig. 4, as the appearance change of decaying is comparatively unclear, the average MAE of this type increases at higher aging degrees. Nevertheless, most cases have the average MAE smaller than 3, which is only about 10% estimation error relative to 25 time steps. The result shows that our proposed approach successfully characterizes the time-variant appearance change. 3.2. Simulation of time-variant appearance synthesis Given an input sample t , we first resize it into the same resolution with the database images, estimate its aging degree, and then synthesize the arbitrary time-variant appearance by equation (8). Fig. 5 shows a synthesized example of brick wall. Note that, even though this sample belongs to a material which does not exist in the database, our proposed approach synthesizes its realistic appearance at different aging degrees. When given a whole image, we have to consider the boundary artifacts between different patches. In addition, when neighboring patches have discontinuous aging degrees, the synthesized appearance change would also be discontinuous and thus result in artifacts. To solve the two problems, we propose to combine the texture synthesis techniques [15-16] in our work. First, as proposed in [15], we decide the minimum error boundary between neighboring patches to eliminate boundary artifacts. For the patches with higher reconstruction error, we then synthesize the time-variant appearance from reliable patches by the pixelwise texture synthesis [16]. (Note that, these techniques are not the main focus in this paper.) Fig. 6 shows our simulation results. In comparison with Fig. 6 (a), Fig. 6 (d) shows that our proposed approach successfully synthesizes the realistic appearance change on the door. 4. CONCLUSION In this paper, we propose a novel time-variant appearance model to estimate the aging degree and synthesize the time-variant appearance of an unknown sample. The contribution of this work includes: (1) we tackle the challenging issue with complete analysis and modeling in 2-D images; (2) instead of using existing techniques, we explicitly formulate the time-variant appearance change as a machine learning problem; and (3)

we combine the two existing texture synthesis techniques to synthesize realistic appearance changes on general object surface. With this framework, in the future, we attempt to extend our approach to facial images, which can be seen as a special case of time-variant appearance. 5. REFERENCES [1] H. W. Jensen, J. Legakis and J. Dorsey, “Rendering of Wet Materials,” Proc. Rendering Techniques Conf., pp. 273-282, 1999. [2] J. Dorsey, H. K. Pedersen and P. Hanrahan, “Flow and Changes in Appearance,” Proc. SIGGRAPH, pp. 411-420, Aug. 1996. [3] J. Dorsey, A. Edelman, H. W. Jensen, J. Legakis, and H. K. Pedersen, “Modeling and Rendering of Weathered Stone,” Proc. SIGGRAPH, vol. 33, pp. 225-234, Aug. 1999. [4] O. Clement, J. Benoit, and E. Paquette, “Efficient Editing of Aged Object Textures,” Proc. the 5th international conference on AFRIGRAPH, pp. 151-158, 2007. [5] B. Sun, K. Sunkavalli, R. Ramamoorthi, P. N. Belhumer, and S. K. Nayar, “Time-Varying BRDFs,” IEEE Trans. Visualization and Computer Graphics, vol. 13, No. 3, 2007. [6] J. Wang, X. Tong, S. Lin, M. Pan, C. Wang, H. Bao, B. Guo, H. Y. Shum, “Appearance Manifolds for Modeling Time-Variant Appearance of Materials,” ACM Trans on Graphics, 2006. [7] J. Gu, C. Tu, R. Ramamoorthi, P. Belhumeur, W. Matusik, and S. Nayar, “Time-Varying Surface Appearance: Acquisition, Modeling and Rendering,” Proc. SIGGRAPH, pp. 762-771, 2006. [8] M. A. O. Vasilescu, D. Terzopoulos, “Multilinear analysis of image ensembles: TensorFaces,” Proc. ECCV, 2002. [9] Hyung-Soo Lee and Daijin, “Tensor based AAM with continous variation estimation: application to variation-robust face recognition,” IEEE Tran. PAMI, 2009. [10] Y. Fu and T. S. Huang, “Image Classification Using Correlation Tensor Analysis,” IEEE Tran. Image Processing, 2008. [11] Dacheng Tao, Xuelong Li, Xindong Wu and S. J. Maybank, “General tensor discriminant analysis and gabor features for gait recognition,” IEEE Tran. PAMI, 2007. [12] M. A. O. Vasilescu, and D. Terzopoulos, “Multilinear projection for appearance-based recognition in the tensor framework,” Proc. ICCV, 2007. [13] T. G. Kolda, “Orthogonal tensor decompositions,” SIAM Journal on Matrix Analysis and Applications, 2001. [14] L. de Lathauwer, B. de Moor, and J. Vandewalle, “On the Best Rank-1 and Rank-(R1;R2;…;RN) Approximation of HigherOrder Tensors,” SIAM J. Matrix Analysis and Applications, vol. 21, no. 4, pp. 1324-1342, 2000. [15] A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” Porc. of 28th annual conf. on Computer graphics and interactive techniques, pp. 341-346, 2001. [16] A. A. Efros and T. K. Leung, “Texture synthesis by nonparametric sampling,” Proc. ICCV, 1999.

(a)

(b)

(c) Figure 5. Synthesized example: (a) brick wall image; (b) a random patch cropped from (a); and (c) the synthesized appearance at 25 different aging degrees.

(a)

(b)

(c) (d) Figure 6. Appearance synthesis of an old door: (a) original image; (b) binary mask; (c) synthesized appearance change; and (d) the synthesized result. (Note that, to visualize the synthesized appearance change in (c), we multiply the value by five.)

Figure 4. Average MAE of estimated aging degree.