Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Video Stabilization of Atmospheric Turbulence Distortion Yifei Lou1 4 Sung Ha Kang2 Stefano Soatto3 Andrea Bertozzi4 1 Department
of Mathematics, UCSD of Mathematics, Georgia Tech 3 Computer Science Department, UCLA 4 Department of Mathematics, UCLA 2 School
May 20, 2012
1/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Examples of turbulence videos
2/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Two effects of turbulence blurry image frames temporal oscillations
3/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction
3/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction
We propose the following PDE model for video stabilization: ut (x, y , k ) = S[u(x, y , k )] + µ4k u where S[·] denotes the Sobolev sharpening method on spatial domain, and 4k is the Laplacian operator acting on time direction k , 4k u = u(x, y , k + 1) − 2u(x, y , k ) + u(x, y , k − 1).
3/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Outline
1
Sobolev sharpening method
2
Video stabilization
3
Reconstruct a latent image
4
Discussion
4/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Sobolev gradient flow The heat equation is the gradient descent on the functional Z 1 E(u) = k∇uk2 , 2 Ω with respect to the L2 metric. Calder-Mansouri-Yezzi consider an inner product on the Sobolev space H01 (Ω) hv , wi −→ gλ (v , w) = (1 − λ)hv , wiL2 + λhv , wiH 1 , for any λ > 0. Then the Sobolev metric gλ on H01 (Ω) is given by ∇gλ E|u = −4(Id − λ4)−1 u .
5/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Sobolev gradient flow
The authors proved the well- posedness of the Sobolev gradient flow (SOB), both in the forward and backward directions. The backward direction can be used for image sharpening. In particular, 1 Es (u) = 4
Z
2
k∇u0 k Ω
!2 R k∇uk2 Ω R −α , 2 Ω k∇u0 k
where u0 is the initial value and α is a scale. For α < 1, we get blurring while we get sharpening for α > 1.
6/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Sobolev gradient flow
The gradient descent to minimize such energy is
ut =
! R k∇uk2 Ω R − α 4(Id − 4)−1 u . 2 k∇u k 0 Ω
(1)
It is a nonlinear PDE. We prove the local and global existence and uniqueness of the solutions to (1).
7/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
(a) original
(b) PM
(c) AM
(d) SOB
Discussion
8/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Our model: ukn+1 − ukn dt
! k∇uk k2 = − α 4(Id − λ4)−1 ukn k∇uk0 k2 +µ ukn+1 + ukn−1 − 2ukn+1 ,
with uk (x, y ) = u(x, y , k ), u 0 be the original video sequence. α > 1 for deblurring. µ is a weighting parameter.
9/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Computations – in spatial domain
Calder et. al. derive an explicit expression for the operator (Id − λ4)−1 on Ω = R2 , i.e., (Id − λ4)−1 f (x) = Sλ ∗ f (x), with ∗ be the convolution operator and 1 Sλ (x) = 4λπ
Z 0
+∞
|x|2
e−t− 4tλ dt . t
10/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Computations – in frequency domain ˆkn (m1 , m2 ) be the discrete Fourier transform of ukn (x, y ). Let u ˆkn+1 − u ˆkn u −4D(m1 , m2 ) ˆ n + µ(u ˆkn+1 + u ˆkn−1 − 2u ˆkn+1 ) , = Ckn u dt 1 + 4λD(m1 , m2 ) k where P Ckn
m1 ,m2
=P
ˆkn (m1 , m2 )|2 D(m1 , m2 )|u
ˆk0 (m1 , m2 )|2 D(m1 , m2 )|u m1 π 2 m2 π 2 D(m1 , m2 ) = sin( ) + sin( ) M1 M2
−α
m1 ,m2
for discrete coordinates m1 = 1, · · · , M1 and m2 = 1, · · · , M2 .
11/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Figure: Raw data.
12/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Figure: SOB.
13/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Figure: SOB+LAP.
14/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Figure: Raw data.
15/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Figure: SOB.
16/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Figure: SOB+LAP.
17/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Finding a latent image
(a)
(b)
(c)
Figure: Latent Images: (a) applying SOB on the temporal mean. (b) The temporal mean of the video sequence after SOB+LAP. (c) Further improvement using our image fusion technique from the video reconstruction by SOB+LAP.
18/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Partition of image domain
Figure: Partition of the image domain Ω. There is one row or one column overlap between two adjacent sub-regions Ωi and Ωj .
19/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Our “lucky” fusion
From these partitions, Select the best patch u(Ωj , kˆ ) from all the frames for each Ωj for 1 ≤ k ≤ N. The best patch is selected by measuring two terms: the similarity to the mean and the sharpness. Why? Similarity is to enforce correct pixel location, while sharpness is to produce sharp edges.
20/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
original
Lucky-region
Mao-Gilles
one of SOB+LAP
mean of SOB+LAP
our method
Discussion
21/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Conclusions
We proposed SOB+LAP, which stabilizes the temporal oscillation and sharpens the video frame at the same time. We use the lucky-region image fusion technique to construct a latent image from results of SOB+LAP. Future direction is to analyze the turbulence behavior in order to resolve fine details in the case of destructive turbulent degradation.
22/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Turbulence motion analysis
(a)
(b)
Figure: The positioning of the key points along a line. The key points are displayed as the blue dots on (a). (b) shows how these points are oscillating can time t changes. The wave movement of the turbulence happens in groups.
23/24
Sobolev sharpening method
Video stabilization
Reconstruct a latent image
Discussion
Reference: Yifei Lou, Sung Ha Kang, Stefano Soatto, and Andrea L. Bertozzi. Video Stabilization of Atmospheric Turbulence Distortion. (CAM report 12-30) Web: sites.google.com/site/louyifei/research/turbulence
Thank You!
24/24