INFORMATIK

Combining Stereo and Visual Hull Information for On-line Reconstruction and Rendering of Dynamic Scenes

Dec 11th, 2002 Ming Ming Li, Li, Hartmut Hartmut Schirmacher Schirmacher Marcus -Peter Seidel Marcus Magnor Magnor,, Hans Hans-Peter Seidel MMSP’02

Ming Li

Introduction

INFORMATIK

The general goal •• Acquiring -world scenes Acquiring dynamic dynamic 3D 3D geometry geometry of of real real-world scenes and -time and render render itit in in real real-time

Previous systems •• Visual Visual hull hull –– Image -Based VH Image-Based VH & & Polyhedral Polyhedral VH VH (MIT) (MIT) •• Stereo Stereo –– 3D -presence ((UPenn.) UPenn.) 3D Dome Dome (CMU), (CMU), Tele Tele-presence

Our work •• Combine Combine two two methods methods at at interactive interactive frame frame rate rate MMSP’02

Ming Li

Visual hull reconstruction

INFORMATIK

Silhouette extraction •• Background Background differencing differencing & & Polygon Polygon approximation approximation

Polyhedral visual hull reconstruction •• Project Project silhouette silhouette polygons polygons back back to to 3D 3D space space •• CSG CSG intersection intersection MMSP’02

Ming Li

Depth from stereo

INFORMATIK

Rectification •• Warps Warps the the stereo stereo images images to to align align them them along along scanlines scanlines

Matching

Dis pari ty Se arch Rang e

Depth recovery Depth =

MMSP’02

Baseline ∗ FocalLengt h Disparity Ming Li

Combination of Stereo & VH

INFORMATIK

Stereo

Visual Hull

Essence

rays radiating from inner surfaces of objects

rays radiating from tangent surfaces of objects

Pros

accurate geometry

robust

Cons

noisy result

approximate geometry

Combine them! •• Use Use visual visual hull hull to to stabilize stabilize stereo stereo recovery recovery •• Use Use stereo stereo algorithm algorithm to to recover recover concave concave geometry geometry details details MMSP’02

Ming Li

Combination of Stereo & VH

INFORMATIK

Basic Idea •• Exploit Exploit visual visual hull hull to to restrict restrict disparity disparity search search range range in in stereo stereo computation computation –– Disparity Disparity range: range: two two disparity disparity maps, maps, upper upper and and lower lower bound bound

Disparity range map from VH •• Use Use OpenGL OpenGL to to render render VH VH into into depth depth buffer buffer •• Convert Convert the the depth depth buffer buffer to to disparity disparity map map MMSP’02

Ming Li

System overview

INFORMATIK

Acquisition Acquisition Setup Setup •• 66 cameras cameras are are grouped grouped pairwise pairwise,, pointing pointing to to aa common common region region •• Pre -calibrated using Pre-calibrated using Tsai’s Tsai’s algorithm algorithm •• Synchronized Synchronized via via external external trigger trigger

MMSP’02

Ming Li

Processing stages

INFORMATIK

Initialization •• Compute Compute aa reference reference background background image image for for segmentation segmentation of of videos videos from from all all views views

Running system •• Processing Processing cycle cycle for for each each frame frame

MMSP’02

Ming Li

Improved reconstruction depth depth map map recovered recovered without without using using visual visual hull hull

MMSP’02

INFORMATIK

Improved Improved depth depth map map with with the the help help of of visual visual hull hull

Ming Li

Rendering

INFORMATIK M

3D forward warping •• Choose Choose one one closest closest view view

p1 ( X1, y1) p2 (X2, y2)

R ef erence vi ewpoi nt

Rendering snapshots

MMSP’02

D esir ed vi ew point

Ming Li

Results

INFORMATIK

System Configuration •• Clients Clients — — Three Three AMD AMD Athlon Athlon 1.1GHz 1.1GHz machines machines •• Server -processor machine Server — — Intel Intel 1.7GHz 1.7GHz dual dual-processor machine •• Input Input image image resolution resolution — — 320x240 320x240 pixels pixels –– About About 18,000 18,000 silhouette silhouette pixels pixels for for 3D 3D warping warping

Performance — Interactive frame rate •• Up Up to to 44 fps fps for for reconstruction reconstruction •• 10 10 fps fps for for rendering rendering MMSP’02

Ming Li

Video show

MMSP’02

INFORMATIK

Ming Li

Conclusion & Future work

INFORMATIK

Conclusions •• Better Better results results using using combined combined reconstruction reconstruction method method •• Reconstruction Reconstruction and and rendering rendering of of dynamic dynamic scenes scenes are are performed performed at at interactive interactive frame frame rates rates

Future work •• Use Use visual visual hull hull reconstruction reconstruction as as aa base base method method and and improve improve the the 3D 3D model model of of visual visual hulls hulls using using stereo stereo information information

MMSP’02

Ming Li