A generative traversability model for monocular robot self-guidance Michael Sapienza and Kenneth P. Camilleri Dept. Systems and Control Engineering University of Malta

July 31, 2012

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

1 / 21

Why? Explore dangerous/unknown environments Assist in household/office chores Intreact with us: Human Robot Interaction (HRI) Exploration

HRI Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

2 / 21

Why? Explore dangerous/unknown environments Assist in household/office chores Intreact with us: Human Robot Interaction (HRI) Exploration

HRI Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

2 / 21

Why? Explore dangerous/unknown environments Assist in household/office chores Intreact with us: Human Robot Interaction (HRI) Exploration

HRI Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

2 / 21

Outline 1

Introduction Computer Vision Monocular Camera Motivation Problem Definition

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

3 / 21

Outline 1

Introduction Computer Vision Monocular Camera Motivation Problem Definition

2

Previous Work Traversability Detection Objectives and Contributions

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

3 / 21

Outline 1

Introduction Computer Vision Monocular Camera Motivation Problem Definition

2

Previous Work Traversability Detection Objectives and Contributions

3

Methods Feature Extraction Classification Vision Algorithm Locomotion

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

3 / 21

Outline 1

Introduction Computer Vision Monocular Camera Motivation Problem Definition

2

Previous Work Traversability Detection Objectives and Contributions

3

Methods Feature Extraction Classification Vision Algorithm Locomotion

4

Experimental Results

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

3 / 21

Outline 1

Introduction Computer Vision Monocular Camera Motivation Problem Definition

2

Previous Work Traversability Detection Objectives and Contributions

3

Methods Feature Extraction Classification Vision Algorithm Locomotion

4

Experimental Results

5

Conclusion Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

3 / 21

Introduction

Computer Vision

Computer Vision “Vision is the process of discovering from images what is present in the world, and where it is”

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

4 / 21

Introduction

Computer Vision

Computer Vision “Vision is the process of discovering from images what is present in the world, and where it is”

192 164 153 178 181 91 1 54 138 177 201 195 99 103 56 78 154 1 92 101 123 167 165 154 163 19 127 196 4 155 191 154 19 99 33 103 56 78 154 192 43

“If vision is really an information processing task, then I should be able to make my computer do it...” [Marr, 1982] Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

4 / 21

Introduction

Monocular Camera Motivation

Monocular Camera Motivation Camera Advantages Passive, low-cost Actions driven by environment semantics Permits communication with robot through images VISAR01

Single Image Advantages Lower processing cost compared to stereo-vision Single images contain enough information for navigation

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

5 / 21

Introduction

Monocular Camera Motivation

Monocular Camera Motivation Camera Advantages Passive, low-cost Actions driven by environment semantics Permits communication with robot through images VISAR01

Single Image Advantages Lower processing cost compared to stereo-vision Single images contain enough information for navigation

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

5 / 21

Introduction

Monocular Camera Motivation

Monocular Camera Motivation Camera Advantages Passive, low-cost Actions driven by environment semantics Permits communication with robot through images VISAR01

Single Image Advantages Lower processing cost compared to stereo-vision Single images contain enough information for navigation

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

5 / 21

Introduction

Monocular Camera Motivation

Monocular Camera Motivation Camera Advantages Passive, low-cost Actions driven by environment semantics Permits communication with robot through images VISAR01

Single Image Advantages Lower processing cost compared to stereo-vision Single images contain enough information for navigation

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

5 / 21

Introduction

Monocular Camera Motivation

Monocular Camera Motivation Camera Advantages Passive, low-cost Actions driven by environment semantics Permits communication with robot through images VISAR01

Single Image Advantages Lower processing cost compared to stereo-vision Single images contain enough information for navigation

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

5 / 21

Introduction

Problem Definition

Problem Definition Traversability Detection Finding a set of pixels that define the boundary between traversable ground regions and obstacle regions.

What does it involve? Binary Image Segmentation, the segmentation of an image into two classes: [0,1]. Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

6 / 21

Previous Work

Traversability Detection

Traversability Detection MIT: Pebbles Robot, [Lorigo et al., 1997] Unstructured environments, safe window, Patch based, threshold on histogram matching.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

7 / 21

Previous Work

Traversability Detection

Traversability Detection MIT: Pebbles Robot, [Lorigo et al., 1997] Unstructured environments, safe window, Patch based, threshold on histogram matching. CMU: Robotic Wheelchair, [Ulrich & Nourbakhsh, 2000] Indoor environments, safe window, Pixel based, histogram thresholding.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

7 / 21

Previous Work

Traversability Detection

Traversability Detection MIT: Pebbles Robot, [Lorigo et al., 1997] Unstructured environments, safe window, Patch based, threshold on histogram matching. CMU: Robotic Wheelchair, [Ulrich & Nourbakhsh, 2000] Indoor environments, safe window, Pixel based, histogram thresholding. Cranfield University, [Katramados et al., 2009] Outdoor environments, safe window, Pixel based, temporal histogram model.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

7 / 21

Previous Work

Traversability Detection

Traversability Detection MIT: Pebbles Robot, [Lorigo et al., 1997] Unstructured environments, safe window, Patch based, threshold on histogram matching. CMU: Robotic Wheelchair, [Ulrich & Nourbakhsh, 2000] Indoor environments, safe window, Pixel based, histogram thresholding. Cranfield University, [Katramados et al., 2009] Outdoor environments, safe window, Pixel based, temporal histogram model. Georgia Institute of Technology, [Kim et al., 2007] Outdoor environments, self-supervised, Superpixel based, probabilistic model. Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

7 / 21

Previous Work

Objectives and Contributions

Objectives and Contributions Project Goals Design real-time vision framework to allow a mobile robot to guide itself in an unknown environment using a low resolution monocular camera.

Contrubutions Complementary set of colour and texture features. Smaller safe-window to allow movement in close proximity with obstacles. Novel generative approach which models the feature dissimilarity distribution. Underlying Assumptions Initially, a safe region in front of the robot is traversable. The ground and obstcales can be differentiated from their appearance.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

8 / 21

Previous Work

Objectives and Contributions

Objectives and Contributions Project Goals Design real-time vision framework to allow a mobile robot to guide itself in an unknown environment using a low resolution monocular camera.

Contrubutions Complementary set of colour and texture features. Smaller safe-window to allow movement in close proximity with obstacles. Novel generative approach which models the feature dissimilarity distribution. Underlying Assumptions Initially, a safe region in front of the robot is traversable. The ground and obstcales can be differentiated from their appearance.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

8 / 21

Previous Work

Objectives and Contributions

Objectives and Contributions Project Goals Design real-time vision framework to allow a mobile robot to guide itself in an unknown environment using a low resolution monocular camera.

Contrubutions Complementary set of colour and texture features. Smaller safe-window to allow movement in close proximity with obstacles. Novel generative approach which models the feature dissimilarity distribution. Underlying Assumptions Initially, a safe region in front of the robot is traversable. The ground and obstcales can be differentiated from their appearance.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

8 / 21

Previous Work

Objectives and Contributions

Objectives and Contributions Project Goals Design real-time vision framework to allow a mobile robot to guide itself in an unknown environment using a low resolution monocular camera.

Contrubutions Complementary set of colour and texture features. Smaller safe-window to allow movement in close proximity with obstacles. Novel generative approach which models the feature dissimilarity distribution. Underlying Assumptions Initially, a safe region in front of the robot is traversable. The ground and obstcales can be differentiated from their appearance.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

8 / 21

Previous Work

Objectives and Contributions

Objectives and Contributions Project Goals Design real-time vision framework to allow a mobile robot to guide itself in an unknown environment using a low resolution monocular camera.

Contrubutions Complementary set of colour and texture features. Smaller safe-window to allow movement in close proximity with obstacles. Novel generative approach which models the feature dissimilarity distribution. Underlying Assumptions Initially, a safe region in front of the robot is traversable. The ground and obstcales can be differentiated from their appearance.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

8 / 21

Methods

Feature Extraction

Descriptive Features Colour Illumination invariant colour features: Hue & Saturation from HSV colour space. Combination of channels from YCbCr & LAB.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

HSV Colour cone

July 31, 2012

9 / 21

Methods

Feature Extraction

Descriptive Features Colour Illumination invariant colour features: Hue & Saturation from HSV colour space. Combination of channels from YCbCr & LAB.

HSV Colour cone

Texture Colour is not always reliable eg: white walls; different objects with similar colours. Texture featrues look at the pixel intensity in relation to its neighbours: Edge magnitudes & Orientation patterns. Local Binary Patterns. Calculating the LBP8,1 code Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

9 / 21

Methods

Feature Extraction

Classification primitives Pixels May result in noisy/spotty classification. Pixels

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

10 / 21

Methods

Feature Extraction

Classification primitives Pixels May result in noisy/spotty classification. Pixels

Patches Allows local feature distributions to be extracted. May contain multiple object boundaries.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

Patches

July 31, 2012

10 / 21

Methods

Feature Extraction

Classification primitives Pixels May result in noisy/spotty classification. Pixels

Patches Allows local feature distributions to be extracted. May contain multiple object boundaries.

Patches

Superpixels Groups of homogeneous pixels: Computationally efficient Preserves image structure Allows rich statistics to be extracted from perceptually meaningful regions Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

Superpixels

July 31, 2012

10 / 21

Methods

Classification

Binary Classification Problem Definition The Classification Objective The classification of each superpixel S with its correct class label Θ ∈ {θ1 , θ2 } given a vector of traversability cues X = hX1 , X2 , ...Xj , ...Xn i, which is a function of dissimilarity between the superpixel S and the model region M .

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

11 / 21

Methods

Classification

Binary Classification Problem Definition The Classification Objective The classification of each superpixel S with its correct class label Θ ∈ {θ1 , θ2 } given a vector of traversability cues X = hX1 , X2 , ...Xj , ...Xn i, which is a function of dissimilarity between the superpixel S and the model region M .

Dissimilarity Metric M The G-statistic g(hS j khj )

=2

B X b=1

hS j [b] log

hS j [b] hM j [b]

Sapienza et al (Dept. SCE)

(1)

Autonomous Visual Guidance

July 31, 2012

11 / 21

Methods

Classification

Simple Prior

(4)

0.6

P(Θ= 1|C)

(3)

0.4

 1  1 − exp(−λC) Y0 1 P (Θ = 1|C) = Y1 1 + Y0 (eλC − 1)

P (C|Θ = 0) =

(2)

0.2

1 exp(−λC) Y1

0.0

P (C|Θ = 1) =

0.8

1.0

Probability of a Traversable Surface? A prior that favours superpixels closer to the robot as being traversable:

0

20

40

60

80

100

120

C

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

12 / 21

Methods

Classification

Simple Prior

(4)

0.6

P(Θ= 1|C)

(3)

0.4

 1  1 − exp(−λC) Y0 1 P (Θ = 1|C) = Y1 1 + Y0 (eλC − 1)

P (C|Θ = 0) =

(2)

0.2

1 exp(−λC) Y1

0.0

P (C|Θ = 1) =

0.8

1.0

Probability of a Traversable Surface? A prior that favours superpixels closer to the robot as being traversable:

0

20

40

60

80

100

120

C

In Practice The height of the superpixel centre point C defines the prior likelihood of finding traversable ground. Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

12 / 21

Methods

Classification

Likelihood Functions A Truncated Exponential Mixture Model What is the probability of a dissimilarity metric value given it was grenerated from the traversable/non-traversable class?  
1 P Xj |Θ = θ1 = exp − αj1 Xj (5) Z1  
1 exp αj2 Xj P Xj |Θ = θ2 = Z0

Sapienza et al (Dept. SCE)

(6)

Autonomous Visual Guidance

July 31, 2012

13 / 21

Methods

Classification

Likelihood Functions A Truncated Exponential Mixture Model What is the probability of a dissimilarity metric value given it was grenerated from the traversable/non-traversable class?  
1 P Xj |Θ = θ1 = exp − αj1 Xj (5) Z1  
1 exp αj2 Xj P Xj |Θ = θ2 = Z0

(6)

The Evidence Normalized traversability cue values accumulated in a histogram. Values obtained from Static Traversability dataset. Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

13 / 21

Methods

Classification

Naive Bayes Classification Posterior Probability To which class Θ does the vector of traversability cues X belong? From Bayes’ rule:
Pˆ (Θ = θl )P (X1 ...Xn |Θ = θl ) P Θ = θl |X1 ...Xn = P ˆ m P (Θ = θm )P (X1 ...Xn |Θ = θm )

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

(7)

14 / 21

Methods

Classification

Naive Bayes Classification Posterior Probability To which class Θ does the vector of traversability cues X belong? From Bayes’ rule:
Pˆ (Θ = θl )P (X1 ...Xn |Θ = θl ) P Θ = θl |X1 ...Xn = P ˆ m P (Θ = θm )P (X1 ...Xn |Θ = θm )

(7)

Assuming the traversability cues are conditionally independent: P (X1 ...Xn |Θ) =

n Y

P (Xj |Θ)

(8)

j=1

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

14 / 21

Methods

Classification

Naive Bayes Classification Posterior Probability To which class Θ does the vector of traversability cues X belong? From Bayes’ rule:
Pˆ (Θ = θl )P (X1 ...Xn |Θ = θl ) P Θ = θl |X1 ...Xn = P ˆ m P (Θ = θm )P (X1 ...Xn |Θ = θm )

(7)

Assuming the traversability cues are conditionally independent: P (X1 ...Xn |Θ) =

n Y

P (Xj |Θ)

(8)

j=1

Using the Maximum a Posteriori (MAP) decision rule: Θ ← arg max P (Θ = θl ) θl

Sapienza et al (Dept. SCE)

n Y

P (Xj |Θ = θl )

(9)

j=1

Autonomous Visual Guidance

July 31, 2012

14 / 21

Methods

Classification

Expectation Maximization Learning Task Estimate a hypothesis hj = hαj1 , αj2 i that describe the rate parameters of the truncated exponential mixture model.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

15 / 21

Methods

Classification

Expectation Maximization Learning Task Estimate a hypothesis hj = hαj1 , αj2 i that describe the rate parameters of the truncated exponential mixture model. EM process E -Step: Calculate the Expected value E[X|Θ = θl ] assuming the current hypothesis h holds. M -Step: Calculate the new ML 0 0 hypothesis h0 = hαj1 , αj2 i assuming that the values for E[X|Θ = θl ] were those calculated from E -Step.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

15 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Vision Algorithm

Algorithm

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

16 / 21

Methods

Locomotion

Locomotion Depth Estimation Orientation and distance of obstacle regions can be calculated using trigonometric identities.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

17 / 21

Methods

Locomotion

Locomotion Depth Estimation Orientation and distance of obstacle regions can be calculated using trigonometric identities.

Sapienza et al (Dept. SCE)

Boundary Interpretation Polar range plot is analysed and largest obstacle free areas beyond predefined distance are identified.

Autonomous Visual Guidance

July 31, 2012

17 / 21

Experimental Results

Results

Play video

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

18 / 21

Conclusion

Summary and Conclusion Design of vision system for a robot to guide itself safely in proximity of obstacles using the smallest reported safe window.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

19 / 21

Conclusion

Summary and Conclusion Design of vision system for a robot to guide itself safely in proximity of obstacles using the smallest reported safe window.

The results demonstrate its competence in various indoor and outdoor environments without the need for prior training, temporal model, or adjustments to system parametrization.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

19 / 21

Conclusion

Summary and Conclusion Design of vision system for a robot to guide itself safely in proximity of obstacles using the smallest reported safe window.

The results demonstrate its competence in various indoor and outdoor environments without the need for prior training, temporal model, or adjustments to system parametrization.

We modelled the feature dissimilarity distribution with a truncated exponential mixture.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

19 / 21

Conclusion

Summary and Conclusion Design of vision system for a robot to guide itself safely in proximity of obstacles using the smallest reported safe window.

The results demonstrate its competence in various indoor and outdoor environments without the need for prior training, temporal model, or adjustments to system parametrization.

We modelled the feature dissimilarity distribution with a truncated exponential mixture.

Simple movement behaviour - move towards largest open space.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

19 / 21

Conclusion

Future Work

Learn multiple ground models; How to transition from one ground surface type to another automatically?.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

20 / 21

Conclusion

Future Work

Learn multiple ground models; How to transition from one ground surface type to another automatically?.

Temporal model of traversable area, how does it change with robot movement?

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

20 / 21

Conclusion

Future Work

Learn multiple ground models; How to transition from one ground surface type to another automatically?.

Temporal model of traversable area, how does it change with robot movement?

Movement driven from probability of each superpixel being traversable, not from binary classification result.

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

20 / 21

Conclusion

Questions?

Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

21 / 21

Conclusion

Katramados, I., Crumpler, S., & Breckon, T. (2009). Real-time traversable surface detection by colour space fusion and temporal analysis. In Int. Conf. Computer Vision Systems, volume 5815 (pp. 265–274). Kim, D., Oh, S., & Rehg, J. (2007). Traversability classification for UGV navigation: a comparison of patch and superpixel representations. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (pp. 3166–3173). Lorigo, L., Brooks, R., & Grimson, W. (1997). Visually-guided obstacle avoidance in unstructured environments. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (pp. 373–379). Marr, D. (1982). Vision. A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman and Company, first edition. Ulrich, I. & Nourbakhsh, I. (2000). Appearance-based obstacle detection with monocular color vision. In AIII Conf. on Artificial Intelligence (pp. 866–871). Sapienza et al (Dept. SCE)

Autonomous Visual Guidance

July 31, 2012

21 / 21