Difference of gaussians type neural image filtering with spiking neurons Bio-inspired preattentional vision system
Sylvain Chevallier LIMSI - CNRS Orsay, France
[email protected]
October, 7th. 2009
Framework Neural image filtering
Visual attention Neural computation
Outline
1. Framework Visual attention Neural computation 2. Neural image filtering Methods Results Saliency extraction
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
2 / 21
Framework Neural image filtering
Visual attention Neural computation
Change blindness
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
3 / 21
Framework Neural image filtering
Visual attention Neural computation
Change blindness
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
3 / 21
Framework Neural image filtering
Visual attention Neural computation
Change blindness
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
3 / 21
Framework Neural image filtering
Visual attention Neural computation
Bio-inspired attentional vision systems Visual features I
Attentional spotlight metaphor
I
Reduce the search space
Definitions I
Preattention and attention
I
Covert attention et overt attention
Bio-inspired vision I
Propose efficient bio-inspired solutions
I
Between realistic models and artificial systems
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
4 / 21
Framework Neural image filtering
Visual attention Neural computation
Existing implementations
Main characteristics 1. Feature decomposition 2. Combination on a saliency map 3. Focus of attention with WTA selection Neural models proposed in (2) and (3) [Itti & Koch, 98]
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
5 / 21
Framework Neural image filtering
Visual attention Neural computation
Existing implementations
Some applications: I
Driver assistance [Michalke, 08]
I
Medical images [Fouquier, 08]
I
Robotics [Frintrop, 06]
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
5 / 21
Framework Neural image filtering
Visual attention Neural computation
Spiking neurons
I
“Third generation” of neural models
I
Precise spike timing Selective information processing
I
I
I
Different behaviors I I
I
Implicite thresholding Temporal integrator Coincidence detector
Anytime computation
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
6 / 21
Framework Neural image filtering
Visual attention Neural computation
Proposed architecture
Inputs I
Luminance
I
Color
Features I
Contrasts/Edges
I
Orientation Color opponency
I S. Chevallier (LIMSI - CNRS)
I
High and low frequencies
I
Saliency extraction
I
Focus of attention
Neural Image Filtering
ICNC’09
7 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Outline
1. Framework Visual attention Neural computation 2. Neural image filtering Methods Results Saliency extraction
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
8 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Neural model
Leaky Integrate-and-Fire (LIF) dV i dt = −λi Vi (t) + ui (t), if Vi < ϑ else trigger a spike and Vi ← Vreset I
Vi (t) : membrane potential
I
λi : relaxation constant
I
ui (t) : command
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
9 / 21
Framework Neural image filtering
Methods Results Saliency extraction
From pixels to spikes dVi dt
with τ = 1/λ
= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset
details
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
10 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Neural image filtering
Neural maps: Input map Pixels to spikes Filter Filtering results Here : difference of gaussians (DOG) type filter
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
11 / 21
Methods Results Saliency extraction
Framework Neural image filtering
Neural image filtering (
dVj dt
P
j = −λj Vj (t) + ∑i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset
DOG filter
Neural filter
details
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
12 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Edge preservation on artificial images
Validation process 1. Noise corruption 2. Neural and DOG filtering 3. Sobel and threshold 4. Comparison with original image
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
13 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Edge preservation on artificial images
"worst" neuronal
"best" neuronal
Gradual results
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
14 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Edge preservation on artificial images
Neural
DOG
Only a few neurons are activated
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
14 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Edge preservation on artificial images
Robust to noise Edge preservation
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
14 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Preattentional architecture
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
15 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Saliency extraction on natural images
Comparison with Itti algorithm
Itti
S. Chevallier (LIMSI - CNRS)
Original
Neural Image Filtering
Neuronal
ICNC’09
16 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Saliency extraction on natural images
Quickly obtain a gradual result
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
16 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Conclusion
I
Bio-inspired neural image filtering
I
Temporal processing
I
Good edge preservation
I
Results obtained gradually
Perspectives I
Mathematical study of neural filtering
I
Extended experimental comparisons
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
17 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Annex
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
18 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Input maps
dVi dt
= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset
with Li the considered pixel value Φi
ˆti = − 1 ln 1 − λi ϑ λi KLi
=
λi = − iϑ ln 1 − λKL i
back
≈
S. Chevallier (LIMSI - CNRS)
1 ˆti
Neural Image Filtering
K Li ϑ
ICNC’09
19 / 21
Framework Neural image filtering
Methods Results Saliency extraction
Input maps
back
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
19 / 21
Methods Results Saliency extraction
Framework Neural image filtering
Integration maps
(
P
dVj dt
j = −λj Vj (t) + ∑i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset
Ni
Si (t) =
∑ δ (t − tif ) f =1
back
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
20 / 21
Methods Results Saliency extraction
Framework Neural image filtering
Integration maps
(
P
dVj dt
j = −λj Vj (t) + ∑i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset
Pj
Vj (t) =
Ni
∑ wij
i=1
j
f =1 Pj
Vj (Tj ) ≈
ˆi
∑ e−λ (t−f t ) H(t, f ˆti )
∑ wij
i=1
1 − e−QNi /Li 1 − e−Q/Li
avec Q =
λj ϑ K
back
S. Chevallier (LIMSI - CNRS)
Neural Image Filtering
ICNC’09
20 / 21
Methods Results Saliency extraction
Framework Neural image filtering
Frequency coding
P1 P2 P3 P4
V ϑ S t ISI
6 ms
S. Chevallier (LIMSI - CNRS)
4 ms
5 ms
Neural Image Filtering
5 ms
4 ms
ICNC’09
21 / 21
