Computational Vision Motion processing

Motion as the most primitive form of vision

image sources: http://www.onekind.org/be_inspired/animals_a_z/zebra/ http://ykonline.yksd.com/distanceedcourses/Courses09/PhysicalScience/Lessons/ThirdQuarter/ Chapter07/07-01/48RabbitFreez.jpg

Motion perception: Navigation

Motion perception: Navigation

source: http://opticflow.bu.edu http://psych.hanover.edu/krantz/motionparallax/motionparallax.html

Motion perception: 
 Figure ground

Motion perception: 
 Structure from motion

Motion perception: Kinetic depth effect

Motion perception: Action recognition

more demos: http://www.biomotionlab.ca/?page_id=11

Motion as correspondence problem Apparent motion

see Ternus demo at http://www.michaelbach.de/ot/mot_Ternus/index.html

Motion as a correspondence problem

Optical flow in computer vision

image source: Militello et al 2014

Optical flow in computer vision Int J Comput Vis (2011) 92: 1–31

source: Baker et al 2011

11

Optical flow in computer vision • Challenge: Non-rigid scenes!

• Uses high-frequency fluorescent texture (UV light)

source: Baker et al 2011

Int J Comput V

Optical flow inetcomputer vision 612 D.J. Butler al.

Fig.et 1.alMPI-Sintel Flow data set. Sample ground truth flow fields and correspondsource: Butler 2012

Ventral vs. dorsal streams

Seeing without motion • Patient LM: 43 yr old, stroke with bilateral posterior parietal and occipital regions

• Complete loss of motion perception

• Comment from scientists that have worked with her for years [Zihl et al ’83]:

- She had difficulty for example, in pouring tea or coffee because the fluid appeared to be frozen like a glacier [...]

- In a room where more than two people were walking [...] she usually left the room because “people were suddenly here or there but [she had] not seen them moving”

Motion in non-primates

source: Lettvin et al 1959

© 2000 Nature

Detecting motion: The Reichardt detector

experiments, using the optomotor response of the beetle Cholorphanus as a behavioral measure. This response is the animal’s tendency to follow the movement of the visual surround to compensate for its mistaken perception of selfmotion in the opposite direction. The beetle was glued to a rod so it could not move its body, head or eyes relative to the surround, but could express its behavior at decision points by rotating a ‘Y-maze globe’ Fig. 1. Tethered Chlorounder its feet (Fig. 1). Their results 3 led to the phanus walking on the Ydevelopment of a model for maze globe (from ref. 10). motion detection that became known as the ‘correlation-type motion sequence of d detector’, the ‘Hassenstein-Reichardt two neighbo model’ or briefly—omitting half the origpreferred or n inal team—the ‘Reichardt detector’ low and Levi (Fig. 2). The core computation in this the null dire model is a delay-and-compare mechacantly reduce nism: delaying the brightness signal as individual res measured by one photoreceptor by a lowto the prefer roughly equ responses. T The author is at the ESPM-Division of Insect mechanism o Biolog y, 201 Wellman Hall, Univ. of California as the basis f Berkeley, Berkeley, California 94720, USA. e-mail: [email protected] this study, the 1168

© 2000 Nature

experiments, using the optomotor response of the beetle Cholorphanus as a behavioral measure. This response is the animal’s tendency to follow the movement of the visual surround to compensate for its mistaken perception of selfmotion in the opposite direction. The beetle was glued to a rod so it could not move its body, head or eyes relative to the surround, but could express its behavior at decision points by rotating a ‘Y-maze globe’ Fig. 1. Tethered Chlorounder its feet (Fig. 1). Their results 3 led to the phanus walking on the Ydevelopment of a model for maze globe (from ref. 10). motion detection that became known as the ‘correlation-type motion sequence of d detector’, the ‘Hassenstein-Reichardt two neighbo model’ or briefly—omitting half the origpreferred or n inal team—the ‘Reichardt detector’ low and Levi (Fig. 2). The core computation in this the null dire model is a delay-and-compare mechacantly reduce nism: delaying the brightness signal as individual res measured by one photoreceptor by a lowto the prefer roughly equ responses. T The author is at the ESPM-Division of Insect mechanism o Biolog y, 201 Wellman Hall, Univ. of California as the basis f Berkeley, Berkeley, California 94720, USA. e-mail: [email protected] this study, the

Detecting motion: The Reichardt detector (1961) 2 mm

photoreceptors

1168

source: Stone & Frisby (2010)

excitatory 
 (e.g., ‘x’, ‘+’)

inhibitory 
 (e.g., ‘/‘, ‘-‘)

Borst & Helmstaedter 2015

Borst & Helmstaedter 2015

Visual motion detection in the fly

Visual motion detection in the fly

Takemura et al 2013

Direction selectivity in V1

Hubel & Wiesel

Motion selectivity in the primary visual cortex

Shmuel & Grinvald ’96

Motion as orientation in space-time

/February 19852/February 1985 A/Vol. 2, No.

E. H. Adelson and J. R. Bergenand J. R. Bergen E. H. Adelson

in rather the frequency domain. n, than in the frequency domain. nce the between the sampled cit difference between the sampled oving we simply ns of bar. the If moving bar. If we simply from(Fig. the 4b) sampled s. 4b) pattern from the sampled patiotemporal plot of the plot of the derive a new spatiotemporal Fig. 4c. Since the 4c. differillustrated in Fig. Since the differer have displayed it ondisplayed a negative, we have it on a dsraytocorresponds zero, white to to zero, posi-white to posithat the sampled-motion ative. Observe that the sampled-motion to be the sum to of be thethe realsum of the real be considered Fig.artifacts 4c. That the of is, Fig.we4c.canThat is, we can continuous motion with motion with motion as being continuous o it. equently in time, the apmpled more frequently in time, the aps improved, i n as shown i n uous motion as is shown improved, ig. 4f) have(Fig. rather4f)little he artifacts have rather little trequencies we can see. sampling 5. a-e (x, t)Fig plots of bars to bars the left or totothe at to the right at thatIf we can see. If Fig sampling 5. a-e (x, t)moving plots of moving the right left or f, Motion is like orientation in (x, t), and ain spaplainly a point come at various speeds. f, Motion is like orientation (x, t), and a spagh, there come will plainly avarious point speeds. at tiotemporally oriented receptive field can be used to detect g, The tiotemporally oriented receptive field can be it.used to detect it. g, The ve so little energy the energy omponents have soinlittle in the same oriented receptive field can respond to can sampled motion just as motion just as same oriented receptive field respond to sampled ncy range that they will mporal-frequency range that itthey will to continuous responds it respondsmotion. to continuous motion. atiotemporal structure of ce the fine spatiotemporal structure of bility the spatial by andthe spatial and Adelson & Bergen ’85 urred tobyinvisibility Fig. 5g, it responds to a sampled version of theversion same of the same is continuous Fig. 5g,well it responds well to a sampled he point, eye. Atthethis point, the continuous

Motion as orientation in space-time Reichardt detector in space-time Smoother Reichardt detector excitatory inhibitory

first RF time

second RF

space

excitatory inhibitory

time

space

2nd neuron has a spatially separated Receptive Field (RF), and a shorter temporal delay Like an oriented V1 receptive field, but oriented in space-time!

if the outputs of a large number of receptors are are temporall filtered in the same way, and the filtered outputs are combine with a spatial weighting function, then again the net respons will be separable. Separability is frequently observed in th early stages of cortical visual processing.25,26 Figure 7 illustrates how the spatiotemporal impulse re sponse may be used to analyze the way a unit will respond t a stimulus. The stimulus here is a light dark edge that i initially stationary, then moves to the right, and then become responsestationary again. The stimulus may be considered to lie o a continuous strip, which is drawn upward over time, as show in Fig. 7a. A picture of the unit's impulse response is overlai on the stimulus, to show how inputs at all points and times ar

Building spatio-temporal RFs (LGN and V1 L4) Spatiotemporally separable impulse

DeValois et al ’00

Fig. 6. A spatiotemporally separable impulse response. The spati and temporal impulse responses are shown along the margins. The product is shown schematically in the center. The spatiotempor impulse re-sponse is a weighting function that sums inputs at variou positions and times to determine the present output.

Adelson & Bergen ’85

Spatiotemporally separable impulse response Spatiotemporally separable vs. inseparable impulse response 290

J. Opt. Soc. Am. A/February 1985

E. H. Adelson and J. R. Bergen

t x

Fig. 8. a, An (x, t) plot of an edge that is stationary, then moves sinusoidally, and then is stationary again. b, A separable spatiotemporal impulse response magnified four times for clarity. c, The convolution of a and b, i,e., the output of a separable channel. There is no selectivity for direction of motion. d The stimulus again. e, A spatiotemporally oriented Gabor function, magnified four times. f, The convolution of d and e. The output is strongly selective for rightward motion.

edge detector of Fig. 5f, this is not an easy unit t o build in a physiological system, but there are ways of approximating it that are physiologically plausible.

cussed how the oscillations might be used to advantage i n computing velocity.) A phase-independent motion detector can be built as shown in Fig. 9. We begin with two units that act as linear spa-

Adelson & Bergen ’85

A variety of receptive field types in the primary visual cortex • Two types of cells in V1:

- tuned to static bars and gratings
 - tuned to drifting bars and gratings

7736 J. Neurosci., December 1, 1996, 16(23):7733–7741

proportion of cells

separable

inseparable

Movshon and Newsome • V1 Neurons Projecting to MT

directionality index

Ventral vs. dorsal streams

Spatiotemporally separable impulse response Spatiotemporally inseparable impulse response 290

J. Opt. Soc. Am. A/February 1985

E. H. Adelson and J. R. Bergen

sensitivity to phase

t x

Fig. 8. a, An (x, t) plot of an edge that is stationary, then moves sinusoidally, and then is stationary again. b, A separable spatiotemporal impulse response magnified four times for clarity. c, The convolution of a and b, i,e., the output of a separable channel. There is no selectivity for direction of motion. d The stimulus again. e, A spatiotemporally oriented Gabor function, magnified four times. f, The convolution of d and e. The output is strongly selective for rightward motion.

edge detector of Fig. 5f, this is not an easy unit t o build in a physiological system, but there are ways of approximating it that are physiologically plausible.

cussed how the oscillations might be used to advantage i n computing velocity.) A phase-independent motion detector can be built as shown in Fig. 9. We begin with two units that act as linear spa-

Adelson & Bergen ’85

compressive nonlinearity following the sum-of-squares stage, in order to keep the outputs within a reasonable range (cf. the research of Pantle and Sekuler28 on the motion aftereffect). Such a monotone transformation would not affect the basic properties of the motion-extraction process but could have an effect on the performance observed in various tasks, such as the accuracy with which changes in speed and contrast could be judged. Energy was extracted in Fig. 9a by using the standard trick

Extracting spatio-temporal motion energy • Goal: To build a RF that takes on a constant value for a constant motion

back to H&W
 pooling over position and phase but otherwise identical tuning properties phase-independent measure of motion information

pha fun ver ture wit 6.

Wa orie out por clud to c one the for spa por a s par asy ma deg A and sep the T to sho (Fi this sec imp

wh abo app N and let

Frequency specific motion aftereffect and opponent channels

https://www.youtube.com/watch?v=OAVXHzAWS60