How fast can neurons encode visual information? Mark Jack Florida A&M University, Physics Dept. / UCLA Neurobiology Department

Clark Atlanta University, Physics Department, March 11, 2004

Our Experimental Setup

Visual Stimulus

In Vitro Mouse Retina

P

P H

H

P B

H

A H

A

B

AB G

A

GG

[1] M.J. Berry et al. (1994)

G

A G

G

H

H

GG

GG

G

G

Multi-electrode recordings from retinal ganglion cells

visual input .

photoreceptors

H

H

ganglion cells

Stimuli • Natural scenes e.g. mouse movie: Visual scene

Ganglion cell responses 1 2 3 4 5 6 7 8 9 10

0

1 2 Time(s)

3

→ Spike trains for repeated stimuli are highly reliable (little spike time jitter, little deviation in spike count).

Stimuli • Full-field flashes with natural temporal statistics:

~1/f power spectrum (FFT)

Intensity

Example of Stimulus/Response Relationship

Stimulus

Iteration Number

1

Response

5

10

0

2 Time (sec)

4

1. Determine whether the cells act independently to encode visual information. 2. Determine what aspects of the spike trains carry visual information. 3. Reconstruct a visual stimulus from the set of spike trains.

1. Determine whether the ganglion cells act independently or interdependently to encode visual information. Why is this relevant? If they act independently, then the activity of each cell can be decoded separately. If not, then the activity of each cell must be evaluated in the context of the firing patterns of other cells.

2. Determine which aspects of the ganglion cell spike trains carry visual information.

Trial 1

repeated presentation of same stimulus

Trial 2 Trial 3 Trial 4 Trial 5 Trial n

… t (time)

3. Reconstruct a visual stimulus from a recorded set of spike trains.

?

Shannon’s Information theory That is, we want to know: What is P (s | r1, r2, r3,...) , where s is the stimulus, and ri is the response of the i th ganglion cell? [2] C.E. Shannon (1948), [3] S.P. Strong et al. (1998)

We want the probability P (s | r1, r2, r3 , ...). But we measure P (r1, r2, r3 , ... | s). Use Bayes’ theorem to get what you want from what you measure: P (s| r1, r2, r3 , ...) = Measured quantity

P (r1, r2, r3 , ... | s) P (s) P (r1, r2, r3 , ...)

Calculated quantity

Determined a priori by nature or by the experimenter.

Mutual information: I = S (s) – S (s | r1, r2, r3 , ...) = S (r1, r2, r3 , ...) – S (r1, r2, r3 , ... | s) (Bayes theorem) S = – P ln P entropy (uncertainty)

Experiment: Present a movie, record ganglion cell responses, measure the amount of information about the movie that could be obtained from the responses when a) their correlations are taken into account. b) their correlations are ignored; i.e. treating P(r1,r2|s) as P(r1|s)P(r2|s). Question: How much information is lost when correlations are ignored?

1. Determine whether the cells act independently to encode visual information [4]. 2. Determine what aspects of the spike trains carry visual information. 3. Reconstruct a visual stimulus from the set of spike trains. [4] S. Nirenberg, S.M. Carcieri, A.L. Jacobs, P.E. Latham, Nature 411 (2001) 698-701.

What aspects of ganglion cell spike trains carry visual information ? • Take a given spike train and bin it in time intervals of different size. =

Most 1 0 0 0 0 1 1 1 1 0 Precise Pattern

=

10121

=

14

=

5

Blurred Pattern

• Do we lose visual information when we go to bigger bins, and if yes, how much?

How quickly after stimulus onset can you tell whether it’s stimulus A or B from looking at the spike pattern? Retinal Ganglion Cell’s Response to Stimulus Stimulus A

t cell’s receptive field

B

Spikes moving down the axon toward the brain t

Possible Encoding Strategies

Rate (spikes/bin) Latency (time to first spike after stimulus presentation) Interspike Intervals (ISI) (average time between spikes)

180 ms

152 ms

7 ms

5 ms

NOTE: Neuronal firing is a stochastic (non-deterministic) process.

Recent model predicts latency as the fastest encoding (R. van Rullen and S.J. Thorpe [5]).

Rate Coding Latency Coding

Our Experimental Setup

Visual Stimulus

In Vitro Mouse Retina

P

P H

H

P B

H

A H

A

B

AB G

A

GG

[1] M.J. Berry et al. (1994)

G

A G

G

H

H

GG

GG

G

G

Multi-electrode recordings from retinal ganglion cells

Step 1: Establish a linear convolution kernel for each cell. Use Reverse Correlation technique for finding the receptive field (kernel) of each cell: 1. Show many iterations of random spatial noise to the retina. 2. Average the images that occurred just prior to a spike. Example of a Spatial Kernel of a Cell

Step 2: Generate a lookup table for weight-to-encoding for each cell. 1. 2. 3. 4.

Show a set of images to the retina. Record the spike patterns in response to each image. Convolve the receptive field (kernel) of each cell with the image. Record the result of the convolution together with the corresponding firing rate, latency, or ISI. 5. Normalize the weights from 0 to 1.

Image

…

Kernel

Effect on cell

Weight (normalized sum of pixels)

Recorded Rate (spikes/sec)

*

=

0.3

20

*

=

0.8

80

…

…

Step 3: “Tile” a virtual retina with the recorded cells and decode an image. P H P

H

A

H A A G G

A

G

P

H B

B

AB

G G G G

G G

H

H

G

G

G

The experimental setup gives us only about a dozen cell recordings for one picture. By “scanning” a larger picture across the retina, each recorded cell has the same effect as having many cells looking at the picture at the same time.

P

H

P

H

P

H

A

H A

H

AB GG

G A

G

A G

G

H

B B

GG GG

G G

Step 4: Reconstruct the image using the weight-to-encoding table. P

H

H P A

H

P

A A G

H

G

H G G

G G G G

H

B B

A A B G G

G

G

Lookup Table

Firing rate = 20

Weight Recorded Rate (sum of pixels) (spikes/sec)

Scale the cell’s kernel by the Weight 0.3 *

0.3

20

0.8

80 …

Add to Reconstruction

=

Repeat this procedure for all cells and for every location in the image.

Sample Decoded Image

Presented Original Image

Theoretical Prediction Based on Mouse Eye Optics

Reconstruction from Neural Responses

Results

Conclusions • Latency encodes visual information faster than other encoding strategies. • Does the brain use this? Some psychophysics experiments say maybe. • What about movies? • Application - Prosthetic Device: Stimulation strategy based on latency encoding – bright stimulation first; dark, last.

Acknowledgements • • • • • •

Sheila Nirenberg Gene Fridman Peter Latham Steve Carcieri John Sinclair Adam Jacobs

References [1] M.J. Berry, D.K. Warland, M. Meister, Proc. Natl. Acad. Sci. USA, 94 (1997) 5411-5416. [2] C.E. Shannon, The Bell System Technical Journal, Vol. 27 (1948) 379-423, 623-656. [3] S.P. Strong et al., Phys. Rev. Lett. 80 (1998) 197-200. [4] S. Nirenberg, S.M. Carcieri, A.L. Jacobs, P.E. Latham, Nature 411 (2001) 698-701. [5] R. van Rullen, S.J. Thorpe, Neural Computation 13 (2001) 1255-1283.