Diversity of E cient Coding Solutions for a Population of Noisy Linear Neurons 1 Eizaburo Doi ,
3 Liam Paninski ,
1,2 Eero P. Simoncelli
1 New York University, 2 Howard Hughes Medical Institute, 3 Columbia University
Purpose
Results
Efficient coding posits that neural resources (cost) should efficiently be utilized for signal representation (objective). (5, 14) • Dependency on the objective function has been well examined. (13, 6, 8) • Dependency on the cost function: - For a single neuron: output range, firing rate, or variance constraints. (11, 4, 14) - For multiple neurons: less studied. We examine the effects of costs that are likely to be relevant in neural populations.
1. Optimal Transformation Depends on the Neural Resource Cost
2. Intuition for the Solution
Solution under the response strength cost
How does the cost change the solution?
It is well-known that the optimal filtering depends on input SNR, exhibiting the transition from band-pass to low-pass. (1-3, 10)
Consider the case of zero sensory noise. The amount of information conveyed by a single neuron is defined by the SNR of the neuron.
10^–2
10^3
Power
Sensory noise
Low-luminance signal
10^–6 1
(high spatial frequencies) x2
10^2
10
10^0
100
High-luminance signal 1
10
Spatial frequency
100
Spatial frequency
x1
The optimal filtering is always low-pass.
10
0 0
θ Principal axis is most informative.
20
10^1
Low-luminance signal
1
10
SNR
Power
Cost function(s): • Individual neuron’s response strength (variance) (1-3, 10) • Individual neuron’s synaptic strength (L2 norm of filter) (7) • Number of neurons
10^2
Synaptic cost
10^3
Objective function: • Transmitted information
π/2
π/4
x2
Optimal Modulation
High-luminance signal
Efficient coding solution can be derived for any size of noisy neural population. The size can be assumed arbitrarily, or selected by optimization with a generalized cost.
•
Efficient codes are generally redundant; even in the undercomplete case.
With noisy neurons, efficient coding is naturally defined in overcomplete cases. • Illustrative example: response cost & no sensory noise (in the complete case, whitening is the solution). # neurons: 3
Feasible lters satisfying the response cost constraint
Definition of efficiency
•
(low spatial frequencies)
Solution under the synaptic strength cost
Methods
Any direction is equally informative.
Signal covariance
θ
Efficient coding can lead to a diverse set of solutions, depending on the relative cost of response and synaptic strengths.
3. Efficient Coding in Overcomplete Cases
20
Filter w
10^1
Response cost
10^–4
Low-luminance signal
•
x1 Feasible lters satisfying the synaptic cost constraint
x2
x2
10
Response cost
High-luminance signal
10^0
One neuron case: • Response cost: any feasible filter is equally good. • Synaptic cost: the one on the principal axis is optimal.
Optimal Modulation (MTF)
SNR
Input Power Spectrum
Power
Efficient coding is commonly simplified and equated to redundancy reduction. (5) • Natural signals are highly correlated. • With no noise, an efficient encoder decorrelates the signal, and transmits independent information. In reality, individual neurons are noisy and precision is limited. In such a case, correlated representation would be beneficial for robust signal representation. (12, 9) We analyze how the redundancy is utilized in a noisy neural population.
Summary
x1
10
x1
uk
uk
uj
π/2
π/4
x2
x1
uk
0 0
100
uj
uj
θ
100
Spatial frequency
Neural noise
Both response and synaptic strength costs should be relevant to neural systems.
Linear lters
(combined cost) = a (synaptic) + (1–a) (response)
u2
x2
with a a parameter to control the relative importance.
Structured input (correlated Gaussian) of high dimension Sensory noise (additive white Gaussian) Neural noise (additive white Gaussian) Arbitrary population size (input : output ratio)
x2
a=0.8
10^2
10^0
a=0
1
10
T
with b another parameter. The optimization remains convex.
2 T σν IN )W ]
(1) J. J. Atick and A. N. Redlich. Towards a theory of early visual processing. Neural Computation, 2:308–320, 1990. (2) J. J. Atick, Z. Li, and A. N. Redlich. Color coding and its interaction with spatiotemporal processing in the retina. Technical Report IASSNS-HEP-90/75, Institute for Advanced Study, November 1990. (3) J. J. Atick and A. N. Redlich. What does the retina know about natural scenes? Neural Computation, 4:196–210, 1992. (4) R. Baddeley, L. F. Abbott, M. J. A. Booth, F. Sengpiel, T. Freeman, E. A. Wakeman, and E. T. Rolls. Responses of neurons in primary and inferior temporal visual cortices to natural scenes. Proc. R. Soc. Lond. B, 264:1775–1783, 1997. (5) H. B. Barlow. Possible principles underlying the transformation of sensory messages. In W. A. Rosenblith, editor, Sensory communication, pages 217–234. MIT Press, MA, 1961. (6) M. Bethge. Factorial coding of natural images: how e ective are linear models in removing higher-order dependencies? J. Opt. Soc. Am. A, 23(6):1253–1268, 2006. (7) A. Campa, P. D. Giudice, N. Parga, and J.-P. Nadal. Maximization of mutual information in a linear noisy network: a detailed study. Network: Computation in Neural Systems, 6:449–468, 1995. (8) E. Doi and M. S. Lewicki. A theory of retinal population coding. In Advances in Neural Information Processing Systems 19 (NIPS*2006), pages 353–360. MIT Press, 2007. (9) E. Doi, D. C. Balcan, and M. S. Lewicki. Robust coding over noisy overcomplete channels. IEEE Transactions on Image Processing, 16:442–452, 2007. (10) J. H. van Hateren. A theory of maximizing sensory information. Biological Cybernetics, 68:23–29, 1992. (11) S. Laughlin. A simple coding procedure enhances a neuron’s information capacity. Z. Naturforsch., 36(c):910–912, 1981. (12) K.-H. Lee and D. P. Petersen. Optimal linear coding for vector channels. IEEE Transactions on Communications, COM-24:1283–1290, 1976. (13) D. L. Ruderman. Designing receptive elds for highest delity. Network: Comput. Neural Syst., 5:147–155, 1994. (14) E. P. Simoncelli and B. A. Olshausen. Natural image statistics and neural representation. Annual Review of Neuroscience, 24:1193–216, 2001.
30
w/o cost for the number of neurons
20
w/ cost for the number of neurons
0
x1
u1
This explains the above low-pass modulation (the principal axis corresponds to the low frequencies).
The general mechanism to maximize information transmission is to best balance: (a) Independent representation of different signal dimensions (serving to add independent channels). (b) Redundant representation of the same signal dimension with noisy neurons (increasing SNR by the factor of the redundancy). Unless the SNR is flat, the selection of the best balance is non-trivial.
40
80
120
Number of neurons
x2
Automatically select the optimal population size given the total resource budget. NOTE: Even if the population size is not large enough to cover the input dimension, efficient codes can be redundant (especially when neurons are noisy). Less noisy neurons
More noisy neurons 30
20
100 complete case
10
0 0
100
Number of neurons
0
30
Whitening completely removes redundancies from the signal. Intuitively, the signal dimension with high variance should be represented more. Under the presence of input noise, this is implemented because amplifying small signal is penalized as it amplifies sensory noise as well. 20
u2 With noisier neurons x1
Direction starts to matter with sensory noise.
Low signal SNR x1
10
High signal SNR
Low signal SNR
Feasible lters at di erent SNR.
0 0
π/4
π/2
θ
Now the same mechanism can be used to maximize information transmission: balancing independent and redundant representations. “The noisier the neurons are, the more redundant the representation should be.” Atick and Redlich proposed an intuitive approximation for efficient coding solution: The optimal filtering can be separable into: (3) 1. Low-pass filtering (to reduce the effect of input noise). 2. Whitening (to defend against corruption by output noise). This approximation is incorrect because whitening (and hence the approximation) does not depend on output noise level.
• Response cost: always whitening; redundancy is never employed. • Synaptic cost: two neurons become completely redundant. 0
“Principal axis should be over-represented.”
High signal SNR
How does the solution vary with higher neural noise?
10
Taking sensory noise into account makes the formulation more realistic, and also resolves otherwise non-intuitive whitening solution with the response cost.
x2
(generalized cost) = a (synaptic) + (1–a) (response) + b (number of neurons)
Represented dimension
(C/M − β)1M = α ρ diag[WW ] + (1 − α) diag[W(Σs + T
u2
Another important cost: number of neuron.
Information [bits]
+ βM
Using SVD, W = PΩQ’. • Qopt: the eigenvector matrix of Σs; roughly speaking, spatial frequencies. • Ωopt: derived with a convex optimization; this is the optimal modulation of the signals in its eigen-space/spatial frequency. • Popt: any orthogonal matrix. In order for the individual neurons to satisfy a fixed cost constraint, we need to find an orthogonal matrix P numerically so that W satisfies: 2
100
Further generalization of the cost
Information [bits]
C = α ρ tr[WW ] + (1 − α) tr[W(Σs +
2 T σν IN )W ]
Feasible lters with di erent a.
Spatial frequency
subject to the generalized cost function:
x2
Synaptic cost
2 T 2 |WΣs W + σν WW + σδ IM | |σν2 WWT + σδ2 IM |
The solution is whitening. This explains the above optimal modulation in high SNR regime.
a=1
Synaptic cost
The problem is to find W that maximizes the mutual information
T
a=0.5
u1
x1
x1
Technical details
2
a=0
High-luminance signal
10^3
10^1
1 I(s; r) = log 2
Response cost
Optimal Modulation with di erent a a=1
Appendix. Presence of Sensory Noise is a Key for Intuitive Solutions
Filter output covariance
Representation
Power
• • • •
Filter output
Filter input
Signal
Filter outputs are highly correlated in the most efficient codes.
Response cost
Combination of the two costs
Optimal modulation w/ more noisy neuron u1
100
Power
Sensory noise
Two neurons case: • Response cost: two independent channels double the information. • Synaptic cost: jointly representing more about the principal axis.
SNR
Linear Gaussian model
Optimal modulation w/ less noisy neuron
10^5
10^5
10^0
10^0
10^–5
10^–5
10^–10
10^–10
Input SNR dB −10 0 10 20 40 80
20
10
0 0
100
0
It is misleading to extrapolate general properties of efficient coding from the specific case of response cost, no sensory noise, and complete representation. In general, efficient coding with noisy neurons leads to redundant representations.
1
10
100
1
10
100
Spatial frequency
If the approximation were true, the left and right panels would be identical.
