A Generalization of the Rate-Distortion Function for Wyner-Ziv Coding of Noisy Sources in the Quadratic-Gaussian Case David Rebollo-Monedero and Bernd Girod
Information Systems Lab. Electrical Eng. Dept. Stanford University
April 2, 2005
Main Results Source Data
X n
Reconstruction
WZ Encoder
WZ Decoder
µ(y)
n
ˆ X n
Y Side Information
Direct Wyner-Ziv (WZ) coding
y y y y
Quadratic distortion No rate loss due to unavailability of side information at encoder Arbitrarily distributed side information Closed-form expression for RD function
Extension to indirect WZ coding Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
2
Outline RD function for WZ coding in the quadratic-Gaussian case
y Previous work y Our generalization y Direct and indirect cases Experimental results
y Simple Gaussian mixture y Information-theoretic vs. operational RD performance
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
3
Direct WZ Coding Source Data
X n
Reconstruction
Encoder n
Decoder n
ˆ X n
Y Side Information
Distortion Information-theoretic RD function
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
4
Previous Work on Direct WZ Coding Source Data
X n
WZ Encoder
Reconstruction WZ Decoder n
ˆ X n
Y Side Information
Quadratic-Gaussian case [Wyner, Ziv, 76-78] Extension [Pradhan, Chou, Ramchandran, 03]
y y y
arbitrarily distributed
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
5
Main Result on Direct WZ Coding Source Data
X n
Reconstruction
WZ Encoder
WZ Decoder
µ(y)
n
ˆ X n
Y Side Information
Slightly relaxes conditions
y y y
arbitrarily distributed
Direct proof, not using duality arguments Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
6
Indirect WZ Coding Indirect Source Data Observation
Source Data
X n
n
n
n
p(z | x , y ) n
Y
X Z n
Reconstruction
Encoder n
Decoder n
ˆ X n
Y Side Information
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
7
Previous Work on Indirect WZ Coding Indirect Observation
Source Data
X n
n
n
n
p(z | x , y )
Z n
NWZ Encoder
Reconstruction NWZ Decoder
n
Y
n
ˆ X n
Y Side Information
Quadratic-Gaussian case [Yamamoto, Itoh, 80]
y y y
jointly Gaussian
Similar, independent work [Flynn, Gray, 87], [Draper, 02]
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
8
Main Result on Indirect WZ Coding Source Data
Indirect Observation
Z
X
n
Reconstruction
NWZ Encoder
NWZ Decoder
µ(y)
ˆ X n
n
α
f (y)
Y Side Information
arbitrarily distributed
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
9
Intuitive Example Indirect Observation
Source Data
Reconstruction
-1 1
X ∼ N (0, 1)
Z n
NWZ Encoder
NWZ Decoder n
Y = Side Information
ˆ X n
½
−1, w.p. 1/2 1, w.p. 1/2
If
available at encoder, encode
If
encoded and decoded ignoring
and _
,
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
10
Operational RD Performance Indirect Observation
Quantization Index
Z
Q
n
q(z n )
SW Encoder
Reconstruction SW Decoder n
Q
ˆn (q, yn ) x n
ˆ X n
Y Side Information
Lloyd algorithm for noisy WZ quantization [Rebollo-Monedero,
Girod, 05] High-rate noisy WZ quantization [Rebollo-Monedero, Rane,
Girod, 04]
y Lattice quantizer and Slepian-Wolf codec asymptotically optimal y ( normalized moment of inertia) Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
11
RD Comparison 9 8 Z W |Y Z N X
1/D [dB]
7 6 5 4 3 2
st Di
t or
e at R ion
)
D
n tio c n Fu
e op l S
(R
≈
Noisy WZ Quantizers (n=1,2,3)
t /bi B d 2 0 6.
High-Rate Approximation (n=1,2,3)
1 0
0.5
1
1.5
R [bit]
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
12
Conclusions No rate loss incurred in WZ coding
y Jointly Gaussian statistics are not necessary y Relaxed conditions in both the direct and indirect case y Side information arbitrarily distributed In the indirect case, condition on data similar to additive
separability in high-rate noisy WZ quantization All conditions determined by conditional joint distribution of
(data, observation) given side information, arbitrarily distributed Paper at www.stanford.edu/~drebollo/publications.htm
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
13
A Generalization of the Rate-Distortion Function for Wyner-Ziv Coding of Noisy Sources in the Quadratic-Gaussian Case David Rebollo-Monedero and Bernd Girod
Information Systems Lab. Electrical Eng. Dept. Stanford University
April 2, 2005
Outline Introduction and previous work Theoretic analysis
y Indirect Wyner-Ziv coding y Generalization of the RD function in the quadratic-Gaussian case Example with experimental results
y Simple Gaussian mixture y Information-theoretic vs. operational RD performance
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
15
Noisy Source Coding with Decoder Side Information Data of Interest
Noisy Medium
Encoder
Decoder
Statistical dependence is known Side information not available at the
encoder Is there a loss in rate-distortion (RD)
performance?
Side Information
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
16
Indirect Wyner-Ziv (WZ) Source Coding Problem Data of Interest
Noisy Medium
Encoder
Decoder
In the quadratic-Gaussian case
y No loss in RD performance y Closed expression for RD function We generalize the conditions in the current
literature for
y Direct case (source data observed) y Indirect case (noisy observation)
Side Information
Proofs do not use duality arguments Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
17
Some Fundamental Results on WZ Coding Theory Directly observed data
y One-letter characterization and quadratic-Gaussian case [Wyner, Ziv, 76-78] y Case in which source data is sum of arbitrarily distributed side information and independent Gaussian noise [Pradhan, Chou, Ramchandran, 03] y Side-information-dependent distortion functions [Csiszár, Körner, 81], [Linder, Zamir, Zeger, 00] Indirectly observed data
y One-letter characterization and quadratic-Gaussian case [Yamamoto, Itoh, 80], [Flynn, Gray, 87], [Draper, 02] y Modified distortion functions to reduce indirect WZ problems to direct WZ problems [Witsenhausen, 80] Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
18
Definitions
r.v. defined on a common probability space, taking values in alphabets i.i.d. drawings of
measurable space. A distortion function is a measurable function . A code consists of two measurable mappings
y Encoder y Decoder Noisy Quantization Reconstruction Observation Index ˆn Encoder Decoder Q Zn X n
Source Data
Xn
p(z n| xn , y ) Yn
q(z n )
ˆn (q, yn ) x
Y n Side Information
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
19
WZ Coding vs. Conditional Coding The indirect WZ RD function is defined as
RATE-DISTORTION FUNCTION Direct Case
Y Also Available at Encoder (Conditional Coding)
Y Available at Decoder Only (WZ Coding)
_
Indirect, Noisy Case
When does
, or more generally, , hold? Closed formula?
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
20
Generalization of the WZ RD Function in the Direct, Quadratic-Gaussian Case Direct case Theorem:
,
,
measurable, and . Suppose that _ , independent from . Then, for all
is arbitrary
, with ,
N ∼ N (0, σ 2 ) Y
µ(y)
X
This slightly relaxes the hypotheses in the literature Direct proof, not using duality arguments Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
21
Generalization of the WZ RD Function in the Indirect, Quadratic-Gaussian Case Theorem: measurable, and
y y
, . ,
Y
. Suppose , independent from , where , measurable, for a.e. ,
_ r.v. satisfying
Then, for all
,
,
Z
µ(y)
f (y)
N ∼ N (0, σ 2 )
α
X
N 0 (E[N 0 | y, z] = 0) Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
22
Modified Distortion Functions Proposition: Let
be measurable. Define . Let be a r.v. in some alphabet . Assume that or , and that there exists a measurable function such that . Then, .
Based on [Witsenhausen, 80] Can be used to reduce indirect WZ problem to direct WZ
problem
becomes direct source data, replacing
Modified distortion function is side-information-dependent Markov conditions correspond to WZ and conditional cases,
respectively Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
23
Overview of the Proofs Indirect RD Function Modified Distortion Function
˜ x, d(z, ˆ y ) = E[d(X, x, ˆ y , z)| y , z] Side-Information-Dependent Distortion Functions of the Form
d(x, x, ˆ y) = (αx − ˆx + f (y))2 Direct RD Function
Conditional Gaussian Maximizes Conditional Differential Entropy Subject to Unconditional Power Constraint Modification of Steps to Prove Shannon Lower Bound Test Channel Technique To Find Non-Distributed RD Function in the Quadratic-Gaussian Case Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
24
Intuitive Example Noisy Observation
Zn
Quantization Index
q (z n )
Q
Reconstruction
ˆn (q, yn ) x
ˆn X
Y n Side Information
discrete random state, uniformly distributed on ,
independent from
RD function
y If
were available at the encoder, encoded, and y If were encoded and decoded ignoring _ , then
could be , and we defined
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
25
Operational RD Performance Operational coding assumes Since
additively separable, according to the high-rate noisy WZ quantization theory in [Rebollo-Monedero, Rane, Girod, 04], for each
y Lattice quantizer followed by SW coder asymptotically optimal y ( normalized moment of inertia) Lloyd algorithm extended for noisy WZ quantization design in
[Rebollo-Monedero, Girod, 05]
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
26
RD Comparison 1/D [dB]
9 8 Z W |Y Z N X
7 6 5 4 3
r sto i D
2
te Ra n tio
on cti n Fu
(R
Noisy WZ Quantization (n=1,2,3)
)
High-Rate Approximation (n=1,2,3)
D
1 0
0
0.5
Slope
1
1.5
R [bit]
Distortion gap w.r.t. RD function Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
27
Conclusions Conditions to ensure that no rate loss incurred in WZ coding
y Jointly Gaussian statistics are not necessary y Relaxed conditions in both the direct and indirect case y Side information arbitrarily distributed, discrete or continuous In the indirect case, condition on the data similar to additive
separability condition in high-rate noisy WZ quantization All conditions determined by conditional joint distribution of
(data, observation) given side information for arbitrarily distributed side information
Rebollo and Girod: A Generalization of the RD Function for WZ Coding of Noisy Sources
28
