PixelGAN Autoencoders Alireza Makhzani, Brendan Frey Machine learning Group University of Toronto

CIFAR Deep Learning Summer School Montreal, Canada

June 29th, 2017

Alireza Makhzani

PixelGAN Autoencoders

1 / 27

Outline

1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders

2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani

PixelGAN Autoencoders

2 / 27

PixelCNNs

✦ ✦ ✦ ✦ ✦

Learn the image statistics directly at the pixel level. Good at modelling low-level pixel statistics. Samples lack global structure. Lacking latent representation. Conditional PixelCNNs can learn conditional densities.

van den Oord et al., 2016 Alireza Makhzani

PixelGAN Autoencoders

3 / 27

Variational Autoencoders

log p(x) >

Eq(z|x) [ log(p(x|z)]

KL(q(z|x)kp(z))

✦ Good at capturing the global structure, but samples are blurry. ✦ Learn hierarchical representations useful for down-stream tasks. ✦ Attempts at combining PixelCNN with VAEs: • PixelVAE (Gulrajani et al., 2016) • VLAE (Chen et al., 2017)

Kingma et al., 2013 Alireza Makhzani

PixelGAN Autoencoders

4 / 27

Adversarial Autoencoders A

C

B

D

Code Space of MNIST:

Makhzani et al., 2015 Alireza Makhzani

Gaussian Prior

Mixture of Gaussians

PixelGAN Autoencoders

5 / 27

Outline

1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders

2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani

PixelGAN Autoencoders

6 / 27

Limitations of Variational/Adversarial Autoencoders

✦ All the image statistics are captured by the single latent vector.

VAE

p(z)

Latent Variable

label, style global and local

p(x|z)

Deterministic (factorized Gaussians)

None

z

x

Alireza Makhzani

PixelGAN Autoencoders

7 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder.

p(z)

Latent Variable

p(x|z)

PixelCNN

z

x

Alireza Makhzani

PixelGAN Autoencoders

8 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder.

p(z)

Latent Variable

p(x|z)

PixelCNN

GAN

z

x

Alireza Makhzani

PixelGAN Autoencoders

9 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder.

p(z)

GAN

z

p(x|z)

PixelGAN (Gaussian)

PixelGAN (Categorical)

Latent Variable

Global (low-frequency)

Discrete (label)

PixelCNN

Local (high-frequency)

Continuous (Style)

x

Alireza Makhzani

PixelGAN Autoencoders

10 / 27

PixelGAN Autoencoders ✦ The image statistics are captured jointly by the latent vector and the autoregressive decoder.

p(z)

GAN

z

p(x|z) x

Alireza Makhzani

PixelGAN (Gaussian)

PixelGAN (Categorical)

Latent Variable

Global (low-frequency)

Discrete (label)

PixelCNN

Local (high-frequency)

Continuous (Style)

Semi-supervised Learning

PixelGAN Autoencoders

11 / 27

PixelGAN Autoencoders

Cost function of PixelGAN = Reconstruction + Adversarial Cost Alireza Makhzani

PixelGAN Autoencoders

12 / 27

Outline

1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders

2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors Clustering Semi-supervised Learning Alireza Makhzani

PixelGAN Autoencoders

13 / 27

Global vs. Local Decomposition

Alireza Makhzani

PixelGAN Autoencoders

14 / 27

Code Space

Code Space of MNIST:

Alireza Makhzani

PixelGAN Autoencoders

15 / 27

Outline

1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders

2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani

PixelGAN Autoencoders

16 / 27

PixelGAN Autoencoders with Categorical Priors

Alireza Makhzani

PixelGAN Autoencoders

17 / 27

Discrete vs. Continuous Decomposition (Clustering)

Alireza Makhzani

PixelGAN Autoencoders

18 / 27

Discrete vs. Continuous Decomposition (Clustering)

0.3% Error rate

Alireza Makhzani

PixelGAN Autoencoders

19 / 27

Unsupervised Clustering

Alireza Makhzani

PixelGAN Autoencoders

20 / 27

Unsupervised Clustering

Alireza Makhzani

PixelGAN Autoencoders

21 / 27

Unsupervised Clustering

5% Error rate

Alireza Makhzani

PixelGAN Autoencoders

22 / 27

Outline

1. Background • PixelCNNs • Variational Autoencoders • Adversarial Autoencoders

2. PixelGAN Autoencoders • Gaussian Priors • Categorical Priors ✦ Clustering ✦ Semi-supervised Learning Alireza Makhzani

PixelGAN Autoencoders

23 / 27

Semi-supervised Learning

Semi-supervised Learning

Alireza Makhzani

PixelGAN Autoencoders

24 / 27

Semi-supervised Learning

Semi-supervised Learning

Alireza Makhzani

PixelGAN Autoencoders

25 / 27

Semi-supervised Classification

Alireza Makhzani

PixelGAN Autoencoders

26 / 27

Thank you!

Alireza Makhzani

PixelGAN Autoencoders

27 / 27