Autoencoders

Variational AE, Disentangled VAE

Autoencoder is a type of neural networks that learn efficient representation of raw unsupervised data.


Problem it solves: dimensionality reduction problem

Solution: train encoder and decoder networks to compress input data then restore it with minimum or even no loss of information

Typical autoencoder architecture

Autoencoder use cases

  1. Denoising

Idea: Train autoencoder to denoise the images:

  1. Given dataset, add white noise -> noisy dataset

  2. use noisy data set to encode & decode

  3. compute loss between AE output and initial dataset

Denoising AE architecture

Example of MNIST denoising

2. Anomaly detection

Idea: we pass the input data through pre-trained AE and check the loss.
If the loss is too high than usual treshhold, we consider data as anomaly.

Note: Typically anomaly detection is a part of ML pipeline to remove outliers.

Example 1: We learn to classify cats vs dogs. We are given image of a car.
We will prefer to skip this image instead of training model on it.

3. Neural inpaiting

Idea: similar as Denoising, we corrupt image by croping some regiong, then we train AE to restore the missing part.

4. Vector math of patterns

Latnt space allows us to encode the input into vectors and recognize the patterns.

A well trained AE can manipulate with concepts like "Smile" or "Glasses" and add or substract patterns from image using vector math.

Variational autoencoders (VAE)

Drawback of simple autoencoder:

  • bad representation of latent space

  • bad sampling from latent space

Idea of VAE: instead of mapping input data to a fixed vector, VAE maps input data to a distribution


AE latent space

VAE latent space

Typical VAE archietcture

Note:

  1. Loss now has two parts:

    • Expectation of MSE loss

    • KL divergence (to ensure the distribution is close to normal)

  2. Reparametrization trick to allow backpropagation:

    • we don't learn the noise (no derivative for gaussian noise)

    • we do learn distribution parameters (propagation)

Disentangled variational autoencoders

Idea: different neurons in VAE are uncorrelated, each one represents some particular data feature.

Result: Perturbing one latent variable (while other are fixed) corresponds to an interpretable changes in output

Example: apply DVAE in RL to find sparse rewards by running trainings inside the DVAE-reduced latent space