Autoencoder and GAN

Autoencoder

Several applications:

1. Dimensional reduction
2. Preprocessing for classification
3. Identifying essential elements of the input data and filtering out noise.

Deal with some of these PCA limitations: PCA has learned features that are linear combinations of original features.

VAE: Variational Autoencoder

Data are assumed to be represented by a set of normally-distributed latent factors. Images can be generated by sampling from these distributions.

The main goal of VAEs: generate images using the decoder The secondary goal is to have similar images be close together in latent space

• Penalty for not reconstructing the image currently
• Penalty for generating vectors of parameters $\mu$ and $\sigma$ that are different from 0 and 1.

The reconstruction error of variational autoencoders can be higher because variational autoencoders are designed to maximize the interpretability of the latent space, not to minimize the reconstruction error.

KL Divergence

• KL Divergence formula for normal distribution: $\frac{1}{2}\times (e^{log(\sigma)}-(log(\sigma)+1)+\mu^2)$

• ??? $KL(q(z

  x)

  p(z)) = \frac{1}{2} \sum_{i=1}^N \log \frac{p(z_i)}{q(z_i)}$

• Including KL Divergence is quite empirical.

GAN

Discriminators would judge whether speech is real or fake. Solve discriminators continually improve at distinguishing between real and fake speech. Feeding the gradient of the resulting discriminator concerning the input.

How to Train?

• Generator:
  • Generate a random noise vector
  • Feed the noise vector to the generator
  • Generate an image $X_G$
• Discriminator:
  • Feed the generated image to the discriminator
    • Two loss function
      • Assuming fake $L_0^{X_G} = f(P_{real}^{X_G}, 0)$
      • Assuming real $L_1^{X_G} = f(P_{real'}^{X_G}, 1)$
    • Update the discriminator weights
      • $\frac{\partial L_0^{X_G}}{\partial W} = \frac{\partial L_0^{X_G}}{\partial X_G}$
  • Distinguish whether the image is real or fake
    • $P_{real}^{X_G}$

Additional Issues

• LIME