homeomorphic vae

2 min read

Explorations in Homeomorphic Variational Auto-Encoding


Revisiting VAE

A variational auto-encoder is a generative model that learns a joint distribution over input data and some latent random variable using variational inference techniques.

$$ p(x, z) = p(x|z)p(z) $$

Say we now have some data $X = \{ x_1 \ldots x_N \}$. Our objective is to learn a set of latent random variables that maximize our ability to reconstruct this set from our latent $\{z_i\}$

$$ \frac{1}{N} \log p(X) = \frac{1}{N} \sum_{i=1}^{N} \log \int p(x_i, z_i) dz $$

However, our computing closed form $p(X)$ is often intractable when each latent $z$ is parameterized by a neural network, so we need to invoke variational inference to approximate our joint $p(x, z)$. Detailed derivation of our lower bound in a future note.

$$ \log p(X) \ge E_{q(z)}[\log p(x|z)] - KL(q(z) \mid\mid p(z)) $$


$$ \log p(X) = \log \int p(x,z) dz \ge E_{q(z)}[log p(x|z)] - KL(q(z) \mid\mid p(z)) $$

Using a reparameterization trick to enable tractable backpropogation, we arrive at:

$$ \log p(X) \ge E_{q(z|x)}[log p(x|z)] - KL(q(z|x) \mid\mid p(z)) $$

Lie Groups + Algebras

A group is a set that is equipped with a binary operation that satisfies properties, namely:

  • associativity over the operation
  • the existence of an identity element invariant to the operation
  • an inverse for each element that will recover the identity

A Lie group $G$ is equipped with a product that satisfies these properties. However, these groups are also smooth manifolds, which means a derivative is globally defined, and group elements can be described continuously with respect to a set of parameters.

There also exists a lie algebra, which is the vector space tangent to the identity element of $G$ and is denoted by $\mathfrak{g}$.

The lie group is connected to the lie algebra through a exponential map:

$$ \exp: \mathfrak{g} \mapsto G $$

The mapping is surjective onto our group $G$.


This notation translates to special orthogonal lie group of 3-dim rotations.

$$ SO(3) := \{ R \in GL(\mathbb{R^3}) : R^T R = \mathbf{I} \wedge \det(A) = 1 \} $$

Where $GL$ is the general linear group using the aforementioned definition of a group in the mathematical sense. Here group members are clearly both square and invertible matrices.

Reparameterizing Lie Groups

Pushforward Measure Proof

$$ \ldots $$