Summary of Advanced Deep Learning Techniques

Advanced Deep Learning Techniques: VGAE & GAE Explained

SummaryKnowledge testFlashcardsPodcastMindmap

Introduction

Variational Graph Autoencoders (VGAE) are models for learning continuous latent representations of nodes in graphs in an unsupervised way. They combine ideas from probabilistic latent-variable models and graph neural networks to encode node information and reconstruct graph structure. This material explains the components, training objective, and practical applications in an accessible way.

Definition: A VGAE is a latent-variable model for graph-structured data that uses an encoder (a graph convolutional network) to approximate a posterior over node embeddings and a decoder (typically an inner product) to reconstruct edges.

Overview of the model components

Graphs and inputs

  • A graph is given as $G=(V,E)$ with $N=|V|$ nodes.
  • The adjacency matrix is $A\in{0,1}^{N\times N}$; we assume diagonal elements set to 1 (self-connections).
  • The degree matrix is $D$ with $D_{ii}=\sum_j A_{ij}$.
  • Node features are in $X\in\mathbb{R}^{N\times D}$.
  • Latent node embeddings are $Z\in\mathbb{R}^{N\times F}$ with rows $z_i$.

Definition: The adjacency matrix $A$ encodes which node pairs have edges, with $A_{ij}=1$ if nodes $i$ and $j$ are connected and $0$ otherwise.

Encoder: Graph Convolutional Network (GCN)

  • The encoder approximates the posterior $q(Z\mid X,A)$ with a factorized Gaussian: $q(Z\mid X,A)=\prod_{i=1}^N q(z_i\mid X,A)$ and $q(z_i\mid X,A)=\mathcal{N}(z_i\mid\mu_i,\operatorname{diag}(\sigma_i^2))$.
  • Means and log-variances are produced by GCNs: $\mu=\mathrm{GCN}\mu(X,A)$ and $\log\sigma=\mathrm{GCN}\sigma(X,A)$.
  • A two-layer GCN has the form

$$\mathrm{GCN}(X,A)=\widetilde{A},\mathrm{ReLU}(\widetilde{A} X W_0) W_1$$

with learnable weight matrices $W_0,W_1$ and $\widetilde{A}=D^{-1/2} A D^{-1/2}$ (symmetric normalization).

Definition: The normalized adjacency matrix $\widetilde{A}=D^{-1/2} A D^{-1/2}$ scales neighbor aggregation to avoid numerical issues and balance contributions from nodes with different degrees.

Decoder: Inner product for edge probabilities

  • The generative model reconstructs edges from latent embeddings via an inner product decoder:

$$p(A\mid Z)=\prod_{i=1}^N\prod_{j=1}^N p(A_{ij}\mid z_i,z_j),$$

with

$$p(A_{ij}=1\mid z_i,z_j)=\sigma\left(z_i^{\top} z_j\right)$$

where $\sigma(\cdot)$ is the logistic sigmoid function.

Definition: The inner product decoder scores the likelihood of an edge between nodes $i$ and $j$ by the sigmoid of the dot product of their latent vectors.

Training objective: Variational lower bound

  • The model is trained by maximizing the evidence lower bound (ELBO):

$$\mathcal{L}=\mathbb{E}_{q(Z\mid X,A)}\left[\log p(A\mid Z)\right]-\mathrm{KL}\left[q(Z\mid X,A),||,p(Z)\right].$$

  • Here $p(Z)$ is a prior (commonly $\mathcal{N}(0,I)$).
  • The expectation is estimated via sampling from the approximate posterior using the reparameterization trick.

Definition: The ELBO balances reconstruction accuracy (first term) with how close the approximate posterior is to the prior (second term).

Step-by-step: From inputs to reconstructed edges

  1. Preprocess graph: set $A_{ii}=1$ and compute $\widetilde{A}=D^{-1/2} A D^{-1/2}$.
  2. Encode: pass $X$ and $\widetilde{A}$ through GCNs to obtain $\mu$ and $\log\sigma$.
  3. Sample embeddings: $z_i=\mu_i+\sigma_i \odot \varepsilon_i$ with $\varepsilon_i\sim\mathcal{N}(0,I)$.
  4. Decode: compute edge probabilities $\hat{p}_{ij}=\sigma\left(z_i^{\top} z_j\right)$.
  5. Optimize: maximize $\mathcal{L}$ w.r.t. encoder parameters using stochastic gradient descent.

Practical examples and applications

  • Link prediction: predict missing or future edges in social networks, citation networks, or knowledge graphs by ranking $\hat{p}_{ij}$.
  • Node representation learning: obtain embeddings $z_i$ for downstream tasks such as node classification or clustering.
  • Recommendation systems: model user-item bipartite graphs to suggest likely interactions.
💡 Did you know?Fun fact: The latent space learned by these models
Sign up for the full summary
FlashcardsKnowledge testSummaryPodcastMindmap
Start for free

Already have an account? Sign in

Graph Autoencoder Basics

Klíčová slova: Variational Graph Autoencoder (VGAE), Variational Graph Autoencoders, Graph and Network Machine Learning

Klíčové pojmy: Graph defined by $G=(V,E)$ with adjacency $A$ and node features $X$, Encoder approximates $q(Z\mid X,A)=\prod_i\mathcal{N}(z_i\mid\mu_i,\mathrm{diag}(\sigma_i^2))$, GCN encoder: $\mathrm{GCN}(X,A)=\widetilde{A}\,\mathrm{ReLU}(\widetilde{A} X W_0) W_1$, Normalized adjacency $\widetilde{A}=D^{-1/2} A D^{-1/2}$, Decoder: $p(A_{ij}=1\mid z_i,z_j)=\sigma\left(z_i^{\top} z_j\right)$, Training objective: ELBO $\mathcal{L}=\mathbb{E}_{q}[\log p(A\mid Z)]-\mathrm{KL}[q||p]$, Use reparameterization trick to sample $z_i=\mu_i+\sigma_i\odot\varepsilon_i$, Incorporate node features $X$ to improve link prediction, Normalize adjacency and use dropout to stabilize training, Evaluate link prediction with proper train/validation/test splits and negative sampling

## Introduction Variational Graph Autoencoders (VGAE) are models for learning continuous latent representations of nodes in graphs in an unsupervised way. They combine ideas from probabilistic latent-variable models and graph neural networks to encode node information and reconstruct graph structure. This material explains the components, training objective, and practical applications in an accessible way. > Definition: A VGAE is a latent-variable model for graph-structured data that uses an encoder (a graph convolutional network) to approximate a posterior over node embeddings and a decoder (typically an inner product) to reconstruct edges. ## Overview of the model components ### Graphs and inputs - A graph is given as $G=(V,E)$ with $N=|V|$ nodes. - The adjacency matrix is $A\in\{0,1\}^{N\times N}$; we assume diagonal elements set to 1 (self-connections). - The degree matrix is $D$ with $D_{ii}=\sum_j A_{ij}$. - Node features are in $X\in\mathbb{R}^{N\times D}$. - Latent node embeddings are $Z\in\mathbb{R}^{N\times F}$ with rows $z_i$. > Definition: The adjacency matrix $A$ encodes which node pairs have edges, with $A_{ij}=1$ if nodes $i$ and $j$ are connected and $0$ otherwise. ### Encoder: Graph Convolutional Network (GCN) - The encoder approximates the posterior $q(Z\mid X,A)$ with a factorized Gaussian: $q(Z\mid X,A)=\prod_{i=1}^N q(z_i\mid X,A)$ and $q(z_i\mid X,A)=\mathcal{N}(z_i\mid\mu_i,\operatorname{diag}(\sigma_i^2))$. - Means and log-variances are produced by GCNs: $\mu=\mathrm{GCN}_\mu(X,A)$ and $\log\sigma=\mathrm{GCN}_\sigma(X,A)$. - A two-layer GCN has the form $$\mathrm{GCN}(X,A)=\widetilde{A}\,\mathrm{ReLU}(\widetilde{A} X W_0) W_1$$ with learnable weight matrices $W_0,W_1$ and $\widetilde{A}=D^{-1/2} A D^{-1/2}$ (symmetric normalization). > Definition: The normalized adjacency matrix $\widetilde{A}=D^{-1/2} A D^{-1/2}$ scales neighbor aggregation to avoid numerical issues and balance contributions from nodes with different degrees. ### Decoder: Inner product for edge probabilities - The generative model reconstructs edges from latent embeddings via an inner product decoder: $$p(A\mid Z)=\prod_{i=1}^N\prod_{j=1}^N p(A_{ij}\mid z_i,z_j),$$ with $$p(A_{ij}=1\mid z_i,z_j)=\sigma\left(z_i^{\top} z_j\right)$$ where $\sigma(\cdot)$ is the logistic sigmoid function. > Definition: The inner product decoder scores the likelihood of an edge between nodes $i$ and $j$ by the sigmoid of the dot product of their latent vectors. ### Training objective: Variational lower bound - The model is trained by maximizing the evidence lower bound (ELBO): $$\mathcal{L}=\mathbb{E}_{q(Z\mid X,A)}\left[\log p(A\mid Z)\right]-\mathrm{KL}\left[q(Z\mid X,A)\,||\,p(Z)\right].$$ - Here $p(Z)$ is a prior (commonly $\mathcal{N}(0,I)$). - The expectation is estimated via sampling from the approximate posterior using the reparameterization trick. > Definition: The ELBO balances reconstruction accuracy (first term) with how close the approximate posterior is to the prior (second term). ## Step-by-step: From inputs to reconstructed edges 1. Preprocess graph: set $A_{ii}=1$ and compute $\widetilde{A}=D^{-1/2} A D^{-1/2}$. 2. Encode: pass $X$ and $\widetilde{A}$ through GCNs to obtain $\mu$ and $\log\sigma$. 3. Sample embeddings: $z_i=\mu_i+\sigma_i \odot \varepsilon_i$ with $\varepsilon_i\sim\mathcal{N}(0,I)$. 4. Decode: compute edge probabilities $\hat{p}_{ij}=\sigma\left(z_i^{\top} z_j\right)$. 5. Optimize: maximize $\mathcal{L}$ w.r.t. encoder parameters using stochastic gradient descent. ## Practical examples and applications - Link prediction: predict missing or future edges in social networks, citation networks, or knowledge graphs by ranking $\hat{p}_{ij}$. - Node representation learning: obtain embeddings $z_i$ for downstream tasks such as node classification or clustering. - Recommendation systems: model user-item bipartite graphs to suggest likely interactions. Fun fact: The latent space learned by these models