An Example of Gamma Convergence

Let $\mathcal{P}_2(\mathbb{R}^d)$ denote the space of probability measures on $\mathbb{R}^d$ with finite second moment and $F: \mathcal{P}_2(\mathbb{R}^d)\to \mathbb{R}$ be the potential functional. We are interested in the following optimization problem

$$
\min_{m\in \mathcal{P}_2(\mathbb{R}^d)} F(m).
$$

But instead, we first study the regularized version, namely, the minimization of the free energy functional
$$
\min_{m\in \mathcal{P}_2(\mathbb{R}^d)} V^{\sigma}(m):= F(m)+\frac{\sigma^2}{2}H(m),
$$

where $H: \mathcal{P}(\mathbb{R}^d)\to [0,+\infty]$ is the relative entropy (Kullback-Leibler divergence) with respect to a given Gibbs measure on $\mathbb{R}^d$, namely,
$$
H(m):= \int_{\mathbb{R}^d} m(x)\log\left(\frac{m(x)}{g(x)}\right)\ dx,
$$
where
$$
g(x)=e^{-U(x)} \text{ with } U \text{ s.t. } \int_{\mathbb{R}^d} e^{-U(x)}dx=1,
$$
is the density of the Gibbs measure and the function $U$ satisfies the following conditions.

Assumption. The function $U:\mathbb{R}^d \to \mathbb{R}$ belongs to $C^\infty$. Further,

(i) there exist constants $C_U>0$ and $C’_U\in\mathbb{R}$ such that
$$
\nabla U(x)\cdot x \ge C_U |x|^2 + C’_U \quad \text{for all } x\in\mathbb{R}^d.
$$
(ii) $\nabla U$ is Lipschitz continuous.

Immediately, we obtain that there exist $0\le C’ \le C$ such that for all $x\in\mathbb{R}^d$
$$
C’|x|^2 - C \le U(x) \le C(1+|x|^2), \qquad |\Delta U(x)| \le C.
$$

Now we will show that the original minimization and the regularized one is connected through the following $\Gamma$-convergence result.

Theorem. Let $\mathcal{P}_2^a(\mathbb{R}^d)$ denote the space of probability measures on $\mathbb{R}^d$ with finite second moment and absolutely continuous with respect to Lebesgue measure. Assume that $F$ is continuous in the topology of weak convergence on $\mathcal{P}_2^a(\mathbb{R}^d)$. Then the sequence of functionals
$$
V^\sigma = F + \frac{\sigma^2}{2} H \xrightarrow{\Gamma} F \quad\text{as } \sigma \downarrow 0\ \text{ on } \mathcal{P}_2^a(\mathbb{R}^d).
$$
In particular, given the minimizer $m^{*,\sigma}$ of $V^\sigma$, we have

$$
\limsup_{\sigma\to 0} F\left(m^{*,\sigma}\right)
=
\inf_{m\in \mathcal{P}_2^a(\mathbb{R}^d)} F(m).
$$

It is a classic property of $\Gamma$-convergence that every cluster point of
$$
\left(\arg\min_m V^\sigma(m)\right)_\sigma
$$
is a minimizer of $F$.

Proof: Let $(\sigma_n)_{n\in\mathbb{N}}$ be a positive sequence decreasing to $0$. By the definition of $\Gamma$-convergence, we first show the liminf inequality. Since $H(m)\ge 0$, we have for all $m_n \to m$,

$$
V^{\sigma_n}(m_n)=F(m_n)+\frac{\sigma_n^2}{2} H(m_n)\ge F(m_n).
$$

Hence, by the continuity of $F$, we have

$$
\liminf_{n\to+\infty} V^{\sigma_n}(m_n) \ge \lim_{n\to+\infty} F(m_n) = F(m).
$$

Next, we show the limsup inequality. Let $f$ be the heat kernel, namely
$$
f(x) = \frac{1}{(2\pi)^{d/2}} e^{-|x|^2/2}
$$
and
$$
f_n(x)=\frac{1}{\sigma_n^d}f\left(\frac{x}{\sigma_n}\right).
$$
Given $m\in\mathcal{P}_2^a(\mathbb{R}^d)$, consider sequence of measures $m_n$ with density $m_n=m\ast f_n$ (with a slight abuse of notation), and we need to show that $m_n$ is just the recovery sequence we want.

$$
H(m\ast f_n)=\int_{\mathbb{R}^d} (m\ast f_n)(x)\log\left(\frac{(m\ast f_n)(x)}{g(x)}\right)\ dx= \int_{\mathbb{R}^d}h((m\ast f_n)(x))\ dx- \int_{\mathbb{R}^d} (m\ast f_n)(x)\log g(x)\ dx,
$$

where $h(x):=x\log(x)$ is also convex. Then it follows from Jensen’s inequality that

$$
\begin{aligned}
\int_{\mathbb{R}^d} h(m\ast f_n),dx&=\int_{\mathbb{R}^d} h\left(\int_{\mathbb{R}^d} f_n(x-y)\ m(dy)\right)\ dx\\\
&\le
\int_{\mathbb{R}^d}\int_{\mathbb{R}^d} h\bigl(f_n(x-y)\bigr)\ m(dy)\ dx\\\
&=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d} h\bigl(f_n(x-y)\bigr)\ dx \ m(dy)\\\
&=
\int_{\mathbb{R}^d} h\bigl(f_n(x)\bigr)\ dx\\\
&=\int_{\mathbb{R}^d} f_n(x)\log f_n(x) \ dx\\\
&=\int_{\mathbb{R}^d}\frac{1}{\sigma_n^d}f\left(\frac{x}{\sigma_n}\right)\log\left( \frac{1}{\sigma_n^d}f\left(\frac{x}{\sigma_n}\right)\right)\ dx\\\
&=\int_{\mathbb{R}^d}\frac{1}{\sigma_n^d}f\left(\frac{x}{\sigma_n}\right)\log f\left(\frac{x}{\sigma_n}\right)\ dx- \int_{\mathbb{R}^d}\frac{1}{\sigma_n^d}f\left(\frac{x}{\sigma_n}\right)\ dx\cdot d\log(\sigma_n)\\\
&=
\int_{\mathbb{R}^d} h\bigl(f(x)\bigr)\ dx - d\log(\sigma_n),
\end{aligned}
$$

Moreover,

$$
\begin{aligned}
\int_{\mathbb{R}^d} (m\ast f_n)(x)\log g(x)\ dx&=\int_{\mathbb{R}^d} \int_{\mathbb{R}^d} f_n(x-y)\ m(dy)\log g(x)\ dx\\\
&=-\int_{\mathbb{R}^d} \int_{\mathbb{R}^d} f_n(x-y)\ U(x)\ dx\ m(dy)\\\
&= -\int_{\mathbb{R}^d} \int_{\mathbb{R}^d} f_n(x)\ U(x+y)\ dx\ m(dy)\\\
&\ge -C_1 \int_{\mathbb{R}^d} \int_{\mathbb{R}^d} f_n(x)\ (1+|x+y|^2)\ dx\ m(dy)\\\
&\ge -C_1\int_{\mathbb{R}^d}\left(1+2\int_{\mathbb{R}^d} |x|^2 f_n(x),dx + 2|y|^2 \right)\ m(dy)\\\
&\ge -C \left(1+\int_{\mathbb{R}^d} |y|^2\ m(dy)\right)
\end{aligned}
$$
Therefore

$$
\limsup_{n\to+\infty} V^{\sigma_n}(m\ast f_n)
\le
F(m)+\limsup_{n\to+\infty}\frac{\sigma_n^2}{2}
\left\{
\int_{\mathbb{R}^d} h(m\ast f_n),dx
-
\int_{\mathbb{R}^d} (m\ast f_n)\log(g),dx
\right\}
\le F(m)+\lim_{n\to+\infty}\frac{\sigma_n^2}{2}\int_{\mathbb{R}^d} h\bigl(f(x)\bigr)\ dx- \lim_{n\to+\infty}\frac{\sigma_n^2}{2}d\log(\sigma_n)+ \lim_{n\to+\infty}\frac{\sigma_n^2}{2} C \left(1+\int_{\mathbb{R}^d} |y|^2\ m(dy)\right)=
F(m).
$$
Hence

$$
V^\sigma = F + \frac{\sigma^2}{2} H \xrightarrow{\Gamma} F \quad\text{as } \sigma \downarrow 0\ \text{ on } \mathcal{P}_2^a(\mathbb{R}^d).
$$

In particular, given a minimizer $m^{\star,\sigma}$ of $V^{\sigma}$, we have

$$
\limsup_{n\to+\infty} F(m^{\star,\sigma_n}) \le \limsup_{n\to+\infty} V^{\sigma_n}(m^{\star,\sigma})\le \limsup_{n\to+\infty} V^{\sigma_n}(m\ast f_n) \le F(m),\quad \text{for all } m\in\mathcal{P}_2^a(\mathbb{R}^d).
$$

Hence,
$$
\limsup_{\sigma\to 0} F\left(m^{*,\sigma}\right)
=
\inf_{m\in \mathcal{P}_2^a(\mathbb{R}^d)} F(m).
$$

Reference

Kaitong Hu. Zhenjie Ren. David Šiška. Łukasz Szpruch. “Mean-field Langevin dynamics and energy landscape of neural networks.” Ann. Inst. H. Poincaré Probab. Statist. 57 (4) 2043 - 2065, November 2021. https://doi.org/10.1214/20-AIHP1140

The cover image in this article was taken at Milford Sound, New Zealand.