Definition of Reproducing Kernel Hilbert Space

In this article we will introduce the definition of Reproducing Kernel Hilbert Space (RKHS).

Let $E$ be a non empty set. Let $\mathcal H$ be a vector space of functions defined on $E$ and taking their values in the set $\mathbb C$ of complex numbers. $\mathcal H$ is endowed with the structure of Hilbert space defined by an inner product $\langle \cdot,\cdot\rangle_{\mathcal H}$,

$$
\begin{aligned}
\mathcal H\times \mathcal H &\longrightarrow \mathbb C\\\
(\varphi,\psi)&\longmapsto \langle \varphi,\psi\rangle_{\mathcal H}.
\end{aligned}
$$

Let $\Vert\cdot\Vert_{\mathcal H}$ denote the associated norm,

$$
\forall \varphi\in\mathcal H,\qquad \Vert\varphi\Vert_{\mathcal H}=\langle \varphi,\varphi\rangle_{\mathcal H}^{1/2}.
$$

For any $t\in E$, we will denote by $e_t$ the evaluation functional at the point $t$, i.e. the mapping

$$
\begin{aligned}
e_t:\mathcal H&\longrightarrow \mathbb C\\\
g&\longmapsto e_t(g):=g(t).
\end{aligned}
$$

Definition 1. (Reproducing kernel) A function

$$
\begin{aligned}
K:E\times E&\longrightarrow \mathbb C\\\
(s,t)&\longmapsto K(s,t)
\end{aligned}
$$

is a reproducing kernel of the Hilbert space $\mathcal H$ if and only if

(a) $\forall t\in E,\quad K(\cdot,t)\in\mathcal H$,

(b) $\forall t\in E,\ \forall \varphi\in\mathcal H$,

$$
\langle \varphi,K(\cdot,t)\rangle_{\mathcal H}=\varphi(t).
$$

Note. The property (b) is called the “reproducing property”: the value of $\varphi$ at point $t$ is reproduced by the inner product of $\varphi$ with $K(\cdot,t)$.

Remark: It is clear that $\forall (s,t)\in E\times E$,

$$
K(s,t)=\langle K(\cdot,t),K(\cdot,s)\rangle.
$$

A Hilbert space of complex-valued functions which possesses a reproducing kernel is called a “reproducing kernel Hilbert space (RKHS)”, or a “proper Hilbert space”.

Example 1: Let $e_1,\ldots,e_n$ be an orthonormal basis in $\mathcal H$ and define

$$
K(x,y)=\sum_{i=1}^n e_i(x)\overline{e_i(y)}.
$$

Then for any $y\in E$,

$$
K(\cdot,y)=\sum_{i=1}^n \overline{e_i(y)},e_i(\cdot)\in\mathcal H.
$$

and for any function

$$
\varphi(\cdot)=\sum_{i=1}^n \lambda_i e_i(\cdot)\in\mathcal H
$$

we have $\forall y\in E$,

$$
\langle \varphi,K(\cdot,y)\rangle_{\mathcal H}
=\left\langle \sum_{i=1}^n \lambda_i e_i(\cdot),\sum_{i=1}^n \overline{e_i(y)},e_i(\cdot)\right\rangle_{\mathcal H}
=\sum_{i=1}^n \lambda_i e_i(y)
=\varphi(y).
$$

Hence any finite dimensional Hilbert space of functions has a reproducing kernel.

Example 2 Let $E=\mathbb N$ be the set of positive integers, and let $\mathcal H=\ell^2(E)$ be the set of complex sequences $(x_i)$ such that

$$
\sum_{i\in\mathbb N} |x_i|^2 <\infty.
$$

$\mathcal H$ is a Hilbert space with inner product, for $x=(x_i)$ and $y=(y_i)$.
$$
\langle x,y\rangle_{\ell^2(E)}=\sum_{i\in\mathbb N} x_i\overline{y_i}
$$

Let $K(i,j)=\delta_{ij}$ (the Kronecker symbol). Then,

$$
\forall j\in\mathbb N,\qquad K(\cdot,j)=(0,\ldots,0,1,0,\ldots)\in\mathcal H\quad \text{($1$ at the $j$-th place),}
$$

and

$$
\forall j\in\mathbb N\quad \forall x=(x_i),
\quad
\langle x,K(\cdot,j)\rangle _{\mathcal H}
=\sum _{i\in\mathbb N} x_i\delta _{ij}
=x_j.
$$

Hence $K$ is the reproducing kernel of $\mathcal H$.

Theorem 1. A Hilbert space of complex valued functions on $E$ has a reproducing kernel if and only if all the evaluation functions $e_t$, $t\in E$ are continuous on $\mathcal H$.

Proof : If $\mathcal H$ has a reproducing kernel $K$, then for any $t\in E$, we have for all $\varphi\in\mathcal H$

$$
e_t(\varphi)=\langle \varphi,K(\cdot,t)\rangle_{\mathcal H}.
$$

Thus, $e_t$ is linear and

$$
|e_t(\varphi)|
\le \Vert\varphi\Vert_{\mathcal H}\Vert K(\cdot,t)\Vert_{\mathcal H}
=\Vert\varphi\Vert_{\mathcal H},\langle K(\cdot,t),K(\cdot,t)\rangle_{\mathcal H}^{1/2}
=\Vert\varphi\Vert_{\mathcal H},[K(t,t)]^{1/2}.
$$

Hence $e_t$ is continuous.

Conversely, if $e_t$, $t\in E$ are continuous, by Riesz’s representation theorem, there exists a function $N_t(\cdot)\in\mathcal H$ such that for all $\varphi\in\mathcal H$

$$
\varphi(t)=e_t(\varphi)=\langle \varphi,N_t(\cdot)\rangle_{\mathcal H}
\qquad \text{for all } t\in E.
$$

Hence $K(s,t)=N_t(s)$ is the reproducing kernel of $\mathcal H$. $\square$

Remark: From the proof above, we get the above $e_t$ has norm $\Vert e_t\Vert_{\mathcal H’}=[K(t,t)]^{1/2}$.

Corollary 1 In a RKHS, a sequence converging in the norm sense converges pointwise to the same limit.

Proof : If $\varphi_n \xrightarrow{\Vert\cdot\Vert_{\mathcal H}} \varphi$, then for each $t\in E$,

$$
|\varphi_n(t)-\varphi(t)|
=|e_t(\varphi_n)-e_t(\varphi)|
\le \Vert e_t\Vert,\Vert \varphi_n-\varphi\Vert_{\mathcal H}
\to 0.
$$

Hence $\varphi_n\to\varphi$ pointwise. $\square$

Question: When is a complex-valued function $K$ defined on $E\times E$ a reproducing kernel?

Answer: If and only if $K$ is a positive type function.

Definition 2. (Positive type function). A function $K:E\times E\to\mathbb C$ is called a positive type function (or a positive definite function) if

$$
\forall n\ge 1,\qquad \forall (a_1,\ldots,a_n)\in\mathbb C^n,\qquad \forall (x_1,\ldots,x_n)\in E^n,
$$

we have

$$
\sum_{i=1}^n\sum_{j=1}^n a_i\overline{a_j},K(x_i,x_j)\in \mathbb R^+,
$$

where $\mathbb R^+$ denotes the set of nonnegative real numbers.

It is worth noting that $K$ is a positive type function if and only if the matrix

$$
\bigl(K(x_i,x_j)\bigr)_{1\le i,j\le n}
$$

is positive definite for any choice of $n\in\mathbb N$ and $(x_1,\ldots,x_n)\in E^n$.

Examples.

(1) Any constant non negative function on $E\times E$ is of positive type.

Proof : If $K:E\times E\to\mathbb C$

$$
K(s,t)=c\ge 0
$$

then for all $n\in\mathbb N$, $(a_1,\ldots,a_n)\in\mathbb C^n$, $(x_1,\ldots,x_n)\in E^n$,

$$
\sum_{i=1}^n\sum_{j=1}^n a_i\overline{a_j},K(x_i,x_j)
=c\sum_{i=1}^n\sum_{j=1}^n a_i\overline{a_j}
=c\left|\sum_{i=1}^n a_i\right|^2\in \mathbb R^+.
$$

(2). The delta function.

$$
\begin{aligned}
\delta:E\times E&\longrightarrow \mathbb C\\\
(x,y)&\longmapsto \delta_{xy}
=
\begin{cases}
1,&\text{if }x=y,\
0,&\text{if }x\ne y.
\end{cases}
\end{aligned}
$$

is of positive type.

Proof : Let $n\in\mathbb N$, $(a_1,\ldots,a_n)\in\mathbb C^n$, $(x_1,\ldots,x_n)\in E^n$.

Let $\{\alpha_1,\ldots,\alpha_p\}$ be the set of different values among $x_1,\ldots,x_n$.

We can write

$$
\sum_{i=1}^n\sum_{j=1}^n a_i\overline{a_j},\delta_{x_i x_j}
=
\sum_{i=1}^n\sum_{x_j=x_i} a_i\overline{a_j}
=
\sum_{k=1}^p \sum_{x_i=x_j=\alpha_k} a_i\overline{a_j}
=
\sum_{k=1}^p \left|\sum_{x_j=\alpha_k} a_j\right|^2
\in \mathbb R^+.
$$

(3) The product $\alpha K$ of a positive type function $K$ with a non negative constant $\alpha$ is a positive type function.

Question: How to prove that a given function is of positive type?

Lemma 1. Let $\mathcal H$ be some Hilbert space with inner product $\langle \cdot,\cdot\rangle_{\mathcal H}$ and let $\varphi:E\to\mathcal H$. Then the function

$$
\begin{aligned}
K:E\times E&\longrightarrow \mathbb C\\\
(x,y)&\longmapsto K(x,y)=\langle \varphi(x),\varphi(y)\rangle_{\mathcal H}
\end{aligned}
$$

is of positive type.

Proof : The conclusion follows easily from the following equalities

$$
\sum_{i=1}^n\sum_{j=1}^n a_i\overline{a_j},K(x_i,x_j)
=
\sum_{i=1}^n\sum_{j=1}^n \langle a_i\varphi(x_i),a_j\varphi(x_j)\rangle_{\mathcal H}
=
\left\Vert \sum_{i=1}^n a_i\varphi(x_i)\right\Vert_{\mathcal H}^2
\in \mathbb R^+.\quad \square
$$

Lemma 1 tells us that writing

$$
K(x,y)=\langle \varphi(x),\varphi(y)\rangle_{\mathcal H}
$$

in some space $\mathcal H$ is sufficient to prove positive definiteness of $K$. $\square$

Reference

Berlinet, A., & Thomas-Agnan, C. (2011). Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media.

The cover image of this article was taken at Lake Tekapo, New Zealand.