Lecture 14.2: Kernels continued

Processing math: 100%

Lecture 14.2: Kernels continued

Well-defined kernels

Here are the most common kernels:

Linear: $\mathsf{k}(\mathbf{x}, \mathbf{z})=\mathbf{x}^\top\mathbf{z}$
RBF: $\mathsf{k}(\mathbf{x},\mathbf{z})=e^{- \frac{(\mathbf{x}-\mathbf{z})^2}{\sigma^2}}$
Polynomial: $\mathsf{k}(\mathbf{x}, \mathbf{z})=(1+\mathbf{x}^\top\mathbf{z})^d$

Kernels built by recursively combining one or more of the following rules are called well-defined kernels:

$\mathsf{k}(\mathbf{x}, \mathbf{z})=\mathbf{x}^\top\mathbf{z}\quad\quad\quad(1)$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=c\mathsf{k_1}(\mathbf{x},\mathbf{z})\quad\quad(2)$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=\mathsf{k_1}(\mathbf{x},\mathbf{z})+\mathsf{k_2}(\mathbf{x},\mathbf{z})$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=g(\mathsf{k}(\mathbf{x},\mathbf{z}))$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=\mathsf{k_1}(\mathbf{x},\mathbf{z})\mathsf{k_2}(\mathbf{x},\mathbf{z})$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=f(\mathbf{x})\mathsf{k_1}(\mathbf{x},\mathbf{z})f(\mathbf{z})\quad(3)$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=e^{\mathsf{k_1}(\mathbf{x},\mathbf{z})}\quad\quad(4)$
$\mathsf{k}(\mathbf{x}, \mathbf{z})=\mathbf{x}^\top A \mathbf{z}$

where

$k_1,k_2$ are well-defined kernels,

$c\geq 0$ ,

$g$ is a polynomial function with positive coefficients,

$f$ is any function and

$\mathbf{A}\succeq 0$ is positive semi-definite. Kernel being well-defined is equivalent to the corresponding kernel matrix,

$K$ , being positive semidefinite (not proved here), which is equivalent to any of the following statement:

All eigenvalues of $K$ are non-negative.
$\exists \text{ real matrix } P \text{ s.t. } K=P^\top P.$
$\forall \text{ real vector } \mathbf{x}, \mathbf{x}^\top K \mathbf{x} \ge 0.$

It is trivial to prove that linear kernel and polynomial kernel with integer

$d$ are both well-defined kernel.

The RBF kernel

$k_{RBF}=e^{\frac{-(\mathbf{x}-\mathbf{z})^2}{\sigma^2}}$ is a well-defined kernel matrix.

$\begin{aligned} \mathsf{k}_1(\mathbf{x},\mathbf{z})&=\mathbf{x}^\top \mathbf{z} \qquad \textrm{well defined by rule (1)}\\ \mathsf{k}_2(\mathbf{x},\mathbf{z})&=\frac{2}{\sigma^2}\mathsf{k}_1(\mathbf{x},\mathbf{z})=\frac{2}{\sigma^2}\mathbf{x}^\top \mathbf{z} \qquad \textrm{well defined by rule (2)}\\ \mathsf{k}_3(\mathbf{x},\mathbf{z})&=e^{\mathsf{k}_2(\mathbf{x},\mathbf{z})}=e^{\frac{2\mathbf{x}^\top \mathbf{z}}{\sigma^2}} \qquad \textrm{well defined by rule (4)}\\ \mathsf{k}_{4}(\mathbf{x},\mathbf{z})&=e^{-\frac{\mathbf{x}^\top\mathbf{x}}{\sigma^2}} \mathsf{k}_3(\mathbf{x},\mathbf{z}) e^{-\frac{\mathbf{z}^\top\mathbf{z}}{\sigma^2}} =e^{-\frac{\mathbf{x}^\top\mathbf{x}}{\sigma^2}} e^{\frac{2\mathbf{x}^\top \mathbf{z}}{\sigma^2}} e^{-\frac{\mathbf{z}^\top\mathbf{z}}{\sigma^2}} \qquad \textrm{well defined by rule (3) with $f(\mathbf{x})= e^{-\frac{\mathbf{x}^\top\mathbf{x}}{\sigma^2}}$}\\ &=e^{\frac{-\mathbf{x}^\top\mathbf{x}+2\mathbf{x}^\top\mathbf{z}-\mathbf{z}^\top\mathbf{z}}{\sigma^2}}=e^{\frac{-(\mathbf{x}-\mathbf{z})^2}{\sigma^2}}=\mathsf{k}_{RBF}(\mathbf{x},\mathbf{z}) \end{aligned}$

You can even define kernels of sets, or strings or molecules. The following kernel is defined on two sets,

$\mathsf{k}(S_1,S_2)=e^{|S_1 \cap S_2|}.$ It can be shown that this is a well-defined kernel. We can list out all possible samples

$\Omega$ and arrange them into a sorted list. We define a vector

$\mathbf{x} \in \{0,1\}^{|\Omega|}$ , where each of its element indicates whether a corresponding sample is included in the set. It is easy to prove that

$\mathsf{k}(S_1,S_2)=e^{x_1^\top x_2},$ which is a well-defined kernel by rules (1) and (4).

Kernel Machines

An algorithm can be kernelized in 2 steps:

Rewrite the algorithm and the classifier entirely in terms of inner-product, $\mathbf{x}_1^\top \mathbf{x}_2$ .
Define a kernel function and substitute $\mathsf{k}(\mathbf{x}_i,\mathbf{x}_j)$ for $\mathbf{x}_i^\top \mathbf{x}_j$ .

Quiz 1: How can you kernelize nearest neighbors (with Euclidean distances)?

$D=\{(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_n,y_n)\}.$
Observations: Nearest neighbor under squared Euclidean distance is the same as nearest neighbor under Euclidean distance. Therefore, out of the original version,

$h(\mathbf{x})= y_j \quad \text{where} \quad j=argmin_{(\vec{x_j},y_j) \in D}(x-x_j)^2$ we can derive the kernel version,

$h(\mathbf{x})= y_j \quad \text{where} \quad j=argmin_{(\vec{x_j},y_j) \in D} (\mathsf{k}(\vec{\mathbf{x}}, \vec{\mathbf{x}})-2\mathsf{k}(\vec{\mathbf{x}}, \vec{\mathbf{x}_j})+\mathsf{k}(\vec{\mathbf{x}_j}, \vec{\mathbf{x}_j}))$ Actually, the kernel nearest neighbor is rarely used in practice, because it brings little improvement since the original

$k$ -NN is already highly non-linear. \subsection{Kernel Regression} Kernel regression is kernelized Ordinary Least Squares Regression (OLS). Vanilla OLS minimizes the following squared loss regression loss function,

$\begin{equation} \min_\mathbf{w} \sum_{i=1}^{n} (\mathbf{w}^\top \mathbf{x}_i -y_i)^2, \end{equation}$ to find the hyper-plane

$\mathbf{w}$ . The prediction at a test-point is simply

$h(\mathbf{x})=\mathbf{w}^\top \mathbf{x}$ . % If we let

$\mathbf{X}=[\mathbf{x}_1,\ldots,\mathbf{x}_n]$ and

$\mathbf{y}=[y_1,\ldots,y_n]^\top$ , the solution of OLS can be written in closed form:

$\begin{equation} \mathbf{w}=(\mathbf{X}\mathbf{X}^\top)^{-1} \mathbf{X} \mathbf{y}\qquad(5)%\label{eq:kernel:OLS} \end{equation}$ \subsubsection{kernelization} We begin by expressing the solution

$\mathbf{w}$ as a linear combination of the training inputs

$\begin{equation} \mathbf{w}=\sum_{i=1}^{n} \alpha_i\mathbf{x}_i=\mathbf{X}\vec{\alpha}. \end{equation}$ You can verify that such a vector

$\vec \alpha$ must always exist by observing the gradient updates that occur if (5) is minimized with gradient descent and the initial vector is set to

$\mathbf{w}_0=\vec 0$ (because the squared loss is convex the solution is independent of its initialization.) We now kernelize the algorithm by substituting

$k(\mathbf{x},\mathbf{z})$ for any inner-product

$\mathbf{x}^\top \mathbf{z}$ . It follows that the prediction at a test point becomes

$\begin{equation} h(\mathbf{x})=\mathbf{w}^\top \mathbf{x} = \sum_{i=1}^{n} \alpha_i \mathbf{x}_i^\top\mathbf{x} =\sum_{i=1}^{n} \alpha_i k(\mathbf{x}_i,\mathbf{x})= K_{X:x}\vec{\alpha}. \end{equation}$ It remains to show that we can also solve for the values of

$\alpha$ in closed form. As it turns out, this is straight-forward.

Kernelized ordinary least squares has the solution

$\vec{\alpha}={\mathbf{K}}^{-1} \mathbf{y}$ .

$\begin{aligned} \mathbf{X}\vec{\alpha}&=\mathbf{w}= (\mathbf{X}\mathbf{X}^\top)^{-1} \mathbf{X} \mathbf{y} \textrm{ \ \ | multiply from left by $(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top$} \\ (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{X}\vec{\alpha} &= (\mathbf{X}^\top\mathbf{X})^{-1}\underbrace{\mathbf{X}^\top(\mathbf{X}\mathbf{X}^\top)^{-1} \mathbf{X}}_{=\mathbf{I}} \mathbf{y}\\ \vec{\alpha}&= (\mathbf{X} ^\top \mathbf{X})^{-1} \mathbf{y} | \textrm{ substitute $\mathbf{K}=\mathbf{X}^\top \mathbf{X}$}\\ \vec{\alpha}&= \mathbf{K}^{-1} \mathbf{y} \\ \end{aligned}$

Kernel regression can be extended to the kernelized version of ridge regression. The solution then becomes

$\begin{equation} \vec{\alpha}=(\mathbf{K}+\sigma^2\mathbf{I})\mathbf{y}. \end{equation}$ In practice a small value of

$\sigma^2>0$ increases stability, especially if

$\mathbf{K}$ is not invertible.

Kernel SVM

The original SVM is a quadratic programming problem

$\begin{aligned} &\min_{\mathbf{w},b}\mathbf{w}^\top\mathbf{w}+C \sum_{i=1}^{n} \xi_i \\ \text{s.t. }\forall i, &\quad y_i(\mathbf{w}^\top\mathbf{x}_i +b) \geq 1 - \xi_i\\ &\quad \xi_i \geq 0 \end{aligned}$ It has a dual form

$\begin{aligned} &\min_{\alpha_1,\cdots,\alpha_n}\frac{1}{2} \sum_{i,j}\alpha_i \alpha_j y_i y_j K_{ij} - \sum_{i=1}^{n}\alpha_i \\ \text{s.t.} &\quad 0 \leq \alpha_i \leq C\\ &\quad \sum_{i=1}^{n} \alpha_i y_i = 0 \end{aligned}$ Easy to show

$0 \leq \alpha_i \leftrightarrow y_i(\mathbf{w}^\top \phi(x_i)+b)=1$ Almost all

$\alpha_i = 0$ (called sparse), support vector

$\alpha_i \neq 0$ .