Language Reference

CALL KPCATRAIN (eigenVal, eigenVec, centroids, rowMeans, scoreTrain, data, <, opt, optCluster> ) ;

This subroutine is supported by the IML procedure and the iml action.

Kernel principal component analysis (kernel PCA) is a nonlinear form of traditional PCA (Schölkopf, Smola, and Müller 1998). It uses the same basic idea as traditional PCA; that is, it seeks to project the set of data onto a low-dimensional subspace that captures the highest possible amount of variance in the data. Whereas traditional PCA performs a linear projection of the data onto a subset of the original space, kernel PCA instead uses a mapping function to embed the data onto a high-dimensional space referred to as by a linear dimensionality reduction through the "kernel trick" in that space. Different kernels correspond to different mapping functions. In this way, you can find a nonlinear subspace that contains the data. The applications of kernel PCA include nonlinear dimensionality reduction, nonlinear data classification, kernel principal component regression, image denoising, and novelty detection.

The KPCATRAIN subroutine calculates the projections of training data onto the principal components in the space ; it also calculates the eigenvector matrix and eigenvalue vector of the kernel matrix. By default, the KPCATRAIN subroutine applies k-means clustering to select representative points (cluster centroids) in the input data matrix and uses a low-rank approximation method for the kernel matrix construction. The low-rank approximation method greatly reduces the complexity of the eigenvector decomposition of the kernel matrix in terms of both computation and memory, and it achieves a performance that is comparable to the exact method.

The input arguments to the KPCATRAIN subroutine are as follows:

The KPCATRAIN subroutine returns the following values:

The KPCATRAIN call implements an eigenvector decomposition on the centered kernel matrix. Currently, this call supports three types of kernel:

In the exact method, the whole kernel matrix is constructed. The centered version can be calculated as , where is an matrix with all elements equal to . Then eigendecomposition is performed on . The normalized version of U with respect to space is , which is returned in eigenVec.

The low-rank approximation algorithm stems from the Nyström method (Baker 1977). The Nyström method can be treated as choosing a subset of c columns, , from the kernel matrix and then reconstructing the complete kernel matrix by , where is the intersection of the selected rows and columns of . The sampling scheme is important to the performance of Nyström method. In (Zhang, Tsang, and Kwok 2008), an error analysis is conducted and the resulting error bound suggests using k-means clustering as the sampling scheme. The greater the number of clusters chosen in k-means clustering, the more accurate the reconstruction. If the number of clusters equals the number of observations, the reconstruction will be exact. Low-rank approximation reduces the memory complexity of kernel PCA from to and reduces computation complexity from to .

The computational details of a low-rank approximation are as follows:

After the eigenvector matrix U and diagonal eigenvalue matrix are obtained, the projection of the training data onto the first q principal components can be calculated as .

Since kernel PCA is a nonlinear generalization of PCA, it can perform traditional PCA if you use a linear kernel and the exact method. For example, suppose you are interested in reproducing the principal component analysis from a call to the PRINCOMP procedure. The following call uses the Sashelp.Iris data set, which has four variables:

proc princomp data=sashelp.iris out=PCComponents cov n=3
              plots(only)=score(ncomp=2);
run;

PROC PRINCOMP creates an output data set that contains the scored data. The scores are the projection of the data onto the first three eigenvectors of the covariance matrix. The following KPCATRAIN call performs an equivalent analysis. The opt vector requests that the analysis use a linear kernel and an exact method and project the data onto three principal components. The SCORE matrix returns the principal component scores. The first two scores are plotted against each other in Figure 210.

proc iml;
use sashelp.iris;
   read all var _NUM_ into X[c=varName];
   read all var "Species" into ID;
close;

/* traditional PCA */
opt = {.,         /* eigenvalue cutoff (default) */
       0,         /* linear kernel */
       .,         /* linear kernel */
       1,         /* exact method for PCA */
       3};        /* project on 3 PCs */
optClus = j(6,1,.);   /* default values */
call KPCATrain(eigenVal, eigenVec, centroids, RowMeans, score,
               X, opt, optClus);
print eigenVal;
title "Score Plot: Kernel PCA";
title2 "Iris Data; Exact Method";
call scatter(score[,1], score[,2]) grid={x y} group=ID label={"PC1" "PC2"};

Because eigenvectors are not unique, the columns of the SCORE matrix might differ in sign from the Prin1, Prin2, and Prin3 variables in the PCComponents data set (obtained from PROC PRINCOMP). For these data, the third column of the SCORE matrix is the negative of the Prin3 variable. For an example of kernel PCA with a nonlinear (Gaussian) kernel, see the example in the KPCASCORE function.