{"id":37854,"date":"2022-11-21T12:35:04","date_gmt":"2022-11-21T11:35:04","guid":{"rendered":"https:\/\/www.inovex.de\/?p=37854"},"modified":"2022-11-21T12:35:04","modified_gmt":"2022-11-21T11:35:04","slug":"dimensionality-reduction-with-principal-component-analysis","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/","title":{"rendered":"Dimensionality Reduction with Principal Component Analysis"},"content":{"rendered":"<p>This article takes a look at principal component analysis (PCA). In the course of the article, we will work our way through the most important theoretical knowledge and create an implementation of PCA in Python.<\/p>\n<p>The content of this blog post was originally <a href=\"https:\/\/github.com\/zotroneneis\/machine_learning_basics\/blob\/master\/principal_component_analysis.ipynb\" target=\"_blank\" rel=\"noopener\">published as a Jupyter notebook here<\/a>. You can <a href=\"https:\/\/mybinder.org\/v2\/gh\/zotroneneis\/machine_learning_basics\/HEAD?filepath=principal_component_analysis.ipynb\" target=\"_blank\" rel=\"noopener\">run the code interactively in your browser <\/a>using Binder.<!--more--><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_83 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\"><p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<\/div><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#What-is-principal-component-analysis\" >What is principal component analysis?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Why-do-we-need-principal-component-analysis\" >Why do we need principal component analysis?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Key-ideas-of-PCA\" >Key ideas of PCA<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Two-ways-of-deriving-PCA\" >Two ways of deriving PCA<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Maximum-variance-perspective\" >Maximum variance perspective<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Projection-perspective\" >Projection perspective<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Interpreting-the-projection-matrix\" >Interpreting the projection matrix<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Computing-the-eigenvectors\" >Computing the eigenvectors<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Summary-of-important-terminology\" >Summary of important terminology<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Implementation\" >Implementation<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Dataset\" >Dataset<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Standardization\" >Standardization<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Computing-the-data-covariance-matrix\" >Computing the data covariance matrix<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Computing-the-eigenvalues-and-eigenvectors\" >Computing the eigenvalues and eigenvectors<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Choosing-the-number-of-eigenvalues\" >Choosing the number of eigenvalues<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Constructing-the-projection-matrix\" >Constructing the projection matrix<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Projecting-the-data\" >Projecting the data<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Plotting-the-transformed-data\" >Plotting the transformed data<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Comparison-to-scikit-learn\" >Comparison to scikit-learn<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#Sources-and-further-reading\" >Sources and further reading<\/a><\/li><\/ul><\/nav><\/div>\n<h2 id=\"1.-What-is-PCA?-\"><span class=\"ez-toc-section\" id=\"What-is-principal-component-analysis\"><\/span>What is principal component analysis?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In simple terms, principal component analysis (PCA) is a technique to perform dimensionality reduction. It has been around for more than 100 years and is still heavily used. Although PCA is most often applied to find a lower-dimensional representation of data, it can also be used for other purposes, e.g., to detect simple patterns in data.<\/p>\n<h2 id=\"2.-Why-do-we-need-PCA?-\"><span class=\"ez-toc-section\" id=\"Why-do-we-need-principal-component-analysis\"><\/span>Why do we need principal component analysis?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>A lot of data is high-dimensional. Imagine patient data collected in a hospital. For each patient, we can have hundreds or even thousands of measurements (blood pressure, heart rate, respiratory rate, etc.). Working with such data is difficult \u2013 it is expensive to store, hard to analyze, and almost impossible to visualize.<\/p>\n<p>Luckily, many dimensions in high-dimensional data are often redundant and can be expressed by combining some of the other dimensions. Also, the key part of the data is often contained in a more compact lower-dimensional structure. Consequently, we can simplify high-dimensional data points using dimensionality reduction techniques like principal component analysis (PCA).<\/p>\n<h2 id=\"3.-Key-ideas-of-PCA-\"><span class=\"ez-toc-section\" id=\"Key-ideas-of-PCA\"><\/span>Key ideas of PCA<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>PCA finds a lower-dimensional representation of data by constructing new features (called principal components) which are linear combinations of the original features<\/li>\n<li>The new features are selected carefully: PCA looks for features that can summarize the data best without losing too much information<\/li>\n<\/ul>\n<h2 id=\"4.-Two-ways-of-deriving-PCA--\"><span class=\"ez-toc-section\" id=\"Two-ways-of-deriving-PCA\"><\/span>Two ways of deriving PCA<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Let&#8217;s say we have an i.i.d dataset \\(\\boldsymbol{X}\\) with \\(D\\) dimensions and a mean value of \\(0\\):<br \/>\n\\(\\mathcal{\\boldsymbol{X}} = \\{\\boldsymbol{x}_1, &#8230;, \\boldsymbol{x}_N\\} \\boldsymbol{x}_n \\in \\mathbb{R}^D\\). The data covariance matrix (which will be needed later on) is computed as follows:<\/p>\n<p>$$ \\boldsymbol{S} = \\frac{1}{N} \\sum_{n=1}^{N} \\boldsymbol{x}_n \\boldsymbol{x}_n^T$$<\/p>\n<p>In PCA our goal is to find projections of the datapoints \\(\\boldsymbol{x}_n\\) that are as similar to the original data points as possible but have lower dimensionality. We can approach this goal from two perspectives:<\/p>\n<ol>\n<li>Maximum variance perspective: We try to find a low-dimensional representation that maximizes the variance of the projected data.<\/li>\n<li>Projection perspective: We try to find a low-dimensional representation that minimizes the average reconstruction error between the original data and the reconstructed data.<\/li>\n<\/ol>\n<p>Both approaches reach the same result.<\/p>\n<h3 id=\"4.1-Maximum-variance-perspective--\"><span class=\"ez-toc-section\" id=\"Maximum-variance-perspective\"><\/span>Maximum variance perspective<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>In the maximum variance perspective, PCA is derived as an algorithm that tries to find a transformation matrix \\(\\boldsymbol{B}\\) which projects the original data \\(\\boldsymbol{X}\\) to a low-dimensional space, ideally without losing information. Let&#8217;s say that this low-dimensional space has dimension \\(M\\).<\/p>\n<p>How can we make sure that we find a matrix \\(B\\) that retains as much information as possible? It turns out that this is achieved when \\(\\boldsymbol{B}\\) projects the data in a way that maximizes the variance of the data in the (new) low-dimensional space.<\/p>\n<p>Let&#8217;s take a closer look at \\(\\boldsymbol{B}\\). We can define \\(\\boldsymbol{B}\\) as a collection of vectors \\(\\boldsymbol{b}_1, &#8230;, \\boldsymbol{b}_M\\):<\/p>\n<p>$$<br \/>\n\\boldsymbol{B} := [\\boldsymbol{b}_1, &#8230;, \\boldsymbol{b}_M] \\in \\mathbb{R}^{D x M}<br \/>\n$$<\/p>\n<p>The vectors \\(\\boldsymbol{b}_m\\) are called <em>basis vectors<\/em> and form the axes of the new \\(M\\)-dimensional space we project our data to.<\/p>\n<p>When deriving which vectors \\(\\boldsymbol{b}_m\\) we should select to maximize the variance, we will find that maximizing the variance is equivalent to selecting those vectors that belong to the largest eigenvalues of the data covariance matrix \\(\\boldsymbol{S}\\). That means that we can construct our matrix \\(\\boldsymbol{B}\\) by first computing the eigenvalues and eigenvectors of the covariance matrix \\(\\boldsymbol{S}\\) and then selecting the \\(M\\) eigenvectors with the largest eigenvalues.<\/p>\n<p>To be more precise: \\(\\boldsymbol{b}_1\\) will be the eigenvector with the largest eigenvalue. In the context of PCA, it is called the <em>first principal component<\/em>. \\(\\boldsymbol{b}_2\\) will be the eigenvector with the second largest eigenvalue and is called <em>second principal component<\/em> and so on.<\/p>\n<p>If you are interested in the derivation, take a look at chapter 10.2 of the book <a href=\"https:\/\/mml-book.com\" target=\"_blank\" rel=\"noopener\">Mathematics for Machine Learning<\/a>.<\/p>\n<p>Once we have found our projection matrix \\(\\boldsymbol{B}\\), we can transform each data vector \\(\\boldsymbol{x}_n\\) by multiplying it with \\(\\boldsymbol{B}\\). This will give us a low-dimensional compressed representation of \\(\\boldsymbol{x}_n\\):<\/p>\n<p>$$<br \/>\n\\boldsymbol{z}_n = \\boldsymbol{B}^T \\boldsymbol{x}_n \\in \\mathbb{R}^M<br \/>\n$$<\/p>\n<p>Using matrix multiplications we could represent the computation as follows:<\/p>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-37862 size-large\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1-1024x362.png\" alt=\"data matrix X, projection matrix B and compressed data matrix Z on a high level\" width=\"640\" height=\"226\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1-1024x362.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1-300x106.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1-768x272.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1-400x141.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1-360x127.png 360w, https:\/\/www.inovex.de\/wp-content\/uploads\/matrix_multiplications-1.png 1196w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/div>\n<h3 id=\"4.2-Projection-perspective--\"><span class=\"ez-toc-section\" id=\"Projection-perspective\"><\/span>Projection perspective<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Another way to derive PCA is to consider the original data points \\(\\boldsymbol{x}_n\\) and their reconstructions \\(\\boldsymbol{\\tilde{x}}_n\\). In this perspective, we are trying to find reconstructions \\(\\boldsymbol{\\tilde{x}}_n\\) that minimize the averaged squared Euclidean distance between the original datapoints and their reconstructions: \\(J = \\frac{1}{N} \\sum_{n=1}^{N}||\\boldsymbol{x}_n &#8211; \\boldsymbol{\\tilde{x}}_n ||^2\\).<\/p>\n<p>You can find more details about this perspective in chapter 10.3 of the book <a href=\"https:\/\/mml-book.com\" target=\"_blank\" rel=\"noopener\">Mathematics for Machine Learning<\/a>.<\/p>\n<h2 id=\"5.-Interpreting-the-projection-matrix--\"><span class=\"ez-toc-section\" id=\"Interpreting-the-projection-matrix\"><\/span>Interpreting the projection matrix<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In the beginning, I mentioned that PCA constructs new features (the principal components) that are linear combinations of the original features. Let&#8217;s take a closer look at what this means.<\/p>\n<p>Each data point is stored in a vector with \\(D\\) elements: \\(\\boldsymbol{x}_n = [x_{n1}, &#8230;, x_{nD}]\\). Each of the \\(D\\) dimensions represents a different feature. For example, think of an image with \\(D = 64\\) pixels. We can describe each data point as a linear combination of the features:<br \/>\n$$<br \/>\n\\boldsymbol{x}_n = x_{n1} \\cdot \\text{feature}_1 + x_{n2} \\cdot \\text{feature}_2 + &#8230; + x_{nD} \\cdot \\text{feature}_D<br \/>\n$$<\/p>\n<p>Using the example of pixels this would correspond to:<br \/>\n$$<br \/>\n\\boldsymbol{x}_n = x_{n1} \\cdot \\text{pixel}_1 + x_{n2} \\cdot \\text{pixel}_2 + &#8230; + x_{nD} \\cdot \\text{pixel}_D<br \/>\n$$<\/p>\n<p>When PCA projects the data to a low-dimensional space it also uses a combination of the features. This becomes evident in the projection equation we have seen above:<\/p>\n<p>$$<br \/>\n\\boldsymbol{z}_n = \\boldsymbol{B}^T \\boldsymbol{x}_n \\in \\mathbb{R}^M<br \/>\n$$<\/p>\n<p>In this equation, we find the compressed representation \\(\\boldsymbol{z}_n\\) of the datapoint \\(\\boldsymbol{x}_n\\) by performing a matrix multiplication. The new, compressed representation of a datapoint \\(\\boldsymbol{x}_n\\) will be given by a linear combination of all its feature values. We can make this clear when considering an example.<\/p>\n<p>Let&#8217;s say our data matrix \\(\\boldsymbol{X}\\) has 3 datapoints and 2 features, so \\(\\boldsymbol{X} \\in \\mathbb{R}^{3&#215;2}\\) and we consider only the first principal component when performing PCA. Hence, we have \\(\\boldsymbol{B} \\in \\mathbb{R}^{2&#215;1}\\). When multiplying the data matrix with the projection matrix we receive the compressed versions of the data points as a matrix \\(\\boldsymbol{Z} \\in \\mathbb{R}^{3&#215;1}\\). Each one-dimensional code \\(\\boldsymbol{z}_n\\) is given by a linear combination of the original feature values:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-37864 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations.png\" alt=\"data matrix X, projection matrix B and compressed data matrix Z in detail\" width=\"1026\" height=\"342\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations.png 1026w, https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations-300x100.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations-1024x341.png 1024w, https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations-768x256.png 768w, https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations-400x133.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/linear_combinations-360x120.png 360w\" sizes=\"auto, (max-width: 1026px) 100vw, 1026px\" \/><\/p>\n<h2 id=\"6.-Computing-the-eigenvectors--\"><span class=\"ez-toc-section\" id=\"Computing-the-eigenvectors\"><\/span>Computing the eigenvectors<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>By now we know that we have to compute eigenvalues and eigenvectors to perform PCA. We have two ways to do that:<\/p>\n<ol>\n<li>Perform an eigendecomposition of the data covariance matrix \\(\\boldsymbol{S}\\)<\/li>\n<li>Perform a singular value decomposition of the data matrix \\(\\boldsymbol{X}\\)<\/li>\n<\/ol>\n<p>The standard approach (which we will use in this tutorial) is the first approach. You can find more on approach two in chapter 10.4 of the <a href=\"https:\/\/mml-book.com\" target=\"_blank\" rel=\"noopener\">Mathematics for Machine Learning<\/a> book.<\/p>\n<p>Note: In many applications, we only need the first few eigenvectors. Performing a full eigendecomposition or singular value decomposition would be computationally wasteful. Therefore, most software packages implement iterative methods which directly optimize the required number of eigenvectors.<\/p>\n<h2 id=\"7.-Summary-of-important-terminology-\"><span class=\"ez-toc-section\" id=\"Summary-of-important-terminology\"><\/span>Summary of important terminology<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>\\(\\boldsymbol{x}_n\\) are our datapoints, stored in the matrix \\(\\mathcal{\\boldsymbol{X}} = \\{\\boldsymbol{x}_1, &#8230;, \\boldsymbol{x}_N\\}, \\boldsymbol{x}_n \\in \\mathbb{R}^D\\).<\/li>\n<li>\\(\\boldsymbol{S}\\) is the data covariance matrix.<\/li>\n<li>\\(\\boldsymbol{B}\\) is the projection matrix and consists of column vectors \\(\\boldsymbol{b}_m\\): \\(\\boldsymbol{B} := [\\boldsymbol{b}_1, &#8230;, \\boldsymbol{b}_M] \\in \\mathbb{R}^{D x M}\\).<\/li>\n<li>\\(\\boldsymbol{z}_n\\) is the low-dimensional representation of \\(\\boldsymbol{x}_n\\): \\(\\boldsymbol{z}_n = \\boldsymbol{B}^T \\boldsymbol{x}_n \\in \\mathbb{R}^M\\).<\/li>\n<\/ul>\n<h2 id=\"8.-Implementation-\"><span class=\"ez-toc-section\" id=\"Implementation\"><\/span>Implementation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<pre>from sklearn.datasets import load_digits\r\nimport matplotlib.pyplot as plt\r\nimport numpy as np\r\n<\/pre>\n<h3 id=\"8.1-Dataset--\"><span class=\"ez-toc-section\" id=\"Dataset\"><\/span>Dataset<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>We will use the digits dataset from scikit-learn for our implementation. It contains 8&#215;8 images of handwritten digits (1797 in total). The data is stored in a data matrix with 1797 rows and 64 columns (corresponding to all 8&#215;8=64 pixel values).<\/p>\n<pre>digits = load_digits()\r\ndata, labels = digits.data, digits.target\r\n\r\nprint(f\"Shape of data matrix: {data.shape}\")\r\nprint(f\"Shape of label matrix: {labels.shape}\")\r\n\r\n# Example image\r\nplt.imshow(digits.images[42], cmap=\"gray\");\r\n<\/pre>\n<pre>Shape of data matrix: (1797, 64)\r\nShape of label matrix: (1797,)\r\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-37878 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/digit_example-1.png\" alt=\"white pixels on black ground that look like a one\" width=\"245\" height=\"248\" \/><\/p>\n<h3 id=\"8.2-Standardization--\"><span class=\"ez-toc-section\" id=\"Standardization\"><\/span>Standardization<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Before we can apply PCA, we should standardize the data. If you are wondering why this is necessary,\u00a0<a href=\"https:\/\/stats.stackexchange.com\/questions\/69157\/why-do-we-need-to-normalize-data-before-principal-component-analysis-pca\" target=\"_blank\" rel=\"noopener\">check out this explanation on StackExchange<\/a>. Standardization rescales the data to have a mean zero and a standard deviation one.<\/p>\n<pre>mean = np.mean(data, axis=0)\r\nstd = np.std(data, axis=0)\r\n# If values have a standard deviation of zero, the normalization step\r\n# will contain a division by zero. To prevent this we replace standard\r\n# deviations of zero with one.\r\nstd[std == 0] = 1.\r\ndata_norm = (data - mean) \/ std\r\n\r\n# We could also perform standardization using scikit learn\r\nfrom sklearn.preprocessing import StandardScaler\r\ndata_norm_sklearn = StandardScaler().fit_transform(data)\r\n<\/pre>\n<h3 id=\"8.3-Computing-the-data-covariance-matrix--\"><span class=\"ez-toc-section\" id=\"Computing-the-data-covariance-matrix\"><\/span>Computing the data covariance matrix<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To find the principal components we first have to compute the data covariance matrix \\(\\boldsymbol{S}\\):<\/p>\n<p>$$<br \/>\n\\boldsymbol{S} = \\frac{1}{N} \\sum_{n=1}^{N} \\boldsymbol{x}_n \\boldsymbol{x}_n^T<br \/>\n$$<\/p>\n<p>We can summarize the summation as follows:<\/p>\n<p>$$<br \/>\n\\boldsymbol{S} = \\frac{1}{N} \\boldsymbol{X}^T \\boldsymbol{X}<br \/>\n$$<\/p>\n<p>&nbsp;<\/p>\n<pre>n_samples, n_features = data_norm.shape\r\ncov_matrix = (data_norm.T.dot(data_norm)) \/ (n_samples)\r\n\r\n# We could also compute the covariance matrix with numpy (see code below)\r\n# Note: the numpy implementation divides by n_samples - 1. This\r\n# is known as Bessel's correction. Refer to the corresponding \r\n# Wikipedia article if interested in more details.\r\ncov_matrix_numpy = np.cov(data_norm.T)\r\n<\/pre>\n<h3 id=\"8.4-Computing-the-eigenvalues-and-eigenvectors--\"><span class=\"ez-toc-section\" id=\"Computing-the-eigenvalues-and-eigenvectors\"><\/span>Computing the eigenvalues and eigenvectors<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To compute eigenvalues and eigenvectors, we perform an eigendecomposition of the covariance matrix \\(\\boldsymbol{S}\\). Doing this manually is messy and it is hard to do the computations correctly. Therefore, we will make use of NumPy to get both eigenvalues and eigenvectors. If you want to learn more about how to perform an eigendecomposition, take a look at this\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Eigendecomposition_of_a_matrix#Numerical_computations\" target=\"_blank\" rel=\"noopener\">Wikipedia article<\/a>.<\/p>\n<pre>eig_vals, eig_vecs = np.linalg.eig(cov_matrix)\r\nprint(f\"Shape of eigenvalue matrix: {eig_vals.shape}\")\r\nprint(f\"Shape of eigenvectors matrix: {eig_vecs.shape}\")\r\nprint(\"=====================================\")\r\n\r\n# We want to find the eigenvectors with the highest eigenvalues. \r\n# To do so, we use argsort which returns the indices that would \r\n# sort the array of eigenvalues. Since we want to sort from largest\r\n# to lowest value we need to reverse the order of the result\r\nsorted_indices = np.argsort(eig_vals)[::-1]\r\n\r\n# Look at the first few eigenvalues to make sure that we sorted them correctly\r\nfor idx in sorted_indices[:3]:\r\n    print(f\"Eigenvalue: {eig_vals[idx]}\")   \r\n<\/pre>\n<pre>Shape of eigenvalue matrix: (64,)\r\nShape of eigenvectors matrix: (64, 64)\r\n=====================================\r\nEigenvalue: 7.340688819618325\r\nEigenvalue: 5.832243185889725\r\nEigenvalue: 5.151093084500989\r\n<\/pre>\n<h3 id=\"8.5-Choosing-the-number-of-eigenvalues-\"><span class=\"ez-toc-section\" id=\"Choosing-the-number-of-eigenvalues\"><\/span>Choosing the number of eigenvalues<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>In order to decide how many eigenvalues we want to use, we should ask ourselves what our goal is. Remember that the number of eigenvalues that we select corresponds to the dimensionality of the new low-dimensional space we are creating.<\/p>\n<p>Most of the time, PCA is used to reduce the dimensionality of data. But at the same time, we want to make sure that our solution is good, meaning that we do not lose a lot of information. Consequently, we should try to select the eigenvalues that explain most of the variance in the data.<\/p>\n<p>Typically, the first few eigenvalues capture most of the variance whereas later ones do not add much information. When using tools like scikit-learn, we can specify how much variance should be explained by the solution so we do not have to choose the number of eigenvalues ourselves.<\/p>\n<p>In our case, we can decide how many principal components to use by computing the <em>explained variance<\/em> of the individual eigenvectors.<\/p>\n<pre>sum_eig_vals = sum(eig_vals)\r\nsorted_eig_vals = sorted(eig_vals, reverse=True)\r\nexplained_variance = [(eig_val \/ sum_eig_vals) *  100 for eig_val in sorted_eig_vals]\r\ncumulative_explained_variance = np.cumsum(explained_variance)\r\n\r\nplt.figure(figsize=(8, 6))\r\nplt.bar(range(len(eig_vals)), explained_variance, label=\"Explained variance\", color=\"red\")\r\nplt.xlabel(\"Index of principal components\")\r\nplt.ylabel(\"Explained variance in percent\")\r\nplt.title(\"Variance captured by the individual principal components\")\r\nplt.show()\r\n\r\nplt.figure(figsize=(8, 6))\r\nplt.bar(range(len(eig_vals)), cumulative_explained_variance, label=\"Cumulative explained variance\", color=\"blue\")\r\nplt.xlabel(\"Number of principal components\")\r\nplt.ylabel(\"Cumulative explained variance in percent\")\r\nplt.title(\"Cumulative variance captured by the principal components\")\r\nplt.show()\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-37872 alignleft\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/explained_variance.png\" alt=\"diagram of variants captured by the individual components, high values at first going lower\" width=\"401\" height=\"314\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/explained_variance.png 494w, https:\/\/www.inovex.de\/wp-content\/uploads\/explained_variance-300x235.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/explained_variance-400x313.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/explained_variance-360x282.png 360w\" sizes=\"auto, (max-width: 401px) 100vw, 401px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-37880 alignright\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/cumulative_explained_variance-1.png\" alt=\"diagram of cumulative variance captured by the principal components with low to high values.\" width=\"400\" height=\"310\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/cumulative_explained_variance-1.png 500w, https:\/\/www.inovex.de\/wp-content\/uploads\/cumulative_explained_variance-1-300x232.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/cumulative_explained_variance-1-400x310.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/cumulative_explained_variance-1-360x279.png 360w\" sizes=\"auto, (max-width: 400px) 100vw, 400px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>The left plot shows that the first principal components explain more variance than the latter ones. However, the amount of variance explained is not very high. Consequently, we would lose a lot of information when using only the first few principal components. The plot to the right shows that we would have to keep 31 principal components to explain 90% of the variance. This is a lot but still better than keeping all 64 values.<\/p>\n<p>For the purpose of this tutorial, we will keep only the 2 largest eigenvalues and their corresponding eigenvectors. This will allow us to visualize the result in two dimensions.<\/p>\n<pre># Select only the number of eigenvalues \/ principal components that we want to use\r\nn_principal_comps = 2\r\neig_vals = eig_vals[sorted_indices[:n_principal_comps]]\r\neig_vecs = eig_vecs[:, sorted_indices[:n_principal_comps]]\r\n\r\nprint(f\"Eigenvalues shape: {eig_vals.shape}\")\r\nprint(f\"Eigenvectors shape: {eig_vecs.shape}\")\r\n<\/pre>\n<pre>Eigenvalues shape: (2,)\r\nEigenvectors shape: (64, 2)\r\n<\/pre>\n<h3 id=\"8.6-Constructing-the-projection-matrix--\"><span class=\"ez-toc-section\" id=\"Constructing-the-projection-matrix\"><\/span>Constructing the projection matrix<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>After finding the eigenvectors\/principal components, we can arrange them into a matrix (our <em>projection matrix<\/em> \\(\\boldsymbol{B}\\)).<\/p>\n<pre>vector_list = [vec.reshape(n_features, 1) for vec in eig_vecs.T]\r\nproj_matrix = np.hstack(vector_list) # This is our projection matrix B\r\nprint(f\"Shape of projection matrix: {proj_matrix.shape}\")\r\n<\/pre>\n<pre>Shape of projection matrix: (64, 2)\r\n<\/pre>\n<h3 id=\"8.7-Projecting-the-data--\"><span class=\"ez-toc-section\" id=\"Projecting-the-data\"><\/span>Projecting the data<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>In order to actually project the original data \\(\\boldsymbol{X}\\) into the lower-dimensional space, we multiply it with the matrix \\(\\boldsymbol{B}\\):<\/p>\n<p>$$<br \/>\n\\boldsymbol{Z} = \\boldsymbol{X} \\boldsymbol{B} \\in \\mathbb{R}^{N x M}<br \/>\n$$<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-37874 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/pca_digits_matrix_multiplications-2.png\" alt=\"Data matrix X: 1798x64, * Projection matrix B: 64x2 =  Compressed data matrix Z: 1997x2\" width=\"694\" height=\"262\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/pca_digits_matrix_multiplications-2.png 694w, https:\/\/www.inovex.de\/wp-content\/uploads\/pca_digits_matrix_multiplications-2-300x113.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/pca_digits_matrix_multiplications-2-400x151.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/pca_digits_matrix_multiplications-2-360x136.png 360w\" sizes=\"auto, (max-width: 694px) 100vw, 694px\" \/><\/p>\n<pre>data_proj = data_norm.dot(proj_matrix) # This is our compressed data matrix Z\r\nprint(f\"Shape of projected data matrix: {data_proj.shape}\")\r\n<\/pre>\n<pre>Shape of projected data matrix: (1797, 2)\r\n<\/pre>\n<h3 id=\"8.8-Plotting-the-transformed-data--\"><span class=\"ez-toc-section\" id=\"Plotting-the-transformed-data\"><\/span>Plotting the transformed data<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>With only two principal components, we can plot the transformed data easily. One axis will correspond to the first principle component, the other axis to the second.<\/p>\n<pre class=\"\">labels_str = [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"]\r\n\r\nplt.figure(figsize=(8, 6))\r\nscatter = plt.scatter(data_proj[:, 0], data_proj[:, 1], c=labels)\r\nplt.legend(handles=scatter.legend_elements()[0], labels=labels_str, loc=\"upper left\")\r\nplt.xlabel('Principal Component 1')\r\nplt.ylabel('Principal Component 2')\r\nplt.title(\"Compressed digit dataset with labels\")\r\nplt.show();\r\n\r\n# Stackoverflow post to add legend: \r\n# https:\/\/stackoverflow.com\/questions\/17411940\/matplotlib-scatter-plot-legend<\/pre>\n<h3><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-37907 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/plot_pca.png\" alt=\"compressed digital dataset with labels\" width=\"512\" height=\"387\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/plot_pca.png 512w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_pca-300x227.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_pca-400x302.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_pca-360x272.png 360w\" sizes=\"auto, (max-width: 512px) 100vw, 512px\" \/><\/h3>\n<p>&nbsp;<\/p>\n<h3 id=\"8.9-Comparison-to-scikit-learn--\"><span class=\"ez-toc-section\" id=\"Comparison-to-scikit-learn\"><\/span>Comparison to scikit-learn<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>In practice, it does not make sense to perform PCA manually. Instead, we can use packages like scikit-learn. This simplifies the computation substantially. Let&#8217;s apply principal component analysis as contained in scikit-learn to the digits dataset and compare it to our solution. As visible in the plot below, our implementation of PCA gives the same result as the scikit-learn version!<\/p>\n<pre>from sklearn.decomposition import PCA\r\nsklearn_pca = PCA(n_components=2)\r\ndata_proj_sklearn = sklearn_pca.fit_transform(data_norm)\r\n# The eigenvectors in scikit-learn will point into the opposite direction\r\n# compared to our eigenvectors (in other words: they are flipped). This\r\n# does not make a difference regarding the solution. However, to make \r\n# sure that we get exactly the same plot we multiply the result by -1.\r\ndata_proj_sklearn *= -1\r\n\r\nplt.figure(figsize=(8, 6))\r\nscatter = plt.scatter(data_proj_sklearn[:, 0], data_proj_sklearn[:, 1], c=labels, label=labels_str)\r\nplt.xlabel('Principal Component 1')\r\nplt.ylabel('Principal Component 2')\r\nplt.legend(handles=scatter.legend_elements()[0], labels=labels_str, loc=\"upper left\")\r\nplt.title(\"Compressed digits dataset with labels (sklearn version)\")\r\nplt.show()\r\n<\/pre>\n<h2><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-37876 size-full\" src=\"https:\/\/www.inovex.de\/wp-content\/uploads\/plot_sklearn.png\" alt=\"compressed digits with labels as scikit learn version\" width=\"512\" height=\"387\" srcset=\"https:\/\/www.inovex.de\/wp-content\/uploads\/plot_sklearn.png 512w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_sklearn-300x227.png 300w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_sklearn-400x302.png 400w, https:\/\/www.inovex.de\/wp-content\/uploads\/plot_sklearn-360x272.png 360w\" sizes=\"auto, (max-width: 512px) 100vw, 512px\" \/><\/h2>\n<h2 id=\"9.-Sources-and-further-reading-\"><span class=\"ez-toc-section\" id=\"Sources-and-further-reading\"><\/span>Sources and further reading<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The basis for this notebook is chapter 10 of the book <a href=\"https:\/\/mml-book.com\" target=\"_blank\" rel=\"noopener\">Mathematics for Machine Learning<\/a>. I can highly recommend reading through the chapter to get a deeper understanding of PCA.<\/p>\n<p>Another fantastic explanation of PCA is given <a href=\"https:\/\/stats.stackexchange.com\/a\/140579\" target=\"_blank\" rel=\"noopener\">in this post on StackExchange<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article takes a look at principal component analysis (PCA). In the course of the article, we will work our way through the most important theoretical knowledge and create an implementation of PCA in Python. The content of this blog post was originally published as a Jupyter notebook here. You can run the code interactively [&hellip;]<\/p>\n","protected":false},"author":194,"featured_media":39459,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"ep_exclude_from_search":false,"footnotes":""},"tags":[511,206,140],"service":[76,431],"coauthors":[{"id":194,"display_name":"Anna-Lena Popkes","user_nicename":"apopkes"}],"class_list":["post-37854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-artificial-intelligence-2","tag-data-science","tag-machine-learning","service-artificial-intelligence","service-data-science"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Dimensionality Reduction with Principal Component Analysis<\/title>\n<meta name=\"description\" content=\"This article takes a look at principal component analysis (PCA) and shows how to create an implementation of PCA in Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Dimensionality Reduction with Principal Component Analysis\" \/>\n<meta property=\"og:description\" content=\"This article takes a look at principal component analysis (PCA) and shows how to create an implementation of PCA in Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/\" \/>\n<meta property=\"og:site_name\" content=\"inovex GmbH\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inovexde\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-21T11:35:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Anna-Lena Popkes\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis-1024x576.png\" \/>\n<meta name=\"twitter:creator\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:site\" content=\"@inovexgmbh\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Anna-Lena Popkes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"16\u00a0Minuten\" \/>\n\t<meta name=\"twitter:label3\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data3\" content=\"Anna-Lena Popkes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/\"},\"author\":{\"name\":\"Anna-Lena Popkes\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/3df260d7f75d6a1efd09edc3741c0fdf\"},\"headline\":\"Dimensionality Reduction with Principal Component Analysis\",\"datePublished\":\"2022-11-21T11:35:04+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/\"},\"wordCount\":2111,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/dimensionality-reduction-with-principal-component-analysis.png\",\"keywords\":[\"Artificial Intelligence\",\"Data Science\",\"Machine Learning\"],\"articleSection\":[\"Analytics\",\"English Content\",\"General\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/\",\"name\":\"Dimensionality Reduction with Principal Component Analysis\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/dimensionality-reduction-with-principal-component-analysis.png\",\"datePublished\":\"2022-11-21T11:35:04+00:00\",\"description\":\"This article takes a look at principal component analysis (PCA) and shows how to create an implementation of PCA in Python.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/dimensionality-reduction-with-principal-component-analysis.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/dimensionality-reduction-with-principal-component-analysis.png\",\"width\":1920,\"height\":1080,\"caption\":\"a cube being reduced in dimensionality\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/dimensionality-reduction-with-principal-component-analysis\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Dimensionality Reduction with Principal Component Analysis\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#website\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"name\":\"inovex GmbH\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#organization\",\"name\":\"inovex GmbH\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/inovex-logo-16-9-1.png\",\"width\":1921,\"height\":1081,\"caption\":\"inovex GmbH\"},\"image\":{\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/inovexde\",\"https:\\\/\\\/x.com\\\/inovexgmbh\",\"https:\\\/\\\/www.instagram.com\\\/inovexlife\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/inovex\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UC7r66GT14hROB_RQsQBAQUQ\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/#\\\/schema\\\/person\\\/3df260d7f75d6a1efd09edc3741c0fdf\",\"name\":\"Anna-Lena Popkes\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/cropped-lena-96x96.jpgb761218ea0c3098c8e9b2c42d8b06a83\",\"url\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/cropped-lena-96x96.jpg\",\"contentUrl\":\"https:\\\/\\\/www.inovex.de\\\/wp-content\\\/uploads\\\/cropped-lena-96x96.jpg\",\"caption\":\"Anna-Lena Popkes\"},\"description\":\"Hi, I'm Anna-Lena, a senior machine learning engineer here at inovex. I am an enthusiastic learner always looking for new challenges to expand my knowledge and skills. I am deeply fascinated by machine learning and its applications to questions that affect and benefit society. This translates into a passion for teaching, where I can share my insights and contribute to the collective growth of the machine learning community.\",\"sameAs\":[\"https:\\\/\\\/alpopkes.com\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/anna-lena-popkes\"],\"url\":\"https:\\\/\\\/www.inovex.de\\\/de\\\/blog\\\/author\\\/apopkes\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Dimensionality Reduction with Principal Component Analysis","description":"This article takes a look at principal component analysis (PCA) and shows how to create an implementation of PCA in Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/","og_locale":"de_DE","og_type":"article","og_title":"Dimensionality Reduction with Principal Component Analysis","og_description":"This article takes a look at principal component analysis (PCA) and shows how to create an implementation of PCA in Python.","og_url":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/","og_site_name":"inovex GmbH","article_publisher":"https:\/\/www.facebook.com\/inovexde","article_published_time":"2022-11-21T11:35:04+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis.png","type":"image\/png"}],"author":"Anna-Lena Popkes","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis-1024x576.png","twitter_creator":"@inovexgmbh","twitter_site":"@inovexgmbh","twitter_misc":{"Verfasst von":"Anna-Lena Popkes","Gesch\u00e4tzte Lesezeit":"16\u00a0Minuten","Written by":"Anna-Lena Popkes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#article","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/"},"author":{"name":"Anna-Lena Popkes","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/3df260d7f75d6a1efd09edc3741c0fdf"},"headline":"Dimensionality Reduction with Principal Component Analysis","datePublished":"2022-11-21T11:35:04+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/"},"wordCount":2111,"commentCount":1,"publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis.png","keywords":["Artificial Intelligence","Data Science","Machine Learning"],"articleSection":["Analytics","English Content","General"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/","url":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/","name":"Dimensionality Reduction with Principal Component Analysis","isPartOf":{"@id":"https:\/\/www.inovex.de\/de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#primaryimage"},"image":{"@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#primaryimage"},"thumbnailUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis.png","datePublished":"2022-11-21T11:35:04+00:00","description":"This article takes a look at principal component analysis (PCA) and shows how to create an implementation of PCA in Python.","breadcrumb":{"@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#primaryimage","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/dimensionality-reduction-with-principal-component-analysis.png","width":1920,"height":1080,"caption":"a cube being reduced in dimensionality"},{"@type":"BreadcrumbList","@id":"https:\/\/www.inovex.de\/de\/blog\/dimensionality-reduction-with-principal-component-analysis\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inovex.de\/de\/"},{"@type":"ListItem","position":2,"name":"Dimensionality Reduction with Principal Component Analysis"}]},{"@type":"WebSite","@id":"https:\/\/www.inovex.de\/de\/#website","url":"https:\/\/www.inovex.de\/de\/","name":"inovex GmbH","description":"","publisher":{"@id":"https:\/\/www.inovex.de\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inovex.de\/de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/www.inovex.de\/de\/#organization","name":"inovex GmbH","url":"https:\/\/www.inovex.de\/de\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/2021\/03\/inovex-logo-16-9-1.png","width":1921,"height":1081,"caption":"inovex GmbH"},"image":{"@id":"https:\/\/www.inovex.de\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inovexde","https:\/\/x.com\/inovexgmbh","https:\/\/www.instagram.com\/inovexlife\/","https:\/\/www.linkedin.com\/company\/inovex","https:\/\/www.youtube.com\/channel\/UC7r66GT14hROB_RQsQBAQUQ"]},{"@type":"Person","@id":"https:\/\/www.inovex.de\/de\/#\/schema\/person\/3df260d7f75d6a1efd09edc3741c0fdf","name":"Anna-Lena Popkes","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-lena-96x96.jpgb761218ea0c3098c8e9b2c42d8b06a83","url":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-lena-96x96.jpg","contentUrl":"https:\/\/www.inovex.de\/wp-content\/uploads\/cropped-lena-96x96.jpg","caption":"Anna-Lena Popkes"},"description":"Hi, I'm Anna-Lena, a senior machine learning engineer here at inovex. I am an enthusiastic learner always looking for new challenges to expand my knowledge and skills. I am deeply fascinated by machine learning and its applications to questions that affect and benefit society. This translates into a passion for teaching, where I can share my insights and contribute to the collective growth of the machine learning community.","sameAs":["https:\/\/alpopkes.com\/","https:\/\/www.linkedin.com\/in\/anna-lena-popkes"],"url":"https:\/\/www.inovex.de\/de\/blog\/author\/apopkes\/"}]}},"_links":{"self":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/37854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/users\/194"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/comments?post=37854"}],"version-history":[{"count":5,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/37854\/revisions"}],"predecessor-version":[{"id":39462,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/posts\/37854\/revisions\/39462"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media\/39459"}],"wp:attachment":[{"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/media?parent=37854"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/tags?post=37854"},{"taxonomy":"service","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/service?post=37854"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.inovex.de\/de\/wp-json\/wp\/v2\/coauthors?post=37854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}