The next part of this course will cover numerical linear algebra.

Linear Algebra Review

Recall that an eigenvalue of a square matrix \(\mathbf{X} \in \mathbb{R}^{d \times d}\) is any vector \(\mathbf{v}\) such that \(\mathbf{Xv} = \lambda \mathbf{v}\) for some scalar \(\lambda\).

The matrix \(\mathbf{X}\) has at most \(d\) linearly independent eigenvectors. If it has a full set of \(d\) eigenvectors \(\mathbf{v}_1, \ldots, \mathbf{v}_d\) with eigenvalues \(\lambda_1 \geq \ldots \geq \lambda_d\), the matrix is called diagonalizable and can be written as \[ \mathbf{X} = \mathbf{V} \mathbf{\Lambda} \mathbf{V}^{-1}. \]

Decomposing a matrix into its eigenvectors and eigenvalues is called eigendecomposition.

While eigendecomposition only applies to square matrices, we can extend the idea to rectangular matrices with a related tool called singular value decomposition. But first, let’s review eigendecomposition. If a square matrix \(\mathbf{V}\) has orthonormal rows, it also has orthonormal columns.

That is, \(\mathbf{V}^\top \mathbf{V} = \mathbf{I}\) and \(\mathbf{V} \mathbf{V}^\top = \mathbf{I}\).

This implies that, for any vector \(\mathbf{x}\), \(\| \mathbf{V x}\|_2^2 = \| \mathbf{x} \|_2^2 = \| \mathbf{V}^\top \mathbf{x} \|_2^2\).

To see this, we can write \[\begin{align*} \| \mathbf{V x}\|_2^2 = (\mathbf{V x})^\top (\mathbf{V x}) = \mathbf{x}^\top \mathbf{V}^\top \mathbf{V x} = \mathbf{x}^\top \mathbf{x} = \| \mathbf{x} \|_2^2. \end{align*}\] A similar set of steps shows that \(\| \mathbf{V}^\top \mathbf{x} \|_2^2 = \| \mathbf{x} \|_2^2\).

We have the same property for the Frobenius norm of a matrix. For any square matrix \(\mathbf{X} \in \mathbb{R}^{d \times d}\), the Frobenius norm \(\| \mathbf{X} \|_F^2\) is defined as the sum of squared entries \(\sum_{i=1}^n \sum_{i,j} x_{ij}^2\). To see that the same property holds for the Frobenius norm, we can write \[\begin{align*} \| \mathbf{V X}\|_F^2 = \sum_{i=1}^d \| \mathbf{V} \mathbf{X}_i \|_2^2 = \sum_{i=1}^d \| \mathbf{X}_i \|_2^2 = \| \mathbf{X} \|_F^2 \end{align*}\] where \(\mathbf{X}_i\) denotes the \(i\)th column of \(\mathbf{X}\) and the second equality follows from the previous result. A similar set of steps shows that \(\| \mathbf{V}^\top \mathbf{X} \|_F^2 = \| \mathbf{X} \|_F^2\).

These properties are not true for rectangular matrices. Let \(\mathbf{V} \in \mathbb{R}^{d \times k}\) with \(d > k\) be a matrix with orthogonal columns. Then \(\mathbf{V}^\top \mathbf{V} = \mathbf{I}\) but \(\mathbf{V} \mathbf{V}^\top \neq \mathbf{I}\).

Similarly, for any \(\mathbf{x}\), \(\| \mathbf{V x}\|_2^2 = \| \mathbf{x} \|_2^2\) but \(\| \mathbf{V}^\top \mathbf{x} \|_2^2 \neq \| \mathbf{x} \|_2^2\).

Multiplying a vector by a matrix \(\mathbf{V}\) with orthonromal columns rotates and/or reflects the vector.