Linear Algebra for Data Science Cookbook

A review of mathematical concepts helpful for data science.

mathematicsdata science

These notes cover key linear algebra topics from the MIT Opencourseware course on Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, using Linear Algebra and Learning from Data as the main reference. All ideas derive from these sources.

Summary

Positive definite symmetric matrices S are particularly useful because they have positive eigenvalues \lambda and orthogonal eigenvectors q. Any such matrix can be written as a combination S = \lambda_1 q_1 q_1^T + \lambda_2 q_2 q_2^T ... of simple rank-one projections qq^T onto those eigenvectors.

If \lambda_1 \geq \lambda_2 \geq \lambda_3 ..., then \lambda_1 q_1 q_1^T captures the greatest variation in a sample covariance matrix.

The singular value decomposition (SVD) extends these ideas to all matrices. It decomposes matrix A into two sets of singular vectors (us and vs) and singular values \sigma, which play the same role as eigenvalues \lambda.

A = \sigma_1 u_1 v_1^T + \sigma_2 u_2 v_2^T + ... A = U \Sigma V^T

Rank-one pieces of A retain their importance ordering as \sigma decreases. The Eckart-Young theorem states that for a rank-k approximation, stop at \sigma_k u_k v_k^T, which complements our understanding of the symmetric matrix A^T A.

Topics from Part 1 of Linear Algebra and Learning from Data

  1. Multiplication Ax using columns of A
  2. Matrix-Matrix multiplication \boldsymbol{AB}
  3. The four fundamental subspaces
  4. Elimination and \boldsymbol{A = LU}
  5. Orthogonal matrices and subspaces
  6. Eigenvalues and eigenvectors
  7. Symmetric positive definite matrices
  8. Singular values and singular vectors in the SVD
  9. Principal components and the best low rank matrix
  10. Norms of vectors and functions and matrices