The final topic in this class is eigenvectors and eigenvalues. These are particular objects attached to a matrix (or linear transformation) that provide an important way of understanding it, and are extremely useful in applications.
Let's start with the basic definitions.
Definition (eigenvector, eigenvalue)
Let \(A\) be an \(n \times n\) matrix. An eigenvector of \(A\) is a nonzero vector \(\mathbf{x} \in \mathbb{R}^n\) such that \(A\mathbf{x}\) is a scalar multiple of \(\mathbf{x}\): \[A\mathbf{x} = \lambda \mathbf{x}\] for some scalar \(\lambda.\) We call \(\lambda\) the eigenvalue of \(\mathbf{x}.\)
Notice that eigenvectors are, by definition, nonzero vectors: the zero vector is not called an eigenvector. Notice also that only square matrices can have eigenvectors and eigenvalues: if \(A\) is not square, then \(A\mathbf{x}\) and \(\mathbf{x}\) do not have the same number of components, so it doesn't make sense for one to be a scalar multiple of the other.
Intuitively, it can be helpful to consider the linear transformation \(T \colon \mathbb{R}^n \to \mathbb{R}^n\) corresponding to \(A.\) Then, an eigenvector is a vector in \(\mathbb{R}^n\) that stays in the same direction, possibly with some scaling, when \(T\) is applied. The corresponding eigenvalue is the factor by which that eigenvector gets scaled when applying \(T.\)
Note: The prefix "eigen-" is a borrowing from German; it means something like "own" or "inherent." In other words, eigenvectors and eigenvalues are objects that are, in some sense, inherent to a matrix or linear transformation.
If given a \(n \times n\) matrix and a nonzero vector in \(\mathbb{R}^n,\) it is pretty easy to check whether that vector is an eigenvector of the matrix, and if so, to compute its corresponding eigenvalue:
Example (eigenvector, eigenvalue)
Let's check that \(A = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix}\) has \(\mathbf{x} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}\) as an eigenvector. To do so, we can multiply \[A \mathbf{x} = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix} \begin{bmatrix} 3 \\ 1 \end{bmatrix} = \begin{bmatrix} 9 \\ 3 \end{bmatrix}.\] We can see that \(A\mathbf{x} = \begin{bmatrix} 9 \\ 3 \end{bmatrix}\) is a scalar multiple of \(\mathbf{x} = \begin{bmatrix} 3 \\ 1 \end{bmatrix},\) so \(\mathbf{x}\) is indeed an eigenvector of \(A.\) To be more specific, we have that \(\begin{bmatrix} 9 \\ 3 \end{bmatrix} = 3 \begin{bmatrix} 3 \\ 1 \end{bmatrix},\) so \(A\mathbf{x} = 3\mathbf{x}.\) Therefore, the corresponding eigenvalue is \(\lambda = 3.\)
However, the reverse question is more difficult.
Question
Given an \(n \times n\) matrix, how can we find its eigenvectors and eigenvalues?
To answer this question, we will first discuss how to compute the eigenvalues. Then, we will discuss how to find the eigenvectors corresponding to each eigenvalue.
By definition, a real number \(\lambda\) is an eigenvalue of an \(n \times n\) matrix \(A\) if and only if there is a nonzero vector \(\mathbf{x} \in \mathbb{R}^n\) such that \(A\mathbf{x} = \lambda \mathbf{x}.\) In other words, \(\lambda\) is an eigenvalue of \(A\) if and only if the equation \(A\mathbf{x} = \lambda \mathbf{x}\) has a nontrivial solution. When does this occur? To answer this question, we can rewrite the equation to make it look more like a homogeneous system: \[A \mathbf{x} - \lambda \mathbf{x} = \mathbf{0}.\] It may now be tempting to factor the left-hand side to get \((A-\lambda)\mathbf{x} = \mathbf{0}.\) However, this doesn't make sense: \(A\) is a matrix and \(\lambda\) is a real number, so we can't subtract them. Instead, we can replace \(\lambda \mathbf{x}\) with \((\lambda I_n) \mathbf{x},\) where \(I_n\) is the identity matrix, because \(I_n \mathbf{x}\) is just \(\mathbf{x}.\) Hence, we have \[A \mathbf{x} - \lambda I_n \mathbf{x} = \mathbf{0},\] which we can now factor as \[(A - \lambda I_n) \mathbf{x} = \mathbf{0}.\] Hence, \(\lambda\) is an eigenvalue if and only if the above system has a nontrivial solution for \(\mathbf{x}.\) Since \(A - \lambda I_n\) is an \(n \times n\) matrix, we know a ton of different statements equivalent to this condition, thanks to the unifying theorem; they are all equivalent to \(A - \lambda I_n\) being non-invertible. The most helpful equivalent statement is the one using the determinant: \((A - \lambda I_n) \mathbf{x} = \mathbf{0}\) has a nontrivial solution if and only if \(\det(A - \lambda I_n) = 0.\) This proves the following result:
Theorem (condition for determining eigenvalues)
Let \(A\) be an \(n \times n\) matrix. Then a real number \(\lambda\) is an eigenvalue of \(A\) if and only if \[\det(A - \lambda I_n) = 0.\]
Let's see this in action for the example above.
Example (computing eigenvalues)
Let's find the eigenvalues of \[A = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix}.\] (From earlier, we know that one of the eigenvalues should be \(3.\))
According to our reasoning so far, we want to write down the matrix \(A - \lambda I_2,\) compute its determinant, and set it equal to zero. We have \[A - \lambda I_2 = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2-\lambda & 3 \\ -1 & 6-\lambda \end{bmatrix}.\] Therefore, the determinant of this matrix is \[\det(A - \lambda I_2) = (2-\lambda)(6-\lambda) - (3)(-1) = \lambda^2 - 8\lambda + 15.\] Hence, \(\lambda\) is an eigenvalue if and only if it satisfies the equation \[\lambda^2 - 8\lambda + 15 = 0.\] Hey, it's just a standard, garden-variety quadratic equation! No vectors or matrices or anything.
As you know, there are several ways to solve a quadratic equation. In this case, we can just factor by inspection as \((\lambda-3)(\lambda-5) = 0.\) Thus, the solutions of the equation, and therefore the eigenvalues of \(A,\) are \(3\) and \(5.\)
Eigenvalues also give us the final part of the unifying theorem. In particular, notice that when \(\lambda = 0,\) the above theorem says that \(0\) is an eigenvalue of \(A\) if and only if \(\det(A) = 0.\) In other words, \(0\) is an eigenvalue of \(A\) if and only if \(A\) is non-invertible. (You can also see this directly from the definition: \(0\) is an eigenvalue of \(A\) if and only if \(A\mathbf{x} = 0\mathbf{x},\) which is just \(A\mathbf{x} = \mathbf{0},\) has a nontrivial solution.) Therefore, we can add the condition that \(0\) is not an eigenvalue of \(A\) to the unifying theorem!
In general, the determinant \(\det(A - \lambda I_n),\) when expanded, will be a polynomial in the variable \(\lambda.\) This polynomial has enormous significance in linear algebra, and it gets a special name:
Definition (characteristic polynomial)
Let \(A\) be an \(n \times n\) matrix. The characteristic polynomial of \(A,\) denoted \(p_A(\lambda),\) is defined by \[p_A(\lambda) = \det(A - \lambda I_n).\] The characteristic equation is the polynomial equation \(p_A(\lambda) = 0.\)
Thus, the eigenvalues of \(A\) are the roots of its characteristic polynomial. Our example above showed that \(A = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix}\) has characteristic polynomial \[p_A(\lambda) = \lambda^2 - 8\lambda + 15.\] Notice that \(A\) is a \(2 \times 2\) matrix, and \(p_A(\lambda)\) is a degree-\(2\) (quadratic) polynomial. This will always be the case:
Fact (degree of characteristic polynomial)
The characteristic polynomial of an \(n \times n\) matrix has degree \(n.\)
To see why, let's write out the general form of the characteristic polynomial: \[\det(A - \lambda I_n) = \left|\begin{matrix} a_{11} - \lambda & a_{12} & \dotsm & a_{1n} \\ a_{21} & a_{22}-\lambda & \dotsm & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \dotsm & a_{nn}-\lambda \end{matrix} \right|.\] In this determinant, each pattern will have \(n\) entries, each of which is either a constant \(a_{ij},\) or a linear factor of the form \(a_{ii} - \lambda.\) Therefore, when we multiply the entries in each pattern and expand, each term will have degree at most \(n.\) Furthermore, the only degree-\(n\) term will come from the pattern consisting of all the diagonal entries: \((a_{11}-\lambda)(a_{22}-\lambda)\dotsm (a_{nn}-\lambda).\)
Note (the leading coefficient of the characteristic polynomial)
The leading coefficient of \(p_A(\lambda)\) — the coefficient of \(\lambda^n\) — is always \((-1)^n,\) since the leading term comes from multiplying out \((a_{11}-\lambda)(a_{22}-\lambda)\dotsm (a_{nn}-\lambda).\) It can be annoying that the leading coefficient is not always \(1,\) so some authors and sources define the characteristic polynomial as the one in our definition but multiplied by \((-1)^n.\) Of course, this doesn't change the roots of the polynomial, so the eigenvalues can be calculated the same way.
Now we know that finding the eigenvalues of a matrix is equivalent to finding the roots of a polynomial. Therefore, we can use algebraic facts about polynomial equations to study eigenvalues.
First, recall that a polynomial of degree \(n\) can have at most \(n\) distinct roots. Therefore, we have the following fact:
Fact (number of eigenvalues)
An \(n \times n\) matrix has at most \(n\) distinct eigenvalues.
In addition, recall that polynomials can have repeated roots: for instance, the polynomial \[\lambda^3 - 15\lambda^2 + 72\lambda - 112 = (\lambda-4)^2 (\lambda-7)\] has two distinct roots, \(4\) and \(7,\) but the root \(4\) is said to have multiplicity \(2\) because two factors of \((\lambda-4)\) appear when the polynomial is factored. This gives rise to the following term related to eigenvalues:
Definition (algebraic multiplicity)
Let \(A\) be an \(n \times n\) matrix and let \(\lambda_1\) be one of its eigenvalues. The algebraic multiplicity of \(\lambda_1\) (with respect to \(A\)) is the multiplicity of \(\lambda_1\) as a root of the characteristic polynomial \(p_A(\lambda).\) In other words, it is the exponent on \((\lambda-\lambda_1)\) in the factored form of \(p_A(\lambda).\)
(When the algebraic multiplicity of an eigenvalue is \(1,\) we often call it a simple eigenvalue.)
Example (algebraic multiplicities)
Suppose \(A\) is a square matrix with characteristic polynomial \[ \lambda^3 - 15\lambda^2 + 72\lambda - 112 = (\lambda-4)^2 (\lambda-7). \] (Notice that from this information alone we can deduce the size of \(A\): it must be a \(3 \times 3\) matrix.) Then \(A\) has two distinct eigenvalues: \(\lambda_1 = 4,\) with algebraic multiplicity \(2,\) and \(\lambda_2 = 7,\) with algebraic multiplicity \(1.\)
We use the term algebraic multiplicity here because this multiplicity comes from solving an algebraic equation, namely \(p_A(\lambda) = 0.\) This will soon be contrasted with another quantity called geometric multiplicity.
Using the notion of algebraic multiplicity, we can state a stronger version of the previous fact about the number of eigenvalues:
Fact (sum of algebraic multiplicities)
The sum of the algebraic multiplicities of the eigenvalues of an \(n \times n\) matrix is at most \(n.\)
In other words, if we count each eigenvalue as many times as it appears as a root in \(p_A(\lambda),\) the sum of those counts is still at most \(n.\)
To close out this section, let's look at another example of the characteristic polynomial.
Example (characteristic polynomial of a triangular matrix)
Suppose that \(A\) is an upper triangular matrix. Then its characteristic polynomial looks like \[p_A(\lambda) = \det(A - \lambda I_n) = \left|\begin{matrix} a_{11} - \lambda & a_{12} & \dotsm & a_{1n} \\ 0 & a_{22}-\lambda & \dotsm & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dotsm & a_{nn}-\lambda \end{matrix} \right|.\] This determinant is triangular, so we know from our determinant facts that it equals the product of the diagonal entries: \[p_A(\lambda) = (a_{11}-\lambda) (a_{22}-\lambda) \dotsm (a_{nn}-\lambda).\] Thus, the characteristic polynomial already comes fully factored! The eigenvalues of \(A\) are just the diagonal entries of \(A,\) and the multiplicity of an eigenvalue is just how many times it appears on the diagonal.
The same is true for lower triangular matrices, and for diagonal matrices (which are both upper and lower triangular).
And finally, here are two remarks about the characteristic polynomial:
Note (non-real roots of the characteristic polynomial)
As you may know, polynomials like \(p_A(\lambda)\) can have roots that are (nonreal) complex numbers, that is, numbers of the form \(a + bi\) where \(i^2 = -1.\) Therefore, one could imagine extending our definitions to allow complex numbers to appear as eigenvalues (or scalars in general) and as entries in eigenvectors (or vectors and matrices in general). This is useful in many important applications, such as the fast Fourier transform. However, we won't do this in this class; we will define eigenvalues to necessarily be real numbers. Thus, if a matrix \(A\) has characteristic polynomial \(\lambda^2 + 1,\) for example, then we will say it has no eigenvalues at all.
Note (computing the roots of the characteristic polynomial)
You may have heard that, in general, there is no formula (like the quadratic formula) for the roots of polynomials of degree \(5\) or higher. (This is the famous Abel–Ruffini theorem.) This means that, for \(5 \times 5\) matrices and larger, there might not be any way to write down exact values for the eigenvalues; they may need to be numerically approximated.
In this class, you'll only be asked to deal with characteristic polynomials for which finding the roots is "easy": they may be quadratics, or occasionally higher-degree polynomials where some of the roots are "obvious," can be easily guessed, or are just given to you. So don't fret too much about solving polynomial equations in this course!
Now that we know how to find the eigenvalues of a matrix, the idea is to take each eigenvalue, one by one, and compute all the corresponding eigenvectors. The collection of all eigenvectors with a particular eigenvalue has a (perhaps predictable) name:
Definition (eigenspace)
Let \(A\) be an \(n \times n\) matrix and let \(\lambda\) be one of its eigenvalues. The eigenspace of \(\lambda\) with respect to \(A\), denoted \(E_\lambda(A),\) is the set of all eigenvectors of \(A\) with eigenvalue \(\lambda,\) and the zero vector. (Remember: \(\mathbf{0}\) is not considered an eigenvector, even though \(A \mathbf{0} = \lambda \mathbf{0}.\)) In other words, \[E_\lambda(A) = \{ \mathbf{x} \in \mathbb{R}^n : A \mathbf{x} = \lambda \mathbf{x} \}.\]
As we saw earlier, the equation \(A \mathbf{x} = \lambda \mathbf{x}\) is equivalent to \[(A - \lambda I_n) \mathbf{x} = \mathbf{0}.\] This represents a homogeneous system, and the set of all solutions is, by definition, the null space of the matrix:
Theorem (eigenspaces as null spaces)
Let \(A\) be an \(n \times n\) matrix and let \(\lambda\) be one of its eigenvalues. Then \[E_\lambda(A) = \text{null}(A - \lambda I_n).\]
In particular, \(E_\lambda(A)\) is a subspace of \(\mathbb{R}^n\) because null spaces are always subspaces. (This is one reason it makes sense to include the zero vector in \(E_\lambda(A),\) since subspaces must contain the zero vector.)
Therefore, to compute the eigenvectors associated to an eigenvalue \(\lambda,\) we just need to write down the matrix \(A - \lambda I_n\) and compute (a basis for) its null space, something we know how to do!
Example (computing eigenspaces)
Earlier, we found that the matrix \(A = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix}\) has eigenvalues \(3\) and \(5.\) Let's find the corresponding eigenspaces.
For the eigenvalue \(\lambda = 3,\) we have \[A - 3I_2 = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix} - \begin{bmatrix} 3 & 0 \\ 0 & 3 \end{bmatrix} = \begin{bmatrix} -1 & 3 \\ -1 & 3 \end{bmatrix}.\] Since \(3\) is an eigenvalue, we expect this matrix to be non-invertible, and indeed it is, since its two rows are the same. By solving the system, we can find that \(\text{null}(A - 3 I_2)\) has basis \(\begin{bmatrix} 3 \\ 1 \end{bmatrix}.\) (This is the eigenvector we were given in the very first example.)
For the eigenvalue \(\lambda = 5,\) we have \[A - 5I_2 = \begin{bmatrix} 2 & 3 \\ -1 & 6 \end{bmatrix} - \begin{bmatrix} 5 & 0 \\ 0 & 5 \end{bmatrix} = \begin{bmatrix} -3 & 3 \\ -1 & 1 \end{bmatrix}.\] By solving the system, we can find that \(\text{null}(A - 5 I_2)\) has basis \(\begin{bmatrix} 1 \\ 1 \end{bmatrix}.\)
Thus, \(A\) has two eigenspaces: \[E_3(A) = \text{span} \left( \begin{bmatrix} 3 \\ 1 \end{bmatrix} \right) \qquad \text{and} \qquad E_5(A) = \text{span} \left( \begin{bmatrix} 1 \\ 1 \end{bmatrix} \right).\]
The process for finding the eigenvalues and eigenvectors is extremely important, and you should make sure you have it down pat: