Exercise
For each matrix \(A,\) find the eigenvalues of \(A\) and a basis for each eigenspace:
- \(A = \begin{bmatrix} 2 & 3 & 3 \\ 0 & -1 & 0 \\ 3 & 3 & 2 \end{bmatrix}.\)
- \(A = \begin{bmatrix} 0 & -3 \\ 3 & 6 \end{bmatrix}.\)
- \(A = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}.\)
(The solutions can be found in the rest of these notes. I suggest you try these for yourself, then check your answers as you go!)
Last time, we defined eigenvalue, eigenvector, and eigenspace and described how to compute these for a given \(n \times n\) matrix. Today, we'll say more about the eigenspaces of a matrix and discuss when we can use eigenvectors to build a basis for \(\mathbb{R}^n.\)
In the example at the end of last class, we computed the eigenvalues and eigenvectors of a \(2 \times 2\) matrix, and found that there were two different eigenvalues, each of which had a \(1\)-dimensional eigenspace. In general, however, the eigenspaces are not always \(1\)-dimensional, as the next example shows:
Example (computing eigenvalues and eigenvectors)
Let \(A = \begin{bmatrix} 2 & 3 & 3 \\ 0 & -1 & 0 \\ 3 & 3 & 2 \end{bmatrix}.\)
To find the eigenvalues of \(A,\) we can compute its characteristic polynomial: \[p_A(\lambda) = \det(A - \lambda I_3) = \left| \begin{matrix} 2-\lambda & 3 & 3 \\ 0 & -1-\lambda & 0 \\ 3 & 3 & 2-\lambda \end{matrix} \right| = -(\lambda+1)^2(\lambda-5).\] (To compute this determinant, it's easiest to expand about the second row, since all but one entry equals zero.) Thus, the eigenvalues of \(A\) are \(-1\) (with algebraic multiplicity \(2\)) and \(5\) (with algebraic multiplicity \(1\); this is a simple eigenvalue).
Now let's compute its eigenvectors. For the eigenvalue \(-1,\) the corresponding eigenspace is \[E_{-1}(A) = \text{null}(A + I_3) = \text{null} \begin{bmatrix} 3 & 3 & 3 \\ 0 & 0 & 0 \\ 3 & 3 & 3 \end{bmatrix}.\] This null space is the set of all vectors whose sum of coordinates is zero; a possible basis for it consists of \(\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}\) and \(\begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}.\)
For the eigenvalue \(5,\) we have \[E_5(A) = \text{null}(A - 5I_3) = \text{null} \begin{bmatrix} -3 & 3 & 3 \\ 0 & 6 & 0 \\ 3 & 3 & -3 \end{bmatrix}.\] After row-reducing, we can find that this null space is 1-dimensional and is spanned by the single vector \(\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}.\)
Thus, our result is that \(A\) has two distinct eigenspaces: \[E_{-1}(A) = \text{span}\left( \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \; \begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}\right) \qquad \text{and} \qquad E_5(A) = \text{span}\left( \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}\right).\] (The corresponding eigenvalues have algebraic multiplicities 2 and 1, respectively.)
In the above example, the eigenspace for \(\lambda = -1\) was \(2\)-dimensional. In general, the dimension of an eigenspace is called the geometric multiplicity of the corresponding eigenvalue:
Definition (geometric multiplicity)
Let \(A\) be an \(n \times n\) matrix and let \(\lambda\) be one of its eigenvalues. The geometric multiplicity of \(\lambda\) is the dimension of the eigenspace \(E_\lambda(A),\) considered as a subspace of \(\mathbb{R}^n.\)
In other words, the geometric multiplicity of \(\lambda\) equals the maximum number of linearly independent eigenvectors we can find with eigenvalue \(\lambda.\) (This is called the geometric multiplicity because it is defined using "geometric" information, namely, the dimension of a subspace.)
You may have noticed that in the above example, the algebraic multiplicity of each eigenvalue matched its geometric multiplicity. As we'll see later, this is in some sense the "ideal" scenario. Unfortunately, this is not always the case:
Example (algebraic vs. geometric multiplicity)
Let \(A = \begin{bmatrix} 0 & -3 \\ 3 & 6 \end{bmatrix}.\) The characteristic polynomial of \(A\) is \[p_A(\lambda) = \left| \begin{matrix} -\lambda & -3 \\ 3 & 6-\lambda \end{matrix} \right| = (-\lambda)(6-\lambda) - (-3)(3) = \lambda^2 - 6\lambda + 9,\] which factors as \((\lambda-3)^2.\) Therefore, \(A\) has only one eigenvalue, \(\lambda = 3,\) with algebraic multiplicity \(2.\)
Let's now compute the eigenvectors corresponding to \(\lambda = 3.\) We have \[A - 3I_2 = \begin{bmatrix} -3 & -3 \\ 3 & 3 \end{bmatrix},\] whose null space has basis \(\left\{ \begin{bmatrix} 1 \\ -1 \end{bmatrix} \right\}.\) Hence, \[E_3(A) = \text{span}\left( \begin{bmatrix} 1 \\ -1 \end{bmatrix} \right),\] so \(\lambda = 3\) has geometric multiplicity \(1.\)
In the above example, the geometric multiplicity of the (only) eigenvalue was strictly less than its algebraic multiplicity, while in the example before, the geometric multiplicity of each eigenvalue was equal to its algebraic multiplicity. It turns out that these are the only two possibilities:
Theorem (algebraic vs. geometric multiplicity)
Let \(A\) be an \(n \times n\) matrix and let \(\lambda\) be an eigenvalue of \(A.\) Then the geometric multiplicity of \(\lambda\) is always less than or equal to the algebraic multiplicity of \(\lambda.\)
(Proving this result requires some more facts involving change-of-basis than we have at this point, so I won't give a proof.)
Here is two easy but important corollaries of this fact:
Corollary (sum of geometric multiplicities)
Let \(A\) be an \(n \times n\) matrix. Then the sum of the geometric multiplicities of the eigenvalues of \(A\) is at most \(n.\)
This follows immediately from the above theorem, plus the fact that the sum of the algebraic multiplicities is at most \(n\) (since they come from the characteristic polynomial of \(A,\) which has degree \(n\)).
Corollary (simple eigenvalues)
Let \(A\) be an \(n \times n\) matrix and let \(\lambda\) be a simple eigenvalue of \(A.\) (Remember, simple means that \(\lambda\) has algebraic multiplicity \(1.\)) Then the eigenspace \(E_\lambda(A)\) must be \(1\)-dimensional; the geometric multiplicity of \(\lambda\) must also be \(1.\)
The reason for this is, well, simple: the dimension of \(E_\lambda(A)\) must be less than or equal to the algebraic multiplicity, which is \(1,\) but it must also be at least \(1\) because it is a nontrivial subspace. Thus the dimension can only be exactly \(1.\)
The above theorem and this corollary give us a good way to do a "sanity check" on our work when finding eigenvalues and eigenvectors. If we find that a matrix has an eigenvalue with algebraic multiplicity \(2,\) then the corresponding eigenspace can only have dimension \(1\) or \(2.\) If a matrix has a simple eigenvalue, then the corresponding eigenspace must have dimension \(1.\)
Earlier we mentioned that the "ideal" scenario for an \(n \times n\) matrix \(A\) is when the geometric multiplicity of each eigenvalue equals its algebraic multiplicity. This is because in this scenario, there are "as many eigenvectors" as possible. In particular, there is a chance that we can find \(n\) linearly independent eigenvectors of \(A,\) which will form a basis for \(\mathbb{R}^n\) by the unifying theorem. Such a basis has a (predictable) name:
Definition (eigenbasis)
Let \(A\) be an \(n \times n\) matrix. An eigenbasis for \(A\) is a basis for \(\mathbb{R}^n\) consisting of eigenvectors of \(A.\)
Please note that, even though we call it an eigenbasis for the matrix \(A,\) it is really a basis of \(\mathbb{R}^n\); it doesn't make sense to speak of a basis of a matrix.
When does an eigenbasis exist, and why are they useful? We will now try to answer these two questions.
Given an \(n \times n\) matrix \(A,\) how would we try to build an eigenbasis for \(A\)? Well, each vector in an eigenbasis is by definition an eigenvector of \(A,\) so to get an eigenbasis, we should choose some vectors from each eigenspace of \(A.\) In order for the chosen vectors to form a basis for \(\mathbb{R}^n,\) we should try to take as many as possible, while keeping them linearly independent. Therefore, the best we can do is to take a basis for each eigenspace and combine them.
Notice, however, that if we combine bases for each eigenspace, it's not clear whether the resulting list of vectors is still linearly independent. The vectors in each eigenspace basis are certainly independent, but there could be linear relations "between" the vectors in different eigenspace bases. Luckily, this cannot happen, as we'll now prove.
Theorem (eigenvectors with different eigenvalues are linearly independent)
Let \(A\) be an \(n \times n\) matrix. Let \(\mathbf{v}_1, \dots, \mathbf{v}_m\) be eigenvectors of \(A\) with eigenvalues \(\lambda_1, \dots, \lambda_m,\) respectively. Suppose that \(\lambda_1, \dots, \lambda_m\) are all different. Then \(\mathbf{v}_1, \dots, \mathbf{v}_m\) are linearly independent.
This theorem says that if we take one eigenvector from each eigenspace, then the chosen vectors will be linearly independent.
Proof of the theorem
Suppose, for the sake of contradiction, that there is a nontrivial linear relation \[c_1 \mathbf{v}_1 + \dotsm + c_m \mathbf{v}_m = \mathbf{0} \qquad (*)\] between the chosen vectors (remember, nontrivial means the coefficients are not all zero). In order to use the fact that \(\mathbf{v}_1, \dots, \mathbf{v}_m\) are eigenvectors, let's multiply both sides of this linear relation (on the left) by the matrix \(A.\) This gives \[c_1 A\mathbf{v}_1 + \dotsm + c_m A\mathbf{v}_m = \mathbf{0}.\] But we know that \(A\mathbf{v}_1 = \lambda_1 \mathbf{v}_1\) and so on for the other eigenvectors, so we can simplify this to \[c_1 \lambda_1 \mathbf{v}_1 + \dotsm + c_m \lambda_m \mathbf{v}_m = \mathbf{0}. \qquad (**)\] This gives a different linear relation between \(\mathbf{v}_1, \dots, \mathbf{v}_m.\)
Now the key step: let's combine our two relations in order to eliminate one of the vectors. Specifically, let's take equation \((**)\) and subtract \(\lambda_m\) times equation \((*).\) This gives \[(c_1 \lambda_1 \mathbf{v}_1 + \dotsm + c_m \lambda_m \mathbf{v}_m) - \lambda_m\left(c_1 \mathbf{v}_1 + \dotsm + c_m \mathbf{v}_m\right) = \mathbf{0},\] which we can rewrite as \[c_1 (\lambda_1 - \lambda_m) \mathbf{v}_1 + \dotsm + c_{m-1} (\lambda_{m-1} - \lambda_m) \mathbf{v}_{m-1} = \mathbf{0}. \qquad (***)\] Notice that the two terms with \(\mathbf{v}_m\) canceled out, and we are now left with a linear relation between \(\mathbf{v}_1, \dots, \mathbf{v}_{m-1}.\)
There's one thing we haven't used yet: that the eigenvalues \(\lambda_1, \dots, \lambda_m\) are different. This means that the differences \(\lambda_1 - \lambda_m,\) \(\dots,\) \(\lambda_{m-1} - \lambda_m\) are nonzero. It follows that the relation \((***)\) is nontrivial: not all its coefficients are zero. (Why? The original relation \((*)\) was nontrivial, so it had at least one nonzero coefficient \(c_i.\) Then, in the new relation \((***),\) the coefficient on \(\mathbf{v}_i\) is \(c_i(\lambda_i - \lambda_m),\) which is nonzero because both \(c_i\) and \(\lambda_i - \lambda_m\) are nonzero.)
Let's now step back and see what we just did. We started with a nontrivial linear relation between \(\mathbf{v}_1, \dots, \mathbf{v}_m\) and, by using the fact that these are eigenvectors, we were able to derive a nontrivial linear relation between \(\mathbf{v}_1, \dots, \mathbf{v}_{m-1},\) a list that is one shorter. Well... what if we do this again?
That is, starting with the nontrivial linear relation between \(\mathbf{v}_1, \dots, \mathbf{v}_{m-1},\) we can apply the same process as we just did to eliminate \(\mathbf{v}_{m-1}\) and end up with a nontrivial linear relation between \(\mathbf{v}_1, \dots, \mathbf{v}_{m-2}.\) We can keep going and going, at each step eliminating the last vector in the list. Eventually, after repeating this process \(m-1\) times in total, we'll end up with a nontrivial linear relation involving just \(\mathbf{v}_1\) on its own: something like \[d \mathbf{v}_1 = \mathbf{0}\] where \(d \neq 0.\) But that's not possible: \(\mathbf{v}_1\) is an eigenvector, so it's nonzero, and so is any nonzero scalar multiple of it. This is the desired contradiction. Phew!
(Note: This technically isn't quite enough for our purposes, because we may want to take more than one vector from a given eigenspace. However, it is possible to extend the theorem to that case; I won't write out all the details.)
We now have a recipe to construct an eigenbasis for \(A\):
Let's see an example.
Example (computing an eigenbasis)
Let \(A = \begin{bmatrix} 2 & 3 & 3 \\ 0 & -1 & 0 \\ 3 & 3 & 2 \end{bmatrix}.\) Previously we found that \(A\) has two distinct eigenspaces: \[E_{-1}(A) = \text{span}\left( \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \; \begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}\right) \qquad \text{and} \qquad E_5(A) = \text{span}\left( \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}\right).\] There are a total of 3 vectors in these bases, and the theorem we just proved guarantees that they are linearly independent. Therefore, \[ \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \quad \begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}, \quad \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} \] is an eigenbasis for \(A.\)
You may wonder when \(A\) will fail to have an eigenbasis. Essentially, this occurs when \(A\) does not have "enough" eigenvectors to fill a basis for \(\mathbb{R}^n.\) There are two possible ways that this can happen:
If not all of the roots of \(p_A(\lambda)\) are real numbers (some are nonreal complex numbers), then the sum of the multiplicities of those roots will be strictly less than \(n.\) In other words, the sum of the algebraic multiplicities of the eigenvalues of \(A\) will be less than \(n.\) However, we know that for each eigenvalue \(\lambda,\) its geometric multiplicity is less than or equal to its algebraic multiplicity. Therefore, the sum of the geometric multiplicities will also be less than \(n.\)
Now remember that the geometric multiplicity of \(\lambda\) equals the dimension of \(E_\lambda(A)\); that is, it equals the maximum number of linearly independent vectors we can find in \(E_\lambda(A).\) Thus, if the sum of the geometric multiplicities is less than \(n,\) then there's no way to find \(n\) linearly independent vectors by only taking vectors inside the eigenspaces of \(A.\)
Example (nonreal roots of the characteristic polynomial)
Let \(A = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}.\) Then we have \[p_A(\lambda) = \left| \begin{matrix} -\lambda & -1 \\ 1 & -\lambda \end{matrix} \right| = \lambda^2 + 1.\] This polynomial has no real roots. (Its complex roots are \(\lambda = \pm i.\)) Therefore, \(A\) has no eigenvalues or eigenvectors at all, since we are restricting ourselves to real numbers; the sum of the algebraic multiplicities is zero.
(Note: If we were using complex numbers as scalars and in our vectors, this would not be an issue. The fundamental theorem of algebra says that a polynomial of degree \(n\) always has \(n\) complex roots, counted with multiplicity. So, over the complex numbers, the sum of the algebraic multiplicities of a matrix is always \(n.\) It's just that, since we're restricting ourselves to real numbers, we can't "see" some of those multiplicities.)
Even if Reason 1 does not apply — that is, the roots of \(p_A(\lambda)\) are all real numbers — it's still not guaranteed that \(A\) has an eigenbasis. In the case that \(p_A(\lambda)\) has all real roots, the sum of the algebraic multiplicities of \(A\) will be exactly \(n.\) However, we saw earlier that some of the geometric multiplicities may be strictly less than their corresponding algebraic multiplicities, in which case the sum of the geometric multiplicities will again be less than \(n.\)
Example (algebraic vs. geometric multiplicity, again)
Let \(A = \begin{bmatrix} 0 & -3 \\ 3 & 6 \end{bmatrix}.\) Previously, we found that \(A\) only has one eigenvalue, \(\lambda = 3,\) with algebraic multiplicity \(2.\) However, we found that its eigenspace is \[E_3(A) = \text{span}\left( \begin{bmatrix} 1 \\ -1 \end{bmatrix} \right),\] so its geometric multiplicity is only \(1.\) Therefore, \(A\) does not have an eigenbasis: all the eigenvectors for \(A\) only span a single line through the origin in \(\mathbb{R}^2.\)
The logic we have just seen essentially constitutes a proof of the following theorem:
Theorem (eigenbases and geometric multiplicities)
Let \(A\) be an \(n \times n\) matrix. Then \(A\) has an eigenbasis if and only if the sum of the geometric multiplicities of \(A\) is equal to \(n.\)