Exercise
Compute the matrix multiplication \[\begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix} \begin{bmatrix} 4 & -3 \\ -1 & 2 \end{bmatrix}.\]
Last time we (finally!) defined matrix multiplication and saw how it is equivalent to composition of linear transformations. However, there were many warnings about matrix multiplication, including that there is no general way to "divide" by a matrix. Today we'll look at a special kind of square matrix, an invertible matrix, where it is actually possible to "divide," or rather, multiply by an inverse.
Let's start by thinking about invertible linear transformations, which will correspond to invertible matrices.
Suppose that we have a linear transformation \(T \colon \mathbb{R}^m \to \mathbb{R}^n\) that is both one-to-one (injective) and onto (surjective). Notice that this can only happen if \(m=n\), since as we learned in a previous class, if \(m<n\) then \(T\) cannot be onto, and if \(m>n\) then \(T\) cannot be one-to-one.
Definition (invertible linear transformation)
A linear transformation \(T \colon \mathbb{R}^n \to \mathbb{R}^n\) is invertible if it is both one-to-one and onto.
Since \(T\) goes from \(\mathbb{R}^n\) to the same \(\mathbb{R}^n\), the unifying theorem tells us that \(T\) is one-to-one if and only if it is onto. So, if \(T \colon \mathbb{R}^n \to \mathbb{R}^n\) is one-to-one, then it is invertible; and if \(T\) is onto, then it is also invertible.
If \(T\) is invertible, this means that for all \(\mathbf{y} \in \mathbb{R}^n,\) there is exactly one \(\mathbf{x} \in \mathbb{R}^n\) that maps to it: \(T(\mathbf{x}) = \mathbf{y}.\) (Remember that one-to-one means that there is at most one \(\mathbf{x}\), and onto means that there is at least one.) In this case, \(T\) has an inverse function, denoted \(T^{-1} \colon \mathbb{R}^n \to \mathbb{R}^n.\) The defining property of the inverse is that \[ T(\mathbf{x}) = \mathbf{y} \qquad \text{if and only if} \qquad \mathbf{x} = T^{-1}(\mathbf{y}). \] An alternative way to characterize the inverse function is via composition: \(T^{-1} \circ T\) and \(T \circ T^{-1}\) are both the identity map from \(\mathbb{R}^n\) to itself. That is, \[ T^{-1}(T(\mathbf{x})) = T(T^{-1}(\mathbf{x})) = \mathbf{x}\] for all \(\mathbf{x} \in \mathbb{R}^n.\)
As a technical detail, just like we did with the composition of linear transformations, we should make sure that the inverse of a linear transformation is also a linear transformation:
Fact (inverse of linear transformation is linear transformation)
If \(T \colon \mathbb{R}^n \to \mathbb{R}^n\) is an invertible linear transformation, then its inverse \(T^{-1} \colon \mathbb{R}^n \to \mathbb{R}^n\) is also linear.
To see why this is true, let's show that \(T^{-1}\) respects addition, one of the defining properties of a linear transformation. The idea is to cleverly use the fact that \(T^{-1} \circ T\) and \(T \circ T^{-1}\) are the identity map, along with the fact that \(T\) is a linear transformation. To start, we use the fact that \(T\) respects addition: \[ T(\mathbf{x} + \mathbf{y}) = T(\mathbf{x}) + T(\mathbf{y})\] for all \(\mathbf{x}\) and \(\mathbf{y}.\) Since these are equal, we can apply the function \(T^{-1}\) to both sides. On the left-hand side, the \(T^{-1}\) will cancel with the \(T\), and we get \[\mathbf{x} + \mathbf{y} = T^{-1}\left(T(\mathbf{x}) + T(\mathbf{y})\right),\] still for all \(\mathbf{x}, \mathbf{y}.\) Now, we make the substitution \(\mathbf{u} = T(\mathbf{x})\) and \(\mathbf{v} = T(\mathbf{y})\), which by definition of the inverse means that \(\mathbf{x} = T^{-1}(\mathbf{u})\) and \(\mathbf{y} = T^{-1}(\mathbf{v}).\) Then, our equation becomes \[T^{-1}(\mathbf{u}) + T^{-1}(\mathbf{v}) = T^{-1}(\mathbf{u} + \mathbf{v}),\] which is exactly the statement that \(T^{-1}\) respects addition. You can show in a similar way that \(T^{-1}\) respects scalar multiplication, but I won't write out the details.
Now, we know that if \(T\) is an invertible linear transformation, then so is \(T^{-1}.\) (Notice that the inverse of \(T^{-1}\) is just \(T\) itself; an invertible function and its inverse come in pairs.) This means that \(T^{-1}\) corresponds to some matrix, which we will call the inverse of the matrix corresponding to \(T.\)
Definition (inverse of a matrix)
An \(n \times n\) matrix \(A\) is invertible if the corresponding linear transformation \(T \colon \mathbb{R}^n \to \mathbb{R}^n\) is invertible (one-to-one and onto).
If \(A\) is invertible, the inverse of \(A\) is the \(n \times n\) matrix \(A^{-1}\) corresponding to \(T^{-1},\) the inverse of \(T.\)
If we want to work only with matrices, we can say that \(A^{-1}\) is the (unique) \(n \times n\) matrix such that \[ A^{-1}A = AA^{-1} = I_n, \] where \(I_n = \begin{bmatrix} 1 & 0 & \dotsm & 0 \\ 0 & 1 & \dotsm & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dotsm & 1 \end{bmatrix}\) is the \(n \times n\) identity matrix. This is because matrix multiplication corresponds to function composition.
Let's quickly stop and add to our unifying theorem:
The Unifying Theorem, version 3
Let \(A\) be an \(n \times n\) matrix with columns \(\mathbf{v}_1, \dots, \mathbf{v}_n\). Let \(T \colon \mathbb{R}^n \to \mathbb{R}^n\) be the linear transformation defined by \(T(\mathbf{x}) = A\mathbf{x}.\) Then, the following are equivalent:
(i) \(\mathbf{v}_1, \dots, \mathbf{v}_n\) span \(\mathbb{R}^n\).
(ii) \(\mathbf{v}_1, \dots, \mathbf{v}_n\) are linearly independent.
(iii) For all \(\mathbf{b} \in \mathbb{R}^n\), the linear system \(A\mathbf{x} = \mathbf{b}\) has exactly one solution.
(iv) \(T\) is one-to-one.
(v) \(T\) is onto.
(vi) \(A\) is invertible.
Given a matrix \(A\), to check that another matrix \(B\) is the inverse of \(A\), you only need to compute \(AB\) and \(BA\) and make sure that both equal the identity matrix. Actually, you only need to compute one of them, because of the following fact:
Fact
Let \(A\) and \(B\) be \(n \times n\) matrices. If \(AB = I_n,\) then \(BA = I_n\) too, and so \(A\) and \(B\) are inverses of each other.
Remember that matrix multiplication is not commutative in general, not even for square matrices! So this fact should be a little surprising.
As promised, invertible matrices are a kind of matrix that you can "divide" by, in a sense. However, when we would want to "divide" by an invertible matrix, what we should actually do is multiply by its inverse.
Example (cancellation of invertible matrix)
Suppose that \[AC = BC,\] where \(A, B, C\) are matrices. Normally we could not conclude that \(A = B\) from this information alone. However, if \(C\) is invertible, then we can take this equation and multiply both sides on the right by \(C^{-1}\). (Because matrix multiplication is not commutative, we need to specify that we are multiplying on the right!) This gives us \[ ACC^{-1} = BCC^{-1}.\] Now, since \(CC^{-1}\) is the identity matrix \(I_n\), we get \(AI_n = BI_n.\) But this is just \(A = B,\) because \(I_n\) is the identity for matrix multiplication. So, by multiplying by the inverse of \(C\), we were able to cancel it from both sides.
Example (invertible linear system)
Suppose we have a linear system of equations in matrix form \(A\mathbf{x} = \mathbf{b},\) and suppose that \(A\) is an invertible \(n \times n\) matrix. Then we can multiply on the left by the inverse \(A^{-1},\) which gives \[A^{-1} A \mathbf{x} = A^{-1} \mathbf{b}.\] As before, \(A^{-1}\) and \(A\) cancel out, so we get a unique solution, \[\mathbf{x} = A^{-1} \mathbf{b}.\] There are a couple of consequences here: first, if \(A\) is invertible, then \(A \mathbf{x} = \mathbf{b}\) always has a unique solution for all \(\mathbf{b}.\) Second, if \(A\) is invertible and we can calculate its inverse \(A^{-1}\), then we can instantly solve any system of the form \(A \mathbf{x} = \mathbf{b}\).
Note: In a way, this is how "division" of real numbers actually works. If I have a linear equation \(ax = b\) and \(a\) is nonzero, then to solve for \(x\) we usually say to "divide" both sides by \(a.\) Another way to think of this, however, is to multiply both sides by the "inverse" of \(a\), AKA the reciprocal \(\frac{1}{a} = a^{-1}.\) Then we get \(a^{-1}ax = a^{-1}b.\) Since \(a^{-1}a = 1,\) we can simplify to get the solution \(x = a^{-1}b,\) which of course we usually write as \(x = \frac{b}{a}.\)
Now that we see some of the uses of matrix inverses, we should discuss how to actually compute the inverse of a matrix.
Let's begin with the smallest nontrivial case, that of a \(2 \times 2\) matrix. Luckily for us, it is very easy to tell when a \(2 \times 2\) matrix is invertible, and to compute its inverse when it is:
Theorem (inverse of a 2x2 matrix)
A \(2 \times 2\) matrix \(A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\) is invertible if and only if \(ad - bc \neq 0.\) Furthermore, if \(A\) is invertible, then its inverse is \[ A^{-1} = \frac{1}{ad-bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}. \]
As a quick check, let's suppose that \(ad-bc \neq 0\) and compute \(A A^{-1}\): \[AA^{-1} = \begin{bmatrix} a & b \\ c & d \end{bmatrix} \cdot \frac{1}{ad-bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix} = \frac{1}{ad-bc} \begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix} = \frac{1}{ad-bc} \begin{bmatrix} ad-bc & 0 \\ 0 & ad-bc \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}.\] This is the identity matrix \(I_2,\) so we know that the formula does give us the inverse of \(A.\) (It turns out that \(ad-bc \neq 0\) is the exact condition needed to ensure that the columns of \(A\) are linearly independent, which means \(A\) is invertible by the unifying theorem.)
This theorem gives us an easy way to solve any \(2 \times 2\) system corresponding to an invertible matrix:
Example (linear system via inverse)
Consider the linear system \[ \left\{ \begin{array}\ 2x_1 &+& 3x_2 &=& 1 \\ x_1 &+& 4x_2 &=& -2 \end{array} \right. \] In matrix form, this is \[ \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}.\] To solve this system, we can multiply on the left by the inverse of \(\begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix}.\) The formula tells us that the inverse is \[ \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix}^{-1} = \frac{1}{(2)(4) - (1)(3)} \begin{bmatrix} 4 & -3 \\ -1 & 2 \end{bmatrix} = \frac{1}{5} \begin{bmatrix} 4 & -3 \\ -1 & 2 \end{bmatrix}. \] Therefore, the unique solution to the system is given by \[ \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \frac{1}{5} \begin{bmatrix} 4 & -3 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 2 \\ -1 \end{bmatrix}. \]
In general, there are formulas for the inverse of a matrix, but they get exponentially more complicated as the size of the matrix increases. Instead, we now describe a general (and, I think, very beautiful) method to compute inverses.
Let \(A\) be an invertible \(n \times n\) matrix. To compute \(A^{-1},\) we can view it as a linear transformation. Then we know that the columns of the matrix are given by the images of the standard basis vectors under that linear transformation. In other words, we have \[ A^{-1} = \begin{bmatrix} | & | & & | \\ A^{-1}\mathbf{e}_1 & A^{-1}\mathbf{e}_2 & \dotsm & A^{-1}\mathbf{e}_n \\ | & | & & | \end{bmatrix} .\] So to compute \(A^{-1},\) it suffices to compute \(A^{-1} \mathbf{e}_i\) for all \(i = 1, 2, \dots, n.\)
But what is \(A^{-1} \mathbf{e}_i\), and how can we compute it? Well, as we saw earlier, this is the unique solution to the linear system given by \(A\mathbf{x} = \mathbf{e}_i.\) So, to compute \(A^{-1} \mathbf{e}_i\), we can simply solve this linear system using the methods we've learned. In particular, we can take the augmented matrix \( [A \;\;|\;\; \mathbf{e}_i] \) and apply Gauss-Jordan elimination to get it into reduced echelon form. Because \(A\) is invertible, we know that \(\text{rref}(A)\) must have a pivot in every row and every column. But \(A\) is square, so the only possibility is that \(\text{rref}(A) = I_n.\) Therefore, Gauss-Jordan elimination will transform this augmented matrix like so: \[ [A \;\;|\;\; \mathbf{e}_i] \longrightarrow \dotsm \longrightarrow [ I_n \;\;|\;\; A^{-1} \mathbf{e}_i ]. \] So, in theory, doing this with each \(\mathbf{e}_i\) will give us the columns of \(A^{-1}\) one at a time.
But we don't need to do it one at a time! If we construct an augmented matrix \[ [A \;\;|\;\; \mathbf{e}_1 \;\; \mathbf{e}_2 \;\; \dotsm \;\; \mathbf{e}_n ] \] with all the standard basis vectors in a row, then we can just row reduce a single time, and each column on the right-hand side will be the corresponding column of \(A^{-1}.\) Furthermore, notice what we're adding on the right side of \(A\): it's just the identity matrix \(I_n\)! This gives us an algorithm for computing the inverse:
Procedure (computing the inverse of a matrix)
Given an \(n \times n\) matrix \(A\), to compute its inverse:
- Form an augmented matrix by putting a copy of the identity matrix on the right: \( [ A \;\;| \;\; I_n ]. \)
- Perform row operations on this augmented matrix to get \(A\) into reduced echelon form.
- If \(A\) is invertible, the result will be \( [ I_n \; \; | \; \; A^{-1} ],\) with \(A^{-1}\) on the right.
- If \(A\) is not invertible, then the left half of the matrix will not be \(I_n\) and this procedure will (obviously) fail to compute \(A^{-1}\).
To see this in action, let's compute the inverse of the \(2 \times 2\) matrix from earlier using this method:
Example (computing inverse by row reduction)
To compute the inverse of \(\begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix},\) we form the augmented matrix \( \left[ \begin{matrix} 2 & 3 \\ 1 & 4 \end{matrix} \; \left| \; \begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix} \right. \right] \) and use Gauss-Jordan elimination to transform the left half into reduced echelon form: \[\begin{aligned}
\left[ \begin{matrix} 2 & 3 \\ 1 & 4 \end{matrix} \; \left| \; \begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix} \right. \right] \quad
\stackrel{-\frac{1}{2}R_1 + R_2 \to R_2}\longrightarrow &\quad\left[ \begin{matrix} 2 & 3 \\ 0 & \frac{5}{2} \end{matrix} \; \left| \; \begin{matrix} 1 & 0 \\ -1/2 & 1 \end{matrix} \right. \right] \\\\
\stackrel{\frac{2}{5}R_2 \to R_2}\longrightarrow &\quad\left[ \begin{matrix} 2 & 3 \\ 0 & 1 \end{matrix} \; \left| \; \begin{matrix} 1 & 0 \\ -1/5 & 2/5 \end{matrix} \right. \right] \\\\
\stackrel{R_1 - 3R_2 \to R_1}\longrightarrow &\quad\left[ \begin{matrix} 2 & 0 \\ 0 & 1 \end{matrix} \; \left| \; \begin{matrix} 8/5 & -6/5 \\ -1/5 & 2/5 \end{matrix} \right. \right] \\\\
\stackrel{\frac{1}{2}R_1 \to R_1}\longrightarrow &\quad\left[ \begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix} \; \left| \; \begin{matrix} 4/5 & -3/5 \\ -1/5 & 2/5 \end{matrix} \right. \right].
\end{aligned}\] Therefore the inverse is \(\begin{bmatrix} 4/5 & -3/5 \\ -1/5 & 2/5 \end{bmatrix} = \dfrac{1}{5} \begin{bmatrix} 4 & -3 \\ -1 & 2 \end{bmatrix},\) which matches what we found earlier.