An Inquiry-Based Introduction to Linear Algebra and Applications

Subsection Application: Modeling the Second Law of Thermodynamics

Pour cream into your cup of coffee and the cream spreads out; straighten up your room and it soon becomes messy again; when gasoline is mixed with air in a car's cylinders, it explodes if a spark is introduced. In each of these cases a transition from a low energy state (your room is straightened up) to a higher energy state (a messy, disorganized room) occurs. This can be described by entropy — a measure of the energy in a system. Low energy is organized (like ice cubes) and higher energy is not (like water vapor). It is a fundamental property of energy (as described by the second law of thermodynamics) that the entropy of a system cannot decrease. In other words, in the absence of any external intervention, things never become more organized.

The Ehrenfest model³⁴ is a Markov process proposed to explain the statistical interpretation of the second law of thermodynamics using the diffusion of gas molecules. This process can be modeled as a problem of balls and bins, as we will do later in this section. The characteristic polynomial of the transition matrix will help us find the eigenvalues and allow us to analyze our model.

Subsection Introduction

We have seen that the eigenvalues of an \(n \times n\) matrix \(A\) are the scalars \(\lambda\) so that \(A - \lambda I_n\) has a nontrivial null space. Since a matrix has a nontrivial null space if and only if the matrix is not invertible, we can also say that \(\lambda\) is an eigenvalue of \(A\) if

This equation is called the characteristic equation of \(A\text{.}\) It provides us an algebraic way to find eigenvalues, which can then be used in finding eigenvectors corresponding to each eigenvalue. Suppose we want to find the eigenvalues of \(A=\left[ \begin{array}{cc} 1 \amp 1 \\ 1\amp 3 \end{array} \right]\text{.}\) Note that

with determinant \((1-\lambda)(3-\lambda)-1=\lambda^2-4\lambda+2\text{.}\) Hence, the eigenvalues \(\lambda_1, \lambda_2\) are the solutions of the characteristic equation \(\lambda^2-4\lambda+2=0\text{.}\) Using quadratic formula, we find that \(\lambda_1=2+\sqrt{2}\) and \(\lambda_2=2-\sqrt{2}\) are the eigenvalues.

In this activity, our goal will be to use the characteristic equation to obtain information about eigenvalues and eigenvectors of a matrix with real entries.

Subsection The Characteristic Equation

Until now, we have been given eigenvalues or eigenvectors of a matrix and determined eigenvectors and eigenvalues from the known information. In this section we use determinants to find (or approximate) the eigenvalues of a matrix. From there we can find (or approximate) the corresponding eigenvectors. The tool we will use is a polynomial equation, the characteristic equation, of a square matrix whose roots are the eigenvalues of the matrix. The characteristic equation will then provide us with an algebraic way of finding the eigenvalues of a square matrix.

We have seen that the eigenvalues of a square matrix \(A\) are the scalars \(\lambda\) so that \(A - \lambda I\) has a nontrivial null space. Since a matrix has a nontrivial null space if and only if the matrix is not invertible, we can also say that \(\lambda\) is an eigenvalue of \(A\) if

Note that if \(A\) is an \(n \times n\) matrix, then \(\det(A - \lambda I)\) is a polynomial of degree \(n\text{.}\) Furthermore, if \(A\) has real entries, the polynomial has real coefficients. This polynomial, and the equation (18.2) are given special names.

So the characteristic equation of \(A\) gives us an algebraic way of finding the eigenvalues of \(A\text{.}\)

As we argued in Preview Activity 18.1, a \(2 \times 2\) matrix can have at most 2 eigenvalues. For an \(n \times n\) matrix, the characteristic polynomial will be a degree \(n\) polynomial, and we know from algebra that a degree \(n\) polynomial can have at most \(n\) roots. Since an eigenvalue of a matrix is a root of the characteristic polynomial of that matrix, we can conclude that an \(n \times n\) matrix can have at most \(n\) distinct eigenvalues. Activity 18.2 (b) shows that a \(4 \times 4\) matrix may have fewer than \(4\) eigenvalues, however. Note that one of these eigenvalues, the eigenvalue 1, appears three times as a root of the characteristic polynomial of the matrix. The number of times an eigenvalue appears as a root of the characteristic polynomial is called the (algebraic) multiplicity of the eigenvalue. More formally:

Thus, in Activity 18.2 (b) the eigenvalue 1 has multiplicity 3 and the eigenvalue 2 has multiplicity 1. Notice that if we count the eigenvalues of an \(n \times n\) matrix with their multiplicities, the total will always be \(n\text{.}\)

If \(A\) is a matrix with real entries, then the characteristic polynomial will have real coefficients. It is possible that the characteristic polynomial can have complex roots, and that the matrix \(A\) has complex eigenvalues. The Fundamental Theorem of Algebra shows us that if a real matrix has complex eigenvalues, then those eigenvalues will appear in conjugate pairs, i.e., if \(\lambda_1=a+ib\) is an eigenvalue of \(A\text{,}\) then \(\lambda_2=a-ib\) is another eigenvalue of \(A\text{.}\) Furthermore, for an odd degree polynomial, since the complex eigenvalues will come in conjugate pairs, we will be able to find at least one real eigenvalue.

We now summarize the information we have so far about eigenvalues of an \(n\times n\) real matrix:

Subsection Eigenspaces, A Geometric Example

Recall that for each eigenvalue \(\lambda\) of an \(n \times n\) matrix \(A\text{,}\) the eigenspace of \(A\) corresponding to the eigenvalue \(\lambda\) is \(\Nul (A - \lambda I_n)\text{.}\) These eigenspaces can tell us important information about the matrix transformation defined by \(A\text{.}\) For example, consider the matrix transformation \(T\) from \(\R^3\) to \(\R^3\) defined by \(T(\vx) = A \vx\text{,}\) where

We are interested in understanding what this matrix transformation does to vectors in \(\R^3\text{.}\) First we note that \(A\) has eigenvalues \(\lambda_1 = 1\) and \(\lambda_2 = 2\text{,}\) with \(\lambda_1\) having multiplicity \(2\text{.}\) There is a pair \(\vv_1 = \left[ \begin{array}{c} 1\\0\\0 \end{array} \right]\) and \(\vv_2 = \left[ \begin{array}{c} 0\\1\\0 \end{array} \right]\) of linearly independent eigenvectors for \(A\) corresponding to the eigenvalue \(\lambda_1\) and an eigenvector \(\vv_3=\left[ \begin{array}{c} 1\\1\\1 \end{array} \right]\) for \(A\) corresponding to the eigenvalue \(\lambda_2\text{.}\) Note that the vectors \(\vv_1\text{,}\) \(\vv_2\text{,}\) and \(\vv_3\) are linearly independent (recall from Theorem that eigenvectors corresponding to different eigenvalues are always linearly independent). So any vector \(\vb\) in \(\R^3\) can be written uniquely as a linear combination of \(\vv_1\text{,}\) \(\vv_2\text{,}\) and \(\vv_3\text{.}\) Let's now consider the action of the matrix transformation \(T\) on a linear combination of \(\vv_1\text{,}\) \(\vv_2\text{,}\) and \(\vv_2\text{.}\) Note that

Equation (18.3) illustrates that it is most convenient to view the action of \(T\) in the coordinate system where \(\Span \{\vv_1\}\) serves as the \(x\)-axis, \(\Span \{\vv_2\}\) serves as the \(y\)-axis, and \(\Span \{\vv_3\}\) as the \(z\)-axis. In this case, we can visualize that when we apply the transformation \(T\) to a vector \(\vb = c_1 \vv_1 + c_2 \vv_2 + c_3 \vv_3\) in \(\R^3\) the result is an output vector that is unchanged in the \(\vv_1\)-\(\vv_2\) plane and scaled by a factor of \(2\) in the \(\vv_3\) direction. For example, consider the box whose sides are determined by the vectors \(\vv_1\text{,}\) \(\vv_2\text{,}\) and \(\vv_3\) as shown in Figure 18.4. The transformation \(T\) stretches this box by a factor of \(2\) in the \(\vv_3\) direction and leaves everything else alone, as illustrated in Figure 18.4. So the entire \(\Span \{\vv_1, \vv_2\})\) is unchanged by \(T\text{,}\) but \(\Span \{\vv_3\})\) is scaled by \(2\text{.}\) In this situation, the eigenvalues and eigenvectors provide the most convenient perspective through which to visualize the action of the transformation \(T\text{.}\)

This geometric perspective illustrates how each eigenvalue and the corresponding eigenspace of \(A\) tells us something important about \(A\text{.}\) So it behooves us to learn a little more about eigenspaces.

Subsection Dimensions of Eigenspaces

There is a connection between the dimension of the eigenspace of a matrix corresponding to an eigenvalue and the multiplicity of that eigenvalue as a root of the characteristic polynomial. Recall that the dimension of a subspace of \(\R^n\) is the number of vectors in a basis for the eigenspace. We investigate the connection between dimension and multiplicity in the next activity.

The examples in Activity 18.3 all provide instances of the principle that the dimension of an eigenspace corresponding to an eigenvalue \(\lambda\) cannot exceed the multiplicity of \(\lambda\text{.}\) Specifically:

The examples we have seen raise another important point. The matrix \(A = \left[ \begin{array}{ccc} 1\amp 0\amp 1\\0\amp 1\amp 1\\0\amp 0\amp 2 \end{array} \right]\) from our geometric example has two eigenvalues \(1\) and \(2\text{,}\) with the eigenvalue 1 having multiplicity 2. If we let \(E_{\lambda}\) represent the eigenspace of \(A\) corresponding to the eigenvalue \(\lambda\text{,}\) then \(\dim(E_1)=2\) and \(\dim(E_2) = 1\text{.}\) If we change this matrix slightly to the matrix \(B = \left[ \begin{array}{crc} 2\amp 0\amp 1 \\ 0\amp 1\amp 1 \\ 0\amp 0\amp 1 \end{array} \right]\) we see that \(B\) has two eigenvalues \(1\) and \(2\text{,}\) with the eigenvalue 1 having multiplicity 2. However, in this case we have \(\dim(E_1) = 1\) (like the example in from Activity 18.2 (a) and Activity 18.3 (a)). In this case the vector \(\vv_1 = [1 \ 0 \ 0]^{\tr}\) forms a basis for \(E_2\) and the vector \(\vv_2 = [0 \ 1 \ 0]^{\tr}\) forms a basis for \(E_1\text{.}\) We can visualize the action of \(B\) on the square formed by \(\vv_1\) and \(\vv_2\) in the \(xy\)-plane as a scaling by 2 in the \(\vv_1\) direction as shown in Figure 18.6, but since we do not have a third linearly independent eigenvector, the action of \(B\) in the direction of \([0 \ 0 \ 1]^{\tr}\) is not so clear.

So the action of a matrix transformation can be more easily visualized if the dimension of each eigenspace is equal to the multiplicity of the corresponding eigenvalue. This geometric perspective leads us to define the geometric multiplicity of an eigenvalue.

Subsection Examples

What follows are worked examples that use the concepts from this section.

Subsection Summary

In this section we studied the characteristic polynomial of a matrix and similar matrices.

• If \(A\) is an \(n \times n\) matrix, the characteristic polynomial of \(A\) is the polynomial

  \begin{equation*} \det(A-\lambda I_n)\text{,} \end{equation*}

  where \(I_n\) is the \(n \times n\) identity matrix.

• If \(A\) is an \(n \times n\) matrix, the characteristic equation of \(A\) is the equation

  \begin{equation*} \det(A-\lambda I_n) = 0\text{.} \end{equation*}

• The characteristic equation of a square matrix provides us an algebraic method to find the eigenvalues of the matrix.

• The eigenvalues of an upper or lower-triangular matrix are the entries on the diagonal.

• There are at most \(n\) eigenvalues of an \(n\times n\) matrix.

• For a real matrix \(A\text{,}\) if an eigenvalue \(\lambda\) of \(A\) is complex, then the complex conjugate of \(\lambda\) is also an eigenvalue.

• The algebraic multiplicity of an eigenvalue \(\lambda\) is the multiplicity of \(\lambda\) as a root of the characteristic equation.

• The dimension of the eigenspace corresponding to an eigenvalue \(\lambda\) is less than or equal to the algebraic multiplicity of \(\lambda\text{.}\)

Subsection Project: The Ehrenfest Model

To realistically model the diffusion of gas molecules we would need to consider a system with a large number of balls as substitutes for the gas molecules. However, the main idea can be seen in a model with a much smaller number of balls, as we will do now. Suppose we have two bins that contain a total of \(4\) balls between them. Label the bins as Bin 1 and Bin 2. In this case we can think of entropy as the number of different possible ways the balls can be arranged in the system. For example, there is only \(1\) way for all of the balls to be in Bin 1 (low entropy), but there are \(4\) ways that we can have one ball in Bin 1 (choose any one of the four different balls, which can be distinguished from each other) and \(3\) balls in Bin 2 (higher entropy). The highest entropy state has the balls equally distributed between the bins (with \(6\) different ways to do this).

We assume that there is a way for balls to move from one bin to the other (like having gas molecules pass through a permeable membrane). A way to think about this is that we select a ball (from ball 1 to ball 4, which are different balls) and move that ball from its current bin to the other bin. Consider a “move” to be any instance when a ball changes bins. A state is any configuration of balls in the bins at a given time, and the state changes when a ball is chosen at random and moved to the other bin. The possible states are to have 0 balls in Bin 1 and 4 balls in Bin 2 (State 0, entropy 1), 1 ball in Bin 1 and 3 in Bin 2 (State 1, entropy 4), 2 balls in each Bin (State 2, entropy 6), 3 balls in Bin 1 and 1 ball in Bin 2 (State 3, entropy 4), and 4 balls in Bin 1 and 0 balls in Bin 2 (State 4, entropy 1). These states are shown in Figure 18.10.

The system we developed in Project Activity 18.4 has matrix form

where \(T\) is the transition matrix

Subsequent moves give probability distribution vectors

This example is an example of a Markov process (see Definition 9.4). There are several questions we can ask about this model. For example, what is the long-term behavior of this system, and how does this model relate to entropy? That is, given an initial probability distribution vector \(\vx_0\text{,}\) the system will have probability distribution vectors \(\vx_1\text{,}\) \(\vx_2\text{,}\) \(\ldots\) after subsequent moves. What happens to the vectors \(\vx_k\) as \(k\) goes to infinity, and what does this tell us about entropy? To answer these questions, we will first explore the sequence \(\{\vx_k\}\) numerically, and then use the eigenvalues and eigenvectors of \(T\) to analyze the sequence \(\{\vx_k\}\text{.}\)

In what follows, we investigate the behavior of the sequence \(\{\vx_k\}\) that we uncovered in Project Activity 18.5.

Now we can analyze the behavior of the sequence \(\{\vx_k\}\text{.}\)

We can now use the eigenvalues and eigenvectors of \(T\) to write the vectors \(\vx_k\) in a convenient form. Let \(\lambda_1 = 0\text{,}\) \(\lambda_2=1\text{,}\) \(\lambda_3=-1\text{,}\) \(\lambda_4=\frac{1}{2}\text{,}\) and \(\lambda_5=-\frac{1}{2}\text{.}\) Notice that

Similarly

We can continue in this manner to ultimately show that for each positive integer \(k\) we have

when \(\vx_0 = \sum_{i=1}^5 a_i \vv_i\text{.}\)