An Inquiry-Based Introduction to Linear Algebra and Applications

Subsection Application: Fitting Functions to Data

Data is all around us. Data is collected on almost everything and it is important to be able to use data to make predictions. However, data is rarely well-behaved and so we generally need to use approximation techniques to estimate from the data. One technique for this is least squares approximations. As we will see, we can use linear algebra to fit a variety of different types of curves to data.

Subsection Introduction

In this section our focus is on fitting linear and polynomial functions to data sets.

Preview Activity 26.1.

NBC was awarded the U.S. television broadcast rights to the 2016 and 2020 summer Olympic games. Table 26.1 lists the amounts paid (in millions of dollars) by NBC sports for the 2008 through 2012 summer Olympics plus the recently concluded bidding for the 2016 and 2020 Olympics, where year 0 is the year 2008. (We will assume a simple model here, ignoring variations such as value of money due to inflation, viewership data which might affect NBC's expected revenue, etc.) Figure 26.2 shows a plot of the data. Our goal in this activity is to find a linear function \(f\) defined by \(f(x) = a_0 + a_1x\) that fits the data well.

Year	Amount
0	894
4	1180
8	1226
12	1418

If the data were actually linear, then the data would satisfy the system

\begin{alignat*}{3} {}a_0 \amp {}+{} \amp {0}a_1 \amp = \amp 894\amp {}\\ {}a_0 \amp {}+{} \amp {4}a_1 \amp = \amp 1180\amp {}\\ {}a_0 \amp {}+{} \amp {8}a_1 \amp = \amp 1226\amp {}\\ {}a_0 \amp {}+{} \amp {12}a_1 \amp = \amp 1418\amp {.} \end{alignat*}

The vector form of this equation is

\begin{equation*} a_0 [1 \ 1 \ 1 \ 1]^{\tr} + a_1[0 \ 4 \ 8 \ 12]^{\tr} = [894 \ 1180 \ 1226 \ 1418]^{\tr}\text{.} \end{equation*}

This equation does not have a solution, so we seek the best approximation to a solution we can find. That is, we want to find \(a_0\) and \(a_1\) so that the line \(f(x) = a_0+a_1x\) provides a best fit to the data.

Letting \(\vv_1 = [1 \ 1 \ 1 \ 1]^{\tr}\) and \(\vv_2 = [0 \ 4 \ 8 \ 12]^{\tr}\text{,}\) and \(\vb = [894 \ 1180 \ 1226 \ 1418]^{\tr}\text{,}\) our vector equation becomes

\begin{equation*} a_0 \vv_1 + a_1\vv_2 = \vb\text{.} \end{equation*}

To make a best fit, we will minimize the square of the distance between \(\vb\) and a vector of the form \(a_0\vv_1 + a_1 \vv_2\text{.}\) That is, minimize

\begin{equation} ||\vb - (a_0 \vv_1 + a_1\vv_2)||^2\text{.}\tag{26.1} \end{equation}

Rephrasing this in terms of projections, we are looking for the vector in \(W = \Span\{\vv_1, \vv_2\}\) that is closest to \(\vb\text{.}\) In other words, the values of \(a_0\) and \(a_1\) will occur as the weights when we write \(\proj_{W} \vb\) as a linear combination of \(\vv_1\) and \(\vv_2\text{.}\) The one wrinkle in this problem is that we need an orthogonal basis for \(W\) to find this projection. Use appropriate technology throughout this activity.

(a)

Find an orthogonal basis \(\CB = \{\vw_1, \vw_2\}\) for \(W\text{.}\)

(b)

Use the basis \(\CB\) to find \(\vy = \proj_{W} \vb\) as illustrated in Figure 26.3.

(c)

Find the values of \(a_0\) and \(a_1\) that give our best fit line by writing \(\vy\) as a linear combination of \(\vv_1\) and \(\vv_2\text{.}\)

(d)

Draw a picture of your line from the previous part on the axes with the data set. How well do you think your line approximates the data? Explain.

Subsection Least Squares Approximations

In Section 25 we saw that the projection of a vector \(\vv\) in \(\R^n\) onto a subspace \(W\) of \(\R^n\) is the best approximation to \(\vv\) of all the vectors in \(W\text{.}\) In fact, if \(\vv = [v_1 \ v_2 \ \ldots \ v_n]^{\tr}\) and \(\proj_W \vv = [w_1 \ w_2 \ w_3 \ \ldots \ w_m]^{\tr}\text{,}\) then the error in approximating \(\vv\) by \(\vw\) is given by

In the context of Preview Activity 26.1, we projected the vector \(\vb\) onto the span of the vectors \(\vv_1 = [1 \ 1 \ 1 \ 1]^{\tr}\) and \(\vv_2 = [0 \ 4 \ 8 \ 12]^{\tr}\text{.}\) The projection minimizes the distance between the vectors in \(W\) and the vector \(\vb\) (as shown in Figure 26.3), and also produces a line which minimizes the sums of the squares of the vertical distances from the line to the data set as illustrated in Figure 26.4 with the olympics data. This is why these approximations are called least squares approximations.

While we can always solve least squares problems using projections, we can often avoid having to create an orthogonal basis when fitting functions to data. We work in a more general setting, showing how to fit a polynomial of degree \(n\) to a set of data points. Our goal is to fit a polynomial \(p(x) = a_0+a_1x+a_2x^2+ \cdots + a_nx^n\) of degree \(n\) to \(m\) data points \((x_1,y_1)\text{,}\) \((x_2,y_2)\text{,}\) \(\ldots\text{,}\) \((x_m,y_m)\text{,}\) no two of which have the same \(x\) coordinate. In the unlikely event that the polynomial \(p(x)\) actually passes through the \(m\) points, then we would have the \(m\) equations

in the \(n+1\) unknowns \(a_0\text{,}\) \(a_1\text{,}\) \(\ldots\text{,}\) \(a_{n-1}\text{,}\) and \(a_n\text{.}\)

The \(m\) data points are known in this situation and the coefficients \(a_0\text{,}\) \(a_1\text{,}\) \(\ldots\text{,}\) \(a_n\) are the unknowns. To write the system in matrix-vector form, the coefficient matrix \(M\) is

while the vectors \(\va\) and \(\vy\) are

Letting \(\vy = [y_1 \ y_2 \ \ldots \ y_m]^{\tr}\) and \(\vv_i = [x_1^{i-1} \ x_2^{i-1} \ \ldots \ x_m^{i-1}]^{\tr}\) for \(1 \leq i \leq n+1\text{,}\) the vector form of the system is

Of course, it is unlikely that the \(m\) data points already lie on a polynomial of degree \(n\text{,}\) so the system will usually have no solution. So instead of attempting to find coefficients \(a_0\text{,}\) \(a_1\text{,}\) \(\ldots\text{,}\) \(a_n\) that give a solution to this system, which may be impossible, we instead look for a vector that is ``close" to a solution. As we have seen, the vector \(\proj_{W} \vy\text{,}\) where \(W\) is the span of the columns of \(M\text{,}\) minimizes the sum of the squares of the differences of the components. That is, our desired approximation to a solution to \(M \vx = \vy\) is the projection of \(\vy\) onto \(\Col M\text{.}\) Now \(\proj_W \vy\) is a linear combination of the columns of \(M\text{,}\) so \(\proj_W \vy = M \va^*\) for some vector \(\va^*\text{.}\) This vector \(\va^*\) then minimizes \(|| \proj_{\perp W} \vy|| = ||\vy - M \va ||\text{.}\) That is, if we let \((M\va)^{\tr} = [b_1 \ b_2 \ b_3 \ \cdots \ b_m]\text{,}\) we are minimizing

The expression \(||\vy - M\va||^2\) measures the error in our approximation.

The question we want to answer is how we can find the vector \(\va^*\) that minimizes \(||\vy -M\va||\) in a way that is more convenient than computing a projection. We answer this question in a general setting in the next activity.

The result of Activity 26.2 is that we can now do least squares polynomial approximations with just matrix operations. We summarize this in the following theorem.

The equations in the system (26.8) are called the normal equations for \(A \vx = \vb\text{.}\) To illustrate, with the Olympics data, our data points are \((0, 894)\text{,}\) \((4, 1180)\text{,}\) \((8, 1226)\text{,}\) \((12,1418)\) with \(\vy = [894 \ 1180 \ 1226 \ 1418]^{\tr}\text{.}\) So \(M = \left[ \begin{array}{cc} 1\amp0 \\ 1\amp4 \\ 1\amp8 \\ 1\amp12 \end{array} \right]\text{.}\) Notice that \(M^{\tr}M\) is invertible, to find the degree 1 approximation to the data, technology shows that

just as in Preview Activity 26.1.

The solution with our Olympics data gave us the situation where \(M^{\tr}M\) was invertible. This corresponded to a unique least squares solution \(\left(M^{\tr}M\right)^{-1}M^{\tr}\vy\text{.}\) It is reasonable to ask when this happens in general. To conclude this section, we will demonstrate that if the columns of a matrix \(A\) are linearly independent, then \(A^{\tr}A\) is invertible.

We summarize the result of Activity 26.4 in the following theorem.

If the columns of \(A\) are linearly dependent, we can still solve the normal equations, but will obtain more than one solution. In a later section we will see that we can also use a pseudoinverse in these situations.

Subsection Examples

What follows are worked examples that use the concepts from this section.

Example 26.7.

According to the Centers for Disease Control and Prevention⁴⁵, the average length of a male infant (in centimeters) in the US as it ages (with time in months from 1.5 to 8.5) is given in Table 26.8.

Age (months)	1.5	2.5	3.5	4.5	5.5	6.5	7.5	8.5
Average Length (cm)	56.6	59.6	62.1	64.2	66.1	67.9	69.5	70.9

In this problem we will find the line and the quadratic of best fit in the least squares sense to this data. We treat age in months as the independent variable and length in centimeters as the dependent variable.

(a)

Find a line that is the best fit to the data in the least squares sense. Draw a picture of your least squares solution against a scatterplot of the data.

Solution.

We assume that a line of the form \(f(x) = a_1x+a_0\) contains all of the data points. The first data point would satisfy \(1.5a_1+a_0 = 56.6\text{,}\) the second \(2.5a_1+a_0 = 59.6\text{,}\) and so on, giving us the linear system

\begin{alignat*}{2} {3} {1.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 56.6{}\\ {2.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 59.6{}\\ {3.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 62.1{}\\ {4.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 64.2{}\\ {5.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 66.1{}\\ {6.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 67.9{}\\ {7.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 69.5{}\\ {8.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 70.9{.} \end{alignat*}

Letting

\begin{equation*} A = \left[ \begin{array}{cc} 1.5\amp 1 \\ 2.5\amp 1 \\ 3.5\amp 1 \\ 4.5\amp 1 \\ 5.5\amp 1 \\ 6.5\amp 1 \\ 7.5\amp 1 \\ 8.5\amp 1 \end{array} \right], \ \vx = \left[ \begin{array}{c} a_1\\a_0 \end{array} \right], \ \text{ and } \ \vb=\left[ \begin{array}{c} 56.6\\59.6\\62.1\\64.2\\66.1\\67.9\\69.5\\70.9 \end{array} \right]\text{,} \end{equation*}

we can write this system in the matrix form \(A \vx = \vb\text{.}\) Neither column of \(A\) is a multiple of the other, so the columns of \(A\) are linearly independent. The least squares solution \(\vx^*\) to the system is then found by

\begin{equation*} \vx^* = \left(A^{\tr}A\right)^{-1}A^{\tr} \vb\text{.} \end{equation*}

Technology shows that (with entries rounded to 3 decimal places), \(\left(A^{\tr}A\right)^{-1}A^{\tr}\) is

\begin{equation*} \left[ \begin{array}{rrrrcrrr} - 0.083\amp - 0.060\amp - 0.036\amp - 0.012\amp 0.012\amp 0.036\amp 0.060\amp 0.083\\ 0.542\amp 0.423\amp 0.304\amp 0.185\amp 0.065\amp - 0.054\amp - 0.173\amp - 0.292 \end{array} \right]\text{,} \end{equation*}

and

\begin{equation*} \vx^* \approx \left[ \begin{array}{cc} 2.011 \\ 54.559 \end{array} \right]\text{.} \end{equation*}

So the least squares linear function to the data is \(f(x) \approx 2.011x + 54.559\text{.}\) A graph of \(f\) against the data points is shown at left in Figure 26.9.

(b)

Now find the least squares quadratic of the form \(q(x) = a_2x^2+a_1x+a_0\) to the data. Draw a picture of your least squares solution against a scatterplot of the data.

Solution.

The first data point would satisfy \((1.5^2)a_2+1.5a_1+a_0 = 56.6\text{,}\) the second \((2.5)^2a_2+2.5a_1+a_0 = 59.6\text{,}\) and so on, giving us the linear system

\begin{alignat*}{3} {1.5^2}a_2 \amp {}+{} \amp {1.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 56.6{}\\ {2.5^2}a_2 \amp {}+{} \amp {2.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 59.6{}\\ {3.5^2}a_2 \amp {}+{} \amp {3.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 62.1{}\\ {4.5^2}a_2 \amp {}+{} \amp {4.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 64.2{}\\ {5.5^2}a_2 \amp {}+{} \amp {5.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 66.1{}\\ {6.5^2}a_2 \amp {}+{} \amp {6.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 67.9{}\\ {7.5^2}a_2 \amp {}+{} \amp {7.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 69.5{}\\ {8.5^2}a_2 \amp {}+{} \amp {8.5}a_1 \amp {}+{} \amp a_0 \amp {}={} 70.9{.} \end{alignat*}

Letting

\begin{equation*} A = \left[ \begin{array}{ccc} 1.5^2\amp 1.5\amp 1 \\ 2.5^2\amp 2.5\amp 1 \\ 3.5^2\amp 3.5\amp 1 \\ 4.5^2\amp 4.5\amp 1 \\ 5.5^2\amp 5.5\amp 1 \\ 6.5^2\amp 6.5\amp 1 \\ 7.5^2\amp 7.5\amp 1 \\ 8.5^2\amp 8.5\amp 1 \end{array} \right], \ \vx = \left[ \begin{array}{c} a_2\\a_1\\a_0 \end{array} \right], \ \text{ and } \ \vb=\left[ \begin{array}{c} 56.6\\59.6\\62.1\\64.2\\66.1\\67.9\\69.5\\70.9 \end{array} \right]\text{,} \end{equation*}

we can write this system in the matrix form \(A \vx = \vb\text{.}\) Technology shows that every column of the reduced row echelon form of \(A\) contains a pivot, so the columns of \(A\) are linearly independent. The least squares solution \(\vx^*\) to the system is then found by

\begin{equation*} \vx^* = \left(A^{\tr}A\right)^{-1}A^{\tr} \vb\text{.} \end{equation*}

Technology shows that (with entries rounded to 3 decimal places) \(\left(A^{\tr}A\right)^{-1}\) is

\begin{equation*} \left[ \begin{array}{rrrrrrrr} 0.042\amp 0.006\amp - 0.018\amp - 0.030\amp - 0.030\amp - 0.018\amp 0.006\amp 0.042\\ - 0.500\amp - 0.119\amp 0.143\amp 0.286\amp 0.310\amp 0.214\amp 0.000\amp -0.333\\ 1.365\amp 0.540\amp - 0.049\amp - 0.403\amp -0.522\amp - 0.406\amp - 0.055\amp 0.531 \end{array} \right]\text{,} \end{equation*}

and

\begin{equation*} \vx^* \approx \left[ \begin{array}{r} -0.118 \\ 3.195 \\ 52.219 \end{array} \right]\text{.} \end{equation*}

So the least squares quadratic function to the data is \(q\) defined by \(q(x) \approx -0.118x^2+3.195x + 52.219\text{.}\) A graph of \(q\) against the data points is shown at right in Figure 26.9.

Subsection Summary

• A least squares approximation to \(A \vx = \vb\) is found by orthogonally projecting \(\vb\) onto \(\Col A\text{.}\)

• If the columns of \(A\) are linearly independent, then the least squares approximation to \(A \vx = \vb\) is \(\left(A^{\tr}A\right)^{-1}A^{\tr} \vb\text{.}\)

• The least squares solution to \(A \vx = \vb\text{,}\) where \(A\vx = [y_1 \ y_2 \ \ldots \ y_m]\) and \(\vb = [b_1 \ b_2 \ \ldots \ b_ m]^{\tr}\text{,}\) minimizes the distance \(||A\vx - \vb||\text{,}\) where

  \begin{equation*} ||A\vx - \vb||^2 = (y_1-b_1)^2 + (y_2-b_2)^2 + \cdots + (y_m-b_m)^2\text{.} \end{equation*}

  So the least squares solution minimizes a sum of squares.

Subsection Project: Other Least Squares Approximations

In this section we learned how to fit a polynomial function to a set of data in the least squares sense. But data takes on many forms, so it is important to be able to fit other types of functions to data sets. We investigate three different types of regression problems in this project.

Index	Age	Temp (\(^\circ\)C)	Length
1	14	25	620
2	28	25	1315
3	41	25	2120
4	55	25	2600
5	69	25	3110
6	83	25	3535
7	97	25	3935
8	111	25	4465
9	125	25	4530
10	139	25	4570
11	153	25	4600
12	14	27	625
13	28	27	1215
14	41	27	2110
15	55	27	2805
16	69	27	3255
17	83	27	4015
18	97	27	4315
19	111	27	4495
20	125	27	4535
21	139	27	4600
22	153	27	4600
23	14	29	590
24	28	29	1305
25	41	29	2140
26	55	29	2890
27	69	29	3920
28	83	29	3920
29	97	29	4515
30	111	29	4520
31	125	29	4525
32	139	29	4565
33	153	29	4566
34	14	31	590
35	28	31	1205
36	41	31	1915
37	55	31	2140
38	69	31	2710
39	83	31	3020
40	97	31	3030
41	111	31	3040
42	125	31	3180
43	139	31	3257
44	153	31	3214