An Inquiry-Based Introduction to Linear Algebra and Applications

Subsection Application: Fourier Series

In calculus, a Taylor polynomial for a function \(f\) is a polynomial approximation that fits \(f\) well around the center of the approximation. For this reason, Taylor polynomials are good local approximations, but they are not in general good global approximations. In particular, if a function \(f\) has periodic behavior it is impossible to model \(f\) well globally with polynomials that have infinite limits at infinity. For these kinds of functions, trigonometric polynomials are better choices. Trigonometric polynomials lead us to Fourier series, and we will investigate how inner products allow us to use trigonometric polynomials to model musical tones later in this section.

Subsection Introduction

In Section 23 we were introduced to inner products in \(\R^n\text{.}\) The concept of an inner product can be extended to vector spaces, as we will see in this section. This will allow us to measure lengths and angles between vectors and define orthogonality in certain vector spaces.

Recall that an inner product on \(\R^n\) assigns to each pair of vectors \(\vu\) and \(\vv\) the scalar \(\langle \vu, \vv \rangle\text{.}\) Thus, an inner product on \(\R^n\) defines a mapping from \(\R^n \times \R^n\) to \(\R\text{.}\) Recall also that an inner product on \(\R^n\) is commutative, distributes over vector addition, and respects scalar multiplication, and the inner product of a vector in \(\R^n\) by itself is always non-negative and is equal to 0 only when the vector is the zero vector. We will investigate this ideas in vector spaces in Preview Activity 35.1.

Subsection Inner Product Spaces

As we saw in Preview Activity 35.1, we can define a mapping from \(\pol_2 \times \pol_2\) to \(\R\) that has the same properties of inner products on \(\R^n\text{.}\) So we can extend the definition of inner product to arbitrary vector spaces.

Since inner products in vector spaces are defined in the same way as inner products in \(\R^n\text{,}\) they will satisfy the same properties. Some of these properties are summarized in the following theorem.

One special inner product is indicated in Preview Activity 35.1. Recall that \(C[a,b]\) is the vector space of continuous functions on the closed interval \([a,b]\text{.}\) Let \(\langle \ , \ \rangle\) map from \(C[a,b] \times C[a,b]\) to \(\R\) be defined by

The verification that this mapping is an inner product is left to the exercises.

Subsection The Length of a Vector

We can use inner products to define the length of any vector in an inner product space and the distance between two vectors in an inner product space. The idea comes right from the relationship between lengths of vectors in \(\R^n\) and inner products on \(\R^n\) (compare to Definition 23.5).

The length of a vector in a vector space is also called magnitude or norm. Just as with inner products on \(\R^n\) we can use the notion of length to define unit vectors in inner product spaces (compare to Definition 23.7).

We can find a unit vector in the direction of a nonzero vector \(\vv\) in an inner product space by dividing by the norm of the vector. That is, the vector \(\ds \frac{\vv}{||\vv||}\) is a unit vector in the direction of the vector \(\vv\text{,}\) provided that \(\vv\) is not zero.

We define the distance between vectors \(\vu\) and \(\vv\) in an inner product space in the same way we defined distance using the dot product (compare to Definition 23.8).

Subsection Orthogonality in Inner Product Spaces

We defined orthogonality in \(\R^n\) using inner products in \(\R^n\) (see Definition 23.11) and the angle between vectors. We can extend those ideas to any inner product space.

If \(\vu\) and \(\vv\) are nonzero vectors in an inner product space, then the angle \(\theta\) between \(\vu\) and \(\vv\) is such that

and \(0\leq \theta \leq \pi\text{.}\) This angle is well-defined due to the Cauchy-Schwarz inequality \(|\langle \vu, \vv \rangle| \leq ||\vu||\, ||\vv||\) whose proof is left to the exercises.

With the angle between vectors in mind, we can define orthogonal vectors in an inner product space.

Note that this defines the zero vector to be orthogonal to every vector.

Using orthogonality we can generalize the notions of orthogonal sets and bases, orthonormal bases and orthogonal complements we defined in \(\R^n\) to all inner product spaces in a natural way.

Subsection Orthogonal and Orthonormal Bases in Inner Product Spaces

As we did with inner products in \(\R^n\text{,}\) we define an orthogonal set to be one in which all of the vectors in the set are orthogonal to each other (compare to Definition 24.1).

As in \(\R^n\text{,}\) an orthogonal set of nonzero vectors is always linearly independent. The proof is similar to that of Theorem 24.3 and is left to the exercises.

A basis that is also an orthogonal set is given a special name (compare to Definition 24.2).

Using inner products in \(\R^n\text{,}\) we saw that the representation of a vector as a linear combination of vectors in an orthogonal basis was quite elegant. The same is true in any inner product space. To see this, let \(\CB = \{\vv_1, \vv_2, \ldots, \vv_m\}\) be an orthogonal basis for a subspace \(W\) of an inner product space and let \(\vx\) be any vector in \(W\text{.}\) We know that

for some scalars \(x_1\text{,}\) \(x_2\text{,}\) \(\ldots\text{,}\) \(x_m\text{.}\) If \(1\leq k\leq m\text{,}\) then, using inner product properties and the orthogonality of the vectors \(\vv_i\text{,}\) we have

So

Thus, we can calculate each weight individually with two simple inner product calculations.

We summarize this discussion in the next theorem (compare to Theorem 24.4).

The decomposition (35.1) is even simpler if \(\langle \vv_k, \vv_k \rangle = 1\) for each \(k\text{.}\) Recall that

so the condition \(\langle \vv, \vv \rangle = 1\) implies that the vector \(\vv\) has norm 1. As with inner products in \(\R^n\text{,}\) an orthogonal basis with this additional condition is given a special name (compare to Definition 24.5).

If \(\CB = \{\vv_1, \vv_2, \ldots, \vv_m\}\) is an orthonormal basis for a subspace \(W\) of an inner product space and \(\vx\) is a vector in \(W\text{,}\) then (35.1) becomes

Recall that we can construct an orthonormal basis from an orthogonal basis by dividing each basis vector by its magnitude.

Subsection Orthogonal Projections onto Subspaces

In Section 25 we saw how to project a vector \(\vv\) in \(\R^n\) onto a subspace \(W\) of \(\R^n\text{.}\) The same process works for vectors in any inner product space.

The notation \(\proj_{W^{\perp}} \vv\) indicates that we expect this vector to be orthogonal to every vector in \(W\text{.}\)

Activity 35.6 shows that the vector \(\vv - \vw\) is orthogonal to vector for \(W\text{.}\) So, in fact, \(\proj_{W^\perp} \vv\) is the projection of \(\vv\) onto the orthogonal complement of \(W\text{,}\) which will be defined shortly.

Subsection Best Approximations in Inner Product Spaces

We have seen, e.g., linear regression to fit a line to a set of data, that we often want to find a vector in a subspace that “best” approximates a given vector in a vector space. As in \(\R^n\text{,}\) the projection of a vector onto a subspace has this important property. That is, \(\proj_W \vv\) is the vector in \(W\) closest to \(\vv\) and therefore the best approximation of \(\vv\) by a vector in \(W\text{.}\) To see that this is true in any inner product space, we first need a generalization of the Pythagorean Theorem that holds in inner product spaces.

Note that replacing \(\vv\) with \(-\vv\) in the theorem also shows that \(||\vu + \vv||^2 = ||\vu||^2 + ||\vv||^2\) if \(\vu\) and \(\vv\) are orthogonal.

Now we will prove that the projection of a vector \(\vu\) onto a subspace \(W\) of an inner product space \(V\) is the best approximation in \(W\) to the vector \(\vu\text{.}\)

Theorem 35.14 shows that the distance from \(\proj_W \vv\) to \(\vv\) is less than the distance from any other vector in \(W\) to \(\vv\text{.}\) So \(\proj_W \vv\) is the best approximation to \(\vv\) of all the vectors in \(W\text{.}\)

In \(\R^n\) using the dot product as inner product, if \(\vv = [v_1 \ v_2 \ v_3 \ \ldots \ v_n]^{\tr}\) and \(\proj_W \vv = [w_1 \ w_2 \ w_3 \ \ldots \ w_n]^{\tr}\text{,}\) then the square of the error in approximating \(\vv\) by \(\proj_W \vv\) is given by

So \(\proj_W \vv\) minimizes this sum of squares over all vectors in \(W\text{.}\) As a result, we call \(\proj_W \vv\) the least squares approximation to \(\vv\text{.}\)

Subsection Orthogonal Complements

If we have a set of vectors \(S\) in an inner product space \(V\text{,}\) we can define the orthogonal complement of \(S\) as we did in \(\R^n\) (see Definition 23.14).

As we saw in \(\R^n\text{,}\) to show that a vector is in the orthogonal complement of a subspace, it is enough to show that the vector is orthogonal to every vector in a basis for that subspace. The same is true in any inner product space. The proof is left to the exercises.

As was the case in \(\R^n\text{,}\) give a subspace \(W\) of an inner product space \(V\text{,}\) any vector in \(V\) can be written uniquely as a sum of a vector in \(W\) and a vector in \(W^{\perp}\text{.}\)

We summarize the result of Activity 35.9.

Theorem 35.17 is useful in many applications. For example, to compress an image using wavelets, we store the image as a collection of data, then rewrite the data using a succession of subspaces and their orthogonal complements. This new representation allows us to visualize the data in a way that compression is possible.

Subsection Examples

What follows are worked examples that use the concepts from this section.

Subsection Summary

• An inner product \(\langle \ , \ \rangle\) on a vector space \(V\) is a mapping from \(V \times V \to \R\) satisfying

  1. \(\langle \vu , \vv \rangle = \langle \vv , \vu \rangle\) for all \(\vu\) and \(\vv\) in \(V\text{,}\)

  2. \(\langle \vu + \vv , \vw \rangle = \langle \vu , \vw \rangle + \langle \vv , \vw \rangle\) for all \(\vu\text{,}\) \(\vv\text{,}\) and \(\vw\) in \(V\text{,}\)

  3. \(\langle c\vu , \vv \rangle = c\langle \vu , \vv \rangle\) for all \(\vu\text{,}\) \(\vv\) in \(V\) and \(c \in \R\text{,}\)

  4. \(\langle \vu , \vu \rangle \geq 0\) for all \(\vu\) in \(V\) and \(\langle \vu , \vu \rangle = 0\) if and only if \(\vu = \vzero\text{.}\)

• An inner product space is a pair \(V\text{,}\) \(\langle \ , \ \rangle\) where \(V\) is a vector space and \(\langle \ , \ \rangle\) is an inner product on \(V\text{.}\)

• The length of a vector \(\vv\) in an inner product space \(V\) is defined to be the real number \(||\vv|| = \sqrt{\langle \vv, \vv \rangle}\text{.}\)

• The distance between two vectors \(\vu\) and \(\vv\) in an inner product space \(V\) is the scalar \(|| \vu - \vv ||\text{.}\)

• The angle \(\theta\) between two vectors \(\vu\) and \(\vv\) is the angle which satisfies \(0\leq \theta \leq \pi\) and

  \begin{equation*} \cos(\theta) = \frac{\langle \vu, \vv \rangle}{||\vu|| ||\vv||} \,\text{.} \end{equation*}

• Two vectors \(\vu\) and \(\vv\) in an inner product space \(V\) are orthogonal if \(\langle \vu, \vv \rangle = 0\text{.}\)

• A subset \(S\) of an inner product space is an orthogonal set if \(\langle \vu, \vv \rangle = 0\) for all \(\vu \neq \vv\) in \(S\text{.}\)

• A basis for a subspace of an inner product space is an orthogonal basis if the basis is also an orthogonal set.

• Let \(\CB = \{\vv_1, \vv_2, \ldots, \vv_m\}\) be an orthogonal basis for a subspace of an inner product space \(V\text{.}\) Let \(\vx\) be a vector in \(W\text{.}\) Then

  \begin{equation*} \vx = \sum_{i=1}^m c_i \vv_i\text{,} \end{equation*}

  where

  \begin{equation*} c_i = \frac{\langle \vx, \vv_i \rangle}{\langle \vv_i, \vv_i \rangle} \end{equation*}

  for each \(i\text{.}\)

• An orthogonal basis \(\CB = \{\vv_1, \vv_2, \ldots, \vv_m\}\) for a subspace \(W\) of an inner product space \(V\) is an orthonormal basis if \(|| \vv_k || = 1\) for each \(k\) from 1 to \(m\text{.}\)

• If \(\CB = \{\vw_1, \vw_2, \ldots, \vw_m\}\) is an orthogonal basis for \(V\) and \(\vx \in V\text{,}\) then

  \begin{equation*} [\vx]_{\CB} = \left[ \begin{array}{c} \frac{\langle \vx, \vw_1 \rangle}{\langle \vw_1, \vw_1 \rangle} \\ \frac{\langle \vx, \vw_2 \rangle}{\langle \vw_2, \vw_2 \rangle} \\ \vdots \\ \frac{\langle \vx, \vw_m \rangle}{\langle \vw_m, \vw_m \rangle} \end{array} \right]\text{.} \end{equation*}

• The projection of the vector \(\vv\) in an inner product space \(V\) onto a subspace \(W\) of \(V\) is the vector

  \begin{equation*} \proj_W \vv = \frac{\langle \vv, \vw_1 \rangle }{\langle \vw_1, \vw_1 \rangle } \vw_1 + \frac{\langle \vv, \vw_2 \rangle }{\langle \vw_2, \vw_2 \rangle } \vw_2 + \cdots + \frac{\langle \vv, \vw_m \rangle}{\langle \vw_m, \vw_m \rangle} \vw_m\text{,} \end{equation*}

  where \(\{\vw_1, \vw_2, \ldots, \vw_m\}\) is an orthogonal basis of \(W\text{.}\) Projections are important in that \(\proj_W \vv\) is the best approximation of the vector \(\vv\) by a vector in \(W\) in the least squares sense.

• With \(W\) as in (a), the projection of \(\vv\) orthogonal to \(W\) is the vector

  \begin{equation*} \proj_{W^{\perp}} \vv = \vv - \proj_W \vv\text{.} \end{equation*}

  The norm of \(\proj_{W^{\perp}} \vv\) provides a measure of how well \(\proj_W \vv\) approximates the vector \(\vv\text{.}\)

• The orthogonal complement of a subset \(S\) of an inner product space \(V\) is the set

  \begin{equation*} S^{\perp} = \{\vv \in V : \langle \vv, \vu \rangle = 0 \text{ for all } \vu \in S\}\text{.} \end{equation*}

Subsection Project: Fourier Series and Musical Tones

Joseph Fourier first studied trigonometric polynomials to understand the flow of heat in metallic plates and rods. The resulting series, called Fourier series, now have applications in a variety of areas including electrical engineering, vibration analysis, acoustics, optics, signal processing, image processing, geology, quantum mechanics, and many more. For our purposes, we will focus on synthesized music.

Pure musical tones are periodic sine waves. Simple electronic circuits can be designed to generate alternating current. Alternating current is current that is periodic, and hence is described by a combination of \(\sin(kx)\) and \(\cos(kx)\) for integer values of \(k\text{.}\) To synthesize an instrument like a violin, we can project the instrument's tones onto trigonometric polynomials — and then we can produce them electronically. As we will see, these projections are least squares approximations onto certain vector spaces. The website falstad.com/fourier/ provides a tool for hearing sounds digitally created by certain functions. For example, you can listen to the sound generated by a sawtooth function \(f\) of the form

Try out some of the tones on this website (click on the Sound button to hear the tones). You can also alter the tones by clicking on any one of the white dots and moving it up or down. and play with the buttons. We will learn much about what this website does in this project.

Pure tones are periodic and so are modeled by trigonometric functions. In general, trigonometric polynomials can be used to produce good approximations to periodic phenomena. A trigonometric polynomial is an object of the form

where the \(c_i\) and \(d_j\) are real constants. With judicious choices of these constants, we can approximate periodic and other behavior with trigonometric polynomials. The first step for us will be to understand the relationships between the summands of these trigonometric polynomials in the inner product space \(C[-\pi, \pi]\)⁶¹ of continuous functions from \([\pi, \pi]\) to \(\R\) with the inner product

Our first order of business is to verify that (35.3) is, in fact, an inner product.

Now we return to our inner product space \(C[-\pi,\pi]\) with inner product (35.3). Given a function \(g\) in \(C[-\pi,\pi]\text{,}\) we approximate \(g\) using only a finite number of the terms in a trigonometric polynomial. Let \(W_n\) be the subspace of \(C[-\pi,\pi]\) spanned by the functions

One thing we need to know is the dimension of \(W_n\text{.}\)

Now we need to see if what happened in Project Activity 35.11 happens in general. A few tables of integrals and some basic facts from trigonometry can help.

Once we have an orthogonal basis for \(W_n\text{,}\) we might want to create an orthonormal basis for \(W_n\text{.}\) Throughout the remainder of this project, unless otherwise specified, you should use a table of integrals or any appropriate technological tool to find integrals for any functions you need.

Now we need to recall how to find the best approximation to a vector by a vector in a subspace, and apply that idea to approximate an arbitrary function \(g\) with a trigonometric polynomial in \(W_n\text{.}\) Recall that the best approximation of a function \(g\) in \(C[-\pi,\pi]\) is the projection of \(g\) onto \(W_n\text{.}\) If we have an orthonormal basis \(\{h_0, h_1, h_2, \ldots, h_{2n}\}\) of \(W_n\text{,}\) then the projection of \(g\) onto \(W_n\) is

With this idea, we can find formulas for the coefficients when we project an arbitrary function onto \(W_n\text{.}\)

Let us return to the sawtooth function defined earlier and find its Fourier coefficients.