A common problem in experimental work is to find a straight line which approximates as well as
possible a collection of points in the plane

{(xi,yi)}

_{i=1}^{p}. The usual way of dealing with these
problems is by the method of least squares and it turns out that all these sorts of approximation
problems can be reduced to Ax = b where the problem is to find the best x for solving this
equation even when there is no solution.

Lemma 11.5.1Let V and W be finite dimensional inner product spaces and let A : V → W belinear. For each y ∈ W there exists x ∈ V such that

|Ax − y| ≤ |Ax1 − y|

for all x_{1}∈ V. Also, x ∈ V is a solution to this minimization problem if and only if x is a solutionto the equation, A^{∗}Ax = A^{∗}y.

Proof: By Theorem 11.2.4 on Page 799 there exists a point, Ax_{0}, in the finite dimensional
subspace, A

(V )

, of W such that for all x ∈ V,

|Ax − y|

^{2}≥

|Ax0 − y|

^{2}. Also, from this theorem,
this happens if and only if Ax_{0}− y is perpendicular to every Ax ∈ A

(V )

. Therefore, the
solution is characterized by

(Ax0 − y,Ax )

= 0 for all x ∈ V which is the same as saying

(A ∗Ax0 − A ∗y,x )

= 0 for all x ∈ V. In other words the solution is obtained by solving
A^{∗}Ax_{0} = A^{∗}y for x_{0}. ■

Consider the problem of finding the least squares regression line in statistics. Suppose you have
given points in the plane,

{(xi,yi)}

_{i=1}^{n} and you would like to find constants m and b such that
the line y = mx + b goes through all these points. Of course this will be impossible
in general. Therefore, try to find m,b such that you do the best you can to solve the
system

( ) ( )
| y1 | | x1 1 | ( )
|( ... |) = |( ... ...|) m
b
yn xn 1

which is of the form y = Ax. In other words try to make

| ( ) |
|| ( ) y1 ||
|| m || . || ||
||A b − ( .. ) ||
| yn |

^{2} as small as
possible. According to what was just shown, it is desired to solve the following for m and
b.

( )
( ) y1
A∗A m = A∗ || ... || .
b ( )
yn

Since A^{∗} = A^{T} in this case,

( ∑n 2 ∑n ) ( ) ( ∑n )
∑ i=1x i i=1xi m = ∑i=1xiyi
ni=1 xi n b ni=1yi

Solving this system of equations for m and b,

− (∑n x )(∑n y) +(∑n x y)n
m = ----i=1∑ni---2i=1--i∑n---i=12-i-i--
( i=1xi)n− ( i=1xi)

One could clearly do a least squares fit for curves of the form y = ax^{2} + bx + c in the same way.
In this case you solve as well as possible for a,b, and c the system

( ) ( )
x21 x1 1 ( a ) y1
|| .. .. .. || | | = || .. ||
( . . . ) ( b ) ( . )
x2n xn 1 c yn