11.5 Least Squares
A common problem in experimental work is to find a straight line which approximates as well as
possible a collection of points in the plane
. The usual way of dealing with these
problems is by the method of least squares and it turns out that all these sorts of approximation
problems can be reduced to Ax
where the problem is to find the best x
for solving this
equation even when there is no solution.
Lemma 11.5.1 Let V and W be finite dimensional inner product spaces and let A : V → W be
linear. For each y ∈ W there exists x ∈ V such that
for all x1 ∈ V. Also, x ∈ V is a solution to this minimization problem if and only if x is a solution
to the equation, A∗Ax = A∗y.
Proof: By Theorem 11.2.4 on Page 799 there exists a point, Ax0, in the finite dimensional
such that for all x ∈ V,
Also, from this theorem,
this happens if and only if Ax0 − y
is perpendicular to every Ax ∈ A
solution is characterized by
= 0 for all
x ∈ V
which is the same as saying
= 0 for all
x ∈ V.
In other words the solution is obtained by solving
for x0. ■
Consider the problem of finding the least squares regression line in statistics. Suppose you have
given points in the plane,
and you would like to find constants m
the line y
goes through all these points. Of course this will be impossible
in general. Therefore, try to find m,b
such that you do the best you can to solve the
which is of the form y = Ax. In other words try to make
as small as
possible. According to what was just shown, it is desired to solve the following for m
Since A∗ = AT in this case,
Solving this system of equations for m and b,
One could clearly do a least squares fit for curves of the form y = ax2 + bx + c in the same way.
In this case you solve as well as possible for a,b, and c the system
using the same techniques.