Least Squares Solutions

Here's the picture of xA x for the 2 × 2 matrix A = (v v ) (s ) (u ) 1 2 1 0 1 s u 0 2 2.

You see that A(u_1 + u_2) = 2v_1 + 0v_2 so the singular values are s_1 = 2 and s_2 = 0.  
Since s_2 = 0, A can't send anything in the direction of v_2.  
This tells you that A sends every vector x to a multiple of v_1.

You can see that y = 4v_1 + v_2.
Since y isn't a multiple of v_1 you know that A x = y has no solution.
Use the graphic to find an x that makes A x as close as possible to y .

Since A sends every x to a multiple of v_1, the closest that A x can come to y = 4v_1 + v_2 is 4v_1. This is the projection of y on the column space of A and is denoted Overscript[y,^].

If y is not in the column space of A, then the closest you can come to solving A x = y is to solve A x = Overscript[y,^].

Since s_1 = 2, you know that x = 4/2u_1 = 2u_1is one solution to A x = Overscript[y,^] = 4v_1.  

And since s_2 = 0, all solutions are given by x = 2u_1 + t u_2.

Use the graphic to verify that the vectors 2u_1 + t u_2 are solutions to A x = Overscript[y,^] for every value of t.

Modified three steps to solving A x = y .
Step 1:  From the plot you see y = 4v_1 + v_2.
    Since s_2 = 0, throw away the v_2 portion of y to get Overscript[y,^] = 4v_1.
    Continue solving A x = Overscript[y,^].
Step 2:  Divide the 4 by the singular value 2 to get 4/2 = 2.
Step 3:  Form x by using 4/2u_1, i.e. x = 2u_1.
Since s_2 = 0, the general solution to A x = Overscript[y,^] is x = 2u_1 + t u_2.

This general solution to A x = Overscript[y,^] is called the general least squares solution to A x = y .

Here's the picture of xA x for the 2 × 2 matrix A = (v v ) (s ) (u ) 1 2 1 0 1 s u 0 2 2.

(Q31) The plot shows that the singular values are s_1 = __ and s_2 = __.

(Q32) The plot shows y = __  v_1 + __  v_2.

(Q33) The vector Overscript[y,^] in the column space of A closest to y is Overscript[y,^] = __  v_1 + __  v_2.

(Q34) How many solutions does A x = y have?  ___
How many solutions will A x = Overscript[y,^] have?   ___

(Q35) The general solution to A x = Overscript[y,^] is x = __ u_1 + __ u_2.

(Q36) The general least squares solution to A x = y is x = __ u_1 + __ u_2.

Up

Copyright © 2007 Todd Will
Created by Mathematica  (April 14, 2007)