"Markku Poysti" <email@example.com_nospam> wrote in message
> I copied the best least squares line fit algorithm from Graphics Gems V
> but now i'm having problems with vertical lines, although the book claims
> the algorithm can handle vertical lines.
The derivation has problems. For example, consider equation (3)
which has x*sin(theta)+y*cos(theta)+p = 0. Figure 1 shows what
p and theta mean. Now choose p = 0 and theta = 45 degrees.
The figure indicates a line y = x. The equation says x + y = 0.
Does this give you warm fuzzies about the rest of the construction?
Let t = ((sxx+syy)-sqrt((sxx-syy)*(sxx-syy)+4*sxy*sxy))/2.
The least squares line contains the average (xmid,ymid) of the
points (using your notation). The direction is chosen to be
either (-sxy, t-syy) or (t-sxx, -sxy), whichever has larger length.
(The GG5 construction essentially chose the wrong direction
that, as you noticed, was (0,0).)
Grundgy details in