They used the formula to find the area of a parallelogram instead of a triangle.
S=sqrt(s1^2+s2^2+s3^2) finds the area of a parallelogram while S=sqrt(s1^2+s2^2+s3^2)/2 is for a triangle. further more I graphed the 3 points out on a 3d grapher and measured the shortest distance from point p to line ab and it is exactly half the answer you would get if you used the book's formula.
my code:
%% 20)
%define variables
xp=2
yp=6
zp=-1
xa=-2
ya=-1.5
za=-3
xb=-2.5
yb=6
zb=4
r=sqrt((xb-xa)^2+(yb-ya)^2+(zb-za)^2) %solve for r
s1=xp*ya+xa*yb+xb*yp-(yp*xa+ya*xb+yb*xp) % solve for s1,s2,s3; this was fun to type
s2=yp*za+ya*zb+yb*zp-(zp*ya+za*yb+zb*yp)
s3=xp*za+xa*zb+xb*zp-(zp*xa+za*xb+zb*xp)
s=sqrt(s1^2+s2^2+s3^2) %solve for s
d=(2*s)/r %distance can now be calculated
%voila
%d=11.3310 (distance)
%s=58.1920 (triangle area)
graph with distance measured:
11.331/2=5.6655
