Find two unit vectors orthogonal to both 8, 5, 1 and −1, 1, 0 .

In order to do this, you must first find the "cross product" of these vectors. To do that, we can use several methods. To simplify this first, I suggest you compute:

‹1, -1, 1› × ‹0, 1, 1›

You are interested in vectors orthogonal to the originals, which don't change when you scale them. Using 0,-1,1 is much easier than 6s and 7s.

So what methods are there to compute this? You can review them here (or presumably in your class notes or textbook):
http://en.wikipedia.org/wiki/Cross_produ...

In addition to these methods, sometimes I like to set up:
‹1, -1, 1› • ‹a, b, c› = 0
‹0, 1, 1› • ‹a, b, c› = 0

That is the dot product, and having these dot products equal zero guarantees orthogonality. You can convert that to:

a - b + c = 0
b + c = 0

This is two equations, three unknowns, so you can solve it with one free parameter:

b = -c
a = c - b = -2c

The computation, regardless of method, yields:
‹1, -1, 1› × ‹0, 1, 1› = ‹-2, -1, 1›

The above method, solving equations, works because you'd just plug in c=1 to obtain this solution. However, it is not a unit vector. There will always be two unit vectors (if you find one, then its negative will be the other of course). To find the unit vector, we need to find the magnitude of our vector:

|| ‹-2, -1, 1› || = √( (-2)² + (-1)² + (1)² ) = √( 4 + 1 + 1 ) = √6

Then we divide that vector by its magnitude to yield one solution:

‹ -2/√6 , -1/√6 , 1/√6 ›

And take the negative for the other:

‹ 2/√6 , 1/√6 , -1/√6 ›