Find a 3x1 matrix x with entries not all zero such that ax =3x ,
where
a= (1,2 ,-1)
(1,0 ,1)
(4,-4,5)

This is called an eigenvector problem. If you search on wikipedia or something you'll find some solutions. I haven't done these for a while, but I think usually they don't give you the 3...
AX = 3X,
then
AX - 3X = 0
(A-3I)X = 0
I is the identity matrix, which is 3*3 in dimensions. Now you can work out the matrix A-3I.
 3I =
 [3, 0, 0
 0,3,0
 0,0,3 ]
So, calculate A-3I. Now you just have the type of problem which is BX = 0. Solve that in the normal way to find the vector X.