Why would an asteroid have a difficult time of holding a probe or person?

The asteroid's mass is so small that it has a much smaller acceleration
due to gravity than Earth has.  That means that things weigh very very little
on the surface of an asteroid. It also means that the "escape velocity" from
an asteroid is very low, and orbital velocities are very low at any distance off
of its surface.

As an extreme example:  You know how when you walk, you naturally rise up
on the toes of one foot while you reach out with the other one to take a step ? 
All of those motions are what you learn in Earth's gravity.  On an asteroid, that
natural action of rising up on your toes might launch you into a long, high arc,
like a golf ball. Or it might even exceed escape velocity and you'd sail up off
of the asteroid and never come back down to it. 

As we gain mass the force of gravity on us increases!