Why is the pitcher considered the most important defensive player on team

Because where and how the pitcher throws the ball has a lot to do with where and how hard the batter hits it.

If the pitcher is throwing the ball on the corners of the plate, moving the ball up and down in the strike zone, and changing speeds generally the batter will not hit as hard as the pitcher who throws the ball, no matter how hard, over the middle of the plate.