VR - Push towards Player

Hey,
this may be a basic problem, but I can’t really put my problem to words.
In my project I have the Player hit a ball (actor) with a racket (pickup actor). The ball should fly towards a wall (actor) hit it and return. The goal is to have as many successive hits as possible. In an attempt to stop the player from running around too much I tried to get the ball aim back at the player once it hits the wall. I put an Add Force node in and used the vector between ball and pawn for force. The result was not what I intended, the ball still flew not directly towards the player. I also tried Add Physical Linear Velocity, that made the ball go rather crazy, in an understandable way. My maths are terrible, which made the calculation for the Velocity vector hell.
I hope this does not sound too complex.
Thanks!