Issues with rotation of components attached to camera in VR mode

Say I have a widget interaction component attached to a camera (pawn camera), the widget interaction component acts fine when playing in view port, following the camera closely and without delays.

Once I play the same scene in VR mode though, the component just disappears whenever I tried to look at any angle below 35-45 degrees downwards. If I just keep my head leveled or looking upwards, it works fine. There’s also a visible lag when the component is following the rotation of the camera.

This issue have been driving me nuts for the past couple of days, looked all over for an answer, so any help will be greatly appreciated, thanks!

Forgot to include some screenshots:
Here you can see that the pointer is working and the hover state of the button is active

Here I’m looking at a low angle with the menu re-aligned, the pointer is gone even though I’m supposed to point directly at the play button.