Prevent pawn with physics from being affected by external physical forces?

I am trying to make a physical pawn/player.

This is my setup:

VRCol is capsule that has gravity and all physics (Like dropping on ground, falling from heights, bumping into walls etc…).

I’ve set rotation constraints for VRCol true for X and Y (So it cant tip over or fall over).

But I left Z rotation constraint as false so user can still turn around.

Problem is that pawn/VRCol is being affected by external forces, like objects hitting them, rubbing against walls etc…

Any ideas how could I prevent by VRCol from being affected physically by physics of other objects?

Like sometimes a will hit VRCol and it will spin forever as if it was in space or something, here is a video: