Physics constraints not constraining

Hello, I am having a problem with Physics constraints not fully locking an object to it’s position.
Essentially I am just trying to male a simulated physics cube that is locked in XYZ position, and XY rotation, so that it only spins in Z rotation. I have all those option checked on the object, but in when I test in VR, I am able to easily “push” the cube in xyz directions. It does pop back to its proper spot, and primarily rotates in z, but the fact that it can be pushed around and is not fully locked to just the Z rotation is my issue. I have tried cranking up values in the linear damping and angular damping, but it doesn’t seem to have any effect on my issue. I’ve also tried increasing the position and velocity solvers numbers but that didn’t seem to help either. Any suggestions on why this is happening and what I can do to fix it would be greatly appreciated, Thanks!