Physics object dont follow blocking collisions

Don’t know if the title says it all but here is a little more details.

I have a hollowed out cube with collision that blocks an object inside. It works to pickup with a motioncontroller and move around the cube and the object is stuck inside but when I shake it fast it lags behind and you can se the mesh inside.

I have searched but didn’t know exactly what to search for. How can I fix this?

I have attached a gif and a picture of the objects seen in wireframe mode.

via GIPHY

127629-capture.png