Physics resets on character movement

I have added a skeletal mesh with physic asset to the camera component of the FPV example character. I set that to physic animate with Set All Bodies Below Simulate Physics. That gets it physic animated all fine but as soon as I move the character the physic animated bone gets reset to its default rotation.
Is there any way to make it work right?