Problem with Physics Handle

I did an object-drag system. using Physics Handle like in “Amnesia”. It works great, but it has one problem. If I shake my camera so fast, it begins to fly away very far from me. I can’t understand why. Has someone the same problem?