Motion Controller Impact Strength

Hi Guys, I have a quick question which has been riddling me, while I have developed a work around, I want to do this the right way. Essentially my scenario is this:

I have Motion Controllers which have hand meshes. The meshes have simulate physics enabled. When those meshes (hands) collide with an environmental object or enemy the result is as expected, force is reasonable and the reaction realistic.

However, if I pickup an object (object is set to physics not simulating + is attached to the motion controller on pickup), and said object while being held collides with the environment (or an enemy simulating physics) the response is extremely unrealistic and very intense. I have tried setting the linear damping and angular damping on the held object but this did not have any effect when the object was being used (likely because physics is turned off when the object is held).

Does anybody have any suggestions as to how I would go about handling this?