Calculating Transform offset between two Actors

Hi, I am working on a VR project and when the player grips an object in the scene with their motioncontroller, I want to use Sweep to simulate the attachment to the hand (which prevents the object from going through walls like if I merely just attached the object as a child component).

My only problem is that using my current method, the picked up object snaps to the exact same location and rotation as the motioncontroller. I want it to retain the original offset from when I grabbed it, so that I can pick it up from the bottom, the top, etc and it will not snap to the same relative location each time matching the motioncontroller exactly.

So basically instead of telling it to SetActorLocationAndRotation equal to the Motioncontroller, I want it to sweep to a location and rotation that represents the initial offset of the object from when I first picked it up. I have been struggling with calculating this desired point based on the initial transforms of the motioncontroller and the object being picked up.

Any help would be appreciated.