Use motion controllers without rendering VR/using HMD

I am currenty working on a prototype for a mixed reality application and my plan was to use the Oculus Touch controllers (motion controllers) without using the HMD. I know the motion controllers wont work without a HMD connected, so I dont have a problem laying down the HMD just on a table or something. My Plan was to use a normal Monitor and set a fixed camera position which matches up with real world objects/positions and then use the motion controllers to “pick up” real world objects and the move them in front of the screen so you can see them in the viewport of the camera.

My problem now is, that as soon as I disable “Lock to HMD” on the camera the motion controllers dont get tracked anymore. So I tried adding a second camera to my character pawn and switching the viewport to it but that didnt seem to work at all (using “Set Active”). Is there any solution to use the motion controllers without tracking the HMD position and fix the camera?