Dragging a lever based on orientation & viewing angle

In my project, I have a lever mesh sticking out of the floor. Holding the left mouse button on this lever and moving the mouse initiates a dragging operation. This dragging operation is responsible for calculating the 2D delta mouse movements, and utilizing this data to rotate the lever, which can only rotate in a single axis (negative or positive, but still only one axis) in local space.

But what if, instead of being in front of the lever, I’m actually behind it? What if I’m on one of its sides? What if the lever is actually on a wall instead of the floor?.. How do I make it so that mouse movements actually rotate the lever appropriate to the angle at which it is viewed from?