Collision Boxes for Punching Always Launches Physics Objects

So for collision detection when my character is punching, I have sphere collision boxes that activate and deactivate during the punching animation. Whenever I punch with them, the box launches no matter what settings I have. Gravity helps, but makes jumping feel awkward so I don’t want to raise it past -3000, but mass and friction do absolutely nothing. I’ve tried disabling physics on the box, but then collision won’t get detected, and restricting rotation and movement just makes it rotate weirdly when it gets hit. I’m really not sure what to do and how to fix this.