Simulate Physics: disallow forces from specific actors

I am currently developing a top down scifi game.

I have a space ship and use phyics simulation with AddForce/AddTorque to move the ship. In general this works fine, if I let Asteorids crash into the ship, it gets pushed away and if I move/rotate the ship near obstacles, the ship gets pushed correctly.

Unfortunately there are some problems when colliding with actors that should not move the space ship, e.g. smaller projectiles from pistols or characters running against the space ship. Sometimes, when characters get spawned into the ship, it reacts so heavy that it get pushed into the next obstacle and stuck there colliding with the obstacle forever.

I want that characters and small projectiles cannot push the ship away but still overlap and collide with it, effectively only switching off the forces getting applied to the ship. Is there a way to do that?