How do I generate pieces of the nav mesh, without losing performance or changing my AI?

Right now I have actors on my map that are supposed to be destroyed by the player. They currently affect the nav mesh so my enemies (who are using a detour crowd AI) will run around them. When the items are destroyed their is still a gap in the nav mesh where the actor using to be. I know that it can be fixed by using dynamic runtime generation, however their is a very noticeable frame rate drop. I can’t have the nav mesh be unaffected by the actor because my enemies will run into it as if it’s not their, and they can’t use RVO avoidance to go around because of the Detour Crowd AI.

In summation, how do I generate pieces of the nav mesh, without losing performance or changing my AI?