Normal map LOD not working correctly.

So I dunno know why this is happening, I’ve got a low poly model and I baked maps in X normal, everything baked fine and such the texture works but as you move around the object sometimes the base normal map just up and disappears making the model all flat and gross. Any suggestions?