CastShadows changes mesh

I have ran into an issue where if I enable cast shadow on my mesh I can see some of the geometry of my low poly model even though I have a normal map applied to it, when I disable cast shadow it fixes the geometry but then it will never cast any shadows. I am somewhat new to UE4 and blender so I have no idea how to fix this and I have been trying for hours, the only thing I have seen to try and fix it was to flip the green channel but it didn’t work.

120979-castshadowon.png

120980-castshadowoff.png

the normal map if that helps in anyway.

I think I found what is causing the issue but not how to fix it. The collision box lines up with the lines on the model so somehow it is messing with the model when I check the “Cast Shadow” option.

121000-collision.png