I tried setting the light map resolution to 1024, the min light map resolution to 1024, I tried blocking it out but nothing seems to fix it.
How can I fix this?
That is Ambien Occlusion. It mimics the tendency of areas where geometry meets to be darker. This is intended but if you want to turn it off you can do so with a post process volume or in Project Settings under Rendering->Default Settings->Ambient Occlusion.