Moving and rotating streamed levels causes light issues

I have it set up streamed level were rooms are endlessly streamed in randomly. As such I’m using set level transform to move and rotate the room to make them flow endlessly. Here is a video showing what i’ve got:

as you may be able to see from it, the lighting in some of the rooms gets darker at certain angles. All the lighting is baked beforehand and look fine but when rotated it from some angles end up looking weird. I was wondering if this is just a problem that rotating streamed levels or if it because the examples i made have outside windows and its the sunlight from different angles.