How to prevent CSM aliasing with dynamic time of day

I am working on a project (using an older version of UE4) where we have a dynamic time of day system that just rotates the directional light. We use cascaded shadow maps for shadows. What happens is the edges of shadows move back and forth a few times a second, which is basically related to the rotation of the directional light at a fairly slow pace (if the light isn’t moving, this doesn’t happen).

My theory is that when the shadow frustum is built to fit the view cascade frustum, the rotation causes the orthogonal projection to move slowly relative to the ground. This movement causes the shadows to creep forward even when the sun hasn’t moved that much (basically the pixel is moving slowly but still covered by the same occluder). Once the pixel stops being rasterized by the occluder, the shadow jumps back a little bit to the pixel behind it. This happens quite often in comparison to the how fast the sun is actually moving.

I was just wondering if anyone has worked with dynamic time of day systems and have a solution to prevent this kind of aliasing besides just jacking up the shadow resolution. Since it jumps about one pixel at a time, resolution is definitely the biggest cause of the issue. I have tried making a larger blur filter using multple PCF samples, but this just makes all my shadows super soft and it doesn’t really fix the issue, just makes it harder to notice because the contrast is lower.