Is there a way to limit shadow frustums? Causing large FPS drop

I am building a level and am getting a large frame rate drop when looking in the direction of the main directional light volume and when behind all of the shadow-casting objects in the level. Turning off “Cast dynamic shadows” will fix the framerate, but will not cast shadows on the rest of the level.

When I look towards where the frustums go to, and the meshes that cast the shadow are not shown in the viewport, the Object shadow projections call count jumps up from 18-20 all the way to 1300’s, causing a major framerate drop of 50-60 frames per second. The level runs smoothly around 70-80 FPS when playing the game normally, and the player does not look in the direction of the directional light when most of the level is not seen on the screen.

Is there anything I can do to limit the distance of the shadow frustum, or is there another way to solve this issue?

http://i.imgur.com/hi1nkQz.jpg