Why use fresnel for refraction in materials?

Why do we use fresnel to determine the refraction value in materials, and not just the physical index of refraction? Is there a physical explanation of this? Is it an engine thing?

Refraction in UE4 is done in screen space. Basically, the material takes a copy of picture behind it, and distorts it according to the value, that you input into refraction slot. It does not account for the surface of the object to its full extent. Given a corner case, where you are looking through a flat plane, perpendicular to a camera, just inputting IOR will give correct result. For actual surfaces, you would need to use Fresnel.
Additionally, as the effect is based on existing picture, material cannot refract objects, that are outside of camera’s field of view. Using Fresnel falloff limits the chances of your seeing boundaries of refraction picture.

It kind of makes sense, but I’m still not entirely sure what the initial issue is.Because the displacement is effected by the surface normal, but too much in many cases it seems.