My "portal" misaligns when you look away from it.

I have a scene capture and render target portal that I need to align with the surrounding tunnel it is placed in.
I have made a video here:

And here is an image on Imgur highlighting my issue:

This should explain and demonstrate my issue much better than I can do in words. Any help appreciated.

I can’t really see the issue, Can you upload a 60fps video

If you watch at 50 seconds into the video and follow my instruction; look at the bottom left edge when I look at it. The ‘real’ tunnel and the ‘projected image’ tunnel are aligned. When I look away from that edge, keep your eye on it, they misalign.
Look at this image to see what I mean. http://i.imgur.com/kxXZATY.jpg