High resolution world displacement

I am currently working on a project where I want to put a hologram of a person into a car and have them talk a bit while driving around for rendering out in stereoscopic 360°. I want to do this using the MS Kinect for getting the depth of the video as well as the video it self. Together with the K4U plugin from OpaqueMedia (Kinect 4 Unreal Introduction in Blueprints - UE Marketplace) getting the RGB stream and depth stream into unreal works like a charm. But the rest doesn’t work as I hoped. The final result should look somewhat like this: http://drdave.co.uk/blog/category/kinect

However, using a wall with a rather high tessellation (5000 Polys) and a world offset material (Unreal Engine 4 Tutorial: Tessellation (english) - YouTube without normal map) the result is far from what I looked for.
The Offset is always either 1 or 0, the is no iterpolation what so ever. Also it seems to affect only every 5th vertecy or so making it seem like an art piece by Picasso.

How can I achieve a look like what Dr. Dave did?
Performance is irrelevant.