Screenspace Portal's on VR

Hi Everyone, I am trying to create a portal system for use on VR, but have come across issues with the way that it renders.

Currently I have a SceneCapture2D component that renders a view to a 1920x1080 render texture. Using screen space UV’s, this is then applied in a material to a plane in the world. This is an amazing method to be used for games that would be played on the monitor, however, when played in VR, there is an aweful amount of skew on the texture.

The problem is not the texture, as when this is applied to a BSP without screen space UV’s, it can be seen as working totally fine without distortion.

Myself and colleagues think that the problem is with the way the screen space UV’s function grabs the screen position. We have tried changing this function that that we calculate the screen position manually/traditionally, however, we still end up with the same results in VR.

We have tried to find out where we can get the camera view matrix or projection matrix. However, this seems to be near impossible with Blueprints and the material shader. We have also tried using the world to clip function.

Does anyone have an appropriate solution to stop the skewing of the render target texture?

Thanks!

My colleague and I managed to fix this (mostly). Hold on to your hats!

We had to create our own BP class in C++ (people who use BP, do not worry, you can find some code to copy and paste below).

In the header, we created a UProperty for a scene capture component 2D that could be fed into a function. We also created a Calculate FOV function and a Update Portal View P Matrix Parameters.

The Calculate FOV function takes an input of a player controller. This then checks to see if a HMD is connected and if so, proceeds to grab the FOV from the VR Camera as it changes between VR and on PC.

The Update Portal View Projection Matrix Parameters function takes dynamic material instance, a player camera transform, and a player controller.
First of all we get the sceneCaptureComponent2D’s capture size (x,y). We then grab the view matrix and view location of the player’s camera transform. We then swap the axis to match unreal’s coord space so Z is up. Lastly we grab the scene capture component 2D’s FOV.

NOW THE MATH BEGINS! :smiley:

If the viewport is wider than it is tall, the XAxisMultiplier is 1 whilst the YAxisMultiplier is viewport.x / viewport.y
Else the XAxisMultiplier is viewport.y / viewport.x and the XAxisMultiplyer is 1.

We then create a projection matrix where the FOV value, axis multipliers and clipping plane values (we gave 10 and 1000) are fed into the constructor.

Then a view projection matrix is created by multiplying the view matrix and the projection matrix.

We then break the VPMatrix up into its column components (for X, Y, Z and W axis) to be fed into out dynamic material instance and feed them into it.

HERE IS ALL THE CODE SO FAR:
.h

// Fill out your copyright notice in the Description page of Project Settings.

#pragma once

#include "GameFramework/Actor.h"
#include "Kismet/GameplayStatics.h"
#include "Camera/PlayerCameraManager.h"
#include "Engine/SceneCapture2D.h"
#include "Classes/Components/SceneCaptureComponent2D.h"

#include "MyTestPortalActor.generated.h"


UCLASS()
class AMyTestPortalActor : public AActor
{
	GENERATED_BODY()
	

public:	
	
	UPROPERTY(VisibleAnywhere, BlueprintReadWrite, Category = "RDSceneCapComp")
	USceneCaptureComponent2D* RDSceneCapComp2D;

	// Sets default values for this actor's properties
	AMyTestPortalActor();

	

	// Called when the game starts or when spawned
	virtual void BeginPlay() override;
	
	// Called every frame
	virtual void Tick( float DeltaSeconds ) override;

	UFUNCTION(BlueprintCallable, Category = "RDMatrix")
	float CalculateFOV(APlayerController* playerController);

	UFUNCTION(BlueprintCallable, Category = "RDMatrix")
	void UpdatePortalVPMParameters(UMaterialInstanceDynamic* material, FTransform PlayerCameraXForm, APlayerController* playerController);
		
};

.cpp

#include "MyTestPortalActor.h"
#include "Kismet/GameplayStatics.h"


#include "Classes/Components/SceneCaptureComponent2D.h"
#include "Classes/Engine/TextureRenderTarget2D.h"
#include "Classes/Camera/CameraComponent.h"
#include "Runtime/HeadMountedDisplay/Public/IHeadMountedDisplay.h"

// Sets default values
AMyTestPortalActor::AMyTestPortalActor()
{
 	// Set this actor to call Tick() every frame.  You can turn this off to improve performance if you don't need it.
	PrimaryActorTick.bCanEverTick = true;
	RDSceneCapComp2D = NULL;
}

// Called when the game starts or when spawned
void AMyTestPortalActor::BeginPlay()
{
	Super::BeginPlay();
	this->SetTickGroup(TG_PostPhysics);
}

// Called every frame
void AMyTestPortalActor::Tick( float DeltaTime )
{
	Super::Tick( DeltaTime );
	this->SetTickGroup(TG_PostPhysics);
	
	
}

float AMyTestPortalActor::CalculateFOV(APlayerController* playerController)
{
	float fov = 90.0f;
	
	if (playerController != NULL)
	{
		if (playerController->PlayerCameraManager != NULL)
		{
			fov = playerController->PlayerCameraManager->GetFOVAngle();
		}
	}

	// FOV changes when we have a VR Headset enabled
	if (GEngine->HMDDevice.IsValid() && GEngine->IsStereoscopic3D())
	{
		float HFOV, VFOV;
		GEngine->HMDDevice->GetFieldOfView(HFOV, VFOV);
		if (VFOV > 0 && HFOV > 0)
		{
			fov = FMath::Max(HFOV, VFOV);
			// AspectRatio won't be used until bConstrainAspectRatio is set to true,
			// but it doesn't really matter since HMD calcs its own projection matrix.
			//OutViewInfo.AspectRatio = HFOV / VFOV;
			//OutViewInfo.bConstrainAspectRatio = true;
		}
	}

	return fov;
}

void AMyTestPortalActor::UpdatePortalVPMParameters(UMaterialInstanceDynamic* material, FTransform PlayerCameraXForm, APlayerController* playerController)
{
	float captureSizeX = RDSceneCapComp2D->TextureTarget->GetSurfaceWidth();
	float captureSizeY = RDSceneCapComp2D->TextureTarget->GetSurfaceHeight();
	const FTransform& Transform = PlayerCameraXForm;
	FMatrix ViewMatrix = Transform.ToInverseMatrixWithScale();
	FVector ViewLocation = Transform.GetTranslation();
	// swap axis st. x=z,y=x,z=y (unreal coord space) so that z is up
	ViewMatrix = ViewMatrix * FMatrix(
		FPlane(0, 0, 1, 0),
		FPlane(1, 0, 0, 0),
		FPlane(0, 1, 0, 0),
		FPlane(0, 0, 0, 1));
	const float FOV = RDSceneCapComp2D->FOVAngle * (float)PI / 360.0f;

	// Build projection matrix
	float XAxisMultiplier;
	float YAxisMultiplier;

	if (captureSizeX > captureSizeY)
	{
		// if the viewport is wider than it is tall
		XAxisMultiplier = 1.0f;
		YAxisMultiplier = captureSizeX / captureSizeY;
	}
	else
	{
		// if the viewport is taller than it is wide
		XAxisMultiplier = captureSizeY / captureSizeX;
		YAxisMultiplier = 1.0f;
	}

	FMatrix ProjectionMatrix = FReversedZPerspectiveMatrix(
		FOV,
		FOV,
		XAxisMultiplier,
		YAxisMultiplier,
		10,
		1000
		);

	const FMatrix ViewProjectionMatrix = ViewMatrix * ProjectionMatrix;

	FVector Xaxis = ViewProjectionMatrix.GetColumn(0);
	FVector Yaxis = ViewProjectionMatrix.GetColumn(1);
	FVector Zaxis = ViewProjectionMatrix.GetColumn(2);
	FVector Waxis = ViewProjectionMatrix.GetColumn(3);

	float XaxisW = ViewProjectionMatrix.M[3][0];
	float YaxisW = ViewProjectionMatrix.M[3][1];
	float ZaxisW = ViewProjectionMatrix.M[3][2];
	float WaxisW = ViewProjectionMatrix.M[3][3];

	material->SetVectorParameterValue("PortalVPM_Xaxis", FLinearColor(Xaxis.X, Xaxis.Y, Xaxis.Z, XaxisW));
	material->SetVectorParameterValue("PortalVPM_Yaxis", FLinearColor(Yaxis.X, Yaxis.Y, Yaxis.Z, YaxisW));
	material->SetVectorParameterValue("PortalVPM_Zaxis", FLinearColor(Zaxis.X, Zaxis.Y, Zaxis.Z, ZaxisW));
	material->SetVectorParameterValue("PortalVPM_Waxis", FLinearColor(Waxis.X, Waxis.Y, Waxis.Z, WaxisW));
}

Now how does the material know how to use these variables I wonder?

We created out own version of the engine’s screen space UV function as it does not work in VR properly.

So below is how we set up the material (with the functions output UV’s going into the vertex shader customised UV’s rather than the pixel shader UV’s)

And here is a screenshot of the function.

In order for the custom node to work, we added some HLSL that could make sense of the crazy variables we are feeding into it.

Here is the HLSL.

float4x4 vpm;vpm = float4x4(PortalVPM_Xaxis.x, PortalVPM_Yaxis.x, PortalVPM_Zaxis.x, PortalVPM_Waxis.x,PortalVPM_Xaxis.y, PortalVPM_Yaxis.y, PortalVPM_Zaxis.y, PortalVPM_Waxis.y,PortalVPM_Xaxis.z, PortalVPM_Yaxis.z, PortalVPM_Zaxis.z, PortalVPM_Waxis.z,PortalVPM_XaxisW, PortalVPM_YaxisW, PortalVPM_ZaxisW, PortalVPM_WaxisW);return mul(float4(In.xyz,1),vpm);

As you can see, it takes all the parameters we are feeding in, and created a matrix to be used in the rest of the custom maths.

Admittedly this got a little out of my depth and my colleague helped out a lot. But if you have any questions about how or why things need to be done, please give me a shout and I will make sure I get an answer for you.

Sure, It looks amazing, and also could help me peek how it works a off-axis projection parallax or whatever this concept it’s called. My main interest it’s to achieve something similars using headtracking and calculating the monitor as one of those portals. I’m researching how to begin and seems like I have a lot to learn from your work, so thanks for sharing.

Could you show the end results of this? I just want to make sure this is worth the time and effort to implement, since I’ve been looking and have found multiple ways to do portals and none of them have worked since they all suffer from the exact issue you had in the OP.

I actually tried to step by step go through this and I’m hitting a lot of bumps along the way. If you could help me out that would be awesome. I’m a little new to C++ in Unreal, so I don’t know how to properly implement those and make them work. Also, is there a Capture Component and Render Target in the scene already? It doesn’t seem as though the code creates one, but I’m also not sure where it would take one over. Questions like these are preventing me from stepping further with this. Any insight would be great, thanks.

Hi ! thanks for the snippet of code, I’m also trying to achieve a portal in VR.

So far with the code above applied to a simple plane I have a different result for each triangle of my plane, any idea ?

What I’m trying to achieve next is to get this portal working in stereo (maybe with 2 scene captures)

Hey Trex,

I haven’t looked at this for a long time, but I have a feeling that you should be able to do it without engine changes. I have a feeling they added a stereo option to scene capture components. As I said I haven’t had the chance to play around with it, but it might be that they added this for creating VR scene captures that have a stereo view (don’t appear flat).

If you get a chance to try it out, let me know how you get on, I would love to revisit this one day, and that is one of the first things I want to investigate.

P.S. I’m not sure if the code snippet is still 100% correct for current engine versions, I think I did this during 4.7 so some of the code might not apply correctly if they have made any changes in the classes mentioned above :slight_smile:

What is PlayerCameraXForm intended to be? World-space transform of the player camera? Would it be possible for you to post a screenshot of the portal blueprint using these nodes?

1 Like

Was anyone able to reproduce this in newer UE versions? Or to come up with some other way? I have tried to implement this in 5.3 but it doesnt seem to work correctly.
(I have the final scene multiplied and rotated by 90 on the “portal”.)