Custom GameViewportClient: Render off-screen

Development Rendering
, , , ,
1

I have created a custom GameViewportClient which is derived from UGameViewportClient.
I am overriding the Draw() call, but instead of calling Super::Draw(Viewport, SceneCanvas); in my custom Draw(), I’d like to do the rendering right there, in my custom Draw().

My custom Draw():

void UCustomGameViewportClient::Draw(FViewport * Viewport, FCanvas * SceneCanvas)
{
       check(SceneCanvas);
	FVector2D viewportSize;
	GetViewportSize(viewportSize);
	TArray<FColor> ColorBuffer;

	//~~~ Invalid View Port Size ~~~
	//if (viewportSize.X <= 0 || viewportSize.Y <= 0) return;
	//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	FSceneViewFamilyContext ViewFamily(FSceneViewFamily::ConstructionValues(
		Viewport,
		GetWorld()->Scene,
		EngineShowFlags)
		.SetRealtimeUpdate(true));

	FGameViewDrawer GameViewDrawer;

	ULocalPlayer* LocalPlayer = 
               Cast<ULocalPlayer>(playerController->GetLocalPlayer());

	if (LocalPlayer)
	{
		FVector	ViewLocation;
		FRotator	ViewRotation;

		EStereoscopicPass PassType = eSSP_LEFT_EYE;

		FSceneView* View = LocalPlayer->CalcSceneView(&ViewFamily, ViewLocation, 
                             ViewRotation, Viewport, &GameViewDrawer, PassType);
	}

	GetRendererModule().BeginRenderingViewFamily(SceneCanvas, &ViewFamily);

	auto renderTarget = SceneCanvas->GetRenderTarget();
	bool PixelsRead = renderTarget->ReadPixels(ColorBuffer);

}

The reason is that I don’t want the scene to be rendered to the UI - I want to render it to an external application.

Could someone please assist me?

2

Ever figured this one out? I tried to create a render target and canvas and then call Super with that but somehow it just does not render into it.

3

Ever figured this one out? I tried to create a render target and canvas and then call Super with that but somehow it just does not render into it.

4

I’ve managed to bypass the Super::Draw(Viewport, SceneCanvas); but that’s about it. I’d really like to be able to just get the raw frame data so that I can stream it or do something else to it, without displaying it… very rarely do you get a response on thsee forums, unfortunately… so not much help :frowning: if you like, I will post how I bypassed the Super::Draw(Viewport, SceneCanvas);

5

What do you mean by bypassed? Can you just post the code you did?

6

I never call Super::Draw(Viewport, SceneCanvas);. I will try post tonight when I get home but if not, I will post it tomorrow.

7

Updated code to show what I mean.

8

Stumbled upon this post when I was looking for a solution for this. Now I found a solution. The code I am posting here is used to apply a Material to a plane and render out an orthographic view onto that plane to save the material as a texture.

class MaterialRenderViewportClient : public FCommonViewportClient {

public:
	virtual void Draw(FViewport *Viewport, FCanvas *Canvas) override {
		FEngineShowFlags EngineShowFlags(ESFIM_Editor);
		EngineShowFlags.DisableAdvancedFeatures();
		EngineShowFlags.SetSnap(0);
		EngineShowFlags.SetSeparateTranslucency(true);
		EngineShowFlags.Rendering = true;

		//Set Viewmode to unlit
		ApplyViewMode(EViewModeIndex::VMI_Unlit, false, EngineShowFlags);

		FSceneViewFamilyContext ViewFamily(FSceneViewFamily::ConstructionValues(
			Canvas->GetRenderTarget(),
			PreviewScene.GetScene(),
			EngineShowFlags)
			.SetRealtimeUpdate(false));

		//Init Views
		FSceneViewInitOptions ViewInitOptions;

		FViewportCameraTransform& ViewTransform = ViewTransformOrthographic;
		const ELevelViewportType EffectiveViewportType = ELevelViewportType::LVT_OrthoFreelook;

		ViewInitOptions.ViewOrigin = ViewTransform.GetLocation();
		FRotator ViewRotation = ViewTransform.GetRotation();
		const FIntPoint ViewportSizeXY = Viewport->GetSizeXY();
		FIntRect ViewRect = FIntRect(0, 0, ViewportSizeXY.X, ViewportSizeXY.Y);
		ViewInitOptions.SetViewRectangle(ViewRect);

		ViewInitOptions.ViewRotationMatrix = FMatrix(
			FPlane(0, 0, 1, 0),
			FPlane(1, 0, 0, 0),
			FPlane(0, 1, 0, 0),
			FPlane(0, 0, ViewInitOptions.ViewOrigin.X, 1));
		
		float ZScale = 0.5f / HALF_WORLD_MAX;
		float ZOffset = HALF_WORLD_MAX;

		//Zoom plane to fill whole Viewport!
		const float Zoom = 100.0f / ViewportSizeXY.X;

		float OrthoWidth = Zoom * ViewportSizeXY.X / 2.0f;
		float OrthoHeight = Zoom * ViewportSizeXY.Y / 2.0f;
		
		ViewInitOptions.ProjectionMatrix = FReversedZOrthoMatrix(
			OrthoWidth,
			OrthoHeight,
			ZScale,
			ZOffset
		);

		ViewInitOptions.ViewFamily = &ViewFamily;

		ViewInitOptions.BackgroundColor = FColor(0, 0, 255);
		ViewInitOptions.OverrideLODViewOrigin = FVector::ZeroVector;
		ViewInitOptions.bUseFauxOrthoViewPos = true;

		FSceneView* View = new FSceneView(ViewInitOptions);

		View->ViewLocation = ViewTransform.GetLocation();
		View->ViewRotation = ViewRotation;

		View->SubduedSelectionOutlineColor = GEngine->GetSubduedSelectionOutlineColor();

		ViewFamily.Views.Add(View);
		//Init Views End

		// Draw the 3D scene
		GetRendererModule().BeginRenderingViewFamily(Canvas, &ViewFamily);
	};
	
	/** Viewport camera transform data for orthographic viewports */
	FViewportCameraTransform		ViewTransformOrthographic;

	virtual UWorld* GetWorld() const override { return PreviewScene.GetWorld(); }
	/** Sets the location of the viewport's camera */
	void SetViewLocation(const FVector& NewLocation)
	{
		FViewportCameraTransform& ViewTransform = ViewTransformOrthographic;
		ViewTransform.SetLocation(NewLocation);
	}
	
	/** Sets the location of the viewport's camera */
	void SetViewRotation(const FRotator& NewRotation)
	{
		FViewportCameraTransform& ViewTransform = ViewTransformOrthographic;
		ViewTransform.SetRotation(NewRotation);
	}
	
	void SetViewLocationForOrbiting(const FVector& LookAtPoint, float DistanceToCamera = 256.f)
	{
		FMatrix Matrix = FTranslationMatrix(-GetViewLocation());
		Matrix = Matrix * FInverseRotationMatrix(GetViewRotation());
		FMatrix CamRotMat = Matrix.InverseFast();
		FVector CamDir = FVector(CamRotMat.M[0][0], CamRotMat.M[0][1], CamRotMat.M[0][2]);
		SetViewLocation(LookAtPoint - DistanceToCamera * CamDir);
		SetLookAtLocation(LookAtPoint);
	}
	
	/**
	* Sets the look at location of the viewports camera for orbit *
	*
	* @param LookAt The new look at location
	* @param bRecalulateView	If true, will recalculate view location and rotation to look at the new point immediatley
	*/
	void SetLookAtLocation(const FVector& LookAt, bool bRecalculateView = false)
	{
		FViewportCameraTransform& ViewTransform = ViewTransformOrthographic;
	
		ViewTransform.SetLookAt(LookAt);
	
		if (bRecalculateView)
		{
			FMatrix OrbitMatrix = ViewTransform.ComputeOrbitMatrix();
			OrbitMatrix = OrbitMatrix.InverseFast();
	
			ViewTransform.SetRotation(OrbitMatrix.Rotator());
			ViewTransform.SetLocation(OrbitMatrix.GetOrigin());
		}
	}
	/** @return the current viewport camera location */
	const FVector& GetViewLocation() const
	{
		const FViewportCameraTransform& ViewTransform = ViewTransformOrthographic;
		return ViewTransform.GetLocation();
	}
	
	/** @return the current viewport camera rotation */
	const FRotator& GetViewRotation() const
	{
		const FViewportCameraTransform& ViewTransform = ViewTransformOrthographic;
		return ViewTransform.GetRotation();
	}

	UMaterial *RenderedMaterial;
	FPreviewScene PreviewScene;
};

The above code was mostly assembled from Editor code (in /Editor/UnrealEd/Private/EditorViewportClient.cpp from void FEditorViewportClient::Draw()).

Then made a Viewport for offscreen rendering:

//copy of FDummyViewport::~FDummyViewport to fix link error, because FDummyViewport's desctructor is not exported in Engine-Module.
FDummyViewport::~FDummyViewport() {
	if (DebugCanvas != NULL)
	{
		delete DebugCanvas;
		DebugCanvas = NULL;
	}
}

class FOffScreenViewport : public FDummyViewport {
public:
	FOffScreenViewport(FViewportClient* InViewportClient) : FDummyViewport(InViewportClient) {};
	virtual ~FOffScreenViewport() override {};
	void RenderOffscreen(int WidthAndHeight) {
		this->SizeX = WidthAndHeight;
		this->SizeY = WidthAndHeight;
		BeginInitResource(this);

		EnqueueBeginRenderFrame();

		FCanvas Canvas(this, NULL, ViewportClient->GetWorld(), ViewportClient->GetWorld()->FeatureLevel);
		{
			ViewportClient->Draw(this, &Canvas);
		}
		Canvas.Flush_GameThread();
		FlushRenderingCommands();
		//Get Frame data here
		TArray<FColor> Bitmap;
		if (ReadPixels(Bitmap, FReadSurfaceDataFlags())) {
			// Create screenshot folder if not already present.
			IFileManager::Get().MakeDirectory(*FPaths::ScreenShotDir(), true);

			const FString ScreenFileName(FPaths::ScreenShotDir() / TEXT("VisualizeTexture"));

			uint32 ExtendXWithMSAA = Bitmap.Num() / WidthAndHeight;

			// Save the contents of the array to a bitmap file. (24bit only so alpha channel is dropped)
			FFileHelper::CreateBitmap(*ScreenFileName, ExtendXWithMSAA, WidthAndHeight, Bitmap.GetData());

			UE_LOG(LogConsoleResponse, Display, TEXT("Content was saved to \"%s\""), *FPaths::ScreenShotDir());
		}
		ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
			EndDrawingCommand,
			FViewport*, Viewport, this,
			{
				Viewport->EndRenderFrame(RHICmdList, false, false);
			});

		BeginReleaseResource(this);
		FlushRenderingCommands();
	};
};

This code was mostly taken from FViewport::HighResScreenshot()

A little HelperClass to initialize everything and kick off the rendering:

class MaterialRenderHelper {
public:
	TSharedPtr<MaterialRenderViewportClient> EditorViewportClient;
	UMaterial* Material;
	UStaticMesh* StaticMesh;
public:
	void RenderOffScreen(int WidthAndHeight) {

		EditorViewportClient = MakeShareable(new MaterialRenderViewportClient());

		EditorViewportClient->SetViewLocation(FVector::ZeroVector);
		EditorViewportClient->SetViewRotation(FRotator::ZeroRotator);
		EditorViewportClient->SetViewLocationForOrbiting(FVector::ZeroVector);

		EditorViewportClient->PreviewScene.SetLightDirection(FRotator(-40.0f, 27.5f, 0.0f));
		UStaticMeshComponent* NewSMComponent = NewObject<UStaticMeshComponent>(GetTransientPackage(), NAME_None, RF_Transient);
		NewSMComponent->SetStaticMesh(StaticMesh);
		FTransform Transform = FTransform::Identity;
		NewSMComponent->OverrideMaterials.Empty();
		NewSMComponent->OverrideMaterials.Add(Material);

		EditorViewportClient->PreviewScene.AddComponent(NewSMComponent, Transform);
		NewSMComponent->SetWorldRotation(FRotator(0.0, 90.0, 90.0));

		//Viewport initialization
		FOffScreenViewport* OffScreenRenderViewport = new FOffScreenViewport(&(*EditorViewportClient));

		OffScreenRenderViewport->RenderOffscreen(WidthAndHeight);

		delete OffScreenRenderViewport;
	};
};

Which I currently use this way:

	MaterialRenderHelper *MatRenderer = new MaterialRenderHelper();
	
	UStaticMesh *Plane = Cast<UStaticMesh>(StaticLoadObject(UStaticMesh::StaticClass(), NULL, TEXT("StaticMesh'/Engine/BasicShapes/Plane.Plane'")));;
	if (IsValid(Plane)) {
		UE_LOG(LogTemp, Warning, TEXT("Asset found!"));
	} else {
		UE_LOG(LogTemp, Warning, TEXT("Couldn't find asset!"));
	}
	MatRenderer->Material = Material;
	MatRenderer->StaticMesh = IsValid(StaticMesh)  ?  StaticMesh : Plane;
	MatRenderer->RenderOffScreen(WidthAndHeight);

I am aware of the fact that this code isn’t very nice, but it’s an early version which includes everything important and I wanted to share this solution.

9

This code is for off screen rendering in editor mode. I want to do it in game mode. How to do?

10

Unfortunately this code wont work in game mode, even though you cut the UnrealEd dependancy.