Bounding Box from specific View (SceneCapture)

[anonymous_user_c3e69a5a]

I want to get the (2D) Bounding Box of an Actor from a fixed view (not player).
So I created a blueprint that inherits SceneCapture2D and has a reference to the actor (as public variable) - so I can get its bounding box.

But the “Project World to Screen” function needs an player as Input, and when I cast the blueprint to a player, a warning pops up saying that the cast will fail:

How do I get the projection from the actor to the Scenecapture?

And how can I draw the bounding box on a texture (for debugging)?

EDIT: Due to [This Post][2] I was wrong when I talked about viewports, since the SceneCapture is no ViewPort in sense of rendering. So I changed it to “view”…

[HarryHighDef]

You can use the get player controller node to get a reference to the player controller.

[anonymous_user_c3e69a5a]

But then I would get the projection from the actor to the players´ coordinate system, not the one of the SceneCapture, wouldn´t I?

[ThompsonN13]

your cast in failing because the actor your currently working in doe not inherit from the player controller class. so you are not casting the player to playercontroller but rather some other class.

[anonymous_user_c3e69a5a]

Hm right - is there a way to convert this blueprint so that it inherits player control?
Or do I have to build it again? =/

[anonymous_user_c3e69a5a]

For now I did find a workaround:
I used Custom Depth Stencil to get a binary render texture (upper left in the picture) with only the Actor being white:

Then I used a simple bounding box algorithm:

FBox2D UTryPixelAccess::calcBoundingFromBinary(UTextureRenderTarget2D * RenderTexture)
{
	TArray<FLinearColor> ImageData;
	FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
	RenderTarget->ReadLinearColorPixels(ImageData);

	FVector2D maxPoint(0, 0);
	FVector2D minPoint(RenderTexture->SizeX, RenderTexture->SizeY);

	for (int x = 0; x < RenderTexture->SizeX; x++) {
		for (int y = 0; y < RenderTexture->SizeY; y++) {
			int i = x + y * RenderTexture->SizeX;
			if (ImageData[i].R > 0.1) {
				// we a have white pixel
				if (x <= minPoint.X) {
					minPoint.X = x;
				}
				if(y <= minPoint.Y) {
					minPoint.Y = y;
				}
				if (x >= maxPoint.X) {
					maxPoint.X = x;
				}
				if (y >= maxPoint.Y) {
					maxPoint.Y = y;
				}
			}
		}
	}

	return FBox2D(minPoint, maxPoint);
}

Though, this is not very performant so if anyone has a better Idea, please reply!

[HarryHighDef]

Why not temporarily move the players camera (viewport) into the position you need?

[anonymous_user_c3e69a5a]

Because I want multiple different viewports having those bounding boxes at the same time, without changing the location of the player.

[anonymous_user_c3e69a5a]

So, I investigated a bit further (e.g. looking into the code of ProjectWorldToScreen and trying to understand it).

This is what came out:

FBox2D UTryPixelAccess::calcBoundingFromBox(USceneCaptureComponent2D * RenderComponent, FBox BoundingBox3D)
{
	UTextureRenderTarget2D* RenderTexture = RenderComponent->TextureTarget;
	FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();

	FVector2D ScreenPosition;
	FVector2D maxPoint(0, 0);
	FVector2D minPoint(RenderTexture->SizeX, RenderTexture->SizeY);
	FIntRect ScreenRect(FIntPoint(0, 0), RenderTarget->GetSizeXY());
	for (int i = 0; i < 8; i++) {
		FVector cornerPoint = BoundingBox3D.GetExtrema(i);
		bool bResult = FSceneView::ProjectWorldToScreen(cornerPoint, ScreenRect, RenderComponent->CustomProjectionMatrix, ScreenPosition);

		minPoint.X = FMath::Min(ScreenPosition.X, minPoint.X);
		minPoint.X = FMath::Min(ScreenPosition.Y, minPoint.Y);
		maxPoint.X = FMath::Max(ScreenPosition.X, maxPoint.X);
		maxPoint.X = FMath::Max(ScreenPosition.Y, maxPoint.Y);

	}

	FBox2D box(minPoint, maxPoint);
	double end = FPlatformTime::Seconds();
	UE_LOG(PixelAccessLog, Log, TEXT("Generated Bounding Box %s in %f ms"), *box.ToString(), (end - start)*1000.0f);
	return box;
}

BUT the problematic thing is, I dont really use a CustomProjectionMatrix in my RenderComponent - so its just the Identitymatrix there when I call it. The projection obviously goes wrong!

Is there any conversion from a “normal” SceneCaptureComponent to it´s equivalent Projection Matrix?

[FIHOTCOOL (weipengsong)]

void YourClass::BuildProjectionMatrix(FIntPoint RenderTargetSize, ECameraProjectionMode::Type ProjectionType, float FOV, float InOrthoWidth, FMatrix& ProjectionMatrix)
{
float const XAxisMultiplier = 1.0f;
float const YAxisMultiplier = RenderTargetSize.X / (float)RenderTargetSize.Y;

	if (ProjectionType == ECameraProjectionMode::Orthographic)
	{
		check((int32)ERHIZBuffer::IsInverted);
		const float OrthoWidth = InOrthoWidth / 2.0f;
		const float OrthoHeight = InOrthoWidth / 2.0f * XAxisMultiplier / YAxisMultiplier;

		const float NearPlane = 0;
		const float FarPlane = WORLD_MAX / 8.0f;

		const float ZScale = 1.0f / (FarPlane - NearPlane);
		const float ZOffset = -NearPlane;

		ProjectionMatrix = FReversedZOrthoMatrix(
			OrthoWidth,
			OrthoHeight,
			ZScale,
			ZOffset
		);
	}
	else
	{
		if ((int32)ERHIZBuffer::IsInverted)
		{
			ProjectionMatrix = FReversedZPerspectiveMatrix(
				FOV,
				FOV,
				XAxisMultiplier,
				YAxisMultiplier,
				GNearClippingPlane,
				GNearClippingPlane
			);
		}
		else
		{
			ProjectionMatrix = FPerspectiveMatrix(
				FOV,
				FOV,
				XAxisMultiplier,
				YAxisMultiplier,
				GNearClippingPlane,
				GNearClippingPlane
			);
		}
	}
}

now you can get the projection matrix by giving FOVAngle in dgree(capture2D->FOVAngle * (float)PI / 360.0f) and OrthoWidth.

[anonymous_user_c3e69a5a]

Hi, thanks for your answer. In fact, I allready fixed the problem using a FMinimalViewInfo and FSceneViewProjectionData.

Your answer is great too, but seems kind of duplicate to the code of FMinimalView:

FMatrix FMinimalViewInfo::CalculateProjectionMatrix() const
{
	FMatrix ProjectionMatrix;

	if (ProjectionMode == ECameraProjectionMode::Orthographic)
	{
		const float YScale = 1.0f / AspectRatio;

		const float HalfOrthoWidth = OrthoWidth / 2.0f;
		const float ScaledOrthoHeight = OrthoWidth / 2.0f * YScale;

		const float NearPlane = OrthoNearClipPlane;
		const float FarPlane = OrthoFarClipPlane;

		const float ZScale = 1.0f / (FarPlane - NearPlane);
		const float ZOffset = -NearPlane;

		ProjectionMatrix = FReversedZOrthoMatrix(
			HalfOrthoWidth,
			ScaledOrthoHeight,
			ZScale,
			ZOffset
			);
	}
	else
	{
		// Avoid divide by zero in the projection matrix calculation by clamping FOV
		ProjectionMatrix = FReversedZPerspectiveMatrix(
			FMath::Max(0.001f, FOV) * (float)PI / 360.0f,
			AspectRatio,
			1.0f,
			GNearClippingPlane );
	}

	if (!OffCenterProjectionOffset.IsZero())
	{
		const float Left = -1.0f + OffCenterProjectionOffset.X;
		const float Right = Left + 2.0f;
		const float Bottom = -1.0f + OffCenterProjectionOffset.Y;
		const float Top = Bottom + 2.0f;
		ProjectionMatrix.M[2][0] = (Left + Right) / (Left - Right);
		ProjectionMatrix.M[2][1] = (Bottom + Top) / (Bottom - Top);
	}

	return ProjectionMatrix;
}

[anonymous_user_c3e69a5a]

I finally found the solution by using a FMinimalViewInfo and a FSceneViewProjectionData. Though there still was some transformation problem, which I fixed by looking at the UGamePlayStatics:ProjectWorldToScreen( APlayerController const* PlayerController, const FVector & WorldPosition, FVector2D & ScreenPosition, bool bPlayerViewportRelative )
code. They applied some magic rotation matrix which I still dont get, but it works! xD

So now, here is my code:

bool UMyPixelUtility::calcBoundingFromViewInfo(USceneCaptureComponent2D * RenderComponent, FVector Origin, FVector Extend, FBox2D & BoxOut, TArray<FVector>& Points, TArray<FVector2D>& Points2D)
{
	bool isCompletelyInView = true;
	// get render target for texture size
	UTextureRenderTarget2D* RenderTexture = RenderComponent->TextureTarget;
	FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
	// initialise viewinfo for projection matrix
	FMinimalViewInfo Info;
	Info.Location = RenderComponent->GetComponentTransform().GetLocation();
	Info.Rotation = RenderComponent->GetComponentTransform().GetRotation().Rotator();
	Info.FOV = RenderComponent->FOVAngle;
	Info.ProjectionMode = RenderComponent->ProjectionType;
	Info.AspectRatio = float(RenderTexture->SizeX) / float(RenderTexture->SizeY);
	Info.OrthoNearClipPlane = 1;
	Info.OrthoFarClipPlane = 1000;
	Info.bConstrainAspectRatio = true;
	// calculate 3D corner Points of bounding box
	Points.Add(Origin + FVector(Extend.X, Extend.Y, Extend.Z));
	Points.Add(Origin + FVector(-Extend.X, Extend.Y, Extend.Z));
	Points.Add(Origin + FVector(Extend.X, -Extend.Y, Extend.Z));
	Points.Add(Origin + FVector(-Extend.X, -Extend.Y, Extend.Z));
	Points.Add(Origin + FVector(Extend.X, Extend.Y, -Extend.Z));
	Points.Add(Origin + FVector(-Extend.X, Extend.Y, -Extend.Z));
	Points.Add(Origin + FVector(Extend.X, -Extend.Y, -Extend.Z));
	Points.Add(Origin + FVector(-Extend.X, -Extend.Y, -Extend.Z));
	// initialize pixel values
	FVector2D MinPixel(RenderTexture->SizeX, RenderTexture->SizeY);
	FVector2D MaxPixel(0, 0);
	FIntRect ScreenRect(0, 0, RenderTexture->SizeX, RenderTexture->SizeY);
	// initialize projection data for sceneview
	FSceneViewProjectionData ProjectionData;
	ProjectionData.ViewOrigin = Info.Location;
	// do some voodoo rotation that is somehow mandatory and stolen from UGameplayStatics::ProjectWorldToScreen
	ProjectionData.ViewRotationMatrix = FInverseRotationMatrix(Info.Rotation) * FMatrix(
		FPlane(0, 0, 1, 0),
		FPlane(1, 0, 0, 0),
		FPlane(0, 1, 0, 0),
		FPlane(0, 0, 0, 1));
	if (RenderComponent->bUseCustomProjectionMatrix == true) {
		ProjectionData.ProjectionMatrix = RenderComponent->CustomProjectionMatrix;
	}
	else {
		ProjectionData.ProjectionMatrix = Info.CalculateProjectionMatrix();;
	}
	ProjectionData.SetConstrainedViewRectangle(ScreenRect);
	// Project Points to pixels and get the corner pixels
	for (FVector& Point : Points) {
		FVector2D Pixel(0, 0);
		FSceneView::ProjectWorldToScreen((Point), ScreenRect, ProjectionData.ComputeViewProjectionMatrix(), Pixel);
		Points2D.Add(Pixel);
		MaxPixel.X = FMath::Max(Pixel.X, MaxPixel.X);
		MaxPixel.Y = FMath::Max(Pixel.Y, MaxPixel.Y);
		MinPixel.X = FMath::Min(Pixel.X, MinPixel.X);
		MinPixel.Y = FMath::Min(Pixel.Y, MinPixel.Y);
	}

	BoxOut = FBox2D(MinPixel, MaxPixel);
	// clamp min point
	if (BoxOut.Min.X < 0) {
		BoxOut.Min.X = 0;
		isCompletelyInView = false;
	}
	else if (BoxOut.Min.X > RenderTexture->SizeX) {
		BoxOut.Min.X = RenderTexture->SizeX;
		isCompletelyInView = false;
	}
	if (BoxOut.Min.Y < 0) {
		BoxOut.Min.Y = 0;
		isCompletelyInView = false;
	}
	else if (BoxOut.Min.Y > RenderTexture->SizeY) {
		BoxOut.Min.Y = RenderTexture->SizeY;
		isCompletelyInView = false;
	}
	// clamp max point
	if (BoxOut.Max.X > RenderTexture->SizeX) {
		BoxOut.Max.X = RenderTexture->SizeX;
		isCompletelyInView = false;
	}
	else if (BoxOut.Max.X < 0) {
		BoxOut.Max.X = 0;
		isCompletelyInView = false;
	}
	if (BoxOut.Max.Y > RenderTexture->SizeY) {
		BoxOut.Max.Y = RenderTexture->SizeY;
		isCompletelyInView = false;
	}
	else if (BoxOut.Max.Y < 0) {
		BoxOut.Max.Y = 0;
		isCompletelyInView = false;
	}
	return isCompletelyInView;
}

(I added some clamping to the texture size when the actor is not in view)

[anonymous_user_c3e69a5a]

If someone wants to see the result: The 3D Bounding box is shown with green lines using DrawDebugBox. Then I projected it onto the RenderTarget of a SceneCapture Component, drawing the projected 2D corner Points in orange and the 2D Bounding Box in violet directly onto the RenderTarget.

[Gigoubigou]

Hi,
I’m facing a similar problem as you did and I try to implement your solution. As I will render my scene, I do not need performance so I wanted to use the custom depth stencil technique.
I was wondering how do you go from your custom depth stencil to a UTextureRenderTarget2D ? I can’t find a way to do that…

[anonymous_user_c3e69a5a]

Whow, this is really old and I haven´t done any work in UE since then

But, you can have a look at my Github, where I had a testing project which you can fork: GitHub - daKenpachi/UnrealProjectionTest: Test of 2D-3D Projections with Unreal Engine 4

Maybe you find what you are looking for, good luck! =)