x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Bounding Box from specific View (SceneCapture)

I want to get the (2D) Bounding Box of an Actor from a fixed view (not player). So I created a blueprint that inherits SceneCapture2D and has a reference to the actor (as public variable) - so I can get its bounding box.

But the "Project World to Screen" function needs an player as Input, and when I cast the blueprint to a player, a warning pops up saying that the cast will fail:

alt text

How do I get the projection from the actor to the Scenecapture?

And how can I draw the bounding box on a texture (for debugging)?

EDIT: Due to This Post I was wrong when I talked about viewports, since the SceneCapture is no ViewPort in sense of rendering. So I changed it to "view"...

Product Version: UE 4.18
Tags:
more ▼

asked Sep 26 '18 at 09:22 AM in Blueprint Scripting

avatar image

DaKenpachi
79 5 7 13

avatar image ThompsonN13 Sep 26 '18 at 01:39 PM

your cast in failing because the actor your currently working in doe not inherit from the player controller class. so you are not casting the player to playercontroller but rather some other class.

avatar image DaKenpachi Sep 26 '18 at 02:00 PM

Hm right - is there a way to convert this blueprint so that it inherits player control? Or do I have to build it again? =/

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

5 answers: sort voted first

I finally found the solution by using a FMinimalViewInfo and a FSceneViewProjectionData. Though there still was some transformation problem, which I fixed by looking at the UGamePlayStatics:ProjectWorldToScreen( APlayerController const* PlayerController, const FVector & WorldPosition, FVector2D & ScreenPosition, bool bPlayerViewportRelative ) code. They applied some magic rotation matrix which I still dont get, but it works! xD

So now, here is my code:

 bool UMyPixelUtility::calcBoundingFromViewInfo(USceneCaptureComponent2D * RenderComponent, FVector Origin, FVector Extend, FBox2D & BoxOut, TArray<FVector>& Points, TArray<FVector2D>& Points2D)
 {
     bool isCompletelyInView = true;
     // get render target for texture size
     UTextureRenderTarget2D* RenderTexture = RenderComponent->TextureTarget;
     FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
     // initialise viewinfo for projection matrix
     FMinimalViewInfo Info;
     Info.Location = RenderComponent->GetComponentTransform().GetLocation();
     Info.Rotation = RenderComponent->GetComponentTransform().GetRotation().Rotator();
     Info.FOV = RenderComponent->FOVAngle;
     Info.ProjectionMode = RenderComponent->ProjectionType;
     Info.AspectRatio = float(RenderTexture->SizeX) / float(RenderTexture->SizeY);
     Info.OrthoNearClipPlane = 1;
     Info.OrthoFarClipPlane = 1000;
     Info.bConstrainAspectRatio = true;
     // calculate 3D corner Points of bounding box
     Points.Add(Origin + FVector(Extend.X, Extend.Y, Extend.Z));
     Points.Add(Origin + FVector(-Extend.X, Extend.Y, Extend.Z));
     Points.Add(Origin + FVector(Extend.X, -Extend.Y, Extend.Z));
     Points.Add(Origin + FVector(-Extend.X, -Extend.Y, Extend.Z));
     Points.Add(Origin + FVector(Extend.X, Extend.Y, -Extend.Z));
     Points.Add(Origin + FVector(-Extend.X, Extend.Y, -Extend.Z));
     Points.Add(Origin + FVector(Extend.X, -Extend.Y, -Extend.Z));
     Points.Add(Origin + FVector(-Extend.X, -Extend.Y, -Extend.Z));
     // initialize pixel values
     FVector2D MinPixel(RenderTexture->SizeX, RenderTexture->SizeY);
     FVector2D MaxPixel(0, 0);
     FIntRect ScreenRect(0, 0, RenderTexture->SizeX, RenderTexture->SizeY);
     // initialize projection data for sceneview
     FSceneViewProjectionData ProjectionData;
     ProjectionData.ViewOrigin = Info.Location;
     // do some voodoo rotation that is somehow mandatory and stolen from UGameplayStatics::ProjectWorldToScreen
     ProjectionData.ViewRotationMatrix = FInverseRotationMatrix(Info.Rotation) * FMatrix(
         FPlane(0, 0, 1, 0),
         FPlane(1, 0, 0, 0),
         FPlane(0, 1, 0, 0),
         FPlane(0, 0, 0, 1));
     if (RenderComponent->bUseCustomProjectionMatrix == true) {
         ProjectionData.ProjectionMatrix = RenderComponent->CustomProjectionMatrix;
     }
     else {
         ProjectionData.ProjectionMatrix = Info.CalculateProjectionMatrix();;
     }
     ProjectionData.SetConstrainedViewRectangle(ScreenRect);
     // Project Points to pixels and get the corner pixels
     for (FVector& Point : Points) {
         FVector2D Pixel(0, 0);
         FSceneView::ProjectWorldToScreen((Point), ScreenRect, ProjectionData.ComputeViewProjectionMatrix(), Pixel);
         Points2D.Add(Pixel);
         MaxPixel.X = FMath::Max(Pixel.X, MaxPixel.X);
         MaxPixel.Y = FMath::Max(Pixel.Y, MaxPixel.Y);
         MinPixel.X = FMath::Min(Pixel.X, MinPixel.X);
         MinPixel.Y = FMath::Min(Pixel.Y, MinPixel.Y);
     }
 
     BoxOut = FBox2D(MinPixel, MaxPixel);
     // clamp min point
     if (BoxOut.Min.X < 0) {
         BoxOut.Min.X = 0;
         isCompletelyInView = false;
     }
     else if (BoxOut.Min.X > RenderTexture->SizeX) {
         BoxOut.Min.X = RenderTexture->SizeX;
         isCompletelyInView = false;
     }
     if (BoxOut.Min.Y < 0) {
         BoxOut.Min.Y = 0;
         isCompletelyInView = false;
     }
     else if (BoxOut.Min.Y > RenderTexture->SizeY) {
         BoxOut.Min.Y = RenderTexture->SizeY;
         isCompletelyInView = false;
     }
     // clamp max point
     if (BoxOut.Max.X > RenderTexture->SizeX) {
         BoxOut.Max.X = RenderTexture->SizeX;
         isCompletelyInView = false;
     }
     else if (BoxOut.Max.X < 0) {
         BoxOut.Max.X = 0;
         isCompletelyInView = false;
     }
     if (BoxOut.Max.Y > RenderTexture->SizeY) {
         BoxOut.Max.Y = RenderTexture->SizeY;
         isCompletelyInView = false;
     }
     else if (BoxOut.Max.Y < 0) {
         BoxOut.Max.Y = 0;
         isCompletelyInView = false;
     }
     return isCompletelyInView;
 }

(I added some clamping to the texture size when the actor is not in view)

more ▼

answered Dec 19 '18 at 12:40 PM

avatar image

DaKenpachi
79 5 7 13

avatar image DaKenpachi Dec 19 '18 at 01:00 PM

alt text

If someone wants to see the result: The 3D Bounding box is shown with green lines using DrawDebugBox. Then I projected it onto the RenderTarget of a SceneCapture Component, drawing the projected 2D corner Points in orange and the 2D Bounding Box in violet directly onto the RenderTarget.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

You can use the get player controller node to get a reference to the player controller.

more ▼

answered Sep 26 '18 at 09:58 AM

avatar image

HarryHighDef
1.4k 2 14 17

avatar image DaKenpachi Sep 26 '18 at 01:11 PM

But then I would get the projection from the actor to the players´ coordinate system, not the one of the SceneCapture, wouldn´t I?

avatar image HarryHighDef Oct 09 '18 at 09:07 AM

Why not temporarily move the players camera (viewport) into the position you need?

avatar image DaKenpachi Oct 19 '18 at 09:21 AM

Because I want multiple different viewports having those bounding boxes at the same time, without changing the location of the player.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

For now I did find a workaround: I used Custom Depth Stencil to get a binary render texture (upper left in the picture) with only the Actor being white:

Differend RenderTargets

Then I used a simple bounding box algorithm:

 FBox2D UTryPixelAccess::calcBoundingFromBinary(UTextureRenderTarget2D * RenderTexture)
 {
     TArray<FLinearColor> ImageData;
     FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
     RenderTarget->ReadLinearColorPixels(ImageData);
 
     FVector2D maxPoint(0, 0);
     FVector2D minPoint(RenderTexture->SizeX, RenderTexture->SizeY);
 
     for (int x = 0; x < RenderTexture->SizeX; x++) {
         for (int y = 0; y < RenderTexture->SizeY; y++) {
             int i = x + y * RenderTexture->SizeX;
             if (ImageData[i].R > 0.1) {
                 // we a have white pixel
                 if (x <= minPoint.X) {
                     minPoint.X = x;
                 }
                 if(y <= minPoint.Y) {
                     minPoint.Y = y;
                 }
                 if (x >= maxPoint.X) {
                     maxPoint.X = x;
                 }
                 if (y >= maxPoint.Y) {
                     maxPoint.Y = y;
                 }
             }
         }
     }
 
     return FBox2D(minPoint, maxPoint);
 }

Though, this is not very performant so if anyone has a better Idea, please reply!

more ▼

answered Oct 09 '18 at 08:51 AM

avatar image

DaKenpachi
79 5 7 13

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

So, I investigated a bit further (e.g. looking into the code of ProjectWorldToScreen and trying to understand it).

This is what came out:

 FBox2D UTryPixelAccess::calcBoundingFromBox(USceneCaptureComponent2D * RenderComponent, FBox BoundingBox3D)
 {
     UTextureRenderTarget2D* RenderTexture = RenderComponent->TextureTarget;
     FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
 
     FVector2D ScreenPosition;
     FVector2D maxPoint(0, 0);
     FVector2D minPoint(RenderTexture->SizeX, RenderTexture->SizeY);
     FIntRect ScreenRect(FIntPoint(0, 0), RenderTarget->GetSizeXY());
     for (int i = 0; i < 8; i++) {
         FVector cornerPoint = BoundingBox3D.GetExtrema(i);
         bool bResult = FSceneView::ProjectWorldToScreen(cornerPoint, ScreenRect, RenderComponent->CustomProjectionMatrix, ScreenPosition);
 
         minPoint.X = FMath::Min(ScreenPosition.X, minPoint.X);
         minPoint.X = FMath::Min(ScreenPosition.Y, minPoint.Y);
         maxPoint.X = FMath::Max(ScreenPosition.X, maxPoint.X);
         maxPoint.X = FMath::Max(ScreenPosition.Y, maxPoint.Y);
 
     }
 
     FBox2D box(minPoint, maxPoint);
     double end = FPlatformTime::Seconds();
     UE_LOG(PixelAccessLog, Log, TEXT("Generated Bounding Box %s in %f ms"), *box.ToString(), (end - start)*1000.0f);
     return box;
 }

BUT the problematic thing is, I dont really use a CustomProjectionMatrix in my RenderComponent - so its just the Identitymatrix there when I call it. The projection obviously goes wrong!

Is there any conversion from a "normal" SceneCaptureComponent to it´s equivalent Projection Matrix?

more ▼

answered Oct 19 '18 at 12:18 PM

avatar image

DaKenpachi
79 5 7 13

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
 void YourClass::BuildProjectionMatrix(FIntPoint RenderTargetSize, ECameraProjectionMode::Type ProjectionType, float FOV, float InOrthoWidth, FMatrix& ProjectionMatrix)
 {
     float const XAxisMultiplier = 1.0f;
     float const YAxisMultiplier = RenderTargetSize.X / (float)RenderTargetSize.Y;
 
     if (ProjectionType == ECameraProjectionMode::Orthographic)
     {
         check((int32)ERHIZBuffer::IsInverted);
         const float OrthoWidth = InOrthoWidth / 2.0f;
         const float OrthoHeight = InOrthoWidth / 2.0f * XAxisMultiplier / YAxisMultiplier;
 
         const float NearPlane = 0;
         const float FarPlane = WORLD_MAX / 8.0f;
 
         const float ZScale = 1.0f / (FarPlane - NearPlane);
         const float ZOffset = -NearPlane;
 
         ProjectionMatrix = FReversedZOrthoMatrix(
             OrthoWidth,
             OrthoHeight,
             ZScale,
             ZOffset
         );
     }
     else
     {
         if ((int32)ERHIZBuffer::IsInverted)
         {
             ProjectionMatrix = FReversedZPerspectiveMatrix(
                 FOV,
                 FOV,
                 XAxisMultiplier,
                 YAxisMultiplier,
                 GNearClippingPlane,
                 GNearClippingPlane
             );
         }
         else
         {
             ProjectionMatrix = FPerspectiveMatrix(
                 FOV,
                 FOV,
                 XAxisMultiplier,
                 YAxisMultiplier,
                 GNearClippingPlane,
                 GNearClippingPlane
             );
         }
     }
 }

now you can get the projection matrix by giving FOVAngle in dgree(capture2D->FOVAngle * (float)PI / 360.0f) and OrthoWidth.

more ▼

answered Dec 17 '18 at 02:57 PM

avatar image

FIHOTCOOL
11 2 2 5

avatar image DaKenpachi Dec 19 '18 at 12:40 PM

Hi, thanks for your answer. In fact, I allready fixed the problem using a FMinimalViewInfo and FSceneViewProjectionData.

Your answer is great too, but seems kind of duplicate to the code of FMinimalView:

 FMatrix FMinimalViewInfo::CalculateProjectionMatrix() const
 {
     FMatrix ProjectionMatrix;
 
     if (ProjectionMode == ECameraProjectionMode::Orthographic)
     {
         const float YScale = 1.0f / AspectRatio;
 
         const float HalfOrthoWidth = OrthoWidth / 2.0f;
         const float ScaledOrthoHeight = OrthoWidth / 2.0f * YScale;
 
         const float NearPlane = OrthoNearClipPlane;
         const float FarPlane = OrthoFarClipPlane;
 
         const float ZScale = 1.0f / (FarPlane - NearPlane);
         const float ZOffset = -NearPlane;
 
         ProjectionMatrix = FReversedZOrthoMatrix(
             HalfOrthoWidth,
             ScaledOrthoHeight,
             ZScale,
             ZOffset
             );
     }
     else
     {
         // Avoid divide by zero in the projection matrix calculation by clamping FOV
         ProjectionMatrix = FReversedZPerspectiveMatrix(
             FMath::Max(0.001f, FOV) * (float)PI / 360.0f,
             AspectRatio,
             1.0f,
             GNearClippingPlane );
     }
 
     if (!OffCenterProjectionOffset.IsZero())
     {
         const float Left = -1.0f + OffCenterProjectionOffset.X;
         const float Right = Left + 2.0f;
         const float Bottom = -1.0f + OffCenterProjectionOffset.Y;
         const float Top = Bottom + 2.0f;
         ProjectionMatrix.M[2][0] = (Left + Right) / (Left - Right);
         ProjectionMatrix.M[2][1] = (Bottom + Top) / (Bottom - Top);
     }
 
     return ProjectionMatrix;
 }
(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question