Scene Capture Image is darker than original game

[AJMW (Arvind Jayaraman)]

I am noticing that the captured image from using a scene-capture 2d component is much darker than the original image in the game? Anyone knows why this could be happening?

Here’s the code to capture this image and transpose it so it can be processed in MATLAB

void UMyBlueprintFunctionLibrary::GetUInt8MWImageFromScene(UTextureRenderTarget2D *texture2DRenderTarget, TArray<uint8> &RGB_MATLAB){
	TArray<FColor> TexturePixels;
	FTextureRenderTargetResource * fTextureRenderTargetResource = texture2DRenderTarget->GameThread_GetRenderTargetResource();
	int32 TexWidth = texture2DRenderTarget->SizeX;
	int32 TexHeight = texture2DRenderTarget->SizeY;
	int32 ImageSize = TexWidth * TexHeight;
	TexturePixels.AddUninitialized(ImageSize);
	RGB_MATLAB.AddUninitialized(ImageSize * 3);
	bool bReadSuccess = fTextureRenderTargetResource->ReadPixels(TexturePixels);
	int32 row = 0;
	int32 col = 0;
	int32 index = 0;
	int32 mlIndex = 0;
	if (bReadSuccess){
		for (col = 0; col < TexWidth; col++){
			for (row = 0; row < TexHeight; row++){
				index = row * TexWidth + col;
				RGB_MATLAB[mlIndex] = TexturePixels[index].R;
				RGB_MATLAB[mlIndex + ImageSize] = TexturePixels[index].G;
				RGB_MATLAB[mlIndex + 2 * ImageSize] = TexturePixels[index].B;
				mlIndex++;
			}
		}
	}
}

When I then write this image to a png file- it appears much darker than the scene in the game.

Here is a comparison of the captured output

and the original scene

Here’s the scenecapture2d’s and render target settings

[anonymous_user_cc69b907]

Hello AJMW,

From what you described and your screenshots you have a Screen Capture 2D in your scene. That Screen Capture is returning a darker value than what you expect to see.

Scene Capture 2D Actors do not work like real reflections; they work more like a TV screen receiving an image from a camera. The Scene Capture 2D captures the scene from its view frustum, stores that view as an image, which is then used within a Material.

In the last screenshot you posted there is an area labeled Adjustments. I would try the brightness and vibrance in that section.

Please let me know if these troubleshoots work for you.

Thank you,

[AJMW (Arvind Jayaraman)]

Hi ,

Thanks for the suggestions. I tried tweaking both the adjustment settings, brightness and vibrance. It seems to have no impact whatsoever on the image captured from the scene. It appears as if the raw pixel values that are obtained with the scene-capture2d + render-target setting are still way too dark to carry out any useful computer-vision tasks.

I am wondering if this is related to how the rendering engine is somehow compensating for gamma correction.

Regards,
Arvind

[anonymous_user_cc69b907]

Hello AJMW,

So, I did a little more research on your issue. In the details panel of your Scene Capture 2D there is a a post processing section.

Tweaking the eye adaptation/auto exposure will help keep your lighting at a steady level. If you go to the auto exposure section and st your min Brightness/Max Brightness and your Exposure Bias all to one that will help.

Also it appears that your car has tinted windows. You may try changing just how dark those windows are. So the issue could lie that you are in a shadow looking through tinted windows.

Let me know if this helps.

Thanks,

[AJMW (Arvind Jayaraman)]

Thanks , I was able to adjust the Exposure-Bias setting and increase it up to ~5 in order to obtain a well-lit and bright image. This solved my problem!

[anonymous_user_9447d77f]

set USceneCaptureComponent2D 's capture source equals SCS_FinalColorLDR; and defalt is Scene Color (HDR).
code is below:
this->GetCaptureComponent2D()->CaptureSource = SCS_FinalColorLDR;

[qiuwch]

I think changing Exposure-Bias setting is not very elegant and sometimes inaccurate.

Another reason to consider is the gamma setting of the SceneCapture2D.
Here are two related snippets from the UnrealCV project

CaptureComponent->TextureTarget->TargetGamma = 1;

FReadSurfaceDataFlags ReadSurfaceDataFlags;
ReadSurfaceDataFlags.SetLinearToGamma(false);
RenderTargetResource->ReadPixels(Image, ReadSurfaceDataFlags);

[MorskoyZmey]

Works perfect for me. Magic numbers…

[OverRated_AU]

Set the marital to UI.

[Rumbleball]

Combined all from above to get good result.

1. Brightness/Max Brightness and your Exposure Bias all to one
2. CaptureComponent2D->CaptureSource = SCS_FinalColorLDR
3. RenderTarget2D->TargetGamma = 1.2

Thanks guys

[Jonas_Molgaard]

Last settings by @Rumbleball does not work for me… or well. If i set to SCS_FinalColorLDR, but that renders my image in 8bit it seems. And obviously i want it to be 16bit. I still can’t find a solution for this. Is it not a bug? I mean, it works for LDR, but not for HDR…

[plastik21]

Try to play with tone mapping parameters of your scene capture component. I reached almost identical picture after an hour of playing with Slope and Toe parameters. Just for example, my values: Slope: 2.5, Toe: -1.5
Edit: I haven’t used gamma correction for render target

[Rumbleball]

Yeah, got good results for a single lightning setting. Was broken after changing the light…

Just played with it again a week ago, took me 6 hours to get some good results. Still not sure what I actually did. Gamma on the RenderTarget for example does not seem to have any effect on the image on my end. While trying to get good results my first used RenderTarget seemed broken, had to make a new one.

I have not done rendering at all yet, but I don’t get it whats so difficult about rendering the scene as the player sees it. The Viewport is a RenderTarget as well. Does the SceneCapture2D use a different renderpipeline?

[Jonas_Molgaard]

Can someone pls post a video showing how they get identical results? This is either a bug or i am doing something wrong. If it is indeed something in the scene that needs to be present in order to get identical lightning, it would show up on a video. So someone pretty pls

[soylent_me]

Hi Qiuwch - Looks like you may have the best solution for this, just wanted to check your current opinion on the best way to get a 1-to-1 match with a scenecap2d. Thanks!

[anonymous_user_9c6a5a9b]

In case it’s useful, the way we got around this issue was updating our post process material’s tonemapping settings to “Before Tonemapping”

[mister-liss]

But this only applies if you’re using a post processing material though I assume. I think myself and most others are using the default UE post process stack.

[phil123456]

how do you achieve that in a blueprint ?