How to sync SceneCaptureComponent2D captures

Hello. I am using UE4 for robot simulation and would like to employ several SceneCapture components as virtual cameras located on the robot. To do this, I have created a custom class, UCamera, which inherits from USceneComponent. When UCamera::BeginPlay is called, I create a internal instance of USceneCaptureComponent2D as follows:

bool UCamera::CreateSceneCapturer()
{
  sceneCapturer_ = NewObject<USceneCaptureComponent2D>();

  if (!sceneCapturer_)
  {
    UE_LOG(LogTemp, Warning, TEXT("Error creating scene capturer"));
    return false;
  }

  sceneCapturer_->AttachTo(this);
  sceneCapturer_->FOVAngle = 90;
  sceneCapturer_->bCaptureEveryFrame = true;
  sceneCapturer_->TextureTarget = renderTarget_;
  sceneCapturer_->DepthBuffer = &depthBuffer_;
  sceneCapturer_->CaptureSource = ESceneCaptureSource::SCS_FinalColorLDR;
  sceneCapturer_->PostProcessSettings.bOverride_AntiAliasingMethod = true;

  sceneCapturer_->PostProcessSettings.AntiAliasingMethod =
      EAntiAliasingMethod::AAM_TemporalAA;

  sceneCapturer_->RegisterComponentWithWorld(this->GetWorld());

  return true;
}

I then create an instance of UCamera in Unreal Editor and attach directly to the FirstPersonCamera in the FirstPersonShooter demo project. The component tree of the FirstPersonCharacter is as follows:

65705-capture.png

Every call to UCamera::TickComponent I call UpdateContent on my internal instance of USceneCaptureComponent2D and send there results of ReadPixels to an external program for visualization. This is my current result (my custom camera on top, the game’s main camera on bottom):

65706-unreal.gif

As you can see, my biggest issue is that my custom camera is not in sync with the other camera and character mesh. I’m not sure if my camera is ahead in time (captures before other objects have moved) or is ahead in position (capturing from the wrong location). I have tried playing with Tick Groups with no success. I have also seen another post that suggested using a SpringArm to make the SceneCapture camera lag behind. This gets closer to the results I want, but is not precise enough. To truly use this in a perception pipeline, the cameras need to be fixed to the robot and cannot be sloshing around.

Any ideas on what I could be doing wrong?

Many thanks!

EDIT: I should also mention that I have tried creating the same setup completely from the UE editor (without writing any C++ code) and I am getting the same results.

Hi,

I’m after the same problem. I investigated the issue on the engine side and came up with “Engine/Source/Runtime/Renderer/Private/SceneCaptureRendering.cpp”. The capture is rendered in a seperated thread. I think it’s a bit time consuming operation so that it lags from the camera view. I don’t think there is nothing to do right now.

Good luck finding the solution…

Thanks for this. I’ll guess I continue investigating the UE4 source code itself. I’ll post any findings.

I can produce this issue in 4.9.2, but not in 4.11. It’s very likely has been fixed in later version of 4.9.2

Did you find a fix for this issue ? I’m having the same problem and it’s really frustrating.

Hi Gwenn. The problem is on the engine side, if you can call it a problem. The latency is proportional to your game FPS. Previously I was using 3 captures for each mirror of a car which killed the FPS. Now, I just use 1 capture and distributed the image to the mirrors by UV mapping.

Whether the capture output has latency or not is a completely different issue - this discussion is about the scene capture location being flat out wrong. Out of sync with the scene, if you will. Latency is fine, inconsistency is not.

FYI, this bug appears to be fixed in 4.11 Preview 2. Some other bugs have appeared (no particle systems are captured) but they’re supposed to be fixed with 4.11 release, so I suggest people who have problems with this move on to this version.

I can confirm that 4.11 Preview 2 doesn’t have the issue, while 4.10.2 does.