UE 4.19.1 memory leak while using blueprint debugger

Development Programming & Scripting Blueprint
, , , , ,
1

When I try to use the blueprint debugger with an Unreal 4.19.1 source build, the Unreal Editor process starts consuming excessive amounts of memory (and cpu) with it rapidly hitting a 64G footprint and crashing. This is only a few seconds after hitting the breakpoint in the blueprint (I paused the editor in VS2017 to catch this)

239191-memory-growth.jpg

I started doing some memory profiling, but figured it would be faster to ask here since I’ve never tried to break apart any of the c++ tick loops in Unreal. One thing I noticed with the first thing ‘leaked’ was a difference in the way it is normally freed vs. what happens while in the blueprint breakpoint. The first item that popped up was a long list of ever growing GpuProfilerEvents. Looking at the Call stack for normal clearing of GpuProfilerEvents:

  UE4Editor-Engine.dll!FRealtimeGPUProfilerFrame::Clear(FRHICommandListImmediate * RHICommandListPtr) Line 378    C++     Symbols loaded.

    UE4Editor-Engine.dll!FRealtimeGPUProfiler::EndFrame(FRHICommandListImmediate & RHICmdList) Line 574     C++     Symbols loaded.
    UE4Editor.exe!`FEngineLoop::Tick'::`50'::EURCMacro_EndFrame::DoTask(ENamedThreads::Type CurrentThread, const TRefCountPtr<FGraphEvent> & MyCompletionGraphEvent) Line 3528      C++     Symbols loaded.
    UE4Editor.exe!TGraphTask<`FEngineLoop::Tick'::`50'::EURCMacro_EndFrame>::ExecuteTask(TArray<FBaseGraphTask *,FDefaultAllocator> & NewTasks, ENamedThreads::Type CurrentThread) Line 829 C++     Symbols loaded.
    [Inline Frame] UE4Editor-Core.dll!FBaseGraphTask::Execute(TArray<FBaseGraphTask *,FDefaultAllocator> & CurrentThread, ENamedThreads::Type) Line 498     C++     Symbols loaded.
    UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksNamedThread(int QueueIndex, bool bAllowStall) Line 665 C++     Symbols loaded.
    UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksUntilQuit(int QueueIndex) Line 574     C++     Symbols loaded.
    UE4Editor-RenderCore.dll!RenderingThreadMain(FEvent * TaskGraphBoundSyncEvent) Line 331 C++     Symbols loaded.
    UE4Editor-RenderCore.dll!FRenderingThread::Run() Line 482       C++     Symbols loaded.
    UE4Editor-Core.dll!FRunnableThreadWin::Run() Line 76    C++     Symbols loaded.
    UE4Editor-Core.dll!FRunnableThreadWin::GuardedRun() Line 25     C++     Symbols loaded.

AND then comparing it to the callstack where it was being allocated (FSlateRHIRenderer::DrawWindow_RenderThread) but never freed, suggests that the loop that is calling this DrawWindow_RenderThread is not properly doing end-of-tick cleanup as seen in the normal flow above.

  UE4Editor-Engine.dll!FRealtimeGPUProfilerFrame::PopEvent(FRHICommandListImmediate & RHICmdList) Line 370        C++     Symbols loaded.

    UE4Editor-Engine.dll!FRealtimeGPUProfiler::PopEvent(FRHICommandListImmediate & RHICmdList) Line 610     C++     Symbols loaded.
    [Inline Frame] UE4Editor-SlateRHIRenderer.dll!FScopedGPUStatEvent::{dtor}() Line 253    C++     Symbols loaded.
    UE4Editor-SlateRHIRenderer.dll!FSlateRHIRenderer::DrawWindow_RenderThread(FRHICommandListImmediate & RHICmdList, FSlateRHIRenderer::FViewportInfo & ViewportInfo, FSlateWindowElementList & WindowElementList, bool bLockToVsync, bool bClear) Line 891    C++     Symbols loaded.
    [Inline Frame] UE4Editor-SlateRHIRenderer.dll!FSlateRHIRenderer::DrawWindows_Private::__l29::<lambda_43e86ee7c51f39979e9a39d40b280024>::operator()(FRHICommandListImmediate &) Line 1050        C++     Symbols loaded.
    [Inline Frame] UE4Editor-SlateRHIRenderer.dll!TEnqueueUniqueRenderCommandType<`FSlateRHIRenderer::DrawWindows_Private'::`29'::SlateDrawWindowsCommandName,<lambda_43e86ee7c51f39979e9a39d40b280024> >::DoTask(ENamedThreads::Type) Line 190        C++     Symbols loaded.
    UE4Editor-SlateRHIRenderer.dll!TGraphTask<TEnqueueUniqueRenderCommandType<`FSlateRHIRenderer::DrawWindows_Private'::`29'::SlateDrawWindowsCommandName,<lambda_43e86ee7c51f39979e9a39d40b280024> > >::ExecuteTask(TArray<FBaseGraphTask *,FDefaultAllocator> & NewTasks, ENamedThreads::Type CurrentThread) Line 829        C++     Symbols loaded.
    [Inline Frame] UE4Editor-Core.dll!FBaseGraphTask::Execute(TArray<FBaseGraphTask *,FDefaultAllocator> & CurrentThread, ENamedThreads::Type) Line 498     C++     Symbols loaded.
    UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksNamedThread(int QueueIndex, bool bAllowStall) Line 665 C++     Symbols loaded.
    UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksUntilQuit(int QueueIndex) Line 574     C++     Symbols loaded.
    UE4Editor-RenderCore.dll!RenderingThreadMain(FEvent * TaskGraphBoundSyncEvent) Line 331 C++     Symbols loaded.
    UE4Editor-RenderCore.dll!FRenderingThread::Run() Line 482       C++     Symbols loaded.
    UE4Editor-Core.dll!FRunnableThreadWin::Run() Line 76    C++     Symbols loaded.
    UE4Editor-Core.dll!FRunnableThreadWin::GuardedRun() Line 25     C++     Symbols loaded.

For a ‘quick’(not) test, I tried removing the new line added to that method that was collecting GPU stats:
//SCOPED_GPU_STAT(RHICmdList, SlateUI);
– doing this eliminated the leaking gpu profile data, but that wasn’t the only leak. It feels as though the loop around FSlateRHIRenderer::DrawWindow_RenderThread is missing some critical per-call cleanup.

Posting here hoping that someone who knows this code better has an aha moment.

Thanks.

2

Hello,

We’ve recently made a switch to a new bug reporting method using a more structured form. Please visit the link below for more details and report the issue using the new Bug Submission Form. Feel free to continue to use this thread for community discussion around the issue.

https://epicsupport.force.com/unrealengine/s/

Thanks

3

Update on this issue. It has been filed as an unreal issue Unreal Engine Issues and Bug Tracker (UE-58566) . I’ve also hacked in a fix for myself here: https://github.com/LanceNeumann/UnrealEngine/commit/86d63a661e585c1daacb454ca29c5f836e013dca

With that fix/hack, in the simplistic example given for the UE-58566 ticket I can sit in the blueprint debugger and not leak memory. Unfortunately, in my real project there are still cases where something leaks while sitting in the blueprint debugger - fewer leaks than without this change and a slower leak when it does leak, but still a leak. When the leak does happen, if I finish blueprint-debugging before running out of memory then the Unreal Editor process does recover at least. This suggests that the remaining leak is of the same style as this already fixed one: gathering pooled buffers without releasing the pool while debugging, but at least leaving them in a state so that when normal processing returns they get released.

I did a test in UE4.18 and realized that it also has a similar leak in the same situation so I suppose this showed up as a ‘big issue’ for me because I randomly tested a blueprint breakpoint at just that bad spot.