Ok, so I’ve been having this crash bug that’s been driving me insane for months now (there is in fact very little sanity left I’m afraid). I spent hours and days trying to find the cause. Then it seemed like it solved itself. Then it came back. Then it solved itself…
It’s back.
I’ve been reading all kinds of engine code, but for the first time now I think I may have an idea of what’s going on.
So here is what I’m doing:
I made this character morphing system that can modify a character’s shape at runtime. It can also add and remove pieces of clothing. Every character’s mesh is a copy of a base mesh that gets procedurally changed: vertex-data and bone-transforms get modified, sections, chunks and materials get added or removed and skeletal mesh resources get re-initialized.
Most of the time everything works fine. But sometimes I get a crash when changing the clothing, which is probably connected to the addition/removal of the clothing’s material(s) from/to the character mesh. It seems to occur completely randomly. But now I realised that certain code that should be executed, because it’s part of the morphing process, never gets reached before the crash. Instead I get an engine error from rendering before the whole process has finished.
I don’t know much about multi threading, but this is the only way I can make any sense of it. Can someone confirm that this may actually be the problem? Is there something I can do to make sure that the mesh I am modifying is not being rendered in this very milisecond?
The last couple of hours I’ve been trying to solve this again. Only this time I focused on the rendering thread. I guess it comes down to a simple question now:
How do I change the number of sections, chunks and materials on a skeletal mesh that is actively being rendered without (sometimes) crashing the rendering thread?
I know that my mesh manipulation is solid. If the modified mesh is not being rendered at the time, there is absolutely no problem and the result looks as it should. It’s related to the MeshComponent’s MeshObject, its “dynamic data”, the fact that the number of sections suddenly changed and the number of VertexFactories isn’t right.
So, how can I fix this?
For every MeshComponent that uses the mesh that is going to change I call SetSkeletalMesh(NULL) before making any changes. Then I mark the component’s render data as dirty, hoping that this will (immediately?) release all GPU resources and avoid the render thread to crash because of outdated information. When all modifications to the mesh are finished, I build the vertex buffers (…), initialize its resources, call SetSkeletalMesh(mesh) on all MeshComponents that use the mesh again, and finally mark their render data as dirty again.
But I still get crashes inside the render thread…
I also played around with GRenderingThread->Suspend and some other functions that control the rendering, but all I managed to achieve this way is to freeze the program.
I am getting different crash reports. I guess it depends on where the render thread is at when I modify the mesh. But this call stack is a typical one:
Assertion failed: LODSection.SectionElements.Num() == LODModel.Sections.Num() [File:D:\BuildFarm\buildmachine_++UE4+Release-4.11\Engine\Source\Runtime\Engine\Private\SkeletalMesh.cpp] [Line: 4525]
UE4Editor_Core!FDebug::AssertFailed() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\core\private\misc\outputdevice.cpp:430]
UE4Editor_Engine!FSkeletalMeshSceneProxy::GetMeshElementsConditionallySelectable() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\engine\private\skeletalmesh.cpp:4527]
UE4Editor_Engine!FSkeletalMeshSceneProxy::GetDynamicMeshElements() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\engine\private\skeletalmesh.cpp:4497]
UE4Editor_Renderer!FProjectedShadowInfo::GatherDynamicMeshElementsArray() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\shadowsetup.cpp:1039]
UE4Editor_Renderer!FProjectedShadowInfo::GatherDynamicMeshElements() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\shadowsetup.cpp:998]
UE4Editor_Renderer!FSceneRenderer::GatherShadowDynamicMeshElements() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\shadowsetup.cpp:2007]
UE4Editor_Renderer!FDeferredShadingSceneRenderer::InitDynamicShadows() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\shadowsetup.cpp:2589]
UE4Editor_Renderer!FDeferredShadingSceneRenderer::InitViews() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\scenevisibility.cpp:2753]
UE4Editor_Renderer!FDeferredShadingSceneRenderer::Render() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\deferredshadingrenderer.cpp:846]
UE4Editor_Renderer!RenderViewFamily_RenderThread() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\renderer\private\scenerendering.cpp:1744]
UE4Editor_Renderer!TGraphTask<`FRendererModule::BeginRenderingViewFamily'::`20'::EURCMacro_FDrawSceneCommand>::ExecuteTask() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\core\public\async\taskgraphinterfaces.h:886]
UE4Editor_Core!FNamedTaskThread::ProcessTasksNamedThread() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\core\private\async\taskgraph.cpp:779]
UE4Editor_Core!FNamedTaskThread::ProcessTasksUntilQuit() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\core\private\async\taskgraph.cpp:526]
UE4Editor_RenderCore!RenderingThreadMain() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\rendercore\private\renderingthread.cpp:310]
UE4Editor_RenderCore!FRenderingThread::Run() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\rendercore\private\renderingthread.cpp:417]
UE4Editor_Core!FRunnableThreadWin::Run() [d:\buildfarm\buildmachine_++ue4+release-4.11\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:74]
I had to post as answer because of the character limit for comments…
I use FlushRenderingCommands() now before I modify my character mesh, because that’s what it says I should do here.
It’s possible that the crash occurs less often, but it still happens…
Are there any other threads I should know about? Because at the moment the updating of the mesh is triggered by some PostEditChangeEvent, meaning I edit the morphing parameters in the details panel and the resulting event leads to the mesh being altered. So now I’m wondering if maybe this is not the (game-) thread I’m supposed to call FlushRenderingCommands() from, but some kind of editor thread. In my limited understanding of the matter this could mean that both the game thread and the render thread (following the game thread) would keep running while I edit the mesh data. Any second opinions?..
I am not 100% sure the crash won’t come back, but I think (just maybe) FSuspendRenderingThread may have done the trick.