Packaged project crashes after 4.13 migration

Development Platform & Builds
, , , , ,
1

After migrating our project from 4.12.5 to 4.13 our packaged game crashes shortly (~1 second) after the player pawn is spawned and possessed. Initially I thought the migration was going very smoothly as the game works completely fine in-editor, but making a packaged version (development or debug) and running the exe causes the game to hard crash to a “Unreal Engine has stopped working” Windows error menu. Unfortunately the log files don’t capture the point of failure (never flushes to disk). The only clues I have right now are:

  1. that I “once” got a normal UE stack trace and crash report window that suggested it might be an issue with a UMG widget on the player pawn blueprint. However removing all UMGs and widgets from the pawn did not resolve the issue.

  2. When choosing ‘debug’ from the “Unreal Engine has stopped working” Windows error menu it sets the execution breakpoint in a Substance plugin c++ file at a point that handles closing down the plugin on engine shutdown (Substance::Details::Engine::releaseEngine). We do make heavy use of Substance so debugging further with nothing to go on will be very difficult…

2

For extra info: I have deleted and rebuilt the build, intermediate and save directories. I have selected the “full rebuild” option in the packaging settings.

3

Packaged game crashes - Programming & Scripting - Unreal Engine Forums did not resolve the issue.

4

So I was able to get a Unreal Crash reporter popup (seemingly randomly) and got this stack:

Access violation - code c0000005 (first/second chance not available)

Snow!FSceneRenderer::GatherShadowsForPrimitiveInner()
Snow!FSceneRenderer::GatherShadowPrimitives()
Snow!FSceneRenderer::InitDynamicShadows()
Snow!FDeferredShadingSceneRenderer::InitViewsPossiblyAfterPrepass()
Snow!FDeferredShadingSceneRenderer::InitViews()
Snow!FDeferredShadingSceneRenderer::Render()
Snow!FSceneInterface::UpdateReflectionCaptureTransform()
Snow!TGraphTask<`FScene::UpdateSceneCaptureContents'::`12'::EURCMacro_CaptureCommand>::ExecuteTask()
Snow!FNamedTaskThread::ProcessTasksNamedThread()
Snow!FNamedTaskThread::ProcessTasksUntilQuit()
Snow!RenderingThreadMain()
Snow!FRenderingThread::Run()
Snow!FRunnableThreadWin::Run()
5

One other potential issue (because it only executes on built versions of the game): The project utilizes the Steam OnlineSubModule. I removed it from my build file and added it as a plugin. However I think that is working correctly as when the built executable starts the steam overlay pops up in the corner and the logs have lines like:

[2016.09.22-23.11.10:791][  0]LogOnline:Warning: STEAM: Failed to obtain steam user stats, user: {my user name} has no stats entries

Which is expected as I haven’t set up user stats yet.

EDIT: removing steam and replacing it with the Null online submodule did not resolve the issue.

6

So narrowed it down to a Substance plugin issue. I noticed that the built package would miraculously start working after trying to start it after 5-6 crashes (the last crash being accompanied by the Unreal Crash Reporter). With that in mind I figured it was something ‘being built’ by the game on startup and was some sort of failing race condition. Looking at the recently modified files brought me to the \Saved\Substance directory. Between crashes more and more .cache files would be added until it finally “finished” and the build would work. By turning the Substance setting (Project Settings > Plugins > Substance) Cooking > Default generation mode for Substances from SGM Platform Default to SGM Baked resolved the issue and we were able to successfully build and run the build with no issues. We will be opening a ticket with Substance to see if this can be resolved as this will remove the ability to do runtime modifications to Substances (which we aren’t using at this time).