Packaged Server Crash Results in No Callstack

HailstoneRyan · April 30, 2016, 4:18am

Building a dedicated server (both our project and ShooterGame) results in no callstack available in the log file when a crash occurs.

We’re building using the following command line, perhaps there is something wrong in our build command.

RunUAT BuildCookRun -project="D:\Documents\Unreal Projects\ShooterGame\ShooterGame.uproject" -build -cook -server -noclient -NoP4 -pak -stage -serverconfig=Debug -serverplatform=Win64 -stagingdirectory="D:\Documents\Unreal Projects\ShooterGame\Build"

What are we doing wrong? It’s incredibly difficult to debug crashes without a callstack.

anonymous_user_a08c0bf2 · May 5, 2016, 7:01pm

Hey Overlawled,

In order to better assist you, I’ll need to know a bit more information.

Is your project blueprint only, C++ only, or a mix of both?
Are you able to replicate this issue on any other project besides Shooter Game and your project?
Has it ever worked properly for you?

Please go through the following documentation to ensure you haven’t missed any steps:

[Dedicated Server Guide (Windows & Linux)][1]
[Command-Line Arguments][2]

A new, community-hosted Unreal Engine Wiki - Announcements - Epic Developer Community Forums
[2]: Command-Line Arguments in Unreal Engine | Unreal Engine 5.1 Documentation

anonymous_user_a08c0bf2 · May 10, 2016, 4:06pm

Hey Overlawled,

We have not heard back from you in a few days, so we are marking this post as Resolved for tracking purposes. If you are still experiencing the issue you reported, please respond to this message with additional information and we will offer further assistance.

Thank you!

HailstoneRyan · May 15, 2016, 4:39am

Sorry , emails from AnswerHub get buried in my inbox.

We haven’t tried it on other projects, no, but felt ShooterGame would be a good place to start to see if it was something project specific.

HailstoneRyan · May 17, 2016, 5:54pm

Sorry , emails from AnswerHub get buried in my inbox.

We haven’t tried it on other projects, no, but felt ShooterGame would be a good place to start to see if it was something project specific.

anonymous_user_a08c0bf2 · May 18, 2016, 8:16pm

Could you please try and replicate this issue in a new project? Upload any logs that you’re able to obtain during testing. By the way, UE-30806 is a bug we’re currently looking in to. It regards a plugin missing error on iOS while the web browser widget is being used, so the fix isn’t released yet.

Thanks!

HailstoneRyan · May 19, 2016, 7:41pm

Ran into the same issue with a blank project and a fresh clone of the UE4 source code.

This is all that exists in the log when it crashes.

[2016.05.19-19.38.24:376][  0]LogWindows:Error: Windows GetLastError: The operation completed successfully. (0)
[2016.05.19-19.38.24:500][  0]LogWindows:Error: === Critical error: ===
Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:D:\Development\UnrealEngine_Source\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 785] 
Array index out of bounds: 0 from an array of size 0




[2016.05.19-19.38.24:504][  0]LogExit: Executing StaticShutdownAfterError
[2016.05.19-19.38.24:505][  0]LogPhysics:Warning: PHYSX: ..\..\PhysX\src\NpScene.cpp (2946) 8 : PxScene::unlockWrite() called without matching call to PxScene::lockWrite(), behaviour will be undefined.
[2016.05.19-19.38.24:506][  0]LogPhysics:Warning: PHYSX: ..\..\PhysX\src\NpScene.cpp (2946) 8 : PxScene::unlockWrite() called without matching call to PxScene::lockWrite(), behaviour will be undefined.
[2016.05.19-19.38.24:507][  0]LogWindows: FPlatformMisc::RequestExit(1)
[2016.05.19-19.38.24:507][  0]Log file closed, 05/19/16 15:38:24

Project for this test is here and includes bat files that show how we build the server and run the server.
https://github.com/joshkay/ServerTest

anonymous_user_a08c0bf2 · May 27, 2016, 8:43pm

The physx issue actually has been fixed in 4.12. Could you please test this in 4.12 and let me know what your results are? Please upload your full error output logs as a .txt file with your next reply.

Make sure you’re adding -log to your server too, so the logs will show up.

Thanks!

HailstoneRyan · May 28, 2016, 1:56pm

Log attached. The PHYSX messages are gone but there’s still no callstack leading to where in the game code this crash is coming from.

[link text][1]

92379-servertest.log (14.6 KB)

anonymous_user_a08c0bf2 · May 31, 2016, 7:30pm

I packaged Shooter Game in 4.11.2 and did not run into any packaging errors. Did you pull down the correct version of Shooter Game for the version of the engine you’re working in? Did you modify anything within the project?

HailstoneRyan · June 1, 2016, 3:43am

Packaging isn’t the problem. Once we’ve packaged a dedicated server and run it, if the server crashes we get no callstack from the server. This wouldn’t be a problem if we were always running the server through VS but the server is hosted in the cloud and we don’t have direct access to the hardware itself.

anonymous_user_a08c0bf2 · June 6, 2016, 8:40pm

We’re investigating this issue, thanks for being so patient.

anonymous_user_a08c0bf2 · June 9, 2016, 6:43pm

I have ran tests with and without your additional command lines and have not ran into any crashes. How long did it take for a crash to show up? Is this still occurring on 4.12.2?

HailstoneRyan · June 10, 2016, 2:37am

Hi ,

The crash isn’t the issue. In the sample we linked we’re forcing a crash. The issue is that when a crash does occur the log contains no callstack.

anonymous_user_a08c0bf2 · June 13, 2016, 5:11pm

Overlawled,

I requested some additional server/crash log information for you to expand on the call stack. I have yet to hear back.

However, I did find a few things I thought that may help you:

Have you looked over the [logs, printing messages during runtime][1] documentation?

You may benefit from reviewing the following too: [Set Crash Handler][2]

Thanks!

A new, community-hosted Unreal Engine Wiki - Announcements - Epic Developer Community Forums
[2]: FGenericPlatformMisc::SetCrashHandler | Unreal Engine Documentation

HailstoneRyan · June 13, 2016, 10:55pm

,
I think we still aren’t on the same page. I’ve done more investigation into this and discovered it is specific to servers packaged for Windows. I’ve attached two sets of logs below, one for Linux and one for Windows. The Linux one has the callstack (or stack trace) as part of the log file, and we can see how we got to the crash which makes it easier for us to track the crash down. The Windows log does not and unfortunately our provider only currently offers Windows servers.

[Linux logs][1]

[Windows logs][2]

94507-servertest.log (18.3 KB)
[2]: 94508-servertest.log (14.6 KB)

anonymous_user_29eab02e1 · June 14, 2016, 12:52pm

I’m having the same issue with the windows builds (using engine built from sources 4.10.4 branch, packaging with editor). Currently asking our testers to copypaste callstack from the crash reporter window, which is really annoying.

anonymous_user_031af941 · June 14, 2016, 4:30pm

If you’re successfully launching the Crash Reporter, you should also have a new crash folder in your WER report queue. You should be able to pull the callstack from CrashContext.runtime-xml. The folder will also contain and copy of the log and the minidump.

HailstoneRyan · June 14, 2016, 5:36pm

Hi ,

I’ve attached the xml, it also only contains that single line rather than a full callstack. Renamed it so it would let me upload it.

[link text][1]

94620-crashcontext.runtime-xml.log (3.19 KB)

anonymous_user_031af941 · June 14, 2016, 7:58pm

Did the CrashReportClient (CRC) run? If your servers run with -unattended the CRC window will not appear and the crash will be sent automatically. Otherwise the dialog should appear. If it ran you should have logs from it too, most likely in AppData\Local\CrashReportClient\Saved\Logs. Looking at your build command, I’d try adding -crashreporter. This should add the CRC into your package as it’s probably not in there.