Game crash report

Hi,

I would like to be notify and get logs/dumps from our users when the game crash.
I can’t find any resource about this topic.

So, I’m looking about the crash report client inside the engine.
It seems to be done for epic only (https://answers.unrealengine.com/questions/680/can-we-get-notifications-of-crash-reports-for-our.html).

Do we have a build-in game crash report? If no, is it a good idea to customize the crash report client and to implement some backend services to handle files coming from the client?

Cheers,

Yohann

I’d also like to know the status of this feature. It would be very useful for bug fixing if there was a way to redirect the crash report client uploads for shipping builds to our own web servers.

You can reconfigure crash report client and the source for our backend services are now available. They’re supplied as is with no proper documentation yet but that will follow. For now, here is a summary.

To implement your own crash report server you will need to:

  1. Configure your project’s Crash Report Client (CRC) to send crashes to your endpoint. The CRC has ini files that allow this to be set.

  2. Set up your own server running a Crash Report Receiver (CRR). This handles the http requests from the client, uploads crashes and saves them to a landing zone. The code can be found in Engine\Source\Programs\CrashReporter\CrashReportReceiver.

  3. Set up a Crash Report Process (CRP) that has access to the landing zone folder from your CRR. The CRP reads crash reports, checks their validity, attempts to symbolicate using crash dumps and stored symbols to produce callstacks. It also passes processed crashes to a website/database, saving out the crash files to a location that the website can link to them. The code can be found in Engine\Source\Programs\CrashReporter\CrashReportProcess.

  4. Set up a Crash Report Website (CRW) that has access to the crash files saved out from the CRP. The crash report website also needs a connection to your crash database (the schema for this in the website source). The code can be found in Engine\Source\Programs\CrashReporter\CrashReportWebSite.

NOTE: We are phasing out the CRR in favour of our Data Router that receives crashes and makes them available to the CRP using Google S3 and an SQS queue. The code for this isn’t available.

Thanks for the detailed reply Chris. If CRC//CRR is being phased out, is there a high likelyhood of things breaking if we did decide to implement our own solution using the approach you mentioned? Cheers! -

The CRC currently uploads to both systems, the old CRR and the new Data Router. At some point ours will stop uploading to the old system but I’ll try to do that with config settings so that the legacy path remains intact. The server-side processor (CRP) is the same. It can take queued crashes from both paths. I’ll look into making Data Router code or at least its spec available for those wishing it run their own.

There are plans to provide an equivalent of our Data Router as SaaS.

Hi Chris,

Let’s say I want to run my own server, are there detailed instructions somewhere to do this? What/how to build/install, etc? And what’s the situation in 4.11/12/13?
For example, I’m having trouble building CrashReportWebSite.sln (in 4.11, 12, or 13… missing files…)
CrashReportServer.sln seems easier to build - presumably CrashReportReceiver.exe and CrashReportProcess.exe should be run as services, but any advice on their config/setup (e.g. CrashReportProcess.config)?

Thanks!

Correct, CrashReportReceiver and CrashReportProcess should run as services. Either on the same machine or on two separate machines with the landing zone shared over your network. I’d use the newest versions if possible, there have been a lot of improvements. You can also run the website on the same machine or on another one, as long as the CRP can contact the website to add crashes and can save the processed crash files to a location that the website can access. I’ll ask someone more familiar with the website code and database setup to help with that stuff.

Hi , which files are reporting missing in the WebSite solution?

When building the solution in 4.13:

Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\IDataRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\FunctionRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\UnitOfWork.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\CrashRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\BuggRespository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\UserRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\UserGroupRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\CallStackRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\DataRepository.cs’ could not be found.
Preview_4.13\Engine\Source\Programs\CrashReporter\CrashReportWebSite\DataModels\Repositories\ErrorMessageRepository.cs’ could not be found.

Maybe I’ve missed a generation step or stg?

Thanks - I got CRR & CRP to work (built from 4.13), even though I’ve improvised a lot about the CRP’s config (e.g. for now I’ve ignored everything p4 / MDD), and of course since I don’t have the CRW yet, CRP classifies everything in Invalid Reports.

NB: found your .zip in another thread Steve, will have a look, thanks :wink:

Hi Chris,

About the Crash Reporter Client, do you have any tips about debugging the symbols lookup?
Even when hardcoding the right symbol server location, I can’t make the CRC come up with a call_stack (“You do not have any debugging symbols required to display the callstack for this crash”) (GetLineByOffsetWide returning E_FAIL…), while the Visual Studio debugger can…
What should I do on top of activating SYMOPT_DEBUG to get details / try to find out what happens?

Thanks,

I’ve not had to debug issues with this code before, sorry. Perhaps you could try registering callbacks using IDebugClient5::SetOutputCallbacksWide and SetEventCallbacksWide?

Yeah I guess - reading the comment

// Disable by default as it can be very spammy/slow. Turn it on if you are debugging symbol look-up!

gave me the hope that messages already did end up somewhere without me writing additional callbacks, but you’re probably right
Thanks!

But maybe we should go back to more basic questions:

  • where do you think the symbol resolution should happen? CRC? Or later by CRP? Or is that capability in CRP just a worse-case scenario?
  • how is one usually supposed to configure the symbols server i.e. what .ini etc files to modify?

We perform symbol resolution in two main scenarios.

  1. Builds with local symbols (dev builds with pdbs in the binary folder) will be resolved in the CRC. No complex symbol lookup needed, they’re right there and they always match. The callstack is added to the uploaded crash data.
  2. Uploads without callstacks are resolved on the server by the CRP. It runs MinidumpDiagnostics (MDD) that matches the build details (branch, engine version, build version, changelist, etc.) to patterns stored in our internal version of basengine.ini. This tells MDD where to find the symbols on our network. We store symbols for engine releases and for internal projects.

Hi Chris,

From googling & experimentation, I think it comes down to dll issues.
To find symbols on a server, with the srv* syntax, it seems that dbghelp.dll & symsrv.dll need to sit alongside (?) CrashReporterClient.exe.

(as kind of confirmed by the code below “// Add in syncing of the Microsoft symbol servers if requested” in C:\DNE\PG2\UE4\Engine\Source\Developer\CrashDebugHelper\Private\Windows\WindowsPlatformStackWalkExt.cpp)

I got a good call stack after copying dbghelp.dll & symsrv.dll from the Windows Debugger Kit alongside the exe (but… I get crashes on SetSymbolPathWide unless I keep it short / to the srv* I want; so it might not be exactly the right dll(s)…)

Sounds like all that is known problems of “trying to make your own debugger”, but they look tricky to get right…

Thanks,

Oh and another question Chris:

I like those scenarios 1&2, but I’m having trouble making 2 work for us, maybe because of our sym store…

When you say that you store symbols (for engine releases and for internal projects), you store pdbs, but do you store binaries there too?

(I fear that’s what’s missing for us… I have trouble getting a call stack when opening a minidump even with Visual Studio… In theory, to analyze a minidump, are the pdbs enough or are the binaries needed too?)

Thanks!

Correct, binaries and pdbs are needed for this method.