Fatal Error crashes on servertravel

Hey MADHOUSE,

I gave this a shot in my test project, but I didn’t see any crashes occur in 4.16.1. Could you please provide the logs so I can take a look at those, as well as give this a shot in a clean project and let me know if you’re seeing it there too.

Thanks

By any chance, are any of your Levels performing a Multicast call from inside a Level Blueprint? If so, it’s possible that’s it’s related to a confirmed 4.16 Engine bug I recently posted here:

Im still getting fatal error crashes. Debugging in visual studio still breaks at this function in IPlatformFilePak.cpp altho I haven’t seen more “Level was reloaded before it even closed the linker from a previous load.” crashes yet, so I cant tell if they are related. I have had quite a few of those in the past aswell.

201733-fatalerrorcrash.jpg

Crash log: [2017.06.28-23.29.06:847][238]LogPackageName: SearchForPackageOnDisk took 0.93 - Pastebin.com

This issue is currently critical and shipping breaking.

Any news on this?

Greetings.
I have also had problems with UE-46854, altho this is a different issue unfortunately. I’ll explore the issue some more this week and see If I can find any more information…

I’ll post back if I have any results. The issue is there obviously but it seems very very low-level and its difficult to debug on my end so it might take some time.

Best Regards

There is definitely something wierd going on there when the engine attempts to read or serialize content from the pak file. At the assertion break the WaitEvent / Trigger event reference/value is null, as well as almost all variables in that section.

         if (WaitEvent)
         {
             WaitEvent->Trigger();
         }

Resulting in crash:
EXCEPTION_ACCESS_VIOLATION reading address 0xffffffff

Please post back if there is any progress on this issue. This problem is still shipping breaking.

Sometimes this is the error crash details, and a bit different from the more usual crash log and messages:

Error: Runnable thread TaskGraphThreadNP 1 crashed.
=== Critical error: ===

Fatal error: [File:H:\UnrealEngine-release\Engine\Source\Runtime\Core\Private\Windows\WindowsPlatformMisc.cpp] [Line: 482] 
Pure virtual function being called


KERNELBASE.dll!0x00000000FD149E5D
Game.exe!FOutputDeviceWindowsError::Serialize() [h:\unrealengine-release\engine\source\runtime\core\private\windows\windowsplatformoutputdevices.cpp:120]
Game.exe!FOutputDevice::Logf__VA() [h:\unrealengine-release\engine\source\runtime\core\private\misc\outputdevice.cpp:70]
Game.exe!FDebug::AssertFailed() [h:\unrealengine-release\engine\source\runtime\core\private\misc\assertionmacros.cpp:349]
VCRUNTIME140.dll!0x00000000EE5C4A74
Game.exe!FPakProcessedReadRequest::RequestIsComplete() [h:\unrealengine-release\engine\source\runtime\pakfile\private\iplatformfilepak.cpp:2740]
Game.exe!FPakAsyncReadFileHandle:oProcessing() [h:\unrealengine-release\engine\source\runtime\pakfile\private\iplatformfilepak.cpp:3079]
Game.exe!TGraphTask<FAsyncIOCPUWorkTask>::ExecuteTask() [h:\unrealengine-release\engine\source\runtime\core\public\async\taskgraphinterfaces.h:886]
Game.exe!FTaskThreadAnyThread::ProcessTasks() [h:\unrealengine-release\engine\source\runtime\core\private\async\taskgraph.cpp:1277]
Game.exe!FTaskThreadAnyThread::ProcessTasksUntilQuit() [h:\unrealengine-release\engine\source\runtime\core\private\async\taskgraph.cpp:1171]
Game.exe!FTaskThreadBase::Run() [h:\unrealengine-release\engine\source\runtime\core\private\async\taskgraph.cpp:643]
Game.exe!FRunnableThreadWin::Run() [h:\unrealengine-release\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:76]
Game.exe!FRunnableThreadWin::GuardedRun() [h:\unrealengine-release\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:33]
kernel32.dll!0x0000000076E9652D
ntdll.dll!0x00000000770CC541
ntdll.dll!0x00000000770CC541

This bug seems to be gone in 4.17 preview 3.

In 4.16.1 there are a constant amount of unknown fatal error crashes that occur on servertravel.

[2017.06.20-17.57.33:751][ 91]LogWindows:Warning: CreateProc failed (2) ../../../Engine/Binaries/Win64/CrashReportClient.exe "H:/GAME/WindowsNoEditor/WindowsNoEditor/Game/Saved/Crashes/UE4CC-Windows-4C804C9F41BF70FD2E8273BEE08A4110_0000" -AppName=UE4-Game -CrashGUID=UE4CC-Windows-4C804C9F41BF70FD2E8273BEE08A4110_0000 -DebugSymbols=..\..\..\Engine\Intermediate\Symbols
[2017.06.20-17.57.33:751][ 91]LogWindows: Could not start CrashReportClient.exe
[2017.06.20-17.57.33:751][ 91]LogMemory: Platform Memory Stats for WindowsNoEditor
[2017.06.20-17.57.33:751][ 91]LogMemory: Process Physical Memory: 1117.67 MB used, 1118.00 MB peak
[2017.06.20-17.57.33:751][ 91]LogMemory: Process Virtual Memory: 2442.87 MB used, 3331.46 MB peak
[2017.06.20-17.57.33:751][ 91]LogMemory: Physical Memory: 8041.95 MB used,  16525.16 MB free, 24567.11 MB total
[2017.06.20-17.57.33:751][ 91]LogMemory: Virtual Memory: 3092.70 MB used,  16525.16 MB free, 8388608.00 MB total
[2017.06.20-17.57.33:751][ 91]LogThreadingWindows:Error: Runnable thread PoolThread 4 crashed.
[2017.06.20-17.57.33:751][ 91]LogWindows:Error: === Critical error: ===
[2017.06.20-17.57.33:751][ 91]LogWindows:Error: 
[2017.06.20-17.57.33:751][ 91]LogWindows:Error: Fatal error!
[2017.06.20-17.57.33:751][ 91]LogWindows:Error: 
[2017.06.20-17.57.33:751][ 91]LogWindows:Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x00000030
[2017.06.20-17.57.33:751][ 91]LogWindows:Error: 

These crashes never occured in 4.15.2, and they are very inconsistent. Sometimes after starting the game on 2 separate machines, you can servertravel between 4-5 levels, and other times it will fatal crash on the very fist servertravel…

Exception thrown at 0x00000001410BADAB in Game-Win64-Shipping.exe: 0xC0000005: Access violation reading location 0x0000000100000030.

After debugging the crash in visual studio the crash seems to happen at this breakpoint:

void RequestIsComplete()
	{
		check(bRequestOutstanding);
		if (!bCanceled && Priority > AIOP_Precache)
		{
			GatherResults();
		}
		SetComplete();
		FScopeLock Lock(&FPakReadRequestEvent);
		bRequestOutstanding = false;
		if (WaitEvent)
		{
			WaitEvent->Trigger();     (<----BREAK POINT)
		}
	}

This function is in IPlatformFilePak.cpp

Since the crash seems related to *.pak file loading/reading at servertravel, I tested to build and package the project with pak files disabled. This stops all crashes at servertravel, altho the resulting project build is twice the size and it is obviously not an acceptable solution that can be shipped…

I will investigate as much as I can, I suspect there might be a need to put larger amount of data into the pak to cause these issues. It could be usefull to have an engineer look at this section of code since this issue did not happen before 4.16. Someone with more insight into what has changed and why this would happen could perhaps shed more light on it.

On my end all I can do is process-of-elimination which might take a while to analyse.

Greetings. Hmm interesting, some of the levels are indeed using multicast RPC’s to reset states on level actors during match restarts, I’ll look into it this week and return with results, it could very well be the related to the same issue.

Thanks for the information!

and sometimes the crash provides less information then in the log above, but still relates to IPlatformFilePak.cpp :

These crashes still occur in 4.16.2…

Hey MADHOUSE,

I’ve reproduced a crash on Server Travel that was reported by nrossol, and entered that ticket here: Unreal Engine Issues and Bug Tracker (UE-46854)

If you’re seeing something different, I haven’t been able to reproduce that on my end, so unfortunately I can’t get a ticket in for that just yet. If you’re able to narrow it down and provide me with a repro case or a project I can use, I’d be glad to take another look at it.

Thanks

We got this one too or somthing similar maybe

https://udn.unrealengine.com/questions/382428/package-crash-assertion-failed-iplatformfilepakcpp.html

Thanks for the heads up, I’ve assigned your ticket out for investigation.