Multiplayer replication crash

Development Programming & Scripting Multiplayer & Networking
, , , ,
1

When running a multiplayer game, we have a random crash in what appears to be engine’s replication logic. Our setup uses a packaged (non-editor) build on two different machines over wired ethernet. At point of crash in UObjectPropertyBase::Identical line 45, ObjectA is not null but appears to be garbage. ObjectB looks fine. This happens when running exact same build copied to both machines. So far it seems to happen on game running as server. Do you know what would cause this or is there a workaround?

So far we have seen this in shipping and development builds but haven’t reproduced it in a debug build. We are running 4.4.0 built from source. Here is crash stack:

UObjectPropertyBase::Identical() 0x3f39653f + O bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\coreuobject\private\uobject\propertybaseobject.cpp:45] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
FRepLayout::CompareProperties() 0x3f95745e + 311 bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\engine\private\replayout.cpp:327] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
FRepLayout::ReplicateProperties() 0x3f96fca8 + 46 bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\engine\private\replayout.cpp:531] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe] }
FObjectReplicator::ReplicateProperties() 0x3f841adf + O bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\engine\private\datareplication.cpp:983] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
UActorChannel::ReplicateActor() 0x3f840b49 + O bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\engine\private\datachannel.cpp:1809] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
UNetDriver::ServerReplicateActors() 0x3f92190f + 8 bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\engine\private\networkdriver.cpp:2422] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
UNetDriver::TickFlush() 0x3f9243a1 + 15 bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\engine\private\networkdriver.cpp:153] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
TBaseUObjectMethodDelegateInstance_OneParam::ExecuteIfSafe() 0x3f93965c + O bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\core\public\delegates\delegateinstancesimpl.inl:532] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
TBaseMulticastDelegate_OneParam::Broadcast() 0x3f8e7975 + 14 bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\core\public\delegates\delegatesignatureimpl.inl:1852] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
UWorld::Tick() 0x3f8fbddf + O bytes [File= c:\gamedev\Game\ue4\engine\source\runtime\engine\private\leveltick.cpp:1233] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
UGameEngine::Tick() 0x3f87e23c + O bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\engine\private\gameengine.cpp:873] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
FEngineLoop::Tick() 0x3f25b13d + O bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\launch\private\launchengineloop.cpp:2098] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
GuardedMain() 0x3f25725c + O bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\launch\private\launch.cpp:133] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
GuardedMainWrapper() 0x3f2572aa + 5 bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\launch\private\windows\launchwindows.cpp:125] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
WinMain() 0x3f25ba20 + 17 bytes [File=c:\gamedev\Game\ue4\engine\source\runtime\launch\private\windows\launchwindows.cpp:201] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
_tmainCRTStartup() 0x4047cd2d + 21 bytes [File=f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c:618] [in C:\gamedev\Game\GameRelease\WindowsNoEditor\Game\Binaries\Win64\Game-Win64-Shipping.exe]
Address = 0x76d459ed (filename not found) [in C:\Windows\system32\kernel32.dll]
Address = 0x76e7c541 (filename not found) [in C:\Wind0ws\SYSTEM32\ntdll.dll]
Address = 0x76e7c541 (filename not found) [in C:\Wind0ws\SYSTEM32\ntdll.dll]
2

Hi Marc,

Thank you for your report. We have assigned this issue to a member of our support staff, and they will begin investigation as soon as possible. If we are unable to reproduce issue, or we need additional information, our staff member will comment here with additional questions. Otherwise, we will post an ‘Answer’ once we have logged issue in our bug database or we have a solution for it.

In meantime, please be sure to review our suggestions for how to report a bug, and feel free to edit your post if you have additional information to provide:

Please be sure to include crash logs and dump files for your project after it crashes.

Thanks!

3

Hey Marc,

I can internally report crash if you give me your [crash logs and dmp files][1], but if you want me to further investigate, I’m going to need more information about your project and objects that aren’t replicating properly. It sounds like you’re working on getting a repro, which would be extremely helpful as well.

[1]:

4

Here is crash dump and logs. ObjectA is always garbage and ObjectB seems to be consistently name of a particular StaticMeshActor that is ground mesh for a part of level. I wouldn’t expect that mesh to be doing any replication. map has many sublevels, but that mesh is in a sublevel that is streamed in on game start and never streamed out. crash usually happens around 1-2 minutes into game. If ObjectB is a red herring, there are other parts of level that stream in closer time crash happens.

5

As a side question, have you tried running a copy of your project in 4.4.2?

Edit: Also, have you submitted a crash report after this crash happened?

6

Hi Marc,

We have not heard back from you in a few days, so we are marking this post as Resolved for tracking purposes. If you are still experiencing issue you reported, please respond to this message with additional information and we will offer further assistance.

Thank you.

7

Some time around 4.5/6 we added this which should fix this issue if it’s related to GC and shadow state:

static FORCEINLINE bool CompareObject( const FRepLayoutCmd& Cmd, const void* A, const void* B )
{
#if 1
	// Until UObjectPropertyBase::Identical is made safe for GC'd objects, we need to do it manually
	// This saves us from having to add referenced objects during GC
	UObjectPropertyBase * ObjProperty = CastChecked< UObjectPropertyBase>( Cmd.Property );

	UObject* ObjectA = ObjProperty->GetObjectPropertyValue( A );
	UObject* ObjectB = ObjProperty->GetObjectPropertyValue( B );

	return ObjectA == ObjectB;
#else
	return Cmd.Property->Identical( A, B );
#endif
}
8

Hello Marc.Hall,

I just wanted to check in and see if you were able to reproduce this issue in latest version of engine with a clean project? If you can, could you please provide a detailed set of steps to reproduce this issue on our end?

9

We decided to remove game modes using unreal’s multiplayer logic and write custom network synchronization logic, so we no longer have a test case for this.