Replication issue of actors placed in "always loaded" sublevels when seamless travel is enabled

Synthesis:

Replication issue of actors placed in “always loaded” sublevels when seamless travel is enabled.

Question:

Should replication work in same way regardless travel mode chosen? (Seamless or not)

Case:

We create a blueprint project.

We create two actors that replicate. One of actors is placed in a sub level.

We travel from level “lobby” to “level_1”.

We expect to see two cubes increase in their height (listening server and client side).

What we observe is that cube placed in sublevel is not properly replicating on client.

(Debugging engine code we notice that UNetDriver::IsLevelInitializedForActor() keeps returning false even when actor is shown in client.)

Steps to reproduce issue:

  • Launch engine.

  • Select New Project.

  • BluePrint ThirdPerson.

  • (Optional Give name).

  • CreateProject.

  • Wait until loading is complete.

  • Remove ThirdPersonCharacter from WorldOutliner of level.

  • Select TemplateLabel in World Outliner and change field named “Text” in Details window to “Lobby”.

  • Save.(Ctrl-s)

  • File SaveAs.

  • Click on ThirdPersonBP folder.

  • Click on Maps.

  • Change field “Name” into “Level_1”.

  • Select TemplateLabel in World Outliner and change field named “Text” in Details window to “Level_1”.

  • Drag a Basic Cube into world.(Fig 1)

69275-fig1.jpg

  • Click on Details panel, Blueprint/Add Script.

  • Click on Create Blueprint.

  • Select Always Relevant and Replicates.(Fig 2)

  • Compile and Save.

  • Click on Event Graph and copy Blueprint from Fig 2.

  • Be aware of flags in “Custom Event Server” node (Run on Server, Reliable) and “Custom Event Client” node (Multicast, Reliable).

  • Compile , Save and close Blueprint editor.

  • Save level.(Ctrl-s)

  • Click on Blueprints.(In top bar, between settings and cinematics[Matinee in version 4.9])

  • Under Project Settings select “GameMode: Edit ThirdPersonGameMode” - “Edit ThirdPersonGameMode”.

  • Enable “Use Seamless Travel”.

  • Compile , Save and close blueprint editor.

  • Click on Window and select Levels.

  • Click On Levels Create New, name it SubLevel and save.

  • RightClick on Sublevel and select Change Streaming Method to Always Loaded. (Fig 3)

69277-fig4.jpg

  • World Outliner select Cube_Blueprint , duplicate it using Ctrl-w. (this copy will go into SubLevel because is one is selected now)

  • Move Cube_Blueprint2.(Fig 4)

69278-fig5.jpg

  • Save. (Ctrl-s)

  • Select down arrow near Play button and press “Change Play Mode and Play Settings”.

  • Select Advanced settings.

  • Scroll down to Multiplayer Options, change Number of Players to 2, remove “Use Single Process”, Select “Play As Listen Server” inside Editor Multiplayer Mode.

  • Close Editor Preferences.

  • Open lobby using File Open Level “ThirdPersonExample”.

  • Select down arrow near Play button and press “Change Play Mode and Play Settings”.

  • Select Standalone Game.

  • Select server window (if you don’t know which one is just try one) (generally client window is one that has focus when you launch game).

  • Open console. (Generally ~key )

  • Issue command “servertravel level_1”. (If it is not doing anything is client window. Select other window and retry)

  • Verify in client window that second box ( one in sublevel) does not grow. (You need to navigate a little with character to see it)

To verify that this works when seamless travel is disabled:

  • Close client and server windows.

  • Click on Blueprints(between settings and cinematics), under Project Settings select “GameMode: Edit ThirdPersonGameMode” - “Edit ThirdPersonGameMode”.

  • Disable “Use Seamless Travel”.

  • Play.

  • Select server window.

  • Open console.

  • Issue command “servertravel level_1”. (if it is not doing anything is client window. Select other window and retry)

Final Notes/Considerations:

replication issue is present on Win64(8.1). Engine version 4.9.2/4.10.0

issue also occurs if a C++ template project is used.

issue does not occur if game is started from that level_1. (It only happens after travelling)

replication seems to work again if client issue a reconnect command after that seamless travel is
finished. (But that is what we are trying to avoid using seamless travel)

Hey Stefano,

Thanks for report! I was able to reproduce this issue in 4.10.1 as well as our internal build, so I’ve created a bug report for developers to consider (UE-24280). I’ll post here if I see any updates on it. Thanks!

Just FYI we are seeing exact same issue in Space Dust Racers using UE4.10.

Thanks a lot of this extensive bug report and repro case. A colleague and I spent a few hours today digging through code with information provided and have found some more information and a temporary workaround.

problem lies in UWorld::AddToWorld (around line 1921), with this call being made on client at end of server travel:

It->PlayerController->ServerUpdateLevelVisibility(PackageName, true);

This RPC never gets sent from client to server, and so server never gets informed about sublevel visibility for client’s connection. If you step into RPC call you’ll see why it’s not sending - inside AActor::ProcessEvent it fails on a check to GetWorld()->AreActorsInitialized().

We looked around elsewhere in codebase and found a gem inside NUTUtilNet.cpp on line 517 -

// Hack-mark  world as having initialized actors (to allow RPC hooks)
ReturnVal->bActorsInitialized = true;

It seems somebody else has experienced same RPC limitation, haha! So back in UWorld::AddToWorld, if you hack bActorsInitialized = true then perform ServerUpdateLevelVisibility RPCs, then restore original bActorsInitialized value afterwards, everything works as expected.

Obviously this is just a workaround, but hopefully it helps Epic team come up with a proper fix.

, you may want to add this information to UE-24280.

Cheers,

Thanks! I’ve added information you provided.

Hey was UE-24280 ever addressed?

In our game we have flag objectives that when placed on a sublevel do not replicate after ServerTravel. For now we will just bring them back up to main level but it adds a bit of redundancy to our map organization.

Hey Mentos,

Here’s link to public tracker so you can keep an eye on this issue: Unreal Engine Issues and Bug Tracker (UE-24280)

Sorry for grave digging, but…it’s been 3 years, and bug still hasn’t been fixed.
Do you have any update Sean?

Because hack is fine and all, but I don’t feel like anyone should have to make a custom engine to have such an important feature working properly…

I’ve done some research into this as we were having this problem with our game:

reason this is happening is that when we are seamless traveling from one map to another, client is loading map faster than server. We had this by restarting a simple map with client already connected to it and seamless travelling to it.

Every time we replicate an actor from server to a client connection we check if UNetConnection::ClientHasInitializedLevelFor. If server thinks client doesn’t have level initialized/visible, we don’t replicate info.

RPC that updates server APlayerController::ServerUpdateLevelVisibility gets sent by client as soon as each sub level is loaded, and on server, we do UNetConnection::UpdateLevelVisibility which adds to set UNetConnection::ClientVisibleLevelNames visible map for that connection. However, since server is loading map slower than that client, this info is received whilst server is still transitioning to TransitionMap. We call UNetDriver::PreSeamlessTravelGarbageCollect() every time server seamless travel which calls UNetConnection::ResetGameWorldState() and in turn empties UNetDriver::ClientVisibleLevelNames. This clears all info that client loaded these sublevels, making levels not replicate.

simplest workaround is to either move all your replicated actors to your persistent level, or you could set your levels to load via Blueprint and try loading them in GameModeBase::PostSeamlessTravel() (not sure if that would actually work). proper solution to this problem most likely would be to queue up RPC calls from client about visible levels and only process them after server has finished travelling. But that’s for Epic people to figure out :wink:

P.S. There is also a memory leak with variable UNetConnection::ClientVisibileActorOuters that it never gets emptied. It’s just a map storing a cache of level actors to its client connection visibility. This essentially means if you seamless server travelled same client connection a lot of times or you had a lot of sub levels it would eventually break. You can probably quickly fix this by emptying it in same place UNetConnection::ClientVisibleLevelNames is done in UNetConnection::ResetGameWorldState()

1 Like

Epic Games, please, fix it when you’ll have resources. We’re experience big problems cause of that bug at present time.

Hey. We have been faced with exact issue described here by Cimmerick, during a seamless travel to same map client were in already, in which case clients tend to load faster than server. Indeed, Client Level Visibility RPCs were sent before server went into ResetGameWorldState() and cleared everything, causing some serious but random replication issues on any of our interactive/dynamic actors in world.
For sharing purposes, we fixed those by doing exactly what you suggested, which is “queue up RPC calls from client about visible levels and only process them after server has finished travelling”. To be exact, we kept RPC as is, but deferred call to UpdateLevelVisibility to after call to ResetGameWorldState. As of now it gave us good results, and fixed our replication bugs.
Cimmerick, thank you for detailed explanation, it definitely saved us time narrowing down, debugging code.
Regards,
Gabriel

Hey Gabriel! What would be awesome is you did a pull request on GitHub with your changes so Engine so everyone can have this fix. Use issue number UE-24280. Contributing to the Unreal Engine | Unreal Engine 5.1 Documentation

That would be great!

Hello,
As requested we submitted this as a pull request, identified as gabriela-funcom:temp_fix_for_UE-24280. I don’t know if you guys can see code for this change.
To be honest, although it’s working fine for us on our project at Funcom, I doubt that Epic is going to take this in, as it still looks like a hack to me. So maybe you can sneak in and check code yourself if you need to.
Thanks. Best regards,
Gabriel

https://github.com/EpicGames/UnrealEngine/pull/6301

Thanks for pulling request! You also pointed in message, that community experiences problems with that bug. That increases chances of Epic Games fix that bug eventually.

After 5 years, it is still a valid bug in 4.24.3.

this says it’ll be in 4.26 Unreal Engine Issues and Bug Tracker (UE-24280)

There’s also a link to git commit so you can try integrating fix manually

Thanks for clarification
TelltheBell.com