More level Streaming with Static NavMesh

Simliar to this: https://udn.unrealengine.com/questions/381719/staticnavmesh-and-level-streaming.html and others.

But, different, larger issues with the approach.

Our understanding is that in order for the master persistent level’s nav mesh to dole out chucks to the sublevels, all the sublevels (with nav boxes) need to be loaded concurrently. This means that they all have to fit in memory, and presently, even with half a dozen levels, its becoming unmanageable. Similarly, requiring all the sublevels to occupy unique bounding boxes, with no overlaps, is also problematic as the # of levels grows.

Given that these concerns are voiced broadly, has this been identified as a priority to resolve? If not, we are willing to invest our dev time to address these issues. We believe the existing nav meshes authored with the sublevels outside of the global persistent context have value, and should be used at runtime. Is this possible? What roadblocks could be forseen with this approach?

Thanks!

Hi Brent,

We actually plan to have a stab at this early next year, so if you can hold off you might just get improvements without spending dev time on it.

If you prefer to work on this instead the way I will approach it is by adding additional options to NavMeshBounds volume that will allow it to figure out where does it reside in relation to other levels and most importantly in relation to persistent’s RecastNavMesh’s origin. The change will also require to store navmesh generation properties (mainly tile sizes) to be able to discard “local” navmeshes that would not fit the whole navmesh tile structure once streamed in. Once this is in it should be relatively straightforward to be able to load in navigation data and append read navigation tiles to the persistent navmesh.

Cheers,

–mieszko

Hi Mieszko,

Appreciate the responsiveness, and glad to hear its on the list. We will probably take a stab at it in the meantime. In our situation, we only ever care about the incoming meshes and volumes, as there are none that carry over between levels. This may simplify our case further over the general one.

Is there a way to follow your dev in this area when it begins?

Thanks,

Brent

Thanks Mieszko, I had a similar brain fart last night but your snippets showed me a much smaller implementation. This does appear to work for us. We’ll have more data for testing later this evening, but my initial cases were positive. Will update when we have more samples, but looking good so far. Thank you!

Hi Mieszko, do you have an approximate date for when improvements to streaming static navmesh will occur?

We are able to get static navmesh streaming to work but it is incredibly inconvenient. Everytime navmesh is rebuilt, every sub-level requires saving. This means that navmesh cannot be built if any of the sub-levels are checked out. Everytime a sub-level is modified, it breaks all navmesh for its P-level (not just the single sub-level).

Hello Mieszko,

I’m the programmer at Toys For Bob who will be investigating and doing this work. I’d love to be able to pick your brain at more length about our best approach.

Thanks!

Steve

My work on this should begin early next year, currently loosely scheduled at the beginning of January.

Regarding picking my brain on this, we can take this “off line”. Shoot me an email. But the overall idea, at least for the first step, is to expand the NavMeshBounds so that they can instruct the navmesh generation process regarding the “world” location of tiles being generated. That required the bounds to know what’s the ‘origin’ of world’s nav mesh (needs to be consistent across all sublevels). It also requires every sublevel to use exactly the same navmesh generation properties, but those can be read from RecastNavMesh’s CDO.

Other approaches I can think of are more generic and therefore won’t help you and will require more work.

Thanks Mieszko,

Can you email me at smariotti@toysforbob.com to set up a dialogue outside of Answers? I have a number of questions about lifetiming and order-of-operations during load. Your knowledge will save me a lot of debugging time to understand the system.

Is that cool?

Thanks!

Steve

Sent.

–mieszko

Thanks Mieszko, but I figured I’d come back here as email may not be the best way to reach you.

Right now I’m trying to identify where the streamed nav mesh data is loaded, and then more importantly, where it is discarded.

To lay out my testing case:

GlobalPersistentLevel is our level that loads into the menu and then loads sub-levels for gameplay. All sublevels brought in this way have Static(statically generated) NavMesh data, and none are set to Dynamic/Create on Load. They are, however, all occupying the same world locations as each other sublevel (and the GlobalPersistentLevel) and are not world-contiguous a la WorldComposition. After loading through our menu system, we see no NavMeshes and I can’t find the code that dumps them anywhere in the streaming load/serializing code in the engine.

If we load one of the sublevels in Editor and then do PIE, the NavMeshes remain intact, and the code path is similar but not quite the same as when loading a sublevel from our UI.

The theory is that these sublevels are having their nav data dumped because during the load process a bounds check is done and it’s determined that none of the sublevels have valid nav data in the locations that the engine cares about.

The proposed fix is to retain the nav mesh data by providing bounds to encompass the sublevels in our GlobalPersistentLevel, and to modify the engine code so that sublevel tile data for the nav mesh gets brought in piecemeal into the GlobalPersistentLevel’s nav mesh actor.

For PIE I see the ARecastNavMesh being serialized in as part of object duplication for PIE, it then makes itself a FPImplRecastNavMesh and they cross-associate to each other by sharing their pointers. In PImplRecastNavMesh’s Serialize() function, the underlying DetourNavMesh is created and then serialized in. The Serialization functions get hit repeatedly for reference counting and loading and saving, which makes for a difficult to trace code path when trying to determine who, after the valid DetourNavMesh has been loaded, it gets dumped again (or just not added to the world.)

So my more direct questions are these:

  1. Where does the code determine whether or not the serialized-in valid pre-generated static nav mesh data can be discarded?
  2. Is this done against bounds in the local level or in the persistent?
  3. Where is the data dumped?
  4. When ULevels are being brought into the world, and the Archive flag shows that it contains a map, is that enough information to identify nav mesh data coming from that archive’s serialize call chain?

With some of this information and the ability to step through and see it, I’ll be ready to ask you even more specific questions. I’m deeper in the bowels of the engine than I’ve been before, and the garbage collection and serialization system are making it hard for me to set clever breakpoints to catch the engine in the act. I’m having to do a lot of engine mods with logging and parse log files. Any better grounding you can supply me would be a big win.

Thanks for help with this,

Steve

Let me give you a high-level description of how navigation data is handled by the NavigationSystem and how static navmesh streaming works.

First thing to remember is that there’s always only one instance of RecastNavMesh per supported agent (see Project Settings → Navigation System → Supported Agents). Then you load a persistent level a RecastNavMesh instance is either loaded or created. If levels being streamed in contain RecastNavMesh instances then those will get destroyed (see UNavigationSystem::ProcessRegistrationCandidates and UNavigationSystem::RegisterNavData).

The way static navmesh streaming works is a bit awkward (new version will be done next year, keeping the legacy approach around for compatibility). Static navmesh is prepared for streaming if in the editor you load all levels, build navigation and save all. The navmesh parts will be saved as URecastNavMeshDataChunk along with NavMeshBoundsVolumes encompassing them in those volumes’ sublevels.

Hope it help.

Thanks for the reply Mieszko,

We only have one supported agent, the default, so there’s only one RecastNavMesh actor per level.

When launching through the UI in our game, I get no valid ARecastNavMesh from the streaming system, even though it stream-requests the level with the nav data in it. When that level loads, I only get the AbstractNavData registered, not the actual nav data. It’s like it doesn’t even try to load it. Or it loads it and discards it elsewhere. This is BEFORE I get to the NavigationSystem::ProcessRegistrationCandidates()/RegisterNavData() code path.

Is there a way to verify on disk that the nav data DID get generated for the sublevel that’s going to be streamed in? Now I’m starting to suspect that the nav data just ain’t there.

I’ve loaded our GlobalPersistentLevel and done Build → Build Paths (to regenerate the nav meshes in sublevels.) Would this be failing because either the sublevels are hidden or the GlobalPersistentLevel lacks navigation bounds that encompass all of the (hidden) sublevels?

EDIT: Brent tells me that trying to generate nav data in the GlobalPersistentLevel is bound to fail since the levels occupy the same 3d space when loaded. So we’re generating nav data for each sublevel individually when we open them, let it generate, and then save all. So that data should be there when streaming in, yes?

More details.

I looked at ProcessRegistrationCandidates()/RegisterNavData() in my use case, and these are not getting called or doing any work when moving from game shell menu to gamplay. I do see them getting called when 1) I run the editor and it loads the default GlobalPersistentLevel in the editor, and 2) I hit Run (either PIE or Standalone) and it loads into our UI.

When we hit Play in the UI to enter gameplay (this is the problem case), it sequentially streams in a bunch of sublevels via Blueprint LoadStreamLevel() in GameplayStatics. I see these FStreamLevelActions get queued and processed for each sublevel including the one with the Nav data.

I see the sublevel with the nav mesh data get streamed in (FPImplRecastNavMesh::Serialize() and ARecastNavMesh::Serialize() reports a valid size for nav data via a log statement I added:

 ([2017.12.21-20.09.00:461][  0]LogTemp: *** RecastNavMesh::Serialize() : NavMeshSizeBytes for LS101_ART_MASTER is 10121996)

by breakpointing in UWorld::UpdateLevelStreamingInner() I see that this map (LS101_ART_MASTER) is indeed added to the UWorld.

BUT. I never see this level ever added via UNavigationSystem::RegisterNavData()

Because the RegisterNavData() mechanism is NOT used when launching into gameplay, the changes you discussed previously don’t seem to apply in this case.

Thank you for your continued help,

Steve

Ok, I did some digging.

You don’t see ProcessRegistrationCandidates nor RegisterNavData calls because ARecastNavMesh::PostInitProperties can (and in your case probably is) call CleanUpAndMarkPendingKill which will result in RecastNavMesh instance getting destroyed before it makes it to the registration phase.

Regarding building navmesh once all the levels are loaded up in the editor, yeah, this won’t work for you. It seems like our navmesh streaming apporoach is not for you.

Now the good news: I made it work :slight_smile: See the attached diff file, it contains the whole related code change. I’m not submitting it yet because it needs some cleaning and testing (plus ensuring full backward compatibility), but (assuming I understand your use case) it will solve your problem.

The way it works with the change is that if you don’t have a RecastNavMesh instance nor any NavMeshBoundsVolumes in the persistent level then the navigation system will not discard RecastNavMesh instance coming from the levels being streamed in. Said navmesh instance will be used as the MainNavData. If you stream in more than one navmesh instance only the first one will be accepted (assuming using a single SupportedAgent). The exception to this rule is loading levels in the editor - the editor world’s navigation system will not discard additional navmesh instances in order to support PIE properly.

In short, as long as you have a single sublevel with RecastNavMesh instance active at a time my change will make it work automagically. Give it a try and let me know if it works for you.

Cheers,

–mieszko

Hello Mieszko, and happy New Year!

I integrated the changes this morning and it works great! I owe you an offering. Maybe one of the ones I made to the gods while working unsuccessfully through the source code.

So as you alluded to, it doesn’t work properly for PIE (advice?) but works fine for standalone. We’re testing packaged builds as well but I suspect that will all be good too.

Thanks again!

Steve

It does work in PIE for me, but my setup might be oversimplified. Can you try disabling auto navmesh rebuilding and see what happens?

We have auto nav mesh rebuilding disabled. The PIE issue is less urgent.

Unfortunately, I spoke too soon before. It does work sometimes, but not consistently. It smacks of either a race condition or some stored state, as we discover the RecastNavMesh-Default’s nav mesh translated way way away from the level that it’s supposed to load into. We suspect it’s at the world location in the PREVIOUSLY loaded sublevel.

Our behavior is not predictable. We are trying to isolate test cases where the nav mesh doesn’t load to the correct location and we still can’t fully predict it. We’re loading between levels and seeing behavior like this within a single run of the game:

Level 1 (valid nav mesh) → Level 2 (valid nav mesh) → Level 3 (no valid nav mesh) → Level 2 (valid nav mesh) → Level 1 (no valid nav mesh)

When the nav mesh is “not valid” it means that either it’s loaded but translated far away, or it’s not displaying anywhere when looking for it with debug camera.

All of our testing is with the Standalone game.

Our current theories are:

  1. a stored offset is causing the NavMesh to load into the wrong place. We don’t know what’s storing it, but it’s acting like state is saved somewhere somehow.
  2. an order of operations problem when loading in the streaming data is causing it to discard or offset the wrong nav mesh. We see an AbstractNavMesh coming in via streaming as well as the valid RecastNavMesh-Default object. In some cases we see the real one, in some cases not, and in some cases the real one is offset.

It’s been hard to find any sort of pattern, but it’s clear the Nav Data isn’t loading into the right place. Maybe it’s loading at a different time during level bringup?

Brent and I have been working on this all day and we’re both pretty good at reversing behavior from output and we don’t see a pattern yet.

Is there something that could account for incorrect nav mesh data offsets with repeated loads? Due to your suggested change, is something sticking around longer than it should be or hiding the valid data?

Thanks again for your help. We’re not out of the woods yet.

Steve

Oh and another point of information: If we load into a level and the nav mesh is not valid, repeatedly loading the level 2-3 times causes it to appear in the correct location.

All loads are done using the same code path/mechanism, so there’s no difference in how we stream in each level in our game code.

It looks like the LevelTransform of the StreamingLevelObject is not getting applied to the NavMesh. If we cycle enough loads, we will eventually end up loading a level at the origin, and the nav mesh appears correct.

I can confirm this. In FLevelUtils::ApplyLevelTransform, if the Actor in question is an ARecastNavMesh, calling ApplyWorldOffset with the LevelTransform.GetTranslation() properly re-positions the nav mesh. Its an ugly hack that we can’t ship, but it does get us working.
I’ll have to look at it a bit harder in the morning to see if there is a better way, hopefully not on main thread, to do this.