HISM UpdateInstanceTransform crashes (with repro)


Background

For a few major versions now, HISM (Hierarchical Instanced Static Mesh) has been particularly crash-prone during run-time updates, i.e. when UpdateInstanceTransform, AddInstance, etc are used on a HISM component after the game has begun. The crash typically happens in the rendering thread due to a HISM “ClusterTree” being invalid.

After chasing this for ages I finally have a clean simple repro project that demonstrates exactly what triggers this HISM crash.


Repro

Run the sample project, it should crash within 8-10 seconds. If not, just wait a bit or try a few times again. This bug is timing related so it behaves a little differently each time you run it. The relevant code is in HISMCrashSimulator.cpp’s Tick function.

HISM is likely to crash whenever UpdateInstanceTransform is called with a location that is exactly the same as the current location of the instance. Furthermore, calling AddInstanceTransform around the same time practically guarantees that the crash will happen (that’s what the repro project does).

In other words, if you need to update only the scale or rotation of a HISM instance and leave the location intact (or if you simply pass the instance’s current transform back to the update function) it eventually crashes and especially so if a new instance is also added at or around the same time.


Crash details

The crash occurs in the rendering thread at HierarchicalInstancedStaticMeshComponent.cpp in the function UpdateInstanceTreeBoundsInternal_RenderThread. Specifically - FClusterNode Node = &ClusterTree[0];* which produces undefined behavior because ClusterTree’s memory is invaild.

Sometimes it crashes right there, other times further ahead, but perhaps the most frightening issue of all is when it freezes the entire unreal process inside some WinEvent thread wait routine after which it is very difficult to even come out and close the program!


Workaround

HISM’s code treats “In-place-updates” (i.e. location unchanged) as a special scenario, this is where the issues manifest.

So the workaround is to add a tiny offset to the location so we can bypass that scenario, like this:

float tinyOffset = KINDA_SMALL_NUMBER * 10;
newTransform.AddToTranslation(FVector(0, 0, tinyOffset));

// newTransform's scale or rotation can now be manipulated as desired 
// and UpdateInstanceTransform can be called safely

Obviously far from ideal, but until there’s a fix I really needed a way to stabilize my release builds so I could move forward and perhaps others using HISM may be in a similar situation as well.

I hope this report helps in fixing the issue!

HISM is one of my favorite features in Unreal Engine and the possibilities are endless when you add dynamic behaviors to it (that’s where the tricky bugs usually come in). I hope to see HISM become even more awesome over time!

Thank you.

HI ,

  • Does this occur in a clean, blank project with no additional content or is it limited to one project?
  • What steps can I take to reproduce this on my end?
  • Are you doing this all in c++ or in blueprints?

I’ve already provided a simple repro project in the first section of this report! :slight_smile:

The repro project uses C++ but if my analysis of the bug is correct it doesn’t matter whether C++ or blueprints are used.

One important update: Further testing reveals the workaround I provided in this report isn’t 100% reliable :frowning: It greatly helps in reducing crashes but doesn’t eliminate it.

, let me know if the repro project link in the OP is not visible to you for some reason or if the instructions in the Repro section above are not clear. This is a simple blank project with just one tiny C++ class in it.

Thank you so much for your time! My release builds are in jeopardy until I can prevent this crash somehow.

Hey ,

Firstly I would like to preface by saying, by updating these HISMC on EventTick, it needs to run on every single game cycle. That means you can’t even draw a frame until the BP is done doing whatever you told it to do, which in your case add another HISMC. Secondly, you are using a for loop to also update the number of instances per tick, which is equally as dangerous. Remember that when making anything on the Event Graph, do your very best to keep things event-based, rather than being driven by Tick and you’ll be fine.

Instead of running on EventTick, try using a Timeline and set it to ‘Auto-Play’ which you can then set a custom time dilation to control when your HISMC updates. This keeps your blueprints needing to wait for the event tick to complete before being able to tick again. With that said, I was able to load your project and did not experience a crash. Could you provide me with your ‘dxdiag’ so I can take a look at your system specifications?

Once you attempt the suggested alterations to how you call the HISMC to be updated, let me know if you still experience the same issues/crash and we can continue with the troubleshooting process.

Thank you,

Hi , I’m certainly not using EventTick or For loops for any HISM work in my actual game! Things are done via conventional event based paradigms.

The repro project was written in that way only because it was the easiest way I could demo the crash with some degree of certainty (and indeed, it still ended up not crashing for you :slight_smile: ). Btw did you try the level a couple of times for more than 10-15 seconds? Reproducible on multiple machines for me (Dx diag attached)

Some background on how my game uses HISM might be relevant: it’s a simulation that uses HISM to model individual plant parts (leaf/stalk/etc) in large numbers. These undergo lifecycle changes that trigger transform updates (mainly to hide instances).

Definitely possible that two unrelated entities ended up calling a HISM update for at least two consecutive ticks. Crash is hard to reproduce but occurs within 5 minutes of gameplay. I tried a centralized queue to control update frequency but admittedly I never tried going slower than at least one update each tick (I need to support a “High-speed simulation” feature of stuff growing and receding at anywhere between 1x to 500x times normal speed). I totally understand such usecases may be beyond what HISM was meant for. I just wish it wouldn’t crash the entire application though!

Note:- My builds are stable now after applying workarounds but I think HISM should guard itself against crashes better, hence this bug report.

Workaround #1 - see bug report.
Workaround #2 - There are some meshes that HISM just doesn’t like regardless of avoiding “in-place” updates. I just use plain ISM for those now.

link text

Just wanted to chime in saying I’m experiencing the same issue. In my case, I’m frequently updating the transforms of various HISM instances, as well as clearing all instances and adding new ones. I tracked it down to the same issue that reported here. The rendering thread is trying to access ClusterTree which could be invalidated at any time in the game thread. There are two places in UpdateInstanceTreeBoundsInternal_RenderThread that retrieve an index of ClusterTree that I’ve seen crashes at in 4.13

This crashes here :

bool UHierarchicalInstancedStaticMeshComponent::UpdateInstanceTransform(int32 InstanceIndex, const FTransform& NewInstanceTransform, bool bWorldSpace, bool bMarkRenderStateDirty, bool bTeleport)
{
	if (!PerInstanceSMData.IsValidIndex(InstanceIndex))
	{
		return false;
	}

	if (IsAsyncBuilding())
	{
		// invalidate the results of the current async build we need to modify the tree
		bConcurrentRemoval = true;
	}

int32 RenderIndex = InstanceReorderTable[InstanceIndex];

InstanceReorderTable can have a size of 0 so it causes a crash.

We have the same issue. Our blueprint calls UpdateInstanceTransform() and crashes on the dedicated server. The InstanceReorderTable can have a size of 0 while PerInstanceSMData.IsValidIndex(InstanceIndex) can be true.

Could this be the fix ? https://github.com/EpicGames/UnrealEngine/commit/efb7531aef6da5a92dfd4c73a7443b627420a9e6

There was a bug entered for a crash dealing with HISMC crashing when called to update.

UE-36188

It has been marked as fixed for 4.14 so I would test your setup in that build to see if it still occurs.

Thanks,

H