We’ve recently worked with a number of developers about reducing patch size. Most of the changes were related to deterministic cooking issues. That is to say, cooking the same asset twice without modifications would not produce exactly the same result. Usually the assets only differ by a small number of bytes because a GUID or other piece of generated information is getting regenerated inappropriately at cook time, or because some variable was not explicitly initialized in a constructor (although it may have been before it was actually used) so the serialized data would be different between subsequent cooks.
I wanted to list the fixes here for developers here to reference in case they need to manually merge these changes back to older engine versions. We can’t guarantee every change to a newer engine version can be successfully merged to older engine versions, but we’ve had good luck so far merging these changes back as far as 4.13.
When you run into deterministic cooking issues, the first thing you should try is re-saving the asset. Often the class that defines an asset changes between releases, and after upgrading when loading that asset the data is automatically fixed up, sometimes generating new unique IDs or random data in the process. If that asset is not then saved, it goes through the upgrade process each time it is loaded, creating the potential for results to differ in subsequent cooks. So re-saving any assets that don’t cook in a deterministic manner is the first step.
3067138 Fix the guid keep generated by adding to the database. This caused worse problem with non-deterministic cooking - This doesn't fix UE-33454 for 100%, but this was the main reason why this was so visible UE-33772, UE-33454
3067168 Smartname guid will be discarded during cooking, and once it's cooked, it's trusted to have correct name. UE-33454
3094606 #ANIM: SmartNAME: the cooking doesn't guarantee the package is saved in the order, so we'll still have to regenerate list without GUID. - assumed the name is all set by now UE-34886
3099251 Deterministic cooking of skeleton - abandon all guid from GuidMap. GuidMap is still important since we have to generate UID from it, but GuidMap only contains name once cooked UE-34834
3121083 Fix for bake distributions deterministic cook issue caused by building bake lighting before postload called on it's children distributions.
3121456 Attempt to fix deterministic cooking issue for particlelodlevel. Ensure the spawn module has had postload called on it before using.
3137054 PR #2628: Fix UAudioComponent SubtitlePriority not being initialised (Contributed by alanedwardes)
3145064 Pulling range update for vector distributions even when UDist is not dirty; some content has a lookup table and a clean dist, but the range values have not been baked; always pulling them should be safe and not significantly costly. UE-36091
3395539 Fixed: Particles with GPU sprites cause deterministic cooking issues UE-44042
3426941 Fix determinstic cooking of LoadAssetClass nodes in macros UE-45488
3451392 Remove UID from smart name serialization to help with non deterministic animation cooking UEAP-249
3453782 Sequencer: Fixed deterministic cooking issues with movie scene data - Movie scene signatures are now initialized in PostInitProperties - A warning is now presented when attempting to cook old data that was never serialized with a signature. - Removed redundant legacy data upgrade logic that could dirty level sequences on load. UE-44912
3462376 Determinism cooking fix from recent work. Make sure we cannot get to compression with invalid curve names
3467508 Add a function to fix a long package name so it matches the case of a file on disk. Fixes deterministic cooking issues when on-disk case changes.
3467510 Fix deterministic cooking issue caused by LODGroup only being initialized in the CDO if it's serialized, causing inconsistent delta serialization for instances.
3469392 Submit sort by UID, dangerous to determinism and will not be needed once smart name reworking is done.
3475788 Fix possible curve UID corruption when baking additive data on animations that haven't hit post load (debug feature enabled by accident, 3477344 fixes debug feature enabled by accident) UE-45731
3515495 Had reports some AnimationBP deterministic cooking errors were being caused by FBakedAnimationState::bAlwaysResetOnEntry not being initialized in the constructor explicitly. Changing that to be explicitly initialized to false in the constructor, as well as UAnimStateNode::bAlwaysResetOnEntry.
3121083 was a project-side change, so the change list isn’t visible to anyone outside epic, but it was a simple change so I took a screenshot of the diff:
Same with 3099251
Another change we have successfully fixed some particle effect deterministic cooking errors with, but haven’t been able to verify enough to commit to source control yet is this change to UParticleModuleColor::CompileModule:
void UParticleModuleColor::CompileModule( FParticleEmitterBuildInfo& EmitterInfo )
{
FVector InitialColor;
float InitialAlpha;
FRandomStream RandomStream(GetTypeHash(GetName()));
InitialColor = StartColor.GetValue(0.0f, nullptr, 0, &RandomStream);
InitialAlpha = StartAlpha.GetValue(0.0f, nullptr, &RandomStream);
EmitterInfo.ColorScale.InitializeWithConstant( InitialColor );
EmitterInfo.AlphaScale.InitializeWithConstant( InitialAlpha );
}
There are a few other deterministic cooking issues that we’ve yet to resolve:
[UE-44910 - Skeletons don’t cook in a deterministic way][3]
[UE-44913 - Some Textures don’t cook deterministically in QAGame][4]
[UE-45856 - Cooking is not deterministic for Destructible Meshes][5]
[UE-44909 - Paper2D Sprite doesn’t cook deterministically][6]
The last issue about the Paper2D sprite, re-saving the asset makes the problem go away. The other issues don’t seem to be causing patching sizes to become overly large, but the status of these issues can be checked by visiting the links to our issues.unrealengine.com site.
Also, there are a few things that can cause issues with deterministic cooking that Epic doesn’t have control over.
One issue we have seen in the past is that some third party tools can cause deterministic cooking errors. One example was a code integration from Enlighten that added a Enlighten-specific GUID to Primitive Components, but regenerated it each time. This caused level files to have deterministic cooking problems, and patch sizes to increase. We’ve heard that Enlighten has a fix for the issue, so hopefully it won’t be a problem for new versions of Enlighten, but it’s something Epic can’t fix since it’s not our code, so please contact Enlighten directly.
Lastly, it’s possible for project side logic to cause deterministic cooking problems. This can happy when, for example, defining a blueprint node in C++ that creates a new GUID, and then calling that node from a blueprint construction script. Blueprint construction scripts get run during the cooking process, so if care is not taken, that GUID will get regenerated with each cook, and the resulting assets will differ so when making a patch the patch size will increase. The generation of GUIDs may not be explicit in the project side code, but come from calling functions from an engine system that you are not 100% familiar with that generates a GUID under the hood. This can be avoided by marking any blueprint nodes defined in your project’s code that don’t need to be called in a construction script to be not place-able in construction scripts. If you do need to define construction script callable nodes in C++ in your project, do so with care. When creating them, test by creating an example blueprint that uses the node in a construction script and then cook that asset twice in a row with no changes to make sure it gives the same binary result.
Thanks.