Is there any work being done on multi-thread or multi-process cooking? That would be a huge win for us.
We currently have almost 28,000 assets that take close to an hour to cook. Is this within the ballpark of what you’d expect? We also noticed large increases in cook times from both 4.16 and 4.17 on all platforms (PC, PS4, Xbox).
We’ve added a lot of instrumentation to our cook to help track this down, but have not seen any smoking guns. Here’s an example of our cook. Do you have any recommendations on what we can look for?
Misc Cook Stats
===============
ShaderCompiler
BlockingTimeSec=0.000000
AsyncCompileTimeSec=0.000000
GlobalBeginCompileShaderTimeSec=0.000000
GlobalBeginCompileShaderCalls=0
ProcessAsyncResultsTimeSec=0.004636
GlobalShader.Misc
ShadersCompiled=0
MeshMaterial.Misc
ShadersCompiled=0
MaterialShader.Misc
ShadersCompiled=0
Package.Load
NumPackagesLoaded=2212
LoadPackageTimeSec=766.839767
Package.Save
NumPackagesSaved=27784
SavePackageTimeSec=1361.424483
BeginCacheForCookedPlatformDataTimeSec=0.187994
StripExportObjectsTimeSec=0.090898
ImportObjectsAndNamesTimeSec=644.905822
TagRelevantNamesTimeSec=0.549502
BuildImportMapTimeSec=0.388614
BuildExportMapTimeSec=0.510751
SaveExportDepenciesTimeSec=61.549071
TagPackageExportsPresaveTimeSec=262.638819
TagPackageExportsTimeSec=247.364260
ResetLoadersForSaveTimeSec=12.554244
TagPackageExportsGetObjectsWithOuter=0.631885
TagPackageExportsGetObjectsWithMarks=0.032586
SerializeImportsTimeSec=0.000000
SortExportsSeekfreeInnerTimeSec=39.703500
SerializeExportsTimeSec=43.810249
SerializeBulkDataTimeSec=3.597077
AsyncWriteTimeSec=0.153850
MBWritten=16628.566468
Cook Profile
============
0.CookWallTimeSec=3263.713983
0. 0.StartupWallTimeSec=344.205438
0. 1.CookByTheBookTimeSec=2919.506500
0. 1. 0.StartCookByTheBookTimeSec=20.913601
0. 1. 0. 0.GameCookModificationDelegateTimeSec=0.185849
0. 1. 1.TickCookOnTheSideTimeSec=2840.968260
0. 1. 1. 0.TickCookOnTheSideLoadPackagesTimeSec=564.124317
0. 1. 1. 1.TickCookOnTheSideSaveCookedPackageTimeSec=2070.635674
0. 1. 1. 1. 0.TickCookOnTheSideResolveRedirectorsTimeSec=0.000000
0. 1. 1. 1. 1.TickCookOnTheSideVerifyCanCookPackageSec=255.172285
0. 1. 1. 1. 2.TickCookOnTheSideConvertBlueprintsSec=0.752136
0. 1. 1. 2.TickCookOnTheSideBeginPackageCacheForCookedPlatformDataTimeSec=65.476262
0. 1. 1. 3.TickCookOnTheSideFinishPackageCacheForCookedPlatformDataTimeSec=0.811819
0. 1. 1. 4.TickCookOnTheSideWaitForAsyncWorkAndCleanUpSec=0.000044
0. 1. 1. 4. 0.TickCookOnTheSideAsyncWriteFilesSec=0.037170
0. 1. 1. 4. 1.TickCookOnTheSideSyncDDCFilesSec=0.009366
0. 1. 1. 4. 2.TickCookOnTheSideSaveShaderCacheSec=0.000000
0. 1. 1. 4. 3.TickCookOnTheSideGenerateManifestSec=0.000000
0. 1. 1. 4. 3. 0.TickCookOnTheSideSaveCookedAssetRegistrySec=0.000000
0. 1. 1. 4. 4.TickCookOnTheSideMapDependencySec=0.000000
0. 1. 2.TickLoopGCTimeSec=0.000000
0. 1. 3.TickLoopRecompileShaderRequestsTimeSec=0.000978
0. 1. 4.TickLoopShaderProcessAsyncResultsTimeSec=0.000800
0. 1. 5.TickLoopProcessDeferredCommandsTimeSec=0.002197
0. 1. 6.TickLoopTickCommandletStatsTimeSec=0.000184
We also have lots of ActiveClassRedirects and other redirector. Do these affect cook times?
Thanks, Joe. We regularly fixup content redirectors because we knew those would be a problem.
There were some regressions in cook time performance over the past few versions, but we’ve recently been looking at them and have made a few fixes that have helped. These changes should all be in our main branch now and should be available in 4.19. If you need some more specifics about cherry picking things that might help, let me know.
Otherwise, I’d be double checking the usual suspects. Are you happy that the assets that are being cooked are all required for your runtime game? (i.e. are you cooking test data or editor only reference data that isn’t necessary for the game?) Do you have a shared DDC server set up to cache intermediates? Have you tried iterative cooking to speed up incremental builds?
If you want to attach a full cook log, there might be a bit more information in there.
I had a quick look over those logs and compared it with some equivalent Fortnite logs. Everything kind of looked similar, so I don’t think you’re a million miles away from what we would expect. I noticed that there are a few minutes taken up by that verifycancookpackage stuff that I mentioned above, so you could save a few minutes by removing that?
Someone else will have to weigh in for the class redirects. For content redirects, they will increase cook time for each content, slightly, since it requires a load, get a new path to the content, and a final load
Thanks, Riley. That’s a useful change, but it doesn’t look like that code path is taken during cooking.
Here the timings we have:
4.12.5:
UE4Editor-Cmd: Execution of commandlet took: 339.50 seconds
4.17.2:
Cook: Execution of commandlet took: 3846.86 seconds
Around 80% of time goes to GEditorSavePackage. Is that something we can expect to be addressed in 4.19?
link text
Looking at your log, that particular run of the cooker did a lot of DDC generation. What kind of timings do you get when you re-run that cook a second time using the primed DDC?
The changes we have made so far that will appear in 4.19 have helped a lot with our own internal project cook times, although they are still slower than several versions ago. It’s difficult to say whether they will “fix” your problems, but they should hopefully improve things.
Cheers,
Graeme
Archiving just creates a complete copy of the build output in a specified directory, and shouldn’t affect cook times noticeably.
It might be worth having a look at this post, as they were reporting similar issues to you and managed to get some good improvements!
https://udn.unrealengine.com/questions/397467/cooking-started-to-take-much-longer-time-after-upd.html
The basic gist was that the VerifyCanCookPackage function on the asset manager class can be slow, and if you’re not using the “never cook” or “development only cook” functionality then there could be a win by deriving your own asset manager class and overriding that function to disable the checks within. The licensee in that linked thread reported a big win from doing that!
link text
This is a fairly fast iteration of our cooker, cooking only maps that we are actively testing. We get 97.5% hit on the local DDC, but still spend a lot of time saving the 13GB of packages.
I had a quick look over those logs and compared it with some equivalent Fortnite logs. Everything kind of looked similar, so I don’t think you’re a million miles away from what we would expect. I noticed that there are a few minutes taken up by that verifycancookpackage stuff that I mentioned above, so you could save a few minutes by removing that?
Thanke, Graeme. We have disabled VerifyCanCookPackage, and it did help. We’re also doing our best to limit the number of maps we’re cooking. One thing we found that made a huge difference was adding memory to our build machines. 64GB just wasn’t enough, and 128GB works much better!
Awesome. Yeah, more memory is a great idea as it will reduce the need to collect garbage, which can flush out loaded assets that will need to be loaded again later on!
And faster NVMe drives also help quite a bit.
Yep we have that change but our cook times are still high.
The timings when the cook ran the second time is pretty much the same. I would guess the cooker is using the cache. Though I assume it does it by default, we don’t do anything specific for that rather than using the same machine for the same configuration.
That’s the arguments we use to cook development build:
-noP4 -platform=Win64 -config=Development -game -build -cook -allmaps -stage
The arguments are copied from :
A new, community-hosted Unreal Engine Wiki - Announcements - Epic Developer Community Forums
With the exception to archive part. The argument didn’t have any effect on time in 4.12.5. Let me know if it’s recommended to use the archive for cooking.
I seem to have no access to the post. Could you please share the details that might be useful?