Hot reloading significantly slower in 4.17? (3sec->13.5sec)

I upgraded from 4.16.1 to 4.17.0 recently, and I noticed that hot reloading is much slower than it used to be.
I tried hot-reloading on a backup of the 4.16.1 version of my project and verified this.

The following is the build log when adding a space and compiling a single bare-bones (25 line) cpp file for an actor in my project via the Build->Build Solution menu in Microsoft Visual Studio Community 2015 (Microsoft Windows 10 Pro). The 4.17 QFE fixing a build issue (QFE-4.17-UBT-CL-3578350-PF-3578389-Job-7937603.exe) has also been applied. I have not made any modifications to the engine’s code.

4.17.0 Build Log:

1>------ Build started: Project: UE4, Configuration: BuiltWithUnrealBuildTool Win32 ------
2>------ Build started: Project: Project, Configuration: Development_Editor x64 ------
2>  Compiling game modules for hot reload
2>  Parsing headers for ProjectEditor
2>    Running UnrealHeaderTool "D:\UnrealProjects\Project\Project.uproject" "D:\UnrealProjects\Project\Intermediate\Build\Win64\ProjectEditor\Development\ProjectEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -installed
2>  Reflection code generated for ProjectEditor in 6.8945103 seconds
2>  Performing 3 actions (4 in parallel)
2>  ShortActorCppFile.cpp
2>  [2/3] Link UE4Editor-Project-3558.dll
2>  [3/3] Link UE4Editor-Project-3558.lib
2>     Creating library D:\UnrealProjects\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-3558.suppressed.lib and object D:\UnrealProjects\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-3558.suppressed.exp
2>     Creating library D:\UnrealProjects\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-3558.lib and object D:\UnrealProjects\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-3558.exp
2>  Deploying ProjectEditor Win64 Development...
2>  Total build time: 13.49 seconds (Local executor: 1.94 seconds)
========== Build: 2 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

4.16.1 Build Log:

1>------ Build started: Project: UE4, Configuration: BuiltWithUnrealBuildTool Win32 ------
2>------ Build started: Project: Project, Configuration: Development_Editor x64 ------
2>  Compiling game modules for hot reload
2>  Performing 3 actions (4 in parallel)
2>  ShortActorCppFile.cpp
2>  [2/3] Link UE4Editor-Project-8506.dll
2>  [3/3] Link UE4Editor-Project-5015.lib
2>     Creating library D:\UnrealProjects\BENCHMARK_4_16_1\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-8506.lib and object D:\UnrealProjects\BENCHMARK_4_16_1\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-8506.exp
2>     Creating library D:\UnrealProjects\BENCHMARK_4_16_1\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-8506.suppressed.lib and object D:\UnrealProjects\BENCHMARK_4_16_1\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-8506.suppressed.exp
2>ERROR : UBT error : Failed to produce item: D:\UnrealProjects\BENCHMARK_4_16_1\Project\Intermediate\Build\Win64\UE4Editor\Development\UE4Editor-Project-5015.lib
2>  Total build time: 2.92 seconds (Local executor: 0.00 seconds)
2>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5): error MSB3075: The command ""C:\Program Files\Epic Games\UE_4.16\Engine\Build\BatchFiles\Build.bat" ProjectEditor Win64 Development "D:\UnrealProjects\BENCHMARK_4_16_1\Project\Project.uproject" -waitmutex" exited with code 5. Please verify that you have sufficient rights to run this command.
========== Build: 1 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

As you can see above, the 4.16.1 build took 2.92 seconds, whereas the 4.17.0 build took 13.49 seconds.

The vast majority of the build time seems to be taken by the following, which executes every time in 4.17.0, but which does not display when building in 4.16.1:

 2>  Parsing headers for ProjectEditor
 2>    Running UnrealHeaderTool "D:\UnrealProjects\Project\Project.uproject" "D:\UnrealProjects\Project\Intermediate\Build\Win64\ProjectEditor\Development\ProjectEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -installed
 2>  Reflection code generated for ProjectEditor in 6.8945103 seconds

I don’t know what’s going on internally, but it seems strange that it needs to take 6.89 seconds doing the above reflection code generation when I add a single whitespace to a cpp file. Perhaps it is superfluously regenerating reflection code for the entire project even though this might be unnecessary? Note that this reflection code generation only occurs when the editor is open (for hot reloading). Building the cpp file with the 4.17 editor closed takes a mere 3 seconds.

Also, you might have noticed that a build error (“UBT error : Failed to produce item”) occurs when building the 4.16.1 version. Please note that even though this error displays, the build and hot reload actually both succeed. This error displaying was a bug that occurred in projects updated to version 4.15 (more info here), and it stopped occurring when I updated to 4.17. I have no idea if this bug being fixed is the reason why the hot reload time increased. However, hot reloading was also fast for me in older versions like 4.13.1, which gives me the impression that this is a new bug introduced into 4.17.

In any case, is this issue (slow hot reloading) occurring with anyone else, and does anyone know of a way to increase hot reload speed? Also, what’s the deal with the ~7sec reflection code generation every time I build?

Just wanted to pitch in and say that I am experiencing the exact same problem. This is a HUGE productivity hit for no apparent reason.

Edit: I’ve opened up a related feedback thread for Epic here: https://forums.unrealengine.com/unreal-engine/feedback-for-epic/1347775-regression-in-compile-hot-reload-times-in-4-17

I’m glad to hear that I’m not the only one experiencing this problem! This is a huge productivity issue for developers who do c++ based gameplay tuning, so I hope we get a response.

By the way, just to be clear, do you happen to be using 4.17.0 with the QFE (like me), or 4.17.1?

Same here… UnrealHeaderTool runs extraneously for unchanged headers for hot loading, but not when the Editor is closed. Hot load compilation has been a mess for several months now… At 4.18p1.

Hey,

Sorry about the inconvenience here.

Does this happen in say a new blank code project in 4.17.1 with no modifications on the engine side? Or is it only with a certain project? If it’s only happening in a specific project, could we get an example project that reproduces the issue with unmodified 4.17.1? That’ll help us investigate faster.

Thanks!

I created a new C++ blank code project as well as a C++ first person project on an unmodified 4.17 engine and confirmed the same behavior in both basic projects as well as in my own project. UnrealHeaderTool runs extraneously long when building for hot reload, and lightning fast when building with the editor closed.

When IWYU was introduced, I noticed a big increase in compile/hot reload times when loading from spin hard drive. When I installed UE4 onto a solid state drive, the compile times when back down to “normal” times.

I never really did figure out why it was an issue but have just gotten used to having the space needed in my SSD.

I can confirm that I have these issues as I mentioned above, and I have everything on a Samsung 850 Pro. This particular issue is definitely not related to hard drive speeds.

Same here. I only build using an SSD and these issues started appearing when I upgraded to 4.17 (build/hot reload speed is fast in 4.16.1. I compared both versions to confirm this.)

Still no progress at 4.18p2. With the Editor and Hot Loading, UnrealHeaderTool runs extraneously on unchanged .h files again and again. What used to take 8 seconds to compile now takes 30! Close the editor, and it is back to normal behavior… change a .cpp file and only that file compiles in 8 seconds.

However there are new bugs… With the editor closed, changing an .h file triggers “System.ArgumentNullException: Value cannot be null.” in UE_4.18\Engine\Build\BatchFiles\Build.bat. Only way out of that is a clean rebuild of everything. This has been documented since 4.17:
https://forums.unrealengine.com/development-discussion/engine-source-github/124586-annoying-compile-issue-in-4-17-preview-3

Another insidious new bug in 4.18 is all the UPROPERTYs Disappearing at the Details panel after a Hotload! It is intermittent but very frequent. Only way out of that is adding or renaming a dummy UPROPERTY to trigger an update on the Details panel.

The state of support for C++ developers at Epic is quite sad. It seems, the developers are hardly testing anything any of this. It is as if, all the company cares about lately is blueprint spaghetti…

I’ve always had my entire project and engine on a very fast SSD. This issue has completely killed my iteration times - 3 sec per change to sometimes 20-26 seconds. I’ve verified the issue in a brand new blank project as well.

I can confirm this problem also.

Compile times are abnormally long on C++ projects (both on binary and source code built versions). My whole project and UE setup is on samsung evo 850 SSD, so staying on SSD instead of hard disk is not a problem.

and looks like things gotten worse on 4.18 previews, I will stay away from new versions until Epic Staff fixes this “SERIOUS” problem that impacts productivity drastically.

I hope 4.16 is good as you guys say for compile times, now downloading 4.16 version.

Edit: Just switched to 4.16, it rocks.

Thank you for finding it. Let’s all vote!

Hi all.

This may be a known issue. See Unreal Engine Issues and Bug Tracker (UE-51472) for the latest status.

Cheers

Looks like the fix was committed: https://github.com/EpicGames/UnrealEngine/commit/470136d2217917370b4c6032c4f371a37a885988

Hoping that 4.18.1 releases soon!

I was able to grab the source and rebuild just the UnrealBuildTool thanks @mpxc’s github link. Simply drop it over top of the version from the launcher and you can enjoy those speedy speedy compile times without waiting for 4.18.1.

Oh boy, finally they fixed this annoying bug. Grabbing source today too, thanks all.

Nothing is fixed. Just applied the hot fix 4.18.1 and compilation is super slow. A rebuild took 461 seconds only for an added boolean variable.

I can confirm it is fixed in 4.18.1. Compile times depend on what you changed. If you change a single cpp file and build (not rebuild) you should significantly faster times as it will only recompile only that one changed cpp file. If you change a single header file, any cpp file which includes that header file must be compiled.

I am getting single file cpp compile times around 2 seconds.

Yeah, it wasn’t a good thing to write before I test well. I still need to spend some time working to confirm everything is ok, but I made a mistake writing prematurely.