Building binary editor from source for artists

Development Platform & Builds
, , ,
1

Hi,

I want to provide our artists and designers with a periodically updated, instantly usable binary version of our custom build of the UE4.11 editor and engine. The only source code that they need to build regularly is our game code.

I also want to minimize the amount of engine code and data that we have to retain in our local Perforce depot while providing hooks into our Jenkins CI server for automated testing.

The first approach I tried was to build the editor from a pared down subset of the Epic Perforce depot biased towards Win64. This weighs in at around 6GB initially (dominated by third party libraries), growing to 40GB after building the Development and Development Editor solution configurations to provide a full complement of binaries.

The build time was around 15 minutes when using distribution and the results, excluding intermediates and debug information, can be compressed to a very portable 1.5GB. This looked promising.

Unfortunately, after running UnrealVersionSelector and opening a 4.10 project from the editor (for conversion to 4.11), UnrealBuildTool instigates a time consuming engine/editor build, ultimately failing to write UE4Editor-Core.dll (which is in use by the running UE4Editor process). The user experience when distribution is enabled is a progress bar that sits at 100% for the majority of the build, leading to confusion/frustration. Invoking UE4Editor with -rocket speeds the process up, but it ultimately fails in the same manner.

I amended the build phase above with -precompile in the hope that statically linked libraries would save the day, but the new binary was identical in size and behavior/failure to the original, suggesting either that I’m doing something wrong, this option has no effect, or it’s not being propagated through the build batch file and UnrealBuildTool.

The runuat gubp -Node=GatherRocket approach fared better but required many more files to be included in the depot (e.g. templates, starter content, and IOS/HTML5 materials) and didn’t take advantage of build distribution, taking around 2 hours to complete, albeit in both regular and DDC flavours.

The resulting editor binary worked as required, converting and opening the legacy project without an engine rebuild, but the lengthy build process is too much of a burden for our programmers when they’re trying to incrementally test and distribute updates to the wider team.

Having already spent quite a lot of time on this and conscious that others have asked similar questions with limited success, I’m hoping that somebody at Epic can provide some assistance.

Key questions include:

  • “is it possible to avoid the engine rebuilds in my first scenario?”
  • “is it possible to dramatically reduce the footprint and overhead of GUBP?”
  • and/or “are there any practical alternatives for providing artists and designers with bespoke editor binaries?”

Best regards,

Scott.

2

Using GUBP is probably our preferred workflow in this case as there are so many caveats when attempting to prevent Engine components from being re-compiled. The -rocket flag is being phased out and in 4.11 you’d have to create a file called Rocket.txt in the Engine/Build folder of version you want to distribute. The -rocket flag used to be passed between the various different .exes that are called when using the editor but it’s much simpler/safer to have a single centralised thing to check. By 4.12 -rocket should hopefully not exist and it will be checking for Engine/Build/InstalledBuild.txt instead, which is a more descriptive name for the file.

You’ll also need to add some .ini file entries to describe the platforms that you’ve pre-compiled with an installed build, something we do during our rocket build process as of 4.11. If you’re only interested in Windows then you could probably get away with adding the following to Engine/Config/BaseEngine.ini:

[InstalledPlatforms]
+InstalledPlatformConfigurations=(PlatformName="Win64", Configuration="DebugGame")
+InstalledPlatformConfigurations=(PlatformName="Win64", Configuration="Development")
+InstalledPlatformConfigurations=(PlatformName="Win64", Configuration="Shipping")
+InstalledPlatformConfigurations=(PlatformName="Win32", Configuration="DebugGame")
+InstalledPlatformConfigurations=(PlatformName="Win32", Configuration="Development")
+InstalledPlatformConfigurations=(PlatformName="Win32", Configuration="Shipping")

with the configurations matching those that you’ve compiled.

I don’t know whether it’s possible to improve the performance of GUBP much, we use it more with our multi machine build system so that different groups of nodes are executed on different machines. It will probably be the simplest method in the long run though as it will take care of creating the right .txt files and the .ini file entries. Not sure where you got the GUBP info from but assume you’re adding -TargetPlatforms=Win32+Win64 to only build those targets? You might also want to add -NoDDC if you don’t need to build DDC for the engine content. I’m afraid the rest of the assets that are needed by GUBP are determined by the nodes that the GatherRocket node depends on, which is centered around making the build that we release from the Epic Launcher. You could possibly look into modifying RocketBuild.Automation.cs to better fit your needs - you’ll see in FilterRocketNode that a dependency on the MakeFeaturePacksNode is added, which you probably don’t need.

Let us know if you have any more issues with either approach and I’ll see if we can help more.

3

Hi Matthew,

Thanks for the info. Adding rocket.txt file and the “InstalledPlatformConfigurations” lines successfully produced an install-like UE4Editor. I’m now rolling this out to our artists.

With subsequent releases of the engine, it would be really handy if somebody from Epic could call out changes and improvements relating to build process in parallel with fleshing out documentation in this area.

Best regards,

Scott.