IWYU prevents build of ShaderCompileWorker

Building stops at ShaderCompileWorker with the following error:

Building ShaderCompileWorker...
Using clang version '3.9.1' (string), 3 (major), 9 (minor), 1 (patch)
Creating makefile for ShaderCompileWorker (UnrealBuildTool.exe is newer)
Using bundled libc++ standard C++ library.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/Async.cpp(1): error: Expected Async.h to be first header included.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Containers/LockFreeList.cpp(1): error: Expected LockFreeList.h to be first header included.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Containers/StackTracker.cpp(1): error: Expected StackTracker.h to be first header included.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Containers/Ticker.cpp(1): error: Expected Ticker.h to be first header included.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Containers/Union.cpp(1): error: Expected Union.h to be first header included.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Containers/Algo/FindSortedStringCaseInsensitive.cpp(1): error: Expected FindSortedStringCaseInsensitive.h to be first header included.
[...and so on for other cpp files...]

Disabling IWYU in the BuildConfiguration.cs fixes the problem. However this does not sound like a long-term acceptable solution. I’m not sure what the root cause may be, I’m new to unreal engine so if there is a better way of reporting/fixing the issue I’ll gladly learn.

Thanks.

Clang: 3.9.1

mono: 4.6.1.3

system: Linux (arch linux)

Edit:

Follow up: this is the output after a fresh checkout of the release branch.

./Setup.sh
./GenerateProjectFiles.sh
make ShaderCompileWorker
bash "/home/qwattash/git/UnrealEngine/Engine/Build/BatchFiles/Linux/Build.sh" ShaderCompileWorker Linux Development  
Building ShaderCompileWorker...
Using clang version '3.9.1' (string), 3 (major), 9 (minor), 1 (patch)
Creating makefile for ShaderCompileWorker (no existing makefile)
Building UnrealHeaderTool...
Using clang version '3.9.1' (string), 3 (major), 9 (minor), 1 (patch)
Creating makefile for UnrealHeaderTool (no existing makefile)
Using bundled libc++ standard C++ library.
Performing 5 actions (4 in parallel)
[1/5] Link libUnrealHeaderTool-Core.so
[2/5] Link libUnrealHeaderTool-Json.so
[3/5] Link libUnrealHeaderTool-Projects.so
[4/5] Link libUnrealHeaderTool-CoreUObject.so
[5/5] Link UnrealHeaderTool
Total build time: 4.04 seconds (Local executor: 2.31 seconds)
Parsing headers for ShaderCompileWorker
  Running UnrealHeaderTool ShaderCompileWorker "/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Development/ShaderCompileWorker.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -NoEnginePlugins
Using binned.
LogPlatformFile: Not using cached  wrapper
LogInit:Display: RandInit(-608031351) SRandInit(-608031351).
LogTaskGraph: Started task graph with 4 named threads and 5 total threads with 1 sets of task threads.
LogDevObjectVersion: Number of dev versions registered: 14
LogDevObjectVersion:   Dev-Blueprints (B0D832E4-1F89-4F0D-ACCF-7EB736FD4AA2): 8
LogDevObjectVersion:   Dev-Build (E1C64328-A22C-4D53-A36C-8E866417BD8C): 0
LogDevObjectVersion:   Dev-Core (375EC13C-06E4-48FB-B500-84F0262A717E): 2
LogDevObjectVersion:   Dev-Editor (E4B068ED-F494-42E9-A231-DA0B2E46BB41): 14
LogDevObjectVersion:   Dev-Framework (CFFC743F-43B0-4480-9391-14DF171D2073): 22
LogDevObjectVersion:   Dev-Mobile (B02B49B5-BB20-44E9-A304-32B752E40360): 0
LogDevObjectVersion:   Dev-Networking (A4E4105C-59A1-49B5-A7C5-40C4547EDFEE): 0
LogDevObjectVersion:   Dev-Online (39C831C9-5AE6-47DC-9A44-9C173E1C8E7C): 0
LogDevObjectVersion:   Dev-Physics (78F01B33-EBEA-4F98-B9B4-84EACCB95AA2): 0
LogDevObjectVersion:   Dev-Platform (6631380F-2D4D-43E0-8009-CF276956A95A): 0
LogDevObjectVersion:   Dev-Rendering (12F88B9F-8875-4AFC-A67C-D90C383ABD29): 12
LogDevObjectVersion:   Dev-Sequencer (7B5AE74C-D270-4C10-A958-57980B212A5A): 3
LogDevObjectVersion:   Dev-VR (D7296918-1DD6-4BDD-9DE2-64A83CC13884): 0
LogDevObjectVersion:   Dev-LoadTimes (C2A15278-BFE7-4AFE-6C17-90FF531DF755): 1
[2017.03.07-00.11.50:297][  0]LogLinux: Skipped benchmarking clocks because the engine is running in a standalone program mode - CLOCK_REALTIME will be used.
[2017.03.07-00.11.50:297][  0]LogTextLocalizationManager: No translations for ('') exist, falling back to 'en' for localization and internationalization data.
[2017.03.07-00.11.50:297][  0]LogCompile: UHT Command Line: /home/qwattash/git/UnrealEngine/Engine/Source/../Binaries/Linux/UnrealHeaderTool ShaderCompileWorker /home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Development/ShaderCompileWorker.uhtmanifest "-LogCmds=loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -NoEnginePlugins
[2017.03.07-00.11.50:298][  0]LogCompile: Loaded manifest: /home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Development/ShaderCompileWorker.uhtmanifest
[2017.03.07-00.11.50:298][  0]LogCompile: Manifest.IsGameTarget=False
[2017.03.07-00.11.50:298][  0]LogCompile: Manifest.RootLocalPath=/home/qwattash/git/UnrealEngine
[2017.03.07-00.11.50:298][  0]LogCompile: Manifest.RootBuildPath=/home/qwattash/git/UnrealEngine/
[2017.03.07-00.11.50:298][  0]LogCompile: Manifest.TargetName=ShaderCompileWorker
[2017.03.07-00.11.50:298][  0]LogCompile: Manifest.Modules=4
[2017.03.07-00.11.50:298][  0]LogCompile:   CoreUObject
[2017.03.07-00.11.50:298][  0]LogCompile:   .BaseDirectory=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/CoreUObject/
[2017.03.07-00.11.50:298][  0]LogCompile:   .IncludeBase=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/CoreUObject/
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedIncludeDirectory=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/CoreUObject/
[2017.03.07-00.11.50:298][  0]LogCompile:   .SaveExportedHeaders=True
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedCPPFilenameBase=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/CoreUObject/CoreUObject.generated
[2017.03.07-00.11.50:298][  0]LogCompile:   .ModuleType=EngineRuntime
[2017.03.07-00.11.50:298][  0]LogCompile:   InputCore
[2017.03.07-00.11.50:298][  0]LogCompile:   .BaseDirectory=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/InputCore/
[2017.03.07-00.11.50:298][  0]LogCompile:   .IncludeBase=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/InputCore/
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedIncludeDirectory=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/InputCore/
[2017.03.07-00.11.50:298][  0]LogCompile:   .SaveExportedHeaders=True
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedCPPFilenameBase=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/InputCore/InputCore.generated
[2017.03.07-00.11.50:298][  0]LogCompile:   .ModuleType=EngineRuntime
[2017.03.07-00.11.50:298][  0]LogCompile:   SlateCore
[2017.03.07-00.11.50:298][  0]LogCompile:   .BaseDirectory=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/SlateCore/
[2017.03.07-00.11.50:298][  0]LogCompile:   .IncludeBase=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/SlateCore/
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedIncludeDirectory=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/SlateCore/
[2017.03.07-00.11.50:298][  0]LogCompile:   .SaveExportedHeaders=True
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedCPPFilenameBase=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/SlateCore/SlateCore.generated
[2017.03.07-00.11.50:298][  0]LogCompile:   .ModuleType=EngineRuntime
[2017.03.07-00.11.50:298][  0]LogCompile:   Slate
[2017.03.07-00.11.50:298][  0]LogCompile:   .BaseDirectory=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Slate/
[2017.03.07-00.11.50:298][  0]LogCompile:   .IncludeBase=/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Slate/
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedIncludeDirectory=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/Slate/
[2017.03.07-00.11.50:298][  0]LogCompile:   .SaveExportedHeaders=True
[2017.03.07-00.11.50:298][  0]LogCompile:   .GeneratedCPPFilenameBase=/home/qwattash/git/UnrealEngine/Engine/Intermediate/Build/Linux/B4D820EA/ShaderCompileWorker/Inc/Slate/Slate.generated
[2017.03.07-00.11.50:298][  0]LogCompile:   .ModuleType=EngineRuntime
[2017.03.07-00.11.50:301][  0]LogCompile: Preparsed module /Script/CoreUObject containing 3 files(s) in 0.00 secs.
[2017.03.07-00.11.50:303][  0]LogCompile: Preparsed module /Script/InputCore containing 1 files(s) in 0.00 secs.
[2017.03.07-00.11.50:317][  0]LogCompile: Preparsed module /Script/SlateCore containing 19 files(s) in 0.01 secs.
[2017.03.07-00.11.50:323][  0]LogCompile: Preparsed module /Script/Slate containing 18 files(s) in 0.01 secs.
[2017.03.07-00.11.50:335][  0]LogCompile: Exporting native class declarations for /Script/CoreUObject
[2017.03.07-00.11.50:341][  0]LogCompile: Autogenerating boilerplate cpp: CoreUObject.generated.cpp
[2017.03.07-00.11.50:344][  0]LogCompile: Success: Parsed 1436 line(s), 85 statement(s) in 0.01 secs.
[2017.03.07-00.11.50:345][  0]LogCompile: Exporting native class declarations for /Script/InputCore
[2017.03.07-00.11.50:346][  0]LogCompile: Autogenerating boilerplate cpp: InputCore.generated.cpp
[2017.03.07-00.11.50:346][  0]LogCompile: Success: Parsed 571 line(s), 21 statement(s) in 0.00 secs.
[2017.03.07-00.11.50:373][  0]LogCompile: Exporting native class declarations for /Script/SlateCore
[2017.03.07-00.11.50:386][  0]LogCompile: Autogenerating boilerplate cpp: SlateCore.generated.cpp
[2017.03.07-00.11.50:392][  0]LogCompile: Success: Parsed 6342 line(s), 324 statement(s) in 0.02 secs.
[2017.03.07-00.11.50:398][  0]LogCompile: Exporting native class declarations for /Script/Slate
[2017.03.07-00.11.50:403][  0]LogCompile: Autogenerating boilerplate cpp: Slate.generated.cpp
[2017.03.07-00.11.50:405][  0]LogCompile: Success: Parsed 1846 line(s), 219 statement(s) in 0.01 secs.
[2017.03.07-00.11.50:405][  0]LogCompile: Preparsing 4 modules took 0.02 seconds
[2017.03.07-00.11.50:405][  0]LogCompile: Parsing took 0.05 seconds
[2017.03.07-00.11.50:405][  0]LogCompile: Code generation took 0.04 seconds
[2017.03.07-00.11.50:405][  0]LogCompile: ScriptPlugin overhead was 0.00 seconds
[2017.03.07-00.11.50:405][  0]LogCompile: Macroize time was 0.00 seconds
[2017.03.07-00.11.50:406][  0]LogObj: Freed 0b from 0 cluster array pools.
Reflection code generated for ShaderCompileWorker in 0.2827769 seconds
Using bundled libc++ standard C++ library.
/home/qwattash/git/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/Async.cpp(1): error: Expected Async.h to be first header included.

Hey qwattash-

I was able to install the engine and run the make ShaderCompileWorker step without issue. Just to verify, are you using the steps shown on the wiki here: A new, community-hosted Unreal Engine Wiki - Announcements - Epic Developer Community Forums ? What was the command used when you received this error?

Thanks for the answer, I think I followed the steps correctly, I added the full output after cleaning the build but the result does not change. I’m wondering what other things can affect the process, I can add all the versions for the dependencies if you think it may help.

I tried to get the expected header name from the build tool where it triggers the error. It seems that is expecting the right header but finding the incorrect one: Expected: Async, Found CorePrivatePCH

Can you open the Async.cpp file (/Engine/Source/Runtime/Core/Private/Async/Async.cpp) and let me know what the first listed include is? If it is not #include "Async/Async.h", please try adding this and then compile the engine again.

The first include file is the one you specified. The same seems to be for the other failing cpp files. I’m not sure how CorePrivatePCH.h ends up in there, possibly through the UnrealHeaderTool generated files?

I tried again locally and was able to use the command git clone https://github.com/EpicGames/UnrealEngine -b release [folder] to clone the release branch (currently 4.15.0) to my machine. I then ran Setup.sh followed by GenerateProjectFiles.sh from the folder. When those finished I was able to run/complete make ShaderCompileWorker. I did notice in my latest attempt that I am currently using Clang 3.8.0 as well as the line Using bundled libc++ standard C++ library. not being present in my terminal window when I start the ShaderCompileWorker step. Please let me know if changing your Clang version has any affect. If you’re still unable to build ShaderCompileWorker after following these steps, please provide as much information about your current setup as possible. I am currently using a vanilla install of Ubuntu 16.04 locally. Please indicate what your environment distro is and any other information you can provide.

It seems that the problem was related to the caching of some value by the build system. While trying to debug the ploblem I cleaned up the build directory multiple times to the point it started working. Now I am not able to reproduce the issue again. I guess that I wasn’t familiar enough with how the UBT works, thanks for the help!