InputTool commandline args

Hey so I have been doing some refactoring of our 2yr old codebase to boost our compile times, had some great increases by switching over from using our PCH setup to using the iwyu style.
our MyProject.Build.cs file now has

MinFilesUsingPrecompiledHeaderOverride = 1;
bFasterWithoutUnity = true;
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
bEnforceIWYU = true;
PrivatePCHHeaderFile = "MyProject.h";

Which works very well, our pch file now just contains,

#include "CoreMinimal.h"
#include "ModuleManager.h"

class FMyProjectModule
    : public IModuleInterface
{
public:
    virtual void StartupModule() override;
    virtual void ShutdownModule() override;
    virtual bool IsGameModule() const override;
};

Which just has some of the startup stuff overridden to add a custom shader directory for our pak file to include correctly. The last thing I wanted to get running was the IncludeTool.exe (IncludeTool | Unreal Engine Documentation) to parse our project for additional dependencies that I could then remove from our compile times.
But atm when running the tool with the options

-Mode=Optimize -Target=MyProjectEditor -Platform=Linux -Configuration=Development -WorkingDir=D:/MyProject/ -OutputDir=D:\Output -SourceFiles=-/Engine/... -

I get an exception thrown in the Program.cs Line 867 which is trying to run the UnrealBuildTool over the project directory which is converted to

if (Utility.Run(UnrealBuildTool, String.Format("-Mode=JsonExport {0} {1} {2}{3} -disableunity -xgeexport -nobuilduht -nopch -nodebuginfo -outputfile=\"{4}\"", Target, Configuration, Platform, Precompile? " -precompile" : "", TaskListFile.ChangeExtension(".json").FullName), WorkingDir, Log) != 0)

Where

UnrealBuildTool = {D:\UnrealEngine\UnrealEngine-release\Engine\Binaries\DotNET\UnrealBuildTool.exe}
Target="MyProjectEditor"
Configuration=Development
Platform=Linux
Precompile=false
TaskListFile =  { CanonicalName = "d:\\MyProject\\targets\\MyProjectEditor linux development.xml"; Directory = {D:\MyProject\Targets}; FullName = "D:\\MyProject\\Targets\\MyProjectEditor Linux Development.xml"; }
WorkingDir = {D:\UnrealEngine\UnrealEngine-release\Engine}

And I don’t know enough about this tool to know if I need to possibly change its working directory (like the visual studio option for the working directory) or if I need to pass a different param, the comments for the arguments are very ‘succinct’ but unfortunately as vague as a programmer can get

/// <summary>
/// Working directory for optimization data and reports
/// </summary>
[CommandLineOption(IsRequired = true)]
public string WorkingDir;

Any help about the environment setup for running this tool over a non-engine folder or setup that anyone else would know any information about would be appreciated, atm I am enjoying the fact that my incremental compile times have already dropped from 85secs to 25secs, which is massive for development, especially without needing to relaunch the editor with the 4.22 live coding (experimental) additions. It has already made development so much less of a headache to change source files, but I am always interested in knowing what our dependency chains in the project currently are and where I might be able to improve our architecture to possibly get this down as close to a 10sec compile as I can. Thanks in advance for any advice.

Ok so doing this in a 4.22 project so its a Unreal’s clang version and build tools have updated slightly, but basically got something running by copying MyProject.Target.cs into the Engine/Source folder, placed my projects Source folder into Engine/Source/Programs/MyProject so that the MyProject.Build.cs was located below that folder as it would have been in my Source folder.

Altered MyProject.Target.cs to have

using UnrealBuildTool;
using System.Collections.Generic;

public class MyProjectTarget : TargetRules
{
    public MyProjectTarget(TargetInfo Target) : base(Target)
	{
		Type = TargetType.Game;
		BuildEnvironment = TargetBuildEnvironment.Shared;
		bBuildAllModules = true;

		ExtraModuleNames.AddRange( new string[] { "MyProject" } );
	}
}

Created two folders to capture my output and input in,

D:/UnrealEngine/Output/

D:/UnrealEngine/Input/

Then I was able to run the tool using the arguments

-Mode=Optimize -Target=MyProject -Platform=Linux -Configuration=Development -WorkingDir=D:\UnrealEngine\Working -OutputDir=D:\UnrealEngine\Output -InputDir=D:\UnrealEngine\UnrealEngine-release -SourceFiles=@D:\UnrealEngine\Working\Filter.txt -SkipCycleCheck -SkipDiagnostics -CleanTaskList

Then I had to step through some catch-22’s with the tool, so that the following lines were updated.

CompileEnvironment.cs
Line 317: Environment.ForceIncludeFiles.Add(new FileReference(Tokens[Idx].Substring(3)));
Line 346: if(!SourceFile.HasExtension(".a") && !SourceFile.HasExtension(".exe"))
Line 372: else if(FileName == "clang++.exe" || FileName == "clang++")

My Input folder has a Filter.txt item in it

\Engine\Source\Programs\GoblinsOfElderstone\...
-\Engine\Source\Developer\...
-\Engine\Source\Editor\...
-\Engine\Source\Runtime\...
-\Engine\Source\ThirdParty\...

And now I’m stuck again, my project finally gets as far as having a valid TaskList, cross references it successfully with my filter list ( had to use the cmd args -SkipCycleCheck -SkipDiagnostics for pieces of code here, that failed and I didn’t think I needed… ) and I finally get to the big boy, bool bResult = ProcessSequenceTree and some probes get successfully made and then…

Creating probes...
[00:00:00] Creating probes... (1/1220; 0%)
[00:00:08] Creating probes... (212/1220; 17%)
[00:00:14] Creating probes... (390/1220; 31%)
[00:00:23] Creating probes... (920/1220; 75%)
[00:00:29] Creating probes... (1190/1220; 97%)
[00:00:29] Creating probes... (1220/1220; 100%)
[00:00:29] Optimize <                                        > 3/1220 probes, 12 active, 3/622784 fragments (0%)
Failed to compile Platform.3of7.h, exit code -1:
    > Compiling attempt 0...
    > Compiler=C:\UnrealToolchains\v13_clang-7.0.1-centos7\x86_64-unknown-linux-gnu\bin\clang++
    > Dependencies=(2, 6, 7)
    > Permutation=(0, 1, 2, 3, 4, 5, 6, 7)
    > PotentialDepdendencyCounts=(0, 1, 2, 4, 5, 6)
    > MinRequired=0, Pivot=5, MaxRequired=5
    > In file included from D:/UnrealEngine/Working/Intermediate/Platform.3of7.h.permutation:9:
    > D:\UnrealEngine\Working\Fragments\Platform.1of7.h(142,7):  err: 'PLATFORM_COMPILER_CLANG' is not defined, evaluates to 0 [-Werror,-Wundef]
    > #elif PLATFORM_COMPILER_CLANG
    >       ^
    > D:\UnrealEngine\Working\Fragments\Platform.1of7.h(676,5):  err: 'PLATFORM_COMPILER_HAS_TCHAR_WMAIN' is not defined, evaluates to 0 [-Werror,-Wundef]
    > #if PLATFORM_COMPILER_HAS_TCHAR_WMAIN
    >     ^
    > D:\UnrealEngine\Working\Fragments\Platform.1of7.h(899,6):  err: 'PLATFORM_TCHAR_IS_CHAR16' is not defined, evaluates to 0 [-Werror,-Wundef]
    >         #if PLATFORM_TCHAR_IS_CHAR16
    >             ^
    > In file included from D:/UnrealEngine/Working/Intermediate/Platform.3of7.h.permutation:17:
    > D:\UnrealEngine\Working\Fragments\Platform.2of7.h(142,7):  err: 'PLATFORM_COMPILER_CLANG' is not defined, evaluates to 0 [-Werror,-Wundef]
    > #elif PLATFORM_COMPILER_CLANG
    >       ^
    > D:\UnrealEngine\Working\Fragments\Platform.2of7.h(676,5):  err: 'PLATFORM_COMPILER_HAS_TCHAR_WMAIN' is not defined, evaluates to 0 [-Werror,-Wundef]
    > #if PLATFORM_COMPILER_HAS_TCHAR_WMAIN
    >     ^
    > D:\UnrealEngine\Working\Fragments\Platform.2of7.h(899,6):  err: 'PLATFORM_TCHAR_IS_CHAR16' is not defined, evaluates to 0 [-Werror,-Wundef]
    >         #if PLATFORM_TCHAR_IS_CHAR16
    >             ^
    > In file included from D:/UnrealEngine/Working/Intermediate/Platform.3of7.h.permutation:20:
    > D:\UnrealEngine\Working\Fragments\Platform.3of7.h(142,7):  err: 'PLATFORM_COMPILER_CLANG' is not defined, evaluates to 0 [-Werror,-Wundef]
    > #elif PLATFORM_COMPILER_CLANG
    >       ^

A whole bunch of clang errors that I got no fking idea how to fix :confused: