ShaderCompileWorker expects different HLSL header format

In master and 4.9 branches, when packed uniforms are not used (or not supported, I’m not sure), GlslBackend in ShaderFormatOpenglModule generates at least ‘@Samplers’ header line in different format than ShaderCompileWorker expects.

Which leads to error “Bad hlslcc header found! Missing ‘#’!” in ShaderCompileWorker, since parsing stops at ‘// @Samplers: t;-1:ps0’ line (ShaderCompileWorker expects something like ‘// @Samplers: t(-1:ps0)’).

I tried to fix it by changing the format in print_extern_vars function in GlslBackend.cpp and parser in ShaderCompileWorker, but it seems parser requires more information than that’s generated by the backend and the fix will require some deeper changes in backend, which are outside of my knowledge.

In branch 4.8 there’s similar issue, but the error generated is ‘Assertion failed: Assertion failed: Match(ShaderSource, ‘(’) in OpenGLShaderCompiler.cpp’

Are your SCW and the engine using the same hlslcc version? (that is, have you [re]built SCW after pulling latest from master?). Also, has DDC been generated by the same version? You can remove the whole Engine/DerivedDataCache and /DerivedDataCache directories.

Otherwise, please add more exact repro steps, preferably starting from a clean clone of the repo to rule out previous state.

Yes, they use the same hlslcc version.

I’m building UE with ‘make’ - this builds editor and SCW.

/DerivedDataCache doesn’t exist, and /Engine/DerivedDataCache is empty.

To reproduce I run:

‘LD_PRELOAD=/usr/lib64/libGL.so.1 ./UE4Editor’

Here:
https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Source/Developer/ShaderFormatOpenGL/Private/GlslBackend.cpp#L2674

has_packed_uniforms is false on my PC (probably GTX 760m doesn’t support this or maybe shader doesn’t use it… dunno), so backend generates @Samplers line in different format than SCW expects:

https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Source/Developer/ShaderFormatOpenGL/Private/GlslBackend.cpp#L2712

https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Source/Developer/ShaderFormatOpenGL/Private/GlslBackend.cpp#L2630

SCW doesn’t expect ‘;’ on that line - instead it expects ‘(’:

https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Source/Developer/ShaderCompilerCommon/Private/ShaderCompilerCommon.cpp#L777

Which leads to these errors:

https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Source/Developer/ShaderFormatOpenGL/Private/OpenGLShaderCompiler.cpp#L525

https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Source/Developer/ShaderFormatOpenGL/Private/OpenGLShaderCompiler.cpp#L530

I’m running into the same issue. But I’m finding with the change to print_extern_vars, the -1 then doesn’t parse as ParseNumber is called which assumes positive numbers only.

This should have nothing to do with your drivers, since the code is being generated by hlslcc. A few more questions: why do you preload libGL and what happens if you don’t? Do you have a conflicting libmcpp installed in your system that might be getting loaded before one from Engine/Binaries/Linux? (Changing LD_LIBRARY_PATH may help with that).

You can disable compiling through SCW and just use the engine itself. Set bAllowCompilingThroughWorkers to false and see whether you can get further.

Without LD_PRELOAD UE4Editor is unable to load libGL.so.1 due to too many TLS objects:

‘Unable to dynamically load libGL: Failed loading libGL.so.1: dlopen: cannot load any more object with static TLS’

I don’t have libmcpp installed in the system.

With bAllowCompilingThroughWorkers=False the result is the same.

Had the same issue with glibc, and increased it to 32 i.e. glibc patch +#define DTV_SURPLUS (32). That was as a result of googling. LD_PRELOAD also worked. But without either of those would fail when worker threads were started. But makes no difference to current problem.

Attaching ShaderCompileWorker input file

When I run (or when UE4Editor runs):

./ShaderCompileWorker …/…/Intermediate/Shaders/WorkingDirectory/1565/5/ 1565 5 WorkerInputOnly.in WorkerOutputOnly.out -communicatethroughfile -Multiprocess

I get ‘Bad hlslcc header found! Missing ‘#’!’ error.

Attaching generated shader source from GlslBackend.cpp - syntax error is in line 4 from ‘;’ character

I hope this will make clear where the problem is and allow you to debug it or at least confirm it.

WorkerInputOnly.txt should be placed in Engine/Intermediate/Shaders/WorkingDirectory/1565/5/WorkerInputOnly.in
and the command should be run from Engine/Binaries/Linux

That’s very helpful - thank you very much! Could you please identify the original .usf file?

In ConsoleVariables.ini, please set

r.ShaderDevelopmentMode=1
r.DumpShaderDebugInfo=1
r.DumpShaderDebugShortNames=1

This will make a dir under YourProject/Saved/ShaderDebugInfo/[shaderplatform]/Material/VertexFactory/Shader.usf or YourProject/Saved/ShaderDebugInfo/[shaderplatform]/Global/Shader.usf and place .glsl files there; there will be also Windows .bat files that have the info how to invoke CrossCompilerTool (feel free to modify code to emit .sh files as well).

You can compile the CCT (make CrossCompilerTool) and run it with the parameters from the bat file to repro the problem outside of the engine. We would be interested to get this repro case (i.e. .usf and CCT invocation that results in the output like above).

The CCT command line is:

./CrossCompilerTool …/…/Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/ScreenSpaceReflections.usf -o=…/…/Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/Output.glsl -ps -entry=ScreenSpaceReflectionsPS -gl3 -separateshaders

Shader is ScreenSpaceReflections.usf (attached).

When I invoke this command manually it crashes with attached output

Attaching also preprocessed shader file which should be placed in Engine/Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/ScreenSpaceReflections.usf

Attaching also GLSL output from CrossCompilerTool and UE4Editor for the same shader.

Thank you very much! These finding turned out useful.

Could you attempt to cherry pick those two commits to see if they fix the crash for you?

https://github.com/EpicGames/UnrealEngine/commit/0b5c4df5dc33ca87e1bb20d72a601328bbf174be

https://github.com/EpicGames/UnrealEngine/commit/7fc32b2afeebb5915a12af10576860c9536d27d8

I wasn’t able to regenerate intermediate shaders debug info due to crash in SCW (attached backtrace).

I ran CCT with previous command line, and previous preprocessed shader.

Now CCT doesn’t crash and displays ‘bad hlslcc header’ error.

Attaching generated GLSL output

To fix SCW crash I applied attached patch.

Now I was able to generate ShaderDebugInfo cache.

Attaching preprocessed USF and generated GLSL files.

The CCT command line is now:

./CrossCompilerTool …/…/Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/ScreenSpaceReflections.usf -o=…/…/Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/Output.glsl -ps -entry=ScreenSpaceReflectionsPS -gl3 -dx11clip -nopp -separateshaders

Attaching also CCT output.

I’m getting basically the same, “Bad hlslcc header found! Missing ‘#’!”.

If I set

r.ShaderDevelopmentMode=1
r.DumpShaderDebugInfo=1
r.DumpShaderDebugShortNames=1

I get only one shader …/…/Saved/ShaderDebugInfo/GLSL_150/Global/SlateElementPSLineSegment_1_1/SlateElementPixelShader.usf. When I run CCT (no .bat file I can see, so using options similar to ones mceier gives)
./CrossCompilerTool …/…/Saved/ShaderDebugInfo/GLSL_150/Global/SlateElementPSLineSegment_1_1/SlateElementPixelShader.usf -o=…/…/Saved/ShaderDebugInfo/GLSL_150/Global/SlateElementPSLineSegment_1_1/Output.glsl -ps -entry=Main -gl4 -dx11clip -nopp

output.glsl attached (basically same as mceier gets)

link text

Did you apply the patch from my previous comment ?

I think other shaders and .bat files weren’t generated on your system because SCW (on master branch) crashes without that patch on linux.

Have now applied the change. Got the .bat files, and all the shaders and almost same outputs as you (differences seem insignificant).
link text

Will add that the generated glsl files seem to have a null byte character at the end - found that when I was doing a diff c.f. output from CCT run manually (diff thinks the file is binary). But does not seem to affect the primary issue.

You’ve probably already done this but setting state->has_packed_uniforms=true in the debugger stops the bad header errors. So now, why is it false to begin with…

It looks like I had stale libhlslcc.a file - I thought that doing “make clean; make” in Engine/Source/ThirdParty/hlslcc/hlslcc/projects/Linux is sufficient to rebuild libhlslcc.a and link with this new version.

But after rebuilding libhlslcc.a it has to be copied from Engine/Source/ThirdParty/hlslcc/hlslcc/lib/Linux to Engine/Source/ThirdParty/hlslcc/hlslcc/lib/Linux/x86_64-unknown-linux-gnu/

When I did this, current version of CCT didn’t show “Bad hlslcc header found! Missing ‘#’!” error and UE4Editor was able to generate all GLSL files.

So this bug report can be marked as resolved.

Finally got it to build and run, hit some other issues apart from CCT, seem to have been solved with later commits. However, I’m still finding I have to manually make and copy libhlslcc.a after make DebugSet. Might just be I’m not familiar enough with the build tools.