x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

ShaderCompileWorker expects different HLSL header format

In master and 4.9 branches, when packed uniforms are not used (or not supported, I'm not sure), GlslBackend in ShaderFormatOpenglModule generates at least '@Samplers' header line in different format than ShaderCompileWorker expects.

Which leads to error "Bad hlslcc header found! Missing '#'!" in ShaderCompileWorker, since parsing stops at '// @Samplers: t;-1:ps0' line (ShaderCompileWorker expects something like '// @Samplers: t(-1:ps0)').

I tried to fix it by changing the format in print_extern_vars function in GlslBackend.cpp and parser in ShaderCompileWorker, but it seems parser requires more information than that's generated by the backend and the fix will require some deeper changes in backend, which are outside of my knowledge.

In branch 4.8 there's similar issue, but the error generated is 'Assertion failed: Assertion failed: Match(ShaderSource, '(') in OpenGLShaderCompiler.cpp'

Product Version: UE 4.8
Tags:
more ▼

asked Jul 25 '15 at 08:39 AM in Bug Reports

avatar image

aukkras
51 1 4 4

avatar image burockcracy Jul 25 '15 at 07:27 PM

I'm running into the same issue. But I'm finding with the change to print_extern_vars, the -1 then doesn't parse as ParseNumber is called which assumes positive numbers only.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

2 answers: sort voted first

Thank you very much! These finding turned out useful.

Could you attempt to cherry pick those two commits to see if they fix the crash for you?

https://github.com/EpicGames/UnrealEngine/commit/0b5c4df5dc33ca87e1bb20d72a601328bbf174be

https://github.com/EpicGames/UnrealEngine/commit/7fc32b2afeebb5915a12af10576860c9536d27d8

more ▼

answered Jul 30 '15 at 06:26 PM

avatar image

RCL STAFF
2.7k 59 6 88

avatar image aukkras Jul 30 '15 at 09:03 PM

I wasn't able to regenerate intermediate shaders debug info due to crash in SCW (attached backtrace).

I ran CCT with previous command line, and previous preprocessed shader.

Now CCT doesn't crash and displays 'bad hlslcc header' error.

Attaching generated GLSL output

backtrace.txt (3.0 kB)
cct_output.txt (5.6 kB)
output2.txt (1.1 kB)
avatar image aukkras Jul 31 '15 at 07:16 AM

To fix SCW crash I applied attached patch.

Now I was able to generate ShaderDebugInfo cache.

Attaching preprocessed USF and generated GLSL files.

The CCT command line is now:

./CrossCompilerTool ../../Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/ScreenSpaceReflections.usf -o=../../Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/Output.glsl -ps -entry=ScreenSpaceReflectionsPS -gl3 -dx11clip -nopp -separateshaders

Attaching also CCT output.

avatar image burockcracy Jul 31 '15 at 12:29 PM

I'm getting basically the same, "Bad hlslcc header found! Missing '#'!".

If I set

r.ShaderDevelopmentMode=1 r.DumpShaderDebugInfo=1 r.DumpShaderDebugShortNames=1

I get only one shader ../../Saved/ShaderDebugInfo/GLSL_150/Global/SlateElementPSLineSegment_1_1/SlateElementPixelShader.usf. When I run CCT (no .bat file I can see, so using options similar to ones mceier gives) ./CrossCompilerTool ../../Saved/ShaderDebugInfo/GLSL_150/Global/SlateElementPSLineSegment_1_1/SlateElementPixelShader.usf -o=../../Saved/ShaderDebugInfo/GLSL_150/Global/SlateElementPSLineSegment_1_1/Output.glsl -ps -entry=Main -gl4 -dx11clip -nopp

output.glsl attached (basically same as mceier gets)

link text

output.glsl.txt (3.4 kB)
avatar image aukkras Jul 31 '15 at 12:50 PM

Did you apply the patch from my previous comment ?

I think other shaders and .bat files weren't generated on your system because SCW (on master branch) crashes without that patch on linux.

avatar image burockcracy Jul 31 '15 at 01:27 PM

Have now applied the change. Got the .bat files, and all the shaders and almost same outputs as you (differences seem insignificant). link text

Will add that the generated glsl files seem to have a null byte character at the end - found that when I was doing a diff c.f. output from CCT run manually (diff thinks the file is binary). But does not seem to affect the primary issue.

avatar image aukkras Aug 03 '15 at 05:24 PM

It looks like I had stale libhlslcc.a file - I thought that doing "make clean; make" in Engine/Source/ThirdParty/hlslcc/hlslcc/projects/Linux is sufficient to rebuild libhlslcc.a and link with this new version.

But after rebuilding libhlslcc.a it has to be copied from Engine/Source/ThirdParty/hlslcc/hlslcc/lib/Linux to Engine/Source/ThirdParty/hlslcc/hlslcc/lib/Linux/x86_64-unknown-linux-gnu/

When I did this, current version of CCT didn't show "Bad hlslcc header found! Missing '#'!" error and UE4Editor was able to generate all GLSL files.

So this bug report can be marked as resolved.

avatar image burockcracy Aug 05 '15 at 01:09 PM

Finally got it to build and run, hit some other issues apart from CCT, seem to have been solved with later commits. However, I'm still finding I have to manually make and copy libhlslcc.a after make DebugSet. Might just be I'm not familiar enough with the build tools.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

Are your SCW and the engine using the same hlslcc version? (that is, have you [re]built SCW after pulling latest from master?). Also, has DDC been generated by the same version? You can remove the whole Engine/DerivedDataCache and /DerivedDataCache directories.

Otherwise, please add more exact repro steps, preferably starting from a clean clone of the repo to rule out previous state.

more ▼

answered Jul 25 '15 at 02:24 PM

avatar image

RCL STAFF
2.7k 59 6 88

avatar image aukkras Jul 25 '15 at 02:58 PM
avatar image RCL STAFF Jul 25 '15 at 09:59 PM

This should have nothing to do with your drivers, since the code is being generated by hlslcc. A few more questions: why do you preload libGL and what happens if you don't? Do you have a conflicting libmcpp installed in your system that might be getting loaded before one from Engine/Binaries/Linux? (Changing LD_LIBRARY_PATH may help with that).

You can disable compiling through SCW and just use the engine itself. Set bAllowCompilingThroughWorkers to false and see whether you can get further.

avatar image aukkras Jul 25 '15 at 10:50 PM

Without LD_PRELOAD UE4Editor is unable to load libGL.so.1 due to too many TLS objects:

'Unable to dynamically load libGL: Failed loading libGL.so.1: dlopen: cannot load any more object with static TLS'

I don't have libmcpp installed in the system.

With bAllowCompilingThroughWorkers=False the result is the same.

avatar image burockcracy Jul 26 '15 at 01:34 AM

Had the same issue with glibc, and increased it to 32 i.e. glibc patch +#define DTV_SURPLUS (32). That was as a result of googling. LD_PRELOAD also worked. But without either of those would fail when worker threads were started. But makes no difference to current problem.

avatar image aukkras Jul 29 '15 at 09:25 AM

Attaching ShaderCompileWorker input file

When I run (or when UE4Editor runs):

./ShaderCompileWorker ../../Intermediate/Shaders/WorkingDirectory/1565/5/ 1565 5 WorkerInputOnly.in WorkerOutputOnly.out -communicatethroughfile -Multiprocess

I get 'Bad hlslcc header found! Missing '#'!' error.

Attaching generated shader source from GlslBackend.cpp - syntax error is in line 4 from ';' character

I hope this will make clear where the problem is and allow you to debug it or at least confirm it.

WorkerInputOnly.txt should be placed in Engine/Intermediate/Shaders/WorkingDirectory/1565/5/WorkerInputOnly.in and the command should be run from Engine/Binaries/Linux

avatar image RCL STAFF Jul 29 '15 at 11:17 PM

That's very helpful - thank you very much! Could you please identify the original .usf file?

In ConsoleVariables.ini, please set

 r.ShaderDevelopmentMode=1
 r.DumpShaderDebugInfo=1
 r.DumpShaderDebugShortNames=1

This will make a dir under YourProject/Saved/ShaderDebugInfo/[shaderplatform]/Material/VertexFactory/Shader.usf or YourProject/Saved/ShaderDebugInfo/[shaderplatform]/Global/Shader.usf and place .glsl files there; there will be also Windows .bat files that have the info how to invoke CrossCompilerTool (feel free to modify code to emit .sh files as well).

You can compile the CCT (make CrossCompilerTool) and run it with the parameters from the bat file to repro the problem outside of the engine. We would be interested to get this repro case (i.e. .usf and CCT invocation that results in the output like above).

avatar image aukkras Jul 30 '15 at 05:41 AM

The CCT command line is:

./CrossCompilerTool ../../Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/ScreenSpaceReflections.usf -o=../../Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/Output.glsl -ps -entry=ScreenSpaceReflectionsPS -gl3 -separateshaders

Shader is ScreenSpaceReflections.usf (attached).

When I invoke this command manually it crashes with attached output

Attaching also preprocessed shader file which should be placed in Engine/Saved/ShaderDebugInfo/GLSL_150/Global/PostProcessScreenSpaceReflectionsPS00/ScreenSpaceReflections.usf

Attaching also GLSL output from CrossCompilerTool and UE4Editor for the same shader.

avatar image burockcracy Aug 03 '15 at 02:11 PM

You've probably already done this but setting state->has_packed_uniforms=true in the debugger stops the bad header errors. So now, why is it false to begin with....

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question