Shaders rebuilding every time I start unreal

For some reason, every time I open Unreal after a day of not using it, all my shaders rebuild. This happens every time. This didn’t happen prior to this version. Is there a known issue?

Hi Floggins -

We do not have any known issues which would cause this issue. Can you let us know if this happens on multiple projects or a specific project? In either case can you also let us know if the project is stored on a Cloud / Synced drive?

Thank You

Eric Ketchum

hi Eric, I’ll try that out with other projects and let you know. Thanks for the reply. And no btw - not using any cloud anything. this is purely local. Well, sorry, not true. It’s via a p4 server that two users are working on, but I’m the only person working on art.

ok, i’ll check this out. I know the DDC is not being synced. But one of the other two might be. I’ll check all this out today. Thanks a lot!

Hi Floggins -

If you are using a P4 Server setup, it sounds like your Intermediate, Saved and Derived Data Caches are being synced across the server. This would cause the engine to need to rebuild after each opening if others are syncing data into that folder which your local PC does not know what to do with. If it happens in all projects across that server then it is the engine’s Intermediate, Saved and Derived Data Cache which needs to be address, however, if it only happens in one project it is probably that particular projects information which needs to be addressed.

Eric Ketchum

checked it out and none of those items are synced. I also fired up a similar project in 4.8 and it’s opening up with no problems. A few shaders were built, but not many.

Do you do a force sync operation when you pull down from P4 or do you do a simple sync and are you re-compiling the engine as well when you pull down or is it just content being updated?

nope. regular sync. not compiling anything. I’m not working with any code. Only updating content.

Hi Floggins -

After experiencing this shader recompile on your project can you pull the log files from your Saved/Log Folder and post the logs here?

Thank You

Eric Ketchum

sure thing. I’ve replaced the naming with other temp names btw. Seeing as anyone can grab this stuff.link text

Hi Floggins -

Your logs show a significant amount of Derived Data Cache errors. Can you look at your Config files and post what is listed under [DerivedDataBackendGraph]? I think what is happening is that the DDC is being invalidate each time you resync and thus requiring a rebuild of the shaders. You might want to take a look at the following as well:

Thank You

Eric Ketchum

sure, but which config files? the ones in the project or for the editor?

I did look into this a while back because I was trying to get this to work on my machine and the others working on the project. The only problem is that it seems to require an absolute path instead of a project relative one. I want to place it in my project folder (but not sync it) so that I can work on both my pc and mac, but the path structures are completely different. Is there a way to set up a local (to each user) DDC but not have it pointing to a url or require a sync?

Sorry Project.

Hi Floggins -

Yes, you will need to set the UE-SharedDataCachePath environment variable in your DefaultEngine.ini to a Local Drive. The important part is to make sure that the DefaultEngine.ini is also not synced or it will overwrite the location of the DDC each time you sync, which might be what is occuring.

Eric Ketchum

it looks like this may have done the trick. Started up Unreal with no shaders being rebuilt. Once time a few rebuilt, but that was after a crash so I assume that’s normal. Thanks for the help.

so we’re having a real problem here. We had to checkin the defaultEngine.ini because a collision profile was added (which for some crazy reason is stored in this file). It was done on a mac, and now the file is no longer plain text readable in windows.

This tells met that this file actually needs to be syncable and Unreal is just not set up to be used on variable setups. How can this be fixed? This was never a problem in the past, I don’t know why it is now. We’ve never had to set the DDC before 4.10 and now it’s causing a major problem for us. What was changed in all this from version 4.8 to 4.10? We easily worked on various platforms with the projects stored in different places, but not anymore.

Hi Floggins -

You would not need to use the DDC, but for the multiple platforms that are actively developing on the project. OSX uses a different level of Shaders than Windows Machines (OpenGL 3.1/4 versus DirectX11).

Now that I know you are using multiple platforms, that explains why you were rebuilding shaders each time. The Shaders were built for OSX and then opened on Windows you would need to rebuild those shaders for Windows and vice versa on the Mac.

The DDC allowed you to store the SM5 Shaders and the OpenGL shaders fully compiled so you didn’t need to recompile them each time.

As far as how shader compilation has changed, the only changes that have occurred are general improvements to shader compiling times and a change to the type of Diffuse Model the engine uses, and those changed between 4.8 and 4.9. Absolutely no Shader Compiling or DDC usage changed between 4.9 and 4.10.

For the ini file, if you used the generic Mac Text Editor, chances are the line endings in code were changed to an OSX specific settings, I believe you can save the ini file with a generic code editor (like Sublime Text) on the Mac and it will not add the Mac specific Rich Text Formatting that the Text Editor does. You will probably have to rebuild the file on a Windows Machine to get it correctly uploaded to Perforce.

Thank You -

Eric Ketchum

As we discussed earlier - the DDC is local so the Mac built shaders are on the Mac and the Windows based shaders are on the Windows machine. There’s no cross-contamination going on here. Also - the file was not edited in a OSX text editor, it was done in Unreal. So OSX Unreal formatted the file with OSX formatting. Possible bug?

Another thing I noticed last night was that when building my project for iOS it was taking MUCH longer than usual. It’s usually about 10 mins if it has to build everything, this time it was well over 1.5 hours and i just killed it at that point. I did notice in output log though, that every time it would find a new engine shader, it would rebuild all the shaders again. So basically it would build ~700 shaders over and over and over.

1 - are you positive that nothing is wrong with the shader building pipeline?
2 - why is a standard build for iOS building all the engine materials that we’re not using? PS - this is carrying a lot of extra data with it.
3 - could there be something wrong with my install because I’ve never had any issues like this in the past (but this is also my first crack at an iOS game in Unreal)?
4 - is there any way to have you guys change the DDC pathing to use a project relative path instead of an absolute one? This would allow folks to have their own unreal copy installed where they want and still work seamlessly on the same project from anywhere around the globe and on any platform.

Last thing (for clarity). We’re two people on this project one guy working on a mac and one guy on a pc. Both of us have and use the two platforms changing back and forth from time to time (depending if we’re at our desk or working on the road). Only the default data that Unreal adds to perforce is synced. So no build folders, cooked data or temp folders. It’s a very basic setup that has worked flawlessly in the past. It’s really since this latest project that we’re having all these problems.

thanks.

Hey Floggins,

I am going to set up a test environment to see if I can get this issue to occur on my end using a Mac and my PC. To make sure we don’t have any differences in the environment, could you provide me with the engine version you are using i.e. GitHub (Source) 4.10.1 or 4.10.2 or Binary?

Could you provide me with some simple steps when you are setting up your workspace on perforce, and the options you have applied in your DefaultEngine.ini for the DDC folder if any?

From reading the Derived Data Cache documentation, if you want to keep your DDC locally you need to pass an argument to the command line.

One suggestion would be to add a ‘Feature Level Swith’ node to your shaders/materials so that when you open the project, the relative Shading Model being used on the platform is already pre-compiled.

Thanks,