Linux and Mac performance roadmap

Development Platform & Builds
, , , , , ,
1

We have a FPS on Early Access (Storm United) and while the windows build general performance are fine, the Linux and mac builds are to be honest still very bad.
We are testing the game on all platforms on the very same computer so its not a hardware issue.

I made a wrong assumption by thinking last year that by the release of the SteamBoxes (next month or so) the Linux builds would be well polished.
Unfortunately, I’m afraid we might have done a mistake to support those platforms, the work overhead is substantial and our forums are riddled with mac and Linux complains.

Bug wise those platforms are still buggy and prone to crashes but we have seen important improvements over the months, unlike the performance area where we have seen little progress to close the gap with windows.

This said, I want to be able to say something about performance to our users for example where the performance drop comes from and hopefully what’s being done by Epic about it.

  1. Why is the framerate so low and unstable as opposed to Windows?
  2. Why are there loading fps hickups every time a new material is drawn on the screen (happens only once for each “new” material after the map loaded)? Is there something we can do about it?
  3. Why is the CPU usage going through the roof on those platforms?
  4. Will Vulcan alone solve those problems? Do we know when we can expect to see it supported?

Any other performance related information is very welcome

Thank you and keep the great work flowing ^^

2

Could you give us some details on what hardware are you testing on and Linux and OS X version you use? Also, some numbers that’d tell us how bad the problem in your game is? Also, do you know if on Linux you’re running the game in OpenGL 4.3 or 3.2 mode?

In general (and not counting the hitches caused by rendering new materials - more on that below), in our demos and games the performance on Linux and Mac is rather OK, disappointingly worse than in Direct3D on Windows, but certainly not very bad (on average about 85-90% of D3D). That said, we just had a report that the new Unreal Tournament’s performance is currently much worse than it was, so we were going to look into this very soon. Our suspiction is that the game started to do something that hits the slow path in the drivers. I suppose your game could have a similar problem. I’ll make sure to let you know what we find.

As for stability, if you have any call stacks, please attach them here to make sure we’re aware of these crashes. Thanks!

And to answer your questions:

  1. It’s a combination of many factors and we’re having a very hard time finding any more culprits in our OpenGL code (not counting the hitching, see answer 2). Part of the reason is that OpenGL drivers are worse at optimizing shaders than D3D is or the way buffer updates and synchronization of these updates in OpenGL are performed. It’s mostly things like that. We do keep trying to improve things, but as I mentioned, it’s not easy.

  2. This is because loading shaders and drawing using a shader and render state combination for the first time is much, much more expensive in OpenGL than it is in D3D. We’re planning to refactor our RHI API to be a better fit for the new APIs (D3D12, Metal, Vulkan) which should also help with this problem, but in a meantime we added a shader cache that works around the problem at the expense of longer load times. It basically saves all the information on what shaders and render states are used during gameplay and the next time you run the game it precompiles and predraws these during loading screen, so that it doesn’t have to do that while you play. For Fortnite we’ve recently been working on automating the process of populating the shader cache and we include the cache file with the game. Unfortunately, the main programmer responsible for this cache is out this week, but we can certainly help you making sure your game makes the best use of this cache when he’s back.

  3. Are you sure? I’m asking, as in the past we had many users confused by how OS X reports CPU usage compared to Windows. Windows shows the percentage of whole CPU that’s in use, OS X shows percentage of a single logical core. This means that on a quad-core i7 with hyperthreading, if the whole CPU is utilized, Windows will show 100% and OS X 800%, while if a single core is fully utilized, Windows will show 12.5% and OS X 100%. If that’s not what you observe, it’s definitely something I’d like to investigate.

  4. Eventually Vulkan (on Linux) and Metal (on OS X) may help, but not without changes to the engine and not without a lot of work on the drivers side to match the performance of D3D, especially in areas like shader execution speed. We are actively working on Vulkan and Metal support with hardware vendors, but while Metal is in (more or less) usable form already, I can’t tell how long it will take Vulkan to get there.

3

Hi and thank you for your insightful answer and sorry for the delay, I had to wait for the scheduled test run with the team to confirm some numbers.

FYI:
Our test Mac and Linux boxes are equipped with an i5 3470 3.4 Ghz CPU , an NVIDIA 650 2gb and 16Gb of ram.
The Linux box runs an Ubuntu 14 with latest official NVIDIA Linux drivers
The Mac is running the latest OSX Yosemite and the latest NVIDIA web drivers

We are running the game with the default OpenGL 3.2

In our case, I would say the performance of those platforms is more around 70-80% of what windows does in terms of framerate when the framerate is stable. This said the framerate isn’t as stable as on windows (I feel its mostly related to some memory related loadings).

I would indeed be very interested to know what caused the latest UT slowdown.

I will report the future crashes and attach the call stacks in separate posts for sake of clarity.
On macOS we currently have an annoying situation where on 3 large maps, the stat unit Frame value goes up to about 70ms, Game goes to about 8ms draw 9ms and gpu is at 35ms. We then get a complete system crashes requiring a reboot (we’re not getting any useful information from them atm).
It happens at some locations of the map. If I’m careful and rotate the camera at the right moment I can see those value shoot up and then go back down without a crash if I rotate back fast enough.
The large maps also crash on linux during loading.
Is it possible there’s some kind of GPU memory overflow?
We have r.streamping.poolsize set to 0, it works great on windows. On mac even if I set it to 500 it will still crash. Are there other console commands we can use to limit memory allocation?

4

  1. Ok I understand.

  2. We are very interested to know more about it. Is it something we will see in a future UE update or does the method require a custom implementation?

  3. When I look at the CPU usage I refer to the data we get from the Stat Unit console command.
    On windows values are stable ranging from 3 to 5ms while on Linux and Mac CPU it shoots to 16ms regularily (it is defenetly occuring more often on Mac)

  4. You say Metal is more or less usable already, does it require anything in particular to be activated like OpenGL 4? or another console command? Or having simply the latest MacOS El Capitan installed does the trick?

On a side note, speaking of mac, is there a way or trick to have a proper splash screen? At this moment we are getting this:
http://puu.sh/l1Jdr/d511ad7926.jpg

Additionally how do we change the default icon of macOS builds? I don’t see any macOS platform in the project settings: http://puu.sh/l1JAb/18b6dd8545.jpg

Thank you very much for your answers

5

On Mac you could try forcing ANSI allocator (see FMacPlatformMemory::BaseAllocator()) and also disable our crash reporter. To do this, modify runGameThread: in LaunchMac.cpp, change this:

if( bIsBuildMachine || ( FPlatformMisc::IsDebuggerPresent() && !GAlwaysReportCrash ) )
to
if( true )

Perhaps system crash reporter would have more luck grabbing the call stack and will shed some light on what’s wrong.

6

  1. It’s already present in 4.9 (see ShaderCache.cpp), but recently we’ve made improvements to this. I’ll make sure to provide you all the info you need soon.

  2. Hmm. This may actually be a red herring, but I’ll check a few things and get back to you on this as well.

  3. Metal support is already in our Main branch (also in GitHub) and it’ll officially be included in binary builds starting in 4.11. No additional work is required to enable it in this branch, it’s used automatically instead of OpenGL on El Capitan.

As for the splash screen, I fixed this recently, for 4.10. We weren’t handling images with DPI != 72 correctly. You can fix it in your build in FMacPlatformSplash::Show(), add this line:

[GSplashScreenImage setSize:NSMakeSize(ImageWidth, ImageHeight)];

after:

NSBitmapImageRep* BitmapRep = [NSBitmapImageRep imageRepWithData: [GSplashScreenImage TIFFRepresentation]];
float ImageWidth = [BitmapRep pixelsWide];
float ImageHeight = [BitmapRep pixelsHigh];

As for the icon, I’m confused, as we definitely have Mac platform properties in project settings. Maybe only in Mac editor? I’ll check.

If you have .icns file, you can rename it to Application.icns and put in your game’s Build/Mac folder or rename it to YourGameName.icns and put it in YourGameName/Source/YourGameName/Resources/Mac, like this:

65290-icon.png