x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Shipping Size is double the project size!

I was about to post this under "Packaging and Deployment" but I am pretty sure this has to be a bug of some sort. A Shipped blueprint project whose project folder amounts to 5gb is being shipped to 10gb (10gb pak file) I am not even sure how is this even possible. I would assume the engine optimizes the assets upon cooking, for example if I import a texture file at 4096*4096 but the "maximum in-game" is set do 2048*2048 I would assume the engine downsamples the texture to 2048*2048 upon shipping to save space. I would assume the engine maybe applies some sort of compression or strips some unnecessary information from the assets as well. I would like clarification on either this happens or not. (Edit: Clarification no longer needed, we've realized this happens, thanks Epic Devs)

But regardless of how optimized the shipping process is I don't see how it can be double of what the assets are, to note this is a blueprint project (so no code) with just static meshes and their materials...and how is this possible that the engine is shipping the project to twice the size of the project folder? Even if the engine packs everything into the .pak file without any regard for optimization it would still never be more than 5gb for this project, so why is this happening? is the engine somehow duplicating every asset inside the .pak file?

Things can get out of hand pretty quickly this way, I don't want to be distributing a 90gb game to my clients, is there something I can do or get clarification on this issue?

This is a x64 windows shipping build.

Product Version: UE 4.15
Tags:
more ▼

asked Apr 30 '17 at 03:17 PM in Bug Reports

avatar image

DavidNSilva
35 4 10 14

avatar image Mobfolios Apr 30 '17 at 05:10 PM

Hey David,

What kind of game is it and what kind of assets you working with?

avatar image DavidNSilva Apr 30 '17 at 06:38 PM

It's a simple project with only static meshes and materiais, there is no blueprint coding or c++ or anything else, just a map with a bunch of static meshes. (quite a few in fact). This is not the main game project but I would like to migrate this assets to the main project but I don't want the shipping size to grow this rapidly.

avatar image Mobfolios May 01 '17 at 02:11 AM

Could you provide a screenshot? Hard to imagine what you're going through based on what you're describing. What's the poly count of your meshes, how many you using what format are your textures in etc?

avatar image DavidNSilva May 01 '17 at 11:34 AM

It's all the meshes from "Crazy Insane Patio Set" from the marketplace.

avatar image TankbusterGames May 01 '17 at 01:43 PM

4096*4096 but the "maximum in-game" is set do 2048*2048 I would assume the engine downsamples the texture to 2048*2048 upon shipping to save space.

I don't think so. You have to resize the texture by yourself.

Since your scene is static.... How big is the lightmap resolution of the meshes? How big is your Light data in general. Have you build light before shipping? Have you triple checked your packaging settings?

avatar image DavidNSilva May 01 '17 at 02:10 PM

Hey thanks for answering, when I mean "maximum in-game" I don't mean I only use 2048*2048 in-game, I mean I set the maximum texture size in the texture properties in-editor, there is no way this setting can be overriden at runtime so I would assume the engine could/should make use of it, specially cause it even reduces the "resource size" label in editor. I might make some tests later on to verify this. Nontheless even without downsampling I don't see how the .pak file can have twice the size of my project folder.

In answer to your other question, I'm not using static lightning at all, it's all fully dynamic so I have no lightmaps.

Yeah I checked the packaging settings...one thing that solves it is to enable .pak compression, reducing the .pak file size from 10gb to 2.5gb....but then again so does compressing the shipped build in a .rar/.zip file :P The disavantage of using .pak compression is well....the .pak is now compressed. The fact a .zip file is able to compress texture information to less than half I think proves the engine in packing redudant data into the .pak file.

avatar image TankbusterGames May 01 '17 at 04:22 PM

https://docs.unrealengine.com/latest/INT/Engine/Content/Types/Textures/SupportAndSettings/index.html

It would increase distributable package size with no rendering quality benefit if a game shipped with 2048 textures assigned to a TextureGroup with a MaxLODSize of 1024

i cannot say anything about the .pak compression, but it wont be the best. Zip will be a little better, because .pak was build to grab data fast in the first place, zip is slower, but better compression.

However.. it's very extreme in your case, so maybe. yeah it's a bug.

avatar image DavidNSilva May 01 '17 at 04:34 PM

Hi Again, I'm not talking about MaxLODSize, it says here: https://docs.unrealengine.com/latest/INT/Engine/Content/Types/Textures/Properties/index.html

Under LOD Bias: The number of miplevels to drop before uploading the Texture. This value can be negative to compensate for the LODGroup's LOD bias for a particular Texture. The LODBias, the LODGroup's LOD bias and the NumCinematicMipLevels are all added together to form a final LOD bias value for the Texture. Mip-levels that cannot be reached in-game because of its LOD bias are removed from the cooked data on disk by Unreal Frontend to save disk space.

I guess this confirms the engine does indeed scrape off unnecessary data.

As for compression...my problem is not that its not compressing enough, my problem is that the .pak file is duplicating the size of the avaliable uncompressed content :/

avatar image Matthew J May 01 '17 at 08:02 PM

Hello DavidNSilva,

When you mention that the size is being doubled, what exact files/folders (please include file paths, as there can be certain folders with the same names in a project directory) are you comparing? Have you tried reproducing this in a fresh project using a few of these assets? If not, please do.

I'm aware this doesn't justify the size being increased, but I would concur with TankbusterGames' comment about the 4096*4096 texture not becoming smaller based off a setting that downsamples the texture, as that 4096*4096 texture will still be packed as is and the downsampling would come later.

Can you also include the log file for your packaging process? The information here may be able to help with looking into what is going on here.

avatar image DavidNSilva May 01 '17 at 08:22 PM

Hi Matthew, What we were talking about was the LOD Bias and it does indeed reduce the .pak file size, it says so in the documentation also TankbusterGames made an experiment with it.

I'll try to clarifify what I mean by "Duplicating": My project folder has 5gb (Checked in windows by going to properties..what I mean is...it's not an estimate and I am not talking about the "Content" folder only, I mean the entireproject folder that contains the .uproject file). I ship the project, the resulting build has a .pak file of 10gb. It would be very difficult for me to try with a few assets only as the difference might not be as noticable.

avatar image Matthew J May 01 '17 at 08:32 PM

I was speaking about TankbusterGames' first comment that quoted your mention of the 4096x4096 textures, I was not referencing the LOD bias conversation.

Trying to reproduce this in a fresh project with the assets that are already giving you this issue would be much easier than me trying to reproduce it with completely different assets. Please take the time to do so.

Another thing that may help diagnose the reason for this increase in size is to try packaging without the "Use pak file" option selected. This will just give you all of the assets uncompressed. This could let you see what assets are being duplicated or added that are increasing the size, if that is what is happening.

avatar image DavidNSilva May 01 '17 at 08:42 PM

Oh sorry about that, I thought you were talking about the LOD bias. The assets that I am using are the "Crazy Insane Patio Set" from the marketplace. I actually have tried what you said before, packaging without the "Use pak file", I got folders and folders full of .uassets files but I am not really sure were to go from there, there are really a lot of assets in there. I am on a fresh project...heres what I did:

I downloaded the "Crazy Insane Patio Set" as mentioned and added the content from the launcher to an empty 4.15 blueprint-only project. I ticked "force no-precomputed lightning" in the provided map/level to make sure lightmaps were not in play. I also disabled "Allow static lightning" in project setting to improve shader performance. I deleted all SM_PatioChair assets (if you try to ship any of them it will fail cooking, something about convex collision and physX). I checked the project folder size (about 3.5gb now without the chairs and stuff) I shipped everything else (So mostly tables). the .pak file is now 9gb.

I know I said 5gb-10gb, but that was before I took all lightmaps.

Is this of use to you? You want me to try to ship with how many assets? if I use only 2 or 3 assets how do I know if the .pak file is acceptable or not?

avatar image Matthew J May 02 '17 at 03:27 PM

One of the main reasons I want you to try this with another project is because we need to determine if is possibly something specific to this marketplace pack. To give an example of why this seems suspect: The content folder for ShooterGame (one of our samples from the Learn tab) is 1.02 GB while the pak file, which also includes light maps, is only 977 MB. This also means it could be caused by some setting that you have enabled. Due to that, could you provide a screenshot of your packaging settings?

While this documentation is focused on APK sizes and packaging for Android, a good amount of this information can still be useful for packaging for other platforms. See if any of the tips here can help out with the size.

Could you test with other assets even if it's one of our free asset packs, such as the Infinity Blade packs on the Marketplace?

avatar image DavidNSilva May 02 '17 at 03:40 PM

Oh that's what you meant by another project. I don't have this problem with any of my other projects, only with this one specifically, but something must be wrong right? the engine shouldn't be doing this under any circunstance, I think it might be related to physx collision cooking cause that used to spam the log until I deleted all chairs. Do you want me to try with the Infinity Blade packs regardless? here's the screenshot alt text

untitled.png (67.6 kB)
avatar image Matthew J May 02 '17 at 03:51 PM

I'm going to continue looking into this and come back with a response but one thing I wanted to clear up first is that, just because this is happening doesn't mean that it's a bug with the engine. The fact that it is only happening in this project actually makes it less likely to be a bug and a settings or content issue instead.

avatar image DavidNSilva May 02 '17 at 04:01 PM

Well I hardly changed any setting as you can see, I think it's a content related also, like something in the content is triggering this abnormal behavior, I find it very strange the .pak file occupies twice as much the .uassets it should have inside. I'll be happy to help in any way I can. Thank you for looking into this issue. I'd also like to add that enabling the "create compressed cooked packages" "solves" the issue...but that's not very ideal because it compresses the content that shouldn't need any compression.

(comments are locked)
10|2000 characters needed characters left

2 answers: sort voted first

While I'm looking into this, the reason I wanted you to try packaging without the .pak file enabled was so that you can compare the content. Are files literally being duplicated? Compare some textures and see if their sizes are bigger.

When it comes to the "create compressed cooked packages" setting, what is wrong with using this? Do you see any sort of detriment in the end result? The pak file is meant for compression in the first place so I don't see why this would be a problem.

more ▼

answered May 02 '17 at 06:24 PM

avatar image DavidNSilva May 02 '17 at 06:35 PM

I did that once, the files did not appear to be literally duplicated but there are a lot of them. I can do that again just need to know how do I know the original size of an asset so I can compare them? as for "create compressed cooked packages" I assume it should affect texture streaming performance etc do you think the difference would be negligible? The engine is so fast I don't notice any loss on my side, but client's specs might be lower than the machines I have available for testing.

avatar image Matthew J May 02 '17 at 06:38 PM

The original size would be the size of the asset in your content folder, so that should be an easy comparison to make. I've never heard of any reduction in performance or quality from the compression setting.

avatar image DavidNSilva May 02 '17 at 06:55 PM

Good to know it doesn't affect performance too much, I'll probably use that option. I'll give the non-pak shipping a go anyways, might take a while, I'll inform back when it's done.

avatar image DavidNSilva May 02 '17 at 07:51 PM

Got it :) alt text As you can see in the screenshot some assets (a lot of them) are a lot bigger after being packed. Not all of them....in fact, some of them are even smaller than the .uasset as it should. But most of them are really small and are getting packed to a much much bigger size! Much more than double.

The original textures folder (under /Content has 3.31gb, the packed textures folders has 8.74gb. both have 519 files (.uasset to .ubulk, not couting the small 1kb .uassets and .uexp files that are present in the .pak folder, those don't seem to be the cause...they amount to 1mb(ish) ).

Edit: Now that I think about it, do you think this is because (maybe) they were imported as .png or some other format and the engine is converting them to .tga upon shipping? I don't know the original format of them...just taking a guess here.

untitled2.png (153.3 kB)
avatar image Matthew J Jun 01 '17 at 08:48 PM

I apologize for the delay, I haven't had a chance to look further into this but what you mentioned about the image format could be the case. Does the size issue only occur with the Texture files and nothing else?

avatar image DavidNSilva Jun 02 '17 at 05:26 PM

Yes only texture files and only Some of them, not all. On any case using the compressed pak file you recommended solves it for me. let me know if you need anything else from me, but my problem is kind of solved and I wouldn't want to waste Epic's resources/time while there might be more important issues. Speaking of which, I am not sure if it's your area or not but If it is maybe you could take a look at my other report about vulkan runtimes being needed (and not included) on directx only projects?

avatar image DavidNSilva Jun 02 '17 at 05:26 PM

Yes only texture files and only Some of them, not all. On any case using the compressed pak file you recommended solves it for me. let me know if you need anything else from me, but my problem is kind of solved and I wouldn't want to waste Epic's resources/time while there might be more important issues. Speaking of which, I am not sure if it's your area or not but If it is maybe you could take a look at my other report about vulkan runtimes being needed (and not included) on directx only projects? Thanks

(comments are locked)
10|2000 characters needed characters left

Oh yeah. I created a blank project with two used 2048x2048 Textures, and used LOD-Bias 0 and 4, and packaged two projects.

The ressource size difference shown in the Texture-Editor is for both textures together -10mb, however, the packaged project (no other settings changed) gets away with only -2mb.

LOD Bias 0 : 192MB LOD Bias 4: 190MB (expected ~182 =( )

So you are right, it will have an impact to package size. Thanks for that!

more ▼

answered May 01 '17 at 05:37 PM

avatar image

TankbusterGames
241 10 11 23

avatar image DavidNSilva May 01 '17 at 07:39 PM

No problem :) One application of this for us game devs is that we can import oversized textures and reduce the lodbias, and as the hardware evolves overtime we can re-ship with a new lod bias to keep up with the "current gen" without much work :)

(comments are locked)
10|2000 characters needed characters left
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question