x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

what are uexp and ubulk files ?

Hi

I am currently in the process of switching from 4.13 to 4.15 and something is surprising me : in my packaged project (and in my cooked folder) uexp and ubulk files appeared. And the texture and material uasset files are now very small. I guess what used to be inside one uasset is now split into those 3 files types but I dind't find any doocumentation on this. Does someone know where I could learn more about it ?

And does it change anything to the process when dynamically mounting a pak file ? I guess those two files have to be mounted too ? Is there something new that should be done ?

Thank you for any input

Product Version: UE 4.15
Tags:
more ▼

asked Apr 11 '17 at 03:41 PM in Everything Else

avatar image

Schopi
46 4 7 11

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

2 answers: sort voted first

Does someone know where I could learn more about it?

o idea. Would like to know too.

And does it change anything to the process when dynamically mounting a pak file?

n my project, the Packaged 4.15 version crashes when trying to mount 4.14 paks.
The opposite is also true. 4.14 project crashes mounting 4.15 pak.
Packaging on 4.15, adding the .uexp and .ubulk files to the pak, mounts successfully on 4.15 project.

So, I think we can say that 4.14 built projects will only mount 4.14- paks. And 4.15 built projects will only mount 4.15 paks. Haven't tested 4.16 yet.

A difference on mounting though

had a bug that after mounting the pak files, local (built with the project) skeletal meshes and some other assets would not be found anymore.

My code was Create FileManager for Paks; Load pak file; Mount; Load assets
To fix it, I had to store the standard FileManager before creating a new one and then, after loading the mounted assets, set the previous FileManager as the current one again.

Working code:

 // 4.15 IWYU includes
 #include "HAL/PlatformFilemanager.h"
 #include "HAL/FileManager.h"
 #include "Engine/StreamableManager.h"
 #include "AssetRegistryModule.h"
 #include "IPlatformFilePak.h"
 
 void UUtils::LoadAssetsFromPakFolder(
     const FString                &DLCFolder,
     TArray<TAssetPtr<UObject> > &Assets,
     TArray<UClass*>                &Classes)
 {
     UE_LOG(PaksLoadingLog, Log, TEXT("============= START LOADING PAKS ============="));
     UE_LOG(PaksLoadingLog, Log, TEXT("\n\n\tGameContentDir: %s\n\tEngineContentDir: %s"),
         *FPaths::GameContentDir(),
         *FPaths::EngineContentDir());
 
     FString Folder = DLCFolder;
     FPaths::NormalizeDirectoryName(Folder);
 
     TArray<FString> FileNames;
 
     IPlatformFile* PreviousPlatformFile = NULL;
     FPakPlatformFile *PlatformFile = NULL;
     if (FString(FPlatformFileManager::Get().GetPlatformFile().GetName()).Equals(FString(TEXT("PakFile"))))
     {
         PlatformFile = static_cast<FPakPlatformFile*>(&FPlatformFileManager::Get().GetPlatformFile());
     }
     else
     {
         PlatformFile = new FPakPlatformFile;
 
         if (!PlatformFile->Initialize(&FPlatformFileManager::Get().GetPlatformFile(), TEXT("")))
         {
             UE_LOG(PaksLoadingLog, Error, TEXT("FPakPlatformFile failed to initialize"));
             return;
         }
         PreviousPlatformFile = &FPlatformFileManager::Get().GetPlatformFile();
         FPlatformFileManager::Get().SetPlatformFile(*PlatformFile);
     }
 
     if (!PlatformFile->DirectoryExists(*Folder))
     {
         UE_LOG(PaksLoadingLog, Error, TEXT("Directory not found: %s"), *Folder);
         return;
     }
     UE_LOG(PaksLoadingLog, Log, TEXT("DLC directory found: %s"), *Folder);
 
     // Get list of files in folder
     IFileManager &FileManager = IFileManager::Get();
     FileManager.FindFiles(FileNames, *Folder);
 
     UE_LOG(PaksLoadingLog, Log, TEXT("Iterating files in DLC folder"));
 
     FStreamableManager StreamableManager;
 
     // Iterates on every file, mount the paks and load the assets found in them
     for (FString &FileName : FileNames)
     {
         // Check if extension match required format
         if (FileName.Right(3).Equals(TEXT("pak"), ESearchCase::IgnoreCase))
         {
             UE_LOG(PaksLoadingLog, Log, TEXT("Pak file found: %s"), *FileName);
             FString PakFullPath( FPaths::Combine(*Folder, *FileName) );
             FString PakName, PakExtension;
             FileName.Split( TEXT("."), &PakName, &PakExtension );
             
             FString PakMountPoint = FPaths::Combine(FPaths::GameContentDir(), TEXT("DLC"), PakName);
 
             if ( !PlatformFile->Mount(*PakFullPath, 0, *PakMountPoint) )
             {
                 UE_LOG(PaksLoadingLog, Log, TEXT("Failed to mount %s"), *FileName);
                 continue;
             }
 
             FString VirtualMountPoint(FString::Printf(TEXT("/Game/DLC/%s/"), *PakName));
             FPackageName::RegisterMountPoint(VirtualMountPoint, PakMountPoint);
             UE_LOG(PaksLoadingLog, Log, TEXT("Mount successful. Iterating assets in %s"), *VirtualMountPoint);
 
             // Print files in pak
             struct FilesDump : public IPlatformFile::FDirectoryVisitor
             {
                 FString mp_PakName;
                 TArray<FString> Files;
 
                 FilesDump(FString &PakName)
                     : mp_PakName(PakName)
                 {}
 
                 virtual bool Visit(const TCHAR *FilenameOrDirectory, bool bIsDirectory)
                 {
                     if (bIsDirectory)
                     {
                         UE_LOG(PaksLoadingLog, Log, TEXT("Found DIR in %s: %s"), *mp_PakName, FilenameOrDirectory);
                     }
                     else
                     {
                         const FString Filename(FilenameOrDirectory);
                         if (Filename.Contains(TEXT(".umap")) || Filename.Contains(TEXT(".uasset")))
                         {
                             Files.Add(FilenameOrDirectory);
                             UE_LOG(PaksLoadingLog, Log, TEXT("Found FILE in %s: %s"), *mp_PakName, FilenameOrDirectory);
                         }
                     }
                     return true;
                 }
             };
 
             FilesDump Visitor(FileName);
             PlatformFile->IterateDirectoryRecursively(*PakMountPoint, Visitor);
             
             for (const FString &AssetFilePath : Visitor.Files)
             {
                 UE_LOG(PaksLoadingLog, Log, TEXT("Asset File: %s"), *AssetFilePath);
 
                 FString AssetFileName = FPackageName::GetShortName(AssetFilePath);
                 FString AssetName, AssetExtension;
                 // Get asset name and extension
                 AssetFileName.Split(TEXT("."), &AssetName, &AssetExtension);
                 // Replace system path for virtual path and extension for asset name
                 FString AssetRefPath(AssetFilePath.Replace(*PakMountPoint, *VirtualMountPoint));
                 AssetRefPath = AssetRefPath.Replace(*AssetExtension, *AssetName);
                 FPaths::RemoveDuplicateSlashes(AssetRefPath);
 
                 // If asset is a Blueprint
                 if (AssetName.StartsWith(TEXT("BP_")))
                 {
                     // Add class postfix
                     AssetRefPath += TEXT("_C");
                     UObject* LoadedClass = StreamableManager.SynchronousLoad(FStringAssetReference(AssetRefPath));
                     if (LoadedClass == NULL)
                     {
                         UE_LOG(PaksLoadingLog, Error, TEXT("Failed to load class: %s"), *AssetRefPath);
                         continue;
                     }
                     UBlueprintGeneratedClass* BPClass = Cast<UBlueprintGeneratedClass>(LoadedClass);
                     if (BPClass == NULL)
                     {
                         UE_LOG(PaksLoadingLog, Error, TEXT("Casting to BlueprintGeneratedClass failed: %s"), *AssetRefPath);
                         continue;
                     }
                     Classes.Add(BPClass);
                 }
                 else
                 {
                     Assets.Add(TAssetPtr<UObject>(FStringAssetReference(AssetRefPath)));
                 }
             }
         }
     }
 
     // return previous platform file manager to the top of the chain, so Unreal doesn't lose it's references
     if (PreviousPlatformFile != NULL)
     {
         FPlatformFileManager::Get().SetPlatformFile(*PreviousPlatformFile);
     }
 
     UE_LOG(PaksLoadingLog, Log, TEXT("============= END LOADING PAKS ============="));
 }

more ▼

answered Apr 27 '17 at 10:43 PM

avatar image

RVillani
1.1k 28 41 90

avatar image Schopi May 02 '17 at 10:39 AM

Thanks a lot for your answer, I will test it as soon as I can.

I don't officially "accept" it only because the main question is still unanswered.

avatar image RVillani May 04 '17 at 07:22 AM

Don't worry :D
I already expected that. I'm also wondering what those files are for.

I just wanted to leave it here for anyone having problems converting their paks between engine versions.

avatar image Hnomer Mar 25 '18 at 11:49 AM

Oh my god! This is just what I was looking for! I can't even find the words to thank you!

avatar image RVillani Mar 26 '18 at 06:59 PM

Send me a million dollars through PayPal and we are square. lol

avatar image MrMcKvarz Apr 02 '18 at 03:34 PM

I don't have any files in the created directory, and IterateDirectoryRecursively returns 0. Do you have any idea what i might have done wrong?

avatar image mseeds1 Apr 07 '18 at 03:09 AM

getting the same @MrMcKvarz, did you find a fix?

avatar image MrMcKvarz Apr 07 '18 at 11:47 AM

Change FPaths::GameContentDir to FPaths::EngineContentDir, then you should probably see those files. Doesn't load any files though. When loading object engine looks if there is actually a file on a disk, and mounting pak file doesn't extract those files to the mount point. Extracting files doesn't help either, since I haven't yet found a way to combine uexp and uasset back to one file, or load them separately.

avatar image Hnomer Apr 07 '18 at 01:02 PM

Depending on the engine version you have, you might want to use "FPaths::ProjectContentDir" instead of "FPaths::GameContentDir", since the second one is obsolete. It works just fine for me, up until the point where I need to cast the object I load from my ".uassets". Couldn't find any solution to this yet. Though, there's another problem that has to do with assets having their hard references to other assets all wrong. I can still load the map from pak file, but all objects inside it are gone. I've also seen errors reporting about missing some of my base classes for their children. I took a break at this point and will certainly come back to the topic during this month, and share everything I'll find.

avatar image mseeds1 Apr 07 '18 at 02:31 PM

@MrMcKvarz I got it working .i.e loading a simple material with no dependencies, but @Hmomer
I think I'm in the same predicament as you. Whenever I mount the pak the target asset becomes available but it breaks references to other dependent assets. When the pak is unmounted the dependent assets become available but the target asset isn't available anymore.

avatar image Hnomer Apr 07 '18 at 04:47 PM

I should try loading a material too because right now I can't even cast my blueprint asset, though it loads just fine. I can hardly think of any simple solution to the dependency problem. I guess this is one of the main issues with UE4 being poorly suited for the mod support. Btw, trying to "trick" dependencies doesn't work. I tried naming both game and mod projects similarly and making the mount point "look" the same as if the mod was running standalone, still my BP couldn't find it's base class, though the path it tried to look for it was literally legit. Btw, does anyone knows the difference between making pak file as a product of the packaged game, and packing the assets manually using "UnrealPak.exe". I tried both and I actually had more success with UnrealPak as I could load the asset into memory at least. The problem is when I use UnrealPak it ignores all the subfolders in the main folder and puts all the assets from subfolders into the root.

avatar image MrMcKvarz Apr 10 '18 at 01:41 PM

I extracted AssetRegistry.bin file from pak, and add it to the current registry. It could theoretically fix broken dependency problem. Still can't load even a broken asset. @Hnomer, can you share your code or tell me how did you modified above code?

avatar image mseeds1 Apr 10 '18 at 02:04 PM

pretty sure asset.bin is loaded into memory when the program starts.

avatar image Hnomer Apr 10 '18 at 02:56 PM

I basically only changed the mount point to fit my needs (instead of "DLC" folder, I use "Mods" folder) and modified the line, where it distinguishes a blueprint from other assets to, again, fit my naming. I've also added additional filtering just to ensure that it doesn't scan my whole pak file, and only the specific folder inside it. As I mentioned in the comment above, I changed "FPaths::GameContentDir" to ""FPaths::ProjectContentDir". And I load the object using UObjects' "TryLoad()" instead of "SynchronousLoad()".

avatar image mseeds1 Apr 10 '18 at 03:29 PM

@Hnomer did you solve the problem with the hard references?

avatar image Hnomer Apr 10 '18 at 03:44 PM

Nope, but I am thinking about taking a look at the UE source and see how it works around those references. I hope I'll be able to modify it. I want it to detect the external reference by maybe keywords and modify this reference as ordered and only then follow it. @MrMcKvarz I don't think that AssetRegistry.bin can fix the dependency problem. I haven't yet figured out its true purpose, but I think it is irrelevant to the topic.

UPD: I ran some tests today and found out that I can't even cast a simple blueprint that inherits from the Actor class, though as before, I could load it as UObject. Tests with a blueprint that has a custom parent showed that it can't find its parent even if I pre-create that parent by the path it is going to be looking at. It still says it can't find the parent... Notice that I ran these tests not in standalone, but in the editor and the pak file I was using was created manually using UnrealPak.exe

avatar image mseeds1 Apr 11 '18 at 01:39 AM

do you think the pak system is bugged? surely it was not designed to work in this manner?

avatar image Hnomer Apr 11 '18 at 07:16 AM

Well, the problem with the references is clearly not a bug, it is that UE4 is not suited for modding. I don't really think that my blueprint loading problem is a bug either. I think that logs are not really giving me enough information and I have to again look at the source of all the functions used in the process of object creation.

avatar image StrangerGwenn Nov 13 '18 at 04:32 AM

Loading paks is not supposed to be supported whithin the editor, you need to build as standalone.

avatar image MrMcKvarz Apr 07 '18 at 09:10 PM

@mseeds1, you got it working with the code above?

avatar image mseeds1 Apr 07 '18 at 11:14 PM

@MrMcKvarz Only materials but I had to make some changes. It still isn't right either.

avatar image mseeds1 Apr 08 '18 at 12:58 AM

@MrMcKvarz I don't know if you can confer this but GameContentDir is a strange path if I use the example code above. It's almost like two paths joined together. e.g ......\path1......\path2 I changed the mount directory to the engine directory that allows me to mount the materials but we really need to get it mounting to the game directory otherwise dependencies are going to cause a lot of problems.

avatar image MrMcKvarz Apr 10 '18 at 02:07 PM

@mseed1, do you mean "../../../ProjectName" or smth like that?

avatar image mseeds1 Apr 10 '18 at 03:34 PM

@MrMcKvarz I mounted to the EngineContentDirectory. It is the only way I have been able to get it to work. I found a way to load static meshes but the internal materials are discarded (i.e so there is no hard references to /Game/materialname/. That may be something you could consider, it is just about serviceable for what I am trying to do but not ideal.

avatar image Wes_song May 18 '19 at 02:51 PM

How to load static meshes?I have mounted .pak successfully,but when ”StreamableManager.SynchronousLoad” excuted ,app crashed,Log end with "Skipping the cache update for the package asset groups due to a cache request from a non-game thread. Some localized packages may be missed for this query." Platform is windowsNoeditor.

avatar image TheMainStark Jun 30 '18 at 07:23 AM

Anyone able to give a blueprint coder a hand in implementing this? I am a little confused on where to add the above code.

Do I add the code above to the UnrealEd Utils class or make a new C++ Class? Creating a C++ class prompts it as it already exists as seen below:

alt text

pakpost.png (2.3 kB)
avatar image MrMcKvarz Jun 30 '18 at 12:08 PM

Hi, any class should be fine, since you are not dealing with the gameplay logic. Try changing name, since utils class is quite common, and as you see, is already taken.

avatar image TheMainStark Jun 30 '18 at 05:26 PM

Thanks for the quick reply, am I correct this is a normal project class (File > New Class) or does this have to be placed in the engine source?

Also, I am on 4.19 and my main error is :

 Cannot open include file: 'IPlatformFilePak.h': No such file or directory
avatar image MrMcKvarz Jun 30 '18 at 05:58 PM

In your build.cs file add "PakFile" to PublicDependencyModuleNames.

avatar image TheMainStark Jul 01 '18 at 03:27 AM

I have it successfully compiled but now I am a bit confused on how to use this?

My class MountPak shows as a cast node... what is the node order to select a pak in a directory, than mount it?

alt text

mountpak.png (48.2 kB)
avatar image MrMcKvarz Jul 01 '18 at 07:47 AM

If your class derived from blueprint function library you need to make your function static and mark it as a ufunction(blueprintcallable).

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

Hey everyone,

Rather than one large asset, these allow us to write an asset's bulk data (.ubulk) and exports (uexp) out into separate files. This system improves perf in certain circumstances where file read contiguity is lost due to the large size of assets. This feature avoids this by enabling the reader to skip over an asset's bulk data when seeking to the next file in a series without having to actually have serialized and seeked past that data (since it's in separate file).

These files aren't 'extras' when the feature is enabled, but rather what used to be a single file is now broken into multiple files.

This system is enabled with the following configuration setting in your engine ini

[Core.System]

UseSeperateBulkDataFiles=True

Furthermore, the event driven loader (EDL) requires split bulk data to be enabled. If EDL is enabled the above setting will therefore be ignored. In 4.15 we introduced the EDL, so separate bulk data was turned on by default. You can see if EDL is enabled with the following configuration setting in your engine ini

[/Script/Engine.StreamingSettings]

s.EventDrivenLoaderEnabled=True

Cheers,

Jonathan

more ▼

answered Nov 15 '17 at 07:56 PM

avatar image

NaturalPanic STAFF
136 4 7 12

avatar image StrangerGwenn Nov 13 '18 at 04:30 AM

Hey NaturalPanic, is there any reference code anywhere to load a pak file and use AssetRegistry to find all assets (including from that pak) ? I'm starting to believe that it's actually entirely impossible since 4.18 onwards, and many posts on AnswerHub and the forums seem to reflect that. I've definitely tried three or four different methods now and it always seem to fail. More on this here : https://answers.unrealengine.com/questions/836122/get-all-actors-from-mounted-pak.html

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question