What are uexp and ubulk files?

Hi

I am currently in the process of switching from 4.13 to 4.15 and something is surprising me : in my packaged project (and in my cooked folder) uexp and ubulk files appeared. And the texture and material uasset files are now very small. I guess what used to be inside one uasset is now split into those 3 files types but I dind’t find any doocumentation on this. Does someone know where I could learn more about it ?

And does it change anything to the process when dynamically mounting a pak file ? I guess those two files have to be mounted too ? Is there something new that should be done ?

Thank you for any input

1 Like

Does someone know where I could learn more about it?

No idea. Would like to know too.

And does it change anything to the process when dynamically mounting a pak file?

On my project, the Packaged 4.15 version crashes when trying to mount 4.14 paks.
The opposite is also true. 4.14 project crashes mounting 4.15 pak.
Packaging on 4.15, adding the .uexp and .ubulk files to the pak, mounts successfully on 4.15 project.

So, I think we can say that 4.14 built projects will only mount 4.14- paks. And 4.15 built projects will only mount 4.15 paks. Haven’t tested 4.16 yet.

A difference on mounting though

I had a bug that after mounting the pak files, local (built with the project) skeletal meshes and some other assets would not be found anymore.

My code was Create FileManager for Paks; Load pak file; Mount; Load assets
To fix it, I had to store the standard FileManager before creating a new one and then, after loading the mounted assets, set the previous FileManager as the current one again.

Working code:

// 4.15 IWYU includes
#include "HAL/PlatformFilemanager.h"
#include "HAL/FileManager.h"
#include "Engine/StreamableManager.h"
#include "AssetRegistryModule.h"
#include "IPlatformFilePak.h"

void UUtils::LoadAssetsFromPakFolder(
	const FString				&DLCFolder,
	TArray<TAssetPtr<UObject> > &Assets,
	TArray<UClass*>				&Classes)
{
	UE_LOG(PaksLoadingLog, Log, TEXT("============= START LOADING PAKS ============="));
	UE_LOG(PaksLoadingLog, Log, TEXT("\n\n\tGameContentDir: %s\n\tEngineContentDir: %s"),
		*FPaths::GameContentDir(),
		*FPaths::EngineContentDir());

	FString Folder = DLCFolder;
	FPaths::NormalizeDirectoryName(Folder);

	TArray<FString> FileNames;

	IPlatformFile* PreviousPlatformFile = NULL;
	FPakPlatformFile *PlatformFile = NULL;
	if (FString(FPlatformFileManager::Get().GetPlatformFile().GetName()).Equals(FString(TEXT("PakFile"))))
	{
		PlatformFile = static_cast<FPakPlatformFile*>(&FPlatformFileManager::Get().GetPlatformFile());
	}
	else
	{
		PlatformFile = new FPakPlatformFile;

		if (!PlatformFile->Initialize(&FPlatformFileManager::Get().GetPlatformFile(), TEXT("")))
		{
			UE_LOG(PaksLoadingLog, Error, TEXT("FPakPlatformFile failed to initialize"));
			return;
		}
		PreviousPlatformFile = &FPlatformFileManager::Get().GetPlatformFile();
		FPlatformFileManager::Get().SetPlatformFile(*PlatformFile);
	}

	if (!PlatformFile->DirectoryExists(*Folder))
	{
		UE_LOG(PaksLoadingLog, Error, TEXT("Directory not found: %s"), *Folder);
		return;
	}
	UE_LOG(PaksLoadingLog, Log, TEXT("DLC directory found: %s"), *Folder);

	// Get list of files in folder
	IFileManager &FileManager = IFileManager::Get();
	FileManager.FindFiles(FileNames, *Folder);

	UE_LOG(PaksLoadingLog, Log, TEXT("Iterating files in DLC folder"));

	FStreamableManager StreamableManager;

	// Iterates on every file, mount the paks and load the assets found in them
	for (FString &FileName : FileNames)
	{
		// Check if extension match required format
		if (FileName.Right(3).Equals(TEXT("pak"), ESearchCase::IgnoreCase))
		{
			UE_LOG(PaksLoadingLog, Log, TEXT("Pak file found: %s"), *FileName);
			FString PakFullPath( FPaths::Combine(*Folder, *FileName) );
			FString PakName, PakExtension;
			FileName.Split( TEXT("."), &PakName, &PakExtension );
			
			FString PakMountPoint = FPaths::Combine(FPaths::GameContentDir(), TEXT("DLC"), PakName);

			if ( !PlatformFile->Mount(*PakFullPath, 0, *PakMountPoint) )
			{
				UE_LOG(PaksLoadingLog, Log, TEXT("Failed to mount %s"), *FileName);
				continue;
			}

			FString VirtualMountPoint(FString::Printf(TEXT("/Game/DLC/%s/"), *PakName));
			FPackageName::RegisterMountPoint(VirtualMountPoint, PakMountPoint);
			UE_LOG(PaksLoadingLog, Log, TEXT("Mount successful. Iterating assets in %s"), *VirtualMountPoint);

			// Print files in pak
			struct FilesDump : public IPlatformFile::FDirectoryVisitor
			{
				FString mp_PakName;
				TArray<FString> Files;

				FilesDump(FString &PakName)
					: mp_PakName(PakName)
				{}

				virtual bool Visit(const TCHAR *FilenameOrDirectory, bool bIsDirectory)
				{
					if (bIsDirectory)
					{
						UE_LOG(PaksLoadingLog, Log, TEXT("Found DIR in %s: %s"), *mp_PakName, FilenameOrDirectory);
					}
					else
					{
						const FString Filename(FilenameOrDirectory);
						if (Filename.Contains(TEXT(".umap")) || Filename.Contains(TEXT(".uasset")))
						{
							Files.Add(FilenameOrDirectory);
							UE_LOG(PaksLoadingLog, Log, TEXT("Found FILE in %s: %s"), *mp_PakName, FilenameOrDirectory);
						}
					}
					return true;
				}
			};

			FilesDump Visitor(FileName);
			PlatformFile->IterateDirectoryRecursively(*PakMountPoint, Visitor);
			
			for (const FString &AssetFilePath : Visitor.Files)
			{
				UE_LOG(PaksLoadingLog, Log, TEXT("Asset File: %s"), *AssetFilePath);

				FString AssetFileName = FPackageName::GetShortName(AssetFilePath);
				FString AssetName, AssetExtension;
				// Get asset name and extension
				AssetFileName.Split(TEXT("."), &AssetName, &AssetExtension);
				// Replace system path for virtual path and extension for asset name
				FString AssetRefPath(AssetFilePath.Replace(*PakMountPoint, *VirtualMountPoint));
				AssetRefPath = AssetRefPath.Replace(*AssetExtension, *AssetName);
				FPaths::RemoveDuplicateSlashes(AssetRefPath);

				// If asset is a Blueprint
				if (AssetName.StartsWith(TEXT("BP_")))
				{
					// Add class postfix
					AssetRefPath += TEXT("_C");
					UObject* LoadedClass = StreamableManager.SynchronousLoad(FStringAssetReference(AssetRefPath));
					if (LoadedClass == NULL)
					{
						UE_LOG(PaksLoadingLog, Error, TEXT("Failed to load class: %s"), *AssetRefPath);
						continue;
					}
					UBlueprintGeneratedClass* BPClass = Cast<UBlueprintGeneratedClass>(LoadedClass);
					if (BPClass == NULL)
					{
						UE_LOG(PaksLoadingLog, Error, TEXT("Casting to BlueprintGeneratedClass failed: %s"), *AssetRefPath);
						continue;
					}
					Classes.Add(BPClass);
				}
				else
				{
					Assets.Add(TAssetPtr<UObject>(FStringAssetReference(AssetRefPath)));
				}
			}
		}
	}

	// return previous platform file manager to the top of the chain, so Unreal doesn't lose it's references
	if (PreviousPlatformFile != NULL)
	{
		FPlatformFileManager::Get().SetPlatformFile(*PreviousPlatformFile);
	}

	UE_LOG(PaksLoadingLog, Log, TEXT("============= END LOADING PAKS ============="));
}
1 Like

Thanks a lot for your answer, I will test it as soon as I can.

I don’t officially “accept” it only because the main question is still unanswered.

1 Like

Don’t worry :smiley:
I already expected that. I’m also wondering what those files are for.

I just wanted to leave it here for anyone having problems converting their paks between engine versions.

1 Like

Hey everyone,

Rather than one large asset, these allow us to write an asset’s bulk data (.ubulk) and exports (uexp) out into separate files. This system improves perf in certain circumstances where file read contiguity is lost due to the large size of assets. This feature avoids this by enabling the reader to skip over an asset’s bulk data when seeking to the next file in a series without having to actually have serialized and seeked past that data (since it’s in separate file).

These files aren’t ‘extras’ when the feature is enabled, but rather what used to be a single file is now broken into multiple files.

This system is enabled with the following configuration setting in your engine ini

[Core.System]

UseSeperateBulkDataFiles=True

Furthermore, the event driven loader (EDL) requires split bulk data to be enabled. If EDL is enabled the above setting will therefore be ignored. In 4.15 we introduced the EDL, so separate bulk data was turned on by default. You can see if EDL is enabled with the following configuration setting in your engine ini

[/Script/Engine.StreamingSettings]

s.EventDrivenLoaderEnabled=True

Cheers,

Jonathan

5 Likes

Oh my god! This is just what I was looking for! I can’t even find the words to thank you!

1 Like

Send me a million dollars through PayPal and we are square. lol

1 Like

I don’t have any files in the created directory, and IterateDirectoryRecursively returns 0. Do you have any idea what i might have done wrong?

1 Like

getting the same @MrMcKvarz, did you find a fix?

1 Like

Change FPaths::GameContentDir to FPaths::EngineContentDir, then you should probably see those files. Doesn’t load any files though. When loading object engine looks if there is actually a file on a disk, and mounting pak file doesn’t extract those files to the mount point. Extracting files doesn’t help either, since I haven’t yet found a way to combine uexp and uasset back to one file, or load them separately.

1 Like

Depending on the engine version you have, you might want to use “FPaths::ProjectContentDir” instead of “FPaths::GameContentDir”, since the second one is obsolete. It works just fine for me, up until the point where I need to cast the object I load from my “.uassets”. Couldn’t find any solution to this yet. Though, there’s another problem that has to do with assets having their hard references to other assets all wrong. I can still load the map from pak file, but all objects inside it are gone. I’ve also seen errors reporting about missing some of my base classes for their children. I took a break at this point and will certainly come back to the topic during this month, and share everything I’ll find.

1 Like

@MrMcKvarz I got it working .i.e loading a simple material with no dependencies, but @Hmomer
I think I’m in the same predicament as you. Whenever I mount the pak the target asset becomes available but it breaks references to other dependent assets. When the pak is unmounted the dependent assets become available but the target asset isn’t available anymore.

1 Like

I should try loading a material too because right now I can’t even cast my blueprint asset, though it loads just fine. I can hardly think of any simple solution to the dependency problem. I guess this is one of the main issues with UE4 being poorly suited for the mod support. Btw, trying to “trick” dependencies doesn’t work. I tried naming both game and mod projects similarly and making the mount point “look” the same as if the mod was running standalone, still my BP couldn’t find it’s base class, though the path it tried to look for it was literally legit. Btw, does anyone knows the difference between making pak file as a product of the packaged game, and packing the assets manually using “UnrealPak.exe”. I tried both and I actually had more success with UnrealPak as I could load the asset into memory at least. The problem is when I use UnrealPak it ignores all the subfolders in the main folder and puts all the assets from subfolders into the root.

1 Like

@mseeds1, you got it working with the code above?

1 Like

@MrMcKvarz I don’t know if you can confer this but GameContentDir is a strange path if I use the example code above. It’s almost like two paths joined together. e.g …\path1…\path2 I changed the mount directory to the engine directory that allows me to mount the materials but we really need to get it mounting to the game directory otherwise dependencies are going to cause a lot of problems.

1 Like

@MrMcKvarz Only materials but I had to make some changes. It still isn’t right either.

1 Like

I extracted AssetRegistry.bin file from pak, and add it to the current registry. It could theoretically fix broken dependency problem. Still can’t load even a broken asset. @Hnomer, can you share your code or tell me how did you modified above code?

1 Like

I basically only changed the mount point to fit my needs (instead of “DLC” folder, I use “Mods” folder) and modified the line, where it distinguishes a blueprint from other assets to, again, fit my naming. I’ve also added additional filtering just to ensure that it doesn’t scan my whole pak file, and only the specific folder inside it.
As I mentioned in the comment above, I changed “FPaths::GameContentDir” to "“FPaths::ProjectContentDir”. And I load the object using UObjects’ “TryLoad()” instead of “SynchronousLoad()”.

1 Like

pretty sure asset.bin is loaded into memory when the program starts.

1 Like

@mseed1, do you mean “…/…/…/ProjectName” or smth like that?

1 Like