Migrating assets from 4.7.6 to 4.9.0 causes out of memory error

I am trying to migrate a project from 4.7.6 to 4.9.0 and I am encountering out of memory errors on packaged assets. It seems that any *.uasset I try to migrate will cause issues. I can migrate an asset from my 4.7.6 project to a new 4.7.6 project, but as soon as I try to migrate it to any other engine version, I get out of memory errors. I even had to hard reboot my computer.

I’m working with a source build of the engine and I made some slight modifications to ObjectVersion.h when I migrated my assets from 4.6.1 to 4.7.6 – I think that these modifications permanently changed the package files during serialization and deserialization. I’d really like to just be able to migrate all of my content assets to the newest version of UE4.

To replicate this problem, I take a single asset from my 4.7.6 project and migrate it into the content folder of a brand new 4.9.0 project and then try to open the 4.9.0 test project. If you do this with the source build, you get this exception:

Ran out of memory allocating 18446744041135534544 bytes with alignment 0

If you run it off of a binary build, I think you’ll just run out of memory and the computer will freeze because it tries to page it out to handle the impossible malloc request.

I have attached my Output Log, my Call Stack, and a sample asset file which causes this crash for me.
The specific place where it crashes is in Array.h at line 1198:

A.Empty(NewNum);

1 Like

Hey -

It is possible that the edits to source code may have affected migration. Creating a blueprint in a 4.7.6 project and migrating it to a 4.9 project was successful for me including opening the 4.9 project after migration. Attempting to import your sample asset returns an error that it can’t the asset type (possibly related). Can you try creating an asset in a new 4.7.6 project and then migrate that asset to a new 4.9 project? If you’re still getting out of memory errors you may need to revert the changes to your 4.7.6 source code or match the changes in your 4.9 source.

Cheers

This is specific to the modified engine I’m running based off of 4.7.6. The short story is that I downloaded the source for the master repo when 4.7.6 was released and a commit in the master caused the version number to go to 4.8.0, and that caused every packaged asset to have a different file signature. This was causing all of my old assets to not be visible within the content browser. So, to get them to become usable again, I had to edit the “ObjectVersion.h” header file. When a package is saved, it takes the package number from the massive enumeration and encodes it into the uasset file. Where exactly, I don’t know.

So… all of my game assets have this version encoded into the file, so I think reverting the source code to vanilla 4.7.6 would just cause 4.7.6 to not recognize the files. Last night I tried to update the ObjectVersion.h file in 4.7.6 to the one used in 4.9.0 so that the assets would be saved with the newer signature. The good news is that the assets didn’t crash the engine or attempt to consume all of the memory, but the bad news is that they still don’t show up in the content browser. I’ll try to work through this more today.

I created a compiled version of 4.7.6 (++depot+UE4-Releases+4.7)

It’s completely unmodified from the release version. I import a small texture asset from a common texture file. The import succeeds. I then try to migrate that asset to 4.9.0 and I get an error:

[2015.09.10-22.33.50:468][359]LogFileManager:Warning: ReadFile failed: Count=0 Length=27738 Error=The operation completed successfully. for file C:/Users/Eric/Documents/Unreal Projects/ZombieGame/Content/TestTexture.uasset
Fatal error: [File:C:\Users\Eric\Documents\Visual Studio 2013\Projects\UE4.9.1\Engine\Source\Runtime\Core\Private\HAL\FileManagerGeneric.cpp] [Line: 647]
Invalid BufferCount=0 while reading C:/Users/Eric/Documents/Unreal Projects/ZombieGame/Content/TestTexture.uasset. File is most likely corrupted, try deleting it if possible. Pos=28762, Size=4301, PrecacheSize=2147483647, PrecacheOffset=28762
UE4Editor.exe has triggered a breakpoint.

I try to import the same texture into 4.9.0 from the original source. The import succeeds. The original file being imported is not an issue, but migrating it from 4.7.6 is.

I will try to migrate from 4.7.6 to 4.8.0 and see if I get the same error, and try to migrate from 4.8.0 to 4.9.0 as well.

Test Asset migration status:

4.7.6 to 4.7.6 (mod) → Success

4.7.6 (mod) to 4.7.6 → Success

4.7.6 && 4.7.6 (mod) to 4.8.0 → Fail

4.8.0 to 4.9.0 → Success

4.7.6 && 4.7.6 (mod) to 4.9.0 → Fail

If I go into my 4.7.6 (mod) and copy the ObjectVersion.h and Version.h from 4.9.0 into it and recompile and resave my test asset, the binary file signature changes. When I try to import the modified binary file created in 4.7.6 into 4.9.0, it doesn’t crash anymore, but it doesn’t show up in my content browser and the editor consumes about 10gb of physical memory.

I have thousands of assets in my project. I’d like to update my engine to 4.9.0 so I can take advantage of the latest market place content and get updated bug fixes. It would take way too long to go through and manually recreate every single game asset (materials, material instances, blueprints, anim blueprints, behavior trees, etc). There’s gotta be a way to do this…

In case you want to replicate my efforts, this is the texture asset I’m using to test migration between different versions:

58150-testtexture.png

It sounds like the changes to your 4.7 engine have caused the assets to only be recognizable by that engine. The other possible solutions that come to mind would be to convert the project’s engine version from your modified 4.7 engine to an unmodified 4.7 engine. Doing so and then migrating the asset to 4.9 may help. Alternatively you could make the same edits to the 4.9 engine files that you made to your 4.7 files and then try to migrate from your 4.7 engine to 4.9.

uh… this hasn’t been resolved yet :frowning:

RE Engine: That’s what I thought as well so I tried to migrate a known good texture asset (from its original source) into the unmodified 4.7.6 engine (see above). Then I tried to migrate that asset from the unmodified 4.7.6 engine to an unmodified 4.9.0 engine and I was getting the same crashes. I also tried to migrate assets from my modified 4.7.6 engine to the unmodified engine, and that worked successfully (and vice versa). So, the long and short of it all is that I can’t move assets from 4.7.6 to either 4.8.0 or 4.9.0.

I’ve resigned myself to the idea that I may have to just manually recreate every asset if this takes too long. It’s gonna be a long weekend :frowning:

When you’re migrating an asset are you selecting the asset in a 4.7.6 project and choosing the migrate option into a 4.9 project? I ask because this is the process that I’ve tried with the sample texture you posted earlier. Also, are you able to make a copy of your 4.7.6 mod-engine project in 4.9? You can do this by opening the 4.9 engine and selecting your project which will prompt you to create a (4.9) copy of the project. Let me know if doing this allows you to open the project / see the textures in the content browser.

Cheers

Yes, I take my texture asset from 4.7.6 (vanilla) and migrate it to 4.9.0 (vanilla) and it fails. It usually says that the asset is corrupted or tries to take up an impossible amount of memory. Both versions of the engine are compiled from source code, if that matters. If you’re able to do it successfully with the same asset on your end, then that gets interesting. I think it might be interesting to compare the uasset binary files your 4.7.6 vanilla engine creates against the 4.7.6 binaries my vanilla engine creates and see if they match up (some file header meta data will be different though). I can also try to import your uasset into my 4.9.0 engine and see if that takes. This would help us isolate the source of the problem and focus in the right place.

One of the first things I tried was to create a copy of my 4.7.6 mod project in 4.9.0 and use the project conversion system. That caused the editor to crash and say my assets were corrupted (or it just caused my ram to spike beyond physical capacity and completely froze my computer, forcing a hard reboot).

I’ve double checked that migrating the asset from 4.7 source to 4.9 source worked for me. Can you let me know if you get the same corrupted asset errors if you are migrating an asset from a 4.7 binary project to 4.9 binary?

I have been able to successfully migrate from the 4.7.6 binary to the latest 4.9.1 binary. I expected this to work, so this gives me more fuel to run tests against. I will try to import from my 4.7.6 source build into the 4.7.6 binary. I’ll also redownload and rebuild the 4.7.6 source just as a sanity check. I’ll post back with updates.

Good news.

My sanity check proved fruitful. I’ve been able to isolate the problem to a single engine modification I made in the 4.7.6 source (and probably forgot about). I did a file and directory comparison against the 4.7.6 source I thought was unmodified against the 4.7.6 source I downloaded. The only difference was about 10 entries at the tail of ObjectVersion.h EUnrealEngineObjectUE4Version enum

After reverting to the original ObjectVersion.H, I was able to successfully import my test file from 4.7.6 to 4.9.1. What this means is that if I can get all of my assets to have the same signatures, I can successfully import them into 4.9.1

So, I have to figure out how to migrate my assets so that 4.9.1 can recognize them. I’m not 100% sure about how to do that yet. My initial thought is that I can copy the missing entries in the 4.7.6 ObjectVersion.h enums into the enums in the 4.9.1 ObjectVersion.h. The harder solution would be to figure out some way to go through each asset and update the serialized binary signature manually to reflect the original 4.7.6 ObjectVersion.h data. Any thoughts on the best approach for this?

Alright, this is crazy, but more good news…

I spent all day trying to isolate and figure this out, but I think I tentatively did it.

I took the original source build of 4.7.6 and imported my test texture successfully into 4.9.1
That’s good. It means I can migrate to a higher version of the engine.

I then used a file and folder comparison tool to see what the differences were between 4.7.6 and 4.7.6 (modified). The difference came down to a few small additions to ObjectVersion.h.

If I revert the ObjectVersion.H to the unmodified one, I have the exact same build as the original source and can migrate assets with no issue into 4.9.1. Adding those changes to 4.9.1 causes a crash when trying to migrate assets.

Okay, so I can’t modify 4.9.1 to accept my file migrations for 4.7.6. So, I need to figure out what changes in the serialized binary between the 4.7.6 modified engine uasset vs. 4.7.6 unmodified uasset. I run the engine source code which generates a broken uasset, save it to a folder. I then revert my engine source to the original source and generate an uasset which can be migrated. I use my file comparison tool to do a binary file compare, and there are three small sections in the file which have differences in byte values. I don’t know exactly what parts of the serialization process these bytes represent, but I can tell you that the engine always crashes on the file compression flag.

Anyways, I decided to go through the broken uasset and began changing one byte value at a time to the byte value which works. Each time, I’d import and try to crash the engine. Low and behold, the very first delta byte fixed the issue! If I change the byte value at 0x0000000C from BF to B2, my test asset migrates to 4.9.1 without a crash.

Now, let’s see if its consistent with all uasset files in the project.

I have solved the problem. It wasn’t easy, but I finally did it.

TL;DR: You have to go through every single uasset in your content library and change the byte value at 0xC from 0xBF or 0xB7 to 0xB2. Don’t do this manually, write a program to do it for you (download my C# script).

Here’s the list of byte values at 0xC and the corresponding action (NC = No Change):

0xBF → 0xB2

0xB7 → 0xB2

0x9D → NC

0x91 → NC

0x65 → NC

0x26 → NC

0xE8 → NC

0x56 → NC

0x3B → NC

0x8D → NC

0x69 → NC

Just to share with anyone else facing a content migration issue, here’s what I did:

  1. Create an original asset which you can import into your working project
  2. Import your asset into a known good engine build (unmodified source build) and attempt to migrate it to your target engine version.
  3. Import your asset into your engine build version which contains your game project. Migration to new engine should crash new engine.
  4. Create a test project in your mod engine and import your asset into it. Verify it crashes target engine. Save this asset to an external folder!
  5. Revert the engine source code to known good source code, then reimport asset into your same test project. It’s very important that you use the same project, because importing an asset into a different project will cause different meta data in the header of the file (such as paths).
  6. Now, you should have two assets from the same source. One crashes the target engine, one does not. Get a file comparison tool (I highly recommend “Beyond Compare”) and do a binary file comparison between the broken and good asset.
  7. Go through each byte difference in the good and bad asset and copy one byte at a time from the good asset to the bad asset. Some combination of bytes will cause it to become uncorrupted. Keep attempting to copy the corrupted asset into your target engine after each byte change. Eventually, it will work.
  1. Note which bytes you changed, then update those bytes in a copy of your content directory and attempt to import them into the target engine. You can manually just copy the *.uasset file.
  2. Look through every single asset in your content directory and note the bytes at the noted memory addresses. If its not a known bad byte, test it by copying the asset into the content directory of your target engine. If it crashes the engine, repeat step 7. If not, ignore the byte.
  3. Create a script which goes through every directory and subdirectory and updates the byte value at the known bad addresses if those byte values are a known bad byte value. Do this on a copy of your game content directory!

If you do this right, you should be able to fix and migrate thousands of corrupted files into a newer project. It will save you weeks of work and you won’t have to manually recreate every single asset.