x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Automation Testing On Binary Engine is Broken

Automation test includes feel like they're completely broken in 4.9.

Problem 1: Including "AutomationEditorCommon.h" breaks build

Just by including this file, my build breaks on AutomationEditorCommon.h:23 because it can't find GLevelEditorModeTools(). This function is defined in UnrealEd.h...

Shouldn't AutomationEditorCommon.h include UnrealEd.h as a dependency?

Problem 2: Including UnrealEd.h and calling FEditorAutomationTestUtilities::LoadMap results in linker failure

The linker can't find FEditorFileUtils::LoadMap

This function is defined in FileHelpers.h but the module is not getting built.

Is Automation Testing simply not designed for the end binary engine user? As a base, what modules do I need to include in my game to get this stuff working?

Product Version: UE 4.9
Tags:
more ▼

asked Sep 16 '15 at 08:33 PM in Bug Reports

avatar image

hyperdr1ve
593 28 51 118

avatar image Adr1c ♦♦ STAFF Sep 16 '15 at 09:37 PM

Hi hyperdr1ve,

Could you provide more info about what kind of tests you're setting up and where in your project you're trying to include the header? AutomationEditorCommon is aimed at testing the main editor functionality, so it's likely that you're either trying to do something that isn't supported by the Binary version (changing engine-side code), or AutomationEditorCommon isn't the right tool for your needs.

avatar image hyperdr1ve Sep 16 '15 at 10:41 PM

Pretty simple really. Copy code for FSetResTest straight from documentation. Hit compile, compile fails because it can't find a function defined in AutomationEditorCommon.h. I'm including AutomationEditorCommon.h directly in the test.

avatar image hyperdr1ve Sep 16 '15 at 10:47 PM

The kind of tests I'm trying to setup are somewhat secondary to just getting a basic test compiling. There aren't any test samples to scour but I'll need to load a specific map, so the sample from the docs seems like a good starting place.

avatar image Adr1c ♦♦ STAFF Sep 17 '15 at 01:44 AM

Those docs are pretty out-of-date. The current version of that test does not use the FEngineAutomationTestUtilities::LoadMap function. Additionally, FEngineAutomationTestUtilities has been renamed FEditorAutomationTestUtilities. Also, most of the methods in AutomationEditorCommon have been exported to the API, so they won't be available outside of the UnrealEd module without making engine-level changes.

The most bare minimum compiling and passing "test" you can get is an IMPLEMENT_SIMPLE_AUTOMATION_TEST macro with a matching RunTest implementation that returns true:

 IMPLEMENT_SIMPLE_AUTOMATION_TEST(FMySimpleTest, "Project.My Simple Test", ATF_Editor | ATF_Game)
 
 bool FMySimpleTest::RunTest(const FString& Parameters)
 {
     return true;
     //TestTrue(TEXT("True is always..."), true); // Uncomment this line to run an actual "test"
 }
avatar image hyperdr1ve Sep 17 '15 at 02:16 AM

So, don't trust anything you see in the official docs, check ;) So how would I get a pointer to the world to get my custom UGameInstance? Or can I manually construct UGameInstance or... what? It's the code in there that I'm trying to write tests for.

avatar image Adr1c ♦♦ STAFF Sep 17 '15 at 02:35 AM

This should get the current game instance:

 UWorld* InWorld = GEngine->GetWorld();
 InWorld->GetGameInstance();
avatar image hyperdr1ve Sep 17 '15 at 04:51 AM

I gave this a try. GEngine->GetWorld() always returns a nullptr. I tried with a level open in the editor and also while PIE. This is with the test defined as ATF_Editor. The full test:

 IMPLEMENT_SIMPLE_AUTOMATION_TEST(FPersistentSaveTest, "Arconia.PersistentSave", EAutomationTestFlags::ATF_Editor)
 
 bool FPersistentSaveTest::RunTest(const FString& Parameters)
 {
     if (GEngine)
     {
         if (GEngine->GetWorld())
         {
             if (GEngine->GetWorld()->GetGameInstance())
             {
                 return true;
             }
         }
         else
         {
             AddError(FString::Printf(TEXT("Getworld is null")));
         }
     }
     else
     {
         AddError(FString::Printf(TEXT("GEngine is null")));
     }    
     return false;
 }

I feel like I'm missing a fundamental step here. If I have the world, I can do pretty much anything I need.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

1 answer: sort voted first

Sorry for the bad info earlier; I'm still getting familiar with the in-game side of this.

Below is a version of the test that compiles and runs for me in game, not editor. Getting it to work in-editor will take more legwork, as you'll need to make sure you've got a game running with a map loaded(eg. PIE), then iterate thru the world contexts to make sure you're checking the right one. I've also reorganized it to use the built-in test-class test methods and removed the unnecessary Printf's.

To test this, run your project with -game and -messaging flags (eg UE4Editor.exe "Path\To\Your\Project.uproject" -game -messaging), then open SessionFrontend > Automation tab and select the game session.

 IMPLEMENT_SIMPLE_AUTOMATION_TEST(FPersistentSaveTest, "CodeTest.PersistentSave", EAutomationTestFlags::ATF_Game )
 
 bool FPersistentSaveTest::RunTest(const FString& Parameters)
 {
     TestNotNull(TEXT("GEngine is not null"), GEngine);
     if (GEngine)
     {
         // Verify only one world exists
         TestEqual(TEXT("GetWorldContexts returns 1"), GEngine->GetWorldContexts().Num(), 1);
         if (GEngine->GetWorldContexts().Num() == 1)
         {
             TestNotNull(TEXT("GetGameInstance is not null"), GEngine->GetWorldContexts()[0].World()->GetGameInstance());
         }
     }
     return true; // Any Test{Result}() failures will override this value
 }
more ▼

answered Sep 17 '15 at 01:45 PM

avatar image

Adr1c ♦♦ STAFF
910 18 7 18

avatar image hyperdr1ve Sep 17 '15 at 05:21 PM

Thanks Adric, you got me to the right place. I've been poking around in Engine.h and am using the following to test while PIE. I can now proceed with the real testing. IMPLEMENT_SIMPLE_AUTOMATION_TEST(FPersistentSaveTest, "Arconia.PersistentSave", EAutomationTestFlags::ATF_Editor)

 UWorld* GetWorld()
 {
     if (GEngine)
     {
         if (FWorldContext* World = GEngine->GetWorldContextFromPIEInstance(0))
         {
             return World->World();
         }
     }
     return nullptr;
 }
 
 UGameInstance* GetGameInstance()
 {
     if (UWorld* World = GetWorld())
     {
         return World->GetGameInstance();
     }
 
     return nullptr;
 }
 
 bool FPersistentSaveTest::RunTest(const FString& Parameters)
 {
     UGameInstance* GameInstance = GetGameInstance();
     TestNotNull(TEXT("GameInstance is not null"), GameInstance);
 
     if (GameInstance)
     {
         // test    
     }
     return true;
 }

Would you be able to file a bug against the automation technical guide? I'd like there to be some official piece of documentation that I can send SDETs to to get started testing a game in UE4, right now this answerhub post is the best source of relevant information.

avatar image Adr1c ♦♦ STAFF Sep 17 '15 at 05:56 PM

Glad to hear that's working for you now. And yes, I'll file a ticket to get our docs updated and expanded.

avatar image Bogustus Mar 03 '16 at 05:44 PM

Any update on this. It's still a real struggle to get automated testing set up for a game.

avatar image Adr1c ♦♦ STAFF Mar 03 '16 at 06:02 PM

Hi Bogustus,

The ticket for updating the technical guide has been triaged and assigned, but no work has been recorded for it. Sorry for the inconvenience.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question