x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Cannot save or change assets after running a automation test?

Hey so recently our project has grown to the point where I need to rely on automation tests to make sure that I do not break assets while working in parts of the game for the most part AI behaviour trees. As such I have implemented some Automation Tests for some of the core tasks for our ai, such as spawning in a world, some units suffering damage, and a single unit with the healing ability to test how he responds etc. I can easily perform situational testing then, debugging blueprints etc. and its fantastic. However after running a automation test the ability to save any assets does not work, neither the Ctrl+s shortcut nor the in editor save button. I am using the perforce plugin for my project, and my tests use the following to queue up the test environments

 IMPLEMENT_COMPLEX_AUTOMATION_TEST(FHealingTest, "MyGame.HealingTest", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter )
 
 DEFINE_LATENT_AUTOMATION_COMMAND( FCreateWorld );
 DEFINE_LATENT_AUTOMATION_COMMAND( FSpawnHexLayout );
 DEFINE_LATENT_AUTOMATION_COMMAND( FWaitForInitialResourceSpawn );
 DEFINE_LATENT_AUTOMATION_COMMAND( FCreateBasicCamp);
 DEFINE_LATENT_AUTOMATION_COMMAND( FCleanupWorkers );
 DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FExecStringCommand, FString, ExecCommand );
 DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FAssignWorkerStates, FString, ExecCommand );
 DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FCheckForPassFail, FString, ExecCommand );
 
 UWorld* g_world = nullptr;
 AMyGameState* g_game_state = nullptr;
 AMyGameMode* g_game_mode = nullptr;
 
 void
 FHealingTest::GetTests( TArray<FString>& OutBeautifiedNames, TArray<FString>& OutTestCommands ) const
 {
     OutBeautifiedNames.Add( FString( "Healing Sick" ) );
     OutBeautifiedNames.Add( FString( "Healing Low Health" ) );
     OutBeautifiedNames.Add( FString( "Healing Diseased" ) );
     OutBeautifiedNames.Add( FString( "Cleansing Dead" ) );
 
     OutTestCommands.Add( FString( "Sick" ) );
     OutTestCommands.Add( FString( "Low" ) );
     OutTestCommands.Add( FString( "Disease" ) );
     OutTestCommands.Add( FString( "Cleanse" ) );
 }
 bool
 FHealingTest::RunTest( const FString& Param )
 {
     SetSuppressLogs( true );
     ADD_LATENT_AUTOMATION_COMMAND( FCreateWorld() );
 
     ADD_LATENT_AUTOMATION_COMMAND( FSpawnHexLayout() );
     ADD_LATENT_AUTOMATION_COMMAND( FWaitForInitialResourceSpawn() );
     SetSuppressLogs( false );
 
     ADD_LATENT_AUTOMATION_COMMAND( FCreateBasicCamp() );
     ADD_LATENT_AUTOMATION_COMMAND( FAssignWorkerStates( Param ) );
     ADD_LATENT_AUTOMATION_COMMAND( FWaitLatentCommand( 360.0f ) );
 
     ADD_LATENT_AUTOMATION_COMMAND( FCheckForPassFail( Param ) );
     ADD_LATENT_AUTOMATION_COMMAND( FCleanupWorkers );
 
     ADD_LATENT_AUTOMATION_COMMAND( FExecStringCommand( TEXT( "quit" ) ) );
     return true;
 }
 
 bool
 FCreateWorld::Update()
 {
     AutomationOpenMap( TEXT( "Procedural/BaseMap" ) );
 
     if ( GEngine == nullptr )
     {
         UE_LOG( LogEngineAutomationTests, Log, TEXT( "GEngine pointer is null" ) );
         return false;
     }
 
     const TIndirectArray<FWorldContext>& WorldContexts = GEngine->GetWorldContexts();
     for ( const FWorldContext& Context : WorldContexts )
     {
         if ( ( ( Context.WorldType == EWorldType::PIE ) || ( Context.WorldType == EWorldType::Game ) ) && ( Context.World() != NULL ) )
         {
             g_world = Context.World();
             break;
         }
     }
 
     if ( g_world == nullptr )
     {
         UE_LOG( LogEngineAutomationTests, Log, TEXT( "Cannot find a valid Game World." ) );
         return false;
     }
     g_game_state = Cast<AMyGameState>( g_world->GetGameState() );
     if ( g_game_state == nullptr )
     {
         UE_LOG( LogEngineAutomationTests, Log, TEXT( "Incorrect game state loaded" ) );
         return false;
     }
     g_game_mode = Cast<AMyGameMode>( g_world->GetAuthGameMode() );
     if ( g_game_mode == nullptr )
     {
         UE_LOG( LogEngineAutomationTests, Log, TEXT( "Incorrect game mode loaded" ) );
         return false;
     }
 
     return true;
 }

For the most part the most complex thing I do is use the AutomationOpenMap() to open a map which launches a Standalone Editor window, then I use the GEngine world contexts to spawn some units in as I would normally. Atm I don't SetSuccess on the tests (I really should but I'm currently not) and I will quite often stop tests midway manually once I identify a problem I need to resolve. Anyway I currently have to restart the editor apply a fix and retest, would love to know if theres anything I can do to remove the 'restart the editor' step from my bug fixing loop.

Thanks ahead of time. :D

Product Version: UE 4.17
Tags:
more ▼

asked Jan 03 '18 at 03:11 AM in C++ Programming

avatar image

Wolflight
46 4 7 8

avatar image Adr1c ♦♦ STAFF Jan 03 '18 at 03:21 PM

Hi Wolflight,

I'm not seeing anything in your code that I'd expect to affect the save system.

How are you running the tests, and how are you stopping them?

Side notes: I'd recommend taking out the final FExecStringCommand(TEXT("quit")), and handle exiting elsewhere, since the current implementation would cause the editor to exit when any of the tests complete.

SetSuccess is optional, and will by default be set by the test framework based on whether any errors/warnings have been logged during the test run.

I suggest not caching the World, GameState, and GameMode for latent commands, and just Get them instead when necessary. Speaking from experience, there's a non-zero risk they'll get nulled between frames, which will cause Bad Things™ if you're not always checking them before use.

avatar image Wolflight Jan 03 '18 at 11:16 PM

Hah! turns out calling quit has been doing nothing anyway because I should have been calling the actual command line 'Exit' of which was causing issues and has since been commented out. The World, GameState and GameMode, I kinda found a workaround for that exact problem you are talking about with a null ptr every now and then and so I had to find a solution (not exactly your suggestion but close enough that I think its fine, I redefined the Latent class macros to have some extra functionality). Atm I am running the tests by selecting them from the Session Frontend window in the Editor usually just a single test at a time selected and run from the Automation tab. Stopping them I have been hitting stop tests then closing the standalone window, but in either order will still cause the issue. The saving issue doesn't apply to assets within my levels, I can move objects round in the level and save them fine. Anything like a Actor Blueprint, Behaviour Tree, DataTable, StaticMesh, nearly anything that opens a seperate tab for editing will not save though.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

1 answer: sort voted first

I was able to repro this, and have entered it as UE-53777. From what I've been able to determine, stopping the test framework in the middle of a long-running latent command blocks the save functionality. Until this is fixed, you should be able to mitigate this be restructuring your tests to return as soon as the behavior under test can be verified instead of relying on extremely long waits.

I strongly recommend against editing the automation test macros. If you need behavior that the macros don't provide, it would be better to create a custom subclass of FAutomationTestBase instead (see FClientFunctionalTestingMaps in ClientFuncTestPerforming.cpp as an example).

more ▼

answered Jan 05 '18 at 04:31 PM

avatar image

Adr1c ♦♦ STAFF
910 18 7 18

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question