Cannot save or change assets after running a automation test?

Hey so recently our project has grown to the point where I need to rely on automation tests to make sure that I do not break assets while working in parts of the game for the most part AI behaviour trees. As such I have implemented some Automation Tests for some of the core tasks for our ai, such as spawning in a world, some units suffering damage, and a single unit with the healing ability to test how he responds etc. I can easily perform situational testing then, debugging blueprints etc. and its fantastic. However after running a automation test the ability to save any assets does not work, neither the Ctrl+s shortcut nor the in editor save button. I am using the perforce plugin for my project, and my tests use the following to queue up the test environments

IMPLEMENT_COMPLEX_AUTOMATION_TEST(FHealingTest, "MyGame.HealingTest", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter )

DEFINE_LATENT_AUTOMATION_COMMAND( FCreateWorld );
DEFINE_LATENT_AUTOMATION_COMMAND( FSpawnHexLayout );
DEFINE_LATENT_AUTOMATION_COMMAND( FWaitForInitialResourceSpawn );
DEFINE_LATENT_AUTOMATION_COMMAND( FCreateBasicCamp);
DEFINE_LATENT_AUTOMATION_COMMAND( FCleanupWorkers );
DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FExecStringCommand, FString, ExecCommand );
DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FAssignWorkerStates, FString, ExecCommand );
DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER( FCheckForPassFail, FString, ExecCommand );

UWorld* g_world = nullptr;
AMyGameState* g_game_state = nullptr;
AMyGameMode* g_game_mode = nullptr;

void
FHealingTest::GetTests( TArray<FString>& OutBeautifiedNames, TArray<FString>& OutTestCommands ) const
{
	OutBeautifiedNames.Add( FString( "Healing Sick" ) );
	OutBeautifiedNames.Add( FString( "Healing Low Health" ) );
	OutBeautifiedNames.Add( FString( "Healing Diseased" ) );
	OutBeautifiedNames.Add( FString( "Cleansing Dead" ) );

	OutTestCommands.Add( FString( "Sick" ) );
	OutTestCommands.Add( FString( "Low" ) );
	OutTestCommands.Add( FString( "Disease" ) );
	OutTestCommands.Add( FString( "Cleanse" ) );
}
bool
FHealingTest::RunTest( const FString& Param )
{
	SetSuppressLogs( true );
	ADD_LATENT_AUTOMATION_COMMAND( FCreateWorld() );

	ADD_LATENT_AUTOMATION_COMMAND( FSpawnHexLayout() );
	ADD_LATENT_AUTOMATION_COMMAND( FWaitForInitialResourceSpawn() );
	SetSuppressLogs( false );

	ADD_LATENT_AUTOMATION_COMMAND( FCreateBasicCamp() );
	ADD_LATENT_AUTOMATION_COMMAND( FAssignWorkerStates( Param ) );
	ADD_LATENT_AUTOMATION_COMMAND( FWaitLatentCommand( 360.0f ) );

	ADD_LATENT_AUTOMATION_COMMAND( FCheckForPassFail( Param ) );
	ADD_LATENT_AUTOMATION_COMMAND( FCleanupWorkers );

	ADD_LATENT_AUTOMATION_COMMAND( FExecStringCommand( TEXT( "quit" ) ) );
	return true;
}

bool
FCreateWorld::Update()
{
	AutomationOpenMap( TEXT( "Procedural/BaseMap" ) );

	if ( GEngine == nullptr )
	{
		UE_LOG( LogEngineAutomationTests, Log, TEXT( "GEngine pointer is null" ) );
		return false;
	}

	const TIndirectArray<FWorldContext>& WorldContexts = GEngine->GetWorldContexts();
	for ( const FWorldContext& Context : WorldContexts )
	{
		if ( ( ( Context.WorldType == EWorldType::PIE ) || ( Context.WorldType == EWorldType::Game ) ) && ( Context.World() != NULL ) )
		{
			g_world = Context.World();
			break;
		}
	}

	if ( g_world == nullptr )
	{
		UE_LOG( LogEngineAutomationTests, Log, TEXT( "Cannot find a valid Game World." ) );
		return false;
	}
	g_game_state = Cast<AMyGameState>( g_world->GetGameState() );
	if ( g_game_state == nullptr )
	{
		UE_LOG( LogEngineAutomationTests, Log, TEXT( "Incorrect game state loaded" ) );
		return false;
	}
	g_game_mode = Cast<AMyGameMode>( g_world->GetAuthGameMode() );
	if ( g_game_mode == nullptr )
	{
		UE_LOG( LogEngineAutomationTests, Log, TEXT( "Incorrect game mode loaded" ) );
		return false;
	}

	return true;
}

For the most part the most complex thing I do is use the AutomationOpenMap() to open a map which launches a Standalone Editor window, then I use the GEngine world contexts to spawn some units in as I would normally. Atm I don’t SetSuccess on the tests (I really should but I’m currently not) and I will quite often stop tests midway manually once I identify a problem I need to resolve. Anyway I currently have to restart the editor apply a fix and retest, would love to know if theres anything I can do to remove the ‘restart the editor’ step from my bug fixing loop.

Thanks ahead of time. :smiley:

Hi Wolflight,

I’m not seeing anything in your code that I’d expect to affect the save system.

How are you running the tests, and how are you stopping them?

Side notes:
I’d recommend taking out the final FExecStringCommand(TEXT(“quit”)), and handle exiting elsewhere, since the current implementation would cause the editor to exit when any of the tests complete.

SetSuccess is optional, and will by default be set by the test framework based on whether any errors/warnings have been logged during the test run.

I suggest not caching the World, GameState, and GameMode for latent commands, and just Get them instead when necessary. Speaking from experience, there’s a non-zero risk they’ll get nulled between frames, which will cause Bad Things™ if you’re not always checking them before use.

Hah! turns out calling quit has been doing nothing anyway because I should have been calling the actual command line ‘Exit’ of which was causing issues and has since been commented out. The World, GameState and GameMode, I kinda found a workaround for that exact problem you are talking about with a null ptr every now and then and so I had to find a solution (not exactly your suggestion but close enough that I think its fine, I redefined the Latent class macros to have some extra functionality). Atm I am running the tests by selecting them from the Session Frontend window in the Editor usually just a single test at a time selected and run from the Automation tab. Stopping them I have been hitting stop tests then closing the standalone window, but in either order will still cause the issue. The saving issue doesn’t apply to assets within my levels, I can move objects round in the level and save them fine. Anything like a Actor Blueprint, Behaviour Tree, DataTable, StaticMesh, nearly anything that opens a seperate tab for editing will not save though.

I was able to repro this, and have entered it as UE-53777. From what I’ve been able to determine, stopping the test framework in the middle of a long-running latent command blocks the save functionality. Until this is fixed, you should be able to mitigate this be restructuring your tests to return as soon as the behavior under test can be verified instead of relying on extremely long waits.

I strongly recommend against editing the automation test macros. If you need behavior that the macros don’t provide, it would be better to create a custom subclass of FAutomationTestBase instead (see FClientFunctionalTestingMaps in ClientFuncTestPerforming.cpp as an example).