Automation tests from command line don't run if AutomationWindow is open

Development Programming & Scripting
1

I tried to run my tests with the following command:

UE4Editor.exe MyProject.uproject -ExecCmds="Automation RunTests MyCategory" LOG="mylog.log" -testexit="Automation Test Queue Empty"

I found that the tests weren’t running, although it said it was. The log was empty and only said that it completed the tests.

After debugging through the source I found out that the Automation Window was blocking my plans. SAutomationWindow is opened by default on my editor and I saved the layout for that. When I run the tests, the function FAutomationControllerManager::SetEnabledTests is called too often and resets my list of tests coming through via “RunTests MyCategory”.

So the code flow is like this:

  • The command line sets the enabled tests via SetEnabledTests
  • The editor opens SAutomationWindow and calls OnRefreshTestCallback(), which sets all tests back to disabled.
  • Tests start running, but the list is actually empty.
  • The log file still reports that tests have run, but they actually haven’t and there is no information in the log.

I have a workaround now by removing the automation window from my default layout, but it’s not a fix. Please fix this issue, so that it will be possible to run the tests with the automation window open.

2

Still happening in 4.17
Basically blocking any automation of the AutomationTesting since we want to habve both, the visualized testing at work time and also the automated testing when running the automation scripts.