How do you run and get output of automation tests?

Now, the simple answer you might be inclined to suggest is:

UE4Editor MyProject.uproject -Game -ExecCmds="Automation RunTests MyTest" -log=Log.txt

However, the reason I ask this question is because this command patently (despite being the answer given in numerous places) Does Not Work.

Here are the things it does:

  • Launch the game

Here are the things it does not do:

  • Run ‘MyTestName’
  • Log output to MyLogFile.txt

(See, I know it should write a log file to Saved/Logs/ but tangibly it does not).

You can certainly run this to execute the test in editor mode and see the output log in the editor:

UE4Editor MyProject.uproject -ExecCmds="Automation RunTests MyTestName"

(notice the absence of the -Game flag, which causes the editor to open, rather than game mode to open).

You can also run:

UE4Editor MyProject.uproject -Game -ExecCmds="Automation RunTests MyTestName" -log

Which pops up the log window and shows commands being run…

…and sits there, on the log message:

[2016.02.11-15.59.15:599][  0]LogLoad: Full Startup: 5.23 seconds (BP compile: 0.01 seconds)
[2016.02.11-15.59.15:671][  2]LogRenderer: Reallocating scene render targets to support 1280x720 (Frame:2).
[2016.02.11-15.59.20:653][259]LogAssetRegistry: FAssetRegistry took 0.0004 seconds to start up

Where is my automation test?

Obviously in the ideal world I wouldn’t have to do this at all, but as per Hot-reload does not re-compile Automation Tests - Programming & Scripting - Unreal Engine Forums, tests aren’t hot reloaded, so they’re impossible to use without closing and opening the editor.

Overall: LAME! This automation stuff seems really rubbish compared to GitHub - google/googletest: GoogleTest - Google Testing and Mocking Framework

Has anyone actually managed to get this working?

How do you:

  1. Run tests

  2. Get the results

  3. Write new tests and repeat (1) and (2) without manually closing and reopening the editor.

Actually, it seems to run the tests, but it just doesn’t tell you which ones failed and which ones succeeded. Whenever I run the tests with the following command: UE4Editor MyProject.uproject -ExecCmds="Automation RunTests MyTest" -log=Log.txt -testexit="Automation Test Queue Empty". And this is part of the log I get. Even when a test is failing. Still broken in 4.16.

[2017.07.21-13.29.38:345][  2]Found 237 Automation Tests
[2017.07.21-13.29.43:250][577]AutomationTestingLog: New page: -----Test Run 8----
[2017.07.21-13.29.43:250][577]AutomationTestingLog: Info -----Test Run 8----
[2017.07.21-13.29.43:252][577]...Automation Test Queue Empty 1 tests performed.