x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Run automated testing from command line

Greetings, In BuildTools presentation (link) I found that I can run tests with command line. Can someone provide an example please?

Product Version: Not Selected
Tags:
more ▼

asked Oct 01 '14 at 12:38 PM in C++ Programming

avatar image

nikitablack
97 26 31 36

avatar image richdadam May 03 '19 at 10:57 PM

You can save presets which are saved into a config file. Can a saved preset be used in a command line invocation? It would be great to configure the list in the Session Frontend and then use it in a command line invocation. Using the name of the preset in the -execcmds is a fail.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

7 answers: sort voted first

My apologies I was wrong about the -execcmd being able to easily run all of the complex tests for an automation test. :-(

You'll wan to run with: -automationtests="Blueprints.Compile-On-Load"

The couple of issue with that is the editor/game will exit once the automation test que is empty which is fine if that is what you want it to do. This also may or may not run all of the available complex tests.

I've logged these issues into our database to hopefully make this easier to use.

To get a full name of a complex test you can run the editor or game with -Execcmds="Automation List" or in the editors console window entering Automation List. That will give you the full list.

more ▼

answered Nov 14 '14 at 09:15 PM

avatar image Fringebits Aug 11 '16 at 01:12 AM

Any chance there is some documentation on -AutomationTests? I was hoping to find some details on it here (https://docs.unrealengine.com/latest/INT/Programming/Basics/CommandLineArguments/index.html).

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

You are correct Mike. Recently there have been some minor changes to how automation is run through the command line.

"-ExecCmds="Automation Run FCompileBlueprintsTest""

In this example you'll need to also specify the asset that is being tested. You can easily find the whole path for that file by running Automation List either as an execcmd or in the console window. The print out will be in your output log.

"-AutomationTests="Blueprints.Compile Blueprints""

Even though the automationtests= command line can use partial arguments you'll need to include everything before the test name itself. For example if I just wanted to run all of the blueprints it would look like this: "-automationtests="Project.Blueprints" or if I wanted to run compile all blueprints that start with the letter A I would use this: -automationtests="Project.Blueprints.Compile Blueprints.A".

Heads up though, this will be changing in the next release. For the better. :-)

more ▼

answered Jun 22 '15 at 06:10 PM

avatar image Samlsewall Jan 11 '16 at 05:02 PM

Hey just a heads up, I do not believe that your answer works with the current version of Unreal.

I believe it was updated, but this is what I've found to work.

What you want to do is to call the following for specific tests from your Engine/Binaries/Win64 directory (or whatever directory your UE4Editor.app... etc is in):

 UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests [TestPrettyName]" -log

In order to get all of the Pretty names for the tests run the following command:

 UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation List" -log


You can call multiple tests using this method if you have a Pretty name that encapsulates multiple tests. for Example:

I have the following tests under my AI code: "TytoDome.AI.2Animals1Plant", "TytoDome.AI.Save The Animals", "TytoDome.AI.Animal Hunting", "TytoDome.AI.Fish Eating". As well as the following tests in my Data Tool: "TytoDome.Data Tool.Unique EcoRole Collection", "TytoDome.Data Tool.Zone Birth and Death and Unlocking", "TytoDome.Data Tool.Animal Collection General".

If I wanted to call EVERY test I have, I would use the following command:

 UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests TytoDome" -log

If I wanted to call ONLY my AI Tests I'd run the following:

 UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests TytoDome.AI" -log

If I wanted to call ONLY my Animal hunting Test I'd run the following:

 UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests TytoDome.AI.Animal Hunting" -log

From here anyone should be able to figure out how to get this to work with their project!

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

To run unit tests from the command line, log to a file (and have a log popup), and exit after the tests, here is what I use:

 UE4Editor.exe Path/To/MyProject.uproject -ExecCmds="Automation RunTests MyProject" -unattended -nopause -testexit="Automation Test Queue Empty" -log -log=RunTests.log

I wanted to also use -NullRHI to avoid seeing the Editor (and the associated Slate load) but this presently crashs UE4.20 at exit

Edit: also, I've updated the command line to get a Tests Report using the option -ReportOutputPath. Here is our current complete commande line:

 UnrealEngine\Engine\Binaries\Win64\UE4Editor.exe "C:\workspace\MyProject\MyProject.uproject" -ExecCmds="Automation RunTests MyProject" -Unattended -NullRHI -TestExit="Automation Test Queue Empty" -ReportOutputPath="C:\workspace\MyProject\Saved\TestReport" -Log -Log=RunTests.log
more ▼

answered Oct 24 '18 at 06:11 PM

avatar image

SRombauts
1.1k 22 17 97

avatar image Zoubi Nov 05 '18 at 10:25 AM

Do you manage to see which tests fail?

I purposely made one test fail. When I run the tests in the editor, the logs state the failure. When I run the tests from the command line, there is no mention of it :/

avatar image SRombauts Nov 06 '18 at 12:50 PM

Yes, in case of failure I do get a error log. I will also make a script to parse the index.json report

avatar image Zoubi Nov 06 '18 at 02:27 PM

Thanks. I found out the issue I had, which is a bug (now fixed in 4.21): when you run the tests and the session frontend is opened, no test is run. Now that it works in the command line, I need to find out why in my Jenkins, the editor tries to compile all the shaders when it did not have to when I cooked the game data...

avatar image SRombauts Nov 06 '18 at 02:42 PM

Ok, good to know! Also, I would like to know how you parse the results. I am currently using the Warnings Plugin but it does not fail the build as it should using thresholds. At the end of the day, the real proper method IMO is to convert the json into JUnit test result so that the proper Jenkins plugin can display all appropriate info

avatar image Zoubi Nov 13 '18 at 12:53 PM

I managed to do just that, and wrote a blog post about it : http://www.emidee.net/ue4/2018/11/13/UE4-Unit-Tests-in-Jenkins.html

avatar image Nicowrnch Jan 02 '19 at 04:05 PM

adding small note:

The opened session frontend issue also affects 4.16 so most likely every version above until 4.21

avatar image Zoubi Nov 06 '18 at 02:52 PM

Right now I did not have a chance to go that far since I do not have (yet!) the unit tests results in the log but my plan was to use the warnings plugin, with a custom regexp, and to fail the build if any test fails

avatar image SRombauts Nov 06 '18 at 03:25 PM

I am doing that using Jenkins Pipeline with the following config:

 warnings failedTotalAll: '0', parserConfigurations: [[parserName: 'UnrealTestReports', pattern: 'Saved\\TestReport\\index.json']]

With a very basic regular expression to catch the global status (so does not count how many failures):

 Regular Expression: ^\s*"state": "Fail",$
avatar image Zoubi Nov 07 '18 at 10:54 AM

I had a look and indeed the cleanest way would be to comvert the JSON file from the report into a valid jUnit XML file. I quickly looked for any plugin which would help do that but did not find any. So for now, I don't export the report, and I configured a Warnings task to parse the log.

Here is the regexp: ^\s*.]AutomationTestingLog: Error:\s(.)((\d+)): (.)(.*)$

And the mapping script: import hudson.plugins.warnings.parser.Warning import hudson.plugins.analysis.util.model.Priority

 String category = "Unit Tests"
 String fileName = matcher.group( 1 )
 String lineNumber = matcher.group( 2 )
 String message = matcher.group( 3 )
 Priority priority = Priority.HIGH;

 return new Warning( fileName, Integer.parseInt(lineNumber), "UE4", category, message, priority );
avatar image Zoubi Nov 06 '18 at 07:08 PM

How do you run the command line in your jenkins pipeline? using the bat command? I have this:

 stage( "UnitTests" ) {
     try {
         bat "\"${env.UE_ROOT_FOLDER}\\Engine\\Binaries\\Win64\\UE4Editor-Cmd.exe\" \"${env.WORKSPACE}\\${env.PROJECT_NAME}.uproject\" -ExecCmds=\"Automation RunTests ${env.PROJECT_NAME}\" -unattended -nopause -testexit=\"Automation Test Queue Empty\" -log -log=RunTests.log"
     } catch ( Exception e ) {
     }
 }

but before the editor tries to run the unit tests, I have this error:

 [2018.11.06-18.45.01:561][  0]LogD3D11RHI: Error: D3DRHI->GetFactory()->CreateSwapChain(DXGIDevice, &SwapChainDesc, SwapChain.GetInitReference()) failed 
 at D:\Build\++UE4\Sync\Engine\Source\Runtime\Windows\D3D11RHI\Private\Windows\WindowsD3D11Viewport.cpp:126 
 with error 887A0022

I guess that's because the editor is created in the bat process and does not have a chance to create the RHI (I tried with -NullRHI, with the same error)

avatar image SRombauts Nov 06 '18 at 08:13 PM

I am running it from a batch script, containing the command line I posted above:

     stage("Run Unit Tests") {
         steps {
             bat 'Tools\\BuildSystem\\run_unit_tests.bat'
         }
     }

Your problem might be that your Jenkins build machine needs to have a DirectX 11 GPU, and also perhaps a user session open. -NullRHI should solve this, I think, but it crashed on UE4.20, I had to backport a fix from 4.21 to have it work

avatar image Zoubi Nov 06 '18 at 09:05 PM

OK I found the issue. The slave was connected to the master from a Windows Service, which does not have the rights to create the DX11 swap chain. When connecting using the JLNP file, it does work :)

I'll have a look at the warnings regexp tomorrow :)

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

Hello,

To run all of the automation tests in a project: UE4Editor.exe GAMENAME -Game -ExecCmds="Automation RunAll"

or

To run a single test: UE4Editor.exe GameName -game -execcmds="Automation Run FSetRestTest"

You can find the list of tests by running -execcmds="Automation List". In the output log you'll see a long list of tests. Each test name has the letter "F" in front of it.

Here's some more info on the automation system:

Creating a test: https://docs.unrealengine.com/latest/INT/Programming/Automation/TechnicalGuide/index.html

Overview: https://docs.unrealengine.com/latest/INT/Programming/Automation/index.html

more ▼

answered Oct 06 '14 at 03:40 PM

avatar image thk123 Nov 12 '14 at 03:37 PM

How would you execute a group of tests (or indeed even a complex test like FCompileBlueprintsTest) Obviously don't want to have to list in the command line all the Blueprints... Alternatively would be a way to trigger a preset collection of auto tests?

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

To run a preset collection through command line is not currently possible but it is on our list of things to do.

You can run the individual tests from a complex test or you can run 'em all.

To run an individual test use this command argument:

-AutomationTests="Pretty.TestName"

Example:

-AutomationTests="Blueprints.Compile Blueprints.My Awesome Blueprint, "Blueprints.Compile Blueprints.MyOKBlueprint"

To run all of a complex test then use:

-Execcmds="Automation Run FCompileBlueprintsTest"

Does this answer your question?

more ▼

answered Nov 12 '14 at 06:15 PM

avatar image thk123 Nov 13 '14 at 10:36 AM

When I run with -Execcmds="Automation Run FCompileBlueprintsTest" I get

ERROR: Failed to find test FCompileBlueprintsTest

I have double checked the name of the test (copied and pasted from a list command). The only way I can get it to work is by specifying a specific BP. We are using UE4.4.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

By chance, are you attempting to run the blueprint test in the game? That test is flagged as Editor only so it won't work if you run it in game.

more ▼

answered Nov 13 '14 at 01:58 PM

avatar image thk123 Nov 14 '14 at 06:07 PM

I am not running with the -game tag and it correctly launches the editor.

I'm not sure how I ever got -Execcmds="Automation Run FCompileBlueprintsTest /some/path/to/blueprint" from the command line to work as even that doesn't work (even that says can't find test when the editor launches).

However, if I run Automation Run FCompileBlueprintsTest /some/path/to/blueprint in the command line from the Session Frontend, that works. But even then, running Automation Run FCompileBlueprintsTest says it can't find that test.

avatar image Mike Kelly Jun 20 '15 at 03:52 AM

-ExecCmds="Automation Run FCompileBlueprintsTest" does not work. -AutomationTests="Blueprints.Compile Blueprints" does not work. But -AutomationTests="Project.Blueprints.Compile Blueprints" does work. That's the PrettyName for FCompileBlueprintsTest, as defined in BlueprintAutomationTests.cpp.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

I've tested, command line "Automation RunTests Project.Blueprints.CompileBlueprints" does work, there's no space between Compile and Blueprints, hope this can help you

more ▼

answered Jul 10 '17 at 05:46 AM

avatar image

Zhenquan
1

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question