x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Gauntlet Automation Framework documentation

Included in the UE4 4.21 patch notes is information about the "Gauntlet" automation framework. Is there any documentation available about this? I would love to use this in a project.

Product Version: UE 4.21
Tags:
more ▼

asked Nov 22 '18 at 09:00 AM in Using UE4

avatar image

JustinVo
26 1 1 3

avatar image sd_pascal Nov 29 '18 at 04:24 PM

Same here. Any external documentation available ?

avatar image woaidaima Dec 11 '18 at 03:35 PM

Same here. Any external documentation available ?

avatar image YogevNa Dec 13 '18 at 01:21 PM

Same here. Any external documentation available ?

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

3 answers: sort voted first

Google Translation of woaidaima's answer into English, for those who don't speak Chinese. It's not a complete answer, and there are still things which I don't understand, but at least it's a bit more info than what I could find elsewhere.

First of all:

Gauntlet Automation Framework is one of the features added in 4.21 and is a framework for automated testing . The outline is described in the 4.21 release notes , but in summary, you can execute various tests by creating and executing test scripts (.cs, .cpp) according to the project. Please note that the configuration and usage may change in later versions because it is an early access function. This article is confirmed for 4.21.1.

Function:

Gauntlet Automation Framework is a function to execute an automatic test using RunUAT (AutomationTool). In other words, it is also possible to execute a test regularly by linking with Jenkins, etc. The following figure illustrates the flow of testing using Framework (Gauntlet Test).

Gauntlet figure

  1. Start the test. Execute the test command from RunUAT to start the test. Although Jenkins is often used as a CI, it is an image similar to creating and executing a test .bat file in the same way.

  2. Run the application. Start the content (Cooked content) in StagedBuild before running the test. If there is a specification of the device to execute, execute the application based on StagedBuild and start the device.

  3. Test execution. Start the test based on the test script created in advance. The application works on a test basis.

  4. Test complete. When the test results are returned from the application, the test ends. Even if the result is not returned, the test script returns the result by itself and completes the test.

  5. Report. Outputs the report to Log or .csv file based on the test result.

Although it is a brief explanation, the above is a series of flow at the time of test execution. The blue / green color in the figure is the function provided by the main Gauntlet Framework, but this will be explained in detail in the later chapters.

How to use

This chapter mainly explains how to execute Gauntlet Test. Here are the basic steps, such as how to run a simple test script and create a batch file yourself.

Run a sample test

First of all, I will describe the procedure for running GauntletTest. Gauntlet Test comes with a simple test script (ElementalDemoTest.cs), so you can try it right after downloading the engine. Here are the steps to run the sample:

[Execution procedure]

  1. Create Staging Build

  2. Test execution

Create Staging Build

Staging Build is created when you create a package, etc. It is created in [PROJECT]/Saved/StagedBuilds. Here we will package the project. If you want to use commands, such as automation, you can create them with the following command:

Build\BatchFiles\RunUAT.bat BuildCookRun -project=ElementalDemo -platform=Win64 -configuration=Development -build -cook -pak -stage

Test execution

After creating the Staging Build, all you have to do is run the tests. Here, the test script created in advance is executed from RunUAT. The following is an example of running ElementalDemo project on Windows:

Engine\Build\BatchFiles\RunUAT.bat RunUnreal -project=ElementalDemo -platform=Win64 -configuration=Development -test=ElementalDemoTest -build=Samples\Showcases\ElementalDemo\Saved\StagedBuilds

If you are successful in running the test, start the application and start the test. A sample test script is a script that simply runs the application and times out after 50 minutes. Therefore, once you start the application, it will stay as it is. If you quit this application (Alt + F4 etc.), you can end the test and get the following result. If you can confirm the operation so far, the procedure to carry out the Gauntlet Test is complete. Test reports are output by default to [ENGINE_ROOT]\GauntletTemp\Logs.

Gauntlet output

Create a batch file for testing

Next, I will explain how to create a batch file for test execution so that you can run automated tests in your own project. Below is a basic batch file to run the test. This is an edited version of "Example command of test execution" described above. A batch file can be created based on this command.

This batch file is an example of a case where the test of the test sample ElementDemoTest is executed with Win64: Development configuration for the own project GauntletTest, using Engine obtained from Github. You can basically use it as it is by rewriting the path or project name, but you can customize it by your project.

 path to rem // RunUAT.bat
 set UAT_PATH=G:\GitHub\UnrealEngine-4.21\Engine\Build\BatchFiles\RunUAT.bat
 rem // project name 
 set PRJ_NAME=GauntletTest
 rem // Staging directory path
 set STAGING_DIR=G:\Projects\GauntletTest\Saved\StagedBuilds
 rem // test execution command
 set TEST_CMD=RunUnreal
 rem // test name
 set TEST_NAME=ElementalDemoTest
 rem // platform
 set PLATFORM=Win64
 rem // build configuration
 set CONFIG=Development
 
 rem ********************* Start Gauntlet Test *********************
 %UAT_PATH% %TEST_CMD% -project=%PRJ_NAME% -platform=%PLATFORM% -configuration=%CONFIG% -test=%TEST_NAME% -build=%STAGING_DIR% 
 rem ********************* End   Gauntlet Test *********************
 pause

Variables:

  1. UAT_PATH: Any specification is acceptable as long as RunUAT.bat can be executed.

  2. PRJ_NAME: Specify a project (does not require .uproject)

  3. TEST_CMD: You can specify the test command and BuildCommand's inherited class (RunUnreal)

  4. TEST_NAME: Inherited class of DefaultNode

  5. PLATFORM: Specify the platform (Win64 / Andorid / PS4 etc.)

  6. CONFIG: Specify build configuration (Shipping / Development etc.)

Framework

  1. Startup script: \Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs

Execution class for GauntletTest. We have the ability to notify Test Executor that the test has started and to set up the device to run the test. By inheriting as needed, you can add steps such as device activation before test execution and account authentication.

  1. Test Execution / Management: \Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs

This class mainly manages test execution. Basically, it receives the start of the test from the start script, executes the test script, and possesses the function to manage the test execution result.

  1. Test script: \Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealTestNode.cs

Base class for test script. TestStart and Tick callback functions are provided, so you can control the behavior during test execution by writing the necessary code for the test.

  1. Test control: \Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Public\GauntletTestController

Base class for performing GauntletTest. It works by enabling the Plugin. Basically, the content is embedded in the application, and is used when accessing the game code directly to control the game. If you want to add an automatic test for each project, create a subclass based on this class and add test code. A base class is a class that provides only basic modules, and you can also implement logic that starts and monitors in derived classes. Use command line parameter (-gauntlet = MyControllerName) to execute TestController.

Customizing sample and source code

If you want to carry out automatic testing in a project, you can create your own test by deriving a class provided by Engine. In the diagram described in "Gauntlet Test Flow", classes that mainly extend are the following reddish classes. This is just an example, but other classes can be extended or modified to provide more efficient tests or tests in line with the project.

The following are samples of scripts and classes provided by default. If you want to create your own test environment or script you can use it as a reference.

GauntletTest sample test 1:

\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Game\Samples\ElementalDemoTest.cs

Inherited class of UnrealTestNode, there is only a 50 minute timeout and the test results fail due to timeout.

GauntletTest sample test 2:

\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Game\Samples\FortGPUTestbedPerfTest.cs

Inherited class of UnrealTestNode, detect a timeout of 5 minutes or specific input and end the test.

Startup Test Controller:

\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletTestControllerBootTest.cpp

This is a test sample that performs application launch check in a subclass of UGauntletTestController.

Error Test Controller:

\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletTestControllerErrorTest.cpp

This is a test sample that performs application error detection in a subclass of UGauntletTestController. It is an execution module of FGauntletModuleImpl GauntletTest. It mainly controls GauntletTestController. Since the Type of Plugin is Development, it can not be executed at runtime by default. You need to change Type to "Runtime" to enable this. (Probably a bug!)

This section describes the sample test and controller class behavior presented above. The following code is a modified version of the sample with additional comments. By referring to the contents described here, you will understand how the sample works and I think it will be a guide for expanding from the sample.

Test sample

 namespace  UE4Game 
 { 
     /// 
     Define test class (name to pass as argument at test execution) /// 
     Define test code as subclass of DefaultNode public  class  ElementalDemoTest  :  DefaultNode 
     { 
         public  ElementalDemoTest ( Gauntlet . UnrealTestContext  InContext ) 
             :  base ( InContext ) 
         { 
         }
 
         /// Define the configuration for test 
         /// For example, MaxDuration specifies the test timeout time 
         /// ※ There is a comment with "5min" but actually 5 * 600 (seconds) = 50 ( Timeout will occur in minutes) 
         public  override  UE4TestConfig  GetConfiguration () 
         { 
             /// just need a single client 
             UE4TestConfig  Config  =  base . GetConfiguration (); 
             UnrealTestRole  ClientRole  =  Config . RequireRole ( UnrealTargetRole . Client ); 
             ClientRole . CommandLine  + =  "- unattended " ; 
             Config . MaxDuration  = 5 * 600;       // 5min should be plenty
             return Config;
         }
 
         /// This is a module that creates a report of test results and is called at the end of the test 
         /// This module describes the process of outputting the result of Health Snapshot (see the next chapter for details) 
         public  override  void  CreateReport ( TestResult  the Result ,  UnrealTestContext  Contex ,  UnrealBuildSource  Build ,  IEnumerable < UnrealRoleArtifacts >  Artifacts ,  string  ArtifactPath ) 
         { 
             UnrealRoleArtifacts  ClientArtifacts  =  Artifacts . the Where ( A  =>  A . SessionRole . RoleType == UnrealTargetRole.Client).FirstOrDefault();
             var SnapshotSummary = new UnrealSnapshotSummary<UnrealHealthSnapshot>(ClientArtifacts.AppInstance.StdOut);
             Log.Info("Elemental Performance Report");
             Log.Info(SnapshotSummary.ToString());
             base.CreateReport(Result, Contex, Build, Artifacts, ArtifactPath);
         }
 
         /// This is a module to save the test result, and is called at the end of the test 
         /// This module performs processing to save the result of FPSChart to the specified Directory with .zip 
         /// as an argument at test execution time Create a .zip file by specifying the output destination with -uploadfolder = [FULLPATH] " 
         /// Run StartFPSChart / StopFPSChart during testing to save the results of FPSChart. 
         public  override  void  SaveArtifacts_DEPRECATED ( string  OutputPath ) 
         { 
             string  UploadFolder  =  Globals . Params . ParseValue ( "uploadfolder" ,  "" ); 
             if  ( UploadFolder . Count ()  >  0  &&  Directory.CreateDirectory(UploadFolder).Exists)
             {
                 string PlatformString = TestInstance.ClientApps[0].Device.Platform.ToString();
                 string ArtifactDir = TestInstance.ClientApps[0].ArtifactPath;
                 string ProfilingDir = Path.Combine(ArtifactDir, "Profiling");
                 string FPSChartsDir = Path.Combine(ProfilingDir, "FPSChartStats").ToLower();
                 string FpsChartsZipPath = Path.Combine(TestInstance.ClientApps[0].ArtifactPath, "FPSCharts.zip").ToLower();
                 if (Directory.Exists(FPSChartsDir))
                 {
                     ZipFile.CreateFromDirectory(FPSChartsDir, FpsChartsZipPath);
                     string DestFileName = "ElementalDemoTest-" + PlatformString + ".zip";
                     string DestZipFile = Path.Combine(UploadFolder, DestFileName);
                     File.Copy(FpsChartsZipPath, DestZipFile);
                 }
             }
             else
             {
                 Log.Info("Not uploading CSV Result UploadFolder: '" + UploadFolder + "'");
             }
         }
     }
 }

Controller sample

 #include "GauntletTestControllerBootTest.h"
  
 /// Called every frame when the test is run 
 void  UGauntletTestControllerBootTest :: OnTick ( float  TimeDelta ) 
 { 
     /// When the boot process is completed Test ends as "Test result: Normal" 
     / // Actually, IsBootProcessComplete () always returns false, so it does not pass 
     if  ( IsBootProcessComplete ()) 
     { 
         EndTest ( 0 ); 
     } 
     else 
     { 
         /// 
         if 300s (seconds) have passed from the start of the test “Test result: The test ends as "abnormal" /// Actually GetTimeInCurrentState () always returns 0 and does not pass 
         if  ( GetTimeInCurrentState ()  >  300 ) 
         { 
             UE_LOG(LogGauntlet, Error, TEXT("Failing boot test after 300 secs!"));
             EndTest(-1);
         }
     }
 }

Performance report

As described in Release Notes, the function to output performance reports as below from 4.21 has been added. This is also available for Blueprints, but can also be incorporated into tests.

The following is an example of acquiring a report in Blueprint. The ability to capture this performance is called Health Snapshot, which allows you to capture performance logs between specific points. The three main events are described below.

Performance report console

Health snapshot in Blueprint

  • Start Performance Snapshots -- Starts acquiring Health Snapshots.

  • Log Performance Snapshot -- Record between 2 points. Change the capture method of Record by the flag of "Reset Stats".

  • Stop Performance Snapshots -- Stops acquisition of Health Snapshot and outputs it.

If at the time to confirm these output in ElementalDemoTest.cs of the sample, in advance Snapshot and Summly Title need to have to match, and then capture the HealthSnapshot during the execution of the test. The following is an example for acquiring Snapshot after matching with the title "LogPerf".

  • Get Snapshot with "Log Perf" with Blueprint:

Log Perf in Blueprint

  • Get Snapshot with "Log Perf" when creating Report:

var SnapshotSummary = new UnrealSnapshotSummary<UnrealHealthSnapshot>(ClientArtifacts.AppInstance.StdOut, "LogPerf");

more ▼

answered Apr 15 '19 at 06:04 AM

avatar image

Jay2645
188 4 13 24

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
more ▼

answered Jan 07 '19 at 08:35 AM

avatar image

woaidaima
6 1

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

Hello, buddy, do you know more about gauntlet test writing, such as automatically assigning android devices from a device pool

more ▼

answered Apr 15 '19 at 06:25 AM

avatar image

woaidaima
6 1

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question