Gauntlet Automation Framework documentation
Included in the UE4 4.21 patch notes is information about the "Gauntlet" automation framework. Is there any documentation available about this? I would love to use this in a project.
asked Nov 22 '18 at 09:00 AM in Using UE4
Google Translation of woaidaima's answer into English, for those who don't speak Chinese. It's not a complete answer, and there are still things which I don't understand, but at least it's a bit more info than what I could find elsewhere.
First of all:
Gauntlet Automation Framework is one of the features added in 4.21 and is a framework for automated testing . The outline is described in the 4.21 release notes , but in summary, you can execute various tests by creating and executing test scripts (.cs, .cpp) according to the project. Please note that the configuration and usage may change in later versions because it is an early access function. This article is confirmed for 4.21.1.
Gauntlet Automation Framework is a function to execute an automatic test using RunUAT (AutomationTool). In other words, it is also possible to execute a test regularly by linking with Jenkins, etc. The following figure illustrates the flow of testing using Framework (Gauntlet Test).
Although it is a brief explanation, the above is a series of flow at the time of test execution. The blue / green color in the figure is the function provided by the main Gauntlet Framework, but this will be explained in detail in the later chapters.
How to use
This chapter mainly explains how to execute Gauntlet Test. Here are the basic steps, such as how to run a simple test script and create a batch file yourself.
Run a sample test
First of all, I will describe the procedure for running GauntletTest. Gauntlet Test comes with a simple test script (ElementalDemoTest.cs), so you can try it right after downloading the engine. Here are the steps to run the sample:
Create Staging Build
Staging Build is created when you create a package, etc. It is created in [PROJECT]/Saved/StagedBuilds. Here we will package the project. If you want to use commands, such as automation, you can create them with the following command:
After creating the Staging Build, all you have to do is run the tests. Here, the test script created in advance is executed from RunUAT. The following is an example of running ElementalDemo project on Windows:
If you are successful in running the test, start the application and start the test. A sample test script is a script that simply runs the application and times out after 50 minutes. Therefore, once you start the application, it will stay as it is. If you quit this application (Alt + F4 etc.), you can end the test and get the following result. If you can confirm the operation so far, the procedure to carry out the Gauntlet Test is complete. Test reports are output by default to [ENGINE_ROOT]\GauntletTemp\Logs.
Create a batch file for testing
Next, I will explain how to create a batch file for test execution so that you can run automated tests in your own project. Below is a basic batch file to run the test. This is an edited version of "Example command of test execution" described above. A batch file can be created based on this command.
This batch file is an example of a case where the test of the test sample ElementDemoTest is executed with Win64: Development configuration for the own project GauntletTest, using Engine obtained from Github. You can basically use it as it is by rewriting the path or project name, but you can customize it by your project.
Execution class for GauntletTest. We have the ability to notify Test Executor that the test has started and to set up the device to run the test. By inheriting as needed, you can add steps such as device activation before test execution and account authentication.
This class mainly manages test execution. Basically, it receives the start of the test from the start script, executes the test script, and possesses the function to manage the test execution result.
Base class for test script. TestStart and Tick callback functions are provided, so you can control the behavior during test execution by writing the necessary code for the test.
Base class for performing GauntletTest. It works by enabling the Plugin. Basically, the content is embedded in the application, and is used when accessing the game code directly to control the game. If you want to add an automatic test for each project, create a subclass based on this class and add test code. A base class is a class that provides only basic modules, and you can also implement logic that starts and monitors in derived classes. Use command line parameter (-gauntlet = MyControllerName) to execute TestController.
Customizing sample and source code
If you want to carry out automatic testing in a project, you can create your own test by deriving a class provided by Engine. In the diagram described in "Gauntlet Test Flow", classes that mainly extend are the following reddish classes. This is just an example, but other classes can be extended or modified to provide more efficient tests or tests in line with the project.
The following are samples of scripts and classes provided by default. If you want to create your own test environment or script you can use it as a reference.
GauntletTest sample test 1:
Inherited class of UnrealTestNode, there is only a 50 minute timeout and the test results fail due to timeout.
GauntletTest sample test 2:
Inherited class of UnrealTestNode, detect a timeout of 5 minutes or specific input and end the test.
Startup Test Controller:
This is a test sample that performs application launch check in a subclass of
Error Test Controller:
This is a test sample that performs application error detection in a subclass of UGauntletTestController. It is an execution module of
This section describes the sample test and controller class behavior presented above. The following code is a modified version of the sample with additional comments. By referring to the contents described here, you will understand how the sample works and I think it will be a guide for expanding from the sample.
As described in Release Notes, the function to output performance reports as below from 4.21 has been added. This is also available for Blueprints, but can also be incorporated into tests.
The following is an example of acquiring a report in Blueprint. The ability to capture this performance is called Health Snapshot, which allows you to capture performance logs between specific points. The three main events are described below.
If at the time to confirm these output in ElementalDemoTest.cs of the sample, in advance Snapshot and Summly Title need to have to match, and then capture the HealthSnapshot during the execution of the test. The following is an example for acquiring Snapshot after matching with the title "LogPerf".
answered Jan 07 '19 at 08:35 AM
Hello, buddy, do you know more about gauntlet test writing, such as automatically assigning android devices from a device pool
answered Apr 15 '19 at 06:25 AM
Follow this question
Once you sign in you will be able to subscribe for any updates here