How can I get a complex test to show up?

Development Programming & Scripting C++
, , , ,
1

I’ve been trying to get automated testing working for my game, but it’s been quite challenging. The documentation is quite outdated - the samples and description don’t match the current 4.10 or 4.11 codebases. Plus, it just looks like major parts of it don’t work.

This question is about “complex tests”.

How can I get a simple example of a “complex test” to appear in the Session’s window’s Automation tab?

I’ve tried every combination of server, client, and editor flags. I’ve tried launching the game in every configuration and attaching the session window to the various processes.

I can see simple tests from the same cpp file appear in this window.

Here is what my “complex test” looks like:

IMPLEMENT_COMPLEX_AUTOMATION_TEST(FTestItShowsUp, “MyGame.TestItShowsUp”, EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::SmokeFilter)

void FTestItShowsUp::GetTests(TArray& OutBeautifiedNames, TArray & OutTestCommands) const
{
}

bool FTestItShowsUp::RunTest(const FString& Parameters)
{
return true;
}

2

If that’s the full text of your tests, then the problem is that your GetTests implementation isn’t adding anything to the OutBeatufiedNames and OutTestCommands arrays.

3

Okay, I added some entries to those arrays. It still doesn’t work.
:slight_smile:

IMPLEMENT_COMPLEX_AUTOMATION_TEST(FTestItShowsUp, “MyGame.TestItShowsUp”, EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::SmokeFilter)

void FTestItShowsUp::GetTests(TArray& OutBeautifiedNames, TArray & OutTestCommands) const
{
OutBeautifiedNames.Add(FString(“See1”));
OutBeautifiedNames.Add(FString(“See2”));
OutBeautifiedNames.Add(FString(“See3”));
OutTestCommands.Add(FString(“See1”));
OutTestCommands.Add(FString(“See2”));
OutTestCommands.Add(FString(“See3”));
}

bool FTestItShowsUp::RunTest(const FString& Parameters)
{
return true;
}

4

A couple of things here:

  1. Instead of using EAutomationTestFlags::ApplicationContextMask, you should be combining the specific flags you want (eg. EAutomationTestFlags::EditorContext | EAutomationTestFlags::ClientContext). Using the mask is discouraged.
  2. You probably don’t want to be using the SmokeFilter. Tests flagged with SmokeFilter are always run on startup in non-Shipping builds, automatically fail if they take longer than 1-2 seconds, and are hidden by default in the Automation Frontend. This may also explain why your test isn’t showing up; check to see whether the test filter dropdown is set to show Smoke tests.
  3. The OutTestCommands may be expecting the strings to be either empty strings or key/value pairs formatted like “Key=Value”. I’m not 100% sure on the k/v-pair thing, but that’s how our internal complex tests that use commands are structured.
5

  1. Okay, good to know.

  2. Thanks for the info about the SmokeFilter. I had no idea it was getting run on startup, but it explains some start-up crashes I was getting when playing with tests.
    Complex tests labeled with SmokeFilter are not showing up when the dropdown is set to show Smoke tests - or at any time really. Simple tests do show up when SmokeFilter is used.
    Changing SmokeFilter to ProductFilter got the complex tests to appear.

  3. OutTestCommands doesn’t seem to require key/value pairs. My simple example does call RunTest() with my test’s test strings.

Thanks Adric! It would be great to see more the documentation updated, but at least we have this question answered. :slight_smile: