Putting JSON in to test parameter stops test ever running

I was planning on using the JSON serializer to allow me to pass in arbitary structs of data to represent specific test cases. However, If I use JSON in the OutTestCommandsArray, the test doesn’t run (the test for now just returns true and never hits the breakpoint in it). I’m not sure what is triggering the problem, I tried a escaped quotation mark but that wasn’t sufficient.

In the header file:

IMPLEMENT_COMPLEX_AUTOMATION_TEST(FMyExampleTest, "My.Example.Test", EAutomationTestFlags::ATF_None)

In the source file:

    void FMyExampleTest::GetTests(TArray<FString>& OutBeautifiedNames, TArray<FString>& OutTestCommands) const
{
	OutBeautifiedNames.Add(TEXT("SpecificTest"));
	OutTestCommands.Add(TEXT("{\"carStartingPosition\": 0,\"lane\": 0,\"windowSize\": 3.4028234663852886e+038}"));
}

bool FMyExampleTest::RunTest(const FString& Parameters)
{
	UE_LOG(LogTemp, Log, TEXT("Parameter %s"), *Parameters);
	return true;
}

I’ve also noticed funny things happening when the beautified names contains dots (not just too many subdivides, but weird corruptions where the test says it requires multiple players). At some point I’ll write up a bug report for that if you’re not able to fix at the same time as this.

It would seem the testing framework hasn’t been tested enough hehe!

Hey there!

Have you only been using ATF_None as your test flag for testing this out? I believe that if you don’t assign it to either Game or Editor, the test will never have an environment in which it thinks it actually needs to be populated, and that could at least be some part of your problem. I’d also be very interested in seeing what happens with your results with dots in the beautified name - we use dots for namespacing our tests in UFE, but in the past I personally have used dots in beautified names for several of my tests and it has properly subdivided my tests.

Thanks!
Ben

I have only been using ATF_None recently, will try with and ATF_Editor but providing I don’t use this JSON stuff the tests run fine, it is only once I I fill in the parameter that it breaks. Were you able to reproduce the problem with the above code?

I was able to reproduce the problem! Part of the problem may be that the paradigm for parsing parameters is the same philosophy as we use for the actual game commandline, where we expect key=value pairs instead of anything json-wise. Would it possible for you to do your json unmarshaling in your GetTests function and convert to key=value pairs to pass in for now as a workaround? I can dig into this deeper at some point next week and see exactly what’s going on with the parsing, I think.

I actually only picked JSON as there is a built in converter. My test parameter is really a USTRUCT. Is there an automatic way to convert that into a key/value string? That said, JSON is already pretty close to key/values anyway, so I’m surprised this is the issue.

I’ll look into this more after Thanksgiving is done, but it definitely seems bizarre that this doesn’t work properly. I’ve filed UE-23818 against myself to look at starting next week. For now, you should be able be able to create a JsonObject, and then PrintF the variable=value pairs into an Fstring and pass that through to OutTestCommands. I haven’t messed with Json parsing much here yet, but that seems like it should be the simplest way to do it.

Hopefully this helps, and I’ll try to have more investigation done on this next week!