Using automation testing for catching expected errors

Suppose I have a piece of program that logs an error.

void DoStuff(bool doit)
{
    if(!doit)
    {
        UE_LOG(MyLogCategory, Error, "Cannot do stuff");
    }
}

And I write an automation test that actually expects this log to be printed.

IMPLEMENT_SIMPLE_AUTOMATION_TEST(DoStuffTest, "DoStuffTest.ThrowsError", EAutomationTestFlags::EngineFilter | EAutomationTestFlags::EditorContext)
bool DoStuffTest::RunTest(const FString& Parameters)
{
    DoStuff(false);

    // TODO: Assert that an error was logged and make the test pass.
}

How can I ensure that my test passes? It now directly fails, because of the UE_LOG in the actual source code.

Hi mtass,

We recently added support for this, and I believe it is expected to ship as part of 4.17. You’d want to use the new AddExpectedError method in the test body, which allows you to define a regex describing the expected error/warning message, the type of comparison to perform, and the number of times you expect the message to occur during the test.

Example from one of the unit tests:

AddExpectedError(TEXT("Response \\(-?\\d+\\)"), EAutomationExpectedErrorFlags::Contains, 4);
AddError(TEXT("Response (0)"));
AddError(TEXT("Response (1)"));
AddError(FString::Printf(TEXT("Response (%d)"), MIN_int64));
AddError(FString::Printf(TEXT("Response (%d)"), MAX_uint64));

From your example, your test would look something like:

IMPLEMENT_SIMPLE_AUTOMATION_TEST(DoStuffTest, "DoStuffTest.ThrowsError", EAutomationTestFlags::EngineFilter | EAutomationTestFlags::EditorContext)
bool DoStuffTest::RunTest(const FString& Parameters)
{
    AddExpectedError(TEXT("Cannot do stuff"), EAutomationExpectedErrorFlags::Exact, 1);
    DoStuff(false);
}