Unreal Engine 4 Automation Tests New API

While digging in the UE4.22 source code I found some strange files with *.spec.cpp extension. You can find them in Plugins/Online/OnlineSubsystem/Source/Test directory. They appears to be used for Automation Tests, but their API looks odd, completely different than the documented one. I was trying to find any informations about them, but without success, so… I researched this new way of writing tests and I must say it is much more convenient and professional than the old one.

In this article I will show some simple tests written using the old Automation Tests and their equivalents written using the new API.

Edit 25.07.2019 – Epic gave us an official documentation for Automation Spec here. But I still recommend this article to compare side by side an old system with a new one.

Project

The project we are testing is a First Person Template with three enemy Characters that are destroyed immediately after being hit by a projectile. Player must be spawned in a way that it aims to the one of the enemies.

Utils

Every time I write Automation Tests I use some of my utility functions:
Automation Test Flags Mask for easier setting up test mask. More about test masks here.

static const int32 TestsFlags =  
    EAutomationTestFlags::EditorContext |
    EAutomationTestFlags::ClientContext |
    EAutomationTestFlags::ProductFilter;

GetWorld - because in many places we need to get an access to the current game world.

static UWorld* GetWorld()
{
    if (GEngine)
    {
        if (FWorldContext* WorldContext = GEngine->GetWorldContextFromPIEInstance(0))
        {
            return WorldContext->World();
        }
    }
    return nullptr;
}

Exit – usually to run test correctly it must be performed on an opened game map and this map should be closed after the test is finished.

static void Exit()
{
    if (UWorld* World = GetWorld())
    {
        if (APlayerController* TargetPC = UGameplayStatics::GetPlayerController(World, 0))
        {
            TargetPC->ConsoleCommand(TEXT("Exit"), true);
        }
    }
}

PressKey - for the best game behaviour simulation we must simulate a key press.

static bool PressKey(const FName& KeyName, EInputEvent InputEvent)
{
    if (GEngine)
    {
        if (GEngine->GameViewport)
        {
            if (FViewport* Viewport = GEngine->GameViewport->Viewport)
            {
                if (FViewportClient* ViewportClient = Viewport->GetClient())
                {
                    return ViewportClient->InputKey(FInputKeyEventArgs(Viewport, 0, KeyName, InputEvent));
                }
            }
        }
    }
    return false;
}

Simple Test

With all of these utils at our disposal we are ready to write a simple test. For example - I want to test if there are three enemies on the map when it starts. I will simply write the code of a test here and I will write every important thing in a comment.

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FEnemyCountTest, "AutoTest.EnemyCountTest", FMyTestUtils::TestsFlags)
bool FEnemyCountTest::RunTest(const FString& Parameters)
{
    // 1. Before we do anything the game map must be opened.
    AutomationOpenMap("/Game/FirstPersonCPP/Maps/FirstPersonExampleMap");
 
    // 2. Check if the game World is valid. If not - end this test immediately.
    UWorld* World = FMyTestUtils::GetWorld();
    TestNotNull("Check if World is properly created", World);
    if (!World) return false;
 
    // 3. Count the number of AEnemyCharacter characters.
    int32 EnemiesCount = 0;
    for (TActorIterator<AActor> It(World, AEnemyCharacter::StaticClass()); It; It++)
    {
        EnemiesCount++;
    }
 
    // 4. Test if there are 3 enemy characters.
    TestTrue("Check if there are 3 enemies on the level", EnemiesCount == 3);
    
    // 5. Exit this map.
    ADD_LATENT_AUTOMATION_COMMAND(FExitGameCommand);
 
    // 6. The test has been finished with success.
    return true;
}

You might be wondering what is this latent automation command under the comment nr 5. The FExitGameCommand is just an another way of closing the game map.

One thing that always bothers me in Automation Tests is that even if any of Tests fails, the final result of the RunTest function is usually true. The RunTest function should return false only when it will not be able to finish the whole tests set.

The other annoying thing is that for every test we must ensure that World, or any other important variable is available and do that extra check if we can continue the test or not.

There must be a better way...

Simple Test – a new way

// 1. We define the test with BEGIN_DEFINE_SPEC and END_DEFINE_SPEC macros. 
//    Everything between these two macros is a variable shared between implemented tests.
BEGIN_DEFINE_SPEC(FNewEnemyCountTest, "AutoTest.ANew.EnemyCountTest", FMyTestUtils::TestsFlags)
UWorld* World;
END_DEFINE_SPEC(FNewEnemyCountTest)
 
void FNewEnemyCountTest::Define()
{
    // 2. BeforeEach - defines what happens before each test.
    BeforeEach([this]() 
    {
        // 3. Before each test we want to open a game map.
        AutomationOpenMap("/Game/FirstPersonCPP/Maps/FirstPersonExampleMap");
        
        // 4. Before each test the World is obtained and tested if is valid.
        World = FMyTestUtils::GetWorld();
        TestNotNull("Check if World is properly created", World);
    });
 
    // 5. It - defines one test.
    It("Test Enemy Count", [this]()
    {
        // 6. Count the number of AEnemyCharacter characters.
        int32 EnemiesCount = 0;
        for (TActorIterator<AActor> It(World, AEnemyCharacter::StaticClass()); It; It++)
        {
            EnemiesCount++;
        }
 
        // 7. Test if there are 3 enemy characters.
        TestTrue("Check if there are 3 enemies on the level", EnemiesCount == 3);
    });
 
    // 8. AfterEach - defines what happens after every test.
    AfterEach([this]()
    {
        // 9. After each test close game map.
        FMyTestUtils::Exit();
    });
}

The issue we had in the old API is solved here. The game World is obtained before each defined test and is checked using TestNotNull function. If the BeforeEach phase fails tests will not be performed. In this example we have only one test, but you can put as many tests as you want by using It functions. Each of these tests will open and close the map and check if the game World is valid. Also, in my opinion, the whole structure of the test is more readable and better standardized.

Complex Test

For a complex test we do the same check of the number of enemies but for multiple maps. Every map is inside /GameFirstPersonCPP/Maps directory.

IMPLEMENT_COMPLEX_AUTOMATION_TEST(FEnemyCountTestMulti, "AutoTest.EnemyCountTestMulti", FMyTestUtils::TestsFlags)
void FEnemyCountTestMulti::GetTests(TArray<FString>& OutBeautifiedNames, TArray <FString>& OutTestCommands) const
{
    // 1. Get the array of assets from Maps directory.
    FAssetRegistryModule& ARM = FModuleManager::LoadModuleChecked<FAssetRegistryModule>("AssetRegistry");
    TArray<FAssetData> AssetDataArray;
    ARM.Get().GetAssetsByPath(TEXT("/Game/FirstPersonCPP/Maps"), AssetDataArray);
 
    for (const auto& AssetData : AssetDataArray)
    {
        // 2. Check if the asset is a World class, which means it is a umap.
        if (AssetData.AssetClass == "World")
        {
            // 3. Use the asset name as a test name and package name (map path) as a test parameter.
            OutBeautifiedNames.Add(AssetData.AssetName.ToString());
            OutTestCommands.Add(AssetData.PackageName.ToString());
        }
    }
}
 
bool FEnemyCountTestMulti::RunTest(const FString& Parameters)
{
    // 4. Open the given map.
    AutomationOpenMap(Parameters);
 
    // ... The rest is the same as Simple Test
}

To run the same test, but with different parameter sets, the GetTests method must be implemented which collects those parameters. There is a proper logic behind this solution but why can’t we just use good old for loop?

Complex Test – a new way

// 1. Get the array of assets from Maps directory.
FAssetRegistryModule& ARM = FModuleManager::LoadModuleChecked<FAssetRegistryModule>("AssetRegistry");
TArray<FAssetData> AssetDataArray;
ARM.Get().GetAssetsByPath(TEXT("/Game/FirstPersonCPP/Maps"), AssetDataArray);
 
for (const auto& AssetData : AssetDataArray)
{
    // 2. Check if the asset is a World class, which means it is a umap.
    if (AssetData.AssetClass == "World")
    {
        // 3. Describe - defines a tests scope.
        Describe(AssetData.AssetName.ToString(), [this, AssetData]()
        {
            BeforeEach([this, AssetData]()
            {
                // 4. Before each test open a different map.
                AutomationOpenMap(AssetData.PackageName.ToString());
                // ... The rest is the same as previous
            });
            // ... The rest is the same as previous
        });
    }
}

In the new API we can simply use a for loop to define multiple tests for different maps. The Describe function here is important as it creates a scope for BeforeEach, It and AfterEach set of functions. Without it, there would be three BeforeEach definitions and all of them would run before each defined test (test maps would be opened 9 times).

Latent Commands

Now it will get more interesting as we will start using Latent Commands in order to check gameplay mechanics. In short – latent commands are functions that runs across multiple frames. One common latent command is FEngineWaitLatentCommand which makes the test to wait the given amount of seconds.

Unfortunately the way latent commands work is not intuitive. When I was learning about them I thought they would make the whole test to wait until it’s finished, but in fact it just enqueues itself and it ticks after the test body is finished!

It means that everything that must wait for a latent command to finish must be a latent command itself.

Consider we want to test if after the shoot the number of enemies will decrease. The test could look like this:

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FShootEnemyTest, "AutoTest.ShootEnemyTest", FMyTestUtils::TestsFlags);
 
// 1. Defines latent command which will check the number of enemies after the shoot.
//    This command accepts the pointer to the tests struct, so it can run test functions.
DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER(FCheckIfEnemyWasShooted, FShootEnemyTest*, Test);
bool FCheckIfEnemyWasShooted::Update()
{
    // 2. Count the number of AEnemyCharacter characters.
    int32 EnemiesCount = 0;
    for (TActorIterator<AActor> It(FMyTestUtils::GetWorld(), AEnemyCharacter::StaticClass()); It; It++)
    {
        EnemiesCount++;
    }
 
    // 3. Test if there are 2 enemy characters.
    Test->TestTrue("Check if there are 2 enemies on the level", EnemiesCount == 2);
 
    // 4. The latent command has finished with success.
    return true;
}
 
bool FShootEnemyTest::RunTest(const FString& Parameters)
{
    // 5. Before we do anything the game map must be opened.
    AutomationOpenMap("/Game/FirstPersonCPP/Maps/FirstPersonExampleMap");
 
    // 6. Check if the game World is valid. If not - end this test immediately.
    UWorld* World = FMyTestUtils::GetWorld();
    TestNotNull("Check if World is properly created", World);
    if (!World) return false;
 
    // 7. Count the number of AEnemyCharacter characters.
    int32 EnemiesCount = 0;
    for (TActorIterator<AActor> It(World, AEnemyCharacter::StaticClass()); It; It++)
    {
        EnemiesCount++;
    }
 
    // 8. Test if there are 3 enemy characters.
    TestTrue("Check if there are 3 enemies on the level", EnemiesCount == 3);
 
    // 9. Press the Left Mouse button (fire).
    const bool bFireButtonWasPressed = FMyTestUtils::PressKey(FName("LeftMouseButton"), EInputEvent::IE_Pressed);
    
    // 10. Test if the button was pressed. If not - end this test immediately.
    TestTrue("Fire button was properly pressed", bFireButtonWasPressed);
    if (bFireButtonWasPressed == false) return false;
    
    // 11. Wait 1 second.
    ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(1.f));
    
    // 12. Wait for our defined latent command to finish.
    ADD_LATENT_AUTOMATION_COMMAND(FCheckIfEnemyWasShooted(this));
    
    // 13. Exit this map.
    ADD_LATENT_AUTOMATION_COMMAND(FExitGameCommand);
 
    // 16. The test has been finished with success.
    return true;
}

As you can imagine with more complex tests it can get pretty messy. You have to divide your test into multiple latent chunks which will wait for each other. You might have already guessed that the new API has solved it much better.

Latent Commands – a new way

To write latent tests in the new API we use LatentIt instead of It.

// 1. LatentIt - defines a latent test.
LatentIt("Test Enemy Shoot", EAsyncExecution::ThreadPool, [this](const FDoneDelegate TestDone)
{
    // 2. Because latent test runs on a separate thread we have to ensure that game logic tests run on a Game Thread. 
    AsyncTask(ENamedThreads::GameThread, [this]() 
    {
        // 3. Count the number of AEnemyCharacter characters.
        int32 EnemiesCount = 0;
        for (TActorIterator<AActor> It(World, AEnemyCharacter::StaticClass()); It; It++)
        {
            EnemiesCount++;
        }
 
        // 4. Test if there are 3 enemy characters.
        TestTrue("Check if there are 3 enemies on the level", EnemiesCount == 3);
 
        // 5. Press the Left Mouse button (fire).
        const bool bFireButtonWasPressed = FMyTestUtils::PressKey(FName("LeftMouseButton"), EInputEvent::IE_Pressed);
        
        // 6. Test if the button was pressed.
        TestTrue("Fire button was properly pressed", bFireButtonWasPressed);
    });
        
    // 7. Wait 1 second.
    FPlatformProcess::Sleep(1.0f);
 
    // 8. Once again - ensure we do tests on the Game Thread.
    AsyncTask(ENamedThreads::GameThread, [this]() 
    {
        // 9. Count the number of AEnemyCharacter characters.
        int32 EnemiesCount = 0;
        for (TActorIterator<AActor> It(FMyTestUtils::GetWorld(), AEnemyCharacter::StaticClass()); It; It++)
        {
            EnemiesCount++;
        }
 
        // 10. Test if there are 2 enemy characters.
        TestTrue("Check if there are 2 enemies on the level", EnemiesCount == 2);
    });
 
    // 11. Execude TestDone delegate to inform Automation System that the latent function has finished.
    TestDone.Execute();
});

As you can see there are no chunks of test logic floating around the whole file, only one, unified test.

Because this test runs on a separate thread it won’t block game logic and it can simply checks interesting us values during the gameplay.

The only issue is that if there is a need to run some logic that must be run on a game thread (like actor iterating) we have to ensure that it will be run on a game thread. AsyncTask function is the easiest way to do it.

However, I still think this is much easier and cleaner way of writing latent tests than in old API.

Conclusion

There you have it – a quick introduction to the new Automation Tests API in UE4. I covered only basics, there are more types of files like *.fake.h and *.mock.h hiding in the Engine/Online/BuildPathServices/Private/Tests directory. It looks like a professional unit testing framework, kinda similar to the Google Test!

You can download the project with all of the described tests from GitHub.

ZOMPI DEV

Categories