Symbolicating crash dumps for Android in Unreal Engine 4

Do you know what is more annoying than a crash? A crash that you don’t know how to track down. You are praying for not being in a situation where no debugger is attached or there is no crash log. Sadly – this is a very common situation when you are working with the external engine, on a mobile device.

Android c++ crash logs are even more nasty, because of their binary format. In fact – they are minidumps, the same format the Microsoft is using for dumping Windows crash data. To encrypt the Android’s minidumps you need to do some voodoo with Google Breakpad.

But what’s this fuss about?

Consider the situation when a game crashes. The first thing you should do is to check crash logs. If you’ve had an Android device connected to a PC – the logcat should give you a valuable stack trace. Or not… it really depends on the model of the device. But if you were lucky you should have seen something like that:

(for those who are curious about what program I’m using for logcat – It’s the Tegra plugin for Visual Studio, it’s really nice)

Ok, but why on some models the log is not so well formatted or there is no log at all? It all depends on how the OS handles Breakpad. What if there is no crash log at all? What if you didn’t have a device connected to a PC and you have to harvest the crash log by yourself? What if the crash happens in the field and you want to gather them to the crash reporting system such as HockeyApp or Crashlytics? This is how.

The Linux.

At the beginning I must inform you – the steps to get readable crash logs from Android requires running Linux. Sadly, I wasn’t able to do this on Windows. If anyone know how to do this on Windows – please collaborate. I was using a virtual machine with 64bit version of Ubuntu 16.04TLS.

Getting the Google Breakpad

To do anything with crash logs on Android in the first place we need a Google Breakpad. To download it properly you wll need a git and a python2.7 installed.

You will also need a depot_tools from a Chromium repository. To get it – run git command in the terminal:

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

Next – export the depot_tools to the PATH:

export PATH=`pwd`/depot_tools:"$PATH"

With depot_tools you can get the Breakpad with all of it’s dependencies by running a fetch command in the breakpad directory

mkdir breakpad
cd breakpad
fetch breakpad

Building the Google Breakpad

With Google Breakpad downloaded you can build it. There are two types of build – the build of a library which will be included to the project to get the crashdump from the device, and the build which builds tools needed to get symbols and symbolicate retrieved crashdumps.

To build tools – you just have to run a following commands from the breakpad directory:

cd src
./configure && make
make install

To check if everything has been built and installed properly try dump_sys and minidump_stackwalk commands.

The instructions of how to build a library for Android are in the last chapter of this article. For now – I’ve made a plugin which contains all of the required third parties: Crash Dump Plugin.

Implementing Breakpad to UE4

Simply put the Crash Dump Plugin into your Engine/Plugins directory and enable it for your project. Have in mind that this plugin works only for non-Shipping, Android builds!

The most important parts of the plugin are:

Creating a directory for crashdumps (it will require a WRITE_EXTERNAL_STORAGE permission!) and setting up the Breakpad handling in the onCreate function of the GameActivity code:

import android.os.Environment;
 
private native void setUpBreakpad(String filepath);
 
@Override
public void onCreate(Bundle savedInstanceState)
{
    super.onCreate(savedInstanceState);
 
    // UE4 onCreate stuff
 
    try {
        String crashDumpDirectoryPath = Environment.getExternalStorageDirectory().getPath() + "/UE4CrashDumps/" + getPackageName() + "/";
        File crashDumpDir = new File(crashDumpDirectoryPath);
        if (crashDumpDir.exists() == false) {
            crashDumpDir.mkdirs();
        }
        setUpBreakpad(crashDumpDirectoryPath);
    } catch (Exception e) {
        Log.debug("Can't create a dir for crash dump logs. Exception: " + e);
    }
}

Implementing the actual setting up of the Breakpad handling in JNI:

bool DumpCallback(const google_breakpad::MinidumpDescriptor& descriptor, void* context, bool succeeded) 
{
    return succeeded;
}
 
static google_breakpad::ExceptionHandler* exceptionHandler;
 
extern "C" void Java_com_epicgames_ue4_GameActivity_setUpBreakpad(JNIEnv* env, jobject obj, jstring filepath)
{
    const char *path = env->GetStringUTFChars(filepath, 0);
    google_breakpad::MinidumpDescriptor descriptor(path);
    exceptionHandler = new google_breakpad::ExceptionHandler(descriptor, NULL, DumpCallback, NULL, true, -1);
}

Now every NDK crash will be handled by the Google Breakpad and saved to the UE4CrashDumps directory in the external storage.

Crashing the game

Yeah, I know this is a blasphemy… But we must somehow test it right? One of the simpliest way to crash an application is to run such a code:

volatile int* a = reinterpret_cast<volatile int*>(NULL);
*a = 1;

Getting crash dumps

To get stored crashdumps you can simply copy them from a device using a file explorer, or by using an adb command:

adb pull /sdcard/UE4CrashDumps/com.yourcompany.yourgame c:\crashdumps

I recommend to rename these crashdump files for some more readable names.

Getting symbols

Now it’s time to use Google Breakpad tools on Linux to get debugging symbols. They are hidden inside the so (shared object) file of the game. You can find it here:

Game\Intermediate\Android\APK\obj\local\armeabi-v7a\libUE4.so

Copy it somewhere to your Linux machine and run the command:

dump_syms libUE4.so > libUE4.so.sym

Next open the libUE4.so.sym file in the text editor and copy the id from the first line:

MODULE Linux arm 9DA4FF1D5571B173E1073D926516DDCB0 libUE4.so

Then create a specific directory structure for the sym file. The structure for this file will be:

Now copy the crashdump file near the symbols directory and run the command:

minidump_stackwalk crash_dump.dmp symbols > crash_log.log

This will produce a really nice and readable crash log! The most insteresting part is the beginning, which points to the place where the game actually crashed. Now we can get it fixed!

What's next?

The next thing to do is to make all of these steps automatic, so we won’t get crazy by doing it all over and over again for every crash we got.

The other thing is to try to do it on Windows, because, well… Windows is still a main operating system for UE4. For now, even if the whole process is automated – there must be a remote Linux machine which gets all of the shared object files and crashdumps.

For me – this is almost everything I wanted to write here. Happy bug hunting!

Building Google Breakpad for Android

If you want to build Google Breakpad library for Android by yourself you will need a Linux system.

Download the NDK of your desired version. I was using the r11c version.
Extract the NDK to the /opt/android-ndk-r11c directory.

Set up Environment variables:

ANDROID_NDK_ROOT=/opt/android-ndk-r11c
PATH=$PATH:$ANDROID_NDK_ROOT

Download the Google Breakpad like in the chapter before.
Inside the breakpad/src directory create android_google_breakpad directory
Put the jni directory into the android_google_breakpad directory. The jni directory can be downloaded from here.

From the android_google_breakpad launch command:

sudo /opt/android-ndk-r11c/ndk-build NDK_PROJECTPATH="." NDK_DEBUG=0 -B

The static library will be in this directory

android_google_breakpad/obj/local/armeabi-v7a/objs/libbreakpad_client.a

For headers – I suggest to simply copy the ones from my github project. When you put the library to the UE4 make sure that the linux_syscall_support.h file from the third parties is the one from my github project, the one with the UE4_HACK definition! Use the UE4_HACK only when building on UE4, not on Linux!

ZOMPI DEV

Categories