Remote desymbolication of iOS crash logs from Unreal Engine 4 games

Let's get straight to the point. You are developing an iOS game with Unreal Engine 4. Let's assume you are using a remote Mac machine to build the game (just like in this post). And the game crashes for unknown reasons. How to deal with it?

Enabling dSYM generation

First of all, UE4 must know if it want to generate dSYM files (which are used for desymbolicating crashlogs). To enable this option modify an DefaultEngine.ini file:

[/Script/IOSRuntimeSettings.IOSRuntimeSettings]
bGeneratedSYMFile=True
Have in mind that build time and build size may increase when this option is set to True.

 

Extracting crash log

To get a crashlog from device on Windows I'm using a set of tool called iOSLogInfo. You can download it from here:

https://www.blackberry.com/blackberrytraining/web/KB_Resources/KB36986_iOSLogInfo_4.3.4.zip 

Using console command run: 

sdsioscrashlog.exe CrashLogs
It will download all of the crashlogs from the device and put them into a CrashLogs directory. When you find your crashlog (it should have *.ips file extension) you will notice that it is not symbolicated. This file in it's current state won't get any useful information. 
Thread 9 Crashed:
0   libsystem_kernel.dylib        	0x00000001addafec4 0x1add8b000 + 151236
1   libsystem_pthread.dylib       	0x00000001adccb774 0x1adcc9000 + 10100
2   libsystem_c.dylib             	0x00000001adc1f844 0x1adbac000 + 473156
3   TestProject                   	0x0000000101675d18 0x1009c4000 + 13311256
4   TestProject                   	0x0000000101758ac0 0x1009c4000 + 14240448
5   TestProject                   	0x00000001017ef33c 0x1009c4000 + 14857020
6   TestProject                   	0x00000001017a4e84 0x1009c4000 + 14552708
7   TestProject                   	0x00000001017a4d30 0x1009c4000 + 14552368
8   TestProject                   	0x000000010152cea4 0x1009c4000 + 11964068
...


Sending crash log to remote machine

Crashlog can be desymbolicated on the machine it's been built on. To send it there you can use a pscp tool (to be downloaded from here: https://the.earth.li/~sgtatham/putty/latest/w64/pscp.exe)
pscp.exe -P [port] -pw [password] [path_to_ips_file] [username]@[address]:/Users/[username]/UE4/Builds/[machine_name]/[project_path]/Binaries/IOS
where username, password, address and port are the parameters of the remote Mac machine the game was built on.

 

Desymbolicating crash log

It's time for the main event - desymbolicating. Connect to your remote Mac machine via ssh (Putty is a great tool for this task, to be downloaded from here: https://the.earth.li/~sgtatham/putty/latest/w64/putty.exe). 
Go to your project's directory
cd /Users/[username]/UE4/Builds/[machine_name]/[project_path]/Binaries/IOS
here you should be able to locate dSYM and ips files. The symbolicatecrash tool will be used, which is a part of xcode. To make your work easier let's create an alias for it: 
alias symbolicatecrash='/Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash'
(have in mind that this path is valid for xcode 8 and newer only). 
Before it can be used an environment variable must be set: 
export DEVELOPER_DIR=$(xcode-select --print-path)
After all of that everything is ready to desymbolicate!
symbolicatecrash -v MyCrashLog.ips TestProject.dSYM > Crashlog.txt
This will desymbolicate the MyCrashLog.ips file and will save results into the Crashlog.txt file.

 

Downloading desymbolicating crash log back to local machine

To download txt file with desymbolicated crash let's just use pscp tool once again on Windows machine: 
pscp.exe -P [port] -pw [password] [username]@[address]:/Users/[username]/UE4/Builds/[machine_name]/[project_path]/Binaries/IOS/Crashlog.txt [download_path]
Now, when the desymbolicated crashlog is downloaded it can finally shows what was going on all the time:
Thread 9 Crashed:
0   libsystem_kernel.dylib        	0x00000001addafec4 0x1add8b000 + 151236
1   libsystem_pthread.dylib       	0x00000001adccb774 0x1adcc9000 + 10100
2   libsystem_c.dylib             	0x00000001adc1f844 0x1adbac000 + 473156
3   TestProject                   	0x0000000101675d18 FGenericPlatformMallocCrash::InitializeSmallPools() + 13311256 (GenericPlatformMallocCrash.cpp:468)
4   TestProject                   	0x0000000101758ac0 FIOSErrorOutputDevice::Serialize(char16_t const*, ELogVerbosity::Type, FName const&) + 14240448 (IOSErrorOutputDevice.cpp:0)
5   TestProject                   	0x00000001017ef33c FOutputDevice::LogfImpl(char16_t const*, ...) + 14857020 (OutputDevice.cpp:71)
6   TestProject                   	0x00000001017a4e84 AssertFailedImplV(char const*, char const*, int, char16_t const*, char*) + 14552708 (AssertionMacros.cpp:105)
7   TestProject                   	0x00000001017a4d30 FDebug::CheckVerifyFailedImpl(char const*, char const*, int, char16_t const*, ...) + 14552368 (AssertionMacros.cpp:0)
8   TestProject                   	0x000000010152cea4 UMyGameplayStatics::LeCrash() + 11964068 (MyGameplayStatics.cpp:117)
...


Conclusion

Making all of these steps is a bit of a chore, but when scripted properly it can be a valuable tool for faster desymbolicating crash logs and, as a result, easier strange crashes fixing. I hope this little article will be a help. Good luck!