Unreal Engine Crash Reporter
Crash report system allows collecting crash information for developer analysis. Crash report system requires a server to process, aggregate and display crash data. This document describes the procedure for setting up the system on your own server.
The document describes the setup for running the system on a single machine. Epic is moving the system to the cloud to handle the increasing load. At the moment of writing, latest version of Unreal engine 4.19 doesn’t have source code for the CrashReportServer. The process is based on source code for 4.17.2 version.
Here is a typical timeline for a single crash:
- An exception (or an assert) is triggered while running game application on the target system.
- A crash folder is generated with log and dump files
- CrashReportClient(CRC) is launched
- CRC sends HTTP request to the crash report server with crash files
- The crash files are received by CrashReportReceiver(CRR) on the server
- The crash files are written to the LandingZone folder
- CrashReportProcess(CRP) scans Landing Zone Folder and discover *new* crash folders
- For every crash CRP:
- If callstack is not symbolized, CRP executes the request to MinidumpDiagnostics (MDD)
- Make an HTTP request to CrashReportWebsite(CRW) to add the crash
- CRW add the crash to the CrashReportDatabase (CRD)
- Move the crash report files to Processed Folder
When the crash has gone through this chain, developer could go to the crash report website in the browser and find the crash.
Note that for MDD to symbolize the callstack, a process need to upload the symbols to the server.
Edit diagram File:CrashReporter-SystemOverview-Source.zip
CRW is written in ASP.NET with use of Entity Framework. That requires the server to run Windows OS + IIS with MS SQL database.
Before proceeding forward choose a database license that suits your demands:
The fork for Unreal Engine with crash report server sources files and fixes:
In CrashReporter branch
The system has been testing on Windows and Linux. Other platforms haven’t been tested.
The configuration process has the following steps:
- Server Machine Setup
- Crash Report Database
- Crash Report Website
- Crash Report Client
- Crash Report Receiver
- Crash Report Process
- Minidump Diagnostics
- Symbol Uploader
- Crash Reporting For Editor
Server Machine Setup
To start come up with the web address for the crash reporter website. Note that system is capable to aggregate crashes from different games made with Unreal Engine if that applies to you.
Verify that web address is reachable by pinging it.
Crash Report Database
Download and install MS SQL server with MS SQL Management Studio.
Modify and execute sql script:
- In CREATE DATABASE CrashReport statement modify the filename to a folder that exists.
- Note that database max size depends on the license and might require editing.
- If the following error occurs: cannot create more than one clustered index on table …
- Modify Statement to create NONCLUSTERED index
When the database is created, execute the following script:
INSERT INTO dbo.UserGroups(Name) VALUES('Group1')
INSERT INTO dbo.UserGroups(Name) VALUES('Group2')
INSERT INTO dbo.UserGroups(Name) VALUES('Coders')
INSERT INTO dbo.UserGroups(Name) VALUES('Group4')
INSERT INTO dbo.UserGroups(Name) VALUES('General')
INSERT INTO dbo.UserGroups(Name) VALUES('Group6')
INSERT INTO dbo.UserGroups(Name) VALUES('Group7')
INSERT INTO dbo.UserGroups(Name) VALUES('Group8')
INSERT INTO dbo.UserGroups(Name) VALUES('Group9')
INSERT INTO dbo.UserGroups(Name) VALUES('Group10')
INSERT INTO dbo.UserGroups(Name) VALUES('Group11')
INSERT INTO dbo.UserGroups(Name) VALUES('Group12')
INSERT INTO dbo.UserGroups(Name) VALUES('Group13')
INSERT INTO dbo.UserGroups(Name) VALUES('Group14')
INSERT INTO dbo.UserGroups(Name) VALUES('Group15')
INSERT INTO dbo.UserGroups(Name) VALUES('Group16')
INSERT INTO dbo.UserGroups(Name) VALUES('Group17')
INSERT INTO dbo.UserGroups(Name) VALUES('Group18')
INSERT INTO dbo.UserGroups(Name) VALUES('Group19')
INSERT INTO dbo.UserGroups(Name) VALUES('Group20')
INSERT INTO dbo.UserGroups(Name) VALUES('Group21')
INSERT INTO dbo.UserGroups(Name) VALUES('Group22')
INSERT INTO dbo.UserGroups(Name) VALUES('Group23')
INSERT INTO dbo.UserGroups(Name) VALUES('Group24')
INSERT INTO dbo.UserGroups(Name) VALUES('Anonymous')
INSERT INTO dbo.Users(UserName, UserGroupId) VALUES('Anonymous', 25)
Add CrashReporter user with SQL authentication:
- In MS SQL Management Studio go to Security -> Logins -> Add New Login</br>
- Add SQL Server authentication
- Uncheck password policy
- Uncheck password expiration
- Default database: CrashReport
- Server Roles
- Check sysadmin
- Check CrashReport with Default schema ‘dbo’
- Right click on the server in Object explorer
- In security tab change to SQL Server and Windows Authentication mode
- From Windows command line go to services and restart the SQL Server service.
- Try to login with CrashReporter user
Crash Report Website
Open CRW solution with Visual Studio 2015:
- Build the solution
- Publish the website to a folder.
- Copy published folder to the server.
- On the server:
- Turn IIS
- Go to Windows Features and enable:
- Internet Information Services -> Web Management Tools -> IIS Management Console
- Internet Information Services -> WWW Services -> App Developement -> ASP.NET 3.5 / 4.7
- Internet Information Services -> WWW Services -> Health and Diagnostics
- Internet Information Services -> WWW Services -> Performance Features
- Internet Information Services -> Common HTTP Features -> Static Content
- Go to Windows Features and enable:
- Modify Web.config:
- DownloadFromS3 = false
- Launch IIS Manager
- Add a new website
- Select physical path to published folder
- Double click on connection strings and set the credentials to CrashReporter user for CrashReportEntities and HangFire
- In IIS add mime types:
.dmp : application/octet-stream
.log : text/plain
.runtime-xml : text/xml
In the browser go to crash reporter website. The page should be loaded with no crashes to present.
Crash Report Client
To test a crash, include a crash function into the code:
checkf(false, TEXT("Crashing on purpose via Crash command.));
A good place for this function could be a project-specific cheat manager.
To package CRC with the game add
-crashreporter switch to cooking the game
Package the game, launch and trigger a crash. If game runs with user interface a CRC window should appear.
Navigate to crashes folder and open the minidump with VisualStudio.
If Visual Studio is unable to match the PDBs that probably indicate the bug on the systems with Windows Fall Creators Update.
Consult this UDN question to resolve the issue:
(Please update to answerhub when it's available)
Crash Report Receiver
By default CRC is wired to Epic services.
UnrealEngine\Engine\Programs\CrashReportClient\Config\DefaultEngine.ini to submit crashes to your server.
- Build solution:
- Copy files to the server:
Create Landing Zone Folder and specify it in CrashReportRepository value inside the CRR config file.
Launch CRR as administartor.
The application will print few lines into console:
CrashReportReceiver, Copyright 2014-2017 Epic Games, Inc.
[STATUS] Saved to ***\ReceivedReports.xml
Landing zone: ***
Launch browser on the server and go to this page http://localhost:57005/CrashReporter/Ping
The reply must be an XML similar to this one:
<CrashReporterResult bSuccess="true" ID="0" Message="" bTimeout="false"/>
Now crash the game and inspect the output for CRR. At some point the following log messages will appear:
[STATUS] A new report has been received into ***
[STATUS] Saved to ***\ReceivedReports.xml
[PERF ] CPU Usage: 08%, Received data: 0MB/00.05 of 10.00 MB, Uploads: 0, Files: 1, 0 : 00%
Inspect Landing Zone Folder and find a new crash folder been created.
To finish this step setup a windows service to run CRR. For details consult:
The logs can be accessed at Logs folder near the CRR executable.
Crash Report Process
- Copy following files to the server:
- Modify CrashReportProcess.config
<ProcessedReports>REPORTS FOLDER </ProcessedReports>
Note that all folders must exist!
<DepotRoot>A dummy folder</DepotRoot>
<InternalLandingZone>LANDING ZONE FOLDER (SEE CRR config)</InternalLandingZone>
<ProcessedReportsIndexPath>Path to index folder</ProcessedReportsIndexPath>
Open IIS Manager:
Edit the website and add virtual folders with name CrashReporterFiles and CrashReporterVideos pointing to processed reports and videos folders
Install CRP as service.
Trigger a crash from the game.
The crash will appear in Landing Zone Folder first and will be transferred to Processed Folder.
Goto CRW and check that crash listed in the crash list
To distinguish two builds version file needs to be updated.
There is a build command with name UpdateLocalVersion that suppose to automate that. However the command requires P4 access and that doesn't work if code is stored in Git.
The idea is simple thought. Something have to update
And specify branch and changelist. A small script could handle this job.
Be aware that the file change adds up to the build time and generate lots cooking warning due to version mismatch.
The configuration above will work but requires PDBs published with executable. MDD allows to process minidumps on the server where all PDBs will have to be stored. Not that this process is not required for Linux platform.
- Open UE4 solution and build MinidumpDiagnostics (MDD)
- Copy executable
UnrealEngine\Engine\Binaries\Win64\MinidumpDiagnostics.exeto a server folder
- Modify CrashReportReceiver.config:
<MDDExecutablePath>Path To MDD exe</MDDExecutablePath>
<MDDPDBCachePath>Path to symbols Root</MDDPDBCachePath>
- Create Engine.ini in folder
- Edit Engine.ini:
To automate the upload a small script can be created that uploads files to crash report server for every packaged game. The script transfers all files that match file mask *.pdb and maintains the folder structure.
The uploader writs to symbol folder of this structure:
<Symbol Root>\Branch-CL-BuildNumber\PlatformName (In UE terms - WindowsClient, WindowsServer, etc)
Crash Reporting For Editor
The process for editor crashes is similar to packaged game with few exceptions.
CrashReportClient settings need to be set in BaseEngine.ini:
When uploading symbols these folders need to be considered: