Difference between revisions of "Unreal Engine Crash Reporter"

From Epic Wiki
Jump to: navigation, search
 
m (1 revision imported)
 
(No difference)

Latest revision as of 17:56, 28 June 2018

Template:Rating

Overview

Crash report system allows collecting crash information for developer analysis. Crash report system requires a server to process, aggregate and display crash data. This document describes the procedure for setting up the system on your own server.

The document describes the setup for running the system on a single machine. Epic is moving the system to the cloud to handle the increasing load. At the moment of writing, latest version of Unreal engine 4.19 doesn’t have source code for the CrashReportServer. The process is based on source code for 4.17.2 version.

Here is a typical timeline for a single crash:

  • An exception (or an assert) is triggered while running game application on the target system.
  • A crash folder is generated with log and dump files
  • CrashReportClient(CRC) is launched
  • CRC sends HTTP request to the crash report server with crash files
  • The crash files are received by CrashReportReceiver(CRR) on the server
  • The crash files are written to the LandingZone folder
  • CrashReportProcess(CRP) scans Landing Zone Folder and discover *new* crash folders
  • For every crash CRP:
    • If callstack is not symbolized, CRP executes the request to MinidumpDiagnostics (MDD)
    • Make an HTTP request to CrashReportWebsite(CRW) to add the crash
    • CRW add the crash to the CrashReportDatabase (CRD)
    • Move the crash report files to Processed Folder

When the crash has gone through this chain, developer could go to the crash report website in the browser and find the crash.
Note that for MDD to symbolize the callstack, a process need to upload the symbols to the server.

CrashReporter-SystemOverview.png Edit diagram File:CrashReporter-SystemOverview-Source.zip

Dependencies

CRW is written in ASP.NET with use of Entity Framework. That requires the server to run Windows OS + IIS with MS SQL database.
Before proceeding forward choose a database license that suits your demands:
https://www.microsoft.com/en-ca/sql-server/sql-server-2017-pricing

The fork for Unreal Engine with crash report server sources files and fixes:
https://github.com/NvizzioAntons/UnrealEngine-CrashReporter
In CrashReporter branch

The system has been testing on Windows and Linux. Other platforms haven’t been tested.

Configuration

The configuration process has the following steps:

  1. Server Machine Setup
  2. Crash Report Database
  3. Crash Report Website
  4. Crash Report Client
  5. Crash Report Receiver
  6. Crash Report Process
  7. Versioning
  8. Minidump Diagnostics
  9. Symbol Uploader
  10. Crash Reporting For Editor

Server Machine Setup

To start come up with the web address for the crash reporter website. Note that system is capable to aggregate crashes from different games made with Unreal Engine if that applies to you.

Verify that web address is reachable by pinging it.

Crash Report Database

Download and install MS SQL server with MS SQL Management Studio.
Modify and execute sql script: UnrealEngine\Engine\Source\Programs\CrashReporter\CrashReportWebSite\Models\CreateDatabase.sql

  • In CREATE DATABASE CrashReport statement modify the filename to a folder that exists.
  • Note that database max size depends on the license and might require editing.
  • If the following error occurs: cannot create more than one clustered index on table …

When the database is created, execute the following script:
INSERT INTO dbo.UserGroups(Name) VALUES('Group1')
INSERT INTO dbo.UserGroups(Name) VALUES('Group2')
INSERT INTO dbo.UserGroups(Name) VALUES('Coders')
INSERT INTO dbo.UserGroups(Name) VALUES('Group4')
INSERT INTO dbo.UserGroups(Name) VALUES('General')
INSERT INTO dbo.UserGroups(Name) VALUES('Group6')
INSERT INTO dbo.UserGroups(Name) VALUES('Group7')
INSERT INTO dbo.UserGroups(Name) VALUES('Group8')
INSERT INTO dbo.UserGroups(Name) VALUES('Group9')
INSERT INTO dbo.UserGroups(Name) VALUES('Group10')
INSERT INTO dbo.UserGroups(Name) VALUES('Group11')
INSERT INTO dbo.UserGroups(Name) VALUES('Group12')
INSERT INTO dbo.UserGroups(Name) VALUES('Group13')
INSERT INTO dbo.UserGroups(Name) VALUES('Group14')
INSERT INTO dbo.UserGroups(Name) VALUES('Group15')
INSERT INTO dbo.UserGroups(Name) VALUES('Group16')
INSERT INTO dbo.UserGroups(Name) VALUES('Group17')
INSERT INTO dbo.UserGroups(Name) VALUES('Group18')
INSERT INTO dbo.UserGroups(Name) VALUES('Group19')
INSERT INTO dbo.UserGroups(Name) VALUES('Group20')
INSERT INTO dbo.UserGroups(Name) VALUES('Group21')
INSERT INTO dbo.UserGroups(Name) VALUES('Group22')
INSERT INTO dbo.UserGroups(Name) VALUES('Group23')
INSERT INTO dbo.UserGroups(Name) VALUES('Group24')
INSERT INTO dbo.UserGroups(Name) VALUES('Anonymous')
INSERT INTO dbo.Users(UserName, UserGroupId) VALUES('Anonymous', 25)

Add CrashReporter user with SQL authentication:

  • In MS SQL Management Studio go to Security -> Logins -> Add New Login</br>
  • Add SQL Server authentication
    • General
      • Uncheck password policy
      • Uncheck password expiration
      • Default database: CrashReport
    • Server Roles
      • Check sysadmin
    • Check CrashReport with Default schema ‘dbo’
  • Right click on the server in Object explorer
    • In security tab change to SQL Server and Windows Authentication mode
    • From Windows command line go to services and restart the SQL Server service.
  • Try to login with CrashReporter user

Crash Report Website

Open CRW solution with Visual Studio 2015:
UnrealEngine\Engine\Source\Programs\CrashReporter\CrashReportWebSite\CrashReportWebSite.sln

  1. Build the solution
  2. Publish the website to a folder.
  3. Copy published folder to the server.
  4. On the server:
  • Turn IIS
    • Go to Windows Features and enable:
      • Internet Information Services -> Web Management Tools -> IIS Management Console
      • Internet Information Services -> WWW Services -> App Developement -> ASP.NET 3.5 / 4.7
      • Internet Information Services -> WWW Services -> Health and Diagnostics
      • Internet Information Services -> WWW Services -> Performance Features
      • Internet Information Services -> Common HTTP Features -> Static Content
  • Modify Web.config:
    • DownloadFromS3 = false
  • Launch IIS Manager
    • Add a new website
    • Select physical path to published folder
    • Double click on connection strings and set the credentials to CrashReporter user for CrashReportEntities and HangFire
    • In IIS add mime types:
      • .dmp : application/octet-stream
      • .log : text/plain
      • .runtime-xml : text/xml

In the browser go to crash reporter website. The page should be loaded with no crashes to present.

Crash Report Client

To test a crash, include a crash function into the code:

UFUNCTION(Exec)
void Crash()
{
checkf(false, TEXT("Crashing on purpose via Crash command.));
}

A good place for this function could be a project-specific cheat manager.

To package CRC with the game add -crashreporter switch to cooking the game
https://wiki.unrealengine.com/How_to_package_your_game_with_commands

Package the game, launch and trigger a crash. If game runs with user interface a CRC window should appear.

Navigate to crashes folder and open the minidump with VisualStudio.
If Visual Studio is unable to match the PDBs that probably indicate the bug on the systems with Windows Fall Creators Update.
Consult this UDN question to resolve the issue:
https://udn.unrealengine.com/questions/401015/unusable-mini-dump-with-windows-10-fall-creators-u.html
(Please update to answerhub when it's available)

Crash Report Receiver

By default CRC is wired to Epic services.

Modify UnrealEngine\Engine\Programs\CrashReportClient\Config\DefaultEngine.ini to submit crashes to your server. DataRouterUrl= CrashReportReceiverIP=webaddress.com:57005

  • Build solution: UnrealEngine\Engine\Source\Programs\CrashReporter\CrashReporterServer.sln
  • Copy files to the server:

UnrealEngine\Engine\Binaries\DotNET\CrashReportCommon.dll UnrealEngine\Engine\Binaries\DotNET\CrashReportReceiver.exe UnrealEngine\Engine\Binaries\DotNET\CrashReportReceiver.exe.config

Create Landing Zone Folder and specify it in CrashReportRepository value inside the CRR config file.

Launch CRR as administartor.
The application will print few lines into console:
CrashReportReceiver, Copyright 2014-2017 Epic Games, Inc.
[STATUS] Saved to ***\ReceivedReports.xml
Landing zone: ***

Launch browser on the server and go to this page http://localhost:57005/CrashReporter/Ping
The reply must be an XML similar to this one:
<CrashReporterResult bSuccess="true" ID="0" Message="" bTimeout="false"/>

Now crash the game and inspect the output for CRR. At some point the following log messages will appear:
[STATUS] A new report has been received into ***
[STATUS] Saved to ***\ReceivedReports.xml
[PERF ] CPU Usage: 08%, Received data: 0MB/00.05 of 10.00 MB, Uploads: 0, Files: 1, 0 : 00%

Inspect Landing Zone Folder and find a new crash folder been created.
To finish this step setup a windows service to run CRR. For details consult:
https://stackoverflow.com/questions/3582108/create-windows-service-from-executable

The logs can be accessed at Logs folder near the CRR executable.

Crash Report Process

  • Copy following files to the server:

UnrealEngine\Engine\Binaries\DotNET\AWSSDK.dll
UnrealEngine\Engine\Binaries\DotNET\CompressionHelper.dll
UnrealEngine\Engine\Binaries\DotNET\CrashReportCommon.dll
UnrealEngine\Engine\Binaries\DotNET\CrashReportProcess.exe
UnrealEngine\Engine\Binaries\DotNET\CrashReportProcess.config

  • Modify CrashReportProcess.config

<ProcessedReports>REPORTS FOLDER </ProcessedReports>
<CrashReportWebSite>WEB ADDRESS</CrashReportWebSite>
<ProcessedVideos>VIDEO FOLDER</ProcessedVideos>
<DepotRoot>A dummy folder</DepotRoot>
<InternalLandingZone>LANDING ZONE FOLDER (SEE CRR config)</InternalLandingZone>
<CrashFilesToDisk>true</CrashFilesToDisk>
<CrashFilesToAWS>false</CrashFilesToAWS>
<ProcessedReportsIndexPath>Path to index folder</ProcessedReportsIndexPath>
Note that all folders must exist!

Open IIS Manager:
Edit the website and add virtual folders with name CrashReporterFiles and CrashReporterVideos pointing to processed reports and videos folders

Install CRP as service.

Trigger a crash from the game.
The crash will appear in Landing Zone Folder first and will be transferred to Processed Folder.
Goto CRW and check that crash listed in the crash list

Versioning

To distinguish two builds version file needs to be updated.
There is a build command with name UpdateLocalVersion that suppose to automate that. However the command requires P4 access and that doesn't work if code is stored in Git.
The idea is simple thought. Something have to update
UnrealEngine\Engine\Source\Runtime\Launch\Resources\Version.h
And specify branch and changelist. A small script could handle this job.

Be aware that the file change adds up to the build time and generate lots cooking warning due to version mismatch.

MinidumpDiagnostics

The configuration above will work but requires PDBs published with executable. MDD allows to process minidumps on the server where all PDBs will have to be stored. Not that this process is not required for Linux platform.

  • Open UE4 solution and build MinidumpDiagnostics (MDD)
  • Copy executable UnrealEngine\Engine\Binaries\Win64\MinidumpDiagnostics.exe to a server folder <MDD Root>\Program\Binaries\Win64\
  • Modify CrashReportReceiver.config:

<MDDExecutablePath>Path To MDD exe</MDDExecutablePath>
<MDDPDBCachePath>Path to symbols Root</MDDPDBCachePath>

  • Create Engine.ini in folder <MDD Root>\Engine\Programs\MinidumpDiagnostics\Saved\Config\Windows\Engine.ini
  • Edit Engine.ini:

PDBCache_0_Branch=BranchName
PDBCache_0_ExecutablePathPattern=<Symbols Root>\%BUILD_VERSION%\%PLATFORM_NAME%
PDBCache_0_SymbolPathPattern=<Symbols Root>\%BUILD_VERSION%\%PLATFORM_NAME%

Symbol Uploader

To automate the upload a small script can be created that uploads files to crash report server for every packaged game. The script transfers all files that match file mask *.pdb and maintains the folder structure.

The uploader writs to symbol folder of this structure:
<Symbol Root>\Branch-CL-BuildNumber\PlatformName (In UE terms - WindowsClient, WindowsServer, etc)

Crash Reporting For Editor

The process for editor crashes is similar to packaged game with few exceptions.
CrashReportClient settings need to be set in BaseEngine.ini:
[CrashReportClient]
CrashReportReceiverIP="http://webaddress:57005"
When uploading symbols these folders need to be considered: UnrealEngine/Engine/Binaries/Win64
UnrealEngine/Engine/Plugins
UnrealProject/Binaries/Win64
UnrealProject/Plugins