Crashpad is a cross platform system for end-to-end crash reporting.
Crashpad supports reporting of native crashes on a variety of operating systems including Windows,
MacOS, Linux, Android and iOS.
Crashpad also provides tools such as dump_syms
, symupload
and
minidump_stackwalk
that provide developers with function names, file names and line
numbers in their crash reports.
Integrating with Crashpad helps software engineers find and fix program crashes in order to develop
more stable applications.
The Chromium depot_tools are a set of tools that are required to build Crashpad. The documentation claims that Python 2.7 is a requirement for depot_tools, however since checking out Crashpad uses only a subset of the depot tools Python 3+ worked fine for BugSplat.
The Crashpad source code can be found here.
Crashpad’s dependencies are managed by gclient
instead of git submodules, so it is best
to use fetch
to get the source code.
Crashpad uses gn
to generate ninja
build files.
By default gn generates a configuration for building static libraries. If you would like to build dynamic libraries see this post on Stack Overflow.
Building Crashpad generates several files that need to be linked with an application in order to generate crash reports.
At minimum MacOS and Linux applications need to be linked with
out/Default/obj/client/libclient.a
,
out/Default/obj/util/libutil.a
, and
out/Default/obj/third_party/mini_chromium/mini_chromium/base/libbase.a
.
MacOS application willl need to link with all of the .o files in
out/Default/obj/out/Default/gen/util/mach
as well.
When building Linux applications, libbase.a
needs to be the last Crashpad file
specified in the build arguments otherwise the application will not build.
Additionally,
~/crashpad/crashpad
and
~/crashpad/crashpad/third_party/mini_chromium/mini_chromium
need to be added as include directories.
Finally,
out/Default/crashpad_handler
needs to be deployed with the application and accessible at runtime.
At minimum Windows applications need to be linked with
out\Default\obj\client\client.lib
,
out\Default\obj\util\util.lib
, and
out\Default\obj\third_party\mini_chromium\mini_chromium\base\base.lib.
Additionally,
out\Default\crashpad_handler.exe
needs to be deployed with the application and accessible at runtime.
Add the following includes to the entry point of the application.
Add a typedef for StringType
,
add using statements for the base
, crashpad
and std
namespace
and declare the following methods at the top of the entry point of the application.
Implement the initializeCrashpad
method replacing the file paths with valid values.
Next, implement the platform specific getExecutableDir
method.
Finally, call the initializeCrashpad
method at the entry point of the application.
Generating sym files requires the dump_syms
tool from the repository of Crashpad’s
predecessor,
Breakpad.
Dump_syms
creates
sym
files
from executable binaries so that minidumps can be symbolicated to determine the function names, file
names
and line numbers in the call stack.
Build the application with symbolic information (preferably in a separate dSYM file) in order to get fully symbolicated crash reports.
Next, build the Xcode project located at
src/src/tools/mac/dump_syms/dump_syms.xcodeproj
. Switch the configuration to dump_syms
and
build the project.
The report navigator tab (icon looks like a chat bubble in Xcode 11) will show the file system
location
with
the compiled executable.
Run the dump_syms executable.
Build the application with symbolic information and a build identifier.
Using clang
this means building with the -g
and -Wl,--build-id
arguments.
Next, run ./configure && make
in the Breakpad directory.
This will generate dump_syms
, symupload
and minidump_stackwalk
.
The dump_syms functionality is built into the
symupload
utility and can be skipped if the application will be symobilcated remotely. Run
dump_syms
only
if the application will be symbolicated locally or the sym file will be uploaded to a remote server
via
some
means other than symupload.
In order for dump_syms.exe
to generate the correct output, applications must be built
with
symbolic information so that each exe and dll file generates a corresponding pdb file.
Generated pdb files must contain full debug information.
With Visual Studio, full debug information can be generated with the
/Zi
compiler argument and the
/DEBUG:FULL
linker argument. Failure to specify either the
/Zi
or
/DEBUG:FULL
arguments will result in dump_syms outputting incorrect sym file data. Additionally the output pdb
file
must
be in the same folder as the corresponding exe or dll file otherwise dump_syms.exe will fail.
In order to run dump_syms.exe a copy of
msdia140.dll
must be placed in the same folder. If
Visual
Studio
is installed this file can be found at
[VisualStudioFolder]\DIA SDK\bin\amd64\msdia140.dll
. Copy msdia140.dll into the same
folder
as
dump_syms.exe and run dump_syms.exe.
Dump_syms can be built from source so that the debugger can be used for troubleshooting. To build
dump_syms
clone the
gyp repository
and run
python setup.py install
. Next, run
gyp ~\breakpad\src\tools\windows\dump_syms\dump_syms.gyp
and use Visual Studio to build the sln file generated by gyp. If the build fails with an error that
unique_pointer is not part of std add
#include <memory>
to the top of the file that contains the error and rebuild.
The symupload
tool is also part of the
Breakpad
repository and can be used to upload sym files to your server. Symbols need to be uploaded to a server
in
order for it to correctly symbolicate a minidump file.
Build the Xcode project located at
src/src/tools/mac/symupload/symupload.xcodeproj
. The report navigator tab (icon looks
like
a
chat bubble in Xcode 11) will show you the file system location with the compiled executable. Copy the
symupload file into your project and run the executable wrapping the arguments to symupload in quotes.
In order to use symupload applications must be built with symbolic information so that each exe and
dll
file generates a corresponding pdb file. Generated pdb files must contain full debug information. Full
debug
information can be generated with the
/Zi
compiler argument and the
/DEBUG:FULL
linker argument. Failure to specify either the
/Zi
or
/DEBUG:FULL
arguments will result in symupload failing entirely. The output pdb file must be in the same folder as
the
corresponding exe or dll file.
In order to run symupload.exe a copy of msdia140.dll
must be placed in the same folder.
If
Visual
Studio
is installed this file can be found at
[VisualStudioFolder]\DIA SDK\bin\amd64\msdia140.dll
.
Copy msdia140.dll into the same folder as symupload.exe and run symupload.exe.
Symupload can be built from source so that the debugger can be used for troubleshooting. To build
symupload
clone the
gyp repository
and run
python setup.py install
. Next, run
gyp ~\breakpad\src\tools\windows\symupload\symupload.gyp
and use Visual Studio to build the sln file generated by gyp. If the build fails with an error that
unique_pointer is not part of std add
#include <memory>
to the top of the file that contains the error and rebuild.
Minidump_stackwalk
is another tool in the
Breakpad
repository that is responsible for the symbolication of minidump files. In order to correctly
symbolicate
minidumps, sym files need to be nested at least 2 folders deep. The topmost parent folder’s name must
equal
the sym files module name. The first child folder’s name must equal the module id. Additionally, the
sym
file name must also match the module name. The module id and module name can be found in the
module
record
of the sym file.
For example the module myApp
with the module id
1A67F3DEAACA3B209D9992871B2620AA0
must be located at
/path/to/symbols/myApp/1A67F3DEAACA3B209D9992871B2620AA0/myApp.sym
.
Minidump_stackwalk is built when building Breakpad.
Run minidump_stackwalk
passing it a path to a dmp file and a path to a symbols
directory.
The folders in the symbols directory need to be laid out following the pattern
module_name/module_id/module_name.sym
:
Minidump_stackwalk is part of the Breakpad processor
sln which is generated by
gyp
At the time of writing the processor.sln
file will not build on Windows 10 with Visual
Studio
2019. However, minidumps generated on the Windows platform can be symbolicated on OS X or by a 3rd
party
service such as
BugSplat. Additionally, minidumps generated by
the
Crashpad library can also be symbolicated via
Debugging
Tools
for Windows given the .exe
, .dll
and .pdb
files are made
available to
WinDBG.
Most issues symbolicating dump files can be traced back to mismatched module ids.
Anything that modifies a given executable (code-signing, anti-cheat) must be performed before
dump_syms
and symupload
are run.
Every time an executable is modified dump_syms
and symupload
need to be
re-run.
The name and id of the module loaded at runtime can be found in the minidump_stackwalk
output.
The module name and id must match the module name and id of the generated sym file in order for
minidump_stackwalk
to correctly symbolicate the minidump file.