QA:Crash Reporting and Analysis

From Camino Wiki
Revision as of 22:00, 3 October 2009 by Sardisson (talk | contribs) (→‎OS Symbols: provide a high-level overview of how the OS symbol generation works and the known caveats of certain steps (ruby version, libraries lists, manually renaming symbol manifests, etc.))
Jump to navigation Jump to search

Camino uses Google Breakpad for its crash-reporting client and Mozilla Crash Stats for the reporting server.

Camino Symbols

Symbol Indices: Camino-2.0b4-2009091516-i386-symbols.txt Camino-2.0b4-2009091516-ppc-symbols.txt

Symbol Files: 8baef9 51 391 10594 - Address, something, line number, file number

OS Symbols

N.B. Even if a crash report has the same build as the OS symbols, we have seen cases where one or more (even all) of the OS binaries has a different hash/Debug Identifier and thus its frames are not symbolized.

Mac OS X 10.4

Mac OS X 10.5

Mac OS X 10.6

Generating Symbols for Mac OS X Libraries

After each relevant software update, we generate Mac OS X symbols, often from a couple of different machines in an attempt to combat the "many Debug Identifiers per library per OS version" issue.

generate_macosx_symbols drives the symbol generation process, using lists of libraries (one per major OS version) from which to extract symbols, and Camino’s custom symbolstore.py, which includes support for generating symbol files for Mac OS X libraries and storing them in non-read-only locations and binaries with spaces in their names.

Call generate_macosx_symbols with three arguments (path to Camino’s custom symbolstore.py, path to the dump_syms binary, and an output path) and then pass a list of libraries to read via stdin.

generate_macosx_symbols requires Ruby 1.8.6 or higher (standard on Mac OS X 10.5), and use with a non-system Ruby requires editing the script to select the appropriate interpreter path. (It may be possible to convince the system Ruby on Mac OS X 10.4 to work by changing readpartial to read and ensuring no warning messages will be generated, e.g. by missing architectures.)

Lists of libraries are generated on new OS versions using dependency_list_generator, Mozilla applications, and a list of extra libraries and plug-ins that are often loaded but not linked against by applications.

Symbol archives are then uploaded to symbols.mozilla.org using a wrapper around upload_symbols.sh (note this script is not spaces-in-paths-friendly). Care should be taken to ensure that the symbol manifest (Mac_OS_X-10_5_8-9L31a-symbols.txt) has a unique filename in order to ensure all symbol files can be referenced afterwards (and removed once their OS is no longer supported by any shipping product); for minor version updates this is usually handled “automatically” by the version number and build number, but updates to QuickTime, Safari, Java, and Security Updates typically introduce new copies of libraries without incrementing the OS build number, so the manifest inside the zip file should manually be renamed with additional identifiers.