QA:Crash Reporting and Analysis

From Camino Wiki
Revision as of 11:11, 28 October 2011 by Sardisson (talk | contribs) (→‎OS Symbols: 10.7 from smorgan, plus some tidying of the process docs)
Jump to navigation Jump to search

Camino uses Google Breakpad for its crash-reporting client and Mozilla Crash Stats for the reporting server.

Camino Symbols

Symbol Indices: Camino-2.0b4-i386-2009091516-symbols.txt Camino-2.0b4-ppc-2009091516-symbols.txt

Symbol Files: 8baef9 51 391 10594 - Address, length in bytes (in hex) of the machine code, line number, file number

Breakpad Symbol File documentation

By way of comparison, and for easy reference, Firefox symbol indices are firefox-3.6.5pre-Darwin-20100510033614-symbols.txt.

Plug-ins and Hacks

Mac Flash Identifiers on wiki.mozilla.org contains a table mapping Flash module IDs to Flash versions.

QA:1Password Versions contains a similar table mapping 1Password module IDs to 1Password versions.

Unsymbolized Frames on 10.4.11 PPC

10.4.11 PPC has a very, very high instance of unsymbolized frames due to non-matching Module IDs. It's a pain to be manually resymbolizing them constantly when examining new topcrashers, so QA:Resymbolized Frames contains some common resymbolized frames.

OS Symbols

N.B. Even if a crash report has the same build as the OS symbols, we have seen cases where one or more (even all) of the OS binaries has a different hash/Debug Identifier and thus its frames are not symbolized.

Mac OS X 10.4

Mac OS X 10.5

Mac OS X 10.6

Mac OS X 10.7

Generating Symbols for Mac OS X Libraries

After each relevant software update, we generate Mac OS X symbols, often from a couple of different machines in an attempt to combat the "many Debug Identifiers per library per OS version and build number" issue.

generate_macosx_symbols drives the symbol generation process, using lists of libraries (one per major OS version) from which to extract symbols, and Camino’s custom symbolstore.py, which includes support both for generating symbol files for Mac OS X libraries and storing them in non-read-only locations and for binaries with spaces in their names.

Call generate_macosx_symbols with three arguments (path to Camino’s custom symbolstore.py, path to the dump_syms binary, and an output path) and then pass a list of libraries to read via stdin (see below for an example invocation on 10.5 wrapped as a convenience script).

generate_macosx_symbols requires Ruby 1.8.6 or higher (standard on Mac OS X 10.5), and use with a non-system Ruby requires editing the script to select the appropriate interpreter path. (It may be possible to convince the system Ruby on Mac OS X 10.4 to work by changing readpartial to read and ensuring no warning messages will be generated, e.g. by missing architectures.)

Lists of libraries are generated on new OS versions using dependency_list_generator, Mozilla applications, and a list of extra libraries and plug-ins that are often loaded but not linked against by applications.

Symbol archives are then uploaded to symbols.mozilla.org using a wrapper around upload_symbols.sh (note this script is not spaces-in-paths-friendly). Care should be taken to ensure that the symbol manifest (Mac_OS_X-10_5_8-9L31a-symbols.txt) has a unique filename in order to ensure all symbol files can be referenced afterwards (and removed once their OS is no longer supported by any shipping product); for minor version updates this is usually handled “automatically” by the version number and build number, but updates to QuickTime, Safari, Java, and Security Updates typically introduce new copies of libraries without incrementing the OS build number, so the manifest inside the zip file should manually be renamed with additional identifiers.

Sample invocation

./ossymbols.sh

ossymbols.sh: 

#!/bin/sh

SRCDIR_PATH="/Users/username/Camino/dev/trunk/mozilla/camino"
DUMP_SYMS_PATH="/Users/username/Camino/dev/trunk/CaminoStaticUniTrunk/i386/camino/google-breakpad/src/tools/mac/dump_syms/build/Release/dump_syms"
OUTPUT_DIR="$SRCDIR_PATH/../.."
LIBRARIES_LIST_PATH="$SRCDIR_PATH/tools/os-symbol-generation/libraries_10_5.txt"

"$SRCDIR_PATH/tools/os-symbol-generation/generate_macosx_symbols" \
  "$SRCDIR_PATH/scripts/symbolstore.py" \
  "$DUMP_SYMS_PATH" "$OUTPUT_DIR" < "$LIBRARIES_LIST_PATH"

For Mac OS X 10.5, 10.6, and 10.7, there's a self-contained AppleScript wrapper app that automates the entire symbol generation process; this is useful if you need to get symbols from a user who is not familiar with this process.

Retrieved from "http://wiki.caminobrowser.org/index.php?title=QA:Crash_Reporting_and_Analysis&oldid=11955"