Difference between revisions of "QA:Crash Reporting and Analysis"

From Camino Wiki
Jump to navigation Jump to search
(→‎OS Symbols: 10.7 from smorgan, plus some tidying of the process docs)
(→‎Mac OS X 10.7: add 10.7.3 from eiichi)
 
(3 intermediate revisions by the same user not shown)
Line 114: Line 114:
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, possibly including Security Update 2011-005) <!-- stuart desktop -->
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, possibly including Security Update 2011-005) <!-- stuart desktop -->
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-Security_Update_2011-006-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, including Security Update 2011-006) <!-- stuart desktop -->
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-Security_Update_2011-006-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, including Security Update 2011-006) <!-- stuart desktop -->
 +
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-Security_Update_2012-001-v1.0-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, including Security Update 2012-001 v1.0) <!-- phiw -->
 +
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-Security_Update_2012-001-v1.1-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, including Security Update 2012-001 v1.1) <!-- phiw -->
 +
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_6_8-10K549-Security_Update_2012-001-v1.1-EY-symbols.txt Mac OS X 10.6.8 10K549] (Universal 32-bit, including Security Update 2012-001 v1.1) <!-- EY -->
  
 
===Mac OS X 10.7===
 
===Mac OS X 10.7===
Line 119: Line 122:
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_7_1-11B26-symbols.txt Mac OS X 10.7.1 11B26] (32-bit)
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_7_1-11B26-symbols.txt Mac OS X 10.7.1 11B26] (32-bit)
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_7_2-11C74-symbols.txt Mac OS X 10.7.2 11C74] (32-bit)
 
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_7_2-11C74-symbols.txt Mac OS X 10.7.2 11C74] (32-bit)
 +
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_7_3-11D50-symbols.txt Mac OS X 10.7.3 11D50] (32-bit) <!-- phiw -->
 +
* [http://symbols.mozilla.org/symbols_os/Mac_OS_X-10_7_3-11D50b-symbols.txt Mac OS X 10.7.3 11D50b] (32-bit) <!-- EY -->
  
 
===Generating Symbols for Mac OS X Libraries===
 
===Generating Symbols for Mac OS X Libraries===
 
After each relevant software update, we generate Mac OS X symbols, often from a couple of different machines in an attempt to combat the "many Debug Identifiers per library per OS version and build number" issue.
 
After each relevant software update, we generate Mac OS X symbols, often from a couple of different machines in an attempt to combat the "many Debug Identifiers per library per OS version and build number" issue.
  
−
<code>[http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/generate_macosx_symbols generate_macosx_symbols]</code> drives the symbol generation process, using lists of libraries (one per [http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_4.txt major] [http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_5.txt OS] [http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_6.txt version]) from which to extract symbols, and Camino’s custom <code>[http://mxr.mozilla.org/camino/source/scripts/symbolstore.py symbolstore.py]</code>, which includes support both for generating symbol files for Mac OS X libraries and storing them in non-read-only locations and for binaries with spaces in their names.
+
<code>[http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/generate_macosx_symbols generate_macosx_symbols]</code> drives the symbol generation process, using lists of libraries ([http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_4.txt one] per [http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_5.txt major] [http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_6.txt OS] [http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/libraries_10_7.txt version]) from which to extract symbols, and Camino’s custom <code>[http://mxr.mozilla.org/camino/source/scripts/symbolstore.py symbolstore.py]</code>, which includes support both for generating symbol files for Mac OS X libraries and storing them in non-read-only locations and for binaries with spaces in their names.
  
 
Call <code>generate_macosx_symbols</code> with three arguments (path to Camino’s custom <code>symbolstore.py</code>, path to the <code>dump_syms</code> binary, and an output path) and then pass a list of libraries to read via <code>stdin</code> (see [[#Sample invocation|below]] for an example invocation on 10.5 wrapped as a convenience script).
 
Call <code>generate_macosx_symbols</code> with three arguments (path to Camino’s custom <code>symbolstore.py</code>, path to the <code>dump_syms</code> binary, and an output path) and then pass a list of libraries to read via <code>stdin</code> (see [[#Sample invocation|below]] for an example invocation on 10.5 wrapped as a convenience script).
Line 131: Line 136:
 
Lists of libraries are generated on new OS versions using <code>[http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/dependency_list_generator dependency_list_generator]</code>, Mozilla applications, and a list of <code>[http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/extra_seed_files.txt extra libraries and plug-ins]</code> that are often loaded but not linked against by applications.
 
Lists of libraries are generated on new OS versions using <code>[http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/dependency_list_generator dependency_list_generator]</code>, Mozilla applications, and a list of <code>[http://mxr.mozilla.org/camino/source/tools/os-symbol-generation/extra_seed_files.txt extra libraries and plug-ins]</code> that are often loaded but not linked against by applications.
  
−
Symbol archives are then uploaded to symbols.mozilla.org using a [http://hg.mozilla.org/users/alqahira_ardisson.org/misc/file/tip/uploadsyms.sh wrapper] around <code>[http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.sh upload_symbols.sh]</code> (note this script is '''not''' spaces-in-paths-friendly).  Care should be taken to ensure that the symbol manifest (<code>Mac_OS_X-10_5_8-9L31a-symbols.txt</code>) has a unique filename in order to ensure all symbol files can be referenced afterwards (and removed once their OS is no longer supported by any shipping product); for minor version updates this is usually handled “automatically” by the version number and build number, '''but updates to QuickTime, Safari, Java, and Security Updates typically introduce new copies of libraries without incrementing the OS build number''', so the manifest inside the zip file should manually be renamed with additional identifiers.
+
Symbol archives are then uploaded to [http://symbols.mozilla.org/symbols_os/ symbols.mozilla.org] using a [http://hg.mozilla.org/users/alqahira_ardisson.org/misc/file/tip/uploadsyms.sh wrapper] around <code>[http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.sh upload_symbols.sh]</code> (note this script is '''not''' spaces-in-paths-friendly).  Care should be taken to ensure that the symbol manifest (<code>Mac_OS_X-10_5_8-9L31a-symbols.txt</code>) has a unique filename in order to ensure all symbol files can be referenced afterwards (and removed once their OS is no longer supported by any shipping product); for minor version updates this is usually handled “automatically” by the version number and build number, '''but updates to QuickTime, Safari, Java, and Security Updates typically introduce new copies of libraries without incrementing the OS build number''', so the manifest inside the zip file should manually be renamed with additional identifiers.
  
 
====Sample invocation====
 
====Sample invocation====

Latest revision as of 14:39, 4 February 2012

Camino uses Google Breakpad for its crash-reporting client and Mozilla Crash Stats for the reporting server.

Camino Symbols

Symbol Indices: Camino-2.0b4-i386-2009091516-symbols.txt Camino-2.0b4-ppc-2009091516-symbols.txt

Symbol Files: 8baef9 51 391 10594 - Address, length in bytes (in hex) of the machine code, line number, file number

Breakpad Symbol File documentation

By way of comparison, and for easy reference, Firefox symbol indices are firefox-3.6.5pre-Darwin-20100510033614-symbols.txt.

Plug-ins and Hacks

Mac Flash Identifiers on wiki.mozilla.org contains a table mapping Flash module IDs to Flash versions.

QA:1Password Versions contains a similar table mapping 1Password module IDs to 1Password versions.

Unsymbolized Frames on 10.4.11 PPC

10.4.11 PPC has a very, very high instance of unsymbolized frames due to non-matching Module IDs. It's a pain to be manually resymbolizing them constantly when examining new topcrashers, so QA:Resymbolized Frames contains some common resymbolized frames.

OS Symbols

N.B. Even if a crash report has the same build as the OS symbols, we have seen cases where one or more (even all) of the OS binaries has a different hash/Debug Identifier and thus its frames are not symbolized.

Mac OS X 10.4

Mac OS X 10.5

Mac OS X 10.6

Mac OS X 10.7

Generating Symbols for Mac OS X Libraries

After each relevant software update, we generate Mac OS X symbols, often from a couple of different machines in an attempt to combat the "many Debug Identifiers per library per OS version and build number" issue.

generate_macosx_symbols drives the symbol generation process, using lists of libraries (one per major OS version) from which to extract symbols, and Camino’s custom symbolstore.py, which includes support both for generating symbol files for Mac OS X libraries and storing them in non-read-only locations and for binaries with spaces in their names.

Call generate_macosx_symbols with three arguments (path to Camino’s custom symbolstore.py, path to the dump_syms binary, and an output path) and then pass a list of libraries to read via stdin (see below for an example invocation on 10.5 wrapped as a convenience script).

generate_macosx_symbols requires Ruby 1.8.6 or higher (standard on Mac OS X 10.5), and use with a non-system Ruby requires editing the script to select the appropriate interpreter path. (It may be possible to convince the system Ruby on Mac OS X 10.4 to work by changing readpartial to read and ensuring no warning messages will be generated, e.g. by missing architectures.)

Lists of libraries are generated on new OS versions using dependency_list_generator, Mozilla applications, and a list of extra libraries and plug-ins that are often loaded but not linked against by applications.

Symbol archives are then uploaded to symbols.mozilla.org using a wrapper around upload_symbols.sh (note this script is not spaces-in-paths-friendly). Care should be taken to ensure that the symbol manifest (Mac_OS_X-10_5_8-9L31a-symbols.txt) has a unique filename in order to ensure all symbol files can be referenced afterwards (and removed once their OS is no longer supported by any shipping product); for minor version updates this is usually handled “automatically” by the version number and build number, but updates to QuickTime, Safari, Java, and Security Updates typically introduce new copies of libraries without incrementing the OS build number, so the manifest inside the zip file should manually be renamed with additional identifiers.

Sample invocation

./ossymbols.sh

ossymbols.sh: 

#!/bin/sh

SRCDIR_PATH="/Users/username/Camino/dev/trunk/mozilla/camino"
DUMP_SYMS_PATH="/Users/username/Camino/dev/trunk/CaminoStaticUniTrunk/i386/camino/google-breakpad/src/tools/mac/dump_syms/build/Release/dump_syms"
OUTPUT_DIR="$SRCDIR_PATH/../.."
LIBRARIES_LIST_PATH="$SRCDIR_PATH/tools/os-symbol-generation/libraries_10_5.txt"

"$SRCDIR_PATH/tools/os-symbol-generation/generate_macosx_symbols" \
  "$SRCDIR_PATH/scripts/symbolstore.py" \
  "$DUMP_SYMS_PATH" "$OUTPUT_DIR" < "$LIBRARIES_LIST_PATH"

For Mac OS X 10.5, 10.6, and 10.7, there's a self-contained AppleScript wrapper app that automates the entire symbol generation process; this is useful if you need to get symbols from a user who is not familiar with this process.

Retrieved from "http://wiki.caminobrowser.org/index.php?title=QA:Crash_Reporting_and_Analysis&oldid=12069"