QA:Triage Policies and Procedures

From Camino Wiki
Revision as of 23:11, 10 January 2011 by Sardisson (talk | contribs) (→‎The very basics - queries, bugzilla search, etc.: fix link, because apparently we renamed that)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
DRAFT
This page is not complete.

Getting Started with Camino Bug Triage

The very basics - queries, bugzilla search, etc.

Getting involved with Camino bug triage requires no prior skills, only a recent nightly build and the willingness to spend a little bit of time (as little as 5 minutes a day, 2-3 days a week). If you're interested in helping out, please read this document (and [specify] some of the linked ones) and contact one of the leads or senior members of the Bug Triage team on #camino on irc.mozilla.org. While you don't have to participate on irc to do triage, it is helpful when starting out [tips] and always good to stop by at least occasionally to stay "in sync" with the rest of the team and the Camino project.

Once you've done your reading and said hello, you're ready to jump right in. At the most basic level, Camino bug triage involves weeding through all the bugs in the UNCONFIRMED state and either trying to reproduce the bug and get a clear set of steps required to do so, or finding a previously filed bug of which the new bug is a duplicate. You should always test in a recent nightly build (no more than a week old), either from the trunk or, if there is ongoing development (often on a branch) towards a release, from the branch (for Camino 2.1, that's the Camino 2.1 nightlies). It's also helpful to have the latest stable release and corresponding Firefox versions on hand, but to start out, simply testing bugs in a recent nightly build is fine.

QA Wanted

There are often bugs that have been previously tagged by triagers as needing additional QA work, often adding a (reduced) testcase, testing on a different OS version or language, or by someone else who has an account on a given site. Other bugs may need more extensive QA work (e.g., testing older nightlies to find a regression window, sampling with Shark or running in gdb). These bugs are all marked with the qawanted keyword (Camino qawanted query). QA and triage team members should check for these bugs regularly and see if they have the skills or hardware required to perform the needed QA.

Querying for UNCOs

Most of the work of the triage and QA team is involved in working with UNCONFIRMED bugs, typically new bugs that have just recently been filed (Camino UNCO query).

While most UNCONFIRMED bugs follow the steps below, there are two classes of UNCONFIRMED bugs that should be handled differently. These are bugs that are filed with the severity of enhancement (or enhancement requests that are misfiled as other severities; anything asking for a new feature, or additions to an existing feature, is an enhancement), and bugs that are "pseudo-support requests." Enhancement requests are only confirmed after consultation among the Camino development team. "Pseudo-support requests" should be closed as INCOMPLETE or INVALID (or occasionally WORKSFORME in cases, e.g., where the reporter cannot find the right setting) and the reporter should be redirected to support resources such as the forum or Camino’s documentation (http://www.caminobrowser.org/help/).

In the beginning, simply try to reproduce the bug and then leave a comment with your findings, e.g., "I can reproduce this bug on 10.4.7/Intel using the reporter's steps and Camino 20060924 1.0+" or "I'm unable to reproduce this bug on 10.3.9 using Camino 20060923 1.2+; can someone check to see if it reproduces on 10.4?"

If you can reproduce the bug and the bug involves something in the web page (the "content area"), it's very likely a Gecko bug and not a Camino bug, and you should search for duplicates and check (or ask someone else to check) the equivalent Firefox version. Bugs in the Camino application itself (the "chrome", and most of the ways Camino interacts with the rest of Mac OS X) can also have duplicates, and they're easier to find (you can search through all the CLOSED Camino bugs for reports similar to the bug you’re triaging). For more information on how Camino QA confirms bugs, see Confirming Bugs below.

Once you've found a bug is a DUPLICATE or has clear steps to reproduce (and a testcase, if relevant, for most bugs involving the content area) and you or another member of the QA or development team can actually reproduce it, comment in the bug with your findings. A development or senior member of the QA team will make the appropriate changes to the bug.

After you have proven yourself to be a useful and responsible member of the triage team, you'll be given Bugzilla privileges that allow you to make substantive changes to bugs yourself, including fixing the severity, keywords, and branch/trunk if relevant.

Anything else in Getting Started?

Common QA Troubleshooting Tips

  • [1:39pm] smorgan: We'll have to start being more diligent about asking about input managers
    [1:39pm] smorgan: up front, that is
    [1:39pm] smorgan: "I have problem x" "Do you have any input managers installed (e.g., 1Password)?"
    [1:40pm] smorgan: Midly annoying, but probably necessary
    [1:41pm] smorgan: We'll have to specifically mention 1Password, since people mostly don't know how it's implemented
  • It’s always useful to have users create a backup of their profile before testing something (since at various times Gecko changes will cause irreversible profile damage), or to have them use Troubleshoot Camino.
  • [01:06am] smorgan: Clean profile is supposed to be a narrowing down tool, not a magical remover of bugs. Only corrupt/invalid profiles that appear to be caused by users mucking around should be considered ignorable.
  • Clean profile/org.mozilla.camino.plist/check for haxies in (~)/Library/InputManagers and (~)/Library/InputManagers/SIMBL
  • When CAMINO_PROFILE_DIR-enabled builds become stable builds, we need a mini app/script that you drag Camino onto to have it launch with a temp directory. Then there's no room for accidental data loss while troubleshooting. [smorgan on #camino]
  • Camino Diagnostics collects system info, installed hacks and plugins, last crash, console.log entries related to Camino, and vnmap output
  • Console.log
  • http headers (browser and server)
    • curl -I 'foo'
    • the JS bookmarklet to catch server response:
    javascript:document.location.href%20=%20'http://webtools.mozilla.org/web-sniffer/?url='%20+%20escape(document.location.href)
  • cookie issues
  • Random (but consistent) crashes, usually in Camino code (org.mozilla.camino or libobjc.dylib/system frameworks as the crashed thread):
    • run with NSZombieEnabled (assumes Camino is installed in the standard location, and will eat lots of RAM and log to the console when you crash):
      NSZombieEnabled=YES /Applications/Camino.app/Contents/MacOS/Camino
      A more extensive version of the above, useful in some cases where the above is ineffective, is:
      MallocScribble=1 MallocPreScribble=1 MallocGuardEdges=1 NSZombieEnabled=YES /Applications/Camino.app/Contents/MacOS/Camino
      • someone remind sardisson to write AppleScript wrappers for these
  • http://kb.mozillazine.org/Standard_diagnostic_%28Firefox%29 (we need a Camino version; see User:Sardisson/To_Do)
  • QA:AppleEvent_Logging for investigating AppleEvent/AppleScript failures (draft; needs work)
  • Blocked cookies (either too restrictive a global setting or some domain set to deny; many sites require cookies from multiple seemingly-unconnected domains in order to function).
    • This is sometimes the case when something works in one browser but inexplicably doesn't work in Camino; users often have more restrictive cookie settings in their main browser but don't mess with the cookie prefs in alternate browsers.
  • Move UA spoofing from hidden prefs to this page, to make it clear that it's a QA tool, not something we "endorse" for everyday use like other hidden prefs. If people are smart enough to find the instructions themselves, we're going to hope they're smart enough to use it responsibly.....
  • [11:40pm] smorgan: Someone should make CaminoConflictCatcher
    [11:41pm] ardissone|in: lol
    [11:41pm] smorgan: Copies the profile to a temp location, launches there, and removes prefs.js lines one by one, relaunching each time
    [11:41pm] smorgan: And it pops up a box when you quit: "Was the bug still there?"
    [11:41pm] smorgan: And then tells you what line it last removed when it said no
    [11:41pm] smorgan: That would be awesome
    [11:42pm] ardissone|in: sounds like a job for Wevah Wonka's Cocoa Factory
    [11:42pm] ardissone|in: or Froodian's Camino Therapy Couch

Searching for Duplicate Bugs

Confirming Bugs

  • Don't confirm RFEs (bugs with severity set to enhancement, or enhancement requests misfiled with some other sort of bug severity, which are very common, particularly if another browser provides such a feature) until there has been discussion among the triage team and, in some cases, with the project lead(s). This helps combat feature creep and keeps Camino from having a kitchen sink.
  • Don't confirm any bug that looks like it has the potential of being a Core bug (anything that has to do with the content area, or with certain app features whose underlying implementation is tightly linked with the Core, e.g. the pop-up blocker, session or global history); see Moving Bugs to Other Products and Components below for more about moving bugs to Core components.
  • Don't confirm bugs that look like they might be resolved DUPLICATE, INVALID, or WONTFIX. Perform a search of old Camino bugs with these resolutions before confirming bugs (especially the latter two; a request or bug may have been WONTFIXed before you began QA work, but that doesn't mean the project lead(s) have changed their opinion). See QA:Bugzilla:Searching For Dupes for more on searching for duplicates, including duplicates of WONTFIXed and INVALID bugs.
  • If the bug has a clean report and is clearly a real bug in the Camino application UI, then by all means confirm the bug. Otherwise, proceed more cautiously, seek additional opinions, etc., before confirming a bug.
  • When you confirm a Camino bug, you should also do the following:
    • Fix other fields (Camino component and QA contact, branch, mis-filed severity)...
    • CCing developers if appropriate; see their areas of expertise in Development:Reviewing
    • Tagging bugs that only appear in one or more major Mac OS X versions...
      • Add [10.4] or [10.3] to the beginning of the summary when it's clear a bug only appears on 10.4 or 10.3. (During periods of time when we support 3 or more Mac OS X versions, a bug that appears in two or more major Mac OS X versions can have multiple version tags.)
    • Adding keywords
    • Regression windows
    • Adding to [meta] trackers, blocks/depends management
  • In most cases, you (nor the reporter) should set the set priority and target milestone; these fields are reserved for the use of release drivers when scoping and driving releases.
  • WONTFIXing bugs: Within the Mozilla project, WONTFIXing bugs is the purview of the module owner. Within Camino, the unanimous agreement of 3 regular QA/triage team members, at least one of whom must be a Camino QA/Triage Lead or the Team Coordinator (Samuel Sidler), is usually also sufficient to WONTFIX a bug. In cases of disagreement, the bug needs to be referred to the Camino lead/module owner (pinkerton) for further discussion and final decision.

Moving Bugs to Other Products and Components

  • Be sure to check the box to reset the assignee and QA contact to the defaults for the new component
  • Status (UNCO vs. NEW; what's required)
  • Testcases (esp. for Core:Layout and Core:Printing (also for DOM-related stuff); link to bugathon doc)
  • Moving to Tech Evangelism product
    • What qualifies as a TE bug
    • TE summary field format (Item 10)
  • Core:General component, for bugs you're sure are Core bugs but unsure of the proper component
  • Setting and Using Flags
  • Setting Milestone, Priority, and Severity [link to bugzilla page on severity]
  • Adding keywords and status whiteboard info
  • http://www.mozilla.org/quality/help/bugzilla-privilege-guide.html covers a lot of this

Regression Windows

One of the common tasks required of triagers is finding a “regression window” for something that used to work correctly but then broke at some point. This is an important but often tedious task that requires downloading older Camino builds, launching them, and trying to reproduce the bug.

For time-saving and ease when trying to find a regression window, Camino builds can be run from the disk image. Camino builds from 26-Mar-2006 support CAMINO_PROFILE_DIR, so they can be run (using the Terminal or Troubleshoot Camino) while your normal Camino is running, or using a custom clean profile with only the relevant preferences (if any) set.

If you have no idea when the behavior regressed, you can narrow down the range to search by checking releases and milestone builds (if the behavior regressed on a branch) and then by checking the nightly from the first of the month, etc., until you get a workable range, and then continue narrowing down the builds until you find the last build in which the bug did not appear and the first build in which it did.

Bonsai queries (module Camino vs. directory mozilla/camino), branch vs. trunk

“CLOSEME”

Often a reporter will file a bug and not provide enough information for us to adequately triage or act upon the bug. Even after we prompt the reporter for more information or suggest the reporter perform common troubleshooting steps (for bugs that clearly work for us), the reporter may not reply with any updates. Rather than let these bugs sit around on the UNCONFIRMED list gathering dust, we periodically (weekly or biweekly) run through the UNCONFIRMED list and mark bugs as “CLOSEME” bugs.

Bugs which are candidates for “CLOSEME” status are UNCONFIRMED bugs that

  1. Are otherwise WORKSFORME for all members of the triage team who have looked at the bug, or
    Are lacking sufficient information to understand or reproduce the bug (e.g., no URL, no steps to reproduce [STR])
  2. Have had an initial pass by the triage team
  3. Have had a triage team member prompt the reporter for whatever additional information is necessary, or
    Have had a triage team member request the reporter to perform certain common troubleshooting techniques (e.g., fresh profile)
  4. Have had no update from the reporter in one week (or more) since last prompted.

If a bug meets all of these criteria, it can be added to the “CLOSEME” list in the following manner:

  1. Prompt the reporter one final time for information, updates, and/or to reply to previous comments in the bug, and
  2. Add [CLOSEME - MM/DD] to the status whiteboard, where MM/DD is the date of the closest Sunday at least five days away.

If a reporter still has not updated the bug by the Sunday night in question, the weekly sweep of “CLOSEME” bugs (query) will catch the bug, and it can be closed in this manner:

  1. Verify that the reporter has not provided additional information or updates as requested and that the current date and the CLOSEME date match (don't close next week's CLOSEMEs).
    (If the reporter has provided additional information, the CLOSEME tag should be removed and the bug should not be closed, even if the bug cannot yet be confirmed or otherwise resolved.)
  2. Close the bug with the appropriate resolution:
    1. INCOMPLETE
      • If the bug has no useful STR or other information (e.g., screenshot, URL), comment in the bug “Closing INCOMPLETE due to lack of updates. If you can supply the requested additional information, you may reopen this bug.” (or similar, depending on the exact nature of the bug).
      • Resolve the bug INCOMPLETE, but do not remove the [CLOSEME] tag from the status whiteboard.
    2. WORKSFORME
      • If the bug has decent STR or other possibly-useful information (e.g., screenshot, URL) but still cannot be reproduced by developers or triage team members, comment in the bug “Closing WORKSFORME due to lack of updates. If you can supply the requested additional information, you may reopen this bug.” (or similar, depending on the exact nature of the bug).
      • Resolve the bug WORKSFORME, but do not remove the [CLOSEME] tag from the status whiteboard.

Bugs closed through this mechanism should not be verified if the bug was closed because we were still lacking key information needed to attempt to reproduce the bug (or similar). If the steps to reproduce are clear and those steps do actually WFM for members of the triage team, the “CLOSEME” bug should have been closed WORKSFORME instead (and can be VERIFIED).

Verifying Bugs

The verification policy in Camino currently tends to omit verification of FIXED bugs, although users and QA are free to verify recently fixed bugs if they like.

Bugs resolved DUPLICATE, WORKSFORME, or INVALID are usually verified, but see above for a caveat when verifying bugs resolved WORKSFORME or INCOMPLETE through the “CLOSEME” system.

Crash Report Analysis

General Mozilla Policies

Retrieved from "http://wiki.caminobrowser.org/index.php?title=QA:Triage_Policies_and_Procedures&oldid=11610"