Releases:Branch Tag Build Release
How to (branch and tag and build and) release a new version of Camino
by Mark Mentovai
on 2007-02-28
Updated for Mercurial by Smokey Ardisson
on 2011-07-25
Pinkerton keeps rooting for me to get hit by a bus. I’ve had pretty good luck so far, in large part because I’ve abandoned my old habit of running after buses while they’re in motion and banging on the doors, but even so, the specifics of our release procedure probably should be documented.
Contents
Source checkout, branching, and tagging for releases in Hg
These instructions correspond to Camino 2.1b1 (Gecko 1.9.2.20pre).
The Gecko base branch for this release was default on releases/mozilla-1.9.2 (normally this would be something like GECKO19218_2011061302_RELBRANCH with corresponding release tag of FIREFOX_3_6_19_RELEASE). This specifies the apex of the minibranch. For the 2.1b1 release, changes to that branch for Camino were made on a new CAMINO_2_1_B1_MINIBRANCH branch, and the release tag for Camino was CAMINO_2_1_B1_RELEASE.
The Camino base branch for this release was default on camino. This specifies the apex of the minibranch. For the 2.1b1 release, changes to that branch for the release were made on a new CAMINO_2_1_B1_MINIBRANCH branch, and the release tag was CAMINO_2_1_B1_RELEASE.
Camino branch and release tags follow this convention (CAMINO_1_1_A2_MINIBRANCH, CAMINO_1_1_B_RELEASE).
Clone the source
For stable releases, you should always start from a Gecko release branch, which has a “final” Gecko version number in config/milestone.txt. Either simply tag Gecko on this release branch—at the Firefox release tag, e.g. the revision corresponding to FIREFOX_3_6_19_RELEASE—with the Camino release tag, or, if you need to minibranch Gecko, create a Camino minibranch based on the Gecko release branch and land code and tag there. (For milestone releases, it is possible to just tag default on releases/mozilla-1.9.2, as was done for Camino 2.1a1.)
Because cloning a Mercurial repository from the source can take more than half an hour, it is possible to “cheat” by reusing your source tree, either by cloning locally to a new folder or ensuring your existing tree is clear of patches and untracked files (use hg stat to find modified or untracked files and back out everything, or hg revert all to throw away all changes).
Before pulling the source, you may want to add the target directory to Spotlight’s “Privacy” list.
hg clone http://hg.mozilla.org/releases/mozilla-1.9.2/ mozilla hg clone http://hg.mozilla.org/camino/ mozilla/camino
Fork a Gecko minibranch and land changes
If necessary, fork a Gecko minibranch before tagging if Camino-specific changes to Gecko are needed, or simply update to the existing Gecko release branch if all you need to do is tag.
cd mozilla hg update $GeckoRelbranchName hg branch
Use hg branch and hg outgoing liberally to make sure that you are on the correct branch after any branch-switching operations like hg update $GeckoRelbranchName and before committing any changes.
If changes to Gecko are required, fork a Camino-specific minibranch:
hg branch CAMINO_2_1_B1_MINIBRANCH hg branch
and land changes:
patch -p1 < ../../10.7scrollbars-1.9.2-complete.diff hg commit -m "Bug 656990 - Ensure compatibility with OS X 10.7's arrowless scrollbar. Original patch by Steven Michaud <smichaud@pobox.com>, r=mstange; backported by smorgan and me. a=smorgan,me for CAMINO_2_1_B1_MINIBRANCH" widget/public/nsILookAndFeel.h widget/src/cocoa/nsLookAndFeel.mm widget/src/cocoa/nsNativeThemeCocoa.mm widget/src/cocoa/nsToolkit.h widget/src/cocoa/nsToolkit.mm hg push -f
Note that if you are working around release time (e.g., you started with a Gecko release branch), the checkin comment for any commits to the Gecko repository may need to contain "CLOSED TREE a=release" in order to be accepted.
Note also that to commit to a new minibranch, you must use the -f argument to hg push in order to create the new head.
If no Gecko changes are required, simply updating to the Gecko release branch (as described in the first set of commands in this section) will put you in good shape for tagging.
Tag the Gecko side of the release
Use hg branch to verify that you are on the branch you think you are on, then tag the correct changeset. For a stable release, that changeset should be the changeset that has the Firefox release tag, or the tip of the Camino minibranch if Camino needs changes to Gecko. For a milestone, you may simply be able to tag the tip changeset of default. Use the pushlog to verify the changeset IDs if necessary. Push to the repository to commit your tag.
hg branch hg tag -r e7afd718aa42 CAMINO_2_1_B1_RELEASE -m "Added tag CAMINO_2_1_B1_RELEASE for changeset e7afd718aa42. CLOSED TREE a=release" hg push
Finally, now that you are done tagging, update your Gecko tree back to the default branch and verify:
hg update default hg branch
Fork a Camino minibranch and land changes
Unlike with Gecko, you will always need to fork a Camino minibranch to bump the version number and tag the release. In most cases you shouldn't have to land other changes, however.
cd camino hg branch CAMINO_2_1_B1_MINIBRANCH hg branch echo 2.1b1 > config/version.txt hg commit -m "Numbers for Camino 2.1b1" config/version.txt hg push -f
Again, note that to commit to the new minibranch, you must use the -f argument to hg push in order to create the new head.
Tag the Camino side of the release
Use hg branch to verify that you are on the branch you think you are on, then tag the correct changeset. Use the pushlog to verify the changeset IDs if necessary.
hg branch hg tag -r 7176ee3c0c26 CAMINO_2_1_B1_RELEASE -m "Added tag CAMINO_2_1_B1_RELEASE for changeset 7176ee3c0c26. CLOSED TREE a=release" hg push
N.B. If this is the first time you are creating a tag in a repository, you may have to commit .hgtags manually (tagging normally automatically generates a commit to the file); in that case, re-use the commit message for the tag when committing the file in order to generate the correct commit message in the remote repository:
hg commit -m "Added tag CAMINO_2_1_B1_RELEASE for changeset 7176ee3c0c26. CLOSED TREE a=release" .hgtags
Finally, now that you are done tagging, update the Camino section of your tree back to the default branch and verify. This is also a good time to change the Camino version number on the default branch:
hg update default hg branch echo 2.1b2pre > config/version.txt hg commit -m "Next release from here may or may not be Camino 2.1b2" config/version.txt hg push
Fix your (my) mistakes
So far we’ve managed not to make mistakes during Hg releases, so these steps are still theoretical and untested in production release-wrangling. If there are problems, in general you would update to the minibranch, make changes and commit, and then move the tag:
hg update $BranchName hg branch <make changes> hg commit
Again, if you’re working on a Camino minibranch in the Gecko repository during release time, your commit message may have to include "CLOSED TREE a=release" in order to be accepted.
If someone else is also making changes to the branch (this might happen if you created a tag on a Gecko release branch and further Gecko changes were necessary for a crash or firedrill, in which case you need to move the Camino release tag), be sure to use hg pull -u -r $BranchName in order to ensure you have the latest changesets from that branch and that your working directory remains set on that branch.
hg update $GeckoRelbranchName hg branch hg pull -u -r $GeckoRelbranchName hg branch
If someone else is also making changes to the Gecko release branch and you already have a Camino minibranch in Gecko, you’ll have to pull the new Gecko changes over to the Camino minibranch manually (the off-by-default-but-shipped-with-Mercurial transplant extension will help in this case; be sure to hg update on the default branch or the minibranch from which you wish to transplant before beginning, in order for Mercurial/transplant to know about those revisions).
hg update default hg pull -u hg update $BranchName hg branch hg pull -u -r $BranchName hg branch hg transplant -b $GeckoRelbranchName $REV1 $REV2 $REV3 … hg branch hg outgoing hg push
If you have pre-commit/pre-push hooks, make sure they account for the commit message syntax in the changesets you are transplanting.
Once you’ve gotten the new Gecko changes onto the Camino minibranch in Gecko, move the Camino release tag as described below.
Moving the Release Tag for Respins
Now that you’ve made changes (or updated to someone else’s changes), move the release tag and push that change. Since the tag already exists, it is again necessary to use the -f argument to hg tag in order to move the tag.
hg branch hg tag -f -r $Revision $TagName -m "Added tag $TagName for changeset $Revision. CLOSED TREE a=release" hg push
If you’re moving a tag in the Gecko repository during release time (on either a Gecko release branch or a Camino minibranch), remember that your tag message may have to include "CLOSED TREE a=release" in order to be accepted.
Again, it’s good practice to return your tree to the default branch now that you are done moving the tag:
hg update default hg branch
After moving the tag, revisit Tinderbox and re-enable the build; as long as $build_hour has passed since the last release candidate build, Tinderbox will automatically produce a clobber release build. If $build_hour has not passed, run the following command on the builder to trigger a new clobber release build:
rm /builds/tinderbox/Cm1.6.7/Darwin_8.10.2_Depend/last-built
Source checkout, branching, and tagging for releases in CVS
These instructions correspond to Camino 1.0.4 (Gecko 1.8.0.10). The base tag for this release was FIREFOX_1_5_0_10_RELEASE. This specifies the apex of the minibranch. For the 1.0.4 release, changes to that tag for Camino were made on a new CAMINO_1_0_4_MINIBRANCH branch, and the release tag was CAMINO_1_0_4_RELEASE. Camino branch and release tags follow this convention (CAMINO_1_1_A2_MINIBRANCH, CAMINO_1_1_B_RELEASE).
Check out the source
Instead of a tag, you can also use a branch name, or a branch name with a date. If using a branch name and a date, use -D when checking out client.mk, and set MOZ_CO_DATE when making client.mk’s checkout target.
If you’re making a release that corresponds to a Gecko “final” version, be sure to pick up source that doesn’t have a “prerelease” Gecko version number in config/milestone.txt. Release tags are good for this, but branch snapshots taken at a known freeze time can work too in a pinch. (KaiRo has a tool that periodically queries tags, or you can use cvs log or perhaps even bonsai (or this link for older branches) to find relevant Gecko tags.)
As of the Gecko 1.8.1.4 and 1.8.0.11 releases, it is possible to base a Camino version on a Gecko release branch forked by the build team. Camino 1.5 is based on GECKO181_20070501_RELBRANCH, for example, the release branch for products based on Gecko 1.8.1.4 (Firefox 2.0.0.4). Note that we’ve been asked not to check in to this branch, even in mozilla/camino, so our own minibranch (below) is still needed.
Before pulling the source, you may want to add the target directory to Spotlight’s “Privacy” list.
cvs checkout -r FIREFOX_1_5_0_10_RELEASE mozilla/client.mk mozilla/camino/config cd mozilla MOZCONFIG=`pwd`/camino/config/mozconfig make -w -f client.mk checkout
In some cases, it’s not proper to base a Camino release off of an entire Gecko tree. For example, Camino 1.5.1 is based on FIREFOX_2_0_0_6_RELEASE in all directories except for mozilla/camino, which is based on then-current code from CAMINO_1_5_BRANCH. To switch the base in mozilla/camino, after the initial checkout above, do:
cvs update -r CAMINO_1_5_BRANCH -Pd camino
However, for the CAMINO_2_0_BRANCH, Mark recommended we always use the CAMINO_2_0_BRANCH’s copy of client.mk. This requires checking out the CAMINO_2_0_BRANCH copy of client.mk, uncommenting the MOZ_CO_TAG and setting it to the appropriate Gecko (Firefox) release tag:
cvs checkout -r CAMINO_2_0_BRANCH mozilla/client.mk mozilla/camino/config cd mozilla sed -e s/#MOZ_CO_TAG =/MOZ_CO_TAG = FIREFOX_3_0_15_RELEASE/ \ < client.mk > client.mk.new && \ mv client.mk.new client.mk MOZCONFIG=`pwd`/camino/config/mozconfig make -w -f client.mk checkout
client.mk will generate conflicts when switching back and forth between tags using this method, but you can ignore them; the right things will be checked out from the right tags, and you’ll be changing client.mk before tagging, anyway.
For Camino releases after Firefox has stopped shipping releases on a given branch (e.g., Camino 2.0.4), you’ll instead want to use the appropriate parent branch tag for the MOZ_CO_TAG, or leave it commented out on the trunk if additional fixes have landed (I seem to recall Mark warning me that setting it to HEAD in CAMINO_*_BRANCH copies of client.mk breaks things). In addition, you’ll want to pull the platform and Firefox version files from the final Firefox release, in order to avoid “pre” strings in the user-agent string (and, therefore, being broken on sites that break with “pre” in the UA):
cvs checkout -r CAMINO_2_0_BRANCH mozilla/client.mk mozilla/camino/config cd mozilla MOZCONFIG=`pwd`/camino/config/mozconfig make -w -f client.mk checkout cvs up -r FIREFOX_3_0_19_RELEASE config/milestone.txt browser/config/version.txt
When re-using a previous minibranch for a subsequent release (e.g., Camino 2.0.7 was Camino 2.0.6 plus a few additional patches landed on the CAMINO_2_0_6_MINIBRANCH), make sure that config/milestone.txt and browser/config/version.txt are present and have the right values.
For whatever reason, the version files for browser, mail, suite, and calendar had disappeared from the re-checkout of the CAMINO_2_0_6_MINIBRANCH (which pulled the CAMINO_2_0_6_RELEASE tag, since the minibranch’s client.mk had not been set back to the minibranch as suggested for full-tree minibranches in Tag it below) before tagging to 2.0.7 release; the lack of browser/config/version.txt caused the release build to fail.
Fork a minibranch
This can be done either by branching the entire checked-out tree, or branching individual files as needed. client.mk always needs to be minibranched to adjust its tags. I tend to branch the entire tree if any other files need to change, and to only branch client.mk if that’s all that’s changing. After branching, use cvs update to change the tag on the files you’ll be working on in your working copy, so that changes will be committed to the right branch.
The usage here branches everything rooted at the current working directory (mozilla):
cvs tag -b CAMINO_1_0_4_MINIBRANCH cvs update -r CAMINO_1_0_4_MINIBRANCH
This usage only branches the individual files which will be modified from the base tag(s) for inclusion in the release tag:
cvs tag -b CAMINO_1_5_1_MINIBRANCH \ client.mk \ camino/Info-Camino.plist \ camino/Info-CaminoStatic.plist \ camino/resources/application/all-camino.js \ camino/resources/localized/English.lproj/InfoPlist.strings cvs update -r CAMINO_1_5_1_MINIBRANCH \ client.mk \ camino/Info-Camino.plist \ camino/Info-CaminoStatic.plist \ camino/resources/application/all-camino.js \ camino/resources/localized/English.lproj/InfoPlist.strings
With recent branches, it is often only necessary to minibranch client.mk and the Camino version file:
cvs tag -b CAMINO_1_6_B3_MINIBRANCH client.mk camino/config/version.txt cvs update -r CAMINO_1_6_B3_MINIBRANCH client.mk camino/config/version.txt
Note that in some cases, a minibranch will be reused for an emergency release. For example, CAMINO_1_6_B1_MINIBRANCH was used for both Camino 1.6b1 and the emergency followup release, 1.6b2. With such a release, you should simply check out a copy of the minibranch and then skip to the land changes section to update it as needed.
Fix your (my) mistakes
At this point, I realized that the FIREFOX_1_5_0_10_RELEASE tag didn’t include changes to two Camino files that had been made recently on the MOZILLA_1_8_0_BRANCH in anticipation of Camino 1.0.4. I realized this because the release notes were missing. I found the other file by querying Bonsai. Here, I updated my local copies of these files to the current 1.8.0 branch versions, and then set the branch tag. I had to use cvs tag -B -F to move the tag because ad_blocking.css had already been branched. cvs tag -B -F is dangerous, but I know what I’m doing. Next time, I’ll remember to check this kind of thing before forking the minibranch.
cvs update -r MOZILLA_1_8_0_BRANCH \ camino/resources/application/ad_blocking.css \ camino/docs/Release\ Notes\ 1-0-4.rtf cvs tag -B -F -b CAMINO_1_0_4_MINIBRANCH \ camino/resources/application/ad_blocking.css \ camino/docs/Release\ Notes\ 1-0-4.rtf cvs update -r CAMINO_1_0_4_MINIBRANCH \ camino/resources/application/ad_blocking.css \ camino/docs/Release\ Notes\ 1-0-4.rtf
Make client.mk apply to the release tag
Now it’s possible to make changes to the working copy. First, update client.mk so that it will pull from the release tag (soon to be created). The relevant section of client.mk should look something like this. In this case, sed won’t change the NSPR and NSS tags to checked out, but that’s all right, because they’re static tags. I used to change those too, but now I think it’s clearer to see which NSPR and NSS releases were used by leaving their original tags intact in client.mk. An obvious exception applies when NSPR or NSS files need to change on the minibranch (like in Camino 2.0.8). Note that I feel entitled to use BS in the more administrative commit messages on my own private minibranch.
sed -e s/FIREFOX_1_5_0_10_RELEASE/CAMINO_1_0_4_RELEASE/ \ < client.mk > client.mk.new && \ mv client.mk.new client.mk cvs commit -m "The 1.0.4 minibranch is now boarding at gate 1.8.0.10" \ client.mk
If you’re using a minibranch based off of a Camino-only branch and Firefox has already stopped releasing off of the underlying Gecko branch and additional Gecko changes have landed, make sure that you adjust client.mk to uncomment the line for the Gecko tag (and set that value to the CAMINO_*_RELEASE tag as well).
Land other changes on the minibranch
I usually find these by querying Bugzilla for bugs with text like camino-1.0.3 in the status whiteboard field, by checking Bonsai for changes I made to the last release minibranch, or by asking Smokey or Sam. Those guys remember everything. These two things come from bugs 325765 and 325964. When landing things on the minibranch, don’t forget to update the status whiteboard fields of the relevant bugs with text like [camino-1.0.4].
sed -e s/0x00010006/0x00010008/ \
< netwerk/cache/src/nsDiskCache.h > netwerk/cache/src/nsDiskCache.h.new && \
mv netwerk/cache/src/nsDiskCache.h.new netwerk/cache/src/nsDiskCache.h
cvs commit -m "325765 Camino 1.0 versioned its cache as 1.8 while the main 1.8.0 branch versions it as 1.6. " \
"Use 1.8 to avoid dumping established caches on upgrading from an earlier 1.0.x Camino." \
netwerk/cache/src/nsDiskCache.h
patch -p1 < ../mathmlprompt.diff
cvs commit -m "325964 MathML missing font dialog contains extra space. " \
"Patch by Smokey Ardisson <alqahira@mindspring.com>. r=me sr=rbs" \
layout/mathml/base/src/mathfont.properties
With recent branches, often the only change after the client.mk change is to update the Camino version file:
echo 1.6b3 > camino/config/version.txt cvs commit -m "Numbers for Camino 1.6b3" client.mk camino/config/version.txt
Release checklist
Run the release checklist in camino/docs/ReleaseChecklist.txt to update all of the version numbers. Then, check it all in. The release checklist and the set of files it modifies may vary slightly by branch.
cat camino/docs/ReleaseChecklist.txt cd camino cvs commit -m "Camino 104, position and hold" \ Info-Camino.plist \ Info-CaminoStatic.plist \ installer/Makefile.in \ resources/application/ChimeraVersion.r \ resources/application/WebsiteDefaults.strings \ resources/application/all-camino.js \ resources/localized/English.lproj/InfoPlist.strings cd ..
On recent branches, if you’ve been following this guide, by the time you run the release checklist, there shouldn’t be any actual work to do. It should be a simple verification that the source tree is ready for a release.
You also might want to update the version numbers on the parent branch (MOZILLA_1_8_0_BRANCH in this case) to indicate that the branch has advanced beyond the new release. This time, I updated the above files on the parent branch to give a version number of 1.0.4+, but this versioning scheme is deprecated. Under the new scheme, the version number should be updated the the next version expected to be released from the branch, with “pre” appended. In this example, the version would be updated to 1.0.5pre. You don’t have to do this now, it can wait until more pressing tasks are done.
Tag it
We’re set. Now tag the tree with the release tag, so that client.mk actually makes sense. Regardless of whether the whole tree or just a few files were forked on the minibranch, we always tag the entire tree needed for a Camino build, even NSPR and NSS. (We don’t need to tag all of Mozilla’s repository, it’s sufficient to tag a full Camino checkout.)
cvs tag CAMINO_1_0_4_RELEASE
If someone pulls client.mk by the CAMINO_1_0_4_RELEASE tag, then they’ll get what they expect. Change the tags in client.mk on the minibranch to correspond to the minibranch instead of the release tag. It doesn’t make much of a difference because the new minibranch is a stub-ended branch and the buck stops here, but I’m something of an anal release-master. (Observe how carefully I placed that hyphen.) This is only recommended if the entire tree was forked to make a minibranch, otherwise, it’s pointless. If using a Gecko or Firefox release branch (recommended), you probably won’t have forked the entire tree for a minibranch, so this step is safe to skip.
sed -e s/CAMINO_1_0_4_RELEASE/CAMINO_1_0_4_MINIBRANCH/ \ < client.mk > client.mk.new && \ mv client.mk.new client.mk cvs commit -m "Please remain seated until your release-master has turned off the fasten seat belt sign." \ client.mk
Now, this part used to be crappy and top-secret, and fortunately, it’s no longer necessary now that tinderbox can use $UsePrebuiltTalkback. The old procedure is included here to confuse you. We used to need to tag the Talkback tree too, so that tinderbox would get the right stuff. We only had access to that from the tinderbox itself. I’m not going to tell you what we used for $CVSROOT, but you could have found it in the build log from any true clobber nightly produced by tinderbox.
CVSROOT=something CVS_RSH=ssh \ cvs checkout -r MOZILLA_1_8_0_BRANCH talkback/fullsoft CVSROOT=something CVS_RSH=ssh \ cvs tag -b CAMINO_1_0_4_MINIBRANCH talkback/fullsoft CVSROOT=something CVS_RSH=ssh \ cvs update -r CAMINO_1_0_4_MINIBRANCH talkback/fullsoft CVSROOT=something CVS_RSH=ssh \ cvs tag CAMINO_1_0_4_RELEASE talkback/fullsoft
Moving the Release Tag for Respins
Sometimes there is a late-breaking change that requires a respin after a release (candidate) build has been produced, or after the tree has been tagged with the _RELEASE tag, as happened with Camino 1.6.7 and Bug 485286. If the change has landed on the underlying Mozilla branch, you will need to update the changed file(s) in your tree to the correct revision and run cvs tag again. Since the tag already exists on the file, you need -F to force the tag to move.
cvs up -j 1.33.28.2 extensions/transformiix/source/xslt/functions/txKeyFunctionCall.cpp cvs tag -F CAMINO_1_6_7_RELEASE extensions/transformiix/source/xslt/functions/txKeyFunctionCall.cpp
After moving the tag, revisit Tinderbox and re-enable the build; as long as $build_hour has passed since the last release candidate build, Tinderbox will automatically produce a clobber release build. If $build_hour has not passed, run the following command to trigger a new clobber release build:
rm /builds/tinderbox/Cm1.6.7/Darwin_8.10.2_Depend/last-built
Make Tinderbox build it
Everything’s set now. Configure tinderbox to produce a new build from the release tag. This is a pain, too, but fortunately, the previous release can be used as a template. The stupid wildcard in the rsync is to avoid copying the whole source and object trees, which have names beginning with Darwin with a capital D. Everything else important has a name beginning with a lowercase letter. rsync is used to preserve symbolic links, of which there are many.
cd /builds/tinderbox mkdir Cm1.0.4 rsync -av Cm1.0.3/[a-z]* Cm1.0.4 sed -e s/1\\.0\\.3/1.0.4/ -e s/1_0_3/1_0_4/ \ < Cm1.0.4/tinder-config.pl > Cm1.0.4/tinder-config.pl.new && \ mv Cm1.0.4/tinder-config.pl.new Cm1.0.4/tinder-config.pl
It’s possible for tinderbox to build from either a branch or a tag. The right way is to build releases from tags, but if you’re sure that a tag corresponds exactly to a branch, you can cheat and build from the branch instead. I usually cheat, but for the purposes of these instructions, I’m doing it the right way. First, check $BuildTag in tinder-config.pl to make sure it corresponds to your chosen release tag (or branch). When building from a tag (instead of a branch), pull-by-timestamp (cvs checkout -D) must be turned off. It’s useful when building from branches. Its behavior is controlled by the $UseTimeStamp tinderbox configuration variable. This time, I’m using:
$BuildTag = 'CAMINO_1_0_4_RELEASE'; $UseTimeStamp = 0; # Use the CVS 'pull-by-timestamp' option, or not
Other significant variables affect the name displayed on the tinderbox status page, and the names of the directories created on the FTP server:
$BuildNameExtra = 'Cm1.0.4'; $milestone = "1.0.4";
If you’re building on a branch that ships Breakpad instead of Talkback, make sure that Talkback is not shipped (and that Breakpad symbols are collected and uploaded):
$shiptalkback = 0;
$crashreporter_buildsymbols = 1;
$crashreporter_pushsymbols = 1;
$ENV{SYMBOL_SERVER_HOST} = 'dm-symbolpush01.mozilla.org';
$ENV{SYMBOL_SERVER_USER} = 'caminobld';
$ENV{SYMBOL_SERVER_PATH} = '/mnt/netapp/breakpad/symbols_camino/';
# this is optional, it's a full path to a ssh private key
$ENV{SYMBOL_SERVER_SSH_KEY} = "$ENV{HOME}/.ssh/id_dsa";
This is also a good time to make sure you have not made any relevant changes to the tinder-config.pl for your release tinderbox since the last release (remember that you are simply copying the previous release’s tinder-config.pl file, not starting with the tinder-config.pl in cvs that is used by the nightly builds, so it is important to check for changes made to nightly configuration that also will be needed by an official release). You’ll want to make sure to check the mozconfig in a similar fashion. (Luckily, tinderbox changes to files other than build-camino.pl and post-mozilla.pl are propagated automatically due to symlinks.) You can sanity-check your changes by diffing these files against their counterparts in the nightly directory and comparing the changes to the ones you just made above.
Now, edit multi-tinderbox’s config so that it knows to build the release. Since we don’t have much spare “room” on the tinderbox, I temporarily disable the base branch build while it’s doing a release. In this case, I’m commenting out Cm1.0-M1.8.0 and adding a Cm1.0.4 entry. Later, when the release is built, I’ll comment out Cm1.0.4 and turn Cm1.0-M1.8.0 back on. multi-tinderbox reloads its configuration when it receives a SIGHUP; when it finishes the build it’s working on, it’ll start over at the top of the list in its config. For that reason, I like to put the release build I’m doing at the top of multi-config.pl.
vi /builds/tinderbox/multi-config.pl
The relevant snippet would look something like this:
$ENV{'CVS_RSH'}='ssh';
$ENV{'TBOX_CLIENT_CVS_DIR'}='/builds/cvs/mozilla/tools/tinderbox';
$BuildSleep = 5;
$Tinderboxes = [
{ tree => "Cm1.0.4" },
# { tree => "Cm1.0-M1.8.0" },
{ tree => "Cm1.1-M1.8" },
{ tree => "CmTrunk" },
];
And then tell multi-tinderbox to reload when the current build finishes:
kill -HUP `cat /builds/tinderbox/multi.pid`
When the release build begins, go to the Camino tinderbox page, click “Administrate Tinderbox Trees” (it should say “Administer,” but whatever), and turn on log scraping for the new build. Do it before the first build completes. If you don’t do this step, it’s no big deal, but the (hopefully green, possibly orange or red) box on tinderbox won’t show test results.
If something goes wrong, fix it. Use the tinderbox logs to your advantage. If you’ve got to move a tag or something, you can refer to the “fix-it” section above where I fixed ad_blocking.css, remembering that at this point, you’ve got both a _MINIBRANCH and _RELEASE tag to worry about.
If you need to trigger a new release build,
rm /builds/tinderbox/Cm1.6.7/Darwin_8.10.2_Depend/last-built
and tinderbox will generate a new release build next cycle.
If Talkback symbol upload appears to fail (as it did with Camino 1.6.7 RC2), you can try again by running
./builds/tinderbox/Cm1.6.7/Darwin_8.10.2_Depend/2009032711/upload-symbols.sh
If Sam is around, you can bug him to check the Talkback symbol server to ensure that your re-upload succeeded.
For Breakpad symbols, you can try again by running the following scripts:
./builds/tinderbox/Cm2.0b3/Darwin_8.10.2_Depend/2009060219/upload-symbols-i386.sh ./builds/tinderbox/Cm2.0b3/Darwin_8.10.2_Depend/2009060219/upload-symbols-ppc.sh
You can check the Breakpad server for symbols by visiting
http://symbols.mozilla.org/symbols_camino/Camino-2.0b3-2009060219-i386-symbols.txt
http://symbols.mozilla.org/symbols_camino/Camino-2.0b3-2009060219-ppc-symbols.txt
and constructing the path to the symbols mentioned in the file.
Since symbol upload failures don't trigger a color change or some sort of visible entry on the waterfall, it is important for you to check the log to make sure that generation and upload succeeded.
If you know when the release date is scheduled, now is a good time to file the Bouncer bug so that you don’t forget to do so later, particularly if the person doing the website updates is doing so at the last moment.
Now is also a good time to get the new Breakpad reports set up on crash-stats.
Stage it
Once the build is done, test it. It’s not evil to get other people to help test the build with you, just pass them the URL to the “nightly” that tinderbox uploaded. If it seems reasonable, stage it and let people know. When possible, it’s nice to let the localization team know that a build is ready and give them time to prepare the multilingual version, and then stage the en-US and multilingual versions together when everything is ready.
ssh stage.mozilla.org cd /home/ftp/pub/camino/releases cp -p ../nightly/2007-02-28-13-1.0.4/Camino.dmg Camino-1.0.4.dmg chmod 644 Camino-1.0.4.dmg cp -p Camino-1.0.4.dmg en-US rm MD5SUMS LC_ALL=C bash md5sum * > MD5SUMS chmod 644 MD5SUMS exit exit
Stage the source tarball and the multilingual version when they’re ready, too. The source tarball will go in /home/ftp/pub/camino/source/camino-1.0.4.tar.bz2, and that directory has an MD5SUMS file as well. The multilingual release will go in
/home/ftp/pub/camino/releases/Camino-1.0.4-MultiLang.dmg, with a copy at /home/ftp/pub/camino/releases/all/Camino-1.0.4.dmg.
scp Camino-1.0.4-MultiLang.dmg username@stage.mozilla.org:/home/ftp/pub/camino/releases scp camino-1.0.4.tar.bz2 username@stage.mozilla.org:/home/ftp/pub/camino/source
Be sure to use the correct filename when copying the MultiLang version to releases/all/. Also be sure to regenerate the MD5SUMS file in releases/ if you generated it before staging the MultiLang version.
After you've successfully staged the release, you can clean up its nightly/latest-1.6.8/ directory. Assuming you’re in /home/ftp/pub/camino/releases on stage:
rm -rf ../nightly/latest-1.6.8/
For Hg-based releases, also be sure to remove the sourcestamp file for the previous “pre”-version from the nightly’s latest directory; e.g., the day after the Camino 2.1b1 build is spun (once the first nightly with the new 2.1b2pre version number has been built), remove the sourcestamp file for the last 2.1a2pre nightly. Assuming you’re in /home/ftp/pub/camino/releases on stage:
rm -rf ../nightly/latest-2.1-M1.9.2/camino-2.1a2pre.txt
Create a source release
We provide source tarballs for major releases (not alphas, betas, or release candidates). Check out a fresh tree using anonymous CVS, so that the CVS/Root files are correct for people bootstrapping anonymous CVS workspaces. If the .tar.bz2 file doesn’t wind up at about 33MB (for the 1.8.0 branch), something’s probably wrong.
touch ~/.cvspass CVSROOT=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot \ cvs checkout -r CAMINO_1_0_4_RELEASE mozilla/client.mk mozilla/camino/config cd mozilla MOZCONFIG=`pwd`/camino/config/mozconfig make -w -f client.mk checkout rm .mozconfig.mk .mozconfig.out cd .. chmod -R a+rX mozilla BZIP2=-9 tar --owner 0 --group 0 -jcf camino-1.0.4.tar.bz2 mozilla
Stage the source release as described above in Stage it. Don't forget to remove the mozilla directory created when checking out the source.
Have someone else set up Breakpad reports
Poke your friendly security-group member to add Socorro reports for the release and the next “pre” version using the Branch Data Sources “add new product version” link on the Admin page on crash-stats.
Other things to do
Your job as a branch tag build release-master is done. It’s time to sit back, relax, and watch as other people do other things. If you needed to modify the procedure in any way, now would be an excellent time to update this document.
- Wait for mirror propagation from
stage.mozilla.orgto all of the systems handlingreleases.mozilla.org,ftp.mozilla.org, and other mirrors. - Have someone on the build team update bouncer. File a bug in the “mozilla.org:Release Engineering” component (release, milestone). Make sure the priority doesn't get cloned, as that may mess up RelEng.
- Someone’s got to update the website, the web site, or both.
- Some releases include a “release notes” URL that needs to be populated with content or a redirect. The release notes URL is defined as
ReleaseNotesDefaultincamino/resources/application/WebsiteDefaults.stringsand is updated as part of the release checklist. The URL used in Camino 1.0.4 is [1], which redirects to [2]. - Don’t forget to restore the Tinderbox configuration in
multi-config.pland forcemulti-tinderboxto reload its configuration withSIGHUP, as described in the Make Tinderbox build it section.
