IT:Setting up Camino Tinderboxen

From Camino Wiki
Jump to navigation Jump to search

How to (configure and set up and start and) run a Camino tinderbox
by Samuel Sidler (with some editing and additions by Smokey Ardisson)
on or before 2008-07-01

10.4 Tinderbox

These instructions were originally written for a 10.4 tinderbox, but they are broadly similar for 10.5 and therefore applicable there, too; any significant points of difference are noted inline.

Configuring the Box

With a new machine, you first need to set up the box. To do this, reinstall the OS (10.4) from a clean disk, including reformatting the machine. For the first run information, make your default user be caminobld with a password of {redacted}. This is the standard user we use for build machines.

Basic Configuration

Next, open System Preferences and begin customizing the machine.

  1. In the Desktop & Screen Save pane, turn off the screen saver and set the desktop to be a solid color.
    • On 10.5 and above, headless machines can't set their desktop backgrounds permanently. Use sudo defaults write /Library/Preferences/ DesktopPicture /Library/Desktop\ Pictures/Solid\ Colors/Solid\ Aqua\ Blue.png instead.
  2. In the Energy Saver pane, change the sliders to "never", indicating you don't want the machine to go to sleep at all.
  3. In the Keyboard & Mouse pane, make sure Full Keyboard Access is set to "Text boxes and lists only".
  4. In the Dock pane, set the Dock to auto-hide.
  5. In the Sharing pane, turn on Remote Login.
  6. Also in the Sharing pane, turn on Apple Remote Desktop (on 10.5, this is "Remote Management"). Here, you'll customize the Access Privileges (on 10.5, "Computer Settings").
    1. Turn "On" the caminobld user.
    2. Check all boxes under "Allow user to do the following on this computer".
    3. Check the VNC box below and set a password of {redacted}.
  7. And one more change in the Sharing pane, change the "Computer Name" to be the name given (such as, cb-minibinus01).
  8. In the Software Update pane, uncheck the box "Check for updates". (We specifically don't automatically check because it can cause unexpected Tp/Ts regressions.)
  9. In the Network pane, be sure your ethernet is set to the appropriate IPs from the IT team.
    1. Make sure that more than one DNS server is set, so that when one goes down for maintenance, the machine doesn't lose the ability to talk to the world.
  10. In the International pane, set a language other than English as the first language and then restore English to the top position; this should eliminate any weirdness with non-ISO values in AppleLanguages
    1. Though this was supposed to have never occurred unless a machine was upgraded from 10.2 or below, we have seen cb-xserve01 afflicted with this bug.

Mail Configuration

Next, make changes to ensure that the mail will work correctly.

The tinderboxen are using their hostname as the domain portion of originated e-mail addresses, so (without any intervention), they'd be sending mail as whatever@cb-xserve01.local. The mail server is getting pissed off because that's not a valid return address. (Note that while cb-xserve01 has a simple hostname, the other tinderboxen all have hostnames. Check with IT if you're not sure of the actual hostname.)

We can fix this by editing /etc/postfix/ in vi or your editor of choice (superuser permissions required) and setting myhostname explicitly:

myhostname =

We should also add these lines, because we've found them to be necessary on tinderboxen:

# Allow large messages (unlimited size, required for tinderbox)
message_size_limit = 0
# Shut up warnings
mailbox_size_limit = 0

Then, reload Postfix to get it to reread its configuration (on 10.5, Postfix appears to run only on demand):

sudo postfix reload

Hostname in other places

One additional change Mark usually makes: edit /etc/hostconfig and set HOSTNAME to the desired hostname, in the case of cb-xserve01, cb-xserve01. At its default setting, -AUTOMATIC-, the system picks a hostname without necessarily doing the best job. The automatic hostname may even wind up changing across reboots. That's kind of annoying. Explicit settings are better here. (Server 10.4.6 was supposed to fix this, but cb-xserve01 regularly displays the problem and even cb-xserve04 once ended up with a hostname of -cb-xserve04- after a reboot, and cb-xserve04 also once switched to cb-xserve04.local randomly in the middle of the night!)

Mark always saves the original files as he found them in whatever.orig files for future diffing pleasure.

N.B. There may be an additional change required to get the right hostname set for sending results to the graph server; this is currently under investigation. cb-xserve01 appears to need HOSTNAME in /etc/hostconfig to be set to to match the DNS name set by Mozilla IT. This may be an artifact of Mac OS X Server, or something else we don't yet know.

This probably should be fixed with sudo scutil --set HostName and rebooting; this TN warns against setting the hostname in /etc/hostconfig. Other useful commands are sudo scutil --get HostName and sudo changeip -checkhostname; recently we’ve found that when scutil reports the right value and changeip reports a mismatch, it’s necessary to run the command suggested by changeip (using sudo) and then reboot.

Build Environment

First, set up the build environment. You should always make sure the system is completely up-to-date by running Software Update. We then install XCode 2.5 (Xcode 3.1 on 10.5), then use MacPorts to install libIDL, autoconf213, and mercurial, as discussed in the Camino build documentation.

Additionally, if you plan to run tests, install wget.

$ sudo port install wget

On Mac OS X 10.4, pay special attention to the instructions on selecting the correct Python version.

Crash Reporter Configuration

Mac OS X Crash Reporter

Because Crash Reporter opens a window when a process crashes, and windows present on the screen affect performance test results, we need to be sure to set Crash Reporter to silently.

Once you’ve installed Xcode, open the CrashReporterPrefs application in /Developer/Applications/Utilities/ and configure Crash Reporter to use Server mode. This will still write crash reports to disk but will suppress the crash reporter window itself.

Breakpad Crash Reporter

On tinderboxen building any Camino branches with Breakpad, execute the following command to ensure that any reports are submitted automatically, allowing the client to both quit and be blown away during rebuild:

defaults write org.mozilla.camino BreakpadSkipConfirm -string YES

If you later need to get the Breakpad report ID, there will be an entry like this in the Console:

Apr 21 14:27:21 Qalaat-Samaan[121]: Breakpad Reporter: Renamed /Users/smokey/Library/Breakpad/Camino/F8D9CA55-E65F-4439-BCA9-8A794A92F055.dmp to /Users/smokey/Library/Breakpad/Camino/CrashID=bp-d1ccccf3-e418-40bd-a3fc-a32c42090421.dmp after successful upload

Since non-nightly builds will not have symbols, you will need to make Breakpad stop suppressing the Mac OS X Crash Reporter in order to have useful crash reports saved on the tinderbox:

defaults write org.mozilla.camino BreakpadSendAndExit -string NO

Note that if you ever remove ~/Library/Preferences/org.mozilla.camino.plist, you will need to execute the defaults commands again.

Software Update

On fast machines, we believe tinderbox kills Camino before we have a chance to write the SULastCheckTime default. This leads to fast tinderboxen checking for updates on every launch. To prevent that (and a corresponding skew of update statistics):

defaults write org.mozilla.camino SUEnableAutomaticChecks -bool NO

Note that if you ever remove ~/Library/Preferences/org.mozilla.camino.plist, you will need to execute the defaults commands again.

Keys and Profile

Now that you have a build environment, you need to add the appropriate private keys and environment variables to continue.


First, add your private key as follows:

Did you really think we'd publish the private key?

Then, add your public key:

Consult your friendly build administrator for this one, too.

Be sure to set permissions on your private key to be 500 by running chmod 500 ~/.ssh/id_dsa.

N.B. Once when resurrecting cb-xserve01, use of the private key kept requiring a passphrase (which may not exist). When all else fails, zip the keys from a working tinderbox, scp them to the new tinderbox, and unzip.

SSH Configuration File

You also need to install a basic SSH configuration file for access to

  User caminobld

Environment Variables

Finally, we want to set environment variables globally. Use vi (or your editor of choice) to edit /etc/profile with the contents below.

LESS='-acdefi~P?f%f:<stdin>.  ?bBByte\: %bB  .?lbLine\: %lb  .?pB%pB\%  .?e(END):--More--.?x+.'
if [ 'x'$BASH != 'x' ];then
  PS1='\u@\h bash\$ '
  shopt -s checkwinsize
  if [ 'x'$USER == 'xroot' ];then
    PS1=$USER'@'`/bin/hostname`' sh# '
    PS1=$USER'@'`/bin/hostname`' sh$ '
umask 22


Now, create a link from ~/.bashrc to /etc/profile using the following command: ln -s /etc/profile ~/.bashrc.


Setting up Tinderbox

The moment you've all been waiting for!

Camino tinderboxen install the tinderbox to /builds/tinderbox/. You'll need to create this directory.

Next, while in /builds/tinderbox/, check out the tinderbox scripts as well as the performance scripts (for perf test running). Camino uses a fork of tinderbox client called “madhatter” that supports both CVS- and Hg-based builds.

hg clone madhatter
cvs co mozilla/tools/performance/

When prompted, type in "yes" to authorize SSH to connect to

Now we need to create some symlinks. These links are created so that when we update the tinderbox script, our builds will use the new version instead of the old. This makes maintenance much easier.

First, in /builds/tinderbox/, we create two symlinks:

ln -s madhatter/
ln -s madhatter/tinderbox tinderbox

Now, before we continue with linking, decide what you're building: branch or trunk (or another branch!). The rest of this documentation assumes you're building Camino with mozilla-1.9.2. (To build a branch, changes are minimal: you just need to create the appropriately-named directory for the branch in the step below, later check out the branch’s tinderbox-configs in that step, and use the appropriate branch information when creating

First, create a directory for the remaining symlinks to live.

mkdir Cm2.1-M1.9.2

Now, cd to that new directory and create a bunch of symlinks:

ln -s ../mozilla/tools/performance/startup/startup-test.html
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/examples
ln -s ../madhatter/
ln -s ../madhatter/install-links
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/
ln -s ../madhatter/tinderbox

Now we need to check out the tinderbox config files. Because we‘re advanced, we have our config files in CVS. To check them out, first go into the /builds/tinderbox/Cm2.1-M1.9.2 directory then run the following command (using the example of minus; substitute the proper directory and branch for the relevant box, remembering that cb-xserve01 uses the macosx directory directly):

cvs co -r BRANCHNAME -d tinderbox-configs mozilla/tools/tinderbox-configs/camino/macosx/cb-minibinus01/

After this, it‘s time to create a few more symlinks! (Note: When setting up the box, it is customary to run it under local config to get it working; to do so, use copies of the checked-out files instead of symlinks. Once the configuration is stable, commit the local copies to CVS, delete the local files, and then add the symlinks below.)

ln -s tinderbox-configs/CLOBBER
ln -s tinderbox-configs/mozconfig
ln -s tinderbox-configs/

Finally, we need to create a couple more files in this directory. First we want to copy the original script here and name it, and then copy one other file.

cp ../madhatter/
cp ../madhatter/

Next, un-comment out the line in (starting with "require"), then make it and executable (chmod 755

Almost there…

You now need to add the file to /builds/tinderbox/. This file will define what builds run on the machine.

The config we use for Camino with mozilla-1.9.2 is:

$BuildSleep = 5;
$Tinderboxes = [
  { tree => "Cm2.1-M1.9.2", args => "--config-cvsup-dir /builds/tinderbox/Cm2.1-M1.9.2/tinderbox-configs" },

To determine whether you're producing nightly builds that get delivered to or just hourly builds (i.e., just running Tp/Ts/Tdhtml), set $OfficialBuildMachinery appropriately in your local The default is 1 for nightly clobber builds (and “hourly” depend tinderbox-builds); uncomment the variable and set its value to 0 to produce only “hourly” builds (which are uploaded to tinderbox-build/ directories). (See the comment for that value; do we have to do anything to that when we're doing hourlies?) Note that some Camino tinderboxen have moved aside (as a hack?); these should all be fixed to use the proper settings.

The last step is only pertinent if this tinderbox will be creating builds to be distributed on SSH to and type in “yes” when prompted. Repeat for if the build contains Talkback, or if the build contains Breakpad. If you don't do this, the tinderbox script will pause at the very end, waiting for someone to enter “yes”.


Finally, copy the prebuilt Talkback binary package to the appropriate location (/builds/tinderbox/Cm2-M1.9 for cvs trunk) and set the appropriate path and filename in

That should be it!

Starting Tinderbox

Try running a tinderbox with ./ and see what happens (Sam prefers this method, since it spews the build process in the Terminal, while Mark and Smokey prefer ./tinderbox multi start which does not spew the build process info, and which can be stopped). This command must be run in the Terminal application on the machine (the script requires a proper display or tests will fail). You do this by accessing the box from a VNC session. Be sure to hide (not minimize) afterwards; otherwise, Ts results will be adversely affected.

After the first build has begun, you need to go to the Camino tinderbox page, click “Administrate Tinderbox Trees” (it should say “Administer,” but whatever), and turn on log scraping for the new build. Do it before the first build completes. Repeat this step for any other builds you are setting up.

If you want to check on the build progress, you can

tail -f /builds/tinderbox/Cm2.1-M1.9.2/Darwin_8.11.0_Depend/Darwin_8.11.0_Depend.log

(where Darwin_8.11.0_Depend is the buildhost+build type).

Fun with the mail queue

Sometimes, MoCo IT will do something that will cause tinderbox to stop accepting mail from our tinderboxen. In these cases, there will be a large backlog of mail built up, and since postfix only runs on demand, for 60s at a time, it will take a long time to clear.

If there are only a few hours or a 1-2 days of backlogged mail, tinderbox will likely be able to handle the backlog sanely. In this case, the easiest way to get postfix running constantly is to touch a junk file in the maildrop directory[1] (use the first for client and the second for Server):

$ sudo touch /var/spool/postfix/maildrop/junk
$ sudo touch /Library/Server/Mail/Data/spool/maildrop/junk

Then once postfix has cleared the backlog, sudo rm the dummy file.

If there are more than a few days of backlogged mail, tinderbox will go crazy trying to process it, so it's better to purge the queue instead:

$ sudo postsuper -d ALL

This will delete everything in the mail queue, new and old, and report back the number of messages deleted. (If it doesn't report a number of messages deleted, then it didn't run; try forcing postfix to start and then quickly clearing the queue.)

Finally, to see the contents of the mail queue:

$ sudo mailq

Fun with vanishing volumes

cb-xserve04 will sometimes lose its secondary (tinderbox) volume unexpectedly[2]. When this happens, the journal will have replayed file system changes into the original mount point, causing the volume to re-appear at a different mount point, breaking tinderbox. If this happens, this Apple KB article explains how to fix the problem and restore the original mount point.

Releasing Camino

Instructions on how to (branch and tag and build and) release a new version of Camino can be found elsewhere on the Camino wiki, in Releases:Branch Tag Build Release (the how-to-mento handbook).