IT:Setting up Camino Tinderboxen

From Camino Wiki
Revision as of 20:41, 26 May 2012 by Sardisson (talk | contribs) (→‎Releasing Camino: various ways of clearing the mail queue)
Jump to navigation Jump to search

How to (configure and set up and start and) run a Camino tinderbox
by Samuel Sidler (with some editing and additions by Smokey Ardisson)
on or before 2008-07-01

10.4 Tinderbox

These instructions were originally written for a 10.4 tinderbox, but they are broadly similar for 10.5 and therefore applicable there, too; any significant points of difference are noted inline.

Configuring the Box

With a new machine, you first need to set up the box. To do this, reinstall the OS (10.4) from a clean disk, including reformatting the machine. For the first run information, make your default user be caminobld with a password of {redacted}. This is the standard user we use for build machines.

Basic Configuration

Next, open System Preferences and begin customizing the machine.

  1. In the Desktop & Screen Save pane, turn off the screen saver and set the desktop to be a solid color.
    • On 10.5 and above, headless machines can't set their desktop backgrounds permanently. Use sudo defaults write /Library/Preferences/com.apple.loginwindow DesktopPicture /Library/Desktop\ Pictures/Solid\ Colors/Solid\ Aqua\ Blue.png instead.
  2. In the Energy Saver pane, change the sliders to "never", indicating you don't want the machine to go to sleep at all.
  3. In the Keyboard & Mouse pane, make sure Full Keyboard Access is set to "Text boxes and lists only".
  4. In the Dock pane, set the Dock to auto-hide.
  5. In the Sharing pane, turn on Remote Login.
  6. Also in the Sharing pane, turn on Apple Remote Desktop (on 10.5, this is "Remote Management"). Here, you'll customize the Access Privileges (on 10.5, "Computer Settings").
    1. Turn "On" the caminobld user.
    2. Check all boxes under "Allow user to do the following on this computer".
    3. Check the VNC box below and set a password of {redacted}.
  7. And one more change in the Sharing pane, change the "Computer Name" to be the name given (such as, cb-minibinus01).
  8. In the Software Update pane, uncheck the box "Check for updates". (We specifically don't automatically check because it can cause unexpected Tp/Ts regressions.)
  9. In the Network pane, be sure your ethernet is set to the appropriate IPs from the IT team.
    1. Make sure that more than one DNS server is set, so that when one goes down for maintenance, the machine doesn't lose the ability to talk to the world.
  10. In the International pane, set a language other than English as the first language and then restore English to the top position; this should eliminate any weirdness with non-ISO values in AppleLanguages
    1. Though this was supposed to have never occurred unless a machine was upgraded from 10.2 or below, we have seen cb-xserve01 afflicted with this bug.

Mail Configuration

Next, make changes to ensure that the mail will work correctly.

The tinderboxen are using their hostname as the domain portion of originated e-mail addresses, so (without any intervention), they'd be sending mail as whatever@cb-xserve01.local. The mail server is getting pissed off because that's not a valid return address. (Note that while cb-xserve01 has a simple mozilla.com hostname, the other tinderboxen all have sj.mozilla.com hostnames. Check with IT if you're not sure of the actual hostname.)

We can fix this by editing /etc/postfix/main.cf in vi or your editor of choice (superuser permissions required) and setting myhostname explicitly:

myhostname = cb-xserve01.mozilla.com

We should also add these lines, because we've found them to be necessary on tinderboxen:

# Allow large messages (unlimited size, required for tinderbox)
message_size_limit = 0
# Shut up warnings
mailbox_size_limit = 0

Then, reload Postfix to get it to reread its configuration (on 10.5, Postfix appears to run only on demand):

sudo postfix reload

Hostname in other places

One additional change Mark usually makes: edit /etc/hostconfig and set HOSTNAME to the desired hostname, in the case of cb-xserve01, cb-xserve01. At its default setting, -AUTOMATIC-, the system picks a hostname without necessarily doing the best job. The automatic hostname may even wind up changing across reboots. That's kind of annoying. Explicit settings are better here. (Server 10.4.6 was supposed to fix this, but cb-xserve01 regularly displays the problem and even cb-xserve04 once ended up with a hostname of -cb-xserve04- after a reboot, and cb-xserve04 also once switched to cb-xserve04.local randomly in the middle of the night!)

Mark always saves the original files as he found them in whatever.orig files for future diffing pleasure.

N.B. There may be an additional change required to get the right hostname set for sending results to the graph server; this is currently under investigation. cb-xserve01 appears to need HOSTNAME in /etc/hostconfig to be set to cb-xserve01.mozilla.com to match the DNS name set by Mozilla IT. This may be an artifact of Mac OS X Server, or something else we don't yet know.

This probably should be fixed with sudo scutil --set HostName cb-xserve01.mozilla.com and rebooting; this TN warns against setting the hostname in /etc/hostconfig. Other useful commands are sudo scutil --get HostName and sudo changeip -checkhostname; recently we’ve found that when scutil reports the right value and changeip reports a mismatch, it’s necessary to run the command suggested by changeip (using sudo) and then reboot.

Build Environment

First, set up the build environment. You should always make sure the system is completely up-to-date by running Software Update. We then install XCode 2.5 (Xcode 3.1 on 10.5), then use MacPorts to install libIDL, autoconf213, and mercurial, as discussed in the Camino build documentation.

Additionally, if you plan to run tests, install wget.

$ sudo port install wget

On Mac OS X 10.4, pay special attention to the instructions on selecting the correct Python version.

Crash Reporter Configuration

Mac OS X Crash Reporter

Because Crash Reporter opens a window when a process crashes, and windows present on the screen affect performance test results, we need to be sure to set Crash Reporter to silently.

Once you’ve installed Xcode, open the CrashReporterPrefs application in /Developer/Applications/Utilities/ and configure Crash Reporter to use Server mode. This will still write crash reports to disk but will suppress the crash reporter window itself.

Breakpad Crash Reporter

On tinderboxen building any Camino branches with Breakpad, execute the following command to ensure that any reports are submitted automatically, allowing the client to both quit and be blown away during rebuild:

defaults write org.mozilla.camino BreakpadSkipConfirm -string YES

If you later need to get the Breakpad report ID, there will be an entry like this in the Console:

Apr 21 14:27:21 Qalaat-Samaan com.apple.launchd.peruser.501[121]: Breakpad Reporter: Renamed /Users/smokey/Library/Breakpad/Camino/F8D9CA55-E65F-4439-BCA9-8A794A92F055.dmp to /Users/smokey/Library/Breakpad/Camino/CrashID=bp-d1ccccf3-e418-40bd-a3fc-a32c42090421.dmp after successful upload

Since non-nightly builds will not have symbols, you will need to make Breakpad stop suppressing the Mac OS X Crash Reporter in order to have useful crash reports saved on the tinderbox:

defaults write org.mozilla.camino BreakpadSendAndExit -string NO

Note that if you ever remove ~/Library/Preferences/org.mozilla.camino.plist, you will need to execute the defaults commands again.

Software Update

On fast machines, we believe tinderbox kills Camino before we have a chance to write the SULastCheckTime default. This leads to fast tinderboxen checking for updates on every launch. To prevent that (and a corresponding skew of update statistics):

defaults write org.mozilla.camino SUEnableAutomaticChecks -bool NO

Note that if you ever remove ~/Library/Preferences/org.mozilla.camino.plist, you will need to execute the defaults commands again.

Keys and Profile

Now that you have a build environment, you need to add the appropriate private keys and environment variables to continue.

Keys

First, add your private key as follows:

~/.ssh/id_dsa:
-----BEGIN DSA PRIVATE KEY-----
Did you really think we'd publish the private key?
-----END DSA PRIVATE KEY-----

Then, add your public key:

~/.ssh/id_dsa.pub:
Consult your friendly build administrator for this one, too.

Be sure to set permissions on your private key to be 500 by running chmod 500 ~/.ssh/id_dsa.

N.B. Once when resurrecting cb-xserve01, use of the private key kept requiring a passphrase (which may not exist). When all else fails, zip the keys from a working tinderbox, scp them to the new tinderbox, and unzip.

SSH Configuration File

You also need to install a basic SSH configuration file for access to hg.mozilla.org: 

Host hg.mozilla.org
  User caminobld

Environment Variables

Finally, we want to set environment variables globally. Use vi (or your editor of choice) to edit /etc/profile with the contents below. 

#!/bin/sh
CC='gcc'
CXX='g++'
EDITOR='vi'
ENV='/etc/profile'
LESS='-acdefi~P?f%f:<stdin>.  ?bBByte\: %bB  .?lbLine\: %lb  .?pB%pB\%  .?e(END):--More--.?x+.'
LESSHISTFILE='-'
PAGER='less'
PATH='/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/X11/bin'
VISUAL='vi'
export CC CXX EDITOR ENV LESS LESSHISTFILE ORGANIZATION PAGER PATH VISUAL
HISTSIZE='1000'
unset HISTFILE
if [ 'x'$BASH != 'x' ];then
  PS1='\u@\h bash\$ '
  shopt -s checkwinsize
else
  if [ 'x'$USER == 'xroot' ];then
    PS1=$USER'@'`/bin/hostname`' sh# '
  else
    PS1=$USER'@'`/bin/hostname`' sh$ '
  fi
fi
umask 22
PATH='/opt/local/bin:/opt/local/sbin:'$PATH
PATH=$HOME'/bin':$PATH

CVS_RSH='ssh'
CVSROOT=':ext:caminobld@cvs.mozilla.org:/cvsroot'
export CVS_RSH CVSROOT

Now, create a link from ~/.bashrc to /etc/profile using the following command: ln -s /etc/profile ~/.bashrc.

Restart Terminal.app.

Setting up Tinderbox

The moment you've all been waiting for!

Camino tinderboxen install the tinderbox to /builds/tinderbox/. You'll need to create this directory.

Next, while in /builds/tinderbox/, check out the tinderbox scripts as well as the performance scripts (for perf test running). Camino uses a fork of tinderbox client called “madhatter” that supports both CVS- and Hg-based builds.

hg clone http://hg.mozilla.org/users/alqahira_ardisson.org/madhatter/ madhatter
cvs co mozilla/tools/performance/

When prompted, type in "yes" to authorize SSH to connect to cvs.mozilla.org.

Now we need to create some symlinks. These links are created so that when we update the tinderbox script, our builds will use the new version instead of the old. This makes maintenance much easier.

First, in /builds/tinderbox/, we create two symlinks:

ln -s madhatter/multi-tinderbox.pl multi-tinderbox.pl
ln -s madhatter/tinderbox tinderbox

Now, before we continue with linking, decide what you're building: branch or trunk (or another branch!). The rest of this documentation assumes you're building Camino with mozilla-1.9.2. (To build a branch, changes are minimal: you just need to create the appropriately-named directory for the branch in the step below, later check out the branch’s tinderbox-configs in that step, and use the appropriate branch information when creating multi-config.pl.)

First, create a directory for the remaining symlinks to live.

mkdir Cm2.1-M1.9.2

Now, cd to that new directory and create a bunch of symlinks:

ln -s ../mozilla/tools/performance/startup/startup-test.html
ln -s ../madhatter/bloatdiff.pl
ln -s ../madhatter/build-camino.pl
ln -s ../madhatter/build-firefox.pl
ln -s ../madhatter/build-galeon.pl
ln -s ../madhatter/build-seamonkey-util.pl
ln -s ../madhatter/build-seamonkey.pl build-seamonkey.pl.orig
ln -s ../madhatter/build-thunderbird.pl
ln -s ../madhatter/examples
ln -s ../madhatter/gettime.pl
ln -s ../madhatter/install-links
ln -s ../madhatter/multi-tinderbox.pl
ln -s ../madhatter/post-mozilla-l10n.pl
ln -s ../madhatter/post-mozilla-rel.pl
ln -s ../madhatter/post-mozilla-sample.pl
ln -s ../madhatter/post-mozilla.pl post-mozilla.pl.orig
ln -s ../madhatter/qatest.pl
ln -s ../madhatter/report-http-status.pl
ln -s ../madhatter/report-tree-status.pl
ln -s ../madhatter/reportdata.pl
ln -s ../madhatter/test-reportdata.pl
ln -s ../madhatter/tinder-defaults.pl
ln -s ../madhatter/tinderbox

Now we need to check out the tinderbox config files. Because we‘re advanced, we have our config files in CVS. To check them out, first go into the /builds/tinderbox/Cm2.1-M1.9.2 directory then run the following command (using the example of minus; substitute the proper directory and branch for the relevant box, remembering that cb-xserve01 uses the macosx directory directly):

cvs co -r BRANCHNAME -d tinderbox-configs mozilla/tools/tinderbox-configs/camino/macosx/cb-minibinus01/

After this, it‘s time to create a few more symlinks! (Note: When setting up the box, it is customary to run it under local config to get it working; to do so, use copies of the checked-out files instead of symlinks. Once the configuration is stable, commit the local copies to CVS, delete the local files, and then add the symlinks below.)

ln -s tinderbox-configs/CLOBBER
ln -s tinderbox-configs/mozconfig
ln -s tinderbox-configs/tinder-config.pl

Finally, we need to create a couple more files in this directory. First we want to copy the original build-camino.pl script here and name it build-seamonkey.pl, and then copy one other file.

cp ../madhatter/build-camino.pl build-seamonkey.pl
cp ../madhatter/post-mozilla.pl post-mozilla.pl

Next, un-comment out the line in post-mozilla.pl (starting with "require"), then make it and build-seamonkey.pl executable (chmod 755 post-mozilla.pl build-seamonkey.pl).

Almost there…

You now need to add the multi-config.pl file to /builds/tinderbox/. This file will define what builds run on the machine.

The config we use for Camino with mozilla-1.9.2 is: 

$ENV{'CVS_RSH'}='ssh';
$ENV{'TBOX_CLIENT_CVS_DIR'}='/builds/tinderbox/mozilla/tools/tinderbox';
$ENV{'TBOX_CLIENT_HG_DIR'}='/builds/tinderbox/madhatter';
$BuildSleep = 5;
$Tinderboxes = [
  { tree => "Cm2.1-M1.9.2", args => "--config-cvsup-dir /builds/tinderbox/Cm2.1-M1.9.2/tinderbox-configs" },
];

To determine whether you're producing nightly builds that get delivered to ftp.mozilla.org or just hourly builds (i.e., just running Tp/Ts/Tdhtml), set $OfficialBuildMachinery appropriately in your local tinder-config.pl. The default is 1 for nightly clobber builds (and “hourly” depend tinderbox-builds); uncomment the variable and set its value to 0 to produce only “hourly” builds (which are uploaded to tinderbox-build/ directories). (See the comment for that value; do we have to do anything to that when we're doing hourlies?) Note that some Camino tinderboxen have moved aside post-mozilla.pl (as a hack?); these should all be fixed to use the proper tinder-config.pl settings.

The last step is only pertinent if this tinderbox will be creating builds to be distributed on ftp.mozilla.org. SSH to stage.mozilla.org and type in “yes” when prompted. Repeat for talkback-upload.mozilla.org if the build contains Talkback, or dm-symbolpush01.mozilla.org if the build contains Breakpad. If you don't do this, the tinderbox script will pause at the very end, waiting for someone to enter “yes”.

ssh stage.mozilla.org
ssh talkback-upload.mozilla.org
ssh symbolpush.mozilla.org

Finally, copy the prebuilt Talkback binary package to the appropriate location (/builds/tinderbox/Cm2-M1.9 for cvs trunk) and set the appropriate path and filename in tinder-config.pl.

That should be it!

Starting Tinderbox

Try running a tinderbox with ./multi-tinderbox.pl and see what happens (Sam prefers this method, since it spews the build process in the Terminal, while Mark and Smokey prefer ./tinderbox multi start which does not spew the build process info, and which can be stopped). This command must be run in the Terminal application on the machine (the script requires a proper display or tests will fail). You do this by accessing the box from a VNC session. Be sure to hide (not minimize) Terminal.app afterwards; otherwise, Ts results will be adversely affected.

After the first build has begun, you need to go to the Camino tinderbox page, click “Administrate Tinderbox Trees” (it should say “Administer,” but whatever), and turn on log scraping for the new build. Do it before the first build completes. Repeat this step for any other builds you are setting up.

If you want to check on the build progress, you can

tail -f /builds/tinderbox/Cm2.1-M1.9.2/Darwin_8.11.0_Depend/Darwin_8.11.0_Depend.log

(where Darwin_8.11.0_Depend is the buildhost+build type).

Fun with the mail queue

Sometimes, MoCo IT will do something that will cause tinderbox to stop accepting mail from our tinderboxen. In these cases, there will be a large backlog of mail built up, and since postfix only runs on demand, for 60s at a time, it will take a long time to clear.

If there are only a few hours or a 1-2 days of backlogged mail, tinderbox will likely be able to handle the backlog sanely. In this case, the easiest way to get postfix running constantly is to touch a junk file in the maildrop directory[1] (use the first for client and the second for Server): 

$ sudo touch /var/spool/postfix/maildrop/junk
$ sudo touch /Library/Server/Mail/Data/spool/maildrop/junk

Then once postfix has cleared the backlog, sudo rm the dummy file.

If there are more than a few days of backlogged mail, tinderbox will go crazy trying to process it, so it's better to purge the queue instead: 

$ sudo postsuper -d ALL

This will delete everything in the mail queue, new and old, and report back the number of messages deleted. (If it doesn't report a number of messages deleted, then it didn't run; try forcing postfix to start and then quickly clearing the queue.)

Finally, to see the contents of the mail queue: 

$ sudo mailq

Releasing Camino

Instructions on how to (branch and tag and build and) release a new version of Camino can be found elsewhere on the Camino wiki, in Releases:Branch Tag Build Release (the how-to-mento handbook).

Retrieved from "http://wiki.caminobrowser.org/index.php?title=IT:Setting_up_Camino_Tinderboxen&oldid=12168"