IT:Setting up Camino Tinderboxen

From Camino Wiki
Revision as of 23:27, 27 February 2010 by Sardisson (talk | contribs) (→‎Basic Configuration: note about fixing desktop bg permanently on 10.5 and up)
Jump to navigation Jump to search

How to (configure and set up and start and) run a Camino tinderbox
by Samuel Sidler (with some editing by Smokey Ardisson)
on or before 2008-07-01

10.4 Tinderbox

Configuring the Box

With a new machine, you first need to set up the box. To do this, reinstall the OS (10.4) from a clean disk, including reformatting the machine. For the first run information, make your default user be caminobld with a password of {redacted}. This is the standard user we use for build machines.

Basic Configuration

Next, open System Preferences and begin customizing the machine.

  1. In the Desktop & Screen Save pane, turn off the screen saver and set the desktop to be a solid color.
    • On 10.5 and above, headless machines can't set their desktop backgrounds permanently. Use sudo defaults write /Library/Preferences/com.apple.loginwindow DesktopPicture /Library/Desktop\ Pictures/Solid\ Colors/Solid\ Aqua\ Blue.png instead.
  2. In the Energy Saver pane, change the sliders to "never", indicating you don't want the machine to go to sleep at all.
  3. In the Keyboard & Mouse pane, make sure Full Keyboard Access is set to "Text boxes and lists only".
  4. In the Dock pane, set the Dock to auto-hide.
  5. In the Sharing pane, turn on Remote Login.
  6. Also in the Sharing pane, turn on Apple Remote Desktop (on 10.5, this is "Remote Management"). Here, you'll customize the Access Privileges (on 10.5, "Computer Settings").
    1. Turn "On" the caminobld user.
    2. Check all boxes under "Allow user to do the following on this computer".
    3. Check the VNC box below and set a password of {redacted}.
  7. And one more change in the Sharing pane, change the "Computer Name" to be the name given (such as, cb-minibinus01).
  8. In the Software Update pane, uncheck the box "Check for updates". (We specifically don't automatically check because it can cause unexpected Tp/Ts regressions.)
  9. In the Network pane, be sure your ethernet is set to the appropriate IPs from the IT team.
    1. Make sure that more than one DNS server is set, so that when one goes down for maintenance, the machine doesn't lose the ability to talk to the world.
  10. In the International pane, set a language other than English as the first language and then restore English to the top position; this should eliminate any weirdness with non-ISO values in AppleLanguages
    1. Though this was supposed to have never occurred unless a machine was upgraded from 10.2 or below, we have seen cb-xserve01 afflicted with this bug.

Mail Configuration

Next, make changes to ensure that the mail will work correctly.

The tinderboxen are using their hostname as the domain portion of originated e-mail addresses, so (without any intervention), they'd be sending mail as whatever@cb-xserve01.local. The mail server is getting pissed off because that's not a valid return address. (Note that while cb-xserve01 has a simple mozilla.com hostname, the other tinderboxen all have sj.mozilla.com hostnames. Check with IT if you're not sure of the actual hostname.)

We can fix this by editing /etc/postfix/main.cf in vi or your editor of choice and setting myhostname explicitly:

myhostname = cb-xserve01.mozilla.com

We should also add these lines, because we've found them to be necessary on tinderboxen:

# Allow large messages (unlimited size, required for tinderbox)
message_size_limit = 0

Then, reload Postfix to get it to reread its configuration:

sudo postfix reload

Hostname in other places

One additional change Mark usually makes: edit /etc/hostconfig and set HOSTNAME to the desired hostname, in the case of cb-xserve01, cb-xserve01. At its default setting, -AUTOMATIC-, the system picks a hostname without necessarily doing the best job. The automatic hostname may even wind up changing across reboots. That's kind of annoying. Explicit settings are better here. (Server 10.4.6 was supposed to fix this, but cb-xserve01 regularly displays the problem and even cb-xserve04 once ended up with a hostname of -cb-sxerve04- after a reboot.)

Mark always saves the original files as he found them in whatever.orig files for future diffing pleasure.

N.B. There may be an additional change required to get the right hostname set for sending results to the graph server; this is currently under investigation. cb-xserve01 appears to need HOSTNAME in /etc/hostconfig to be set to cb-xserve01.mozilla.com to match the DNS name set by Mozilla IT. This may be an artifact of Mac OS X Server, or something else we don't yet know.

This probably should be fixed with sudo scutil --set HostName cb-xserve01.mozilla.com and rebooting; this TN warns against setting the hostname in /etc/hostconfig. Other useful commands are sudo scutil --get HostName and sudo changeip -checkhostname.

Build Environment

First, set up the build environment. You should always make sure the system is completely up-to-date by running Software Update. We then install XCode 2.5, then use MacPorts to install libIDL and Glib, as discussed in the Camino build documentation.

Additionally, if you plan to run tests, install wget.

sudo port install wget

Crash Reporter Configuration

Mac OS X Crash Reporter

Because Crash Reporter opens a window when a process crashes, and windows present on the screen affect performance test results, we need to be sure to set Crash Reporter to silently.

Once you’ve installed Xcode, open the CrashRepoterPrefs application in /Developer/Applications/Utilities/ and configure Crash Reporter to use Server mode. This will still write crash reports to disk but will suppress the crash reporter window itself.

Breakpad Crash Reporter

On tinderboxen building any Camino branches with Breakpad, execute the following command to ensure that any reports are submitted automatically, allowing the client to both quit and be blown away during rebuild:

defaults write org.mozilla.camino BreakpadSkipConfirm -string YES

If you later need to get the Breakpad report ID, there will be an entry like this in the Console:

Apr 21 14:27:21 Qalaat-Samaan com.apple.launchd.peruser.501[121]: Breakpad Reporter: Renamed /Users/smokey/Library/Breakpad/Camino/F8D9CA55-E65F-4439-BCA9-8A794A92F055.dmp to /Users/smokey/Library/Breakpad/Camino/CrashID=bp-d1ccccf3-e418-40bd-a3fc-a32c42090421.dmp after successful upload

Since non-nightly builds will not have symbols, you will need to make Breakpad stop suppressing the Mac OS X Crash Reporter in order to have useful crash reports saved on the tinderbox:

defaults write org.mozilla.camino BreakpadSendAndExit -string NO

Keys and Profile

Now that you have a build environment, you need to add the appropriate private keys and environment variables to continue.

First, add your private key as follows:

~/.ssh/id_dsa:
-----BEGIN DSA PRIVATE KEY-----
Did you really think we'd publish the private key?
-----END DSA PRIVATE KEY-----

Then, add your public key:

~/.ssh/id_dsa.pub:
Consult your friendly build administrator for this one, too.

Be sure to set permissions on your private key to be 500 by running chmod 500 ~/.ssh/id_dsa.

Finally, we want to set environment variables globally. Use vi (or your editor of choice) to edit /etc/profile with the contents below.

#!/bin/sh

CC='gcc' CXX='g++' EDITOR='vi' ENV='/etc/profile' LESS='-acdefi~P?f%f:<stdin>. ?bBByte\: %bB .?lbLine\: %lb .?pB%pB\% .?e(END):--More--.?x+.' LESSHISTFILE='-' PAGER='less' PATH='/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/X11/bin' VISUAL='vi' export CC CXX EDITOR ENV LESS LESSHISTFILE ORGANIZATION PAGER PATH VISUAL HISTSIZE='1000' unset HISTFILE if [ 'x'$BASH != 'x' ];then

 PS1='\u@\h bash\$ '
 shopt -s checkwinsize

else

 if [ 'x'$USER == 'xroot' ];then
   PS1=$USER'@'`/bin/hostname`' sh# '
 else
   PS1=$USER'@'`/bin/hostname`' sh$ '
 fi

fi umask 22 PATH='/opt/local/bin:/opt/local/sbin:'$PATH PATH=$HOME'/bin':$PATH

CVS_RSH='ssh' CVSROOT=':ext:caminobld@cvs.mozilla.org:/cvsroot'

export CVS_RSH CVSROOT

Now, create a link from ~/.bashrc to /etc/profile using the following command: ln -s /etc/profile ~/.bashrc.

Restart Terminal.app.

Setting up Tinderbox

The moment you've all been waiting for!

Camino tinderboxen install the tinderbox to /builds/tinderbox/. You'll need to create this directory.

Next, while in /builds/tinderbox/, check out the tinderbox scripts as well as the performance scripts (for perf test running).

cvs co mozilla/tools/tinderbox/
cvs co mozilla/tools/performance/

When prompted, type in "yes" to authorize SSH to connect.

Now we need to create some symlinks. These links are created so that when we update the tinderbox script, our builds will use the new version instead of the old. This makes maintenance much easier.

First, in /builds/tinderbox/, we create two symlinks:

ln -s mozilla/tools/tinderbox/multi-tinderbox.pl multi-tinderbox.pl
ln -s mozilla/tools/tinderbox/tinderbox tinderbox

Now, before we continue with linking, decide what you're building: branch or trunk (or another branch!). The rest of this documentation assumes you're building cvs trunk. (To build a branch, changes are minimal: you just need to create the appropriately-named directory for the branch in the step below, later check out the branch’s tinderbox-configs in that step, and use the appropriate branch information when creating multi-config.pl.)

First, create a directory for the remaining symlinks to live.

mkdir Cm2-M1.9

Now, in that new directory, create a bunch of symlinks:

ln -s ../mozilla/tools/tinderbox/bloatdiff.pl
ln -s ../mozilla/tools/tinderbox/build-camino.pl
ln -s ../mozilla/tools/tinderbox/build-firefox.pl
ln -s ../mozilla/tools/tinderbox/build-galeon.pl
ln -s ../mozilla/tools/tinderbox/build-seamonkey-util.pl
ln -s ../mozilla/tools/tinderbox/build-seamonkey.pl build-seamonkey.pl.orig
ln -s ../mozilla/tools/tinderbox/build-thunderbird.pl
ln -s ../mozilla/tools/tinderbox/examples
ln -s ../mozilla/tools/tinderbox/gettime.pl
ln -s ../mozilla/tools/tinderbox/install-links
ln -s ../mozilla/tools/tinderbox/multi-tinderbox.pl
ln -s ../mozilla/tools/tinderbox/post-mozilla-l10n.pl
ln -s ../mozilla/tools/tinderbox/post-mozilla-rel.pl
ln -s ../mozilla/tools/tinderbox/post-mozilla-sample.pl
ln -s ../mozilla/tools/tinderbox/post-mozilla.pl post-mozilla.pl.orig
ln -s ../mozilla/tools/tinderbox/qatest.pl
ln -s ../mozilla/tools/tinderbox/report-http-status.pl
ln -s ../mozilla/tools/tinderbox/report-tree-status.pl
ln -s ../mozilla/tools/tinderbox/reportdata.pl
ln -s ../mozilla/tools/performance/startup/startup-test.html
ln -s ../mozilla/tools/tinderbox/test-reportdata.pl
ln -s ../mozilla/tools/tinderbox/tinder-defaults.pl
ln -s ../mozilla/tools/tinderbox/tinderbox

Now we need to check out the tinderbox config file. Because we‘re advanced, we have our config files in CVS. To check them out, first go into the /builds/tinderbox/Cm2-M1.9 directory then run the following command (using the example of minus; substitute the proper directory and branch for the relevant box, remembering that cb-xserve01 uses the macosx directory directly):

cvs co -r BRANCHNAME -d tinderbox-configs mozilla/tools/tinderbox-configs/camino/macosx/cb-minibinus01/

After this, it‘s time to create a few more symlinks! (Note: When setting up the box, it is customary to run it under local config to get it working; to do so, use copies of the checked-out files instead of symlinks. Once the configuration is stable, commit the local copies to CVS, delete the local files, and then add the symlinks below.)

ln -s tinderbox-configs/CLOBBER
ln -s tinderbox-configs/mozconfig
ln -s tinderbox-configs/tinder-config.pl

Finally, we need to create a couple more files in this directory. First we want to copy the original build-camino.pl script here and name it build-seamonkey.pl, and then copy one other file.

cp ../mozilla/tools/tinderbox/build-camino.pl build-seamonkey.pl
cp ../mozilla/tools/tinderbox/post-mozilla.pl post-mozilla.pl

Next, un-comment out the line in post-mozilla.pl (starting with "require"), then make it and build-seamonkey.pl executable (chmod 755 post-mozilla.pl build-seamonkey.pl).

Almost there…

You now need to add the multi-config.pl file to /builds/tinderbox/. This file will define what builds run on the machine.

The config we use for cvs trunk is:

$ENV{'CVS_RSH'}='ssh';

$ENV{'TBOX_CLIENT_CVS_DIR'}='/builds/tinderbox/mozilla/tools/tinderbox'; $BuildSleep = 5; $Tinderboxes = [

 { tree => "CmTrunk", args => "--config-cvsup-dir /builds/tinderbox/CmTrunk/tinderbox-configs" },

];

To determine whether you're producing nightly builds that get delivered to ftp.mozilla.org or just hourly builds (i.e., just running Tp/Ts/Tdhtml), set $OfficialBuildMachinery appropriately in your local tinder-config.pl. The default is 1 for nightly clobber builds (and “hourly” depend tinderbox-builds); uncomment the variable and set its value to 0 to produce only “hourly” builds (which are uploaded to tinderbox-build/ directories). (See the comment for that value; do we have to do anything to that when we're doing hourlies?) Note that some Camino tinderboxen have moved aside post-mozilla.pl (as a hack?); these should all be fixed to use the proper tinder-config.pl settings.

The last step is only pertinent if this tinderbox will be creating builds to be distributed on ftp.mozilla.org. SSH to stage.mozilla.org and type in “yes” when prompted. Repeat for talkback-upload.mozilla.org if the build contains Talkback, or dm-symbolpush01.mozilla.org if the build contains Breakpad. If you don't do this, the tinderbox script will pause at the very end, waiting for someone to enter “yes”.

ssh stage.mozilla.org
ssh talkback-upload.mozilla.org
ssh dm-symbolpush01.mozilla.org

Finally, copy the prebuilt Talkback binary package to the appropriate location (/builds/tinderbox/Cm2-M1.9 for cvs trunk) and set the appropriate path and filename in tinder-config.pl.

That should be it!

Starting Tinderbox

Try running a tinderbox with ./multi-tinderbox.pl and see what happens (Sam prefers this method, since it spews the build process in the Terminal, while Mark prefers ./tinderbox multi start which does not spew the build process info). This command needs to be run in the Terminal application on the machine (the script requires a proper display or tests will fail). You do this by accessing the box from a VNC session. Be sure to hide (not minimize) Terminal.app afterwards; otherwise, Ts results will be adversely affected.

After the first build has begun, you need to go to the Camino tinderbox page, click “Administrate Tinderbox Trees” (it should say “Administer,” but whatever), and turn on log scraping for the new build. Do it before the first build completes. Repeat this step for any other builds you are setting up.

If you want to check on the build progress, you can

tail -f /builds/tinderbox/Cm2-M1.9/Darwin_8.11.0_Depend/Darwin_8.11.0_Depend.log

(where Darwin_8.11.0_Depend is the buildhost+build type).

Releasing Camino

Instructions on how to (branch and tag and build and) release a new version of Camino can be found elsewhere on the Camino wiki, in Releases:Branch Tag Build Release (the how-to-mento handbook).