From RAD Lab

Contents 1 RAD Lab Golden Image 1.1 Known Issues 1.2 Latest AMIs (last release: 4/28/09) 1.3 Release History: 4/11/09 1.4 Release History: 2/25/09 1.5 Release History: 11/09/08 1.6 Release History: 11/02/08 1.7 Release History: 10/25/08 1.8 Build Preparation 1.9 Building a 32-bit AMI (notes/comments in bold) 1.10 Building a 64-bit AMI (notes/comments in bold) 1.11 Registering an AMI 1.12 Extending the build framework

RAD Lab Golden Image

Known Issues

• Many packages aren't available for install until apt-get update is run.
• Outdated Chef Gem (http://github.com/opscode/chef/tree/0.7.4)
• Modify install-cloudstone
  • Downgrade Rack to 0.9.0 (Olio does not support Rack 1.0.0.)
  • Configure syslog to listen for HAProxy logs (see Chef recipes).
  • Create nginx service script in /etc/init.d.
• Create install-postgresql
  • Install PostgreSQL 8.4 to /mnt/postgresql (see Chef recipes).
  • Downgrade the libpq5 package (see Chef recipes).
  • Install the postgres gem (see Chef recipes).
  • Increase kernel shared buffer size (see Chef recipes).
• Install the IRB package.
• Modify install-R
  • Add: echo " install.packages('RMySQL',dependencies=TRUE,repos='http://cran.cnr.berkeley.edu/') " | R --vanilla
• Create install-java6
  • Install Java 1.6 from Sun on the image
• Increase default max number of files open to 65535 in /etc/security/limits.conf
  • soft nofile 65535
  • hard nofile 65535

Latest AMIs (last release: 4/28/09)

• Latest golden image AMIs available: ami-e7a2448e (32-bit AMI) and ami-e4a2448d (64-bit AMI)
  • Known additions/improvements to these AMIs:
    • Fixed broken mysql gem install (thanks Hubert, David, Aaron)
    • Fixed broken mysql (thanks Hubert, David, Aaron)
    • Updated R installation (thanks Peter)
    • Fixed the locale settings error messages
    • Installed and configured chef so it works out of the box (thanks Michael, Jesse)
    • Restricted the golden image to use only the RAD Lab repos (thanks Michael, Jesse, Nick) - use enableapt if you need to install software from an outside repo on an instance
  • AMIs obsoleted
    • 32-bit ami-e7a2448e replaces ami-8d7a9de4
    • 64-bit ami-e4a2448d replaces ami-8c7a9de5
  • If something is broken/missing from the AMIs email: rean AT eecs so it can be resolved in a future build/release.

Release History: 4/11/09

• Latest golden image AMIs available: ami-8d7a9de4 (32-bit AMI) and ami-8c7a9de5 (64-bit AMI)
  • Known additions/improvements to these AMIs:
    • Golden image only talks to RAD Lab repos by default (thanks Michael, Nick, Jesse), run enableapt if you need to contact outside repos
    • Updated mysql gem (thanks Hubert)
    • Installed and configured chef so it works out of the box
  • AMIs obsoleted
    • 32-bit ami-8d7a9de4 replaces ami-99cc2bf0
    • 64-bit ami-8c7a9de5 replaces ami-84cc2bed
  • If something is broken/missing from the AMIs email: rean AT eecs so it can be resolved in a future build/release.

Release History: 2/25/09

• Latest golden image AMIs available: ami-99cc2bf0 (32-bit AMI) and ami-84cc2bed (64-bit AMI)
• Release Notes
  • Known additions/improvements to these AMIs:
    • Installed runit
    • Installed chef gem
    • Installed R
    • Added pointer to the radlab apt repo to /etc/apt/sources.list
  • AMIs obsoleted
    • 32-bit ami-99cc2bf0 replaces ami-de15f1b7
    • 64-bit ami-84cc2bed replaces ami-d115f1b8
  • If something is broken/missing from the AMIs email: rean AT eecs so it can be resolved in a future build/release.

Release History: 11/09/08

• Latest golden image AMIs available: ami-de15f1b7 (32-bit AMI) and ami-d115f1b8 (64-bit AMI). Both AMIs are built on an Ubuntu 8.04 LTS Hardy base.
• Release Notes:
  • Known additions/improvements to these AMIs:
    • MySQL 5.1.28-rc automatically configured with innodb (thanks Will)
    • 32-bit and 64-bit AMIs both use the same director SSH key (stored on s3 in the radlab_goldenimage_build bucket)
    • will_paginate gem installed via install-cloudstone script
  • AMIs obsoleted:
    • 32-bit ami-de15f1b7 replaces ami-2102e648
    • 64-bit ami-d115f1b8 replaces ami-2002e649
  • If something is broken/missing from the AMIs email: rean AT eecs so it can be resolved in a future build/release.

Release History: 11/02/08

• Latest golden image AMIs available: ami-2102e648 (32-bit AMI) and ami-2002e649 (64-bit AMI). Both AMIs are built on an Ubuntu 8.04 LTS Hardy base.
• Release Notes:
  • Known additions/improvements to these AMIs:
    • Added quiet version of ssh-keygen command to generate key used by the Director to ssh into machines
    • Edited mysql build script (in install-cloudstone) to make sure it creates /etc/init.d/mysql
    • Added install "gem install amazon-ec2" to the install-cloudstone script
    • Copied Amazon secret keys to ami and set appropriate env vars in .bash_profile script. ec2-describe-instances works out-of-the-box.
    • Update path of root user to include /usr/local/mysql/bin
    • Move /ec2-api-tools-1.3-26369/ into /root rather and added bin directory to path of root user
    • Fixed open jdk 7 install script (install-java7). openjdk binaries stored on s3 (radlab_java bucket) and pulled from there using s3cmd utility
    • /root/build dir created by install-cloudstone deleted/cleaned up before bundle and upload to s3
    • Added nginx to the path of the root user
    • mysql-5.1.28-rc tarball stored on S3 (radlab_goldenimage_build bucket) and install-cloudstone now pulls it from s3 using s3cmd utility
    • ruby gem tarball stored on S3 (radlab_goldenimage_build bucket) and install-cloudstone now pulls it from s3 using s3cmd utility
    • Added apt-get install for libreadline-dev (thanks Wei)
  • AMIs obsoleted:
    • 32-bit ami-2102e648 replaces ami-b83dd9d1
    • 64-bit ami-2002e649 replaces ami-813dd9e8
  • If something is broken/missing from the AMIs email: rean AT eecs so it can be resolved in a future build/release.

Release History: 10/25/08

• AMIs available: ami-b83dd9d1 (32-bit AMI) and ami-813dd9e8 (64-bit AMI). Both AMIs are built on an Ubuntu 8.04 LTS Hardy base. These AMIs will be deleted on 11/7/2008 unless otherwise announced. Please update to the latest AMIs.
• Release Notes:
  • Known additions/improvements to these AMIs:
    • capistrano, sqlite3, libsqlite3-dev, sqlite3-ruby installed [thanks Peter, Charles R for pointing out these omitted packages]
    • ec2-api-tools installed (in /, this will probably move to /root in future AMI builds)
    • Fixed MySQL 5.1.28-rc post-install initialization
  • AMIs obsoleted:
    • 32-bit AMI ami-b83dd9d1 replaces ami-0a32d663
  • If something is broken/missing from the AMIs email: rean AT eecs so it can be resolved in a future build/release.

Build Preparation

• Overview

Our build script is based on the ec2ubuntu-build-ami described here, original version can be found here. We have made some alterations to tailor this script for our needs. The RAD Lab's version of this script and the additional build tools required can be obtained using git (see below).

• Build Prerequisites

1. An R-cluster account (see EC2 Bootstrap Guide | Getting Access to R Cluster)
2. An EC2 keypair (see EC2 Bootstrap Guide | Getting EC2 Credentials)
3. Golden Image Build tools
   1. All our build tools can be obtained using git:

      git clone ssh://<username>@<R-cluster machine>/project/cs/radlab/src/goldenimage.git/ goldenimage.git

   2. If you are working on an R-cluster machine then you can use:

       git clone /project/cs/radlab/src/goldenimage.git/ goldenimage.git 

4. A Build Machine AMI
   1. You need a 32-bit or 64-bit AMI to use as your build machine/environment depending on whether you are creating a 32-bit golden image AMI or a 64-bit golden image AMI. We use ami-f51aff9c (a 32-bit, 2.6.21 kernel, Fedora Core AMI) and ami-f21aff9b (a 64-bit, 2.6.21 kernel, Fedora Core AMI).

Building a 32-bit AMI (notes/comments in bold)

1. ec2-run-instances ami-f51aff9c -k <user keypair> -z us-east-1a -g director --instance-type m1.small [use ec2-describe-instances to find your instance IP address, e.g., ec2-describe-instances | grep rean-keypair]
2. scp -i <path to user keypair> ~/.ec2/cert-RPTDFBBI3SAOFH5VQQHMJKXJSUNPBMZW.pem ~/.ec2/pk-RPTDFBBI3SAOFH5VQQHMJKXJSUNPBMZW.pem root@<EC2 Instance Public IP>:/root
3. ssh -i <path to user keypair> root@<EC2 Instance Public IP>
4. yum install git -y
5. yum install emacs -y [if you prefer to have emacs around for editing]
6. git clone ssh://<R-cluster user name>@scm.millennium.berkeley.edu/project/cs/radlab/src/goldenimage.git goldenimage.git
7. goldenimage.git/bin/install-s3cmd-tool [Install the s3cmd tool so we can fetch things from s3, e.g., JDK binary installers]
8. cp goldenimage.git/bin/.s3cfg ~ [Copy the configuration file with the RAD Lab's EC2 credentials. This file is used by s3cmd]
9. s3cmd get s3://radlab_java/jdk-6u10-linux-i586-rpm.bin jdk-6u10-linux-i586-rpm.bin
10. chmod 755 jdk-6u10-linux-i586-rpm.bin
11. echo yes | ./jdk-6u10-linux-i586-rpm.bin > /dev/null
12. export JAVA_HOME=/usr/java/jdk1.6.0_10/ [required by ec2-register to register your AMI with S3]
13. cp goldenimage.git/bin/ec2-api-tools.zip .
14. unzip ec2-api-tools.zip
15. cd goldenimage.git/bin/
16. source set-ec2-AMI-build-env.sh [only if you want the AMI automatically bundled and uploaded to the "radlab" bucket on S3 otherwise the file system for your image is in /mnt/build/ubuntu]
17. ./ec2ubuntu-build-ami 2>&1 | tee build.log [Start the build and dump all the messages into build.log]
18. Find something else to do for the ~1.5 hrs it takes to build the image

Building a 64-bit AMI (notes/comments in bold)

1. ec2-run-instances ami-f21aff9b -k <user keypair> -z us-east-1a -g director --instance-type m1.large [use ec2-describe-instances to find your instance IP address, e.g., ec2-describe-instances | grep rean-keypair]
2. scp -i <path to user keypair> ~/.ec2/cert-RPTDFBBI3SAOFH5VQQHMJKXJSUNPBMZW.pem ~/.ec2/pk-RPTDFBBI3SAOFH5VQQHMJKXJSUNPBMZW.pem root@<EC2 Instance Public IP>:/root
3. ssh -i <path to user keypair> root@<EC2 Instance Public IP>
4. rpm -e --nodeps --allmatches curl-7.17.1-2.fc8.i386
5. yum install git -y
6. yum install emacs -y [if you prefer using emacs for editing]
7. git clone ssh://<R-cluster user name>@scm.millennium.berkeley.edu/project/cs/radlab/src/goldenimage.git goldenimage.git
8. goldenimage.git/bin/install-s3cmd-tool [Install the s3cmd tool so we can fetch things from s3, e.g., JDK binary installers]
9. cp goldenimage.git/bin/.s3cfg ~ [Copy the configuration file with the RAD Lab's EC2 credentials. This file is used by s3cmd]
10. s3cmd get s3://radlab_java/jdk-6u10-linux-x64-rpm.bin jdk-6u10-linux-x64-rpm.bin
11. chmod 755 jdk-6u10-linux-x64-rpm.bin
12. echo yes | ./jdk-6u10-linux-x64-rpm.bin > /dev/null
13. export JAVA_HOME=/usr/java/jdk1.6.0_10/ [required by ec2-register to register your AMI with S3]
14. cp goldenimage.git/bin/ec2-api-tools.zip .
15. unzip ec2-api-tools.zip
16. cd goldenimage.git/bin/
17. source set-ec2-AMI-build-env.sh [only if you want the AMI automatically bundled and uploaded to the "radlab" bucket on S3 otherwise the file system for your image is in /mnt/build/ubuntu]
18. ./ec2ubuntu-build-ami 2>&1 | tee build.log [Start the build and dump all the messages into build.log]
19. Find something else to do for the ~1.5 hrs it takes to build the image

Registering an AMI

1. ec2-register <bucket name>/<image manifest> [bucket name = radlab and the console output from the ec2ubuntu-build-ami script is the image manifest name]

Extending the build framework