EC2 Bootstrap Guide
From RAD Lab
Contents |
Bootstrap Procedure
Getting access to R Cluster
You'll need an account on the R Cluster to get access to the EC2 credentials file (plus, this tutorial assumes your setting up EC2 access from a R Cluster account)
Request an account at the following address
- https://www.millennium.berkeley.edu/account/
- Remember to mark the "RAD Lab" entry under "Private Clusters"
- The /work directory is only mounted if you're a member of the R Cluster and that's where the EC2 Credential file lives
- It'd be a good idea to request being added to the 'radlab' and 'grads' groups, not that I know how to do this since there's no comment field... email radlab-support@cs?
To login:
ssh <loginname>@r10.millennium.berkeley.edu
Getting EC2 Credentials
You'll need a copy of the RAD Lab's EC2 credentials to use the ec2 interaction tools. To get them:
cp -r /work/ec2_credentials/.ec2 ~/.ec2
Setting up your environment
Next comes some environment variables magic.
Changing your shell
I'm assuming you're using some bash-compatible shell here. Alas, the default shell on the R Cluster is csh. To change your shell, use ypchsh:
% ypchsh Changing NIS account information for jtrutna on admin.Millennium.Berkeley.EDU. Please enter password: Changing login shell for jtrutna on admin.Millennium.Berkeley.EDU. To accept the default, simply press return. To use the system's default shell, type the word "none". Login shell [/bin/csh]: /bin/bash The login shell has been changed on admin.Millennium.Berkeley.EDU.
It seems to take a little while for the preferences to percolate. Until that happens, just manually change shells
bash
Setting Necessary Environment variables
Add the following to your ~/.profile
export EC2_HOME=/work/ec2_tools export PATH=$PATH:$EC2_HOME/bin export EC2_PRIVATE_KEY=`ls ~/.ec2/pk-*.pem` export EC2_CERT=`ls ~/.ec2/cert-*.pem` export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun
- (JAVA_HOME Found by tracking down 'which java'->/etc/alternatives/java->/usr/lib/jvm/java-1.5.0-sun/bin/java)
Now load in those changes.
source .profile
EC2 should be working now, type
ec2ver
and it should report the version number. Ta-Da!
Generating a Keypair
EC2 uses public/private-key cryptography to control access to new instances, so you'll need to generate a pair to launch new instances (unless you're using an existing pair, in which case, why are you reading this tutorial?).
The ec2-add-keypair command is used to generate keypairs. It takes a label, generates a public-private keypair, stores and associates the public key with it's name, and prints the private key to standard out. When you start a new instance, you give it the name of the keypair you want to use. Amazon will add the public key associated with that name to the ~/.ssh/authorized_key file. You can then log onto your newly created instance using the private key.
You'll want to choose something descriptive for the keypair name, since keypair names are only way to differentiate running instances. Using your own name or the name of a project are good choices. We redirect stdout to a file to save the private key for later.
ec2-add-keypair <keypair_name> > <privatekey_file>
Since ssh is going to use this later and it's paranoid, you'll need to modify the permissions.
chmod 400 <privatekey_file>
Starting/Stopping Instances
Listing Available Images
To see a list of available images (created by members of the radlab):
ec2-describe-images -o self -H --show-empty-fields
In particular, the golden master should be in this list (hopefully). Find the one you care about and note the value in the second column, the ImageID. You'll use it to identify the image you want to launch.
Starting an Instance
To launch an instance, type
ec2-run-instances <imageID> -k <keypair_name>
- Note: -k specifies the name of the keypair.
This should return output that looks something like:
jtrutna@r10:~$ ec2-run-instances ami-0ec02467 -k <keypair_name> RESERVATION r-7f2efe16 117716615155 default INSTANCE i-1a298873 ami-0ec02467 pending <keypair_name> 0 m1.small 2008-09-19T02:44:58+0000 us-east-1a
(if it says 'pending' instead of running, you need to wait longer) Wait a bit to give the instance a chance to boot, then type:
ec2-describe-instances
Which will give you a huge list of the currently running instances, your's should be near the bottom.
...<truncated>... RESERVATION r-1626f67f 117716615155 manager INSTANCE i-8e5ffee7 ami-b19074d8 ec2-75-101-245-64.compute-1.amazonaws.com ip-10-251-158-31.ec2.internal running perf-keypair 0 m1.large 2008-09-18T22:08:11+0000 us-east-1a RESERVATION r-7f2efe16 117716615155 default INSTANCE i-1a298873 ami-0ec02467 ec2-67-202-22-42.compute-1.amazonaws.com ip-10-251-75-193.ec2.internal running <keypair_name> 0 m1.small 2008-09-19T02:44:58+0000 us-east-1a
- The 2nd column of the line starting with "INSTANCE" (i-1a298873, in this case) is the instance ID and is needed to access or shutdown the image.
- The 4th column is the external url (ec2-67-202-22-42.compute-1.amazonaws.com), used to access the machine from outside the cloud
- The 5th column is the internal url (ip-10-251-75-193.ec2.internal), and is used for instances to talk to each other or to ssh from one machine to another.
If you forget it later on, you can get a list of instances currently running by typing
ec2-describe-instances (or 'ec2din')
Terminating an Instance
When your finished running your experiments, you can shutdown an instance using
ec2-terminate-instances <instanceId> (instanceId := second column from ec2-describe-instances)
Logging into an Instance
Opening ports
Since you've launched your instance in the default group, ports 22 and 80 have probably already been opened in the group firewall, but if they haven't...
ec2-authorize default -p 22 ec2-authorize default -p 80
SSH'ing into your instance(s)
Finally, to connect to your box
ssh -i <privatekey_file> root@<external url>
- Note: The -i argument is the actual path to the keypair file created by ec2-add-keypair.
- The external url is the 3rd column of the INSTANCE line from ec2-describe-instances
Congratulations, you're part of the utility computing revolution!
