From RAD Lab

The Manager Script is a Ruby script that allows you to deploy and manage multiple Cloudstone clusters simultaneously. It supports colocation and all of the Cloudstone Chef Recipes as of July 8, 2009.

Contents 1 Preparation 2 Instructions 3 States 4 Error Messages 5 Extending the Manager 5.1 Adding New Chef Recipes 5.2 Changing Chef Configurations 6 Customizing the Chef Repository

Preparation

The Manager assumes that your Amazon EC2 key pair is added to a running ssh-agent.

• Start up an ssh-agent: eval `ssh-agent -s`
• Add your EC2 keypair: ssh-add <path/to/EC2/keypair>

Instructions

1. Check out the Manager: git clone ssh://<user>@scm.millennium.berkeley.edu/project/cs/radlab/src/manager.git manager.git
2. Obtain the Chef repository: git clone ssh://<user>@scm.millennium.berkeley.edu/project/cs/radlab/src/chef-repo.git chef-repo.git
3. Change directory to the Ruby script: cd manager.git/ruby
4. Create a copy of the configuration file template: cp config.js.template config.js
5. Edit the Manager configuration file you just created: config.js
   • Edit the "key-pair" attribute and set if to the name of your EC2 Key Pair (i.e. the one used to obtain EC2 instances)
   • Edit the "instances" attribute to specify your Cluster Configuration. (Read the configuration file for comments that describe how to do this.)
   • If you checked out the Chef repository above, you don't need to edit "chef-repo". Otherwise, set it to the relative path containing the repository (e.g. ../../repo.tar.gz or ../../chef-repo.git).
   • Edit the "auth" section and enter the RAD Lab EC2 authentication information (AMAZON_ACCESS_KEY_ID and AMAZON_SECRET_ACCESS_KEY values)
6. Run the manager to see all possible options: ./manager
   • The following gems need to be installed for the manager to work. If you are running the manager from your own machine, make sure these gems are installed. On the R cluster you can install them (if you have sudo privileges) using sudo gem install [gem-name]
     • net-ssh (Tested with 2.0.11)
     • net-scp (Tested with 1.0.2)
     • json (Tested with 1.1.6)
     • amazon-ec2 (Tested with 0.4.5)
     • right_aws (Tested with 1.10.0)
7. To start an experiment, run the start task: ./manager start
   • If the manager exits unexpectedly, you can re-attempt deployment without allocating new machines: ./manager deploy
   • Once the manager runs successfully it will out put the URL for the Faban web interface go to the URL indicated by the "Point your browser to:" message.
8. To check that everything is running, ./manager status
9. To download logs, ./manager logs
10. To shut down the cluster and return the instances: ./manager stop

States

Each cluster is organized into States that persist across tasks. States keep track of the instances as well as their roles. The intended use of States is to permit the management of simultaneous experiments. As an example:

1. Start an experiment (using the default State): ./manager start
2. Check on the status of the first experiment: ./manager status
3. Start a second experiment (named e2): ./manager -s e2 start
4. Check on the status of the second experiment: ./manager -s e2 status
5. List the two experiments currently active: ./manager states
6. Shut down the first experiment: ./manager stop
7. Shut down the second experiment: ./manager -s e2 stop

Error Messages

• .../net/ssh/transport/session.rb:65:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
  • You probably forgot to add your EC2 keypair to the SSH Agent.
  • To fix: ssh-add <path/to/EC2/keypair>

Extending the Manager

When new Chef recipes are created or when a Chef configuration is changed, the Manager Script must be updated.

Adding New Chef Recipes

1. Create the Service subclass for the new recipe in lib/classes/service.rb
2. Add the class you created in Step 1 to Service#subclasses, which returns a list of the services in the order of deployment.
3. Update the comment in config.js.template to reflect the new service the Manager now supports.

Changing Chef Configurations

1. Look up the Service subclass that deploys the service of which the configuration has changed.
2. Modify the class's #config method.

Customizing the Chef Repository

If you want to customize the chef repository when deploying using the Manager Script, keep in mind that the chef repository used to deploy the services/roles is local. As an example, if you want to use the xtrace.js configuration for the Chukwa recipe:

1. Check out the latest chef repository.
2. Rename xtrace.js to default.js
3. Deploy using the Manager Script.