Projects/Cloudstone/Deployment Library

From RAD Lab

Jump to: navigation, search

The Deployment Library is a Scala (Java-compatible) Library that allows you to manage instances and services in general deployments. The original target deployments for the Deployment Library are Cloudstone and SCADS. The library relies on the use of Chef to deploy and manage services.

Contents

[edit]

Setup

[edit]

Pre-requirements

  • The source code:
    git clone ssh://username@scm.millennium.berkeley.edu/project/cs/radlab/src/deploy_library.git
  • Maven, version 2 or greater. Check that you have it installed and which version with mvn --version.
    If you need to install it, the package is called maven2 in both aptitude and MacPorts.
  • Scala - optional. I don't believe it is necessary to have Scala installed to just compile the source and use it with java code, but it is probably necessary if you would like to play around with the library with the scala interpreter.
[edit]

Install/Use

There are several ways to use the library.

  • Experimenting in the scala interpreter:
    mvn scala:console: Starts the Scala interpreter with all the Deployment Library and its dependencies in the classpath, so you can play with it interactively.
    Just type in
    import deploylib._
    and you are good to go.
  • Running a scala script that depends on the library:
    mvn scala:script -DscriptFile=pathto/script.scala
  • Using the library in a different java or scala project:
    • mvn package: Compiles and packages the code and all dependencies into one jar, target/deploy-1.0-SNAPSHOT.jar.
      Now code that uses any piece of the Deployment Library can do so as long as that jar file is in the classpath during compile and run time.
    • mvn install: This does the same as above, but puts the jar in your local maven directory, ~/.m2, so that other projects that use maven can easily include the Deployment Library.
  • Writing code that you want included with the deploylib project:
    Put your code in src/main/<language>/<package> where <language> is likely scala or java and <package> is recommended to match the package you specify in your java or scala code. The <package> directory is optional, but recommended as it keeps things more organized. Maven allows arbitrary directory structure after src/main/<language>.
    With your code organized in this way mvn package and mvn install will work as described above and will include your code. In addition, you might find the following useful:
    • mvn compile: Compiles everything in the src directory and places the resulting class files in target/classes. This is useful to check for compile time errors.
    • mvn clean: Deletes the target directory. Sometimes mvn won't recompile everything it should if it has already done a compilation, doing a clean will fix this.
[edit]

Documentation

Scala Doc
Click on the deploylib package link to get a summary of the different classes and objects.

[edit]

Example Cloudstone Deployment Using Scala 

 /**
  * An example use of the deployment library to deploy Cloudstone
  */

 package ScalaCloudstoneExample

 import deploylib._ /* Imports all files in the deployment library */
 import org.json.JSONObject
 import org.json.JSONArray
 import scala.collection.jcl.Conversions._

 object Cloudstone {
   def start = {
     /**
      * The stack will have the following defaults for the other roles:
      * 1 MySQL server on an c1.xlarge instance
      * 1 HAProxy server on a m1.small instance
      * 1 nginx server on a m1.small instance
      * 1 Faban master/driver server on a c1.xlarge
      */


     val railsSettings   = (2, "c1.xlarge", InstanceType.cores("c1.xlarge") * 2)

     val mysqlSettings   = (1, "c1.xlarge")
     val haproxySettings = (1, "m1.small")
     val nginxSettings   = (1, "m1.small")
     val fabanSettings   = (1, "c1.xlarge")


     /* A shortcut method for DataCenter.runInstances provided in API */
     def runInstances(count: Int, typeString: String): InstanceGroup = {
       val imageId = InstanceType.bits(typeString) match {
         case "32-bit" => "ami-e7a2448e"
         case "64-bit" => "ami-e4a2448d"
       }
       val keyName = "abeitch"
       val keyPath = "/Users/aaron/.ec2/id_rsa-abeitch"
       val location = "us-east-1a"
     
       DataCenter.runInstances(imageId, count, keyName, keyPath, typeString, location)
     }


     val rails   = runInstances(railsSettings._1, railsSettings._2)
     val mysql   = runInstances(mysqlSettings._1, mysqlSettings._2)
     val haproxy = runInstances(haproxySettings._1, haproxySettings._2)
     val nginx   = runInstances(nginxSettings._1, nginxSettings._2)
     val faban   = runInstances(fabanSettings._1, fabanSettings._2)
   
     val allInstances = new InstanceGroup()
     allInstances ++ rails ++ mysql ++ haproxy ++ nginx ++ faban
   
     println("Waiting until all instances are ready.")
     allInstances.parallelMap((instance: Instance) => instance.waitUntilReady)
   
     println("Building configurations")
     val mysqlPort = 3306
     /* Rails Configuration */
     val railsConfig = new JSONObject()
     railsConfig.put("recipes", new JSONArray().put("cloudstone::rails"))
     val railsRails = new JSONObject()
   
     val railsRailsPorts = new JSONObject()
     railsRailsPorts.put("start", 3000)
     railsRailsPorts.put("count", railsSettings._3)
     railsRails.put("ports", railsRailsPorts)
   
     val railsRailsDatabase = new JSONObject()
     railsRailsDatabase.put("host", mysql.getFirst().privateDnsName)
     railsRailsDatabase.put("adapter", "mysql")
     railsRailsDatabase.put("port", mysqlPort)
     railsRails.put("database", railsRailsDatabase)
   
     val railsRailsMemcached = new JSONObject()
     railsRailsMemcached.put("host", "localhost")
     railsRailsMemcached.put("port", 1211)
     railsRails.put("memcached", railsRailsMemcached)
   
     val railsRailsGeocoder = new JSONObject()
     railsRailsGeocoder.put("host", faban.getFirst().privateDnsName)
     railsRailsGeocoder.put("port", 9980)
     railsRails.put("geocoder", railsRailsGeocoder)
   
     railsConfig.put("rails", railsRails)
   
     /* mysql configuration */
     val mysqlConfig = new JSONObject()
     val mysqlRecipes = new JSONArray()
     mysqlRecipes.put("cloudstone::mysql")
     mysqlRecipes.put("cloudstone::faban-agent")
     mysqlConfig.put("recipes", mysqlRecipes)
     val mysqlMysql = new JSONObject()
   
     mysqlMysql.put("server_id", 1)
     mysqlMysql.put("port", mysqlPort)
     mysqlConfig.put("mysql", mysqlMysql)
   
     val mysqlFabanAgent = new JSONObject()
     mysqlFabanAgent.put("jdbc", "mysql")
     mysqlConfig.put("faban", mysqlFabanAgent)
   
     /* haproxy configuration */
     val haproxyConfig = new JSONObject()
     haproxyConfig.put("recipes", new JSONArray().put("cloudstone::haproxy"))
     val haproxyHaproxy = new JSONObject()
     haproxyHaproxy.put("port", 4000)
   
     val haproxyHaproxyServers = new JSONObject()
     rails.foreach(instance => {
       val server = new JSONObject()
       server.put("start", 3000)
       server.put("count", railsSettings._3)
       haproxyHaproxyServers.put(instance.privateDnsName, server)
     })
     haproxyHaproxy.put("servers", haproxyHaproxyServers)
   
     haproxyConfig.put("haproxy", haproxyHaproxy)
   
     /* nginx configuration */
     val nginxConfig = new JSONObject()
     val nginxRecipes = new JSONArray()
     nginxRecipes.put("cloudstone::nginx")
     nginxRecipes.put("cloudstone::faban-agent")
     nginxConfig.put("recipes", nginxRecipes)
     val nginxNginx = new JSONObject()
     val nginxNginxServers = new JSONObject()
   
     haproxy.foreach(instance => {
       val server = new JSONObject()
       server.put("start", 4000)
       server.put("count", 1)
       nginxNginxServers.put(instance.privateDnsName, server)
     })
     nginxNginx.put("servers", nginxNginxServers)
     nginxConfig.put("nginx", nginxNginx)
   
     val nginxFaban = new JSONObject()
     nginxFaban.put("mysql", false)
     nginxFaban.put("postgresql", false)
     nginxConfig.put("faban", nginxFaban)
   
     /* faban configuration */
     val fabanConfig = new JSONObject()
     fabanConfig.put("recipes", new JSONArray().put("cloudstone::faban"))
     val fabanFaban = new JSONObject()
     val fabanFabanHosts = new JSONObject()
     fabanFabanHosts.put("driver", faban.getFirst().privateDnsName)
     fabanFabanHosts.put("webserver", nginx.getFirst().privateDnsName)
     fabanFabanHosts.put("database", mysql.getFirst().privateDnsName)
     fabanFabanHosts.put("storage", "")
     fabanFabanHosts.put("cache", "")
   
     val fabanFabanDatabase = new JSONObject()
     fabanFabanDatabase.put("adapter", "mysql")
     fabanFabanDatabase.put("port", mysqlPort)
   
     fabanFaban.put("hosts", fabanFabanHosts)
     fabanFaban.put("database", fabanFabanDatabase)
     fabanFaban.put("debug", false)
     fabanConfig.put("faban", fabanFaban)
   
     def deployMaker(jsonConfig: JSONObject): (Instance) => Unit = {
       (instance: Instance) => {
         instance.deploy(jsonConfig)
       }
     }
   
     println("Deploying mysql.")
     mysql.parallelMap(deployMaker(mysqlConfig))
     println("Deploying rails.")
     rails.parallelMap(deployMaker(railsConfig))
     println("Deploying haproxy.")
     haproxy.parallelMap(deployMaker(haproxyConfig))
     println("Deploying nginx.")
     nginx.parallelMap(deployMaker(nginxConfig))
     println("Deploying faban.")
     faban.parallelMap(deployMaker(fabanConfig))
   
     def startAllServices(instance: Instance): Unit = {
       def startService(service: Service): Unit = {service.start}
       instance.getAllServices.foreach(startService)
     }
   
     //allInstances.parallelMap(startAllServices)
     println("All done")
   }
 }

[edit]

Example Cloudstone Deployment Using Java 

package JavaCloudstoneExample;

import deploylib.*;
import org.json.JSONObject;
import org.json.JSONArray;
import scala.*;

public class Cloudstone {
    public void run() throws org.json.JSONException {
        /**
         * This is a simple example where you pass on the command line
         * the number and size of rails servers you would like in the 
         * following form:
         *
         * scala cloudstone --count 2 --type c1.xlarge
         *
         * The stack will have the following defaults for the other roles:
         * 1 MySQL server on an c1.xlarge instance
         * 1 HAProxy server on a m1.small instance
         * 1 nginx server on a m1.small instance
         * 1 Faban master/driver server on a c1.xlarge
         */
        
        Object[] railsSettings   = {1, "m1.small", InstanceType.cores("m1.small") * 2};
        
        Object[] mysqlSettings   = {1, "c1.xlarge"};
        Object[] haproxySettings = {1, "m1.small"};
        Object[] nginxSettings   = {1, "m1.small"};
        Object[] fabanSettings   = {1, "c1.xlarge"};
        
        
        InstanceGroup rails = runInstances((Integer)railsSettings[0],
            (String)railsSettings[1]);
                                           
        InstanceGroup mysql = runInstances((Integer)mysqlSettings[0],
            (String)mysqlSettings[1]);
        InstanceGroup haproxy = runInstances((Integer)haproxySettings[0],
            (String)haproxySettings[1]);
        InstanceGroup nginx = runInstances((Integer)nginxSettings[0],
            (String)nginxSettings[1]);
        InstanceGroup faban = runInstances((Integer)fabanSettings[0],
            (String)fabanSettings[1]);
            
        InstanceGroup allInstances = new InstanceGroup();
        allInstances.addAll(rails);
        allInstances.addAll(mysql);
        allInstances.addAll(haproxy);
        allInstances.addAll(nginx);
        allInstances.addAll(faban);
        
        
        /**************************************************
         * Don't forget to wait on the instances          *
         **************************************************/
        System.out.println("Waiting on instances.");
        allInstances.waitUntilReady();
        System.out.println("Instances ready.");

        /* Rails Configuration */
        JSONObject railsConfig = new JSONObject();
        railsConfig.put("recipes", new JSONArray().put("cloudstone::rails"));
        JSONObject railsRails = new JSONObject();

        JSONObject railsRailsPorts = new JSONObject();
        railsRailsPorts.put("start", 3000);
        railsRailsPorts.put("count", (Integer)railsSettings[2]);
        railsRails.put("ports", railsRailsPorts);

        JSONObject railsRailsDatabase = new JSONObject();
        railsRailsDatabase.put("host", mysql.getFirst().privateDnsName());
        railsRailsDatabase.put("adapter", "mysql");
        railsRailsDatabase.put("port", 3306);
        railsRails.put("database", railsRailsDatabase);

        JSONObject railsRailsMemcached = new JSONObject();
        railsRailsMemcached.put("host", "localhost");
        railsRailsMemcached.put("port", 1211);
        railsRails.put("memcached", railsRailsMemcached);

        JSONObject railsRailsGeocoder = new JSONObject();
        railsRailsGeocoder.put("host", faban.getFirst().privateDnsName());
        railsRailsGeocoder.put("port", 9980);
        railsRails.put("geocoder", railsRailsGeocoder);

        railsConfig.put("rails", railsRails);

        /* mysql configuration */
        JSONObject mysqlConfig = new JSONObject();
        JSONArray mysqlRecipes = new JSONArray();
        mysqlRecipes.put("cloudstone::mysql");
        mysqlRecipes.put("cloudstone::faban-agent");
        mysqlConfig.put("recipes", mysqlRecipes);
        JSONObject mysqlMysql = new JSONObject();

        mysqlMysql.put("server_id", 1);
        mysqlConfig.put("mysql", mysqlMysql);

        JSONObject mysqlFaban = new JSONObject();
        mysqlFaban.put("jdbc", "mysql");
        mysqlConfig.put("faban", mysqlFaban);

        /* haproxy configuration */
        JSONObject haproxyConfig = new JSONObject();
        haproxyConfig.put("recipes", new JSONArray().put("cloudstone::haproxy"));
        JSONObject haproxyHaproxy = new JSONObject();
        haproxyHaproxy.put("port", 4000);

        JSONObject haproxyHaproxyServers = new JSONObject();
        for (Instance instance : rails) {
            JSONObject server = new JSONObject();
            server.put("start", 3000);
            server.put("count", (Integer)railsSettings[2]);
            haproxyHaproxyServers.put(instance.privateDnsName(), server);
        }
        haproxyHaproxy.put("servers", haproxyHaproxyServers);

        haproxyConfig.put("haproxy", haproxyHaproxy);

        /* nginx configuration */
        JSONObject nginxConfig = new JSONObject();
        JSONArray nginxRecipes = new JSONArray();
        nginxRecipes.put("cloudstone::nginx");
        nginxRecipes.put("cloudstone::faban-agent");
        nginxConfig.put("recipes", nginxRecipes);
        JSONObject nginxNginx = new JSONObject();
        JSONObject nginxNginxServers = new JSONObject();

        for (Instance instance : haproxy) {
            JSONObject server = new JSONObject();
            server.put("start", 4000);
            server.put("count", 1);
            nginxNginxServers.put(instance.privateDnsName(), server);
        }
        nginxNginx.put("servers", nginxNginxServers);
        nginxConfig.put("nginx", nginxNginx);

        // JSONObject nginxFaban = new JSONObject();
        // nginxFaban.put("mysql", false);
        // nginxConfig.put("faban", nginxFaban);

        /* faban configuration */
        JSONObject fabanConfig = new JSONObject();
        fabanConfig.put("recipes", new JSONArray().put("cloudstone::faban"));
        JSONObject fabanFaban = new JSONObject();
        JSONObject fabanFabanHosts = new JSONObject();
        fabanFabanHosts.put("driver", faban.getFirst().privateDnsName());
        fabanFabanHosts.put("webserver", nginx.getFirst().privateDnsName());
        fabanFabanHosts.put("database", mysql.getFirst().privateDnsName());
        fabanFabanHosts.put("storage", "");
        fabanFabanHosts.put("cache", "");

        fabanFaban.put("hosts", fabanFabanHosts);
 
        JSONObject fabanFabanDatabase = new JSONObject();
        fabanFabanDatabase.put("adapter", "mysql");
        fabanFabanDatabase.put("port", 3306);
        fabanFaban.put("database", fabanFabanDatabase);
        
        fabanConfig.put("faban", fabanFaban);

        System.out.println("Deploying mysql.");
        mysql.parallelMap(new Deploy(mysqlConfig));
        System.out.println("Deploying rails.");
        rails.parallelMap(new Deploy(railsConfig));
        System.out.println("Deploying haproxy.");
        haproxy.parallelMap(new Deploy(haproxyConfig));
        System.out.println("Deploying nginx.");
        nginx.parallelMap(new Deploy(nginxConfig));
        System.out.println("Deploying faban.");
        faban.parallelMap(new Deploy(fabanConfig));
        
        System.out.println("All done.");
        
    }
    
    private InstanceGroup runInstances(int count, String typeString) {
        return DataCenter.runInstances(count, typeString);
    }
    
    private class Deploy implements InstanceExecute {
        private JSONObject config;
        
        public Deploy(JSONObject config) {
            this.config = config;
        }
        
        public Object execute(Instance instance) {
            instance.deploy(config);
            return null;
        }
    }
}

Retrieved from "http://radlab.cs.berkeley.edu/wiki/Projects/Cloudstone/Deployment_Library"
Personal tools