RADSClassFall06/labs/cheat sheet

From RAD Lab

Jump to: navigation, search

Contents

[edit]

Request a Clustered Computing account

[edit]

Virtual Machines

  • when properly set up, each VM will have a hostname "vmN.vm" with a corresponding IP address 192.168.7.(100+N).
  • each VM is visible from each server on the X cluster and each other VM
[edit]

reservation system

  • there are other people using the cluster except us; please reserve the machines you'll be using for your labs here (the account/password is the same as for this wiki)
  • turn your VMs off when you're not using them because they could consume CPU even when idle and they take up a lot of memory. You can turn a VM on/off from the vmware-server-console (see below)
[edit]

how to clone a VM

  1. shut down the VM you want to clone and don't turn it on until you finished configuring the new VM (see below "how to shut down a VM")
  2. copy the directory with the VM to a new server (you don't want to copy a running VM)
  3. start vmware-server-console on the new server
  4. connect to localhost, click on "Open a virtual machine", click on "Browse ...", go to the directory with your new VM, click on bodikp-fc5.vmx, click "Open". The virtual machine should show up in the inventory on the left.
  5. now we need to change the IP address and hostname of the new VM: "Power on the virtual machine" and click "Create" to create a new ID for this cloned VM. The VM should boot up into a GUI login prompt.
  6. log in as root/rootroot, open System -> Administration -> Network. In the Devices tab, double-click on "eth0", and set IP address to one of the IP addresses allocated to your team (for example, 192.168.7.107). Click OK.
  7. click on the DNS tab and change the hostname to the hostname corresponding to the new IP address (for example, vm7.vm). In the File menu click on Save, OK. Restart the VM from within the OS (System, Shut down, restart).
  8. now you boot up the original VM
[edit]

disk space

  • the size of the disk for the VM is set to 8GB, that should be enough. Please don't add another disk to the VM because we might run out of disk space on the physical disk.
[edit]

starting and shutting down a VM

You can control your VM from vmware-server-console that runs on the physical host with the VM:

  1. SSH into that physical host with X11 forwarding turned on (ssh -Y x??) and start 'vmware-server-console&', or
  2. (if you're on Windows or Linux) install vmware-server-console on your machine (installers are in /work/vmware-console) and connect to the physical server with your VM

In the VMWare console you should see all the VMs installed on that server; you can start/stop a VM or change its settings.

[edit]

starting/stopping a Rails app

  • lighttpdctl start|stop|restart|reload will start/stop/restart/reload the Lighttpd web server
    • lighttpdctl start will start the web server and 5 dispatch.fcgi processes that will handle all the Rails dynamic pages
  • you'll find the app on http://localhost


[edit]

Workload Generation

  • ab
  • bench -u http://localhost/papers/show/2 -r 50 -c 2
    • bench will hit the specified URL (/papers/show/2) 50 times and will simulate the specified number of concurrent users (the -c option)
    • easy to use, but will only hit 1 URL
  • crawl -u http://localhost/ -c 2
    • crawl will crawl the website: request a URL, parse the response, pick a random link, make another request, ... and will simulate the specified number of concurrent users (the -c) option
    • will hit all available links, but won't do POST requests (eg, adding new data or searching) and can't log in
    • bench and crawl are part of Rails Analyzer Tools: http://dev.robotcoop.com/Tools/rails_analyzer_tools/index.html
  • researchindex_load
    • This is a ruby script found in the Workload Generation VM (in ~railsdev/researchindex_load. It is like crawl, but also performs searches and is slightly more powerful.
    • A few parameters can be tuned by editing the script: number of threads, number of iterations, pause after each action, etc.
    • We've (Greg Gibeling and Team3) created an Advanced ResearchIndex Load script which will even collect results from the production log.
[edit]

Workload Generation VM

A smaller VM is available and is pre-installed with all the workload tools. This can be setup by the following step:

  • Copy from /work/bodikp/RADSClassVMs/loadVM to /scratch/Virtual Machines/ on the physical machine you want the VM to reside (and rename the directory to reflect that it belongs to your team).
  • Start VMware Server Console and connect to the physical machine in question.
  • Open the VM that you just copied and when prompted, create a new ID.
  • Start the VM and login using railsdev and password railsdev.
  • Run: sudo fixmac (use railsdev as the password when prompted)
  • Edit the /etc/network/interfaces file
    • sudo nano /etc/network/interfaces
    • change eth0 to eth1 (team5 note: using eth1 didnt work for us, but eth0 did!)
    • change the address for eth1 to one of the IPs assigned to you
  • Edit the /etc/hostname file
    • sudo nano /etc/hostname
    • change the hostname to the one corresponding to the IP you chose (ie. IP of 192.168.7.190 has hostname of vm90)
  • Edit the /etc/hosts file to reflect the change in hostname
  • Restart VM from the VMWare server console (sudo reboot won't work)
[edit]

Lighttpd

  • lighttpdctl start|stop|restart|reload will start/stop/restart/reload the Lighttpd web server
    • lighttpdctl start will start the web server and 5 dispatch.fcgi processes that will handle all the Rails dynamic pages. The number of dispatchers is specifies in the lighttpd config file
  • you'll find the app on http://localhost
[edit]

config file

  • /etc/lighttpd/lighttpd.conf

The important configuration is here: 

fastcgi.server = (
  ".fcgi" => (
        "localhost" => (
           "min-procs" => 1,
           "max-procs" => 5,
           "socket" => "/tmp/fcgi.socket",
           "bin-path" => "/www/dispatch.fcgi"
        )
  )
)

This says that lighttpd should launch 5 dispatcher processes (on localhost) that will handle FastCGI requests. The path to the FCGI dispatcher is "/www/dispatch.fcgi" (/www is a link to the directory of the web site). You can change the "max-procs", restart the web server, and run your benchmark again to test the performance.

[edit]

log files

  • error log: /logs/lighttpd.error.log
  • access log: /logs/researchindex.access.log
    • the first number on each line in the access log is the execution time (in seconds) of that request in the web server
[edit]

mod_status


[edit]

Rails

[edit]

basics

  • the basic building block of a Rails application is a controller. In our case, for example, PaperController, CollectionsController, or CommentsController. Each controller contains various actions that user can invoke:
    • list: list all papers
    • show: show one papers
    • new/create: create a new paper
    • edit/update: update a paper
  • these code corresponding to these actions is in app/controllers/papers_controller.rb. The corresponding views are in app/views/papers/
  • the default URL for a particular controller/action pair is http://vm7.vm/controller/action, followed by arguments (for example, /papers/show/30)
  • when Rails receives a request to /papers/show/30, it first executes the show action in app/controllers/papers_controller.rb and then renders the HTML using the code in app/views/papers/show.rhtml. render :partial => comments will generate HTML using _comments.rhtml in the same views/papers directory.
[edit]

configuration files

  • #{RAILS_ROOT}/config/database.yml
    • to change the database used for the app, change database: researchindex_prod to any other available database
    • available databases:
      • researchindex_prod: original database from spring 2006 (tiny, about 1MB of data)
      • RI_fake_tiny: fake data (1.6MB)
      • RI_fake_small: fake data (15MB)
      • RI_fake_med: fake data (50MB)
  • #{RAILS_ROOT}/config/environment.rb
  • #{RAILS_ROOT}/config/environments/production.rb, development.rb, ...
[edit]

Rails logs

  • Rails logs can be found in #{RAILS_ROOT}/log, which in our case is something like /home/railsdev/websites/researchindex/production/log
  • production vs. development log: how to turn on/off, what information
[edit]

MySQL

  • MySQL service will start automatically after boot
  • sudo /sbin/service mysqld restart to restart MySQL (after config change)
[edit]

configuration files

  • /etc/my.cnf
[edit]

logs

  • error log: /var/log/mysqld-err.log
  • general query log: /var/log/mysqld.log
    • has to be enabled in the config file: uncomment the following line log=/var/log/mysqld.log
[edit]

SVN

A subversion repository has been setup for each team, and you may find it useful when modifying the code for the researchindex application. This will allow you to use IDEs such as Eclipse to modify the code and deploy the changes at any target VM.

The URL of the svn repository is at: svn+ssh://x[nn].millennium.berkeley.edu/project/cs/iram/a/radlab/cs294-1-f06/subversion/team[n]/

The repository is currently empty, and you need to put the research index files into it first:

  • Log into your VM
  • Edit the subversion config file (if you don't see a .subversion directory, you can skip this part, but if you encounter an error when you do svn ci later come back and do this):
    • cd ~/.subversion/
    • nano config
    • goto the line that says: editor-cmd = editor , uncomment, change it to your favorite editor.
  • cd /home/railsdev/websites/researchindex/production
  • Remove all .svn subdirectories in the production directory
    • I did this by doing: rm -rf .svn, rm -rf */.svn, rm -rf */*/.svn, rm -rf */*/*/.svn, (4 levels ought to be enough).
  • svn co svn+ssh://[username]@x[nn].millennium.berkeley.edu/project/cs/iram/a/radlab/cs294-1-f06/subversion/team[n]/ .
    • type your password when prompted.
  • Add the relevant files to SVN
    • svn add app/ config/ ...
    • DO NOT add log/ temp/ and tmp/ !!!
  • Commit the files
    • svn ci

So, now when this is done, your files will be in SVN. After you modify them and committed the changes, you can update the copy in your VM by doing: svn up.

[edit]

monitoring/alarming

  • both Ganglia and Cacti are PHP applications running on Apache on port 81
  • code in /var/www/html/ganglia and /cacti
[edit]

Ganglia

  • distributed monitoring service, running on http://localhost:81/ganglia
  • http://ganglia.sourceforge.net/
  • by executing a command like gmetric --name temperature --value `cputemp` --type int16 --units Celsius you can add your own custom metrics. Adding this command to crontab will continuously update the values and graph in Ganglia. Take a look at /usr/local/sbin/tcp_est and the following line in /etc/crontab: * * * * * root tcp_est
[edit]

Monitoring a cluster with Ganglia

When monitoring multiple machines in the same cluster, gmetad is the web frontend exchanging data with gmond on "worker" nodes. Assume vm0.vm is the frontend node, vm(1-10).vm are the nodes you want to monitor in the cluster.

On vm0.vm edit /etc/gmetad.conf:

  • Add vm(1-10).vm to data_source and trusted_hosts
  • run service gmetad restart to restart the service.

On vm(1-10).vm edit /etc/gmond.conf

  • Find section udp_send_channel change to the following (e.g. vm1.vm) 
 udp_send_channel {
  host = vm1.vm
  port = 8649
 
  host = vm0.vm
  port = 8649
 }
  • In tcp_accept_channel remove bind = 127.0.0.1 and change to the following 
 tcp_accept_channel {
  port = 8649
  acl {
  default = allow
 }
}
  • Restart gmond by service gmond restart. Once gmond is restarted you can telnet to port 8649 to see if it is configured properly.

Start ntpd by service ntpd start to synchronize time on each vm and run rdate -s time.nist.gov.

Have a look at http://vm80.vm:81/ganglia

[edit]

Cacti


[edit]

Improving performance

[edit]

benchmarking

  • Production log analyzer: here
    • installed; run agains the production log to get a nice summary of performance
  • there's also Railsbench, but we couldn't get that to work here
[edit]

useful links

  • a series of 4 articles on scaling a Rails application. The first one is here
[edit]

Tuning parameters

  • number of dispatchers: you can change the number of dispatchers that Lighttpd starts in lighttpd config file (default is 5). Is it better to run more dispatchers (each takes up some memory) or use that memory for caching?
  • MySQL caches: see below
[edit]

Caching in Rails

[edit]

Fragment caching

Caches fragments of the final HTML code. "This is useful when certain elements of an action change frequently or depend on complicated state while other parts rarely change or can be shared amongst multiple parties."

[edit]

2nd-level cache

Caching of pieces of data from the database (or Rails ActiveRecord objects)

  • http://dev.robotcoop.com/Libraries/cached_model/index.html: CachedModel stores Rails ActiveRecord objects in memcache allowing for very fast retrievals. This sounds great, but only works for simple queries (read the docs) and it's time-based (although you should be able to expire objects manually).

If you want to cache more complex queries, you're on your own.

[edit]

query cache in MySQL

Cache of individual SQL queries in MySQL. If the result of query Q depends on table T and T is changed in any way (even if not affecting result of Q), the cache entry for Q is expired. Currently our workload generation is read-only which means that this query cache might help quite a bit. We're working on a read/write workload generator.

[edit]

memcached

This is a in-memory distributed hash-table that supports get/set methods. In Rails it's used for fragment or any other type of caching and as a session store. The ResearchIndex app is already set up to use it as a session store.

Retrieved from "http://radlab.cs.berkeley.edu/wiki/RADSClassFall06/labs/cheat_sheet"
Personal tools