From RAD Lab

CS294 Fall 06 - Lab 1 - Get Something Running

The thread running through all these labs is that you are expecting your site to grow: the number of users will increase, possibly the size of your database will increase, and you must try to stay one step ahead of the growth rather than suffering the consequences of overload.

Goals of this lab:

• Drive the application to saturation (maximum workload) on a single machine. Understand how application behaves as you drive it to saturation. Deliverable: Demo of workload generation, and measurements and analysis illustrating your app's behavior as you saturate it. A typical way to graph workload is to graph response time (y-axis) vs. offered load (number of concurrent users, x-axis); acceptable behavior means that the response time (or some weighted-mean computation of it) is below a certain threshold, but for this lab you don't need to identify a specific threshold, just showing the relationship will be enough.
  
• Identify the software bottlenecks leading to saturation: where are most of the resources going as you approach saturation, and what software tuning is available? (In Lab 2 you will actually try doing this tuning.)  Deliverable: metrics and analysis used to justify your conclusions of what the bottlenecks are. Hint: use the Rails Analyzer to find the slowest URLs.

Procedure Outline

1. Get your favorite version control (CVS, RCS, Subversion, etc.) running.  Be conservative and use version control on everything, and do frequent checkins. If you mysteriously break something you can always diff or roll back to an earlier version.
   
2. Get your VM up and running, point your browser at the ResearchIndex app, and play with the app features to get familiar with it.
   
3. Configure the Workload Generator to automatically generate some workload for the app
4. Collect basic offered-workload
   
5. Get monitoring tools set up (Ganglia, Nagios, etc.) and learn how to draw pretty pictures of relevant app metrics. Part of this will involve monitoring the app and deciding which metrics are "relevant". As you do this, think about how and whether this process of metric selection could be improved or automated.
6. Use the workload generator to experiment with driving your app to saturation. Note that the app may behave differently in production mode than in development mode. Make sure the correct mode is set for Rails when you're doing measurements.
   
7. Use monitoring, application-level profiling, scanning the application logs, etc. to understand where most of the application's time is being spent. What functions, database queries, etc. dominate the resource consumption?
8. Can you split out the resource consumption by component?  Consider the three relevant tiers to be the lighttpd web server, the Rails dispatcher and application, and the database.
9. Experiment with the available datasets: the original from Spring 2006 and the fake datasets of various sizes. How does the size of the data affect the performance?

Specific questions to answer

1. When you start lighttpd web server, it launches 5 dispatcher processes (this number can be changed in the lighttpd config file, see the cheat sheet) -- it's these dispatchers that will execute the requests, not lighttpd. When lighttpd receives a request that's not a static file (image, CSS style, javascript), it will forward that request to one of the dispatchers (to the one that has the fewest number of waiting requests). The dispatcher will execute the requests assigned to it one at a time and will send the final HTML code back to lighttpd. If you use 5 dispatchers, at most 5 requests are being executed concurrently; all the other ones will be queued at the web server. It's important to realize that the response time reported by lighttpd (in the access log) includes the time the request spent in the queue. The times reported by Rails (in the production log) don't include this, they only include the execution time in Rails. Use the crawl workload generator (see the cheat sheet) to generate different levels of workload (corresponding to different numbers of concurrent users). Look at the total response time as seen from the "client's" point of view and make a rough graph of response time vs. offered load (=number of concurrent users).
2. How much of the time is spent in the Web server process (lighttpd) vs. the different parts of the application (controller action, rendering, database)? Note: Remember that there are multiple Rails dispatchers running, but all writing to the same log file. The first field of each logfile entry is the process ID of the dispatcher that wrote it. You may want to write a short sed or perl script to massage the logfile to group the lines by process ID.
   
3. Look at the RailsAnalyzer [or similar] output to compare the relative performance of different types of requests. Do some consume markedly more (or fewer) resources than others? If necessary, adjust the workload generator to focus on specific request types, and see how the offered load vs. response time graph changes as the mix of requests is biased towards the more resource-intensive ones.
4. If there are requests that seem to be particularly heavy, turn on development-mode logging in order to see the correspondence between those requests and the database queries they generate.
   
5. Advanced/optional: look at the application source to see how and why these queries are generated and whether the code could be more efficient.  The development log shows the SQL query source corresponding to each controller action; in general for controller foo you can find the controller functions in app/controllers/{foo|application}_controller.rb and the additional logic in app/models/{foo|application}.rb and app/helpers/{foo|application}_helper.rb. The view files are in app/views/foo/
6. Repeat some of the measurements using different (larger) databases. You can do this by changing the database information in database.yml.
   

Possibly Useful Readings and Resources:

• Rails Framework Documentation and Ruby Manual online
• Rails tutorials worth a skim: Rolling with Ruby on Rails and this link farm - check out especially the "screencasts" and a couple of the quick tutorials to get the flavor of RoR.
  
• More rails tutorials: Top 12 RoR tutorials
• Take a look at the cheat sheet as well.