From RAD Lab

Lab 2: Find and fix bottlenecks

The approach to this lab should be roughly:

1. Pick a defensible workload mix that includes both expensive and inexpensive operations <li> Using one or more of the workload generators playing back that worklaod, identify the point beyond which your system starts to saturate.
   One way to do this would be to define a threshold response latency and consider anything longer to be unacceptable (this is what most sites do in practice). Another would be to look for any evidence of queueing behind the application servers, such as queues waiting for a Dispatcher to become available. (Recall that in steady state, if the arrival rate exceeds the service rate, the queues will grow without bound. As a colleague used to say, "and then pretty soon you'll have an infinitely long queue.")
   <li> Try various of the scaling techniques we discussed on Monday (the slides are linked to class calendar) and measure the improvement of each; the goal is to create some headroom, i.e. to be able to accommodate a higher volume of workload while staying below your threshold of performance. <li> Report on the effects of each technique and the total improvement you were able to achieve---i.e., what was your maximum workload originally (given your chosen threshold of response time) and how much more workload were you eventually able to handle while staying below that same threshold. Which techniques were most/least effective? Which required tuning of parameters and how did you go about it? (e.g. amount of memory allocated for a particular type of caching, default lifetime of cached objects, number of dispatchers, etc.) </ol> Techniques we talked about include but are not limited to:
   <li> Caching -- at various levels and using various techniques <li> Prefetching -- see note below <li> Changing source code to improve query behavior and/or eliminate extra computation where possible <li> Utilization: making sure the 3 stage "pipeline" (Web server, dispatchers, DB) is "balanced". In other words, you don't want to be leaving performance on the table because (eg) you can't keep the database busy or because you're spending a lot of time waiting for I/O without overlapping computation, etc.

   
   

   [edit]

   Useful readings/activities/etc:

   <li> Adventures in Scaling <li> log in as GoogleGuest and benchmark some of the pages related to the networking class (lot of comments, deadlines, ...) <li>Be a hero: If you develop useful scripts or interesting techniques or hints, add them to the Lab 2 hints page (in lieu of a blog). In your lab writeups, if you used someone else's tools or hints, give them credit; you won't lose points for this. Remember you're not in competition here.

   
   

   [edit]

   Some things to keep in mind

   <li>Rails production mode disables a lot of logging and other stuff that is turned on in development mode to help you debug. Do your benchmarking in production mode but use development or debugging logs to get information about what to try. The file environment.rb contains information and settings relating to the different modes; you may want to look through it as you might look through an httpd.conf file. <li> The largest DB tables in the large fake DB total less than about 50MB. THis means in principle all tables can fit in memory. Of course, since the number of possible queries and query results is much larger, you shouldn't expect that you'll never go to disk or recompute a fragment, but it would be interesting to see how close to that ideal you can get, and what prevents you from going further. <li> If in the process of doing focused load testing you want to focus on a small number of specific URL's, check out the benchmark facility of Rails.

   
   

   BIG INEFFICIENT QUERIES AND WHAT TO DO ABOUT THEM: This came up in class at the very end and I didn't have time to discuss it. Recall that in Rails you can talk about one object "belonging to" another or "having many of" another. eg, any Order belngs to exactly one Customer; every Customer can have zero or more Orders; would be expressed (in the model files, Customer.rb and Order.rb) as:

    class Customer <<ActiveRecord::Base
    has_many :orders
   
    class Order <<ActiveRecord::Base
    belongs_to :customer
   

   Remember that you don't HAVE to do this in Rails, but it gives you the abiltiy to refer (eg) to:

   c = Customer.find(id=500)
   order_array = c.orders
   ...dereference order_array[] elements
   

   Cool. but unfortunately, as you may have realized, the first find() operation does a query on customers, but each dereference of the order_array potentially produces another query to the Orders table. if there's a lot of orders per customer, this is wasteful if you have a loop that's iterating over order_array[] and you end up with a query inside the loop.

   Used with caution, the :include keyword (an optional parameter to the Find method) can help...basically, you can hint Rails to do some prefetching of orders at the time you do a find on Customers (because a customer has_many orders). the relationship has to be defined in the direction of the prefetch for this to work, ie prefetching orders when finding a customer requires that customer has_many orders, not just order belongs_to customer (because in the latter case, the relationship info is contained in the order class but not the customer class). in this case, future dereferences of order_array[] don't result in separate queries.

   The usual caveats apply of course: if the orders table changes under you during the actual loop, or if you plan to hold these prefeteched values for a long time, you're on your own.

   This also works with many-to-many relationships (as in has_and_belongs_to_many). if you use it judiciously, it may be able to save a lot of extra queries. if used recklessly, it can cause needless prefetching (queries that could have been avoided).

   Read more about it in the Rails manual online at rubyonrails.org.