Projects/Cloudstone/Debugging Faban

From RAD Lab

Jump to: navigation, search

Faban is used to generate workload against the web application in Cloudstone. In order to debug Faban, you must deploy a Cloudstone cluster, reconfigure Faban, and then attach a debugger to the Faban instance. However, since the full trace of Faban involves multiple Java processes, attaching the debugger is a feat on its own.

Contents

Creating a Workload Driver

The Faban workload driver can be developed separately from the Faban Harness/Kit. In order to use the workload driver, your workload source will have to build a DriverName.jar file. If this file is present in faban/benchmarks/ when Faban is started (e.g. via faban/master/bin/startup.sh, it will automatically be loaded.

Instructions for building a new Faban workload can be found here.

Known Issues

  • Stale Faban Configuration Files
    • When Faban is started with a packaged workload driver, the configuration files are "put into the correct place".
    • However, when a new workload driver is "installed" (i.e. placed in faban/benchmarks/), the new configuration files will not be loaded.
    • To solve this, you must wipe out the Faban installation and install a vanilla Faban followed by the new workload driver.

Debugging the Faban Web Server

The Faban Web Server exists on the Master instance. Attaching a debugger to this part of Faban allows you to trace the web interface and what happens as a result of the actions submitted there. The Chef recipe for Faban can be configured to make Faban listen for debuggers on port 8000 (i.e. by setting debug to true). If you would like to reconfigure an existing Faban instance:

  1. Shut down Faban: /root/faban/master/bin/shutdown.sh
  2. Change the last line of /root/faban/master/bin/startup.sh to: exec "$PRGDIR"/"$EXECUTABLE" jpda start "$@"
  3. Change line 141 of /root/faban/master/bin/catalina.sh to: JPDA_OPTS="-Xdebug -Xrunjdwp:transport=$JPDA_TRANSPORT,address=$JPDA_ADDRESS,server=y,suspend=y"
  4. Start up Faban: /root/faban/master/bin/startup.sh
  5. Attach your debugger to port 8000.

Debugging a Faban Agent

When a run is scheduled using the Faban web interface, a series of calls are made that eventually lead to the execution of the Agent script in /root/faban/bin/agent. This agent script is responsible for launching the Java command responsible for the starting the Faban agent. In order to configure it to listen for debuggers on port 8001:

  1. Before scheduling a run, modify the Agent script at /root/faban/bin/agent by adding these two lines before line 87:
    • -Xdebug \
    • -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y \
  2. Schedule a run and attach your debugger to port 8001. If it fails, try again; you may have attached the debugger too soon (i.e. before the agent was started).

Faban Error Log Messages

  • Retry connecting to CmdAgent@ip-*-*-*-*, count *.
    1. Log into the machine, kill all Faban instances: pkill java
    2. Make sure JAVA_HOME is set correctly: export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
    3. Restart Faban agent: faban/bin/agent