StarTrace/Instrumentation Backplane(2)

From RAD Lab

Filter-Map-Reduce... 'LogReduce?'

Ideals: Simple (... possibly violated)
Minimally intrusive
Arbitrary queries

Weaknesses: Doesn't handle path based information yet - Just want to kick the ball around.

Basic Idea: Distributing scripts to providers

With Filtering

Infrastructure

Following the google model, we have a single master controller. This controller keeps track of the network location of all the providers on the backplane, plus whatever extra information is required. When a query is presented to the backplane it is presented in the format of three separate scripts; filter, map, & reduce.

Map and reduce function like expected, except map blocks for additional input and may not terminate. The map task resides on the provider and performs (possibly) blocking operations, generally reads on a log file. Once activated, the map task processes the input and may emit a value, which is received by the reduce task on the client. The filter script is an optional component which limits the number of providers the map tasks executes on to those satisfying some condition, like those having the appropriate scripting language or log file. The filter task is essentially a map-reduce task of a specific form, namely it filters some of the entries from the IBP Coordinators table of BackPlane Providers. It is important to note here that the 'mapping' we are performing here is unlike Google's MapReduce in that we are applying a particular computation onto a number of machines, not distributing computation on a set of files OVER a set of machines.

Conceptual execution path
1. An IBP Task is received.
2. The BackPlane Provider (BPP) Table is retrieved from the IBP Coordinator
3. The filter script is distributed to all providers listed in the (now local) BPP table and executed there.
4. Providers passing the filter return positively, those not passing either return negatively or fail to return
5. The map task is distributed to all the Providers listed in the filtered BPP table and run.
6. Running map tasks periodically emit values until self-termination, or IBP Task terminated.
7. Reduce script running on client receives values, aggregates and periodically outputs results

In the simplest implementation, communication between the consumers and provider could occur over SSH connections, started and terminated by the client machine requiring no provider configuration changes if an ssh daemon is running. For security purposes, some sort of access limits should probably be set account on the IBP communicates over.

Basically, I was trying to think around the hesitancy a lot of people expressed, especially at the second breakout, about allowing configuration changes to production servers. We could even have log format normalization scripts written and stored on the client side and loaded as needed onto the producer. I don't know if the additional network traffic and/or logistical difficulties are worth it, but I wanted to get some feedback.

Goals: Simple - Duck Typing - Instead of maintaining a lot of state about the providers, i.e. what services they provide, just try them. Run a program on each that determines presence or absence of relevant capabilities for each query.

Minimally Intrusive - "provider" could be as little as a chrooted user with permissions that only allow reading symlinks to particular log files. Still too intrusive?

Arbitrary queries - Allows arbitrary questions to be posed by consumers in the form of arbitrary scripts. (Limited by each machines security policy)

Weaknesses/Questions/Thoughts
Doesn't address path based tracing - how do we integrate this? Efficient implementation possible in this configuration?
Filtering unnecessary?
Same filter could be running many times on single machine.
Same log parsing routine could be running many times on single machine.
\__ General problem, map tasks can't share resources. Addition of daemon helps alleviate...
Won't detect providers who satisfy filter after filter has run.
\__ Unless map task specifically initiates
Viable for highly promiscuous filters?
Deleterious effect of runaway filter/map scripts?
\__ Monitoring daemon could be installed, requires additional software
Added value of resident software on provider outweighs modifications required?

JesseTrutna - 1/16/07 jtrutna@gmail.com