A Common Substrate for Cluster Computing

TitleA Common Substrate for Cluster Computing
Publication TypeConference Paper
Year of Publication2009
AuthorsHindman, Benjamin, Andy Konwinski, Matei Zaharia, & Ion Stoica
Conference NameProceedings of USENIX HotCloud'09: Workshop on Hot Topics in Cloud Computing
Date Published06/2009
Conference LocationSan Diego, CA

The success of MapReduce has sparked many efforts to design cluster computing frameworks. We argue that no single framework will be optimal for all applications, and that we should instead enable organizations to run multiple frameworks efficiently in the same cloud. Furthermore, to ease development of new frameworks, it is critical to identify common abstractions and modularize their architectures. To achieve these goals, we propose Nexus, a low-level substrate that provides isolation and efficient resource sharing across frameworks running on the same cluster, while giving each framework freedom to implement its own programming model and fully control the execution of its jobs. Nexus fosters innovation in the cloud by letting organizations run new frameworks alongside existing ones and by letting framework developers focus on specific applications rather than building onesize-fits-all frameworks.