Noah Watkins, Carlos Maltzahn, Scott Brandt, Adam Manzanares
07/16/2012 09:21 PM
Computer Science
Cloud-based services have become an attractive alternative to in-house data centers because of their flexible, on-demand availability of compute and storage resources. This is also true for scientific high-performance computing (HPC) appli- cations that are currently being run on expensive, dedicated hardware. One important challenge of HPC applications is their need to perform periodic global checkpoints of execu- tion state to stable storage in order to recover from failures, but the checkpoint process can dominate the total run-time of HPC applications even in the failure-free case! In HPC architectures, dedicated stable storage is highly tuned for this type of workload using locality and physical layout poli- cies, which are generally unknown in typical cloud environ- ments. In this paper we introduce DataMods, an extended version of the Ceph file system and associated distributed object store RADOS, which are widely used in open source cloud stacks. DataMods extends object-based storage with extended services take advantage of common cloud data cen- ter node hardware configurations (i.e. CPU and local storage resources), and that can be used to construct efficient, scal- able middleware services that span the entire storage stack and utilize asynchronous services for offline data manage- ment services.
