■ ExpREsS: EneRgy Efficient Scheduling of Mixed Stream and Batch Processing Workloads

Abstract.

Nowadays we see the wide adoption of novel distributed processing frameworks such as Apache Spark for handling batch and stream processing big data applications. An important aspect that has not been examined in these systems is their energy consumption during the application execution. Reducing the power consumption of modern datacenters is a necessity as datacenters contribute over 2% of the total US electric usage. One way of addressing this energy issue is by scheduling the applications in an energy-efficient way. However, efficiently scheduling applications can be challenging as we need to consider the trade-off between the datacenter's energy usage and per application performance requirements. In this work we propose, ExpREsS, a scheduler for orchestrating the execution of Spark applications so that it both minimizes the energy consumption and satisfies the applications' performance requirements. Our approach exploits time-series prediction models for capturing the applications' energy usage and execution times, and then applies a novel DVFS technique to minimize the energy consumption. Our detailed experimental evaluation using realistic workloads on our local cluster illustrates the working and benefits of our approach.

BibTex Entry.

@INPROCEEDINGS{8005324,
author={S. Maroulis and N. Zacheilas and V. Kalogeraki},
booktitle={2017 IEEE International Conference on Autonomic Computing (ICAC)},
title={ExpREsS: EneRgy Efficient Scheduling of Mixed Stream and Batch Processing Workloads},
year={2017},
volume={},
number={},
pages={27-32},
keywords={Big Data;distributed processing;scheduling;time series;ExpREsS;energy efficient scheduling of mixed stream and batch processing;novel distributed processing frameworks;Apache Spark;novel DVFS technique;time-series prediction models;Energy consumption;Sparks;Power demand;Time-frequency analysis;Batch production systems;Radio spectrum management;Distributed Systems;Scheduling;Green Computing},
doi={10.1109/ICAC.2017.43},
ISSN={2474-0756},
month={July},}