SLA-based adaptation schemes in distributed stream processing engines

Hanif, M. and Kim, E. and Helal, S. and Lee, C. (2019) SLA-based adaptation schemes in distributed stream processing engines. Applied Sciences (Switzerland), 9 (6). ISSN 2076-3417

Full text not available from this repository.

Abstract

With the upswing in the volume of data, information online, and magnanimous cloud applications, big data analytics becomes mainstream in the research communities in the industry as well as in the scholarly world. This prompted the emergence and development of real-time distributed stream processing frameworks, such as Flink, Storm, Spark, and Samza. These frameworks endorse complex queries on streaming data to be distributed across multiple worker nodes in a cluster. Few of these stream processing frameworks provides fundamental support for controlling the latency and throughput of the system as well as the correctness of the results. However, none has the ability to handle them on the fly at runtime. We present a well-informed and efficient adaptive watermarking and dynamic buffering timeout mechanism for the distributed streaming frameworks. It is designed to increase the overall throughput of the system by making the watermarks adaptive towards the stream of incoming workload, and scale the buffering timeout dynamically for each task tracker on the fly while maintaining the Service Level Agreement (SLA)-based end-to-end latency of the system. This work focuses on tuning the parameters of the system (such as window correctness, buffering timeout, and so on) based on the prediction of incoming workloads and assesses whether a given workload will breach an SLA using output metrics including latency, throughput, and correctness of both intermediate and final results. We used Apache Flink as our testbed distributed processing engine for this work. However, the proposed mechanism can be applied to other streaming frameworks as well. Our results on the testbed model indicate that the proposed system outperforms the status quo of stream processing. With the inclusion of learning models like naïve Bayes, multilayer perceptron (MLP), and sequential minimal optimization (SMO)., the system shows more progress in terms of keeping the SLA intact as well as quality of service (QoS).

Item Type: Journal Article
Journal or Publication Title: Applied Sciences (Switzerland)
Subjects:
Departments: Faculty of Science and Technology > School of Computing & Communications
ID Code: 133197
Deposited By: ep_importer_pure
Deposited On: 03 May 2019 14:45
Refereed?: Yes
Published?: Published
Last Modified: 25 Feb 2020 08:53
URI: https://eprints.lancs.ac.uk/id/eprint/133197

Actions (login required)

View Item View Item