Scalable Pipeline Orchestration Through Intelligent Dependency-Aware Recovery and Performance Optimization
Main Article Content
Abstract
The new data and machine learning pipelines are increasingly being deployed in distributed and heterogeneous environments with limited resources, and where task dependencies and dynamical load and transient failures can become a critical factor in run-time performance and resilience. This paper will introduce a Scalable Pipeline Orchestration Framework including Intelligent Dependency-Aware Recovery and Performance Optimization which is a design that incorporates dependency analysis, runtime monitoring, failure prediction, adaptive recovery, and resource-aware scheduling as a single orchestration architecture. The framework describes pipeline tasks as a dynamically increasing dependency graph and maintains an ever-refined assessment on the state of tasks, resource usage, the latency of communicating the execution state and the failure state indicator. The recovery scheme can be based on dependency which allows the determination of the scope of the failure and selective re-execution of tasks which are affected and without necessarily re-executing irrelevant parts. To address the workload conditions and the urgency of the pipeline, a smart optimization layer dynamically adjusts the priority and resource assignment to the decisions of workload and execution parallelism as well as workload scheduling. The frame work also includes on historical information on the execution with the aim of enhancing on future recovery plans and reduce future prediction of failures. The key performance indicators that may be applied to design an evaluation are execution time, recovery latency, resource utilization, throughput and failure recovery rate and scalability. The objective of the proposed plan will be to decrease the overhead of recoveries and introduce uniformity in the pipeline and better performance of the work with addition in the work load and performance breakdown in case of failure. The framework offers a viable basis on how to orchestrate complex data-intensive workflows in resilient, scalable and adaptive ways.
Article Details
Section
How to Cite
References
[1] S. R. Mirampalli, R. Wankar, and S. N. Srirama, “Evaluating NiFi and MQTT based serverless data pipelines in fog computing environments,” Future Generation Computer Systems, vol. 150, pp. 341–353, 2024, doi: 10.1016/j.future.2023.09.014.
[2] M. Ghorbian, M. Ghobaei-Arani, and L. Esmaeili, “A survey on the scheduling mechanisms in serverless computing: A taxonomy, challenges, and trends,” Cluster Computing, 2024, doi: 10.1007/s10586-023-04264-8.
[3] Z. Shojaee Rad, M. Ghobaei-Arani, and R. Ahsan, “Memory orchestration mechanisms in serverless computing: A taxonomy, review and future directions,” Cluster Computing, 2024, doi: 10.1007/s10586-023-04251-z.
[4] A. Ebrahimi, M. Ghobaei-Arani, and H. Saboohi, “Cold start latency mitigation mechanisms in serverless computing: Taxonomy, review, and future directions,” Journal of Systems Architecture, vol. 151, Art. no. 103115, 2024, doi: 10.1016/j.sysarc.2024.103115.
[5] M. Ghorbian, M. Ghobaei-Arani, and M. Asadolahpour-Karimi, “Function placement approaches in serverless computing: A survey,” Journal of Systems Architecture, vol. 157, Art. no. 103291, 2024.
[6] M. Tari, M. Ghobaei-Arani, M. Pouramini, M. Ghorbian, et al., “Auto-scaling mechanisms in serverless computing: A comprehensive review,” Computer Science Review, Art. no. 100650, 2024, doi: 10.1016/j.cosrev.2024.100650.
[7] C. K. Dehury, P. Jakovits, S. N. Srirama, G. Giotis, and G. Garg, “TOSCAdata: Modeling data pipeline applications in TOSCA,” Journal of Systems and Software, vol. 186, Art. no. 111164, 2022, doi: 10.1016/j.jss.2021.111164.
[8] S. R. Poojara, C. K. Dehury, P. Jakovits, and S. N. Srirama, “Serverless data pipeline approaches for IoT data in fog and cloud computing,” Future Generation Computer Systems, vol. 130, pp. 91–105, 2022, doi: 10.1016/j.future.2021.12.012.
[9] T. Rausch, A. Rashed, and S. Dustdar, “Optimized container scheduling for data-intensive serverless edge computing,” Future Generation Computer Systems, vol. 114, pp. 259–271, 2021, doi: 10.1016/j.future.2020.07.017.
[10] S. Eismann, J. Grohmann, E. Van Eyk, N. Herbst, and S. Kounev, “Predicting the costs of serverless workflows,” in Proc. ACM/SPEC Int. Conf. Performance Engineering, 2020, pp. 265–276.
[11] J. Enes, R. R. Expósito, and J. Touriño, “Real-time resource scaling platform for big data workloads on serverless environments,” Future Generation Computer Systems, vol. 105, pp. 361–379, 2020, doi: 10.1016/j.future.2019.11.037.
[12] A. Pogiatzis and G. Samakovitis, “An event-driven serverless ETL pipeline on AWS,” Applied Sciences, vol. 11, no. 1, Art. no. 191, 2021, doi: 10.3390/app11010191.