Designing In-Memory Distributed Systems for Billion-Row Healthcare Data Processing
Main Article Content
Abstract
Processing patient-level healthcare data at national or global scale forces distributed systems to sustain billions of rows while still meeting the latency and operational expectations of commercial analytics products. This paper presents the design and evaluation of a real-time, in-memory computing cluster engineered for complex cohort workloads over massive claims and reference pipelines—on the order of 100+ billion rows ingested through a modern extract-transform-load (ETL) stack. Where prior work often optimises a single dimension, this study foregrounds the joint balancing of three competing operational forces: memory footprint, fault tolerance, and ingest throughput. The architecture combines stream-oriented ingestion and orchestration patterns (Scala/Java, Apache Spark, Apache Kafka, PostgreSQL) with Kubernetes-based operations and a deliberately cloud-agnostic footprint, so that the same capability runs for different clients and regions without re-platforming. Using a design-science methodology, the artefact is evaluated through production telemetry and controlled fault-injection experiments. Results show that in-memory execution converts iterative join-and-window cohort logic from a disk-bound bottleneck into a memory-bound, tunable workload; that checkpoint-and-replay fault tolerance restores state within bounded recovery times under node loss; and that back-pressure-aware streaming stabilises ingest throughput under bursty load. The study quantifies the memory–fault-tolerance–throughput trade-off surface, identifies a garbage-collection-driven latency inflection, and derives capacity-planning heuristics. Contributions include a reference design for billion-row in-memory healthcare processing, an empirical characterisation of its operational envelope, and transferable engineering lessons for cloud-agnostic, multi-region deployment. Limitations include single-organisation, observational measurement, which constrains causal and external generalisation. The work advances distributed-systems practice by demonstrating how streaming orchestration and in-memory compute can meet product-grade latency at healthcare scale.
Article Details
Section
How to Cite
References
1. Brewer, E. (2012) 'CAP twelve years later: how the "rules" have changed', Computer, 45(2), pp. 23–29. Available at: https://doi.org/10.1109/MC.2012.37
2. Burns, B., Grant, B., Oppenheimer, D., Brewer, E. & Wilkes, J. (2016) 'Borg, Omega, and Kubernetes', Communications of the ACM, 59(5), pp. 50–57. Available at: https://doi.org/10.1145/2890784
3. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S. & Tzoumas, K. (2015) 'Apache Flink: stream and batch processing in a single engine', IEEE Data Engineering Bulletin, 38(4), pp. 28–38.
4. Chen, M., Mao, S. & Liu, Y. (2020) 'Big data analytics in healthcare: promise and potential', Health Information Science and Systems, 8(1), pp. 1–15. Available at: https://doi.org/10.1007/s13755-020-00120-w
5. Creswell, J.W. & Creswell, J.D. (2018) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 5th edn. Thousand Oaks: SAGE.
6. Dean, J. & Ghemawat, S. (2008) 'MapReduce: simplified data processing on large clusters', Communications of the ACM, 51(1), pp. 107–113. Available at: https://doi.org/10.1145/1327452.1327492
7. Gannon, D., Barga, R. & Sundaresan, N. (2017) 'Cloud-native applications', IEEE Cloud Computing, 4(5), pp. 16–21. Available at: https://doi.org/10.1109/MCC.2017.4250939
8. Hevner, A.R., March, S.T., Park, J. & Ram, S. (2004) 'Design science in information systems research', MIS Quarterly, 28(1), pp. 75–105. Available at: https://doi.org/10.2307/25148625
9. Hripcsak, G. et al. (2015) 'Observational Health Data Sciences and Informatics (OHDSI): opportunities for observational researchers', Studies in Health Technology and Informatics, 216, pp. 574–578.
10. Kleppmann, M. (2017) Designing Data-Intensive Applications. Sebastopol: O'Reilly Media.
11. Kreps, J., Narkhede, N. & Rao, J. (2011) 'Kafka: a distributed messaging system for log processing', Proceedings of the NetDB Workshop, pp. 1–7.
12. Marz, N. & Warren, J. (2015) Big Data: Principles and Best Practices of Scalable Real-Time Data Systems. Shelter Island: Manning Publications.
13. Ousterhout, K., Rasti, R., Ratnasamy, S., Shenker, S. & Chun, B.G. (2015) 'Making sense of performance in data analytics frameworks', Proceedings of USENIX NSDI, pp. 293–307.
14. Shvachko, K., Kuang, H., Radia, S. & Chansler, R. (2010) 'The Hadoop distributed file system', Proceedings of IEEE MSST, pp. 1–10. Available at: https://doi.org/10.1109/MSST.2010.5496972
15. Stonebraker, M. & Çetintemel, U. (2018) '"One size fits all": an idea whose time has come and gone', in Making Databases Work. New York: ACM Books, pp. 441–462.
16. Vavilapalli, V.K. et al. (2013) 'Apache Hadoop YARN: yet another resource negotiator', Proceedings of the ACM Symposium on Cloud Computing (SoCC), pp. 1–16. Available at: https://doi.org/10.1145/2523616.2523633
17. Wang, S.V., Schneeweiss, S. et al. (2022) 'Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies', Value in Health, 25(4), pp. 481–495. Available at: https://doi.org/10.1016/j.jval.2021.10.017
18. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S. & Stoica, I. (2013) 'Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing', Proceedings of USENIX NSDI, pp. 15–28.
19. Zaharia, M. et al. (2016) 'Apache Spark: a unified engine for big data processing', Communications of the ACM, 59(11), pp. 56–65. Available at: https://doi.org/10.1145/2934664
20. Akidau, T. et al. (2015) 'The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing', Proceedings of the VLDB Endowment, 8(12), pp. 1792–1803. Available at: https://doi.org/10.14778/2824032.2824076