Designing In-Memory Distributed Systems for Billion-Row Healthcare Data Processing

Abstract

Processing patient-level healthcare data at national or global scale forces distributed systems to sustain billions of rows while still meeting the latency and operational expectations of commercial analytics products. This paper presents the design and evaluation of a real-time, in-memory computing cluster engineered for complex cohort workloads over massive claims and reference pipelines—on the order of 100+ billion rows ingested through a modern extract-transform-load (ETL) stack. Where prior work often optimises a single dimension, this study foregrounds the joint balancing of three competing operational forces: memory footprint, fault tolerance, and ingest throughput. The architecture combines stream-oriented ingestion and orchestration patterns (Scala/Java, Apache Spark, Apache Kafka, PostgreSQL) with Kubernetes-based operations and a deliberately cloud-agnostic footprint, so that the same capability runs for different clients and regions without re-platforming. Using a design-science methodology, the artefact is evaluated through production telemetry and controlled fault-injection experiments. Results show that in-memory execution converts iterative join-and-window cohort logic from a disk-bound bottleneck into a memory-bound, tunable workload; that checkpoint-and-replay fault tolerance restores state within bounded recovery times under node loss; and that back-pressure-aware streaming stabilises ingest throughput under bursty load. The study quantifies the memory–fault-tolerance–throughput trade-off surface, identifies a garbage-collection-driven latency inflection, and derives capacity-planning heuristics. Contributions include a reference design for billion-row in-memory healthcare processing, an empirical characterisation of its operational envelope, and transferable engineering lessons for cloud-agnostic, multi-region deployment. Limitations include single-organisation, observational measurement, which constrains causal and external generalisation. The work advances distributed-systems practice by demonstrating how streaming orchestration and in-memory compute can meet product-grade latency at healthcare scale.