Tracing Across the Seam: OpenTelemetry-Based Observability for Hybrid Estates Where Message-Oriented Middleware and Kubernetes Microservices Coexist
Main Article Content
Abstract
Enterprises that modernize payment and settlement platforms seldom do so in one step. For years during a strangler-style migration, a single transaction crosses IBM MQ queue managers, broker flows, and application-server consumers as well as containerized microservices on Kubernetes. Distributed tracing covers the microservices, but trace context is usually lost at the first queue, so no team sees the full path and incident triage begins with an argument about ownership. We present a telemetry architecture that closes this seam with standards-based mechanisms. W3C Trace Context is propagated through MQ message properties (the MQRFH2 usr folder, exposed as JMS user properties); channel hops are reconstructed from queue-manager activity trace; broker spans are derived from existing monitoring events; property-unaware legacy programs are bridged through a MsgId-keyed context registry; and batch consumers use span links with per-message process spans. An OpenTelemetry Collector topology correlates middleware logs, statistics, and events with spans and applies tail sampling tuned to queue dwell time. On an illustrative eight-hop payment-style workflow in a controlled environment, end-to-end trace completeness rose from 0% to 84.9% (91.3% excluding derived channel spans), and the mean of per-category median time to identify fell from 61.5 to 18.8 minutes across six incident categories. The added cost was 12.7 ms at p99, 130 to 180 bytes per message, and 9.6 GB per day of span storage. We also identify a structural conflict between tail-sampling decision windows and asynchronous dwell time, and we offer practitioner guidance for multi-year hybrid estates.
Article Details
Section
How to Cite
References
[1] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, "Borg, Omega, and Kubernetes," ACM Queue, vol. 14, no. 1, pp. 70–93, 2016.
[2] M. Fowler, "StranglerFigApplication," martinfowler.com, Jun. 2004. [Online]. Available: https://martinfowler.com/bliki/StranglerFigApplication.html
[3] S. Newman, Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith. Sebastopol, CA, USA: O'Reilly Media, 2019.
[4] B. H. Sigelman et al., "Dapper, a large-scale distributed systems tracing infrastructure," Google, Inc., Tech. Rep. dapper-2010-1, Apr. 2010.
[5] R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica, "X-Trace: A pervasive network tracing framework," in Proc. 4th USENIX Symp. Networked Systems Design and Implementation (NSDI), 2007, pp. 271–284.
[6] OpenTelemetry Authors, "OpenTelemetry Specification: Context, Propagators API, and Tracing API," Cloud Native Computing Foundation, 2023. [Online]. Available: https://opentelemetry.io/docs/specs/otel/
[7] S. Kanzhelev, M. McLean, A. Reitbauer, B. Drutu, N. Molnar, and Y. Shkuro, Eds., "Trace Context," W3C Recommendation, World Wide Web Consortium, Nov. 2021.
[8] X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, W. Li, and D. Ding, "Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study," IEEE Trans. Software Engineering, vol. 47, no. 2, pp. 243–260, 2021.
[9] M. Chow, D. Meisner, J. Flinn, D. Peek, and T. F. Wenisch, "The Mystery Machine: End-to-end performance analysis of large-scale Internet services," in Proc. 11th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2014, pp. 217–231.
[10] J. Kaldor et al., "Canopy: An end-to-end performance tracing and analysis system," in Proc. 26th ACM Symp. Operating Systems Principles (SOSP), 2017, pp. 34–50.
[11] J. Mace, R. Roelke, and R. Fonseca, "Pivot tracing: Dynamic causal monitoring for distributed systems," in Proc. 25th ACM Symp. Operating Systems Principles (SOSP), 2015, pp. 378–393.
[12] J. Mace and R. Fonseca, "Universal context propagation for distributed system instrumentation," in Proc. 13th European Conf. Computer Systems (EuroSys), 2018, Art. no. 8.
[13] OpenZipkin, "B3 propagation specification," OpenZipkin project, 2023. [Online]. Available: https://github.com/openzipkin/b3-propagation
[14] OpenTelemetry Authors, "Semantic conventions for messaging spans," OpenTelemetry Semantic Conventions, Cloud Native Computing Foundation, 2023. [Online]. Available: https://opentelemetry.io/docs/specs/semconv/messaging/
[15] OpenTelemetry Authors, "OpenTelemetry Collector documentation," Cloud Native Computing Foundation, 2023. [Online]. Available: https://opentelemetry.io/docs/collector/
[16] The Jaeger Authors, "Jaeger documentation: Architecture, sampling, and span references," Cloud Native Computing Foundation, 2023. [Online]. Available: https://www.jaegertracing.io/docs/
[17] P. Las-Casas, J. Mace, D. Guedes, and R. Fonseca, "Weighted sampling of execution traces: Capturing more needles and less hay," in Proc. ACM Symp. Cloud Computing (SoCC), 2018, pp. 326–332.
[18] P. Las-Casas, G. Papakerashvili, V. Anand, and J. Mace, "Sifter: Scalable sampling for distributed traces, without feature engineering," in Proc. ACM Symp. Cloud Computing (SoCC), 2019, pp. 312–324.
[19] G. Hohpe and B. Woolf, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Boston, MA, USA: Addison-Wesley, 2003.
[20] IBM Corporation, "IBM MQ 9.3 documentation: Message properties, the MQRFH2 header, and the PROPCTL attribute," IBM Documentation, 2022.
[21] N. Deakin, "Java Message Service, Version 2.0," JSR 343 Final Release, Java Community Process, Oracle, 2013.
[22] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds., Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA, USA: O'Reilly Media, 2016.
[23] C. Majors, L. Fong-Jones, and G. Miranda, Observability Engineering: Achieving Production Excellence. Sebastopol, CA, USA: O'Reilly Media, 2022.
[24] Y. Shkuro, Mastering Distributed Tracing: Analyzing Performance in Microservices and Complex Systems. Birmingham, U.K.: Packt Publishing, 2019.
[25] Y. Gan et al., "An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems," in Proc. 24th Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2019, pp. 3–18.
[26] IBM Corporation, "IBM MQ 9.3 documentation: Application activity trace, event monitoring, and accounting and statistics messages," IBM Documentation, 2022.