Stirring Apache Kafka into the Middleware Mix

(First of a six-part blog series that describes how a team of IT pros and managers at one of the world’s largest global banks accommodated a bank acquisition and mastered a complex hybrid messaging environment spanning mainframes, distributed systems, and clouds.)

About the Author

VP – Product Management, and a published author and sought-after industry speaker .Charley Rich has decades of product management experience spanning Middleware Messaging, Analytics, APM, and SaaS. Charley contributed to four successful startups and was worldwide product manager at IBM Tivoli, earning the General Manager’s Award. At Nastel Technologies he is VP – Product Management, and a published author and sought-after industry speaker .

In the Beginning, There Was Darkness…

Ok, I’m being a bit overly dramatic. What was happening in New York a couple of years ago at a really big global bank—I’ll call it BigBank—with almost U.S. $2 trillion in total assets was a lack of visibility into a complex hybrid messaging environment.

Let me paint the scene. BigBank’s IT team had been getting along pretty well… Among other development and operations activities, they were successfully monitoring and managing their IBM MQ middleware family of products as well as an additional homegrown messaging solution as part of their cash management and payment processing systems. All was well with bank applications utilizing MQ and its associated tools.

So what was the problem? BigBank had just acquired another firm—I’ll dub that one TargetBank—that employed TIBCO EMS (a fully compliant JMS service implementation) for reliable messaging and TIBCO Rendezvous (messaging middleware) for rapid publishing of market data instead of MQ for their middleware and messaging layer. Further complicating the process of combining two completely different infrastructures was the fact that both organizations were rolling out Apache Kafka implementations.

BigBank’s IT team quickly set about putting DataPower appliances to work acting as a translation bridge between the two technology platforms, transforming and routing messages coming from BigBank applications using MQ into the native TIBCO EMS format that TargetBank’s applications employed, and vice versa.

To monitor and manage performance of the combined middleware and messaging infrastructure end-to-end, the team needed visibility for:

…and they wanted a single product to provide monitoring, tracking, and analytics capabilities for it all.

Figure 1: BigBank required monitoring, tracking, and analytics for their complex combined infrastructure


BigBank’s current mix of vendor-supplied and home-grown tools couldn’t handle the complexity of the new combined technology environment. Something better was needed to help maintain improved productivity, reduce MTTR (mean time to repair) for problems, minimize support costs associated with problem resolution, and provide visualization tools to supply IT pros and business analysts “the Big Picture.”

The search was on for a global monitoring solution that could provide a single point of control for:

  • Unified operational procedures that span multiple technology platforms
  • Run-time operations, MQ administration, rapid problem detection and isolation, and operational review and planning
  • Seamless integration of Apache Kafka monitoring capabilities (including performance and latency metrics, message content surveillance and payload examination, plus full audit tracking)
  • Message auditing – used to trace the granting or denying of permissions
  • Tracking – stitching together a flow of transactions, each defined as a group of activities targeted at achieving a common goal or task. These activities are deemed related based on observation of messages exchanged between them or more precisely between logical units of work (LUWs). LUWs represent a collection of operations and messages within a session that must all be done or none done, and are typically delimited by START/CMMIT calls
  • Transformation – used in IBM Message Broker (IBM Integration Bus, or IIB) where the structure of a message is changed from one XML format to another. This is often required to provide communications between applications that read different formats
  • Routing – employed in IBM Message Broker (Integration Bus) to send a message to a receiver based on message contents
  • Performance – measuring application throughput in terms of metrics such as messages/second or transactions/second

The right tool would greatly improve interoperability between DataPower, Solace, IBM MQ, TIBCO, and Apache Kafka messaging components. Last but certainly not least on the team’s wish list was the ability to track messages through JMS, IBM MQ, DataPower, and Apache Kafka.

Would they be able to find a single software product that could do what was necessary?

Yup! (Otherwise I wouldn’t be writing this blog.) Stay tuned next week for my next blog installment in Achieving Complete Visibility in a Complex Hybrid Messaging Environment – Part 2.