There’s a class of scalability challenges that are best approached using an asynchronous, event-driven architecture — particularly when it comes to doing data integrations. To support these uses cases Salesforce recently added a feature called “Platform Events” which exposes a time-ordered immutable event stream to our customers. Our multitenant setup posed some challenges for traditional messaging systems and luckily Apache Kafka came to the rescue by inspiring us to leverage the time-ordered event log paradigm. This post will detail the unique needs we encountered and how event logs helped us address those.
Enterprise Developers Need Events
As Enterprise systems grow, they tend to become more complex. Teams gradually add more and more independent services, with different development lifecycles, and eventually the system invariably reaches a point where decoupling these services becomes critical. Thus, a shift towards asynchronous interactions is a natural evolution; enterprise developers love event-driven architectures because they provide the runtime independence you need to tame this complexity.
Event Log to the Rescue
As we were first architecting this system, Apache Kafka was getting more and more popular inside Salesforce, and its simplicity and reliability seemed appealing. Kafka does not use a traditional queuing paradigm, but instead arranges events in the form of an immutable time-ordered log. It inspired a breakthrough in our search for a multi-tenant pub/sub architecture — in all the ways that a traditional message queue wouldn’t work for us, the Kafka paradigm seemed to fit the bill.
This article originally appeared on Salesforce Engineering. To read the full article, click here.
AutoPilot for Apache Kafka offers powerful forensics to diagnose Kafka problems. Kafka performance and availability monitoring is accomplished via end-to-end stream monitoring and tracking of metrics from brokers, consumers, producers and Zookeeper, Kafka’s configuration service. AutoPilot for Apache Kafka examines the metrics collected for Kafka topics, producers, consumers and brokers while simultaneously offering deep-dive insight into the JVM itself.