Kafka

Expanding Visibility With Apache Kafka

Expanding Visibility With Apache Kafka - What would you do if you had terabytes of operational data being generated in production each day, and hundreds of engineering teams wanting to use that data to improve their services … but no way to connect the two? A Broader Vision This started us thinking bigger. There are a vast [...]

By |Kafka, logging|

How Apache Kafka Inspired Our Platform Events Architecture

There’s a class of scalability challenges that are best approached using an asynchronous, event-driven architecture — particularly when it comes to doing data integrations. To support these uses cases Salesforce recently added a feature called “Platform Events” which exposes a time-ordered immutable event stream to our customers. Our multitenant setup posed some challenges for traditional messaging systems and luckily Apache Kafka [...]

By |Kafka, logging, Middleware|

Developing a Deeper Understanding of Apache Kafka Architecture

The Apache Kafka distributed streaming platform features an architecture that – ironically, given the name – provides application messaging that is markedly clearer and less Kafkaesque when compared with alternatives. In this article, we’ll take a detailed look at how Kafka’s architecture accomplishes this. The Kafka Components – Universal Modeling Language (UML) Kafka’s main architectural components include Producers, Topics, [...]

By |Kafka|

AutoPilot Monitoring for Apache Kafka

AutoPilot for Apache Kafka provides operational and transactional monitoring for Apache Kafka, the open-source stream processing platform developed by the Apache Software Foundation, written in Scala and Java. Apache Kafka is a unified, high-throughput, low-latency platform for handling real-time data feeds. It utilizes a massively scalable pub/sub message queue—designed as a distributed transaction log—as its storage [...]

By |Apache, Kafka, open-source|