Kafka Security – First Steps
Apache Kafka provides an unified, high-throughput, low-latency platform for handling real-time data feeds. Installing Apache Kafka, especially the right configuration of Kafka Security including authentication and encryption is kind of a challenge. This should give a brief summary about our experience and lessons learned when trying to install and configure Apache Kafka, the right way.
Do you really need Apache Kafka?
Think about this carefully before you start using Apache Kafka. Don’t get me wrong: Apache Kafka is great in doing stuff it is intended to do but for some use-cases Apache Kafka tend to be a overkill. Going into this topic is outside the scope. I’ve found a good article about this topic: Apache Kafka Use Cases: When To Use It & When Not To. To sum it up in a few words:
Use Kafka for:
- Real-time data processing: Transmitting data from producers to data handlers and then to data storages.
- Application activity tracking: This is the use case Kafka was originally developed for, to be used in LinkedIn: Publishing events in applications to dedicated Kafka topics.
- Logging and/or monitoring: Publishing logs into Kafka topics.
Don’t use Kafka if:
- You need to process only a small amount of messages per day. Think about using traditional message queues like RabbitMQ.
- You need to apply on-the-fly data transformations or doing ETL jobs (extract, transform, load).
- You need a task queue: Again, RabbitMQ is designed for being a task queue.
- You need a database: There is no need for an explanation: Kafka is not designed for persistent data storage.
What do we have?
There are several ways how to setup Apache Kafka but the easiest way to install Apache Kafka is using Ansible and the official Kafka playjobs from Confluent: The Easiest Way to Install Apache Kafka and Confluent Platform – Using Ansible. This can also be used for production i.e. to setup a multi-node cluster including 3 nodes with an Kafka Broker and a Apache ZooKeeper instance on each node. The Kafka Broker is doing the work i.e. consuming and producing messages based on using topics. The Apache ZooKeeper instances are used for maintaining configuration information, naming, providing distributed synchronization, and providing group services. So you can say, that the Apache ZooKeeper is used to manage our Kafka Cluster.
Now the first challenge: You need authN/authZ and SSL encryption for all connections:
- Kafka clients (i.e. your software) are connecting to the Kafka Brokers.
- Each Kafka Broker is connecting to the other brokers.
- Each Kafka Broker is connecting to the ZooKeeper instances.
Each of these connections need SSL encryption, authentication and authorization. Since authorization is done via ACLs the focus of this article is on SSL encryption and authentication.
Kafka Broker Authentication
There are several methods to authenticate Kafka clients against Kafka Brokers. Note that for inter-node communication a Kafka Broker can also be a Kafka client that is connecting to another Kafka Broker.
Mutual TLS (mTLS)
Each Kafka Broker and each Kafka client gets a SSL certificate that supports client authentication. This is easy and bullet-proofed but you need a client certificate for each Kafka Broker and each Kafka client and this is out of scope what e.g. Let’s Encrypt offers. So you need your own PKI infrastructure and probably an own certificate authority to do so.
In SASL PLAIN, user and passwords are stored in each Kafka Broker using SASL PLAIN (not to mix up with no SSL encryption being called PLAINTEXT). So SSL encryption is a must or this doesn’t make sense at all. Further think about what will happen if you add or change a user or a password: You must update and restart each Kafka Broker for each user or password change. So this is still simple to setup but this does not scale well with Kafka Brokers and clients.
Salted Challenge Response Authentication Mechanism (SCRAM) is a family password-based challenge–response authentication mechanisms providing authentication of a user (here our Kafka client or Kafka Broker) to a server (Kafka Broker). If enabled, salted usernames and passwords are stored encrypted in Apache ZooKeeper. So scaling clients without restarting the Kafka Brokers should work. Further no passwords will be send unencrypted over the wire but SSL encryption is still required to prevent interception of SCRAM exchange.
This article originally appeared on awesome-it.de.com, to read the full article, click here.
Check out this page to see how Nastel’s i2M platform helps you to secure Kafka environments.
Nastel Technologies is the global leader in Integration Infrastructure Management (i2M). It helps companies achieve flawless delivery of digital services powered by integration infrastructure by delivering tools for Middleware Management, Monitoring, Tracking, and Analytics to detect anomalies, accelerate decisions, and enable customers to constantly innovate, to answer business-centric questions, and provide actionable guidance for decision-makers. It is particularly focused on IBM MQ, Apache Kafka, Solace, TIBCO EMS, ACE/IIB and also supports RabbitMQ, ActiveMQ, Blockchain, IOT, DataPower, MFT, IBM Cloud Pak for Integration and many more.
The Nastel i2M Platform provides:
- Secure self-service configuration management with auditing for governance & compliance
- Message management for Application Development, Test, & Support
- Real-time performance monitoring, alerting, and remediation
- Business transaction tracking and IT message tracing
- AIOps and APM
- Automation for CI/CD DevOps
- Analytics for root cause analysis & Management Information (MI)
- Integration with ITSM/SIEM solutions including ServiceNow, Splunk, & AppDynamics