Managing Kafka on AWS with Nastel Navigator

Apache Kafka is a very popular open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Amazon MSK is a fully managed Kafka service that makes using Kafka even easier. But how do you get started with Kafka on AWS? How do you migrate your existing Kafka environment to the cloud? How do you manage many Kafka environments at once?  How do you give the application teams easy audited access to their data without putting other teams at risk?

Nastel Navigator provides a single GUI for migration and management of multiple middleware products including Apache Kafka and Amazon MSK. It enables you to reduce operational risk with automated audited deployment, migration, secure DevOps self-service with full lifecycle management, integrated into DevOps tooling for debugging, root cause analysis, diagnostics, and remediation. You can securely manage multiple middleware products including Kafka, IBM MQ, and TIBCO EMS on multiple servers on multiple clouds and on-premises.

Here you can see a typical screenshot of the GUI. It is entirely customizable. In this case, the user has Kafka, TIBCO, and MQ on different tabs. On the Kafka tab, they have viewlets that allow them to view only the topics, partitions, and consumers that are relevant to their applications.

In this blog post, we explain how to connect Nastel Navigator, available in AWS Marketplace, with a Kafka server. Nastel Navigator uses a central server environment that connects to and collects data remotely from the Kafka clusters and provides access to the entire configuration.

Prerequisites

To run this solution, you must have the following prerequisites:

• An AWS account – if you don’t already have one, create one.
• An installation of Apache Kafka. This could be one that you’ve installed yourself from the Kafka community or it could be the AWS MSK managed service.
• Nastel Navigator can be installed from AWS Marketplace. There is a choice of editions depending on the number of Kafka brokers. Nastel Navigator Express is the completely free version that can be used for this walkthrough.

Solution overview

The Navigator platform consists of a Complex Event Processor (CEP) Server, a Workgroup server, a Domain Server, and a Web Server. The platform interacts with multiple middleware instances over TCPIP and is accessed by multiple users via the Web Server.

Solution walkthrough

For information about the installation and setup of this central server, see the Nastel documentation. This post starts at the point where the server has been installed and is ready to operate.

Because Navigator connects as a client connection, we use standard connection details and options. Although this post uses Amazon MSK, you have other options for Kafka. In this case, we get these details from the Amazon MSK cluster configuration.

1. Log in to the Amazon MSK Cluster and select ‘View Client Information’
2. Copy the bootstrap server for the TLS or Plaintext bootstrap server to the clipboard.
3. Log in to the Nastel Navigator interface and create a definition for this Kafka cluster. To do this, go to the Workspace tab, select the WGS line, click on Create, then Remote Kafka Managers.
4. Provide the details from the information that we collected, namely the name of the cluster and the bootstrap server.
5. If necessary, add other properties on the Config tab. This could include credentials and connection attributes.

Now you’re ready to manage this Kafka cluster. As well as viewing authorized topics and partitions, you can drill down to see individual messages in them, add new test messages, and perform other actions such as creating, deleting, and changing Kafka definitions.

Conclusion

For more information about usage, see the Nastel Resource Center. For more information on the Nastel platform and other capabilities, contact us at the Nastel website.