Monitor and Manage Apache Kafka

Effective and Efficient Monitoring, Management, Tracing and Tracking with Nastel’s® AutoPilot® for Apache Kafka

Nastel for kafka from Nastel Technologies on Vimeo.

NASTEL AUTOPILOT FOR KAFKA

Nastel’s® AutoPilot® for Apache Kafka provides operational and transactional monitoring for Apache Kafka.

Kafka is the open-source stream processing platform developed by the Apache Software Foundation, written in Scala and Java.

MONITOR APACHE KAFKA

Apache Kafka is a unified, high-throughput, low-latency platform for handling real-time data feeds.

It utilizes a massively scalable publish / consume message queue designed as a distributed transaction log as its storage layer.

It is often used by Nastel clients as a transport mechanism for streaming data interconnected with other messaging and processing systems.

Key Benefits

SINGLE VERSION OF THE TRUTH

Nastel’s® AutoPilot® for Apache Kafka delivers a single-point-of-truth to track performance, latency, and logs, along with full message auditing and content surveillance capabilities.

It provides complete message flow analytics, relating applications to the messages they publish to Kafka, and the applications that subscribe to those messages.Powerful network auto-discovery capabilities are included with matching visualization tools, as seen below. In the Viewlet below, solid lines (edges) represent send-to relationships; dotted lines represent an Acts-On relationship.Each edge has statistics showing average elapsed time and count.
POWERFUL FORENSIC ANALYSIS

Nastel’s® AutoPilot® for Apache Kafka offers powerful forensics to diagnose Kafka problems.

Kafka performance and availability monitoring is accomplished via end-to-end stream monitoring and tracking of metrics from brokers, consumers, producers and Zookeeper (Kafka’s configuration service).
EXAMINE METRICS 

Nastel’s® AutoPilot® for Apache Kafka examines the metrics collected for Kafka topics, producers, consumers and brokers while simultaneously offering deep-dive insight into the JVM itself.

MONITOR KAFKA 

Monitor all Kafka Clusters, Producers, Consumers, Brokers, Topics, Partitions, Leaders, Followers

FOR DEVELOPERS

Make Kafka apps faster and more stable 

– Optimizing Kafka apps
– Identifying latency and performance bottlenecks
– Identifying spots for data loss

Improve quality and help debug apps 
– Capturing Kafka exceptions
– Enabling easy message capture and profiling
– Generating message flow charts and topology displays

FOR OPERATIONS

Reduce MTTR (mean time to repair) 

– Finding problems and bottlenecks
– Generating alerts based on user-defined conditions
– Providing anomaly detection in flows
– Identifying problems, data loss, and latencies

Where is my message?
– Capture messages for problem identification Message flow analytics
– see the longest and shortest paths

FOR BUSINESS

Message Content Analytics

– Extract and summarize relevant business tokens. E.g., payment amounts (max, min, avg, etc.)

Alerts based on business conditions


– SLAs and OLAs

SPOG

– Single-pane-of-glass display for business flows over Kafka infrastructure Risk management and audit capabilities for relevant business transactions

Key Features

AUTO DISCOVERY

Auto discovery of end-to-end transactions spanning Kafka and other technologies such as IBM MQ

MESSAGE PARSING

Parsing of Kafka messages which are tokenized and utilized for analytics and transaction stitching

MONITOR SLA’S

SLA Monitoring, Analytics and Reporting

DEEP DIVE

Deep-dive monitoring of composite application components that include Kafka

PROACTIVE ALERTING

Proactive alerting and reduction in false alarms

Note: Nastel’s® AutoPilot® for Apache Kafka also utilizes Kafka internally as an integration technology for data transport.

Nastel AutoPilot for Kafka

Nastel AutoPilot for Kafka provides the industries deepest pro-active and real-time Kafka monitoring.
Nastel AutoPilot for Karka monitors queues, channels, managers, listeners and queue statistics.

While simultaneously managing DLQs (Dead Letter Queues) Nastel delivers self-service capabilities that allow your
development, QA and Operations teams
to manage their own queues as well as message tracking, Kafka configuration management and capacity planning in a single easy to use solution for Apache Kafka.

Nastel AutoPilot for Kafka lowers your operational risk while improving user experience and reducing support costs, including Mean Time to Repair (MTTR).
READ MORE

Nastel XRay for Kafka

Profile your Kafka middleware deployments. Know where your messages are instantly. Find applications which abuse Kafka resources.

Search messages by content, header and more. Visualize message flows across apps, queues, channels. Optimize performance of your Kafka apps and messaging topologies.

READ MORE

Nastel Navigator for Kafka

Nastel’s Navigator gives you the visibility and diagnostics of middleware including objects and messages you need to address your Middleware issues.
all within a web browser
with absolutely no impact on production.


                     (COMING SOON)

READ MORE

Kafka is an open-source solution used broadly by all sizes of enterprise to allow disparate applications to share real-time data.

Kafka creates streams of data in channels called Kafka Topics

Nastel AutoPilot, Nastel XRay and Nastel Navigator is your 360-degree situational awareness solution for Kafka

KAFKA ARCHITECTURE

Kafka servers (the system that is Kafka) are called Kafka Brokers 

Systems that publish data to a Kafka topic are called Kafka Producers Systems that read content from a Kafka topic are called Kafka Consumers

TOPICS

Topics contain series of data referred to as partitions, partitions are each assigned to a specific Kafka broker, the broker that owns a partition is referred to as the partition leader.
Other copies of a partition are replicated to additional Kafka servers, these are referred to as followers.

KAFKA CLUSTER

A Kafka Cluster will contain one (or more) of Brokers, Producers and Consumers.

WHAT DOES NASTEL SOLVE FOR KAFKA ADMINISTRATORS AND USERS

Nastel AutoPilot for Kafka is able to monitor the entire Kafka Cluster and all of its underlying components.

Nastel XRay for Kafka is able to present and analyze the entire pathway each user’s transaction makes through the entire application stack Analytically see how users are experiencing your applications and to compare real-time performance to historical performance allowing you to predict and improve the entire experience.

Nastel Navigator for Kafka provides your administrators with a web based command and control environment to manage all of your Kafka systems and to securely allow your users to self-serve their administrative needs.