AutoPilot Insight: A Fresh Approach to Analytics
It’s a given that IT organizations and other lines of business want to find data outliers faster and sense problem conditions before they actually affect users. But ever-growing volumes of business data and never-ending streams of event data overwhelm traditional analytic methods. These approaches just don’t scale. A fresh approach to analytics is to leverage machine learning which improves over time and addresses novel situations.
Although many IT organizations field basic analytic tools sufficient to keep MTTR to an acceptable level, they need more sophisticated capabilities to answer questions like: “How does the performance of IT activities and operations impact our business?” And, “Is there a way to understand these dynamic interplays in real-time to optimize intelligent day-to-day management of the business?”
To answer these business-centric questions and provide actionable guidance for decision-makers, Nastel’s AutoPilot for Analytics fuses:
- Advanced predictive anomaly detection, Bayesian Classification and other machine learning algorithms
- Raw information handling and analytics speed
- End-to-end business transaction tracking that spans technologies, tiers, and organizations
- Intuitive, easy-to-use data visualizations and dashboards
All of these capabilities fuse seamlessly across dynamic IT environments, from mobile to mainframe, and provide the broad array of analytic and decision-support capabilities needed by developers, IT admins, and business analysts to satisfy enterprise-grade operations intelligence and APM needs.
Nastel provides two types of machine learning methodologies that learn and improve their analysis over time without any dependency on writing rules. The two methodologies are anomaly detection and Bayesian Conditional Probability.
- This algorithm is based on Netflix’s Robust Anomaly Detection (RAD)
- Detects data outliers within massive amounts of event ‘noise’ using machine-learning-based anomaly detection
- Detect anomaly in a period of time
- The function defects unusual elapsed times and learns if they become normal
- If this anomaly keeps reoccurring in this season, it becomes designated as “Not Normal”
Figure 1: This query analyzes activities and detects hours in the day with unusual elapsed times
- Classifies behavior and learns over time
- Addresses the problem of judging documents as belonging to one category or the other (such as spam or legitimate, sports or politics, etc.)
- Used to categorize sentiment or behavior
- Can provide predictive assessments using conditional probability
- This can be trained on either the results of a query or data manually specified by users
Figure 2: This query reads the note the Customer Service Rep entered about the customer and uses this to analyze customer sentiment. This is used to categorize users and predict their future behavior based on their level of satisfaction.
Predictive Anomaly Detection and Machine Learning
AutoPilot’s real-time “smart” analytics for anomaly detection makes the task of detecting problems and combining emerging trends and subtle behavior patterns into a clear picture of how IT operations are affecting business outcomes much easier and faster. IT professionals and business stakeholders can now make more informed and rapid decisions based on business insights formerly hidden in a multitude of high-volume data sources.
This major advance in anomaly detection and machine learning is based on extensive enhancements and extensions to the open source code contribution by Netflix in their Robust Anomaly Detection (RAD) project. Utilizing machine learning technology, AutoPilot rapidly improves over time at predicting, sensing, and evaluating the exact nature of performance issues.
Potential use cases include almost any industry vertical where subtle data outliers demand fast reactions and decisions. Some examples include actual or potential financial system security breaches, non-compliance, supply chain issues that potentially cascade rapidly into major delivery problems affecting customers…the list is almost endless.
- Developers can leverage AutoPilot’s ready-to-use smart analytics without the burden of constantly writing rules to make sense of data
- Business analysts are armed with advanced tools to better understand the behavior of users and immediately understand what is normal or expected behavior, and what is not
Enterprise-grade “Fast Data” Handling & Analytics Speed
The presence of fast data — high-velocity streaming information with short-term value generated from a large multitude of IT infrastructure devices and instrumentation sources—can easily overwhelm existing analytic and APM tools.
Nastel has released the next generation of its high-speed analytics platform incorporating real-time parallel data processing grids and FatPipes™ orchestration: AutoPilot Insight.
AutoPilot Insight analytics platform architecture
Designed for extreme scalability, AutoPilot Insight’s Lambda architecture provides the data management and processing bandwidth necessary to handle the largest enterprise use cases, regardless of data source or level of high-velocity data flows.
Business Transaction Tracking
Because transactions range from simple synchronous message exchanges between point-to-point application connections, to much more complex asynchronous communications, a sophisticated tracking and monitoring solution is required.
In the latter instance, long-running, multi-step asynchronous transactions transit your IT infrastructure, spanning multiple technologies, tiers, and even different organizations. They frequently involve a broker routing the messages associated with a single transaction to many discrete destinations (DBs, servers, apps, mainframes, etc.). And because a message broker is involved, these complex transactions often morph and split, thus defying standard tracking and analysis via tagging or statistical sampling techniques.
AutoPilot stitches together complete end-to-end transactions by examining method calls and individual message payload contents, correlating them and presenting intuitive visualizations of any pending or existing breaches in expected behavior and performance.
AutoPilot’s deep examination of transaction message payloads enables linkage of IT activities and behavior to expected and actual business outcomes
Following are several examples of actual Business Milestones displays drawing upon Nastel’s transaction tracking capabilities.
Example 1: Business Milestones display reflecting a SWIFT-based financial payment topology with red color bars flagging problem areas for the user
Example 2: Business Milestones for product order transactions, from initiation to shipment, showing clickable drilldown detail screen
For analysts and other non-technical business users, Nastel’s new Launchpad makes it easy to send data to AutoPilot, automatically create intuitive dashboard displays, or simply walk users through sample business use case setups that could be used as templates for their own purposes. Enterprise organizations now have an easy means of applying analytics to the data streaming from their applications to AutoPilot® Insight.
By adding intuitive workflows plus wizards, use-case templates, and dashboard launch-pad features, the net result is an extremely fast and capable APM platform that serves the analysis and reporting needs of everyone from developers, IT admins, telecom managers, and business analysts.”
Example of AutoPilot Launchpad, showing access to analytics, demos, data dashboard, and explanations of sample use cases
Example of wizard-guided data analysis process for business specialists
Example of candlestick data presentation in the context of memory utilization
360° Situational Awareness®
AutoPilot’s real-time CEP analytical engine collects, aggregates, filters and correlates metrics and events from infrastructure systems, then augments and enriches that data with information obtained from external sources such as RSS, news feeds, financial data and even email messages for complete 360° situational awareness.
AutoPilot’s CEP analytics engine automatically calculates dynamic trends and creates its own metrics, such as “change in rate-of-change.” It can determine what is truly outside of “business normal” for a given environment, instantly differentiating intermittent spikes in, say, resource consumption from a true problem that might cause an outage or impede the delivery of a business service. It is used for:
- Compliance – detecting potential or actual breaches in responsibility
- Preventing false problem alarms
- Understanding trends across composite applications
- Determining the ability to handle rapid increases in load