AIOps: The State of Full Packet Capture Enters the Age of Practicality
How machine learning and artificial intelligence are changing the game of acting on large volumes of network data in near real time.
AIOps – It’s a great time to be a security analyst, but those who serve in the role today are facing much higher expectations from their organizations compared with when I started out. Many are teetering on the edge of burnout because their companies need to get to the truth sooner, leaving analysts stuck with traditional approaches and tactics associated with full packet capture as the high-speed network’s bandwidth increases by the day.
The state of full packet capture — fundamental to enabling security analysts to hunt for threats, discover anomalies, or respond to incidents — has seen a few incremental advancements over the several decades but nothing that has allowed the analyst to allocate less time to it because there is still a bit of heavy lifting required.
As a security analyst in the military, my first experience with full packet capture in the late ’90s was the SHADOW system, an open source project dubbed an intrusion-detection system but really a full packet capture system designed for retrospective analysis, also known as threat hunting. The project was essentially a framework built with tcpdump and a collection of Perl scripts. However, SHADOW lacked any form of indexing, so mining the data was quite painful.
The next breakthrough in full packet capture was Time Machine, which introduced the notion of connection cutoff and indexing for faster search and retrieval. A sister project to Zeek (formerly known as Bro), Time Machine was an interesting project with lots of promise. Unfortunately, Time Machine did not scale beyond a few gigabits per second. Finally, there is Moloch, a full packet capture and search application integrated with advanced visualization that scales to 10Gbps and more. Moloch represents the state-of-the-art in open source, full packet capture, but it is yet to be determined if it can scale to 100Gbps.
These incremental improvements were made in the background while the high-speed network expanded and has grown in importance within the organization. While the number of servers on-premises might have decreased, the quantity of mobile devices, Internet of Things sensors and cloud applications that organizations are utilizing today to improve operations is increasing to create an even more complex network environment, making the traditional approaches to full packet capture even more impractical.
Adding to the problem is the recent rise in overall traffic. which is forecasted to continue. According to Cisco, companies can expect to see their network traffic triple by 2022. This will require organizations to make a proportional increase in data storage and maintain a brute force, record-everything approach for network forensics that will cost companies significantly more in terms of time and money. This runs counter to most companies’ digital transformation journeys where the bigger objective is to save on operational costs, increase IT agility, and improve responsiveness.
Fortunately, full packet capture is finally entering the age of practicality because of the introduction of AIOps. Gartner defines AIOps as the application of machine learning (ML) and data science to IT operations problems. The firm also predicts that large enterprises use of AIOps tools will reach 30% by 2023. The adoption of AIOps will pave the way to security automation like intelligent packet capture. This is an exciting development that our company is pursuing, along with others in the industry, to enable security analysts to utilize AIOps for network forensics.
This article originally appeared on darkreading.com To read the full article and see the images, click here.
Nastel Technologies uses machine learning to detect anomalies, behavior and sentiment, accelerate decisions, satisfy customers, innovate continuously. To answer business-centric questions and provide actionable guidance for decision-makers, Nastel’s AutoPilot® for Analytics fuses:
- Advanced predictive anomaly detection, Bayesian Classification and other machine learning algorithms
- Raw information handling and analytics speed
- End-to-end business transaction tracking that spans technologies, tiers, and organizations
- Intuitive, easy-to-use data visualizations and dashboards
If you would like to learn more, click here.