Enhancing Anomaly Detection Techniques for Emerging Threats

Mills, Ryan and Race, Nicholas and Broadbent, Matthew (2022) Enhancing Anomaly Detection Techniques for Emerging Threats. PhD thesis, Lancaster University.

Text (2022MillsPhD)
2022millsphd.pdf - Published Version
Available under License Creative Commons Attribution-NonCommercial.
Download (7MB)

Abstract

Despite the Internet being an apex of human achievement for many years, criminal behaviour and malicious activity are continuing to propagate at an alarming rate. This juxtaposition can be loosely attributed to the myriad of vulnerabilities identified in existing software. Cyber criminals leverage these innovative infection and exploitation techniques to author pervasive malware and propagate devastating attacks. These malicious actors are motivated by the financial or political gain achieved upon successful infiltration into computer systems as the resources held within are often very valuable in nature. With the widespread developments in the Internet of Things (IoT), 5G, and Starlink satellites, unserved areas of the world will experience a pervasive expansion of connected devices to the Internet. Consequently, a barrage of potential new attack vectors and victims are unfolding which requires constant monitoring in order to manage this ever growing problem. Conventional rule-based intrusion detection mechanisms used by network management solutions rely on pre-defined attack signatures and hence are unable to identify new attacks. In parallel, anomaly detection solutions tend to suffer from high false positive rates due to the limited statistical validation of ground truth data, which is used for profiling normal network behaviour. When considering the explosive threat landscape and the expanse of connected devices, current security solutions also face challenges relating to the scale at which attacks need to be monitored and detected. However, recent innovations in Big Data processing have revealed a promising avenue in which scale is addressed through cluster computing and parallel processing. This thesis advances beyond current solutions and leverages the coupling of anomaly detection and Cyber Threat Intelligence (CTI) with parallel processing for the profiling and detection of emerging cyber attacks. This is demonstrated through the design, implementation, and evaluation of Citrus: a novel intrusion detection framework which is adept at tackling emerging threats through the collection and labelling of live attack data by utilising diverse Internet vantage points in order to detect and classify malicious behaviour using graph-based metrics, as well as a range of Machine Learning (ML) algorithms. This research provides innovative contributions to the cyber security field, including the public release of an open flow-based intrusion detection data set. This data set encompasses emerging attack patterns and is supported by a robust ground truth. Furthermore, Citrus advances the current state of the art through a novel ground truth development method. Citrus also enables both near real-time and offline detection of emerging cyber attacks under optimal computational costs. These properties demonstrate that it is a viable and practical solution for next generation network defence and resilience strategies.