Abstract:

In modern large-scale distributed networks, the rapid increase in traffic complexity and the emergence of sophisti cated cyber-attacks have made traditional anomaly detection sys tems inadequate. Existing detection techniques such as signature based and statistical threshold models are limited to recognizing known attack patterns and fail to identify zero-day or evolving threats. While Finite State Automata (FSA)–based detection methods provide a structured and interpretable representation of protocol behaviors, they lack adaptability and scalability when faced with dynamic and heterogeneous traffic environments. Conversely, machine learning and deep learning–based systems such as Support Vector Machines (SVM), Autoencoders, and Convolutional Neural Networks (CNN) have improved accuracy but often act as opaque “black-box” models that are difficult to interpret and prone to high false-positive rates. These short comings collectively hinder the deployment of reliable, real-time network anomaly detection mechanisms capable of addressing modern cyber threats. To overcome these limitations, this paper proposes a two-level hybrid anomaly detection architecture that integrates the formal modeling power of Finite State Automata (FSA) with the adaptive intelligence of a Generative Artificial Neural Network (Gen-ANN). The FSA layer models the standard TCP protocol state transitions and identifies deviations such as SYN flood and Xmas scan attacks, providing explainable, protocol-level anomaly recognition. The Gen-ANN layer then revalidates these detections, refining classification accuracy and significantly reducing false positives by learning complex flow correlations. The architecture also includes role-based access control (RBAC) to ensure secure data management, batch-mode traffic analysis for scalability under API constraints, and a React based real-time visualization dashboard for monitoring network behavior and anomaly trends