I
n modern large-scale distributed networks, the
rapid increase in traffic complexity and the emergence of sophisti-
cated cyber-attacks have made traditional anomaly detection sys-
tems inadequate. Existing detection techniques such as signature-
based and statistical threshold models are limited to recognizing
known attack patterns and fail to identify zero-day or evolving
threats. While Finite State Automata (FSA)–based detection
methods provide a structured and interpretable representation of
protocol behaviors, they lack adaptability and scalability when
faced with dynamic and heterogeneous traffic environments.
Conversely, machine learning and deep learning–based systems
such as Support Vector Machines (SVM), Autoencoders, and
Convolutional Neural Networks (CNN) have improved accuracy
but often act as opaque “black-box” models that are difficult
to interpret and prone to high false-positive rates. These short-
comings collectively hinder the deployment of reliable, real-time
network anomaly detection mechanisms capable of addressing
modern cyber threats. To overcome these limitations, this paper
proposes a two-level hybrid anomaly detection architecture that
integrates the formal modeling power of Finite State Automata
(FSA) with the adaptive intelligence of a Generative Artificial
Neural Network (Gen-ANN). The FSA layer models the standard
TCP protocol state transitions and identifies deviations such
as SYN flood and Xmas scan attacks, providing explainable,
protocol-level anomaly recognition. The Gen-ANN layer then
revalidates these detections, refining classification accuracy and
significantly reducing false positives by learning complex flow
correlations. The architecture also includes role-based access
control (RBAC) to ensure secure data management, batch-mode
traffic analysis for scalability under API constraints, and a React-
based real-time visualization dashboard for monitoring network
behavior and anomaly trends.
