Abstract:

I

n modern large-scale distributed networks, the

rapid increase in traffic complexity and the emergence of sophisti-

cated cyber-attacks have made traditional anomaly detection sys-

tems inadequate. Existing detection techniques such as signature-

based and statistical threshold models are limited to recognizing

known attack patterns and fail to identify zero-day or evolving

threats. While Finite State Automata (FSA)–based detection

methods provide a structured and interpretable representation of

protocol behaviors, they lack adaptability and scalability when

faced with dynamic and heterogeneous traffic environments.

Conversely, machine learning and deep learning–based systems

such as Support Vector Machines (SVM), Autoencoders, and

Convolutional Neural Networks (CNN) have improved accuracy

but often act as opaque “black-box” models that are difficult

to interpret and prone to high false-positive rates. These short-

comings collectively hinder the deployment of reliable, real-time

network anomaly detection mechanisms capable of addressing

modern cyber threats. To overcome these limitations, this paper

proposes a two-level hybrid anomaly detection architecture that

integrates the formal modeling power of Finite State Automata

(FSA) with the adaptive intelligence of a Generative Artificial

Neural Network (Gen-ANN). The FSA layer models the standard

TCP protocol state transitions and identifies deviations such

as SYN flood and Xmas scan attacks, providing explainable,

protocol-level anomaly recognition. The Gen-ANN layer then

revalidates these detections, refining classification accuracy and

significantly reducing false positives by learning complex flow

correlations. The architecture also includes role-based access

control (RBAC) to ensure secure data management, batch-mode

traffic analysis for scalability under API constraints, and a React-

based real-time visualization dashboard for monitoring network

behavior and anomaly trends.