Project Overview

This project presents a modular, dockerized pipeline for real-time network anomaly detection leveraging live packet capture, intrusion detection systems, and machine learning. The system is designed for Security Operations Centers (SOCs), research environments, and organizations requiring advanced network monitoring capabilities.

Key Capabilities

Live Packet CaptureContinuous network traffic ingestion using tcpdump, tshark, or scapy with automated rotation
IDS EnrichmentZeek, Suricata, or Snort processing extracts HTTP, DNS, SSL, and alert logs
ML DetectionUnsupervised algorithms (Isolation Forest, Autoencoders, LSTM) identify novel threats
Dockerized DeploymentConsistent, portable environment across any infrastructure

Pipeline Architecture

1. Live Capture

tcpdump / tshark / scapy
5-minute .pcap rotation

2. IDS Enrichment

Zeek / Suricata / Snort
HTTP, DNS, SSL, Alerts

3. Feature Extraction

Connection durations
Byte counts, protocols

4. ML Detection

Isolation Forest
Autoencoders, LSTM

5. Alerting

Real-time notifications
SOC integration

How It Works

  • 1. Live Packet Capture: Network packets are captured in real time using tools like tcpdump, tshark, or scapy. Captured traffic is periodically saved into .pcap files (e.g., every 5 minutes) through an automated script or cron job.
  • 2. Traffic Enrichment with IDS: Each .pcap file is processed by powerful IDS tools such as Zeek, Suricata, or Snort. These tools extract detailed logs (e.g., HTTP, DNS, SSL, alerts), providing enriched insights into network behavior and potential threats.
  • 3. Machine Learning-Based Anomaly Detection: Extracted logs and features—such as connection durations, byte counts, protocols, signatures, and network flow statistics—are fed into unsupervised anomaly detection algorithms (e.g., Isolation Forest, Autoencoders, LSTM-based models) implemented with frameworks such as scikit-learn, TensorFlow, or PyTorch.
  • 4. Automated and Near Real-Time Analysis: The system continuously analyzes enriched network data, flags suspicious activity, and can notify users or security systems, enabling timely response to emerging anomalies.
  • 5. Dockerized Deployment: The entire pipeline is containerized with Docker to simplify deployment, scalability, and environment consistency. This enables you to easily run the system across different machines or cloud environments without complex setup.

Technology Stack

Packet Capture

tcpdump tshark scapy

IDS Tools

Zeek (Bro) Suricata Snort

ML Frameworks

scikit-learn TensorFlow PyTorch

Algorithms

Isolation Forest Autoencoders LSTM

Infrastructure

Docker Docker Compose Python 3.8+ Cron

Anomaly Detection Algorithms

Isolation Forest

Ensemble method that isolates anomalies instead of profiling normal points. Efficient for high-dimensional network flow data with linear time complexity.

Autoencoders

Neural networks that learn compressed representations. Anomalies exhibit high reconstruction error, ideal for complex pattern detection in traffic.

LSTM Networks

Recurrent neural networks capturing temporal dependencies in network traffic. Effective for detecting behavioral anomalies over time sequences.

Feature Engineering

  • • Connection duration and packet counts
  • • Byte ratios and protocol distributions
  • • Flow statistics (inter-arrival times, burst patterns)
  • • DNS/HTTP metadata from Zeek logs
  • • Alert signatures from Suricata/Snort

Dockerized Deployment

The entire pipeline is containerized using Docker, ensuring consistent execution across development, testing, and production environments. This architecture enables:

  • ✓ One-command deployment across any Linux host or cloud instance
  • ✓ Isolated dependencies with no conflicts
  • ✓ Scalable microservices architecture
  • ✓ Easy integration with orchestration tools (Kubernetes, Docker Swarm)
Quick Start

# Clone the repository
git clone https://github.com/MukeshKumarSingh1/Real-Time-Network-Anomaly-Detection.git
cd Real-Time-Network-Anomaly-Detection

# Build and run with Docker Compose
docker-compose up -d

The Docker Compose configuration orchestrates all pipeline components, including packet capture services, IDS processors, ML inference engines, and alerting modules.

Use Cases & Applications

SOC IntegrationDeploy as a sensor within security operations centers to detect zero-day threats and insider activity.
Research & EducationPlatform for studying network behavior, testing detection algorithms, and cybersecurity training.
Cloud EnvironmentsMonitor east-west traffic within VPCs for anomalous lateral movement.
Critical InfrastructureProtect industrial networks with tailored detection for OT/IoT protocols.

📚 This is an overview of the Real-Time Network Anomaly Detection project.

Access the Full Repository