aiwaf 0.1.8.7__tar.gz → 0.1.8.9__tar.gz

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aiwaf
-Version: 0.1.8.7
+Version: 0.1.8.9
 Summary: AI-powered Web Application Firewall
 Home-page: https://github.com/aayushgauba/aiwaf
 Author: Aayush Gauba
@@ -27,6 +27,18 @@ Dynamic: requires-python
 
 ---
 
+## 🚀 Quick Installation
+
+```bash
+pip install aiwaf
+```
+
+**⚠️ Important:** Add `'aiwaf'` to your Django `INSTALLED_APPS` to avoid setup errors.
+
+**📋 Complete Setup Guide:** See [INSTALLATION.md](INSTALLATION.md) for detailed installation instructions and troubleshooting.
+
+---
+
 ## System Requirements
 
 No GPU needed—AI-WAF runs entirely on CPU with just Python 3.8+, Django 3.2+, a single vCPU and ~512 MB RAM for small sites; for moderate production traffic you can bump to 2–4 vCPUs and 2–4 GB RAM, offload the daily detect-and-train job to a worker, and rotate logs to keep memory use bounded.
@@ -83,7 +95,28 @@ aiwaf/
   - Submit forms faster than `AIWAF_MIN_FORM_TIME` seconds (default: 1 second)
 
 - **UUID Tampering Protection**  
-  Blocks guessed or invalid UUIDs that don’t resolve to real models.
+  Blocks guessed or invalid UUIDs that don't resolve to real models.
+
+- **Built-in Request Logger**  
+  Optional middleware logger that captures requests to CSV:
+  - **Automatic fallback** when main access logs unavailable
+  - **CSV format** for easy analysis and training
+  - **Captures response times** for better anomaly detection
+  - **Zero configuration** - works out of the box
+
+
+**Exempt Path & IP Awareness**
+
+**Exempt Paths:**
+AI‑WAF automatically exempts common login paths (`/admin/`, `/login/`, `/accounts/login/`, etc.) from all blocking mechanisms. You can add additional exempt paths in your Django `settings.py`:
+
+```python
+AIWAF_EXEMPT_PATHS = [
+    "/api/webhooks/",
+    "/health/",
+    "/special-endpoint/",
+]
+```
 
 
 **Exempt Path & IP Awareness**
@@ -213,6 +246,42 @@ AIWAF_CSV_DATA_DIR = "aiwaf_data"  # Directory for CSV files
 
 ---
 
+### Built-in Request Logger (Optional)
+
+Enable AI-WAF's built-in request logger as a fallback when main access logs aren't available:
+
+```python
+# Enable middleware logging
+AIWAF_MIDDLEWARE_LOGGING = True                    # Enable/disable logging
+AIWAF_MIDDLEWARE_LOG = "aiwaf_requests.log"        # Log file path
+AIWAF_MIDDLEWARE_CSV = True                        # Use CSV format (recommended)
+```
+
+**Then add middleware to MIDDLEWARE list:**
+
+```python
+MIDDLEWARE = [
+    # ... your existing middleware ...
+    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Add near the end
+]
+```
+
+**Manage middleware logging:**
+
+```bash
+python manage.py aiwaf_logging --status    # Check logging status
+python manage.py aiwaf_logging --enable    # Show setup instructions  
+python manage.py aiwaf_logging --clear     # Clear log files
+```
+
+**Benefits:**
+- **Automatic fallback** when `AIWAF_ACCESS_LOG` unavailable
+- **CSV format** with precise timestamps and response times
+- **Zero configuration** - trainer automatically detects and uses CSV logs
+- **Lightweight** - fails silently to avoid breaking your application
+
+---
+
 ### Optional (defaults shown)
 
 ```python
@@ -244,14 +313,40 @@ Add in **this** order to your `MIDDLEWARE` list:
 ```python
 MIDDLEWARE = [
     "aiwaf.middleware.IPAndKeywordBlockMiddleware",
-    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware", 
     "aiwaf.middleware.AIAnomalyMiddleware",
     "aiwaf.middleware.HoneypotTimingMiddleware",
     "aiwaf.middleware.UUIDTamperMiddleware",
     # ... other middleware ...
+    "aiwaf.middleware_logger.AIWAFLoggerMiddleware",  # Optional: Add if using built-in logger
 ]
 ```
 
+> **⚠️ Order matters!** AI-WAF protection middleware should come early. The logger middleware should come near the end to capture final response data.
+
+---
+
+##  Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+### What happens:
+1. Read access logs (incl. rotated or gzipped) **OR** AI-WAF middleware CSV logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+6. Remove any keywords associated with newly exempt paths
+
+**Note:** If main access log (`AIWAF_ACCESS_LOG`) is unavailable, trainer automatically falls back to AI-WAF middleware CSV logs.
+
+---
+
+## 🧠 How It Works
+```
+
 ---
 
 ##  Running Detection & Training

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/README.md

@@ -6,6 +6,18 @@
 
 ---
 
+## 🚀 Quick Installation
+
+```bash
+pip install aiwaf
+```
+
+**⚠️ Important:** Add `'aiwaf'` to your Django `INSTALLED_APPS` to avoid setup errors.
+
+**📋 Complete Setup Guide:** See [INSTALLATION.md](INSTALLATION.md) for detailed installation instructions and troubleshooting.
+
+---
+
 ## System Requirements
 
 No GPU needed—AI-WAF runs entirely on CPU with just Python 3.8+, Django 3.2+, a single vCPU and ~512 MB RAM for small sites; for moderate production traffic you can bump to 2–4 vCPUs and 2–4 GB RAM, offload the daily detect-and-train job to a worker, and rotate logs to keep memory use bounded.
@@ -62,7 +74,28 @@ aiwaf/
   - Submit forms faster than `AIWAF_MIN_FORM_TIME` seconds (default: 1 second)
 
 - **UUID Tampering Protection**  
-  Blocks guessed or invalid UUIDs that don’t resolve to real models.
+  Blocks guessed or invalid UUIDs that don't resolve to real models.
+
+- **Built-in Request Logger**  
+  Optional middleware logger that captures requests to CSV:
+  - **Automatic fallback** when main access logs unavailable
+  - **CSV format** for easy analysis and training
+  - **Captures response times** for better anomaly detection
+  - **Zero configuration** - works out of the box
+
+
+**Exempt Path & IP Awareness**
+
+**Exempt Paths:**
+AI‑WAF automatically exempts common login paths (`/admin/`, `/login/`, `/accounts/login/`, etc.) from all blocking mechanisms. You can add additional exempt paths in your Django `settings.py`:
+
+```python
+AIWAF_EXEMPT_PATHS = [
+    "/api/webhooks/",
+    "/health/",
+    "/special-endpoint/",
+]
+```
 
 
 **Exempt Path & IP Awareness**
@@ -192,6 +225,42 @@ AIWAF_CSV_DATA_DIR = "aiwaf_data"  # Directory for CSV files
 
 ---
 
+### Built-in Request Logger (Optional)
+
+Enable AI-WAF's built-in request logger as a fallback when main access logs aren't available:
+
+```python
+# Enable middleware logging
+AIWAF_MIDDLEWARE_LOGGING = True                    # Enable/disable logging
+AIWAF_MIDDLEWARE_LOG = "aiwaf_requests.log"        # Log file path
+AIWAF_MIDDLEWARE_CSV = True                        # Use CSV format (recommended)
+```
+
+**Then add middleware to MIDDLEWARE list:**
+
+```python
+MIDDLEWARE = [
+    # ... your existing middleware ...
+    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Add near the end
+]
+```
+
+**Manage middleware logging:**
+
+```bash
+python manage.py aiwaf_logging --status    # Check logging status
+python manage.py aiwaf_logging --enable    # Show setup instructions  
+python manage.py aiwaf_logging --clear     # Clear log files
+```
+
+**Benefits:**
+- **Automatic fallback** when `AIWAF_ACCESS_LOG` unavailable
+- **CSV format** with precise timestamps and response times
+- **Zero configuration** - trainer automatically detects and uses CSV logs
+- **Lightweight** - fails silently to avoid breaking your application
+
+---
+
 ### Optional (defaults shown)
 
 ```python
@@ -223,14 +292,40 @@ Add in **this** order to your `MIDDLEWARE` list:
 ```python
 MIDDLEWARE = [
     "aiwaf.middleware.IPAndKeywordBlockMiddleware",
-    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware", 
     "aiwaf.middleware.AIAnomalyMiddleware",
     "aiwaf.middleware.HoneypotTimingMiddleware",
     "aiwaf.middleware.UUIDTamperMiddleware",
     # ... other middleware ...
+    "aiwaf.middleware_logger.AIWAFLoggerMiddleware",  # Optional: Add if using built-in logger
 ]
 ```
 
+> **⚠️ Order matters!** AI-WAF protection middleware should come early. The logger middleware should come near the end to capture final response data.
+
+---
+
+##  Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+### What happens:
+1. Read access logs (incl. rotated or gzipped) **OR** AI-WAF middleware CSV logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+6. Remove any keywords associated with newly exempt paths
+
+**Note:** If main access log (`AIWAF_ACCESS_LOG`) is unavailable, trainer automatically falls back to AI-WAF middleware CSV logs.
+
+---
+
+## 🧠 How It Works
+```
+
 ---
 
 ##  Running Detection & Training

aiwaf-0.1.8.9/aiwaf/management/commands/aiwaf_logging.py

@@ -0,0 +1,166 @@
+from django.core.management.base import BaseCommand
+from django.conf import settings
+import os
+
+class Command(BaseCommand):
+    help = 'Manage AI-WAF middleware logging settings and view log status'
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--enable',
+            action='store_true',
+            help='Enable middleware logging (shows settings to add)'
+        )
+        parser.add_argument(
+            '--disable',
+            action='store_true',
+            help='Disable middleware logging (shows settings to remove)'
+        )
+        parser.add_argument(
+            '--status',
+            action='store_true',
+            help='Show current middleware logging status'
+        )
+        parser.add_argument(
+            '--clear',
+            action='store_true',
+            help='Clear/delete middleware log files'
+        )
+
+    def handle(self, *args, **options):
+        if options['enable']:
+            self._show_enable_instructions()
+        elif options['disable']:
+            self._show_disable_instructions()
+        elif options['clear']:
+            self._clear_logs()
+        else:
+            self._show_status()
+
+    def _show_status(self):
+        """Show current middleware logging configuration"""
+        self.stdout.write(self.style.HTTP_INFO("🔍 AI-WAF Middleware Logging Status"))
+        self.stdout.write("")
+        
+        # Check settings
+        logging_enabled = getattr(settings, 'AIWAF_MIDDLEWARE_LOGGING', False)
+        log_file = getattr(settings, 'AIWAF_MIDDLEWARE_LOG', 'aiwaf_requests.log')
+        csv_format = getattr(settings, 'AIWAF_MIDDLEWARE_CSV', True)
+        csv_file = log_file.replace('.log', '.csv') if csv_format else None
+        
+        # Status
+        status_color = self.style.SUCCESS if logging_enabled else self.style.WARNING
+        self.stdout.write(f"Status: {status_color('ENABLED' if logging_enabled else 'DISABLED')}")
+        self.stdout.write(f"Log File: {log_file}")
+        if csv_format:
+            self.stdout.write(f"CSV File: {csv_file}")
+        self.stdout.write(f"Format: {'CSV' if csv_format else 'Text'}")
+        self.stdout.write("")
+        
+        # File existence and sizes
+        if logging_enabled:
+            self.stdout.write("📁 Log Files:")
+            
+            if csv_format and csv_file:
+                if os.path.exists(csv_file):
+                    size = os.path.getsize(csv_file)
+                    lines = self._count_csv_lines(csv_file)
+                    self.stdout.write(f"  ✅ {csv_file} ({size:,} bytes, {lines:,} entries)")
+                else:
+                    self.stdout.write(f"  ❌ {csv_file} (not found)")
+            
+            if os.path.exists(log_file):
+                size = os.path.getsize(log_file)
+                self.stdout.write(f"  ✅ {log_file} ({size:,} bytes)")
+            else:
+                self.stdout.write(f"  ❌ {log_file} (not found)")
+        
+        # Middleware check
+        middleware_list = getattr(settings, 'MIDDLEWARE', [])
+        middleware_installed = 'aiwaf.middleware_logger.AIWAFLoggerMiddleware' in middleware_list
+        
+        self.stdout.write("")
+        middleware_color = self.style.SUCCESS if middleware_installed else self.style.ERROR
+        self.stdout.write(f"Middleware: {middleware_color('INSTALLED' if middleware_installed else 'NOT INSTALLED')}")
+        
+        if logging_enabled and not middleware_installed:
+            self.stdout.write(self.style.WARNING("⚠️  Logging is enabled but middleware is not installed!"))
+
+    def _show_enable_instructions(self):
+        """Show instructions for enabling middleware logging"""
+        self.stdout.write(self.style.SUCCESS("🚀 Enable AI-WAF Middleware Logging"))
+        self.stdout.write("")
+        self.stdout.write("Add these settings to your Django settings.py:")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("# Enable AI-WAF middleware logging"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_LOGGING = True"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_LOG = 'aiwaf_requests.log'  # Optional"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_CSV = True  # Optional (default: True)"))
+        self.stdout.write("")
+        self.stdout.write("Add middleware to MIDDLEWARE list (preferably near the end):")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("MIDDLEWARE = ["))
+        self.stdout.write(self.style.HTTP_INFO("    # ... your existing middleware ..."))
+        self.stdout.write(self.style.HTTP_INFO("    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',"))
+        self.stdout.write(self.style.HTTP_INFO("]"))
+        self.stdout.write("")
+        self.stdout.write("Benefits:")
+        self.stdout.write("  ✅ Fallback when main access logs unavailable")
+        self.stdout.write("  ✅ CSV format for easy analysis") 
+        self.stdout.write("  ✅ Automatic integration with AI-WAF trainer")
+        self.stdout.write("  ✅ Captures response times for better detection")
+
+    def _show_disable_instructions(self):
+        """Show instructions for disabling middleware logging"""
+        self.stdout.write(self.style.WARNING("⏹️  Disable AI-WAF Middleware Logging"))
+        self.stdout.write("")
+        self.stdout.write("To disable, update your Django settings.py:")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("# Disable AI-WAF middleware logging"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_LOGGING = False"))
+        self.stdout.write("")
+        self.stdout.write("And remove from MIDDLEWARE list:")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("MIDDLEWARE = ["))
+        self.stdout.write(self.style.HTTP_INFO("    # ... your existing middleware ..."))
+        self.stdout.write(self.style.HTTP_INFO("    # 'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Remove this line"))
+        self.stdout.write(self.style.HTTP_INFO("]"))
+
+    def _clear_logs(self):
+        """Clear/delete middleware log files"""
+        log_file = getattr(settings, 'AIWAF_MIDDLEWARE_LOG', 'aiwaf_requests.log')
+        csv_format = getattr(settings, 'AIWAF_MIDDLEWARE_CSV', True)
+        csv_file = log_file.replace('.log', '.csv') if csv_format else None
+        
+        files_deleted = 0
+        
+        # Delete CSV file
+        if csv_file and os.path.exists(csv_file):
+            try:
+                os.remove(csv_file)
+                self.stdout.write(self.style.SUCCESS(f"✅ Deleted {csv_file}"))
+                files_deleted += 1
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to delete {csv_file}: {e}"))
+        
+        # Delete text log file
+        if os.path.exists(log_file):
+            try:
+                os.remove(log_file)
+                self.stdout.write(self.style.SUCCESS(f"✅ Deleted {log_file}"))
+                files_deleted += 1
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to delete {log_file}: {e}"))
+        
+        if files_deleted == 0:
+            self.stdout.write(self.style.WARNING("ℹ️  No log files found to delete"))
+        else:
+            self.stdout.write(self.style.SUCCESS(f"🗑️  Deleted {files_deleted} log file(s)"))
+
+    def _count_csv_lines(self, csv_file):
+        """Count lines in CSV file (excluding header)"""
+        try:
+            with open(csv_file, 'r', encoding='utf-8') as f:
+                return sum(1 for line in f) - 1  # Subtract header
+        except:
+            return 0

aiwaf-0.1.8.9/aiwaf/middleware_logger.py

@@ -0,0 +1,160 @@
+# aiwaf/middleware_logger.py
+
+import os
+import csv
+import time
+from datetime import datetime
+from django.conf import settings
+from django.utils.deprecation import MiddlewareMixin
+from .utils import get_ip
+
+class AIWAFLoggerMiddleware(MiddlewareMixin):
+    """
+    Middleware that logs requests to a CSV file for AI-WAF training.
+    Acts as a fallback when main access logs are unavailable.
+    """
+    
+    def __init__(self, get_response):
+        super().__init__(get_response)
+        self.log_file = getattr(settings, "AIWAF_MIDDLEWARE_LOG", "aiwaf_requests.log")
+        self.csv_format = getattr(settings, "AIWAF_MIDDLEWARE_CSV", True)
+        self.log_enabled = getattr(settings, "AIWAF_MIDDLEWARE_LOGGING", False)
+        
+        # CSV file path (if using CSV format)
+        if self.csv_format and self.log_enabled:
+            self.csv_file = self.log_file.replace('.log', '.csv')
+            self._ensure_csv_header()
+    
+    def _ensure_csv_header(self):
+        """Ensure CSV file has proper header row"""
+        if not os.path.exists(self.csv_file):
+            os.makedirs(os.path.dirname(self.csv_file), exist_ok=True) if os.path.dirname(self.csv_file) else None
+            with open(self.csv_file, 'w', newline='', encoding='utf-8') as f:
+                writer = csv.writer(f)
+                writer.writerow([
+                    'timestamp', 'ip_address', 'method', 'path', 'status_code', 
+                    'response_time', 'user_agent', 'referer', 'content_length'
+                ])
+    
+    def process_request(self, request):
+        """Store request start time"""
+        request._aiwaf_start_time = time.time()
+        return None
+    
+    def process_response(self, request, response):
+        """Log the completed request"""
+        if not self.log_enabled:
+            return response
+            
+        # Calculate response time
+        start_time = getattr(request, '_aiwaf_start_time', time.time())
+        response_time = time.time() - start_time
+        
+        # Extract request data
+        log_data = {
+            'timestamp': datetime.now().strftime('%d/%b/%Y:%H:%M:%S +0000'),
+            'ip_address': get_ip(request),
+            'method': request.method,
+            'path': request.path,
+            'status_code': response.status_code,
+            'response_time': f"{response_time:.3f}",
+            'user_agent': request.META.get('HTTP_USER_AGENT', '-'),
+            'referer': request.META.get('HTTP_REFERER', '-'),
+            'content_length': response.get('Content-Length', '-')
+        }
+        
+        if self.csv_format:
+            self._log_to_csv(log_data)
+        else:
+            self._log_to_text(log_data)
+            
+        return response
+    
+    def _log_to_csv(self, data):
+        """Write log entry to CSV file"""
+        try:
+            with open(self.csv_file, 'a', newline='', encoding='utf-8') as f:
+                writer = csv.writer(f)
+                writer.writerow([
+                    data['timestamp'], data['ip_address'], data['method'],
+                    data['path'], data['status_code'], data['response_time'],
+                    data['user_agent'], data['referer'], data['content_length']
+                ])
+        except Exception as e:
+            # Fail silently to avoid breaking the application
+            pass
+    
+    def _log_to_text(self, data):
+        """Write log entry in common log format"""
+        try:
+            # Common Log Format with response time
+            log_line = f'{data["ip_address"]} - - [{data["timestamp"]}] "{data["method"]} {data["path"]} HTTP/1.1" {data["status_code"]} {data["content_length"]} "{data["referer"]}" "{data["user_agent"]}" response-time={data["response_time"]}\n'
+            
+            with open(self.log_file, 'a', encoding='utf-8') as f:
+                f.write(log_line)
+        except Exception as e:
+            # Fail silently to avoid breaking the application
+            pass
+
+
+class AIWAFCSVLogParser:
+    """
+    Parser for AI-WAF CSV logs that converts them to the format expected by trainer.py
+    """
+    
+    @staticmethod
+    def parse_csv_log(csv_file_path):
+        """
+        Parse CSV log file and return records in the format expected by trainer.py
+        Returns list of dictionaries with keys: ip, timestamp, path, status, referer, user_agent, response_time
+        """
+        records = []
+        
+        if not os.path.exists(csv_file_path):
+            return records
+        
+        try:
+            with open(csv_file_path, 'r', newline='', encoding='utf-8') as f:
+                reader = csv.DictReader(f)
+                for row in reader:
+                    try:
+                        # Convert timestamp to datetime object
+                        timestamp = datetime.strptime(row['timestamp'], '%d/%b/%Y:%H:%M:%S +0000')
+                        
+                        record = {
+                            'ip': row['ip_address'],
+                            'timestamp': timestamp,
+                            'path': row['path'],
+                            'status': row['status_code'],
+                            'referer': row['referer'],
+                            'user_agent': row['user_agent'],
+                            'response_time': float(row['response_time'])
+                        }
+                        records.append(record)
+                    except (ValueError, KeyError) as e:
+                        # Skip malformed rows
+                        continue
+        except Exception as e:
+            # Return empty list if file can't be read
+            pass
+            
+        return records
+    
+    @staticmethod 
+    def get_log_lines_for_trainer(csv_file_path):
+        """
+        Convert CSV log to format compatible with trainer.py's _read_all_logs()
+        Returns list of log line strings
+        """
+        records = AIWAFCSVLogParser.parse_csv_log(csv_file_path)
+        log_lines = []
+        
+        for record in records:
+            # Convert back to common log format that trainer.py expects
+            timestamp_str = record['timestamp'].strftime('%d/%b/%Y:%H:%M:%S +0000')
+            content_length = '-'  # We don't track this in our format
+            
+            log_line = f'{record["ip"]} - - [{timestamp_str}] "GET {record["path"]} HTTP/1.1" {record["status"]} {content_length} "{record["referer"]}" "{record["user_agent"]}" response-time={record["response_time"]:.3f}'
+            log_lines.append(log_line)
+            
+        return log_lines

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/aiwaf/storage.py

@@ -3,7 +3,18 @@ import numpy as np
 import pandas as pd
 from django.conf import settings
 from django.utils import timezone
-from .models import FeatureSample, BlacklistEntry, IPExemption, DynamicKeyword
+
+# Only import models if aiwaf is in INSTALLED_APPS
+try:
+    from django.apps import apps
+    if apps.is_installed('aiwaf'):
+        from .models import FeatureSample, BlacklistEntry, IPExemption, DynamicKeyword
+    else:
+        # Create dummy classes to avoid import errors
+        FeatureSample = BlacklistEntry = IPExemption = DynamicKeyword = None
+except (ImportError, RuntimeError):
+    # Handle cases where Django isn't fully initialized yet
+    FeatureSample = BlacklistEntry = IPExemption = DynamicKeyword = None
 
 # Configuration
 STORAGE_MODE = getattr(settings, "AIWAF_STORAGE_MODE", "models")  # "models" or "csv"
@@ -52,22 +63,25 @@ class CsvFeatureStore:
 class DbFeatureStore:
     @staticmethod
     def persist_rows(rows):
-        objs = []
-        for ip,pl,kw,rt,si,bc,t404,label in rows:
-            objs.append(FeatureSample(
-                ip=ip, path_len=pl, kw_hits=kw,
-                resp_time=rt, status_idx=si,
-                burst_count=bc, total_404=t404,
-                label=label
-            ))
-        FeatureSample.objects.bulk_create(objs, ignore_conflicts=True)
+        if FeatureSample is not None:
+            objs = []
+            for ip,pl,kw,rt,si,bc,t404,label in rows:
+                objs.append(FeatureSample(
+                    ip=ip, path_len=pl, kw_hits=kw,
+                    resp_time=rt, status_idx=si,
+                    burst_count=bc, total_404=t404,
+                    label=label
+                ))
+            FeatureSample.objects.bulk_create(objs, ignore_conflicts=True)
 
     @staticmethod
     def load_matrix():
-        qs = FeatureSample.objects.all().values_list(
-            "path_len","kw_hits","resp_time","status_idx","burst_count","total_404"
-        )
-        return np.array(list(qs), dtype=float)
+        if FeatureSample is not None:
+            qs = FeatureSample.objects.all().values_list(
+                "path_len","kw_hits","resp_time","status_idx","burst_count","total_404"
+            )
+            return np.array(list(qs), dtype=float)
+        return np.empty((0,6))
 
 def get_store():
     if getattr(settings, "AIWAF_FEATURE_STORE", "csv") == "db":
@@ -266,8 +280,12 @@ def get_blacklist_store():
     if STORAGE_MODE == "csv":
         return CsvBlacklistStore
     else:
-        # Return a wrapper for Django models
-        return ModelBlacklistStore
+        # Return a wrapper for Django models (only if models are available)
+        if BlacklistEntry is not None:
+            return ModelBlacklistStore
+        else:
+            # Fallback to CSV if models aren't available
+            return CsvBlacklistStore
 
 
 def get_exemption_store():
@@ -275,7 +293,10 @@ def get_exemption_store():
     if STORAGE_MODE == "csv":
         return CsvExemptionStore
     else:
-        return ModelExemptionStore
+        if IPExemption is not None:
+            return ModelExemptionStore
+        else:
+            return CsvExemptionStore
 
 
 def get_keyword_store():
@@ -283,7 +304,10 @@ def get_keyword_store():
     if STORAGE_MODE == "csv":
         return CsvKeywordStore
     else:
-        return ModelKeywordStore
+        if DynamicKeyword is not None:
+            return ModelKeywordStore
+        else:
+            return CsvKeywordStore
 
 
 # ============= Django Model Wrappers =============
@@ -293,19 +317,25 @@ class ModelBlacklistStore:
     
     @staticmethod
     def add_ip(ip_address, reason):
-        BlacklistEntry.objects.get_or_create(ip_address=ip_address, defaults={"reason": reason})
+        if BlacklistEntry is not None:
+            BlacklistEntry.objects.get_or_create(ip_address=ip_address, defaults={"reason": reason})
     
     @staticmethod
     def is_blocked(ip_address):
-        return BlacklistEntry.objects.filter(ip_address=ip_address).exists()
+        if BlacklistEntry is not None:
+            return BlacklistEntry.objects.filter(ip_address=ip_address).exists()
+        return False
     
     @staticmethod
     def get_all():
-        return list(BlacklistEntry.objects.values("ip_address", "reason", "created_at"))
+        if BlacklistEntry is not None:
+            return list(BlacklistEntry.objects.values("ip_address", "reason", "created_at"))
+        return []
     
     @staticmethod
     def remove_ip(ip_address):
-        BlacklistEntry.objects.filter(ip_address=ip_address).delete()
+        if BlacklistEntry is not None:
+            BlacklistEntry.objects.filter(ip_address=ip_address).delete()
 
 
 class ModelExemptionStore:
@@ -313,19 +343,25 @@ class ModelExemptionStore:
     
     @staticmethod
     def add_ip(ip_address, reason=""):
-        IPExemption.objects.get_or_create(ip_address=ip_address, defaults={"reason": reason})
+        if IPExemption is not None:
+            IPExemption.objects.get_or_create(ip_address=ip_address, defaults={"reason": reason})
     
     @staticmethod
     def is_exempted(ip_address):
-        return IPExemption.objects.filter(ip_address=ip_address).exists()
+        if IPExemption is not None:
+            return IPExemption.objects.filter(ip_address=ip_address).exists()
+        return False
     
     @staticmethod
     def get_all():
-        return list(IPExemption.objects.values("ip_address", "reason", "created_at"))
+        if IPExemption is not None:
+            return list(IPExemption.objects.values("ip_address", "reason", "created_at"))
+        return []
     
     @staticmethod
     def remove_ip(ip_address):
-        IPExemption.objects.filter(ip_address=ip_address).delete()
+        if IPExemption is not None:
+            IPExemption.objects.filter(ip_address=ip_address).delete()
 
 
 class ModelKeywordStore:
@@ -333,19 +369,24 @@ class ModelKeywordStore:
     
     @staticmethod
     def add_keyword(keyword, count=1):
-        obj, created = DynamicKeyword.objects.get_or_create(keyword=keyword, defaults={"count": count})
-        if not created:
-            obj.count += count
-            obj.save()
+        if DynamicKeyword is not None:
+            obj, created = DynamicKeyword.objects.get_or_create(keyword=keyword, defaults={"count": count})
+            if not created:
+                obj.count += count
+                obj.save()
     
     @staticmethod
     def get_top_keywords(limit=10):
-        return list(DynamicKeyword.objects.order_by("-count").values_list("keyword", flat=True)[:limit])
+        if DynamicKeyword is not None:
+            return list(DynamicKeyword.objects.order_by("-count").values_list("keyword", flat=True)[:limit])
+        return []
     
     @staticmethod
     def remove_keyword(keyword):
-        DynamicKeyword.objects.filter(keyword=keyword).delete()
+        if DynamicKeyword is not None:
+            DynamicKeyword.objects.filter(keyword=keyword).delete()
     
     @staticmethod
     def clear_all():
-        DynamicKeyword.objects.all().delete()
+        if DynamicKeyword is not None:
+            DynamicKeyword.objects.all().delete()

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/aiwaf/trainer.py

@@ -65,16 +65,31 @@ def remove_exempt_keywords() -> None:
 
 def _read_all_logs() -> list[str]:
     lines = []
+    
+    # First try to read from main access log
     if LOG_PATH and os.path.exists(LOG_PATH):
         with open(LOG_PATH, "r", errors="ignore") as f:
             lines.extend(f.readlines())
-    for p in sorted(glob.glob(f"{LOG_PATH}.*")):
-        opener = gzip.open if p.endswith(".gz") else open
-        try:
-            with opener(p, "rt", errors="ignore") as f:
-                lines.extend(f.readlines())
-        except OSError:
-            continue
+        for p in sorted(glob.glob(f"{LOG_PATH}.*")):
+            opener = gzip.open if p.endswith(".gz") else open
+            try:
+                with opener(p, "rt", errors="ignore") as f:
+                    lines.extend(f.readlines())
+            except OSError:
+                continue
+    
+    # If no lines found from main log, try AI-WAF middleware CSV log
+    if not lines:
+        middleware_csv = getattr(settings, "AIWAF_MIDDLEWARE_LOG", "aiwaf_requests.log").replace('.log', '.csv')
+        if os.path.exists(middleware_csv):
+            try:
+                from .middleware_logger import AIWAFCSVLogParser
+                csv_lines = AIWAFCSVLogParser.get_log_lines_for_trainer(middleware_csv)
+                lines.extend(csv_lines)
+                print(f"📋 Using AI-WAF middleware CSV log: {middleware_csv} ({len(csv_lines)} entries)")
+            except Exception as e:
+                print(f"⚠️  Failed to read middleware CSV log: {e}")
+    
     return lines
 
 

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/aiwaf.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aiwaf
-Version: 0.1.8.7
+Version: 0.1.8.9
 Summary: AI-powered Web Application Firewall
 Home-page: https://github.com/aayushgauba/aiwaf
 Author: Aayush Gauba
@@ -27,6 +27,18 @@ Dynamic: requires-python
 
 ---
 
+## 🚀 Quick Installation
+
+```bash
+pip install aiwaf
+```
+
+**⚠️ Important:** Add `'aiwaf'` to your Django `INSTALLED_APPS` to avoid setup errors.
+
+**📋 Complete Setup Guide:** See [INSTALLATION.md](INSTALLATION.md) for detailed installation instructions and troubleshooting.
+
+---
+
 ## System Requirements
 
 No GPU needed—AI-WAF runs entirely on CPU with just Python 3.8+, Django 3.2+, a single vCPU and ~512 MB RAM for small sites; for moderate production traffic you can bump to 2–4 vCPUs and 2–4 GB RAM, offload the daily detect-and-train job to a worker, and rotate logs to keep memory use bounded.
@@ -83,7 +95,28 @@ aiwaf/
   - Submit forms faster than `AIWAF_MIN_FORM_TIME` seconds (default: 1 second)
 
 - **UUID Tampering Protection**  
-  Blocks guessed or invalid UUIDs that don’t resolve to real models.
+  Blocks guessed or invalid UUIDs that don't resolve to real models.
+
+- **Built-in Request Logger**  
+  Optional middleware logger that captures requests to CSV:
+  - **Automatic fallback** when main access logs unavailable
+  - **CSV format** for easy analysis and training
+  - **Captures response times** for better anomaly detection
+  - **Zero configuration** - works out of the box
+
+
+**Exempt Path & IP Awareness**
+
+**Exempt Paths:**
+AI‑WAF automatically exempts common login paths (`/admin/`, `/login/`, `/accounts/login/`, etc.) from all blocking mechanisms. You can add additional exempt paths in your Django `settings.py`:
+
+```python
+AIWAF_EXEMPT_PATHS = [
+    "/api/webhooks/",
+    "/health/",
+    "/special-endpoint/",
+]
+```
 
 
 **Exempt Path & IP Awareness**
@@ -213,6 +246,42 @@ AIWAF_CSV_DATA_DIR = "aiwaf_data"  # Directory for CSV files
 
 ---
 
+### Built-in Request Logger (Optional)
+
+Enable AI-WAF's built-in request logger as a fallback when main access logs aren't available:
+
+```python
+# Enable middleware logging
+AIWAF_MIDDLEWARE_LOGGING = True                    # Enable/disable logging
+AIWAF_MIDDLEWARE_LOG = "aiwaf_requests.log"        # Log file path
+AIWAF_MIDDLEWARE_CSV = True                        # Use CSV format (recommended)
+```
+
+**Then add middleware to MIDDLEWARE list:**
+
+```python
+MIDDLEWARE = [
+    # ... your existing middleware ...
+    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Add near the end
+]
+```
+
+**Manage middleware logging:**
+
+```bash
+python manage.py aiwaf_logging --status    # Check logging status
+python manage.py aiwaf_logging --enable    # Show setup instructions  
+python manage.py aiwaf_logging --clear     # Clear log files
+```
+
+**Benefits:**
+- **Automatic fallback** when `AIWAF_ACCESS_LOG` unavailable
+- **CSV format** with precise timestamps and response times
+- **Zero configuration** - trainer automatically detects and uses CSV logs
+- **Lightweight** - fails silently to avoid breaking your application
+
+---
+
 ### Optional (defaults shown)
 
 ```python
@@ -244,14 +313,40 @@ Add in **this** order to your `MIDDLEWARE` list:
 ```python
 MIDDLEWARE = [
     "aiwaf.middleware.IPAndKeywordBlockMiddleware",
-    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware", 
     "aiwaf.middleware.AIAnomalyMiddleware",
     "aiwaf.middleware.HoneypotTimingMiddleware",
     "aiwaf.middleware.UUIDTamperMiddleware",
     # ... other middleware ...
+    "aiwaf.middleware_logger.AIWAFLoggerMiddleware",  # Optional: Add if using built-in logger
 ]
 ```
 
+> **⚠️ Order matters!** AI-WAF protection middleware should come early. The logger middleware should come near the end to capture final response data.
+
+---
+
+##  Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+### What happens:
+1. Read access logs (incl. rotated or gzipped) **OR** AI-WAF middleware CSV logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+6. Remove any keywords associated with newly exempt paths
+
+**Note:** If main access log (`AIWAF_ACCESS_LOG`) is unavailable, trainer automatically falls back to AI-WAF middleware CSV logs.
+
+---
+
+## 🧠 How It Works
+```
+
 ---
 
 ##  Running Detection & Training

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/aiwaf.egg-info/SOURCES.txt

@@ -7,6 +7,7 @@ aiwaf/apps.py
 aiwaf/blacklist_manager.py
 aiwaf/decorators.py
 aiwaf/middleware.py
+aiwaf/middleware_logger.py
 aiwaf/models.py
 aiwaf/storage.py
 aiwaf/trainer.py
@@ -19,6 +20,7 @@ aiwaf.egg-info/top_level.txt
 aiwaf/management/__init__.py
 aiwaf/management/commands/__init__.py
 aiwaf/management/commands/add_ipexemption.py
+aiwaf/management/commands/aiwaf_logging.py
 aiwaf/management/commands/aiwaf_reset.py
 aiwaf/management/commands/detect_and_train.py
 aiwaf/resources/model.pkl

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "aiwaf"
-version = "0.1.8.7"
+version = "0.1.8.9"
 description = "AI-powered Web Application Firewall"
 readme = "README.md"
 requires-python = ">=3.8"

{aiwaf-0.1.8.7 → aiwaf-0.1.8.9}/setup.py

@@ -9,7 +9,7 @@ long_description = (HERE / "README.md").read_text(encoding="utf-8")
 
 setup(
     name="aiwaf",
-    version="0.1.8.7",
+    version="0.1.8.9",
     description="AI‑driven, self‑learning Web Application Firewall for Django",
     long_description=long_description,
     long_description_content_type="text/markdown",