aiwaf 0.1.8.6__tar.gz → 0.1.8.8__tar.gz

{aiwaf-0.1.8.6 → aiwaf-0.1.8.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aiwaf
-Version: 0.1.8.6
+Version: 0.1.8.8
 Summary: AI-powered Web Application Firewall
 Home-page: https://github.com/aayushgauba/aiwaf
 Author: Aayush Gauba
@@ -83,7 +83,28 @@ aiwaf/
   - Submit forms faster than `AIWAF_MIN_FORM_TIME` seconds (default: 1 second)
 
 - **UUID Tampering Protection**  
-  Blocks guessed or invalid UUIDs that don’t resolve to real models.
+  Blocks guessed or invalid UUIDs that don't resolve to real models.
+
+- **Built-in Request Logger**  
+  Optional middleware logger that captures requests to CSV:
+  - **Automatic fallback** when main access logs unavailable
+  - **CSV format** for easy analysis and training
+  - **Captures response times** for better anomaly detection
+  - **Zero configuration** - works out of the box
+
+
+**Exempt Path & IP Awareness**
+
+**Exempt Paths:**
+AI‑WAF automatically exempts common login paths (`/admin/`, `/login/`, `/accounts/login/`, etc.) from all blocking mechanisms. You can add additional exempt paths in your Django `settings.py`:
+
+```python
+AIWAF_EXEMPT_PATHS = [
+    "/api/webhooks/",
+    "/health/",
+    "/special-endpoint/",
+]
+```
 
 
 **Exempt Path & IP Awareness**
@@ -188,6 +209,67 @@ AIWAF_ACCESS_LOG = "/var/log/nginx/access.log"
 
 ---
 
+### Storage Configuration
+
+**Choose storage backend:**
+
+```python
+# Use Django models (default) - requires database tables
+AIWAF_STORAGE_MODE = "models"
+
+# OR use CSV files - no database required
+AIWAF_STORAGE_MODE = "csv"
+AIWAF_CSV_DATA_DIR = "aiwaf_data"  # Directory for CSV files
+```
+
+**CSV Mode Features:**
+- No database migrations required
+- Files stored in `aiwaf_data/` directory:
+  - `blacklist.csv` - Blocked IP addresses
+  - `exemptions.csv` - Exempt IP addresses  
+  - `keywords.csv` - Dynamic keywords
+  - `access_samples.csv` - Feature samples for ML training
+- Perfect for lightweight deployments or when you prefer file-based storage
+- Management commands work identically in both modes
+
+---
+
+### Built-in Request Logger (Optional)
+
+Enable AI-WAF's built-in request logger as a fallback when main access logs aren't available:
+
+```python
+# Enable middleware logging
+AIWAF_MIDDLEWARE_LOGGING = True                    # Enable/disable logging
+AIWAF_MIDDLEWARE_LOG = "aiwaf_requests.log"        # Log file path
+AIWAF_MIDDLEWARE_CSV = True                        # Use CSV format (recommended)
+```
+
+**Then add middleware to MIDDLEWARE list:**
+
+```python
+MIDDLEWARE = [
+    # ... your existing middleware ...
+    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Add near the end
+]
+```
+
+**Manage middleware logging:**
+
+```bash
+python manage.py aiwaf_logging --status    # Check logging status
+python manage.py aiwaf_logging --enable    # Show setup instructions  
+python manage.py aiwaf_logging --clear     # Clear log files
+```
+
+**Benefits:**
+- **Automatic fallback** when `AIWAF_ACCESS_LOG` unavailable
+- **CSV format** with precise timestamps and response times
+- **Zero configuration** - trainer automatically detects and uses CSV logs
+- **Lightweight** - fails silently to avoid breaking your application
+
+---
+
 ### Optional (defaults shown)
 
 ```python
@@ -219,14 +301,40 @@ Add in **this** order to your `MIDDLEWARE` list:
 ```python
 MIDDLEWARE = [
     "aiwaf.middleware.IPAndKeywordBlockMiddleware",
-    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware", 
     "aiwaf.middleware.AIAnomalyMiddleware",
     "aiwaf.middleware.HoneypotTimingMiddleware",
     "aiwaf.middleware.UUIDTamperMiddleware",
     # ... other middleware ...
+    "aiwaf.middleware_logger.AIWAFLoggerMiddleware",  # Optional: Add if using built-in logger
 ]
 ```
 
+> **⚠️ Order matters!** AI-WAF protection middleware should come early. The logger middleware should come near the end to capture final response data.
+
+---
+
+##  Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+### What happens:
+1. Read access logs (incl. rotated or gzipped) **OR** AI-WAF middleware CSV logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+6. Remove any keywords associated with newly exempt paths
+
+**Note:** If main access log (`AIWAF_ACCESS_LOG`) is unavailable, trainer automatically falls back to AI-WAF middleware CSV logs.
+
+---
+
+## 🧠 How It Works
+```
+
 ---
 
 ##  Running Detection & Training

{aiwaf-0.1.8.6 → aiwaf-0.1.8.8}/README.md

@@ -62,7 +62,28 @@ aiwaf/
   - Submit forms faster than `AIWAF_MIN_FORM_TIME` seconds (default: 1 second)
 
 - **UUID Tampering Protection**  
-  Blocks guessed or invalid UUIDs that don’t resolve to real models.
+  Blocks guessed or invalid UUIDs that don't resolve to real models.
+
+- **Built-in Request Logger**  
+  Optional middleware logger that captures requests to CSV:
+  - **Automatic fallback** when main access logs unavailable
+  - **CSV format** for easy analysis and training
+  - **Captures response times** for better anomaly detection
+  - **Zero configuration** - works out of the box
+
+
+**Exempt Path & IP Awareness**
+
+**Exempt Paths:**
+AI‑WAF automatically exempts common login paths (`/admin/`, `/login/`, `/accounts/login/`, etc.) from all blocking mechanisms. You can add additional exempt paths in your Django `settings.py`:
+
+```python
+AIWAF_EXEMPT_PATHS = [
+    "/api/webhooks/",
+    "/health/",
+    "/special-endpoint/",
+]
+```
 
 
 **Exempt Path & IP Awareness**
@@ -167,6 +188,67 @@ AIWAF_ACCESS_LOG = "/var/log/nginx/access.log"
 
 ---
 
+### Storage Configuration
+
+**Choose storage backend:**
+
+```python
+# Use Django models (default) - requires database tables
+AIWAF_STORAGE_MODE = "models"
+
+# OR use CSV files - no database required
+AIWAF_STORAGE_MODE = "csv"
+AIWAF_CSV_DATA_DIR = "aiwaf_data"  # Directory for CSV files
+```
+
+**CSV Mode Features:**
+- No database migrations required
+- Files stored in `aiwaf_data/` directory:
+  - `blacklist.csv` - Blocked IP addresses
+  - `exemptions.csv` - Exempt IP addresses  
+  - `keywords.csv` - Dynamic keywords
+  - `access_samples.csv` - Feature samples for ML training
+- Perfect for lightweight deployments or when you prefer file-based storage
+- Management commands work identically in both modes
+
+---
+
+### Built-in Request Logger (Optional)
+
+Enable AI-WAF's built-in request logger as a fallback when main access logs aren't available:
+
+```python
+# Enable middleware logging
+AIWAF_MIDDLEWARE_LOGGING = True                    # Enable/disable logging
+AIWAF_MIDDLEWARE_LOG = "aiwaf_requests.log"        # Log file path
+AIWAF_MIDDLEWARE_CSV = True                        # Use CSV format (recommended)
+```
+
+**Then add middleware to MIDDLEWARE list:**
+
+```python
+MIDDLEWARE = [
+    # ... your existing middleware ...
+    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Add near the end
+]
+```
+
+**Manage middleware logging:**
+
+```bash
+python manage.py aiwaf_logging --status    # Check logging status
+python manage.py aiwaf_logging --enable    # Show setup instructions  
+python manage.py aiwaf_logging --clear     # Clear log files
+```
+
+**Benefits:**
+- **Automatic fallback** when `AIWAF_ACCESS_LOG` unavailable
+- **CSV format** with precise timestamps and response times
+- **Zero configuration** - trainer automatically detects and uses CSV logs
+- **Lightweight** - fails silently to avoid breaking your application
+
+---
+
 ### Optional (defaults shown)
 
 ```python
@@ -198,14 +280,40 @@ Add in **this** order to your `MIDDLEWARE` list:
 ```python
 MIDDLEWARE = [
     "aiwaf.middleware.IPAndKeywordBlockMiddleware",
-    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware", 
     "aiwaf.middleware.AIAnomalyMiddleware",
     "aiwaf.middleware.HoneypotTimingMiddleware",
     "aiwaf.middleware.UUIDTamperMiddleware",
     # ... other middleware ...
+    "aiwaf.middleware_logger.AIWAFLoggerMiddleware",  # Optional: Add if using built-in logger
 ]
 ```
 
+> **⚠️ Order matters!** AI-WAF protection middleware should come early. The logger middleware should come near the end to capture final response data.
+
+---
+
+##  Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+### What happens:
+1. Read access logs (incl. rotated or gzipped) **OR** AI-WAF middleware CSV logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+6. Remove any keywords associated with newly exempt paths
+
+**Note:** If main access log (`AIWAF_ACCESS_LOG`) is unavailable, trainer automatically falls back to AI-WAF middleware CSV logs.
+
+---
+
+## 🧠 How It Works
+```
+
 ---
 
 ##  Running Detection & Training

aiwaf-0.1.8.8/aiwaf/blacklist_manager.py

@@ -0,0 +1,24 @@
+# aiwaf/blacklist_manager.py
+
+from .storage import get_blacklist_store
+
+class BlacklistManager:
+    @staticmethod
+    def block(ip, reason):
+        store = get_blacklist_store()
+        store.add_ip(ip, reason)
+
+    @staticmethod
+    def is_blocked(ip):
+        store = get_blacklist_store()
+        return store.is_blocked(ip)
+
+    @staticmethod
+    def all_blocked():
+        store = get_blacklist_store()
+        return store.get_all()
+    
+    @staticmethod
+    def unblock(ip):
+        store = get_blacklist_store()
+        store.remove_ip(ip)

{aiwaf-0.1.8.6 → aiwaf-0.1.8.8}/aiwaf/management/commands/add_ipexemption.py

@@ -1,5 +1,5 @@
 from django.core.management.base import BaseCommand, CommandError
-from aiwaf.models import IPExemption
+from aiwaf.storage import get_exemption_store
 
 class Command(BaseCommand):
     help = 'Add an IP address to the IPExemption list (prevents blacklisting)'
@@ -11,12 +11,13 @@ class Command(BaseCommand):
     def handle(self, *args, **options):
         ip = options['ip']
         reason = options['reason']
-        obj, created = IPExemption.objects.get_or_create(ip_address=ip, defaults={'reason': reason})
-        if not created:
+        
+        store = get_exemption_store()
+        
+        if store.is_exempted(ip):
             self.stdout.write(self.style.WARNING(f'IP {ip} is already exempted.'))
         else:
+            store.add_ip(ip, reason)
             self.stdout.write(self.style.SUCCESS(f'IP {ip} added to exemption list.'))
-        if reason:
-            obj.reason = reason
-            obj.save()
-            self.stdout.write(self.style.SUCCESS(f'Reason set to: {reason}'))
+            if reason:
+                self.stdout.write(self.style.SUCCESS(f'Reason: {reason}'))

aiwaf-0.1.8.8/aiwaf/management/commands/aiwaf_logging.py

@@ -0,0 +1,166 @@
+from django.core.management.base import BaseCommand
+from django.conf import settings
+import os
+
+class Command(BaseCommand):
+    help = 'Manage AI-WAF middleware logging settings and view log status'
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--enable',
+            action='store_true',
+            help='Enable middleware logging (shows settings to add)'
+        )
+        parser.add_argument(
+            '--disable',
+            action='store_true',
+            help='Disable middleware logging (shows settings to remove)'
+        )
+        parser.add_argument(
+            '--status',
+            action='store_true',
+            help='Show current middleware logging status'
+        )
+        parser.add_argument(
+            '--clear',
+            action='store_true',
+            help='Clear/delete middleware log files'
+        )
+
+    def handle(self, *args, **options):
+        if options['enable']:
+            self._show_enable_instructions()
+        elif options['disable']:
+            self._show_disable_instructions()
+        elif options['clear']:
+            self._clear_logs()
+        else:
+            self._show_status()
+
+    def _show_status(self):
+        """Show current middleware logging configuration"""
+        self.stdout.write(self.style.HTTP_INFO("🔍 AI-WAF Middleware Logging Status"))
+        self.stdout.write("")
+        
+        # Check settings
+        logging_enabled = getattr(settings, 'AIWAF_MIDDLEWARE_LOGGING', False)
+        log_file = getattr(settings, 'AIWAF_MIDDLEWARE_LOG', 'aiwaf_requests.log')
+        csv_format = getattr(settings, 'AIWAF_MIDDLEWARE_CSV', True)
+        csv_file = log_file.replace('.log', '.csv') if csv_format else None
+        
+        # Status
+        status_color = self.style.SUCCESS if logging_enabled else self.style.WARNING
+        self.stdout.write(f"Status: {status_color('ENABLED' if logging_enabled else 'DISABLED')}")
+        self.stdout.write(f"Log File: {log_file}")
+        if csv_format:
+            self.stdout.write(f"CSV File: {csv_file}")
+        self.stdout.write(f"Format: {'CSV' if csv_format else 'Text'}")
+        self.stdout.write("")
+        
+        # File existence and sizes
+        if logging_enabled:
+            self.stdout.write("📁 Log Files:")
+            
+            if csv_format and csv_file:
+                if os.path.exists(csv_file):
+                    size = os.path.getsize(csv_file)
+                    lines = self._count_csv_lines(csv_file)
+                    self.stdout.write(f"  ✅ {csv_file} ({size:,} bytes, {lines:,} entries)")
+                else:
+                    self.stdout.write(f"  ❌ {csv_file} (not found)")
+            
+            if os.path.exists(log_file):
+                size = os.path.getsize(log_file)
+                self.stdout.write(f"  ✅ {log_file} ({size:,} bytes)")
+            else:
+                self.stdout.write(f"  ❌ {log_file} (not found)")
+        
+        # Middleware check
+        middleware_list = getattr(settings, 'MIDDLEWARE', [])
+        middleware_installed = 'aiwaf.middleware_logger.AIWAFLoggerMiddleware' in middleware_list
+        
+        self.stdout.write("")
+        middleware_color = self.style.SUCCESS if middleware_installed else self.style.ERROR
+        self.stdout.write(f"Middleware: {middleware_color('INSTALLED' if middleware_installed else 'NOT INSTALLED')}")
+        
+        if logging_enabled and not middleware_installed:
+            self.stdout.write(self.style.WARNING("⚠️  Logging is enabled but middleware is not installed!"))
+
+    def _show_enable_instructions(self):
+        """Show instructions for enabling middleware logging"""
+        self.stdout.write(self.style.SUCCESS("🚀 Enable AI-WAF Middleware Logging"))
+        self.stdout.write("")
+        self.stdout.write("Add these settings to your Django settings.py:")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("# Enable AI-WAF middleware logging"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_LOGGING = True"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_LOG = 'aiwaf_requests.log'  # Optional"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_CSV = True  # Optional (default: True)"))
+        self.stdout.write("")
+        self.stdout.write("Add middleware to MIDDLEWARE list (preferably near the end):")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("MIDDLEWARE = ["))
+        self.stdout.write(self.style.HTTP_INFO("    # ... your existing middleware ..."))
+        self.stdout.write(self.style.HTTP_INFO("    'aiwaf.middleware_logger.AIWAFLoggerMiddleware',"))
+        self.stdout.write(self.style.HTTP_INFO("]"))
+        self.stdout.write("")
+        self.stdout.write("Benefits:")
+        self.stdout.write("  ✅ Fallback when main access logs unavailable")
+        self.stdout.write("  ✅ CSV format for easy analysis") 
+        self.stdout.write("  ✅ Automatic integration with AI-WAF trainer")
+        self.stdout.write("  ✅ Captures response times for better detection")
+
+    def _show_disable_instructions(self):
+        """Show instructions for disabling middleware logging"""
+        self.stdout.write(self.style.WARNING("⏹️  Disable AI-WAF Middleware Logging"))
+        self.stdout.write("")
+        self.stdout.write("To disable, update your Django settings.py:")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("# Disable AI-WAF middleware logging"))
+        self.stdout.write(self.style.HTTP_INFO("AIWAF_MIDDLEWARE_LOGGING = False"))
+        self.stdout.write("")
+        self.stdout.write("And remove from MIDDLEWARE list:")
+        self.stdout.write("")
+        self.stdout.write(self.style.HTTP_INFO("MIDDLEWARE = ["))
+        self.stdout.write(self.style.HTTP_INFO("    # ... your existing middleware ..."))
+        self.stdout.write(self.style.HTTP_INFO("    # 'aiwaf.middleware_logger.AIWAFLoggerMiddleware',  # Remove this line"))
+        self.stdout.write(self.style.HTTP_INFO("]"))
+
+    def _clear_logs(self):
+        """Clear/delete middleware log files"""
+        log_file = getattr(settings, 'AIWAF_MIDDLEWARE_LOG', 'aiwaf_requests.log')
+        csv_format = getattr(settings, 'AIWAF_MIDDLEWARE_CSV', True)
+        csv_file = log_file.replace('.log', '.csv') if csv_format else None
+        
+        files_deleted = 0
+        
+        # Delete CSV file
+        if csv_file and os.path.exists(csv_file):
+            try:
+                os.remove(csv_file)
+                self.stdout.write(self.style.SUCCESS(f"✅ Deleted {csv_file}"))
+                files_deleted += 1
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to delete {csv_file}: {e}"))
+        
+        # Delete text log file
+        if os.path.exists(log_file):
+            try:
+                os.remove(log_file)
+                self.stdout.write(self.style.SUCCESS(f"✅ Deleted {log_file}"))
+                files_deleted += 1
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to delete {log_file}: {e}"))
+        
+        if files_deleted == 0:
+            self.stdout.write(self.style.WARNING("ℹ️  No log files found to delete"))
+        else:
+            self.stdout.write(self.style.SUCCESS(f"🗑️  Deleted {files_deleted} log file(s)"))
+
+    def _count_csv_lines(self, csv_file):
+        """Count lines in CSV file (excluding header)"""
+        try:
+            with open(csv_file, 'r', encoding='utf-8') as f:
+                return sum(1 for line in f) - 1  # Subtract header
+        except:
+            return 0

{aiwaf-0.1.8.6 → aiwaf-0.1.8.8}/aiwaf/management/commands/aiwaf_reset.py

@@ -1,5 +1,5 @@
 from django.core.management.base import BaseCommand
-from aiwaf.models import BlacklistEntry, IPExemption
+from aiwaf.storage import get_blacklist_store, get_exemption_store
 
 class Command(BaseCommand):
     help = 'Reset AI-WAF by clearing all blacklist and exemption (whitelist) entries'
@@ -26,9 +26,12 @@ class Command(BaseCommand):
         exemptions_only = options['exemptions_only']
         confirm = options['confirm']
         
+        blacklist_store = get_blacklist_store()
+        exemption_store = get_exemption_store()
+        
         # Count current entries
-        blacklist_count = BlacklistEntry.objects.count()
-        exemption_count = IPExemption.objects.count()
+        blacklist_count = len(blacklist_store.get_all())
+        exemption_count = len(exemption_store.get_all())
         
         if blacklist_only and exemptions_only:
             self.stdout.write(self.style.ERROR('Cannot use both --blacklist-only and --exemptions-only flags'))
@@ -61,10 +64,18 @@ class Command(BaseCommand):
         deleted_counts = {'blacklist': 0, 'exemptions': 0}
         
         if clear_blacklist:
-            deleted_counts['blacklist'], _ = BlacklistEntry.objects.all().delete()
+            # Clear blacklist entries
+            blacklist_entries = blacklist_store.get_all()
+            for entry in blacklist_entries:
+                blacklist_store.remove_ip(entry['ip_address'])
+            deleted_counts['blacklist'] = len(blacklist_entries)
         
         if clear_exemptions:
-            deleted_counts['exemptions'], _ = IPExemption.objects.all().delete()
+            # Clear exemption entries
+            exemption_entries = exemption_store.get_all()
+            for entry in exemption_entries:
+                exemption_store.remove_ip(entry['ip_address'])
+            deleted_counts['exemptions'] = len(exemption_entries)
         
         # Report results
         if clear_blacklist and clear_exemptions:

{aiwaf-0.1.8.6 → aiwaf-0.1.8.8}/aiwaf/middleware.py

@@ -16,8 +16,9 @@ from django.apps import apps
 from django.urls import get_resolver
 from .trainer import STATIC_KW, STATUS_IDX, path_exists_in_django
 from .blacklist_manager import BlacklistManager
-from .models import DynamicKeyword, IPExemption
+from .models import IPExemption
 from .utils import is_exempt, get_ip, is_ip_exempted
+from .storage import get_keyword_store
 
 MODEL_PATH = getattr(
     settings,
@@ -74,15 +75,14 @@ class IPAndKeywordBlockMiddleware:
             return self.get_response(request)
         if BlacklistManager.is_blocked(ip):
             return JsonResponse({"error": "blocked"}, status=403)
+        
+        keyword_store = get_keyword_store()
         segments = [seg for seg in re.split(r"\W+", path) if len(seg) > 3]
+        
         for seg in segments:
-            obj, _ = DynamicKeyword.objects.get_or_create(keyword=seg)
-            DynamicKeyword.objects.filter(pk=obj.pk).update(count=F("count") + 1)
-        dynamic_top = list(
-            DynamicKeyword.objects
-            .order_by("-count")
-            .values_list("keyword", flat=True)[: getattr(settings, "AIWAF_DYNAMIC_TOP_N", 10)]
-        )
+            keyword_store.add_keyword(seg)
+            
+        dynamic_top = keyword_store.get_top_keywords(getattr(settings, "AIWAF_DYNAMIC_TOP_N", 10))
         all_kw = set(STATIC_KW) | set(dynamic_top)
         suspicious_kw = {
             kw for kw in all_kw
@@ -172,10 +172,11 @@ class AIAnomalyMiddleware(MiddlewareMixin):
         data.append((now, request.path, response.status_code, resp_time))
         data = [d for d in data if now - d[0] < self.WINDOW]
         cache.set(key, data, timeout=self.WINDOW)
+        
+        keyword_store = get_keyword_store()
         for seg in re.split(r"\W+", request.path.lower()):
             if len(seg) > 3:
-                obj, _ = DynamicKeyword.objects.get_or_create(keyword=seg)
-                DynamicKeyword.objects.filter(pk=obj.pk).update(count=F("count") + 1)
+                keyword_store.add_keyword(seg)
 
         return response