aiwaf 0.1.9.1.9__tar.gz → 0.1.9.2.1__tar.gz

{aiwaf-0.1.9.1.9 → aiwaf-0.1.9.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aiwaf
-Version: 0.1.9.1.9
+Version: 0.1.9.2.1
 Summary: AI-powered Web Application Firewall
 Home-page: https://github.com/aayushgauba/aiwaf
 Author: Aayush Gauba
@@ -25,7 +25,13 @@ Dynamic: requires-python
 # AI‑WAF
 
 > A self‑learning, Django‑friendly Web Application Firewall  
-> with rate‑limiting, anomaly detection, honeypots, UUID‑tamper protection, dynamic keyword extraction, file‑extension probing detection, exempt path awareness, and daily retraining.
+> with **enhanced context-aware protection**, rate‑limiting, anomaly detection, honeypots, UUID‑tamper protection, **smart keyword learning**, file‑extension probing detection, exempt path awareness, and daily retraining.
+
+**🆕 Latest Enhancements:**
+- ✅ **Smart Keyword Filtering** - Prevents blocking legitimate pages like `/profile/`
+- ✅ **Granular Reset Commands** - Clear specific data types (`--blacklist`, `--keywords`, `--exemptions`)
+- ✅ **Context-Aware Learning** - Only learns from suspicious requests, not legitimate site functionality
+- ✅ **Enhanced Configuration** - `AIWAF_ALLOWED_PATH_KEYWORDS` and `AIWAF_EXEMPT_KEYWORDS`
 
 ---
 
@@ -88,9 +94,19 @@ aiwaf/
   - Burst count  
   - Total 404s  
 
-- **Dynamic Keyword Extraction & Cleanup**  
-  - Every retrain adds top 10 keyword segments from 4xx/5xx paths  
-  - **If a path is added to `AIWAF_EXEMPT_PATHS`, its keywords are automatically removed from the database**
+- **Enhanced Dynamic Keyword Learning with Django Route Protection**  
+  - **Smart Context-Aware Learning**: Only learns keywords from suspicious requests on non-existent paths
+  - **Automatic Django Route Extraction**: Automatically excludes keywords from:
+    - Valid Django URL patterns (`/profile/`, `/admin/`, `/api/`, etc.)
+    - Django app names and model names (users, posts, categories)
+    - View function names and URL namespaces
+  - **Unified Logic**: Both trainer and middleware use identical legitimate keyword detection
+  - **Configuration Options**: 
+    - `AIWAF_ALLOWED_PATH_KEYWORDS` - Explicitly allow certain keywords in legitimate paths
+    - `AIWAF_EXEMPT_KEYWORDS` - Keywords that should never trigger blocking
+  - **Automatic Cleanup**: Keywords from `AIWAF_EXEMPT_PATHS` are automatically removed from the database
+  - **False Positive Prevention**: Stops learning legitimate site functionality as "malicious"
+  - **Inherent Malicious Detection**: Middleware also blocks obviously malicious keywords (`hack`, `exploit`, `attack`) even if not yet learned
 
 - **File‑Extension Probing Detection**  
   Tracks repeated 404s on common extensions (e.g. `.php`, `.asp`) and blocks IPs.
@@ -196,20 +212,44 @@ python manage.py add_ipexemption <ip-address> --reason "optional reason"
 
 ### Resetting AI-WAF
 
-Clear all blacklist and exemption entries:
+The `aiwaf_reset` command provides **granular control** for clearing different types of data:
 
 ```bash
-# Clear everything (with confirmation prompt)
+# Clear everything (default - backward compatible)
 python manage.py aiwaf_reset
 
-# Clear everything without confirmation
+# Clear everything without confirmation prompt
 python manage.py aiwaf_reset --confirm
 
-# Clear only blacklist entries
-python manage.py aiwaf_reset --blacklist-only
+# 🆕 GRANULAR CONTROL - Clear specific data types
+python manage.py aiwaf_reset --blacklist      # Clear only blocked IPs
+python manage.py aiwaf_reset --exemptions     # Clear only exempted IPs  
+python manage.py aiwaf_reset --keywords       # Clear only learned keywords
+
+# 🔧 COMBINE OPTIONS - Mix and match as needed
+python manage.py aiwaf_reset --blacklist --keywords      # Keep exemptions
+python manage.py aiwaf_reset --exemptions --keywords     # Keep blacklist
+python manage.py aiwaf_reset --blacklist --exemptions    # Keep keywords
+
+# 🚀 COMMON USE CASES
+# Fix false positive keywords (like "profile" blocking legitimate pages)
+python manage.py aiwaf_reset --keywords --confirm
+python manage.py detect_and_train  # Retrain with enhanced filtering
+
+# Clear blocked IPs but preserve exemptions and learning
+python manage.py aiwaf_reset --blacklist --confirm
+
+# Legacy support (still works for backward compatibility)
+python manage.py aiwaf_reset --blacklist-only    # Legacy: blacklist only
+python manage.py aiwaf_reset --exemptions-only   # Legacy: exemptions only
+```
 
-# Clear only exemption entries  
-python manage.py aiwaf_reset --exemptions-only
+**Enhanced Feedback:**
+```bash
+$ python manage.py aiwaf_reset --keywords
+🔧 AI-WAF Reset: Clear 15 learned keywords
+Are you sure you want to proceed? [y/N]: y
+✅ Reset complete: Deleted 15 learned keywords
 ```
 
 ### Checking Dependencies
@@ -482,6 +522,21 @@ AIWAF_EXEMPT_PATHS = [          # optional but highly recommended
     "/media/",
     "/health/",
 ]
+
+# 🆕 ENHANCED KEYWORD FILTERING OPTIONS
+AIWAF_ALLOWED_PATH_KEYWORDS = [  # Keywords allowed in legitimate paths
+    "profile", "user", "account", "settings", "dashboard",
+    "admin", "api", "auth", "search", "contact", "about",
+    # Add your site-specific legitimate keywords
+    "buddycraft", "sc2", "starcraft",  # Example: gaming site keywords
+]
+
+AIWAF_EXEMPT_KEYWORDS = [        # Keywords that never trigger blocking
+    "api", "webhook", "health", "static", "media",
+    "upload", "download", "backup", "profile"
+]
+
+AIWAF_DYNAMIC_TOP_N = 10        # Number of dynamic keywords to learn (default: 10)
 ```
 
 > **Note:** You no longer need to define `AIWAF_MALICIOUS_KEYWORDS` or `AIWAF_STATUS_CODES` — they evolve dynamically.
@@ -680,6 +735,65 @@ python manage.py detect_and_train
 
 ---
 
+## 🔧 Troubleshooting
+
+### Legitimate Pages Being Blocked
+
+**Problem**: Users can't access legitimate pages like `/en/profile/` due to keyword blocking.
+
+**Cause**: AIWAF learned legitimate keywords (like "profile") as suspicious from previous traffic.
+
+**Solution**:
+```bash
+# 1. Clear problematic learned keywords
+python manage.py aiwaf_reset --keywords --confirm
+
+# 2. Add legitimate keywords to settings
+# In settings.py:
+AIWAF_ALLOWED_PATH_KEYWORDS = [
+    "profile", "user", "account", "dashboard",
+    # Add your site-specific keywords
+]
+
+# 3. Retrain with enhanced filtering (won't learn legitimate keywords)
+python manage.py detect_and_train
+
+# 4. Test - legitimate pages should now work!
+```
+
+### Preventing Future False Positives
+
+Configure AIWAF to recognize your site's legitimate keywords:
+
+```python
+# settings.py
+AIWAF_ALLOWED_PATH_KEYWORDS = [
+    # Common legitimate keywords
+    "profile", "user", "account", "settings", "dashboard",
+    "admin", "search", "contact", "about", "help",
+    
+    # Your site-specific keywords
+    "buddycraft", "sc2", "starcraft",  # Gaming site example
+    "shop", "cart", "checkout",        # E-commerce example  
+    "blog", "article", "news",         # Content site example
+]
+```
+
+### Reset Command Options
+
+```bash
+# Clear everything (safest for troubleshooting)
+python manage.py aiwaf_reset --confirm
+
+# Clear only problematic keywords
+python manage.py aiwaf_reset --keywords --confirm
+
+# Clear blocked IPs but keep exemptions
+python manage.py aiwaf_reset --blacklist --confirm
+```
+
+---
+
 ## 🧠 How It Works
 
 | Middleware                         | Purpose                                                         |

{aiwaf-0.1.9.1.9 → aiwaf-0.1.9.2.1}/README.md

@@ -2,7 +2,13 @@
 # AI‑WAF
 
 > A self‑learning, Django‑friendly Web Application Firewall  
-> with rate‑limiting, anomaly detection, honeypots, UUID‑tamper protection, dynamic keyword extraction, file‑extension probing detection, exempt path awareness, and daily retraining.
+> with **enhanced context-aware protection**, rate‑limiting, anomaly detection, honeypots, UUID‑tamper protection, **smart keyword learning**, file‑extension probing detection, exempt path awareness, and daily retraining.
+
+**🆕 Latest Enhancements:**
+- ✅ **Smart Keyword Filtering** - Prevents blocking legitimate pages like `/profile/`
+- ✅ **Granular Reset Commands** - Clear specific data types (`--blacklist`, `--keywords`, `--exemptions`)
+- ✅ **Context-Aware Learning** - Only learns from suspicious requests, not legitimate site functionality
+- ✅ **Enhanced Configuration** - `AIWAF_ALLOWED_PATH_KEYWORDS` and `AIWAF_EXEMPT_KEYWORDS`
 
 ---
 
@@ -65,9 +71,19 @@ aiwaf/
   - Burst count  
   - Total 404s  
 
-- **Dynamic Keyword Extraction & Cleanup**  
-  - Every retrain adds top 10 keyword segments from 4xx/5xx paths  
-  - **If a path is added to `AIWAF_EXEMPT_PATHS`, its keywords are automatically removed from the database**
+- **Enhanced Dynamic Keyword Learning with Django Route Protection**  
+  - **Smart Context-Aware Learning**: Only learns keywords from suspicious requests on non-existent paths
+  - **Automatic Django Route Extraction**: Automatically excludes keywords from:
+    - Valid Django URL patterns (`/profile/`, `/admin/`, `/api/`, etc.)
+    - Django app names and model names (users, posts, categories)
+    - View function names and URL namespaces
+  - **Unified Logic**: Both trainer and middleware use identical legitimate keyword detection
+  - **Configuration Options**: 
+    - `AIWAF_ALLOWED_PATH_KEYWORDS` - Explicitly allow certain keywords in legitimate paths
+    - `AIWAF_EXEMPT_KEYWORDS` - Keywords that should never trigger blocking
+  - **Automatic Cleanup**: Keywords from `AIWAF_EXEMPT_PATHS` are automatically removed from the database
+  - **False Positive Prevention**: Stops learning legitimate site functionality as "malicious"
+  - **Inherent Malicious Detection**: Middleware also blocks obviously malicious keywords (`hack`, `exploit`, `attack`) even if not yet learned
 
 - **File‑Extension Probing Detection**  
   Tracks repeated 404s on common extensions (e.g. `.php`, `.asp`) and blocks IPs.
@@ -173,20 +189,44 @@ python manage.py add_ipexemption <ip-address> --reason "optional reason"
 
 ### Resetting AI-WAF
 
-Clear all blacklist and exemption entries:
+The `aiwaf_reset` command provides **granular control** for clearing different types of data:
 
 ```bash
-# Clear everything (with confirmation prompt)
+# Clear everything (default - backward compatible)
 python manage.py aiwaf_reset
 
-# Clear everything without confirmation
+# Clear everything without confirmation prompt
 python manage.py aiwaf_reset --confirm
 
-# Clear only blacklist entries
-python manage.py aiwaf_reset --blacklist-only
+# 🆕 GRANULAR CONTROL - Clear specific data types
+python manage.py aiwaf_reset --blacklist      # Clear only blocked IPs
+python manage.py aiwaf_reset --exemptions     # Clear only exempted IPs  
+python manage.py aiwaf_reset --keywords       # Clear only learned keywords
+
+# 🔧 COMBINE OPTIONS - Mix and match as needed
+python manage.py aiwaf_reset --blacklist --keywords      # Keep exemptions
+python manage.py aiwaf_reset --exemptions --keywords     # Keep blacklist
+python manage.py aiwaf_reset --blacklist --exemptions    # Keep keywords
+
+# 🚀 COMMON USE CASES
+# Fix false positive keywords (like "profile" blocking legitimate pages)
+python manage.py aiwaf_reset --keywords --confirm
+python manage.py detect_and_train  # Retrain with enhanced filtering
+
+# Clear blocked IPs but preserve exemptions and learning
+python manage.py aiwaf_reset --blacklist --confirm
+
+# Legacy support (still works for backward compatibility)
+python manage.py aiwaf_reset --blacklist-only    # Legacy: blacklist only
+python manage.py aiwaf_reset --exemptions-only   # Legacy: exemptions only
+```
 
-# Clear only exemption entries  
-python manage.py aiwaf_reset --exemptions-only
+**Enhanced Feedback:**
+```bash
+$ python manage.py aiwaf_reset --keywords
+🔧 AI-WAF Reset: Clear 15 learned keywords
+Are you sure you want to proceed? [y/N]: y
+✅ Reset complete: Deleted 15 learned keywords
 ```
 
 ### Checking Dependencies
@@ -459,6 +499,21 @@ AIWAF_EXEMPT_PATHS = [          # optional but highly recommended
     "/media/",
     "/health/",
 ]
+
+# 🆕 ENHANCED KEYWORD FILTERING OPTIONS
+AIWAF_ALLOWED_PATH_KEYWORDS = [  # Keywords allowed in legitimate paths
+    "profile", "user", "account", "settings", "dashboard",
+    "admin", "api", "auth", "search", "contact", "about",
+    # Add your site-specific legitimate keywords
+    "buddycraft", "sc2", "starcraft",  # Example: gaming site keywords
+]
+
+AIWAF_EXEMPT_KEYWORDS = [        # Keywords that never trigger blocking
+    "api", "webhook", "health", "static", "media",
+    "upload", "download", "backup", "profile"
+]
+
+AIWAF_DYNAMIC_TOP_N = 10        # Number of dynamic keywords to learn (default: 10)
 ```
 
 > **Note:** You no longer need to define `AIWAF_MALICIOUS_KEYWORDS` or `AIWAF_STATUS_CODES` — they evolve dynamically.
@@ -657,6 +712,65 @@ python manage.py detect_and_train
 
 ---
 
+## 🔧 Troubleshooting
+
+### Legitimate Pages Being Blocked
+
+**Problem**: Users can't access legitimate pages like `/en/profile/` due to keyword blocking.
+
+**Cause**: AIWAF learned legitimate keywords (like "profile") as suspicious from previous traffic.
+
+**Solution**:
+```bash
+# 1. Clear problematic learned keywords
+python manage.py aiwaf_reset --keywords --confirm
+
+# 2. Add legitimate keywords to settings
+# In settings.py:
+AIWAF_ALLOWED_PATH_KEYWORDS = [
+    "profile", "user", "account", "dashboard",
+    # Add your site-specific keywords
+]
+
+# 3. Retrain with enhanced filtering (won't learn legitimate keywords)
+python manage.py detect_and_train
+
+# 4. Test - legitimate pages should now work!
+```
+
+### Preventing Future False Positives
+
+Configure AIWAF to recognize your site's legitimate keywords:
+
+```python
+# settings.py
+AIWAF_ALLOWED_PATH_KEYWORDS = [
+    # Common legitimate keywords
+    "profile", "user", "account", "settings", "dashboard",
+    "admin", "search", "contact", "about", "help",
+    
+    # Your site-specific keywords
+    "buddycraft", "sc2", "starcraft",  # Gaming site example
+    "shop", "cart", "checkout",        # E-commerce example  
+    "blog", "article", "news",         # Content site example
+]
+```
+
+### Reset Command Options
+
+```bash
+# Clear everything (safest for troubleshooting)
+python manage.py aiwaf_reset --confirm
+
+# Clear only problematic keywords
+python manage.py aiwaf_reset --keywords --confirm
+
+# Clear blocked IPs but keep exemptions
+python manage.py aiwaf_reset --blacklist --confirm
+```
+
+---
+
 ## 🧠 How It Works
 
 | Middleware                         | Purpose                                                         |

{aiwaf-0.1.9.1.9 → aiwaf-0.1.9.2.1}/aiwaf/__init__.py

@@ -1,6 +1,6 @@
 default_app_config = "aiwaf.apps.AiwafConfig"
 
-__version__ = "0.1.9.1.9"
+__version__ = "0.1.9.2.1"
 
 # Note: Middleware classes are available from aiwaf.middleware
 # Import them only when needed to avoid circular imports during Django app loading

aiwaf-0.1.9.2.1/aiwaf/management/commands/aiwaf_reset.py

@@ -0,0 +1,183 @@
+from django.core.management.base import BaseCommand
+from aiwaf.storage import get_blacklist_store, get_exemption_store, get_keyword_store
+import sys
+
+class Command(BaseCommand):
+    help = 'Reset AI-WAF by clearing blacklist, exemption, and/or keyword entries'
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--blacklist',
+            action='store_true',
+            help='Clear blacklist entries (default: all)'
+        )
+        parser.add_argument(
+            '--exemptions',
+            action='store_true',
+            help='Clear exemption entries (default: all)'
+        )
+        parser.add_argument(
+            '--keywords',
+            action='store_true',
+            help='Clear learned dynamic keywords (default: all)'
+        )
+        parser.add_argument(
+            '--confirm',
+            action='store_true',
+            help='Skip confirmation prompt'
+        )
+        
+        # Legacy flags for backward compatibility
+        parser.add_argument(
+            '--blacklist-only',
+            action='store_true',
+            help='(Legacy) Clear only blacklist entries'
+        )
+        parser.add_argument(
+            '--exemptions-only',
+            action='store_true',
+            help='(Legacy) Clear only exemption entries'
+        )
+
+    def handle(self, *args, **options):
+        # Parse arguments
+        blacklist_flag = options.get('blacklist', False)
+        exemptions_flag = options.get('exemptions', False)
+        keywords_flag = options.get('keywords', False)
+        confirm = options.get('confirm', False)
+        
+        # Legacy support
+        blacklist_only = options.get('blacklist_only', False)
+        exemptions_only = options.get('exemptions_only', False)
+        
+        # Handle legacy flags
+        if blacklist_only:
+            blacklist_flag = True
+            exemptions_flag = False
+            keywords_flag = False
+        elif exemptions_only:
+            blacklist_flag = False
+            exemptions_flag = True
+            keywords_flag = False
+        
+        # If no specific flags, clear everything
+        if not (blacklist_flag or exemptions_flag or keywords_flag):
+            blacklist_flag = exemptions_flag = keywords_flag = True
+        
+        try:
+            blacklist_store = get_blacklist_store()
+            exemption_store = get_exemption_store()
+            keyword_store = get_keyword_store()
+        except Exception as e:
+            self.stdout.write(self.style.ERROR(f'Error initializing stores: {e}'))
+            return
+        
+        # Count current entries safely
+        counts = {'blacklist': 0, 'exemptions': 0, 'keywords': 0}
+        entries = {'blacklist': [], 'exemptions': [], 'keywords': []}
+        
+        if blacklist_flag:
+            try:
+                entries['blacklist'] = blacklist_store.get_all()
+                counts['blacklist'] = len(entries['blacklist'])
+            except Exception as e:
+                self.stdout.write(self.style.WARNING(f'Warning: Could not count blacklist entries: {e}'))
+        
+        if exemptions_flag:
+            try:
+                entries['exemptions'] = exemption_store.get_all()
+                counts['exemptions'] = len(entries['exemptions'])
+            except Exception as e:
+                self.stdout.write(self.style.WARNING(f'Warning: Could not count exemption entries: {e}'))
+        
+        if keywords_flag:
+            try:
+                entries['keywords'] = keyword_store.get_all_keywords()
+                counts['keywords'] = len(entries['keywords'])
+            except Exception as e:
+                self.stdout.write(self.style.WARNING(f'Warning: Could not count keyword entries: {e}'))
+        
+        # Build action description
+        actions = []
+        if blacklist_flag:
+            actions.append(f"{counts['blacklist']} blacklist entries")
+        if exemptions_flag:
+            actions.append(f"{counts['exemptions']} exemption entries")
+        if keywords_flag:
+            actions.append(f"{counts['keywords']} learned keywords")
+        
+        action = "Clear " + ", ".join(actions)
+        
+        # Show what will be cleared
+        self.stdout.write(f"🔧 AI-WAF Reset: {action}")
+        
+        if not confirm:
+            try:
+                response = input("Are you sure you want to proceed? [y/N]: ")
+                if response.lower() not in ['y', 'yes']:
+                    self.stdout.write(self.style.WARNING('Operation cancelled'))
+                    return
+            except (EOFError, KeyboardInterrupt):
+                self.stdout.write(self.style.WARNING('\nOperation cancelled'))
+                return
+        
+        # Perform the reset
+        deleted_counts = {'blacklist': 0, 'exemptions': 0, 'keywords': 0, 'errors': []}
+        
+        if blacklist_flag:
+            # Clear blacklist entries
+            try:
+                for entry in entries['blacklist']:
+                    try:
+                        blacklist_store.remove_ip(entry['ip_address'])
+                        deleted_counts['blacklist'] += 1
+                    except Exception as e:
+                        deleted_counts['errors'].append(f"Error removing blacklist IP {entry.get('ip_address', 'unknown')}: {e}")
+            except Exception as e:
+                deleted_counts['errors'].append(f"Error clearing blacklist: {e}")
+        
+        if exemptions_flag:
+            # Clear exemption entries
+            try:
+                for entry in entries['exemptions']:
+                    try:
+                        exemption_store.remove_ip(entry['ip_address'])
+                        deleted_counts['exemptions'] += 1
+                    except Exception as e:
+                        deleted_counts['errors'].append(f"Error removing exemption IP {entry.get('ip_address', 'unknown')}: {e}")
+            except Exception as e:
+                deleted_counts['errors'].append(f"Error clearing exemptions: {e}")
+        
+        if keywords_flag:
+            # Clear keyword entries
+            try:
+                for keyword in entries['keywords']:
+                    try:
+                        keyword_store.remove_keyword(keyword)
+                        deleted_counts['keywords'] += 1
+                    except Exception as e:
+                        deleted_counts['errors'].append(f"Error removing keyword '{keyword}': {e}")
+            except Exception as e:
+                deleted_counts['errors'].append(f"Error clearing keywords: {e}")
+        
+        # Report results
+        if deleted_counts['errors']:
+            for error in deleted_counts['errors']:
+                self.stdout.write(self.style.WARNING(f"⚠️  {error}"))
+        
+        # Build success message
+        success_parts = []
+        if blacklist_flag:
+            success_parts.append(f"{deleted_counts['blacklist']} blacklist entries")
+        if exemptions_flag:
+            success_parts.append(f"{deleted_counts['exemptions']} exemption entries")
+        if keywords_flag:
+            success_parts.append(f"{deleted_counts['keywords']} learned keywords")
+        
+        success_message = "✅ Reset complete: Deleted " + ", ".join(success_parts)
+        self.stdout.write(self.style.SUCCESS(success_message))
+        
+        if deleted_counts['errors']:
+            self.stdout.write(
+                self.style.WARNING(f"⚠️  Completed with {len(deleted_counts['errors'])} errors (see above)")
+            )