aiwaf 0.1.2__tar.gz → 0.1.3__tar.gz

aiwaf-0.1.3/PKG-INFO

@@ -0,0 +1,181 @@
+Metadata-Version: 2.4
+Name: aiwaf
+Version: 0.1.3
+Summary: AI-powered Web Application Firewall
+Author: Aayush Gauba
+Author-email: Aayush Gauba <gauba.aayush@gmail.com>
+License: MIT
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Dynamic: author
+
+# AI‑WAF
+
+> A self‑learning, Django‑friendly Web Application Firewall  
+> with rate‑limiting, anomaly detection, honeypots, UUID‑tamper protection, dynamic keyword extraction, file‑extension probing detection, and daily retraining.
+
+---
+
+## Package Structure
+
+```
+aiwaf/
+├── __init__.py
+├── blacklist_manager.py
+├── middleware.py
+├── trainer.py                   # exposes train()
+├── utils.py
+├── template_tags/
+│   └── aiwaf_tags.py
+├── resources/
+│   ├── model.pkl                # pre‑trained base model
+│   └── dynamic_keywords.json    # evolves daily
+├── management/
+│   └── commands/
+│       └── detect_and_train.py  # `python manage.py detect_and_train`
+└── LICENSE
+```
+
+---
+
+## Features
+
+- **IP Blocklist**  
+  Instantly blocks suspicious IPs (supports CSV fallback or Django model).
+
+- **Rate Limiting**  
+  Sliding‑window blocks flooders (> `AIWAF_RATE_MAX` per `AIWAF_RATE_WINDOW`), then blacklists them.
+
+- **AI Anomaly Detection**  
+  IsolationForest on features:
+  - Path length  
+  - Keyword hits (static + dynamic)  
+  - Response time  
+  - Status‑code index  
+  - Burst count  
+  - Total 404s  
+
+- **Dynamic Keyword Extraction**  
+  Every retrain: top 10 most frequent “words” from 4xx/5xx paths are appended to your malicious keyword set.
+
+- **File‑Extension Probing Detection**  
+  Tracks repeated 404s on common web‑extensions (e.g. `.php`, `.asp`) and auto‑blocks after a burst.
+
+- **Honeypot Field**  
+  Hidden form field (via template tag) that bots fill → instant block.
+
+- **UUID Tampering Protection**  
+  Any `<uuid:…>` URL that doesn’t map to **any** model in its Django app gets blocked.
+
+- **Daily Retraining**  
+  Reads rotated/gzipped logs, auto‑blocks 404 floods (≥6), retrains the model, updates `model.pkl` + `dynamic_keywords.json`.
+
+---
+
+## Installation
+
+```bash
+# From PyPI
+pip install aiwaf
+
+# Or for local development
+git clone https://github.com/aayushgauba/aiwaf.git
+cd aiwaf
+pip install -e .
+```
+
+---
+
+## ⚙️ Configuration (`settings.py`)
+
+```python
+INSTALLED_APPS += ["aiwaf"]
+
+# Required
+AIWAF_ACCESS_LOG = "/var/log/nginx/access.log"
+
+# Optional (defaults shown)
+AIWAF_MODEL_PATH         = BASE_DIR / "aiwaf" / "resources" / "model.pkl"
+AIWAF_HONEYPOT_FIELD     = "hp_field"
+AIWAF_RATE_WINDOW        = 10         # seconds
+AIWAF_RATE_MAX           = 20         # max reqs/window
+AIWAF_RATE_FLOOD         = 10         # flood threshold
+AIWAF_WINDOW_SECONDS     = 60         # anomaly window
+AIWAF_FILE_EXTENSIONS    = [".php", ".asp", ".jsp"]  # 404‑burst tracked extensions
+```
+
+> **Note:** You no longer need to define `AIWAF_MALICIOUS_KEYWORDS` or `AIWAF_STATUS_CODES` in your settings — they’re built in and evolve dynamically.
+
+---
+
+## Middleware Setup
+
+Add in **this** order to your `MIDDLEWARE` list:
+
+```python
+MIDDLEWARE = [
+    "aiwaf.middleware.IPBlockMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.AIAnomalyMiddleware",
+    "aiwaf.middleware.HoneypotMiddleware",
+    "aiwaf.middleware.UUIDTamperMiddleware",
+    # ... other middleware ...
+]
+```
+
+---
+
+## Honeypot Field (in your template)
+
+```django
+{% load aiwaf_tags %}
+
+<form method="post">
+  {% csrf_token %}
+  {% honeypot_field %}
+  <!-- your real fields -->
+</form>
+```
+
+> Renders a hidden `<input name="hp_field" style="display:none">`.  
+> Any non‑empty submission → IP blacklisted.
+
+---
+
+## Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+**What happens:**
+1. Read access logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+
+---
+
+## How It Works
+
+| Middleware               | Purpose                                                         |
+|--------------------------|------------------------------------------------------------------|
+| IPBlockMiddleware        | Blocks requests from known blacklisted IPs                      |
+| RateLimitMiddleware      | Enforces burst & flood thresholds                               |
+| AIAnomalyMiddleware      | ML‑driven behavior analysis + block on anomaly                  |
+| HoneypotMiddleware       | Detects bots filling hidden inputs in forms                     |
+| UUIDTamperMiddleware     | Blocks guessed/nonexistent UUIDs across all models in an app    |
+
+---
+
+## License
+
+This project is licensed under the **MIT License**. See the [LICENSE](LICENSE) file for details.
+
+---
+
+## Credits
+
+**AI‑WAF** by [Aayush Gauba](https://github.com/aayushgauba)  
+> “Let your firewall learn and evolve — keep your site a fortress.”

aiwaf-0.1.3/README.md

@@ -0,0 +1,170 @@
+# AI‑WAF
+
+> A self‑learning, Django‑friendly Web Application Firewall  
+> with rate‑limiting, anomaly detection, honeypots, UUID‑tamper protection, dynamic keyword extraction, file‑extension probing detection, and daily retraining.
+
+---
+
+## Package Structure
+
+```
+aiwaf/
+├── __init__.py
+├── blacklist_manager.py
+├── middleware.py
+├── trainer.py                   # exposes train()
+├── utils.py
+├── template_tags/
+│   └── aiwaf_tags.py
+├── resources/
+│   ├── model.pkl                # pre‑trained base model
+│   └── dynamic_keywords.json    # evolves daily
+├── management/
+│   └── commands/
+│       └── detect_and_train.py  # `python manage.py detect_and_train`
+└── LICENSE
+```
+
+---
+
+## Features
+
+- **IP Blocklist**  
+  Instantly blocks suspicious IPs (supports CSV fallback or Django model).
+
+- **Rate Limiting**  
+  Sliding‑window blocks flooders (> `AIWAF_RATE_MAX` per `AIWAF_RATE_WINDOW`), then blacklists them.
+
+- **AI Anomaly Detection**  
+  IsolationForest on features:
+  - Path length  
+  - Keyword hits (static + dynamic)  
+  - Response time  
+  - Status‑code index  
+  - Burst count  
+  - Total 404s  
+
+- **Dynamic Keyword Extraction**  
+  Every retrain: top 10 most frequent “words” from 4xx/5xx paths are appended to your malicious keyword set.
+
+- **File‑Extension Probing Detection**  
+  Tracks repeated 404s on common web‑extensions (e.g. `.php`, `.asp`) and auto‑blocks after a burst.
+
+- **Honeypot Field**  
+  Hidden form field (via template tag) that bots fill → instant block.
+
+- **UUID Tampering Protection**  
+  Any `<uuid:…>` URL that doesn’t map to **any** model in its Django app gets blocked.
+
+- **Daily Retraining**  
+  Reads rotated/gzipped logs, auto‑blocks 404 floods (≥6), retrains the model, updates `model.pkl` + `dynamic_keywords.json`.
+
+---
+
+## Installation
+
+```bash
+# From PyPI
+pip install aiwaf
+
+# Or for local development
+git clone https://github.com/aayushgauba/aiwaf.git
+cd aiwaf
+pip install -e .
+```
+
+---
+
+## ⚙️ Configuration (`settings.py`)
+
+```python
+INSTALLED_APPS += ["aiwaf"]
+
+# Required
+AIWAF_ACCESS_LOG = "/var/log/nginx/access.log"
+
+# Optional (defaults shown)
+AIWAF_MODEL_PATH         = BASE_DIR / "aiwaf" / "resources" / "model.pkl"
+AIWAF_HONEYPOT_FIELD     = "hp_field"
+AIWAF_RATE_WINDOW        = 10         # seconds
+AIWAF_RATE_MAX           = 20         # max reqs/window
+AIWAF_RATE_FLOOD         = 10         # flood threshold
+AIWAF_WINDOW_SECONDS     = 60         # anomaly window
+AIWAF_FILE_EXTENSIONS    = [".php", ".asp", ".jsp"]  # 404‑burst tracked extensions
+```
+
+> **Note:** You no longer need to define `AIWAF_MALICIOUS_KEYWORDS` or `AIWAF_STATUS_CODES` in your settings — they’re built in and evolve dynamically.
+
+---
+
+## Middleware Setup
+
+Add in **this** order to your `MIDDLEWARE` list:
+
+```python
+MIDDLEWARE = [
+    "aiwaf.middleware.IPBlockMiddleware",
+    "aiwaf.middleware.RateLimitMiddleware",
+    "aiwaf.middleware.AIAnomalyMiddleware",
+    "aiwaf.middleware.HoneypotMiddleware",
+    "aiwaf.middleware.UUIDTamperMiddleware",
+    # ... other middleware ...
+]
+```
+
+---
+
+## Honeypot Field (in your template)
+
+```django
+{% load aiwaf_tags %}
+
+<form method="post">
+  {% csrf_token %}
+  {% honeypot_field %}
+  <!-- your real fields -->
+</form>
+```
+
+> Renders a hidden `<input name="hp_field" style="display:none">`.  
+> Any non‑empty submission → IP blacklisted.
+
+---
+
+## Running Detection & Training
+
+```bash
+python manage.py detect_and_train
+```
+
+**What happens:**
+1. Read access logs
+2. Auto‑block IPs with ≥ 6 total 404s
+3. Extract features & train IsolationForest
+4. Save `model.pkl`
+5. Extract top 10 dynamic keywords from 4xx/5xx
+
+---
+
+## How It Works
+
+| Middleware               | Purpose                                                         |
+|--------------------------|------------------------------------------------------------------|
+| IPBlockMiddleware        | Blocks requests from known blacklisted IPs                      |
+| RateLimitMiddleware      | Enforces burst & flood thresholds                               |
+| AIAnomalyMiddleware      | ML‑driven behavior analysis + block on anomaly                  |
+| HoneypotMiddleware       | Detects bots filling hidden inputs in forms                     |
+| UUIDTamperMiddleware     | Blocks guessed/nonexistent UUIDs across all models in an app    |
+
+---
+
+## License
+
+This project is licensed under the **MIT License**. See the [LICENSE](LICENSE) file for details.
+
+---
+
+## Credits
+
+**AI‑WAF** by [Aayush Gauba](https://github.com/aayushgauba)  
+> “Let your firewall learn and evolve — keep your site a fortress.”

{aiwaf-0.1.2 → aiwaf-0.1.3}/aiwaf/middleware.py

@@ -1,23 +1,40 @@
+# aiwaf/middleware.py
+
 import time
+import re
+import os
 import numpy as np
 import joblib
+
 from collections import defaultdict
 from django.utils.deprecation import MiddlewareMixin
 from django.http import JsonResponse
 from django.conf import settings
 from django.core.cache import cache
-from django.urls import resolve
+from django.db.models import F
 from django.apps import apps
-from .blacklist_manager import BlacklistManager
 
-try:
-    MODEL_PATH = settings.AIWAF_MODEL_PATH
-except AttributeError:
-    import importlib.resources
-    MODEL_PATH = importlib.resources.files("aiwaf").joinpath("resources/model.pkl")
+from .blacklist_manager import BlacklistManager
+from .models import DynamicKeyword
 
+# ─── Model loading with fallback ────────────────────────────────────────────
+MODEL_PATH = getattr(
+    settings,
+    "AIWAF_MODEL_PATH",
+    os.path.join(os.path.dirname(__file__), "resources", "model.pkl")
+)
 MODEL = joblib.load(MODEL_PATH)
 
+# ─── Static keywords default ────────────────────────────────────────────────
+STATIC_KW = getattr(
+    settings,
+    "AIWAF_MALICIOUS_KEYWORDS",
+    [
+        ".php", "xmlrpc", "wp-", ".env", ".git", ".bak",
+        "conflg", "shell", "filemanager"
+    ]
+)
+
 def get_ip(request):
     xff = request.META.get("HTTP_X_FORWARDED_FOR")
     if xff:
@@ -37,18 +54,21 @@ class IPBlockMiddleware:
 
 
 class RateLimitMiddleware:
-    WINDOW = getattr(settings, "AIWAF_RATE_WINDOW", 10)
-    MAX = getattr(settings, "AIWAF_RATE_MAX", 20)
-    FLOOD = getattr(settings, "AIWAF_RATE_FLOOD", 10)
+    WINDOW = 10
+    MAX    = 20
+    FLOOD  = 10
+
     def __init__(self, get_response):
         self.get_response = get_response
         self.logs = defaultdict(list)
+
     def __call__(self, request):
-        ip = get_ip(request)
+        ip  = get_ip(request)
         now = time.time()
         recs = [t for t in self.logs[ip] if now - t < self.WINDOW]
         recs.append(now)
         self.logs[ip] = recs
+
         if len(recs) > self.MAX:
             return JsonResponse({"error": "too_many_requests"}, status=429)
         if len(recs) > self.FLOOD:
@@ -59,40 +79,57 @@ class RateLimitMiddleware:
 
 
 class AIAnomalyMiddleware(MiddlewareMixin):
-    WINDOW_SECONDS = getattr(settings, "AIWAF_WINDOW_SECONDS", 60)
+    WINDOW = getattr(settings, "AIWAF_WINDOW_SECONDS", 60)
+    TOP_N  = getattr(settings, "AIWAF_DYNAMIC_TOP_N", 10)
+
     def process_request(self, request):
         ip = get_ip(request)
         if BlacklistManager.is_blocked(ip):
             return JsonResponse({"error": "blocked"}, status=403)
+
         now = time.time()
         key = f"aiwaf:{ip}"
         data = cache.get(key, [])
+        # TODO: you may want to capture real status & response_time in process_response
         data.append((now, request.path, 0, 0.0))
-        data = [d for d in data if now - d[0] < self.WINDOW_SECONDS]
-        cache.set(key, data, timeout=self.WINDOW_SECONDS)
+        data = [d for d in data if now - d[0] < self.WINDOW]
+        cache.set(key, data, timeout=self.WINDOW)
+
+        # update dynamic‐keyword counts
+        for seg in re.split(r"\W+", request.path.lower()):
+            if len(seg) > 3:
+                obj, _ = DynamicKeyword.objects.get_or_create(keyword=seg)
+                DynamicKeyword.objects.filter(pk=obj.pk).update(count=F("count") + 1)
+
         if len(data) < 5:
             return None
-        total = len(data)
-        ratio_404 = sum(1 for (_, _, st, _) in data if st == 404) / total
-        hits = sum(
-            any(k in path.lower() for k in settings.AIWAF_MALICIOUS_KEYWORDS)
-            for (_, path, _, _) in data
+
+        # pull top‐N dynamic tokens
+        top_dynamic = list(
+            DynamicKeyword.objects
+            .order_by("-count")
+            .values_list("keyword", flat=True)[: self.TOP_N]
         )
-        avg_rt = np.mean([rt for (_, _, _, rt) in data]) if data else 0.0
-        intervals = [
-            data[i][0] - data[i-1][0] for i in range(1, total)
-        ]
-        avg_iv = np.mean(intervals) if intervals else 0.0
-        X = np.array([[total, ratio_404, hits, avg_rt, avg_iv]], dtype=float)
+        ALL_KW = set(STATIC_KW) | set(top_dynamic)
+
+        total    = len(data)
+        ratio404 = sum(1 for (_, _, st, _) in data if st == 404) / total
+        hits     = sum(any(kw in path.lower() for kw in ALL_KW) for (_, path, _, _) in data)
+        avg_rt   = np.mean([rt for (_, _, _, rt) in data]) if data else 0.0
+        ivs      = [data[i][0] - data[i - 1][0] for i in range(1, total)]
+        avg_iv   = np.mean(ivs) if ivs else 0.0
+
+        X = np.array([[total, ratio404, hits, avg_rt, avg_iv]], dtype=float)
         if MODEL.predict(X)[0] == -1:
             BlacklistManager.block(ip, "AI anomaly")
             return JsonResponse({"error": "blocked"}, status=403)
+
         return None
 
 
 class HoneypotMiddleware(MiddlewareMixin):
     def process_view(self, request, view_func, view_args, view_kwargs):
-        trap = request.POST.get(settings.AIWAF_HONEYPOT_FIELD, "")
+        trap = request.POST.get(getattr(settings, "AIWAF_HONEYPOT_FIELD", "hp_field"), "")
         if trap:
             ip = get_ip(request)
             BlacklistManager.block(ip, "HONEYPOT triggered")
@@ -105,11 +142,13 @@ class UUIDTamperMiddleware(MiddlewareMixin):
         uid = view_kwargs.get("uuid")
         if not uid:
             return None
+
         ip = get_ip(request)
-        app_label = view_kwargs.get("app_label") or view_func.__module__.split('.')[0]
-        app_config = apps.get_app_config(app_label)
-        for Model in app_config.get_models():
+        app_label = view_func.__module__.split(".")[0]
+        app_cfg   = apps.get_app_config(app_label)
+        for Model in app_cfg.get_models():
             if Model.objects.filter(pk=uid).exists():
                 return None
+
         BlacklistManager.block(ip, "UUID tampering")
-        return JsonResponse({"error": "blocked"}, status=403)
+        return JsonResponse({"error": "blocked"}, status=403)

{aiwaf-0.1.2 → aiwaf-0.1.3}/aiwaf/models.py

@@ -26,3 +26,11 @@ class BlacklistEntry(models.Model):
 
     def __str__(self):
         return f"{self.ip_address} ({self.reason})"
+
+class DynamicKeyword(models.Model):
+    keyword      = models.CharField(max_length=100, unique=True)
+    count        = models.PositiveIntegerField(default=0)
+    last_updated = models.DateTimeField(auto_now=True)
+
+    class Meta:
+        ordering = ['-count']

aiwaf-0.1.3/aiwaf/trainer.py

@@ -0,0 +1,175 @@
+import os
+import glob
+import gzip
+import re
+import json
+import joblib
+
+from datetime import datetime
+from collections import defaultdict, Counter
+
+import pandas as pd
+from sklearn.ensemble import IsolationForest
+
+from django.conf import settings
+from django.apps import apps
+
+# ─── CONFIG ────────────────────────────────────────────────────────────────
+
+# Where to read your access logs (and rotated/.gz siblings)
+LOG_PATH = settings.AIWAF_ACCESS_LOG
+
+# Where we save our trained model
+MODEL_PATH = os.path.join(
+    os.path.dirname(__file__),
+    "resources",
+    "model.pkl"
+)
+
+# Static “malicious” path keywords & file extensions
+MALICIOUS_KEYWORDS = [
+    ".php", "xmlrpc", "wp-", ".env", ".git", ".bak",
+    "conflg", "shell", "filemanager"
+]
+STATUS_CODES = ["200", "403", "404", "500"]
+
+# Regex for combined log with response-time=…
+_LOG_RX = re.compile(
+    r'(\d+\.\d+\.\d+\.\d+).*\[(.*?)\].*"(?:GET|POST) (.*?) HTTP/.*?" '
+    r'(\d{3}).*?"(.*?)" "(.*?)".*?response-time=(\d+\.\d+)'
+)
+
+# Your Django model for storing blocked IPs
+BlacklistEntry = apps.get_model("aiwaf", "BlacklistEntry")
+
+
+# ─── READ & PARSE LOG LINES ─────────────────────────────────────────────────
+
+def _read_all_logs():
+    lines = []
+    if LOG_PATH and os.path.exists(LOG_PATH):
+        with open(LOG_PATH, "r", errors="ignore") as f:
+            lines += f.readlines()
+    for path in sorted(glob.glob(LOG_PATH + ".*")):
+        opener = gzip.open if path.endswith(".gz") else open
+        try:
+            with opener(path, "rt", errors="ignore") as f:
+                lines += f.readlines()
+        except OSError:
+            continue
+    return lines
+
+def _parse(line):
+    m = _LOG_RX.search(line)
+    if not m:
+        return None
+    ip, ts_str, path, status, ref, ua, rt = m.groups()
+    try:
+        ts = datetime.strptime(ts_str.split()[0], "%d/%b/%Y:%H:%M:%S")
+    except ValueError:
+        return None
+    return {
+        "ip": ip,
+        "timestamp": ts,
+        "path": path,
+        "status": status,
+        "ua": ua,
+        "response_time": float(rt),
+    }
+
+
+# ─── TRAIN ENTRYPOINT ───────────────────────────────────────────────────────
+
+def train():
+    raw = _read_all_logs()
+    if not raw:
+        print("❌ No log lines found – check settings.AIWAF_ACCESS_LOG")
+        return
+
+    parsed  = []
+    ip_404   = defaultdict(int)
+    ip_times = defaultdict(list)
+
+    # parse + accumulate timestamps & 404 counts
+    for ln in raw:
+        rec = _parse(ln)
+        if not rec:
+            continue
+        parsed.append(rec)
+        ip_times[rec["ip"]].append(rec["timestamp"])
+        if rec["status"] == "404":
+            ip_404[rec["ip"]] += 1
+
+    # auto-block IPs with >=6 total 404s
+    newly_blocked = []
+    for ip, cnt in ip_404.items():
+        if cnt >= 6:
+            obj, created = BlacklistEntry.objects.get_or_create(
+                ip_address=ip,
+                defaults={"reason": "Excessive 404s (≥6)"}
+            )
+            if created:
+                newly_blocked.append(ip)
+    if newly_blocked:
+        print(f"🔒 Blocked {len(newly_blocked)} IPs for 404 flood: {newly_blocked}")
+
+    # build feature vectors
+    rows = []
+    for r in parsed:
+        ip         = r["ip"]
+        burst      = sum(
+            1 for t in ip_times[ip]
+            if (r["timestamp"] - t).total_seconds() <= 10
+        )
+        total404   = ip_404[ip]
+        kw_hits    = sum(k in r["path"].lower() for k in MALICIOUS_KEYWORDS)
+        status_idx = STATUS_CODES.index(r["status"]) if r["status"] in STATUS_CODES else -1
+
+        rows.append([
+            len(r["path"]),
+            kw_hits,
+            r["response_time"],
+            status_idx,
+            burst,
+            total404
+        ])
+
+    if not rows:
+        print("⚠️ No entries to train on.")
+        return
+
+    df = pd.DataFrame(
+        rows,
+        columns=[
+            "path_len", "kw_hits", "resp_time",
+            "status_idx", "burst_count", "total_404"
+        ]
+    ).fillna(0).astype(float)
+
+    # train & save
+    clf = IsolationForest(contamination=0.01, random_state=42)
+    clf.fit(df.values)
+    os.makedirs(os.path.dirname(MODEL_PATH), exist_ok=True)
+    joblib.dump(clf, MODEL_PATH)
+    print(f"✅ Model trained on {len(df)} samples → {MODEL_PATH}")
+
+    # extract top‑10 dynamic keywords from 4xx/5xx paths
+    tokens = Counter()
+    for r in parsed:
+        if r["status"].startswith(("4", "5")):
+            segs = re.split(r"\W+", r["path"].lower())
+            for seg in segs:
+                if len(seg) > 3 and seg not in MALICIOUS_KEYWORDS:
+                    tokens[seg] += 1
+
+    new_kw = [kw for kw, _ in tokens.most_common(10)]
+    DK_FILE = os.path.join(os.path.dirname(__file__), "resources", "dynamic_keywords.json")
+    try:
+        existing = set(json.load(open(DK_FILE)))
+    except FileNotFoundError:
+        existing = set()
+    updated = sorted(existing | set(new_kw))
+    with open(DK_FILE, "w") as f:
+        json.dump(updated, f, indent=2)
+
+    print(f"📝 Updated dynamic keywords: {new_kw}")