redsafe 0.1.0__tar.gz

redsafe-0.1.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.4
+Name: redsafe
+Version: 0.1.0
+Summary: Local AI-safe redaction engine for security data
+Author: AI Safe Redaction Engine
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: spacy
+Requires-Dist: pytesseract
+Requires-Dist: opencv-python
+Requires-Dist: lxml
+Requires-Dist: numpy
+Requires-Dist: scikit-learn
+
+# AI Safe Redaction Engine
+
+A local-first, modular redaction system for security data before AI processing.
+
+## Single-Word CLI Name
+This project is packaged for `pipx` as:
+- Command: `redsafe`
+- Package name: `redsafe`
+
+Other good single-word alternatives if you want to rename later:
+- `safescrub`
+- `cloaknet`
+- `privashield`
+
+## Supported Inputs
+- Burp Suite XML exports
+- Raw HTTP requests/responses
+- Network logs
+- Basic PCAP parsing
+- Screenshots (OCR + masking)
+
+## Features
+- Analysis-based sensitive data detection (not regex-only)
+- Named Entity Recognition (spaCy)
+- Secret detection with heuristics and entropy scoring
+- Context-aware detection from header/parameter names
+- Consistent placeholder mapping (`<EMAIL_1>`, `<JWT_TOKEN_1>`, etc.)
+- Local-only processing, no external API calls
+
+## Project Structure
+- `parsers/`: input ingestion modules
+- `detection/`: entity, secret, entropy, and context detectors
+- `redaction/`: placeholder and redaction logic
+- `core/`: data models and orchestration pipeline
+- `utils/`: file/encoding helpers
+- `tests/`: sample files + pytest coverage
+
+## Install With pipx (Recommended)
+```bash
+cd /home/gss/Desktop/Codex/ai-redaction-engine
+pipx install .
+```
+
+Run:
+```bash
+redsafe --input tests/sample_burp.xml --type burp
+redsafe --input tests/sample_http.txt --type http
+redsafe --input tests/sample_log.txt --type log
+redsafe --input tests/sample_image.png --type image
+```
+
+To reinstall after local code changes:
+```bash
+pipx reinstall redsafe
+```
+
+## Install With venv (Alternative)
+```bash
+cd /home/gss/Desktop/Codex/ai-redaction-engine
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python -m spacy download en_core_web_sm
+```
+
+Run examples:
+```bash
+python main.py --input tests/sample_burp.xml --type burp
+python main.py --input tests/sample_http.txt --type http
+python main.py --input tests/sample_log.txt --type log
+python main.py --input tests/sample_image.png --type image
+```
+
+Outputs are written to `sanitized_output/`.
+
+## Redaction Tuning (False Positive Control)
+Secret entropy detection can be tuned via environment variables:
+
+```bash
+export REDACTION_ENTROPY_THRESHOLD=4.2
+export REDACTION_MIN_SECRET_LEN=24
+export REDACTION_MIN_BASE64_LEN=28
+export REDACTION_IGNORE_VALUES="application/x-www-form-urlencoded,text/plain"
+```
+
+These values are consumed by `SecretDetectionConfig` in `detection/secret_detection.py`.
+
+## Run Tests
+```bash
+pytest -q
+```
+
+## Example: Burp XML Sanitization
+Input Burp request header:
+```text
+Authorization: Bearer eyJhbGciOiJIUzI1Ni.eyJzdWIiOiIxMjM0NTYifQ.signature123456789
+Cookie: sessionid=abcDEF1234567890
+```
+
+Sanitized output:
+```text
+Authorization: Bearer <JWT_TOKEN_1>
+Cookie: sessionid=<SESSION_COOKIE_1>
+```
+
+## Example: HTTP Sanitization
+Input:
+```text
+POST /login HTTP/1.1
+Host: app.local
+Content-Type: application/x-www-form-urlencoded
+
+email=john@example.com&password=SuperSecret123
+```
+
+Sanitized output:
+```text
+POST /login HTTP/1.1
+Host: app.local
+Content-Type: application/x-www-form-urlencoded
+
+email=<EMAIL_1>&password=<PASSWORD_1>
+```
+
+## Example: Image Sanitization
+- OCR extracts text from screenshot.
+- Detection engine flags sensitive values.
+- Matching OCR boxes are masked in output image.
+
+## Notes
+- Designed for integration into a future AI pentesting engine.
+- All processing is local.
+- If `en_core_web_sm` is unavailable, regex/heuristic detection still works.
+- Image redaction needs local `opencv-python`, `pytesseract`, and system `tesseract` binary installed.

redsafe-0.1.0/README.md

@@ -0,0 +1,134 @@
+# AI Safe Redaction Engine
+
+A local-first, modular redaction system for security data before AI processing.
+
+## Single-Word CLI Name
+This project is packaged for `pipx` as:
+- Command: `redsafe`
+- Package name: `redsafe`
+
+Other good single-word alternatives if you want to rename later:
+- `safescrub`
+- `cloaknet`
+- `privashield`
+
+## Supported Inputs
+- Burp Suite XML exports
+- Raw HTTP requests/responses
+- Network logs
+- Basic PCAP parsing
+- Screenshots (OCR + masking)
+
+## Features
+- Analysis-based sensitive data detection (not regex-only)
+- Named Entity Recognition (spaCy)
+- Secret detection with heuristics and entropy scoring
+- Context-aware detection from header/parameter names
+- Consistent placeholder mapping (`<EMAIL_1>`, `<JWT_TOKEN_1>`, etc.)
+- Local-only processing, no external API calls
+
+## Project Structure
+- `parsers/`: input ingestion modules
+- `detection/`: entity, secret, entropy, and context detectors
+- `redaction/`: placeholder and redaction logic
+- `core/`: data models and orchestration pipeline
+- `utils/`: file/encoding helpers
+- `tests/`: sample files + pytest coverage
+
+## Install With pipx (Recommended)
+```bash
+cd /home/gss/Desktop/Codex/ai-redaction-engine
+pipx install .
+```
+
+Run:
+```bash
+redsafe --input tests/sample_burp.xml --type burp
+redsafe --input tests/sample_http.txt --type http
+redsafe --input tests/sample_log.txt --type log
+redsafe --input tests/sample_image.png --type image
+```
+
+To reinstall after local code changes:
+```bash
+pipx reinstall redsafe
+```
+
+## Install With venv (Alternative)
+```bash
+cd /home/gss/Desktop/Codex/ai-redaction-engine
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python -m spacy download en_core_web_sm
+```
+
+Run examples:
+```bash
+python main.py --input tests/sample_burp.xml --type burp
+python main.py --input tests/sample_http.txt --type http
+python main.py --input tests/sample_log.txt --type log
+python main.py --input tests/sample_image.png --type image
+```
+
+Outputs are written to `sanitized_output/`.
+
+## Redaction Tuning (False Positive Control)
+Secret entropy detection can be tuned via environment variables:
+
+```bash
+export REDACTION_ENTROPY_THRESHOLD=4.2
+export REDACTION_MIN_SECRET_LEN=24
+export REDACTION_MIN_BASE64_LEN=28
+export REDACTION_IGNORE_VALUES="application/x-www-form-urlencoded,text/plain"
+```
+
+These values are consumed by `SecretDetectionConfig` in `detection/secret_detection.py`.
+
+## Run Tests
+```bash
+pytest -q
+```
+
+## Example: Burp XML Sanitization
+Input Burp request header:
+```text
+Authorization: Bearer eyJhbGciOiJIUzI1Ni.eyJzdWIiOiIxMjM0NTYifQ.signature123456789
+Cookie: sessionid=abcDEF1234567890
+```
+
+Sanitized output:
+```text
+Authorization: Bearer <JWT_TOKEN_1>
+Cookie: sessionid=<SESSION_COOKIE_1>
+```
+
+## Example: HTTP Sanitization
+Input:
+```text
+POST /login HTTP/1.1
+Host: app.local
+Content-Type: application/x-www-form-urlencoded
+
+email=john@example.com&password=SuperSecret123
+```
+
+Sanitized output:
+```text
+POST /login HTTP/1.1
+Host: app.local
+Content-Type: application/x-www-form-urlencoded
+
+email=<EMAIL_1>&password=<PASSWORD_1>
+```
+
+## Example: Image Sanitization
+- OCR extracts text from screenshot.
+- Detection engine flags sensitive values.
+- Matching OCR boxes are masked in output image.
+
+## Notes
+- Designed for integration into a future AI pentesting engine.
+- All processing is local.
+- If `en_core_web_sm` is unavailable, regex/heuristic detection still works.
+- Image redaction needs local `opencv-python`, `pytesseract`, and system `tesseract` binary installed.

redsafe-0.1.0/core/data_models.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional
+
+
+@dataclass
+class HTTPMessage:
+    method: str = ""
+    path: str = ""
+    version: str = ""
+    headers: Dict[str, str] = field(default_factory=dict)
+    parameters: Dict[str, str] = field(default_factory=dict)
+    body: str = ""
+
+
+@dataclass
+class HTTPResponse:
+    status_code: int = 0
+    reason: str = ""
+    version: str = ""
+    headers: Dict[str, str] = field(default_factory=dict)
+    body: str = ""
+
+
+@dataclass
+class HTTPExchange:
+    request: HTTPMessage
+    response: Optional[HTTPResponse] = None
+    source: str = ""
+
+
+@dataclass
+class Detection:
+    kind: str
+    value: str
+    start: int = -1
+    end: int = -1
+    confidence: float = 1.0
+    source: str = ""
+
+
+@dataclass
+class RedactionSummary:
+    detected_count: int = 0
+    redacted_count: int = 0
+    mappings: Dict[str, str] = field(default_factory=dict)
+
+
+@dataclass
+class ImageOCRToken:
+    text: str
+    left: int
+    top: int
+    width: int
+    height: int
+
+
+@dataclass
+class PipelineResult:
+    sanitized_text: Optional[str] = None
+    sanitized_image_path: Optional[str] = None
+    detections: List[Detection] = field(default_factory=list)
+    summary: RedactionSummary = field(default_factory=RedactionSummary)

redsafe-0.1.0/core/redaction_pipeline.py

@@ -0,0 +1,169 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Dict, Iterable, List
+
+from core.data_models import Detection, HTTPExchange, HTTPMessage, HTTPResponse, PipelineResult, RedactionSummary
+from detection.context_detection import ContextDetector
+from detection.entity_detection import EntityDetector
+from detection.secret_detection import SecretDetector
+from parsers.burp_xml_parser import parse_burp_xml
+from parsers.http_parser import parse_http_file
+from parsers.image_parser import extract_ocr_tokens
+from parsers.log_parser import parse_network_log
+from parsers.pcap_parser import parse_pcap_basic
+from redaction.placeholder_mapper import PlaceholderMapper
+from redaction.redact_image import redact_image_tokens
+from redaction.redact_text import redact_text_content
+from utils.file_utils import ensure_dir, read_text, write_text
+
+
+class RedactionPipeline:
+    def __init__(self) -> None:
+        self.entity_detector = EntityDetector()
+        self.secret_detector = SecretDetector()
+        self.context_detector = ContextDetector()
+        self.mapper = PlaceholderMapper()
+
+    def _serialize_request(self, req: HTTPMessage) -> str:
+        start = f"{req.method} {req.path} {req.version}".strip()
+        lines = [start] if start else []
+        lines.extend([f"{k}: {v}" for k, v in req.headers.items()])
+        if req.body:
+            lines.append("")
+            lines.append(req.body)
+        return "\n".join(lines).strip()
+
+    def _serialize_response(self, resp: HTTPResponse) -> str:
+        start = f"{resp.version} {resp.status_code} {resp.reason}".strip()
+        lines = [start] if start else []
+        lines.extend([f"{k}: {v}" for k, v in resp.headers.items()])
+        if resp.body:
+            lines.append("")
+            lines.append(resp.body)
+        return "\n".join(lines).strip()
+
+    def _serialize_exchanges(self, exchanges: Iterable[HTTPExchange]) -> str:
+        blocks: List[str] = []
+        for ex in exchanges:
+            parts = []
+            if ex.request:
+                req_txt = self._serialize_request(ex.request)
+                if req_txt:
+                    parts.append(req_txt)
+            if ex.response:
+                resp_txt = self._serialize_response(ex.response)
+                if resp_txt:
+                    parts.append(resp_txt)
+            if parts:
+                blocks.append("\n\n".join(parts))
+        return "\n\n###\n\n".join(blocks)
+
+    def _collect_detections_from_text(self, text: str) -> List[Detection]:
+        detections = []
+        detections.extend(self.entity_detector.detect(text))
+        detections.extend(self.secret_detector.detect(text))
+        return detections
+
+    def _collect_context_from_exchanges(self, exchanges: Iterable[HTTPExchange]) -> List[Detection]:
+        detections: List[Detection] = []
+        for ex in exchanges:
+            detections.extend(self.context_detector.detect_pairs(ex.request.headers, source="request_headers"))
+            detections.extend(self.context_detector.detect_pairs(ex.request.parameters, source="request_params"))
+            if ex.response:
+                detections.extend(self.context_detector.detect_pairs(ex.response.headers, source="response_headers"))
+        return detections
+
+    def _collect_context_from_logs(self, entries: Iterable[Dict[str, str]]) -> List[Detection]:
+        detections: List[Detection] = []
+        for entry in entries:
+            pairs = {k: v for k, v in entry.items() if k != "raw"}
+            detections.extend(self.context_detector.detect_pairs(pairs, source="log_fields"))
+        return detections
+
+    def _dedupe(self, detections: Iterable[Detection]) -> List[Detection]:
+        unique = {}
+        for d in detections:
+            key = (d.kind, d.value)
+            if d.value and key not in unique:
+                unique[key] = d
+        return list(unique.values())
+
+    def _build_summary(self, detections: List[Detection]) -> RedactionSummary:
+        return RedactionSummary(
+            detected_count=len(detections),
+            redacted_count=len(self.mapper.mapping()),
+            mappings=self.mapper.mapping(),
+        )
+
+    def process_burp(self, input_path: str, output_dir: str) -> PipelineResult:
+        exchanges = parse_burp_xml(input_path)
+        original_text = self._serialize_exchanges(exchanges)
+        detections = self._collect_detections_from_text(original_text)
+        detections.extend(self._collect_context_from_exchanges(exchanges))
+        detections = self._dedupe(detections)
+
+        redacted = redact_text_content(original_text, detections, self.mapper)
+        out = ensure_dir(output_dir) / (Path(input_path).stem + ".sanitized.txt")
+        write_text(out, redacted)
+
+        return PipelineResult(sanitized_text=redacted, detections=detections, summary=self._build_summary(detections))
+
+    def process_http(self, input_path: str, output_dir: str) -> PipelineResult:
+        raw = read_text(input_path)
+        exchanges = parse_http_file(raw, source=input_path)
+        original_text = self._serialize_exchanges(exchanges) if exchanges else raw
+
+        detections = self._collect_detections_from_text(original_text)
+        detections.extend(self._collect_context_from_exchanges(exchanges))
+        detections = self._dedupe(detections)
+
+        redacted = redact_text_content(original_text, detections, self.mapper)
+        out = ensure_dir(output_dir) / (Path(input_path).stem + ".sanitized.txt")
+        write_text(out, redacted)
+
+        return PipelineResult(sanitized_text=redacted, detections=detections, summary=self._build_summary(detections))
+
+    def process_log(self, input_path: str, output_dir: str) -> PipelineResult:
+        raw = read_text(input_path)
+        entries = parse_network_log(raw)
+
+        detections = self._collect_detections_from_text(raw)
+        detections.extend(self._collect_context_from_logs(entries))
+        detections = self._dedupe(detections)
+
+        redacted = redact_text_content(raw, detections, self.mapper)
+        out = ensure_dir(output_dir) / (Path(input_path).stem + ".sanitized.txt")
+        write_text(out, redacted)
+
+        return PipelineResult(sanitized_text=redacted, detections=detections, summary=self._build_summary(detections))
+
+    def process_pcap(self, input_path: str, output_dir: str) -> PipelineResult:
+        exchanges = parse_pcap_basic(input_path)
+        original_text = self._serialize_exchanges(exchanges)
+
+        detections = self._collect_detections_from_text(original_text)
+        detections.extend(self._collect_context_from_exchanges(exchanges))
+        detections = self._dedupe(detections)
+
+        redacted = redact_text_content(original_text, detections, self.mapper)
+        out = ensure_dir(output_dir) / (Path(input_path).stem + ".sanitized.txt")
+        write_text(out, redacted)
+
+        return PipelineResult(sanitized_text=redacted, detections=detections, summary=self._build_summary(detections))
+
+    def process_image(self, input_path: str, output_dir: str) -> PipelineResult:
+        tokens = extract_ocr_tokens(input_path)
+        full_text = "\n".join(t.text for t in tokens)
+
+        detections = self._collect_detections_from_text(full_text)
+        detections = self._dedupe(detections)
+
+        out = ensure_dir(output_dir) / (Path(input_path).stem + ".sanitized.png")
+        redact_image_tokens(input_path, str(out), tokens, detections, self.mapper)
+
+        return PipelineResult(
+            sanitized_image_path=str(out),
+            detections=detections,
+            summary=self._build_summary(detections),
+        )

redsafe-0.1.0/detection/context_detection.py

@@ -0,0 +1,31 @@
+from __future__ import annotations
+
+from typing import Dict, List
+
+from core.data_models import Detection
+
+
+SENSITIVE_KEYS = {
+    "password": "PASSWORD",
+    "passwd": "PASSWORD",
+    "secret": "SECRET",
+    "token": "TOKEN",
+    "apikey": "API_KEY",
+    "api_key": "API_KEY",
+    "authorization": "AUTHORIZATION",
+    "session": "SESSION",
+    "cookie": "COOKIE",
+    "private_key": "PRIVATE_KEY",
+}
+
+
+class ContextDetector:
+    def detect_pairs(self, pairs: Dict[str, str], source: str = "context") -> List[Detection]:
+        detections: List[Detection] = []
+        for k, v in pairs.items():
+            key_lower = k.lower()
+            for skey, kind in SENSITIVE_KEYS.items():
+                if skey in key_lower and v:
+                    detections.append(Detection(kind=kind, value=v, source=source, confidence=0.95))
+                    break
+        return detections

redsafe-0.1.0/detection/entity_detection.py

@@ -0,0 +1,72 @@
+from __future__ import annotations
+
+import re
+from typing import List
+
+from core.data_models import Detection
+
+try:
+    import spacy  # type: ignore
+except Exception:  # pragma: no cover
+    spacy = None
+
+
+EMAIL_RE = re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b")
+PHONE_RE = re.compile(r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]?)?\d{3}[\s.-]?\d{4}\b")
+ADDRESS_HINT_RE = re.compile(r"\b\d{1,6}\s+[A-Za-z0-9\s]{3,}\s(?:Street|St|Road|Rd|Avenue|Ave|Lane|Ln|Boulevard|Blvd|Drive|Dr)\b", re.IGNORECASE)
+
+
+class EntityDetector:
+    def __init__(self) -> None:
+        self._nlp = None
+        self._load_model()
+
+    def _load_model(self) -> None:
+        if spacy is None:
+            return
+        for model in ("en_core_web_sm", "en_core_web_md"):
+            try:
+                self._nlp = spacy.load(model)
+                return
+            except Exception:
+                continue
+
+    def detect(self, text: str) -> List[Detection]:
+        detections: List[Detection] = []
+
+        for m in EMAIL_RE.finditer(text):
+            detections.append(Detection(kind="EMAIL", value=m.group(0), start=m.start(), end=m.end(), source="regex"))
+
+        for m in PHONE_RE.finditer(text):
+            if len(re.sub(r"\D", "", m.group(0))) >= 7:
+                detections.append(Detection(kind="PHONE", value=m.group(0), start=m.start(), end=m.end(), source="regex"))
+
+        for m in ADDRESS_HINT_RE.finditer(text):
+            detections.append(Detection(kind="ADDRESS", value=m.group(0), start=m.start(), end=m.end(), source="regex"))
+
+        if self._nlp is None:
+            return detections
+
+        doc = self._nlp(text)
+        for ent in doc.ents:
+            label_map = {
+                "PERSON": "NAME",
+                "ORG": "ORGANIZATION",
+                "GPE": "ADDRESS",
+                "LOC": "ADDRESS",
+                "FAC": "ADDRESS",
+            }
+            mapped = label_map.get(ent.label_)
+            if mapped:
+                detections.append(
+                    Detection(
+                        kind=mapped,
+                        value=ent.text,
+                        start=ent.start_char,
+                        end=ent.end_char,
+                        source="spacy",
+                        confidence=0.9,
+                    )
+                )
+
+        return detections

redsafe-0.1.0/detection/entropy_detection.py

@@ -0,0 +1,24 @@
+from __future__ import annotations
+
+import math
+from collections import Counter
+
+
+class EntropyDetector:
+    @staticmethod
+    def shannon_entropy(value: str) -> float:
+        if not value:
+            return 0.0
+        counts = Counter(value)
+        length = len(value)
+        entropy = 0.0
+        for count in counts.values():
+            p = count / length
+            entropy -= p * math.log2(p)
+        return entropy
+
+    @staticmethod
+    def is_high_entropy(value: str, threshold: float = 3.5, min_len: int = 16) -> bool:
+        if len(value) < min_len:
+            return False
+        return EntropyDetector.shannon_entropy(value) >= threshold