shadowshield 0.6.2__tar.gz → 0.7.0__tar.gz

{shadowshield-0.6.2 → shadowshield-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: shadowshield
-Version: 0.6.2
+Version: 0.7.0
 Summary: Unified open-source security shield for agentic AI systems — inspired by Sentinel & ShadowClaw.
 Project-URL: Homepage, https://shadowshield.xyz
 Project-URL: Documentation, https://github.com/0xsl1m/shadowshield#readme
@@ -28,7 +28,6 @@ Requires-Dist: httpx>=0.27
 Requires-Dist: pydantic>=2.5
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: structlog>=24.1
-Requires-Dist: tiktoken>=0.6
 Provides-Extra: all
 Requires-Dist: datasets>=2.18; extra == 'all'
 Requires-Dist: fastapi>=0.110; extra == 'all'
@@ -135,8 +134,10 @@ print(result.safe_text)            # safe fallback message
   false-positive rate on hard negatives** in the bundled benchmark.
 - **Proven, reproducibly.** Ships an eval harness + offline benchmark:
   `shadowshield benchmark`. Loads public datasets (PINT/deepset/InjecAgent) too.
-- **Drop-in integrations.** OpenAI-compatible clients, LangChain, decorators,
-  context managers, **async** (`ascan`). Or call `shield.scan()` directly.
+- **Drop-in integrations.** A zero-code **reverse-proxy gateway** for any
+  OpenAI-compatible endpoint, plus OpenAI/Anthropic/LiteLLM/LangChain wrappers,
+  pure-ASGI middleware, RAG retriever guards, decorators, context managers, and
+  **async** (`ascan`). Or call `shield.scan()` directly.
 - **Extensible & lightweight.** Add a detector/responder in ~10 lines or ship a
   plugin. Tiny core dependency set; ML/PII/datasets are optional extras.
 
@@ -149,9 +150,10 @@ print(result.safe_text)            # safe fallback message
 > in-distribution **regression baseline, not a SOTA claim**. We publish the humbling
 > external numbers on purpose — a credible security tool shows its homework.
 > The frozen blind semantic snapshots are harder still: v1 reaches 26.7% recall /
-> 13.3% FPR, v2 reaches 0% / 10%, and v3 reaches 30% / 30%; the 90-row aggregate
-> is 22.2% / 20%. Run `shadowshield benchmark --generalization all`; these gaps
-> are public by design.
+> 13.3% FPR, v2 reaches 0% / 10%, v3 reaches 30% / 30%, and the 58-example v4
+> snapshot (indirect tool-result, multilingual, and semantic-pretext attacks)
+> reaches 58.6% / 6.9%; the 148-row aggregate is 36.5% / 14.9%. Run
+> `shadowshield benchmark --generalization all`; these gaps are public by design.
 
 ---
 
@@ -207,8 +209,8 @@ pip install "shadowshield[dashboard]"      # + FastAPI HTTP server & dashboard
 pip install "shadowshield[all]"            # everything
 ```
 
-Core deps are intentionally small: `pydantic`, `structlog`, `pyyaml`, `httpx`,
-`tiktoken`. The ML classifier, Presidio PII, dataset loaders, and dashboard live
+Core deps are intentionally small: `pydantic`, `structlog`, `pyyaml`, and
+`httpx`. The ML classifier, Presidio PII, dataset loaders, and dashboard live
 behind extras — the default install pulls **no** heavy ML stack.
 
 ---
@@ -297,7 +299,9 @@ shadowshield init > shield.yaml # write an annotated default config
 shadowshield benchmark          # run the bundled offline benchmark
 shadowshield benchmark --adversarial
 shadowshield benchmark --generalization all # all frozen blind semantic snapshots
+shadowshield calibrate --input benchmark.json --output calibration.json # isotonic score calibration
 shadowshield serve              # HTTP server + live dashboard (needs [dashboard])
+shadowshield proxy --upstream https://api.openai.com --port 8100  # gateway mode (needs [dashboard])
 ```
 
 ### 9. HTTP server (any language / a browser dashboard)
@@ -317,6 +321,62 @@ Endpoints: `GET /health` (liveness), `GET /ready` (readiness), `POST /scan`,
 `api_keys` is supplied; local-only trusted embeddings must explicitly pass
 `allow_insecure_local=True`.
 
+### 10. Gateway mode — guardrails without code changes
+
+Put ShadowShield *in front of* any OpenAI-compatible endpoint and point your
+existing SDK at the proxy instead. Chat messages are scanned pre-flight (a
+blocked request returns an OpenAI-style `403` and never reaches the upstream),
+completions are scanned post-flight, and malicious **SSE streams are cut
+mid-flight** with a conventional `finish_reason="content_filter"` chunk:
+
+```bash
+pip install "shadowshield[dashboard]"
+shadowshield proxy --upstream https://api.openai.com --port 8100 \
+  --api-key "$GATEWAY_KEY"        # proxy auth; upstream key stays in Authorization
+```
+
+```python
+client = OpenAI(base_url="http://localhost:8100/v1", api_key=os.environ["OPENAI_API_KEY"])
+```
+
+The same protection embeds in-process via `ShieldASGIMiddleware`
+(`shadowshield.middleware.asgi`) for any ASGI app — SSE passes through there,
+so use the proxy when you stream.
+
+### 11. Streaming completions — cut the stream mid-flight
+
+`StreamScanner` scans a completion *while it streams* (bounded memory,
+carry-over window so split signatures still match) and returns a terminal
+verdict the moment the stream must stop:
+
+```python
+scanner = shield.stream_scanner(scan_interval_chars=256)
+for event in openai_stream:                    # any streaming SDK
+    terminal = scanner.feed(event.delta_text)
+    if terminal is not None:
+        break                                  # close the stream NOW
+final = scanner.finalize()
+```
+
+### 12. Anthropic, LiteLLM, and RAG pipelines
+
+```python
+from shadowshield.middleware import (
+    ShieldedAnthropicClient, shielded_completion, scan_retrieved_chunks,
+)
+
+anthropic_client = ShieldedAnthropicClient(Anthropic(), shield)  # messages.create guarded
+completion = shielded_completion(shield)                         # wraps litellm.completion
+
+# Retrieved chunks are untrusted: drop poisoned documents before the prompt.
+report = scan_retrieved_chunks(shield, retrieved_chunks, on_threat="drop")
+prompt = build_prompt(query, report.safe_chunks)
+```
+
+Duck-typed wrappers for LlamaIndex (`ShieldedLlamaIndexRetriever`) and Haystack
+(`ShieldedHaystackRetriever`) filter poisoned nodes/documents inside each
+framework's retriever contract.
+
 ### Production container
 
 The included container runs the full control plane as a non-root user with a
@@ -330,7 +390,7 @@ export SHADOWSHIELD_ADMIN_KEY="$(openssl rand -hex 32)"
 export SHADOWSHIELD_POLICY_KEY="$(openssl rand -hex 32)"
 export SHADOWSHIELD_POLICY_STATE_KEY="$(openssl rand -hex 32)"
 export SHADOWSHIELD_IMAGE_DIGEST="$(curl -fsSL \
-  https://github.com/0xsl1m/shadowshield/releases/download/v0.6.2/container-digest.txt)"
+  https://github.com/0xsl1m/shadowshield/releases/download/v0.7.0/container-digest.txt)"
 docker compose pull
 docker compose up -d
 ```
@@ -345,20 +405,31 @@ it beyond localhost. See the
 [production-readiness roadmap](docs/PRODUCTION_READINESS.md) for launch gates,
 known scale limits, and the operator checklist.
 
+**Kubernetes:** `deploy/helm/shadowshield/` ships a chart with the same
+hardening (digest-pinned image — install fails without it, read-only root
+filesystem, all capabilities dropped, non-root, seccomp, bounded `/tmp`,
+external secrets, policy-state PVC, health probes):
+
+```bash
+helm install shadowshield deploy/helm/shadowshield \
+  --set image.digest=sha256:<release-digest> \
+  --set secrets.existingSecret=shadowshield-secrets
+```
+
 Upgrading a control-plane volume from 0.6.0 requires an offline re-key because
 0.6.0 authenticated durable state with the policy-signing key:
 
 ```bash
 # Load the existing scan/admin keys first so the new Compose file can resolve.
 export SHADOWSHIELD_IMAGE_DIGEST="$(curl -fsSL \
-  https://github.com/0xsl1m/shadowshield/releases/download/v0.6.2/container-digest.txt)"
+  https://github.com/0xsl1m/shadowshield/releases/download/v0.7.0/container-digest.txt)"
 export SHADOWSHIELD_POLICY_KEY="<existing-0.6.0-policy-key>"
 export SHADOWSHIELD_POLICY_STATE_KEY="$(openssl rand -hex 32)"
 # Stop every writer and snapshot the volume before running the migration.
 docker compose stop shadowshield
 docker compose run --rm --no-deps shadowshield \
   shadowshield migrate-policy-state --path /var/lib/shadowshield/policy-state.json
-# Preserve the reported .pre-0.6.1.bak file, then start 0.6.2.
+# Preserve the reported .pre-0.6.1.bak file, then start 0.7.0.
 docker compose up -d
 ```
 

{shadowshield-0.6.2 → shadowshield-0.7.0}/README.md

@@ -62,8 +62,10 @@ print(result.safe_text)            # safe fallback message
   false-positive rate on hard negatives** in the bundled benchmark.
 - **Proven, reproducibly.** Ships an eval harness + offline benchmark:
   `shadowshield benchmark`. Loads public datasets (PINT/deepset/InjecAgent) too.
-- **Drop-in integrations.** OpenAI-compatible clients, LangChain, decorators,
-  context managers, **async** (`ascan`). Or call `shield.scan()` directly.
+- **Drop-in integrations.** A zero-code **reverse-proxy gateway** for any
+  OpenAI-compatible endpoint, plus OpenAI/Anthropic/LiteLLM/LangChain wrappers,
+  pure-ASGI middleware, RAG retriever guards, decorators, context managers, and
+  **async** (`ascan`). Or call `shield.scan()` directly.
 - **Extensible & lightweight.** Add a detector/responder in ~10 lines or ship a
   plugin. Tiny core dependency set; ML/PII/datasets are optional extras.
 
@@ -76,9 +78,10 @@ print(result.safe_text)            # safe fallback message
 > in-distribution **regression baseline, not a SOTA claim**. We publish the humbling
 > external numbers on purpose — a credible security tool shows its homework.
 > The frozen blind semantic snapshots are harder still: v1 reaches 26.7% recall /
-> 13.3% FPR, v2 reaches 0% / 10%, and v3 reaches 30% / 30%; the 90-row aggregate
-> is 22.2% / 20%. Run `shadowshield benchmark --generalization all`; these gaps
-> are public by design.
+> 13.3% FPR, v2 reaches 0% / 10%, v3 reaches 30% / 30%, and the 58-example v4
+> snapshot (indirect tool-result, multilingual, and semantic-pretext attacks)
+> reaches 58.6% / 6.9%; the 148-row aggregate is 36.5% / 14.9%. Run
+> `shadowshield benchmark --generalization all`; these gaps are public by design.
 
 ---
 
@@ -134,8 +137,8 @@ pip install "shadowshield[dashboard]"      # + FastAPI HTTP server & dashboard
 pip install "shadowshield[all]"            # everything
 ```
 
-Core deps are intentionally small: `pydantic`, `structlog`, `pyyaml`, `httpx`,
-`tiktoken`. The ML classifier, Presidio PII, dataset loaders, and dashboard live
+Core deps are intentionally small: `pydantic`, `structlog`, `pyyaml`, and
+`httpx`. The ML classifier, Presidio PII, dataset loaders, and dashboard live
 behind extras — the default install pulls **no** heavy ML stack.
 
 ---
@@ -224,7 +227,9 @@ shadowshield init > shield.yaml # write an annotated default config
 shadowshield benchmark          # run the bundled offline benchmark
 shadowshield benchmark --adversarial
 shadowshield benchmark --generalization all # all frozen blind semantic snapshots
+shadowshield calibrate --input benchmark.json --output calibration.json # isotonic score calibration
 shadowshield serve              # HTTP server + live dashboard (needs [dashboard])
+shadowshield proxy --upstream https://api.openai.com --port 8100  # gateway mode (needs [dashboard])
 ```
 
 ### 9. HTTP server (any language / a browser dashboard)
@@ -244,6 +249,62 @@ Endpoints: `GET /health` (liveness), `GET /ready` (readiness), `POST /scan`,
 `api_keys` is supplied; local-only trusted embeddings must explicitly pass
 `allow_insecure_local=True`.
 
+### 10. Gateway mode — guardrails without code changes
+
+Put ShadowShield *in front of* any OpenAI-compatible endpoint and point your
+existing SDK at the proxy instead. Chat messages are scanned pre-flight (a
+blocked request returns an OpenAI-style `403` and never reaches the upstream),
+completions are scanned post-flight, and malicious **SSE streams are cut
+mid-flight** with a conventional `finish_reason="content_filter"` chunk:
+
+```bash
+pip install "shadowshield[dashboard]"
+shadowshield proxy --upstream https://api.openai.com --port 8100 \
+  --api-key "$GATEWAY_KEY"        # proxy auth; upstream key stays in Authorization
+```
+
+```python
+client = OpenAI(base_url="http://localhost:8100/v1", api_key=os.environ["OPENAI_API_KEY"])
+```
+
+The same protection embeds in-process via `ShieldASGIMiddleware`
+(`shadowshield.middleware.asgi`) for any ASGI app — SSE passes through there,
+so use the proxy when you stream.
+
+### 11. Streaming completions — cut the stream mid-flight
+
+`StreamScanner` scans a completion *while it streams* (bounded memory,
+carry-over window so split signatures still match) and returns a terminal
+verdict the moment the stream must stop:
+
+```python
+scanner = shield.stream_scanner(scan_interval_chars=256)
+for event in openai_stream:                    # any streaming SDK
+    terminal = scanner.feed(event.delta_text)
+    if terminal is not None:
+        break                                  # close the stream NOW
+final = scanner.finalize()
+```
+
+### 12. Anthropic, LiteLLM, and RAG pipelines
+
+```python
+from shadowshield.middleware import (
+    ShieldedAnthropicClient, shielded_completion, scan_retrieved_chunks,
+)
+
+anthropic_client = ShieldedAnthropicClient(Anthropic(), shield)  # messages.create guarded
+completion = shielded_completion(shield)                         # wraps litellm.completion
+
+# Retrieved chunks are untrusted: drop poisoned documents before the prompt.
+report = scan_retrieved_chunks(shield, retrieved_chunks, on_threat="drop")
+prompt = build_prompt(query, report.safe_chunks)
+```
+
+Duck-typed wrappers for LlamaIndex (`ShieldedLlamaIndexRetriever`) and Haystack
+(`ShieldedHaystackRetriever`) filter poisoned nodes/documents inside each
+framework's retriever contract.
+
 ### Production container
 
 The included container runs the full control plane as a non-root user with a
@@ -257,7 +318,7 @@ export SHADOWSHIELD_ADMIN_KEY="$(openssl rand -hex 32)"
 export SHADOWSHIELD_POLICY_KEY="$(openssl rand -hex 32)"
 export SHADOWSHIELD_POLICY_STATE_KEY="$(openssl rand -hex 32)"
 export SHADOWSHIELD_IMAGE_DIGEST="$(curl -fsSL \
-  https://github.com/0xsl1m/shadowshield/releases/download/v0.6.2/container-digest.txt)"
+  https://github.com/0xsl1m/shadowshield/releases/download/v0.7.0/container-digest.txt)"
 docker compose pull
 docker compose up -d
 ```
@@ -272,20 +333,31 @@ it beyond localhost. See the
 [production-readiness roadmap](docs/PRODUCTION_READINESS.md) for launch gates,
 known scale limits, and the operator checklist.
 
+**Kubernetes:** `deploy/helm/shadowshield/` ships a chart with the same
+hardening (digest-pinned image — install fails without it, read-only root
+filesystem, all capabilities dropped, non-root, seccomp, bounded `/tmp`,
+external secrets, policy-state PVC, health probes):
+
+```bash
+helm install shadowshield deploy/helm/shadowshield \
+  --set image.digest=sha256:<release-digest> \
+  --set secrets.existingSecret=shadowshield-secrets
+```
+
 Upgrading a control-plane volume from 0.6.0 requires an offline re-key because
 0.6.0 authenticated durable state with the policy-signing key:
 
 ```bash
 # Load the existing scan/admin keys first so the new Compose file can resolve.
 export SHADOWSHIELD_IMAGE_DIGEST="$(curl -fsSL \
-  https://github.com/0xsl1m/shadowshield/releases/download/v0.6.2/container-digest.txt)"
+  https://github.com/0xsl1m/shadowshield/releases/download/v0.7.0/container-digest.txt)"
 export SHADOWSHIELD_POLICY_KEY="<existing-0.6.0-policy-key>"
 export SHADOWSHIELD_POLICY_STATE_KEY="$(openssl rand -hex 32)"
 # Stop every writer and snapshot the volume before running the migration.
 docker compose stop shadowshield
 docker compose run --rm --no-deps shadowshield \
   shadowshield migrate-policy-state --path /var/lib/shadowshield/policy-state.json
-# Preserve the reported .pre-0.6.1.bak file, then start 0.6.2.
+# Preserve the reported .pre-0.6.1.bak file, then start 0.7.0.
 docker compose up -d
 ```
 

shadowshield-0.7.0/docs/AUDIT_REPORT_2026-08-05.md

@@ -0,0 +1,190 @@
+# ShadowShield — Comprehensive Audit Report
+
+**Project:** shadowshield v0.6.3 (`C:\Users\jhwil\Documents\OpenClaw VPS\shadowshield`)
+**Audit date:** 2026-08-05 · **Auditor:** Kimi Work (automated code audit)
+**Scope:** full repository — source, tests, packaging, CI/CD, container, docs, dependency & secret hygiene
+
+---
+
+## 1. Executive summary
+
+ShadowShield is a unified open-source security shield for agentic AI systems
+(prompt-injection / jailbreak / PII / exfiltration detection with a
+detect → decide → respond engine, plus an optional FastAPI control plane).
+
+**Overall verdict: strong.** This is an unusually disciplined codebase for a
+0.x project: the full test suite passes, strict mypy and Ruff are clean, the
+threat model is documented and consistently implemented, and the release
+pipeline is among the most hardened seen in open-source Python (hash-locked
+builds, reproducible-build gate, pinned actions, Trivy gates, OIDC publishing,
+fail-closed container startup tests). Findings are limited to one
+medium-severity API design issue and a set of low-severity gaps.
+
+**Verification results from this audit run (2026-08-05):**
+
+| Check | Result |
+|---|---|
+| Test suite (`pytest tests`) | **320 passed, 2 skipped** (opt-in real-model tests), 38.8 s |
+| Ruff lint | **Clean** (`All checks passed!`) |
+| mypy (strict, 54 source files) | **Clean** (`no issues found`) |
+| Coverage | **86%** total (branch), gate `fail_under = 80` met |
+| pip-audit (installed environment) | **No known vulnerabilities** |
+| Dangerous-sink scan (`eval/exec/pickle/subprocess/shell=True/yaml.load`) | **None in `src/`** |
+| Committed-secret scan (AWS/GitHub/OpenAI key patterns, private keys) | **None** (2 hits are intentional test fixtures) |
+| Git working tree | Clean; 146 tracked files; build artifacts correctly ignored |
+
+---
+
+## 2. Strengths confirmed
+
+### 2.1 Detection core
+- **ReDoS-conscious regexes.** Signature patterns use bounded quantifiers
+  (`[\w\s,'’]{0,40}?` etc.); multilingual regex groups are pre-filtered by
+  cheap vocabulary cues before running (`_candidate_signatures`,
+  `prompt_injection.py:487`). Exfiltration detector explicitly documents a
+  rewrite away from a quadratic pattern.
+- **Defense in depth.** Normalization (zero-width, homoglyph, bidi stripping),
+  base64/hex payload decoding with re-scan and severity bump, canary tokens for
+  detecting *successful* injections, optional transformer/vector/LLM-judge
+  layers that are lazy-imported and never pulled in by the base install.
+- **Bounded outputs.** `MAX_FINDINGS_PER_DETECTOR`, truncated matched text
+  (`m.group(0)[:160]`), content-free telemetry — detection failures cannot
+  amplify into memory or leak content into logs.
+
+### 2.2 HTTP control plane (`_security.py`, `control.py`, `server.py`)
+- **Fail-closed by default:** the app factory raises if API/admin keys are
+  missing; insecure mode requires an explicit `allow_insecure_local=True`.
+- **Early authentication** before body reads; constant-time key comparison via
+  `hmac.compare_digest` (byte-safe for non-ASCII input).
+- **Intake hardening:** 1 MiB body cap, 8,192-frame cap, single 15 s total read
+  deadline (defeats slow/chunked-body starvation of admission slots),
+  503-with-`Retry-After` concurrency cap (16) on scan paths, `OPTIONS`
+  authenticated like other requests.
+- **Credential hygiene enforced at startup:** scan keys, admin keys, policy
+  signing key, and policy-state key must all be pairwise distinct
+  (constant-time overlap check); policy-state key has a minimum length.
+- **Browser hardening:** CSP, `nosniff`, `DENY` framing, `no-store`, COOP,
+  conditional HSTS; docs/redoc/openapi endpoints disabled.
+
+### 2.3 Policy push (`core/policy.py`) — the standout design
+Signed (HMAC-SHA256, pluggable verifier) config bundles with a **structural
+protection floor**: allow-listed fields only, always-on detectors cannot be
+disabled or de-weighted below baseline, block-threshold ceiling, aggregate
+degradation cap, fail-safe (never fail-open) semantics with durable HMAC'd
+anti-replay state. A compromised control plane cannot remotely weaken the
+fleet — this is the correct threat model, implemented end-to-end.
+
+### 2.4 Supply chain & CI
+- All GitHub Actions **SHA-pinned**; `persist-credentials: false`; read-only
+  token permissions; actionlint gate.
+- **Reproducible-build gate** (double build + `cmp`), Twine metadata check,
+  installed-wheel smoke test.
+- Hash-locked build/container lockfiles (`uv pip compile --generate-hashes`),
+  `--require-hashes` installs in the Dockerfile; base image **digest-pinned**.
+- Per-extra dependency-audit matrix (dynamically discovered), Trivy SBOM +
+  fail-on-fixable-CRITICAL/HIGH gate, PyPI OIDC publishing, SLSA/CycloneDX
+  attestations, Dependabot, pre-commit.
+- Container: non-root user, read-only root fs, `cap_drop: ALL`,
+  `no-new-privileges`, pids/mem/cpu limits, healthcheck; CI proves the image
+  **refuses to start without keys** and exercises 401/200/413 paths end-to-end.
+
+### 2.5 Honest posture
+`docs/PRODUCTION_READINESS.md` reports blind-generalization detection honestly
+(v3: 30% ASR / 30% FPR — labeled **Beta**, not oversold), separates
+library-ready vs operator-owned items, and the changelog is precise about what
+each release changed.
+
+---
+
+## 3. Findings
+
+### M-1 (Medium) — `apply_bundle` accepts unsigned bundles when no verifier is passed
+`src/shadowshield/core/policy.py:250` — `if verifier is not None and not
+verifier(bundle): raise PolicyRejected`. When the caller omits `verifier`, an
+**unsigned bundle is silently applied**, contradicting the module's own
+documented invariant ("an unsigned/badly-signed bundle is rejected"). The
+control plane compensates (`control.py:1286` rejects remote updates when no
+verifier exists unless `allow_insecure_local`), so the deployed path is safe —
+but the **library API itself is fail-open by default**, and any future caller
+that forgets the verifier gets no protection.
+
+**Recommendation:** make verification mandatory by default — e.g. raise unless
+`verifier` is provided, or add an explicit `allow_unsigned: bool = False`
+opt-in so skipping signature checks is a deliberate, greppable decision.
+
+### L-1 (Low) — Coverage gaps in glue/integration code
+86% overall, but: `middleware/langchain.py` **0%**, `plugins/manager.py` 33%,
+`integrations/agentdojo.py` 23%, `middleware/base.py` 57%, `detectors/pii.py`
+77%. LangChain middleware and the plugin manager are user-facing extension
+points; their error paths are exactly where misuse bugs live. The AgentDojo
+adapter is acceptable to leave low (needs API keys), but langchain middleware
+and plugin-manager state transitions deserve unit tests with fakes.
+
+### L-2 (Low) — `control.py` is a 1,448-line god-file
+App factory, auth, policy endpoints, metrics/Prometheus, dashboard HTML, and
+CLI glue in one module. It is well organized internally, but this size raises
+audit cost and regression risk per edit. Consider splitting into
+`control/auth.py`, `control/policy_api.py`, `control/metrics.py`,
+`control/dashboard.py`.
+
+### L-3 (Low) — Reporter transport hardening
+`reporter.py:_http_transport`: TLS verification relies on httpx defaults
+(fine), but there is no scheme enforcement — an `http://` endpoint would ship
+telemetry plus the `x-api-key` header in cleartext. Add an https-only check
+(or loud warning) at construction. The bounded queue with silent drop-counting
+is good; consider emitting a log line when drops begin so operators notice
+collector outages.
+
+### L-4 (Low) — Sanitizer overlapping spans
+`sanitizer.py` replaces spans right-to-left with index clamping — correct for
+nested/disjoint spans, but two overlapping detector spans can yield nested
+`[redacted:…]` placeholders. Harmless (no corruption), but a span-merge pass
+would produce cleaner output.
+
+### L-5 (Info) — Local pip-audit of `container.lock` fails offline
+`pip-audit -r requirements/container.lock` errors locally because the lock
+contains a direct-URL `colorama` entry combined with `--require-hashes`. CI
+audits it with `--no-deps`, which passes — so this is a reproduction quirk,
+not a defect. Worth one line in `requirements/` docs for anyone auditing
+locally.
+
+### L-6 (Info) — Local environment drift
+`uvicorn` is not installed in the local `.venv` despite being in the
+`dashboard` extra (tests pass regardless). Also `.coverage`/`coverage.xml`/
+`dist/`/`production-dist/` artifacts exist locally (all correctly gitignored).
+Housekeeping only.
+
+---
+
+## 4. Non-findings (checked, no issue)
+
+- **Committed secrets:** none. Two regex hits are deliberate detector test
+  fixtures (`tests/test_detectors.py:131`, `tests/test_telemetry.py:19`).
+- **`.env.example`:** empty values by design; `compose.yaml` uses
+  `${VAR:?…}` so copying the example can never boot with placeholder creds.
+- **Dangerous dynamic execution:** no `eval/exec/pickle/subprocess/shell=True`
+  in `src/`; YAML usage is safe; FastAPI apps disable introspection endpoints.
+- **Skipped tests:** both skips are opt-in real-model tests covered by the
+  dedicated `ml-integration` CI job, not silent gaps.
+- **Site headers:** `site/vercel.json` ships a strict CSP with per-script
+  hashes, HSTS preload, and a restrictive Permissions-Policy.
+
+## 5. Recommended next actions (priority order)
+
+1. **M-1:** make bundle signature verification non-skippable by default in
+   `core/policy.py` (small change, closes the only fail-open path).
+2. **L-1:** add unit tests for `middleware/langchain.py` and
+   `plugins/manager.py` error paths.
+3. **L-2:** split `control.py` before it grows further.
+4. **L-3:** enforce https (or warn) on `Reporter` endpoints.
+5. Keep the blind-benchmark program running; the honest Beta label is a
+   strength — protect it.
+
+---
+
+*Verification commands: `.venv/Scripts/python.exe -m pytest tests -q`,
+`-m ruff check .`, `-m mypy`, `-m coverage report`,
+`uv tool run pip-audit --path .venv/Lib/site-packages`,
+plus targeted source review of `_security.py`, `core/policy.py`,
+`core/engine.py`, `responders/sanitizer.py`, `reporter.py`, `control.py`,
+CI workflows, Dockerfile, and `compose.yaml`.*

{shadowshield-0.6.2 → shadowshield-0.7.0}/docs/BENCHMARKS.md

@@ -26,14 +26,40 @@ pip install "shadowshield[transformers]"
 shadowshield benchmark --hf deepset/prompt-injections --split test --transformer
 ```
 
+The CLI warms every configured detector before starting the latency clock. Its
+JSON and text reports include a runtime-integrity section covering warmup,
+readiness, and bounded per-detector failure counts. A benchmark with a warmup
+failure, a detector that remains unready, or any detector error exits non-zero
+and is marked **unreliable**; its confusion counts remain available for
+diagnosis but must not be published as quality evidence. CLI benchmark shields
+disable normal audit emission so logging I/O does not distort latency or corrupt
+the command's machine-readable JSON stdout.
+
+Programmatic callers can request the same behavior with
+`evaluate_shield(shield, examples, warmup=True)`. The default remains
+`warmup=False` for API compatibility. In both modes, readiness is checked after
+the scans and per-scan detector error counters are aggregated.
+
+Per-category output retains the original `total`, `flagged`, and flag-rate
+fields, and additionally reports class-conditional TP/FP/TN/FN, recall, FPR,
+and balanced accuracy. Recall is `null`/`n/a` for categories with no attack
+rows; FPR is `null`/`n/a` for categories with no benign rows; balanced accuracy
+is defined only when both classes are present.
+
+Aggregate and per-category recall/FPR also include two-sided 95% Wilson score
+intervals. Wilson intervals remain informative at small sample sizes and at
+observed rates of 0% or 100%; an interval is `null`/`n/a` when that category has
+no rows for the corresponding class. Existing scalar metric fields are
+unchanged, and the interval fields are additive.
+
 ## 1. Bundled benchmark (in-distribution — a regression baseline)
 
 75 curated examples (40 attack / 35 benign, incl. 16 NotInject-style hard
 negatives). `balanced` mode:
 
-| recall | FPR | precision | F1 | p50 |
+| recall (95% CI) | FPR (95% CI) | precision | F1 | p50 |
 |---:|---:|---:|---:|---:|
-| 100% | 0% | 100% | 100% | 0.16 ms |
+| 100% [91.2%, 100%] | 0% [0%, 9.9%] | 100% | 100% | 0.16 ms |
 
 **This is a regression baseline and a smoke test — NOT a claim of real-world
 accuracy.** 100% on our own set just means we don't regress on the attack
@@ -45,9 +71,9 @@ The 36-row adversarial catalogue includes obfuscation, multilingual and indirect
 attacks, plus benign trigger-heavy counterexamples. It improved from
 15/2/16/3 to 18/0/18/0 (TP/FP/TN/FN):
 
-| recall | FPR | precision | F1 |
+| recall (95% CI) | FPR (95% CI) | precision | F1 |
 |---:|---:|---:|---:|
-| 100% | 0% | 100% | 100% |
+| 100% [82.4%, 100%] | 0% [0%, 17.6%] | 100% | 100% |
 
 This is still a curated regression set. The signatures were developed with these
 cases visible, so its perfect score is not a generalization claim.
@@ -67,12 +93,12 @@ v1 SHA-256 `b3281ba1a42d266bb930bbb41943016d47b38dbc822ff7cff5131f3448a0248f`;
 v2 SHA-256 `aa8b8c81c00a55bb65180e15ff743b6241d24845b3886e8e60b52b9b23db47fa`;
 v3 SHA-256 `2285031e8143572311a522a4b6ec1a39c96a34ac2b42785f17818e8a145342bf`.
 
-| snapshot | rows | TP/FP/TN/FN | recall | FPR | balanced accuracy |
+| snapshot | rows | TP/FP/TN/FN | recall (95% CI) | FPR (95% CI) | balanced accuracy |
 |---|---:|---:|---:|---:|---:|
-| v1 | 30 | 4/2/13/11 | 26.7% | 13.3% | 56.7% |
-| v2 | 20 | 0/1/9/10 | 0% | 10% | 45% |
-| v3 | 40 | 6/6/14/14 | 30% | 30% | 50% |
-| **aggregate** | **90** | **10/9/36/35** | **22.2%** | **20%** | **51.1%** |
+| v1 | 30 | 4/2/13/11 | 26.7% [10.9%, 52.0%] | 13.3% [3.7%, 37.9%] | 56.7% |
+| v2 | 20 | 0/1/9/10 | 0% [0%, 27.8%] | 10% [1.8%, 40.4%] | 45% |
+| v3 | 40 | 6/6/14/14 | 30% [14.5%, 51.9%] | 30% [14.5%, 51.9%] | 50% |
+| **aggregate** | **90** | **10/9/36/35** | **22.2% [12.5%, 36.3%]** | **20% [10.9%, 33.8%]** | **51.1%** |
 
 The v3 snapshot was opened only after a detector candidate and acceptance bar
 were frozen. That candidate raised aggregate recall to 55.6% but failed the