stackone-defender 0.7.3__tar.gz → 0.8.2__tar.gz

stackone_defender-0.8.2/.release-please-manifest.json

@@ -0,0 +1 @@
+{".":"0.8.2"}

{stackone_defender-0.7.3 → stackone_defender-0.8.2}/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## [0.8.2](https://github.com/StackOneHQ/defender-py/compare/stackone-defender-v0.7.4...stackone-defender-v0.8.2) (2026-08-19)
+
+
+### Features
+
+* **ENG-1084:** detect-and-gate + sentence-level cleaning — Tier 1 detects without mutating content (gate on `allowed`); `sanitized` is a sentence-cleaned copy of high/critical fields ([60b01d0](https://github.com/StackOneHQ/defender-py/commit/60b01d0117c04106a3017dc4bf536cce6cc7a6c4))
+* **ENG-1084:** mark dropped runs with `[CONTENT SANITISED]` in the cleaned copy ([fdcdb94](https://github.com/StackOneHQ/defender-py/commit/fdcdb94e27698ac71a8dd8933d54ad47455be376))
+* **ENG-1084:** `detected_field_count` threat-count signal; object-key scanning; Tier 2 availability (`require_tier2` / `tier2_available`) ([6f1b52a](https://github.com/StackOneHQ/defender-py/commit/6f1b52afd9d2932aab9462776c102d05e5bd9626))
+* **ENG-1084:** NFKC-fold unicode before Tier 2 classification ([50741ee](https://github.com/StackOneHQ/defender-py/commit/50741eeb7c9d7e435a2a504d2f83c3bd9b2b8c04))
+
+
+### Bug Fixes
+
+* **ENG-1084:** return large arrays in full — 0.7.4 truncated arrays >1000 items to 100 + a sentinel, dropping the rest; Tier 1 detection is now bounded by the `max_size` budget (flagged via `coverage_degraded`), never drops data ([26c7eec](https://github.com/StackOneHQ/defender-py/commit/26c7eecb10f55be0a9e7cb614a67629fa485d199))
+* **ENG-1084:** scan strings inside risky array fields (`{"name": [INJ]}`) ([6f1b52a](https://github.com/StackOneHQ/defender-py/commit/6f1b52afd9d2932aab9462776c102d05e5bd9626))
+* **ENG-1084:** evidence-driven encoding escalation; token-degeneracy (OOD) guard; decorative-run normalization ([f450992](https://github.com/StackOneHQ/defender-py/commit/f45099241e36ee4210daaf39f2573c2a88d3071b))
+* **ENG-1084:** gate sentence-cleaning on aggregate risk; verbatim when nothing is dropped ([9e25335](https://github.com/StackOneHQ/defender-py/commit/9e2533576f90574f8bab5476728d79c5275b9060))
+* **ENG-1084:** count objects/arrays once in size metrics + iterate `obj.items()` directly ([26c7eec](https://github.com/StackOneHQ/defender-py/commit/26c7eecb10f55be0a9e7cb614a67629fa485d199))
+* **ENG-1084:** warmup fail-open + dict-subclass stripping ([f49ffcf](https://github.com/StackOneHQ/defender-py/commit/f49ffcf6e93e322623447d217994cc8215e02410))
+
+## [0.7.4](https://github.com/StackOneHQ/defender-py/compare/stackone-defender-v0.7.3...stackone-defender-v0.7.4) (2026-08-13)
+
+
+### Features
+
+* **ENG-1761:** port Tier 2 buckets, dedupe, and cost telemetry from TS 0.7.4 ([b92b9f6](https://github.com/StackOneHQ/defender-py/commit/b92b9f661ef927b27542a664d92477e039279758))
+* **ENG-1761:** port Tier 2 fixed-width buckets, dedupe, cost telemetry from TS 0.7.4 ([1e63d09](https://github.com/StackOneHQ/defender-py/commit/1e63d0999d003348c17d6700bf6a03b65a7b2242))
+
+
+### Bug Fixes
+
+* **ENG-1761:** address review — cold_load on failure paths + stronger tests ([e897786](https://github.com/StackOneHQ/defender-py/commit/e897786ff013b95b7b74eecd8df554ee978dd98b))
+
 ## [0.7.3](https://github.com/StackOneHQ/defender-py/compare/stackone-defender-v0.7.2...stackone-defender-v0.7.3) (2026-07-22)
 
 

{stackone_defender-0.7.3 → stackone_defender-0.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: stackone-defender
-Version: 0.7.3
+Version: 0.8.2
 Summary: Indirect prompt injection defense for AI agents using tool calls
 Project-URL: Homepage, https://github.com/StackOneHQ/stackone-defender
 Project-URL: Repository, https://github.com/StackOneHQ/stackone-defender
@@ -116,10 +116,12 @@ else:
 
 ### Tier 1 — Pattern detection (sync, ~1 ms)
 
-- **Unicode normalization** — homoglyph resistance (e.g. Cyrillic `а` → ASCII `a`)
-- **Role stripping** — `SYSTEM:`, `ASSISTANT:`, `<system>`, `[INST]`, etc.
-- **Pattern removal** — phrases like “ignore previous instructions”
-- **Encoding detection** — suspicious Base64/URL-shaped payloads
+Detects (never rewrites) injection signals and records them as evidence:
+
+- **Unicode normalization** (for matching only) — homoglyph resistance (e.g. Cyrillic `а` → ASCII `a`)
+- **Role markers** — `SYSTEM:`, `ASSISTANT:`, `<system>`, `[INST]`, etc.
+- **Instruction-override patterns** — phrases like “ignore previous instructions”
+- **Encoding detection** — decode-then-detect: a decoded payload is escalated only if it trips a real attack pattern
 - **Boundary annotation (opt-in)** — `[UD-{id}]…[/UD-{id}]` wrappers when `annotate_boundary=True` (npm: `annotateBoundary`). Use `generate_boundary_instructions` from the package root in prompts when you enable wrapping.
 
 ### Tier 2 — ML classification (ONNX)
@@ -133,7 +135,7 @@ Packed-chunk MiniLM classifier (int8 ONNX ~22 MB, bundled):
 ### Optional SFE preprocessor
 
 - `use_sfe=True` runs a field-level FastText pass to build a **classifier-only** view of the payload
-- **Tier 1** always sanitizes the **original** tool value; **`sanitized`** in `DefenseResult` is unchanged by SFE drops
+- **Tier 1** detects on the raw tool value; SFE drops are classifier-only and never remove fields from the returned `sanitized` payload
 - **Tier 2** extracts strings from the SFE-filtered tree; `fields_dropped` lists paths omitted from that extraction (not removed from `sanitized`)
 - Fails open if the runtime/model is unavailable: payload continues unfiltered
 
@@ -152,7 +154,7 @@ Authoritative LLM-based classification for the cases Tier 2 finds ambiguous. The
 
 Two modes, selected via `defender_mode`:
 - **`"cascade"`** (default): Tier 1 → Tier 2 → Tier 3, with Tier 3 invoked only when the Tier 2 effective score falls in the gray band (default `[0.3, 0.85)`). The Tier 3 verdict overrides Tier 2 on the escalated chunk — a `block` forces a block, an `allow` rescues it. Outside the band defender skips the round trip.
-- **`"tier3_only"`**: skip Tier 2; the block/allow decision is the Tier 3 verdict alone. Tier 1 still sanitizes the returned `sanitized` payload.
+- **`"tier3_only"`**: skip Tier 2; the block/allow decision is the Tier 3 verdict alone. Tier 1 still runs detection; the returned `sanitized` payload is the original content.
 
 Register a provider once at startup, then opt in per instance:
 
@@ -189,8 +191,9 @@ defense = create_prompt_defense(
 
 ### `allowed` vs `risk_level`
 
+- **`DefenseResult.sanitized`** is a **sentence-level cleaned** copy of the tool result (high-scoring sentences dropped within high-risk fields, optionally `[UD-…]` boundary-wrapped). Cleaning is best-effort (capped by detection) — still gate on `allowed`. Set `sanitize_content=False` for pure detect-and-gate: `sanitized` is then the input verbatim.
 - Use **`allowed`** for gating when `block_high_risk=True`: `False` means do not pass `sanitized` to the model as-is.
-- **`risk_level`** is diagnostic: it starts at `default_risk_level` (default `"medium"`) and is **escalated** by Tier 1 / Tier 2 signals — not reduced. Use it for logging, not as the sole block signal unless you implement your own policy.
+- **`risk_level`** is diagnostic: it starts at `default_risk_level` (default `"low"`) and is **escalated** by Tier 1 / Tier 2 signals — not reduced. Use it for logging, not as the sole block signal unless you implement your own policy.
 
 | Level | Typical trigger |
 |-------|------------------|
@@ -206,8 +209,9 @@ defense = create_prompt_defense(
 defense = create_prompt_defense(
     enable_tier1=True,
     enable_tier2=True,
+    require_tier2=False,  # True: raise if Tier 2 can't load (fail closed) instead of degrading to Tier 1
     block_high_risk=False,
-    default_risk_level="medium",
+    default_risk_level="low",
     annotate_boundary=False,  # True: wrap risky strings with [UD-…] tags (npm: annotateBoundary)
     tier2_fields=["subject", "body", "snippet"],  # optional: scope Tier 2 to these JSON keys (default: all strings)
     use_sfe=True,  # optional: enable semantic field extractor preprocessing
@@ -229,18 +233,31 @@ from dataclasses import dataclass, field
 
 @dataclass
 class DefenseResult:
-    allowed: bool
-    risk_level: RiskLevel
-    sanitized: Any
-    detections: list[str]
-    fields_sanitized: list[str]
-    patterns_by_field: dict[str, list[str]]
+    allowed: bool                             # gating decision (respects block_high_risk)
+    risk_level: RiskLevel                     # diagnostic; max of Tier 1 / Tier 2
+    sanitized: Any                            # sentence-cleaned copy (input verbatim when sanitize_content=False); dropped runs leave a [CONTENT SANITISED] marker; best-effort, still gate on allowed
+    detections: list[str]                     # Tier 1 pattern names detected
+    fields_sanitized: list[str]               # fields whose content the cleaner changed in sanitized (empty when sanitize_content=False or no Tier 2); for detections read detections/patterns_by_field
+    patterns_by_field: dict[str, list[str]]   # patterns detected per field
+    detected_field_count: int                 # count of fields with a Tier-1 detection (keys of patterns_by_field); threat-count signal (fields_sanitized len no longer tracks this)
     tier2_score: float | None = None
+    tier2_raw_score: float | None = None
+    tier2_aux_score: float | None = None      # multi-head models only
+    tier2_multihead_blocked: bool | None = None
     tier2_skip_reason: str | None = None
     max_sentence: str | None = None
+    tier3: Tier3Result | None = None          # present when Tier 3 ran
     fields_dropped: list[str] = field(default_factory=list)
     truncated_at_depth: bool | None = None
     latency_ms: float = 0.0
+    # Cost telemetry — present only when the batched Tier 2 classifier ran
+    phase_timings: PhaseTimings | None = None       # prepare / infer / aggregate ms
+    tier2_stats: Tier2Stats | None = None           # string/chunk/unique counts, real/padded tokens
+    tier1_ms: float | None = None
+    cold_load: bool | None = None
+    # Operational signals
+    tier2_available: bool | None = None       # False when Tier 2 enabled but failed to load
+    coverage_degraded: bool | None = None      # True when Tier 1 detection coverage was capped
 ```
 
 ### `defense.defend_tool_results(items)`
@@ -255,7 +272,7 @@ results = defense.defend_tool_results([
 ])
 for r in results:
     if not r.allowed:
-        print("Blocked:", ", ".join(r.fields_sanitized))
+        print("Blocked:", ", ".join(r.detections))
 ```
 
 ### `await defense.defend_tool_results_async(items)`
@@ -300,7 +317,7 @@ sanitized = run_tool_and_defend(gmail_api.get_message(msg_id), "gmail_get_messag
 
 ## Risky field detection
 
-Only **string** values under configured “risky” keys are scanned and sanitized. [`RiskyFieldConfig`](https://github.com/StackOneHQ/stackone-defender/blob/main/src/stackone_defender/types.py) provides global names/patterns plus **`tool_overrides`** (wildcard tool names → field list), same idea as the npm package.
+Only **string** values under configured “risky” keys are Tier-1-scanned — including strings nested inside arrays/objects under those keys (e.g. `{"name": ["…"]}`). [`RiskyFieldConfig`](https://github.com/StackOneHQ/stackone-defender/blob/main/src/stackone_defender/types.py) provides global names/patterns plus **`tool_overrides`** (wildcard tool names → field list), same idea as the npm package. (Tier 2 scans all extracted strings regardless.)
 
 | Tool pattern | Scanned fields |
 |--------------|----------------|

{stackone_defender-0.7.3 → stackone_defender-0.8.2}/README.md

@@ -90,10 +90,12 @@ else:
 
 ### Tier 1 — Pattern detection (sync, ~1 ms)
 
-- **Unicode normalization** — homoglyph resistance (e.g. Cyrillic `а` → ASCII `a`)
-- **Role stripping** — `SYSTEM:`, `ASSISTANT:`, `<system>`, `[INST]`, etc.
-- **Pattern removal** — phrases like “ignore previous instructions”
-- **Encoding detection** — suspicious Base64/URL-shaped payloads
+Detects (never rewrites) injection signals and records them as evidence:
+
+- **Unicode normalization** (for matching only) — homoglyph resistance (e.g. Cyrillic `а` → ASCII `a`)
+- **Role markers** — `SYSTEM:`, `ASSISTANT:`, `<system>`, `[INST]`, etc.
+- **Instruction-override patterns** — phrases like “ignore previous instructions”
+- **Encoding detection** — decode-then-detect: a decoded payload is escalated only if it trips a real attack pattern
 - **Boundary annotation (opt-in)** — `[UD-{id}]…[/UD-{id}]` wrappers when `annotate_boundary=True` (npm: `annotateBoundary`). Use `generate_boundary_instructions` from the package root in prompts when you enable wrapping.
 
 ### Tier 2 — ML classification (ONNX)
@@ -107,7 +109,7 @@ Packed-chunk MiniLM classifier (int8 ONNX ~22 MB, bundled):
 ### Optional SFE preprocessor
 
 - `use_sfe=True` runs a field-level FastText pass to build a **classifier-only** view of the payload
-- **Tier 1** always sanitizes the **original** tool value; **`sanitized`** in `DefenseResult` is unchanged by SFE drops
+- **Tier 1** detects on the raw tool value; SFE drops are classifier-only and never remove fields from the returned `sanitized` payload
 - **Tier 2** extracts strings from the SFE-filtered tree; `fields_dropped` lists paths omitted from that extraction (not removed from `sanitized`)
 - Fails open if the runtime/model is unavailable: payload continues unfiltered
 
@@ -126,7 +128,7 @@ Authoritative LLM-based classification for the cases Tier 2 finds ambiguous. The
 
 Two modes, selected via `defender_mode`:
 - **`"cascade"`** (default): Tier 1 → Tier 2 → Tier 3, with Tier 3 invoked only when the Tier 2 effective score falls in the gray band (default `[0.3, 0.85)`). The Tier 3 verdict overrides Tier 2 on the escalated chunk — a `block` forces a block, an `allow` rescues it. Outside the band defender skips the round trip.
-- **`"tier3_only"`**: skip Tier 2; the block/allow decision is the Tier 3 verdict alone. Tier 1 still sanitizes the returned `sanitized` payload.
+- **`"tier3_only"`**: skip Tier 2; the block/allow decision is the Tier 3 verdict alone. Tier 1 still runs detection; the returned `sanitized` payload is the original content.
 
 Register a provider once at startup, then opt in per instance:
 
@@ -163,8 +165,9 @@ defense = create_prompt_defense(
 
 ### `allowed` vs `risk_level`
 
+- **`DefenseResult.sanitized`** is a **sentence-level cleaned** copy of the tool result (high-scoring sentences dropped within high-risk fields, optionally `[UD-…]` boundary-wrapped). Cleaning is best-effort (capped by detection) — still gate on `allowed`. Set `sanitize_content=False` for pure detect-and-gate: `sanitized` is then the input verbatim.
 - Use **`allowed`** for gating when `block_high_risk=True`: `False` means do not pass `sanitized` to the model as-is.
-- **`risk_level`** is diagnostic: it starts at `default_risk_level` (default `"medium"`) and is **escalated** by Tier 1 / Tier 2 signals — not reduced. Use it for logging, not as the sole block signal unless you implement your own policy.
+- **`risk_level`** is diagnostic: it starts at `default_risk_level` (default `"low"`) and is **escalated** by Tier 1 / Tier 2 signals — not reduced. Use it for logging, not as the sole block signal unless you implement your own policy.
 
 | Level | Typical trigger |
 |-------|------------------|
@@ -180,8 +183,9 @@ defense = create_prompt_defense(
 defense = create_prompt_defense(
     enable_tier1=True,
     enable_tier2=True,
+    require_tier2=False,  # True: raise if Tier 2 can't load (fail closed) instead of degrading to Tier 1
     block_high_risk=False,
-    default_risk_level="medium",
+    default_risk_level="low",
     annotate_boundary=False,  # True: wrap risky strings with [UD-…] tags (npm: annotateBoundary)
     tier2_fields=["subject", "body", "snippet"],  # optional: scope Tier 2 to these JSON keys (default: all strings)
     use_sfe=True,  # optional: enable semantic field extractor preprocessing
@@ -203,18 +207,31 @@ from dataclasses import dataclass, field
 
 @dataclass
 class DefenseResult:
-    allowed: bool
-    risk_level: RiskLevel
-    sanitized: Any
-    detections: list[str]
-    fields_sanitized: list[str]
-    patterns_by_field: dict[str, list[str]]
+    allowed: bool                             # gating decision (respects block_high_risk)
+    risk_level: RiskLevel                     # diagnostic; max of Tier 1 / Tier 2
+    sanitized: Any                            # sentence-cleaned copy (input verbatim when sanitize_content=False); dropped runs leave a [CONTENT SANITISED] marker; best-effort, still gate on allowed
+    detections: list[str]                     # Tier 1 pattern names detected
+    fields_sanitized: list[str]               # fields whose content the cleaner changed in sanitized (empty when sanitize_content=False or no Tier 2); for detections read detections/patterns_by_field
+    patterns_by_field: dict[str, list[str]]   # patterns detected per field
+    detected_field_count: int                 # count of fields with a Tier-1 detection (keys of patterns_by_field); threat-count signal (fields_sanitized len no longer tracks this)
     tier2_score: float | None = None
+    tier2_raw_score: float | None = None
+    tier2_aux_score: float | None = None      # multi-head models only
+    tier2_multihead_blocked: bool | None = None
     tier2_skip_reason: str | None = None
     max_sentence: str | None = None
+    tier3: Tier3Result | None = None          # present when Tier 3 ran
     fields_dropped: list[str] = field(default_factory=list)
     truncated_at_depth: bool | None = None
     latency_ms: float = 0.0
+    # Cost telemetry — present only when the batched Tier 2 classifier ran
+    phase_timings: PhaseTimings | None = None       # prepare / infer / aggregate ms
+    tier2_stats: Tier2Stats | None = None           # string/chunk/unique counts, real/padded tokens
+    tier1_ms: float | None = None
+    cold_load: bool | None = None
+    # Operational signals
+    tier2_available: bool | None = None       # False when Tier 2 enabled but failed to load
+    coverage_degraded: bool | None = None      # True when Tier 1 detection coverage was capped
 ```
 
 ### `defense.defend_tool_results(items)`
@@ -229,7 +246,7 @@ results = defense.defend_tool_results([
 ])
 for r in results:
     if not r.allowed:
-        print("Blocked:", ", ".join(r.fields_sanitized))
+        print("Blocked:", ", ".join(r.detections))
 ```
 
 ### `await defense.defend_tool_results_async(items)`
@@ -274,7 +291,7 @@ sanitized = run_tool_and_defend(gmail_api.get_message(msg_id), "gmail_get_messag
 
 ## Risky field detection
 
-Only **string** values under configured “risky” keys are scanned and sanitized. [`RiskyFieldConfig`](https://github.com/StackOneHQ/stackone-defender/blob/main/src/stackone_defender/types.py) provides global names/patterns plus **`tool_overrides`** (wildcard tool names → field list), same idea as the npm package.
+Only **string** values under configured “risky” keys are Tier-1-scanned — including strings nested inside arrays/objects under those keys (e.g. `{"name": ["…"]}`). [`RiskyFieldConfig`](https://github.com/StackOneHQ/stackone-defender/blob/main/src/stackone_defender/types.py) provides global names/patterns plus **`tool_overrides`** (wildcard tool names → field list), same idea as the npm package. (Tier 2 scans all extracted strings regardless.)
 
 | Tool pattern | Scanned fields |
 |--------------|----------------|

{stackone_defender-0.7.3 → stackone_defender-0.8.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "stackone-defender"
-version = "0.7.3"
+version = "0.8.2"
 description = "Indirect prompt injection defense for AI agents using tool calls"
 readme = "README.md"
 requires-python = ">=3.11"

{stackone_defender-0.7.3 → stackone_defender-0.8.2}/src/stackone_defender/classifiers/onnx_classifier.py

@@ -11,11 +11,20 @@ from __future__ import annotations
 import logging
 import math
 import threading
+from dataclasses import dataclass
 from pathlib import Path
-from typing import Literal
+from typing import Literal, cast
 
 _logger = logging.getLogger(__name__)
 
+
+@dataclass
+class BatchTokenStats:
+    """Padding accounting accumulated across a batched classification (#6 telemetry)."""
+
+    real_tokens: int = 0  # sum of real (non-pad) tokens across all chunks
+    padded_tokens: int = 0  # tokens actually run through ONNX, including padding
+
 # Shared across all OnnxClassifier instances (keyed by resolved model dir path).
 # Tuple is (session, tokenizer, count_tokenizer); the count tokenizer is
 # non-truncating so count_tokens reports true length (see _load_model).
@@ -61,8 +70,28 @@ class OnnxClassifier:
     """
 
     _MAX_BATCH_CHUNK = 32
-
-    def __init__(self, model_path: str | None = None, temperature_t: float | None = None):
+    # Fixed pad widths (ascending). A string pads to the smallest bucket >= its
+    # token length, so its padded length — and thus its quantized score — is a
+    # function of the string alone, not its batch neighbours (deterministic).
+    _PAD_BUCKETS = (32, 64, 128, 256)
+
+    # Token-degeneracy (OOD) guard. Below this many content tokens, token-share
+    # is too coarse to mean anything (a 1-2 token row is trivially "dominated");
+    # matches the shortest decorative run that still false-fires post-collapse
+    # (``─``x3 -> 3 content tokens). Damp only when the row draws on at most
+    # _DEGENERACY_MAX_DISTINCT_TOKENS distinct tokens — padding an attack with
+    # copies of one token can push the share past 2/3 but cannot remove the
+    # attack's own vocabulary, so the distinct-token floor is a structural, not
+    # tuned, backstop against that bypass.
+    _DEGENERACY_MIN_CONTENT_TOKENS = 3
+    _DEGENERACY_MAX_DISTINCT_TOKENS = 4
+
+    def __init__(
+        self,
+        model_path: str | None = None,
+        temperature_t: float | None = None,
+        degeneracy_max_token_share: float | None = None,
+    ):
         self._model_path = model_path or get_default_model_path()
         self._session = None
         self._tokenizer = None
@@ -70,6 +99,14 @@ class OnnxClassifier:
         self._count_tokenizer = None
         self._max_length = 256
         self._load_failed = False
+        # Token-degeneracy guard threshold; > 1 disables. See _is_degenerate.
+        self._degeneracy_max_token_share = 2 / 3
+        if degeneracy_max_token_share is not None and math.isfinite(degeneracy_max_token_share):
+            self._degeneracy_max_token_share = float(degeneracy_max_token_share)
+        # Cached [UNK] id, resolved lazily from the loaded tokenizer. ``None`` =
+        # not yet resolved or no [UNK] concept; see _get_unk_token_id.
+        self._unk_token_id: int | None = None
+        self._unk_resolved = False
         # Output mode is detected lazily from the logits shape on the first
         # inference call. ``None`` until then.
         self._output_mode: Literal["single", "multi"] | None = None
@@ -132,8 +169,10 @@ class OnnxClassifier:
                 import onnxruntime as ort
                 from tokenizers import Tokenizer
             except ImportError as e:
+                # No warning here -- the ImportError propagates to the caller,
+                # which owns user-facing messaging (PromptDefense warns once per
+                # instance). Warning here logged a line on every failed call.
                 self._load_failed = True
-                _logger.warning("[defender] ONNX model failed to load: %s", e)
                 raise ImportError(
                     "ONNX dependencies not installed. Install with: pip install stackone-defender[onnx]"
                 ) from e
@@ -142,7 +181,10 @@ class OnnxClassifier:
                 tokenizer_path = str(Path(self._model_path) / "tokenizer.json")
                 self._tokenizer = Tokenizer.from_file(tokenizer_path)
                 self._tokenizer.enable_truncation(max_length=self._max_length)
-                self._tokenizer.enable_padding(length=self._max_length)
+                # No fixed padding: classify_batch_pair pads each batch per bucket
+                # width (see _PAD_BUCKETS), so a string's padded length depends only
+                # on itself. Single classify runs at the string's real length.
+                self._tokenizer.no_padding()
 
                 # tokenizer.json bakes in truncation+padding, so disable both
                 # here to count true length rather than a value capped at
@@ -182,6 +224,10 @@ class OnnxClassifier:
         import numpy as np
 
         encoding = self._tokenizer.encode(text)
+        # Fix 3: token-degeneracy guard — skip inference on off-distribution
+        # input; its mean-pooled score is arbitrary. Damp to a benign 0.
+        if self._is_degenerate(encoding.ids):
+            return 0.0, None
         input_ids = np.array([encoding.ids], dtype=np.int64)
         attention_mask = np.array([encoding.attention_mask], dtype=np.int64)
 
@@ -199,36 +245,130 @@ class OnnxClassifier:
         main_logit = float(row[0]) if hasattr(row, "__len__") and len(row) > 0 else float(row)
         return _sigmoid(main_logit / t), None
 
-    def classify_batch(self, texts: list[str]) -> list[float]:
+    def classify_batch(self, texts: list[str], stats: BatchTokenStats | None = None) -> list[float]:
         """Classify multiple texts; returns main-head scores only.
 
         Back-compat wrapper around :meth:`classify_batch_pair`.
         """
-        return [main for main, _ in self.classify_batch_pair(texts)]
+        return [main for main, _ in self.classify_batch_pair(texts, stats)]
 
-    def classify_batch_pair(self, texts: list[str]) -> list[tuple[float, float | None]]:
+    def classify_batch_pair(
+        self, texts: list[str], stats: BatchTokenStats | None = None
+    ) -> list[tuple[float, float | None]]:
         """Classify multiple texts, returning ``(main, aux)`` per row.
 
-        Aux is ``None`` per-row for single-head models. Chunks the input to
-        bound native memory; the attention matrix is ``O(chunk * seq_len^2)``,
-        and for MiniLM (``max_length=256``) a chunk of 32 keeps memory
-        under ~50MB per call.
+        Aux is ``None`` per-row for single-head models. Groups strings into fixed
+        pad-width buckets by token length, batches within each bucket, and pads
+        every row to the BUCKET width (not the batch max). This bounds padding
+        waste and makes a string's padded length — hence its quantized score —
+        depend only on itself, not its batch neighbours. Chunks of 32 bound
+        native memory (attention is ``O(chunk * seq_len^2)``). Results scatter
+        back to input order. ``stats`` accumulates padding counts when provided.
         """
         if not texts:
             return []
         self._ensure_loaded()
-        all_pairs: list[tuple[float, float | None]] = []
-        for offset in range(0, len(texts), self._MAX_BATCH_CHUNK):
-            chunk = texts[offset : offset + self._MAX_BATCH_CHUNK]
-            all_pairs.extend(self._classify_batch_chunk_pair(chunk))
-        return all_pairs
 
-    def _classify_batch_chunk_pair(self, texts: list[str]) -> list[tuple[float, float | None]]:
+        encodings = self._tokenizer.encode_batch(texts)
+        last_bucket = self._PAD_BUCKETS[-1]
+        buckets: dict[int, list[int]] = {}
+        for i, enc in enumerate(encodings):
+            length = len(enc.ids)
+            width = next((b for b in self._PAD_BUCKETS if length <= b), last_bucket)
+            buckets.setdefault(width, []).append(i)
+
+        pairs: list[tuple[float, float | None] | None] = [None] * len(texts)
+        for width in self._PAD_BUCKETS:
+            bucket_idxs = buckets.get(width)
+            if not bucket_idxs:
+                continue
+            for offset in range(0, len(bucket_idxs), self._MAX_BATCH_CHUNK):
+                idxs = bucket_idxs[offset : offset + self._MAX_BATCH_CHUNK]
+                chunk = [encodings[i] for i in idxs]
+                if stats is not None:
+                    # Every row pads to the bucket width; real_tokens is useful work.
+                    stats.padded_tokens += len(chunk) * width
+                    for enc in chunk:
+                        stats.real_tokens += len(enc.ids)
+                chunk_pairs = self._classify_batch_chunk_pair(chunk, width)
+                for k, orig_idx in enumerate(idxs):
+                    pairs[orig_idx] = chunk_pairs[k]
+
+        # Fix 3: token-degeneracy guard — damp off-distribution rows to a benign
+        # 0 so they drop out of any upstream max. Reuses the already-computed ids.
+        for i, enc in enumerate(encodings):
+            if self._is_degenerate(enc.ids):
+                pairs[i] = (0.0, None)
+
+        return cast(list[tuple[float, float | None]], pairs)
+
+    def _get_unk_token_id(self) -> int | None:
+        """Resolve the tokenizer's ``[UNK]`` id, cached. Returns ``None`` when the
+        tokenizer has no ``[UNK]`` concept (the guard then skips its factor-3 check)."""
+        if self._unk_resolved:
+            return self._unk_token_id
+        self._unk_resolved = True
+        try:
+            self._unk_token_id = self._tokenizer.token_to_id("[UNK]")
+        except Exception:
+            self._unk_token_id = None
+        return self._unk_token_id
+
+    def _is_degenerate(self, ids: list[int]) -> bool:
+        """Token-degeneracy (OOD) test over a tokenized row. Damps only when ALL
+        of these hold over the content tokens (excluding [CLS]/[SEP]):
+
+        1. the single most-frequent token covers >= ``degeneracy_max_token_share``,
+        2. the row draws on <= ``_DEGENERACY_MAX_DISTINCT_TOKENS`` distinct tokens, and
+        3. the dominant token is NOT [UNK].
+
+        Factor 2 blocks a padding attack; factor 3 blocks a homoglyph attack —
+        fullwidth / zero-width / other OOV chars collapse to repeated [UNK], the
+        signature of encoding evasion (more suspicious, not less), so those rows
+        are left to score rather than suppressed. Reuses the ids the model runs on.
+        """
+        threshold = self._degeneracy_max_token_share
+        if not (0 < threshold <= 1):  # disabled
+            return False
+        has_specials = len(ids) >= 2
+        content = ids[1:-1] if has_specials else ids
+        n = len(content)
+        if n < self._DEGENERACY_MIN_CONTENT_TOKENS:
+            return False
+        counts: dict[int, int] = {}
+        max_freq = 0
+        dominant_id = -1
+        for tok in content:
+            c = counts.get(tok, 0) + 1
+            counts[tok] = c
+            if c > max_freq:
+                max_freq = c
+                dominant_id = tok
+        if max_freq / n < threshold or len(counts) > self._DEGENERACY_MAX_DISTINCT_TOKENS:
+            return False
+        unk = self._get_unk_token_id()
+        return unk is None or dominant_id != unk
+
+    def _classify_batch_chunk_pair(
+        self, encodings: list, pad_to: int | None = None
+    ) -> list[tuple[float, float | None]]:
+        """Run one pre-tokenized chunk through ONNX, padding rows to ``pad_to``.
+
+        Rows are padded to at least the chunk's real max, so a caller can never
+        under-pad. Inputs are pre-tokenized so ``classify_batch_pair`` can
+        length-bucket without re-tokenizing.
+        """
         import numpy as np
 
-        encodings = self._tokenizer.encode_batch(texts)
-        input_ids = np.array([e.ids for e in encodings], dtype=np.int64)
-        attention_mask = np.array([e.attention_mask for e in encodings], dtype=np.int64)
+        actual_max = max(len(e.ids) for e in encodings)
+        max_len = max(pad_to, actual_max) if pad_to is not None else actual_max
+        batch_size = len(encodings)
+        input_ids = np.zeros((batch_size, max_len), dtype=np.int64)
+        attention_mask = np.zeros((batch_size, max_len), dtype=np.int64)
+        for i, enc in enumerate(encodings):
+            n = len(enc.ids)
+            input_ids[i, :n] = enc.ids
+            attention_mask[i, :n] = enc.attention_mask
 
         results = self._session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})
         logits = results[0]
@@ -237,12 +377,12 @@ class OnnxClassifier:
         t = self._temperature_t
         pairs: list[tuple[float, float | None]] = []
         if self._output_mode == "multi":
-            for i in range(len(texts)):
+            for i in range(batch_size):
                 main = _sigmoid(float(logits[i][0]) / t)
                 aux = _sigmoid(float(logits[i][1]) / t)
                 pairs.append((main, aux))
         else:
-            for i in range(len(texts)):
+            for i in range(batch_size):
                 row = logits[i]
                 # ``row`` may be a scalar (shape ``[batch]``) or 1-vector.
                 main_logit = float(row[0]) if hasattr(row, "__len__") and len(row) > 0 else float(row)

{stackone_defender-0.7.3 → stackone_defender-0.8.2}/src/stackone_defender/classifiers/patterns.py

@@ -134,7 +134,7 @@ ENCODING_SUSPICIOUS_PATTERNS: list[PatternDefinition] = [
     PatternDefinition("html_entity_abuse", re.compile(r"(?:&#\d{2,4};){4,}|(?:&#x[0-9a-fA-F]{2,4};){4,}", re.I), "encoding_suspicious", "medium", "HTML entity encoding (potential obfuscation)"),
     PatternDefinition("rot13_mention", re.compile(r"rot13|caesar\s+cipher|decode\s+this", re.I), "encoding_suspicious", "medium", "Mention of ROT13 or similar encoding schemes"),
     PatternDefinition("binary_string_encoding", re.compile(r"\b[01]{8}(?:\s+[01]{8}){2,}\b"), "encoding_suspicious", "medium", "Binary-encoded string (potential obfuscation)"),
-    PatternDefinition("morse_code_encoding", re.compile(r"(?:[.-]+\s){4,}[.-]+"), "encoding_suspicious", "low", "Morse code pattern (potential obfuscation)"),
+    PatternDefinition("morse_code_encoding", re.compile(r"(?:[.-]{1,8}\s){4,}[.-]{1,8}"), "encoding_suspicious", "low", "Morse code pattern (potential obfuscation)"),
     PatternDefinition("leetspeak_injection", re.compile(r"1gn0r3|f0rg3t|byp4ss|syst3m|4dm1n|h4ck", re.I), "encoding_suspicious", "medium", "Leetspeak obfuscation of injection keywords"),
 ]
 
@@ -170,10 +170,14 @@ INDIRECT_INJECTION_PATTERNS: list[PatternDefinition] = [
     # ``[config](https://.../system-setup)`` triggered. Real smuggled-
     # instruction attacks include the full "ignore (all|the|previous|prior)"
     # phrasing in the URL/anchor.
+    # Negated, bounded char classes ([^\]] / [^)]) instead of ``.*?`` so the
+    # regex is linear -- the lazy-dot form could backtrack on long unclosed
+    # link/URL spans (ReDoS-prone).
     PatternDefinition(
         "markdown_hidden_instruction",
         re.compile(
-            r"\[.*?\]\(.*?(?:ignore|disregard|forget|override)\W+(?:all|the|previous|prior)\W+.*?\)",
+            r"\[[^\]\n]{0,200}\]\([^)\n]{0,300}(?:ignore|disregard|forget|override)\W{1,8}"
+            r"(?:all|the|previous|prior)\W{1,8}[^)\n]{0,300}\)",
             re.I,
         ),
         "structural",