controlzero 1.13.9__tar.gz → 1.13.10__tar.gz

{controlzero-1.13.9 → controlzero-1.13.10}/CHANGELOG.md

@@ -1,5 +1,71 @@
 # Changelog
 
+## 1.13.10 (2026-07-30): DLP now covers prompts and tool output, not just tool arguments
+
+Until this release, Control Zero scanned the ARGUMENTS of a tool call and
+nothing else. A secret could reach a model or leave the machine by three other
+routes that looked identical to a working setup. All three are closed here.
+
+### Fixed
+
+- **A secret pasted into a prompt was sent to the model unscanned (#1966).**
+  Prompt submission is now a scanned surface on the hosts that expose one
+  (Claude Code `UserPromptSubmit`, Gemini CLI `BeforeAgent`). Previously only
+  tool calls were checked, so pasting a key into the chat box egressed it.
+
+- **Credentials from dashboard-configured DLP rules were written to the audit
+  log in cleartext (#1881 follow-up).** The scanner hashed matches only when
+  the rule category was the literal string `secret`; dashboard rules arrive as
+  `credential`, so those matches were stored verbatim. The audit trail was
+  keeping the secret it had just detected. Now gated on a named set of
+  value-bearing categories, so a category added later cannot reach cleartext.
+
+- **Spooled audit records were replayed without ever being scanned (#1967).**
+  Replay validated JSON shape, tool name and decision -- never payload content
+  -- then transmitted the stored bytes as-is, possibly days later on a
+  different network. Records written before this release are treated as
+  unscanned rather than trusted.
+
+- **The auth cache dropped `key_type`, silently disabling the audit
+  anti-spoof guarantee.** `key_type == "extension"` is server-authoritative
+  for a row's `surface_class`; on every cache HIT the field was empty, so the
+  wire-supplied value was honoured instead. The guarantee held only on the
+  first request after a cache miss.
+
+### Added
+
+- **Sensitive data in tool OUTPUT is now detected and recorded (#1965).**
+  Reading a file that contains a secret is caught. The tool call is NOT
+  blocked -- an agent reading a file is normal work -- so this is deliberately
+  observe-and-record: one deduplicated message per tool call, and an audit row
+  carrying `effect=recorded_only` that is never labelled a block.
+
+- **DLP runs before export to third-party observability (#1967).** Braintrust
+  and Langfuse were receiving full tool arguments and output previews with no
+  scan in between.
+
+### Changed
+
+- API-key creation rejects a `key_type` outside `{live, test}` with
+  `INVALID_CREDENTIAL_KIND` (400) instead of accepting any string and
+  interpolating it into the stored key prefix. Bulk extension-key prefixes are
+  limited to 1-8 characters of `[a-z0-9-]`, which is what the database column
+  actually fits.
+
+### Notes for anyone writing a policy by hand
+
+DLP rules go under `dlp_rules:`, and **every rule needs a `pattern:`**. A rule
+without one is dropped silently. Our own published recipe used `dlp:` with
+`pattern_group:` -- neither is read by anything -- so it configured no DLP at
+all. That recipe is corrected, and CI now fails if a shipped example uses a key
+the loader does not consume.
+
+### Not in this release
+
+Tool output is observed, not blocked. Control Zero's own audit egress (the path
+where we ship your data to our backend) is still unscanned; that is tracked and
+is the last uncovered egress path.
+
 ## 1.13.9 (2026-07-28): Kiro IDE hooks execute on 1.0.x and stay in the correct project
 
 ### Fixed

{controlzero-1.13.9 → controlzero-1.13.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: controlzero
-Version: 1.13.9
+Version: 1.13.10
 Summary: AI agent governance: policies, audit, and observability for tool calls. Works locally with no signup.
 Project-URL: Homepage, https://controlzero.ai
 Project-URL: Documentation, https://docs.controlzero.ai
@@ -130,8 +130,8 @@ Templates available (`controlzero init -t <name>`):
 - `autogen` — AutoGen starter policy
 - `codex-cli` — Codex CLI hook starter
 - `gemini-cli` — Gemini CLI hook starter
-- `kiro` — Kiro (AWS) hook starter (CLI: GA; IDE: limited preview)
-- `antigravity` — Google Antigravity (IDE + `agy` CLI) hook starter (Beta)
+- `kiro` — Kiro (AWS) hook starter (CLI: GA; IDE: PRIVATE PREVIEW)
+- `antigravity` — Google Antigravity (IDE + `agy` CLI) hook starter (BETA)
 
 ## Loading a policy
 

{controlzero-1.13.9 → controlzero-1.13.10}/README.md

@@ -75,8 +75,8 @@ Templates available (`controlzero init -t <name>`):
 - `autogen` — AutoGen starter policy
 - `codex-cli` — Codex CLI hook starter
 - `gemini-cli` — Gemini CLI hook starter
-- `kiro` — Kiro (AWS) hook starter (CLI: GA; IDE: limited preview)
-- `antigravity` — Google Antigravity (IDE + `agy` CLI) hook starter (Beta)
+- `kiro` — Kiro (AWS) hook starter (CLI: GA; IDE: PRIVATE PREVIEW)
+- `antigravity` — Google Antigravity (IDE + `agy` CLI) hook starter (BETA)
 
 ## Loading a policy
 

{controlzero-1.13.9 → controlzero-1.13.10}/controlzero/__init__.py

@@ -41,7 +41,7 @@ from controlzero.hitl.grant_protocol import (
 from controlzero.policy_loader import load_policy
 from controlzero.secrets import SecretRef, SecretStr, secret_ref
 
-__version__ = "1.13.9"
+__version__ = "1.13.10"
 
 __all__ = [
     "Client",

controlzero-1.13.10/controlzero/_internal/dlp_output_observer.py

@@ -0,0 +1,95 @@
+"""Observe-only DLP scanning for data produced after a call starts.
+
+This deliberately reuses :class:`DLPScanner`.  It does not evaluate policy
+effects and cannot return a deny: the data has already been produced.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+from controlzero._internal.dlp_scanner import (
+    DLPScanner,
+    extract_text_from_args,
+)
+from controlzero._internal.enforcer import (
+    DLP_STATUS_CLEAN,
+    DLP_STATUS_FINDINGS,
+    DLP_STATUS_SKIPPED,
+    DLP_STATUS_UNAVAILABLE,
+)
+
+DLP_OUTPUT_EFFECT_RECORDED_ONLY = "recorded_only"
+
+
+@dataclass(frozen=True)
+class DLPOutputObservation:
+    """Terminal, non-enforcing result of scanning one output payload."""
+
+    scanned: bool
+    status: str
+    reason: str
+    details: dict[str, Any] = field(default_factory=dict)
+    findings: list[dict] = field(default_factory=list)
+    effect: str = DLP_OUTPUT_EFFECT_RECORDED_ONLY
+
+
+def observe_dlp_output(
+    scanner: Optional[DLPScanner],
+    output: Any,
+) -> DLPOutputObservation:
+    """Scan one complete output and collapse all findings into one observation.
+
+    The caller owns user messaging and audit emission.  Returning one object
+    per invocation makes the deduplication boundary explicit: ten matches in
+    one payload are still one observation and therefore one notice.
+    """
+
+    if scanner is None:
+        return DLPOutputObservation(
+            scanned=False,
+            status=DLP_STATUS_UNAVAILABLE,
+            reason="scanner_uninitialized",
+        )
+
+    try:
+        text = extract_text_from_args(output)
+    except Exception as exc:  # observation must never affect the tool result
+        return DLPOutputObservation(
+            scanned=False,
+            status=DLP_STATUS_UNAVAILABLE,
+            reason="output_extraction_error",
+            details={"error_type": type(exc).__name__},
+        )
+
+    if not text:
+        return DLPOutputObservation(
+            scanned=False,
+            status=DLP_STATUS_SKIPPED,
+            reason="no_scannable_output",
+        )
+
+    try:
+        matches = scanner.scan(text)
+    except Exception as exc:  # observation must never affect the tool result
+        return DLPOutputObservation(
+            scanned=False,
+            status=DLP_STATUS_UNAVAILABLE,
+            reason="scanner_error",
+            details={"error_type": type(exc).__name__},
+        )
+
+    if not matches:
+        return DLPOutputObservation(
+            scanned=True,
+            status=DLP_STATUS_CLEAN,
+            reason="scan_clean",
+        )
+
+    return DLPOutputObservation(
+        scanned=True,
+        status=DLP_STATUS_FINDINGS,
+        reason="scan_findings",
+        findings=scanner.get_findings_for_audit(matches),
+    )

{controlzero-1.13.9 → controlzero-1.13.10}/controlzero/_internal/dlp_scanner.py

@@ -11,7 +11,7 @@ Two pattern sources:
   2. Custom rules: loaded from the policy file's `dlp_rules:` section
      or pulled from backend for enrolled SDKs.
 
-Secret-category matches store a SHA-256 hash of the matched text, never the
+Matched sensitive values store a SHA-256 hash of the matched text, never the
 plaintext. This is consistent with the browser extension's HMAC approach.
 
 Pattern categories:
@@ -28,6 +28,36 @@ import re
 from dataclasses import dataclass, field
 from typing import Any, Optional
 
+# Categories the backend can emit in SDK bundle rules. This is the union of
+# dashboard-authored custom rule categories (validDLPCategories in
+# dlp_handler.go) and canonical built-in detector categories
+# (internal/dlpcatalog/patterns/*.json). Keep this as a named positive set:
+# every accepted category carries a matched sensitive VALUE and must be hashed.
+# A category introduced by a newer backend is rejected at scan time rather than
+# silently taking a plaintext fallback.
+BACKEND_DLP_VALUE_CATEGORIES = frozenset(
+    {
+        "compliance",
+        "contact",
+        "credential",
+        "crypto",
+        "custom",
+        "document",
+        "financial",
+        "ip",
+        "medical",
+        "national_id",
+        "network",
+        "pii",
+        "secret",
+        "tax",
+    }
+)
+
+# ``healthcare`` is the older SDK-local spelling used by the embedded NPI/DEA
+# built-ins. The backend catalog spells the corresponding category ``medical``.
+SENSITIVE_VALUE_CATEGORIES = BACKEND_DLP_VALUE_CATEGORIES | {"healthcare"}
+
 # ---------------------------------------------------------------------------
 # Built-in patterns grouped by category.
 # All builtins use action="detect". Admins add action="block" via custom rules.
@@ -575,7 +605,7 @@ class DLPRule:
     id: str
     name: str
     pattern: str
-    category: str  # "pii", "financial", "secret", "healthcare"
+    category: str
     action: str  # "detect", "block", "mask"
     region: str = "global"
     compiled: Optional[re.Pattern] = field(default=None, repr=False)
@@ -593,14 +623,14 @@ class DLPMatch:
     rule_name: str
     category: str
     action: str
-    matched_text: str  # plaintext for pii/financial, SHA-256 hash for secret
+    matched_text: str  # SHA-256 hash of the matched sensitive value
     offset: int
     count: int = 1
     # Length of the matched span in the SOURCE text. Carried so the
     # masker can splice the exact span out of a string value without
     # re-running the regex. Defaults to len(matched_text) so DLPMatch
-    # objects built by older callers (and the secret path, where
-    # matched_text is a hash, not the raw value) still report a sensible
+    # objects built by older callers (where matched_text may not be the raw
+    # value) still report a sensible
     # length -- but the masker only ever uses the span length recorded at
     # scan time, never the hash length.
     length: int = 0
@@ -669,7 +699,7 @@ class DLPScanner:
         """Number of loaded rules."""
         return len(self._rules)
 
-    def scan(self, text: str) -> list[DLPMatch]:
+    def _scan_impl(self, text: str) -> list[DLPMatch]:
         """Scan text against all loaded DLP rules.
 
         Args:
@@ -678,7 +708,7 @@ class DLPScanner:
 
         Returns:
             List of DLPMatch objects for each rule that matched.
-            For secret-category rules, matched_text is a SHA-256 hash.
+            ``matched_text`` is a SHA-256 hash for every supported category.
         """
         if not text or not self._rules:
             return []
@@ -694,13 +724,19 @@ class DLPScanner:
             for m in found:
                 raw_text = m.group(0)
 
-                # Secret-category: hash the matched text, never store plaintext
-                if rule.category == "secret":
-                    display_text = hashlib.sha256(
-                        raw_text.encode("utf-8")
-                    ).hexdigest()
-                else:
-                    display_text = raw_text
+                # Every supported DLP category carries a sensitive matched
+                # VALUE. Enumerate the backend vocabulary positively, and
+                # fail loudly on forward drift so a newer category can never
+                # inherit a cleartext fallback (#1881, same failure shape as
+                # #1927).
+                if rule.category not in SENSITIVE_VALUE_CATEGORIES:
+                    raise ValueError(
+                        f"unknown DLP value category {rule.category!r}; "
+                        "refusing to serialize matched text"
+                    )
+                display_text = hashlib.sha256(
+                    raw_text.encode("utf-8")
+                ).hexdigest()
 
                 matches.append(
                     DLPMatch(
@@ -711,7 +747,7 @@ class DLPScanner:
                         matched_text=display_text,
                         offset=m.start(),
                         # Span length in the SOURCE text (NOT len(display_text),
-                        # which for secrets is a 64-char hash). The masker splices
+                        # which is a 64-char hash). The masker splices
                         # exactly [offset, offset+length) out of the string value.
                         length=len(raw_text),
                     )
@@ -719,6 +755,39 @@ class DLPScanner:
 
         return matches
 
+    def scan(self, text: str) -> list[DLPMatch]:
+        """Scan text on the ENFORCEMENT path -- "should this call be allowed".
+
+        A thin wrapper over :meth:`_scan_impl` on purpose. The egress
+        conformance matrix measures host-surface coverage by patching THIS
+        method, so anything that patches or spies on ``scan`` observes only
+        enforcement scanning. :meth:`scan_for_egress` goes straight to
+        ``_scan_impl`` and is therefore invisible to that measurement -- which
+        is correct, because our own outbound hygiene is not host coverage and
+        counting it as such would inflate the published number.
+        """
+        return self._scan_impl(text)
+
+    def scan_for_egress(self, text: str) -> list[DLPMatch]:
+        """Scan text we are about to SEND OURSELVES, not text we are policing.
+
+        Same detection, deliberately a different entry point from :meth:`scan`.
+
+        These are two different jobs. ``scan`` answers "should this tool call be
+        allowed" -- a host-surface enforcement question. This answers "are we
+        about to put a customer's secret on our own wire" -- a hygiene question
+        about Control Zero's own egress (path 5, #1967).
+
+        Keeping them distinguishable matters beyond tidiness: the egress
+        conformance matrix measures host-surface coverage by spying on
+        ``scan``. When redaction shared that entry point, a cell with no host
+        surface at all started reporting as "scanned", which would have
+        promoted a declaration and inflated the published coverage number on
+        the strength of our own outbound hygiene. The measurement has to
+        measure what it says it measures.
+        """
+        return self._scan_impl(text)
+
     def has_blocking_match(self, matches: list[DLPMatch]) -> bool:
         """Check if any match has action='block'."""
         return any(m.action == "block" for m in matches)
@@ -836,12 +905,12 @@ class DLPScanner:
     def get_findings_for_audit(self, matches: list[DLPMatch]) -> list[dict]:
         """Convert matches to audit-safe findings.
 
-        For secret-category matches, only count is included (no text).
+        For detect/block matches, ``matched_text`` is the SHA-256 hash produced
+        by :meth:`scan`, for every supported sensitive-value category.
         For mask-action matches, the raw value is REPLACED by the value-free
-        mask placeholder ("audit shows the masked value, never the secret") --
+        mask placeholder ("audit shows the masked value, never the value") --
         consistent with the gateway, which renders a mask as a value-free
-        ``[REDACTED-...]`` token everywhere. For other (detect) categories the
-        matched text is included.
+        ``[REDACTED-...]`` token everywhere.
         """
         findings: list[dict] = []
         for m in matches:
@@ -854,22 +923,26 @@ class DLPScanner:
             if m.action == "mask":
                 # Mask is enforcement, not observation: the audit row records
                 # WHAT the value was masked TO (the value-free placeholder),
-                # never the raw bytes -- even for pii/financial categories that
-                # would otherwise echo plaintext under "detect".
+                # never the raw bytes. Detect/block findings carry a digest,
+                # while mask findings carry the actual replacement token.
                 entry["masked"] = True
                 entry["matched_text"] = mask_placeholder(m.rule_id)
-            elif m.category != "secret":
-                # Never include matched text for secrets in audit
+            else:
+                # scan() has already replaced every supported category's raw
+                # value with its SHA-256 digest.
                 entry["matched_text"] = m.matched_text
             findings.append(entry)
         return findings
 
 
-def extract_text_from_args(args: Any) -> str:
+def extract_text_from_args(args: Any, *, include_keys: bool = False) -> str:
     """Recursively extract all string values from a nested dict/list structure.
 
     Handles nested dicts, lists, and mixed structures. Non-string leaf
-    values are converted to their string representation.
+    values are converted to their string representation. Mapping keys are
+    omitted by default because tool-argument keys are schema, not user data.
+    Egress callers that forward user-controlled mappings can set
+    ``include_keys=True`` to scan every key that will cross the boundary too.
 
     Args:
         args: The tool arguments (typically a dict from JSON).
@@ -878,20 +951,27 @@ def extract_text_from_args(args: Any) -> str:
         A single string with all extracted values joined by newlines.
     """
     parts: list[str] = []
-    _extract_recursive(args, parts)
+    _extract_recursive(args, parts, include_keys=include_keys)
     return "\n".join(parts)
 
 
-def _extract_recursive(obj: Any, parts: list[str]) -> None:
+def _extract_recursive(
+    obj: Any,
+    parts: list[str],
+    *,
+    include_keys: bool = False,
+) -> None:
     """Walk a nested structure and collect string representations."""
     if isinstance(obj, str):
         parts.append(obj)
     elif isinstance(obj, dict):
-        for v in obj.values():
-            _extract_recursive(v, parts)
+        for key, value in obj.items():
+            if include_keys:
+                _extract_recursive(key, parts, include_keys=True)
+            _extract_recursive(value, parts, include_keys=include_keys)
     elif isinstance(obj, (list, tuple)):
         for item in obj:
-            _extract_recursive(item, parts)
+            _extract_recursive(item, parts, include_keys=include_keys)
     elif obj is not None:
         # Convert numbers, bools, etc. to string for scanning
         parts.append(str(obj))

{controlzero-1.13.9 → controlzero-1.13.10}/controlzero/_internal/enforcer.py

@@ -634,6 +634,16 @@ class PolicyEvaluator:
         """Set or replace the DLP scanner."""
         self._dlp_scanner = scanner
 
+    def observe_output(self, output):
+        """DLP-scan already-produced data without changing its disposition.
+
+        Kept on the evaluator so this path uses the exact scanner instance
+        configured for argument DLP, including built-ins and policy rules.
+        """
+        from controlzero._internal.dlp_output_observer import observe_dlp_output
+
+        return observe_dlp_output(self._dlp_scanner, output)
+
     def set_default_action(self, action: str) -> None:
         """Replace the no-match default action.
 
@@ -1447,16 +1457,11 @@ class PolicyEvaluator:
             residual = self._dlp_scanner.scan(masked_text) if masked_text else []
             if self._dlp_scanner.has_masking_match(residual):
                 # Build the deny's audit findings from ONLY the residual
-                # MASK-action matches. A co-matching DETECT-action rule (e.g. a
-                # detect rule on "SECRET" that also fires on the un-maskable
-                # value) would otherwise have get_findings_for_audit echo its
-                # RAW matched_text into the audit row -- raw PII in the audit
-                # even on a block is a #202 leak (codex re-review of #1427).
+                # MASK-action matches so the row records the exact fail-closed
+                # condition, not unrelated detect findings on the same bytes.
                 # Mask findings render as the value-free [REDACTED-...]
-                # placeholder, so this records THAT a mask rule fired and was
-                # blocked, never the matched bytes. (Block-action matches can't
-                # appear here: a block hit is handled earlier and short-circuits
-                # to deny before the mask branch.)
+                # placeholder. (Block-action matches cannot appear here: a
+                # block hit short-circuits before the mask branch.)
                 residual_mask_only = [m for m in residual if m.action == "mask"]
                 offending = residual_mask_only[0]
                 return PolicyDecision(
@@ -1479,8 +1484,8 @@ class PolicyEvaluator:
                     # Audit the firing MASK rule(s) WITHOUT echoing the raw
                     # value: get_findings_for_audit renders mask findings as the
                     # value-free placeholder, so the deny row records that a
-                    # mask rule fired, never the secret. Detect findings (which
-                    # carry raw text) are deliberately excluded from this path.
+                    # mask rule fired, never the sensitive value. Unrelated
+                    # detect findings are deliberately excluded from this path.
                     dlp_findings=self._dlp_scanner.get_findings_for_audit(
                         residual_mask_only
                     ),