babelqueue 1.12.0__tar.gz → 1.13.0__tar.gz

{babelqueue-1.12.0 → babelqueue-1.13.0}/CHANGELOG.md

@@ -9,6 +9,37 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.13.0] - 2026-06-21
+
+### Added
+- **Runtime GDPR field encryption** (ADR-0030) — the optional `babelqueue.gdpr` module ports the Go
+  reference to Python: the SDK-enforcement half of governed PII. The babelqueue-registry only
+  **declares** and **audits** `x-gdpr-sensitive` fields (and can **mask** them for safe logging);
+  this module **enforces** them on the wire — a producer encrypts each marked leaf before publish, a
+  consumer decrypts it after decode. Strictly **opt-in**: a producer/consumer that never calls
+  `protect`/`unprotect` is unchanged. The **envelope stays frozen** (GR-1): only **values inside
+  `data`** change — a sensitive leaf's value becomes a ciphertext **string** — so `data` stays pure
+  JSON (an SDK without the key still carries the envelope), `meta.schema_version` stays `1`, and
+  `trace_id` is untouched (GR-4). The core stays **stdlib-only** (GR-7): Python's standard library
+  has **no AES-GCM**, so the module ships **no** concrete cipher and pulls **no** crypto dependency
+  — `Cipher` is a caller-provided `typing.Protocol` (`encrypt(bytes) -> str` / `decrypt(str) ->
+  bytes`) bound to a KMS / Vault / HSM / tokenisation service, or to a local AES-256-GCM via the
+  optional `cryptography` library (new `babelqueue[gdpr]` extra, documented in the README — still
+  **not** a core dependency). `protect(data, schema, cipher)` canonically JSON-encodes each marked
+  leaf and replaces it with the cipher's ciphertext string **in place**; `unprotect(...)` is the
+  exact inverse, restoring `data` **byte-for-byte**. The marked paths come from the **same per-URN
+  schema** the validator already loads — `babelqueue.schema.sensitive_paths` walks nested objects
+  (`profile.full_name`), array items (`addresses[].line`), container, and root marks, and the
+  `x-gdpr-sensitive` keyword is **validation-neutral** (annotating a schema is never breaking). An
+  absent marked field is skipped (not an error); a re-run `unprotect` on already-cleartext data is a
+  no-op (non-string leaves are left alone); a wrong key / tampered / non-ciphertext value raises the
+  new `babelqueue.DecryptError` so the consumer fails the message (retry / dead-letter) rather than
+  handle unreadable PII. Validate cleartext **before** `protect` / **after** `unprotect`, since a
+  schema constraining a sensitive field would reject the ciphertext string. Unit-tested without any
+  crypto dependency (a stdlib-only authenticated fake cipher) for round-trip exactness across
+  nested/array/container/root marks, skips, idempotent re-runs, frozen-envelope decode/validate, and
+  wrong-key failure. The envelope is unchanged (`schema_version: 1`); this is purely additive.
+
 ## [1.12.0] - 2026-06-21
 
 ### Added

{babelqueue-1.12.0 → babelqueue-1.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: babelqueue
-Version: 1.12.0
+Version: 1.13.0
 Summary: Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers.
 Project-URL: Homepage, https://babelqueue.com
 Project-URL: Source, https://github.com/BabelQueue/babelqueue-python
@@ -38,6 +38,8 @@ Requires-Dist: pytest>=7; extra == 'dev'
 Requires-Dist: ruff>=0.5; extra == 'dev'
 Provides-Extra: django
 Requires-Dist: django>=4.2; extra == 'django'
+Provides-Extra: gdpr
+Requires-Dist: cryptography>=42; extra == 'gdpr'
 Provides-Extra: kafka
 Requires-Dist: confluent-kafka>=2.3; extra == 'kafka'
 Provides-Extra: otel
@@ -311,6 +313,65 @@ with bare-value back-compat), RabbitMQ (AMQP header table) and SQS (`MessageAttr
 transports; where a transport can't carry it, propagation degrades cleanly to v0.1 `trace_id`
 correlation with no error.
 
+## GDPR field encryption (optional)
+
+The optional `babelqueue.gdpr` module is the runtime, SDK-enforcement half of ADR-0030: it
+encrypts the `data` fields a registry declared `x-gdpr-sensitive`. The babelqueue-registry only
+**declares** and **audits** sensitivity (and can **mask** for safe logging); this module
+**enforces** it on the wire — a producer encrypts each marked leaf before publish, a consumer
+decrypts it after decode. It is strictly **opt-in**: if you never call `protect`/`unprotect`,
+nothing changes.
+
+The envelope stays **frozen** (`schema_version: 1`): only the **values inside `data`** change — a
+sensitive leaf's value becomes a ciphertext **string**. `data` stays pure JSON, so an SDK without
+the key still carries the envelope (it just can't read the protected fields), and `trace_id` is
+never touched. Which fields are sensitive lives in the **schema**, not the message.
+
+Because Python's standard library has **no AES-GCM**, the core ships **no** cipher and pulls **no**
+crypto dependency (GR-7). `Cipher` is a caller-provided `Protocol` you bind to a KMS / Vault / HSM /
+tokenisation service, or to a local AES-256-GCM via the optional `cryptography` library
+(`pip install "babelqueue[gdpr]"` — still **not** a core dependency):
+
+```python
+import base64, os
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+
+class AesGcmCipher:
+    def __init__(self, key: bytes) -> None:       # 16/24/32-byte key — the CALLER's
+        self._aead = AESGCM(key)
+
+    def encrypt(self, plaintext: bytes) -> str:
+        nonce = os.urandom(12)
+        return base64.b64encode(nonce + self._aead.encrypt(nonce, plaintext, None)).decode("ascii")
+
+    def decrypt(self, ciphertext: str) -> bytes:
+        raw = base64.b64decode(ciphertext)
+        return self._aead.decrypt(raw[:12], raw[12:], None)   # raises on wrong key / tamper
+```
+
+Wire it around the codec — validate **cleartext** (before `protect`, after `unprotect`), since a
+schema that constrains a sensitive field would reject the ciphertext string:
+
+```python
+from babelqueue import EnvelopeCodec
+from babelqueue.gdpr import protect, unprotect
+
+# producer: encrypt marked leaves in place, then encode
+protect(data, schema, cipher)
+body = EnvelopeCodec.encode(EnvelopeCodec.make("urn:babel:orders:created", data, queue="orders"))
+
+# consumer: decode, then decrypt marked leaves in place before the handler reads data
+envelope = EnvelopeCodec.decode(body)
+unprotect(envelope["data"], schema, cipher)
+```
+
+`schema` is the same per-URN JSON Schema the validator uses; `babelqueue.schema.sensitive_paths`
+locates every `x-gdpr-sensitive` leaf (nested objects, `addresses[].line` array items, container or
+root marks). An absent marked field is skipped; a re-run `unprotect` on cleartext is a no-op
+(non-string leaves are left alone); a wrong key / tampered ciphertext raises
+`babelqueue.DecryptError` so the consumer fails the message (retry / dead-letter) rather than
+process unreadable PII. The round-trip restores `data` **byte-for-byte**.
+
 ## What's here
 
 The codec/contracts/dead-letter (zero-dep core), the `BabelQueue` runtime

{babelqueue-1.12.0 → babelqueue-1.13.0}/README.md

@@ -259,6 +259,65 @@ with bare-value back-compat), RabbitMQ (AMQP header table) and SQS (`MessageAttr
 transports; where a transport can't carry it, propagation degrades cleanly to v0.1 `trace_id`
 correlation with no error.
 
+## GDPR field encryption (optional)
+
+The optional `babelqueue.gdpr` module is the runtime, SDK-enforcement half of ADR-0030: it
+encrypts the `data` fields a registry declared `x-gdpr-sensitive`. The babelqueue-registry only
+**declares** and **audits** sensitivity (and can **mask** for safe logging); this module
+**enforces** it on the wire — a producer encrypts each marked leaf before publish, a consumer
+decrypts it after decode. It is strictly **opt-in**: if you never call `protect`/`unprotect`,
+nothing changes.
+
+The envelope stays **frozen** (`schema_version: 1`): only the **values inside `data`** change — a
+sensitive leaf's value becomes a ciphertext **string**. `data` stays pure JSON, so an SDK without
+the key still carries the envelope (it just can't read the protected fields), and `trace_id` is
+never touched. Which fields are sensitive lives in the **schema**, not the message.
+
+Because Python's standard library has **no AES-GCM**, the core ships **no** cipher and pulls **no**
+crypto dependency (GR-7). `Cipher` is a caller-provided `Protocol` you bind to a KMS / Vault / HSM /
+tokenisation service, or to a local AES-256-GCM via the optional `cryptography` library
+(`pip install "babelqueue[gdpr]"` — still **not** a core dependency):
+
+```python
+import base64, os
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+
+class AesGcmCipher:
+    def __init__(self, key: bytes) -> None:       # 16/24/32-byte key — the CALLER's
+        self._aead = AESGCM(key)
+
+    def encrypt(self, plaintext: bytes) -> str:
+        nonce = os.urandom(12)
+        return base64.b64encode(nonce + self._aead.encrypt(nonce, plaintext, None)).decode("ascii")
+
+    def decrypt(self, ciphertext: str) -> bytes:
+        raw = base64.b64decode(ciphertext)
+        return self._aead.decrypt(raw[:12], raw[12:], None)   # raises on wrong key / tamper
+```
+
+Wire it around the codec — validate **cleartext** (before `protect`, after `unprotect`), since a
+schema that constrains a sensitive field would reject the ciphertext string:
+
+```python
+from babelqueue import EnvelopeCodec
+from babelqueue.gdpr import protect, unprotect
+
+# producer: encrypt marked leaves in place, then encode
+protect(data, schema, cipher)
+body = EnvelopeCodec.encode(EnvelopeCodec.make("urn:babel:orders:created", data, queue="orders"))
+
+# consumer: decode, then decrypt marked leaves in place before the handler reads data
+envelope = EnvelopeCodec.decode(body)
+unprotect(envelope["data"], schema, cipher)
+```
+
+`schema` is the same per-URN JSON Schema the validator uses; `babelqueue.schema.sensitive_paths`
+locates every `x-gdpr-sensitive` leaf (nested objects, `addresses[].line` array items, container or
+root marks). An absent marked field is skipped; a re-run `unprotect` on cleartext is a no-op
+(non-string leaves are left alone); a wrong key / tampered ciphertext raises
+`babelqueue.DecryptError` so the consumer fails the message (retry / dead-letter) rather than
+process unreadable PII. The round-trip restores `data` **byte-for-byte**.
+
 ## What's here
 
 The codec/contracts/dead-letter (zero-dep core), the `BabelQueue` runtime

{babelqueue-1.12.0 → babelqueue-1.13.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.12.0"
+version = "1.13.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -37,6 +37,9 @@ pulsar = ["pulsar-client>=3.4"]
 kafka = ["confluent-kafka>=2.3"]
 artemis = ["python-qpid-proton>=0.39"]
 otel = ["opentelemetry-api>=1.20"]
+# Only for the documented bring-your-own AES-256-GCM reference cipher (ADR-0030). The core never
+# imports it; protect/unprotect take a caller-provided Cipher Protocol, so this is NOT a core dep.
+gdpr = ["cryptography>=42"]
 celery = ["celery>=5"]
 django = ["django>=4.2"]
 dev = ["pytest>=7", "pytest-cov>=4", "mypy>=1.8", "ruff>=0.5"]

{babelqueue-1.12.0 → babelqueue-1.13.0}/src/babelqueue/__init__.py

@@ -11,10 +11,11 @@ and dead-letter helpers. Framework adapters (Celery, Django, ...) build on this.
 
 from __future__ import annotations
 
-from . import dead_letter, headers, idempotency, outbox, redrive, replay
+from . import dead_letter, gdpr, headers, idempotency, outbox, redrive, replay
 from .app import BabelQueue
 from .codec import SCHEMA_VERSION, SOURCE_LANG, EnvelopeCodec
 from .contracts import HasTraceId, PolyglotMessage
+from .gdpr import Cipher, protect, unprotect
 from .headers import headers_from_context
 from .idempotency import IdempotencyStore, InMemoryStore
 from .outbox import (
@@ -25,12 +26,12 @@ from .outbox import (
     OutboxRelayResult,
     OutboxStore,
 )
-from .exceptions import BabelQueueError, UnknownUrnError
+from .exceptions import BabelQueueError, DecryptError, UnknownUrnError
 from .replay import HEADER_REPLAY_BYPASS, bypass_external_effects, is_replay
 from .routing import UnknownUrnStrategy
 from .transport import HeaderPublisher, InMemoryTransport, ReceivedMessage, Transport
 
-__version__ = "1.12.0"
+__version__ = "1.13.0"
 
 __all__ = [
     "BabelQueue",
@@ -46,6 +47,11 @@ __all__ = [
     "HeaderPublisher",
     "BabelQueueError",
     "UnknownUrnError",
+    "DecryptError",
+    "gdpr",
+    "Cipher",
+    "protect",
+    "unprotect",
     "dead_letter",
     "headers",
     "idempotency",

{babelqueue-1.12.0 → babelqueue-1.13.0}/src/babelqueue/exceptions.py

@@ -23,3 +23,15 @@ class InvalidPayloadError(BabelQueueError):
         super().__init__(f"data for {urn!r} does not match its URN schema: {violation}")
         self.urn = urn
         self.violation = violation
+
+
+class DecryptError(BabelQueueError):
+    """A protected ``x-gdpr-sensitive`` field could not be restored on the consume side (ADR-0030).
+
+    Raised by :func:`babelqueue.gdpr.unprotect` when a marked leaf is a ciphertext string the
+    :class:`~babelqueue.gdpr.Cipher` cannot open — a wrong key, a tampered/garbled ciphertext, or a
+    value whose decrypted bytes are not the JSON the producer encoded. The Python mirror of the Go
+    ``gdpr.ErrDecrypt``: a missing field is skipped (not an error), but an *unreadable* one stops
+    ``unprotect`` so the consumer fails the message (retry / dead-letter) rather than handle
+    unreadable PII.
+    """

babelqueue-1.13.0/src/babelqueue/gdpr.py

@@ -0,0 +1,245 @@
+"""Optional runtime GDPR field encryption (ADR-0030): the SDK-enforcement half.
+
+The Python mirror of the Go ``gdpr`` package — the reference the SDKs share. The
+babelqueue-registry only **declares** and **audits** which ``data`` fields are sensitive (the
+``x-gdpr-sensitive`` schema keyword) and can **mask** them for safe logging; this module
+**enforces** that on the wire — a producer encrypts each marked leaf before publish, a consumer
+decrypts it after decode. It is strictly **opt-in**: a producer/consumer that never calls
+:func:`protect`/:func:`unprotect` behaves exactly as before.
+
+The contract is deliberately tight so every SDK stays byte-compatible:
+
+- **The envelope stays frozen (GR-1).** :func:`protect` mutates only **values inside ``data``**: a
+  sensitive leaf's value becomes a ciphertext **string**. It never adds, renames, removes or
+  retypes an envelope field; ``meta.schema_version`` stays ``1``; ``trace_id`` is untouched (GR-4).
+  ``data`` stays **pure JSON** (GR-3) — a JSON string is still pure JSON, so any SDK can carry the
+  envelope even without the key (it just can't read the protected fields).
+- **Zero heavy dependencies (GR-7).** Python's standard library has **no AES-GCM**, so the core
+  ships **no** concrete cipher and pulls **no** crypto dependency. :class:`Cipher` is a
+  caller-provided :class:`typing.Protocol` — bind it to a KMS, Vault transit, an HSM, a
+  tokenisation service, or a local AES-GCM built on the optional ``cryptography`` library (see the
+  bring-your-own-cipher example below). The crypto dependency lives in the *caller's* code, never
+  in this core.
+
+The sensitive paths come from the **same per-URN schema** the produce/consume validation path
+already loads (:func:`babelqueue.schema.sensitive_paths`) — the ``x-gdpr-sensitive`` marks ride on
+it. Validate **cleartext**: run :func:`babelqueue.schema.validate` **before** :func:`protect` on
+the producer and **after** :func:`unprotect` on the consumer, because a schema that constrains a
+sensitive field (``minLength``, ``enum``, …) would otherwise reject the ciphertext string.
+
+Typical wiring (producer)::
+
+    from babelqueue import EnvelopeCodec
+    from babelqueue.gdpr import protect
+
+    data = {"order_id": 1042, "email": "alice@example.com"}
+    # validate(provider, urn, data)          # optional: validate cleartext first
+    protect(data, schema, cipher)            # encrypt marked leaves IN PLACE
+    body = EnvelopeCodec.encode(EnvelopeCodec.make(urn, data))   # ciphertext rides inside data
+
+and the inverse on the consumer, after decode and before the handler reads ``data``::
+
+    from babelqueue.gdpr import unprotect
+
+    envelope = EnvelopeCodec.decode(body)
+    unprotect(envelope["data"], schema, cipher)   # decrypt marked leaves IN PLACE
+    # validate(provider, urn, envelope["data"])   # optional: validate cleartext after
+
+Bring-your-own AES-256-GCM cipher (NOT part of this core — ``pip install cryptography``)::
+
+    import base64, os
+    from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+
+    class AesGcmCipher:
+        '''A reference Cipher: AES-256-GCM, random 12-byte nonce prepended, base64 out.
+        The 32-byte key is the CALLER's; this does no key management or rotation.'''
+
+        def __init__(self, key: bytes) -> None:
+            self._aead = AESGCM(key)          # key must be 16, 24, or 32 bytes
+
+        def encrypt(self, plaintext: bytes) -> str:
+            nonce = os.urandom(12)
+            sealed = self._aead.encrypt(nonce, plaintext, None)
+            return base64.b64encode(nonce + sealed).decode("ascii")
+
+        def decrypt(self, ciphertext: str) -> bytes:
+            raw = base64.b64decode(ciphertext)
+            nonce, sealed = raw[:12], raw[12:]
+            return self._aead.decrypt(nonce, sealed, None)   # raises on wrong key/tamper
+
+The same plaintext MAY encrypt to a different string each call (the random nonce is expected and
+good); :meth:`Cipher.decrypt` restores the exact bytes :meth:`Cipher.encrypt` was given, and a
+wrong-key / tampered input MUST raise (GCM authentication) — :func:`unprotect` turns any such raise
+into :class:`~babelqueue.exceptions.DecryptError`.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any, Callable, List, MutableMapping, MutableSequence, Optional, Protocol, Tuple, runtime_checkable
+
+from .exceptions import DecryptError
+from .schema import sensitive_paths
+
+__all__ = ["Cipher", "protect", "unprotect"]
+
+
+@runtime_checkable
+class Cipher(Protocol):
+    """The field-level protection primitive the **caller** provides — a seam onto a KMS, Vault
+    transit, an HSM, a tokenisation service, or a local AES-GCM (see the module docstring). Keeping
+    this a Protocol is what holds GR-7: this core never imports a crypto library; only the caller's
+    concrete cipher does.
+
+    Contract for an implementation:
+
+    - :meth:`encrypt` takes the canonical JSON bytes of one field value (see :func:`protect`) and
+      returns the ciphertext as a **str** that is valid for placement inside a JSON document (e.g.
+      base64). The same plaintext MAY encrypt to a different string each call (a random nonce/IV is
+      expected and good).
+    - :meth:`decrypt` is the exact inverse: given a string :meth:`encrypt` produced it returns the
+      original JSON bytes byte-for-byte. A string it did not produce, or one produced under a
+      different key, MUST **raise** rather than return silent garbage, so a wrong-key consume fails
+      loudly (:func:`unprotect` wraps the raise in :class:`~babelqueue.exceptions.DecryptError`).
+    - Both SHOULD be safe for concurrent use; a producer/consumer may fan one cipher across threads.
+    """
+
+    def encrypt(self, plaintext: bytes) -> str:
+        """Protect one field value (its canonical JSON bytes) → a JSON-safe ciphertext string."""
+        ...
+
+    def decrypt(self, ciphertext: str) -> bytes:
+        """Reverse :meth:`encrypt`, returning the original field-value JSON bytes."""
+        ...
+
+
+def protect(data: MutableMapping[str, Any], schema: Any, cipher: Cipher) -> None:
+    """Encrypt, in place, every value in ``data`` at a path the schema marked ``x-gdpr-sensitive``.
+
+    The producer-side step — run it after building ``data`` and before encode/publish. Each marked
+    leaf's value is canonically JSON-encoded and replaced by ``cipher.encrypt``'s ciphertext
+    **string**; the envelope frame, non-sensitive fields, and everything else are untouched (GR-1).
+
+    A marked path absent from ``data`` is skipped (not an error) — schemas evolve and a message need
+    not carry every optional field. ``data`` may be empty (no-op). A ``None`` schema or one with no
+    marks is a no-op (nothing is sensitive). A container mark (a whole object/array marked
+    sensitive) is supported: the entire sub-value is encoded and encrypted as one ciphertext string.
+
+    On any cipher error this propagates and leaves ``data`` partially protected; treat a raised
+    error as fatal for that message (do not publish it).
+    """
+    _walk(data, schema, cipher, _encrypt_leaf)
+
+
+def unprotect(data: MutableMapping[str, Any], schema: Any, cipher: Cipher) -> None:
+    """Decrypt, in place, every value in ``data`` at an ``x-gdpr-sensitive`` path — the consumer-side
+    inverse of :func:`protect`. Run it after decode and before the handler reads ``data``.
+
+    An absent path is skipped. A leaf that is **not a string** — it was never protected, or this is
+    a re-run after a successful :func:`unprotect` — is left as-is, so re-invoking on already-cleartext
+    data is safe (idempotent for non-string leaves). A string the cipher cannot open (wrong key,
+    tampered, or not a ciphertext) raises :class:`~babelqueue.exceptions.DecryptError` — the consumer
+    should fail the message (retry / dead-letter) rather than process unreadable PII.
+    """
+    _walk(data, schema, cipher, _decrypt_leaf)
+
+
+# A leaf op transforms one sensitive leaf value (encrypt or decrypt). It returns ``(new_value, ok)``:
+# ``ok=False`` means the value should be left in place untouched.
+_LeafOp = Callable[[Any, Cipher], Tuple[Any, bool]]
+
+
+def _walk(data: Any, schema: Any, cipher: Optional[Cipher], op: _LeafOp) -> None:
+    """Drive ``op`` over every ``x-gdpr-sensitive`` path the schema declares, resolving each path
+    against ``data`` itself (NOT by re-walking the schema over the value), so the operation touches
+    exactly the declared leaves — non-sensitive siblings are never read or copied."""
+    if data is None or schema is None or cipher is None:
+        return
+    for sp in sensitive_paths(schema):
+        _apply_at_path(data, _parse_path(sp.path), cipher, op)
+
+
+# One step of a sensitive path: ``(key, is_array)``. ``"addresses[].line"`` parses to
+# ``[("addresses", True), ("line", False)]``; the ``[]`` marker means array-descent into every
+# element before the next segment.
+_Segment = Tuple[str, bool]
+
+
+def _parse_path(path: str) -> List[_Segment]:
+    """Split a sensitive path (``"email"``, ``"profile.full_name"``, ``"addresses[].line"``) into
+    segments. A trailing ``[]`` on a part binds to it as array-descent. A root mark (path ``""``)
+    yields no segments — there is no addressable leaf for it inside ``data``, so it is skipped."""
+    if path == "":
+        return []
+    segments: List[_Segment] = []
+    for part in path.split("."):
+        if len(part) >= 2 and part.endswith("[]"):
+            segments.append((part[:-2], True))
+        else:
+            segments.append((part, False))
+    return segments
+
+
+def _apply_at_path(node: Any, segments: List[_Segment], cipher: Cipher, op: _LeafOp) -> None:
+    """Resolve ``segments`` against ``node`` and run ``op`` on the leaf(s). It descends objects by
+    key and, when a segment is an array, fans out over every element. An absent key or a type
+    mismatch (a path that does not exist in this particular message) is skipped silently — schemas
+    describe the union of possible shapes; a given message need not contain every field."""
+    if not segments:
+        return  # root mark or exhausted path with no leaf key — nothing addressable in data
+    key, is_array = segments[0]
+    if not isinstance(node, MutableMapping):
+        return  # expected an object here but the message has something else — skip
+    if key not in node:
+        return  # absent field — skip (not an error)
+
+    child = node[key]
+    last = len(segments) == 1
+
+    if is_array:
+        if not isinstance(child, MutableSequence):
+            return  # declared array but message has a non-array — skip
+        for i, elem in enumerate(child):
+            if last:
+                new_value, ok = op(elem, cipher)
+                if ok:
+                    child[i] = new_value
+            else:
+                _apply_at_path(elem, segments[1:], cipher, op)
+        return
+
+    if last:
+        new_value, ok = op(child, cipher)
+        if ok:
+            node[key] = new_value
+        return
+    _apply_at_path(child, segments[1:], cipher, op)
+
+
+def _encrypt_leaf(value: Any, cipher: Cipher) -> Tuple[Any, bool]:
+    """Canonically JSON-encode one field value and replace it with the cipher's ciphertext string.
+    The JSON encoding is what makes the round-trip exact: :func:`unprotect`'s ``json.loads`` restores
+    the same decoded-JSON value (``float`` for numbers, ``dict`` for objects, …) the codec would
+    have produced, so protect → unprotect is byte-for-byte. ``sort_keys`` + compact separators give
+    one canonical encoding, mirroring the Go reference's ``json.Marshal``."""
+    plaintext = json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")
+    return cipher.encrypt(plaintext), True
+
+
+def _decrypt_leaf(value: Any, cipher: Cipher) -> Tuple[Any, bool]:
+    """Reverse :func:`_encrypt_leaf`. A non-string leaf is left untouched (``ok=False``) so
+    :func:`unprotect` is safe to re-run on already-cleartext data; a string that fails to open or to
+    JSON-decode raises :class:`~babelqueue.exceptions.DecryptError` so the consumer fails the message
+    rather than handle unreadable PII."""
+    if not isinstance(value, str):
+        # Not a ciphertext string (already cleartext, or never protected) — leave as-is.
+        return None, False
+    try:
+        plaintext = cipher.decrypt(value)
+    except Exception as exc:  # noqa: BLE001 — any cipher failure is a decrypt failure for the caller
+        raise DecryptError(f"cannot decrypt a protected field: {exc}") from exc
+    try:
+        restored = json.loads(plaintext)
+    except (ValueError, TypeError) as exc:
+        raise DecryptError(f"decrypted plaintext is not JSON: {exc}") from exc
+    return restored, True

{babelqueue-1.12.0 → babelqueue-1.13.0}/src/babelqueue/schema.py

@@ -35,7 +35,7 @@ import functools
 import json
 import os
 import threading
-from typing import Any, Callable, Mapping, Optional, Protocol, runtime_checkable
+from typing import Any, Callable, List, Mapping, NamedTuple, Optional, Protocol, runtime_checkable
 
 from .exceptions import InvalidPayloadError
 
@@ -259,3 +259,61 @@ def _violation(path: str, reason: str) -> str:
 
 def _join(path: str, key: str) -> str:
     return key if path == "" else f"{path}.{key}"
+
+
+class SensitivePath(NamedTuple):
+    """One property a schema marked ``x-gdpr-sensitive`` (ADR-0030), located by its dotted path
+    from the schema root. Array elements use the ``field[]`` segment the validator and registry
+    ``compat`` linter use (e.g. ``addresses[].line``). ``category`` is the optional
+    ``"x-gdpr-sensitive": "<category>"`` string, or ``""`` when the keyword was the boolean
+    ``true``. A mark on the root schema itself is reported with ``path == ""``."""
+
+    path: str
+    category: str
+
+
+def _gdpr_mark(schema: Mapping[str, Any]) -> Optional[str]:
+    """Read the ``x-gdpr-sensitive`` extension keyword (ADR-0030) from one schema node.
+
+    Returns the category string for a marked node (``""`` when the keyword was the boolean
+    ``true``, a non-empty category when it was a non-empty string), or ``None`` when the node is
+    not marked. The keyword is **validation-neutral**: it never makes a value valid or invalid, so
+    annotating a schema is never a breaking change (GR-1). Any other shape — ``false``, ``""``, a
+    number — leaves the node unmarked. Mirrors the Go ``fromMap`` and the registry's parser.
+    """
+    mark = schema.get("x-gdpr-sensitive")
+    if mark is True:
+        return ""
+    if isinstance(mark, str) and mark != "":
+        return mark
+    return None
+
+
+def sensitive_paths(schema: Mapping[str, Any]) -> List[SensitivePath]:
+    """Every property a schema marked ``x-gdpr-sensitive``, in sorted path order (ADR-0030).
+
+    Descends nested objects (dotted paths like ``profile.full_name``) and array item schemas
+    (``addresses[].line``); a mark on the root schema itself is reported with ``path == ""``. It is
+    the value-level counterpart to the registry's inventory: :mod:`babelqueue.gdpr` uses these paths
+    to locate the leaves it encrypts on produce and decrypts on consume. The Python mirror of the
+    Go ``Schema.SensitivePaths``. Non-mapping / malformed nodes are skipped, never raised on.
+    """
+    out: List[SensitivePath] = []
+    _collect_sensitive(schema, "", out)
+    out.sort(key=lambda sp: sp.path)
+    return out
+
+
+def _collect_sensitive(schema: Any, path: str, out: List[SensitivePath]) -> None:
+    if not isinstance(schema, Mapping):
+        return
+    category = _gdpr_mark(schema)
+    if category is not None:
+        out.append(SensitivePath(path, category))
+    properties = schema.get("properties")
+    if isinstance(properties, Mapping):
+        for name, sub in properties.items():
+            _collect_sensitive(sub, _join(path, str(name)), out)
+    items = schema.get("items")
+    if isinstance(items, Mapping):
+        _collect_sensitive(items, path + "[]", out)

babelqueue-1.13.0/tests/test_gdpr.py

@@ -0,0 +1,328 @@
+"""Tests for the optional runtime GDPR field encryption (ADR-0030).
+
+The cipher used here is a STDLIB-ONLY fake — it deliberately imports no ``cryptography`` library,
+proving the core needs no crypto dependency (GR-7). It still honours the full :class:`Cipher`
+contract: a random per-call nonce so the same plaintext encrypts differently each time, a
+byte-for-byte inverse, and a wrong-key ``decrypt`` that raises. A production cipher (AES-256-GCM via
+``cryptography``) is documented in ``babelqueue/gdpr.py``; the contract these tests pin is the same.
+"""
+
+from __future__ import annotations
+
+import base64
+import hashlib
+import json
+import os
+import unittest
+from typing import Any, Dict
+
+from babelqueue import EnvelopeCodec
+from babelqueue.exceptions import DecryptError
+from babelqueue.gdpr import Cipher, protect, unprotect
+from babelqueue.schema import SensitivePath, sensitive_paths
+
+
+class XorCipher:
+    """A reversible, authenticated, STDLIB-ONLY reference :class:`Cipher` for tests.
+
+    NOT cryptographically strong — it is a keystream XOR with a keyed digest tag — but it exercises
+    every property the contract depends on: a random per-call nonce (same plaintext → different
+    ciphertext), a byte-exact inverse, base64 output that drops into JSON, and a wrong-key/tampered
+    ``decrypt`` that raises (the tag check fails). This is the seam :func:`protect`/:func:`unprotect`
+    rely on; the production cipher is the caller's AES-GCM.
+    """
+
+    _NONCE = 16
+
+    def __init__(self, key: bytes) -> None:
+        self._key = key
+
+    def _keystream(self, nonce: bytes, length: int) -> bytes:
+        out = bytearray()
+        counter = 0
+        while len(out) < length:
+            out += hashlib.sha256(self._key + nonce + counter.to_bytes(4, "big")).digest()
+            counter += 1
+        return bytes(out[:length])
+
+    def _tag(self, nonce: bytes, ciphertext: bytes) -> bytes:
+        return hashlib.sha256(self._key + b"tag" + nonce + ciphertext).digest()[:16]
+
+    def encrypt(self, plaintext: bytes) -> str:
+        nonce = os.urandom(self._NONCE)
+        ks = self._keystream(nonce, len(plaintext))
+        ct = bytes(p ^ k for p, k in zip(plaintext, ks))
+        sealed = nonce + ct + self._tag(nonce, ct)
+        return base64.b64encode(sealed).decode("ascii")
+
+    def decrypt(self, ciphertext: str) -> bytes:
+        raw = base64.b64decode(ciphertext)
+        nonce, body, tag = raw[: self._NONCE], raw[self._NONCE : -16], raw[-16:]
+        if self._tag(nonce, body) != tag:
+            raise ValueError("authentication failed (wrong key or tampered ciphertext)")
+        ks = self._keystream(nonce, len(body))
+        return bytes(c ^ k for c, k in zip(body, ks))
+
+
+KEY = b"0123456789abcdef0123456789abcdef"
+
+
+def _cipher() -> Cipher:
+    return XorCipher(KEY)
+
+
+# A schema marking the leaf `email`, the nested `profile.full_name`, the array-item
+# `addresses[].line`, and a container `secret_blob` (a whole object marked sensitive).
+SCHEMA: Dict[str, Any] = {
+    "type": "object",
+    "properties": {
+        "order_id": {"type": "integer"},
+        "email": {"type": "string", "x-gdpr-sensitive": "email"},
+        "profile": {
+            "type": "object",
+            "properties": {
+                "full_name": {"type": "string", "x-gdpr-sensitive": True},
+                "tier": {"type": "string"},
+            },
+        },
+        "addresses": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "properties": {
+                    "line": {"type": "string", "x-gdpr-sensitive": True},
+                    "city": {"type": "string"},
+                },
+            },
+        },
+        "secret_blob": {"type": "object", "x-gdpr-sensitive": True},
+    },
+}
+
+
+def _data() -> Dict[str, Any]:
+    return {
+        "order_id": 1042,
+        "email": "alice@example.com",
+        "profile": {"full_name": "Alice Liddell", "tier": "gold"},
+        "addresses": [
+            {"line": "1 Rabbit Hole", "city": "Wonderland"},
+            {"line": "2 Looking Glass", "city": "Mirror"},
+        ],
+        "secret_blob": {"pan": "4111111111111111", "exp": 1230},
+    }
+
+
+class SensitivePathsTest(unittest.TestCase):
+    def test_walks_nested_object_array_item_and_container(self) -> None:
+        paths = sensitive_paths(SCHEMA)
+        # Sorted by path, with the array-item using the "field[]" segment.
+        self.assertEqual(
+            paths,
+            [
+                SensitivePath("addresses[].line", ""),
+                SensitivePath("email", "email"),
+                SensitivePath("profile.full_name", ""),
+                SensitivePath("secret_blob", ""),
+            ],
+        )
+
+    def test_root_mark_reports_empty_path(self) -> None:
+        self.assertEqual(sensitive_paths({"x-gdpr-sensitive": True}), [SensitivePath("", "")])
+
+    def test_validation_neutral_shapes_are_ignored(self) -> None:
+        # false, "", and a number never mark a property (mirrors the Go parser).
+        schema = {
+            "type": "object",
+            "properties": {
+                "a": {"type": "string", "x-gdpr-sensitive": False},
+                "b": {"type": "string", "x-gdpr-sensitive": ""},
+                "c": {"type": "string", "x-gdpr-sensitive": 1},
+                "d": {"type": "string"},
+            },
+        }
+        self.assertEqual(sensitive_paths(schema), [])
+
+    def test_malformed_schema_is_skipped_not_raised(self) -> None:
+        self.assertEqual(sensitive_paths({"properties": "nope", "items": 5}), [])
+
+
+class ProtectRoundTripTest(unittest.TestCase):
+    def test_round_trip_restores_data_byte_for_byte(self) -> None:
+        original = _data()
+        data = _data()
+        protect(data, SCHEMA, _cipher())
+
+        # Marked leaves became ciphertext STRINGS; everything else is untouched.
+        self.assertIsInstance(data["email"], str)
+        self.assertNotEqual(data["email"], original["email"])
+        self.assertIsInstance(data["profile"]["full_name"], str)
+        self.assertIsInstance(data["addresses"][0]["line"], str)
+        self.assertIsInstance(data["secret_blob"], str)  # the whole container is one ciphertext
+        # Non-sensitive siblings are bit-identical.
+        self.assertEqual(data["order_id"], 1042)
+        self.assertEqual(data["profile"]["tier"], "gold")
+        self.assertEqual(data["addresses"][0]["city"], "Wonderland")
+
+        unprotect(data, SCHEMA, _cipher())
+        self.assertEqual(data, original)
+
+    def test_nested_object_only(self) -> None:
+        schema = {"type": "object", "properties": {"profile": {"type": "object",
+                  "properties": {"full_name": {"type": "string", "x-gdpr-sensitive": True}}}}}
+        data = {"profile": {"full_name": "Bob", "tier": "silver"}}
+        protect(data, schema, _cipher())
+        self.assertNotEqual(data["profile"]["full_name"], "Bob")
+        self.assertEqual(data["profile"]["tier"], "silver")
+        unprotect(data, schema, _cipher())
+        self.assertEqual(data, {"profile": {"full_name": "Bob", "tier": "silver"}})
+
+    def test_array_items(self) -> None:
+        schema = {"type": "object", "properties": {"addresses": {"type": "array", "items":
+                  {"type": "object", "properties": {"line": {"type": "string", "x-gdpr-sensitive": True}}}}}}
+        data = {"addresses": [{"line": "a", "city": "x"}, {"line": "b", "city": "y"}]}
+        protect(data, schema, _cipher())
+        self.assertNotEqual(data["addresses"][0]["line"], "a")
+        self.assertNotEqual(data["addresses"][1]["line"], "b")
+        self.assertEqual(data["addresses"][0]["city"], "x")
+        unprotect(data, schema, _cipher())
+        self.assertEqual(data, {"addresses": [{"line": "a", "city": "x"}, {"line": "b", "city": "y"}]})
+
+    def test_array_leaf_directly_marked(self) -> None:
+        # A path ending in "[]" marks the array ELEMENTS themselves (e.g. a list of phone strings).
+        schema = {"type": "object", "properties":
+                  {"phones": {"type": "array", "items": {"type": "string", "x-gdpr-sensitive": True}}}}
+        self.assertEqual(sensitive_paths(schema), [SensitivePath("phones[]", "")])
+        data = {"phones": ["+1-555-0100", "+1-555-0199"]}
+        protect(data, schema, _cipher())
+        self.assertNotEqual(data["phones"][0], "+1-555-0100")
+        self.assertNotEqual(data["phones"][1], "+1-555-0199")
+        unprotect(data, schema, _cipher())
+        self.assertEqual(data, {"phones": ["+1-555-0100", "+1-555-0199"]})
+
+    def test_non_string_values_round_trip_exactly(self) -> None:
+        # A sensitive leaf need not be a string: number/bool/null/object restore to the same type.
+        schema = {"type": "object", "properties": {k: {"x-gdpr-sensitive": True}
+                  for k in ("n", "b", "z", "o")}}
+        data: Dict[str, Any] = {"n": 1230, "b": True, "z": None, "o": {"k": [1, 2, 3]}}
+        protect(data, schema, _cipher())
+        for k in ("n", "b", "z", "o"):
+            self.assertIsInstance(data[k], str)
+        unprotect(data, schema, _cipher())
+        self.assertEqual(data, {"n": 1230, "b": True, "z": None, "o": {"k": [1, 2, 3]}})
+
+    def test_root_mark_is_skipped_no_addressable_leaf(self) -> None:
+        # A mark on the root schema has no addressable leaf inside data — a no-op, not an error.
+        data = {"a": 1}
+        protect(data, {"x-gdpr-sensitive": True}, _cipher())
+        self.assertEqual(data, {"a": 1})
+
+
+class ProtectSkipsTest(unittest.TestCase):
+    def test_absent_marked_field_is_skipped(self) -> None:
+        data = {"order_id": 7}  # no `email`, `profile`, etc.
+        protect(data, SCHEMA, _cipher())
+        self.assertEqual(data, {"order_id": 7})
+
+    def test_type_mismatch_is_skipped(self) -> None:
+        # Schema says addresses is an array of objects, but the message carries a scalar — skip.
+        data = {"addresses": "not-an-array", "profile": "not-an-object"}
+        protect(data, SCHEMA, _cipher())
+        self.assertEqual(data, {"addresses": "not-an-array", "profile": "not-an-object"})
+
+    def test_no_marks_or_none_schema_is_noop(self) -> None:
+        data = _data()
+        snapshot = json.dumps(data, sort_keys=True)
+        protect(data, {"type": "object", "properties": {"email": {"type": "string"}}}, _cipher())
+        protect(data, None, _cipher())
+        self.assertEqual(json.dumps(data, sort_keys=True), snapshot)
+
+    def test_empty_data_is_noop(self) -> None:
+        data: Dict[str, Any] = {}
+        protect(data, SCHEMA, _cipher())
+        self.assertEqual(data, {})
+
+
+class UnprotectEdgeTest(unittest.TestCase):
+    def test_non_string_leaf_left_untouched_idempotent(self) -> None:
+        # Re-running unprotect is a no-op for NON-STRING leaves: once restored to number/bool/null/
+        # object they are no longer ciphertext candidates, so a second pass leaves them as-is.
+        schema = {"type": "object", "properties": {k: {"x-gdpr-sensitive": True}
+                  for k in ("n", "b", "z", "o")}}
+        data: Dict[str, Any] = {"n": 1230, "b": True, "z": None, "o": {"k": 1}}
+        protect(data, schema, _cipher())
+        unprotect(data, schema, _cipher())  # first restore
+        restored = json.dumps(data, sort_keys=True)
+        unprotect(data, schema, _cipher())  # second pass is a no-op (non-string leaves left alone)
+        unprotect(data, schema, _cipher())  # and again — still stable
+        self.assertEqual(json.dumps(data, sort_keys=True), restored)
+
+    def test_rerun_on_restored_string_leaf_raises(self) -> None:
+        # A restored STRING leaf IS a ciphertext candidate, so a naive double-unprotect on a string
+        # field fails cleanly (it is not the cipher's output) rather than silently corrupting data —
+        # callers run unprotect exactly once, matching the documented contract.
+        data = {"email": "alice@example.com"}
+        protect(data, SCHEMA, _cipher())
+        unprotect(data, SCHEMA, _cipher())
+        self.assertEqual(data, {"email": "alice@example.com"})
+        with self.assertRaises(DecryptError):
+            unprotect(data, SCHEMA, _cipher())
+
+    def test_wrong_key_raises_decrypt_error(self) -> None:
+        data = _data()
+        protect(data, SCHEMA, XorCipher(KEY))
+        with self.assertRaises(DecryptError):
+            unprotect(data, SCHEMA, XorCipher(b"f" * 32))  # different key → tag check fails
+
+    def test_decrypted_non_json_raises_decrypt_error(self) -> None:
+        # A cipher that opens but yields bytes that are not JSON also fails cleanly as DecryptError —
+        # the consumer must never feed corrupt-but-decrypted bytes to the handler.
+        class GarbageCipher:
+            def encrypt(self, plaintext: bytes) -> str:
+                return "garbage"
+
+            def decrypt(self, ciphertext: str) -> bytes:
+                return b"\xff\xfe not json"
+
+        data = {"email": "ct"}
+        with self.assertRaises(DecryptError):
+            unprotect(data, SCHEMA, GarbageCipher())
+
+    def test_non_ciphertext_string_raises_decrypt_error(self) -> None:
+        # A marked leaf that is a plain (never-protected) string is still a string, so unprotect
+        # tries to open it and fails cleanly rather than silently passing garbage through.
+        data = {"email": "alice@example.com"}
+        with self.assertRaises(DecryptError):
+            unprotect(data, SCHEMA, _cipher())
+
+
+class FrozenEnvelopeTest(unittest.TestCase):
+    def test_protected_envelope_still_decodes_and_preserves_frame(self) -> None:
+        data = _data()
+        protect(data, SCHEMA, _cipher())
+
+        envelope = EnvelopeCodec.make("urn:babel:orders:created", data,
+                                      queue="orders", trace_id="trace-xyz")
+        body = EnvelopeCodec.encode(envelope)
+
+        # data stays pure JSON (the ciphertext is just a string) — encode/decode round-trips.
+        decoded = EnvelopeCodec.decode(body)
+        self.assertTrue(EnvelopeCodec.accepts(decoded))
+        self.assertEqual(decoded["meta"]["schema_version"], 1)
+        self.assertEqual(decoded["trace_id"], "trace-xyz")  # GR-4: trace_id untouched
+        self.assertEqual(decoded["job"], "urn:babel:orders:created")
+
+        # The consumer decrypts in place and recovers the original cleartext exactly.
+        unprotect(decoded["data"], SCHEMA, _cipher())
+        self.assertEqual(decoded["data"], _data())
+
+    def test_ciphertext_is_a_json_string(self) -> None:
+        # GR-3: a protected value is a JSON string, so an SDK without the key still carries the frame.
+        data = {"email": "alice@example.com"}
+        protect(data, SCHEMA, _cipher())
+        reparsed = json.loads(json.dumps(data))
+        self.assertIsInstance(reparsed["email"], str)
+
+
+if __name__ == "__main__":
+    unittest.main()