babelqueue 1.12.0__tar.gz → 1.14.0__tar.gz

babelqueue-1.14.0/.github/dependabot.yml

@@ -0,0 +1,11 @@
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

{babelqueue-1.12.0 → babelqueue-1.14.0}/.github/workflows/ci.yml

@@ -118,7 +118,8 @@ jobs:
       - uses: actions/checkout@v5
       - name: Verify vendored conformance matches the canonical suite
         run: |
-          git clone --depth 1 https://github.com/BabelQueue/conformance.git "$RUNNER_TEMP/conformance"
+          git clone --depth 1 --branch v1.0.0 https://github.com/BabelQueue/conformance.git "$RUNNER_TEMP/conformance"
+          diff -u "$RUNNER_TEMP/conformance/CONFORMANCE_VERSION" "tests/conformance/CONFORMANCE_VERSION"
           diff -ru "$RUNNER_TEMP/conformance/manifest.json" "tests/conformance/manifest.json"
           diff -ru "$RUNNER_TEMP/conformance/fixtures"      "tests/conformance/fixtures"
           diff -ru "$RUNNER_TEMP/conformance/schema"        "tests/conformance/schema"

{babelqueue-1.12.0 → babelqueue-1.14.0}/CHANGELOG.md

@@ -9,6 +9,86 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.14.0] - 2026-10-03
+
+### Added
+- **Graceful shutdown** of `BabelQueue.consume()`: on SIGTERM/SIGINT the loop sets a stop flag,
+  takes no new message, lets the in-flight handler finish (or releases the message unchanged once
+  the new `shutdown_timeout` — default 30 s — expires), restores the previous signal handlers and
+  closes the transport. Handlers are installed only on the main thread (`handle_signals=False`
+  opts out); a second signal forces an immediate exit with the message released. A plain
+  `KeyboardInterrupt` keeps its previous behaviour. New `BabelQueue.stop()` ends the loop from any
+  thread after the in-flight message.
+- `Redeliverer` protocol (`redeliver(message, body, delay)`) — a transport's native
+  release; the runtime uses it for retries, unknown-URN `RELEASE` and shutdown releases, and
+  falls back to publish + ack otherwise. New `retry_backoff` / `unknown_urn_release_delay`
+  settings feed its delay.
+- Conformance runners for the `roundtrip`, `data_shape`, `forbidden_keys` and
+  `payload_schema_unicode` sections (none skipped).
+- `.github/dependabot.yml` (pip + GitHub Actions, weekly).
+
+### Changed
+- **SQS release** (broker-bindings §3.5): retries, unknown-URN `RELEASE` and shutdown releases
+  **always** call `ChangeMessageVisibility` instead of re-sending, so the broker's
+  `ApproximateReceiveCount` carries the attempt count. There is no send + delete fallback. The
+  default delay for `retry_backoff` and `unknown_urn_release_delay` is **0 s** (immediate
+  redelivery); a delay outside 0..43200 s (including `inf`/`nan`) is clamped with a warning on the
+  `babelqueue.sqs` logger. A message without a receipt handle is not released (warning); one
+  without a receive count is released but its attempts cannot advance (warning).
+  **Poison-loop risk:** a message that always fails is redelivered immediately until
+  `max_attempts` — configure a native SQS `RedrivePolicy` (`maxReceiveCount` ≥ `max_attempts`) as
+  the backstop and a non-zero `retry_backoff` where needed. A shutdown release also counts as a
+  receive on SQS, so keep `shutdown_timeout` above the longest handler time.
+- Delivery semantics are documented as **at-least-once**: a handler that finishes right as the
+  shutdown deadline fires may be released and run again.
+- The SQS, Kafka, Pulsar, Azure Service Bus and Artemis transports' header/property projections
+  read the envelope without logging a false forbidden-key drop (the body is sent unchanged).
+
+### Fixed
+- The five forbidden envelope keys (`timestamp`, `meta.max_retries`, `meta.attempts`,
+  `meta.source`, `meta.ts` — message-envelope §10) are dropped on decode with a warning on the
+  `babelqueue.codec` logger, and never written on encode. **Transitional:** encode currently drops
+  them with a warning; a future MINOR (K-15 / R1-E0) will reject them instead. A raw-body release
+  republish now strips them too (a clean body is re-sent byte for byte); DLQ redrive still restores
+  the original bytes unchanged.
+- `BabelQueue.stop()` called before `consume()` starts is now honoured instead of being cleared.
+- A failed acknowledgement of an already-processed message (e.g. SQS `DeleteMessage` after a
+  successful handler, an unknown-URN `DELETE`, or a dead-letter publish) is no longer treated as a
+  handler failure: the message is **not** released or retried; the failure is logged at `ERROR`
+  on the new `babelqueue.app` logger and the broker redelivers it on its own (SQS: after the
+  visibility timeout). The consume loop keeps running.
+
+## [1.13.0] - 2026-06-21
+
+### Added
+- **Runtime GDPR field encryption** (ADR-0030) — the optional `babelqueue.gdpr` module ports the Go
+  reference to Python: the SDK-enforcement half of governed PII. The babelqueue-registry only
+  **declares** and **audits** `x-gdpr-sensitive` fields (and can **mask** them for safe logging);
+  this module **enforces** them on the wire — a producer encrypts each marked leaf before publish, a
+  consumer decrypts it after decode. Strictly **opt-in**: a producer/consumer that never calls
+  `protect`/`unprotect` is unchanged. The **envelope stays frozen** (GR-1): only **values inside
+  `data`** change — a sensitive leaf's value becomes a ciphertext **string** — so `data` stays pure
+  JSON (an SDK without the key still carries the envelope), `meta.schema_version` stays `1`, and
+  `trace_id` is untouched (GR-4). The core stays **stdlib-only** (GR-7): Python's standard library
+  has **no AES-GCM**, so the module ships **no** concrete cipher and pulls **no** crypto dependency
+  — `Cipher` is a caller-provided `typing.Protocol` (`encrypt(bytes) -> str` / `decrypt(str) ->
+  bytes`) bound to a KMS / Vault / HSM / tokenisation service, or to a local AES-256-GCM via the
+  optional `cryptography` library (new `babelqueue[gdpr]` extra, documented in the README — still
+  **not** a core dependency). `protect(data, schema, cipher)` canonically JSON-encodes each marked
+  leaf and replaces it with the cipher's ciphertext string **in place**; `unprotect(...)` is the
+  exact inverse, restoring `data` **byte-for-byte**. The marked paths come from the **same per-URN
+  schema** the validator already loads — `babelqueue.schema.sensitive_paths` walks nested objects
+  (`profile.full_name`), array items (`addresses[].line`), container, and root marks, and the
+  `x-gdpr-sensitive` keyword is **validation-neutral** (annotating a schema is never breaking). An
+  absent marked field is skipped (not an error); a re-run `unprotect` on already-cleartext data is a
+  no-op (non-string leaves are left alone); a wrong key / tampered / non-ciphertext value raises the
+  new `babelqueue.DecryptError` so the consumer fails the message (retry / dead-letter) rather than
+  handle unreadable PII. Validate cleartext **before** `protect` / **after** `unprotect`, since a
+  schema constraining a sensitive field would reject the ciphertext string. Unit-tested without any
+  crypto dependency (a stdlib-only authenticated fake cipher) for round-trip exactness across
+  nested/array/container/root marks, skips, idempotent re-runs, frozen-envelope decode/validate, and
+  wrong-key failure. The envelope is unchanged (`schema_version: 1`); this is purely additive.
+
 ## [1.12.0] - 2026-06-21
 
 ### Added

{babelqueue-1.12.0 → babelqueue-1.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: babelqueue
-Version: 1.12.0
+Version: 1.14.0
 Summary: Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers.
 Project-URL: Homepage, https://babelqueue.com
 Project-URL: Source, https://github.com/BabelQueue/babelqueue-python
@@ -35,9 +35,11 @@ Provides-Extra: dev
 Requires-Dist: mypy>=1.8; extra == 'dev'
 Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
-Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: ruff<0.16,>=0.5; extra == 'dev'
 Provides-Extra: django
 Requires-Dist: django>=4.2; extra == 'django'
+Provides-Extra: gdpr
+Requires-Dist: cryptography>=42; extra == 'gdpr'
 Provides-Extra: kafka
 Requires-Dist: confluent-kafka>=2.3; extra == 'kafka'
 Provides-Extra: otel
@@ -172,6 +174,27 @@ app.run()                               # consume forever (Ctrl-C to stop)
   and `memory://` (in-process, great for tests/local). Bring your own by passing
   `transport=...`.
 
+- **Graceful shutdown:** SIGTERM/SIGINT finish the in-flight handler, or release
+  the message unchanged once `shutdown_timeout` expires. Delivery is
+  **at-least-once** — keep handlers idempotent (see the idempotency helper).
+
+### SQS release and poison messages
+
+On a handler failure, unknown-URN `release` or shutdown release, the SQS
+transport **always** releases with `ChangeMessageVisibility` (broker-bindings
+§3.5) — it never re-sends a copy. The broker's `ApproximateReceiveCount` is the
+attempt counter, so a shutdown release also consumes an attempt. The default
+delay is **0 s** (`retry_backoff` / `unknown_urn_release_delay`), i.e. the
+message is visible again immediately; delays outside 0..43200 s are clamped with
+a warning.
+
+**Poison-loop risk:** a message that always fails is redelivered at once until
+`max_attempts` is reached. If the receive count is unavailable the SDK cannot
+advance attempts at all. **Configure a native `RedrivePolicy`**
+(`maxReceiveCount` ≥ `max_attempts`, pointing at `<queue>.dlq`) on every SQS
+queue as the broker-side backstop, and set a non-zero `retry_backoff` if the
+handler's dependencies need time to recover.
+
 ### Sharing a Redis queue with Laravel
 
 By default the Redis transport owns its queue end-to-end (`RPUSH` to produce;
@@ -311,6 +334,65 @@ with bare-value back-compat), RabbitMQ (AMQP header table) and SQS (`MessageAttr
 transports; where a transport can't carry it, propagation degrades cleanly to v0.1 `trace_id`
 correlation with no error.
 
+## GDPR field encryption (optional)
+
+The optional `babelqueue.gdpr` module is the runtime, SDK-enforcement half of ADR-0030: it
+encrypts the `data` fields a registry declared `x-gdpr-sensitive`. The babelqueue-registry only
+**declares** and **audits** sensitivity (and can **mask** for safe logging); this module
+**enforces** it on the wire — a producer encrypts each marked leaf before publish, a consumer
+decrypts it after decode. It is strictly **opt-in**: if you never call `protect`/`unprotect`,
+nothing changes.
+
+The envelope stays **frozen** (`schema_version: 1`): only the **values inside `data`** change — a
+sensitive leaf's value becomes a ciphertext **string**. `data` stays pure JSON, so an SDK without
+the key still carries the envelope (it just can't read the protected fields), and `trace_id` is
+never touched. Which fields are sensitive lives in the **schema**, not the message.
+
+Because Python's standard library has **no AES-GCM**, the core ships **no** cipher and pulls **no**
+crypto dependency (GR-7). `Cipher` is a caller-provided `Protocol` you bind to a KMS / Vault / HSM /
+tokenisation service, or to a local AES-256-GCM via the optional `cryptography` library
+(`pip install "babelqueue[gdpr]"` — still **not** a core dependency):
+
+```python
+import base64, os
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+
+class AesGcmCipher:
+    def __init__(self, key: bytes) -> None:       # 16/24/32-byte key — the CALLER's
+        self._aead = AESGCM(key)
+
+    def encrypt(self, plaintext: bytes) -> str:
+        nonce = os.urandom(12)
+        return base64.b64encode(nonce + self._aead.encrypt(nonce, plaintext, None)).decode("ascii")
+
+    def decrypt(self, ciphertext: str) -> bytes:
+        raw = base64.b64decode(ciphertext)
+        return self._aead.decrypt(raw[:12], raw[12:], None)   # raises on wrong key / tamper
+```
+
+Wire it around the codec — validate **cleartext** (before `protect`, after `unprotect`), since a
+schema that constrains a sensitive field would reject the ciphertext string:
+
+```python
+from babelqueue import EnvelopeCodec
+from babelqueue.gdpr import protect, unprotect
+
+# producer: encrypt marked leaves in place, then encode
+protect(data, schema, cipher)
+body = EnvelopeCodec.encode(EnvelopeCodec.make("urn:babel:orders:created", data, queue="orders"))
+
+# consumer: decode, then decrypt marked leaves in place before the handler reads data
+envelope = EnvelopeCodec.decode(body)
+unprotect(envelope["data"], schema, cipher)
+```
+
+`schema` is the same per-URN JSON Schema the validator uses; `babelqueue.schema.sensitive_paths`
+locates every `x-gdpr-sensitive` leaf (nested objects, `addresses[].line` array items, container or
+root marks). An absent marked field is skipped; a re-run `unprotect` on cleartext is a no-op
+(non-string leaves are left alone); a wrong key / tampered ciphertext raises
+`babelqueue.DecryptError` so the consumer fails the message (retry / dead-letter) rather than
+process unreadable PII. The round-trip restores `data` **byte-for-byte**.
+
 ## What's here
 
 The codec/contracts/dead-letter (zero-dep core), the `BabelQueue` runtime

{babelqueue-1.12.0 → babelqueue-1.14.0}/README.md

@@ -120,6 +120,27 @@ app.run()                               # consume forever (Ctrl-C to stop)
   and `memory://` (in-process, great for tests/local). Bring your own by passing
   `transport=...`.
 
+- **Graceful shutdown:** SIGTERM/SIGINT finish the in-flight handler, or release
+  the message unchanged once `shutdown_timeout` expires. Delivery is
+  **at-least-once** — keep handlers idempotent (see the idempotency helper).
+
+### SQS release and poison messages
+
+On a handler failure, unknown-URN `release` or shutdown release, the SQS
+transport **always** releases with `ChangeMessageVisibility` (broker-bindings
+§3.5) — it never re-sends a copy. The broker's `ApproximateReceiveCount` is the
+attempt counter, so a shutdown release also consumes an attempt. The default
+delay is **0 s** (`retry_backoff` / `unknown_urn_release_delay`), i.e. the
+message is visible again immediately; delays outside 0..43200 s are clamped with
+a warning.
+
+**Poison-loop risk:** a message that always fails is redelivered at once until
+`max_attempts` is reached. If the receive count is unavailable the SDK cannot
+advance attempts at all. **Configure a native `RedrivePolicy`**
+(`maxReceiveCount` ≥ `max_attempts`, pointing at `<queue>.dlq`) on every SQS
+queue as the broker-side backstop, and set a non-zero `retry_backoff` if the
+handler's dependencies need time to recover.
+
 ### Sharing a Redis queue with Laravel
 
 By default the Redis transport owns its queue end-to-end (`RPUSH` to produce;
@@ -259,6 +280,65 @@ with bare-value back-compat), RabbitMQ (AMQP header table) and SQS (`MessageAttr
 transports; where a transport can't carry it, propagation degrades cleanly to v0.1 `trace_id`
 correlation with no error.
 
+## GDPR field encryption (optional)
+
+The optional `babelqueue.gdpr` module is the runtime, SDK-enforcement half of ADR-0030: it
+encrypts the `data` fields a registry declared `x-gdpr-sensitive`. The babelqueue-registry only
+**declares** and **audits** sensitivity (and can **mask** for safe logging); this module
+**enforces** it on the wire — a producer encrypts each marked leaf before publish, a consumer
+decrypts it after decode. It is strictly **opt-in**: if you never call `protect`/`unprotect`,
+nothing changes.
+
+The envelope stays **frozen** (`schema_version: 1`): only the **values inside `data`** change — a
+sensitive leaf's value becomes a ciphertext **string**. `data` stays pure JSON, so an SDK without
+the key still carries the envelope (it just can't read the protected fields), and `trace_id` is
+never touched. Which fields are sensitive lives in the **schema**, not the message.
+
+Because Python's standard library has **no AES-GCM**, the core ships **no** cipher and pulls **no**
+crypto dependency (GR-7). `Cipher` is a caller-provided `Protocol` you bind to a KMS / Vault / HSM /
+tokenisation service, or to a local AES-256-GCM via the optional `cryptography` library
+(`pip install "babelqueue[gdpr]"` — still **not** a core dependency):
+
+```python
+import base64, os
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+
+class AesGcmCipher:
+    def __init__(self, key: bytes) -> None:       # 16/24/32-byte key — the CALLER's
+        self._aead = AESGCM(key)
+
+    def encrypt(self, plaintext: bytes) -> str:
+        nonce = os.urandom(12)
+        return base64.b64encode(nonce + self._aead.encrypt(nonce, plaintext, None)).decode("ascii")
+
+    def decrypt(self, ciphertext: str) -> bytes:
+        raw = base64.b64decode(ciphertext)
+        return self._aead.decrypt(raw[:12], raw[12:], None)   # raises on wrong key / tamper
+```
+
+Wire it around the codec — validate **cleartext** (before `protect`, after `unprotect`), since a
+schema that constrains a sensitive field would reject the ciphertext string:
+
+```python
+from babelqueue import EnvelopeCodec
+from babelqueue.gdpr import protect, unprotect
+
+# producer: encrypt marked leaves in place, then encode
+protect(data, schema, cipher)
+body = EnvelopeCodec.encode(EnvelopeCodec.make("urn:babel:orders:created", data, queue="orders"))
+
+# consumer: decode, then decrypt marked leaves in place before the handler reads data
+envelope = EnvelopeCodec.decode(body)
+unprotect(envelope["data"], schema, cipher)
+```
+
+`schema` is the same per-URN JSON Schema the validator uses; `babelqueue.schema.sensitive_paths`
+locates every `x-gdpr-sensitive` leaf (nested objects, `addresses[].line` array items, container or
+root marks). An absent marked field is skipped; a re-run `unprotect` on cleartext is a no-op
+(non-string leaves are left alone); a wrong key / tampered ciphertext raises
+`babelqueue.DecryptError` so the consumer fails the message (retry / dead-letter) rather than
+process unreadable PII. The round-trip restores `data` **byte-for-byte**.
+
 ## What's here
 
 The codec/contracts/dead-letter (zero-dep core), the `BabelQueue` runtime

{babelqueue-1.12.0 → babelqueue-1.14.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.12.0"
+version = "1.14.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -37,9 +37,12 @@ pulsar = ["pulsar-client>=3.4"]
 kafka = ["confluent-kafka>=2.3"]
 artemis = ["python-qpid-proton>=0.39"]
 otel = ["opentelemetry-api>=1.20"]
+# Only for the documented bring-your-own AES-256-GCM reference cipher (ADR-0030). The core never
+# imports it; protect/unprotect take a caller-provided Cipher Protocol, so this is NOT a core dep.
+gdpr = ["cryptography>=42"]
 celery = ["celery>=5"]
 django = ["django>=4.2"]
-dev = ["pytest>=7", "pytest-cov>=4", "mypy>=1.8", "ruff>=0.5"]
+dev = ["pytest>=7", "pytest-cov>=4", "mypy>=1.8", "ruff>=0.5,<0.16"]
 
 [project.urls]
 Homepage = "https://babelqueue.com"

{babelqueue-1.12.0 → babelqueue-1.14.0}/src/babelqueue/__init__.py

@@ -11,10 +11,11 @@ and dead-letter helpers. Framework adapters (Celery, Django, ...) build on this.
 
 from __future__ import annotations
 
-from . import dead_letter, headers, idempotency, outbox, redrive, replay
+from . import dead_letter, gdpr, headers, idempotency, outbox, redrive, replay
 from .app import BabelQueue
 from .codec import SCHEMA_VERSION, SOURCE_LANG, EnvelopeCodec
 from .contracts import HasTraceId, PolyglotMessage
+from .gdpr import Cipher, protect, unprotect
 from .headers import headers_from_context
 from .idempotency import IdempotencyStore, InMemoryStore
 from .outbox import (
@@ -25,12 +26,12 @@ from .outbox import (
     OutboxRelayResult,
     OutboxStore,
 )
-from .exceptions import BabelQueueError, UnknownUrnError
+from .exceptions import BabelQueueError, DecryptError, UnknownUrnError
 from .replay import HEADER_REPLAY_BYPASS, bypass_external_effects, is_replay
 from .routing import UnknownUrnStrategy
-from .transport import HeaderPublisher, InMemoryTransport, ReceivedMessage, Transport
+from .transport import HeaderPublisher, InMemoryTransport, ReceivedMessage, Redeliverer, Transport
 
-__version__ = "1.12.0"
+__version__ = "1.14.0"
 
 __all__ = [
     "BabelQueue",
@@ -44,8 +45,14 @@ __all__ = [
     "InMemoryTransport",
     "ReceivedMessage",
     "HeaderPublisher",
+    "Redeliverer",
     "BabelQueueError",
     "UnknownUrnError",
+    "DecryptError",
+    "gdpr",
+    "Cipher",
+    "protect",
+    "unprotect",
     "dead_letter",
     "headers",
     "idempotency",