@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/data-platform-architecture.md

@@ -0,0 +1,211 @@
+# Data Platform Architecture (Python)
+
+Use when designing the data-platform substrate of a service or service-fleet: DB engine choice (single-instance OLTP, managed cloud DB, distributed SQL, sharding middleware), HA topology, read scaling and replica routing, sharding and resharding strategy, cross-region replication, backup and restore (with rehearsed recovery), cluster lifecycle (provision / scale / decommission / re-shard), capacity planning, fleet-wide schema-migration coordination, and connection-pool / proxy topology.
+
+This complements `data-modeling-and-migrations.md` (which owns schema, index, transaction, outbox, and per-service migration concerns): this file owns the **substrate** that schema and queries sit on. Load both when designing a new data-bound service or auditing an existing one.
+
+> **Sibling sync.** A parallel `go-microservice-architecture/references/data-platform-architecture.md` mirrors **all non-stack-specific sections** of this file. Only the *Python-specific implementation patterns* section diverges by stack. The mirrored sections stay free of three categories of stack-specific token: DB-engine-specific syntax, runtime/concurrency-mechanic names, and library/framework API names. The concrete token list and grep command live in the *Mirrored-section grep gate* subsection at the end of this file's stack-glue.
+
+> **Sanitization boundary.** Vendor names (PostgreSQL, MySQL, Vitess, TiDB, CockroachDB, Aurora, Cloud Spanner, AlloyDB, Cloud SQL, DynamoDB, RDS Proxy, PgBouncer, ProxySQL, S3, Glacier, gp3, io2, etc.) below are illustrative; concrete topology choices, region names, cluster identifiers, and capacity numbers live only in the maintainer's private alias map. The sanitization audience list is positive (external / client / regulator / SOC / procurement / internal-compliance / sales-engineering / partner draft / forwardable-internal); sanitize before any document leaves the implementation team's approved audience.
+>
+> **Sanitization vs grep gate are separate concerns.** The mirrored-section grep gate at the end of this file's stack-glue forbids *stack-specific implementation syntax* in mirrored content; "zero hits" on the gate is not "safe to forward externally." The illustrative vendor / cloud-service / storage-tier names above are *intentionally* in mirrored content (an architect must reason about engine choice across stacks); they require **manual sanitization review** before this file or excerpts are forwarded to the audiences above. Maintain the vendor-name list above as the canonical set of names that require manual replacement before external publication.
+
+## When this applies / does not apply
+
+Apply when:
+- the service owns a durable relational store (its own DB instance, schema, or shared cluster),
+- the team owns operations of that store (per the business-team-owns-data-infra ownership model),
+- the service or fleet faces a sharding, HA, replication, backup, or capacity decision that goes beyond schema design,
+- the team is choosing between standard OLTP + manual sharding, sharding middleware, or a distributed SQL engine.
+
+Skip when:
+- the service uses a fully-managed external DB whose lifecycle is owned by the cloud provider (RDS Aurora / Cloud Spanner / DynamoDB), and the team is only choosing schema and queries — route to `data-modeling-and-migrations.md`,
+- the service is stateless and only consumes data via the data platform owned by another team — route to that team's contract.
+
+## DB engine choice axis
+
+The choice is not "PostgreSQL or MySQL" — it is the position on this axis:
+
+- **Single-instance OLTP** (PostgreSQL / MySQL on one node, with replicas). Familiar, broad ecosystem, low operational complexity. Sharding becomes a migration project once one instance maxes out.
+- **Single-instance OLTP + sharding middleware** (Vitess in front of MySQL, ShardingSphere, application-layer sharding). Keeps the operational model of single-instance OLTP per shard while spreading load across shards. Adds middleware layer to operate; routing complexity moves into the proxy.
+- **Distributed SQL engine** (TiDB, CockroachDB, YugabyteDB, Spanner-shape). Native horizontal scale, transactions across shards, native HA. Trades latency (Raft / Paxos quorum write path) and operational model (cluster of stateful nodes with consensus protocols) for transparent scale.
+- **Cloud-managed equivalents** (Aurora, Cloud SQL, AlloyDB, DynamoDB, Spanner). The provider owns operations; the team owns schema, queries, and contract. Cost model differs; vendor-specific scaling and pricing characteristics matter.
+
+Pick by:
+- **Expected scale** — single-instance maxes out at the box's IOPS / connection ceiling; sharding middleware scales horizontally but each shard is still single-instance ops; distributed SQL scales nodes transparently but at consensus latency cost.
+- **Transaction shape** — single-shard transactions are cheap on all engines; multi-shard transactions are expensive on middleware (two-phase commit, distributed coordinator), built-in on distributed SQL, impossible without an outbox/saga on per-shard apps.
+- **Operational ownership** — single-instance is straightforward; sharding middleware adds a layer to monitor; distributed SQL needs in-house consensus-protocol experience or vendor support; managed cloud DB outsources ops.
+- **Migration path** — choose so the next-tier migration is reachable from this one (start single-instance with a sharding-key contract so a future shard migration does not require schema rewrite).
+
+Document the chosen engine, the next-tier migration trigger, and the operational footprint per cluster.
+
+## HA topology
+
+State the HA model explicitly per cluster:
+
+- **Failover model** — single-primary + synchronous standby with automatic failover (most common); single-primary + async replicas (no automatic failover, RTO = manual promotion time); multi-primary (rare, conflict-resolution required); consensus-based (Raft / Paxos with N replicas, leader election internal to the engine).
+- **Synchronous vs asynchronous replication** — sync gives RPO = 0 at the cost of write latency (every write waits for the standby to ack); async writes are faster but the standby lags. Mixed: sync to one standby (RPO = 0) + async to others (read scaling).
+- **Failover trigger** — health check threshold, automatic promotion, split-brain protection (fencing of the demoted primary). Define the failover time budget; test it on a real schedule.
+- **Quorum semantics** — for consensus engines, the write quorum (e.g., majority of N replicas). Loss of quorum = no writes. The minimum healthy replica count to remain writable is part of the topology contract.
+- **Cross-AZ vs single-AZ** — single-AZ failover handles instance failure; cross-AZ handles AZ failure; cross-region handles region failure. Each tier costs more in latency.
+
+Document the failover RTO and RPO targets per cluster, and the last date a failover was tested in production-like conditions (tested vs theoretical).
+
+## Read scaling and replica routing
+
+Once read load exceeds a single primary's capacity, read replicas spread the load:
+
+- **Replica lag** — async replicas lag the primary by milliseconds to seconds; sync replicas lag by zero but slow writes. Define the maximum acceptable lag per use case.
+- **Read-your-writes consistency** — a read replica may not yet show a write the same client just made. Either route those reads to the primary (the standard "session pinning" pattern), use a per-tenant or per-session replica with a lag budget, or accept eventual consistency on that path.
+- **Replica routing** — at the application layer (the service chooses primary or replica per query), at the proxy layer (the proxy routes by SQL shape), or at the engine layer (the engine routes read-only transactions to replicas). The proxy / engine path is more transparent but moves correctness into infrastructure.
+- **Staleness budget** — per query class, declare the maximum acceptable replica lag; monitor and alert when exceeded; fall back to primary when budget is breached.
+
+Replica use is not free: failure modes include lagging replicas serving stale rows, replica connection-pool exhaustion, and replicas falling out of sync after primary failover (must re-attach).
+
+## Sharding and resharding
+
+When a single primary cannot handle write volume or storage, shard:
+
+- **Shard key** — the column or hash that determines which shard a row lives on. Once committed it is hard to change. Choose carefully: tenant_id for SaaS; resource_id for partitioned workloads; time bucket for time-series; composite (tenant_id, year) for both.
+- **Shard count and growth** — start with more shards than nodes (each node holds N shards), so adding nodes redistributes existing shards rather than re-keying. Shard count growth requires a re-shard migration.
+- **Sharding model** — hash (uniform distribution, no range queries cross-shard), range (range queries cheap, hot range risk), lookup-table / directory (flexible, indirection layer to maintain), composite (tenant + sub-shard).
+- **Cross-shard transactions** — expensive (two-phase commit or distributed coordinator) or impossible (no XA support). Design the domain so cross-shard transactions are rare; route those to outbox/saga workflows when needed.
+- **Resharding path** — the upgrade from N shards to N+M, or from one engine to another. Define: new-shard provisioning, dual-write window, reconciliation, cutover, decommission. Resharding is a multi-week project; estimate it before starting.
+
+## Cross-region replication
+
+When the service serves users in multiple regions, or compliance requires data residency, replicate across regions:
+
+- **Sync vs async cross-region** — sync gives RPO = 0 but adds inter-region RTT to every write (50–100 ms typical, business-critical or not). Async lets writes complete in the originating region; the secondary region lags.
+- **Multi-region writes** — global tables (each region writes locally, conflict resolution per-row), per-region partitions (each tenant pinned to one region, no cross-region writes for that tenant), single-primary-with-read-replicas-elsewhere (writes only in the home region, reads anywhere).
+- **Data residency** — when "tenant X's data stays in region Y" is a contractual obligation (see `multi-tenant-isolation.md`), the data plane is region-per-tenant or region-pinned by tenant.
+- **Cross-region failure scope** — what happens when a region goes down? Define which clusters fail over to which, which tenants are affected, and the time budget for restore-to-secondary.
+
+## Backup, restore, and tested recovery
+
+Backup strategy is not the backup itself; it is the **tested ability to restore**:
+
+- **Backup types** — full snapshots (consistent point-in-time, large), incremental snapshots (diff from last snapshot), WAL / binlog archive (continuous, supports PITR), logical exports (portable, slower restore).
+- **Recovery objectives** — RPO (max data loss in seconds/minutes; depends on backup frequency and WAL archival), RTO (max time to restore; depends on backup size, restore mechanism, and tested practice).
+- **Cross-region backup** — store backups in a different region than the primary so a region outage does not also lose the backups.
+- **Encryption at rest in backups** — backups carry the same encryption boundary as the primary; key rotation includes backup re-encryption (or accept that old backups remain on old keys).
+- **Tested recovery (the rule that distinguishes real from theatre)** — restore from backup on a schedule. Validate the restored state matches expected. Time the restore and compare to RTO. A backup that has never been restored is a backup of unknown quality. Date the last successful restore in the cluster contract.
+
+## Cluster lifecycle
+
+The cluster has a lifecycle as concrete as any service:
+
+- **Provision** — declarative infra (Terraform / equivalent), parameter group, encryption-at-rest configuration, network placement, audit log destination, monitoring scrape config, identity / role setup. Provisioning is reproducible; one-off manual clusters are technical debt.
+- **Scale up** — increasing instance class (vertical) without downtime requires planned maintenance windows on most engines; budget for it.
+- **Scale out** — adding nodes (replicas, shards) requires re-balancing and may briefly affect write latency. Define the scale-out runbook.
+- **Decommission** — taking a cluster out of service: drain traffic, verify zero writes/reads against it for an observation window, snapshot for retention, then destroy. Premature destroy after "looks idle" is a real outage class.
+- **Cluster identity** — the cluster has a name, owner, lifecycle stage, and a connection contract documented; services that connect to it are listed.
+
+## Capacity planning
+
+A data cluster has multiple capacity dimensions, each can become the bottleneck:
+
+- **Storage** — current usage, growth rate, headroom; alert at 70% / 80% / 90% with an explicit response. Storage growth past auto-extend limits is a hard outage.
+- **IOPS / throughput** — provisioned (e.g., AWS gp3 / io2) or burst-limited; monitor utilization vs limit; right-size before the limit is hit.
+- **Connection ceiling** — the engine's max connections; the proxy's connection pool size; the per-service pool. A connection storm at startup (every instance opens 100 connections) can exceed the ceiling instantly.
+- **Query latency budget** — p50 / p95 / p99 latency; growth in p99 is a leading indicator before throughput saturates.
+- **Replica lag headroom** — lag spikes during heavy writes are normal; sustained lag indicates the replica cannot keep up.
+
+Each dimension has a documented limit, current usage, growth rate, and the action when the threshold trips. Capacity planning is monthly minimum, weekly during growth.
+
+## Fleet-wide schema migration coordination
+
+When the fleet has more than ~20 service DBs and migration tooling is per-DB, fleet-wide coordination becomes its own concern:
+
+- **Migration registry** — a catalog of which service owns which DB, which schema version each is on, and which migrations are pending. Without this, a fleet-wide change (e.g., adding a tenant_id column for compliance) cannot be tracked.
+- **Coordinated change rollout** — when the change spans services (a new column in shared semantics, a deprecation of a cross-service contract), define the order: which service migrates first, which dual-reads, when the old shape is retired.
+- **Migration tool unification** — fleet-wide migrations work best when every service uses the same migration tool with the same conventions; mixed tooling makes fleet operations brittle.
+- **Migration approval gate** — at fleet scale, migrations need pre-merge review (does it break replicas? does it lock tables? does it require downtime?). A "migration approval" workflow + checklist beats heroics.
+
+## Connection pool and proxy topology
+
+The path from app to DB has its own architecture:
+
+- **Per-service pool** — each service instance holds its own connection pool. Simple, but fleet-wide connection count = (services × instances × pool_size). Watch the engine's max-connections ceiling.
+- **Proxy layer** (PgBouncer, ProxySQL, Vitess gateway, RDS Proxy) — a proxy multiplexes many service connections into fewer DB connections. Reduces the connection count seen by the engine. Adds a hop (latency) and a layer to operate.
+- **Transaction-mode vs session-mode pooling** — transaction-mode is denser (more service connections per DB connection) but breaks features that rely on session state (prepared statements, advisory locks, session variables). Session-mode preserves features at the cost of density.
+- **Connection lifecycle** — the pool's idle timeout, max lifetime, and reconnect-on-error policy. A connection storm on app start (every instance opens 50 connections at once) is a common outage trigger.
+- **Proxy HA** — the proxy must be HA-paired or per-AZ; a single proxy is a single point of failure for every service behind it.
+
+## Cost and efficiency
+
+Data layer cost grows with scale; explicit cost ownership prevents drift:
+
+- **Right-sizing** — instance class, storage tier (provisioned IOPS vs gp3 vs gp2), backup retention. Over-provisioned clusters are real money.
+- **Cold storage and archival** — old rows that are rarely read move to cheaper storage (S3 / Glacier / equivalent); the archival path must preserve tenant scope and support re-hydration (see `multi-tenant-isolation.md`).
+- **Read-replica tax** — replicas cost as much as primaries; only run replicas that have a real reader.
+- **Cross-region transfer** — egress and inter-region replication traffic costs add up; budget per cluster.
+
+## Anti-patterns
+
+Block these:
+
+- **Sharding decided after a single-instance outage** — emergency sharding under load is a real outage class. Decide shard key and shard count before the migration is forced.
+- **Backups that have never been restored** — backups of unknown quality; the first restore is during the incident. Schedule restore drills.
+- **Failover that has never been tested in production-like conditions** — RTO is theoretical until proven. Test on a schedule with realistic load.
+- **Cross-region sync writes used to hide application bugs** — using cross-region sync replication to mask consistency bugs in the app layer; the latency tax is permanent. Fix the bug.
+- **Hot shard** — one shard absorbs the majority of writes (popular tenant, hot resource, time-bucket clustering). Audit shard key cardinality before launch.
+- **Long-running transactions on primary** — analytical queries that hold long locks, blocking writes. Route analytical traffic to replicas or a separate warehouse.
+- **Connection storm on app start** — every instance opens its full pool at boot, exceeding the engine ceiling. Stagger pool warmup or use a proxy.
+- **Proxy as single point of failure** — one PgBouncer instance fronting the whole cluster. Pair or per-AZ.
+- **Engine choice driven by hype, not by transaction shape** — picking distributed SQL for a 100-write/sec workload, or single-instance OLTP for a workload that needs distributed transactions. The transaction shape determines the engine, not vice versa.
+- **Schema migration that locks the table on a hot path** — a migration that holds a strong lock for minutes; the service is effectively down. Use online-DDL tooling and review migration locking behavior pre-merge.
+- **Backup retention shorter than the deletion / regulatory clock** — restoring a 30-day backup to recover yesterday's data only works if the backup is within retention. Coordinate backup retention with data-deletion SLAs (see `multi-tenant-isolation.md` per-store deletion modes).
+- **Primary DB as cross-service queue** — using a service's primary OLTP table as the substrate for cross-service async messaging via polling. Adds queue load to the primary's connection pool and IOPS budget; couples the broker semantics to the DB's locking and transaction model; lacks fanout, replay, and lag visibility that a proper broker provides. Route durable cross-service async messaging to `event-driven-architecture.md` (broker + outbox poller); allow only low-volume same-service jobs with an explicit capacity budget against the primary.
+
+## Operations checklist (data platform launch)
+
+Each item is a verifiable action:
+
+- Engine choice declared with the next-tier migration trigger (e.g., "single-instance until 50k QPS sustained; migrate to sharding middleware at that threshold").
+- HA topology documented: failover model, sync/async configuration, quorum semantics, cross-AZ placement, failover RTO/RPO targets, last tested-failover date.
+- Read-replica routing decision documented per query class with staleness budget and fallback-to-primary path.
+- Sharding model declared (shard key, shard count, growth path) before the first shard is provisioned; not retrofitted.
+- Cross-region replication mode declared per cluster (sync / async / multi-region / per-region partitioned); residency commitments enforced at the data plane.
+- Backup strategy declared: type (snapshot / WAL / logical), frequency, retention, cross-region location, encryption-at-rest, RPO target.
+- Restore drill scheduled on a documented cadence; last successful restore dated; restore time vs RTO target measured.
+- Cluster lifecycle steps documented: provision (declarative IaC), scale up/out (runbook), decommission (drain + observe + snapshot + destroy); no one-off manual clusters.
+- Capacity dimensions monitored (storage, IOPS, connections, latency p99, replica lag) with documented alert thresholds and response runbooks.
+- Fleet-wide migration registry exists; per-service migration tool + version recorded; coordinated migrations have an order and approval gate.
+- Connection pool sizes documented per service; proxy topology declared (per-service / proxy layer / mixed) with HA pairing where a proxy is used.
+- Cost reviewed monthly; right-sizing reviewed quarterly; cold-storage / archival path tested.
+
+## Python-specific implementation patterns
+
+Stack-localized recipes; the sibling Go file localizes the same patterns differently.
+
+- **DB driver and pool** — choose sync or async driver explicitly: `psycopg` (v3, supports both sync and async) or `psycopg2` (legacy sync) or `asyncpg` (async only) for PostgreSQL; for new MySQL async services prefer `asyncmy` and allow `aiomysql`; for MySQL sync prefer `mysqlclient` and allow `PyMySQL`. Do not replace an existing working `aiomysql` driver solely to comply with the default. ORM layer: SQLAlchemy 2.x (async with asyncpg / asyncmy / aiomysql; sync with psycopg / mysqlclient / PyMySQL); Django ORM for Django services; SQLModel when the service deliberately chooses SQLAlchemy + Pydantic or already uses SQLModel. Configure pool size explicitly (`pool_size`, `max_overflow`, `pool_recycle`, `pool_pre_ping`); do not rely on defaults.
+- **Sync-vs-async-pool decision** — pick one model per service. An asyncio service that occasionally calls a sync DB driver via `to_thread` works but lose efficiency; a sync service spawning asyncio just for DB is over-engineered. The choice is part of `async-execution-model.md` and should be settled before this file is consulted.
+- **Sharding middleware integration** — Vitess via MySQL wire protocol; the Python client (`asyncmy` / `aiomysql` / `mysqlclient` / `PyMySQL`) connects as if to MySQL. Sharding routing is at the gateway; the app's job is to include the shard key in every query.
+- **Replica routing** — SQLAlchemy 2.x supports binds (`session.execute(stmt, bind=replica_engine)`) or separate engine instances per replica; route per query at the service layer. Session pinning for read-your-writes: keep a primary engine for the same async session after a write. The Python file's `data-modeling-and-migrations.md` `Read Replica And Routing Boundary` section names the binding mechanics in more detail.
+- **Migration tooling** — Alembic for SQLAlchemy; Django migrations for Django ORM. Pick one and stick to it across the fleet for the migration registry to be useful. Alembic + asyncpg works (use the sync driver for Alembic, async for runtime).
+- **Outbox poller integration** — the data-platform-architecture decisions (which engine, what HA, replica lag budget) feed into the outbox poller's behavior described in `event-driven-architecture.md` and `data-modeling-and-migrations.md` (Outbox And Dual-Write Consistency).
+- **PgBouncer with Python** — `asyncpg` and `psycopg` both prepare statements by default, which can break PgBouncer transaction-mode pooling. Disable prepared statements at the connection level (`prepare_threshold=None` for psycopg; `statement_cache_size=0` for asyncpg) for transaction-mode pools. Session-mode pools work with all features. Choose mode per service's feature usage.
+- **Vitess gateway** — looks like MySQL on the wire; the Python MySQL driver (e.g., `asyncmy`) connects normally. Transactions across shards require explicit 2PC or routing to a single shard.
+- **Health checks for HA** — a FastAPI dependency or Django readiness view pings the DB; a failed ping → mark un-ready. Distinguish "primary unreachable" (fail) from "replica lagging" (degrade but still ready for non-critical reads).
+- **Connection storm mitigation** — staggered pool warmup (sleep N × instance_index ms before opening connections at boot); `pool_pre_ping=True` to handle stale connections; exponential backoff on reconnect. For an asyncio service, use a single `AsyncEngine` per service instance and share via dependency injection — do not create per-request engines.
+- **Test substitution** — define a repository protocol; provide a real-DB integration test using `testcontainers-python` for PostgreSQL / MySQL containers. SQLAlchemy's in-memory SQLite is convenient for unit tests but loses many semantics (transactions, locking, FK enforcement); use it cautiously and have integration tests on a real engine.
+
+### Mirrored-section grep gate
+
+The sibling-sync header forbids three categories of stack-specific token in mirrored sections (everything from "When this applies" through "Operations checklist"; everything *before* the `## Python-specific implementation patterns` H2). Run this grep against the mirrored region before every commit; zero hits required.
+
+Forbidden tokens for this Python file's mirrored sections:
+
+- **DB-engine syntax** — `SET LOCAL`, `set_config\(`, `current_setting\(`, `pg_try_advisory`, `pg_stat_activity`, `BYPASSRLS`, `FORCE ROW LEVEL SECURITY`, `search_path`, `GET_LOCK\(`.
+- **Runtime / concurrency mechanic names** — `context\.Context`, `\bgoroutine\b`, `\bgoroutines\b`, `ctx\.Done`, `database/sql`, `\bsqlx\b`, `contextvars`, `\basyncio\b`, `run_in_executor`, `to_thread`, `ThreadPoolExecutor`, `ProcessPoolExecutor`, `copy_context`, `async with`, `after_commit`, `listens_for`, `asyncio\.Queue`, `asyncio\.Event`, `asyncio\.create_task`, `asyncio\.Task`.
+- **Library / framework API names** — `GORM`, `Hertz`, `Kitex`, `golang-migrate`, `\bgoose\b`, `\bAtlas\b`, `pgx`, `lib/pq`, `go-sql-driver/mysql`, `SetMaxOpenConns`, `SetMaxIdleConns`, `SetConnMaxLifetime`, `SetConnMaxIdleTime`, `PingContext`, `(^|[^[:alnum:]_])\*?sql\.DB\b`, `(^|[^[:alnum:]_])sql\.Tx\b`, `BeginTx`, `QueryContext`, `ExecContext`, `(^|[^[:alnum:]_])sql\.Rows\b`, `(^|[^[:alnum:]_])sql\.NullString\b`, `DB\.Stats`, `PrimaryDB\(`, `ReplicaDB\(`, `testcontainers-go`, `\btestcontainers\b`, `SQLAlchemy`, `FastAPI`, `Starlette`, `Pydantic`, `httpx`, `Alembic`, `asyncpg`, `psycopg`, `aiomysql`, `asyncmy`, `mysqlclient`, `PyMySQL`, `Django`, `databases`, `sqlmodel`, `tortoise`, `pool_pre_ping`.
+
+Run:
+
+```
+awk '/^## Python-specific implementation patterns/{exit} 1' data-platform-architecture.md \
+  | grep -nE '(SET LOCAL|set_config\(|current_setting\(|pg_try_advisory|pg_stat_activity|BYPASSRLS|FORCE ROW LEVEL SECURITY|search_path|GET_LOCK\(|context\.Context|\bgoroutine\b|\bgoroutines\b|ctx\.Done|database/sql|\bsqlx\b|contextvars|\basyncio\b|run_in_executor|to_thread|ThreadPoolExecutor|ProcessPoolExecutor|copy_context|async with|after_commit|listens_for|asyncio\.Queue|asyncio\.Event|asyncio\.create_task|asyncio\.Task|GORM|Hertz|Kitex|golang-migrate|\bgoose\b|\bAtlas\b|pgx|lib/pq|go-sql-driver/mysql|SetMaxOpenConns|SetMaxIdleConns|SetConnMaxLifetime|SetConnMaxIdleTime|PingContext|(^|[^[:alnum:]_])\*?sql\.DB\b|(^|[^[:alnum:]_])sql\.Tx\b|BeginTx|QueryContext|ExecContext|(^|[^[:alnum:]_])sql\.Rows\b|(^|[^[:alnum:]_])sql\.NullString\b|DB\.Stats|PrimaryDB\(|ReplicaDB\(|testcontainers-go|\btestcontainers\b|SQLAlchemy|FastAPI|Starlette|Pydantic|httpx|Alembic|asyncpg|psycopg|aiomysql|asyncmy|mysqlclient|PyMySQL|Django|databases|sqlmodel|tortoise|pool_pre_ping)'
+```
+
+Allowed exception: the *Sibling sync* header itself names the three category classes (without tokens) and references this gate; the *Sanitization boundary* header does not contain any of these tokens. Vendor names in mirrored sections that name a DB engine class (PostgreSQL / MySQL / Vitess / TiDB / CockroachDB / PgBouncer / ProxySQL) are allowed because they are engine choices an architect must reason about across stacks; the gate forbids stack-specific *implementation syntax*, not generic engine names.

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/event-driven-architecture.md

@@ -0,0 +1,263 @@
+# Event-Driven Architecture (Python)
+
+Use when designing event-driven systems, async message contracts, producer/consumer ownership, transactional outbox/inbox, sagas, schema evolution, dead-letter and replay strategy, or end-to-end delivery guarantees for a Python service that publishes or consumes messages.
+
+This complements `async-execution-model.md` (concurrency model choice), `background-jobs-and-scheduling.md` (in-tenant scheduled jobs), `reliability-and-error-contract.md` (error contract), and `observability-and-ops.md` (OpenTelemetry / startup wiring / propagation): this file owns the **architecture of durable, cross-service event-driven boundaries** — delivery semantics taxonomy, producer-side patterns, transactional outbox, idempotency design, schema evolution, saga/choreography, fanout, replay, and end-to-end "exactly-once" illusion.
+
+> **Conforms to the parallel-stack references pattern.** This file follows the layout documented in `skill-extraction-workflow/references/parallel-stack-references-pattern.md`: mirrored stack-agnostic core (when-applies through operations checklist), stack-specific implementation patterns section, and the embedded `### Mirrored-section grep gate` at the end of the stack-glue. The sibling `go-microservice-architecture/references/event-driven-architecture.md` mirrors the same structure. Either this file or `multi-tenant-isolation.md` may be used as a template for new parallel-stack extractions; multi-tenant additionally demonstrates the `## Topic-extension backlog` H2 for topic-wider-than-loop cases.
+
+> **Sibling sync.** A parallel `go-microservice-architecture/references/event-driven-architecture.md` mirrors **all non-stack-specific sections** of this file (when-applies/not-applies, delivery semantics, event vs command vs query, idempotency, outbox, ordering, schema evolution, retry/DLQ/replay, backpressure, fanout, saga, end-to-end exactly-once, anti-patterns, operations checklist). Only the *Python-specific implementation patterns* section diverges by stack. Maintainers updating any mirrored section here must update the sibling in the same change to prevent drift.
+
+> **Sanitization boundary.** The named brokers (Kafka, Pulsar, RabbitMQ, NATS JetStream, Redis Streams) and libraries below are concrete examples for **internal** implementation guidance, scoped to the implementation team's approved audience. Before this file (or excerpts) is copied into any document leaving that audience — external / client-facing materials, customer-specific deliverables, regulator or auditor evidence packages, SOC / compliance reports, procurement responses, or partner architecture appendices — replace the named choices with generic categories (`the broker`, `a partitioned log`, `a confirm-mode AMQP queue`) unless the vendor selection is already approved for disclosure to that specific audience.
+
+## When this applies / does not apply
+
+Apply when the service:
+- publishes durable events or commands to a broker (Kafka, Pulsar, RabbitMQ, NATS JetStream, Redis Streams, cloud-managed equivalents),
+- consumes from a durable subscription (consumer group, durable subscriber, queue binding),
+- needs cross-service atomicity between a DB write and a published message,
+- needs a documented replay or backfill story for the event log.
+
+Skip when the service:
+- only does in-process pub-sub or fire-and-forget logging,
+- uses synchronous HTTP/RPC with no durable async boundary (use `api-contract-and-schema.md` instead),
+- uses a job queue purely for in-tenant background work where loss is acceptable (use `background-jobs-and-scheduling.md` or `batch-and-pipeline-architecture.md`).
+
+## Delivery semantics taxonomy
+
+State the semantics of every event-driven boundary explicitly. Default to **at-least-once** unless the broker contract proves otherwise.
+
+- **At-most-once** — broker may drop; consumer never sees duplicates. Never acceptable for source-of-truth state changes, money, audit, or non-idempotent external effects. **Is acceptable** for derived or rebuildable state changes (cache-invalidation hints, search-index refresh signals, sampled traces, approximate counters) when each of (a) the downstream state can be repaired by TTL / periodic rebuild / source-of-truth reconciliation, (b) the freshness SLA is documented, and (c) the repair path is itself instrumented and alerted. Otherwise telemetry-only.
+- **At-least-once** — broker may redeliver; consumer must be idempotent. The default for durable brokers. Every consumer needs an idempotency key and dedup design.
+- **"Exactly-once" illusion** — not a broker property; an *end-to-end* property assembled from (a) transactional or idempotent producer, (b) idempotent consumer with dedup storage, (c) atomic commit-and-publish (outbox + transactional message acknowledgement, or the broker's transactional-producer feature), and **(d) the side effect under the claim must be inside the same atomic domain** (DB + broker via transactional consumer, or a single DB transaction). External side effects — third-party API calls, S3 writes, secondary publishes to a different broker, sent emails — are *outside* the atomic domain and require their own provider-side idempotency keys, an outbox/process-manager step, or honest documentation as at-least-once. Do not claim end-to-end exactly-once for a flow whose externally visible side effect is not in the atomic domain.
+
+Record the chosen semantics in the event contract; downstream consumers reason about retries and dedup against it.
+
+## Event vs command vs query
+
+The semantic shape determines ownership, schema, and retry posture:
+
+- **Event** — a fact about something that happened. Past tense. Owned by the producer. Many consumers may subscribe. Schema evolves with backward-compatible additions; semantic meaning is fixed once published.
+- **Command** — a request to do something. Imperative. Owned by the recipient's contract. Typically one consumer (a command handler). May fail validation and be rejected; the sender is told.
+- **Query** — a request for state. Synchronous HTTP/RPC or query API, not a durable message. If you find yourself sending a query as a message, you probably want a query API plus an event subscription for change notifications.
+
+Mixing these confuses ownership: a command consumer that drops the message because "it's an event, consumers are best-effort" is a bug; an event producer that retries indefinitely because "it's a command, must deliver" creates head-of-line blocking.
+
+## Idempotency design
+
+Every at-least-once consumer needs idempotency. Decide each axis explicitly:
+
+- **Idempotency key source** — event id from the producer, natural resource key (`order_id` + `state_transition`), or a derived hash. The producer-generated event id is preferred because it survives intermediate retries and is stable across consumer redeploys.
+- **Dedup window** — how long the consumer remembers seen keys. Shorter window = cheaper storage, larger duplicate risk if a slow retry arrives after window expiry. The window must exceed the broker's maximum redelivery interval and any expected outage/replay window.
+- **Dedup storage** — tier by impact:
+  - *Lossy / rebuildable effects* (the same effects allowed under at-most-once above): Redis `SET NX EX` with TTL ≥ window is sufficient. Loss of a dedup key just causes one duplicate side effect, repaired by the same path that handles at-most-once loss.
+  - *Source-of-truth state changes, money, audit, regulated, or non-idempotent external effects*: the **authority** is a durable DB row keyed by `event_id` (or natural key). Redis may sit in front, but only as a *negative cache* (a miss does not skip the durable check) or as an *optimization cache for keys known to have been populated only after the durable commit succeeded* (write order: durable insert → commit → cache set; never cache before commit). A positive Redis hit cannot bypass the durable check unless the writer guarantees the cache entry only appears post-commit. If Redis is used at all on this path, configure `maxmemory-policy noeviction`, enable persistence with replication, monitor eviction/replication-lag counters, and alert on loss.
+  - Never use Redis as the only dedup for state changes by default.
+- **Side-effect ordering** — there are two side-effect classes; treat them separately:
+  - *DB-local effects*: write the dedup record in the same DB transaction as the side effect. A consumer that performs the side effect, then writes the dedup record, can lose the dedup on crash and re-process on redelivery.
+  - *Cross-system effects (external API call, broker publish to another topic, S3/object-store write, email send)*: the DB transaction cannot include these. Use the **intent-then-execute** pattern: write a durable intent row + dedup key in one DB tx, mark `pending`; the executor (an outbox poller or process manager) calls the external system using a provider-supplied idempotency key; on success it updates the row to `done`. The dedup record and the terminal status are separate columns. The crash window — executor calls the external system, the call succeeds, the executor dies before updating `done` — is the dangerous case: on restart the row is still `pending` and a naive retry duplicates the effect. Handle it explicitly: add an `unknown` / `reconcile` state and require **the external system to support either (a) idempotency-key replay that returns the same outcome for the same key, or (b) status lookup by the idempotency key**. If the external system supports neither, this pattern is not safe; document the boundary as at-least-once with manual reconciliation, alert on rows stuck in `pending` past a deadline, and run a separate reconciliation job. A redelivery sees `pending` / `unknown` / `done` and either looks up status, runs the call with the idempotent key, or skips. Treat any flow that needs cross-system atomicity as a process-manager workflow (see *Saga / process manager* below), not as inline consumer code.
+
+## Transactional outbox / inbox
+
+The outbox pattern makes "write to DB and publish event" atomic without distributed transactions across DB and broker:
+
+- Producer writes the event to an `outbox` table in the **same DB transaction** as the state change.
+- A separate poller (a long-running worker in the same service, or a separate worker process) reads the outbox in order, publishes to the broker, and marks the row sent. The poller is at-least-once: idempotent re-publishing is fine because consumers dedup. The stack-glue section names the specific runtime primitive used to host the poller.
+- Use a monotonic `outbox_id` for ordering within a partition key; publish in `outbox_id` order **per key** (not globally).
+- Poller failure modes: crash mid-publish, broker unavailable, slow broker. Each must leave the outbox in a recoverable state.
+- The "never post-commit publish" rule applies to **durable cross-process events**. An after-commit hook or post-commit callback (the ORM's transaction-commit event) that publishes to a remote broker silently drops the event on a crash between commit and publish. *Exception*: in-process, same-instance post-commit notifications that are intentionally best-effort and rebuildable (local cache invalidation, in-process projection refresh, an in-process signal to other workers in the same process) need no outbox, because there is no durable consumer claim — but only when no remote consumer exists on that channel.
+
+### SKIP LOCKED and per-key ordering
+
+`SELECT … FOR UPDATE SKIP LOCKED` lets multiple poller replicas run safely **only when outbox rows are independent or ordering is not required**. `SKIP LOCKED` is *defined* to let later rows overtake a locked earlier row; if poller A locks `outbox_id=100` for key `K` and stalls on broker publish, poller B will skip 100, lock 101 for key `K`, and publish 101 first — consumers see `K`'s events out of order. To preserve per-key ordering across HA pollers, choose one strategy, **and implement the prerequisites that go with it**; the strategies are not safe on their own:
+
+- *Single active publisher per key/partition* — route by `hash(partition_key) MOD N == poller_index` (NOT by `outbox_id`; `outbox_id` is monotonic and would split a key's rows across publishers). Each row's serializer is the publisher that owns its partition key. **Prerequisites:** stable ownership epoch (each publisher knows its index and the current N), drain on rebalance (when N changes, old owners must finish in-flight publishes before new owners take the key), and a fenced epoch token in the publish path so a stale owner cannot publish after its index has been reassigned.
+- *Per-key advisory lease* — acquire a short-TTL DB lock or row-lock on the partition key before publishing; release on success or on stall. **Prerequisites:** bounded lease TTL with explicit renewal, fencing token written with the publish so a stuck holder cannot resume after its lease expired, and stale-lock recovery (a stuck holder's lease must expire and another poller must take over without duplicate publish — duplicate publish is acceptable because the consumer dedups, but lock leak that blocks the key forever is not). The DB-specific advisory-lock primitive lives in the stack-glue section below; engines differ in lock lifetime and transaction scope.
+- *Dispatcher that refuses to publish `id=N+1` while `id=N` for the same key is unsent* — query `MIN(outbox_id) WHERE key=K AND sent_at IS NULL` and refuse newer ids until that row's `sent_at` is set. **Prerequisites:** an `abandoned` status that the row can move into after K publish failures (otherwise a poisoned unsent row blocks the key forever), an alert on abandonment, and an explicit policy for cross-key dependencies (workflow steps that span keys K1 and K2 can deadlock if K1's next row waits on K2 and vice versa; the dispatcher must not couple keys' progress).
+
+Without one of these (and its prerequisites), `SKIP LOCKED` and per-key ordering are mutually exclusive — pick which property the outbox actually delivers and document it on the event contract.
+
+Inbox (the dual) is rarer: when a consumer must read a message and produce a side effect in a different system atomically, the consumer writes the inbox record + side effect in one DB tx, then acks the broker. On crash before ack, redelivery hits the inbox row and skips the side effect.
+
+## Delivery ordering
+
+Most brokers guarantee per-partition or per-key ordering, not global ordering. Design accordingly:
+
+- **Partition key** — choose a key that aligns with the consistency boundary (per-resource, per-tenant). All events for the same boundary land on the same partition and are processed in order.
+- **Hot partition risk** — a key with high cardinality at the head of the distribution (one tenant's traffic, one popular resource) creates a hot partition that blocks the consumer group. Audit key cardinality before launch.
+  - *If the ordering boundary can be sharded* (the key is finer than the consistency boundary needs), shard the known hot keys (sub-key suffix, time-windowed re-keying, dedicated topic for the hot tenant).
+  - *If the ordering boundary cannot be sharded* (regulated per-tenant audit log, per-resource state machine that must remain serializable), do not shard — sharding destroys the invariant. Mitigate with tenant isolation (dedicated partition or topic for the hot tenant), admission control (per-tenant rate limit at the producer), batching, or a serializing writer; alert on partition-skew metrics so the operator knows when to allocate dedicated capacity.
+  - *Throughput ceiling escape hatch* — these mitigations move or throttle the bottleneck; they do not make a non-shardable serial invariant scale beyond a single serializer's physical throughput. If the required write rate genuinely exceeds one serial lane's capacity, no broker tuning fixes it: redesign the invariant (does it really need total ordering, or just per-sub-key ordering?), split the domain (multiple smaller consistency boundaries), pre-aggregate at the producer (one summary event per N), use a stronger consistency model (a write-ordered log service), or reject the SLA. Do not let a fictional broker mitigation hide an unscalable invariant.
+- **Cross-partition ordering** — does not exist for free. If a consumer needs to see events from two different keys in a specific order, that ordering must be encoded in the events (causal links, vector clocks, or a serializing component) or relaxed.
+- **Scaling and repartition** — broker-specific; document the broker's mechanics on the event contract. Kafka-like partitioned logs are expensive to repartition (changes hash placement and ordering); pre-plan partition count for expected growth. Pulsar topics scale via partition addition with key-shared subscription semantics; NATS JetStream streams reshape via mirroring; RabbitMQ queues and Redis Streams have no partition concept — scale via sharded queues/streams managed by the application. The "plan for years" rule is Kafka-specific; do not apply it to NATS / RabbitMQ / Redis Streams without mapping to their actual scaling story.
+
+## Schema evolution
+
+Events outlive the producer's current code. Schema discipline is non-negotiable:
+
+- **Compatibility mode** — choose `backward` (new producers, old consumers), `forward` (old producers, new consumers), or `full` (both). For multi-team async fanout, full compatibility is the default; pick the others only with explicit consumer coordination.
+- **Additive by default** — new fields are optional with safe defaults; do not remove a field or repurpose its meaning **on the existing version**. Deprecate by creating a new event type or version and migrating consumers.
+- **Security / compliance exception** — when continuing to emit a field is itself the problem (leaky PII field, secret accidentally embedded, regulator-mandated retraction), additive-only is overruled. The path is: create a new event version that omits the field, inventory active consumers, deploy a coordinated emergency migration plan (consumer flips first, producer flips second, old version retired), and define historical-data handling (purge, redaction in archives, controlled access). Document the security trigger; this path is not the default deprecation flow.
+- **Rolling deprecation (the default)** — create a new event type or version, dual-publish or dual-read across a compatibility window, migrate consumers with monitoring and a rollback path, then retire the old version after explicit approval. Do not stop-the-world.
+- **Schema discovery model** — choose by audience:
+  - *Single-team, single-broker boundary*: in-payload schema or a model-derived schema (from the service's typed model layer) in the envelope header is acceptable.
+  - *Multi-team, single-broker fanout*: a schema registry (Confluent-style or self-hosted) validates compatibility at build/deploy time.
+  - *Multi-broker, multi-tenant, or external consumers* (webhooks, partner integrations, tenant-private contracts): hybrid — versioned envelope (`event_type`, `event_version`) in payload + per-tenant contract catalog/registry where available + signed schema references for external consumers + broker-specific validation gates. Neither a single registry nor in-payload alone is sufficient.
+- **Metadata leakage in cross-trust boundaries** — for external or multi-tenant contracts, the schema discovery layer itself can leak. `event_type` strings, version names, registry paths, enum labels, and catalog visibility can reveal unreleased products, regulated workflows, internal team structure, or tenant-specific capabilities even when payload fields are field-level protected. Before publishing to an external boundary: classify each of (`event_type`, `event_version`, schema-reference path, registry namespace, enum values, error codes, catalog listings) by audience; use opaque public aliases (`event_type=ext.<opaque-name>`, namespace-by-tenant-without-tenant-name) where the internal name is sensitive; never let an internal event type cross the boundary as-is.
+- **Breaking change discipline** — a semantic-level breaking change (a field's meaning changes, an enum value is repurposed) is not caught by schema compatibility checks. Document semantic changes; coordinate consumer rollout before publishing the new shape.
+
+For event payloads modelled with the service's typed-model layer, freeze the model at publish time and version the envelope; the producer's evolving model must not reach the wire without a registered new version. The stack-glue section names the specific model library and freezing pattern.
+
+## Retry, dead-letter, replay
+
+Distinguish three failure dispositions explicitly:
+
+- **Retryable** — transient (dependency 5xx, timeout, lock contention). Bounded retries with exponential backoff + jitter, capped attempt count, retry budget.
+- **Permanent drop / dead-letter** — malformed payload, unknown event type, expired delivery, domain conflict that will not resolve on retry. Send to DLQ with the original payload + failure metadata; alert. Do not silently ack.
+- **Poison** — a message that consistently fails after retries. Quarantine to DLQ; alert with the consumer + payload class. Replay decision is manual.
+
+**Retry budget scope must match the constrained resource:**
+- *Isolated handler* (the failure costs only this consumer): per-consumer retry budget.
+- *Shared downstream* (multiple consumers call the same payment provider, LLM vendor, third-party API, regulated quota): global or per-provider or per-tenant budget. Three consumers each independently burning their per-consumer budget against the same downstream produces a retry storm at the provider; the budget belongs to the constrained resource, not to the consumer.
+
+**Replay tooling is part of the architecture, not a runbook detail.** Every event-driven boundary needs a documented replay path: read from DLQ or from a checkpointed offset, transform if needed (e.g., bump version), republish or invoke the consumer directly. Without replay, DLQ becomes a graveyard and outages produce permanent data loss.
+
+**Replay fixtures must be representative by event class.** A synthetic poison message exercises parser failures fine, but is insufficient for:
+- *Signed callbacks* (vendor webhooks with signature + timestamp + raw-body validation): keep captured sanitized raw-body + signature fixtures, or use the provider's test fixtures.
+- *PII or regulated payloads*: privacy-approved fixtures only; the replay tool must accept them without round-tripping through unsanitized logs.
+- *Schema-validated payloads*: fixtures must pass the same registry validation as production messages, or the replay tool will diverge from production semantics.
+- *Stateful sequences* (create/update/delete chains, reserve/release pairs, workflow timeout sequences, multi-event sagas): fixtures must include the ordered sequence, plus the duplicate, missing-predecessor, out-of-order, and replay-after-terminal-state cases. A single in-sequence event replayed in isolation can pass the consumer while the real production failure is a sequence-level invariant violation.
+
+Document which fixture class is used and which classes are deliberately not covered.
+
+## Backpressure
+
+Consumer lag, broker queue depth, and producer rate are the three backpressure signals. Wire them:
+
+- **Consumer-side** — bound the work-in-flight, but **the shape depends on ordering**:
+  - *Unordered consumer*: a bounded work-queue sized to the worker pool, N workers reading from the queue. A shared shutdown signal stops the workers (the stack-glue section names the specific primitive). Order of completion is undefined.
+  - *Ordered partitioned log* (Kafka, Pulsar key-shared, NATS JetStream ordered consumer): **one serial work lane per assigned partition/key**. Feed each partition into its own bounded work-queue + single worker, or process messages serially within the partition's poll loop. Feeding multiple ordered partitions into a single shared work-queue + worker pool loses per-partition ordering, and a slow message on one partition can starve cold partitions or let later offsets overtake earlier ones. The bounded-queue-plus-pool shape is correct for unordered work queues; not for ordered partitioned logs.
+  - *Rebalance handling for partition-assigned consumers* — when a Kafka / Pulsar key-shared / similar consumer group rebalances and a partition is revoked, "one lane per partition" is unsafe without explicit rebalance discipline. The revoked owner must (1) stop fetching from the partition immediately, (2) drain or cancel its in-flight lane (await handler completion to a bounded deadline, or cancel with an explicit `partial-failure` disposition), (3) commit or abort offsets according to the handler outcome (commit only completed offsets; do not commit `last poll` blindly), and (4) be fenced so it cannot still publish a side effect after the new owner has started — typically by tagging each in-flight message with the assignment epoch and refusing side effects whose epoch is stale. Without fencing, the new owner and the old owner can process the same business key concurrently; per-partition ordering at steady state is meaningless if the rebalance window allows concurrent processing.
+  - Avoid unbounded worker-per-message fanout in all cases (the stack-glue section names the specific anti-pattern API).
+- **Producer-side** — when the broker buffer fills (Kafka producer queue, RabbitMQ unconfirmed-publishes limit, NATS slow-consumer warning), block the producer's caller with a bounded wait or shed load at the producer entry point. Never block forever; surface a typed exception after a bounded wait so upstream can backpressure further.
+- **Cross-service** — a slow consumer is an upstream producer's problem to know about. Consumer lag must be exposed as a metric and alerted; producers cannot fix what they cannot see.
+
+## Fanout patterns
+
+- **Pub-sub** — one publish, many independent subscribers. Each subscriber owns its own consumer group/durable subscription, processes at its own pace, has its own DLQ. Add a subscriber by deploying it; no producer change.
+- **Work queue** — one publish, exactly one of N workers handles it. All workers share a consumer group. Add capacity by adding workers in the same group.
+- **CDC (change-data-capture)** — the DB is the source of truth; a CDC connector (Debezium-style) produces events from the WAL/binlog. Useful when many downstream consumers need to react to a DB-owned domain and the writing service does not own a transactional outbox.
+- **Producer-held subscriber list** — generally an anti-pattern (couples producer to consumers; defeats the point of pub-sub). *Legitimate exceptions*: CDC bridges and webhook dispatchers do interact with external targets (a CDC sink config, a per-tenant webhook URL list). The ownership rule for those external lists: the **authoritative list lives in the owning configuration / control-plane system** (the tenant-config service, the connector-config service, the customer-admin app — whichever owns target lifecycle, approval, rotation, and audit). The producer or dispatcher reads a snapshot from that system; it does not own the list unless it *is* the authoritative config owner. Hardcoded subscriber lists in producer code, or producer-owned lists that bypass the tenant-config policy plane, are still the anti-pattern.
+
+## Saga / process manager vs choreography
+
+For multi-step workflows that span services:
+
+- **Choreography** — each service listens for events and emits its own. No central coordinator. Loose coupling; the workflow exists only as the set of subscriptions. Hard to reason about end-to-end; debugging requires tracing across services.
+- **Orchestration (process manager / saga)** — a coordinator service drives the workflow: emits commands, listens for responses, compensates on failure. The workflow is explicit in one place.
+
+**Orchestration is required by workflow invariants, not by step count.** Use orchestration when any of these is true, regardless of whether the workflow has two steps or twenty:
+- Compensation across services is needed (one step's failure requires undoing another step's effect).
+- Audit needs **a single controllable workflow state or a resume/decision point** — not just append-only traceability. Append-only audit (every step emits an immutable audit event tagged with a workflow id) can be served by choreography; a single state view that an operator must inspect or resume requires orchestration.
+- An external irreversible effect is involved (see below).
+- Timeout handling needs a coordinator with a clock.
+- A single business owner needs one place to inspect or resume stuck state.
+
+Choreography is acceptable when none of those hold and the workflow is small and fixed. It is also acceptable for **broadcast fanout** (one event, many independent subscribers, no compensation across subscribers) even when auditability is required — each subscriber owns its own DLQ, immutable audit events carry the shared workflow / trace id, and there is no central state to inspect; the audit story is "did every subscriber see and process this," answered by traces and per-subscriber metrics.
+
+**Compensation, where possible; prevention, where not.** Distributed transactions are not available; design compensation as the default:
+- *Compensable steps* — idempotent compensating action (refund, cancel, undo) that is itself at-least-once and idempotent. Document the compensation per step.
+- *Irreversible steps* — sent email, filed regulatory report, shipped physical item, called external action with no reversal API. No real compensation exists; at best there is a corrective follow-up (apology email, retraction filing, refund + recovery offer) which is a separate workflow. For these:
+  - Add prevention gates before the irreversible step (validation, manual review, approval, dual-confirm, dry-run).
+  - Use `pending` / `confirmed` / `committed` state machine so the orchestrator can fail-stop before the irreversible action.
+  - Define explicit irreversible-state handling: how the workflow records the irreversible commit, what the corrective workflow looks like, who is paged.
+  - Do not design a fictional compensation that "reverses" an irreversible action; document the reality.
+
+If the workflow involves money, audit, or a regulatory requirement, default to orchestration with an explicit state machine.
+
+## End-to-end "exactly-once" illusion
+
+Stack-agnostic recipe; document each clause for every event-driven boundary that claims it:
+
+1. Transactional or outbox-based publish — the message is durable iff the originating state change is durable, atomically.
+2. Idempotent consumer with dedup storage that survives consumer restart and broker redelivery.
+3. Commit-and-publish ordering — consumer's side effect + dedup record + offset commit are atomic from the consumer's point of view (DB transaction including offset, or broker transactional consumer + producer pairing). Brokers without a transactional offset semantic (NATS JetStream's per-ack model, RabbitMQ classic queues without publisher confirms + tx) cannot satisfy this clause; document the gap.
+4. **Side effect under the claim is inside the atomic domain.** "Atomic domain" means the DB transaction that includes the offset commit, or the broker transactional producer/consumer pair. External side effects — third-party API calls, S3/object-store writes, secondary publishes to a different broker, sent emails, SMS, payments — are *outside* the atomic domain. For them you need provider-side idempotency keys, an outbox/process-manager step that records terminal status, or honest documentation as at-least-once. Claiming exactly-once for a flow whose externally visible effect is not in the atomic domain is the most common false claim.
+5. Replay path respects idempotency — a manual replay of a DLQ message lands on the consumer's existing dedup and does not double-apply.
+
+If any clause is missing, the boundary is at-least-once with duplicates. Tell consumers honestly.
+
+## Anti-patterns
+
+- **Post-commit publish (durable cross-process)** — publishing the event after the DB transaction commits, without an outbox, when consumers are in another process. A crash between commit and publish silently drops the event. In-process, same-instance, rebuildable post-commit hooks are not this anti-pattern.
+- **DB-as-queue** — using a DB table as the message broker via polling without an outbox-style design. Fine for low-volume single-instance background jobs in the same service; breaks under load, lacks fanout, and entangles application reads with queue mechanics when used as a cross-service broker.
+- **No idempotency key** — consumer relies on broker's "exactly-once" claim or on hope. Every redelivery becomes a duplicate side effect.
+- **Naive retry without dedup** — retry on the producer (republishing the same logical event) without the consumer recognising it as a duplicate. Doubles the side effect.
+- **Hot partition / hot key** — one key concentrates traffic, blocking the consumer group. Symptom: consumer lag concentrated on a single partition.
+- **Schema drift without registry or envelope** — producer adds a field without coordinating; an older consumer crashes on unknown required field.
+- **DLQ as graveyard** — messages land in DLQ, nobody looks, no replay tooling. The DLQ becomes a silent data-loss channel.
+- **Exactly-once claimed by broker badge** — broker config has an "exactly-once" mode, but the consumer is not idempotent and the producer is not transactional, OR the side effect is outside the atomic domain. The claim is wrong; record correct end-to-end semantics.
+- **Producer-held subscriber list as code** — subscriber list hardcoded at the producer when broker-side subscription is possible. (CDC bridges and webhook dispatchers are exceptions; their lists must live as config with audit and rotation.)
+- **Sync HTTP/RPC as command** — a "command" sent via blocking HTTP/RPC with retry and no replay path. If the call needs the durability of a queue, use a queue; if it needs the latency of HTTP/RPC, accept best-effort.
+
+## Operations checklist (event-driven boundary launch)
+
+Before a new event-driven boundary goes live:
+
+- Delivery semantics declared explicitly in the event contract, including which side effects are inside the atomic domain and which require provider idempotency keys.
+- Idempotency key, dedup storage tier (lossy/Redis vs source-of-truth/durable), and dedup window documented per consumer; Redis-as-authority disallowed for source-of-truth flows.
+- Outbox table + poller in place when atomic publish is required; per-key ordering strategy declared if ordering is part of the contract; no post-commit cross-process publish without outbox.
+- Partition key + estimated cardinality; hot-partition mitigation declared (shard if the ordering boundary allows; tenant-isolation/quota/dedicated partition if it does not); broker-specific scaling/repartition path documented.
+- Schema compatibility mode declared; schema registered or versioned envelope in payload; multi-broker / multi-tenant / external boundaries documented with the hybrid model; rolling deprecation plan in place for known breaking changes; security/compliance retraction path documented.
+- Retry policy (max attempts, base delay, jitter, retry budget) declared per consumer; retry budget scope (per-consumer vs per-provider vs per-tenant) matches the constrained resource.
+- DLQ topic + replay tool documented; replay tested with **representative fixtures** for each event class on this boundary (synthetic for parser; raw-body+signature for signed callbacks; privacy-approved for PII flows).
+- Consumer lag metric exposed; SLO and alert threshold defined; per-partition lag visible for ordered partitions.
+- Backpressure path: bounded work-in-flight in the right shape (per-partition lane for ordered logs; shared pool only for unordered queues); producer behaviour when broker is slow or full is documented and observable.
+- End-to-end exactly-once claim, if made, validated against the five-clause recipe above — including the atomic-domain clause — otherwise the boundary is documented as at-least-once.
+- Workflow that involves money, audit, regulatory, or external irreversible effects uses orchestration with explicit `pending`/`confirmed`/`committed` state machine and named prevention gates; no fictional compensation for irreversible steps.
+
+## Python-specific implementation patterns
+
+These are stack-localized recipes that implement the stack-agnostic patterns above; the sibling Go file localizes the same patterns differently.
+
+- **Library choice axis** — pick by feature, not popularity. For Kafka in Python: `aiokafka` for native asyncio, `confluent-kafka-python` for higher throughput and lower-level offset/transaction control; do not mix in one service. For RabbitMQ: `aio-pika` for asyncio, `pika` for sync. For NATS: `nats-py`. For Redis Streams as a lightweight broker: `redis-py` Stream APIs or `arq` for an opinionated job/queue layer. For Pulsar: `pulsar-client`. Standardize within the service; multi-broker services need an abstraction that does not hide delivery semantics.
+- **Asyncio consumer shape** — depends on ordering (see *Backpressure* above):
+  - *Unordered work queue*: one `asyncio.Task` polls the broker into a bounded `asyncio.Queue`; a fixed-size worker pool of N tasks reads from the queue.
+  - *Ordered partitioned log*: one `asyncio.Task` per assigned partition that processes serially, or a per-partition bounded `asyncio.Queue` with a single worker task. A shared `asyncio.Event` triggers shutdown; never feed multiple ordered partitions into a shared worker pool.
+  - Avoid `asyncio.create_task(handle(msg))` inside a `for msg in consumer` loop — unbounded fanout will OOM under load and discards ordering.
+- **Sync vs async consumers** — if the handler is CPU-bound or calls a sync DB driver (psycopg2, sync SQLAlchemy), use a thread pool or process pool; do not block the event loop. For mixed workloads, route async-friendly handlers to the asyncio worker pool and CPU/sync handlers to a `ProcessPoolExecutor` or to Celery/RQ.
+- **Context propagation** — extract correlation id, trace context, lane/env from message headers into `contextvars` at the consumer boundary. OpenTelemetry's `aiokafka`/`pika` instrumentations restore the trace context automatically; verify they are wired in `observability-and-ops.md`'s OTel/startup section, not in `async-execution-model.md`.
+- **Outbox poller with SQLAlchemy** — an asyncio task that runs a **claim → commit → publish → mark-sent** loop, not "publish inside the DB tx" (a broker call inside a SQLAlchemy session held open for the broker round-trip violates the short-transaction rule in `data-modeling-and-migrations.md`):
+  1. **Claim tx (short)**: `BEGIN; SELECT … FROM outbox WHERE sent_at IS NULL AND (processing_until IS NULL OR processing_until < NOW()) ORDER BY id LIMIT N FOR UPDATE SKIP LOCKED; UPDATE outbox SET processing_until = NOW() + lease, owner = :owner WHERE id IN (…); COMMIT;` — the row is now leased to this poller; session closes immediately.
+  2. **Publish (outside any session)**: `await broker.publish(...)` for each leased row. Duplicate publish on retry is acceptable because the consumer dedups.
+  3. **Mark-sent tx (short)**: `BEGIN; UPDATE outbox SET sent_at = NOW(), processing_until = NULL WHERE id = :id AND owner = :owner; COMMIT;` — guarded by `owner` so a re-leased row (after the original lease expired) is not double-marked.
+  4. **Abandon tx**: after K publish failures, mark row `abandoned` and alert; do not block the partition key forever on a poison row (see the *Dispatcher that refuses to publish `id=N+1`* strategy).
+  For per-key ordering across HA pollers, layer one of the strategies in *SKIP LOCKED and per-key ordering* (hash-routed publisher by `hash(partition_key) MOD N`, per-key advisory lease via the DB's advisory-lock primitive — PostgreSQL `pg_try_advisory_xact_lock(hashtext(partition_key))` for tx-scoped locks bound to the connection's current transaction; MySQL `GET_LOCK(name, timeout)` with explicit session-scoped semantics (release explicitly on success or stall, or rely on session close), and lock names server-wide-scoped so use a fully-bounded namespaced name `lk:<env8>:<svc8>:<purpose8>:<hash16>` (total length = 46 chars including separators, fits inside MySQL's 64-char limit; `<env8>`, `<svc8>`, `<purpose8>` are generated from the canonical environment / service / purpose identities by a **documented deterministic function** (e.g., first-8-of-base32(SHA256(canonical_identity))) or allocated from a **collision-checked registry** — human-readable abbreviations are NOT acceptable unless the registry proves uniqueness in the MySQL server-wide lock namespace; `<hash16>` is the first 16 chars of base32(SHA256(length-prefixed-encoding(canonical_partition_key, versioned_namespace))) — e.g., `SHA256(len(pk) + ':' + pk + len(ns) + ':' + ns)` or canonical JSON/CBOR over `[canonical_partition_key, versioned_namespace]`; raw `pk + ':' + ns` concatenation is **not** acceptable because real partition keys (`acme:prod`, `order:123`, user-supplied ids) can contain `:` and the resulting hash input is not injective. **Namespace-migration safety**: changing `versioned_namespace` requires either a drain / stop-the-world for the poller lane, or a dual-lock period (acquire old + new lock names in canonical order) so that mixed namespace versions across a rolling deploy / rollback cannot acquire different locks for the same partition key and publish concurrently. Without this, a version bump silently splits the per-key serialization lane.), and avoid MySQL NDB / multi-mysqld setups where `GET_LOCK` is not cluster-wide; TiDB supports MySQL-style user-level locks (`GET_LOCK`) cluster-wide in supported versions — verify timeout / deadlock semantics for the deployed TiDB version against a scenario-specific compatibility source: **pinned cluster** → checked-in version pin; **managed channel (TiDB Cloud)** → provider channel/SLA *and* current cluster version (channel alone is insufficient — the version still varies inside the channel); **rolling-upgrade fleet** → min/max active versions across the fleet plus the documented rolling-upgrade policy; **ad-hoc verification** → recorded `tidb_version()` output that includes cluster identity and timestamp. Otherwise fall back to an external coordinator (etcd lease, Redis `SET NX` with TTL) with fencing — with bounded TTL and a fencing token written with each publish; or refuse-newer-id dispatcher with abandoned-row state. Use `asyncpg`-backed SQLAlchemy 2.x async sessions for the poller path.
+- **Idempotency storage** — pick by impact (see *Idempotency design*):
+  - Lossy/rebuildable: `redis-py` `SET dedup:<key> 1 EX <window> NX`, branch on returned value.
+  - Source-of-truth: insert into a `processed_events` table with `event_id` as primary key inside the side-effect SQLAlchemy session; on `IntegrityError(unique violation)`, skip the side effect. Optionally cache the recent N keys in Redis as a hot-path filter, but Redis is not the authority.
+  - Cross-system effects: intent-then-execute pattern — write `processed_events` row with status `pending` in a SQLAlchemy session, run the external call with a provider idempotency key, update status to `done` on success.
+- **Retry policy** — bounded exponential backoff with jitter; `tenacity` is the standard library for this, but configure it explicitly per consumer (max attempts, base delay, jitter, retryable exceptions). Never retry inside the handler with an inline `await asyncio.sleep` past tens of seconds — let the broker redeliver after nack, or push to a delay queue/scheduled topic. Scope the retry budget to the actual constrained resource: per-consumer for isolated handlers, **per-provider / per-tenant / per-region / per-API-method / global** for shared downstreams (the budget belongs to whichever resource has the hard quota; "per-provider" alone is insufficient when the real limit is per-region or per-method).
+- **Exception classification** — define a small exception hierarchy (`RetryableError`, `PermanentError`, `PoisonError`) and classify at the boundary; the consumer dispatcher branches into `retry` / `drop` / `dlq` dispositions. Never let an un-typed `Exception` reach the dispatcher and decide by message-matching.
+- **Test substitution** — define a `Broker` protocol at the service boundary; provide an in-memory implementation for unit tests and an integration test against a real broker (Kafka container, RabbitMQ container) via `testcontainers-python`. Mocking the broker client directly leaks library specifics into tests.
+- **Process model** — when the broker client is sync-only (some Kafka clients, legacy RabbitMQ usage), run the consumer in a dedicated process or thread; do not import a sync broker client into an asyncio service and hope. `async-execution-model.md` covers the decision matrix.
+- **Graceful shutdown order** — signal handler → set shutdown `asyncio.Event` → poller stops fetching → worker pool drains (bounded by `shutdown_grace_period_s`) → outbox poller finishes its in-flight publish → broker client closes → DB engine disposes connections. If the last outbox publish needs an in-flight DB read, hold the engine open until the outbox poller acknowledges drain; the rule is "no new work in flight," not "rigid component order."
+
+### Mirrored-section grep gate
+
+The sibling-sync header forbids three categories of stack-specific token in mirrored sections (everything from "When this applies" through "Operations checklist"; everything *before* the `## Python-specific implementation patterns` H2). Run this grep against the mirrored region before every commit; zero hits required.
+
+Forbidden tokens for this Python file's mirrored sections:
+
+- **DB-engine syntax** — `SET LOCAL`, `set_config\(`, `current_setting\(`, `pg_try_advisory`, `pg_stat_activity`, `BYPASSRLS`, `FORCE ROW LEVEL SECURITY`, `search_path`, `GET_LOCK\(`.
+- **Runtime / concurrency mechanic names** — `context\.Context`, `\bgoroutine\b`, `\bgoroutines\b`, `ctx\.Done`, `database/sql`, `\bsqlx\b`, `contextvars`, `\basyncio\b`, `run_in_executor`, `to_thread`, `ThreadPoolExecutor`, `ProcessPoolExecutor`, `copy_context`, `async with`, `after_commit`, `listens_for`, `asyncio\.Queue`, `asyncio\.Event`, `asyncio\.create_task`, `asyncio\.Task`, `asyncio\.gather`.
+- **Library / framework API names** — `GORM`, `Hertz`, `Kitex`, `golang\.org/x/time/rate`, `SQLAlchemy`, `FastAPI`, `Starlette`, `Pydantic`, `httpx`, `aiokafka`, `\bpika\b`, `aio-pika`, `redis-py`, `tenacity`, `structlog`, `testcontainers`, `Alembic`, `async-lru`, `asyncpg`, `psycopg`.
+
+Run:
+
+```
+awk '/^## Python-specific implementation patterns/{exit} 1' event-driven-architecture.md \
+  | grep -nE '(SET LOCAL|set_config\(|current_setting\(|pg_try_advisory|pg_stat_activity|BYPASSRLS|FORCE ROW LEVEL SECURITY|search_path|GET_LOCK\(|context\.Context|\bgoroutine\b|\bgoroutines\b|ctx\.Done|database/sql|\bsqlx\b|contextvars|\basyncio\b|run_in_executor|to_thread|ThreadPoolExecutor|ProcessPoolExecutor|copy_context|async with|after_commit|listens_for|asyncio\.Queue|asyncio\.Event|asyncio\.create_task|asyncio\.Task|asyncio\.gather|GORM|Hertz|Kitex|golang\.org/x/time/rate|SQLAlchemy|FastAPI|Starlette|Pydantic|httpx|aiokafka|\bpika\b|aio-pika|redis-py|tenacity|structlog|testcontainers|Alembic|async-lru|asyncpg|psycopg)'
+```
+
+Allowed exception: the *Sibling sync* header itself names the three category classes (without tokens) and references this gate; the *Sanitization boundary* header does not contain any of these tokens. The sibling Go file maintains the matching gate for Go-side tokens, so a token forbidden here may legitimately appear in the *Go-specific implementation patterns* section of the sibling, and vice versa.