@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/multi-tenant-isolation.md

@@ -0,0 +1,281 @@
+# Multi-Tenant Isolation (Python)
+
+Use when designing the tenant-isolation architecture of a SaaS service: how tenants are kept apart at the data, compute, network, identity, observability, lifecycle, and compliance layers; how queries, jobs, caches, and external calls carry tenant context safely; how a tenant's data can be exported or deleted on demand; and how shared services keep cross-tenant aggregation auditable.
+
+> **Sibling sync.** A parallel `go-microservice-architecture/references/multi-tenant-isolation.md` mirrors **all non-stack-specific sections** of this file. Only the *Python-specific implementation patterns* section diverges by stack. Maintainers updating any mirrored section here must update the sibling in the same change. The mirrored sections stay free of three categories of stack-specific token: DB-engine-specific syntax, runtime/concurrency-mechanic names, and library / framework API names. The concrete token list and grep command live in the *Mirrored-section grep gate* subsection at the end of this file's stack-glue section. Before commit, run that grep against the file's mirrored sections; zero hits required. The same gate lives in the Go sibling. Routing-reference text (which sibling file each "route to" link points to) may differ between Go and Python because the sibling skill trees differ — this is the only mirrored-section divergence explicitly allowed.
+
+> **Sanitization boundary.** Tenant identifiers, customer names, lane / region names, regulator labels, and quota numbers below are illustrative; concrete values live only in the private alias map. The list of audiences that require sanitization is **positive** (these audiences require it unless explicitly approved otherwise): external / client-facing materials, customer-specific deliverables, regulator or auditor evidence, SOC / compliance reports, procurement responses, internal compliance reviews, sales-engineering or security-questionnaire appendices, partner architecture drafts, and any document that could be forwarded to any of those. "Internal" by itself is not safety; internal documents are routinely forwarded.
+
+## When this applies / does not apply
+
+Apply when:
+- the service has more than one tenant (customer, organization, workspace) sharing any layer (DB, compute, broker, cache, storage),
+- a single tenant's data, behavior, or failure must not be visible to or affect another tenant,
+- tenants have differing commitments (residency, deletion SLA, quota tier, isolation tier),
+- regulatory or compliance scopes (PCI, HIPAA, SOC, residency requirements) require provable tenant separation.
+
+Skip when:
+- the service is single-tenant by deployment (per-customer dedicated stack with no shared layer); route to `platform-release-engineering/SKILL.md` and to this file's *Compliance, residency, sovereignty* section for residency commitments,
+- the service is internal-only with a single owning team (employees of one org are not "tenants" for this purpose); identity/permission boundaries still apply but route to `web-framework-boundaries.md` or `api-contract-and-schema.md`,
+- tenant-equivalent isolation is owned entirely by a platform layer above the service (e.g., per-tenant namespace owned by the platform); route to `platform-service-connectivity/SKILL.md` for the platform contract.
+
+## Tenant isolation tiers (decision tree)
+
+Treat isolation as **two orthogonal placement decisions**: data placement (where the tenant's bytes live) and compute placement (where the tenant's CPU/memory runs). Real systems combine the two — Azure's elastic-pool SaaS pattern, for example, gives each tenant a single-tenant database (high data isolation) while pooling compute (high density). Picking a "tier" globally without naming both placements leads to over- or under-isolating one of them.
+
+**Data placement** — lowest to highest isolation:
+- *Shared with row-level scope* — one DB, one schema, every row carries `tenant_id`, database-level Row-Level Security (RLS) policies enforce filtering at the engine. Cheapest, highest density.
+- *Schema-per-tenant* — one DB instance, one schema per tenant; queries route through the tenant's selected schema namespace. Useful when per-tenant DDL diverges or per-tenant backup is required.
+- *Single-tenant data container on pooled compute* — one database (or equivalent isolated container) per tenant inside a pooled engine. Data is physically separable for export / delete / encryption; compute remains shared.
+- *Database-per-tenant on dedicated compute* — one DB instance per tenant (or per tenant group) on its own compute. Adds blast-radius isolation on the storage engine itself.
+- *Region-per-tenant (or per-tenant-group)* — one full stack per region/cluster, with explicit routing. Required for residency, sovereignty, and the strongest blast-radius isolation.
+
+**Compute placement** — lowest to highest isolation:
+- *Quota-protected shared compute* — one pool of workers serves all tenants; per-tenant quotas (CPU, memory, concurrency, OS-level cgroups) prevent noisy-neighbor.
+- *Pool-per-tenant-group* — bounded pools serve groups of tenants with similar profiles (free tier, pro tier, enterprise tier).
+- *Pool-per-tenant* — a dedicated compute pool per tenant; the strongest noisy-neighbor isolation that still amortizes platform cost.
+- *Dedicated cluster / region* — full stack isolation per tenant; pairs naturally with region-per-tenant data.
+
+**Hybrid placements are first-class.** Real combinations: shared row-scoped data + pool-per-tenant compute (high compute isolation for noisy workloads while keeping data dense); single-tenant data container + pooled compute (high data isolation for compliance while keeping compute dense). The tenant directory records both placements per tenant; the data layer, compute layer, observability, and lifecycle layers each read the placement that applies to them.
+
+**Compatibility constraints between placements (residency / sovereignty boundary).** When a tenant has region-pinned or sovereignty-pinned data, the compute that can read, decrypt, cache, log, index, export, transform, or administer that data **must be pinned to the same region / sovereignty boundary**. A shared global worker pool in `us-east` that pulls jobs for an `EU` residency tenant violates the residency commitment even when the bytes are at-rest in EU. The compatibility rule: for every (data_placement, compute_placement) pair the directory records, the compute placement's region/sovereignty must be a subset of (or equal to) the data placement's. Exceptions exist only for a documented *control-plane-only* path that touches no tenant data (a purely metadata refresh, a placement record write) and that has explicit compliance approval. Observability, logging, audit, and analytics pipelines are *not* control-plane-only; they touch tenant content and must follow the compatibility rule.
+
+Pick each placement by:
+- **Compliance/residency commitments** — the strictest commitment determines the floor (residency demands region-per-tenant data for that region's tenants).
+- **Blast radius requirement** — what failure mode must one tenant be protected from? Compute blast radius is solved by compute placement; data blast radius is solved by data placement.
+- **Per-tenant ops needs** — own backup window, own DDL, own version, own deletion SLA → higher data placement.
+- **Density and cost** — lower placements serve more tenants per dollar; do not pick a higher placement than the commitment justifies.
+- **Migration cost** — the placement must be reachable from where the service is now; designing for a placement the service cannot deliver is worse than picking a lower one with a documented promotion path.
+
+Document each tenant's placement per layer in a tenant directory (a `tenants` table or a config source-of-truth); the directory is the contract that the data, compute, observability, and lifecycle layers all read.
+
+## Tenant context as a first-class value
+
+Tenant id is not derived, inferred from session, or recovered from a payload field consumed downstream. Treat it as a first-class request-scoped value with explicit propagation:
+
+- **Entry-point extraction** — every request, message, job, or trigger names where its tenant id comes from (JWT claim, request header, message header, parent job context). The extraction is one function per entry point; never re-derived later.
+- **Reject missing tenant** — if the tenant id is absent and the operation needs one (almost all of them do), fail at the boundary with an explicit `missing_tenant` error. Do not fall back to a default tenant, a `0` sentinel, or "the only tenant we know about." The rare *legitimately tenant-less* operations — health checks from a load balancer, scheduled cron jobs that operate across tenants, system bootstrap, platform background work, evaluation jobs — are explicitly enumerated by name; each carries an explicit `no_tenant` / `cross_tenant` marker so the default reject path stays default.
+- **Propagation through every hop** — the tenant id rides on the request context, message headers, RPC metadata, log fields, trace baggage, and outbox rows. Concurrent work spawned from a request must carry the same propagation; see the stack-specific section for the work-spawning propagation rules, which differ by mechanism per stack.
+- **No tenant in path params for high-trust operations** — for a privileged operation that names the tenant in the URL or payload, the tenant in the URL must match the tenant id derived from the authenticated principal; never trust the path alone. The mismatch is a security event, not a 400. *Legitimate exception:* a principal authorized to operate on a different tenant (parent-org admin acting on a child tenant, partner integration scoped to multiple tenants) — this is itself a cross-tenant capability and follows the cross-tenant rules below; never a silent allowance.
+- **Cross-tenant operations are explicit** — when an operation legitimately spans tenants (admin queries, marketplace aggregation), it carries an explicit `cross_tenant` capability claim and an explicit list of tenants; the default deny path means no operation can quietly become cross-tenant by losing scope.
+
+## Tenant-aware data access
+
+Once the tenant context is reliable, data access enforces it at multiple layers (defense in depth — the application layer can be bypassed; the DB layer is the engine-level authority *for the application's normal role*, and the application's normal role is what serves end-user requests):
+
+- **DB engine enforcement** — for shared data placements, the engine enforces tenant filtering on every row read or written by the application role. The engine policy is authoritative for that role; the ORM filter is a hot-path optimization. **Bypass paths exist and must be controlled.** A normal engine policy is bypassed by: privileged roles equivalent to superuser, roles holding the engine's bypass attribute, the table owner unless table-level forced enforcement is set, bulk truncate or schema-level operations, logical-replication streams, view-execution semantics where the policy belongs to the viewer vs the definer, and triggers running as a different role. The architecture must (a) enable forced policy enforcement on every tenant-scoped table, (b) use distinct DB roles for application / migration / admin / replication, none of which carry the bypass attribute for the production app role, (c) require explicit tenant filters or offline approval for any path that legitimately runs as a bypass role (data migration, regulator export, repair script), and (d) audit every connection that runs under a bypass role. The stack-glue section names the engine-specific syntax for each control.
+- **Bypass-role audit independence — full control-plane scope, with degraded mode for small orgs.** "Audit every bypass connection" is meaningless if the bypass role (or the operator who can assume it) can read, modify, or *prevent delivery of* the audit. Independence has two parts: **data-plane** (the bypass role cannot `SELECT` / `UPDATE` / `DELETE` / `TRUNCATE` audit rows; ingestion uses a separate write-only role with `INSERT`-only privilege; reads of audit go through a distinct read-only auditor role the bypass roles cannot assume), and **control-plane** (the operator who can use the bypass role cannot modify network policy, security groups, firewall rules, DNS, service-account bindings, sidecars, queues, or routing between the bypass-role host and the audit sink). A bypass operator who can drop egress to the audit DB, run a migration under bypass, and restore egress has defeated the audit even though no audit row was visible. Verify both planes: attempted audit-table `SELECT` / `DELETE` with each bypass role is denied; attempted modification of the audit transport (firewall, DNS, service account, queue) from each bypass operating role is denied; and an alert exists on **audit delivery gaps** (the audit pipeline notices when expected events stop arriving from a bypass-host source) so that a successful tamper of the transport is detected even when no audit row appears.
+  **Degraded mode for small orgs where control-plane independence is impossible.** A small org with one root operator (or one ops team that owns both bypass-role access and infra controls) cannot mechanically separate the two planes. The architecture must not block forever on impossible separation; instead, classify bypass as **break-glass only** under the degraded mode, and replace mechanical independence with: (a) the audit sink delivers to an **external append-only audit store outside the operator's control** (a regulator-grade evidence service, a separate vendor's audit log, a tamper-evident ledger), (b) **two-person approval** is required before any bypass operation runs (one person initiates, a second approves; both identities recorded), (c) **delivery-gap alerting** runs from outside the operator's infrastructure so a tamper of local transport is still alerted by the external system, and (d) a **time-boxed remediation plan** is documented (when the org grows, the mechanical control-plane independence is restored). Record explicitly whether each environment runs in the strong-independence mode or the degraded mode; un-recorded environments default to strong-independence required.
+- **Catalog-level coverage gate by classification of every relation, not just by column.** A representative cross-tenant-read denial test catches the obvious case but misses newly added tenant-scoped tables — especially join tables (e.g., `user_roles(user_id, role_id)`) whose tenancy is inferred via a parent table's `tenant_id`, denormalized projections, composite-key tables where tenant is implicit, **runtime-created tables** (worker-managed temp tables, queue inboxes), **extension-created child tables** (partition-management extensions that materialize child partitions, materialized view refreshes), and views. The schema source-of-truth must **classify every relation** (table, view, materialized view, partition child) as `tenant_scoped` / `global` / `cross_tenant_aggregate` / `system`; the schema-CI gate **fails any migration that adds or modifies a relation without a classification**, including extension-created relations classified by **owner/template rule** (every child partition of a tenant-scoped parent inherits `tenant_scoped`; every relation owned by a tenant-scoped extension inherits the extension's class; relations created by an unrecognized owner block merge). For tenant-scoped relations: if the relation has a `tenant_id` column, the gate verifies engine-policy enabled + forced enforcement + application-role policy + missing-tenant query returns zero / errors; if no `tenant_id` column (inferred-tenancy via parent), the classification names the parent / join path and the gate verifies the policy expression resolves tenant correctly via that join path (the stack-glue section gives the engine-specific syntax). The catalog of classifications is a checked-in source-of-truth artifact reviewed on every migration; unclassified relations block merge.
+- **Tenant session state must survive the actual unit of work** — the session-level tenant variable that the engine filter reads must be set so that *every query in the request* sees it. Transaction-local settings vanish at statement end if no transaction is open; session-scoped settings leak across connection check-in unless reset. The stack-glue section names the safe pattern per stack: either wrap all tenant-scoped work in an explicit transaction with the tenant variable set inside it, or set the session variable on every connection checkout and reset it on every check-in with a default-deny engine policy when the variable is missing. Default-deny on missing tenant is the only correct fail mode at the engine layer.
+- **Connection pool tenancy** — a connection cannot leak across tenants between two checkouts. Either issue an explicit reset on check-in, or use a connection-per-request pattern with explicit tenant set on checkout, paired with a default-deny engine policy. Reset hooks are *cleanup*, not *correctness*: the correctness comes from setup on checkout / transaction start plus default-deny, not from trusting every reset path (invalidation, async-connection GC, terminating connections may skip the reset). Alert on connections returned to the pool with a non-default tenant setting.
+- **Cache classification, not "every key carries tenant"** — caches divide into classes; each class has its own keying and access rule:
+  - *Tenant-scoped cache* — every key includes the tenant id; cross-tenant read is a leak.
+  - *Global immutable or config cache* — values truly global to the product, immutable or refreshed centrally; no tenant key; intentionally cross-tenant.
+  - *Cross-tenant aggregate cache* — pre-computed aggregates (top-N, billing rollups); access is gated by the cross-tenant capability; keys do not carry tenant but reads are authorized.
+  - *Prewarm / system cache* — operational caches without specific tenant; access is system-only.
+  **Differential test before classifying a cache as global, with declared dependency domains covering external sources.** Before declaring a cache "global," answer: *if I served this exact key to tenant A and to tenant B from the same cache entry, could the value, the visibility, or the authorization differ?* Causes of divergence include tenant contracts, entitlements, feature-flag targeting (per-tenant rollout), AB-test assignment, residency/region, model entitlement, partner tier, jurisdiction-specific behavior, and tenant-overridden config. If any can differ, the cache is tenant-scoped, not global, even when the data is called "config." Pricing, entitlement, model-routing, flag-targeted config, and partner-product caches are almost always tenant-scoped despite looking like global config. **Future-proofing: the classification is not a one-time decision.** Each cache declares its **dependency domains** — the input domains whose schema changes can introduce tenant-varying divergence (entitlements, feature flags, contracts, pricing, model routing, residency, tenant config). Dependency declarations **must include external and runtime-config sources**, not only the team's own checked-in schemas: third-party feature-flag SaaS, runtime A/B-test config services, partner-managed entitlement APIs, vendor-side billing or pricing surfaces. For each external/runtime dependency, declare a change signal: a webhook-backed drift trigger that fires on the dependency's change events, a scheduled attestation job that re-verifies the differential test on a documented cadence, or — when no reliable change signal is available — **default downgrade to tenant-scoped**. A change to any declared dependency domain (internal schema or external/runtime config) triggers a CI gate that reviews and either re-runs the differential test or downgrades the cache. Without dependency declarations that cover external sources, a global cache silently becomes wrong the moment a remote schema/config change ships. Audit each cache for its class; raw cross-tenant reads on a tenant-scoped cache without tenant in the key are the most common cross-tenant data leak in practice.
+- **Cross-tenant queries are exceptional, audited, and capability-gated** — operations that legitimately span tenants (search across customers for an admin, billing aggregation across tenants) bypass the per-tenant filter via a named capability. Every cross-tenant query writes an audit row (who, what, why, which tenants, what was returned in aggregate); the capability is rate-limited; the audit table is read by the privacy team. Do not let cross-tenant become a runtime hatch with no record. (See *Cross-tenant features* below for the modes that handle high-volume legitimate cases.)
+- **Read replicas carry the same enforcement** — engine filtering, schema routing, and bypass-role controls all apply to replicas; a replica that bypasses enforcement is a leak. Verify on every replica including logical-replication downstreams.
+- **Backup and analytics pipelines carry the same enforcement** — exporting data to a warehouse, a backup, or an analytics broker is still tenant-scoped; the export tool either preserves the tenant column or partitions by tenant. Where enforcement cannot happen at the source (a warehouse that loads via bulk insert), it must happen at the consumer (warehouse-side row policies, view filters). An analytics dashboard that loses tenant scope is a cross-tenant leak with a delayed timer.
+
+## Quota and rate limit per tenant
+
+A noisy or hostile tenant must not consume more than its share of any contended resource. Apply per-tenant budgets at every layer where contention is possible:
+
+- **API gateway / ingress** — per-tenant requests/sec; surface a `429 Too Many Requests` with a `Retry-After` header; tag the rejection with the tenant for observability.
+- **Background worker / job runner** — per-tenant concurrent job budget; per-tenant queue depth limit; per-tenant CPU/memory hint where the runtime supports it.
+- **DB connections** — per-tenant connection share; a hostile tenant cannot starve the pool. Connection pool with per-tenant tag is the simplest; per-tenant DB user with role-level limits is stricter.
+- **Broker / messaging** — per-tenant publish rate; per-tenant consumer lag budget (one tenant's slow consumer cannot block the rest).
+- **Object storage / egress** — per-tenant request and byte budgets; egress cost attribution is a tenant attribute.
+- **External provider quotas** — when calling a shared downstream (LLM vendor, payment provider), the per-provider budget (see `event-driven-architecture.md`) is partitioned per tenant; a single tenant cannot burn the shared quota.
+- **Shared accelerator / GPU inference** — modern inference systems rely on continuous or dynamic batching for throughput; hard-partitioning the batcher per tenant destroys utilization economics without necessarily improving fairness. For shared accelerators, the default isolation is **fair scheduling + admission control + per-tenant spend / request budgets + abuse-isolation rate limits**, not per-tenant physical or batch partitioning. Document the fairness algorithm (token bucket per tenant feeding a fair scheduler; per-tenant fallback when over budget) and the abuse-isolation cutoff (a runaway tenant is degraded, not allowed to starve the batch). **Exception for SLO-contracted or large-prefill workloads:** named tenants with latency-SLO contracts, or workload classes with rare but very large requests (long prefill, big batches), starve under pure fair queuing when many small requests fill the batcher. For those, layer one of: reserved accelerator capacity for the named tenant, a priority lane with bounded share, max-wait admission control that preempts smaller jobs after a deadline, or a dedicated partition. Document which mechanism applies per tenant / workload class; fair scheduling is the default for shared long-tail traffic, not proof that partitioning is wrong.
+- **Resources without quota** — file descriptors, ephemeral ports, in-memory buffer pools, broker per-topic quotas, OS-level limits. Per-tenant quota does not exist for these by default; the architecture protects them with global guardrails plus per-tenant attribution so that exhaustion is traceable to a tenant.
+- **Quota tier alignment** — quotas align with the tenant's tier in the tenant directory; an enterprise tier carries higher budgets than a free tier. *Exception:* free-tier abuse protection sometimes requires *stricter* limits than enterprise baseline, and trial tenants may carry elevated quotas. Quotas are tenant-directory attributes, not hardcoded in code.
+
+For each quota, define: the budget, the enforcement layer, the response when exceeded, the alert when the budget is approached, and the escalation path when a tenant legitimately needs more.
+
+## Tenant-aware observability
+
+Every telemetry record carries tenant scope. Metrics use `tenant_id` only for the bounded named cohorts and heavy-hitters defined below (full per-id labeling at long-tail scale is infeasible); logs and traces carry `tenant_id` on every record; all three carry an explicit `cross_tenant` / `platform` marker when the operation is not single-tenant. The compromise model below balances the operator's need for "which tenant is affected" against the cost of unbounded cardinality:
+
+- **Headline metrics by tenant id** — RED / USE metrics on the most operationally critical paths (request rate, error rate, latency) carry `tenant_id` for **bounded, named cohorts**: paid tenants, high-risk tenants, internal tenants. Free-tier and long-tail tenants do not get individual labels.
+- **Long-tail metrics by tenant tier** — for tenants outside the named cohorts, metrics carry `tenant_tier` (free / pro / enterprise) instead of `tenant_id`; the operator can detect tier-wide issues without per-tenant cardinality.
+- **Heavy-hitter detection** — a separate fast path identifies the top-N tenants by traffic in a rolling window; a noisy tenant from the long tail is promoted to per-id observation when detected.
+- **Exemplars / trace links / log joins** — for the long tail, traces and logs carry `tenant_id` even though metrics do not; the operator joins from a tier-level metric alert to traces / logs filtered by tenant id when investigating.
+- **Cohort SLOs** — per-tenant SLO computation is infeasible at millions of tenants; define SLOs per cohort (per-tier, per-region, per-product-line) with the named-tenant exceptions for paid / high-risk getting their own SLO. The single-tenant SLO is reserved for tenants whose contract names one.
+- **Log fields** — `tenant_id` as a structured field on every log line; the log aggregator filters by tenant. Logs without tenant scope are usable only for stack-trace forensics, not for incident triage.
+- **Trace attributes** — `tenant.id` as a span attribute (OpenTelemetry semantic convention); trace baggage carries tenant id across service hops.
+- **Audit log scope** — security-relevant events (cross-tenant query, admin god-mode, deletion, export, regulated transactions) write to an append-only audit stream with stricter retention and access control. For regulated domains (books-and-records under FINRA/SEC and similar regimes), specific structured app logs and ops logs may *also* be classified as audit records; the audit scope is set by the compliance mapping, not by a default "audit is a separate stream from logs." Cross-index ops logs into the audit retention plane where regulators require it.
+
+## Per-tenant lifecycle
+
+The lifecycle operations — provision, suspend, export, delete, retention, archive — are tenant-scoped and contractual.
+
+- **Provision** — creating a tenant is an idempotent workflow: tenant directory row, isolation placement resources (schema / DB / region per placement), default quotas, seed data, identity provider linking. Each step must be retryable; partial-provision must be recoverable, not leave a half-tenant.
+- **Suspend** — pause access without deleting data. Used for non-payment, security incidents, or compliance holds. Suspension is reversible; a suspended tenant's data is preserved according to the retention policy.
+- **Export (right to portability)** — the tenant can request all of their data in a documented format. The export must include data across every store (DB, broker offsets if relevant, object storage, analytics warehouse extracts) and must not leak other tenants' data; the export job runs under the same tenant-scoped enforcement as normal queries. Track export SLA per tenant tier.
+- **Serving suppression before physical delete (per-path inventory + propagation barrier)** — most deletes are async across stores; the production DB row may be deleted while a search index, CDN edge cache, hot in-memory cache, or downstream materialized view still serves the document for minutes or hours. Mark the tenant (or the specific object) in an **authoritative denylist** at the moment deletion starts; every serving path (API read, search query, CDN origin, cache layer, replica read, analytics dashboard) consults the denylist before serving. **The denylist write must be propagated to every serving path before any destructive source action runs** — fire-and-forget denylist replication is not enough; if any serving path receives the denylist update through an eventually consistent config channel whose lag can exceed source delete latency, stale tenant content keeps being served. Maintain a **serving-path inventory** that lists every path serving tenant data; for each path, one of these **suppression modes** is declared and tested before launch:
+  - *Ack barrier* — the path implements an ack endpoint; the deletion workflow blocks source destruction until the path acks the denylist version.
+  - *Central strong check* — every serving read calls a central consistent suppression store before serving until async replicas catch up.
+  - *Path disabled / removed from serving* — the path is removed from the serving topology before destructive action (acceptable when the path's content is rebuildable).
+  - *Vendor purge with verifiable completion* — for third-party CDN / hosted analytics / managed warehouse paths that cannot ack or call back, invoke the vendor's purge API and wait for its completion signal; the vendor's purge API itself must be auditable.
+  - *Wait-out max TTL* — block destructive action until the path's documented max-content-TTL has elapsed since the denylist write (acceptable only when TTL is bounded and short; record the max-TTL per path).
+  Denylist entries are removed only after the per-store physical-deletion modes below complete. The workflow refuses to advance past denylist write until each path's declared mode has been satisfied. A path with no declared mode blocks the workflow; un-ackable paths must explicitly pick vendor purge, disablement, or wait-out, not silently fall back to fire-and-forget.
+- **Delete (right to erasure) with per-store modes** — define explicitly per store *which* deletion mode applies; "deletion complete" means the committed policy state for each store, not always physical absence:
+  - *Hard-delete verified* — rows are physically removed; verification queries confirm absence (production DB, hot indexes, hot caches).
+  - *Anonymized* — identifiers and PII are stripped; rows that must outlive the tenant for regulatory reasons (audit records, financial transaction history) become non-attributable.
+  - *Beyond-use / suppressed until purge* — the store cannot delete on demand (backups, WORM archives, snapshot-only warehouses); access is suppressed and the data is included in the next regularly scheduled purge (which has its own SLA per tier).
+  - *Retained under legal basis* — explicit hold for a legal or regulatory reason; the hold itself is documented and revisited.
+  - *Unverifiable immutable append* — write-only audit logs, blockchain-style ledgers; deletion is impossible by design; the tenant is informed.
+  Each per-store step records `pending` / `done` / `unknown`; an `unknown` triggers a reconciliation job and alert; per-store mode is encoded in the workflow so completion can be reasoned about.
+- **Retention** — per-tenant or per-tier retention windows for production data, logs, audit, and backups. Retention drives storage cost and compliance; it is a tenant-directory attribute.
+- **Archive** — moving cold tenants to cheaper storage; the archive path preserves tenant scope and supports re-hydration on request.
+
+The export, delete, and archive workflows are the highest-stakes lifecycle paths — they are also the easiest to half-implement. The `unknown` / `reconcile` state pattern from `event-driven-architecture.md` applies; the per-store deletion modes above prevent the workflow from either falsely marking complete or blocking forever.
+
+## Per-tenant rollout
+
+Code changes that touch a tenant's data or behavior do not roll out everywhere at once.
+
+- **Per-tenant feature flags** — a flag system that can target by tenant id, tenant tier, or tenant tag. Flag service is itself a tenant-aware service.
+- **Per-tenant canary** — promote a tenant or tenant group to the new version first; observe per-tenant SLO (where it exists) or per-cohort SLO; promote next group only on green.
+- **Per-tier promotion ordering** — typical order is internal/synthetic tenants → free tier → pro tier → enterprise tier; never test on enterprise first.
+- **Per-tenant rollback** — when a change harms a single tenant, rolling back for that tenant must be possible without rolling back the whole fleet (config flip, route override, schema rollback). Document the rollback path per change class.
+- **Per-tenant schema migration** — for schema-per-tenant or DB-per-tenant placements, migrations run per tenant; the runner tracks progress per tenant and isolates failures.
+
+## Cross-tenant features (admin views, aggregations, investigations)
+
+A small set of legitimate operations cross tenant boundaries. Each is an exception, not a default. Different volumes need different modes — a single rule cannot serve both an admin's single search and a security team's 24/7 fraud monitoring:
+
+- **Normal-volume admin operations (per-call audit)** — admin search, individual support actions, single-tenant repair scripts. Each call: explicit capability claim, audit row per call (who, when, what, which tenants), short TTL access token, rate-limited, surface-limited (aggregates not raw rows where possible).
+- **High-volume investigations (per-job / per-session audit)** — security forensics, fraud monitoring, regulator response, customer-requested cross-tenant joint debugging. Per-call audit would either flood the audit pipeline or be down-sampled to uselessness; audit operates at the **session** or **job** granularity (who started it, when, what tenants are in scope, what was the purpose, when it ended), with per-call records summarized at the session level. The session itself is short-TTL and renewable with approval.
+- **Break-glass raw access** — incident response where an operator legitimately needs raw cross-tenant rows. Allowed under a named break-glass capability with after-the-fact review (the operation runs, the audit happens immediately, the review happens within a documented SLA), separate retention, and a post-incident report.
+- **Audit capacity controls** — the audit emission rate itself is rate-limited per capability so that legitimate high-volume use cannot DoS the audit system; over-quota events page the security team.
+
+"Admin god-mode" with no audit, no rate limit, no TTL, and no review SLA is the single most common path to a cross-tenant data leak. Never ship it.
+
+## Compliance, residency, sovereignty
+
+Tenant commitments around where data lives and who can see it shape the isolation placement directly.
+
+- **Residency** — "tenant X's data stays in region Y" is a region-per-tenant or region-pinned commitment; the data plane (DB, object storage, backup, analytics) all honor it; the routing layer enforces it.
+- **Sovereignty** — government / regulated tenants may require a separate stack with no cross-border access; this is a deployment-level isolation, not a runtime knob.
+- **Encryption** — at-rest encryption per tenant (separate keys per tenant) is a stronger model than a shared key.
+- **Crypto-deletion is conditional, not universal** — destroying the per-tenant key is acceptable proof of erasure **only when** the key hierarchy, key backups, envelope keys, restore paths, and the relevant regulator's interpretation all support it. NIST SP 800-88 treats cryptographic erase as a sanitization technique with conditions; some interpretations of GDPR distinguish anonymization (irreversible) from pseudonymization (key-linkable encrypted data may still be personal data). Where any condition fails, crypto-deletion is a *beyond-use / suppression* control (one of the per-store deletion modes above), not proof of erasure; the deletion workflow records the actual mode achieved per tenant per store, and the tenant is told what was achieved. **Required evidence before recording "erasure" via crypto-deletion**, per store and per tenant; each artifact must be authentic to *this* deletion, not theatrical paperwork:
+  - *(a) key hierarchy diagram* — scoped to this store and this tenant's key version, dated within a defined freshness window (e.g., last 30 days), showing every key that wraps or could reconstruct the data.
+  - *(b) key-backup inventory* — for this store, scoped to this tenant's keys, naming every backup location, rotation policy, and the holder of each backup; dated within the freshness window.
+  - *(c) restore-path test result* — run against *this* backup store with *this* tenant's key-version destroyed, confirming the restore fails because the key is gone. A restore test on a different store or a different key-version is not evidence.
+  - *(d) envelope-key destruction proof* — for every wrapping key in the hierarchy, signed or countersigned by a role that is **independent of the role executing the deletion** (no operator may self-attest their own destruction).
+  - *(e) backup-key purge SLA* — the crypto-erasure is not complete until backup-key purge completes; the SLA names the date by which all backups containing the key will be purged, and the workflow remains in `beyond-use` until that date.
+  - *(f) compliance / regulator acceptance* — explicit acceptance recorded for this store and this jurisdiction, dated, naming the regulatory regime; a generic "GDPR" acceptance from headquarters does not cover a regional regulator.
+  Each artifact's chain-of-custody must use **a non-forgeable workflow execution identifier**, not a bare run id: the workflow engine signs each execution record with a nonce or content-addressed event id (the hash of the execution's immutable input state), and each artifact references that signed/hashed identifier. Artifact hashes are themselves recorded in the append-only audit store at production time. A sequential or predictable run id alone — which a CI system could let any operator re-trigger with arbitrary metadata — is not chain-of-custody. Without all six artifacts at the required scope and freshness, and a non-forgeable execution-record link, the mode is *beyond-use* until they are produced.
+- **Data subject rights** — GDPR-style erasure, portability, access requests are tenant lifecycle operations (see *Per-tenant lifecycle*). The compliance team owns the policy; this skill owns the executable workflows.
+- **Audit retention vs data retention** — separate from data retention; audit logs typically have longer required retention (years) than operational data. *Conflict case:* when audit retention preserves data that the tenant has a deletion right over, the deletion workflow's `anonymized` mode is the typical resolution: audit records survive without tenant-attributable PII. Document the resolution per regulator.
+
+Residency and sovereignty commitments are made at the customer-contract layer (product / legal); the architecture enforces what was promised. Do not make commitments the architecture cannot deliver, and do not under-deliver isolation that was promised. Silent migrations (an acquired tenant assigned to a placement the architecture cannot deliver) need an audit-and-remediation workflow before the commitment lapses.
+
+## Migration between tiers
+
+Tenants graduate between placements when their commitments, load, or risk profile changes. The migration path must be designed up front; retrofitting it is the painful case.
+
+- **Promotion** — move one tenant from a lower placement to a higher one. Steps: provision new placement resources, copy data with tenant-scoped enforcement, cut over reads, drain writes from the old placement, cut over writes, decommission the old tenant's footprint.
+- **Classify each step's reversibility before starting.** Two classes of irreversibility apply:
+  - *Structural irreversibility* — the step itself rewrites a primitive the system cannot un-do: ID-scheme reshape, shard-key rewrite, encryption-key envelope change, object-prefix change, warehouse re-partition.
+  - *Time-based irreversibility* — the step looks reversible at start, but a TTL, retention job, token-expiry, object-lifecycle rule, log-purge cadence, or backup-rotation window will delete the old state during the migration window. By the time rollback is attempted, the old state is gone even though no structural irreversible step ran.
+  For each irreversible step (either class), define the pre-step checkpoint or export snapshot, the reconciliation query that verifies post-state, and the **last safe rollback point** before that step. The last safe rollback point is the **earliest** of the structural-irreversible boundary and any time-based deadline (the soonest TTL, retention purge, or lifecycle expiry that touches state needed for rollback). Either freeze / extend those timers across the migration window, or compute the rollback point from them. Beyond the last safe rollback point, recovery is forward-fix, not rollback.
+- **Live dual-write is itself a small process-manager workflow** — dual-write between placements diverges under load, partial failure, and ordering reordering. Prefer write-freeze windows or outbox-replay-from-checkpoint over ad-hoc dual-write; if dual-write is used, the divergence-detection and reconciliation job is part of the migration design, not the operations runbook.
+- **Per-tenant migration windows** — a single tenant's migration does not require a fleet-wide window; the system supports per-tenant maintenance.
+- **Audit and verification** — verify per-store that all data is in the new placement before decommissioning the old; "did we leave anything behind" is the most common defect.
+- **Demotion is rare** — moving a tenant down a placement is usually for cost recovery on suspended tenants; the path is rarely automated and is usually a manual operation.
+
+## Anti-patterns
+
+Block these:
+
+- **Tenant id derived at boundary, then trusted as a payload field downstream** — extracted from a JWT at the gateway, passed down as a payload field that downstream code treats as authoritative. Either re-verify at the data layer (engine-level filter), or document explicitly that a specific hot-path runs under a verified upstream filter and cannot be reached by an unverified path.
+- **Tenant-scoped cache without tenant in the key** — the single most common cross-tenant data leak. Audit every cache against the cache classification above.
+- **Engine-level tenant filter disabled "for performance"** — turning it off makes the application layer the only barrier. Tune indexes and queries first; never disable the engine filter as the fix.
+- **Bypass-role connections without explicit tenant scoping** — migration runs, repair scripts, regulator exports, replication streams that run as a bypass role without an explicit per-row tenant filter. Treat any bypass-role connection as if there is no engine filter and add the application-layer filter, the audit, and the approval.
+- **`tenant_id` as nullable / optional column** — every tenant-scoped table has `tenant_id NOT NULL`; nullable means a row that can be returned to any tenant.
+- **Cross-tenant query as ad-hoc DB read in a debugging path** — the privacy boundary does not have a "debug mode" exception. Use the audited cross-tenant capability, or do not run the query.
+- **Admin god-mode without audit, rate limit, or TTL** — a sustained admin session can see anything. Audit per call (or per session for high-volume); rate-limit; short-lived tokens; review SLA.
+- **Logs and metrics without tenant scope** — incident response cannot answer "which tenant" without the label; aggregate dashboards quietly mix tenant impact.
+- **Tenant-scoped data in shared object-store buckets without per-tenant prefix and policy** — a misrouted object can be visible cross-tenant. Per-tenant prefix + IAM policy; never rely on the application path alone.
+- **Deletion that leaves data behind past the documented mode** — "we deleted from prod" is not deletion if a backup still has it past the backup-purge SLA. The deletion workflow names the mode per store and is honest about what was achieved.
+- **Tier commitment in contracts that the architecture cannot deliver** — promising residency-in-region-X when the deployment does not have a stack in region X. Product / legal must consult the architecture before committing; silent migrations need a remediation workflow.
+- **Cross-tenant feature added without capability gate, audit, or mode** — every cross-tenant code path needs an explicit capability claim and an audit mode appropriate to its volume (per-call / per-session / break-glass); no exceptions.
+- **"Crypto-deletion" claimed without validating the conditions** — destroying a key is not erasure when key copies exist, when envelope keys would reconstruct, when backups still hold the key, or when the regulator does not accept it.
+
+## Operations checklist (multi-tenant boundary launch)
+
+Each item below is a verifiable action; "complete" requires the artifact named, not an acceptance statement.
+
+- Data placement and compute placement declared per tenant in the tenant directory; the directory is queryable by the data, compute, observability, and lifecycle layers and one query returns the placement set for any tenant id.
+- Tenant context extraction documented per entry point; a synthetic test request with no tenant id at each entry point produces a `missing_tenant` rejection; cross-tenant entry points require the explicit capability and the test without the capability is rejected.
+- Engine-level enforcement verified by an end-to-end test that attempts a cross-tenant read with the application role and is denied; bypass roles enumerated; one connection per bypass role tested and an audit row confirmed.
+- Schema-CI catalog gate runs on every migration: **every relation** (table, view, materialized view, partition child, runtime-created or extension-created relation) is classified `tenant_scoped` / `global` / `cross_tenant_aggregate` / `system` in the checked-in catalog; the build fails on any unclassified relation; extension-created relations follow the owner/template rule. For tenant-scoped relations, the gate verifies engine-policy enabled + forced + application-role policy covers the relation + missing-tenant query returns zero/errors; tenant-scoped relations without a `tenant_id` column declare the parent / join path and the gate verifies the policy resolves tenant via that path.
+- Bypass-role audit independence verified at **both planes**: (a) data-plane — `SELECT` and `DELETE` against the audit sink with each bypass role denied; reads of audit routed through a distinct read-only auditor role; (b) control-plane — attempted modification of audit transport (firewall, security group, DNS, service-account binding, sidecar, queue, routing) from each bypass operating role denied; (c) audit-delivery-gap alert exercised by a synthetic transport tamper (audit pipeline notices when events stop arriving from a bypass source). Environments running the degraded mode for small orgs record: external append-only audit store, two-person approval, external-system delivery-gap alerting, time-boxed remediation plan.
+- (Compatibility) every (data_placement, compute_placement) pair in the directory passes the residency / sovereignty compatibility rule: compute that can read / decrypt / cache / log / index / export / administer region-pinned data is itself region-pinned to the same boundary; control-plane-only exceptions have compliance approval recorded.
+- Tenant session variable set inside an explicit transaction or on every checkout with reset on check-in; pool-reset hook in place; an alert exists on the pool-state metric for connections returned with non-default tenant.
+- Cache classification documented per cache (tenant-scoped / global / cross-tenant aggregate / system); each cache declares its dependency domains (internal + external/runtime config sources, e.g., third-party feature-flag SaaS); a CI gate re-runs the differential test or downgrades the cache when any declared domain changes; for external dependencies without a reliable change signal, the cache defaults to tenant-scoped. A grep/lint gate records zero raw cache-read calls outside the keying helper for tenant-scoped caches; cross-tenant aggregates have an authorization gate that an end-to-end test exercises.
+- Per-tenant quotas listed by layer (API, worker, DB connections, broker, storage, external providers, accelerators, OS-level resources without quota); each quota has a documented budget, enforcement layer, exceeded-response, alert threshold, and escalation owner; a synthetic over-quota test exercises each.
+- Observability: a sampled request, message, and job each show `tenant_id` (for named cohorts) or `tenant_tier` (for long tail) on the headline metric, in the structured log, and in the trace; the per-tenant or per-cohort SLO dashboard URL is linked from the runbook; heavy-hitter detection produces a tenant id within N minutes of synthetic traffic.
+- Cross-tenant capability gates verified: per-call audit for normal admin, per-session for high-volume, break-glass with post-hoc review; the audit emission rate-limit is configured; over-quota emission triggers a security page.
+- Per-tenant lifecycle: provision is idempotent (a synthetic re-provision is a no-op); export job runs under tenant-scoped enforcement and a synthetic test confirms zero cross-tenant rows in the output; delete workflow maintains a **serving-path inventory** with one declared suppression mode per path (ack barrier / central strong check / disabled-from-serving / vendor purge / wait-out max TTL); the workflow blocks source destruction until each path's mode is satisfied (the test exercises a synthetic path failing to ack and confirms the workflow does not advance); per-store deletion modes (hard-delete-verified / anonymized / beyond-use / retained-under-legal-basis / unverifiable-append) each have a test; **crypto-deletion mode** requires the six evidence artifacts at per-store/per-tenant/per-key-version scope, within the freshness window, countersigned by an independent role, linked to the workflow's signed/content-addressed execution identifier (not a bare run id), with artifact hashes in the append-only audit store; archive and re-hydration tested.
+- Per-tenant rollout: feature flags target by tenant / tier / tag; canary order documented; per-tenant rollback path tested for at least one representative change class.
+- Compliance commitments per tenant in the directory; residency / sovereignty / encryption / data-subject-rights workflows in place where promised; crypto-deletion mode classified per store; audit retention vs data retention conflict resolution documented per regulator.
+- Migration path between placements (promotion) designed; structural and time-based irreversible steps classified; TTL / retention / lifecycle deadlines that intersect the migration window are frozen or extended **only where legally permitted** (regulator-mandated retention maxima and data-subject deletion SLAs cannot be paused — in those cases the deadline is immutable and the last safe rollback point is computed from it); last safe rollback point computed as the earliest of structural and time-based boundaries, accepting that beyond it recovery is forward-fix; a migration dry-run on a synthetic tenant has run end-to-end.
+
+## Python-specific implementation patterns
+
+Stack-localized recipes; the sibling Go file localizes the same patterns differently.
+
+- **Tenant context on `contextvars.ContextVar`** — define a module-level `tenant_id_var: ContextVar[UUID | None] = ContextVar("tenant_id", default=None)` and helpers `set_tenant(id) -> Token`, `get_tenant() -> UUID | None`, `require_tenant() -> UUID` (raises on missing). Every entry point (FastAPI dependency, Starlette middleware, message handler) sets the var once; downstream code reads via `get_tenant()` / `require_tenant()` and never accepts tenant as a separate argument.
+- **Async task spawning** — in Python 3.11+, `asyncio.create_task(coro())` **copies the current context by default**; the tenant var propagates without explicit `copy_context()`. Pass an explicit context only when overriding (e.g., starting a long-lived worker with a different tenant scope). The dangerous propagation gaps are **not** `create_task`; they are:
+  - `loop.run_in_executor(executor, fn)` and raw `concurrent.futures.ThreadPoolExecutor.submit(fn)` — the executor thread does **not** see the caller's contextvars. Wrap the callable: `loop.run_in_executor(executor, contextvars.copy_context().run, fn, *args)`.
+  - `asyncio.to_thread(fn, *args)` — does propagate contextvars (it uses `copy_context().run` internally); the gap is `run_in_executor`, not `to_thread`.
+  - `ProcessPoolExecutor` — separate processes never share contextvars; pass the tenant id as an explicit argument to the submitted function.
+  - Long-lived background workers (Celery, arq, custom consumers) — establish per-message context inside the worker; do not rely on the parent process's contextvars.
+  - Callbacks from foreign threads (signal handlers, third-party callback APIs) — contextvars are per-thread; a foreign-thread callback enters with the foreign thread's contextvars, not the request's.
+- **FastAPI dependency for tenant** — a single dependency `tenant: UUID = Depends(require_tenant_from_jwt)` validates the JWT, looks up the tenant in the directory, sets `tenant_id_var`, and returns the id. Endpoints declare the dependency explicitly; missing-tenant raises `HTTPException(status_code=401)` before the endpoint body runs. Cross-tenant endpoints use a distinct dependency `cross_tenant: CrossTenantCap = Depends(require_cross_tenant_cap)` that reads a different claim, writes an audit row before the handler runs, and tags the request with the audit mode (per-call / per-session / break-glass).
+- **PostgreSQL tenant session variable with FORCE RLS** — schema setup: `ALTER TABLE <t> ENABLE ROW LEVEL SECURITY; ALTER TABLE <t> FORCE ROW LEVEL SECURITY;` for every tenant-scoped table (FORCE makes the table owner subject to RLS too). Create distinct DB roles: `app_user` (no BYPASSRLS), `migration_user` (BYPASSRLS, used only by Alembic/SQLAlchemy migration sessions), `replication_user`, `admin_user` (used only via audited bastion). Application connects as `app_user`. Tenant variable: set inside an explicit transaction per request with `await session.execute(text("SELECT set_config('app.tenant_id', :tid, true)"), {"tid": str(tenant_id)})`. The `true` third argument is tx-local; **outside an explicit transaction the setting vanishes at statement end**. Wrap every tenant-scoped request in `async with session.begin():` (or use `async_session_factory()` with explicit `begin()`); SQLAlchemy 2.x's autobegin gives you that as long as you use the unit-of-work pattern, but a `session.execute` outside `begin()` is the failure mode to avoid. Default-deny policy: `CREATE POLICY tenant_isolation ON <t> USING (tenant_id = current_setting('app.tenant_id', true)::uuid)`; the `true` second arg to `current_setting` makes a missing setting return NULL, which fails the policy and rejects the row — default-deny without crash.
+- **Connection pool reset** — `event.listens_for(pool, "reset")` issues `RESET app.tenant_id; RESET ROLE` on connection return. Treat reset as cleanup, not correctness; SQLAlchemy's reset event has paths (terminating connections, async-connection GC, invalidated connections) where the hook is not guaranteed to execute the SQL. Correctness comes from setup on every transaction (the `set_config` above) plus the default-deny policy that rejects rows when the setting is missing. A periodic sampler queries `pg_stat_activity` and alerts on sessions with `app.tenant_id` set outside an active query.
+- **Cache key construction** — a single helper per cache class. `tenant_key(namespace, *parts)` for tenant-scoped caches (calls `require_tenant()`); `global_key(namespace, *parts)` for global immutable caches; `aggregate_key(namespace, *parts)` paired with a `require_cross_tenant()` check for cross-tenant aggregates. Lint enforcement via a ruff custom rule or a CI grep gate forbidding raw `await redis.get(key)` outside these helpers for tenant-scoped caches.
+- **Starlette / FastAPI middleware** — a `TenantMiddleware` extracts tenant from the verified JWT/session, validates against the tenant directory, sets `tenant_id_var`, and yields to the next app. Missing tenant short-circuits with a typed error response before the route runs.
+- **Async message consumer** — every message header carries `tenant_id`; the consumer's first action after decode is `tenant_id_var.set(msg.headers["tenant_id"])`. A message without a tenant id goes to DLQ with `missing_tenant` disposition unless the message type is explicitly cross-tenant.
+- **Outbox rows carry tenant** — the `outbox` table has `tenant_id` (UUID, NOT NULL); the poller publishes with the tenant id in the broker header. Downstream consumers re-enter their own tenant context from the header. The claim-commit-publish-mark-sent loop (see `event-driven-architecture.md`) reads tenant from each row.
+- **Quota enforcement** — per-tenant token bucket in Redis keyed by `tenant:<id>:bucket:<resource>`; an async helper `await acquire_token(tenant_id, resource) -> bool` is called at the entry of each rate-limited operation. For per-process limits use `asyncio.Semaphore` indexed by tenant; for cross-process use Redis-backed bucket via a `redis-py` Lua script. For shared accelerators (LLM inference, GPU), the per-tenant bucket feeds a fair scheduler that interleaves tenants without partitioning the batch.
+- **Observability** — `structlog` processor that pulls `tenant_id_var.get()` and attaches `tenant_id` to every log record; OpenTelemetry baggage (`baggage.set_baggage("tenant.id", str(tenant_id))`) propagates across spans; emit Prometheus metrics with `tenant_id` only on headline RED metrics for named cohorts; tag tenant tier for long tail; heavy-hitter detection runs in a sidecar that promotes a long-tail tenant to per-id observation when its traffic crosses a threshold. Wire OTel startup in `observability-and-ops.md`.
+- **Cross-tenant capability** — a typed `CrossTenantCap` Pydantic model attached to the request via a dependency, with a mode field (`per_call` / `per_session` / `break_glass`); a helper `audit_cross_tenant(tenants: list[UUID], operation: str, mode: AuditMode)` writes an audit row per call (per-call mode) or starts a session record (per-session mode); break-glass mode also queues a post-hoc review task.
+- **Tenant directory client with revision-aware refresh + atomic-snapshot semantics** — `httpx.AsyncClient` over the tenant directory service. A monotonic `generation` alone is not enough when the directory's backing store is sharded or eventually consistent — a global revision can advance while one tenant's update has not yet reached the replica the directory serves from. The directory must expose either (a) a true atomic-snapshot read with a watermark covering all tenants, or (b) per-tenant / per-shard versions plus a checksum the client can use to detect partial snapshots. The client caches the last-known-good snapshot tagged with the generation + completeness proof; the background refresher polls and replaces the snapshot only when the new snapshot is provably complete (atomic watermark, or all shard versions ≥ the previous snapshot's). On a partial-snapshot signal, keep the old snapshot, alert, and retry; do not replace. Readiness fails if the snapshot is older than the configured staleness budget; the client returns `unknown_tenant` (fail-closed) when the directory itself is unreachable past readiness budget. Tier downgrades propagate within one refresh interval; alert when refresh duration exceeds the budget. `async-lru` is acceptable for the in-memory snapshot; the refresh task is the authoritative path, not the TTL eviction.
+- **Test substitution** — define a `TenantDirectory` Protocol; provide an in-memory implementation for pytest with a few synthetic tenants; integration tests run against a real directory via `testcontainers-python` or a dev instance.
+- **Graceful tenant offboarding** — the deletion workflow is a state machine (see `background-jobs-and-scheduling.md` or your workflow engine); each per-store step has an async handler that records `pending` / `done` / `unknown`; each step is tagged with one of the per-store deletion modes (hard-delete-verified / anonymized / beyond-use / retained-under-legal-basis / unverifiable-append); the workflow's `unknown` path triggers a reconciliation job and alerts. Schema-per-tenant or DB-per-tenant deletion paths drop the tenant's schema/DB after data export verification; the drop is itself audited.
+
+### Mirrored-section grep gate
+
+The sibling-sync header forbids three categories of stack-specific token in mirrored sections (everything from "When this applies" through "Operations checklist"; everything *before* the `## Python-specific implementation patterns` H2). Run this grep against the mirrored region before every commit; zero hits required (except inside fenced code blocks, which mirrored sections do not use).
+
+Forbidden tokens for this Python file's mirrored sections:
+
+- **DB-engine syntax** — `SET LOCAL`, `RESET ROLE`, `RESET app\.tenant_id`, `set_config\(`, `current_setting\(`, `pg_try_advisory`, `pg_stat_activity`, `BYPASSRLS`, `FORCE ROW LEVEL SECURITY`, `search_path`, `CREATE POLICY`, `ENABLE ROW LEVEL SECURITY`, `GET_LOCK\(`.
+- **Runtime / concurrency mechanic names** — `context\.Context`, `\bgoroutine\b`, `\bgoroutines\b`, `ctx\.Done`, `ResetSession`, `database/sql`, `\bsqlx\b`, `contextvars`, `\basyncio\b`, `run_in_executor`, `to_thread`, `ThreadPoolExecutor`, `ProcessPoolExecutor`, `copy_context`, `async with`, `executor/thread-pool/process-pool`.
+- **Library / framework API names** — `GORM`, `\bsqlx\b`, `Hertz`, `Kitex`, `golang\.org/x/time/rate`, `SQLAlchemy`, `FastAPI`, `Starlette`, `Pydantic`, `httpx`, `aiokafka`, `pika`, `aio-pika`, `redis-py`, `tenacity`, `structlog`, `testcontainers`, `Alembic`, `async-lru`, `asyncpg`, `psycopg`, `cache\.Get`.
+
+Run the command below. The regex is **generated from** the token list above and must match it exactly — token list changes require regenerating the regex. Word boundaries (`\b…\b`) preserve the declared semantics (`asyncio` as a word, not as a substring of `asyncio.create_task` in a routing-reference text).
+
+```
+awk '/^## Python-specific implementation patterns/{exit} 1' multi-tenant-isolation.md \
+  | grep -nE '(SET LOCAL|RESET ROLE|RESET app\.tenant_id|set_config\(|current_setting\(|pg_try_advisory|pg_stat_activity|BYPASSRLS|FORCE ROW LEVEL SECURITY|search_path|CREATE POLICY|ENABLE ROW LEVEL SECURITY|GET_LOCK\(|context\.Context|\bgoroutine\b|\bgoroutines\b|ctx\.Done|ResetSession|database/sql|\bsqlx\b|contextvars|\basyncio\b|run_in_executor|to_thread|ThreadPoolExecutor|ProcessPoolExecutor|copy_context|async with|executor/thread-pool/process-pool|GORM|Hertz|Kitex|SQLAlchemy|FastAPI|Starlette|Pydantic|httpx|aiokafka|\bpika\b|aio-pika|redis-py|tenacity|structlog|testcontainers|Alembic|async-lru|asyncpg|psycopg|cache\.Get)'
+```
+
+Allowed exception: the *Sibling sync* header itself names the three category classes (without tokens) and references this gate; the *Sanitization boundary* header does not contain any of these tokens. The sibling Go file maintains the matching gate for Go-side tokens (`context.Context`, `goroutine`, `GORM`, `database/sql`, etc.), so a token forbidden here may legitimately appear in the *Go-specific implementation patterns* section of the sibling, and vice versa.
+
+## Topic-extension backlog
+
+Two multi-tenant isolation areas are known to be wider than the current rules close. Extend each via **incident-driven evidence**, not a generic re-review pass — a generic round only surfaces them at wording level without resolving them. Each names the general rule already in place + the residual gap + the incident evidence that would unblock the next extension.
+
+- **External / runtime cache dependency change-signal contracts.** Rule in place: each global cache declares its dependency domains including external / runtime sources; the gate uses webhook-backed drift triggers, scheduled attestation, or default-downgrade-to-tenant-scoped. Residual: the **specific change-signal contract** between this codebase and each external dependency is per-integration and cannot be generalized. Extend when: a real incident occurs where a remote dependency schema shifted and a global cache silently broke. Evidence to bring: the specific external provider, its change-event surface (or lack of one), the cache that broke, the drift detection that should have caught it.
+- **Catalog gate coverage for non-relational tenant stores.** Rule in place: the schema-CI gate enumerates every relation (table, view, materialized view, partition child, runtime-created, extension-created) and fails on unclassified ones. Residual: the rule is DB-relation-only. Non-relational tenant stores — including but not limited to message-queue topics that fan out per-tenant, object-store paths with per-tenant prefixes, search indexes, vector embeddings stores, document DB collections (MongoDB, DocumentDB), graph DB labels/edges, time-series and metrics stores (Prometheus / Mimir / VictoriaMetrics tenant series), ML feature stores, primary key-value stores (Redis / DynamoDB used as system-of-record, not as cache), secrets and config stores (Vault, KMS, parameter stores), CDN origins, analytics-warehouse extracts, and cache namespaces — each have their own catalog-and-classification gap that the DB-CI gate does not see. Treat the list as open-ended; any per-tenant store class outside the DB-relations boundary needs its own enumeration mechanism. Extend when: a real incident occurs where a non-DB store accumulated unclassified tenant-scoped content (a search index with tenant rows that bypassed the deletion workflow, an object-store path with tenant data not in the deletion serving-suppression denylist). Evidence to bring: the non-DB store class, its enumeration mechanism (or lack of one), the tenant-classification gap, and the gate that should have caught it.
+
+A related open concern: **workflow-execution-identifier forgery / replay** in crypto-deletion chain-of-custody — use a signed nonce or content-addressed event id rather than a bare run id; the specific signing primitive depends on the workflow engine in use, so extend when the engine surfaces a real replay or relabeling case.

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/observability-and-ops.md

@@ -0,0 +1,26 @@
+# Observability And Ops
+
+Use this for logs, metrics, traces, health checks, readiness, and operational controls.
+
+## Required Surfaces
+
+- Structured logs with request ID, trace ID, user/tenant/resource scope where safe, route/job name, dependency target, and outcome.
+- Metrics for request latency, error rate, dependency latency, queue depth, retry/drop counts, cache hit/miss, and job duration.
+- Traces for inbound requests, outbound HTTP, DB calls, queue work, and inference calls where feasible.
+- Health checks distinguish process liveness from dependency readiness.
+- Debug endpoints and profilers must be disabled or protected in production.
+
+## Python Notes
+
+- Use OpenTelemetry or framework-supported instrumentation when available.
+- Make logging configuration deterministic at startup; avoid libraries configuring global logging unexpectedly.
+- Use JSON logs for service environments where log aggregation expects structured fields.
+
+## Cross-Stack Conventions
+
+When the portfolio contains Go services and Python services that share dashboards and alerts, the architecture layer fixes a common shape:
+
+- Metric naming convention: `{category}_{operation}_{suffix}` snake_case, identical across stacks. Base label set for RPC/HTTP metrics is platform-fixed (`caller`, `caller_cluster`, `caller_env`, `caller_method`, `callee`, `callee_cluster`, `callee_env`, `method`, `err_code`); high-cardinality identifiers are forbidden as labels in either stack.
+- Logger trace-linkage contract: every service-internal logger accepts `ctx` (Go) or carries `contextvars` (Python), automatically emits the same four trace-identity fields, and adds `span.AddEvent` / `span.add_event` on warn/error. The wire shape is identical; only the call site differs. A shared foundation framework may propagate this cross-cutting context, but it does not own the application's logger API surface or codegen output meant for wire consumers; keep that ownership boundary explicit.
+- OTel resource attributes (`service.name`, `service.namespace`, `service.version`, `lane`, `pod.ip`, `pod.name`) are set once per process at startup in both stacks. Per-call attributes belong on spans, not on the provider.
+- Context propagation tiers (global trace/log identity, post-auth API context, inbound header mappings) follow the same three-tier shape as the Go side. Typed accessors hide the bare-string assertion in both stacks.

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/packaging-runtime-readiness.md

@@ -0,0 +1,20 @@
+# Packaging Runtime Readiness
+
+Use this for pyproject, uv/poetry/pip, lockfiles, tooling, containers, and deployment process.
+
+## Packaging
+
+- Prefer `pyproject.toml` as the canonical project metadata and tool config surface.
+- Use lockfiles for deployable services.
+- Separate runtime and dev dependency groups.
+- In monorepos, make workspace packages explicit and avoid import success that depends on developer-local `PYTHONPATH` tricks.
+- Generated clients and shared packages should be versioned or workspace-pinned deliberately.
+- **For new Python service architectures, `uv` (Astral, Rust-based) is the current default packaging tool** — single static binary replacing pip + pip-tools + virtualenv + pipx + most of poetry + pyenv (per Astral docs). Architecture impact beyond "faster install": (a) Cargo-style workspaces map cleanly to monorepo subpackages so workspace pinning is first-class rather than ad-hoc; (b) built-in Python version management means CI / Docker base images no longer need separate `pyenv` / system-Python juggling; (c) `uv.lock` is cross-platform reproducible (single lockfile across macOS / Linux / Windows / arch variants), removing per-platform lockfile drift that plagued poetry; (d) pip-compatible interface (`uv pip`) makes migration low-risk because existing requirements.txt / setup.py / pyproject can be consumed without restructuring. For existing services on poetry / pip-tools / pdm / rye / hatch, plan the uv switch as a dedicated architecture migration (lockfile parity check, CI pipeline flip, Dockerfile rebuild, dev-onboarding doc update in one slice), not as drive-by during feature work.
+
+## Runtime
+
+- Define entrypoint: `uvicorn`/`gunicorn`, framework command, worker command, scheduler command, or CLI.
+- Define process model: worker count, async event loop, thread/process workers, memory limits, graceful shutdown, and health probes.
+- Release readiness includes migrations, startup validation, smoke tests, canary, rollback, and observability checks.
+- **ASGI server choice has expanded beyond `uvicorn` / `gunicorn+uvicorn` / `hypercorn`** — `granian` (emmett-framework, Rust-based) is the current credible high-throughput alternative for ASGI services, supporting ASGI/3, RSGI, WSGI, HTTP/1, HTTP/2, TLS, WebSockets (HTTP/3 planned per the project README's "eventually 3" roadmap note — verified not shipped as of May 2026). Per the Granian project's own `benchmarks/vs.md` and third-party load-test repos (e.g., `piccolo-orm/asgi_server_performance`, `synodriver/asgi-server-benchmark`), ASGI echo on 10KB payload typically lands granian > uvicorn-httptools > hypercorn by roughly the ratios 58k / 51k / 8k RPS in April-2026-era runs; file-serving gap is wider (granian ~47k vs uvicorn ~18k via `pathsend`). Treat the absolute numbers as benchmark-snapshot-specific; re-run against your workload before basing a switch on them. Architecture impact: when serving throughput is the binding constraint, granian can buy headroom without rewriting the **plain ASGI path**. **"Without rewriting" caveats**: granian's worker / process model differs from `gunicorn+uvicorn` fork-based workers (Rust runtime + Python interpreters with different lifecycle hooks); ASGI lifespan events, contextvars propagation across worker boundaries, custom signal handlers, prometheus/metrics exporters tied to uvicorn internals, and ASGI middleware that depends on uvicorn-specific behavior all need smoke-testing on granian before a switch. If the team plans to adopt granian's bespoke RSGI protocol for max performance (rather than ASGI), application code that uses ASGI-specific middleware, ASGI scope manipulation, or third-party ASGI libraries WILL need rewriting — RSGI is a different protocol, not a faster ASGI. Trade-offs: smaller operational maturity, fewer community recipes, Rust-runtime-on-the-side observability differs from a pure-Python server. **Choose uvicorn** for ecosystem maturity, broad reference material, and known operational patterns; **choose granian** when (a) profiled benchmarks on your workload show uvicorn saturation, (b) the team has Rust-toolchain debugging capacity, (c) the deployment story can absorb a less-common runtime. Hypercorn remains the choice when HTTP/2 + ASGI under pure-Python ops matters more than peak throughput.
+- **Python runtime version baseline (2025-2026)**: Python 3.13 (released October 2024) ships **experimental** free-threaded build per PEP 703 — GIL-disabled, ~40% single-threaded perf hit per python.org "What's New in 3.13" notes, used at the team's risk for parallel-CPU workloads. Python 3.14 (released 7 October 2025) advances free-threading to **supported (Phase II of PEP 703)** per PEP 779 — meaning the free-threaded build is a first-class supported configuration, NOT that it is the default Python build or the default production choice. Per the python.org free-threading howto, the single-threaded penalty narrowed to ~5-10% (specializing adaptive interpreter re-enabled thread-safely); PEP 803 defines the `abi3t` stable ABI for free-threaded C extensions. Architecture impact: for services where parallel CPU work matters (in-process ML inference fan-out, heavy parsing, parallel compression), 3.14 free-threading is the first version where adoption is **a defensible experiment for a selected service**, not yet a defensible default. Pre-flight burn-in required before any production switch: (a) C-extension readiness — all extensions in the service's dependency tree must declare free-threading support; many popular extensions (numpy, pandas, lxml, psycopg native bits, asyncpg native bits, pillow, cryptography) were still mid-migration at 2026-Q1, verify per-version; mixing GIL-only and free-threading-aware extensions in one process is unsupported; (b) GC and runtime behavior under sustained threading load differs from GIL build — measure tail latency, memory residency, and CPU efficiency on the actual workload; (c) debugger / profiler ergonomics (gdb, pdb, py-spy, scalene, prometheus exporters) may have rough edges on free-threaded builds; (d) library-level thread-safety: code paths that were "implicitly safe because of the GIL" can race in free-threaded mode (singletons built at import time, module-level mutable caches, third-party libraries that rely on GIL-protected dict mutation). For pure I/O-bound services, stay on stock GIL build — the 5-10% overhead is pure cost. For services on 3.13 or older, treat free-threading as opt-in research, not default.

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/redis-cache-coordination.md

@@ -0,0 +1,41 @@
+# Redis Cache Coordination
+
+Use this for Redis cache, locks, rate limits, counters, idempotency windows, and ephemeral coordination.
+
+## Rules
+
+- Redis is not durable source of truth for auditable state.
+- Centralize key names, TTLs, namespaces, lock leases, and rate-limit windows.
+- Use unique lock values and compare-and-delete unlock semantics.
+- TTL policy is part of architecture, not an implementation afterthought.
+- Cache invalidation must have an owner: write-through, event-driven invalidation, explicit delete, or time-bounded staleness.
+- If Redis is used for idempotency, separate pending claim, completed marker, duplicate, and retryable-failure states. Pending claims need a lease/TTL and completed markers need a dedupe window that matches the external retry horizon.
+- Redis Streams, Pub/Sub, lists, and sorted sets are not interchangeable. Name the delivery guarantee: fire-and-forget notification, replayable stream, deduplicated pending work, delayed scheduling, or durable queue backed by DB/MQ.
+- Broad pattern deletion and stream cleanup need a bounded strategy. Architecture should state whether cleanup is batched, TTL-driven, event-driven, or an admin/backfill operation.
+- Redis failure policy must be per responsibility: fail open for optional caches, fail closed for authorization/risk controls, return duplicate/pending states for idempotency, and expose repair/retry for workflow coordination.
+
+## Python Notes
+
+- Choose sync or async Redis clients according to the service stack.
+- Do not use a sync Redis client directly in async endpoints unless isolated.
+- Connection pools, timeouts, and retries should be configured once through dependency assembly.
+
+## Cache Stampede Architecture
+
+- For any cache fronting an expensive source of truth, architecture names the stampede defense: single-flight (one fetcher per key), negative caching with shorter TTL for misses, and randomized TTL jitter. Architecture also names the staleness budget so the team can pick the right combination.
+- Multi-tier caches (process-local in front of Redis in front of DB) need a coherent invalidation contract: Pub/Sub fan-out, `CLIENT TRACKING` (RESP3) for short-staleness reads, or explicit delete-on-write. Architecture names the chosen mechanism per cache tier.
+
+## Cluster Topology Boundary
+
+- If Redis is deployed as Cluster (or Cluster-compatible managed service), architecture declares the hash-tag convention and the keys that must share a slot. Cross-slot Lua, `MULTI/EXEC`, and pipeline-with-dependencies are architectural decisions, not coincidence; the key schema is the contract.
+- Cluster failover, slot migration, and topology refresh behavior is owned at the platform layer; this skill names the client's expected behavior (refresh on `MOVED`/`ASK`, retry on topology change, idempotency on the retry path).
+
+## Authentication And Encryption Responsibility
+
+- Production Redis must be authenticated and encrypted. TLS is required for any non-loopback traffic; the auth model is whichever the chosen Redis provider supports — ACL user + password, mTLS client certificates, or short-lived IAM/AAD tokens. Architecture names the trust anchor (managed CA, internal PKI) and the credential lifetime, and treats the provider's strongest supported option as the default.
+- Redis ACL users follow least-privilege: per-service users with explicit command lists. Cluster-wide admin commands (`FLUSHDB`, `CONFIG`, `DEBUG`) are not part of application user grants.
+
+## Retry And Circuit Breaker Ownership
+
+- Retry policy and circuit breaker for Redis access live at the dependency-assembly layer, not scattered through call sites. Architecture names the transient-error catalog (timeout, connection reset, `LOADING`, `BUSY`) and the breaker scope (per-endpoint, per-shard).
+- Breaker-open behavior is a product contract: cache-miss path may return stale-OK, authz path must fail closed, idempotency path returns "duplicate uncertain". Architecture names the per-responsibility breaker semantics; do not let the resilience layer pick silently.

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/reliability-and-error-contract.md

@@ -0,0 +1,17 @@
+# Reliability And Error Contract
+
+Use this for timeout, retry, fallback, exception mapping, and canonical errors.
+
+## Error Contract
+
+- Define canonical error shape once.
+- Map Pydantic validation, framework HTTP exceptions, ORM errors, Redis errors, queue errors, HTTP client errors, and inference-client errors into that shape. A pluggable or runtime error classifier's output is untrusted input at the boundary: validate/normalize it and fail closed to a safe generic code; never index the code→status mapping with an unvalidated classifier or external value.
+- Keep public error details safe: no secrets, stack traces, raw provider payloads, or internal IDs unless explicitly allowed. Gate every code through an allow-list separating client-safe from internal-only codes before writing it to a client-facing response body.
+
+## Reliability
+
+- Every external call needs timeout, retry/fallback policy, and observability.
+- Retries require idempotency or a clear reason the operation is safe.
+- Circuit breakers, rate limits, and backpressure belong to architecture when a dependency can overload.
+- Fail closed for auth, permissions, money, irreversible actions, and data-integrity checks.
+- Use graceful degradation only for non-critical cache, analytics, recommendations, or telemetry paths.

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/source-evidence-map.md

@@ -0,0 +1,55 @@
+# Python Architecture Source Evidence Map (Methodology Template)
+
+This file describes the **shape** of source-evidence tracking for Python architecture extraction. Project-specific provenance (real repo names, branch state, dates, module counts, extraction logs) lives in the maintainer's private alias file at `~/.<host>/.private-aliases/<project>.yaml`, never in this shared skill tree — see `skill-extraction-workflow` Core Rule "Extraction lifecycle handoff".
+
+## Scope boundary
+
+This shared template contains reusable methodology only. Do not add concrete repository names, branch state, module counts, package descriptors, or extraction logs.
+
+## Source coverage dimensions (architecture-level)
+
+When extracting Python architecture guidance from a new source set, organize evidence by these dimensions:
+
+| Dimension | What to collect | What to extract at architecture layer |
+| --- | --- | --- |
+| Service boundary inventory | How services are split (by domain / by traffic class / by data ownership); inter-service contract count | Boundary criteria, when to split vs merge, ownership rules |
+| API contract shape | Pydantic / OpenAPI / framework schemas in use, public-vs-internal API split, contract evolution practice | Contract-first discipline, additive evolution rule, public/internal split when stability needs differ |
+| Data ownership | Per-service write model, cross-service read patterns, migration discipline (Alembic / framework migrations) | Each service owns its write model; cross-service reads through API; relational source-of-truth |
+| Runtime platform | Service discovery integration, framework choice (FastAPI / Flask / Django / equivalent), dynamic config, secret management, observability, mesh integration | Workload identity, registry-based vs DNS-based SD, mesh ownership boundary, dependency client contracts |
+| Async / sync model | asyncio call-chain hygiene, blocking work isolation (thread pool / `asyncio.to_thread` / process pool), framework async support | Async only when call chain and libraries are truly non-blocking; isolate blocking I/O / CPU / GPU explicitly |
+| Worker / job design | Celery / RQ / arq / framework workers, schedule, retry, idempotency, backfill | Worker design as first-class architecture; idempotency key, lease, max execution |
+| Error / context contract | Error code enum, exception → code map, ctx propagation through async paths | Stable enum, framework exception mapping, contextvars-as-source-of-truth for correlation |
+| Reliability surfaces | Rate limit, backpressure, circuit breaker, idempotency, durable status | Reliability is architecture, not bolt-on |
+| AI / LLM service hosting | Inference adapter shape, streaming protocol, replay, token cost, fallback | Service-side boundary only; inference design routes to `llm-inference-integration` |
+| Generated code ownership | OpenAPI client gen, protobuf gen, ORM autogenerate, ownership boundary | Generated code is output; generation commands documented and portable |
+| Packaging | Dependency manager (uv / poetry / pip), lockfile, container image, process model | Reproducible builds, pinned dependencies, documented process model |
+| Release runtime | Deployment unit, mesh routing, canary, approval, rollback | Architecture surface, not afterthought |
+
+## Sibling-generalization mini-map (template)
+
+For each Python architecture extraction, record decisions across sibling skills:
+
+- `python-service-dev` — does the architectural rule have an implementation pattern? note in dev reference.
+- `go-microservice-architecture` — cross-language counterpart? mirror at architecture level.
+- `go-microservice-dev` — implementation counterpart?
+- `llm-inference-integration` — does the AI/inference-adjacent rule belong there instead?
+- `testing-strategy` — what test layer proves the architectural property?
+- `platform-observability` / `platform-service-connectivity` / `platform-release-engineering` — does the rule belong at platform layer?
+
+Decisions go in the private alias map alongside the per-batch extraction log.
+
+## Coverage discipline rules
+
+- Architecture extractions describe **decision criteria, invariants, ownership boundaries, and acceptance checks**. Implementation patterns go to `python-service-dev`.
+- Mark each dimension `pending` / `read` / `deep-read` / `excluded` / `unavailable` / `routed` with the actual artifacts inspected, in the private alias map.
+- Re-read source artifacts when changing rules; use `wording cleanup` / `no new source read` labels honestly.
+
+## Where the live provenance lives
+
+For the maintainer's current Python extraction project, see:
+- Private alias map: `~/.<host>/.private-aliases/<project>.yaml`
+- Per-batch extraction logs: `~/.<host>/skills/.extraction-work/<batch>-completion.md`
+
+## Audit gate
+
+Before committing a change to this skill, run the private alias map's `audit_cmd` against the changed shared-skill files. Any hit on real repo names, branch states, dates, file counts, or business-domain nouns is either fixed or recorded as `known_debt` (pre-existing hits only — new/modified content must stay zero-hit per the R0 rule).

package/dist/assets/marketplace/plugins/ccl-skills/skills/python-service-architecture/references/web-framework-boundaries.md

@@ -0,0 +1,26 @@
+# Web Framework Boundaries
+
+Use this when choosing or reviewing FastAPI, Flask, Django, Starlette, ASGI, or WSGI boundaries.
+
+## Framework Choice
+
+- Prefer FastAPI for typed HTTP APIs, OpenAPI-first contracts, async-compatible endpoints, dependency injection, and service-style applications.
+- Prefer Django when the product needs admin, ORM conventions, forms, templating, user/session features, and a cohesive framework.
+- Prefer Flask for small synchronous tools, simple internal UIs, or minimal APIs where explicit wiring is more valuable than framework breadth.
+- Use Starlette-level primitives only when the service needs lower-level ASGI control.
+- **Consider Litestar (formerly Starlite) as a structured alternative to FastAPI** when the architecture wants framework-owned ORM integration, server-side sessions, caching, plugin system, and OpenTelemetry built in rather than assembled per-project; per Litestar docs the framework does more work at bootstrap (precomputed per-route requirements) so runtime cost is lower, and dependency injection is a dict-based `Provide(...)` model that allows transparent override at app/router/controller/route level. The architecture-level trade-off is: more framework convention vs more ecosystem maturity. Choose Litestar when the team wants framework opinions to outpace ad-hoc choices; stay on FastAPI when ecosystem maturity (third-party integrations, hiring pool, community examples) outweighs the structural delta. **Validate production-adoption risks before committing**: Litestar's plugin / middleware ecosystem is smaller than FastAPI's (which inherits much of Starlette's middleware library); the most common production-critical pieces — observability/OTel exporter (Litestar has first-party support, but the specific exporter version + auth-header forwarding behavior may differ), auth middleware (Litestar plugins exist but parity with FastAPI-specific extensions is per-feature), deployment-runtime quirks (lifespan event ordering, ASGI middleware chaining, custom serializer hooks) — should each be smoke-tested on a real Litestar app before the team commits. Litestar does NOT extend Starlette, so third-party "Starlette middleware" packages may need a Litestar-native equivalent or wrapper. Either choice is a long-term commitment — framework switches mid-product are expensive and rarely worth it. Implementation-level Litestar specifics live in `python-service-dev/references/web-framework-patterns.md`.
+
+## Boundaries
+
+- Keep routers/views thin. They should bind request data, authenticate/authorize, call application services, and map responses/errors.
+- Put business decisions and data transactions outside route handlers.
+- Use app factories or explicit `create_app()` functions when middleware, clients, settings, and test substitution matter.
+- Use framework dependency mechanisms, but do not bury heavy construction, network I/O, or mutable global state inside dependency functions.
+- Treat ASGI/WSGI choice as runtime architecture: worker count, event loop, sync adapters, lifespan hooks, and shutdown cleanup all matter.
+
+## Anti-Patterns
+
+- Mixing sync ORM calls inside async endpoints without isolation.
+- Starting clients, schedulers, or background loops at import time.
+- Returning raw ORM objects or unvalidated dictionaries as public API contracts.
+- Letting framework folder conventions define domain ownership without an explicit boundary.