hatch3r 1.7.0 → 1.7.5

package/rules/hatch3r-i18n.md

@@ -93,3 +93,16 @@ ICU MessageFormat 2.0 reached Final Candidate status in CLDR 46.1 (January 2025)
 - **Migration strategy:** New translation keys should use MF2 syntax. Existing MF1 keys can be migrated incrementally — both syntaxes can coexist during transition.
 - **Tooling:** Verify that your translation management system (TMS) supports MF2 syntax before migrating. Test with a small key set first.
 - **Stability:** The MF2 specification has stability guarantees post-approval (mid-2025). Syntax and semantics will not change incompatibly after that point.
+
+## Microcopy and Tone
+
+Translation strings are user-facing copy — write them as product copy, not as technical labels.
+
+- Use plain language. Default to second person ("you", "your") for end-user surfaces.
+- Use a corrective verb in error messages: "Try again", "Reconnect", "Enter a valid email" — not "Error" or "Oops".
+- Never expose to end users: protocol acronyms ("FIDO2", "WebAuthn"), raw HTTP status codes ("500", "401"), language sentinel values (`null`, `undefined`), or internal record/ID strings. Translate these into a user-visible cause + recovery.
+- CTA labels are action-oriented and specific: "Save changes" beats "Submit"; "Delete project" beats "Confirm"; "Send invite" beats "OK".
+- Error tone explains the cause and offers a recovery path. Do not blame the user. Replace "You entered an invalid value" with "This field needs a valid email address — for example, name@example.com".
+- Use ICU MessageFormat (1.0 or 2.0 per the MF2 section above) for every plural, gender, and select pattern. Never concatenate translated fragments to build a sentence — each complete sentence is a single translation key with its own placeholders.
+- Tone source-of-truth: the GOV.UK Design System content style guide (https://design-system.service.gov.uk/styles/) and IBM Carbon Design System voice and tone guidance (https://carbondesignsystem.com/guidelines/content/general/) — cite both when reviewing tone or training a translator.
+- Cross-reference `rules/hatch3r-ux-states-and-flows.md` Microcopy subsection for the state-driven copy patterns (loading, empty, error, partial) that share this tone contract.

package/rules/hatch3r-i18n.mdc

@@ -88,3 +88,16 @@ ICU MessageFormat 2.0 reached Final Candidate status in CLDR 46.1 (January 2025)
 - **Migration strategy:** New translation keys should use MF2 syntax. Existing MF1 keys can be migrated incrementally — both syntaxes can coexist during transition.
 - **Tooling:** Verify that your translation management system (TMS) supports MF2 syntax before migrating. Test with a small key set first.
 - **Stability:** The MF2 specification has stability guarantees post-approval (mid-2025). Syntax and semantics will not change incompatibly after that point.
+
+## Microcopy and Tone
+
+Translation strings are user-facing copy — write them as product copy, not as technical labels.
+
+- Use plain language. Default to second person ("you", "your") for end-user surfaces.
+- Use a corrective verb in error messages: "Try again", "Reconnect", "Enter a valid email" — not "Error" or "Oops".
+- Never expose to end users: protocol acronyms ("FIDO2", "WebAuthn"), raw HTTP status codes ("500", "401"), language sentinel values (`null`, `undefined`), or internal record/ID strings. Translate these into a user-visible cause + recovery.
+- CTA labels are action-oriented and specific: "Save changes" beats "Submit"; "Delete project" beats "Confirm"; "Send invite" beats "OK".
+- Error tone explains the cause and offers a recovery path. Do not blame the user. Replace "You entered an invalid value" with "This field needs a valid email address — for example, name@example.com".
+- Use ICU MessageFormat (1.0 or 2.0 per the MF2 section above) for every plural, gender, and select pattern. Never concatenate translated fragments to build a sentence — each complete sentence is a single translation key with its own placeholders.
+- Tone source-of-truth: the GOV.UK Design System content style guide (https://design-system.service.gov.uk/styles/) and IBM Carbon Design System voice and tone guidance (https://carbondesignsystem.com/guidelines/content/general/) — cite both when reviewing tone or training a translator.
+- Cross-reference `rules/hatch3r-ux-states-and-flows.md` Microcopy subsection for the state-driven copy patterns (loading, empty, error, partial) that share this tone contract.

package/rules/hatch3r-iteration-summary.md

@@ -16,6 +16,8 @@ Every iteration with the user ends with the canonical block defined below — no
 
 Every user-facing iteration, regardless of size — multi-step coding tasks, single-file edits, read-only answers, failed or blocked attempts. No exceptions.
 
+The per-turn pipeline-state header (defined in `hatch3r-agent-orchestration` → Per-Turn Pipeline-State Header) is a separate start-of-turn artifact and does not replace this end-of-turn block.
+
 ## The Required Block
 
 Use this exact shape with these exact field names:

package/rules/hatch3r-iteration-summary.mdc

@@ -11,6 +11,8 @@ Every iteration with the user ends with the canonical block defined below — no
 
 Every user-facing iteration, regardless of size — multi-step coding tasks, single-file edits, read-only answers, failed or blocked attempts. No exceptions.
 
+The per-turn pipeline-state header (defined in `hatch3r-agent-orchestration` → Per-Turn Pipeline-State Header) is a separate start-of-turn artifact and does not replace this end-of-turn block.
+
 ## The Required Block
 
 Use this exact shape with these exact field names:

package/rules/hatch3r-migrations.md

@@ -1,7 +1,7 @@
 ---
 id: hatch3r-migrations
 type: rule
-description: Database migration and schema change patterns for the project
+description: Database migration and schema change patterns — expand-contract, online DDL, backfills, compatibility windows, reversibility, multi-region, tooling
 scope: "**/migrations/**,**/*migration*,**/migrate/**,**/seeds/**,**/seeders/**,**/prisma/migrations/**,**/drizzle/**,**/knex/**"
 tags: [implementation, brownfield]
 quality_charter: agents/shared/quality-charter.md
@@ -9,23 +9,68 @@ cache_friendly: true
 ---
 # Migrations
 
-- Schema changes must be backward-compatible. Add fields with defaults; never remove or rename without migration.
 - Migration scripts live in a dedicated `migrations/` directory. One script per migration.
-- Every migration is idempotent. Safe to re-run. Use document version or `migratedAt` to skip.
-- Migration functions log progress and are resumable. Handle partial failures.
-- Test migrations against emulator or staging before deploying. Verify data integrity.
-- Order: deploy new code (handles old + new schema) → run migration → remove old schema handling.
-- Document schema changes in project data model spec.
-- Rollback plan required for every migration. Never run destructive migrations without backup verification.
-- Hot documents must stay within size limits after migration.
+- Every migration is idempotent (re-running produces the same result). Use a version column, `migratedAt` timestamp, or migration ledger row to skip already-applied work.
+- Test every migration against an emulator or staging dataset before production. Verify data integrity after each step, not just at the end.
+- Document the schema change in the project data model spec. Hot documents must stay within size limits after migration.
 
-## Data Validation During Migration
+## Expand-Contract Pattern (mandatory for non-trivial schema changes)
 
-- Validate data integrity after each migration step, not just at the end. Check that migrated records match the expected schema, required fields are populated, and no data was silently dropped.
-- Include count checks: the number of records processed should match the number of records in the source collection. Log discrepancies as errors, not warnings.
-- For large datasets, migrate in batches with progress checkpoints. If a batch fails, resume from the last checkpoint rather than restarting the entire migration.
+Non-trivial = anything beyond pure-additive nullable columns on small tables, or any rename/drop/type-change. Use a 3-deploy cadence; split Migrate into two deploys when dual-write is required (4 deploys total).
 
-## Migration Coordination in Multi-Service Environments
+1. **Deploy 1 — Expand.** Add new column nullable, add new table, or `CREATE INDEX CONCURRENTLY`. Add new constraints with `NOT VALID` first. Old code paths still work. No app behavior change in this deploy.
+2. **Deploy 2 — Migrate (backfill + dual-write).** Run a batched, idempotent, resumable backfill job. If the change is a column rename / type swap, app code writes to both old and new columns during this phase. Validate row counts and per-block checksums on the new shape before proceeding.
+3. **Deploy 3 — Contract.** Switch reads to the new shape (feature-flag-gated; flip is the rollback). Drop the old column, old table, or old index. Wait at least one full release cycle plus one on-call rotation between Expand and Contract — old code must remain executable to roll back inside the deploy window.
 
-- When a migration affects shared data (e.g., a schema used by multiple services), coordinate the migration order across services. The consuming services must be deployed with backward-compatible readers before the migration runs.
-- Never assume that all service instances will be running the same code version during a migration window. Design migrations to tolerate mixed-version reads and writes during the rollout period.
+Hard rules: never rename a column in a single step; never add a `NOT NULL` column to a populated table without a default or a deferred `SET NOT NULL NOT VALID` → `VALIDATE`; every phase must be valid in isolation so that any deploy is independently rollbackable.
+
+## Online Schema Changes
+
+Set `lock_timeout` and `statement_timeout` before every DDL statement to bound blast radius. Selection by engine:
+
+- **Postgres 18.x.** Use `CREATE INDEX CONCURRENTLY` (outside any transaction block — disable the migration tool's transaction wrapper). On failure, the index is left `INVALID`; emit a `DROP INDEX IF EXISTS` + retry step. For FK and CHECK constraints, use `ALTER TABLE ... ADD CONSTRAINT ... NOT VALID` followed later by `VALIDATE CONSTRAINT` (skips full scan, downgrades to `SHARE UPDATE EXCLUSIVE`). Postgres 18 also supports `SET NOT NULL NOT VALID` for column nullability. Use `pg_repack` 1.5.x for bloat removal instead of `VACUUM FULL`. Avoid `ALTER TABLE ... ADD COLUMN ... DEFAULT non_constant_expression` on large tables — it rewrites every row.
+- **MySQL 8.4 LTS.** `ALGORITHM=INSTANT` is the default for many metadata ops (ADD COLUMN at end, RENAME COLUMN, some index meta) — verify against the 8.4 online DDL operations matrix. Hard limit: 64 row versions per table in 8.4. When `INSTANT` is rejected, fall back to `ALGORITHM=INPLACE`. For `ALGORITHM=COPY` operations on large tables, use `gh-ost` v1.1.8 (trigger-free, binlog-based, checkpoint + resume + revert) when the table has no incoming FKs and the cluster is not Galera / Percona XtraDB. Use `pt-online-schema-change` when FKs are present (`--alter-foreign-keys-method`) or under Galera. `lhm` is unmaintained — do not propose it for new code.
+
+## Backfill Jobs
+
+Every backfill must be batched, idempotent, resumable, throttled, and observable.
+
+- **Batched.** Order by PK or a monotonic key. Chunk by `id BETWEEN ? AND ?` (range), not `LIMIT/OFFSET` — offsets drift under concurrent writes. Default chunk 1k–10k rows; tune by table width.
+- **Idempotent.** Write `UPDATE ... SET new = f(old) WHERE id = ? AND new IS NULL` (or upsert with a deterministic source-derived value). Re-running on the same range must produce the same final state.
+- **Resumable.** Persist the last-processed boundary (`last_id` or timestamp cursor) to a control table after each batch commit. Resume from the checkpoint on restart; never restart from zero on partial failure.
+- **Throttled.** Poll replication lag (`pg_stat_replication`, `SHOW REPLICA STATUS`) between batches; pause when lag exceeds 30 seconds or the SLO threshold. Cap concurrency at the IO budget of the slowest replica.
+- **Observable.** Emit `migration.backfill.rows_processed` (counter), `migration.backfill.error_rate` (counter), `migration.backfill.eta_seconds` (gauge), and `migration.backfill.current_boundary` (gauge). Wire dashboards before launch. Avoid single mega-DML — one `UPDATE` over 50M+ rows produces multi-hour locks and table bloat.
+
+## Compatibility Window
+
+Schema changes deploy before the code that depends on them when widening (add column, add table, add index). Schema changes deploy after the code that no longer depends on them when narrowing (drop column, drop table). During the window, app code reads both shapes — the new shape if populated, fall back to the old shape otherwise. Rollback compatibility (old code remains executable against the current schema) must hold for at least 1 full release cycle plus 1 on-call rotation — minimum 7 calendar days, longer when the on-call rotation is longer.
+
+## Reversibility
+
+Every migration ships a tested down-migration script. Forward-only migrations are permitted only when the operation is data-destructive (e.g., a `DROP COLUMN` after Contract) — these require an explicit `IRREVERSIBLE: <reason>` annotation in the migration header and reviewer sign-off. A compensating forward migration that restores the prior shape is acceptable in place of a down-script for tools that lack reversibility (Prisma Migrate, Drizzle Kit — surface the gap to the reviewer). Default for every migration: reversible.
+
+## Data Integrity Verification
+
+Apply layered verification from cheapest to most thorough; stop at the cheapest layer that detects no drift.
+
+1. **Pre-migration backup drill.** Full restore to staging plus a smoke query within 24 hours prior to a destructive migration. "Backup exists" is not verification.
+2. **Row-count parity per chunk.** Source rows processed equals target rows written. Log discrepancies as errors, not warnings.
+3. **Aggregate checks.** SUM, MIN, MAX, COUNT(DISTINCT) on numeric and date columns per partition or batch.
+4. **Per-block checksums.** SHA-256 or MD5 over concatenated key columns for blocks of N rows (e.g., `md5(string_agg(id::text || col::text, ',' ORDER BY id))`).
+5. **Cross-system diff.** Datafold Reconcile, dbt-data-diff, or a hand-rolled sample-then-drill comparison for value-level differences.
+6. **Canary dual-read.** Read both shapes in production for 24–72 hours before cutover; shadow-diff and alert on mismatch.
+7. **Reconciliation control table.** Per-batch row count plus checksum stored alongside the checkpoint; auto-stop the backfill on drift above the configured threshold.
+
+## Multi-Region & Replica Lag
+
+- Pause backfill writes when any replica lag exceeds 30 seconds (or the project's lag SLO, whichever is lower). Resume only after lag returns to baseline for 5 consecutive minutes.
+- Roll migrations across regions sequentially; never alter an active partition during the peak traffic window of any region.
+- FK validation (`VALIDATE CONSTRAINT`) reads the entire dependent table — schedule outside peak read windows on replica-heavy topologies.
+- For Postgres major-version upgrades, use native logical replication (PG17+ preserves slots through `pg_upgrade`); advance sequences manually at cutover — logical replication does not replicate sequences, DDL, or large objects.
+- For ongoing cross-system replication, prefer Debezium (when Kafka is already deployed) or AWS DMS (managed, AWS-native). DMS hard limit: 200 tasks per replication instance — relevant for schema-per-tenant designs.
+
+## Tooling Mandate
+
+Pick one schema-management tool per project and commit the schema declaration to the repo. Greenfield default: Atlas (50+ destructive/locking linters, auto-generated down migrations, GitHub Actions approval policies) or dbmate (plain-SQL portability with first-class `-- migrate:down`). Existing-project default: whatever already ships migrations in the repo. Acceptable tools: Atlas, Prisma Migrate (forward-only — surface to reviewer), Drizzle Kit (forward-only — surface), Flyway 11+, Liquibase 4.27+ Pro, sqitch, Alembic, Knex, dbmate, Bytebase. Run a migration linter in CI — Atlas analyze, `squawk` for raw Postgres SQL — fail the PR on destructive operations without an explicit `IRREVERSIBLE:` annotation.
+
+Cross-references: see `hatch3r-data-classification` (PII / encrypted-column migration requirements), `hatch3r-feature-flags` (read-path switchover gating), `hatch3r-observability-metrics` (backfill progress metrics).

package/rules/hatch3r-migrations.mdc

@@ -1,27 +1,72 @@
 ---
-description: Database migration and schema change patterns for the project
+description: Database migration and schema change patterns — expand-contract, online DDL, backfills, compatibility windows, reversibility, multi-region, tooling
 globs: ["**/migrations/**", "**/*migration*", "**/migrate/**", "**/seeds/**", "**/seeders/**", "**/prisma/migrations/**", "**/drizzle/**", "**/knex/**"]
 alwaysApply: false
 ---
 # Migrations
 
-- Schema changes must be backward-compatible. Add fields with defaults; never remove or rename without migration.
 - Migration scripts live in a dedicated `migrations/` directory. One script per migration.
-- Every migration is idempotent. Safe to re-run. Use document version or `migratedAt` to skip.
-- Migration functions log progress and are resumable. Handle partial failures.
-- Test migrations against emulator or staging before deploying. Verify data integrity.
-- Order: deploy new code (handles old + new schema) → run migration → remove old schema handling.
-- Document schema changes in project data model spec.
-- Rollback plan required for every migration. Never run destructive migrations without backup verification.
-- Hot documents must stay within size limits after migration.
+- Every migration is idempotent (re-running produces the same result). Use a version column, `migratedAt` timestamp, or migration ledger row to skip already-applied work.
+- Test every migration against an emulator or staging dataset before production. Verify data integrity after each step, not just at the end.
+- Document the schema change in the project data model spec. Hot documents must stay within size limits after migration.
 
-## Data Validation During Migration
+## Expand-Contract Pattern (mandatory for non-trivial schema changes)
 
-- Validate data integrity after each migration step, not just at the end. Check that migrated records match the expected schema, required fields are populated, and no data was silently dropped.
-- Include count checks: the number of records processed should match the number of records in the source collection. Log discrepancies as errors, not warnings.
-- For large datasets, migrate in batches with progress checkpoints. If a batch fails, resume from the last checkpoint rather than restarting the entire migration.
+Non-trivial = anything beyond pure-additive nullable columns on small tables, or any rename/drop/type-change. Use a 3-deploy cadence; split Migrate into two deploys when dual-write is required (4 deploys total).
 
-## Migration Coordination in Multi-Service Environments
+1. **Deploy 1 — Expand.** Add new column nullable, add new table, or `CREATE INDEX CONCURRENTLY`. Add new constraints with `NOT VALID` first. Old code paths still work. No app behavior change in this deploy.
+2. **Deploy 2 — Migrate (backfill + dual-write).** Run a batched, idempotent, resumable backfill job. If the change is a column rename / type swap, app code writes to both old and new columns during this phase. Validate row counts and per-block checksums on the new shape before proceeding.
+3. **Deploy 3 — Contract.** Switch reads to the new shape (feature-flag-gated; flip is the rollback). Drop the old column, old table, or old index. Wait at least one full release cycle plus one on-call rotation between Expand and Contract — old code must remain executable to roll back inside the deploy window.
 
-- When a migration affects shared data (e.g., a schema used by multiple services), coordinate the migration order across services. The consuming services must be deployed with backward-compatible readers before the migration runs.
-- Never assume that all service instances will be running the same code version during a migration window. Design migrations to tolerate mixed-version reads and writes during the rollout period.
+Hard rules: never rename a column in a single step; never add a `NOT NULL` column to a populated table without a default or a deferred `SET NOT NULL NOT VALID` → `VALIDATE`; every phase must be valid in isolation so that any deploy is independently rollbackable.
+
+## Online Schema Changes
+
+Set `lock_timeout` and `statement_timeout` before every DDL statement to bound blast radius. Selection by engine:
+
+- **Postgres 18.x.** Use `CREATE INDEX CONCURRENTLY` (outside any transaction block — disable the migration tool's transaction wrapper). On failure, the index is left `INVALID`; emit a `DROP INDEX IF EXISTS` + retry step. For FK and CHECK constraints, use `ALTER TABLE ... ADD CONSTRAINT ... NOT VALID` followed later by `VALIDATE CONSTRAINT` (skips full scan, downgrades to `SHARE UPDATE EXCLUSIVE`). Postgres 18 also supports `SET NOT NULL NOT VALID` for column nullability. Use `pg_repack` 1.5.x for bloat removal instead of `VACUUM FULL`. Avoid `ALTER TABLE ... ADD COLUMN ... DEFAULT non_constant_expression` on large tables — it rewrites every row.
+- **MySQL 8.4 LTS.** `ALGORITHM=INSTANT` is the default for many metadata ops (ADD COLUMN at end, RENAME COLUMN, some index meta) — verify against the 8.4 online DDL operations matrix. Hard limit: 64 row versions per table in 8.4. When `INSTANT` is rejected, fall back to `ALGORITHM=INPLACE`. For `ALGORITHM=COPY` operations on large tables, use `gh-ost` v1.1.8 (trigger-free, binlog-based, checkpoint + resume + revert) when the table has no incoming FKs and the cluster is not Galera / Percona XtraDB. Use `pt-online-schema-change` when FKs are present (`--alter-foreign-keys-method`) or under Galera. `lhm` is unmaintained — do not propose it for new code.
+
+## Backfill Jobs
+
+Every backfill must be batched, idempotent, resumable, throttled, and observable.
+
+- **Batched.** Order by PK or a monotonic key. Chunk by `id BETWEEN ? AND ?` (range), not `LIMIT/OFFSET` — offsets drift under concurrent writes. Default chunk 1k–10k rows; tune by table width.
+- **Idempotent.** Write `UPDATE ... SET new = f(old) WHERE id = ? AND new IS NULL` (or upsert with a deterministic source-derived value). Re-running on the same range must produce the same final state.
+- **Resumable.** Persist the last-processed boundary (`last_id` or timestamp cursor) to a control table after each batch commit. Resume from the checkpoint on restart; never restart from zero on partial failure.
+- **Throttled.** Poll replication lag (`pg_stat_replication`, `SHOW REPLICA STATUS`) between batches; pause when lag exceeds 30 seconds or the SLO threshold. Cap concurrency at the IO budget of the slowest replica.
+- **Observable.** Emit `migration.backfill.rows_processed` (counter), `migration.backfill.error_rate` (counter), `migration.backfill.eta_seconds` (gauge), and `migration.backfill.current_boundary` (gauge). Wire dashboards before launch. Avoid single mega-DML — one `UPDATE` over 50M+ rows produces multi-hour locks and table bloat.
+
+## Compatibility Window
+
+Schema changes deploy before the code that depends on them when widening (add column, add table, add index). Schema changes deploy after the code that no longer depends on them when narrowing (drop column, drop table). During the window, app code reads both shapes — the new shape if populated, fall back to the old shape otherwise. Rollback compatibility (old code remains executable against the current schema) must hold for at least 1 full release cycle plus 1 on-call rotation — minimum 7 calendar days, longer when the on-call rotation is longer.
+
+## Reversibility
+
+Every migration ships a tested down-migration script. Forward-only migrations are permitted only when the operation is data-destructive (e.g., a `DROP COLUMN` after Contract) — these require an explicit `IRREVERSIBLE: <reason>` annotation in the migration header and reviewer sign-off. A compensating forward migration that restores the prior shape is acceptable in place of a down-script for tools that lack reversibility (Prisma Migrate, Drizzle Kit — surface the gap to the reviewer). Default for every migration: reversible.
+
+## Data Integrity Verification
+
+Apply layered verification from cheapest to most thorough; stop at the cheapest layer that detects no drift.
+
+1. **Pre-migration backup drill.** Full restore to staging plus a smoke query within 24 hours prior to a destructive migration. "Backup exists" is not verification.
+2. **Row-count parity per chunk.** Source rows processed equals target rows written. Log discrepancies as errors, not warnings.
+3. **Aggregate checks.** SUM, MIN, MAX, COUNT(DISTINCT) on numeric and date columns per partition or batch.
+4. **Per-block checksums.** SHA-256 or MD5 over concatenated key columns for blocks of N rows (e.g., `md5(string_agg(id::text || col::text, ',' ORDER BY id))`).
+5. **Cross-system diff.** Datafold Reconcile, dbt-data-diff, or a hand-rolled sample-then-drill comparison for value-level differences.
+6. **Canary dual-read.** Read both shapes in production for 24–72 hours before cutover; shadow-diff and alert on mismatch.
+7. **Reconciliation control table.** Per-batch row count plus checksum stored alongside the checkpoint; auto-stop the backfill on drift above the configured threshold.
+
+## Multi-Region & Replica Lag
+
+- Pause backfill writes when any replica lag exceeds 30 seconds (or the project's lag SLO, whichever is lower). Resume only after lag returns to baseline for 5 consecutive minutes.
+- Roll migrations across regions sequentially; never alter an active partition during the peak traffic window of any region.
+- FK validation (`VALIDATE CONSTRAINT`) reads the entire dependent table — schedule outside peak read windows on replica-heavy topologies.
+- For Postgres major-version upgrades, use native logical replication (PG17+ preserves slots through `pg_upgrade`); advance sequences manually at cutover — logical replication does not replicate sequences, DDL, or large objects.
+- For ongoing cross-system replication, prefer Debezium (when Kafka is already deployed) or AWS DMS (managed, AWS-native). DMS hard limit: 200 tasks per replication instance — relevant for schema-per-tenant designs.
+
+## Tooling Mandate
+
+Pick one schema-management tool per project and commit the schema declaration to the repo. Greenfield default: Atlas (50+ destructive/locking linters, auto-generated down migrations, GitHub Actions approval policies) or dbmate (plain-SQL portability with first-class `-- migrate:down`). Existing-project default: whatever already ships migrations in the repo. Acceptable tools: Atlas, Prisma Migrate (forward-only — surface to reviewer), Drizzle Kit (forward-only — surface), Flyway 11+, Liquibase 4.27+ Pro, sqitch, Alembic, Knex, dbmate, Bytebase. Run a migration linter in CI — Atlas analyze, `squawk` for raw Postgres SQL — fail the PR on destructive operations without an explicit `IRREVERSIBLE:` annotation.
+
+Cross-references: see `hatch3r-data-classification` (PII / encrypted-column migration requirements), `hatch3r-feature-flags` (read-path switchover gating), `hatch3r-observability-metrics` (backfill progress metrics).

package/rules/hatch3r-observability-logging.md

@@ -3,7 +3,7 @@ id: hatch3r-observability-logging
 type: rule
 description: Structured logging and error reporting conventions for the project
 scope: conditional
-globs: "**/*log*,**/*logger*,**/*logging*,**/*error*,**/observability/**"
+globs: "**/*log*,**/*logger*,**/*logging*,**/*error*,**/observability/**,**/routes/**,**/handlers/**,**/services/**,**/api/**,**/middleware/**,**/controllers/**,**/lib/**"
 tags: [devops]
 quality_charter: agents/shared/quality-charter.md
 cache_friendly: true

package/rules/hatch3r-observability-logging.mdc

@@ -1,6 +1,6 @@
 ---
 description: Structured logging and error reporting conventions for the project
-globs: ["**/*log*", "**/*logger*", "**/*logging*", "**/*error*", "**/observability/**"]
+globs: ["**/*log*", "**/*logger*", "**/*logging*", "**/*error*", "**/observability/**", "**/routes/**", "**/handlers/**", "**/services/**", "**/api/**", "**/middleware/**", "**/controllers/**", "**/lib/**"]
 alwaysApply: false
 ---
 # Observability -- Logging & Error Reporting

package/rules/hatch3r-observability-metrics.md

@@ -3,7 +3,7 @@ id: hatch3r-observability-metrics
 type: rule
 description: Metrics, SLO/SLI definitions, alerting, and dashboard conventions for the project
 scope: conditional
-globs: "**/*metric*,**/*slo*,**/*sli*,**/*alert*,**/*dashboard*,**/observability/**"
+globs: "**/*metric*,**/*slo*,**/*sli*,**/*alert*,**/*dashboard*,**/observability/**,**/routes/**,**/handlers/**,**/services/**,**/api/**,**/middleware/**,**/controllers/**,**/lib/**"
 tags: [devops]
 quality_charter: agents/shared/quality-charter.md
 cache_friendly: true

package/rules/hatch3r-observability-metrics.mdc

@@ -1,6 +1,6 @@
 ---
 description: Metrics, SLO/SLI definitions, alerting, and dashboard conventions for the project
-globs: ["**/*metric*", "**/*slo*", "**/*sli*", "**/*alert*", "**/*dashboard*", "**/observability/**"]
+globs: ["**/*metric*", "**/*slo*", "**/*sli*", "**/*alert*", "**/*dashboard*", "**/observability/**", "**/routes/**", "**/handlers/**", "**/services/**", "**/api/**", "**/middleware/**", "**/controllers/**", "**/lib/**"]
 alwaysApply: false
 ---
 # Observability -- Metrics, SLOs & Alerting

package/rules/hatch3r-observability-tracing-detail.md

@@ -3,7 +3,7 @@ id: hatch3r-observability-tracing-detail
 type: rule
 description: Extended tracing reference -- AI agent instrumentation, tool call audit trails, LLM request tracing, and correlation ID patterns
 scope: conditional
-globs: "**/*trac*,**/*span*,**/*telemetry*,**/*otel*,**/*agent*,**/observability/**"
+globs: "**/*trac*,**/*span*,**/*telemetry*,**/*otel*,**/*agent*,**/observability/**,**/routes/**,**/handlers/**,**/services/**,**/api/**,**/middleware/**,**/controllers/**,**/lib/**"
 tags: [devops]
 quality_charter: agents/shared/quality-charter.md
 cache_friendly: true

package/rules/hatch3r-observability-tracing-detail.mdc

@@ -1,6 +1,6 @@
 ---
 description: Extended tracing reference -- AI agent instrumentation, tool call audit trails, LLM request tracing, and correlation ID patterns
-globs: ["**/*trac*", "**/*span*", "**/*telemetry*", "**/*otel*", "**/*agent*", "**/observability/**"]
+globs: ["**/*trac*", "**/*span*", "**/*telemetry*", "**/*otel*", "**/*agent*", "**/observability/**", "**/routes/**", "**/handlers/**", "**/services/**", "**/api/**", "**/middleware/**", "**/controllers/**", "**/lib/**"]
 alwaysApply: false
 ---
 # Observability -- Tracing Extended Reference

package/rules/hatch3r-observability-tracing.md

@@ -3,7 +3,7 @@ id: hatch3r-observability-tracing
 type: rule
 description: Distributed tracing and OpenTelemetry core conventions for the project
 scope: conditional
-globs: "**/*trac*,**/*span*,**/*telemetry*,**/*otel*,**/observability/**"
+globs: "**/*trac*,**/*span*,**/*telemetry*,**/*otel*,**/observability/**,**/routes/**,**/handlers/**,**/services/**,**/api/**,**/middleware/**,**/controllers/**,**/lib/**"
 tags: [devops]
 quality_charter: agents/shared/quality-charter.md
 cache_friendly: true

package/rules/hatch3r-observability-tracing.mdc

@@ -1,6 +1,6 @@
 ---
 description: Distributed tracing and OpenTelemetry core conventions for the project
-globs: ["**/*trac*", "**/*span*", "**/*telemetry*", "**/*otel*", "**/observability/**"]
+globs: ["**/*trac*", "**/*span*", "**/*telemetry*", "**/*otel*", "**/observability/**", "**/routes/**", "**/handlers/**", "**/services/**", "**/api/**", "**/middleware/**", "**/controllers/**", "**/lib/**"]
 alwaysApply: false
 ---
 # Observability -- Distributed Tracing & OpenTelemetry

package/rules/hatch3r-observability.md

@@ -3,6 +3,7 @@ id: hatch3r-observability
 type: rule
 description: "[Deprecated] Observability conventions -- split into hatch3r-observability-logging, hatch3r-observability-metrics, and hatch3r-observability-tracing"
 scope: conditional
+globs: "**/routes/**,**/handlers/**,**/services/**,**/api/**,**/middleware/**,**/controllers/**,**/lib/**,**/observability/**"
 tags: [devops]
 quality_charter: agents/shared/quality-charter.md
 deprecated: true

package/rules/hatch3r-observability.mdc

@@ -1,5 +1,6 @@
 ---
 description: "[Deprecated] Observability conventions -- split into hatch3r-observability-logging, hatch3r-observability-metrics, and hatch3r-observability-tracing"
+globs: ["**/routes/**", "**/handlers/**", "**/services/**", "**/api/**", "**/middleware/**", "**/controllers/**", "**/lib/**", "**/observability/**"]
 alwaysApply: false
 ---
 # Observability (Deprecated Redirect)

package/rules/hatch3r-operability.md

@@ -0,0 +1,149 @@
+---
+id: hatch3r-operability
+type: rule
+description: Operability patterns in user code — liveness / readiness / startup probes, graceful shutdown, feature flags, runbook URL annotations, health endpoints
+scope: "**/services/**,**/handlers/**,**/health*,**/probes/**,**/k8s/**,**/manifests/**,**/charts/**,**/feature*,**/flags/**"
+tags: [implementation, devops]
+quality_charter: agents/shared/quality-charter.md
+cache_friendly: true
+---
+# Operability
+
+## Liveness, Readiness, and Startup Probes (Kubernetes)
+
+Three probe kinds; each answers a different question and triggers a different action. Conflating them is the most common operability bug — a liveness probe that reaches downstream causes pod-restart cascades on dependency outages.
+
+- **Liveness** — shallow self-check: process alive, event loop responsive, no deadlock on the request handler. NEVER checks external dependencies. Failure → kubelet restarts the pod. Default: `httpGet /health/live`, `periodSeconds: 10`, `failureThreshold: 3`, `timeoutSeconds: 1`.
+- **Readiness** — deep dependency check: DB reachable, cache reachable, required downstream healthy, migrations applied. Failure → endpoint controller removes the pod from Service load-balancer rotation; pod stays running and recovers when dependencies return. Default: `httpGet /health/ready`, `periodSeconds: 5`, `failureThreshold: 2`, `timeoutSeconds: 2`.
+- **Startup** — same checks as readiness but with longer initial allowance for slow-starting apps (large model load, warm cache build, JIT warmup, schema migration). Once succeeds once, liveness and readiness take over. Default: `httpGet /health/startup`, `periodSeconds: 5`, `failureThreshold: 60` (allowing 5 minutes), `timeoutSeconds: 2`.
+
+Concrete examples per ecosystem:
+
+- **Node Express** — `/health/live`, `/health/ready`, `/health/startup` on the same router with different handler chains. The live handler returns 200 unconditionally if the event loop is responsive; the ready handler awaits `Promise.allSettled` over DB ping, cache ping, downstream ping with per-check 1s timeout.
+- **Spring Boot Actuator** — `/actuator/health/liveness` and `/actuator/health/readiness` exposed out of the box; add startup via a custom `HealthIndicator`. Configure `management.endpoint.health.probes.enabled=true`.
+- **Go (net/http)** — three handlers on `http.ServeMux`; ready handler aggregates checks via `errgroup.Group.Wait()` with per-check `context.WithTimeout`.
+
+Anti-pattern: a single `/health` endpoint that both Kubernetes probes hit. The pod is killed during DB outage because the liveness probe failed on the same deep check the readiness probe is supposed to own.
+
+## Graceful Shutdown
+
+Handle `SIGTERM` per the sequence below. Skipping the preStop delay causes the well-known endpoint-propagation race that drops up to several seconds of in-flight traffic.
+
+- Step 1: Stop accepting new connections (close the HTTP listener; stop pulling from the queue).
+- Step 2: Mark `/health/ready` to return 503 (the endpoint controller removes the pod from the Service).
+- Step 3: Wait `preStop` window of 1–3s for endpoint-removal propagation. Kubernetes does NOT propagate endpoint removal before delivering SIGTERM — without an explicit `preStop` `sleep`, traffic continues to arrive for the first 1–3s of shutdown.
+- Step 4: Drain in-flight requests (configurable deadline 30–45s; cap by `terminationGracePeriodSeconds`).
+- Step 5: Close DB connections, drain queue consumers, flush log buffers, close OpenTelemetry exporters.
+- Step 6: Exit 0.
+
+Pod manifest baseline:
+
+```
+terminationGracePeriodSeconds: 45
+lifecycle:
+  preStop:
+    exec:
+      command: ["sh", "-c", "sleep 3"]
+```
+
+Force kill after grace period defeats the drain. If 45s is insufficient, raise `terminationGracePeriodSeconds` rather than skip the drain — but investigate why the service holds long-running requests at shutdown.
+
+For queue consumers (Kafka, NATS, SQS): on SIGTERM stop pulling new messages first, finish in-flight processing, then commit offsets and disconnect. Skipping the commit step on shutdown produces duplicate processing on the next pod's first poll.
+
+## Feature Flags — OpenFeature
+
+OpenFeature is a CNCF Incubating SDK (pre-1.0 spec) that wraps any provider — LaunchDarkly, Unleash, Flagsmith, GrowthBook, Statsig — behind a consistent API. New code targets OpenFeature; provider-specific SDK imports become a finding.
+
+Flag types by lifecycle:
+
+- **Release flag** — gate new code path during rollout. Remove within 1 sprint of full enablement; otherwise it becomes flag debt.
+- **Experiment flag** — A/B test traffic split for measurement. Retire after the experiment concludes and the decision is shipped.
+- **Ops flag** — kill switch for risky features and feature-level circuit breakers. Permanent by design. Cross-reference the kill-switch pattern below.
+- **Permission flag** — entitlement gating (per-tenant feature availability). Prefer reusing the authorization layer where possible; flag only the surface that the auth system does not naturally cover.
+
+Flag-debt budget: 50–100 active flags per service. Run a monthly cleanup cadence — list flags with `enabled = true for 100% of traffic for >30 days` and either remove them or convert to permanent config.
+
+Default-off for new release flags; default-on only for kill switches (so the absence of provider connectivity does not silently disable the feature the on-call needs to disable). Evaluate flags at request entry once and pass the resolved value down — re-evaluating mid-request risks split-brain behavior when the flag flips during the call.
+
+## Kill-Switch Pattern
+
+Every risky feature ships with a kill switch (Ops flag) and a documented procedure for flipping it. On-call must be able to disable the feature without redeploying. Document the flag name in `docs/runbooks/<service>.md` alongside the alert.
+
+Test the kill switch on every release — a kill switch that nobody has flipped in production is unverified. Quarterly drill: flip the flag, observe the metric drop, restore.
+
+Cross-reference `rules/hatch3r-progressive-delivery.md` — kill switch is the rollback mechanism when the staged rollout has already finished and the regression is discovered after 100%.
+
+## Runbook URL on Every Alert
+
+Alert without a runbook is a finding under this rule. Every alert carries a runbook URL annotation:
+
+- **Prometheus:** `annotations.runbook_url: "https://internal.example.com/runbooks/<alert-name>.md"`.
+- **Datadog / Grafana:** equivalent `runbook_url` or `notification_message` template field.
+
+Runbook format:
+
+- **Symptoms** — what the on-call sees on the dashboard or in the alert payload.
+- **Triage** — first three commands or queries to narrow the cause.
+- **Mitigation** — kill switch, rollback command, scale action.
+- **Root cause** — links to past postmortems for similar symptoms.
+- **Follow-ups** — open issues, related dashboards, owner team.
+
+Empirical observation from 2024–2026 incidents: LLM-driven auto-diagnosis quality is dominated by runbook quality rather than by model choice. A high-fidelity runbook with concrete commands beats a generic prompt against a frontier model on the same dataset.
+
+Runbooks live in the service repository under `docs/runbooks/<alert-name>.md` so they ship with the code that emits the alert. Renaming an alert without updating the runbook URL produces a 404 link from the alert payload — a CI check on the alert manifest catches this on every PR that touches alert names.
+
+## Health Endpoint Conventions
+
+- `/health/live` — 200 + `{ "status": "ok", "version": "<semver>", "build_sha": "<short-sha>" }` on success; 503 + `{ "status": "down", "reason": "<diagnostic>" }` on failure.
+- `/health/ready` — same shape; 503 includes the failing downstream name (e.g. `reason: "postgres-primary unreachable"`).
+- `/health/startup` — same as ready but with allowance for warmup.
+
+Never expose secrets, connection strings, or per-request internals from health endpoints — they are unauthenticated by default. Detailed diagnostics live behind authentication on a separate admin endpoint.
+
+Probe response time budget: under 100ms for `/health/live`, under 500ms for `/health/ready` and `/health/startup`. A probe that times out the kubelet causes pod restart cascades unrelated to the underlying issue.
+
+Cache the ready-check result for 1–2s — Kubernetes polls every 5s by default; checking each downstream on every poll multiplies dependency load by the replica count.
+
+## Capacity Planning
+
+- Nightly load test in CI against a staging environment representative of production. Compare baseline-vs-current p50, p95, p99 latency and error rate; fail the run on a 20% regression vs the previous green baseline.
+- Saturation tracking — alert when CPU sustained above 70% for 15 minutes, memory above 80% for 15 minutes, connection pool above 80% utilization for 5 minutes.
+- Rightsizing — target 20–30% headroom on CPU and memory at peak. Below 10% headroom is undersized; above 50% sustained is oversized and a cost finding.
+- HPA (Horizontal Pod Autoscaler) target CPU at 60–70% — leaves room for the scale-up lag to bring new replicas online before the existing pods saturate.
+- Cold-start budget for serverless and JVM services: measure p95 cold-start latency; provision the warm-pool to keep cold-start traffic below 1% of total. KEDA or platform autoscaler keeps the warm-pool at the right size.
+
+## Multi-Region and Disaster Recovery
+
+- RTO (Recovery Time Objective) and RPO (Recovery Point Objective) documented per service tier in the service catalog.
+- Active-active for tier-1 services targeting RTO under 5 minutes and RPO under 1 minute.
+- Active-passive acceptable for tier-2 services targeting RTO under 1 hour.
+- Failover drill quarterly; the drill is the test of the runbook. A runbook that has not been executed in a drill is a draft, not a runbook.
+- Data residency — honor regional data boundaries (EU-US DPF, regional cloud regions). Cross-region replication respects the residency contract.
+- DNS-based failover (Route 53, Cloud DNS) has a propagation tail measured in minutes; for sub-minute RTO use load-balancer-level failover (cross-region target groups) or anycast.
+
+## 2024–2026 Outage Lessons
+
+- **CrowdStrike, July 2024** — global config push without canary. Mitigation: staged rollout (cross-reference `rules/hatch3r-progressive-delivery.md`).
+- **AWS us-east-1, October 2025** — cascading failure across services with hidden us-east-1 control-plane dependency. Mitigation: dependency mapping, multi-region for control plane.
+- **Azure East-US2, September 2025** — single-region outage on customer-perceived multi-region service. Mitigation: active-active across regions.
+- **Cloudflare, November 2025** — config-change incident. Mitigation: treat config as code, canary config changes.
+
+Most outages have an organizational root cause (change management, capacity planning, ownership). The patterns above defend against the technical leg of the failure; organizational defenses are out of scope for this rule.
+
+Postmortem cadence: within 5 business days of incident close, blameless, owned by the on-call rotation, action items tracked in the service backlog with target due dates. Postmortem reviewers cross-check that the runbook was updated and that an automated detection (alert, dashboard, or unit test) was added for the failure mode that escaped detection.
+
+## Cross-References
+
+- `rules/hatch3r-resilience-patterns.md` — circuit breakers, retries, timeouts.
+- `rules/hatch3r-progressive-delivery.md` — canary, blue-green, kill-switch usage during rollout.
+- `rules/hatch3r-observability-metrics.md` — SLOs, RED metrics, burn-rate alerts feed the runbook.
+
+## References
+
+- Kubernetes probes — `kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes`
+- Kubernetes pod termination — `kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination`
+- Google SRE workbook, graceful-shutdown chapter — `sre.google/workbook`
+- OpenFeature specification — `openfeature.dev`
+- LaunchDarkly docs — `docs.launchdarkly.com`
+- Unleash docs — `docs.getunleash.io`
+- 2024–2026 outage postmortems: CrowdStrike Jul 2024, AWS us-east-1 Oct 2025, Azure East-US2 Sep 2025, Cloudflare Nov 2025.