@tamsensedev/runner 0.13.0 → 0.18.0

package/library/skills/safe-database-changes/SKILL.md

@@ -1,111 +1,111 @@
----
-name: safe-database-changes
-description: Use when a change touches a database schema, migration files, ORM models, seeds, backfill scripts or SQL that writes stored data. Covers which databases a run may touch, never editing applied migrations, expand-and-contract for renames and drops, lock-safe DDL, batched backfills and the deploy order to report.
----
-
-# Safe database changes
-
-During a deploy, old and new code run against the same database, and deleted data does not come back. Nobody is watching this run, so every database action must be safe by construction, not by luck.
-
-The project's own conventions win: its migration tool, folder layout, naming, review rules and anything in CLAUDE.md, AGENTS.md, CONTRIBUTING or the README about migrations and deploys. This skill fills the gaps.
-
-## Hard rules
-
-1. **Only a throwaway database.** Never run migrations, seeds, resets, backfills, fixtures or any other write against a real, shared, staging or production database. Allowed targets: a database you created for this run and will delete (for example a container you started), an in-memory or temp-file database (SQLite in a temp folder), or the one the project's own test setup creates and destroys (testcontainers, a `docker compose` test service, a test script that creates a fresh database).
-2. **Know the target before you connect.** Before any command that opens a database connection (migrate, seed, `db push`, a test run that hits the database, an ORM console), find the URL it will use: `.env*` files, config files, environment variables already set in the shell, the tool's config. If you cannot show it points at a throwaway database, do not run it. Write the migration file anyway and say in the report that it was not applied. A localhost URL is not proof: a developer's local database holds their data too.
-3. **Never edit an applied migration.** A migration that exists on the base branch, or may have run anywhere, is frozen: do not edit, rename, reorder, squash or delete it. Fix forward with a new migration. You may edit only migrations this branch added (check with `git log origin/<base> -- <migration path>`: no output means it is new).
-4. **No destructive shortcuts.** Do not use commands that make the database match the schema by throwing data away: `migrate reset`, `db push --accept-data-loss` or `--force-reset`, schema "push" commands against anything but a throwaway database, `DROP DATABASE`, `TRUNCATE`, `flush`. Never bypass a tool's safety checks: no `--force` you were not told to use, no consent environment variables (Prisma blocks resets from AI agents until a human consents; never set that variable yourself), no hiding the variables tools use to detect an agent.
-5. **No secrets in output.** When you mention a database URL in the report or logs, show only the host and database name.
-
-## Process
-
-### 1. Learn how this project does migrations
-
-- Find the tool and the folder: `prisma/migrations`, `drizzle/`, `db/migrate` (Rails), `*/migrations/` (Django), `alembic/versions`, `migrations/` with up and down SQL files, Flyway `V1__*.sql`, Liquibase changelogs, EF Core `Migrations/`, Laravel `database/migrations`.
-- Read two or three recent migrations: naming, one change per file or many, whether schema and data changes are separated, whether they write raw SQL for indexes, whether down migrations exist.
-- Find the commands in package scripts, Makefile, CI workflows or docs. Note which are dev-only (they may reset or diff against a live database) and which are deploy commands. See [references/migration-tools.md](references/migration-tools.md) for the common tools.
-- Look for a documented deploy order (migrate before or after the app starts) and a migration linter the project already uses (for example squawk). Run the linter if it exists.
-
-### 2. Classify the change
-
-| Kind | Examples | Safe in one deploy? |
-|---|---|---|
-| Additive | new table, nullable column, column with a constant default, new index (built without blocking) | Yes |
-| Breaking | rename or drop a column or table, change a type, new NOT NULL or UNIQUE on existing data, tighter check | No: use expand and contract |
-| Data | backfill, transform, dedupe, delete rows | Separate migration or script, batched |
-| Heavy | index or constraint on a large table, type change that rewrites the table | Yes only with non-blocking forms |
-
-Old code must keep working against the new schema (it runs until the deploy finishes), and the new code must cope with the moment before the migration ran if the project deploys code first.
-
-### 3. Breaking changes: expand and contract
-
-Never rename or drop something the running code still uses in the same deploy. Split it:
-
-1. **Expand:** add the new column or table (nullable or with a default). Deploy code that writes both old and new.
-2. **Migrate:** backfill existing rows in batches. Deploy code that reads the new one.
-3. **Contract:** once nothing reads or writes the old one, drop it in a later, separate change.
-
-A task usually covers one phase. Do that phase and list the remaining phases as follow-ups in the report, with the order they must ship in. Do not squeeze all phases into one migration to "finish the job".
-
-Watch the ORM: some tools turn a renamed field into DROP plus ADD, which deletes the data. Always read the SQL the tool generated (step 4).
-
-### 4. Write the migration
-
-- Generate it the project's way, then **read the generated SQL** line by line. Look for DROP, type changes, NOT NULL on existing columns, new unique constraints, defaults that rewrite the table, indexes built in a blocking way, and anything you did not intend.
-- One concern per migration. Keep schema changes and data changes in separate migrations or scripts.
-- Make it re-runnable where the tool allows it (`IF NOT EXISTS`, guarded blocks), and give it a down migration if the project writes them. If a step cannot be undone (dropping data, lossy casts), say so in the migration's description and in the report.
-- Large or busy tables need lock-aware DDL: build indexes without blocking writes, add constraints as NOT VALID and validate separately, set a short lock timeout. PostgreSQL details, with the lock level of each operation: [references/postgres-ddl.md](references/postgres-ddl.md). For MySQL, prefer `ALGORITHM=INSTANT` or `INPLACE, LOCK=NONE` where supported and state the table size risk; for SQLite, many changes rebuild the table, which is fine for local files but say so.
-- Keep integrity rules in the database where the project does (NOT NULL, UNIQUE, foreign keys, CHECK), and still validate input at the application boundary.
-- New queries need indexes, and new foreign keys usually need an index on the referencing column. Query tuning lives in the `postgres-performance` skill.
-
-### 5. Data changes and backfills
-
-Backfill in batches, never one UPDATE over the whole table: each batch short, keyed on the primary key, idempotent (safe to run twice), resumable after a crash, with a pause between batches. Patterns for SQL and for ORM scripts: [references/backfills.md](references/backfills.md).
-
-Never run the backfill against real data from this run. Ship it as a migration or script and describe how and when it should run.
-
-### 6. Update everything that depends on the schema
-
-ORM models and generated clients, types, validation schemas, seeds, fixtures and factories, API serializers, admin screens, analytics queries, and tests. Search for the old column and table names across the repository (including raw SQL strings) before you call a rename or drop done.
-
-### 7. Verify against a throwaway database
-
-1. Start from an empty throwaway database and apply all migrations from scratch. This proves the migration chain still works.
-2. If the tool supports it, roll back the new migration and apply it again.
-3. Insert a few realistic rows before your migration when it touches existing data (NULLs, duplicates, long strings, old enum values) and check the result. Run the validation queries in [references/postgres-ddl.md](references/postgres-ddl.md#checks-before-and-after) on that data.
-4. Run the project's checks: generate the client, type check, tests.
-5. Delete what you created: stop and remove the container, delete the temp database file.
-
-If no throwaway database is possible (no Docker, no test setup), use offline checks the tool offers (schema validation, `--create-only`, SQL diff output), read the SQL carefully, and state plainly in the report that the migration was not applied anywhere.
-
-## Report
-
-Include in your final report:
-
-- Each schema change and its migration file.
-- Whether it was applied, and where (only a throwaway database, named as such), or why not.
-- Lock and duration risk on big tables, and how the migration avoids it.
-- Deploy order: migrate before or after the code, and which later changes (backfill run, contract phase) must follow.
-- Anything irreversible, and how to roll back the rest.
-- Assumptions you made (table sizes, database version) because you could not check them.
-
-## Common mistakes
-
-- Running `migrate dev` or a schema push with the default `.env`, which points at a shared database.
-- Editing an old migration to "fix" it instead of adding a new one.
-- Accepting the ORM's DROP plus ADD for a rename.
-- Adding NOT NULL or UNIQUE to a column with existing bad data, without checking for NULLs or duplicates first.
-- Backfilling in the same transaction as the schema change, or in one giant UPDATE.
-- Dropping a column in the same deploy that stops using it.
-- Creating an index in a blocking way on a big table, or concurrently inside a transaction (it fails).
-- Forgetting seeds, fixtures and raw SQL that still use the old name.
-
-## Done when
-
-- The migration follows the project's tool and conventions and its SQL has been read.
-- No real or shared database was touched; anything applied went to a throwaway database that is now gone.
-- Old and new code both work during the deploy, or the report explains the required order.
-- Backfills are batched, idempotent and separate from schema changes.
-- The report lists changes, deploy order, follow-ups and anything irreversible.
-
-Sources: timescale/pg-aiguide postgres-database-migration (Apache-2.0, commit 187be00), affaan-m/ECC database-migrations (MIT, commit 30b6892), facts restated from prisma/skills (MIT, commit ba3f9d7). Changed: rewritten for unattended runs against local throwaway databases only.
+---
+name: safe-database-changes
+description: Use when a change touches a database schema, migration files, ORM models, seeds, backfill scripts or SQL that writes stored data. Covers which databases a run may touch, never editing applied migrations, expand-and-contract for renames and drops, lock-safe DDL, batched backfills and the deploy order to report.
+---
+
+# Safe database changes
+
+During a deploy, old and new code run against the same database, and deleted data does not come back. Nobody is watching this run, so every database action must be safe by construction, not by luck.
+
+The project's own conventions win: its migration tool, folder layout, naming, review rules and anything in CLAUDE.md, AGENTS.md, CONTRIBUTING or the README about migrations and deploys. This skill fills the gaps.
+
+## Hard rules
+
+1. **Only a throwaway database.** Never run migrations, seeds, resets, backfills, fixtures or any other write against a real, shared, staging or production database. Allowed targets: a database you created for this run and will delete (for example a container you started), an in-memory or temp-file database (SQLite in a temp folder), or the one the project's own test setup creates and destroys (testcontainers, a `docker compose` test service, a test script that creates a fresh database).
+2. **Know the target before you connect.** Before any command that opens a database connection (migrate, seed, `db push`, a test run that hits the database, an ORM console), find the URL it will use: `.env*` files, config files, environment variables already set in the shell, the tool's config. If you cannot show it points at a throwaway database, do not run it. Write the migration file anyway and say in the report that it was not applied. A localhost URL is not proof: a developer's local database holds their data too.
+3. **Never edit an applied migration.** A migration that exists on the base branch, or may have run anywhere, is frozen: do not edit, rename, reorder, squash or delete it. Fix forward with a new migration. You may edit only migrations this branch added (check with `git log origin/<base> -- <migration path>`: no output means it is new).
+4. **No destructive shortcuts.** Do not use commands that make the database match the schema by throwing data away: `migrate reset`, `db push --accept-data-loss` or `--force-reset`, schema "push" commands against anything but a throwaway database, `DROP DATABASE`, `TRUNCATE`, `flush`. Never bypass a tool's safety checks: no `--force` you were not told to use, no consent environment variables (Prisma blocks resets from AI agents until a human consents; never set that variable yourself), no hiding the variables tools use to detect an agent.
+5. **No secrets in output.** When you mention a database URL in the report or logs, show only the host and database name.
+
+## Process
+
+### 1. Learn how this project does migrations
+
+- Find the tool and the folder: `prisma/migrations`, `drizzle/`, `db/migrate` (Rails), `*/migrations/` (Django), `alembic/versions`, `migrations/` with up and down SQL files, Flyway `V1__*.sql`, Liquibase changelogs, EF Core `Migrations/`, Laravel `database/migrations`.
+- Read two or three recent migrations: naming, one change per file or many, whether schema and data changes are separated, whether they write raw SQL for indexes, whether down migrations exist.
+- Find the commands in package scripts, Makefile, CI workflows or docs. Note which are dev-only (they may reset or diff against a live database) and which are deploy commands. See [references/migration-tools.md](references/migration-tools.md) for the common tools.
+- Look for a documented deploy order (migrate before or after the app starts) and a migration linter the project already uses (for example squawk). Run the linter if it exists.
+
+### 2. Classify the change
+
+| Kind | Examples | Safe in one deploy? |
+|---|---|---|
+| Additive | new table, nullable column, column with a constant default, new index (built without blocking) | Yes |
+| Breaking | rename or drop a column or table, change a type, new NOT NULL or UNIQUE on existing data, tighter check | No: use expand and contract |
+| Data | backfill, transform, dedupe, delete rows | Separate migration or script, batched |
+| Heavy | index or constraint on a large table, type change that rewrites the table | Yes only with non-blocking forms |
+
+Old code must keep working against the new schema (it runs until the deploy finishes), and the new code must cope with the moment before the migration ran if the project deploys code first.
+
+### 3. Breaking changes: expand and contract
+
+Never rename or drop something the running code still uses in the same deploy. Split it:
+
+1. **Expand:** add the new column or table (nullable or with a default). Deploy code that writes both old and new.
+2. **Migrate:** backfill existing rows in batches. Deploy code that reads the new one.
+3. **Contract:** once nothing reads or writes the old one, drop it in a later, separate change.
+
+A task usually covers one phase. Do that phase and list the remaining phases as follow-ups in the report, with the order they must ship in. Do not squeeze all phases into one migration to "finish the job".
+
+Watch the ORM: some tools turn a renamed field into DROP plus ADD, which deletes the data. Always read the SQL the tool generated (step 4).
+
+### 4. Write the migration
+
+- Generate it the project's way, then **read the generated SQL** line by line. Look for DROP, type changes, NOT NULL on existing columns, new unique constraints, defaults that rewrite the table, indexes built in a blocking way, and anything you did not intend.
+- One concern per migration. Keep schema changes and data changes in separate migrations or scripts.
+- Make it re-runnable where the tool allows it (`IF NOT EXISTS`, guarded blocks), and give it a down migration if the project writes them. If a step cannot be undone (dropping data, lossy casts), say so in the migration's description and in the report.
+- Large or busy tables need lock-aware DDL: build indexes without blocking writes, add constraints as NOT VALID and validate separately, set a short lock timeout. PostgreSQL details, with the lock level of each operation: [references/postgres-ddl.md](references/postgres-ddl.md). For MySQL, prefer `ALGORITHM=INSTANT` or `INPLACE, LOCK=NONE` where supported and state the table size risk; for SQLite, many changes rebuild the table, which is fine for local files but say so.
+- Keep integrity rules in the database where the project does (NOT NULL, UNIQUE, foreign keys, CHECK), and still validate input at the application boundary.
+- New queries need indexes, and new foreign keys usually need an index on the referencing column. Query tuning lives in the `postgres-performance` skill.
+
+### 5. Data changes and backfills
+
+Backfill in batches, never one UPDATE over the whole table: each batch short, keyed on the primary key, idempotent (safe to run twice), resumable after a crash, with a pause between batches. Patterns for SQL and for ORM scripts: [references/backfills.md](references/backfills.md).
+
+Never run the backfill against real data from this run. Ship it as a migration or script and describe how and when it should run.
+
+### 6. Update everything that depends on the schema
+
+ORM models and generated clients, types, validation schemas, seeds, fixtures and factories, API serializers, admin screens, analytics queries, and tests. Search for the old column and table names across the repository (including raw SQL strings) before you call a rename or drop done.
+
+### 7. Verify against a throwaway database
+
+1. Start from an empty throwaway database and apply all migrations from scratch. This proves the migration chain still works.
+2. If the tool supports it, roll back the new migration and apply it again.
+3. Insert a few realistic rows before your migration when it touches existing data (NULLs, duplicates, long strings, old enum values) and check the result. Run the validation queries in [references/postgres-ddl.md](references/postgres-ddl.md#checks-before-and-after) on that data.
+4. Run the project's checks: generate the client, type check, tests.
+5. Delete what you created: stop and remove the container, delete the temp database file.
+
+If no throwaway database is possible (no Docker, no test setup), use offline checks the tool offers (schema validation, `--create-only`, SQL diff output), read the SQL carefully, and state plainly in the report that the migration was not applied anywhere.
+
+## Report
+
+Include in your final report:
+
+- Each schema change and its migration file.
+- Whether it was applied, and where (only a throwaway database, named as such), or why not.
+- Lock and duration risk on big tables, and how the migration avoids it.
+- Deploy order: migrate before or after the code, and which later changes (backfill run, contract phase) must follow.
+- Anything irreversible, and how to roll back the rest.
+- Assumptions you made (table sizes, database version) because you could not check them.
+
+## Common mistakes
+
+- Running `migrate dev` or a schema push with the default `.env`, which points at a shared database.
+- Editing an old migration to "fix" it instead of adding a new one.
+- Accepting the ORM's DROP plus ADD for a rename.
+- Adding NOT NULL or UNIQUE to a column with existing bad data, without checking for NULLs or duplicates first.
+- Backfilling in the same transaction as the schema change, or in one giant UPDATE.
+- Dropping a column in the same deploy that stops using it.
+- Creating an index in a blocking way on a big table, or concurrently inside a transaction (it fails).
+- Forgetting seeds, fixtures and raw SQL that still use the old name.
+
+## Done when
+
+- The migration follows the project's tool and conventions and its SQL has been read.
+- No real or shared database was touched; anything applied went to a throwaway database that is now gone.
+- Old and new code both work during the deploy, or the report explains the required order.
+- Backfills are batched, idempotent and separate from schema changes.
+- The report lists changes, deploy order, follow-ups and anything irreversible.
+
+Sources: timescale/pg-aiguide postgres-database-migration (Apache-2.0, commit 187be00), affaan-m/ECC database-migrations (MIT, commit 30b6892), facts restated from prisma/skills (MIT, commit ba3f9d7). Changed: rewritten for unattended runs against local throwaway databases only.

package/library/skills/safe-database-changes/references/backfills.md

@@ -1,75 +1,75 @@
-# Backfills and data migrations
-
-Read this when existing rows must be filled, transformed, deduplicated or deleted. Write the backfill; do not run it against real data from a task run.
-
-## Rules
-
-- **Separate from schema changes.** The schema migration adds the column; the backfill is its own migration or script. A failed backfill must not roll back or block the schema change.
-- **Batches.** Each batch touches a bounded number of rows (start around 1,000 to 10,000) and commits. Never one UPDATE over the whole table: it holds row locks for the whole run, bloats the table and can lag replicas.
-- **Keyset by primary key.** Walk the table with `WHERE id > last_id ORDER BY id LIMIT n`, not OFFSET, which gets slower with every page.
-- **Idempotent.** Only touch rows that still need it (`WHERE new_col IS NULL`, or compare values), so running it twice or resuming after a crash is safe.
-- **Resumable.** Log or store progress (last id processed). A crash halfway must not require starting over or leave rows half-done.
-- **Gentle.** Pause briefly between batches (50 to 200 ms) on busy tables, and keep each batch well under a second.
-- **New writes are covered.** Code that writes the new column ships before or with the backfill, otherwise rows created during the backfill are missed.
-- **Verify.** After the run: count rows still needing the change (expect 0) and spot-check a sample.
-
-## SQL (PostgreSQL)
-
-One batch, run repeatedly until it updates 0 rows:
-
-```sql
-WITH batch AS (
-  SELECT id FROM users
-  WHERE normalized_email IS NULL
-  ORDER BY id
-  LIMIT 5000
-  FOR UPDATE SKIP LOCKED
-)
-UPDATE users u
-SET normalized_email = lower(u.email)
-FROM batch
-WHERE u.id = batch.id;
-```
-
-A loop that commits per batch (PostgreSQL 11+, run outside an explicit transaction; `COMMIT` inside `DO` fails if the caller wrapped it in one, which many migration runners do, so this usually belongs in a script, not a migration file):
-
-```sql
-DO $$
-DECLARE
-  last_id bigint := 0;
-  max_id bigint;
-BEGIN
-  SELECT coalesce(max(id), 0) INTO max_id FROM users;
-  WHILE last_id < max_id LOOP
-    UPDATE users
-    SET normalized_email = lower(email)
-    WHERE id > last_id AND id <= last_id + 5000
-      AND normalized_email IS NULL;
-    last_id := last_id + 5000;
-    RAISE NOTICE 'processed up to id %', last_id;
-    COMMIT;
-    PERFORM pg_sleep(0.1);
-  END LOOP;
-END $$;
-```
-
-Range batches (`id > a AND id <= b`) cope with gaps in ids; they just process fewer rows in sparse ranges. After a large backfill, run `ANALYZE` on the table (plain `VACUUM` too if many rows changed); never `VACUUM FULL` on a live table.
-
-## In application code
-
-Use the project's language and ORM, keep the same rules: fetch ids in keyset order, update a batch, commit, repeat.
-
-- **Rails:** `Model.where(new_col: nil).in_batches(of: 5_000) { |rel| rel.update_all(...) }` in a data migration or a rake task, with `disable_ddl_transaction!` if it lives in a migration.
-- **Django:** a `RunPython` migration (or management command) that loops over `values_list("pk", flat=True)` in slices and calls `bulk_update` or `update()`; set `atomic = False` on the migration so each batch commits.
-- **Alembic:** a separate revision that runs batched `UPDATE` statements through `op.get_bind()`, or a standalone script.
-- **Node ORMs (Prisma, Drizzle, Knex, Kysely, TypeORM):** a script in the project's scripts folder that loops `findMany` / `select` with `where id > lastId order by id take n`, then updates by id list; run outside long transactions.
-
-Name the script clearly, make the batch size and a dry-run mode configurable if the project's other scripts do, and document in the report how to run it after the schema migration is deployed.
-
-## Deletes and dedupes
-
-- Delete in batches by id, same as updates. Check what cascades (`ON DELETE CASCADE`) and how many child rows that pulls in.
-- For dedupes, write the rule for which row survives (oldest, most complete), repoint foreign keys to the survivor first, then delete the rest.
-- Irreversible: say so in the report and recommend a backup or an export of affected rows before running.
-
-Adapted from timescale/pg-aiguide `skills/postgres-database-migration/references/backfill-strategies.md` (Apache-2.0, commit 187be00d317a51210c132010d947f4546ffb5eef, Copyright 2025 Timescale, Inc., d/b/a Tiger Data) and affaan-m/ECC `skills/database-migrations` (MIT, commit 30b68924ae9729159f44326aa3134b1d9d67c0e2, Copyright (c) 2026 Affaan Mustafa). Changed: condensed, rewritten examples, added idempotency, resumability and ORM guidance, run-from-task restrictions.
+# Backfills and data migrations
+
+Read this when existing rows must be filled, transformed, deduplicated or deleted. Write the backfill; do not run it against real data from a task run.
+
+## Rules
+
+- **Separate from schema changes.** The schema migration adds the column; the backfill is its own migration or script. A failed backfill must not roll back or block the schema change.
+- **Batches.** Each batch touches a bounded number of rows (start around 1,000 to 10,000) and commits. Never one UPDATE over the whole table: it holds row locks for the whole run, bloats the table and can lag replicas.
+- **Keyset by primary key.** Walk the table with `WHERE id > last_id ORDER BY id LIMIT n`, not OFFSET, which gets slower with every page.
+- **Idempotent.** Only touch rows that still need it (`WHERE new_col IS NULL`, or compare values), so running it twice or resuming after a crash is safe.
+- **Resumable.** Log or store progress (last id processed). A crash halfway must not require starting over or leave rows half-done.
+- **Gentle.** Pause briefly between batches (50 to 200 ms) on busy tables, and keep each batch well under a second.
+- **New writes are covered.** Code that writes the new column ships before or with the backfill, otherwise rows created during the backfill are missed.
+- **Verify.** After the run: count rows still needing the change (expect 0) and spot-check a sample.
+
+## SQL (PostgreSQL)
+
+One batch, run repeatedly until it updates 0 rows:
+
+```sql
+WITH batch AS (
+  SELECT id FROM users
+  WHERE normalized_email IS NULL
+  ORDER BY id
+  LIMIT 5000
+  FOR UPDATE SKIP LOCKED
+)
+UPDATE users u
+SET normalized_email = lower(u.email)
+FROM batch
+WHERE u.id = batch.id;
+```
+
+A loop that commits per batch (PostgreSQL 11+, run outside an explicit transaction; `COMMIT` inside `DO` fails if the caller wrapped it in one, which many migration runners do, so this usually belongs in a script, not a migration file):
+
+```sql
+DO $$
+DECLARE
+  last_id bigint := 0;
+  max_id bigint;
+BEGIN
+  SELECT coalesce(max(id), 0) INTO max_id FROM users;
+  WHILE last_id < max_id LOOP
+    UPDATE users
+    SET normalized_email = lower(email)
+    WHERE id > last_id AND id <= last_id + 5000
+      AND normalized_email IS NULL;
+    last_id := last_id + 5000;
+    RAISE NOTICE 'processed up to id %', last_id;
+    COMMIT;
+    PERFORM pg_sleep(0.1);
+  END LOOP;
+END $$;
+```
+
+Range batches (`id > a AND id <= b`) cope with gaps in ids; they just process fewer rows in sparse ranges. After a large backfill, run `ANALYZE` on the table (plain `VACUUM` too if many rows changed); never `VACUUM FULL` on a live table.
+
+## In application code
+
+Use the project's language and ORM, keep the same rules: fetch ids in keyset order, update a batch, commit, repeat.
+
+- **Rails:** `Model.where(new_col: nil).in_batches(of: 5_000) { |rel| rel.update_all(...) }` in a data migration or a rake task, with `disable_ddl_transaction!` if it lives in a migration.
+- **Django:** a `RunPython` migration (or management command) that loops over `values_list("pk", flat=True)` in slices and calls `bulk_update` or `update()`; set `atomic = False` on the migration so each batch commits.
+- **Alembic:** a separate revision that runs batched `UPDATE` statements through `op.get_bind()`, or a standalone script.
+- **Node ORMs (Prisma, Drizzle, Knex, Kysely, TypeORM):** a script in the project's scripts folder that loops `findMany` / `select` with `where id > lastId order by id take n`, then updates by id list; run outside long transactions.
+
+Name the script clearly, make the batch size and a dry-run mode configurable if the project's other scripts do, and document in the report how to run it after the schema migration is deployed.
+
+## Deletes and dedupes
+
+- Delete in batches by id, same as updates. Check what cascades (`ON DELETE CASCADE`) and how many child rows that pulls in.
+- For dedupes, write the rule for which row survives (oldest, most complete), repoint foreign keys to the survivor first, then delete the rest.
+- Irreversible: say so in the report and recommend a backup or an export of affected rows before running.
+
+Adapted from timescale/pg-aiguide `skills/postgres-database-migration/references/backfill-strategies.md` (Apache-2.0, commit 187be00d317a51210c132010d947f4546ffb5eef, Copyright 2025 Timescale, Inc., d/b/a Tiger Data) and affaan-m/ECC `skills/database-migrations` (MIT, commit 30b68924ae9729159f44326aa3134b1d9d67c0e2, Copyright (c) 2026 Affaan Mustafa). Changed: condensed, rewritten examples, added idempotency, resumability and ORM guidance, run-from-task restrictions.

package/library/skills/safe-database-changes/references/migration-tools.md

@@ -1,69 +1,69 @@
-# Migration tools: what is safe to run
-
-Read this when you need the commands for the project's migration tool. The project's own scripts and docs come first; these are examples. "Connects" means the command opens a connection to the configured database: check the URL first and run it only against a throwaway database.
-
-## Questions to answer for any tool
-
-1. How is a migration file created, and does creating it connect to a database?
-2. Which command applies migrations in deploys, and which ones are dev-only?
-3. Which commands can drop data (reset, push, clean, fresh, wipe)? Never run those outside a throwaway database.
-4. Does the runner wrap each migration in a transaction? If yes, how do you opt out for `CREATE INDEX CONCURRENTLY`?
-5. How does it treat renames: a real rename, or drop plus add? Does it ask an interactive question (which cannot be answered in this run)?
-6. Is there a generated artifact to commit alongside (schema dump, snapshot, generated client)?
-
-## Tools
-
-**Prisma**
-- Create: `prisma migrate dev --name <name> --create-only` writes the SQL without applying it, but still connects (it uses a shadow database). Without `--create-only` it applies the migration and may offer to reset the database on drift. `migrate dev` may refuse to run in a non-interactive shell like this one; then generate the SQL with `prisma migrate diff` from the existing migrations to the schema (flags differ between Prisma versions: read `prisma migrate diff --help`), using a throwaway database as the shadow database, and save it as a new timestamped folder under `prisma/migrations` matching the existing naming.
-- Deploy: `prisma migrate deploy`. Dev-only or destructive: `migrate reset`, `db push` (no migration file; `--accept-data-loss`, `--force-reset`). Prisma blocks resets started by AI agents until a human consents; never provide that consent yourself.
-- A renamed field becomes DROP plus ADD in the generated SQL: edit the new migration to `RENAME` (or use expand and contract).
-- After schema changes run `prisma generate` so types match. Check how the project's migration runner treats transactions before adding `CONCURRENTLY`; keep such a statement alone in its own migration and confirm on the throwaway database.
-
-**Drizzle**
-- Create: `drizzle-kit generate` (works from the schema and stored snapshots). For ambiguous renames it asks an interactive question, which cannot be answered here: write that migration by hand (`drizzle-kit generate --custom` creates an empty one) and keep the snapshot consistent.
-- Apply: `drizzle-kit migrate`. `drizzle-kit push` applies schema directly with no migration file: dev and throwaway only.
-
-**Rails (Active Record)**
-- Create: `bin/rails generate migration <Name>`. Apply: `bin/rails db:migrate`; undo: `db:rollback`. Destructive: `db:reset`, `db:drop`, `db:schema:load` on a live database.
-- Commit the updated `db/schema.rb` or `db/structure.sql`.
-- Concurrent index: `disable_ddl_transaction!` in the migration and `add_index ..., algorithm: :concurrently`. If the project uses a gem like strong_migrations, follow its messages.
-
-**Django**
-- Create: `python manage.py makemigrations <app> --noinput` (no database needed). Without `--noinput` it asks questions about renames and defaults, which nobody can answer here. With it, a rename silently becomes `RemoveField` plus `AddField` (data loss) and a missing default makes it fail: read the generated operations and write `RenameField` by hand where you meant a rename. Show SQL: `sqlmigrate <app> <number>`. Apply: `migrate`.
-- Concurrent index: `AddIndexConcurrently` from `django.contrib.postgres.operations` in a migration with `atomic = False`.
-- Contract phase: `SeparateDatabaseAndState` removes a field from the model state first and drops the column in a later migration.
-
-**Alembic (SQLAlchemy)**
-- Create: `alembic revision --autogenerate -m "<msg>"` connects to compare models with the database; `alembic revision -m` (empty) does not. Always review autogenerated operations.
-- Apply: `alembic upgrade head`; offline SQL: `alembic upgrade head --sql`. Undo: `alembic downgrade -1`.
-- Concurrent index: inside `with op.get_context().autocommit_block():` call `op.create_index(..., postgresql_concurrently=True)`.
-
-**Knex / Kysely**
-- Knex: `knex migrate:make <name>`, `migrate:latest`, `migrate:rollback`. Opt out of the transaction per file with `exports.config = { transaction: false }`.
-- Kysely: migrations are TypeScript files with `up` and `down`; type them as `Kysely<any>` so old migrations do not depend on today's schema types.
-
-**golang-migrate and plain SQL folders**
-- Paired `NNN_name.up.sql` / `.down.sql`. Keep a `CONCURRENTLY` statement alone in its file. `migrate force <version>` only marks a version after a failed run has been fixed by hand; never use it to skip a migration.
-
-**Flyway / Liquibase**
-- Applied files and changesets are checksummed: editing one breaks every environment that ran it. Add a new version instead.
-- Destructive: `flyway clean`, Liquibase `dropAll`. Liquibase changesets that need to run outside a transaction use `runInTransaction="false"`.
-
-**Entity Framework Core**
-- Create: `dotnet ef migrations add <Name>` (no database needed). Offline SQL: `dotnet ef migrations script --idempotent`. Apply: `dotnet ef database update`.
-- Check the generated `Up` for `DropColumn` plus `AddColumn` where you meant `RenameColumn`. Raw SQL that must run outside a transaction: `migrationBuilder.Sql(..., suppressTransaction: true)`.
-
-**Laravel**
-- Create: `php artisan make:migration <name>`. Apply: `php artisan migrate`. Destructive: `migrate:fresh`, `migrate:reset`, `migrate:refresh`, `db:wipe`.
-
-## Getting a throwaway database
-
-In order of preference:
-
-1. The project's own test database setup (a test script, testcontainers, a compose file for tests). Read it first to confirm it creates and destroys its own database.
-2. A container you start yourself, only if Docker and the database image are already present on the machine: unique name, random host port, removed at the end (`docker run --rm -d --name tmp-migrate-<random> -p 127.0.0.1::5432 -e POSTGRES_PASSWORD=<random> postgres:<version the project uses>`, then `docker port` to find the port, and `docker rm -f` when done). Point the tool at it with an explicit URL on the command line or in the environment of that one command, never by editing the project's `.env`. Do not touch containers you did not start.
-3. SQLite projects: a copy or a new file in a temp folder.
-
-If none of these is available, do not apply the migration. Generate it, read it, and report that it was not applied.
-
-Written for this library. Tool behaviour summarised from the tools' public documentation; the Prisma agent-consent note restates prisma/skills `prisma-cli/references/agent-safety.md` (MIT, commit ba3f9d73200bbb62e49733ab93a961fd138a2c18, Copyright (c) 2025 Prisma) in our own words.
+# Migration tools: what is safe to run
+
+Read this when you need the commands for the project's migration tool. The project's own scripts and docs come first; these are examples. "Connects" means the command opens a connection to the configured database: check the URL first and run it only against a throwaway database.
+
+## Questions to answer for any tool
+
+1. How is a migration file created, and does creating it connect to a database?
+2. Which command applies migrations in deploys, and which ones are dev-only?
+3. Which commands can drop data (reset, push, clean, fresh, wipe)? Never run those outside a throwaway database.
+4. Does the runner wrap each migration in a transaction? If yes, how do you opt out for `CREATE INDEX CONCURRENTLY`?
+5. How does it treat renames: a real rename, or drop plus add? Does it ask an interactive question (which cannot be answered in this run)?
+6. Is there a generated artifact to commit alongside (schema dump, snapshot, generated client)?
+
+## Tools
+
+**Prisma**
+- Create: `prisma migrate dev --name <name> --create-only` writes the SQL without applying it, but still connects (it uses a shadow database). Without `--create-only` it applies the migration and may offer to reset the database on drift. `migrate dev` may refuse to run in a non-interactive shell like this one; then generate the SQL with `prisma migrate diff` from the existing migrations to the schema (flags differ between Prisma versions: read `prisma migrate diff --help`), using a throwaway database as the shadow database, and save it as a new timestamped folder under `prisma/migrations` matching the existing naming.
+- Deploy: `prisma migrate deploy`. Dev-only or destructive: `migrate reset`, `db push` (no migration file; `--accept-data-loss`, `--force-reset`). Prisma blocks resets started by AI agents until a human consents; never provide that consent yourself.
+- A renamed field becomes DROP plus ADD in the generated SQL: edit the new migration to `RENAME` (or use expand and contract).
+- After schema changes run `prisma generate` so types match. Check how the project's migration runner treats transactions before adding `CONCURRENTLY`; keep such a statement alone in its own migration and confirm on the throwaway database.
+
+**Drizzle**
+- Create: `drizzle-kit generate` (works from the schema and stored snapshots). For ambiguous renames it asks an interactive question, which cannot be answered here: write that migration by hand (`drizzle-kit generate --custom` creates an empty one) and keep the snapshot consistent.
+- Apply: `drizzle-kit migrate`. `drizzle-kit push` applies schema directly with no migration file: dev and throwaway only.
+
+**Rails (Active Record)**
+- Create: `bin/rails generate migration <Name>`. Apply: `bin/rails db:migrate`; undo: `db:rollback`. Destructive: `db:reset`, `db:drop`, `db:schema:load` on a live database.
+- Commit the updated `db/schema.rb` or `db/structure.sql`.
+- Concurrent index: `disable_ddl_transaction!` in the migration and `add_index ..., algorithm: :concurrently`. If the project uses a gem like strong_migrations, follow its messages.
+
+**Django**
+- Create: `python manage.py makemigrations <app> --noinput` (no database needed). Without `--noinput` it asks questions about renames and defaults, which nobody can answer here. With it, a rename silently becomes `RemoveField` plus `AddField` (data loss) and a missing default makes it fail: read the generated operations and write `RenameField` by hand where you meant a rename. Show SQL: `sqlmigrate <app> <number>`. Apply: `migrate`.
+- Concurrent index: `AddIndexConcurrently` from `django.contrib.postgres.operations` in a migration with `atomic = False`.
+- Contract phase: `SeparateDatabaseAndState` removes a field from the model state first and drops the column in a later migration.
+
+**Alembic (SQLAlchemy)**
+- Create: `alembic revision --autogenerate -m "<msg>"` connects to compare models with the database; `alembic revision -m` (empty) does not. Always review autogenerated operations.
+- Apply: `alembic upgrade head`; offline SQL: `alembic upgrade head --sql`. Undo: `alembic downgrade -1`.
+- Concurrent index: inside `with op.get_context().autocommit_block():` call `op.create_index(..., postgresql_concurrently=True)`.
+
+**Knex / Kysely**
+- Knex: `knex migrate:make <name>`, `migrate:latest`, `migrate:rollback`. Opt out of the transaction per file with `exports.config = { transaction: false }`.
+- Kysely: migrations are TypeScript files with `up` and `down`; type them as `Kysely<any>` so old migrations do not depend on today's schema types.
+
+**golang-migrate and plain SQL folders**
+- Paired `NNN_name.up.sql` / `.down.sql`. Keep a `CONCURRENTLY` statement alone in its file. `migrate force <version>` only marks a version after a failed run has been fixed by hand; never use it to skip a migration.
+
+**Flyway / Liquibase**
+- Applied files and changesets are checksummed: editing one breaks every environment that ran it. Add a new version instead.
+- Destructive: `flyway clean`, Liquibase `dropAll`. Liquibase changesets that need to run outside a transaction use `runInTransaction="false"`.
+
+**Entity Framework Core**
+- Create: `dotnet ef migrations add <Name>` (no database needed). Offline SQL: `dotnet ef migrations script --idempotent`. Apply: `dotnet ef database update`.
+- Check the generated `Up` for `DropColumn` plus `AddColumn` where you meant `RenameColumn`. Raw SQL that must run outside a transaction: `migrationBuilder.Sql(..., suppressTransaction: true)`.
+
+**Laravel**
+- Create: `php artisan make:migration <name>`. Apply: `php artisan migrate`. Destructive: `migrate:fresh`, `migrate:reset`, `migrate:refresh`, `db:wipe`.
+
+## Getting a throwaway database
+
+In order of preference:
+
+1. The project's own test database setup (a test script, testcontainers, a compose file for tests). Read it first to confirm it creates and destroys its own database.
+2. A container you start yourself, only if Docker and the database image are already present on the machine: unique name, random host port, removed at the end (`docker run --rm -d --name tmp-migrate-<random> -p 127.0.0.1::5432 -e POSTGRES_PASSWORD=<random> postgres:<version the project uses>`, then `docker port` to find the port, and `docker rm -f` when done). Point the tool at it with an explicit URL on the command line or in the environment of that one command, never by editing the project's `.env`. Do not touch containers you did not start.
+3. SQLite projects: a copy or a new file in a temp folder.
+
+If none of these is available, do not apply the migration. Generate it, read it, and report that it was not applied.
+
+Written for this library. Tool behaviour summarised from the tools' public documentation; the Prisma agent-consent note restates prisma/skills `prisma-cli/references/agent-safety.md` (MIT, commit ba3f9d73200bbb62e49733ab93a961fd138a2c18, Copyright (c) 2025 Prisma) in our own words.