@prairielearn/migrations 1.0.0 → 1.2.0

package/CHANGELOG.md

@@ -0,0 +1,24 @@
+# @prairielearn/migrations
+
+## 1.2.0
+
+### Minor Changes
+
+- 4cc962358: Add support for asynchronous batched migrations
+
+### Patch Changes
+
+- Updated dependencies [4cc962358]
+  - @prairielearn/named-locks@1.3.0
+
+## 1.1.0
+
+### Minor Changes
+
+- 400a0b901: Use automatically-renewing named lock
+- 751010ea3: Support JavaScript migration files
+
+### Patch Changes
+
+- Updated dependencies [400a0b901]
+  - @prairielearn/named-locks@1.2.0

package/README.md

@@ -0,0 +1,145 @@
+# `@prairielearn/migrations`
+
+This package runs two types of database migrations:
+
+- **Regular migrations**, which run immediately and typically change the database schema (adding/removing tables, columns, indexes, etc.)
+- **Batched migrations**, which run asynchronously over batches of data within a given table.
+
+## Usage
+
+### Regular migrations
+
+Regular migrations can be authored as either SQL or JavaScript. They should be located within one or more directories. They are uniquely identified by a 14-character timestamp at the start of their filename.
+
+```sql
+-- migrations/20230411002409_example_migration.sql
+CREATE TABLE IF NOT EXISTS
+  examples (id BIGSERIAL PRIMARY KEY, value TEXT NOT NULL);
+```
+
+```ts
+// migrations/20230411002409_example_migration.ts
+module.exports = async function () {
+  console.log('something useful.');
+};
+```
+
+### Batched migrations
+
+Batched migrations are useful for when one needs to make changes to many rows within a table, for instance backfilling a new column from existing data. While one could technically do this with the schema migrations machinery, that has a number of disadvantages:
+
+- Doing an update all in one go (e.g. `UPDATE table_name SET column = 'some value' WHERE column IS NULL`) has the potential to lock the table for a long time. For zero-downtime deploys, this is unacceptable.
+- Schema migrations are expected to run synchronously during deploy. So even if you wrote JavaScript code to manually batch up a table to avoid locks, you'd have to babysit a long-running process.
+- If errors are encountered, you'll have to figure out how to manually retry the change for the failing batches.
+
+By using batched migrations, these problems are avoided:
+
+- Work is done in small batches, so large numbers of rows (or even entire tables) are not locked for long periods of time.
+- Work is done asynchronously in the background, so migrations that operate on very large tables won't block deploys.
+- Each batch is tracked independently and failing batched can be easily retried.
+
+#### Writing batched migrations
+
+Batched migrations are written as an object with two functions:
+
+- `getParameters()`: returns the minimum and maximum IDs to operate on, as well as a batch size. `min` defaults to `1` and `batchSize` defaults to `1000`. If `max === null`, that indicates that there are no rows to operate on.
+- `execute(min, max)`: runs the migration on the given range of IDs, inclusive of its endpoints.
+
+A `makeBatchedMigration()` function is available to help ensure you're writing an object with the correct shape.
+
+```ts
+// batched-migrations/20230411002409_example_migration.ts
+import { makeBatchedMigration } from '@prairielearn/migrations';
+import { queryOneRowAsync, queryAsync } from '@prairielearn/postgres';
+
+export default makeBatchedMigration({
+  async getParameters() {
+    const result = await queryOneRowAsync('SELECT MAX(id) as max from examples;', {});
+    return {
+      max: result.rows[0].max,
+      batchSize: 1000,
+    };
+  },
+
+  async execute(min: bigint, max: bigint) {
+    await queryAsync('UPDATE examples SET text = TRIM(text) WHERE id >= $min AND id <= $max', {
+      min,
+      max,
+    });
+  },
+});
+```
+
+Batched migration `execute()` functions **must** be idempotent, as they may run multiple times on the same ID range in the case of retries after failure.
+
+#### Executing batched migrations
+
+Unlike regular migrations, batched migrations aren't automatically started. Instead, you must write a regular migration to call `enqueueBatchedMigration()` to explicitly start a given batched migration. This provides precise control over execution order.
+
+```ts
+// migrations/20230411002409_start_batched_migration__example_migration.ts
+import { enqueueBatchedMigration } from '@prairielearn/migrations';
+
+export default async function () {
+  await enqueueBatchedMigration('20230411002409_example_migration');
+}
+```
+
+This will queue the batched migration for execution.
+
+You may need to ensure that a given batched migration has succeeded before running a subsequent regular migration. For instance, you might have a batched migration that copies a column from one table to another, and you want to ensure that all data has been copied before you delete the original column. You can achieve this by "finalizing" the migration with `finalizeBatchedMigration()`. This will synchronously execute any remaining batches, and will error if the migration ends up in a failed state. This gives you a chance to fix any errors and retry the failed jobs.
+
+```ts
+// migrations/20230411002409_finalize_batched_migration__example_migration.ts
+import { finalizeBatchedMigration } from '@prairielearn/migrations';
+
+export default async function () {
+  await finalizeBatchedMigration('20230411002409_finalize_batched_migration__example_migration');
+}
+```
+
+In most cases, you'll want to do your best to ensure that the given batched migration has finished _before_ deploying a migration that finalizes it. That way, `finalizeBatchedMigration()` will just have to assert that the migration has already successfully executed. However, finalizing a migration is still an important part of preventing data loss or inconsistencies for many migrations.
+
+### Server setup
+
+To execute any pending regular migrations, call `init()` early on in your application startup code. The first argument is an array of directory paths containing migration files as described above. The second argument is a project identifier, which is used to isolate multiple migration sequences from each other when two or more applications share a single database.
+
+```ts
+import { init } from '@prairielearn/migrations';
+
+await init([path.join(__dirname, 'migrations')], 'prairielearn');
+```
+
+If you want to make use of batched migrations, you'll need to do some additional setup. Since batched migrations are typically used with regular migrations, you'll need to take care to call `init()` after `initBatchedMigrations()` but before `startBatchedMigrations()`.
+
+```ts
+import {
+  init,
+  initBatchedMigrations,
+  startBatchedMigrations,
+  stopBatchedMigrations,
+} from '@prairielearn/migrations';
+
+const runner = initBatchedMigrations({
+  project: 'prairielearn',
+  directories: [path.join(__dirname, 'batched-migrations')],
+});
+runner.on('error', (error) => {
+  // Handle error, e.g. by reporting to Sentry.
+});
+
+await init([path.join(__dirname, 'migrations')], 'prairielearn');
+
+startBatchedMigrations({
+  workDurationMs: 60_000,
+  sleepDurationMs: 30_000,
+});
+```
+
+If you want to gracefully shut down your server, you can stop processing batched migrations and wait for any in-progress jobs to finish.
+
+```ts
+import { stopBatchedMigrations } from '@prairielearn/migrations';
+
+await stopBatchedMigrations();
+```

package/dist/batched-migrations/batched-migration-job.d.ts

@@ -0,0 +1,42 @@
+import { z } from 'zod';
+export declare const BatchedMigrationJobStatusSchema: z.ZodEnum<["pending", "failed", "succeeded"]>;
+export type BatchedMigrationJobStatus = z.infer<typeof BatchedMigrationJobStatusSchema>;
+export declare const BatchedMigrationJobRowSchema: z.ZodObject<{
+    id: z.ZodString;
+    batched_migration_id: z.ZodString;
+    min_value: z.ZodBigInt;
+    max_value: z.ZodBigInt;
+    status: z.ZodEnum<["pending", "failed", "succeeded"]>;
+    attempts: z.ZodNumber;
+    created_at: z.ZodDate;
+    updated_at: z.ZodDate;
+    started_at: z.ZodNullable<z.ZodDate>;
+    finished_at: z.ZodNullable<z.ZodDate>;
+    data: z.ZodUnknown;
+}, "strip", z.ZodTypeAny, {
+    status: "pending" | "failed" | "succeeded";
+    id: string;
+    min_value: bigint;
+    max_value: bigint;
+    created_at: Date;
+    updated_at: Date;
+    started_at: Date | null;
+    batched_migration_id: string;
+    attempts: number;
+    finished_at: Date | null;
+    data?: unknown;
+}, {
+    status: "pending" | "failed" | "succeeded";
+    id: string;
+    min_value: bigint;
+    max_value: bigint;
+    created_at: Date;
+    updated_at: Date;
+    started_at: Date | null;
+    batched_migration_id: string;
+    attempts: number;
+    finished_at: Date | null;
+    data?: unknown;
+}>;
+export type BatchedMigrationJobRow = z.infer<typeof BatchedMigrationJobRowSchema>;
+export declare function selectRecentJobsWithStatus(batchedMigrationId: string, status: BatchedMigrationJobStatus, limit: number): Promise<BatchedMigrationJobRow[]>;

package/dist/batched-migrations/batched-migration-job.js

@@ -0,0 +1,25 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.selectRecentJobsWithStatus = exports.BatchedMigrationJobRowSchema = exports.BatchedMigrationJobStatusSchema = void 0;
+const zod_1 = require("zod");
+const postgres_1 = require("@prairielearn/postgres");
+const sql = (0, postgres_1.loadSqlEquiv)(__filename);
+exports.BatchedMigrationJobStatusSchema = zod_1.z.enum(['pending', 'failed', 'succeeded']);
+exports.BatchedMigrationJobRowSchema = zod_1.z.object({
+    id: zod_1.z.string(),
+    batched_migration_id: zod_1.z.string(),
+    min_value: zod_1.z.bigint({ coerce: true }),
+    max_value: zod_1.z.bigint({ coerce: true }),
+    status: exports.BatchedMigrationJobStatusSchema,
+    attempts: zod_1.z.number(),
+    created_at: zod_1.z.date(),
+    updated_at: zod_1.z.date(),
+    started_at: zod_1.z.date().nullable(),
+    finished_at: zod_1.z.date().nullable(),
+    data: zod_1.z.unknown(),
+});
+async function selectRecentJobsWithStatus(batchedMigrationId, status, limit) {
+    return (0, postgres_1.queryValidatedRows)(sql.select_recent_jobs_with_status, { batched_migration_id: batchedMigrationId, status, limit }, exports.BatchedMigrationJobRowSchema);
+}
+exports.selectRecentJobsWithStatus = selectRecentJobsWithStatus;
+//# sourceMappingURL=batched-migration-job.js.map

package/dist/batched-migrations/batched-migration-job.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"batched-migration-job.js","sourceRoot":"","sources":["../../src/batched-migrations/batched-migration-job.ts"],"names":[],"mappings":";;;AAAA,6BAAwB;AACxB,qDAA0E;AAE1E,MAAM,GAAG,GAAG,IAAA,uBAAY,EAAC,UAAU,CAAC,CAAC;AAExB,QAAA,+BAA+B,GAAG,OAAC,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC,CAAC;AAG7E,QAAA,4BAA4B,GAAG,OAAC,CAAC,MAAM,CAAC;IACnD,EAAE,EAAE,OAAC,CAAC,MAAM,EAAE;IACd,oBAAoB,EAAE,OAAC,CAAC,MAAM,EAAE;IAChC,SAAS,EAAE,OAAC,CAAC,MAAM,CAAC,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;IACrC,SAAS,EAAE,OAAC,CAAC,MAAM,CAAC,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;IACrC,MAAM,EAAE,uCAA+B;IACvC,QAAQ,EAAE,OAAC,CAAC,MAAM,EAAE;IACpB,UAAU,EAAE,OAAC,CAAC,IAAI,EAAE;IACpB,UAAU,EAAE,OAAC,CAAC,IAAI,EAAE;IACpB,UAAU,EAAE,OAAC,CAAC,IAAI,EAAE,CAAC,QAAQ,EAAE;IAC/B,WAAW,EAAE,OAAC,CAAC,IAAI,EAAE,CAAC,QAAQ,EAAE;IAChC,IAAI,EAAE,OAAC,CAAC,OAAO,EAAE;CAClB,CAAC,CAAC;AAGI,KAAK,UAAU,0BAA0B,CAC9C,kBAA0B,EAC1B,MAAiC,EACjC,KAAa;IAEb,OAAO,IAAA,6BAAkB,EACvB,GAAG,CAAC,8BAA8B,EAClC,EAAE,oBAAoB,EAAE,kBAAkB,EAAE,MAAM,EAAE,KAAK,EAAE,EAC3D,oCAA4B,CAC7B,CAAC;AACJ,CAAC;AAVD,gEAUC"}

package/dist/batched-migrations/batched-migration-job.sql

@@ -0,0 +1,12 @@
+-- BLOCK select_recent_jobs_with_status
+SELECT
+  *
+FROM
+  batched_migration_jobs
+WHERE
+  batched_migration_id = $batched_migration_id
+  AND status = $status
+ORDER BY
+  max_value DESC
+LIMIT
+  $limit;

package/dist/batched-migrations/batched-migration-runner.d.ts

@@ -0,0 +1,28 @@
+import { BatchedMigrationRow, BatchedMigrationImplementation } from './batched-migration';
+interface BatchedMigrationRunnerOptions {
+    logProgress?: boolean;
+}
+export declare class BatchedMigrationRunner {
+    private options;
+    private migration;
+    private migrationImplementation;
+    private migrationStatus;
+    constructor(migration: BatchedMigrationRow, migrationImplementation: BatchedMigrationImplementation, options?: BatchedMigrationRunnerOptions);
+    private log;
+    private hasIncompleteJobs;
+    private hasFailedJobs;
+    private refreshMigrationStatus;
+    private finishRunningMigration;
+    private getNextBatchBounds;
+    private startJob;
+    private serializeJobData;
+    private finishJob;
+    private getOrCreateNextMigrationJob;
+    private runMigrationJob;
+    run({ signal, iterations, durationMs, }?: {
+        signal?: AbortSignal;
+        iterations?: number;
+        durationMs?: number;
+    }): Promise<void>;
+}
+export {};

package/dist/batched-migrations/batched-migration-runner.js

@@ -0,0 +1,136 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.BatchedMigrationRunner = void 0;
+const postgres_1 = require("@prairielearn/postgres");
+const logger_1 = require("@prairielearn/logger");
+const serialize_error_1 = require("serialize-error");
+const zod_1 = require("zod");
+const batched_migration_1 = require("./batched-migration");
+const batched_migration_job_1 = require("./batched-migration-job");
+const sql = (0, postgres_1.loadSqlEquiv)(__filename);
+class BatchedMigrationRunner {
+    constructor(migration, migrationImplementation, options = {}) {
+        this.options = options;
+        this.migration = migration;
+        this.migrationImplementation = migrationImplementation;
+        this.migrationStatus = migration.status;
+    }
+    log(message, ...meta) {
+        if (this.options.logProgress) {
+            logger_1.logger.info(`[${this.migration.filename}] ${message}`, ...meta);
+        }
+    }
+    async hasIncompleteJobs(migration) {
+        return (0, postgres_1.queryValidatedSingleColumnOneRow)(sql.batched_migration_has_incomplete_jobs, { batched_migration_id: migration.id }, zod_1.z.boolean());
+    }
+    async hasFailedJobs(migration) {
+        return (0, postgres_1.queryValidatedSingleColumnOneRow)(sql.batched_migration_has_failed_jobs, { batched_migration_id: migration.id }, zod_1.z.boolean());
+    }
+    async refreshMigrationStatus(migration) {
+        this.migrationStatus = await (0, postgres_1.queryValidatedSingleColumnOneRow)(sql.get_migration_status, {
+            id: migration.id,
+        }, batched_migration_1.BatchedMigrationStatusSchema);
+    }
+    async finishRunningMigration(migration) {
+        // Safety check: if there are any pending jobs, don't mark this
+        // migration as finished.
+        if (await this.hasIncompleteJobs(migration)) {
+            this.log(`Incomplete jobs found, not marking as finished`);
+            return;
+        }
+        const hasFailedJobs = await this.hasFailedJobs(migration);
+        const finalStatus = hasFailedJobs ? 'failed' : 'succeeded';
+        await (0, batched_migration_1.updateBatchedMigrationStatus)(migration.id, finalStatus);
+        this.log(`Finished with status '${finalStatus}'`);
+    }
+    async getNextBatchBounds(migration) {
+        const lastJob = await (0, postgres_1.queryValidatedZeroOrOneRow)(sql.select_last_batched_migration_job, { batched_migration_id: migration.id }, batched_migration_job_1.BatchedMigrationJobRowSchema);
+        const nextMin = lastJob ? lastJob.max_value + 1n : migration.min_value;
+        if (nextMin > migration.max_value)
+            return null;
+        let nextMax = nextMin + BigInt(migration.batch_size) - 1n;
+        if (nextMax > migration.max_value)
+            nextMax = migration.max_value;
+        return [nextMin, nextMax];
+    }
+    async startJob(job) {
+        await (0, postgres_1.queryAsync)(sql.start_batched_migration_job, { id: job.id });
+        const jobRange = `[${job.min_value}, ${job.max_value}]`;
+        const migrationRange = `[${this.migration.min_value}, ${this.migration.max_value}]`;
+        this.log(`Started job ${job.id} for range ${jobRange} in ${migrationRange}`);
+    }
+    serializeJobData(data) {
+        if (data == null)
+            return null;
+        // Return JSON-stringified data. Convert BigInts to strings.
+        return JSON.stringify(data, (_key, value) => {
+            if (typeof value === 'bigint')
+                return value.toString();
+            return value;
+        });
+    }
+    async finishJob(job, status, data) {
+        await (0, postgres_1.queryAsync)(sql.finish_batched_migration_job, {
+            id: job.id,
+            status,
+            data: this.serializeJobData(data),
+        });
+        this.log(`Job ${job.id} finished with status '${status}'`);
+    }
+    async getOrCreateNextMigrationJob(migration) {
+        const nextBatchBounds = await this.getNextBatchBounds(migration);
+        if (nextBatchBounds) {
+            return (0, postgres_1.queryValidatedOneRow)(sql.insert_batched_migration_job, {
+                batched_migration_id: migration.id,
+                min_value: nextBatchBounds[0],
+                max_value: nextBatchBounds[1],
+            }, batched_migration_job_1.BatchedMigrationJobRowSchema);
+        }
+        else {
+            // Pick up any old pending jobs from this migration. These will only exist if
+            // an admin manually elected to retry all failed jobs; we'll never automatically
+            // transition failed jobs back to pending.
+            return (0, postgres_1.queryValidatedZeroOrOneRow)(sql.select_first_pending_batched_migration_job, { batched_migration_id: migration.id }, batched_migration_job_1.BatchedMigrationJobRowSchema);
+        }
+    }
+    async runMigrationJob(migration, migrationImplementation) {
+        const nextJob = await this.getOrCreateNextMigrationJob(migration);
+        if (nextJob) {
+            await this.startJob(nextJob);
+            let error = null;
+            try {
+                // We'll only handle errors thrown by the migration itself. If any of
+                // our own execution machinery throws an error, we'll let it bubble up.
+                await migrationImplementation.execute(nextJob.min_value, nextJob.max_value);
+            }
+            catch (err) {
+                error = err;
+            }
+            if (error) {
+                await this.finishJob(nextJob, 'failed', { error: (0, serialize_error_1.serializeError)(error) });
+            }
+            else {
+                await this.finishJob(nextJob, 'succeeded');
+            }
+        }
+        else {
+            await this.finishRunningMigration(migration);
+        }
+    }
+    async run({ signal, iterations, durationMs, } = {}) {
+        let iterationCount = 0;
+        const endTime = durationMs ? Date.now() + durationMs : null;
+        while (!signal?.aborted &&
+            (iterations ? iterationCount < iterations : true) &&
+            (endTime ? Date.now() < endTime : true) &&
+            (this.migrationStatus === 'running' || this.migrationStatus === 'finalizing')) {
+            await this.runMigrationJob(this.migration, this.migrationImplementation);
+            iterationCount += 1;
+            // Always refresh the status so we can detect if the migration was marked
+            // as paused by another process.
+            await this.refreshMigrationStatus(this.migration);
+        }
+    }
+}
+exports.BatchedMigrationRunner = BatchedMigrationRunner;
+//# sourceMappingURL=batched-migration-runner.js.map

package/dist/batched-migrations/batched-migration-runner.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"batched-migration-runner.js","sourceRoot":"","sources":["../../src/batched-migrations/batched-migration-runner.ts"],"names":[],"mappings":";;;AAAA,qDAMgC;AAChC,iDAA8C;AAC9C,qDAAiD;AACjD,6BAAwB;AAExB,2DAM6B;AAC7B,mEAIiC;AAEjC,MAAM,GAAG,GAAG,IAAA,uBAAY,EAAC,UAAU,CAAC,CAAC;AAMrC,MAAa,sBAAsB;IAMjC,YACE,SAA8B,EAC9B,uBAAuD,EACvD,UAAyC,EAAE;QAE3C,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;QACvB,IAAI,CAAC,SAAS,GAAG,SAAS,CAAC;QAC3B,IAAI,CAAC,uBAAuB,GAAG,uBAAuB,CAAC;QACvD,IAAI,CAAC,eAAe,GAAG,SAAS,CAAC,MAAM,CAAC;IAC1C,CAAC;IAEO,GAAG,CAAC,OAAe,EAAE,GAAG,IAAW;QACzC,IAAI,IAAI,CAAC,OAAO,CAAC,WAAW,EAAE;YAC5B,eAAM,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,SAAS,CAAC,QAAQ,KAAK,OAAO,EAAE,EAAE,GAAG,IAAI,CAAC,CAAC;SACjE;IACH,CAAC;IAEO,KAAK,CAAC,iBAAiB,CAAC,SAA8B;QAC5D,OAAO,IAAA,2CAAgC,EACrC,GAAG,CAAC,qCAAqC,EACzC,EAAE,oBAAoB,EAAE,SAAS,CAAC,EAAE,EAAE,EACtC,OAAC,CAAC,OAAO,EAAE,CACZ,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,aAAa,CAAC,SAA8B;QACxD,OAAO,IAAA,2CAAgC,EACrC,GAAG,CAAC,iCAAiC,EACrC,EAAE,oBAAoB,EAAE,SAAS,CAAC,EAAE,EAAE,EACtC,OAAC,CAAC,OAAO,EAAE,CACZ,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,sBAAsB,CAAC,SAA8B;QACjE,IAAI,CAAC,eAAe,GAAG,MAAM,IAAA,2CAAgC,EAC3D,GAAG,CAAC,oBAAoB,EACxB;YACE,EAAE,EAAE,SAAS,CAAC,EAAE;SACjB,EACD,gDAA4B,CAC7B,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,sBAAsB,CAAC,SAA8B;QACjE,+DAA+D;QAC/D,yBAAyB;QACzB,IAAI,MAAM,IAAI,CAAC,iBAAiB,CAAC,SAAS,CAAC,EAAE;YAC3C,IAAI,CAAC,GAAG,CAAC,gDAAgD,CAAC,CAAC;YAC3D,OAAO;SACR;QAED,MAAM,aAAa,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC1D,MAAM,WAAW,GAAG,aAAa,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,WAAW,CAAC;QAC3D,MAAM,IAAA,gDAA4B,EAAC,SAAS,CAAC,EAAE,EAAE,WAAW,CAAC,CAAC;QAC9D,IAAI,CAAC,GAAG,CAAC,yBAAyB,WAAW,GAAG,CAAC,CAAC;IACpD,CAAC;IAEO,KAAK,CAAC,kBAAkB,CAC9B,SAA8B;QAE9B,MAAM,OAAO,GAAG,MAAM,IAAA,qCAA0B,EAC9C,GAAG,CAAC,iCAAiC,EACrC,EAAE,oBAAoB,EAAE,SAAS,CAAC,EAAE,EAAE,EACtC,oDAA4B,CAC7B,CAAC;QAEF,MAAM,OAAO,GAAG,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,GAAG,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,SAAS,CAAC;QACvE,IAAI,OAAO,GAAG,SAAS,CAAC,SAAS;YAAE,OAAO,IAAI,CAAC;QAE/C,IAAI,OAAO,GAAG,OAAO,GAAG,MAAM,CAAC,SAAS,CAAC,UAAU,CAAC,GAAG,EAAE,CAAC;QAC1D,IAAI,OAAO,GAAG,SAAS,CAAC,SAAS;YAAE,OAAO,GAAG,SAAS,CAAC,SAAS,CAAC;QAEjE,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;IAC5B,CAAC;IAEO,KAAK,CAAC,QAAQ,CAAC,GAA2B;QAChD,MAAM,IAAA,qBAAU,EAAC,GAAG,CAAC,2BAA2B,EAAE,EAAE,EAAE,EAAE,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC;QAClE,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,SAAS,KAAK,GAAG,CAAC,SAAS,GAAG,CAAC;QACxD,MAAM,cAAc,GAAG,IAAI,IAAI,CAAC,SAAS,CAAC,SAAS,KAAK,IAAI,CAAC,SAAS,CAAC,SAAS,GAAG,CAAC;QACpF,IAAI,CAAC,GAAG,CAAC,eAAe,GAAG,CAAC,EAAE,cAAc,QAAQ,OAAO,cAAc,EAAE,CAAC,CAAC;IAC/E,CAAC;IAEO,gBAAgB,CAAC,IAAa;QACpC,IAAI,IAAI,IAAI,IAAI;YAAE,OAAO,IAAI,CAAC;QAE9B,4DAA4D;QAC5D,OAAO,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE;YAC1C,IAAI,OAAO,KAAK,KAAK,QAAQ;gBAAE,OAAO,KAAK,CAAC,QAAQ,EAAE,CAAC;YACvD,OAAO,KAAK,CAAC;QACf,CAAC,CAAC,CAAC;IACL,CAAC;IAEO,KAAK,CAAC,SAAS,CACrB,GAA2B,EAC3B,MAAkE,EAClE,IAAc;QAEd,MAAM,IAAA,qBAAU,EAAC,GAAG,CAAC,4BAA4B,EAAE;YACjD,EAAE,EAAE,GAAG,CAAC,EAAE;YACV,MAAM;YACN,IAAI,EAAE,IAAI,CAAC,gBAAgB,CAAC,IAAI,CAAC;SAClC,CAAC,CAAC;QACH,IAAI,CAAC,GAAG,CAAC,OAAO,GAAG,CAAC,EAAE,0BAA0B,MAAM,GAAG,CAAC,CAAC;IAC7D,CAAC;IAEO,KAAK,CAAC,2BAA2B,CACvC,SAA8B;QAE9B,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACjE,IAAI,eAAe,EAAE;YACnB,OAAO,IAAA,+BAAoB,EACzB,GAAG,CAAC,4BAA4B,EAChC;gBACE,oBAAoB,EAAE,SAAS,CAAC,EAAE;gBAClC,SAAS,EAAE,eAAe,CAAC,CAAC,CAAC;gBAC7B,SAAS,EAAE,eAAe,CAAC,CAAC,CAAC;aAC9B,EACD,oDAA4B,CAC7B,CAAC;SACH;aAAM;YACL,6EAA6E;YAC7E,gFAAgF;YAChF,0CAA0C;YAC1C,OAAO,IAAA,qCAA0B,EAC/B,GAAG,CAAC,0CAA0C,EAC9C,EAAE,oBAAoB,EAAE,SAAS,CAAC,EAAE,EAAE,EACtC,oDAA4B,CAC7B,CAAC;SACH;IACH,CAAC;IAEO,KAAK,CAAC,eAAe,CAC3B,SAA8B,EAC9B,uBAAuD;QAEvD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,2BAA2B,CAAC,SAAS,CAAC,CAAC;QAClE,IAAI,OAAO,EAAE;YACX,MAAM,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;YAE7B,IAAI,KAAK,GAAG,IAAI,CAAC;YACjB,IAAI;gBACF,qEAAqE;gBACrE,uEAAuE;gBACvE,MAAM,uBAAuB,CAAC,OAAO,CAAC,OAAO,CAAC,SAAS,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC;aAC7E;YAAC,OAAO,GAAG,EAAE;gBACZ,KAAK,GAAG,GAAG,CAAC;aACb;YAED,IAAI,KAAK,EAAE;gBACT,MAAM,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,QAAQ,EAAE,EAAE,KAAK,EAAE,IAAA,gCAAc,EAAC,KAAK,CAAC,EAAE,CAAC,CAAC;aAC3E;iBAAM;gBACL,MAAM,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,WAAW,CAAC,CAAC;aAC5C;SACF;aAAM;YACL,MAAM,IAAI,CAAC,sBAAsB,CAAC,SAAS,CAAC,CAAC;SAC9C;IACH,CAAC;IAED,KAAK,CAAC,GAAG,CAAC,EACR,MAAM,EACN,UAAU,EACV,UAAU,MAC4D,EAAE;QACxE,IAAI,cAAc,GAAG,CAAC,CAAC;QACvB,MAAM,OAAO,GAAG,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC;QAC5D,OACE,CAAC,MAAM,EAAE,OAAO;YAChB,CAAC,UAAU,CAAC,CAAC,CAAC,cAAc,GAAG,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC;YACjD,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC;YACvC,CAAC,IAAI,CAAC,eAAe,KAAK,SAAS,IAAI,IAAI,CAAC,eAAe,KAAK,YAAY,CAAC,EAC7E;YACA,MAAM,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,uBAAuB,CAAC,CAAC;YACzE,cAAc,IAAI,CAAC,CAAC;YACpB,yEAAyE;YACzE,gCAAgC;YAChC,MAAM,IAAI,CAAC,sBAAsB,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;SACnD;IACH,CAAC;CACF;AAxLD,wDAwLC"}

package/dist/batched-migrations/batched-migration-runner.sql

@@ -0,0 +1,93 @@
+-- BLOCK select_last_batched_migration_job
+SELECT
+  *
+FROM
+  batched_migration_jobs
+WHERE
+  batched_migration_id = $batched_migration_id
+ORDER BY
+  id DESC
+LIMIT
+  1;
+
+-- BLOCK insert_batched_migration_job
+INSERT INTO
+  batched_migration_jobs (
+    batched_migration_id,
+    status,
+    min_value,
+    max_value
+  )
+VALUES
+  (
+    $batched_migration_id,
+    'pending'::enum_batched_migration_job_status,
+    $min_value,
+    $max_value
+  )
+RETURNING
+  *;
+
+-- BLOCK start_batched_migration_job
+UPDATE batched_migration_jobs
+SET
+  attempts = attempts + 1,
+  updated_at = CURRENT_TIMESTAMP,
+  started_at = CURRENT_TIMESTAMP
+WHERE
+  id = $id;
+
+-- BLOCK finish_batched_migration_job
+UPDATE batched_migration_jobs
+SET
+  status = $status::enum_batched_migration_job_status,
+  updated_at = CURRENT_TIMESTAMP,
+  finished_at = CURRENT_TIMESTAMP,
+  data = $data
+WHERE
+  id = $id;
+
+-- BLOCK select_first_pending_batched_migration_job
+SELECT
+  *
+FROM
+  batched_migration_jobs
+WHERE
+  batched_migration_id = $batched_migration_id
+  AND status = 'pending'
+ORDER BY
+  id ASC
+LIMIT
+  1;
+
+-- BLOCK batched_migration_has_incomplete_jobs
+SELECT
+  EXISTS (
+    SELECT
+      1
+    FROM
+      batched_migration_jobs
+    WHERE
+      batched_migration_id = $batched_migration_id
+      AND status = 'pending'
+  ) as exists;
+
+-- BLOCK batched_migration_has_failed_jobs
+SELECT
+  EXISTS (
+    SELECT
+      1
+    FROM
+      batched_migration_jobs
+    WHERE
+      batched_migration_id = $batched_migration_id
+      AND status = 'failed'
+  ) as exists;
+
+-- BLOCK get_migration_status
+SELECT
+  status
+FROM
+  batched_migrations
+WHERE
+  id = $id;