@filoz/repair-cli 0.1.2 → 0.2.1

package/readme.md

@@ -4,10 +4,10 @@
 
 > Early repair for faulty service providers and datasets
 
-The `repair` CLI helps prepare and run repair jobs that move pieces away from a faulty Filecoin service provider and into a target PDP provider. It uses:
+The `repair` CLI helps prepare and run repair and replication jobs on Filecoin PDP datasets. It uses:
 
 - an indexer Postgres database as the read-only source of providers, datasets, and pieces
-- a local SQLite database to track repair jobs and per-piece operations
+- a local SQLite database to track repair and replication jobs and per-piece operations
 - a configured Filecoin wallet to create datasets and submit on-chain add-piece transactions
 
 ## Installation
@@ -37,6 +37,7 @@ Setup prompts for:
 - calibration indexer Postgres URL
 - chain, either Filecoin Mainnet `314` or Filecoin Calibration `314159`
 - local SQLite database path
+- dataset source label, used when creating repair target datasets (defaults to `early-repair`)
 
 The command stores these values in the CLI config and runs the local SQLite schema migration. It returns the configured wallet address.
 
@@ -58,7 +59,7 @@ Interactive configuration and local database setup.
 repair setup
 ```
 
-Use this whenever you need to initialize the CLI, change the active chain, update indexer URLs, or move the local SQLite database.
+Use this whenever you need to initialize the CLI, change the active chain, update indexer URLs, change the dataset source label, or move the local SQLite database.
 
 ### `repair wallet fund`
 
@@ -140,6 +141,26 @@ Filter by provider ID:
 repair datasets list --provider-id 123
 ```
 
+### `repair datasets show <id>`
+
+Shows a dataset from the configured indexer.
+
+```bash
+repair datasets show 42
+```
+
+The output includes the dataset record, its provider, and active pieces.
+
+### `repair datasets terminate <id>`
+
+Terminates a dataset owned by the configured wallet.
+
+```bash
+repair datasets terminate 42
+```
+
+The command submits the on-chain terminate transaction, waits for it to be mined, and returns the result.
+
 ### `repair repair create`
 
 Creates a local repair plan for a source provider and a target provider.
@@ -165,12 +186,14 @@ repair repair list
 Each repair includes:
 
 - repair ID and status
-- source provider ID
+- repair provider ID
 - target provider ID and target provider URL
 - target dataset ID, when one has been created or found
 - block number used when the repair was created
 - total operations and counts by `pending`, `failed`, `completed`, and `skipped`
 
+This command lists provider-level repairs only. Dataset replications are listed with `repair replicate list`.
+
 ### `repair repair run <repairId>`
 
 Runs a pending repair.
@@ -179,18 +202,17 @@ Runs a pending repair.
 repair repair run 1
 ```
 
-The command first ensures the target repair dataset exists for the configured wallet and target provider. If no matching dataset exists, it creates one with IPFS indexing enabled and CDN disabled. Then it processes pending `add_piece` operations by pulling pieces from alternate providers into the target provider and committing them on-chain.
+The command first ensures the target repair dataset exists for the configured wallet and target provider. If no matching dataset exists, it creates one with IPFS indexing enabled and CDN disabled, using the configured source label. Then it processes pending `add_piece` operations by pulling pieces from alternate providers into the target provider and committing them on-chain.
 
 Options:
 
 - `--concurrency <number>` controls how many pull batches run at once. Defaults to `4`.
 - `--batch-size <number>` controls the maximum number of `add_piece` operations per batch. Defaults to `40`.
-- `--reset` retries failed `add_piece` operations as well as pending operations.
 
 Example:
 
 ```bash
-repair repair run 1 --concurrency 8 --batch-size 40 --reset
+repair repair run 1 --concurrency 8 --batch-size 40
 ```
 
 ### `repair repair delete <repairId>`
@@ -203,6 +225,68 @@ repair repair delete 1
 
 This only deletes local SQLite state. It does not delete on-chain datasets or remove pieces from a provider.
 
+### `repair replicate create`
+
+Creates a local replication plan for a source dataset and a target provider.
+
+```bash
+repair replicate create --data-set-id 42 --target-provider-id 202
+```
+
+`--data-set-id` is the source dataset whose pieces should be replicated.
+
+`--target-provider-id` is the provider that should receive the replicated pieces. It must be different from the source dataset's provider.
+
+The command snapshots the current chain block number, creates a local replication row, and creates local `add_piece` operations for every active piece in the source dataset. Pieces are pulled from the source dataset's provider, not from alternate providers. The command returns a `replicateId`.
+
+### `repair replicate list`
+
+Lists local dataset replications.
+
+```bash
+repair replicate list
+```
+
+Each replication includes:
+
+- replication ID and status
+- source provider ID and source dataset ID
+- target provider ID and target provider URL
+- target dataset ID, when one has been created
+- block number used when the replication was created
+- total operations and counts by `pending`, `failed`, `completed`, and `skipped`
+
+### `repair replicate run <replicateId>`
+
+Runs a pending replication.
+
+```bash
+repair replicate run 1
+```
+
+The command creates a new target dataset on the target provider, copying CDN and metadata settings from the source dataset. Then it processes pending `add_piece` operations by pulling pieces from the source provider into the target provider and committing them on-chain.
+
+Options:
+
+- `--concurrency <number>` controls how many pull batches run at once. Defaults to `4`.
+- `--batch-size <number>` controls the maximum number of `add_piece` operations per batch. Defaults to `40`.
+
+Example:
+
+```bash
+repair replicate run 1 --concurrency 8 --batch-size 40
+```
+
+### `repair replicate delete <replicateId>`
+
+Deletes a local replication and its operations.
+
+```bash
+repair replicate delete 1
+```
+
+This only deletes local SQLite state. It does not delete on-chain datasets or remove pieces from a provider.
+
 ## Typical Workflow
 
 1. Configure the CLI.
@@ -238,6 +322,25 @@ repair repair list
 repair repair run 1
 ```
 
+### Replicating a dataset
+
+Use replication when you want to copy an entire dataset to another provider, rather than repairing all pieces on a faulty provider.
+
+1. Find the source dataset.
+
+```bash
+repair datasets list
+repair datasets show 42
+```
+
+1. Create, inspect, and run the replication.
+
+```bash
+repair replicate create --data-set-id 42 --target-provider-id 202
+repair replicate list
+repair replicate run 1
+```
+
 ## Contributing
 
 Read contributing [guidelines](../../.github/CONTRIBUTING.md).

package/src/cli.ts

@@ -3,6 +3,7 @@ import { Cli } from 'incur'
 import { datasets } from './commands/datasets.ts'
 import { providers } from './commands/providers.ts'
 import { repair } from './commands/repair.ts'
+import { replicate } from './commands/replicate.ts'
 import { setup } from './commands/setup.ts'
 import { wallet } from './commands/wallet.ts'
 import { version } from './utils.ts'
@@ -15,6 +16,7 @@ const cli = Cli.create('repair', {
 cli.command(setup)
 cli.command(wallet)
 cli.command(repair)
+cli.command(replicate)
 cli.command(datasets)
 cli.command(providers)
 cli.serve()

package/src/commands/datasets.ts

@@ -1,7 +1,7 @@
 import * as p from '@clack/prompts'
 import * as SP from '@filoz/synapse-core/sp'
 import { getPdpDataSet } from '@filoz/synapse-core/warm-storage'
-import { and, eq } from 'drizzle-orm'
+import { and, asc, eq } from 'drizzle-orm'
 import { Cli, z } from 'incur'
 import { contextMiddleware, contextSchema } from '../middleware.ts'
 import { globalOptions, hashLink } from '../utils.ts'
@@ -32,8 +32,11 @@ datasets.command('list', {
         where: and(...conditions),
         with: {
           provider: true,
-          pieces: true,
+          pieces: {
+            where: eq(c.var.indexerDb._.fullSchema.pieces.removed, false),
+          },
         },
+        orderBy: [asc(schema.dataSets.dataSetId)],
       })
 
       const datasetsFlattened = datasets.map((dataset) => {
@@ -133,7 +136,16 @@ datasets.command('show', {
   middleware: [contextMiddleware],
   run: async (c) => {
     try {
-      const dataset = await getPdpDataSet(c.var.client, { dataSetId: c.args.id })
+      const dataset = await c.var.indexerDb.query.dataSets.findFirst({
+        where: eq(c.var.indexerDb._.fullSchema.dataSets.dataSetId, c.args.id),
+        with: {
+          provider: true,
+          pieces: {
+            where: eq(c.var.indexerDb._.fullSchema.pieces.removed, false),
+          },
+        },
+      })
+
       if (!dataset) {
         return c.error({
           code: 'DATASET_NOT_FOUND',
@@ -141,7 +153,6 @@ datasets.command('show', {
           retryable: false,
         })
       }
-
       return c.ok(dataset)
     } catch (error) {
       if (c.options.debug) {

package/src/commands/providers.ts

@@ -30,7 +30,6 @@ providers.command('list', {
       const filters: (SQLWrapper | undefined)[] = [
         eq(schema.providers.providerActive, true),
         eq(schema.providers.pdpProductActive, true),
-        // or(eq(schema.providers.approved, true), eq(schema.providers.endorsed, true)),
       ]
       if (!c.options.all) {
         filters.push(or(eq(schema.providers.approved, true), eq(schema.providers.endorsed, true)))

package/src/commands/repair.ts

@@ -1,11 +1,12 @@
-import { and, desc, eq, inArray } from 'drizzle-orm'
+import { and, desc, eq, inArray, isNull } from 'drizzle-orm'
 import { Cli, z } from 'incur'
 import { repairCreate } from '../db/repair-create.ts'
 import { repairDelete } from '../db/repair-delete.ts'
 import { contextMiddleware, contextSchema } from '../middleware.ts'
+import { runAddPieces } from '../pipeline/add-pieces.ts'
 import { ensureRepairDataset } from '../pipeline/create-datasets.ts'
-import { runPullPiecesPhase } from '../pipeline/pull.ts'
 import { globalOptions } from '../utils.ts'
+
 export const repair = Cli.create('repair', {
   description: 'Repair commands',
   vars: contextSchema,
@@ -50,6 +51,7 @@ repair.command('list', {
     try {
       const localSchema = c.var.localDb._.fullSchema
       const repairs = await c.var.localDb.query.repairs.findMany({
+        where: isNull(localSchema.repairs.repairDataSetId),
         orderBy: [desc(localSchema.repairs.createdAt)],
         with: {
           operations: true,
@@ -131,16 +133,19 @@ repair.command('run', {
     repairId: z.coerce.number().describe('Repair ID to run'),
   }),
   options: globalOptions.extend({
-    concurrency: z.coerce.number().default(4).describe('Concurrency level'),
-    batchSize: z.coerce.number().default(40).describe('Max add_piece operations per pull batch'),
-    reset: z.boolean().default(false).describe('Reset the repair'),
+    concurrency: z.coerce.number().min(1).max(10).default(4).describe('Concurrency level'),
+    batchSize: z.coerce.number().min(1).max(40).default(40).describe('Max pieces per batch'),
   }),
   middleware: [contextMiddleware],
   run: async (c) => {
     try {
       const schema = c.var.localDb._.fullSchema
       const repair = await c.var.localDb.query.repairs.findFirst({
-        where: and(eq(schema.repairs.id, c.args.repairId), inArray(schema.repairs.status, ['pending'])),
+        where: and(
+          eq(schema.repairs.id, c.args.repairId),
+          inArray(schema.repairs.status, ['pending', 'failed']),
+          isNull(schema.repairs.repairDataSetId)
+        ),
       })
       if (!repair) {
         return c.error({
@@ -155,12 +160,11 @@ repair.command('run', {
         repair,
       })
 
-      await runPullPiecesPhase({
+      await runAddPieces({
         ...c.var,
         repair,
         concurrency: c.options.concurrency,
         batchSize: c.options.batchSize,
-        reset: c.options.reset,
       })
       return c.ok({
         repairId: repair.id,

package/src/commands/replicate.ts

@@ -0,0 +1,183 @@
+import { and, desc, eq, inArray, isNotNull } from 'drizzle-orm'
+import { Cli, z } from 'incur'
+import { repairDelete } from '../db/repair-delete.ts'
+import { replicateCreate } from '../db/replicate-create.ts'
+import { contextMiddleware, contextSchema } from '../middleware.ts'
+import { runAddPieces } from '../pipeline/add-pieces.ts'
+import { ensureReplicateDataset } from '../pipeline/create-datasets.ts'
+import { globalOptions } from '../utils.ts'
+
+export const replicate = Cli.create('replicate', {
+  description: 'Replicate dataset commands',
+  vars: contextSchema,
+})
+
+replicate.command('create', {
+  description: 'Create a new dataset replication',
+  options: globalOptions.extend({
+    dataSetId: z.coerce.bigint().describe('Dataset ID to replicate'),
+    targetProviderId: z.coerce.bigint().describe('Target provider ID for replication'),
+  }),
+  middleware: [contextMiddleware],
+  run: async (c) => {
+    try {
+      const { dataSetId, targetProviderId } = c.options
+
+      const replicateId = await replicateCreate({
+        ...c.var,
+        dataSetId,
+        targetProviderId,
+      })
+
+      return c.ok({
+        replicateId,
+      })
+    } catch (error) {
+      console.error(error)
+      return c.error({
+        code: 'REPLICATE_FAILED',
+        message: error instanceof Error ? error.message : 'Failed to replicate the dataset',
+        retryable: true,
+      })
+    }
+  },
+})
+
+replicate.command('list', {
+  description: 'List all replications',
+  options: globalOptions,
+  middleware: [contextMiddleware],
+  run: async (c) => {
+    try {
+      const localSchema = c.var.localDb._.fullSchema
+      const replications = await c.var.localDb.query.repairs.findMany({
+        where: isNotNull(localSchema.repairs.repairDataSetId),
+        orderBy: [desc(localSchema.repairs.createdAt)],
+        with: {
+          operations: true,
+        },
+      })
+
+      const replicationsFlattened = replications.map((repair) => {
+        const { operations, ...replicationWithoutOperations } = repair
+        return {
+          id: replicationWithoutOperations.id,
+          status: replicationWithoutOperations.status,
+          sourceProviderId: replicationWithoutOperations.repairProviderId,
+          sourceDataSetId: replicationWithoutOperations.repairDataSetId,
+          targetProviderId: replicationWithoutOperations.targetProviderId,
+          targetProviderUrl: replicationWithoutOperations.targetProviderUrl,
+          targetDataSetId: replicationWithoutOperations.targetDataSetId,
+          blockNumber: replicationWithoutOperations.blockNumber,
+          operations: operations.length,
+          pending: operations.filter((operation) => operation.status === 'pending').length,
+          failed: operations.filter((operation) => operation.status === 'failed').length,
+          completed: operations.filter((operation) => operation.status === 'completed').length,
+          skipped: operations.filter((operation) => operation.status === 'skipped').length,
+        }
+      })
+
+      return c.ok({
+        replications: replicationsFlattened,
+      })
+    } catch (error) {
+      console.error(error)
+      return c.error({
+        code: 'REPLICATE_FAILED',
+        message: error instanceof Error ? error.message : 'Failed to list replications',
+        retryable: true,
+      })
+    }
+  },
+})
+
+replicate.command('delete', {
+  description: 'Delete a replication',
+  args: z.object({
+    replicateId: z.coerce.number().describe('Replication ID to delete'),
+  }),
+  options: globalOptions,
+  middleware: [contextMiddleware],
+  run: async (c) => {
+    try {
+      const { deleted, operationsDeleted } = await repairDelete({
+        localDb: c.var.localDb,
+        repairId: c.args.replicateId,
+      })
+
+      if (!deleted) {
+        return c.error({
+          code: 'REPLICATE_NOT_FOUND',
+          message: 'Replication not found',
+          retryable: false,
+        })
+      }
+
+      return c.ok({
+        replicateId: c.args.replicateId,
+        operationsDeleted,
+      })
+    } catch (error) {
+      console.error(error)
+      return c.error({
+        code: 'REPLICATE_FAILED',
+        message: error instanceof Error ? error.message : 'Failed to delete the replication',
+        retryable: true,
+      })
+    }
+  },
+})
+
+replicate.command('run', {
+  description: 'Run a replication',
+  args: z.object({
+    replicateId: z.coerce.number().describe('Replication ID to run'),
+  }),
+  options: globalOptions.extend({
+    concurrency: z.coerce.number().min(1).max(10).default(4).describe('Concurrency level'),
+    batchSize: z.coerce.number().min(1).max(40).default(40).describe('Max pieces per batch'),
+  }),
+  middleware: [contextMiddleware],
+  run: async (c) => {
+    try {
+      const schema = c.var.localDb._.fullSchema
+      const repair = await c.var.localDb.query.repairs.findFirst({
+        where: and(
+          eq(schema.repairs.id, c.args.replicateId),
+          inArray(schema.repairs.status, ['pending', 'failed']),
+          isNotNull(schema.repairs.repairDataSetId)
+        ),
+      })
+      if (!repair) {
+        return c.error({
+          code: 'REPLICATE_NOT_FOUND',
+          message: 'Replication not found, it may have already been run or completed',
+          retryable: false,
+        })
+      }
+
+      await ensureReplicateDataset({
+        ...c.var,
+        repair,
+        source: c.var.source,
+      })
+
+      await runAddPieces({
+        ...c.var,
+        repair,
+        concurrency: c.options.concurrency,
+        batchSize: c.options.batchSize,
+      })
+      return c.ok({
+        replicateId: repair.id,
+      })
+    } catch (error) {
+      console.error(error)
+      return c.error({
+        code: 'REPLICATE_FAILED',
+        message: error instanceof Error ? error.message : 'Failed to replicate the dataset',
+        retryable: true,
+      })
+    }
+  },
+})

package/src/commands/setup.ts

@@ -22,9 +22,7 @@ function validatePostgresUrl(value: string) {
 
 export const setup = Cli.create('setup', {
   description: 'Setup the CLI',
-  options: globalOptions.extend({
-    privateKey: z.string().optional().describe('Private key to use'),
-  }),
+  options: globalOptions,
   run: async (c) => {
     try {
       // Private key
@@ -32,7 +30,7 @@ export const setup = Cli.create('setup', {
         message: 'Enter your private key',
         validate(value) {
           if (!value || !/^0x[a-fA-F0-9]{64}$/.test(value)) {
-            return `Invalid private key!`
+            return `Invalid private key! (must be 64 hex characters) starting with 0x`
           }
         },
         initialValue: config.get('privateKey'),
@@ -114,12 +112,28 @@ export const setup = Cli.create('setup', {
         })
       }
 
+      const source = await p.text({
+        message: 'Enter your source',
+        placeholder: 'Dataset source',
+        defaultValue: 'early-repair',
+        initialValue: config.get('source'),
+        withGuide: false,
+      })
+      if (p.isCancel(source)) {
+        return c.error({
+          code: 'SETUP_CANCELLED',
+          message: 'Setup cancelled',
+          retryable: false,
+        })
+      }
+
       // Set config
       config.set('privateKey', pk)
       config.set('indexerMainnetUrl', indexerMainnetUrl)
       config.set('indexerCalibrationUrl', indexerCalibrationUrl)
       config.set('chainId', chainId)
       config.set('dbPath', dbPath)
+      config.set('source', source)
 
       // setup database
       const db = await createLocalDatabase(dbPath)

package/src/db/dedupe-cids.ts

@@ -0,0 +1,49 @@
+import { and, eq, inArray } from 'drizzle-orm'
+import type { OperationSelect } from '../local-schema.ts'
+import type { IndexerDatabase, LocalDatabase } from '../types.ts'
+import { upsertOperations } from './upsert-operations.ts'
+
+export type SyncPiecesOnchainOptions = {
+  indexerDb: IndexerDatabase
+  localDb: LocalDatabase
+  dataSetId: bigint
+  operations: OperationSelect[]
+}
+
+/**
+ * Dedupe operations by CID on the target repair dataset.
+ * Returns operations that are not onchain.
+ */
+export async function dedupeCids({ indexerDb, localDb, dataSetId, operations }: SyncPiecesOnchainOptions) {
+  const cids = operations.map((operation) => operation.cid)
+  const schema = indexerDb._.fullSchema
+
+  const rows = await indexerDb
+    .select({ cid: schema.pieces.cid })
+    .from(schema.pieces)
+    .where(
+      and(eq(schema.pieces.dataSetId, dataSetId), eq(schema.pieces.removed, false), inArray(schema.pieces.cid, cids))
+    )
+
+  const existingCids = new Set<string>()
+  const completedOperations: OperationSelect[] = []
+
+  for (const row of rows) {
+    completedOperations.push(...operations.filter((operation) => operation.cid === row.cid))
+    existingCids.add(row.cid)
+  }
+
+  if (completedOperations.length > 0) {
+    await upsertOperations({
+      localDb,
+      operations: completedOperations.map((operation) => ({
+        ...operation,
+        status: 'completed',
+        error: null,
+      })),
+    })
+  }
+
+  // return operations that are not onchain
+  return operations.filter((operation) => !existingCids.has(operation.cid))
+}

package/src/db/{get-providers-by-cid.ts → find-providers-by-cid.ts}

@@ -1,10 +1,9 @@
-import { and, asc, eq, inArray, isNull, lte, notInArray, or } from 'drizzle-orm'
+import { and, asc, eq, inArray, isNull, lte, or } from 'drizzle-orm'
 import type { IndexerDatabase, RepairProvider } from '../types.ts'
 
-export type GetProvidersByCidOptions = {
+export type FindProvidersByCidOptions = {
   indexerDb: IndexerDatabase
   cids: readonly string[]
-  excludedProviderIds: readonly bigint[]
   blockNumber: bigint
 }
 
@@ -20,12 +19,11 @@ export type ProvidersByCid = Record<string, RepairProvider[]>
  * included. Every requested CID is present in the result; CIDs with no alternate providers
  * map to an empty array.
  */
-export async function getProvidersByCid({
+export async function findProvidersByCid({
   indexerDb,
   cids,
-  excludedProviderIds,
   blockNumber,
-}: GetProvidersByCidOptions): Promise<ProvidersByCid> {
+}: FindProvidersByCidOptions): Promise<ProvidersByCid> {
   const schema = indexerDb._.fullSchema
   const providersByCid = Object.fromEntries(cids.map((cid) => [cid, []])) as ProvidersByCid
   if (cids.length === 0) return providersByCid
@@ -35,11 +33,8 @@ export async function getProvidersByCid({
     eq(schema.dataSets.deleted, false),
     or(isNull(schema.dataSets.pdpEndEpoch), lte(schema.dataSets.pdpEndEpoch, blockNumber)),
     eq(schema.pieces.removed, false),
-    // or(eq(schema.providers.approved, true), eq(schema.providers.endorsed, true)),
+    or(eq(schema.providers.approved, true), eq(schema.providers.endorsed, true)),
   ]
-  if (excludedProviderIds.length > 0) {
-    filters.push(notInArray(schema.dataSets.providerId, [...excludedProviderIds]))
-  }
 
   // Join through datasets because providers own datasets, while pieces only reference dataset IDs.
   const rows = await indexerDb

package/src/db/{get-repair-dataset.ts → find-repair-dataset.ts}

@@ -1,12 +1,12 @@
 import { and, asc, eq, isNull } from 'drizzle-orm'
 import type { Address } from 'viem'
 import type { IndexerDatabase } from '../types.ts'
-import { EARLY_REPAIR_SOURCE } from '../utils.ts'
 
-export type GetRepairDatasetOptions = {
+export type FindRepairDatasetOptions = {
   indexerDb: IndexerDatabase
   providerId: bigint
   payer: Address
+  source: string
 }
 
 /**
@@ -14,11 +14,12 @@ export type GetRepairDatasetOptions = {
  *
  * When multiple datasets match, the lowest `dataSetId` is returned.
  */
-export async function getRepairDataset({
+export async function findRepairDataset({
   indexerDb,
   providerId,
   payer,
-}: GetRepairDatasetOptions): Promise<bigint | null> {
+  source,
+}: FindRepairDatasetOptions): Promise<bigint | null> {
   const schema = indexerDb._.fullSchema
 
   const result = await indexerDb
@@ -32,7 +33,7 @@ export async function getRepairDataset({
         eq(schema.dataSets.deleted, false),
         isNull(schema.dataSets.pdpEndEpoch),
         eq(schema.dataSets.payer, payer.toLowerCase()),
-        eq(schema.dataSets.source, EARLY_REPAIR_SOURCE),
+        eq(schema.dataSets.source, source),
         eq(schema.dataSets.withCdn, false),
         eq(schema.dataSets.withIpfsIndexing, true)
       )