@filoz/repair-cli 0.4.0 → 1.1.0

package/readme.md

@@ -6,7 +6,7 @@
 
 The `repair` CLI helps prepare and run repair and replication jobs on Filecoin PDP datasets. It uses:
 
-- an indexer Postgres database as the read-only source of providers, datasets, and pieces
+- the [Filecoin API](https://github.com/filecoin-project/agents/tree/main/apps/filecoin-api) (`https://api.filecoin.io` by default) as the read-only source of providers, datasets, and pieces
 - a local SQLite database to track repair and replication jobs and per-piece operations
 - a configured Filecoin wallet to create datasets and submit on-chain add-piece transactions
 
@@ -24,7 +24,7 @@ repair --help
 
 ## Setup
 
-Run setup before using any command that talks to the indexer, local database, or wallet.
+Run setup before using any command that talks to the Filecoin API, local database, or wallet.
 
 ```bash
 repair setup
@@ -33,8 +33,7 @@ repair setup
 Setup prompts for:
 
 - private key for the repair wallet
-- mainnet indexer Postgres URL
-- calibration indexer Postgres URL
+- Filecoin API URL (defaults to `https://api.filecoin.io`)
 - chain, either Filecoin Mainnet `314` or Filecoin Calibration `314159`
 - local SQLite database path
 - dataset source label, used when creating repair target datasets (defaults to `early-repair`)
@@ -59,7 +58,7 @@ Interactive configuration and local database setup.
 repair setup
 ```
 
-Use this whenever you need to initialize the CLI, change the active chain, update indexer URLs, change the dataset source label, or move the local SQLite database.
+Use this whenever you need to initialize the CLI, change the active chain, change the Filecoin API URL, change the dataset source label, or move the local SQLite database.
 
 ### `repair wallet fund`
 
@@ -136,7 +135,7 @@ The command waits for the revoke transaction to be mined and returns the session
 
 ### `repair providers list`
 
-Lists PDP providers from the configured indexer.
+Lists PDP providers from the Filecoin API.
 
 ```bash
 repair providers list
@@ -149,8 +148,6 @@ By default, the command returns active PDP providers that are approved or endors
 - `serviceUrl`
 - `approved`
 - `endorsed`
-- `pieceCount`, the number of active indexed pieces for that provider
-- `totalSize`, the sum of active piece raw sizes formatted in decimal GB
 
 Use `--all` to include every active PDP provider, even if it is not approved or endorsed.
 
@@ -158,6 +155,12 @@ Use `--all` to include every active PDP provider, even if it is not approved or
 repair providers list --all
 ```
 
+Use `--stats` to also return `pieceCount`, the number of active indexed pieces for each provider, and `totalSize`, the sum of their raw sizes in decimal GB. The API has no aggregate endpoint, so this pages through every piece of every listed provider and can take a long time.
+
+```bash
+repair providers list --stats
+```
+
 ### `repair datasets list`
 
 Lists datasets owned by the configured repair wallet.
@@ -166,7 +169,7 @@ Lists datasets owned by the configured repair wallet.
 repair datasets list
 ```
 
-Each dataset includes its ID, CDN/IPFS indexing flags, source, provider URL, PDP end epoch, and piece count.
+Each dataset includes its ID, CDN/IPFS indexing flags, source, provider URL, and PDP end epoch. Add `--stats` to include each dataset's active piece count, which pages through every piece.
 
 Filter by provider ID:
 
@@ -176,7 +179,7 @@ repair datasets list --provider-id 123
 
 ### `repair datasets show <id>`
 
-Shows a dataset from the configured indexer.
+Shows a dataset from the Filecoin API.
 
 ```bash
 repair datasets show 42
@@ -206,7 +209,7 @@ repair repair create --provider-id 101 --target-provider-id 202
 
 `--target-provider-id` is the provider that should receive the repaired pieces. It must be different from `--provider-id`.
 
-The command snapshots the current chain block number, creates a local repair row, scans active pieces for the source provider, deduplicates them by CID, and creates local `add_piece` operations. Pieces with no alternate provider are marked `skipped`. The command returns a `repairId`.
+The command snapshots the current chain block number, creates a local repair row, scans active pieces for the source provider, deduplicates them by CID, and creates local `add_piece` operations. For each piece it probes the approved and endorsed providers (endorsed first) with `HEAD /piece/{cid}`, then the provider being repaired as a last resort, and uses the first one that serves it as the pull source. Pieces that no provider serves are marked `skipped`. The command returns a `repairId`.
 
 ### `repair repair list`
 
@@ -235,18 +238,18 @@ Runs a pending repair.
 repair repair run 1
 ```
 
-The command first ensures the target repair dataset exists for the configured wallet and target provider. If no matching dataset exists, it creates one with IPFS indexing enabled and CDN disabled, using the configured source label. Then it processes pending `add_piece` operations by pulling pieces from alternate providers into the target provider and committing them on-chain.
+The command first ensures the target repair dataset exists for the configured wallet and target provider. If no matching dataset exists, it creates one with IPFS indexing enabled and CDN disabled, using the configured source label. Then it reads the CIDs already in the target dataset once, marks matching operations completed, and processes the remaining pending `add_piece` operations by pulling pieces from alternate providers into the target provider and committing them on-chain.
 
 Options:
 
 - `--concurrency <number>` controls how many pull batches run at once. Defaults to `4`.
-- `--batch-size <number>` controls the maximum number of `add_piece` operations per batch. Defaults to `40`.
+- `--batch-size <number>` caps the number of `add_piece` operations per batch. By default each batch holds as many pieces as fit in one `addPieces` message (about 400 without piece metadata, 80 for legacy datasets).
 - `--payer <address>` overrides the payer address used when creating or finding the target repair dataset. Defaults to the configured wallet address.
 
 Example:
 
 ```bash
-repair repair run 1 --concurrency 8 --batch-size 40
+repair repair run 1 --concurrency 8 --batch-size 100
 repair repair run 1 --payer 0x1234567890123456789012345678901234567890
 ```
 
@@ -304,13 +307,13 @@ The command creates a new target dataset on the target provider, copying CDN and
 Options:
 
 - `--concurrency <number>` controls how many pull batches run at once. Defaults to `4`.
-- `--batch-size <number>` controls the maximum number of `add_piece` operations per batch. Defaults to `40`.
+- `--batch-size <number>` caps the number of `add_piece` operations per batch. By default each batch holds as many pieces as fit in one `addPieces` message (about 400 without piece metadata, 80 for legacy datasets).
 - `--payer <address>` overrides the payer address used when creating the target replication dataset. Defaults to the configured wallet address.
 
 Example:
 
 ```bash
-repair replicate run 1 --concurrency 8 --batch-size 40
+repair replicate run 1 --concurrency 8 --batch-size 100
 repair replicate run 1 --payer 0x1234567890123456789012345678901234567890
 ```
 
@@ -337,7 +340,7 @@ repair second-copy create \
 
 `--pieces-file` is a JSON file with an `aggregates` array. Each aggregate must include a `piece_cid` string. Only aggregate `piece_cid` values are used; `sub_pieces` are ignored.
 
-`--source-provider-url` is the provider URL pieces are pulled from. It is not looked up in the indexer, so it can point at a provider on another network (for example Mainnet while the CLI is configured for Calibration).
+`--source-provider-url` is the provider URL pieces are pulled from. It is not looked up in the Filecoin API, so it can point at a provider on another network (for example Mainnet while the CLI is configured for Calibration).
 
 `--target-provider-id` is the provider on the configured chain that should receive the copied pieces. Its service URL must differ from `--source-provider-url`.
 
@@ -373,13 +376,13 @@ The command first ensures the target dataset exists for the configured wallet an
 Options:
 
 - `--concurrency <number>` controls how many pull batches run at once. Defaults to `4`.
-- `--batch-size <number>` controls the maximum number of `add_piece` operations per batch. Defaults to `40`.
+- `--batch-size <number>` caps the number of `add_piece` operations per batch. By default each batch holds as many pieces as fit in one `addPieces` message (about 400 without piece metadata, 80 for legacy datasets).
 - `--payer <address>` overrides the payer address used when creating or finding the target dataset. Defaults to the configured wallet address.
 
 Example:
 
 ```bash
-repair second-copy run 1 --concurrency 8 --batch-size 40
+repair second-copy run 1 --concurrency 8 --batch-size 100
 repair second-copy run 1 --payer 0x1234567890123456789012345678901234567890
 ```
 

package/src/api.ts

@@ -0,0 +1,220 @@
+import {
+  ApiError,
+  createApiClient,
+  type DataSet,
+  DEFAULT_API_URL,
+  type Piece,
+  type PieceWithDataSet,
+  type Provider,
+  unwrap,
+} from '@filecoin-project/api-client'
+import { pipeline, withRateLimit, withRetry, withTimeout } from 'fetch-extras'
+import type { IndexerApi } from './types.ts'
+
+export { ApiError, DEFAULT_API_URL } from '@filecoin-project/api-client'
+export type { DataSet, Piece, PieceWithDataSet, Provider }
+
+/** filecoin-api network path segment. */
+export type Network = 'mainnet' | 'calibration'
+
+/** Largest page size filecoin-api accepts. */
+export const MAX_PAGE_SIZE = 200
+
+/** Retries for network errors and `408`, `429` and `5xx` responses. */
+const MAX_RETRIES = 5
+
+/** Per-attempt request timeout; a timed-out request fails instead of retrying. */
+const REQUEST_TIMEOUT_MS = 30_000
+
+/** filecoin-api allows 120 requests per minute per client IP. */
+const RATE_LIMIT = { requestsPerInterval: 120, interval: 60_000 }
+
+/** Map a chain ID to its filecoin-api network. */
+export function networkFromChainId(chainId: number): Network {
+  return chainId === 314 ? 'mainnet' : 'calibration'
+}
+
+/**
+ * Create a filecoin-api client for a chain.
+ *
+ * @param options.baseUrl - API origin; defaults to the client's production URL.
+ * @param options.chainId - Chain ID used to pick the network.
+ */
+export function createIndexerApi({ baseUrl, chainId }: { baseUrl?: string; chainId: number }): IndexerApi {
+  // Pace requests under the API's limit, and still retry 429s (other clients may share the IP),
+  // honouring Retry-After.
+  const fetchWithRetry = pipeline(
+    fetch,
+    withTimeout(REQUEST_TIMEOUT_MS),
+    withRateLimit(RATE_LIMIT),
+    withRetry({ retries: MAX_RETRIES })
+  )
+  return {
+    client: createApiClient({ baseUrl: baseUrl ?? DEFAULT_API_URL, fetch: fetchWithRetry }),
+    network: networkFromChainId(chainId),
+  }
+}
+
+/** Return `null` instead of throwing for a filecoin-api 404. */
+async function orNull<T>(promise: Promise<T>): Promise<T | null> {
+  try {
+    return await promise
+  } catch (error) {
+    if (ApiError.isNotFound(error)) return null
+    throw error
+  }
+}
+
+/** Collect every item from a cursor-paginated walk. */
+async function collect<T>(pages: AsyncIterable<T[]>): Promise<T[]> {
+  const items: T[] = []
+  for await (const page of pages) items.push(...page)
+  return items
+}
+
+/**
+ * Walk a cursor-paginated list, yielding one page of items at a time.
+ *
+ * @param fetchPage - Fetches the page after `cursor` (`undefined` for the first page).
+ */
+async function* paginate<T>(
+  fetchPage: (cursor: string | undefined) => Promise<{ data: T[]; nextCursor: string | null }>
+): AsyncGenerator<T[]> {
+  let cursor: string | undefined
+  do {
+    const page = await fetchPage(cursor)
+    yield page.data
+    cursor = page.nextCursor ?? undefined
+  } while (cursor)
+}
+
+/** Get a provider by ID, or `null` when it does not exist. */
+export function getProvider({ client, network }: IndexerApi, providerId: bigint): Promise<Provider | null> {
+  return orNull(
+    client
+      .GET('/{network}/providers/{providerId}', {
+        params: { path: { network, providerId: providerId.toString() } },
+      })
+      .then((result) => unwrap(result).data)
+  )
+}
+
+/** List every provider matching the filters. */
+export function listProviders(
+  { client, network }: IndexerApi,
+  query: { active?: boolean; approved?: boolean; endorsed?: boolean } = {}
+): Promise<Provider[]> {
+  return collect(
+    paginate((cursor) =>
+      client
+        .GET('/{network}/providers', {
+          params: {
+            path: { network },
+            query: {
+              limit: MAX_PAGE_SIZE,
+              cursor,
+              active: bool(query.active),
+              approved: bool(query.approved),
+              endorsed: bool(query.endorsed),
+            },
+          },
+        })
+        .then(unwrap)
+    )
+  )
+}
+
+/** Get a data set by ID, or `null` when it does not exist. */
+export function getDataSet({ client, network }: IndexerApi, dataSetId: bigint): Promise<DataSet | null> {
+  return orNull(
+    client
+      .GET('/{network}/data-sets/{dataSetId}', {
+        params: { path: { network, dataSetId: dataSetId.toString() } },
+      })
+      .then((result) => unwrap(result).data)
+  )
+}
+
+/** List every data set matching the filters, newest ID first. */
+export function listDataSets(
+  { client, network }: IndexerApi,
+  query: { owner?: string; providerId?: bigint; deleted?: boolean; withCdn?: boolean }
+): Promise<DataSet[]> {
+  return collect(
+    paginate((cursor) =>
+      client
+        .GET('/{network}/data-sets', {
+          params: {
+            path: { network },
+            query: {
+              limit: MAX_PAGE_SIZE,
+              cursor,
+              owner: query.owner,
+              provider_id: query.providerId?.toString(),
+              deleted: bool(query.deleted),
+              with_cdn: bool(query.withCdn),
+            },
+          },
+        })
+        .then(unwrap)
+    )
+  )
+}
+
+/** Walk the pieces of one data set page by page, newest piece ID first. */
+export function dataSetPiecePages(
+  { client, network }: IndexerApi,
+  dataSetId: bigint,
+  query: { removed?: boolean } = {}
+): AsyncGenerator<Piece[]> {
+  return paginate((cursor) =>
+    client
+      .GET('/{network}/data-sets/{dataSetId}/pieces', {
+        params: {
+          path: { network, dataSetId: dataSetId.toString() },
+          query: { limit: MAX_PAGE_SIZE, cursor, removed: bool(query.removed) },
+        },
+      })
+      .then(unwrap)
+  )
+}
+
+/** List every piece in one data set, newest piece ID first. */
+export function listDataSetPieces(
+  api: IndexerApi,
+  dataSetId: bigint,
+  query: { removed?: boolean } = {}
+): Promise<Piece[]> {
+  return collect(dataSetPiecePages(api, dataSetId, query))
+}
+
+/**
+ * Walk the pieces of every data set a provider stores, page by page, newest data set and piece
+ * first. One walk covers all of the provider's data sets.
+ */
+export function providerPiecePages(
+  { client, network }: IndexerApi,
+  providerId: bigint,
+  query: { removed?: boolean } = {}
+): AsyncGenerator<PieceWithDataSet[]> {
+  return paginate((cursor) =>
+    client
+      .GET('/{network}/pieces', {
+        params: {
+          path: { network },
+          query: { limit: MAX_PAGE_SIZE, cursor, provider_id: providerId.toString(), removed: bool(query.removed) },
+        },
+      })
+      .then(unwrap)
+  )
+}
+
+/** Encode an optional boolean filter as a query string value. */
+function bool(value: boolean | undefined): 'true' | 'false' | undefined {
+  return value === undefined ? undefined : value ? 'true' : 'false'
+}
+
+/** Convert API piece metadata to the string record stored on local operations. */
+export function toStringMetadata(metadata: Record<string, unknown> | null): Record<string, string> {
+  return Object.fromEntries(Object.entries(metadata ?? {}).map(([key, value]) => [key, String(value)]))
+}

package/src/commands/datasets.ts

@@ -1,9 +1,9 @@
 import * as p from '@clack/prompts'
 import * as SP from '@filoz/synapse-core/sp'
 import { getPdpDataSet } from '@filoz/synapse-core/warm-storage'
-import { and, asc, eq } from 'drizzle-orm'
 import { Cli, z } from 'incur'
 import { isAddress } from 'viem'
+import { getDataSet, getProvider, listDataSetPieces, listDataSets, listProviders } from '../api.ts'
 import { contextMiddleware, contextSchema } from '../middleware.ts'
 import { globalOptions, hashLink } from '../utils.ts'
 
@@ -18,42 +18,33 @@ datasets.command('list', {
   options: globalOptions.extend({
     providerId: z.coerce.bigint().optional().describe('Filter datasets by provider ID'),
     payer: z.string().refine(isAddress, 'Invalid address').optional().describe('Filter datasets by payer address'),
+    stats: z.boolean().optional().default(false).describe('Include piece counts (slow: walks every piece)'),
   }),
   middleware: [contextMiddleware],
   run: async (c) => {
     try {
-      const schema = c.var.indexerDb._.fullSchema
+      const { api } = c.var
       const payer = c.options.payer?.toLowerCase() ?? c.var.client.account.address.toLowerCase()
-      const conditions = [eq(schema.dataSets.deleted, false), eq(schema.dataSets.payer, payer)]
+      const [rows, providers] = await Promise.all([
+        listDataSets(api, { owner: payer, providerId: c.options.providerId, deleted: false }),
+        listProviders(api),
+      ])
+      const serviceUrls = new Map(providers.map((provider) => [provider.providerId, provider.serviceUrl]))
 
-      if (c.options.providerId != null) {
-        conditions.push(eq(schema.dataSets.providerId, c.options.providerId))
-      }
-
-      const datasets = await c.var.indexerDb.query.dataSets.findMany({
-        where: and(...conditions),
-        with: {
-          provider: true,
-          pieces: {
-            where: eq(c.var.indexerDb._.fullSchema.pieces.removed, false),
-          },
-        },
-        orderBy: [asc(schema.dataSets.dataSetId)],
-      })
-
-      const datasetsFlattened = datasets.map((dataset) => {
-        const { provider, pieces } = dataset
-        return {
+      const datasetsFlattened = []
+      for (const dataset of rows.reverse()) {
+        datasetsFlattened.push({
           id: dataset.dataSetId,
           withCdn: dataset.withCdn,
           withIpfsIndexing: dataset.withIpfsIndexing,
           source: dataset.source,
-          provider: provider.serviceUrl,
+          provider: dataset.providerId == null ? null : serviceUrls.get(dataset.providerId),
           pdpEndEpoch: dataset.pdpEndEpoch,
-          pieces: pieces.length,
-          //   metadata: JSON.stringify(dataset.metadata),
-        }
-      })
+          ...(c.options.stats && {
+            pieces: (await listDataSetPieces(api, BigInt(dataset.dataSetId), { removed: false })).length,
+          }),
+        })
+      }
 
       return c.ok({
         datasets: datasetsFlattened,
@@ -92,12 +83,19 @@ datasets.command('terminate', {
           retryable: false,
         })
       }
+      if (!dataset.provider) {
+        return c.error({
+          code: 'PROVIDER_NOT_FOUND',
+          message: `Provider ${dataset.providerId} for dataset ${dataset.dataSetId} is not available`,
+          retryable: false,
+        })
+      }
       if (isInteractive) {
-        spinner.message(`Terminating dataset ${dataset?.dataSetId} at ${dataset?.provider.pdp.serviceURL}...`)
+        spinner.message(`Terminating dataset ${dataset.dataSetId} at ${dataset.provider.pdp.serviceURL}...`)
       }
       const result = await SP.terminateService(c.var.client, {
         dataSetId: c.args.id,
-        serviceURL: dataset?.provider.pdp.serviceURL,
+        serviceURL: dataset.provider.pdp.serviceURL,
       })
 
       const waitForResult = await SP.waitForTerminateService({
@@ -138,15 +136,8 @@ datasets.command('show', {
   middleware: [contextMiddleware],
   run: async (c) => {
     try {
-      const dataset = await c.var.indexerDb.query.dataSets.findFirst({
-        where: eq(c.var.indexerDb._.fullSchema.dataSets.dataSetId, c.args.id),
-        with: {
-          provider: true,
-          pieces: {
-            where: eq(c.var.indexerDb._.fullSchema.pieces.removed, false),
-          },
-        },
-      })
+      const { api } = c.var
+      const dataset = await getDataSet(api, c.args.id)
 
       if (!dataset) {
         return c.error({
@@ -155,7 +146,11 @@ datasets.command('show', {
           retryable: false,
         })
       }
-      return c.ok(dataset)
+      const [provider, pieces] = await Promise.all([
+        dataset.providerId == null ? null : getProvider(api, BigInt(dataset.providerId)),
+        listDataSetPieces(api, c.args.id, { removed: false }),
+      ])
+      return c.ok({ ...dataset, provider, pieces: pieces.reverse() })
     } catch (error) {
       if (c.options.debug) {
         console.error(error)

package/src/commands/providers.ts

@@ -1,6 +1,7 @@
-import { and, asc, count, eq, inArray, isNull, lte, or, type SQLWrapper, sum } from 'drizzle-orm'
 import { Cli, z } from 'incur'
 import { getBlockNumber } from 'viem/actions'
+import { listProviders, providerPiecePages } from '../api.ts'
+import { getActiveDataSetIds } from '../db/get-active-data-set-ids.ts'
 import { contextMiddleware, contextSchema } from '../middleware.ts'
 import { globalOptions } from '../utils.ts'
 
@@ -12,6 +13,13 @@ function formatBytesAsGb(bytes: bigint): string {
   return `${whole}.${fraction.toString().padStart(2, '0')} GB`
 }
 
+/** Compare decimal ID strings numerically. */
+function compareIds(a: string, b: string): number {
+  const x = BigInt(a)
+  const y = BigInt(b)
+  return x < y ? -1 : x > y ? 1 : 0
+}
+
 export const providers = Cli.create('providers', {
   description: 'Provider commands',
   options: globalOptions,
@@ -22,51 +30,36 @@ providers.command('list', {
   description: 'List all providers from the indexer',
   options: globalOptions.extend({
     all: z.boolean().optional().default(false).describe('Include all providers'),
+    stats: z
+      .boolean()
+      .optional()
+      .default(false)
+      .describe('Include piece count and total size (slow: walks every piece of each provider)'),
   }),
   middleware: [contextMiddleware],
   run: async (c) => {
     try {
-      const schema = c.var.indexerDb._.fullSchema
-      const filters: (SQLWrapper | undefined)[] = [
-        eq(schema.providers.providerActive, true),
-        eq(schema.providers.pdpProductActive, true),
-      ]
-      if (!c.options.all) {
-        filters.push(or(eq(schema.providers.approved, true), eq(schema.providers.endorsed, true)))
-      }
-      const blockNumber = await getBlockNumber(c.var.client)
-      const rows = await c.var.indexerDb.query.providers.findMany({
-        orderBy: [asc(schema.providers.providerId)],
-        where: and(...filters),
-      })
-
-      const providerIds = rows.map((provider) => provider.providerId)
-      const statsByProviderId = new Map<bigint, { pieceCount: number; totalSize: bigint }>()
+      const { api } = c.var
+      const rows = (await listProviders(api, { active: true }))
+        .filter((provider) => provider.pdpProductActive && (c.options.all || provider.approved || provider.endorsed))
+        .sort((a, b) => compareIds(a.providerId, b.providerId))
 
-      if (providerIds.length > 0) {
-        const stats = await c.var.indexerDb
-          .select({
-            providerId: schema.dataSets.providerId,
-            pieceCount: count(schema.pieces.pieceId),
-            totalSize: sum(schema.pieces.rawSize),
-          })
-          .from(schema.pieces)
-          .innerJoin(schema.dataSets, eq(schema.pieces.dataSetId, schema.dataSets.dataSetId))
-          .where(
-            and(
-              inArray(schema.dataSets.providerId, providerIds),
-              eq(schema.dataSets.deleted, false),
-              or(isNull(schema.dataSets.pdpEndEpoch), lte(schema.dataSets.pdpEndEpoch, blockNumber)),
-              eq(schema.pieces.removed, false)
-            )
-          )
-          .groupBy(schema.dataSets.providerId)
+      const statsByProviderId = new Map<string, { pieceCount: number; totalSize: bigint }>()
 
-        for (const stat of stats) {
-          statsByProviderId.set(stat.providerId, {
-            pieceCount: stat.pieceCount,
-            totalSize: stat.totalSize == null ? 0n : BigInt(stat.totalSize),
-          })
+      if (c.options.stats && rows.length > 0) {
+        const blockNumber = await getBlockNumber(c.var.client)
+        for (const provider of rows) {
+          const providerId = BigInt(provider.providerId)
+          const activeDataSetIds = await getActiveDataSetIds({ api, providerId, blockNumber })
+          const stats = { pieceCount: 0, totalSize: 0n }
+          for await (const pieces of providerPiecePages(api, providerId, { removed: false })) {
+            for (const piece of pieces) {
+              if (!activeDataSetIds.has(piece.dataSetId)) continue
+              stats.pieceCount++
+              stats.totalSize += BigInt(piece.rawSize ?? 0)
+            }
+          }
+          statsByProviderId.set(provider.providerId, stats)
         }
       }
 
@@ -78,8 +71,7 @@ providers.command('list', {
           serviceUrl: provider.serviceUrl,
           approved: provider.approved,
           endorsed: provider.endorsed,
-          pieceCount: stats?.pieceCount ?? 0,
-          totalSize: formatBytesAsGb(stats?.totalSize ?? 0n),
+          ...(stats && { pieceCount: stats.pieceCount, totalSize: formatBytesAsGb(stats.totalSize) }),
         }
       })
 

package/src/commands/repair.ts

@@ -135,7 +135,11 @@ repair.command('run', {
   }),
   options: globalOptions.extend({
     concurrency: z.coerce.number().min(1).max(10).default(4).describe('Concurrency level'),
-    batchSize: z.coerce.number().min(1).max(40).default(40).describe('Max pieces per batch'),
+    batchSize: z.coerce
+      .number()
+      .min(1)
+      .optional()
+      .describe('Max pieces per batch. Defaults to as many as fit in one addPieces message'),
     payer: z.string().refine(isAddress, 'Invalid address').optional().describe('Payer address'),
   }),
   middleware: [contextMiddleware],

package/src/commands/replicate.ts

@@ -136,7 +136,11 @@ replicate.command('run', {
   }),
   options: globalOptions.extend({
     concurrency: z.coerce.number().min(1).max(10).default(4).describe('Concurrency level'),
-    batchSize: z.coerce.number().min(1).max(40).default(40).describe('Max pieces per batch'),
+    batchSize: z.coerce
+      .number()
+      .min(1)
+      .optional()
+      .describe('Max pieces per batch. Defaults to as many as fit in one addPieces message'),
     payer: z.string().refine(isAddress, 'Invalid address').optional().describe('Payer address'),
   }),
   middleware: [contextMiddleware],

package/src/commands/second-copy.ts

@@ -137,7 +137,11 @@ secondCopy.command('run', {
   }),
   options: globalOptions.extend({
     concurrency: z.coerce.number().min(1).max(10).default(4).describe('Concurrency level'),
-    batchSize: z.coerce.number().min(1).max(40).default(40).describe('Max pieces per batch'),
+    batchSize: z.coerce
+      .number()
+      .min(1)
+      .optional()
+      .describe('Max pieces per batch. Defaults to as many as fit in one addPieces message'),
     payer: z.string().refine(isAddress, 'Invalid address').optional().describe('Payer address'),
   }),
   middleware: [contextMiddleware],