@inixiative/archive 0.12.1 → 0.14.0

package/README.md

@@ -33,7 +33,7 @@ archive list
 archive search --query 'migration'
 ```
 
-Use `--source claude-code` for Claude histories. Collection matches the exact working-directory metadata recorded by the provider. Repeat `--project-root` to map several checkouts to one project; `--worktrees` also accepts every current git worktree of each root, re-read on each scan. Nested checkouts are separate mappings. Unknown directories, symlinks and sessions without usable metadata are skipped. Incomplete or changing files remain at source and retry on the next scan. The collector rescans files every 30 seconds; it deduplicates normalized content rather than maintaining byte-offset ingestion cursors. A watching collector waits out a server that is still starting.
+Use `--source claude-code` for Claude histories. Collection matches the exact working-directory metadata recorded by the provider. Repeat `--project-root` to map several checkouts to one project; `--worktrees` also accepts every current git worktree of each root, re-read on each scan. Nested checkouts are separate mappings unless `--subdirectories` lets a root also take the directories below it; the longest matching root wins. `--catch-all` (no roots) takes every session no other collector of that source claims, as projectId `unrouted`; no destination may take `unrouted`, so those sessions never leave the machine. Subdirectory and catch-all matches skip the exclusions: temp directories (test fixtures, agent scratchpads) and `.foundry/decision-receipts` by default, plus each `--exclude` (absolute paths match below them, relative ones match those path segments anywhere); an exact root is explicit and still matches. Unknown directories, symlinks and sessions without usable metadata are skipped. Incomplete or changing files remain at source and retry on the next scan. The collector rescans every 30 seconds. A session whose digest the server already holds is not sent again, and a watching collector does not re-read a file whose size and mtime have not moved since its last outcome (it re-checks a stored session against the server every 10 minutes, so one deleted there is captured again). A session is captured into one project; a collector refuses one another project holds, except one the catch-all holds as `unrouted`, which moves to the project that now claims it. Collectors compete only with collectors over the same history directory. A watching collector waits out a server that is still starting. `serve --sync` backs off failed deliveries: 30 seconds after the first failure, doubling to at most 30 minutes, cleared on delivery. An archive a destination refuses waits alone, and is tried again at once when it has a new revision; a destination that does not answer (or answers 5xx) is skipped for all its archives until it is due.
 
 To run the server without Docker, point `DATABASE_URL` at a Postgres with the `vector` and `pg_trgm` extensions available and run `archive serve [--port 4700] [--sync]`. Set `ARCHIVE_DEBUG=1` to see the underlying error when a command fails.
 
@@ -62,7 +62,7 @@ archive agents install
 archive agents status
 ```
 
-`add-collector` upserts by `--name` (lowercase letters, digits, `.` and `-`); `--directory` defaults to `~/.claude/projects` or `~/.codex/sessions`, and `--project-root`, `--worktrees` and `--atlas` behave as for `collect`. `remove-collector --name N` and `server --url URL` edit the same file. `install` writes one unit per collector, `com.inixiative.archive.collect.<name>`. Each runs the current Bun with this package's own `src/cli.ts`, restarts on exit (30 second throttle), works in the archive home and logs to `logs/<label>.{out,err}.log` there. Re-running `install` reloads only changed or stopped units and removes Archive units no longer declared (including the retired `local` and `sync` units); other launchd agents are untouched. `uninstall` removes every Archive unit; `status` reports each unit's load state, PID and last log entry. Re-run `install` after upgrading the package if its install path changes.
+`add-collector` upserts by `--name` (lowercase letters, digits, `.` and `-`); `--directory` defaults to `~/.claude/projects` or `~/.codex/sessions`, and `--project-root`, `--worktrees`, `--subdirectories`, `--catch-all` and `--atlas` behave as for `collect`. `remove-collector --name N`, `exclude [--exclude PATH ...]` (replaces the extra exclusions) and `server --url URL` edit the same file. `install` writes one unit per collector, `com.inixiative.archive.collect.<name>`, running `collect --agent <name> --watch`: the unit re-reads `agents.json` on every scan and routes each session against every collector of its source, so edits to roots and exclusions apply without a reinstall. Each runs the current Bun with this package's own `src/cli.ts`, restarts on exit (30 second throttle), works in the archive home and logs to `logs/<label>.{out,err}.log` there. Re-running `install` reloads only changed or stopped units and removes Archive units no longer declared. Units are written and loaded one at a time; a unit that fails to load gets its previous file back, so the next `install` loads it again, and `install` reports every failure after trying the rest (including the retired `local` and `sync` units); other launchd agents are untouched. `uninstall` removes every Archive unit; `status` reports each unit's load state, PID and last log entry. Re-run `install` after upgrading the package if its install path changes.
 
 `@inixiative/archive/agents` exports the pure pieces for integrations: `agentsConfigSchema`, `agentUnits`, `renderPlist`, `renderSystemdUnit` and `planAgents`, plus `installAgents`, `uninstallAgents` and `agentStatus` with an injectable command runner and supervisor directory.
 
@@ -128,7 +128,7 @@ The archive owns its tag system. It keeps two different things apart:
 bun run archive tag --id ARCHIVE_ID --tag reviewed --untag debugging
 ```
 
-Tag edits apply to that archive without a new revision, survive re-capture, and do not sync. Offered tags are defined archive-wide or for one actor. Each session can name its **actor**: a `user`, a `service` such as an API-key run, or a `job` (its id is the job name), with the `integrationId` it ran under when there is one. An organization's archive mostly holds its members' work, filterable per person. Entries the model produced record its `model` and `effort` (from Claude Code assistant records and Codex turn context, per turn); listings summarize them per session as `models` and filter by `model` and `effort`.
+Tag edits apply to that archive without a new revision, survive re-capture, and do not sync. Offered tags are defined archive-wide or for one actor. Each session can name its **actor**: a `user`, a `service` such as an API-key run, or a `job` (its id is the job name), with the `integrationId` it ran under when there is one. An organization's archive mostly holds its members' work, filterable per person. Entries the model produced record its `model` and `effort` (from Claude Code assistant records and Codex turn context, per turn); listings summarize them per session as `models` and filter by `model` and `effort`. Each session also records its `runtime`: `provider` (`anthropic` for Claude Code, Codex's `model_provider`) and `accessPath` (`subscription`, `api` or `unknown`), with `evidence` `transcript` when the transcript proves it (a subscription's rate-limit window: Claude Code `quotaLimits`, Codex `rate_limits.plan_type`) or `harness` when the collector read the harness login instead (Claude Code's `oauthAccount` unless its settings supply an API key; Codex `auth.json`). A subagent run is its own session with `lineage`: `kind` (`subagent` or `fork`), `parentSessionId`, and what the harness recorded (`agentType`, `agentName`, `description`, `toolUseId`, `depth`; Claude Code's `agent-<id>.meta.json`, Codex's subagent `session_meta`). Example: `models: [{model: 'claude-opus-5-5', effort: 'medium'}]`, `runtime: {provider: 'anthropic', accessPath: 'subscription', evidence: 'harness'}`.
 
 The **retention** setting (`retentionDays`, off by default) deletes archives whose latest capture is older than that, at server start and hourly. It deletes only from the archive it is set on; deletions never sync.
 
@@ -148,7 +148,7 @@ All data endpoints are `POST /api/v1/archive/<action>` with a JSON body and `Aut
 | `tags/list`, `tags/define`, `tags/remove` | `tag`, `actorId?`, `description?` | offered tags with usage counts |
 | `destinations/list`, `destinations/libraries`, `destinations/connect`, `destinations/remove`, `destinations/routes`, `destinations/sync` | see [Routing](#routing) | routes |
 
-Filters are `projectId`, `source`, `tag`, `actorId`, `model`, `effort` and `reference` (`{integration, ref}`). Jev integration is not enabled. Session content must not be sent to Jev without selecting that service for the relevant ownership boundary.
+Filters are `projectId`, `source`, `tag`, `actorId`, `model`, `effort`, `provider`, `accessPath` and `reference` (`{integration, ref}`); `archive list` and `archive search` take them as flags. Jev integration is not enabled. Session content must not be sent to Jev without selecting that service for the relevant ownership boundary.
 
 ## Limits
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@inixiative/archive",
-  "version": "0.12.1",
+  "version": "0.14.0",
   "type": "module",
   "exports": {
     ".": "./src/index.ts",
@@ -43,7 +43,7 @@
   },
   "devDependencies": {
     "@biomejs/biome": "2.5.2",
-    "@inixiative/config": "^0.5.66",
+    "@inixiative/config": "^0.5.70",
     "@types/bun": "1.4.2",
     "lefthook": "2.1.9",
     "typescript": "6.0.3"

package/prisma/migrations/20261012120000_runtime/migration.sql

@@ -0,0 +1,10 @@
+-- AlterTable
+ALTER TABLE "archives" ADD COLUMN     "accessPath" TEXT,
+ADD COLUMN     "provider" TEXT;
+
+-- CreateIndex
+CREATE INDEX "archives_provider_idx" ON "archives"("provider");
+
+-- CreateIndex
+CREATE INDEX "archives_accessPath_idx" ON "archives"("accessPath");
+

package/prisma/schema.prisma

@@ -28,6 +28,9 @@ model Archive {
   source     String
   projectId  String?
   actorId    String?
+  /// The model provider (anthropic, openai) and the access path (subscription, api, unknown).
+  provider   String?
+  accessPath String?
   capturedAt Float
   summary    Json
 
@@ -38,6 +41,8 @@ model Archive {
 
   @@index([projectId])
   @@index([actorId])
+  @@index([provider])
+  @@index([accessPath])
   @@map("archives")
 }
 

package/src/agents.ts

@@ -17,6 +17,7 @@ import { homedir } from 'node:os';
 import { dirname, isAbsolute, join, resolve } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { z } from 'zod';
+import { type CollectionSource, defaultExclusions, UNROUTED } from './collector';
 
 export const defaultHome = () => join(homedir(), '.local/share/archive');
 export const AGENT_LABEL_PREFIX = 'com.inixiative.archive.';
@@ -25,22 +26,40 @@ export const defaultHistoryDirectory = (source: 'claude-code' | 'codex') =>
   join(homedir(), source === 'codex' ? '.codex/sessions' : '.claude/projects');
 
 const absolute = z.string().max(4096).refine(isAbsolute, 'Path must be absolute');
-export const collectorAgentSchema = z.strictObject({
-  name: z
-    .string()
-    .max(64)
-    .regex(/^[a-z0-9]+(?:[.-][a-z0-9]+)*$/),
-  source: z.enum(['claude-code', 'codex']),
-  directory: absolute,
-  projectId: z.string().min(1).max(256),
-  projectRoots: z.array(absolute).min(1).max(100),
-  worktrees: z.boolean().optional(),
-  atlas: z.boolean().optional(),
-});
+export const collectorAgentSchema = z
+  .strictObject({
+    name: z
+      .string()
+      .max(64)
+      .regex(/^[a-z0-9]+(?:[.-][a-z0-9]+)*$/),
+    source: z.enum(['claude-code', 'codex']),
+    directory: absolute,
+    projectId: z.string().min(1).max(256),
+    projectRoots: z.array(absolute).max(100),
+    worktrees: z.boolean().optional(),
+    subdirectories: z.boolean().optional(),
+    /** Every session no other collector of this source claims; projectId `unrouted`, never synced. */
+    catchAll: z.boolean().optional(),
+    atlas: z.boolean().optional(),
+  })
+  .superRefine((c, ctx) => {
+    if (c.catchAll && (c.projectId !== UNROUTED || c.projectRoots.length))
+      ctx.addIssue({
+        code: 'custom',
+        message: `A catch-all collector has projectId ${UNROUTED} and no roots`,
+      });
+    if (!c.catchAll && (c.projectId === UNROUTED || !c.projectRoots.length))
+      ctx.addIssue({
+        code: 'custom',
+        message: `A collector needs a root and a projectId other than ${UNROUTED}`,
+      });
+  });
 export const agentsConfigSchema = z
   .strictObject({
     /** The Archive server collectors write to; the token is the home's `server.token`. */
     server: z.strictObject({ url: z.url() }).optional(),
+    /** Added to the default exclusions (temp directories, Foundry decision receipts). */
+    exclude: z.array(z.string().min(1).max(4096)).max(100).optional(),
     collectors: z.array(collectorAgentSchema).max(100),
   })
   .refine(
@@ -82,21 +101,36 @@ export interface AgentUnitOptions {
 }
 export const packageCliPath = () => fileURLToPath(new URL('./cli.ts', import.meta.url));
 
-export function collectorArgs(c: CollectorAgent, server?: { url: string }) {
-  return [
-    'collect',
-    ...(server ? ['--url', server.url] : []),
-    '--directory',
-    c.directory,
-    '--source',
-    c.source,
-    ...c.projectRoots.flatMap((root) => ['--project-root', root]),
-    ...(c.worktrees ? ['--worktrees'] : []),
-    ...(c.atlas ? ['--atlas'] : []),
-    '--project-id',
-    c.projectId,
-    '--watch',
-  ];
+/** A unit runs one named collector; it re-reads agents.json each scan, so edits apply live. */
+export function collectorArgs(c: CollectorAgent) {
+  return ['collect', '--agent', c.name, '--watch'];
+}
+
+/** A named collector, the peers it is routed against, its exclusions and server, from agents.json. */
+export function collectorPlan(config: AgentsConfig, name: string) {
+  const collector = config.collectors.find((c) => c.name === name);
+  if (!collector) throw new Error(`No collector named ${name} in agents.json`);
+  const source = (c: CollectorAgent): CollectionSource => ({
+    directory: c.directory,
+    source: c.source,
+    projectRoots: c.projectRoots,
+    projectId: c.projectId,
+    ...(c.worktrees ? { worktrees: true } : {}),
+    ...(c.subdirectories ? { subdirectories: true } : {}),
+    ...(c.catchAll ? { catchAll: true } : {}),
+    ...(c.atlas ? { atlas: true } : {}),
+  });
+  // Collectors over the same history compete for its sessions; another directory is another history.
+  const named = config.collectors.filter(
+    (c) => c.source === collector.source && resolve(c.directory) === resolve(collector.directory),
+  );
+  const peers = named.map(source);
+  return {
+    collector: peers[named.indexOf(collector)]!,
+    peers,
+    exclude: [...defaultExclusions(), ...(config.exclude ?? [])],
+    server: config.server,
+  };
 }
 
 /** One supervised unit per declared agent. */
@@ -114,7 +148,7 @@ export function agentUnits(config: AgentsConfig, options: AgentUnitOptions): Age
       stderr: join(home, 'logs', `${label}.err.log`),
     };
   };
-  return config.collectors.map((c) => unit(`collect.${c.name}`, collectorArgs(c, config.server)));
+  return config.collectors.map((c) => unit(`collect.${c.name}`, collectorArgs(c)));
 }
 
 const xml = (value: string) =>
@@ -310,23 +344,49 @@ export function installAgents(config: AgentsConfig, options: AgentHostOptions =
   }));
   const plan = planAgents(units, installed, h.platform, h.dir);
   h.remove(plan.remove);
-  for (const { file, content } of plan.write) {
+  const previous = new Map(installed.map((i) => [i.label, i.content]));
+  const write = (file: string, content: string) => {
     const temp = `${file}.${randomUUID()}.tmp`;
     writeFileSync(temp, content, { mode: 0o644, flag: 'wx' });
     renameSync(temp, file);
-  }
-  if (h.platform === 'linux' && plan.write.length) h.systemctl('daemon-reload');
+  };
+  const loaded: string[] = [];
+  const failed: { label: string; error: string }[] = [];
+  // Write and load one unit at a time. A unit that fails to load gets its old file back (or none),
+  // so the next install sees it as changed and loads it, instead of trusting a stale running copy.
+  const install = (label: string, content?: string) => {
+    try {
+      if (content !== undefined) {
+        write(h.file(label), content);
+        if (h.platform === 'linux') h.systemctl('daemon-reload');
+      }
+      h.load(label);
+      loaded.push(label);
+    } catch (error) {
+      failed.push({ label, error: error instanceof Error ? error.message : String(error) });
+      if (content === undefined) return;
+      const old = previous.get(label);
+      try {
+        if (old === undefined) rmSync(h.file(label), { force: true });
+        else write(h.file(label), old);
+        if (h.platform === 'linux') h.systemctl('daemon-reload');
+        // Loading stopped the unit that ran before; bring it back as it was.
+        if (old !== undefined) h.load(label);
+      } catch {}
+    }
+  };
+  for (const { label, content } of plan.write) install(label, content);
   // Unchanged units that are not running are loaded again.
-  const reload = [
-    ...plan.write.map((w) => w.label),
-    ...plan.unchanged.filter((label) => !h.loaded(label).loaded),
-  ];
-  for (const label of reload) h.load(label);
+  for (const label of plan.unchanged) if (!h.loaded(label).loaded) install(label);
+  if (failed.length)
+    throw new Error(
+      `Archive agents failed to load: ${failed.map((f) => `${f.label} (${f.error})`).join('; ')}`,
+    );
   return {
     platform: h.platform,
     dir: h.dir,
-    loaded: reload,
-    unchanged: plan.unchanged.filter((l) => !reload.includes(l)),
+    loaded,
+    unchanged: plan.unchanged.filter((l) => !loaded.includes(l)),
     removed: plan.remove,
   };
 }

package/src/cli.ts

@@ -7,6 +7,7 @@ import { fileURLToPath } from 'node:url';
 import { parseArgs } from 'node:util';
 import {
   agentStatus,
+  collectorPlan,
   defaultHistoryDirectory,
   defaultHome,
   installAgents,
@@ -16,8 +17,8 @@ import {
   writeAgents,
 } from './agents';
 import { importChatGPTThread } from './chatgpt';
-import { archiveRequest, searchRemotes, syncArchives } from './client';
-import { collectSessions } from './collector';
+import { archiveRequest, type SyncBackoff, searchRemotes, syncArchives } from './client';
+import { type CollectorCache, collectSessions, defaultExclusions, UNROUTED } from './collector';
 import {
   archiveDestinationSchema,
   connectDestination,
@@ -66,6 +67,14 @@ export async function runCli(args = Bun.argv.slice(2)) {
       query: { type: 'string', default: '' },
       'project-root': { type: 'string', multiple: true },
       worktrees: { type: 'boolean', default: false },
+      subdirectories: { type: 'boolean', default: false },
+      'catch-all': { type: 'boolean', default: false },
+      exclude: { type: 'string', multiple: true },
+      agent: { type: 'string' },
+      provider: { type: 'string' },
+      model: { type: 'string' },
+      effort: { type: 'string' },
+      'access-path': { type: 'string' },
       atlas: { type: 'boolean', default: false },
       tag: { type: 'string', multiple: true },
       untag: { type: 'string', multiple: true },
@@ -91,14 +100,16 @@ export async function runCli(args = Bun.argv.slice(2)) {
         'up | down   (the local Archive: bundled compose.yaml with Postgres, data in --home)\n' +
         'serve [--port 4700 --hostname 127.0.0.1] [--sync]   (DATABASE_URL; token from ARCHIVE_SERVER_TOKEN or --token-file)\n' +
         'import --file PATH --source codex|claude-code|chatgpt --project-id ID [--tag TAG]\n' +
-        'collect --directory HISTORY --source codex|claude-code --project-root EXACT_CWD [--project-root ...] [--worktrees] [--atlas] --project-id ID [--watch]\n' +
-        'list | export --id ID | tag --id ID [--tag TAG ...] [--untag TAG ...] | search --query TEXT [--remote]\n' +
+        'collect --directory HISTORY --source codex|claude-code (--project-root CWD [--project-root ...] --project-id ID [--worktrees] [--subdirectories] | --catch-all) [--exclude PATH ...] [--atlas] [--watch]\n' +
+        'collect --agent NAME [--watch]   (one collector from agents.json, routed against the others, re-read each scan)\n' +
+        'list | search --query TEXT [--remote]   [--project-id ID] [--source S] [--tag T] [--provider P] [--model M] [--effort E] [--access-path subscription|api|unknown]\n' +
+        'export --id ID | tag --id ID [--tag TAG ...] [--untag TAG ...]\n' +
         'pair --kingdom KINGDOM_API_URL [--name NAME] [--yes]   (register this Archive with Kingdom; claim the review code there, then confirm the owner)\n' +
         'connect --url HTTPS_URL --project-id ID (--token-env ENV | --token-file PATH)   (a remote Archive, for serve --sync)\n' +
         'connect --kind kingdom --url KINGDOM_API_URL --credential-file PATH --integration-id UUID --resource-id UUID --project-id ID   (a hosted Archive through Kingdom)\n' +
         'sync | routes   (the server publishes to, or previews, its destinations once)\n' +
-        'agents add-collector --name N --source codex|claude-code --project-id ID --project-root DIR [--project-root ...] [--directory HISTORY] [--worktrees] [--atlas]\n' +
-        'agents remove-collector --name N | server --url URL | install | uninstall | status\n' +
+        'agents add-collector --name N --source codex|claude-code (--project-id ID --project-root DIR [--project-root ...] [--worktrees] [--subdirectories] | --catch-all) [--directory HISTORY] [--atlas]\n' +
+        'agents remove-collector --name N | exclude [--exclude PATH ...] | server --url URL | install | uninstall | status\n' +
         `Data commands talk to the Archive server at --url (default ${DEFAULT_URL}) with the token in --token-file (default <home>/server.token) or ARCHIVE_TOKEN.`,
     );
     return;
@@ -170,10 +181,12 @@ export async function runCli(args = Bun.argv.slice(2)) {
     output({ listening: instance.server.url.href, sync: v.sync });
     let stopped = false;
     // One process owns the database connection: the server and its sync share it.
+    // Failed deliveries back off across cycles instead of being re-sent every 30 seconds.
+    const backoff: SyncBackoff = new Map();
     const sync = (async () => {
       while (v.sync && !stopped) {
         try {
-          const results = await syncArchives(instance.store, readDestinations(config));
+          const results = await syncArchives(instance.store, readDestinations(config), { backoff });
           const failed = results.filter((r) => r.status.startsWith('failed'));
           if (failed.length) console.error(JSON.stringify({ sync: 'failed', archives: failed }));
         } catch {
@@ -258,8 +271,11 @@ export async function runCli(args = Bun.argv.slice(2)) {
     const action = positionals[1];
     const base = { collectors: [], ...current };
     if (action === 'add-collector') {
-      if (!v.name || !v['project-id'] || !v['project-root'] || !v.source)
-        throw new Error('add-collector requires --name, --source, --project-id and --project-root');
+      const catchAll = v['catch-all'];
+      if (!v.name || !v.source || (!catchAll && (!v['project-id'] || !v['project-root'])))
+        throw new Error(
+          'add-collector requires --name, --source, and --project-id with --project-root (or --catch-all)',
+        );
       const source = v.source as 'codex' | 'claude-code';
       output(
         writeAgents(
@@ -268,13 +284,18 @@ export async function runCli(args = Bun.argv.slice(2)) {
             name: v.name,
             source,
             directory: resolve(v.directory ?? defaultHistoryDirectory(source)),
-            projectId: v['project-id'],
-            projectRoots: v['project-root'].map((root) => resolve(root)),
+            projectId: catchAll ? UNROUTED : v['project-id']!,
+            projectRoots: catchAll ? [] : v['project-root']!.map((root) => resolve(root)),
             ...(v.worktrees ? { worktrees: true } : {}),
+            ...(v.subdirectories ? { subdirectories: true } : {}),
+            ...(catchAll ? { catchAll: true } : {}),
             ...(v.atlas ? { atlas: true } : {}),
           }),
         ),
       );
+    } else if (action === 'exclude') {
+      const { exclude: _, ...rest } = base;
+      output(writeAgents(file, { ...rest, ...(v.exclude?.length ? { exclude: v.exclude } : {}) }));
     } else if (action === 'remove-collector') {
       if (!v.name) throw new Error('remove-collector requires --name');
       output(
@@ -313,7 +334,21 @@ export async function runCli(args = Bun.argv.slice(2)) {
   }
   const token = process.env.ARCHIVE_TOKEN ?? readToken();
   if (!token) throw new Error('No Archive token: run `archive init` or set ARCHIVE_TOKEN');
+  const agentsFile = join(home, 'agents.json');
   const archive = new ArchiveClient({ url: v.url ?? DEFAULT_URL, token });
+  if ((command === 'list' || command === 'search') && (v.tag?.length ?? 0) > 1)
+    throw new Error('Filter by one --tag at a time');
+  const filter = {
+    ...(v['project-id'] ? { projectId: v['project-id'] } : {}),
+    ...(v.source ? { source: v.source as 'claude-code' | 'codex' | 'foundry' | 'chatgpt' } : {}),
+    ...(v.tag?.length ? { tag: v.tag[0] } : {}),
+    ...(v.provider ? { provider: v.provider } : {}),
+    ...(v.model ? { model: v.model } : {}),
+    ...(v.effort ? { effort: v.effort } : {}),
+    ...(v['access-path']
+      ? { accessPath: v['access-path'] as 'subscription' | 'api' | 'unknown' }
+      : {}),
+  };
   if (command === 'routes') output(await archive.routes());
   else if (command === 'sync') {
     const results = await archive.sync();
@@ -349,15 +384,48 @@ export async function runCli(args = Bun.argv.slice(2)) {
       entries: snapshot.entries.length,
       coverage: snapshot.coverage,
     });
-  } else if (command === 'list') output(await archive.list());
+  } else if (command === 'list') output(await archive.list(filter));
   else if (command === 'collect') {
-    if (
-      !v.directory ||
-      !v['project-root'] ||
-      !v['project-id'] ||
-      !['codex', 'claude-code'].includes(v.source ?? '')
-    )
-      throw new Error('Collect requires --directory, --source, --project-root and --project-id');
+    const flagged = () => {
+      const catchAll = v['catch-all'];
+      if (
+        !v.directory ||
+        !['codex', 'claude-code'].includes(v.source ?? '') ||
+        (!catchAll && (!v['project-root'] || !v['project-id']))
+      )
+        throw new Error(
+          'Collect requires --agent, or --directory, --source and --project-root with --project-id (or --catch-all)',
+        );
+      const collector = {
+        directory: v.directory,
+        source: v.source as 'codex' | 'claude-code',
+        projectRoots: catchAll ? [] : v['project-root']!,
+        projectId: catchAll ? UNROUTED : v['project-id']!,
+        worktrees: v.worktrees,
+        subdirectories: v.subdirectories,
+        catchAll,
+        atlas: v.atlas,
+        tags: v.tag,
+      };
+      return {
+        collector,
+        peers: [collector],
+        exclude: [...defaultExclusions(), ...(v.exclude ?? [])],
+        server: undefined as { url: string } | undefined,
+      };
+    };
+    // A named collector re-reads agents.json each scan, so its routing follows edits without a restart.
+    const plan = () => {
+      if (!v.agent) return flagged();
+      const config = readAgents(agentsFile);
+      if (!config) throw new Error('No agents.json; declare agents first');
+      return collectorPlan(config, v.agent);
+    };
+    plan();
+    const cache: CollectorCache = new Map();
+    // A named collector writes to the server agents.json declares (unless --url), re-read each scan.
+    let client = archive;
+    let clientUrl = v.url ?? DEFAULT_URL;
     let stopped = false;
     const stop = () => {
       stopped = true;
@@ -365,17 +433,27 @@ export async function runCli(args = Bun.argv.slice(2)) {
     process.once('SIGINT', stop);
     process.once('SIGTERM', stop);
     do {
+      let current: ReturnType<typeof plan>;
+      try {
+        current = plan();
+      } catch (error) {
+        if (!v.watch) throw error;
+        console.error(
+          JSON.stringify({ collect: 'agents.json unusable', error: (error as Error).message }),
+        );
+        for (let i = 0; !stopped && i < 30; i++) await Bun.sleep(1000);
+        continue;
+      }
       try {
-        const result = await collectSessions(archive, {
-          directory: v.directory,
-          source: v.source as 'codex' | 'claude-code',
-          projectRoots: v['project-root'],
-          worktrees: v.worktrees,
-          atlas: v.atlas,
-          projectId: v['project-id'],
-          tags: v.tag,
-        });
-        output(result);
+        const { collector, peers, exclude, server } = current;
+        const url = v.url ?? server?.url ?? DEFAULT_URL;
+        if (url !== clientUrl) {
+          client = new ArchiveClient({ url, token });
+          clientUrl = url;
+        }
+        const started = Date.now();
+        const result = await collectSessions(client, collector, { peers, exclude, cache });
+        output({ ...result, ms: Date.now() - started });
         if (!v.watch && result.failed) process.exitCode = 1;
       } catch (error) {
         // The server may be starting; a watching collector retries.
@@ -390,6 +468,7 @@ export async function runCli(args = Bun.argv.slice(2)) {
   } else if (command === 'search')
     output(
       await archive.search({
+        ...filter,
         query: v.query,
         limit: Number(v.limit) > 100 ? 100 : Number(v.limit),
       }),

package/src/client.ts

@@ -126,28 +126,81 @@ export async function routingPreview(
   }));
 }
 
+/**
+ * Deliveries that failed, and when each may be tried again: per archive and destination, and per
+ * destination while it is unreachable. A retry waits 30 seconds, doubling per failure up to 30
+ * minutes; a new revision of the archive is tried at once.
+ */
+export type SyncBackoff = Map<string, { failures: number; until: number; digest?: string }>;
+const BACKOFF_BASE_MS = 30_000;
+const BACKOFF_MAX_MS = 30 * 60_000;
+/** No HTTP answer, or a 5xx: the destination is down, not refusing this archive. */
+const outage = (message: string) => {
+  const status = Number(/\((\d{3})\)/.exec(message)?.[1]);
+  return !status || status >= 500;
+};
+
 export async function syncArchives(
   store: ArchiveStore,
   destinations: ArchiveDestination[],
+  options: { backoff?: SyncBackoff; now?: () => number } = {},
 ): Promise<SyncResult[]> {
+  const now = options.now ?? Date.now;
+  const backoff = options.backoff;
   const results: SyncResult[] = [];
-  for (const archive of await store.list())
+  const seen = new Set<string>();
+  const delay = (failures: number) =>
+    Math.min(BACKOFF_BASE_MS * 2 ** (failures - 1), BACKOFF_MAX_MS);
+  const archives = await store.list();
+  for (const archive of archives)
     for (const destination of destinations.filter((d) => d.projectId === archive.projectId)) {
+      const identity = destinationIdentity(destination);
+      const key = JSON.stringify([archive.id, identity]);
+      const down = JSON.stringify(['destination', identity]);
+      seen.add(key).add(down);
+      const deferred = (until: number) =>
+        results.push({
+          id: archive.id,
+          destination: destination.url,
+          status: `deferred until ${new Date(until).toISOString()}`,
+        });
+      const circuit = backoff?.get(down);
+      if (circuit && now() < circuit.until) {
+        deferred(circuit.until);
+        continue;
+      }
+      const waiting = backoff?.get(key);
+      if (waiting && now() < waiting.until && waiting.digest === archive.digest) {
+        deferred(waiting.until);
+        continue;
+      }
       try {
         const result = await publishArchive(store, archive.id, destination);
+        backoff?.delete(key);
+        backoff?.delete(down);
         results.push({
           id: archive.id,
           destination: destination.url,
           status: result.unchanged ? 'unchanged' : 'published',
         });
       } catch (error) {
+        const message = error instanceof Error ? error.message : String(error);
+        if (outage(message)) {
+          const failures = (circuit?.failures ?? 0) + 1;
+          backoff?.set(down, { failures, until: now() + delay(failures) });
+        } else {
+          const failures = (waiting?.failures ?? 0) + 1;
+          backoff?.set(key, { failures, until: now() + delay(failures), digest: archive.digest });
+        }
         results.push({
           id: archive.id,
           destination: destination.url,
-          status: `failed; local revision retained: ${error instanceof Error ? error.message : String(error)}`,
+          status: `failed; local revision retained: ${message}`,
         });
       }
     }
+  // Archives that were deleted, or routes that were removed, leave no entries behind.
+  for (const key of backoff?.keys() ?? []) if (!seen.has(key)) backoff?.delete(key);
   return results;
 }