@inixiative/archive 0.11.0 → 0.12.0

package/README.md

@@ -21,7 +21,7 @@ archive up      # creates ~/.local/share/archive/server.token, then docker compo
 archive down
 ```
 
-`up` runs the `compose.yaml` that ships in the package (project `archive`) with `ARCHIVE_DATA_DIR` set to the archive home and the server token from its `server.token`. Postgres data lives in `$ARCHIVE_DATA_DIR/postgres` and the server's configuration (`destinations.json`, destination token files, the Kingdom pairing) in `$ARCHIVE_DATA_DIR/config`, so both survive rebuilds and volume prunes. Compose mounts the config directory at the same absolute path, so the CLI and the server read one `destinations.json` and every path in it resolves for both. Running Compose directly requires an absolute `ARCHIVE_DATA_DIR`. Pin a version with `ARCHIVE_VERSION`; build from a checkout with `docker compose -f compose.yaml -f compose.build.yaml up -d --build`.
+`up` runs the `compose.yaml` that ships in the package (project `archive`) with `ARCHIVE_DATA_DIR` set to the archive home and the server token from its `server.token`. Postgres data lives in `$ARCHIVE_DATA_DIR/postgres` and the server's configuration (`destinations.json`, destination token files, the Kingdom pairing) in `$ARCHIVE_DATA_DIR/config`, so both survive rebuilds and volume prunes. Compose mounts the config directory at the same absolute path, so the CLI and the server read one `destinations.json` and every path in it resolves for both. `up` runs the image tagged with the CLI's own version, so a cached `latest` can never sit under a newer config layout; set `ARCHIVE_VERSION` to run another. Running Compose directly requires an absolute `ARCHIVE_DATA_DIR` and an `ARCHIVE_VERSION`; build from a checkout with `ARCHIVE_VERSION=local docker compose -f compose.yaml -f compose.build.yaml up -d --build`.
 
 The server applies its schema migrations on start and requires bearer authentication for data endpoints; `/health` has no session data. Every other command talks to it over HTTP at `--url` (default `http://127.0.0.1:4700`) with the token in `--token-file` (default `~/.local/share/archive/server.token`) or `ARCHIVE_TOKEN`:
 
@@ -48,7 +48,7 @@ archive connect --url https://your-archive.example --project-id inixiative --tok
 archive search --remote --query 'migration'
 ```
 
-`setup` is an alias for `connect`. Setup verifies access before saving `<home>/config/destinations.json` and does not upload; the Compose server (`serve --sync`) reads it and publishes every 30 seconds. `sync` and `routes` run the same publication, or preview it, once against `DATABASE_URL`. The URL locates the server; the token grants access. Configuration stores only the token file path (`tokenFile`) or environment variable name (`--token-env`, `tokenEnv`), never the token. A token file is read on every request, so rotating it needs no restart; symlinks and files readable by group or others are refused. Redirects are refused; only HTTPS or loopback HTTP is allowed.
+`setup` is an alias for `connect`. Setup verifies access before saving `<home>/config/destinations.json` and does not upload; the Compose server (`serve --sync`) reads it and publishes every 30 seconds. `sync` and `routes` ask that server to run the same publication, or preview it, once. The URL locates the server; the token grants access. Configuration stores only the token file path (`tokenFile`) or environment variable name (`--token-env`, `tokenEnv`), never the token. A token file is read on every request, so rotating it needs no restart; symlinks and files readable by group or others are refused. Redirects are refused; only HTTPS or loopback HTTP is allowed.
 
 ## Always-on local archiving
 
@@ -112,8 +112,10 @@ The server manages routes over HTTP too, so Foundry can read and set them. Only
 | `destinations/libraries` | none | `paired`, and the `libraries` (`integrationId`, `resourceId`, `name`) the held Signets may write to |
 | `destinations/connect` | `projectId`, `integrationId`, `resourceId` | the saved destination, after Kingdom confirms `sessions.write` on the library (403 otherwise) |
 | `destinations/remove` | `projectId`, `integrationId`, `resourceId` | `removed` |
+| `destinations/routes` | none | each archive with the destinations its project routes to; nothing is sent |
+| `destinations/sync` | none | one publication to every destination: `id`, `destination`, `status` per archive and route |
 
-`@inixiative/archive/remote`'s `ArchiveClient` wraps these as `destinations()`, `libraries()`, `connectDestination()` and `removeDestination()`. The sync loop picks up changes within 30 seconds.
+`@inixiative/archive/remote`'s `ArchiveClient` wraps these as `destinations()`, `libraries()`, `connectDestination()`, `removeDestination()`, `routes()` and `sync()`. The sync loop picks up changes within 30 seconds.
 
 ## Tags, references, actors and retention
 
@@ -144,7 +146,7 @@ All data endpoints are `POST /api/v1/archive/<action>` with a JSON body and `Aut
 | `delete` | `archiveId` | deleted |
 | `settings/read`, `settings/update` | `integrations?`, `retentionDays?` | settings |
 | `tags/list`, `tags/define`, `tags/remove` | `tag`, `actorId?`, `description?` | offered tags with usage counts |
-| `destinations/list`, `destinations/libraries`, `destinations/connect`, `destinations/remove` | see [Routing](#routing) | routes |
+| `destinations/list`, `destinations/libraries`, `destinations/connect`, `destinations/remove`, `destinations/routes`, `destinations/sync` | see [Routing](#routing) | routes |
 
 Filters are `projectId`, `source`, `tag`, `actorId`, `model`, `effort` and `reference` (`{integration, ref}`). Jev integration is not enabled. Session content must not be sent to Jev without selecting that service for the relevant ownership boundary.
 

package/compose.yaml

@@ -19,7 +19,7 @@ services:
       timeout: 5s
       retries: 20
   archive:
-    image: ghcr.io/inixiative/archive:${ARCHIVE_VERSION:-latest}
+    image: ghcr.io/inixiative/archive:${ARCHIVE_VERSION:?archive up sets ARCHIVE_VERSION to the CLI's own version}
     command: ["bun", "run", "src/cli.ts", "serve", "--home", "${ARCHIVE_DATA_DIR}", "--hostname", "0.0.0.0", "--sync"]
     depends_on:
       postgres:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@inixiative/archive",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "type": "module",
   "exports": {
     ".": "./src/index.ts",
@@ -43,7 +43,7 @@
   },
   "devDependencies": {
     "@biomejs/biome": "2.5.2",
-    "@inixiative/config": "^0.5.64",
+    "@inixiative/config": "^0.5.66",
     "@types/bun": "1.4.2",
     "lefthook": "2.1.9",
     "typescript": "6.0.3"

package/src/cli.ts

@@ -16,7 +16,7 @@ import {
   writeAgents,
 } from './agents';
 import { importChatGPTThread } from './chatgpt';
-import { archiveRequest, routingPreview, searchRemotes, syncArchives } from './client';
+import { archiveRequest, searchRemotes, syncArchives } from './client';
 import { collectSessions } from './collector';
 import {
   archiveDestinationSchema,
@@ -30,7 +30,6 @@ import { pairWithKingdom, verifyKingdomDestination } from './kingdom';
 import { previewImports } from './preview';
 import { ArchiveClient, DEFAULT_URL } from './remote';
 import { startArchiveServer } from './server';
-import { ArchiveStore } from './store';
 
 /** Applies pending schema migrations to the archive database. */
 export function migrate(databaseUrl: string) {
@@ -97,7 +96,7 @@ export async function runCli(args = Bun.argv.slice(2)) {
         'pair --kingdom KINGDOM_API_URL [--name NAME] [--yes]   (register this Archive with Kingdom; claim the review code there, then confirm the owner)\n' +
         'connect --url HTTPS_URL --project-id ID (--token-env ENV | --token-file PATH)   (a remote Archive, for serve --sync)\n' +
         'connect --kind kingdom --url KINGDOM_API_URL --credential-file PATH --integration-id UUID --resource-id UUID --project-id ID   (a hosted Archive through Kingdom)\n' +
-        'sync | routes   (DATABASE_URL: publish to, or preview, destinations once)\n' +
+        'sync | routes   (the server publishes to, or previews, its destinations once)\n' +
         'agents add-collector --name N --source codex|claude-code --project-id ID --project-root DIR [--project-root ...] [--directory HISTORY] [--worktrees] [--atlas]\n' +
         'agents remove-collector --name N | server --url URL | install | uninstall | status\n' +
         `Data commands talk to the Archive server at --url (default ${DEFAULT_URL}) with the token in --token-file (default <home>/server.token) or ARCHIVE_TOKEN.`,
@@ -117,6 +116,12 @@ export async function runCli(args = Bun.argv.slice(2)) {
     output({ initialized: true, home, tokenFile });
     return;
   }
+  const packageVersion = () =>
+    (
+      JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')) as {
+        version: string;
+      }
+    ).version;
   if (command === 'up' || command === 'down') {
     // The machine's one local Archive: the bundled compose file, with its data in the home.
     if (command === 'up') init();
@@ -135,6 +140,7 @@ export async function runCli(args = Bun.argv.slice(2)) {
         env: {
           ...process.env,
           ARCHIVE_DATA_DIR: home,
+          ARCHIVE_VERSION: process.env.ARCHIVE_VERSION ?? packageVersion(),
           // Compose requires the variable even to stop; down never needs the real token.
           ARCHIVE_SERVER_TOKEN: readToken() ?? 'x'.repeat(32),
         },
@@ -305,24 +311,15 @@ export async function runCli(args = Bun.argv.slice(2)) {
     if (results.some((r) => 'error' in r)) process.exitCode = 1;
     return;
   }
-  if (command === 'sync' || command === 'routes') {
-    const store = new ArchiveStore(databaseUrl());
-    try {
-      if (command === 'routes') output(await routingPreview(store, readDestinations(config)));
-      else {
-        const results = await syncArchives(store, readDestinations(config));
-        output(results);
-        if (results.some((r) => r.status.startsWith('failed'))) process.exitCode = 1;
-      }
-    } finally {
-      await store.close();
-    }
-    return;
-  }
   const token = process.env.ARCHIVE_TOKEN ?? readToken();
   if (!token) throw new Error('No Archive token: run `archive init` or set ARCHIVE_TOKEN');
   const archive = new ArchiveClient({ url: v.url ?? DEFAULT_URL, token });
-  if (command === 'import') {
+  if (command === 'routes') output(await archive.routes());
+  else if (command === 'sync') {
+    const results = await archive.sync();
+    output(results);
+    if (results.some((r) => r.status.startsWith('failed'))) process.exitCode = 1;
+  } else if (command === 'import') {
     if (
       !v.file ||
       !['codex', 'claude-code', 'chatgpt'].includes(v.source ?? '') ||

package/src/import.ts

@@ -7,6 +7,47 @@ export interface ImportOptions {
   title?: string;
   projectId?: string;
 }
+/** Content the harness wraps in or adds to a user message; the person never typed it. */
+const injectedTags = [
+  'system-reminder',
+  'local-command-caveat',
+  'local-command-stdout',
+  'local-command-stderr',
+  'command-name',
+  'command-message',
+  'command-args',
+  'bash-input',
+  'bash-stdout',
+  'bash-stderr',
+  'task-notification',
+  'user-prompt-submit-hook',
+  'fork-boilerplate',
+  'recommended_plugins',
+  'environment_context',
+  'user_instructions',
+  'permissions',
+  'INSTRUCTIONS',
+].join('|');
+const injectedBlock = new RegExp(`<(${injectedTags})(?:\\s[^>]*)?>[\\s\\S]*?</\\1>`, 'g');
+const injectedOpening = new RegExp(`^<(?:${injectedTags})(?:>|\\s)`);
+/** Foundry's native-session envelope; the person's text is the first User Message section. */
+const foundryUserMessage =
+  /^# User Message[ \t]*\n([\s\S]*?)(?=\n# (?:User Message|Assistant Message|Foundry Injection)[ \t]*\n|(?![\s\S]))/m;
+
+/** What the person typed in a user message, without injected content; undefined when nothing is left. */
+function typedPrompt(text: string) {
+  let prompt = text
+    .replace(injectedBlock, '')
+    .replace(/<\/?pasted_content(?:\s[^>]*)?>/g, '')
+    .replace(/^\s*# AGENTS\.md instructions for [^\n]*/, '')
+    .trim();
+  if (/^# (?:System Context|User Message)[ \t]*\n/.test(prompt))
+    prompt = prompt.match(foundryUserMessage)?.[1].trim() ?? '';
+  if (!prompt || injectedOpening.test(prompt) || /^\[Request interrupted by user/.test(prompt))
+    return undefined;
+  return prompt;
+}
+
 export function importTranscript(text: string, options: ImportOptions): ArchiveSnapshot {
   return importTranscriptLines(text.split('\n'), options);
 }
@@ -19,6 +60,8 @@ export function importTranscriptLines(
   let sessionId = options.sessionId;
   let reasoning = false;
   const seen = new Map<string, string>();
+  let prompt: string | undefined;
+  let metaPrompt: string | undefined;
   let index = -1;
   let contentBytes = 0;
   // Codex states the model per turn; Claude Code per assistant record.
@@ -75,6 +118,11 @@ export function importTranscriptLines(
           'Supported transcript content exceeds the archive size limit; no archive imported',
         );
       seen.set(id, identity);
+      // Claude Code marks expanded commands and caveats as meta: a title only when nothing was typed.
+      if (kind === 'user' && !prompt && !row.isCompactSummary) {
+        if (!row.isMeta) prompt = typedPrompt(content);
+        else metaPrompt ??= typedPrompt(content);
+      }
       entries.push({
         id,
         kind,
@@ -165,14 +213,7 @@ export function importTranscriptLines(
   }
   if (!sessionId || !entries.length)
     throw new Error('Transcript must identify a session and contain supported records');
-  const firstPrompt = entries.find(
-    (entry) =>
-      entry.kind === 'user' &&
-      !/^<(?:recommended_plugins|environment_context|user_instructions|permissions)(?:>|\s)/.test(
-        entry.text.trimStart(),
-      ),
-  );
-  const title = options.title ?? firstPrompt?.text.slice(0, 160) ?? sessionId;
+  const title = options.title ?? (prompt ?? metaPrompt)?.slice(0, 160) ?? sessionId;
   return archiveSnapshotSchema.parse({
     schemaVersion: 1,
     sourceId: options.sourceId,

package/src/remote.ts

@@ -1,6 +1,7 @@
 import { existsSync, readFileSync } from 'node:fs';
 import { join } from 'node:path';
 import { defaultHome } from './agents';
+import type { routingPreview, syncArchives } from './client';
 import type { DescribedDestination } from './config';
 import type { ArchiveChunk, ArchiveSnapshot } from './index';
 import { archiveKey, archiveSnapshotSchema } from './index';
@@ -22,6 +23,8 @@ export type DestinationStatus = DescribedDestination & { delivered: number; pend
 /** A hosted Archive library a held Signet may write to. */
 export type ArchiveLibrary = { integrationId: string; resourceId: string; name: string };
 export type ArchiveLibraries = { paired: boolean; libraries: ArchiveLibrary[] };
+export type RoutingPreview = Awaited<ReturnType<typeof routingPreview>>;
+export type SyncResult = Awaited<ReturnType<typeof syncArchives>>[number];
 /** A project routed to a hosted Archive library through Kingdom. */
 export type KingdomRoute = { projectId: string; integrationId: string; resourceId: string };
 
@@ -162,6 +165,14 @@ export class ArchiveClient {
   libraries() {
     return this.request<ArchiveLibraries>('destinations/libraries');
   }
+  /** Each archive and the destinations its project routes to; nothing is sent. */
+  routes() {
+    return this.request<RoutingPreview>('destinations/routes');
+  }
+  /** Publishes every archive to its project's destinations once. */
+  sync() {
+    return this.request<SyncResult[]>('destinations/sync', {}, { timeoutMs: 600_000 });
+  }
   /** Routes a project to a library; the server verifies its Signet grants sessions.write first. */
   connectDestination(route: KingdomRoute) {
     return this.request<DescribedDestination>('destinations/connect', route);

package/src/server.ts

@@ -1,5 +1,6 @@
 import { timingSafeEqual } from 'node:crypto';
 import { z } from 'zod';
+import { routingPreview, syncArchives } from './client';
 import {
   connectDestination,
   describeDestination,
@@ -195,6 +196,16 @@ export function createArchiveHandler(
             });
         return json({ data: { destinations } });
       }
+      if (config && path === '/api/v1/archive/destinations/routes') {
+        z.strictObject({}).parse(body);
+        return json({
+          data: await routingPreview(store, readDestinations(config.destinationsFile)),
+        });
+      }
+      if (config && path === '/api/v1/archive/destinations/sync') {
+        z.strictObject({}).parse(body);
+        return json({ data: await syncArchives(store, readDestinations(config.destinationsFile)) });
+      }
       if (config && path === '/api/v1/archive/destinations/libraries') {
         z.strictObject({}).parse(body);
         return json({ data: await pairedLibraries(config.kingdomDirectory) });