@jarenjs/db 0.67.0 → 0.72.0

package/README.md

@@ -688,6 +688,15 @@ opinion: <!--fact:vector.ceiling-->5.142 ns per vector component — one query r
 Past that this design is the wrong tool and no margin changes it; what
 lies beyond is an approximate index, and this store does not have one.
 
+The [labelled retrieval comparison](../../benchmark/README.md#labelled-recall-and-repeated-refinement)
+also scores an optional storage adapter with approximate candidates against
+exact cosine on cached live embeddings. The AI ledger reports the adapter's
+algorithm, exhaustive flag and returned candidate count, validates candidate
+identity/vector shape, and re-scores candidates itself. The DB ledger recipe
+continues to normalize as an exact adapter. No ANN runtime dependency is added.
+
+Index decision: <!--fact:recall.annDecision-->0/6 contender rows cleared all bars; retain exact. Required exact-top-10 recall ≥ 0.95, p95 speedup ≥ 2×, and a measured exact p95 ≥ 100 ms. The largest reference corpus contains 5183 documents; scale beyond it remains unmeasured.<!--/fact-->
+
 **One document, three executors, proven to agree.** As with the spatial
 family, the k-nearest shapes of a committed corpus
 (`test/json/fixtures/vector-corpus.json`) run through the JavaScript
@@ -1028,6 +1037,10 @@ shutdown limits, the browser persistence matrix and measured latency/memory loss
   and `worker.stop()` quiesces every claim, renewal, checkpoint and
   settlement before it resolves, so closing the store releases the
   database file deterministically.
+  DAG jobs inspect persisted workflow/input/task identity before reading
+  checkpoint values. `jobs.reset(id, { expectedGeneration })` explicitly
+  discards an inactive run's checkpoints and restarts it under a new fence;
+  live leases and stale observations refuse (JOBS-FORMAT §10).
 - **The browser** (`@jarenjs/db/wasm`): the same store, the same
   queries, the same live updates run on the official SQLite wasm build
   over a probed persistence ladder: isolated SharedArrayBuffer OPFS,
@@ -1335,3 +1348,19 @@ Every subpath a consumer can import, derived from the manifest by
 | `@jarenjs/db/node-worker` | JavaScript | declared |
 | `@jarenjs/db/node-pool` | JavaScript | declared |
 <!--/fact-->
+
+
+The [durable AI ledger recipe](../ai/README.md#a-durable-ledger-over-jarenjsdb)
+exposes optional atomic namespace mutation as well as its base storage and rank
+capabilities. It reads and publishes through one immediate transaction, using the
+transaction's synchronous collection facade so another SQLite connection on the
+same event loop cannot block an awaiting writer. A rejected callback changes
+neither records nor counters; ledger id minting and guarded multi-record updates
+use that boundary.
+
+Document-only migrations also accept named files:
+`jaren-db documents --migrations migrations --in users=users.json --in events=events.jsonl --out migrated.json`.
+The output is one atomically published collection bundle; read it with
+`--format collections`. Assertion plans expose provider, ordered-fold and
+bounded materialization strategies through `onAssertionPlan`.
+See [MIGRATION-FORMAT §6 and §11](docs/MIGRATION-FORMAT.md).

package/docs/JOBS-FORMAT.md

@@ -390,7 +390,10 @@ await store.jobs.enqueue('sync-report', { input: { day: '2026-08-05' } });
 
   **The legacy rule is deterministic, and never reads unknown as equal.**
   A run checkpointed by a release that did not record task identity has
-  no `taskVersionsHash`. If it recorded no node value yet, there is
+  no `taskVersionsHash`. Identity inspection reads only the reserved metadata
+  value and probes whether other rows exist; it does not load or parse node
+  values. Missing metadata with saved values refuses `JD2069`. Refused
+  legacy upgrades leave the original identity untouched. If it recorded no node value yet, there is
   nothing that could be replayed wrongly, so the identity is upgraded in
   place and the run proceeds. If it DID record values, the implementation
   that produced them cannot be confirmed and the resume is refused,
@@ -471,6 +474,16 @@ a cancellation policy, a retry policy are the host's.
   and counts it under `stats().cancellations`; an attempt held by
   another process meets the fence at its next settling call and records
   a loss, as any superseded attempt does.
+- **`reset(id, { expectedGeneration })`** — explicitly discard one inactive
+  run's checkpoints and restart its attempts at zero. Read the job's
+  `leaseGeneration` first and pass that observation. One transaction
+  conditionally updates the job, increments its fence generation and deletes
+  its checkpoint rows. A concurrent generation change refuses `JD2066`, a
+  live lease `JD2068`, an unknown or completed job `JD2065`; no refusal
+  discards values. The next claim recomputes the workflow under current
+  identity. Reset does not undo external task effects; their idempotency
+  keys remain the host's responsibility. It is root-only administration,
+  with the same signal/deadline options as requeue.
 - **`requeue(id)`** — returns a failed, dead, cancelled or lease-expired
   job to `pending` at the clock's instant (the queue's own order). The
   attempt history is KEPT: `attempts` counts every claim the job ever

package/docs/MIGRATION-FORMAT.md

@@ -235,25 +235,34 @@ hash of the `baseline` model when no migration has run.
 
   | strategy | which assertions | what it costs |
   |---|---|---|
-  | per-document | a FLWOR over `$[*]` whose `$where`/`$return` read only the binding | one keyset batch at a time; fails fast at the first batch that violates |
-  | fold | exactly one of `$count`, `$sum`, `$min`, `$max` over the root | one batch at a time; each batch is answered by the ENGINE and the partial answers combine |
-  | materialize | everything else (`$let`, `$distinct`, a nested `$for`, two aggregates) | every document at once, under `assertionBounds` |
-
-  A fold is sound because the operator is associative: the answer over a
-  collection is the combination of the answers over any partition of it.
-  Nothing reimplements an operator — each batch is evaluated by the same
-  compiled query the whole-collection path would use, and only the
-  COMBINE step is written here, so null handling, empty-sequence answers
-  and type coercions are the engine's. A suite runs every fold shape both
-  ways, over ten corpora and six partitions, and requires the value and
-  the verdict to be indistinguishable; a shape that cannot pass it is not
-  in the set.
+  | provider | a collection `$count` that the existing query planner proves native without a typed intermediate schema | one aggregate result, with no assertion document fetch |
+  | per-document | an independent, unwindowed FLWOR over `$[*]`, with the default empty-sequence expectation | one keyset batch; fails at the first violation |
+  | fold | `$count`, `$sum`, `$avg`, `$min`, `$max` over a partition-independent operand; sequence EBV of an independent FLWOR | one batch plus fixed accumulator state |
+  | fold with bounds | `$distinct` over such an operand with an explicit positive `maxDistinct` | one batch plus unique items under cardinality and byte bounds |
+  | materialize | global or positional operands, nested scans, unsupported shapes, or unbounded distinct | the collection under `assertionBounds` |
+
+  Folds feed the operand's sequence items, in original order, into the
+  query engine's shared accumulator. Floating-point addition is **not
+  associative**: combining batch totals changes answers. The ordered state
+  preserves the whole-query result, including singleton arrays, negative
+  zero, NaN, mixed-type refusals and Unicode code-point ordering. Sequence
+  EBV is checked once across all batches, including an empty source.
+  `$[0]`, root-dependent filters and positional bindings never qualify as
+  independent merely because they occur inside an aggregate.
+
+  `classifyAssertion(query, { expect, maxDistinct })` exposes the portable
+  strategy. `onAssertionPlan(plan)` reports migration, step, collection,
+  strategy, shape, reason and bounds before the assertion reads documents;
+  Store dry-run query statements also expose strategy and reason. Provider
+  promotion reuses the query planner with no assumed typed columns. Typed
+  SQL aggregates still need a trustworthy intermediate schema and proof
+  that their numeric/order semantics match; otherwise the ordered fold runs.
 
   **A materializing assertion is bounded.** `options.assertionBounds`
   defaults to `{ maxRows: 100000, maxBytes: 67108864 }` and is crossed
   BEFORE the excess is held — the walk stops at the row that would break
   it, refusing `JD2007` (rows) or `JD2076` (bytes) and naming the two
-  assertion shapes that are answered in batches instead. `null` on either
+  assertion strategies that are answered in batches instead. `null` on either
   member removes that bound, which a caller must ask for: an unbounded
   read nobody declared is exactly what this classification removes. This
   is a deliberate behavior change — a migration that used to read a very
@@ -265,14 +274,11 @@ hash of the `baseline` model when no migration has run.
   transformed | derived | asserted }`, one event per batch), and never
   hold the whole collection in memory. A PER-DOCUMENT assertion — a
   FLWOR over `$[*]` whose `$where` and `$return` read only the binding
-  — walks the same batches and fails fast at the first batch that
-  violates, because its answer over each batch is its answer over the
-  whole. A cross-document assertion (one that reads the root: `$count:
-  '$[*]'`, a `$let`, a `$distinct`, a nested `$for`) reads the whole
-  collection into one array — a stated cost; keep such assertions
-  early, before the data grows. A cross-document assertion that is one
-  associative aggregate no longer costs that read at all — see the
-  classification table above.
+  — with the default empty expectation walks the same batches and fails
+  fast. Other assertions follow the classification table above.
+  `batchSize` must be a positive safe integer. `maxDistinct` opts into a
+  distinct fold and bounds retained unique items; `maxBytes` applies to
+  those items. Duplicates do not consume additional cardinality.
 - A transform MUST NOT change a caller-keyed document's key member —
   the key column would go stale; the run refuses (`JD0023`).
 
@@ -510,10 +516,23 @@ jaren-db documents --migrations <dir> --in <file|-> (--out <file|-> | --in-place
   line-delimited, everything else one JSON array) unless `--format` /
   `--out-format` says otherwise, and stdio defaults to JSONL. Input and
   output encodings are independent, so this is also the converter.
-  - **A file holds ONE collection.** The migrations name it; a chain
-    whose document steps touch more than one cannot be applied to a
-    file, and is refused rather than partly run. `--collection` asserts
-    which collection the file holds and refuses a mismatch.
+  - **Named sources share one collection bundle.** Repeat
+    `--in users=users.json --in events=events.jsonl --out migrated.json`.
+    Output defaults to `--out-format collections`: a standard JSON object
+    mapping collection names to arrays, including empty collections.
+    Publication is one atomic rename after every collection succeeds.
+    Multiple source files cannot use `--in-place`, because separate
+    renames cannot give this guarantee. Single-source `--collection`
+    still asserts the collection name and refuses a mismatch.
+  - **A bundle can be read explicitly** with `--format collections`, then
+    replaced using `--in-place --yes`. Bundle input is materialized under
+    a whole-file `maxBytes` ceiling before parsing and per-collection
+    `maxRows` validation after parsing. JSON and JSONL single-collection
+    sources remain streaming. `--max-rows`, `--max-bytes`, and
+    `--max-distinct` configure assertion admission too; the first two
+    accept `none` to remove their limit deliberately. A materializing
+    file run admits rows before retaining them. Physical-step and query
+    compilation refusals precede every input read.
   - **`--out` writes elsewhere; `--in-place` replaces the input and
     needs `--yes`.** Either way the documents land in a sibling
     temporary that is renamed over the target only once every document

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@jarenjs/db",
   "private": false,
-  "version": "0.67.0",
+  "version": "0.72.0",
   "type": "module",
   "main": "./src/index.js",
   "types": "./types/index.d.ts",
@@ -84,9 +84,9 @@
     "prepack": "npm run build:types"
   },
   "dependencies": {
-    "@jarenjs/core": "^0.67.0",
-    "@jarenjs/json": "^0.67.0",
-    "@jarenjs/validate": "^0.67.0"
+    "@jarenjs/core": "^0.72.0",
+    "@jarenjs/json": "^0.72.0",
+    "@jarenjs/validate": "^0.72.0"
   },
   "bin": {
     "jaren-db": "./src/cli.js"

package/src/cli.js

@@ -21,8 +21,10 @@ import {
   migrateDocuments, streamDocuments, classifyAssertion,
 } from './index.js';
 import { nodeDriver } from './drivers/node.js';
+import { prepareDocumentRun } from './documents.js';
+import { normalizeAssertionBounds, createAssertionBoundGuard } from './document-steps.js';
 import {
-  readDocuments, openAtomicTarget, openStreamTarget, openNullTarget,
+  readDocuments, readCollectionBundle, openAtomicTarget, openStreamTarget, openNullTarget,
   formatOf, DOCUMENT_FORMATS,
 } from './document-files.js';
 
@@ -37,7 +39,9 @@ Usage:
   jaren-db check    --model <model> --store <db> [--migrations <dir>] [--snapshot <file>]
   jaren-db shape    --model <model>
   jaren-db documents --migrations <dir> --in <file|-> (--out <file|-> | --in-place --yes | --check)
-                     [--format json|jsonl] [--out-format json|jsonl] [--collection <name>] [--batch-size <n>]
+                     [--format json|jsonl|collections] [--out-format json|jsonl|collections]
+                     [--in collection=file ...] [--collection <name>] [--batch-size <n>]
+                     [--max-rows <n|none>] [--max-bytes <n|none>] [--max-distinct <n>]
 
 A <model> or a migration is a .json file, or a MODULE (.js, .mjs, .cjs —
 or .ts where Node strips types) whose default export, or its 'model' /
@@ -96,6 +100,7 @@ function parseArgs(argv) {
     command: argv[2], from: null, to: null, model: null, store: null,
     baseline: null, migrations: null, id: null, out: null,
     snapshot: null, types: null,
+    inputs: [], maxRows: undefined, maxBytes: undefined, maxDistinct: undefined,
     in: null, format: null, outFormat: null, collection: null,
     batchSize: null, inPlace: false, check: false,
     dryRun: false, yes: false, help: false,
@@ -112,7 +117,10 @@ function parseArgs(argv) {
       case '--out': options.out = argv[++i]; break;
       case '--snapshot': options.snapshot = argv[++i]; break;
       case '--types': options.types = argv[++i]; break;
-      case '--in': options.in = argv[++i]; break;
+      case '--in': options.in = argv[++i]; options.inputs.push(options.in); break;
+      case '--max-rows': options.maxRows = argv[++i]; break;
+      case '--max-bytes': options.maxBytes = argv[++i]; break;
+      case '--max-distinct': options.maxDistinct = argv[++i]; break;
       case '--out-format': options.outFormat = argv[++i]; break;
       case '--format': options.format = argv[++i]; break;
       case '--collection': options.collection = argv[++i]; break;
@@ -435,94 +443,107 @@ async function commandShape(options) {
  */
 async function commandDocuments(options) {
   if (options.migrations === null) misuse('documents needs --migrations <dir>');
-  if (options.in === null) misuse('documents needs --in <file> (or - for standard input)');
+  if (options.inputs.length === 0) misuse('documents needs --in <file> (or - for standard input)');
+  if (options.inputs.some((input) => typeof input !== 'string' || input.startsWith('--')))
+    misuse('--in requires a file or collection=file');
+  const multiple = options.inputs.length > 1 || /^[A-Za-z_][A-Za-z0-9_]*=/.test(options.in);
+  if (multiple && options.inPlace)
+    misuse('multiple sources require one --out collection bundle; several in-place renames are not atomic');
   const sinks = [options.out !== null, options.inPlace, options.check].filter(Boolean).length;
-  if (sinks === 0)
-    misuse('documents needs one of --out <file>, --in-place or --check');
-  if (sinks > 1)
-    misuse('documents takes exactly one of --out, --in-place and --check');
-  if (options.inPlace && options.in === '-')
-    misuse('--in-place needs a file to replace, not standard input');
-  if (options.inPlace && !options.yes)
-    misuse('--in-place rewrites the input file — pass --yes to confirm, or --out to write elsewhere');
+  if (sinks !== 1) misuse('documents takes exactly one of --out <file>, --in-place and --check');
+  if (options.inPlace && options.in === '-') misuse('--in-place needs a file to replace, not standard input');
+  if (options.inPlace && !options.yes) misuse('--in-place rewrites the input file — pass --yes to confirm, or --out to write elsewhere');
   const batchSize = options.batchSize === null ? 500 : Number(options.batchSize);
-  if (!Number.isInteger(batchSize) || batchSize < 1)
-    misuse(`--batch-size must be a positive integer, not '${options.batchSize}'`);
-
-  const fromStdin = options.in === '-';
-  const inFormat = options.format ?? (fromStdin ? 'jsonl' : formatOf(options.in));
-  if (!DOCUMENT_FORMATS.includes(inFormat))
-    misuse(`--format must be one of ${DOCUMENT_FORMATS.join(', ')}, not '${inFormat}'`);
+  if (!Number.isSafeInteger(batchSize) || batchSize < 1) misuse('--batch-size must be a positive integer');
+  const declared = {};
+  for (const name of ['maxRows', 'maxBytes', 'maxDistinct']) {
+    if (options[name] === undefined) continue;
+    declared[name] = options[name] === 'none' ? null : Number(options[name]);
+  }
+  let assertionBounds;
+  try { assertionBounds = normalizeAssertionBounds(declared); }
+  catch (error) { misuse(error.message); }
+  const inFormat = options.format;
+  const bundleInput = inFormat === 'collections';
+  if (inFormat !== null && ![...DOCUMENT_FORMATS, 'collections'].includes(inFormat))
+    misuse('--format must be one of json, jsonl or collections');
+  if (bundleInput && multiple) misuse('--format collections takes one bundle file');
   const target = options.inPlace ? options.in : options.out;
   const toStdout = target === '-';
-  const outFormat = options.outFormat
-    ?? (options.check || toStdout ? inFormat : formatOf(/** @type {string} */ (target)));
-  if (!DOCUMENT_FORMATS.includes(outFormat))
-    misuse(`--out-format must be one of ${DOCUMENT_FORMATS.join(', ')}, not '${outFormat}'`);
-  if (!fromStdin && !fs.existsSync(options.in)) fail(`no such file: '${options.in}'`);
-
   const migrations = await loadMigrationsDir(options.migrations);
   if (migrations.length === 0) fail(`no migration documents in '${options.migrations}'`);
-
-  // A document file holds ONE collection. The migrations say which:
-  // every document step must name it, or this chain cannot be applied to
-  // a file at all — running only the steps that match would leave the
-  // rest silently unapplied, which is the one outcome a migration runner
-  // may never produce.
   const documentSteps = migrations.flatMap((migration) => migration.steps)
     .filter((step) => step.kind === 'jslt' || step.kind === 'query');
   const named = [...new Set(documentSteps.map((step) => step.collection))];
-  if (named.length > 1) {
-    fail(`a document file holds one collection, and these migrations touch ${named.length} `
-      + `(${named.join(', ')}) — run them against a store, or split the chain so each `
-      + 'migration touches the collection its file holds');
-  }
-  if (options.collection !== null && named.length === 1 && options.collection !== named[0]) {
-    misuse(`--collection names '${options.collection}', but these migrations touch `
-      + `'${named[0]}' — is this the right file for them?`);
-  }
-  // with no document step at all there is no collection to infer; the
-  // run still proceeds, because a physical step must be REFUSED by name
-  // rather than reported as a missing collection
-  const collection = named[0] ?? options.collection ?? 'documents';
-
-  // only a MATERIALIZING assertion needs the collection at once; a
-  // per-document predicate and an associative aggregate are both
-  // answered one batch at a time, so they stream
-  const materializes = documentSteps.some((step) => step.collection === collection
-    && step.kind === 'query' && classifyAssertion(step.assert).strategy === 'materialize');
-
-  const source = () => (fromStdin ? process.stdin : options.in);
+  const files = {};
+  if (multiple) {
+    if (options.collection !== null) misuse('named --in sources already declare their collections');
+    for (const input of options.inputs) {
+      const match = /^([A-Za-z_][A-Za-z0-9_]*)=(.+)$/.exec(input);
+      if (match === null) misuse('each named --in must be collection=file');
+      if (Object.hasOwn(files, match[1])) misuse(`duplicate input collection '${match[1]}'`);
+      Object.defineProperty(files, match[1], { value: match[2], enumerable: true });
+    }
+    if (Object.values(files).filter((file) => file === '-').length > 1) misuse('standard input can supply only one collection');
+  }
+  else if (!bundleInput) {
+    if (named.length > 1) fail(`a document file holds one collection, but these migrations touch ${named.join(', ')}; supply --in collection=file for every collection, or --format collections`);
+    const name = named[0] ?? options.collection ?? 'documents';
+    if (options.collection !== null && options.collection !== name) misuse(`--collection names '${options.collection}', but these migrations touch '${name}'`);
+    Object.defineProperty(files, name, { value: options.in, enumerable: true });
+  }
+  const runOptions = { batchSize, assertionBounds };
+  // Physical and compilation refusals precede every document read, including
+  // the materializing route. A bundle's actual collection inventory is checked
+  // again after its bounded parse.
+  prepareDocumentRun(bundleInput ? named : Object.keys(files), migrations, runOptions);
+  let state;
+  if (bundleInput) state = await readCollectionBundle(options.in === '-' ? process.stdin : options.in, assertionBounds);
+  const names = bundleInput ? Object.keys(state) : Object.keys(files);
+  prepareDocumentRun(names, migrations, runOptions);
+  const outFormat = options.outFormat ?? (multiple || bundleInput ? 'collections'
+    : options.check || toStdout ? inFormat ?? (options.in === '-' ? 'jsonl' : formatOf(options.in)) : formatOf(target));
+  if (!['json', 'jsonl', 'collections'].includes(outFormat)) misuse('--out-format must be json, jsonl or collections');
+  if (names.length > 1 && outFormat !== 'collections') misuse('multiple collections require --out-format collections');
+  const materializes = bundleInput || documentSteps.some((step) => step.kind === 'query'
+    && classifyAssertion(step.assert, { expect: step.expect, maxDistinct: assertionBounds.maxDistinct }).strategy === 'materialize');
+  const sources = Object.fromEntries(Object.entries(files).map(([name, file]) => [name,
+    readDocuments(file === '-' ? process.stdin : file, inFormat ?? (file === '-' ? 'jsonl' : formatOf(file)))]));
   let sink;
   if (options.check) sink = openNullTarget();
-  else if (toStdout) sink = openStreamTarget(process.stdout, outFormat);
-  else sink = await openAtomicTarget(/** @type {string} */ (target), outFormat);
-
+  else if (toStdout) sink = openStreamTarget(process.stdout, outFormat, { collections: names });
+  else sink = await openAtomicTarget(target, outFormat, { collections: names });
   try {
     let report;
     if (materializes) {
-      const documents = [];
-      for await (const document of readDocuments(source(), inFormat)) documents.push(document);
-      const out = await migrateDocuments({ [collection]: documents }, migrations, { batchSize });
+      if (!bundleInput) {
+        state = {};
+        for (const name of names) {
+          const documents = [], guard = createAssertionBoundGuard(assertionBounds, name, 'the file runner');
+          for await (const document of sources[name]) { guard.admit(document); documents.push(document); }
+          Object.defineProperty(state, name, { value: documents, enumerable: true });
+        }
+      }
+      const out = await migrateDocuments(state, migrations, runOptions);
       report = out.report;
-      for (const document of out.documents[collection]) await sink.write(document);
-    }
-    else {
-      report = await streamDocuments({ [collection]: readDocuments(source(), inFormat) },
-        migrations, { batchSize, write: (name, document) => sink.write(document) });
+      for (const name of names) for (const document of out.documents[name]) await sink.write(document, name);
     }
+    else report = await streamDocuments(sources, migrations,
+      { ...runOptions, write: (name, document) => sink.write(document, name) });
     const written = await sink.commit();
-    const counts = report.counts[collection] ?? { read: 0, transformed: 0, asserted: 0 };
-    console.log(`${options.check ? 'checked' : 'migrated'} '${collection}': `
-      + `${counts.read} read, ${counts.transformed} transformed, ${counts.asserted} asserted `
-      + `(${report.strategy[collection]})`);
+    for (const name of names) {
+      const counts = report.counts[name] ?? { read: 0, transformed: 0, asserted: 0 };
+      console.log(`${options.check ? 'checked' : 'migrated'} '${name}': ${counts.read} read, `
+        + `${counts.transformed} transformed, ${counts.asserted} asserted (${report.strategy[name]})`);
+    }
     console.log(`applied: ${report.applied.join(', ')}`);
     if (options.check) console.log('checked only — nothing was written');
     else if (toStdout) console.log(`wrote ${written.documents} document(s) to standard output`);
     else console.log(`wrote ${written.documents} document(s) to ${target} (${written.bytes} bytes)`);
   }
   catch (error) {
-    await sink.abort();
+    try { await sink.abort(); }
+    catch (cleanupError) { error.cleanupError = cleanupError; }
     return fail(error.message);
   }
 }

package/src/dag-job.js

@@ -48,7 +48,7 @@ const fingerprint = (value) => hashContent(canonicalizeJson(value ?? null));
  * @param {any} store - an open store with `{ jobs: true }`
  * @param {{ compileDag: Function,
  *   documents: Record<string, any>,
- *   tasks?: Record<string, Function>,
+ *   tasks?: Record<string, Function | { run: Function, version?: string, taskVersions?: Record<string, string> }>,
  *   concurrency?: number, pollInterval?: number, leaseMs?: number,
  *   owner?: string, renew?: boolean, onOutcome?: (event: any) => void,
  *   backoffBase?: number, backoffCap?: number,
@@ -130,14 +130,20 @@ export function createDagJobRunner(store, options) {
    * just as surely as an edited document does.
    */
   const requireSameRun = async (context, jobId, revision, inputHash, taskVersions) => {
-    const loaded = await context.checkpoints.load(jobId);
-    const stored = loaded?.values?.[RUN_IDENTITY_NODE];
+    const loaded = await context.checkpoints.inspect(jobId, RUN_IDENTITY_NODE);
+    const stored = loaded.value;
     const taskVersionsHash = fingerprint(taskVersions);
     const identity = { revision, inputHash, taskVersionsHash, taskVersions };
     if (stored === undefined) {
+      if (loaded.hasValues) throw new DbRuntimeError('JD2069',
+        `run '${jobId}' cannot resume: checkpoint values have no recorded workflow, input or task identity`,
+        { docPath: '/jobs', collection: jobId });
       await context.checkpoints.save(jobId, RUN_IDENTITY_NODE, identity);
       return;
     }
+    if (stored === null || typeof stored !== 'object' || Array.isArray(stored))
+      throw new DbRuntimeError('JD2069', `run '${jobId}' has invalid checkpoint identity metadata`,
+        { docPath: '/jobs', collection: jobId });
     const differs = [];
     if (stored.revision !== revision) {
       differs.push(`the workflow (checkpointed under revision ${stored.revision}, `
@@ -152,13 +158,8 @@ export function createDagJobRunner(store, options) {
       // equal: the upgrade is allowed only where nothing can be replayed
       // wrongly — when no node value has been recorded yet, so the run has
       // nothing to inherit from an implementation nobody can name.
-      const recorded = Object.keys(loaded?.values ?? {})
-        .filter((nodeId) => nodeId !== RUN_IDENTITY_NODE);
-      if (recorded.length === 0) {
-        await context.checkpoints.save(jobId, RUN_IDENTITY_NODE, identity);
-      }
-      else {
-        differs.push(`the task versions (this run recorded ${recorded.length} node value(s) `
+      if (loaded.hasValues) {
+        differs.push(`the task versions (this run recorded node value(s) `
           + 'before task identity was persisted, so the implementation that produced them '
           + 'cannot be confirmed)');
       }
@@ -169,10 +170,14 @@ export function createDagJobRunner(store, options) {
         : `checkpointed under ${stored.taskVersionsHash}, this runner compiles `
           + `${taskVersionsHash}`})`);
     }
-    if (differs.length === 0) return;
+    if (differs.length === 0) {
+      if (stored.taskVersionsHash === undefined)
+        await context.checkpoints.save(jobId, RUN_IDENTITY_NODE, identity);
+      return;
+    }
     throw new DbRuntimeError('JD2069',
       `run '${jobId}' cannot resume: ${differs.join(' and ')} changed since its `
-      + 'checkpoints were written. Enqueue it under a new id, or drop the run — '
+      + 'checkpoints were written. Enqueue it under a new id, or explicitly reset this inactive run with its observed generation — '
       + 'reusing them would answer for a computation nobody asked for',
       { docPath: '/jobs', collection: jobId });
   };

package/src/document-files.js

@@ -208,6 +208,65 @@ export function readDocuments(source, format) {
   throw refuse(`unknown document format '${format}' — one of ${DOCUMENT_FORMATS.join(', ')}`);
 }
 
+/**
+ * Read an explicitly materialized collection-map bundle under a byte ceiling.
+ * JSON/JSONL collection readers remain streaming; a bundle holds named arrays.
+ * @param {string | AsyncIterable<any>} source
+ * @param {{ maxBytes: number | null, maxRows: number | null }} bounds
+ */
+export async function readCollectionBundle(source, bounds) {
+  let text = '', bytes = 0;
+  for await (const chunk of textChunks(source)) {
+    bytes += Buffer.byteLength(chunk);
+    if (bounds.maxBytes !== null && bytes > bounds.maxBytes)
+      throw refuse(`the collection bundle exceeds its maxBytes bound of ${bounds.maxBytes}`);
+    text += chunk;
+  }
+  const value = parseDocument(text, 'collection bundle');
+  if (value === null || typeof value !== 'object' || Array.isArray(value)
+    || Object.values(value).some((documents) => !Array.isArray(documents)))
+    throw refuse('a collection bundle is an object mapping collection names to document arrays');
+  for (const [name, documents] of Object.entries(value))
+    if (bounds.maxRows !== null && documents.length > bounds.maxRows)
+      throw refuse(`collection '${name}' exceeds its maxRows bound of ${bounds.maxRows}`);
+  return value;
+}
+
+/** One encoder for file and stream output, including atomic collection bundles. */
+function documentEncoder(format, collections = []) {
+  if (!['json', 'jsonl', 'collections'].includes(format)) throw refuse(`unknown document format '${format}'`);
+  if (format === 'collections' && (new Set(collections).size !== collections.length
+    || collections.some((name) => typeof name !== 'string')))
+    throw refuse('collection bundle names must be unique strings');
+  let documents = 0, index = -1, members = 0;
+  const advance = (next) => {
+    let text = '';
+    while (index < next) {
+      text += index < 0 ? '{\n' : '\n],\n';
+      index++;
+      text += `${JSON.stringify(collections[index])}: [`;
+      members = 0;
+    }
+    return text;
+  };
+  return {
+    write: (document, collection) => {
+      const value = JSON.stringify(document);
+      if (value === undefined) throw refuse('a document must be JSON serializable');
+      if (format === 'jsonl') { documents++; return `${value}\n`; }
+      if (format === 'json') return documents++ === 0 ? `[\n${value}` : `,\n${value}`;
+      const next = collections.indexOf(collection);
+      if (next < index || next < 0) throw refuse('collection bundle writes must follow the declared collection order');
+      const prefix = advance(next);
+      documents++;
+      return prefix + (members++ === 0 ? '\n' : ',\n') + value;
+    },
+    finish: () => format === 'jsonl' ? '' : format === 'json' ? documents === 0 ? '[]\n' : '\n]\n'
+      : collections.length === 0 ? '{}\n' : advance(collections.length - 1) + '\n]\n}\n',
+    count: () => documents,
+  };
+}
+
 /**
  * A sink that publishes whole or not at all.
  *
@@ -217,17 +276,18 @@ export function readDocuments(source, format) {
  * a target that never changed.
  *
  * @param {string} target - the file to replace
- * @param {'json' | 'jsonl'} format
- * @returns {Promise<{ write: (document: any) => Promise<void>,
+ * @param {'json' | 'jsonl' | 'collections'} format
+ * @param {{ collections?: string[] }} [options]
+ * @returns {Promise<{ write: (document: any, collection?: string) => Promise<void>,
  *   commit: () => Promise<{ bytes: number, documents: number }>,
  *   abort: () => Promise<void>, temporary: string }>}
  */
-export async function openAtomicTarget(target, format) {
+export async function openAtomicTarget(target, format, options = {}) {
+  const encoder = documentEncoder(format, options.collections);
   const directory = path.dirname(path.resolve(target));
   const temporary = path.join(directory,
     `.${path.basename(target)}.${process.pid}.${Date.now()}.tmp`);
   const handle = await fsp.open(temporary, 'wx');
-  let documents = 0;
   let bytes = 0;
   let settled = false;
 
@@ -243,21 +303,20 @@ export async function openAtomicTarget(target, format) {
 
   return {
     temporary,
-    write: async (document) => {
-      const text = JSON.stringify(document);
-      if (format === 'jsonl') { documents++; return put(`${text}\n`); }
-      return put(documents++ === 0 ? `[\n${text}` : `,\n${text}`);
+    write: async (document, collection) => {
+      if (settled) throw refuse('this target was already settled');
+      await put(encoder.write(document, collection));
     },
     commit: async () => {
       if (settled) throw refuse('this target was already settled');
       settled = true;
       try {
-        if (format === 'json') await put(documents === 0 ? '[]\n' : '\n]\n');
+        await put(encoder.finish());
         // Publish only after the temporary has been durably flushed.
         await handle.sync();
         await handle.close();
         await fsp.rename(temporary, target);
-        return { bytes, documents };
+        return { bytes, documents: encoder.count() };
       }
       catch (error) {
         try { await discard(); }
@@ -277,23 +336,20 @@ export async function openAtomicTarget(target, format) {
  * A sink that writes to an open stream (standard output) and can never
  * be taken back — `abort` is honest that what left has left.
  * @param {{ write: (chunk: string, callback: (error?: any) => void) => any }} stream
- * @param {'json' | 'jsonl'} format
+ * @param {'json' | 'jsonl' | 'collections'} format
+ * @param {{ collections?: string[] }} [options]
  */
-export function openStreamTarget(stream, format) {
-  let documents = 0;
+export function openStreamTarget(stream, format, options = {}) {
+  const encoder = documentEncoder(format, options.collections);
   const put = (text) => new Promise((resolve, reject) => {
     stream.write(text, (error) => (error ? reject(error) : resolve(undefined)));
   });
   return {
     temporary: null,
-    write: async (document) => {
-      const text = JSON.stringify(document);
-      if (format === 'jsonl') { documents++; return put(`${text}\n`); }
-      return put(documents++ === 0 ? `[\n${text}` : `,\n${text}`);
-    },
+    write: async (document, collection) => put(encoder.write(document, collection)),
     commit: async () => {
-      if (format === 'json') await put(documents === 0 ? '[]\n' : '\n]\n');
-      return { bytes: 0, documents };
+      await put(encoder.finish());
+      return { bytes: 0, documents: encoder.count() };
     },
     abort: async () => undefined,
   };