@awebai/oats 0.22.0 → 0.22.2

package/package-catalog.json

@@ -2,12 +2,12 @@
   "packages": {
     "oats.okf": {
       "url": "https://github.com/awebai/oats-okf.git",
-      "ref": "v1.4.1",
+      "ref": "v1.5.0",
       "path": "oats-package"
     },
     "oats.aweb": {
       "url": "https://github.com/awebai/oats-aweb.git",
-      "ref": "v1.8.0",
+      "ref": "v1.9.0",
       "path": "oats-package"
     },
     "oats.jira": {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@awebai/oats",
-  "version": "0.22.0",
+  "version": "0.22.2",
   "description": "OATS (Open Agent Team Specification) — durable souls, disposable instances, targetable capability packages, and the runtime-neutral oats CLI/kernel.",
   "keywords": [
     "agents",

package/packages/record/README.md

@@ -6,12 +6,17 @@ contract is [`docs/turn-record-sot.md`](docs/turn-record-sot.md) in this
 package; the conformance vectors under `test/vectors/` pin the format
 (`node test/vectors/validate.mjs`, dependency-free).
 
-Everything is a signed turn in an append-only record. This package is the
-core: gathering every conversation, keeping it durably, and finding it
-again. The experimental tools that select and synthesize over the record
-(dress, spawn, segments, mind) live in `packages/experimental` of the oats
-repo and run as `oats experimental <cmd>`; they are deliberately not part
-of this package.
+Turns in this append-only record are content-addressed. Native session turns
+are not signed. Projected aweb mail and chat keep their original message
+signatures verbatim.
+
+After `turn-record setup` runs on a machine, this package gathers Claude Code,
+Pi, and Codex transcripts plus aw client logs. It skips sources matched by the
+local record's ignore list. The package keeps captured conversations durably
+and finds them again. The experimental tools that select and synthesize over
+the record (dress, spawn, segments, mind) live in `packages/experimental` of
+the oats repo and run as `oats experimental <cmd>`. They are deliberately not
+part of this package.
 
 - **`lib/canonical.mjs`** — canonical JSON (integers only in the core),
   `t1:` content ids, did:key Ed25519 verification. Byte-compatible with awid
@@ -126,16 +131,71 @@ non-prefix copy of a stream quarantines it loudly.
 
 ## Durability and concurrency
 
-Appends and merges run under a per-stream lockfile
-(`streams/<id>/.lock`, exclusive-create, stale after 30 s), because the
-torn-tail repair is a read-truncate-write sequence and hooks, watchers and
-manual passes can fire concurrently for the same owner. Sync tools may copy
-a `.lock` file; that cannot corrupt data, but a synced-in stale lock can
-delay a local `mergeStreamCopy` on that stream by up to the stale threshold
-(30 s), so excluding `.lock` from sync patterns is the right configuration. Journal writes fsync;
-note that on macOS `fsync(2)` does not guarantee media durability (that
-would need `F_FULLFSYNC`, which Node's fs API does not expose) — the
-guarantee is OS-crash-level, not power-loss-level.
+Appends and merges run under a per-stream lockfile (`streams/<id>/.lock`),
+because the torn-tail repair is a read-truncate-write sequence and hooks,
+watchers and manual passes can fire concurrently for the same owner.
+
+The lock carries an **owner token** — the holder's pid, its hostname, and a
+random nonce — written by hard-linking a fully-written temp file into place,
+so a lock that exists is always readable in full. Two guarantees rest on
+that token:
+
+- **A live holder is never stolen from, at any age.** Staleness is proven
+  from the holder's liveness, not from how long ago the lock was created.
+  A contender that finds the holder's pid alive on this host waits, and
+  fails with a timeout naming the holder — it does not reclaim. (The
+  earlier design judged a lock stale by age while the waiter's timeout
+  started when the contender arrived; a contender arriving late therefore
+  reclaimed a lock whose holder was still inside its critical section, and
+  both ran at once.)
+- **A holder releases only its own lock.** Release unlinks `.lock` only
+  while it still carries the releaser's nonce, so a holder whose lock was
+  reclaimed cannot delete the replacement lock out from under its new
+  owner.
+
+**Crash recovery is immediate, not timed**: a lock whose pid is gone from
+this host is provably stale and is reclaimed at once, without waiting out
+any threshold.
+
+A lock that cannot be read at all (anything but "not there" — a permission
+error, a directory in its place) is not retried: acquisition fails at once
+with the underlying error code and the lock's path, because retrying cannot
+clear such a condition. Everything else that fails to acquire — including a
+lock judged stale whose removal keeps failing — is bounded by
+`lockTimeoutMs` and reports why on timeout.
+
+`lockStaleMs` (30 s by default) is the fallback for the two cases where
+liveness cannot be checked here: a lock created on **another host** (file
+sync can copy one in) and a lock with **no readable token** (written by an
+older version of this package, or by hand). Those are still reclaimed by
+age. `lockTimeoutMs` (10 s) is how long a contender waits; the two
+thresholds are independent, and neither ordering of them is required or
+assumed.
+
+Honest limits:
+
+- **Pid reuse across a reboot.** A lock left by a process whose pid was
+  later reused by an unrelated live process reads as held forever, and
+  contenders on that stream time out until it is deleted by hand. The
+  timeout message names the lock path and the recorded holder. This is the
+  chosen direction of failure: refusing to write is recoverable, writing
+  concurrently with a live holder is not.
+- **Reclaim is not perfectly atomic.** A contender re-checks the lock's
+  identity (inode and mtime) immediately before removing it. That recheck
+  **detects** a replacement that landed while the old lock was being proven
+  stale — the common cascade, one contender reclaiming from whoever
+  reclaimed first — and the contender gives up and retries. It is a
+  detection, not an exclusion: a replacement landing in the window between
+  the recheck and the unlink is not seen and is removed. Nothing here
+  defends against that window.
+- **Sync tools may copy a `.lock` file.** That cannot corrupt data, but a
+  synced-in lock from another host is judged by age, so it can delay a local
+  `mergeStreamCopy` on that stream by up to 30 s — excluding `.lock` from
+  sync patterns remains the right configuration.
+
+Journal writes fsync; note that on macOS `fsync(2)` does not guarantee media
+durability (that would need `F_FULLFSYNC`, which Node's fs API does not
+expose) — the guarantee is OS-crash-level, not power-loss-level.
 
 ## Upgrading
 

package/packages/record/bin/capture.mjs

@@ -14,11 +14,12 @@
 
 import { watch } from "node:fs";
 import { homedir, hostname } from "node:os";
-import { join } from "node:path";
+import { dirname, join } from "node:path";
 import process from "node:process";
 
 import { RecordStore } from "../lib/store.mjs";
-import { captureAllSessions } from "../lib/capture-cc.mjs";
+import { captureAllSessions, captureSessions } from "../lib/capture-cc.mjs";
+import { sessionsForHome } from "../lib/sessions-for-home.mjs";
 import { SESSION_FORMATS } from "../lib/formats.mjs";
 import { captureAwLogs, defaultCommLogDir } from "../lib/capture-aw.mjs";
 import { RecordIndex } from "../lib/index-db.mjs";
@@ -41,7 +42,7 @@ function loadIgnoreOrExit(recordRoot) {
 // hostname-derived owner, which forked the whole record into a second owner
 // namespace: 571 duplicate journals from one typo. Parsing must refuse what
 // it does not understand before anything can be written.
-const VALUE_FLAGS = new Set(["root", "owner"]);
+const VALUE_FLAGS = new Set(["root", "owner", "home"]);
 const BOOL_FLAGS = new Set([
   "watch",
   "status",
@@ -61,6 +62,12 @@ const USAGE = `capture — land sessions and aw client logs in the turn record.
   capture --watch              pass now, then re-pass on filesystem change
                                (debounced) and every 15 minutes regardless
   capture --status             show store/stream summary, capture nothing
+  capture --home <dir>         capture the sessions that ran inside <dir> (an
+                               OATS instance home) and print them as JSON:
+                               thread, stream, turn count, first/last turn id.
+                               Tombstoned turns are never a boundary. Codex
+                               keeps a day's rollouts in one directory, so the
+                               pass captures that day; the list is filtered.
   capture --install-hint       print the Claude Code hook snippet
   capture --help               this text
   capture --quiet              suppress per-pass progress
@@ -224,6 +231,55 @@ if (args.status) {
   process.exit(0);
 }
 
+// One instance home: capture its own sessions and report them with exact
+// sequence boundaries (first and last captured turn id), so a consumer such
+// as the OKF harvester can name what it read without timestamps, which tie
+// and which late capture appends behind. Output is JSON, always: this mode
+// exists for programs.
+if (args.home) {
+  warnOnStrangerOwner();
+  const ignore = loadIgnoreOrExit(root);
+  const unattributed = [];
+  const found = sessionsForHome(args.home, { onUnattributed: (source, path) => unattributed.push({ source, path }) });
+  const sessions = [];
+  const dirs = new Map(); // one capture pass per (format, directory)
+  for (const s of found) dirs.set(`${s.source}\0${dirname(s.path)}`, { format: s.source, dir: dirname(s.path) });
+  let appended = 0;
+  for (const { format, dir } of dirs.values()) {
+    appended += captureSessions(store, { owner, roots: [dir], format, ignore }).appended;
+  }
+  if (appended > 0 && !args["no-index"]) {
+    const index = new RecordIndex(store);
+    try {
+      index.update();
+    } finally {
+      index.close();
+    }
+  }
+  // A tombstoned turn is hidden everywhere; a boundary naming one would be
+  // refused by recall, so boundaries come from the visible turns only.
+  const claims = store.tombstoneClaims();
+  for (const s of found) {
+    const stream = `${owner}~${s.source}.${s.sessionId}`;
+    const turns = store.readStream(stream).filter((t) => !store.claimHides(claims, t));
+    if (!turns.length) continue; // ignored by rule, nothing capturable yet, or all hidden
+    sessions.push({
+      thread: s.thread,
+      source: s.source,
+      sessionId: s.sessionId,
+      path: s.path,
+      cwd: s.cwd,
+      stream,
+      turns: turns.length,
+      firstTurnId: turns[0].id,
+      lastTurnId: turns[turns.length - 1].id,
+      lastTs: turns[turns.length - 1].ts,
+    });
+  }
+  console.log(JSON.stringify({ home: args.home, owner, appended, sessions, ...(unattributed.length ? { unattributed } : {}) }, null, 2));
+  process.exit(0);
+}
+
 warnOnStrangerOwner();
 pass();
 

package/packages/record/bin/recall.mjs

@@ -4,6 +4,13 @@
 //   recall <query...>                 FTS5 query over mail, chat, sessions
 //   recall --kind mail <query...>     filter by kind (mail|chat|session|note)
 //   recall --thread <thread> [query]  filter/list by thread
+//   recall --thread <thread> --json   the thread's turns in journal order,
+//     [--after <id>] [--until <id>]   with extracted text; exact id bounds
+//     [--limit n] [--ids-only]        (after is exclusive, until inclusive);
+//                                     --ids-only lists id, ts and the bytes
+//                                     each turn occupies in the --json output
+//                                     (text extracted but not emitted), so a
+//                                     consumer can size a window it will read
 //   recall --from <name> <query...>   filter by speaker
 //   recall --limit N                  max results (default 20)
 //   recall --show <turn-id>           print one turn as JSON
@@ -22,7 +29,7 @@ function parseArgs(argv) {
   const args = { _: [] };
   for (let i = 0; i < argv.length; i++) {
     const a = argv[i];
-    if (["--root", "--kind", "--thread", "--from", "--role", "--limit", "--show"].includes(a)) {
+    if (["--root", "--kind", "--thread", "--from", "--role", "--limit", "--show", "--after", "--until"].includes(a)) {
       args[a.slice(2)] = argv[++i];
     } else if (a.startsWith("--")) args[a.slice(2)] = true;
     else args._.push(a);
@@ -65,6 +72,57 @@ try {
     process.exit(2);
   }
 
+  // Thread turns straight from the journal, in capture SEQUENCE: the order a
+  // consumer can bound exactly with turn ids. Timestamps tie within a turn
+  // and late capture appends older stamps behind newer ones, so a window
+  // named by ids is the only one two passes agree on. after is exclusive,
+  // until inclusive; an unknown id is an error, never an empty answer.
+  if (args.json && args.thread && !query) {
+    // Session streams are not in readAll(), so hiddenIds() cannot see them:
+    // resolve tombstone claims once and test each session turn directly
+    // (store.mjs, tombstoneClaims). A redacted line must never reach the
+    // harvester, which promotes what it reads into a durable soul.
+    const claims = store.tombstoneClaims();
+    const turns = [];
+    const sessionStreams = store.sessionStreamsFor(args.thread);
+    for (const streamId of sessionStreams) {
+      for (const t of store.readStream(streamId)) if (!store.claimHides(claims, t)) turns.push(t);
+    }
+    if (!sessionStreams.length) {
+      // not a session thread (mail, chat, note): the bulk read has them
+      for (const { turn } of store.readAll().values()) if (turn.thread === args.thread && !store.claimHides(claims, turn)) turns.push(turn);
+    }
+    const ids = turns.map((t) => t.id);
+    let start = 0;
+    let end = turns.length;
+    if (args.after) {
+      const i = ids.indexOf(args.after);
+      if (i < 0) { console.error(`--after: no turn ${args.after} in thread ${args.thread}`); process.exit(1); }
+      start = i + 1;
+    }
+    if (args.until) {
+      const i = ids.indexOf(args.until);
+      if (i < 0) { console.error(`--until: no turn ${args.until} in thread ${args.thread}`); process.exit(1); }
+      end = i + 1;
+    }
+    const cap = args.limit ? Number(args.limit) : Infinity;
+    const stop = Math.min(end, start + cap);
+    const window = turns.slice(start, stop);
+    // A window is a bounded read: a consumer that plans one (the OKF
+    // harvester) sizes it with --ids-only first, then reads exactly that.
+    const out = window.map((t) => {
+      const docs = index.extractText(t);
+      const base = { id: t.id, ts: t.ts, thread: t.thread, kind: t.kind, source: t.provenance?.source ?? null };
+      const full = { ...base, text: docs.map((d) => ({ role: d.role, text: d.text })) };
+      // bytes = what this turn occupies in the pretty-printed --json answer,
+      // so a consumer's byte cap bounds what it will actually receive.
+      if (args["ids-only"]) return { ...base, bytes: Buffer.byteLength(JSON.stringify(full, null, 2), "utf8") + 8 };
+      return full;
+    });
+    console.log(JSON.stringify({ thread: args.thread, total: turns.length, from: start, to: stop, remaining: end - stop, turns: out }, null, 2));
+    process.exit(0);
+  }
+
   const limit = args.limit ? Number(args.limit) : 20;
   let rows;
   if (query) {
@@ -86,10 +144,18 @@ try {
       .all(args.thread, limit);
   }
 
+  if (rows.length === 0 && args.json) {
+    console.log("[]");
+    process.exit(0);
+  }
   if (rows.length === 0) {
     console.error("no matches");
     process.exit(1);
   }
+  if (args.json) {
+    console.log(JSON.stringify(rows, null, 2));
+    process.exit(0);
+  }
   for (const r of rows) {
     const where = r.loc ? ` @${r.loc}` : "";
     const to = r.to_name ? ` -> ${r.to_name}` : "";

package/packages/record/docs/turn-record-sot.md

@@ -8,7 +8,7 @@ aweb mail and chat messages into it. It is the contract implemented by the
 record tools (`capture`, `recall`, and the experimental tools over them); this
 package (`@awebai/turn-record`, in the oats repo) is the reference
 implementation, and the spec lives beside it on purpose: the record spans
-agent sessions from every harness, and the aweb server is one projected
+agent sessions from supported harnesses, and the aweb server is one projected
 source, not the record's home. The architecture decision it implements is
 `2026-08-18-turn-record-and-tools.md` in the strategy repo
 (github.com/awebai/strategy); the identity and messaging contracts it builds

package/packages/record/lib/sessions-for-home.mjs

@@ -0,0 +1,130 @@
+// sessionsForHome — the session transcripts one OATS instance home produced.
+//
+// Nothing in a captured session turn names the instance that ran it: turns
+// carry owner, thread, kind and text. What every harness DOES record is the
+// working directory the session started in — Claude Code on each line, pi in
+// its session header, Codex in session_meta — and an OATS instance runs with
+// its home as that directory. So "the instance's own sessions" is exactly
+// "the session files whose recorded cwd is the home or a directory below it".
+//
+// The match is on canonical paths and is exact-or-descendant, never looser:
+// a home is disposable and unique, so anything that ran inside it is the
+// instance's own, and nothing outside it — not the parent workspace, not a
+// sibling home — is ever swept in.
+
+import { closeSync, openSync, readSync, realpathSync, statSync } from "node:fs";
+import { resolve, sep } from "node:path";
+import { StringDecoder } from "node:string_decoder";
+
+import { SESSION_FORMATS } from "./formats.mjs";
+
+// A session's first lines can be large (Claude Code queue operations and
+// file-history snapshots run to 100 KB and more) and the first cwd-bearing
+// line can sit past 100 KB of bookkeeping lines, so the scan is incremental:
+// whole lines only, chunk by chunk, until the first cwd or the byte bound.
+// A file whose bound is exhausted without a cwd is reported as unattributable
+// through the optional `onUnattributed` hook, never silently dropped.
+const CHUNK_BYTES = 64 * 1024;
+export const CWD_SCAN_BOUND_BYTES = 8 * 1024 * 1024;
+
+function* wholeLines(path, bound) {
+  const fd = openSync(path, "r");
+  try {
+    const buf = Buffer.alloc(CHUNK_BYTES);
+    const decoder = new StringDecoder("utf8"); // a multi-byte character may straddle two chunks
+    let carry = "";
+    let offset = 0;
+    while (offset < bound) {
+      const n = readSync(fd, buf, 0, Math.min(CHUNK_BYTES, bound - offset), offset);
+      if (n === 0) break;
+      offset += n;
+      carry += decoder.write(buf.subarray(0, n));
+      let nl;
+      while ((nl = carry.indexOf("\n")) >= 0) {
+        yield carry.slice(0, nl);
+        carry = carry.slice(nl + 1);
+      }
+    }
+    // The remainder is a fragment unless the file ended exactly there.
+    if (carry && offset < bound) yield carry;
+  } finally {
+    closeSync(fd);
+  }
+}
+
+function cwdOfLine(source, line) {
+  if (!line.trim()) return undefined;
+  let d;
+  try {
+    d = JSON.parse(line);
+  } catch {
+    return undefined; // a non-JSON native line
+  }
+  if (source === "cc" && typeof d.cwd === "string") return d.cwd;
+  if (source === "pi" && d.type === "session" && typeof d.cwd === "string") return d.cwd;
+  if (source === "codex" && d.type === "session_meta" && typeof d.payload?.cwd === "string") return d.payload.cwd;
+  return undefined;
+}
+
+/** The working directory a session file records, scanning whole lines from
+ *  the start until the first one that carries it, or undefined when none
+ *  does within `bound` bytes (unknown format, torn file, no cwd at all). */
+export function sessionCwd(source, path, { bound = CWD_SCAN_BOUND_BYTES } = {}) {
+  try {
+    for (const line of wholeLines(path, bound)) {
+      const cwd = cwdOfLine(source, line);
+      if (cwd) return cwd;
+    }
+  } catch {
+    return undefined;
+  }
+  return undefined;
+}
+
+function canonical(p) {
+  try {
+    return realpathSync(p);
+  } catch {
+    return resolve(p); // a retired home's cwd no longer exists; compare the lexical path
+  }
+}
+
+function within(child, parent) {
+  return child === parent || child.startsWith(parent + sep);
+}
+
+/** Session files whose recorded cwd is `home` or below it, oldest first.
+ *  `roots` may override the per-format search roots ({ cc, pi, codex }),
+ *  otherwise each format's default roots under the current HOME are used.
+ *  `onUnattributed(source, path)` is called for a file that carries no cwd
+ *  within the scan bound, so a caller can report it instead of losing it.
+ *  Each entry: { source, sessionId, thread, path, cwd, bytes, mtime }. */
+export function sessionsForHome(home, { roots, onUnattributed, bound } = {}) {
+  const target = canonical(home);
+  const out = [];
+  for (const fmt of Object.values(SESSION_FORMATS)) {
+    const rs = roots?.[fmt.source] ?? fmt.defaultRoots();
+    for (const path of fmt.listFiles(rs)) {
+      const cwd = sessionCwd(fmt.source, path, bound ? { bound } : {});
+      if (!cwd) { if (onUnattributed) onUnattributed(fmt.source, path); continue; }
+      if (!within(canonical(cwd), target)) continue;
+      let stat;
+      try {
+        stat = statSync(path);
+      } catch {
+        continue; // vanished between listing and stat
+      }
+      const sessionId = fmt.sessionId(path);
+      out.push({
+        source: fmt.source,
+        sessionId,
+        thread: `${fmt.source}:session:${sessionId}`,
+        path,
+        cwd,
+        bytes: stat.size,
+        mtime: stat.mtime.toISOString(),
+      });
+    }
+  }
+  return out.sort((a, b) => a.mtime.localeCompare(b.mtime) || a.path.localeCompare(b.path));
+}