@stravica-ai/digital-operator-pa 0.1.21 → 0.1.23

package/dist/engineering-pa-cli/docs/estate-sync.md

@@ -128,11 +128,74 @@ id/tag is estate-side work, not part of this package.
    proceeds to push. The audit record adds
    `autoResolved: <N>` + `autoResolvedPaths: [...]` to the `ok` line.
 
-   If **any** conflicted path is outside the allowlist,
-   `git rebase --abort` runs and sync exits 2 with status `fail-rebase`
-   as before. Real content conflicts still surface for human resolution;
-   the audit record includes `conflictedPaths: [...]` for the disallowed
-   paths so the on-call has a starting point.
+   If **any** conflicted path is outside the allowlist, the tool
+   attempts **self-heal on rebase-abort** (R2, w-2026-08-12-004, replaces
+   the pre-R2 "abort and wedge" behaviour): estate-sync's own version of
+   each conflicting path is queued to
+   `<repoRoot>/.cache/estate-sync/pending/<pathSlug>__<firstSeenTs>` with
+   a JSON sidecar carrying `originalPath`, `conflictingUpstreamCommitSha`,
+   `upstreamBlobShaAtQueue`, `attemptCount`, `firstSeen`, and
+   `alertLevel`. The pending content is fsynced BEFORE any git mutation
+   runs, so a queue failure never triggers a reset. Then four cascading
+   guards fire (HEAD sha equals the sha recorded immediately after
+   `git commit`; HEAD^ exists; exactly one commit in `HEAD^..HEAD`;
+   exactly one commit ahead of `origin/<branch>`); on any guard failure
+   the tool falls through to the classic `fail-rebase` behaviour and
+   stays wedged loudly. When all guards pass, `git reset --soft HEAD^`
+   drops estate-sync's own commit while leaving BOTH the index and the
+   working tree untouched — every non-conflicting path in the dropped
+   commit stays staged and its bytes stay on disk for the next tick to
+   recommit. Then surgically per conflicting path: newly-added paths get
+   `git rm --cached -- <path>` + `fs.unlink` (unstaged and removed from
+   disk; their bytes live in pending); modified paths get
+   `git checkout HEAD -- <path>` (restored to the parent's version in
+   both index and working tree). The audit record uses one of
+   `fail-rebase-queued` (alertLevel `warn`, attempts < 5),
+   `fail-rebase-stuck` (alertLevel `error`, attempts >= 5), or
+   `fail-rebase-needs-human` (divergence detected during drain) with
+   exit code **0** — the process exits healthy because the workspace
+   itself is fast-forwardable again, while the `fail-*` status routes
+   through librarian-alerts (when wired to estate-sync's runs.jsonl).
+
+   > **Why `--soft` and not `--keep`** (correction landed 2026-08-12 via
+   > the PR-120 fresh-context review): `--keep <ref>` updates the
+   > working tree to match `<ref>` on every file that differs between
+   > HEAD and `<ref>`. In a multi-file estate-sync commit (which is the
+   > design — step 6 scoops every in-scope dirty path), that means
+   > every non-conflicting path in the dropped commit would be silently
+   > destroyed (newly-added files deleted, modifications reverted).
+   > `--soft` leaves the index and working tree entirely alone; the
+   > surgical scope above touches only the conflicting paths, so
+   > non-conflicting content survives across the drop.
+
+   On every subsequent tick, **DRAIN** runs at the top (inside the PID
+   lock, after fetch so `origin/<branch>` refs are fresh), oldest-first.
+   For each pending entry the tool compares the sidecar's
+   `upstreamBlobShaAtQueue` against origin's blob NOW and picks one of
+   three actions:
+   - **`already-delivered`** — upstream's current blob bytes exactly
+     match the pending content (another writer pushed our version for
+     us). Pending removed cleanly, no workspace write.
+   - **`staged-for-retry`** — upstream blob is unchanged since queue
+     time. Copy the pending bytes back into the workspace at
+     `originalPath`; step 6 will stage them and step 7 will rebase
+     normally. Pending removed ONLY after commit + push are confirmed
+     on HEAD (`finaliseDrainOnPushSuccess`); a crash between drain
+     and commit leaves the pending file intact so the next tick
+     re-evaluates.
+   - **`divergence-held`** — upstream now has a DIFFERENT resolution
+     than the sidecar recorded at queue time (a third writer intervened,
+     a human reconciled with different bytes, etc.). The sidecar is
+     stamped `alertLevel: 'needs-human'` (sticky — never de-escalates
+     back to warn or error), and the workspace is NOT touched. This
+     blocks the silent-clobber hazard the earlier ratified drain
+     sketch had, where re-applying stale bytes would overwrite the
+     winner's resolution.
+
+   Pending queue lives at `.cache/estate-sync/pending/`, gitignored per
+   the tooling standard §16. Content-safety invariant: no reset ever
+   runs when the queue write fails, and no workspace write ever happens
+   when upstream has diverged.
 
 ### Registry-scoped commit set
 
@@ -229,7 +292,13 @@ One JSON object per line, camelCase keys (§11 of the tooling standard):
 | `branch` | string | The current branch name. Present on `ok` and `noop-nothing-to-do`. |
 | `autoResolved` | number | Number of allowlisted regenerated-artifact paths auto-resolved during a rebase conflict. `ok` only, present iff > 0. |
 | `autoResolvedPaths` | string[] | The specific paths that got the `--ours` treatment. `ok` only, paired with `autoResolved`. |
-| `conflictedPaths` | string[] | The disallowed conflicted paths that forced the abort. `fail-rebase` only, when at least one conflict was outside the allowlist. |
+| `conflictedPaths` | string[] | The disallowed conflicted paths. Present on `fail-rebase` (self-heal declined, wedged loud), and on any `fail-rebase-queued` / `fail-rebase-stuck` / `fail-rebase-needs-human` (self-heal ran, workspace healthy again). |
+| `selfHealed` | bool | Set to `true` on any self-heal audit record (`fail-rebase-queued`, `-stuck`, or `-needs-human`). |
+| `droppedCommit` | string | 40-hex sha of estate-sync's own commit that self-heal dropped. Present on all self-heal records. |
+| `queued` | object[] | Summary of each queued pending entry (id, originalPath, attemptCount, alertLevel, upstreamBlobShaAtQueue, conflictingUpstreamCommitSha, pendingPath). Present on all self-heal records. |
+| `drained` | object[] | Summary of drain results for this tick (id, originalPath, action, attemptCount, optional alertLevel + upstream blob shas). Present on any tick that saw pending entries at start-of-tick. |
+| `drainFinalised` | string[] | Ids of pending entries removed after commit + push were confirmed on HEAD. Present on `ok` records that carried `drained` entries with action `staged-for-retry`. |
+| `selfHealDeclined` | object | `{ stage, reason }` when self-heal was invoked but refused (queue-write failure, guard failure, reset failure). Paired with `status: fail-rebase` and `exitCode: 2`. |
 
 Status vocabulary (closed set):
 
@@ -238,13 +307,25 @@ Status vocabulary (closed set):
 | `ok` | 0 | Commit + push (or pull-only) succeeded. |
 | `noop-nothing-to-do` | 0 | Clean tree, no local commits ahead, up-to-date with origin. |
 | `noop-locked` | 0 | Another sync run holds the lock. |
-| `fail-rebase` | 2 | `git pull --rebase` conflicted; rebase aborted. Human reconciliation required. |
+| `fail-rebase` | 2 | `git pull --rebase` conflicted; self-heal either not attempted (regenerated-artifacts branch already resolved) or declined (guard-failure / queue-write-failure / reset-failure). Workspace is wedged, human reconciliation required. |
+| `fail-rebase-queued` | 0 | Self-heal ran on a non-allowlist conflict, dropped estate-sync's own commit under guard, queued the conflicting content for retry. Workspace is fast-forwardable again; librarian-alerts should surface the queued path (`alertLevel: warn`, `attemptCount < 5`). |
+| `fail-rebase-stuck` | 0 | Same as `fail-rebase-queued` but the pending entry's `attemptCount >= 5` — escalated alert (`alertLevel: error`). |
+| `fail-rebase-needs-human` | 0 | Self-heal ran, drain-time upstream divergence detected on a pending entry — someone else pushed a different resolution while ours was queued. Workspace NOT touched; pending held indefinitely (`alertLevel: needs-human`, sticky). |
 | `fail-push` | 2 | Push rejected even after one rebase-and-retry. Commit is local-only. |
 | `fail-git` | 3 | A `git status`/`add`/`commit` command itself errored (transport or IO). |
 | `fail-cli-bootstrap` | varies | parseArgs / import / startup failure before runSync was reached. |
 
-Consumer contract: monitors should treat `status == 'ok'` and any
-`noop-*` as healthy; any `fail-*` as failing.
+Consumer contract:
+- **Process health** — check `exitCode`. `0` = the sync process itself is
+  healthy (nothing to fix at cron / systemd level). This is the correct
+  signal for "is the cron process wedged?"
+- **Alert routing** — check `status.startsWith('fail-')`. Every `fail-*`
+  status is a human-visible alert regardless of exitCode, because
+  self-heal statuses (`fail-rebase-queued`, `fail-rebase-stuck`,
+  `fail-rebase-needs-human`) intentionally exit 0 (workspace is healthy,
+  process is healthy) while surfacing that a pending entry needs
+  attention. Escalation ladder inside self-heal: `queued` → `stuck` at
+  attempt >= 5 → `needs-human` on drain-time divergence (sticky).
 
 ## Exit codes
 

package/dist/engineering-pa-cli/src/pa/io.mjs

@@ -24,6 +24,24 @@ export async function writeJson(path, value) {
   await writeFile(path, body, 'utf8');
 }
 
+/**
+ * Write a JSON file with O_CREAT|O_EXCL semantics. Refuses to overwrite
+ * an existing file (throws a Node error with `code === 'EEXIST'`).
+ *
+ * Same shape rules as `writeJson`: 2-space indent, trailing newline,
+ * parent directories created on demand. Callers that follow an
+ * allocate-then-write pattern use this to turn a same-author write race
+ * into an EEXIST they can retry at NNN+1, rather than the plain flag
+ * `'w'` behaviour where the second write silently clobbered the first.
+ * Same primitive `escalate.mjs` in `librarian-pr-intake` uses for the
+ * librarian's own allocator — the two writers now share one race model.
+ */
+export async function writeJsonExclusive(path, value) {
+  await mkdir(dirname(path), { recursive: true });
+  const body = JSON.stringify(value, null, 2) + '\n';
+  await writeFile(path, body, { encoding: 'utf8', flag: 'wx' });
+}
+
 /**
  * Ensure a directory exists. Returns the absolute path.
  */

package/dist/engineering-pa-cli/src/pa/work.mjs

@@ -26,7 +26,22 @@ import {
 } from '../../../engineering-pa-core/src/workItemId/index.mjs';
 
 import { getRepoRoot } from './repo-root.mjs';
-import { readJson, writeJson, ensureDir, listJsonFiles, today } from './io.mjs';
+import {
+  readJson,
+  writeJson,
+  writeJsonExclusive,
+  ensureDir,
+  listJsonFiles,
+  today,
+} from './io.mjs';
+
+// Upper bound on the O_CREAT|O_EXCL retry loop in `createWork`. Same
+// value the librarian-pr-intake allocator uses (`escalate.mjs`); both
+// close the same class of same-author race between allocator scan and
+// exclusive write. 99 is comfortably above any realistic burst — 999 is
+// the per-author-per-day NNN ceiling so a runaway loop still terminates
+// well below it.
+const MAX_ALLOCATE_RETRIES = 99;
 
 const STATUSES = new Set(['draft', 'planned', 'in-progress', 'blocked', 'done', 'archived']);
 
@@ -63,29 +78,22 @@ function validatePriority(priority) {
 }
 
 /**
- * Highest NNN already allocated for `dateStr` in THIS author's per-day
- * sequence. Legacy items (`w-YYYY-MM-DD-NNN` with no author segment)
- * do not count toward any author's sequence — each writer starts from
- * 001 on the day writer-scoping takes effect and existing legacy items
- * simply occupy a separate, older space.
+ * Highest NNN currently visible on disk for `dateStr` in THIS author's
+ * per-day sequence. Legacy items (`w-YYYY-MM-DD-NNN` with no author
+ * segment) do not count toward any author's sequence — each writer
+ * starts from 001 on the day writer-scoping takes effect and existing
+ * legacy items simply occupy a separate, older space.
  *
- * If 999 is already taken, throws — callers should treat as an
- * operational failure and report. In practice this is unreachable
- * because NNN is per-author-per-day, not shared.
+ * The value is a starting point for `createWork`'s allocate-write-retry
+ * loop, not a final answer: between the scan and the exclusive write,
+ * another same-author `pa work create` may have landed a same-NNN file
+ * (same-author write race, structurally possible when two operator
+ * sessions run concurrently on shared work/). The O_CREAT|O_EXCL retry
+ * in `createWork` is what closes that residual race.
  */
-async function nextWorkNumber(repoRoot, dateStr, author) {
-  const dir = join(repoRoot, 'work');
-  const files = await listJsonFiles(dir);
-  const max = maxNForAuthorOnDate(files, dateStr, author);
-  const next = max + 1;
-  if (next > 999) {
-    const err = new Error(
-      `no free work-item slot for ${dateStr} in author=${author} sequence (NNN exhausted)`,
-    );
-    err.exitCode = 2;
-    throw err;
-  }
-  return next;
+async function scanMaxNInWorkDir(workDir, dateStr, author) {
+  const files = await listJsonFiles(workDir);
+  return maxNForAuthorOnDate(files, dateStr, author);
 }
 
 /**
@@ -101,6 +109,15 @@ async function nextWorkNumber(repoRoot, dateStr, author) {
  * with an actionable message naming all three sources — a mint with no
  * author identity is the exact defect R1 exists to close, so we refuse
  * to fall back to any hard-coded default.
+ *
+ * Write is O_CREAT|O_EXCL via `writeJsonExclusive`. If two same-author
+ * `createWork` invocations race (same estate, same day, two operator
+ * sessions), the allocator scan may hand both callers the same base
+ * NNN; the first exclusive write lands and the second gets EEXIST,
+ * which the retry loop below resolves by trying NNN+1, NNN+2, ... up
+ * to `MAX_ALLOCATE_RETRIES`. Cross-writer collisions (different author
+ * tokens) are already structurally impossible after R1
+ * (w-2026-08-12-001) — this closes the residual same-author window.
  */
 export async function createWork(opts) {
   if (!opts.title) {
@@ -127,28 +144,48 @@ export async function createWork(opts) {
   await ensureDir(workDir);
 
   const dateStr = today();
-  const nnn = await nextWorkNumber(repoRoot, dateStr, author);
-  const id = buildWorkItemId({ date: dateStr, author, nnn });
-  const path = join(workDir, `${id}.json`);
-
-  const item = {
-    id,
-    projectId: opts.projectId || null,
-    title: opts.title,
-    category: opts.category || 'chore',
-    status: 'draft',
-    priority: opts.priority == null ? 2 : opts.priority,
-    owner: opts.owner || 'system-architect',
-    assignee: opts.assignee || opts.owner || 'system-architect',
-    created: dateStr,
-    updated: dateStr,
-    completedAt: null,
-    summary: opts.summary || '',
-    blocked: null,
-    links: [],
-  };
-  await writeJson(path, item);
-  return { id, path };
+  const baseN = await scanMaxNInWorkDir(workDir, dateStr, author);
+
+  for (let offset = 1; offset <= MAX_ALLOCATE_RETRIES; offset += 1) {
+    const nnn = baseN + offset;
+    if (nnn > 999) {
+      const err = new Error(
+        `no free work-item slot for ${dateStr} in author=${author} sequence (NNN exhausted)`,
+      );
+      err.exitCode = 2;
+      throw err;
+    }
+    const id = buildWorkItemId({ date: dateStr, author, nnn });
+    const path = join(workDir, `${id}.json`);
+    const item = {
+      id,
+      projectId: opts.projectId || null,
+      title: opts.title,
+      category: opts.category || 'chore',
+      status: 'draft',
+      priority: opts.priority == null ? 2 : opts.priority,
+      owner: opts.owner || 'system-architect',
+      assignee: opts.assignee || opts.owner || 'system-architect',
+      created: dateStr,
+      updated: dateStr,
+      completedAt: null,
+      summary: opts.summary || '',
+      blocked: null,
+      links: [],
+    };
+    try {
+      await writeJsonExclusive(path, item);
+      return { id, path };
+    } catch (e) {
+      if (e && e.code === 'EEXIST') continue;
+      throw e;
+    }
+  }
+  const err = new Error(
+    `pa work create: gave up after ${MAX_ALLOCATE_RETRIES} EEXIST retries from base NNN ${baseN + 1} in ${dateStr} author=${author}`,
+  );
+  err.exitCode = 2;
+  throw err;
 }
 
 /**

package/dist/engineering-pa-cli/src/sync/git.mjs

@@ -198,3 +198,91 @@ export async function rebaseContinue(runner, { cwd }) {
     { cwd, env: { ...process.env, GIT_EDITOR: 'true' } },
   );
 }
+
+// Resolve a ref (default HEAD) to its full sha. Returns { code, sha, stderr }.
+// The self-heal path uses this both to record the sha of estate-sync's own
+// commit at commit time AND to verify HEAD is still that sha immediately
+// before dropping it — the guard that stops the reset ever touching a
+// commit we did not just make.
+export async function revParseSha(runner, { cwd, ref = 'HEAD' } = {}) {
+  const r = await runner.spawn('git', ['rev-parse', '--verify', `${ref}^{commit}`], { cwd });
+  return { code: r.code, sha: r.stdout.trim(), stderr: r.stderr };
+}
+
+// Count commits in a range (e.g. HEAD^..HEAD). Returns { code, count, stderr }.
+// count === 1 is the drop-guard invariant: exactly one commit between HEAD's
+// parent and HEAD (never a merge-commit stack, never zero, never more).
+export async function revListCount(runner, { cwd, range }) {
+  const r = await runner.spawn('git', ['rev-list', '--count', range], { cwd });
+  const n = Number.parseInt(r.stdout.trim(), 10);
+  return { code: r.code, count: Number.isFinite(n) ? n : null, stderr: r.stderr };
+}
+
+// git reset --soft <ref>. Moves HEAD to <ref>, leaves BOTH the index and
+// the working tree untouched. This is the load-bearing primitive for the
+// R2 self-heal path: after dropping our commit, the non-conflicting
+// portion of that commit's diff MUST remain staged in the index so the
+// next tick recommits it. --mixed would unstage (still fine for our
+// purposes but noisier), --hard would obliterate everything, and --keep
+// (which we originally used, corrected 2026-08-12 per the reviewer's
+// blocker on PR #120) actively UPDATES the working tree to match <ref>
+// on every file that differs between HEAD and <ref> — meaning every
+// non-conflicting path in the dropped commit gets silently destroyed.
+// --soft leaves the surgical work to the caller: keep what we want,
+// remove or restore only the conflicting paths we've already queued.
+export async function resetSoft(runner, { cwd, ref }) {
+  return runner.spawn('git', ['reset', '--soft', ref], { cwd });
+}
+
+// git rm --cached -- <path>. Unstages a single path — removes it from the
+// index without touching the working tree. Paired with `fs.unlink` in
+// self-heal to surgically un-stage a newly-added conflicting path after
+// the --soft reset (the addition is queued to pending, so we want the
+// path GONE from the index AND from disk without touching any other
+// staged content).
+export async function rmCached(runner, { cwd, paths }) {
+  if (paths.length === 0) return { code: 0, stdout: '', stderr: '' };
+  return runner.spawn('git', ['rm', '--cached', '--', ...paths], { cwd });
+}
+
+// git hash-object <path>. Computes git's own blob sha for the byte
+// contents of a file, WITHOUT staging or writing anything to the object
+// store. Used by the drain to compare a pending file's bytes against
+// `origin/<branch>:<path>` via matching sha values — binary-safe, no
+// utf8 round-trip, and apples-to-apples (git blob sha both sides).
+export async function hashObject(runner, { cwd, path }) {
+  const r = await runner.spawn('git', ['hash-object', '--', path], { cwd });
+  return { code: r.code, sha: r.stdout.trim(), stderr: r.stderr };
+}
+
+// Does <ref>:<path> exist? `git cat-file -e` exits 0 if yes, non-zero if no.
+// Used by self-heal to decide the working-tree cleanup shape:
+//   - path existed at HEAD^   → checkout HEAD^ to restore parent's version
+//   - path did NOT exist there → the file was newly added by our commit;
+//                                remove it from the working tree entirely
+// Getting this branch right is what stops the wedge-loop where our own
+// content, still in the tree after reset, re-conflicts on every tick.
+export async function refPathExists(runner, { cwd, ref, path }) {
+  const r = await runner.spawn('git', ['cat-file', '-e', `${ref}:${path}`], { cwd });
+  return r.code === 0;
+}
+
+// git checkout <ref> -- <paths>. Copies the ref's version of each path into
+// BOTH the index and the working tree. Used by self-heal after the
+// reset --keep HEAD^ to overwrite the (now unstaged) conflicting paths with
+// their pre-commit versions, so the next tick's step 6 doesn't re-stage the
+// content we've deliberately queued out of the way.
+export async function checkoutFromRef(runner, { cwd, ref, paths }) {
+  if (paths.length === 0) return { code: 0, stdout: '', stderr: '' };
+  return runner.spawn('git', ['checkout', ref, '--', ...paths], { cwd });
+}
+
+// Does the ref exist locally? Used to check origin/<branch> before we try
+// to inspect it (a fresh clone with no fetches wouldn't have it). We use
+// `git show-ref --verify --quiet` because it's the ref-existence idiom that
+// doesn't fall back to name-rev heuristics and produces no output on either
+// branch.
+export async function refExists(runner, { cwd, ref }) {
+  const r = await runner.spawn('git', ['show-ref', '--verify', '--quiet', ref], { cwd });
+  return r.code === 0;
+}