iit-mcp-autotask 0.7.2 → 0.8.0

package/README.md

@@ -9,9 +9,9 @@ packaged playbooks, strong security boundaries, and predictable per-tenant in-pr
 client + SDK-owned TTL caches).
 
 > **Package status (2026-10-19):** the runtime dependency `node-autotask` is published to public npm
-> at `0.18.0`; the committed `pnpm-lock.yaml` pins that
+> at `0.19.0`; the committed `pnpm-lock.yaml` pins that
 > release with its integrity hash, so installs resolve deterministically. The dependency is always npm
-> (node-autotask pinned exactly at `0.18.0`) — never a `file:`, `git:`, `link:` or workspace reference
+> (node-autotask pinned exactly at `0.19.0`) — never a `file:`, `git:`, `link:` or workspace reference
 > to a local checkout.
 
 > **0.6.0 is a breaking release** (unreleased): tool names, result shapes and configuration all
@@ -120,7 +120,7 @@ An operator deployment needs:
 |----------|---------|
 | `AUTOTASK_PROFILE=operator` (or an operator token) | the profile |
 | `AUTOTASK_POLICY_FILE` | the write policy: a closed JSON schema, default deny ([example](examples/policy.time-entries.json)) |
-| `AUTOTASK_STATE_DIR` | mode 0700; holds encrypted plans, the idempotency ledger, the hash-chained audit log and a lock (one process per state dir) |
+| `AUTOTASK_STATE_DIR` | mode 0700; holds encrypted plans, the idempotency ledger, the hash-chained audit log, the primary lock and the holder registry (many processes may share one dir — see below) |
 | `AUTOTASK_RUN_SCOPE` | stdio: `{"companyIds":[...]}` or `{"all":true}` (the policy's own scope); it can only narrow the policy. On HTTP each operator token carries `runScope` instead |
 | `AUTOTASK_PRINCIPAL` | stdio: the name plans and the audit log are bound to (default `stdio`) |
 | `AUTOTASK_KILL_FILE` | optional: while this file exists every preview and apply is denied (default `<state dir>/KILL`) |
@@ -159,19 +159,25 @@ the same key only replays `unknown`. Re-read the target; if the write is absent,
 a new `idempotency_key` (a new attempt) after the 10-minute cooldown; its duplicate query refuses
 it if the first write did land. Payload field names are matched case-insensitively, and keys that
 differ only by case, or are not plain ASCII field names, are refused. `STATE_UNAVAILABLE` means the
-state dir could not be checked (`retry`) or the server lost its state lock (`stop`; see below). Use a separate Autotask API user for operator deployments, and run
+state dir could not be checked (`retry`) or the server's own registration in the state dir was removed (`stop`; see below). Use a separate Autotask API user for operator deployments, and run
 the live write leg on throwaway data before relying on it (B7 in the 0.6.0 plan).
 
 #### The state lock: recovery
 
-One process owns a state dir through `<state dir>/lock` (its host, pid and a per-boot id). Run
-exactly one container per state dir: never share it between replicas or a rolling deploy's old and
-new containers.
+Many processes may share a state dir (issue #68): every process registers its boot in a holder
+registry (`<state dir>/holders/<boot>` plus a 5 s lease) before any other state work, and a
+per-operation lock (`<state dir>/op.lock`) serialises the short shared-state critical sections —
+the boot ledger rewrite, an apply's claim (the fresh ledger scan + the write-ahead `sending`
+line), the apply's result append and every audit append. The dir still names ONE primary in
+`<state dir>/lock` (host, pid and a per-boot id): a boot that finds a LIVE primary (same host,
+a fresh lease) no longer refuses — it boots as a SECONDARY and shares the dir under the op lock.
 
-The holder refreshes a lease heartbeat (`<state dir>/lock.lease`) every 5 s, and a lease older
-than 15 s (3 refresh cycles) is expired. The takeover rules:
+The primary keeps the #55 recovery machinery unchanged. The primary refreshes
+`<state dir>/lock.lease` every 5 s, and a lease older than 15 s (3 refresh cycles) is expired.
+The succession rules:
 
-- A live holder is never taken: a FRESH lease vetoes any takeover, whatever the pid probe says.
+- A live primary is never taken: a FRESH lease vetoes any takeover, whatever the pid probe says.
+  A secondary boot under a live primary simply registers and shares; it never touches the lock.
 - After a crash, a SIGKILL or an OOM kill the next boot takes the lock over by itself when the
   old holder is gone, including a container where node is PID 1 again (the boot id tells the runs
   apart).
@@ -182,16 +188,16 @@ than 15 s (3 refresh cycles) is expired. The takeover rules:
   its own. With a new hostname (the docker default: the container id) the R6 host refusal applies
   and the operator deletes `<state dir>/lock` (the case below). Every takeover is appended to the
   `audit.jsonl` hash chain as a `lock_takeover` line, and `/ready` reports the lock's age, holder
-  id and lease freshness, so an orchestrator can tell a stale lock from a live one.
+  id, lease freshness and the holder count, so an orchestrator can tell a stale lock from a live
+  one.
+
+Boots sweep the dead registrations, and a `sending` ledger entry becomes `unknown` only when its
+origin is not a live holder: a live peer's in-flight write stays `sending` (an apply of the same
+key replays it, it is never re-sent), while a dead peer's — or a pre-0.8.0, unstamped — entry
+becomes `unknown` exactly as in the single-process case.
 
 Cases that need an operator:
 
-- **`locked by a running process (pid N)`** (with or without `; its lease is X ms old`): another
-  live server holds this state dir (the lease-less form is a pre-0.6.0 server, which never wrote
-  one). Confirm with `ps -p N` that it is not a server you forgot, stop it, and start again. If it
-  is really gone and the lock's pid is dead, no unlock is needed: the next boot takes the lock
-  over immediately (a lease-less pre-0.6.0 lock has no lease to wait out). A live pid with no
-  lease is never taken over - stop that server, or delete `<state dir>/lock`.
 - **`locked by host H (pid N); one container per state dir`**: the lock belongs to another host or
   container, whose pids mean nothing here, so it is never taken over automatically. Make sure the
   server on H is stopped (or the container is gone), then delete `<state dir>/lock` and start
@@ -201,12 +207,25 @@ Cases that need an operator:
   atomic renames, so this is not a crash remnant of our own). Make sure no server is running on
   this state dir, delete `<state dir>/lock.lease`, and start again.
 - **`another process is taking over the lock (<state dir>/lock.takeover)`**: a boot crashed in the
-  middle of a takeover, a disk error hit mid-create, or two boots raced. Make sure no other server is starting on this state
-  dir, delete `<state dir>/lock.takeover`, and start again.
-
-Never delete `lock` while a server is running: writes stop (`STATE_UNAVAILABLE`) the moment the
-running server notices it no longer holds the lock, and it stops refreshing the lease. Keep the
-rest of the state dir; `audit.jsonl` is the write history.
+  middle of a takeover, a disk error hit mid-create, or two boots raced and the wait ran out.
+  Make sure no other server is starting on this state dir, delete `<state dir>/lock.takeover`,
+  and start again.
+- **`<state dir>/lock disappeared or is unreadable` / `names no pid`**: the lock file is gone or
+  corrupt (our writes are atomic, so this is not a crash remnant of our own). Make sure no server
+  is running on this state dir, delete `<state dir>/lock` (and `lock.lease`), and start again.
+- **`the op lock could not be taken at boot: …`**: the per-operation lock was already stuck when
+  this boot started (the `op.lock` case below, seen at boot). Make sure no server is running on
+  this state dir, delete `<state dir>/op.lock` (and `op.lock.lease`), and start again.
+- **`op.lock names no pid` / `a peer process holds <state dir>/op.lock; the wait ran out`**: the
+  per-operation lock is stuck — a boot crashed mid-operation and the 10 s wait ran out, or the
+  file is corrupt. Make sure no server is running on this state dir, delete `<state dir>/op.lock`
+  (and `op.lock.lease`), and start again.
+
+Never delete a running server's registration (`<state dir>/holders/<boot>`) while it is running:
+writes stop (`STATE_UNAVAILABLE`) the moment the server notices the file is gone, and it stops
+refreshing the registration's lease. Deleting the primary `lock` out from under a running primary
+is not a stop signal (the next boot simply takes the lock over) — the registration is the stop
+signal. Keep the rest of the state dir; `audit.jsonl` is the write history.
 
 ### Query results: TSV, `output_path` and the saved-file shape
 
@@ -410,7 +429,7 @@ preview/apply path; a plan previewed under 0.6.0 cannot be applied after a rollb
 |----------|---------|------|
 | `POST /mcp` | the canonical MCP endpoint (Streamable HTTP transport); Host/Origin validated, 1 MiB body cap | a bearer token in `tokens` mode; per-request `X-Autotask-*` headers for tool calls |
 | `GET /health` | process alive — pre-auth, **no vendor call** (container healthcheck) | none |
-| `GET /ready` | config valid, SDK plan-hash pin checked, server initialised, client factory ready — **no live vendor call** (credentials are request-scoped). In an operator deployment the payload also carries `lock` (the state lock's age, holder id and lease freshness — see "The state lock: recovery") | none |
+| `GET /ready` | config valid, SDK plan-hash pin checked, server initialised, client factory ready — **no live vendor call** (credentials are request-scoped). In an operator deployment the payload also carries `lock` (the state lock's age, holder id, lease freshness and the holder count — see "The state lock: recovery") | none |
 | `GET /doctor` | zero-wire credential/config preflight: per credential field `present`/`len`/`prefix2`/`shapeOk` (`AUTOTASK_SECRET`'s prefix is masked to `[REDACTED]` — no character of a secret is ever emitted; identifiers keep their format prefix), the base-url + allow-list resolution (`default_only` / `allow_list` / `default_and_list` / `any`), the SDK plan-hash gate status and the host-policy mode — **no vendor call, by construction** | Host/Origin validated; a bearer token in `tokens` mode |
 | `GET /metadata` | server identity + the MCP protocol **era pair** (baseline `2026-07-28` + legacy fallback `2025-11-25`) + capability-manifest pointer + the `deployment` block (profile, toolsets, approval mode, caller auth, rate limiting) — the served payload is the repo's `server.json` + build stamp | Host/Origin validated; a bearer token in `tokens` mode |
 
@@ -655,7 +674,7 @@ independent review with its resolutions, [`docs/reviews/0.6.0.md`](docs/reviews/
 
 ## SDK dependency
 
-`iit-mcp-autotask` depends on **`node-autotask` from npm at exactly `0.18.0`** (an exact pin —
+`iit-mcp-autotask` depends on **`node-autotask` from npm at exactly `0.19.0`** (an exact pin —
 no range operator; subpaths `.` / `/capabilities` / `/operations` / `/resources` / `/types` /
 `/pool`) — the only **vendor** dependency; the other two runtime dependencies are
 `@modelcontextprotocol/server` (the MCP framework) and `zod` (schema validation). It never
@@ -665,7 +684,7 @@ user — the published tarball ships no lockfile) must resolve to the release th
 reviewed against, or the plan-hash boot check refuses to start.
 
 **Publishing `node-autotask` is the maintainer's action, done manually.** It is published on the
-public registry at `0.18.0` (published 2026-10-19); the
+public registry at `0.19.0` (published 2026-10-19); the
 exact pin and the committed `pnpm-lock.yaml` (with `pnpm-workspace.yaml`) pin the real published
 tarball with its integrity hash, so installs resolve to exactly that release — the lockfile
 freezes the pnpm/git/Docker paths, the pin freezes the npm/npx paths. The docs are written against this

package/dist/main.js

@@ -189,6 +189,19 @@ function main() {
             log.error('uncaught exception', { err: err instanceof Error ? (err.stack ?? err.message) : String(err) });
             void handle.close().finally(() => process.exit(1));
         });
+        // SIGTERM/SIGINT: close the transport and exit through process.exit, whose 'exit' hook
+        // releases the state dir (issue #68: a terminated process must not leave its holder
+        // registration behind for the next boot to sweep).
+        let stopping = false;
+        const stop = (signal) => {
+            if (stopping)
+                return; // a second signal mid-close: the first one is finishing it
+            stopping = true;
+            log.info('shutting down', { signal });
+            void handle.close().finally(() => process.exit(0));
+        };
+        process.on('SIGTERM', () => stop('SIGTERM'));
+        process.on('SIGINT', () => stop('SIGINT'));
         return;
     }
     // tokens mode: the tokens file is read and validated once, before the listener binds.

package/dist/tools/catalog-data.d.ts

@@ -96,7 +96,7 @@ export declare function cacheOptionsForPreset(preset: 'on' | 'off', fieldCacheTt
  * fails the boot (or the CI test) instead of quietly shipping a tool surface that no longer
  * describes the SDK behind it.
  */
-export declare const BUILT_AGAINST_PLAN_HASH = "5176d37fc9bae34139c151ae0ea8336dab63a30382f04acbcaed0e24cdd71d5f";
+export declare const BUILT_AGAINST_PLAN_HASH = "078f288c280029abcbc632ac9a842ba952296ef28872c28da67a08af6ed76475";
 /**
  * Fail fast when the installed SDK's plan has moved out from under this surface.
  * Called at boot (main.ts) and safe to call repeatedly (pure comparison).

package/dist/tools/catalog-data.js

@@ -159,7 +159,7 @@ export function cacheOptionsForPreset(preset, fieldCacheTtlMs) {
  * fails the boot (or the CI test) instead of quietly shipping a tool surface that no longer
  * describes the SDK behind it.
  */
-export const BUILT_AGAINST_PLAN_HASH = '5176d37fc9bae34139c151ae0ea8336dab63a30382f04acbcaed0e24cdd71d5f';
+export const BUILT_AGAINST_PLAN_HASH = '078f288c280029abcbc632ac9a842ba952296ef28872c28da67a08af6ed76475';
 /**
  * Fail fast when the installed SDK's plan has moved out from under this surface.
  * Called at boot (main.ts) and safe to call repeatedly (pure comparison).

package/dist/tools/server.d.ts

@@ -28,7 +28,7 @@ import { type TokenStore } from '../http-guard.js';
 import type { WriteRuntime } from '../write/runtime.js';
 /** The wire identity of this server (the initialize answer + the `/metadata` payload). */
 export declare const SERVER_NAME = "iit-mcp-autotask";
-export declare const SERVER_VERSION = "0.7.2";
+export declare const SERVER_VERSION = "0.8.0";
 export declare const SERVER_TITLE = "Autotask MCP Server";
 /**
  * The MCP protocol era this server advertises (the `/metadata` stamp, both levels).
@@ -60,7 +60,7 @@ export declare function serverInstructions(config: Pick<Config, 'AUTOTASK_PROFIL
 /** The full identity stamp (the `/metadata` build block — the era pair, FR-2). */
 export declare const SERVER_IDENTITY: {
     readonly name: "iit-mcp-autotask";
-    readonly version: "0.7.2";
+    readonly version: "0.8.0";
     readonly mcpProtocolBaseline: "2026-07-28";
     readonly legacyProtocolFallback: "2025-11-25";
 };

package/dist/tools/server.js

@@ -29,7 +29,7 @@ import { registerTools } from './register.js';
 import { configForToken } from '../http-guard.js';
 /** The wire identity of this server (the initialize answer + the `/metadata` payload). */
 export const SERVER_NAME = 'iit-mcp-autotask';
-export const SERVER_VERSION = '0.7.2';
+export const SERVER_VERSION = '0.8.0';
 export const SERVER_TITLE = 'Autotask MCP Server';
 /**
  * The MCP protocol era this server advertises (the `/metadata` stamp, both levels).

package/dist/tools/write.js

@@ -604,21 +604,37 @@ export function buildApply(ctx) {
                 return refused('denied', 'stop', reason('PRINCIPAL_MISMATCH', 'this plan belongs to another principal or another Autotask tenant credential'), plan);
             }
             return store.withApplyLock(async () => {
-                const hit = store.lookup(plan.idemKey);
-                if (hit !== undefined) {
+                const base = { idemKey: plan.idemKey, planHash: plan.planHash, targetKey: plan.facts.targetKey, op: plan.op };
+                // The hit answer — shared by the FIRST lookup (its old position) and the claim's rescan:
+                // the same plan hash answers with the recorded outcome (`replayed: true`), except a
+                // retryable failure that wrote nothing (a 429) which may run again under the same key;
+                // another plan hash is `IDEMPOTENCY_CONFLICT`, `verify_then_repreview` (pre-release V4:
+                // not a dead end). `undefined` means the hit does not end the apply.
+                const hitAnswer = (hit) => {
                     if (hit.planHash !== plan.planHash) {
-                        // Pre-release V4: not a dead end. The key already ran another plan (for example a retry
-                        // re-previewed after the state moved on): re-read, and run anything still needed as a new attempt.
                         const was = hit.state === 'sending' ? 'unknown' : hit.state;
                         return refused('denied', 'verify_then_repreview', reason('IDEMPOTENCY_CONFLICT', `this idempotency key already ran a different plan (outcome ${was}). Do not re-apply under this key: re-read the target to see whether that write landed; if a write is still needed, preview again under a new idempotency_key (a new run id or attempt)`), plan);
                     }
-                    // F1: a retryable failure (a 429, or refused before it was sent) wrote nothing, so the same
-                    // plan may run again under the same key; the SDK's own cooldown still gates the send.
-                    // Everything else replays: succeeded, a deterministic failure, and above all `unknown` and a
-                    // lingering `sending`, which are never re-sent (a new attempt needs a new key and a fresh
-                    // preview, whose duplicate query finds a write that did land).
+                    // F1: a retryable failure (a 429, or refused before it was sent) wrote nothing, so the
+                    // same plan may run again under the same key; the SDK's own cooldown still gates the
+                    // send. Everything else replays: succeeded, a deterministic failure, and above all
+                    // `unknown` and a lingering `sending` (a peer's in-flight write, issue #68), which are
+                    // never re-sent (a new attempt needs a new key and a fresh preview, whose duplicate
+                    // query finds a write that did land).
                     if (!(hit.state === 'failed' && hit.outcome?.retryable === true))
                         return outcomeAnswer(plan, hit, { replayed: true });
+                    return undefined;
+                };
+                // GATES in the old relative order (issue #68): the first lookup is a FRESH from-disk
+                // scan, no op lock — a same-key replay NEVER dials the vendor, so an apply after a
+                // landed write answers from the ledger; a snapshot re-read before this gate would read
+                // the CHANGED vendor state, drift, and refuse before reaching the replay path (the
+                // ordering flaw that broke acceptance criterion 3).
+                const hit = store.lookup(plan.idemKey);
+                if (hit !== undefined) {
+                    const early = hitAnswer(hit);
+                    if (early !== undefined)
+                        return early;
                 }
                 if (now() > plan.expiresAt)
                     return refused('error', 'repreview', reason('PLAN_EXPIRED', 'the plan is older than 15 minutes; preview again'), plan);
@@ -632,6 +648,9 @@ export function buildApply(ctx) {
                 if (store.coolingDown(plan.facts.targetKey, now())) {
                     return refused('denied', 'verify_then_repreview', reason('COOLDOWN_ACTIVE', 'an earlier write to this target has an unknown outcome; re-read it, then preview again after the 10-minute cooldown'), plan);
                 }
+                // The snapshot re-read (the only vendor call before the send) stays OUTSIDE the op lock
+                // — the claim's critical section must never span a vendor round-trip (issue #68) — in
+                // its old position: after the gates above, before the claim.
                 const record = ctx.record(plan.op);
                 const shape = writeShape(record);
                 const parsed = { args: plan.args, targets: plan.targets, ...(SHAPE_ARGS[shape].includes('data') ? { data: plan.args.at(-1) } : {}) };
@@ -649,14 +668,34 @@ export function buildApply(ctx) {
                 if ('code' in snap || snap.hash !== plan.snapshotHash) {
                     return refused('error', 'repreview', reason('STATE_DRIFT', 'the target changed since the preview; preview again'), plan);
                 }
-                // Last point a cancellation or the kill switch stops the write: nothing has been sent.
+                // Last point a cancellation stops the write before anything is claimed: nothing has been sent.
                 if (signal?.aborted === true)
                     return abortedResult();
-                const late = store.blocked();
-                if (late !== undefined)
-                    return refused(late.next === 'retry' ? 'error' : 'denied', late.next, reason(late.code, late.detail), plan, { retryable: late.next === 'retry' });
-                const base = { idemKey: plan.idemKey, planHash: plan.planHash, targetKey: plan.facts.targetKey, op: plan.op };
-                store.append({ ...base, state: 'sending', at: now() });
+                // The CLAIM (issue #68): one op-lock critical section — the blocked() re-check, a FRESH
+                // ledger re-scan and the `sending` append. The re-scan closes the cross-process race: a
+                // peer's claim landing between the first lookup and here is visible inside the lock,
+                // and the loser gets the honest hit answer (replay/unknown) without sending. The
+                // expiry/policy/counters/cooldown checks stay OUTSIDE it — they already ran above, and
+                // the file is the single source of truth, so a locked re-check would add nothing.
+                const opBlocked = store.acquireOpLock();
+                if (opBlocked !== undefined) {
+                    return refused('error', 'retry', reason(opBlocked.code, opBlocked.detail), plan, { retryable: true });
+                }
+                try {
+                    const late = store.blocked();
+                    if (late !== undefined)
+                        return refused(late.next === 'retry' ? 'error' : 'denied', late.next, reason(late.code, late.detail), plan, { retryable: late.next === 'retry' });
+                    const rescan = store.lookup(plan.idemKey);
+                    if (rescan !== undefined) {
+                        const lateHit = hitAnswer(rescan);
+                        if (lateHit !== undefined)
+                            return lateHit;
+                    }
+                    store.append({ ...base, state: 'sending', at: now() });
+                }
+                finally {
+                    store.releaseOpLock();
+                }
                 store.audit({ event: 'apply_start', principal: w.principal, op: plan.op, targets: plan.targets, planHash: plan.planHash, decision: 'sending', reasons: [] }, now());
                 // The SDK write: no MCP signal (M7, never aborted once sending), capped at the apply wait.
                 // Retries are off via the PER-CALL maxRetries: 0 on the tenant client (node-autotask#105:
@@ -700,23 +739,38 @@ export function buildApply(ctx) {
                 finally {
                     clearTimeout(timer);
                 }
-                store.append(entry);
-                // The last blocked() gate was before the vendor send: the lock may have been taken over
-                // while the write was in flight. A GENUINE loss (STATE_UNAVAILABLE with next === 'stop':
-                // the lock was deleted or replaced) throws - the completed write's audit line can no
-                // longer be appended by us, and answering success for an unaudited write is worse than a
-                // loud failure (S2-M2). A kill switch that comes up during the send does NOT throw: the
-                // lock is still held, the audit line appends, and the kill switch stops the NEXT write
-                // (the main-branch behaviour, kept deliberately). A transient unreadable lock
-                // (STATE_UNAVAILABLE with next === 'retry', V7) proceeds to the audit as normal: the
-                // write completed, and the audit's skip-and-log guard keeps the chain honest if the lock
-                // truly was lost in the meantime.
+                // The result append takes the op lock (issue #68 review): one O_APPEND line is not
+                // enough — a peer that boots between our open('a') and writeSync renames the ledger out
+                // from under the write, and the `succeeded` line (with its `created` block) is orphaned
+                // on the old inode. Under the op lock the append serialises with the boot rewrite and
+                // with the peer's claim, and the apply_result audit rides the SAME acquisition (audit is
+                // re-entrant-aware: it appends directly while the lock is ours).
+                const blockedAfter = store.acquireOpLock();
+                if (blockedAfter !== undefined) {
+                    throw new ConfigError(`AUTOTASK_STATE_DIR: the write may have landed on the vendor side, but its result could not be recorded (${blockedAfter.detail}); re-read the target before any retry`);
+                }
+                try {
+                    store.append(entry);
+                    const head = store.audit({ event: 'apply_result', principal: w.principal, op: plan.op, targets: plan.targets, planHash: plan.planHash, decision: entry.state, reasons: entry.outcome?.code === undefined ? [] : [entry.outcome.code] }, now());
+                    ctx.log.info('write applied', { op: plan.op, outcome: entry.state, auditHead: head });
+                }
+                finally {
+                    store.releaseOpLock();
+                }
+                // The last blocked() gate was before the vendor send: this process's holder registration
+                // may have been removed while the write was in flight. A GENUINE stop (STATE_UNAVAILABLE
+                // with next === 'stop': the registration was deleted) throws — answering success for a
+                // write that an external actor has already stopped is worse than a loud failure (the
+                // ledger and audit lines WERE written; the failure is about this process's standing).
+                // A kill switch that comes up during the send does NOT throw: the registration is still
+                // present, the audit line appends, and the kill switch stops the NEXT write (the
+                // main-branch behaviour, kept deliberately). A transient unreadable registration
+                // (STATE_UNAVAILABLE with next === 'retry') proceeds to the audit as normal: the write
+                // completed.
                 const after = store.blocked();
                 if (after !== undefined && after.code === 'STATE_UNAVAILABLE' && after.next === 'stop') {
-                    throw new ConfigError(`AUTOTASK_STATE_DIR: the state lock was lost while the write was in flight; the ledger entry was written to this process's ledger descriptor (the successor's boot may have rewritten the ledger since, so it may not be visible in the file), and the audit line was not appended (${after.detail})`);
+                    throw new ConfigError(`AUTOTASK_STATE_DIR: this process's holder registration was removed while the write was in flight; the write completed on the vendor side and its ledger and audit lines were recorded, but this process is stopped (${after.detail})`);
                 }
-                const head = store.audit({ event: 'apply_result', principal: w.principal, op: plan.op, targets: plan.targets, planHash: plan.planHash, decision: entry.state, reasons: entry.outcome?.code === undefined ? [] : [entry.outcome.code] }, now());
-                ctx.log.info('write applied', { op: plan.op, outcome: entry.state, auditHead: head });
                 if (entry.state !== 'succeeded') {
                     const data = error === undefined ? {} : { error: honestError(typedErrorPayload(error, ctx.correlationId, requestSecrets(ctx)), error, entry) };
                     return outcomeAnswer(plan, entry, { replayed: false, data });

package/dist/write/store.d.ts

@@ -4,8 +4,9 @@ export declare const PLAN_TTL_MS: number;
 export declare const LEDGER_RETENTION_MS: number;
 export declare const COOLDOWN_MS: number;
 /**
- * The lease heartbeat: while it holds the lock, the holder rewrites `<state>/lock.lease`
- * atomically at this interval (issue #55).
+ * The lease heartbeat: while it is primary, the holder rewrites `<state>/lock.lease` atomically
+ * at this interval (issue #55); EVERY process also refreshes its own registration's lease here
+ * (issue #68) — one timer, both heartbeats.
  *
  * 5 s: one small 0600 file rewrite + fsync every 5 s is negligible against the apply path, and it
  * keeps the takeover delay after a container replacement short (the lease is dead within one
@@ -16,14 +17,16 @@ export declare const LEASE_REFRESH_MS = 5000;
  * A lease older than this is expired: the holder has been silent for three full refresh cycles,
  * so its lock may be taken over. A healthy event loop does not stall for 15 s (GC, fsync and OS
  * scheduling are far shorter), so a live holder is never taken in practice; a pathological stall
- * longer than this CAN be taken over, and the stalled process detects the lost lock and stops
- * new writes (`STATE_UNAVAILABLE`) the moment it resumes; an apply already in flight at the
- * takeover writes its ledger entry to its own descriptor (the honest outcome - the successor's
- * boot rewrites the ledger, so it may not be visible in the file) and skips its audit append
- * (a forked chain is worse than a missing line), while the explicit re-check in the tool fails
- * a genuinely lost lock loudly so the chain is never forked.
+ * longer than this CAN be taken over, and the stalled process detects the loss and stops new
+ * writes (`STATE_UNAVAILABLE`) the moment it resumes. The same expiry window is the op lock's
+ * dead-man's switch: holds are milliseconds, so a holder whose op lease went stale is a crashed
+ * mid-op process, and the next acquire takes the op lock over.
  */
 export declare const LEASE_EXPIRY_MS: number;
+/** The op lock's poll interval while a peer holds it (holds are milliseconds; 10 ms is plenty). */
+export declare const OP_LOCK_POLL_MS = 10;
+/** The op lock's wait cap: a live holder almost always releases long before this. */
+export declare const OP_LOCK_WAIT_MS = 10000;
 /** What a plan names as its target: the resource and the ids a snapshot read. */
 export interface PlanTargets {
     resource: string;
@@ -72,6 +75,10 @@ export interface LedgerEntry {
         principal: string;
         tenantId: string;
     };
+    /** The per-boot identity of the process that appended this line (issue #68); absent on legacy entries. */
+    boot?: string;
+    /** The pid of the process that appended this line (issue #68); absent on legacy entries. */
+    pid?: number;
 }
 /** The identity fields of a credential (never its secret): what the tenant id is an HMAC of. */
 export interface TenantIdentity {
@@ -96,7 +103,7 @@ export interface AuditEvent {
 }
 /** The facts /ready reports on the state lock (issue #55): age, holder id, lease freshness. */
 export interface LockInfo {
-    /** This process holds the lock. */
+    /** This process is the primary (holds the lock). */
     held: boolean;
     /** The holder the lock currently names (absent when the lock is gone or unreadable). */
     holder?: {
@@ -110,7 +117,15 @@ export interface LockInfo {
     leaseAgeMs?: number;
     /** True when a readable lease is older than {@link LEASE_EXPIRY_MS}. */
     leaseExpired?: boolean;
+    /** How many processes have registered in `<state>/holders/` (including this one, issue #68). No liveness filter: dead registrations are swept at the next boot. */
+    holders: number;
 }
+/** Why an op-lock acquisition gave up: the bounded wait ran out, or the op lock needs an operator. */
+export type OpLockBlocked = {
+    code: 'STATE_UNAVAILABLE';
+    next: 'retry';
+    detail: string;
+};
 export interface StateStore {
     readonly dir: string;
     tenantId(identity: TenantIdentity): string;
@@ -118,9 +133,9 @@ export interface StateStore {
     savePlan(plan: PlanRecord): void;
     /** The decrypted plan, or undefined when absent or not authentic (forged, edited, swapped). */
     loadPlan(handle: string): PlanRecord | undefined;
-    /** The latest ledger entry for an idempotency key. */
+    /** The latest ledger entry for an idempotency key (a fresh scan of the file). */
     lookup(idemKey: string): LedgerEntry | undefined;
-    /** Append + fsync. */
+    /** Append + fsync (one O_APPEND line; the line is stamped with this boot's identity). */
     append(entry: LedgerEntry): void;
     counters(op: string, now: number): Counters;
     /** True while an unknown outcome on this target is younger than the cooldown. */
@@ -132,31 +147,45 @@ export interface StateStore {
     createdHere(resource: string, id: number, now: number, days: number, principal: string, tenantId: string): {
         companyIds: number[];
     } | undefined;
-    /** Append one chained audit line; returns the new head hash. When the lock is no longer held, the line is skipped (logged) and the current head is returned (a forked chain is worse than a missing line). */
+    /**
+     * Append one chained audit line; returns the new head hash. The append runs under the op lock,
+     * which adopts the file's tail when the chain advanced under a peer; the line is skipped (logged)
+     * and the current head returned only when the store is closed or the op lock cannot be taken.
+     */
     audit(event: AuditEvent, now: number): string;
     /** The facts /ready reports on the state lock: who holds it, how long, and how fresh its lease is. */
     lockInfo(): LockInfo;
     killed(): boolean;
     /**
-     * Why writes may not run right now: the kill switch (stop), the state lock lost (stop), or the
-     * lock unreadable for the moment (retry); undefined when writes may run.
+     * Why writes may not run right now: the kill switch (stop), this process's holder registration
+     * gone (stop) or unreadable for the moment (retry); undefined when writes may run.
      */
     blocked(): {
         code: 'KILL_SWITCH' | 'STATE_UNAVAILABLE';
         next: 'stop' | 'retry';
         detail: string;
     } | undefined;
-    /** One apply at a time per process (and, through the lock file, per state dir). */
+    /**
+     * Take the per-operation lock (issue #68). Returns undefined when acquired — the caller must
+     * then call {@link releaseOpLock} — or the reason the bounded wait gave up.
+     */
+    acquireOpLock(): OpLockBlocked | undefined;
+    /** Release the per-operation lock if it is still this process's. */
+    releaseOpLock(): void;
+    /** One apply at a time per process. */
     withApplyLock<T>(fn: () => Promise<T>): Promise<T>;
     close(): void;
 }
 /**
- * Take the state lock: O_EXCL, or a takeover of a holder that is no longer active.
+ * Take the state lock: O_EXCL, or a takeover of a holder that is no longer active, or — when the
+ * current holder is a LIVE primary — a plain secondary boot (issue #68): the lock stays with the
+ * primary and this process shares the dir under the op lock.
  *
  * The takeover is serialised by a second O_EXCL file, `lock.takeover` (review F7): only its holder
  * may replace the stale lock, and it re-reads the lock first, so two processes booting after a crash
- * cannot both take over (the loser sees either the takeover file or the winner's live pid + fresh
- * lease). A takeover file left by a crash mid-takeover refuses boot until an operator removes it.
+ * cannot both take over (the loser sees either the takeover file — and now WAITS for the taker
+ * instead of refusing — or the winner's live pid + fresh lease). A takeover file that survives the
+ * wait cap (a crash mid-takeover left it behind) refuses boot until an operator removes it.
  *
  * The verdict has three parts (issue #55): another host's lock is never taken (R6); a FRESH lease
  * (the holder rewrote `lock.lease` within {@link LEASE_EXPIRY_MS}) vetoes the takeover outright —
@@ -182,6 +211,8 @@ export interface OpenStoreOptions {
     /** `AUTOTASK_KILL_FILE`; default `<state>/KILL`. */
     killFile?: string;
     now?: () => number;
+    /** The op lock's wait cap (and the lock-takeover contention wait). Default {@link OP_LOCK_WAIT_MS}. */
+    opLockWaitMs?: number;
 }
 /** Open (or create) the state directory; throws ConfigError on every boot refusal. */
 export declare function openStateStore(dir: string, opts: OpenStoreOptions): StateStore;