c8ctl-plugin-nano 1.61.0 → 1.61.1

package/README.md

@@ -728,7 +728,7 @@ only `latest`, so the key is the reproducibility handle).
 
 On completion the plugin writes an **output envelope** back under
 `io.nanobpm.agentResult` (`{schemaVersion, status, sandbox, image, output, truncated, stderrTruncated, exitCode, signal, error, promptResourceKey?}`). When a repository was
-provisioned (below) it also carries `{repository, branch, baseSha, headSha, commits[], pushed, pushError?, gitError?, pr?}`.
+provisioned (below) it also carries `{repository, branch, baseSha, headSha, commits[], pushed, pushError?, pushFailed?, strandedCommits?, branchMismatch?, scanError?, gitError?, pr?}`. `pushFailed` is the explicit "push failed" flag — set for a non-zero `git push` **whose result could not be confirmed as landed at the remote** (a non-fast-forward rejection, or an auth, hook, or network error), not only a server rejection. **`pushFailed` does not by itself prove a remote push was attempted or rejected:** `finalizeGit` also sets it (with `strandedCommits` and a `branchMismatch` `{expected, actual}`) when it *refuses to push at all* because the harness moved HEAD off the provisioned work branch — or left commits abandoned on another local branch — so pushing the work branch would publish stale work and strand those commits (no `git push` runs, so `pushError` is absent in that case). As a guard against false strands, a non-zero push is re-checked with `ls-remote`: if `origin/<branch>` already points at `headSha` — or the remote tip is a **descendant** of `headSha` (another actor pushed a further commit after ours landed) — the push is treated as a transport hiccup that landed after the ref was accepted (`pushed: true`, no `pushFailed`/`pushError`); otherwise `pushFailed` is set and `strandedCommits` lists the SHAs of the new commits left UNPUSHED in the throwaway workspace — together they are the recovery handle for a failed push, so consumers must not treat `pushed: false` alone as the only signal. On such a failure (a rejected push **or** a branch-mismatch refusal) the throwaway workspace is preserved **best-effort** (even under the default `--keep-runs=false`) so those SHAs stay recoverable, but this is a *recovery window, not a durable archive*: the run-directory reaper still ages it out by mtime and worker shutdown removes the namespace — copy the stranded commits out promptly (or run with `--keep-runs`). **`scanError`** flags a **best-effort/PARTIAL** `strandedCommits` list: a commit or remote-reachability scan did **not** complete, so a consumer should recover the whole preserved workspace rather than trust the list exactly. It arises two ways: (a) a **pre-push** commit-enumeration scan failed, so `finalizeGit` refused to push on an incomplete graph — **no push was attempted** (`pushError` is absent) and `strandedCommits` is a partial surfacing; or (b) a push was **ATTEMPTED and rejected** but the remote-reachability FILTER that trims already-published commits from the strand set itself failed — so `scanError` is present **alongside** `pushError`, marking the rejected-push strand list as best-effort (it may falsely include an already-landed commit). Either way `pushFailed` stays authoritative; `scanError` is how a consumer knows the strand list is inexact (recover the whole workspace) versus an exact list (a rejected push with no `scanError`).
 
 **Git provisioning (host).** When `--sandbox none` (the default) and the envelope
 carries a `repository.url`, the plugin provisions a workspace on the host around
@@ -764,7 +764,17 @@ the harness:
    per envelope (default 120s, or the `--clone-timeout` worker flag) as a backstop
    for repos big enough to approach the cap even when shallow; a timeout is now
    reported *as a timeout* rather than an opaque `exit 128`;
-3. create `branch.create` (if set) off that target;
+3. create `branch.create` (if set) off that target — or, when `branch.create` is
+   **absent** (or names the effective base itself) **and push is enabled**, cut a
+   generated fallback work branch `nano/agent-work/<base>-<runId>` so commits are
+   never made directly on the base branch; the branch actually used (configured or
+   generated) rides back in the result envelope (`branch`) and is exported to the
+   harness as `AGENT_REPO_BRANCH`. **Exception — a detached tag/SHA checkout with
+   no `branch.create`:** with no symbolic branch to name a fallback from, it stays
+   `branch: null` and its commits are **not** pushed even when push is enabled —
+   this holds for *any* detached no-`branch.create` checkout, **including** when a
+   `branch.base` is configured (the base name is not used to synthesize a fallback
+   here). Supply `branch.create` to publish work committed off a tag/SHA base;
 4. set a **committer identity** on the workspace, preferring the operator's own
    (`GIT_AUTHOR_*` env → global `git config user.name/email` → the
    `gh`-authenticated GitHub user), and only falling back to `nano-agent` when
@@ -793,10 +803,36 @@ config is neutralized (`GIT_CONFIG_GLOBAL` → the platform null device,
 or `url.*.insteadOf` can't silently inject operator credentials. (An **SSH**
 remote — `git@…`/`ssh://…` — can still authenticate via the host's SSH
 agent/config; use HTTPS URLs if you need a guaranteed-anonymous clone.)
-Token-backed jobs keep global config (e.g. `http.proxy`). A push failure is
-reported as `pushError` (the job still completes) so a later BPMN step can drive
-the merge; a clone/checkout failure sheds the job (retryable). Workspaces are
-deleted after each job (keep them with `--keep-runs`).
+Token-backed jobs keep global config (e.g. `http.proxy`). **`pushFailed` is the
+authoritative "work is stranded, workspace preserved" flag** (the job still
+completes, so a later BPMN step can drive the merge); a clone/checkout failure
+sheds the job (retryable). `pushError` is present **only** for a push that was
+ATTEMPTED and rejected/unconfirmed — when `finalizeGit` refuses to push at all (a
+branch mismatch, or a partial/incomplete commit scan), it sets `pushFailed`
+(with `branchMismatch` or `scanError`) but **no** `pushError`, so consumers must
+key recovery off `pushFailed`, not `pushError`. **`scanError` flags a
+best-effort/PARTIAL `strandedCommits` list** — a commit/reachability scan did not
+complete, so the list may omit or (after a rejected push) falsely include commits
+and a consumer should recover the whole preserved workspace rather than trust the
+list exactly. It arises two ways: (a) a pre-push commit-enumeration scan failed, so
+`finalizeGit` refused to push on an incomplete graph (**no** `git push` ran, so
+`pushError` is absent); or (b) a push was ATTEMPTED and rejected but the
+remote-reachability FILTER that trims already-published commits from the strand set
+itself failed — so `scanError` is present **alongside** `pushError`, marking the
+otherwise-complete rejected-push strand list as best-effort. Either way `pushFailed`
+stays authoritative. Workspaces are
+deleted after each job (keep them with `--keep-runs`) — **except** a job whose
+`git push` **could not be confirmed as landed** (a non-fast-forward rejection, or
+an auth/hook/network error whose `ls-remote` re-check did not find `origin/<branch>`
+at or ahead of `headSha` — `pushFailed` + `pushError`), **or** one where
+`finalizeGit` refused to push (no `git push` ran, so `pushError` is absent)
+because HEAD moved off the provisioned work branch — or left commits on another
+local branch or an abandoned detached HEAD reachable only via the reflog (a
+`branchMismatch` strand) — **or** because a critical local commit scan did not
+complete (a `scanError` strand, with a best-effort PARTIAL `strandedCommits`),
+whose workspace is preserved best-effort so its
+`strandedCommits` stay recoverable; that preservation is still age-gated by the
+reaper and cleared on worker shutdown, so recover the SHAs promptly.
 
 ```bash
 # The harness sees a cloned repo at $AGENT_WORKSPACE; branch/push/PR are handled for it.

package/agent-instance.mjs

@@ -28,6 +28,57 @@ const SDK_UPDATE = 'updateAgentInstance';
 
 const isNonBlank = (v) => v != null && String(v).trim() !== '';
 const isPlainObject = (v) => v != null && typeof v === 'object' && !Array.isArray(v);
+// #229: collapse CR/LF (and other line/para separators) to a single space so a
+// multiline engine error can't split one correlation record across several
+// worker-log lines — that would both dilute the status/body diagnostic and let a
+// crafted error body spoof extra log lines. Used when rendering SDK errors.
+const oneLine = (v) => String(v).replace(/[\r\n\t\f\v\u0085\u2028\u2029]+/g, ' ');
+
+/**
+ * Extract the HTTP status + response body from an SDK rejection (#229).
+ *
+ * A create/append failure logged as an opaque `status 400` is useless — the body
+ * is what distinguishes a lease-fence rejection from a schema error from a 404.
+ * The host `@camunda8/orchestration-cluster-api` client surfaces these on a few
+ * shapes depending on the transport, so probe the common ones defensively and
+ * never throw (this runs on the best-effort logging path).
+ *
+ * @param {*} err
+ * @returns {{ status: (number|string|undefined), body: (string|undefined), message: string }}
+ */
+export function describeSdkError(err) {
+  if (err == null) return { status: undefined, body: undefined, message: String(err) };
+  const status =
+    err.status ??
+    err.statusCode ??
+    err?.response?.status ??
+    err?.response?.statusCode ??
+    (typeof err.code === 'number' ? err.code : undefined);
+  let body =
+    err.body ??
+    err.responseBody ??
+    err?.response?.data ??
+    err?.response?.body ??
+    undefined;
+  if (body != null && typeof body !== 'string') {
+    try {
+      body = JSON.stringify(body);
+    } catch {
+      body = String(body);
+    }
+  }
+  const message = err.message ? String(err.message) : String(err);
+  return { status, body, message };
+}
+
+/** One-line rendering of {@link describeSdkError} for a log line (body capped). */
+function formatSdkError(err) {
+  const { status, body, message } = describeSdkError(err);
+  const parts = [`status ${status ?? 'unknown'}`];
+  if (isNonBlank(body)) parts.push(`body ${oneLine(String(body).slice(0, 600))}`);
+  parts.push(`msg ${oneLine(message)}`);
+  return parts.join('; ');
+}
 
 /**
  * Is this activated job an `external` (job-backed) agent job — i.e. one whose
@@ -175,6 +226,24 @@ export function createAgentInstanceProducer(opts = {}) {
   const leaseToken = job?.leaseToken != null ? String(job.leaseToken) : '';
   const elementInstanceKey = job?.elementInstanceKey != null ? String(job.elementInstanceKey) : '';
   const elementId = job?.elementId != null ? String(job.elementId) : null;
+  // #229: cross-channel correlation. Stamp jobKey + elementInstanceKey +
+  // processInstanceKey on every producer log so the AgentInstance channel can be
+  // joined to the job / relay / git channels (today AgentInstance logs carry no
+  // processInstanceKey, so the four channels can't be reconciled).
+  const processInstanceKey = job?.processInstanceKey != null ? String(job.processInstanceKey) : '';
+  const corr = () =>
+    `job ${jobKey || '?'} eik ${elementInstanceKey || '?'} pik ${processInstanceKey || '?'}`;
+  // The lease token is a secret-ish fence token — never log it whole. `slice(-6)`
+  // would print a short/malformed lease (≤6 chars) IN FULL, so only surface a tail
+  // when the token is long enough that the tail still hides most of it; otherwise
+  // emit a safe digest (presence + length). Enough either way to tell "present"
+  // from "absent" and to correlate the activation without exposing the token.
+  const leaseNote = () =>
+    !leaseToken
+      ? 'lease absent'
+      : leaseToken.length > 8
+        ? `lease …${leaseToken.slice(-4)}`
+        : `lease present (len ${leaseToken.length})`;
 
   // The producer is a no-op unless every precondition holds: a usable SDK client,
   // an external agent job, and the ACP classifier. Any missing piece leaves the
@@ -191,6 +260,16 @@ export function createAgentInstanceProducer(opts = {}) {
   let activated = false;
   let loopIteration = 0;
   let queue = Promise.resolve();
+  // #229 turn accounting: how many AgentHistory turns were actually appended, and
+  // when the instance was minted — so `complete()` can log "N turns over Xm" and
+  // separate "created but nothing ingested" (the 0-turns husk) from "create failed".
+  let turnsAppended = 0;
+  let activatedAt = 0;
+  // #229 first-failure elevation: the FIRST per-turn append failure (400/404) for
+  // this instance is logged at `warn` (with the SDK verb + status); repeats stay at
+  // `debug` so a persistently-rejecting instance doesn't flood the log.
+  let appendFailureLogged = false;
+  let ingestFailureLogged = false;
   // Coalesce streamed message chunks (same messageId + role) into one turn, flushed
   // on a role/message boundary, a tool event, or completion — the engine dedups on
   // historyItemId (it does NOT merge), so a turn must be appended exactly once, whole.
@@ -201,10 +280,30 @@ export function createAgentInstanceProducer(opts = {}) {
   const iso = () => new Date(now()).toISOString();
 
   // Serialize an SDK call onto the queue so appends preserve order and `complete`
-  // can drain them. A rejection is swallowed (best-effort) but never breaks the chain.
-  const enqueue = (fn) => {
+  // can drain them. A rejection is best-effort (never breaks the chain), but the
+  // first append failure per instance is elevated to `warn` with the SDK verb +
+  // HTTP status (#229) — per-turn append failures were invisible at `debug`.
+  const enqueue = (fn, label = 'updateAgentInstance') => {
     queue = queue.then(fn).catch((err) => {
-      logger?.debug?.(`AgentInstance producer: SDK call failed — ${err?.message || err}`);
+      const { status, message } = describeSdkError(err);
+      // The first-failure elevation is for per-turn APPEND failures ONLY. A
+      // completion status update (status→COMPLETED) rides this same queue, so if
+      // it were allowed to consume the one-shot flag it would suppress the FIRST
+      // real per-turn append failure down to `debug` — the exact regression the
+      // #229 warning exists to prevent. Gate the elevation on the append label;
+      // completion failures are diagnosed separately in `complete()` (they render
+      // a failed/unknown terminal transition), so here they only ever log at debug.
+      const isAppend = label.includes('append');
+      if (isAppend && !appendFailureLogged) {
+        appendFailureLogged = true;
+        logger?.warn?.(
+          `AgentInstance producer: ${label} failed (${corr()}) — status ${status ?? 'unknown'}: ${oneLine(message)}; further append failures for this instance stay at debug.`,
+        );
+      } else {
+        logger?.debug?.(
+          `AgentInstance producer: ${label} failed (${corr()}) — status ${status ?? 'unknown'}: ${oneLine(message)}`,
+        );
+      }
     });
     return queue;
   };
@@ -222,8 +321,16 @@ export function createAgentInstanceProducer(opts = {}) {
         history: [turn],
       };
       if (status) req.status = status;
-      await camunda[SDK_UPDATE](req);
-    });
+      const res = await camunda[SDK_UPDATE](req);
+      // #229/#232: the engine dedups appends by historyItemId, so a retry or a
+      // reactivation can return 200 while creating ZERO new history entries. Count
+      // what the engine actually CREATED (`res.createdHistory`) — not the attempt —
+      // so the completion counter separates a real append from a deduplicated no-op
+      // and keeps the 0-turns husk diagnosis honest. Fall back to +1 only when the
+      // response omits the field (older engine), so a genuine append is never
+      // under-counted.
+      turnsAppended += Array.isArray(res?.createdHistory) ? res.createdHistory.length : 1;
+    }, 'updateAgentInstance(append)');
   };
 
   const flushMessage = () => {
@@ -289,6 +396,18 @@ export function createAgentInstanceProducer(opts = {}) {
     appendTurn(turn);
   };
 
+  // #229: first-failure elevation for ingest faults (classifier OR handler). The
+  // FIRST ingest failure per instance logs at `warn`; repeats stay at `debug` so a
+  // persistently-faulting instance doesn't flood the log.
+  const noteIngestFailure = (err) => {
+    if (!ingestFailureLogged) {
+      ingestFailureLogged = true;
+      logger?.warn?.(`AgentInstance producer: ingest failed (${corr()}) — ${oneLine(err?.message || err)}; further ingest failures for this instance stay at debug.`);
+    } else {
+      logger?.debug?.(`AgentInstance producer: ingest failed (${corr()}) — ${oneLine(err?.message || err)}`);
+    }
+  };
+
   return {
     /** True once the AgentInstance has been minted (or is being minted). */
     get active() {
@@ -339,15 +458,22 @@ export function createAgentInstanceProducer(opts = {}) {
           // result carries no key, fall back to the elementInstanceKey correlation is
           // not possible for updates (they need the agentInstanceKey), so disable.
           disabled = true;
-          logger?.warn?.('AgentInstance producer: create returned no agentInstanceKey; disabling durable transcript for this job.');
+          logger?.warn?.(`AgentInstance producer: create returned no agentInstanceKey (${corr()}); disabling durable transcript for this job.`);
           return false;
         }
         loopIteration = 1;
-        logger?.info?.(`AgentInstance ${agentInstanceKey} minted for element instance ${elementInstanceKey} (job ${jobKey}).`);
+        activatedAt = now();
+        logger?.info?.(`AgentInstance ${agentInstanceKey} minted (${corr()}; ${leaseNote()}; model ${oneLine(def.model)}/${oneLine(def.provider)}).`);
         return true;
       } catch (err) {
         disabled = true;
-        logger?.warn?.(`AgentInstance producer: createAgentInstance failed — ${err?.message || err}; continuing without a durable transcript (job completion unaffected).`);
+        // #229: the single line that would have root-caused the 20974 work-loss.
+        // Log the engine's HTTP status + response body (a lease-fence 400 vs a
+        // schema 400 vs a 404) plus the request correlation actually sent —
+        // elementInstanceKey/jobKey/processInstanceKey, whether the lease was
+        // present + its tail, and the model/provider. An opaque "status 400" alone
+        // is useless.
+        logger?.warn?.(`AgentInstance producer: createAgentInstance REJECTED (${corr()}; ${leaseNote()}; model ${oneLine(def.model)}/${oneLine(def.provider)}) — ${formatSdkError(err)}; continuing without a durable transcript (job completion unaffected).`);
         return false;
       }
     },
@@ -362,7 +488,11 @@ export function createAgentInstanceProducer(opts = {}) {
       let classified;
       try {
         classified = classify(rawUpdate);
-      } catch {
+      } catch (err) {
+        // A classifier/translation fault is an ingest failure too — route it
+        // through the same first-failure elevation (#229) instead of returning
+        // silently, or the ingest path can still drop every turn with no warning.
+        noteIngestFailure(err);
         return;
       }
       if (!classified || typeof classified !== 'object') return;
@@ -401,7 +531,10 @@ export function createAgentInstanceProducer(opts = {}) {
             break;
         }
       } catch (err) {
-        logger?.debug?.(`AgentInstance producer: ingest failed — ${err?.message || err}`);
+        // #229: elevate the FIRST ingest failure per instance to `warn` (repeats
+        // stay `debug`) so a translation/append fault that silently drops every
+        // turn is visible at normal verbosity.
+        noteIngestFailure(err);
       }
     },
 
@@ -425,6 +558,12 @@ export function createAgentInstanceProducer(opts = {}) {
         return;
       }
       flushMessage();
+      // Track whether the COMPLETED status request actually resolved. The update
+      // rides the best-effort queue (whose catch swallows SDK rejections), so we
+      // must NOT infer the terminal transition from `ok` alone — a 400/404 on the
+      // status update would otherwise be logged as a successful COMPLETED (#229),
+      // defeating the husk diagnosis. `null` ⇒ no status update attempted.
+      let statusResolved = ok ? false : null;
       if (ok) {
         enqueue(async () => {
           await camunda[SDK_UPDATE]({
@@ -434,9 +573,25 @@ export function createAgentInstanceProducer(opts = {}) {
             jobLease: leaseToken,
             status: 'COMPLETED',
           });
-        });
+          statusResolved = true;
+        }, 'updateAgentInstance(status→COMPLETED)');
       }
       try { await queue; } catch { /* best effort */ }
+      // #229: log a turn counter — "N turns appended over Xm, status→…" — so the
+      // 0-turns husk ("created but nothing ingested") is distinguishable from a
+      // healthy run at a glance, separate from the "create failed" line above.
+      const elapsedMs = activatedAt ? Math.max(0, now() - activatedAt) : 0;
+      const mins = (elapsedMs / 60000).toFixed(1);
+      // Render the ACTUAL terminal transition: COMPLETED only when the status
+      // update resolved; on a rejected update say so (the instance stays
+      // non-terminal, so a retry/reactivation still continues it); on a failed run
+      // no update was attempted at all.
+      const transition = ok
+        ? (statusResolved
+          ? 'COMPLETED'
+          : 'COMPLETED update FAILED — left non-terminal (retry/reactivation continues it)')
+        : 'left non-terminal (retry/reactivation continues it)';
+      logger?.info?.(`AgentInstance ${agentInstanceKey} (${corr()}): ${turnsAppended} turn(s) appended over ${mins}m, status→${transition}.`);
     },
   };
 }