@link-assistant/hive-mind 2.19.0 → 2.19.1

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # @link-assistant/hive-mind
 
+## 2.19.1
+
+### Patch Changes
+
+- 6cdf9e2: Keep the Formal AI release a verified update accepted, and record the backend that actually served each task rather than the local wrapper (issues #2207, #2208, delivered together for #2209).
+
+  Two defects sat on either side of the same handoff, and each one hid the other. The updater verified a new Formal AI image against the persisted memory volume, accepted it, and wrote it into `formal-ai-sidecar.json` — and then the next cold task threw it away, because `acquireFormalAiSidecar` resolved its image from the environment alone and re-selected the bootstrap release. Meanwhile the task's provenance reported `formal-ai --version` from the local wrapper, a binary that answers no requests, so the log could not have revealed the downgrade even when it happened.
+
+  - **The accepted image is what the next task boots.** `readAcceptedFormalAiImage` reads the acceptance back out of state and acquire passes it into image resolution. Precedence is an operator pin (exact and exclusive), then the accepted image (also exclusive), then the published bootstrap image only when nothing has been accepted yet, then the local Hive Mind image. The accepted candidates lead with the **immutable digest** and fall back to the tag only with `expectDigest` attached, so a tag that moved after acceptance is refused instead of booted — which is not hypothetical: `ghcr.io/link-assistant/formal-ai:latest` pointed at 0.346.0 when this was reported and points at 0.347.0 now. An accepted image that is no longer on the host is a refusal with the pin named in the message, never a silent downgrade onto memory that has already been migrated.
+  - **The serving backend is asked, not assumed.** `probeFormalAiBackend` queries `/health` on the configured endpoint before any client runs, within a bounded deadline, presenting the configured API key and returning immediately on 401/403 instead of retrying a credential that will still be wrong. `assertSupportedFormalAiBackend` gives each answer its own refusal — unreachable, unauthorized, non-JSON, JSON without a version, below `FORMAL_AI_MINIMUM_VERSION`, incompatible persisted memory — and never guesses a version.
+  - **The two versions have two names.** `runtime.formalAiVersion` is the release that answers the requests and is what the logs and provenance report; `runtime.formalAiWrapperVersion` is the local executable that started the server and wrote the config, keeping its own floor check and its own log line. `validateFormalAiToolConnection` is unchanged.
+  - **The halves are joined.** When Hive Mind manages the sidecar, the leased image, digest, source and verified version are published into the task environment, read back by the runtime, and required to agree with what `/health` reports; an endpoint serving something other than the accepted release is refused rather than recorded. The runtime cache is keyed by endpoint, not by the release behind it, so a cache hit re-probes and says so when the backend changed underneath a reused URL.
+
+  Three regression suites cover it: `tests/test-issue-2207-accepted-image-persistence.mjs` (candidate selection, update → stopped sidecar → acquire → release → acquire including a process restart against persisted state, a moved tag, a rollback, a pruned accepted image, an operator pin with a busy lease), `tests/test-issue-2208-serving-backend-provenance.mjs` (ten cases against real loopback HTTP backends), and `tests/test-issue-2209-verified-release-serving-provenance.mjs`, which runs the Docker simulator and a real serving process together so neither half can look correct while the pair fails. The same transition was then run against a real Docker daemon and the real published images; the evidence is in `docs/case-studies/issue-2209/`.
+
 ## 2.19.0
 
 ### Minor Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-assistant/hive-mind",
-  "version": "2.19.0",
+  "version": "2.19.1",
   "description": "AI-powered issue solver and hive mind for collaborative problem solving",
   "main": "src/hive.mjs",
   "type": "module",

package/src/formal-ai-image.lib.mjs

@@ -35,15 +35,20 @@
  * - `HIVE_MIND_FORMAL_AI_IMAGE` is an **exact** operator pin: it is the only
  *   candidate, and if it cannot be resolved the acquire fails. An operator who
  *   names an image means that image.
- * - Otherwise the published image is preferred, and the local Hive Mind image is
- *   the fallback. The fallback is only *used*; it is never pulled, because its
- *   whole point is that it is already on the host.
+ * - Otherwise the image accepted by the last verified idle update wins, by its
+ *   immutable digest (issue #2207). Before that rule the bootstrap version was
+ *   re-selected on every cold start, so an accepted update was discarded by the
+ *   next task and the sidecar returned to the release Hive Mind ships with.
+ * - Otherwise the published bootstrap image is preferred, and the local Hive
+ *   Mind image is the fallback. The fallback is only *used*; it is never pulled,
+ *   because its whole point is that it is already on the host.
  * - Nothing here downgrades a Formal AI task to another model. Issue #2146's
  *   fail-closed rule still holds: when no candidate resolves, the task is
  *   refused with an actionable message.
  *
  * @see https://github.com/link-assistant/hive-mind/issues/2154
  * @see https://github.com/link-assistant/hive-mind/issues/2146
+ * @see https://github.com/link-assistant/hive-mind/issues/2207
  */
 
 import { execFile } from 'node:child_process';
@@ -62,6 +67,7 @@ const DEFAULT_DOCKER_TIMEOUT_MS = 600_000;
 /** Where each candidate came from, so logs and errors can explain themselves. */
 export const FORMAL_AI_IMAGE_SOURCES = Object.freeze({
   PINNED: 'operator-pin',
+  ACCEPTED: 'accepted-update',
   PUBLISHED: 'published-image',
   HIVE_MIND: 'hive-mind-image',
 });
@@ -114,15 +120,82 @@ export const classifyDockerRegistryError = message => {
  */
 export const resolveFormalAiFallbackImage = (env = process.env) => String(env.HIVE_MIND_FORMAL_AI_FALLBACK_IMAGE || '').trim() || getDockerIsolationImage({ env });
 
+/**
+ * The last image an idle update actually accepted, read out of the durable
+ * sidecar record (issue #2207).
+ *
+ * `state.image`/`state.imageDigest` are a *cache of what is running* — every
+ * acquire overwrites them with the container it found — so they cannot answer
+ * "which image did we verify?". `state.lastUpdate` can: it is written only by
+ * `updateFormalAiSidecarWhenIdle` after the pull, the side-effect-free memory
+ * preflight, the backed-up migration and the `/health` memory-compatibility
+ * check have all succeeded, and it is left untouched by a rollback.
+ *
+ * @param {object|null} state - A `formal-ai-sidecar.json` record.
+ * @returns {{image: string|null, digest: string|null, version: string|null, memorySchemaVersion: (number|string|null), updatedAt: string|null}|null}
+ */
+export const readAcceptedFormalAiImage = state => {
+  const update = state?.lastUpdate;
+  if (!update || typeof update !== 'object') return null;
+  const image = String(update.image || '').trim() || null;
+  const digest = String(update.digest || '').trim() || null;
+  if (!image && !digest) return null;
+  return { image, digest, version: update.version ?? null, memorySchemaVersion: update.memorySchemaVersion ?? null, updatedAt: update.updatedAt ?? null };
+};
+
+/**
+ * Candidates that reproduce an accepted update, most trustworthy first.
+ *
+ * Two rules make this safe:
+ *
+ *  - **The digest wins.** `docker image inspect --format {{.Id}}` returns the
+ *    local content address, which Docker also accepts as a run reference. Using
+ *    it means a tag that moved after acceptance (`:latest` is the update tag)
+ *    cannot smuggle an unverified image past the memory verification that
+ *    earned the acceptance.
+ *  - **The tag is only a recovery path.** If the accepted image ID has been
+ *    pruned, the recorded reference may be pulled again, but the result is
+ *    accepted only when its digest still matches; otherwise the candidate is
+ *    refused rather than booted.
+ *
+ * @param {object|null} accepted - Output of {@link readAcceptedFormalAiImage}.
+ * @returns {Array<object>}
+ */
+export const resolveAcceptedFormalAiImageCandidates = (accepted = null) => {
+  if (!accepted) return [];
+  const candidates = [];
+  if (accepted.digest) candidates.push({ image: accepted.digest, reference: accepted.image || accepted.digest, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: false, accepted: true });
+  if (accepted.image && accepted.digest) candidates.push({ image: accepted.image, reference: accepted.image, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: true, accepted: true, expectDigest: accepted.digest });
+  // An acceptance recorded before digests were captured: the reference is all
+  // there is, so it may be used but never re-pulled behind a moving tag.
+  if (accepted.image && !accepted.digest) candidates.push({ image: accepted.image, reference: accepted.image, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: false, accepted: true });
+  return candidates;
+};
+
 /**
  * Ordered list of images to try, most preferred first.
  *
+ * Precedence, and why:
+ *
+ *  1. `HIVE_MIND_FORMAL_AI_IMAGE` — an operator who names an image means that
+ *     image; the pin is exact and exclusive.
+ *  2. The last accepted, verified update (issue #2207). Also exclusive: once an
+ *     image has been verified against the migrated persisted memory, quietly
+ *     booting the older bootstrap release against that memory is exactly the
+ *     silent downgrade the fail-closed rule forbids.
+ *  3. The bootstrap release, then the local Hive Mind image — the cold-start
+ *     path for a host that has never completed an update.
+ *
  * @param {object} [env]
+ * @param {object} [options]
+ * @param {object|null} [options.accepted] - Output of {@link readAcceptedFormalAiImage}.
  * @returns {Array<{image: string, source: string, pullable: boolean}>}
  */
-export const resolveFormalAiSidecarImageCandidates = (env = process.env) => {
+export const resolveFormalAiSidecarImageCandidates = (env = process.env, { accepted = null } = {}) => {
   const pinned = String(env.HIVE_MIND_FORMAL_AI_IMAGE || '').trim();
   if (pinned) return [{ image: pinned, source: FORMAL_AI_IMAGE_SOURCES.PINNED, pullable: true }];
+  const acceptedCandidates = resolveAcceptedFormalAiImageCandidates(accepted);
+  if (acceptedCandidates.length > 0) return acceptedCandidates;
   const candidates = [{ image: `${FORMAL_AI_IMAGE_REPOSITORY}:${FORMAL_AI_BOOTSTRAP_VERSION}`, source: FORMAL_AI_IMAGE_SOURCES.PUBLISHED, pullable: true }];
   const fallback = resolveFormalAiFallbackImage(env);
   // Never pulled: the fallback's value is that it is already on the host. A
@@ -132,7 +205,7 @@ export const resolveFormalAiSidecarImageCandidates = (env = process.env) => {
 };
 
 /** The preferred image, kept for callers and logs that only need the headline reference. */
-export const resolveFormalAiSidecarImage = (env = process.env) => resolveFormalAiSidecarImageCandidates(env)[0].image;
+export const resolveFormalAiSidecarImage = (env = process.env, options = {}) => resolveFormalAiSidecarImageCandidates(env, options)[0].image;
 
 const inspectLocalImage = async (image, { run, timeoutMs }) => {
   try {
@@ -148,7 +221,8 @@ const inspectLocalImage = async (image, { run, timeoutMs }) => {
  * bot log knows the cause and the fix without opening a shell.
  */
 const describeFailure = attempts => {
-  const lines = ['No Formal AI image could be resolved, so the sidecar was not started.'];
+  const accepted = attempts.some(attempt => attempt.source === FORMAL_AI_IMAGE_SOURCES.ACCEPTED);
+  const lines = [accepted ? 'The Formal AI image accepted by the last verified update could not be resolved, so the sidecar was not started.' : 'No Formal AI image could be resolved, so the sidecar was not started.'];
   for (const attempt of attempts) {
     if (attempt.source === FORMAL_AI_IMAGE_SOURCES.HIVE_MIND && attempt.kind === 'absent') {
       lines.push(`• ${attempt.image} (local Hive Mind image, fallback): not present on this host — Docker-isolated tasks would have to pull it too.`);
@@ -157,6 +231,13 @@ const describeFailure = attempts => {
     lines.push(`• ${attempt.image} (${attempt.source}): ${attempt.reason}${attempt.error ? ` — ${attempt.error}` : ''}`);
     for (const step of attempt.remediation ?? []) lines.push(`    → ${step}`);
   }
+  if (accepted) {
+    // Falling back to the bootstrap release here would run an *older* binary
+    // against memory that the accepted release already migrated (issue #2207).
+    lines.push('Hive Mind will not fall back to an older Formal AI release against memory an accepted update already migrated.');
+    lines.push('    → restore the accepted image on this host (`docker pull` it, or copy it back), or');
+    lines.push('    → set HIVE_MIND_FORMAL_AI_IMAGE to the image you want this host to run, which overrides the accepted update.');
+  }
   lines.push('Formal AI tasks fail closed by design (issue #2146): Hive Mind will not silently run them on another model.');
   return lines.join('\n');
 };
@@ -172,17 +253,27 @@ const describeFailure = attempts => {
  * @param {Function|null} [params.log]
  * @param {boolean} [params.verbose]
  * @param {boolean} [params.pull] - Set false to accept only images already on the host.
- * @returns {Promise<{image: string, source: string, digest: string|null, pulled: boolean, attempts: object[]}>}
+ * @param {object|null} [params.accepted] - Output of {@link readAcceptedFormalAiImage}, used to build the default candidates.
+ * @returns {Promise<{image: string, reference: string, source: string, digest: string|null, pulled: boolean, attempts: object[]}>}
  * @throws {Error} When no candidate resolves; the message names every attempt and its fix.
  */
-export const ensureFormalAiSidecarImage = async ({ env = process.env, run = execFileAsync, timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS, log = null, verbose = false, pull = true, candidates = resolveFormalAiSidecarImageCandidates(env) } = {}) => {
+export const ensureFormalAiSidecarImage = async ({ env = process.env, run = execFileAsync, timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS, log = null, verbose = false, pull = true, accepted = null, candidates = resolveFormalAiSidecarImageCandidates(env, { accepted }) } = {}) => {
   const attempts = [];
+  const reference = candidate => candidate.reference || candidate.image;
+  // A candidate that names an expected digest is a *recovery* path for an
+  // accepted update whose image ID was pruned: the reference may be re-fetched,
+  // but only the recorded content address is allowed to run (issue #2207).
+  const mismatch = (candidate, digest) => ({ ...candidate, kind: 'digest-mismatch', reason: `resolved to ${digest ?? 'no digest'} instead of the accepted ${candidate.expectDigest}`, error: null, remediation: ['the tag moved after the update was accepted; restore the accepted image or repin with HIVE_MIND_FORMAL_AI_IMAGE'] });
 
   for (const candidate of candidates) {
     const localDigest = await inspectLocalImage(candidate.image, { run, timeoutMs });
+    if (localDigest && candidate.expectDigest && localDigest !== candidate.expectDigest) {
+      attempts.push(mismatch(candidate, localDigest));
+      continue;
+    }
     if (localDigest) {
       if (verbose && log) await log(`[VERBOSE] formal-ai-image: using '${candidate.image}' (${candidate.source}) already present locally, digest=${localDigest}`);
-      return { image: candidate.image, source: candidate.source, digest: localDigest, pulled: false, attempts };
+      return { image: candidate.image, reference: reference(candidate), source: candidate.source, digest: localDigest, pulled: false, attempts };
     }
 
     if (!candidate.pullable || !pull) {
@@ -202,8 +293,13 @@ export const ensureFormalAiSidecarImage = async ({ env = process.env, run = exec
     }
 
     const digest = await inspectLocalImage(candidate.image, { run, timeoutMs });
+    if (candidate.expectDigest && digest !== candidate.expectDigest) {
+      attempts.push(mismatch(candidate, digest));
+      if (log) await log(`⚠️ ${candidate.image} no longer resolves to the accepted digest ${candidate.expectDigest}; refusing to boot it.`);
+      continue;
+    }
     if (verbose && log) await log(`[VERBOSE] formal-ai-image: pulled '${candidate.image}' (${candidate.source}), digest=${digest ?? 'unknown'}`);
-    return { image: candidate.image, source: candidate.source, digest, pulled: true, attempts };
+    return { image: candidate.image, reference: reference(candidate), source: candidate.source, digest, pulled: true, attempts };
   }
 
   const error = new Error(describeFailure(attempts));
@@ -216,6 +312,8 @@ export default {
   FORMAL_AI_IMAGE_SOURCES,
   classifyDockerRegistryError,
   ensureFormalAiSidecarImage,
+  readAcceptedFormalAiImage,
+  resolveAcceptedFormalAiImageCandidates,
   resolveFormalAiFallbackImage,
   resolveFormalAiSidecarImage,
   resolveFormalAiSidecarImageCandidates,

package/src/formal-ai-isolation.lib.mjs

@@ -16,6 +16,7 @@
  * @see https://github.com/link-assistant/hive-mind/issues/2146
  */
 
+import { buildFormalAiSidecarProvenanceEnv } from './formal-ai-runtime.lib.mjs';
 import { acquireFormalAiSidecar, attachTaskToFormalAiNetwork, isFormalAiSidecarEnabled, isFormalAiTask, releaseFormalAiSidecar } from './formal-ai-sidecar.lib.mjs';
 
 const logToConsole = message => console.log(message);
@@ -43,6 +44,19 @@ export const acquireFormalAiSidecarForTask = async ({ backend, args = [], model
   }
 };
 
+/**
+ * The environment a Formal AI task is launched with.
+ *
+ * Besides the endpoint, this carries the identity of the image the lease pinned
+ * (issue #2208). The runtime inside the container still asks the endpoint which
+ * release is answering; these values let it refuse when the answer is not the
+ * accepted image Hive Mind leased, and let the task record the digest rather
+ * than a mutable tag.
+ *
+ * @returns {object} The task environment (`env` itself when Formal AI is not involved).
+ */
+export const buildFormalAiTaskEnv = ({ sidecar, env = process.env } = {}) => (sidecar ? { ...env, HIVE_MIND_FORMAL_AI_BASE_URL: sidecar.baseUrl, ...buildFormalAiSidecarProvenanceEnv(sidecar) } : env);
+
 /**
  * Add the internal Formal AI network to the freshly-created task container.
  *
@@ -66,4 +80,4 @@ export const releaseFormalAiSidecarForTask = async ({ sidecar, sessionId, env =
   }
 };
 
-export default { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, releaseFormalAiSidecarForTask };
+export default { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, buildFormalAiTaskEnv, releaseFormalAiSidecarForTask };

package/src/formal-ai-runtime.lib.mjs

@@ -53,7 +53,7 @@ import { homedir } from 'node:os';
 import { dirname, join } from 'node:path';
 import { promisify } from 'node:util';
 
-import { assertSupportedFormalAiVersion, FORMAL_AI_MINIMUM_VERSION, readFormalAiBinaryVersion } from './formal-ai-version.lib.mjs';
+import { assertSupportedFormalAiVersion, FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast, readFormalAiBinaryVersion } from './formal-ai-version.lib.mjs';
 
 const execFileAsync = promisify(execFile);
 
@@ -65,6 +65,26 @@ export const FORMAL_AI_MODEL_NAME = 'formal-ai';
 export const FORMAL_AI_DEFAULT_HOST = '127.0.0.1';
 export const FORMAL_AI_SERVER_READY_TIMEOUT_MS = 90_000;
 
+/** Where a Formal AI server reports its own version and memory compatibility. */
+export const FORMAL_AI_HEALTH_PATH = '/health';
+
+/**
+ * Bounded budget for the backend probe (issue #2208).
+ *
+ * Deliberately short: this runs immediately before the client executes, so a
+ * hanging endpoint must surface as a refusal rather than as a task that appears
+ * to be thinking. The server-start path has its own, much longer budget.
+ */
+export const FORMAL_AI_BACKEND_PROBE_TIMEOUT_MS = 15_000;
+
+/** Environment carrying the Hive-Mind-managed sidecar's identity into the task container (issue #2207/#2208). */
+export const FORMAL_AI_SIDECAR_PROVENANCE_ENV = Object.freeze({
+  image: 'HIVE_MIND_FORMAL_AI_SIDECAR_IMAGE',
+  imageDigest: 'HIVE_MIND_FORMAL_AI_SIDECAR_DIGEST',
+  version: 'HIVE_MIND_FORMAL_AI_SIDECAR_VERSION',
+  imageSource: 'HIVE_MIND_FORMAL_AI_SIDECAR_SOURCE',
+});
+
 export const resolveFormalAiApiKey = (env = process.env) => env.FORMAL_AI_API_KEY?.trim() || FORMAL_AI_DEFAULT_API_KEY;
 
 /**
@@ -164,6 +184,109 @@ export const waitForFormalAiServerReady = async ({ baseUrl, probePath = '/api/op
   return { ready: false, error: lastError || 'timed out' };
 };
 
+/**
+ * Ask the endpoint that will actually answer the model requests who it is
+ * (issue #2208).
+ *
+ * Until this existed, `prepareFormalAiRuntime` read `formal-ai --version` from
+ * the *local* executable and logged the answer as "the Formal AI version" even
+ * when `HIVE_MIND_FORMAL_AI_BASE_URL` pointed at a container running a
+ * completely different release. Every provenance record therefore named a binary
+ * that never saw the request.
+ *
+ * Properties that matter:
+ *
+ *  - **Bounded.** The whole probe shares one deadline, and each request carries
+ *    its own abort signal, so an endpoint that accepts the connection and then
+ *    goes quiet cannot stall the task.
+ *  - **Authenticated.** The configured key is presented the way the served API
+ *    expects it. A 401/403 is reported as an authentication problem and is *not*
+ *    retried — retrying a rejected credential only delays the failure.
+ *  - **Honest about malformed answers.** A body that is not JSON, or that
+ *    carries no version, is a distinct outcome from "unreachable"; guessing a
+ *    version here would recreate the defect this function exists to fix.
+ *
+ * @returns {Promise<{ok: boolean, kind: string, status: number|null, version: string|null, memory: object|null, health: object|null, error: string|null}>}
+ */
+export const probeFormalAiBackend = async ({ baseUrl, apiKey = null, path = FORMAL_AI_HEALTH_PATH, timeoutMs = FORMAL_AI_BACKEND_PROBE_TIMEOUT_MS, fetchImpl = globalThis.fetch, intervalMs = 500, now = () => Date.now(), sleepImpl = delay } = {}) => {
+  if (!baseUrl) return { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: 'no base URL to probe' };
+  const url = `${String(baseUrl).replace(/\/+$/, '')}${path}`;
+  const headers = apiKey ? { Authorization: `Bearer ${apiKey}`, 'X-Api-Key': apiKey } : {};
+  const deadline = now() + timeoutMs;
+  let last = { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: 'not probed' };
+
+  for (;;) {
+    const remaining = deadline - now();
+    if (remaining <= 0) break;
+    try {
+      const response = await fetchImpl(url, { headers, signal: AbortSignal.timeout(Math.min(remaining, timeoutMs)) });
+      const status = response?.status ?? null;
+      if (status === 401 || status === 403) {
+        // Not retried: the credential is wrong, and it will still be wrong in 500ms.
+        return { ok: false, kind: 'unauthorized', status, version: null, memory: null, health: null, error: `the Formal AI endpoint rejected the configured credentials (HTTP ${status})` };
+      }
+      if (!response?.ok) {
+        last = { ok: false, kind: 'http-error', status, version: null, memory: null, health: null, error: `HTTP ${status}` };
+      } else {
+        const text = await response.text();
+        let health;
+        try {
+          health = JSON.parse(text);
+        } catch {
+          return { ok: false, kind: 'malformed', status, version: null, memory: null, health: null, error: `${path} did not return JSON: ${text.slice(0, 200)}` };
+        }
+        const version = typeof health?.version === 'string' ? health.version.trim() || null : null;
+        if (!version) return { ok: false, kind: 'no-version', status, version: null, memory: health?.memory ?? null, health, error: `${path} answered without a version field` };
+        return { ok: true, kind: 'ok', status, version, memory: health?.memory ?? null, health, error: null };
+      }
+    } catch (error) {
+      last = { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: error?.message || String(error) };
+    }
+    if (deadline - now() <= intervalMs) break;
+    await sleepImpl(intervalMs);
+  }
+  return last;
+};
+
+/**
+ * Turn a probe result into either the accepted backend description or an
+ * actionable refusal.
+ *
+ * Fail-closed (issue #2146) applies here too: a Formal AI task that cannot prove
+ * which release is serving it must stop, not proceed and record a guess.
+ *
+ * @param {object} probe - Output of {@link probeFormalAiBackend}.
+ * @param {object} context
+ * @param {string} context.baseUrl
+ * @param {string} [context.minimumVersion]
+ * @param {string|null} [context.expectedVersion] - Version the leased sidecar image was verified at.
+ * @returns {{version: string, memory: object|null}}
+ */
+export const assertSupportedFormalAiBackend = (probe, { baseUrl, minimumVersion = FORMAL_AI_MINIMUM_VERSION, expectedVersion = null } = {}) => {
+  const where = `the Formal AI endpoint ${baseUrl}`;
+  if (!probe?.ok) {
+    const detail = probe?.error ? `: ${probe.error}` : '';
+    if (probe?.kind === 'unauthorized') throw new Error(`${where} refused the configured credentials${detail}. Set FORMAL_AI_API_KEY to a key the server accepts.`);
+    if (probe?.kind === 'malformed') throw new Error(`${where} answered ${FORMAL_AI_HEALTH_PATH} with something other than JSON${detail}. Hive Mind will not guess which Formal AI release is serving this task.`);
+    if (probe?.kind === 'no-version') throw new Error(`${where} answered ${FORMAL_AI_HEALTH_PATH} without a version${detail}. Hive Mind requires a serving backend that reports its version (Formal AI >= ${minimumVersion}).`);
+    if (probe?.kind === 'http-error') throw new Error(`${where} did not serve ${FORMAL_AI_HEALTH_PATH}${detail}. Check that HIVE_MIND_FORMAL_AI_BASE_URL points at a Formal AI server >= ${minimumVersion}.`);
+    throw new Error(`${where} could not be reached${detail}. Check HIVE_MIND_FORMAL_AI_BASE_URL and that the Formal AI server is running.`);
+  }
+  if (!isFormalAiVersionAtLeast(probe.version, minimumVersion)) {
+    throw new Error(`${where} serves Formal AI ${probe.version}, but Hive Mind requires >= ${minimumVersion}. Upgrade the server; the local wrapper's version does not change what answers the requests.`);
+  }
+  if (probe.memory?.compatible === false) {
+    throw new Error(`${where} reports incompatible persisted memory (migration_state=${probe.memory?.migration_state ?? 'unknown'}). Refusing to run a task against memory the serving release cannot read.`);
+  }
+  if (expectedVersion && expectedVersion !== probe.version) {
+    // A lease pins the sidecar's image for the whole task, so the endpoint
+    // answering with a different release means it is not the container Hive
+    // Mind verified and leased.
+    throw new Error(`${where} serves Formal AI ${probe.version}, but the leased Hive Mind sidecar image was verified as ${expectedVersion}. Refusing to record provenance for a backend that is not the accepted release.`);
+  }
+  return { version: probe.version, memory: probe.memory ?? null };
+};
+
 /** Read the machine-readable client registry (`formal-ai clients --format json`). */
 export const loadFormalAiClientRegistry = async ({ formalAiPath = 'formal-ai', run = execFileAsync, env = process.env, timeoutMs = 30_000 } = {}) => {
   const result = await run(formalAiPath, ['clients', '--format', 'json'], { encoding: 'utf8', env: { ...process.env, ...env }, timeout: timeoutMs, maxBuffer: 32 * 1024 * 1024 });
@@ -321,6 +444,69 @@ export const startFormalAiServer = async ({ cwd, host = FORMAL_AI_DEFAULT_HOST,
   };
 };
 
+/**
+ * The provenance Hive Mind's sidecar lifecycle publishes for the container it
+ * leased to this task (issues #2207, #2208).
+ *
+ * The lease pins one verified image for the whole task, so these values say
+ * which release *should* be answering. They are a cross-check, never a
+ * substitute for asking the endpoint itself.
+ */
+export const readFormalAiSidecarProvenance = (env = process.env) => {
+  const read = name => String(env[name] || '').trim() || null;
+  const image = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.image);
+  const imageDigest = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageDigest);
+  const version = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.version);
+  const imageSource = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageSource);
+  if (!image && !imageDigest && !version) return null;
+  return { image, imageDigest, version, imageSource };
+};
+
+/**
+ * Publish the leased sidecar's identity into a task's environment so the runtime
+ * inside the container can cross-check the endpoint it is pointed at.
+ *
+ * @param {object|null} sidecar - An `acquireFormalAiSidecar` result.
+ * @returns {object} Environment entries (empty when nothing is known).
+ */
+export const buildFormalAiSidecarProvenanceEnv = (sidecar = null) => {
+  if (!sidecar) return {};
+  const entries = {
+    [FORMAL_AI_SIDECAR_PROVENANCE_ENV.image]: sidecar.imageReference || sidecar.image || null,
+    [FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageDigest]: sidecar.imageDigest || null,
+    [FORMAL_AI_SIDECAR_PROVENANCE_ENV.version]: sidecar.servingVersion || sidecar.health?.version || null,
+    [FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageSource]: sidecar.imageSource || null,
+  };
+  return Object.fromEntries(Object.entries(entries).filter(([, value]) => value));
+};
+
+/** One-line description of the backend for logs and session provenance. */
+const describeFormalAiBackend = backend => [`${backend.version} at ${backend.baseUrl}`, backend.image ? `image ${backend.image}` : null, backend.imageDigest ? `digest ${backend.imageDigest}` : null].filter(Boolean).join(', ');
+
+/**
+ * Query the endpoint that will serve this task and build its provenance record.
+ *
+ * @returns {Promise<object>} `{ baseUrl, version, memory, image, imageDigest, imageSource, leased, startedLocally, probedAt }`
+ */
+const resolveFormalAiBackend = async ({ baseUrl, apiKey, env, deps, startedLocally }) => {
+  const sidecar = readFormalAiSidecarProvenance(env);
+  const probe = await (deps.probeBackendImpl || probeFormalAiBackend)({ baseUrl, apiKey, env });
+  const { version, memory } = assertSupportedFormalAiBackend(probe, { baseUrl, expectedVersion: sidecar?.version ?? null });
+  return {
+    baseUrl,
+    version,
+    memory,
+    image: sidecar?.image ?? null,
+    imageDigest: sidecar?.imageDigest ?? null,
+    imageSource: sidecar?.imageSource ?? null,
+    /** True when Hive Mind leased this endpoint from its own sidecar. */
+    leased: !!sidecar,
+    /** True when this process started the server it is now talking to. */
+    startedLocally,
+    probedAt: new Date().toISOString(),
+  };
+};
+
 const runtimeCache = new Map();
 let exitHookInstalled = false;
 
@@ -357,16 +543,32 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
   const resolvedFormalAiPath = formalAiPath || env.HIVE_MIND_FORMAL_AI_PATH?.trim() || 'formal-ai';
   const cacheKey = `${tool}::${workdir}::${env.HIVE_MIND_FORMAL_AI_BASE_URL || ''}`;
   const cached = runtimeCache.get(cacheKey);
-  if (cached) return cached.runtime;
+  if (cached) {
+    // Issue #2208: the cache key is the endpoint, not the release behind it. An
+    // external base URL can be re-pointed at a different container between
+    // tasks, so the cached provenance is re-checked instead of replayed.
+    const backend = await resolveFormalAiBackend({ baseUrl: cached.runtime.baseUrl, apiKey: resolveFormalAiApiKey(env), env, deps, startedLocally: cached.runtime.serverStarted });
+    if (backend.version !== cached.runtime.backend?.version || backend.imageDigest !== cached.runtime.backend?.imageDigest) {
+      await log(`🧠 Formal AI: serving backend changed to ${describeFormalAiBackend(backend)}`);
+    }
+    cached.runtime.backend = backend;
+    cached.runtime.formalAiVersion = backend.version;
+    return cached.runtime;
+  }
 
   installExitHook();
 
   // Issue #2146: `--no-tool-check` skipped the only version probe, allowing an
   // old Formal AI build to return the same unexecuted plan through all five
   // Claude/Codex restarts. Runtime safety cannot depend on preflight options.
-  const formalAiVersion = await (deps.readVersionImpl || readFormalAiBinaryVersion)({ formalAiPath: resolvedFormalAiPath, env });
-  assertSupportedFormalAiVersion(formalAiVersion);
-  await log(`🧠 Formal AI: version ${formalAiVersion} (minimum ${FORMAL_AI_MINIMUM_VERSION})`);
+  //
+  // This is the *local wrapper*: the executable that starts the server and
+  // writes the client configuration. Issue #2208: it is not necessarily the
+  // release that answers the model requests, so it keeps its own name and its
+  // own compatibility check, and it is never reported as the serving version.
+  const formalAiWrapperVersion = await (deps.readVersionImpl || readFormalAiBinaryVersion)({ formalAiPath: resolvedFormalAiPath, env });
+  assertSupportedFormalAiVersion(formalAiWrapperVersion);
+  await log(`🧠 Formal AI: local wrapper version ${formalAiWrapperVersion} (minimum ${FORMAL_AI_MINIMUM_VERSION})`);
 
   const apiKey = resolveFormalAiApiKey(env);
   const externalBaseUrl = env.HIVE_MIND_FORMAL_AI_BASE_URL?.trim() || null;
@@ -386,6 +588,13 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
       await log(`🧠 Formal AI: using the configured server ${baseUrl}`, { verbose: true });
     }
 
+    // Before any client configuration is written, ask the endpoint who it is.
+    const backend = await resolveFormalAiBackend({ baseUrl, apiKey, env, deps, startedLocally: !!server });
+    await log(`🧠 Formal AI: serving backend ${describeFormalAiBackend(backend)}`);
+    if (backend.version !== formalAiWrapperVersion) {
+      await log(`🧠 Formal AI: local wrapper ${formalAiWrapperVersion} differs from the serving backend ${backend.version}; provenance records the backend`, { verbose: true });
+    }
+
     const clients = await (deps.loadRegistryImpl || loadFormalAiClientRegistry)({ formalAiPath: resolvedFormalAiPath, env });
     const client = findFormalAiClient(clients, tool);
     if (!client) throw new Error(`Formal AI does not list a client configuration for "${tool}"`);
@@ -429,7 +638,11 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
       client,
       notes,
       serverStarted: !!server,
-      formalAiVersion,
+      /** The release that actually answers this task's model requests. */
+      formalAiVersion: backend.version,
+      /** The local executable that started the server and wrote the config. */
+      formalAiWrapperVersion,
+      backend,
       stop: async () => {
         runtimeCache.delete(cacheKey);
         await server?.stop?.();

package/src/formal-ai-sidecar.lib.mjs

@@ -46,7 +46,7 @@ import fs from 'node:fs';
 import { promisify } from 'node:util';
 
 import { FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast } from './formal-ai-version.lib.mjs';
-import { ensureFormalAiSidecarImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
+import { ensureFormalAiSidecarImage, readAcceptedFormalAiImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
 import { isFormalAiModel } from './formal-ai-model.lib.mjs';
 import { getModelFromArgs } from './model-args.lib.mjs';
 import { withStateLock } from './state-lock.lib.mjs';
@@ -90,7 +90,7 @@ export const FORMAL_AI_MEMORY_PATH = `${FORMAL_AI_MEMORY_MOUNT}/memory.lino`;
  * issue #2154 taught us that "which image" and "is it actually pullable" are the
  * same question.
  */
-export { FORMAL_AI_IMAGE_REPOSITORY, resolveFormalAiSidecarImage, resolveFormalAiSidecarImageCandidates } from './formal-ai-image.lib.mjs';
+export { FORMAL_AI_IMAGE_REPOSITORY, readAcceptedFormalAiImage, resolveAcceptedFormalAiImageCandidates, resolveFormalAiSidecarImage, resolveFormalAiSidecarImageCandidates } from './formal-ai-image.lib.mjs';
 
 /** Applied to the sidecar, its network and its volume so reconciliation can find them. */
 export const FORMAL_AI_SIDECAR_LABEL = 'com.link-assistant.hive-mind.formal-ai';
@@ -102,7 +102,13 @@ const SIDECAR_LOCK_NAME = 'formal-ai-sidecar';
 const DEFAULT_HEALTH_ATTEMPTS = 60;
 const DEFAULT_HEALTH_DELAY_MS = 1000;
 
-const EMPTY_STATE = Object.freeze({ version: 1, image: null, imageDigest: null, startedAt: null, leases: [], lastUpdate: null });
+/**
+ * `lastUpdate` is the accepted-release record (issue #2207) and `serving` is the
+ * provenance of the container that last answered a lease (issue #2208). Neither
+ * is derivable from `image`/`imageDigest`, which are only a cache of whatever is
+ * running right now.
+ */
+const EMPTY_STATE = Object.freeze({ version: 1, image: null, imageReference: null, imageDigest: null, startedAt: null, leases: [], lastUpdate: null, serving: null });
 
 /**
  * True when a task will be driven by Formal AI.
@@ -350,14 +356,21 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
       // #2154 the reference went straight into `docker run`, so a registry that
       // refused the pull surfaced as an unreadable `Command failed: docker run …`
       // dump and the task died even though a usable image sat on the host.
-      const resolved = container.exists && container.image ? { image: container.image, source: 'running-sidecar', pulled: false } : await ensureFormalAiSidecarImage({ env, run, timeoutMs: imageTimeoutMs, log, verbose });
+      //
+      // `accepted` is what issue #2207 was missing: without it the candidate
+      // list was rebuilt from the bootstrap pin on every cold start, so an
+      // update that had already been pulled, migrated and verified was silently
+      // discarded by the very next task.
+      const accepted = readAcceptedFormalAiImage(state);
+      const resolved = container.exists && container.image ? { image: container.image, reference: container.image, source: 'running-sidecar', pulled: false } : await ensureFormalAiSidecarImage({ env, accepted, run, timeoutMs: imageTimeoutMs, log, verbose });
       const image = resolved.image;
+      const imageReference = resolved.reference || image;
 
       await ensureFormalAiNetwork({ run, timeoutMs, log, verbose });
       await ensureFormalAiMemoryVolume({ image, run, timeoutMs, log, verbose });
 
       if (!container.exists) {
-        if (log) await log(`🧠 Starting the Formal AI sidecar (${image}, ${resolved.source}) on the internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
+        if (log) await log(`🧠 Starting the Formal AI sidecar (${imageReference}${imageReference === image ? '' : ` @ ${image}`}, ${resolved.source}) on the internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
         await dockerText(run, buildFormalAiSidecarRunArgs({ image, env }), { timeoutMs });
         container = await inspectDockerContainer(FORMAL_AI_SIDECAR_CONTAINER_NAME, { run, timeoutMs });
       }
@@ -375,15 +388,27 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
       // and then fail on the agent-mode API the tasks depend on.
       const reportedVersion = health.health?.version ?? null;
       if (reportedVersion && !isFormalAiVersionAtLeast(reportedVersion, FORMAL_AI_MINIMUM_VERSION)) {
-        throw new Error(`Formal AI sidecar image ${image} (${resolved.source}) runs formal-ai ${reportedVersion}, but Hive Mind requires >= ${FORMAL_AI_MINIMUM_VERSION}. Rebuild or repin the image (HIVE_MIND_FORMAL_AI_IMAGE) before running Formal AI tasks.`);
+        throw new Error(`Formal AI sidecar image ${imageReference} (${resolved.source}) runs formal-ai ${reportedVersion}, but Hive Mind requires >= ${FORMAL_AI_MINIMUM_VERSION}. Rebuild or repin the image (HIVE_MIND_FORMAL_AI_IMAGE) before running Formal AI tasks.`);
       }
 
       const address = await readFormalAiSidecarAddress({ run, timeoutMs });
       const acquiredAt = now().toISOString();
       const nextLeases = [...leases.filter(lease => lease.sessionId !== sessionId), { sessionId, tool, model, acquiredAt }];
-      writeFormalAiSidecarState({ ...state, image: container.image || image, imageDigest: container.imageDigest, startedAt: state.startedAt || acquiredAt, leases: nextLeases }, { env, fsImpl });
+      // The provenance of what is *actually serving* this lease, so a task's
+      // evidence can name the release that answered its requests rather than
+      // whatever binary happens to sit next to the wrapper (issue #2208).
+      const serving = {
+        image: imageReference,
+        imageDigest: container.imageDigest ?? resolved.digest ?? null,
+        imageSource: resolved.source,
+        version: reportedVersion,
+        memorySchemaVersion: health.health?.memory?.schema_version ?? null,
+        acceptedAt: accepted?.updatedAt ?? null,
+        observedAt: acquiredAt,
+      };
+      writeFormalAiSidecarState({ ...state, image: container.image || image, imageReference, imageDigest: container.imageDigest, startedAt: state.startedAt || acquiredAt, leases: nextLeases, serving }, { env, fsImpl });
 
-      if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: lease '${sessionId}' acquired (${nextLeases.length} active), image=${container.image || image} (${resolved.source}), digest=${container.imageDigest ?? 'unknown'}, address=${address ?? 'unknown'}, formal-ai=${health.health?.version ?? 'unknown'}, memory schema=${health.health?.memory?.schema_version ?? 'unknown'}`);
+      if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: lease '${sessionId}' acquired (${nextLeases.length} active), image=${imageReference} (${resolved.source}), digest=${container.imageDigest ?? 'unknown'}, address=${address ?? 'unknown'}, formal-ai=${reportedVersion ?? 'unknown'}, memory schema=${serving.memorySchemaVersion ?? 'unknown'}`);
 
       return {
         address,
@@ -394,8 +419,14 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
         containerName: FORMAL_AI_SIDECAR_CONTAINER_NAME,
         memoryVolume: FORMAL_AI_MEMORY_VOLUME_NAME,
         image: container.image || image,
+        imageReference,
         imageSource: resolved.source,
-        imageDigest: container.imageDigest,
+        imageDigest: serving.imageDigest,
+        acceptedImage: accepted?.image ?? null,
+        acceptedDigest: accepted?.digest ?? null,
+        acceptedVersion: accepted?.version ?? null,
+        servingVersion: reportedVersion,
+        memorySchemaVersion: serving.memorySchemaVersion,
         health: health.health,
         leaseCount: nextLeases.length,
       };
@@ -453,6 +484,7 @@ export default {
   inspectDockerContainer,
   isFormalAiSidecarEnabled,
   isFormalAiTask,
+  readAcceptedFormalAiImage,
   readDockerImageDigest,
   readFormalAiSidecarAddress,
   readFormalAiSidecarState,

package/src/formal-ai.lib.mjs

@@ -92,6 +92,11 @@ export const resolveFormalAiToolExecution = async ({ tool, model, toolPath, work
     env: runtime.env,
     home: runtime.home,
     client: runtime.client,
+    // Issue #2208: provenance names the backend that answered, and the local
+    // wrapper under its own name — the two are not interchangeable.
+    backend: runtime.backend,
+    formalAiVersion: runtime.formalAiVersion,
+    formalAiWrapperVersion: runtime.formalAiWrapperVersion,
     stop: runtime.stop,
   };
 };

package/src/isolation-runner.lib.mjs

@@ -19,7 +19,7 @@ import fs from 'node:fs';
 import os from 'node:os';
 import path from 'node:path';
 import { isExecutingSessionStatus, isTerminalSessionStatus } from './session-status.lib.mjs';
-import { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, releaseFormalAiSidecarForTask } from './formal-ai-isolation.lib.mjs';
+import { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, buildFormalAiTaskEnv, releaseFormalAiSidecarForTask } from './formal-ai-isolation.lib.mjs';
 // The image references live in their own module so the Formal AI sidecar can
 // resolve the locally present Hive Mind image (which bakes `formal-ai`) without
 // importing this runner and creating a cycle. Re-exported here because callers
@@ -399,7 +399,7 @@ export async function executeWithIsolation(command, args, options = {}) {
     await releaseFormalAiSidecarForTask({ sidecar, sessionId, env: hostEnv, verbose });
     return failLaunch(formalAiRoutingError);
   }
-  const taskEnv = sidecar ? { ...hostEnv, HIVE_MIND_FORMAL_AI_BASE_URL: sidecar.baseUrl } : hostEnv;
+  const taskEnv = buildFormalAiTaskEnv({ sidecar, env: hostEnv });
   const effectiveOptions =
     backend === 'docker'
       ? {