@link-assistant/hive-mind 2.19.0 → 2.19.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,33 @@
1
1
  # @link-assistant/hive-mind
2
2
 
3
+ ## 2.19.2
4
+
5
+ ### Patch Changes
6
+
7
+ - 57b8258: Never merge hive-mind's own placeholder file, and restart when a pull request changes nothing.
8
+
9
+ `konard/audio-decomposer#1` was solved twice, both pull requests were auto-merged, and the complete diff of both was the `.gitkeep` hive-mind writes only so that an empty branch has something to open a pull request from. Two defects had to line up for that, and both are fixed here.
10
+
11
+ - **The placeholder is reverted before anything can merge it.** `cleanupClaudeFile()` ran after `startAutoRestartUntilMergeable()`, so with `--auto-merge` it was structurally guaranteed to be too late: on that pull request the revert commit is timestamped four seconds after the merge commit. It now runs before the watch loop and still after `verifyResults()`, which is the ordering issue #1516 actually asked for. As defense in depth, the watch loop reverts a placeholder that survived into its own diff — a crashed session, a resumed run — instead of merging it.
12
+ - **An appended placeholder is recognised as a placeholder.** The empty-pull-request detector matched an added `# .gitkeep file auto-generated at …` line, which the solver only writes when it _creates_ the file; when the file already exists it appends `# Updated: <timestamp>` instead, and that reads as an ordinary modification. The diff was counted as real work, so the pull request looked mergeable and the auto-restart from issue #2119 never fired. The measurement now reconstructs both sides of the file and compares them with hive-mind's own generated lines removed, so created, appended and re-appended placeholders are all caught — while a genuine edit to a `.gitkeep` or `CLAUDE.md` the repository owns still counts as work.
13
+
14
+ The leak was not a one-off: `.gitkeep` on the default branch of `link-foundation/rust-ai-driven-development-pipeline-template` had accumulated eight solver-generated lines from eight merged pull requests, one of which also carried real changes. Each miss makes the next one certain, because a surviving file forces the append path that the detector could not see. The full reconstruction is in `docs/case-studies/issue-2211`.
15
+
16
+ ## 2.19.1
17
+
18
+ ### Patch Changes
19
+
20
+ - 6cdf9e2: Keep the Formal AI release a verified update accepted, and record the backend that actually served each task rather than the local wrapper (issues #2207, #2208, delivered together for #2209).
21
+
22
+ Two defects sat on either side of the same handoff, and each one hid the other. The updater verified a new Formal AI image against the persisted memory volume, accepted it, and wrote it into `formal-ai-sidecar.json` — and then the next cold task threw it away, because `acquireFormalAiSidecar` resolved its image from the environment alone and re-selected the bootstrap release. Meanwhile the task's provenance reported `formal-ai --version` from the local wrapper, a binary that answers no requests, so the log could not have revealed the downgrade even when it happened.
23
+
24
+ - **The accepted image is what the next task boots.** `readAcceptedFormalAiImage` reads the acceptance back out of state and acquire passes it into image resolution. Precedence is an operator pin (exact and exclusive), then the accepted image (also exclusive), then the published bootstrap image only when nothing has been accepted yet, then the local Hive Mind image. The accepted candidates lead with the **immutable digest** and fall back to the tag only with `expectDigest` attached, so a tag that moved after acceptance is refused instead of booted — which is not hypothetical: `ghcr.io/link-assistant/formal-ai:latest` pointed at 0.346.0 when this was reported and points at 0.347.0 now. An accepted image that is no longer on the host is a refusal with the pin named in the message, never a silent downgrade onto memory that has already been migrated.
25
+ - **The serving backend is asked, not assumed.** `probeFormalAiBackend` queries `/health` on the configured endpoint before any client runs, within a bounded deadline, presenting the configured API key and returning immediately on 401/403 instead of retrying a credential that will still be wrong. `assertSupportedFormalAiBackend` gives each answer its own refusal — unreachable, unauthorized, non-JSON, JSON without a version, below `FORMAL_AI_MINIMUM_VERSION`, incompatible persisted memory — and never guesses a version.
26
+ - **The two versions have two names.** `runtime.formalAiVersion` is the release that answers the requests and is what the logs and provenance report; `runtime.formalAiWrapperVersion` is the local executable that started the server and wrote the config, keeping its own floor check and its own log line. `validateFormalAiToolConnection` is unchanged.
27
+ - **The halves are joined.** When Hive Mind manages the sidecar, the leased image, digest, source and verified version are published into the task environment, read back by the runtime, and required to agree with what `/health` reports; an endpoint serving something other than the accepted release is refused rather than recorded. The runtime cache is keyed by endpoint, not by the release behind it, so a cache hit re-probes and says so when the backend changed underneath a reused URL.
28
+
29
+ Three regression suites cover it: `tests/test-issue-2207-accepted-image-persistence.mjs` (candidate selection, update → stopped sidecar → acquire → release → acquire including a process restart against persisted state, a moved tag, a rollback, a pruned accepted image, an operator pin with a busy lease), `tests/test-issue-2208-serving-backend-provenance.mjs` (ten cases against real loopback HTTP backends), and `tests/test-issue-2209-verified-release-serving-provenance.mjs`, which runs the Docker simulator and a real serving process together so neither half can look correct while the pair fails. The same transition was then run against a real Docker daemon and the real published images; the evidence is in `docs/case-studies/issue-2209/`.
30
+
3
31
  ## 2.19.0
4
32
 
5
33
  ### Minor Changes
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@link-assistant/hive-mind",
3
- "version": "2.19.0",
3
+ "version": "2.19.2",
4
4
  "description": "AI-powered issue solver and hive mind for collaborative problem solving",
5
5
  "main": "src/hive.mjs",
6
6
  "type": "module",
@@ -35,15 +35,20 @@
35
35
  * - `HIVE_MIND_FORMAL_AI_IMAGE` is an **exact** operator pin: it is the only
36
36
  * candidate, and if it cannot be resolved the acquire fails. An operator who
37
37
  * names an image means that image.
38
- * - Otherwise the published image is preferred, and the local Hive Mind image is
39
- * the fallback. The fallback is only *used*; it is never pulled, because its
40
- * whole point is that it is already on the host.
38
+ * - Otherwise the image accepted by the last verified idle update wins, by its
39
+ * immutable digest (issue #2207). Before that rule the bootstrap version was
40
+ * re-selected on every cold start, so an accepted update was discarded by the
41
+ * next task and the sidecar returned to the release Hive Mind ships with.
42
+ * - Otherwise the published bootstrap image is preferred, and the local Hive
43
+ * Mind image is the fallback. The fallback is only *used*; it is never pulled,
44
+ * because its whole point is that it is already on the host.
41
45
  * - Nothing here downgrades a Formal AI task to another model. Issue #2146's
42
46
  * fail-closed rule still holds: when no candidate resolves, the task is
43
47
  * refused with an actionable message.
44
48
  *
45
49
  * @see https://github.com/link-assistant/hive-mind/issues/2154
46
50
  * @see https://github.com/link-assistant/hive-mind/issues/2146
51
+ * @see https://github.com/link-assistant/hive-mind/issues/2207
47
52
  */
48
53
 
49
54
  import { execFile } from 'node:child_process';
@@ -62,6 +67,7 @@ const DEFAULT_DOCKER_TIMEOUT_MS = 600_000;
62
67
  /** Where each candidate came from, so logs and errors can explain themselves. */
63
68
  export const FORMAL_AI_IMAGE_SOURCES = Object.freeze({
64
69
  PINNED: 'operator-pin',
70
+ ACCEPTED: 'accepted-update',
65
71
  PUBLISHED: 'published-image',
66
72
  HIVE_MIND: 'hive-mind-image',
67
73
  });
@@ -114,15 +120,82 @@ export const classifyDockerRegistryError = message => {
114
120
  */
115
121
  export const resolveFormalAiFallbackImage = (env = process.env) => String(env.HIVE_MIND_FORMAL_AI_FALLBACK_IMAGE || '').trim() || getDockerIsolationImage({ env });
116
122
 
123
+ /**
124
+ * The last image an idle update actually accepted, read out of the durable
125
+ * sidecar record (issue #2207).
126
+ *
127
+ * `state.image`/`state.imageDigest` are a *cache of what is running* — every
128
+ * acquire overwrites them with the container it found — so they cannot answer
129
+ * "which image did we verify?". `state.lastUpdate` can: it is written only by
130
+ * `updateFormalAiSidecarWhenIdle` after the pull, the side-effect-free memory
131
+ * preflight, the backed-up migration and the `/health` memory-compatibility
132
+ * check have all succeeded, and it is left untouched by a rollback.
133
+ *
134
+ * @param {object|null} state - A `formal-ai-sidecar.json` record.
135
+ * @returns {{image: string|null, digest: string|null, version: string|null, memorySchemaVersion: (number|string|null), updatedAt: string|null}|null}
136
+ */
137
+ export const readAcceptedFormalAiImage = state => {
138
+ const update = state?.lastUpdate;
139
+ if (!update || typeof update !== 'object') return null;
140
+ const image = String(update.image || '').trim() || null;
141
+ const digest = String(update.digest || '').trim() || null;
142
+ if (!image && !digest) return null;
143
+ return { image, digest, version: update.version ?? null, memorySchemaVersion: update.memorySchemaVersion ?? null, updatedAt: update.updatedAt ?? null };
144
+ };
145
+
146
+ /**
147
+ * Candidates that reproduce an accepted update, most trustworthy first.
148
+ *
149
+ * Two rules make this safe:
150
+ *
151
+ * - **The digest wins.** `docker image inspect --format {{.Id}}` returns the
152
+ * local content address, which Docker also accepts as a run reference. Using
153
+ * it means a tag that moved after acceptance (`:latest` is the update tag)
154
+ * cannot smuggle an unverified image past the memory verification that
155
+ * earned the acceptance.
156
+ * - **The tag is only a recovery path.** If the accepted image ID has been
157
+ * pruned, the recorded reference may be pulled again, but the result is
158
+ * accepted only when its digest still matches; otherwise the candidate is
159
+ * refused rather than booted.
160
+ *
161
+ * @param {object|null} accepted - Output of {@link readAcceptedFormalAiImage}.
162
+ * @returns {Array<object>}
163
+ */
164
+ export const resolveAcceptedFormalAiImageCandidates = (accepted = null) => {
165
+ if (!accepted) return [];
166
+ const candidates = [];
167
+ if (accepted.digest) candidates.push({ image: accepted.digest, reference: accepted.image || accepted.digest, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: false, accepted: true });
168
+ if (accepted.image && accepted.digest) candidates.push({ image: accepted.image, reference: accepted.image, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: true, accepted: true, expectDigest: accepted.digest });
169
+ // An acceptance recorded before digests were captured: the reference is all
170
+ // there is, so it may be used but never re-pulled behind a moving tag.
171
+ if (accepted.image && !accepted.digest) candidates.push({ image: accepted.image, reference: accepted.image, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: false, accepted: true });
172
+ return candidates;
173
+ };
174
+
117
175
  /**
118
176
  * Ordered list of images to try, most preferred first.
119
177
  *
178
+ * Precedence, and why:
179
+ *
180
+ * 1. `HIVE_MIND_FORMAL_AI_IMAGE` — an operator who names an image means that
181
+ * image; the pin is exact and exclusive.
182
+ * 2. The last accepted, verified update (issue #2207). Also exclusive: once an
183
+ * image has been verified against the migrated persisted memory, quietly
184
+ * booting the older bootstrap release against that memory is exactly the
185
+ * silent downgrade the fail-closed rule forbids.
186
+ * 3. The bootstrap release, then the local Hive Mind image — the cold-start
187
+ * path for a host that has never completed an update.
188
+ *
120
189
  * @param {object} [env]
190
+ * @param {object} [options]
191
+ * @param {object|null} [options.accepted] - Output of {@link readAcceptedFormalAiImage}.
121
192
  * @returns {Array<{image: string, source: string, pullable: boolean}>}
122
193
  */
123
- export const resolveFormalAiSidecarImageCandidates = (env = process.env) => {
194
+ export const resolveFormalAiSidecarImageCandidates = (env = process.env, { accepted = null } = {}) => {
124
195
  const pinned = String(env.HIVE_MIND_FORMAL_AI_IMAGE || '').trim();
125
196
  if (pinned) return [{ image: pinned, source: FORMAL_AI_IMAGE_SOURCES.PINNED, pullable: true }];
197
+ const acceptedCandidates = resolveAcceptedFormalAiImageCandidates(accepted);
198
+ if (acceptedCandidates.length > 0) return acceptedCandidates;
126
199
  const candidates = [{ image: `${FORMAL_AI_IMAGE_REPOSITORY}:${FORMAL_AI_BOOTSTRAP_VERSION}`, source: FORMAL_AI_IMAGE_SOURCES.PUBLISHED, pullable: true }];
127
200
  const fallback = resolveFormalAiFallbackImage(env);
128
201
  // Never pulled: the fallback's value is that it is already on the host. A
@@ -132,7 +205,7 @@ export const resolveFormalAiSidecarImageCandidates = (env = process.env) => {
132
205
  };
133
206
 
134
207
  /** The preferred image, kept for callers and logs that only need the headline reference. */
135
- export const resolveFormalAiSidecarImage = (env = process.env) => resolveFormalAiSidecarImageCandidates(env)[0].image;
208
+ export const resolveFormalAiSidecarImage = (env = process.env, options = {}) => resolveFormalAiSidecarImageCandidates(env, options)[0].image;
136
209
 
137
210
  const inspectLocalImage = async (image, { run, timeoutMs }) => {
138
211
  try {
@@ -148,7 +221,8 @@ const inspectLocalImage = async (image, { run, timeoutMs }) => {
148
221
  * bot log knows the cause and the fix without opening a shell.
149
222
  */
150
223
  const describeFailure = attempts => {
151
- const lines = ['No Formal AI image could be resolved, so the sidecar was not started.'];
224
+ const accepted = attempts.some(attempt => attempt.source === FORMAL_AI_IMAGE_SOURCES.ACCEPTED);
225
+ const lines = [accepted ? 'The Formal AI image accepted by the last verified update could not be resolved, so the sidecar was not started.' : 'No Formal AI image could be resolved, so the sidecar was not started.'];
152
226
  for (const attempt of attempts) {
153
227
  if (attempt.source === FORMAL_AI_IMAGE_SOURCES.HIVE_MIND && attempt.kind === 'absent') {
154
228
  lines.push(`• ${attempt.image} (local Hive Mind image, fallback): not present on this host — Docker-isolated tasks would have to pull it too.`);
@@ -157,6 +231,13 @@ const describeFailure = attempts => {
157
231
  lines.push(`• ${attempt.image} (${attempt.source}): ${attempt.reason}${attempt.error ? ` — ${attempt.error}` : ''}`);
158
232
  for (const step of attempt.remediation ?? []) lines.push(` → ${step}`);
159
233
  }
234
+ if (accepted) {
235
+ // Falling back to the bootstrap release here would run an *older* binary
236
+ // against memory that the accepted release already migrated (issue #2207).
237
+ lines.push('Hive Mind will not fall back to an older Formal AI release against memory an accepted update already migrated.');
238
+ lines.push(' → restore the accepted image on this host (`docker pull` it, or copy it back), or');
239
+ lines.push(' → set HIVE_MIND_FORMAL_AI_IMAGE to the image you want this host to run, which overrides the accepted update.');
240
+ }
160
241
  lines.push('Formal AI tasks fail closed by design (issue #2146): Hive Mind will not silently run them on another model.');
161
242
  return lines.join('\n');
162
243
  };
@@ -172,17 +253,27 @@ const describeFailure = attempts => {
172
253
  * @param {Function|null} [params.log]
173
254
  * @param {boolean} [params.verbose]
174
255
  * @param {boolean} [params.pull] - Set false to accept only images already on the host.
175
- * @returns {Promise<{image: string, source: string, digest: string|null, pulled: boolean, attempts: object[]}>}
256
+ * @param {object|null} [params.accepted] - Output of {@link readAcceptedFormalAiImage}, used to build the default candidates.
257
+ * @returns {Promise<{image: string, reference: string, source: string, digest: string|null, pulled: boolean, attempts: object[]}>}
176
258
  * @throws {Error} When no candidate resolves; the message names every attempt and its fix.
177
259
  */
178
- export const ensureFormalAiSidecarImage = async ({ env = process.env, run = execFileAsync, timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS, log = null, verbose = false, pull = true, candidates = resolveFormalAiSidecarImageCandidates(env) } = {}) => {
260
+ export const ensureFormalAiSidecarImage = async ({ env = process.env, run = execFileAsync, timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS, log = null, verbose = false, pull = true, accepted = null, candidates = resolveFormalAiSidecarImageCandidates(env, { accepted }) } = {}) => {
179
261
  const attempts = [];
262
+ const reference = candidate => candidate.reference || candidate.image;
263
+ // A candidate that names an expected digest is a *recovery* path for an
264
+ // accepted update whose image ID was pruned: the reference may be re-fetched,
265
+ // but only the recorded content address is allowed to run (issue #2207).
266
+ const mismatch = (candidate, digest) => ({ ...candidate, kind: 'digest-mismatch', reason: `resolved to ${digest ?? 'no digest'} instead of the accepted ${candidate.expectDigest}`, error: null, remediation: ['the tag moved after the update was accepted; restore the accepted image or repin with HIVE_MIND_FORMAL_AI_IMAGE'] });
180
267
 
181
268
  for (const candidate of candidates) {
182
269
  const localDigest = await inspectLocalImage(candidate.image, { run, timeoutMs });
270
+ if (localDigest && candidate.expectDigest && localDigest !== candidate.expectDigest) {
271
+ attempts.push(mismatch(candidate, localDigest));
272
+ continue;
273
+ }
183
274
  if (localDigest) {
184
275
  if (verbose && log) await log(`[VERBOSE] formal-ai-image: using '${candidate.image}' (${candidate.source}) already present locally, digest=${localDigest}`);
185
- return { image: candidate.image, source: candidate.source, digest: localDigest, pulled: false, attempts };
276
+ return { image: candidate.image, reference: reference(candidate), source: candidate.source, digest: localDigest, pulled: false, attempts };
186
277
  }
187
278
 
188
279
  if (!candidate.pullable || !pull) {
@@ -202,8 +293,13 @@ export const ensureFormalAiSidecarImage = async ({ env = process.env, run = exec
202
293
  }
203
294
 
204
295
  const digest = await inspectLocalImage(candidate.image, { run, timeoutMs });
296
+ if (candidate.expectDigest && digest !== candidate.expectDigest) {
297
+ attempts.push(mismatch(candidate, digest));
298
+ if (log) await log(`⚠️ ${candidate.image} no longer resolves to the accepted digest ${candidate.expectDigest}; refusing to boot it.`);
299
+ continue;
300
+ }
205
301
  if (verbose && log) await log(`[VERBOSE] formal-ai-image: pulled '${candidate.image}' (${candidate.source}), digest=${digest ?? 'unknown'}`);
206
- return { image: candidate.image, source: candidate.source, digest, pulled: true, attempts };
302
+ return { image: candidate.image, reference: reference(candidate), source: candidate.source, digest, pulled: true, attempts };
207
303
  }
208
304
 
209
305
  const error = new Error(describeFailure(attempts));
@@ -216,6 +312,8 @@ export default {
216
312
  FORMAL_AI_IMAGE_SOURCES,
217
313
  classifyDockerRegistryError,
218
314
  ensureFormalAiSidecarImage,
315
+ readAcceptedFormalAiImage,
316
+ resolveAcceptedFormalAiImageCandidates,
219
317
  resolveFormalAiFallbackImage,
220
318
  resolveFormalAiSidecarImage,
221
319
  resolveFormalAiSidecarImageCandidates,
@@ -16,6 +16,7 @@
16
16
  * @see https://github.com/link-assistant/hive-mind/issues/2146
17
17
  */
18
18
 
19
+ import { buildFormalAiSidecarProvenanceEnv } from './formal-ai-runtime.lib.mjs';
19
20
  import { acquireFormalAiSidecar, attachTaskToFormalAiNetwork, isFormalAiSidecarEnabled, isFormalAiTask, releaseFormalAiSidecar } from './formal-ai-sidecar.lib.mjs';
20
21
 
21
22
  const logToConsole = message => console.log(message);
@@ -43,6 +44,19 @@ export const acquireFormalAiSidecarForTask = async ({ backend, args = [], model
43
44
  }
44
45
  };
45
46
 
47
+ /**
48
+ * The environment a Formal AI task is launched with.
49
+ *
50
+ * Besides the endpoint, this carries the identity of the image the lease pinned
51
+ * (issue #2208). The runtime inside the container still asks the endpoint which
52
+ * release is answering; these values let it refuse when the answer is not the
53
+ * accepted image Hive Mind leased, and let the task record the digest rather
54
+ * than a mutable tag.
55
+ *
56
+ * @returns {object} The task environment (`env` itself when Formal AI is not involved).
57
+ */
58
+ export const buildFormalAiTaskEnv = ({ sidecar, env = process.env } = {}) => (sidecar ? { ...env, HIVE_MIND_FORMAL_AI_BASE_URL: sidecar.baseUrl, ...buildFormalAiSidecarProvenanceEnv(sidecar) } : env);
59
+
46
60
  /**
47
61
  * Add the internal Formal AI network to the freshly-created task container.
48
62
  *
@@ -66,4 +80,4 @@ export const releaseFormalAiSidecarForTask = async ({ sidecar, sessionId, env =
66
80
  }
67
81
  };
68
82
 
69
- export default { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, releaseFormalAiSidecarForTask };
83
+ export default { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, buildFormalAiTaskEnv, releaseFormalAiSidecarForTask };
@@ -53,7 +53,7 @@ import { homedir } from 'node:os';
53
53
  import { dirname, join } from 'node:path';
54
54
  import { promisify } from 'node:util';
55
55
 
56
- import { assertSupportedFormalAiVersion, FORMAL_AI_MINIMUM_VERSION, readFormalAiBinaryVersion } from './formal-ai-version.lib.mjs';
56
+ import { assertSupportedFormalAiVersion, FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast, readFormalAiBinaryVersion } from './formal-ai-version.lib.mjs';
57
57
 
58
58
  const execFileAsync = promisify(execFile);
59
59
 
@@ -65,6 +65,26 @@ export const FORMAL_AI_MODEL_NAME = 'formal-ai';
65
65
  export const FORMAL_AI_DEFAULT_HOST = '127.0.0.1';
66
66
  export const FORMAL_AI_SERVER_READY_TIMEOUT_MS = 90_000;
67
67
 
68
+ /** Where a Formal AI server reports its own version and memory compatibility. */
69
+ export const FORMAL_AI_HEALTH_PATH = '/health';
70
+
71
+ /**
72
+ * Bounded budget for the backend probe (issue #2208).
73
+ *
74
+ * Deliberately short: this runs immediately before the client executes, so a
75
+ * hanging endpoint must surface as a refusal rather than as a task that appears
76
+ * to be thinking. The server-start path has its own, much longer budget.
77
+ */
78
+ export const FORMAL_AI_BACKEND_PROBE_TIMEOUT_MS = 15_000;
79
+
80
+ /** Environment carrying the Hive-Mind-managed sidecar's identity into the task container (issue #2207/#2208). */
81
+ export const FORMAL_AI_SIDECAR_PROVENANCE_ENV = Object.freeze({
82
+ image: 'HIVE_MIND_FORMAL_AI_SIDECAR_IMAGE',
83
+ imageDigest: 'HIVE_MIND_FORMAL_AI_SIDECAR_DIGEST',
84
+ version: 'HIVE_MIND_FORMAL_AI_SIDECAR_VERSION',
85
+ imageSource: 'HIVE_MIND_FORMAL_AI_SIDECAR_SOURCE',
86
+ });
87
+
68
88
  export const resolveFormalAiApiKey = (env = process.env) => env.FORMAL_AI_API_KEY?.trim() || FORMAL_AI_DEFAULT_API_KEY;
69
89
 
70
90
  /**
@@ -164,6 +184,109 @@ export const waitForFormalAiServerReady = async ({ baseUrl, probePath = '/api/op
164
184
  return { ready: false, error: lastError || 'timed out' };
165
185
  };
166
186
 
187
+ /**
188
+ * Ask the endpoint that will actually answer the model requests who it is
189
+ * (issue #2208).
190
+ *
191
+ * Until this existed, `prepareFormalAiRuntime` read `formal-ai --version` from
192
+ * the *local* executable and logged the answer as "the Formal AI version" even
193
+ * when `HIVE_MIND_FORMAL_AI_BASE_URL` pointed at a container running a
194
+ * completely different release. Every provenance record therefore named a binary
195
+ * that never saw the request.
196
+ *
197
+ * Properties that matter:
198
+ *
199
+ * - **Bounded.** The whole probe shares one deadline, and each request carries
200
+ * its own abort signal, so an endpoint that accepts the connection and then
201
+ * goes quiet cannot stall the task.
202
+ * - **Authenticated.** The configured key is presented the way the served API
203
+ * expects it. A 401/403 is reported as an authentication problem and is *not*
204
+ * retried — retrying a rejected credential only delays the failure.
205
+ * - **Honest about malformed answers.** A body that is not JSON, or that
206
+ * carries no version, is a distinct outcome from "unreachable"; guessing a
207
+ * version here would recreate the defect this function exists to fix.
208
+ *
209
+ * @returns {Promise<{ok: boolean, kind: string, status: number|null, version: string|null, memory: object|null, health: object|null, error: string|null}>}
210
+ */
211
+ export const probeFormalAiBackend = async ({ baseUrl, apiKey = null, path = FORMAL_AI_HEALTH_PATH, timeoutMs = FORMAL_AI_BACKEND_PROBE_TIMEOUT_MS, fetchImpl = globalThis.fetch, intervalMs = 500, now = () => Date.now(), sleepImpl = delay } = {}) => {
212
+ if (!baseUrl) return { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: 'no base URL to probe' };
213
+ const url = `${String(baseUrl).replace(/\/+$/, '')}${path}`;
214
+ const headers = apiKey ? { Authorization: `Bearer ${apiKey}`, 'X-Api-Key': apiKey } : {};
215
+ const deadline = now() + timeoutMs;
216
+ let last = { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: 'not probed' };
217
+
218
+ for (;;) {
219
+ const remaining = deadline - now();
220
+ if (remaining <= 0) break;
221
+ try {
222
+ const response = await fetchImpl(url, { headers, signal: AbortSignal.timeout(Math.min(remaining, timeoutMs)) });
223
+ const status = response?.status ?? null;
224
+ if (status === 401 || status === 403) {
225
+ // Not retried: the credential is wrong, and it will still be wrong in 500ms.
226
+ return { ok: false, kind: 'unauthorized', status, version: null, memory: null, health: null, error: `the Formal AI endpoint rejected the configured credentials (HTTP ${status})` };
227
+ }
228
+ if (!response?.ok) {
229
+ last = { ok: false, kind: 'http-error', status, version: null, memory: null, health: null, error: `HTTP ${status}` };
230
+ } else {
231
+ const text = await response.text();
232
+ let health;
233
+ try {
234
+ health = JSON.parse(text);
235
+ } catch {
236
+ return { ok: false, kind: 'malformed', status, version: null, memory: null, health: null, error: `${path} did not return JSON: ${text.slice(0, 200)}` };
237
+ }
238
+ const version = typeof health?.version === 'string' ? health.version.trim() || null : null;
239
+ if (!version) return { ok: false, kind: 'no-version', status, version: null, memory: health?.memory ?? null, health, error: `${path} answered without a version field` };
240
+ return { ok: true, kind: 'ok', status, version, memory: health?.memory ?? null, health, error: null };
241
+ }
242
+ } catch (error) {
243
+ last = { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: error?.message || String(error) };
244
+ }
245
+ if (deadline - now() <= intervalMs) break;
246
+ await sleepImpl(intervalMs);
247
+ }
248
+ return last;
249
+ };
250
+
251
+ /**
252
+ * Turn a probe result into either the accepted backend description or an
253
+ * actionable refusal.
254
+ *
255
+ * Fail-closed (issue #2146) applies here too: a Formal AI task that cannot prove
256
+ * which release is serving it must stop, not proceed and record a guess.
257
+ *
258
+ * @param {object} probe - Output of {@link probeFormalAiBackend}.
259
+ * @param {object} context
260
+ * @param {string} context.baseUrl
261
+ * @param {string} [context.minimumVersion]
262
+ * @param {string|null} [context.expectedVersion] - Version the leased sidecar image was verified at.
263
+ * @returns {{version: string, memory: object|null}}
264
+ */
265
+ export const assertSupportedFormalAiBackend = (probe, { baseUrl, minimumVersion = FORMAL_AI_MINIMUM_VERSION, expectedVersion = null } = {}) => {
266
+ const where = `the Formal AI endpoint ${baseUrl}`;
267
+ if (!probe?.ok) {
268
+ const detail = probe?.error ? `: ${probe.error}` : '';
269
+ if (probe?.kind === 'unauthorized') throw new Error(`${where} refused the configured credentials${detail}. Set FORMAL_AI_API_KEY to a key the server accepts.`);
270
+ if (probe?.kind === 'malformed') throw new Error(`${where} answered ${FORMAL_AI_HEALTH_PATH} with something other than JSON${detail}. Hive Mind will not guess which Formal AI release is serving this task.`);
271
+ if (probe?.kind === 'no-version') throw new Error(`${where} answered ${FORMAL_AI_HEALTH_PATH} without a version${detail}. Hive Mind requires a serving backend that reports its version (Formal AI >= ${minimumVersion}).`);
272
+ if (probe?.kind === 'http-error') throw new Error(`${where} did not serve ${FORMAL_AI_HEALTH_PATH}${detail}. Check that HIVE_MIND_FORMAL_AI_BASE_URL points at a Formal AI server >= ${minimumVersion}.`);
273
+ throw new Error(`${where} could not be reached${detail}. Check HIVE_MIND_FORMAL_AI_BASE_URL and that the Formal AI server is running.`);
274
+ }
275
+ if (!isFormalAiVersionAtLeast(probe.version, minimumVersion)) {
276
+ throw new Error(`${where} serves Formal AI ${probe.version}, but Hive Mind requires >= ${minimumVersion}. Upgrade the server; the local wrapper's version does not change what answers the requests.`);
277
+ }
278
+ if (probe.memory?.compatible === false) {
279
+ throw new Error(`${where} reports incompatible persisted memory (migration_state=${probe.memory?.migration_state ?? 'unknown'}). Refusing to run a task against memory the serving release cannot read.`);
280
+ }
281
+ if (expectedVersion && expectedVersion !== probe.version) {
282
+ // A lease pins the sidecar's image for the whole task, so the endpoint
283
+ // answering with a different release means it is not the container Hive
284
+ // Mind verified and leased.
285
+ throw new Error(`${where} serves Formal AI ${probe.version}, but the leased Hive Mind sidecar image was verified as ${expectedVersion}. Refusing to record provenance for a backend that is not the accepted release.`);
286
+ }
287
+ return { version: probe.version, memory: probe.memory ?? null };
288
+ };
289
+
167
290
  /** Read the machine-readable client registry (`formal-ai clients --format json`). */
168
291
  export const loadFormalAiClientRegistry = async ({ formalAiPath = 'formal-ai', run = execFileAsync, env = process.env, timeoutMs = 30_000 } = {}) => {
169
292
  const result = await run(formalAiPath, ['clients', '--format', 'json'], { encoding: 'utf8', env: { ...process.env, ...env }, timeout: timeoutMs, maxBuffer: 32 * 1024 * 1024 });
@@ -321,6 +444,69 @@ export const startFormalAiServer = async ({ cwd, host = FORMAL_AI_DEFAULT_HOST,
321
444
  };
322
445
  };
323
446
 
447
+ /**
448
+ * The provenance Hive Mind's sidecar lifecycle publishes for the container it
449
+ * leased to this task (issues #2207, #2208).
450
+ *
451
+ * The lease pins one verified image for the whole task, so these values say
452
+ * which release *should* be answering. They are a cross-check, never a
453
+ * substitute for asking the endpoint itself.
454
+ */
455
+ export const readFormalAiSidecarProvenance = (env = process.env) => {
456
+ const read = name => String(env[name] || '').trim() || null;
457
+ const image = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.image);
458
+ const imageDigest = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageDigest);
459
+ const version = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.version);
460
+ const imageSource = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageSource);
461
+ if (!image && !imageDigest && !version) return null;
462
+ return { image, imageDigest, version, imageSource };
463
+ };
464
+
465
+ /**
466
+ * Publish the leased sidecar's identity into a task's environment so the runtime
467
+ * inside the container can cross-check the endpoint it is pointed at.
468
+ *
469
+ * @param {object|null} sidecar - An `acquireFormalAiSidecar` result.
470
+ * @returns {object} Environment entries (empty when nothing is known).
471
+ */
472
+ export const buildFormalAiSidecarProvenanceEnv = (sidecar = null) => {
473
+ if (!sidecar) return {};
474
+ const entries = {
475
+ [FORMAL_AI_SIDECAR_PROVENANCE_ENV.image]: sidecar.imageReference || sidecar.image || null,
476
+ [FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageDigest]: sidecar.imageDigest || null,
477
+ [FORMAL_AI_SIDECAR_PROVENANCE_ENV.version]: sidecar.servingVersion || sidecar.health?.version || null,
478
+ [FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageSource]: sidecar.imageSource || null,
479
+ };
480
+ return Object.fromEntries(Object.entries(entries).filter(([, value]) => value));
481
+ };
482
+
483
+ /** One-line description of the backend for logs and session provenance. */
484
+ const describeFormalAiBackend = backend => [`${backend.version} at ${backend.baseUrl}`, backend.image ? `image ${backend.image}` : null, backend.imageDigest ? `digest ${backend.imageDigest}` : null].filter(Boolean).join(', ');
485
+
486
+ /**
487
+ * Query the endpoint that will serve this task and build its provenance record.
488
+ *
489
+ * @returns {Promise<object>} `{ baseUrl, version, memory, image, imageDigest, imageSource, leased, startedLocally, probedAt }`
490
+ */
491
+ const resolveFormalAiBackend = async ({ baseUrl, apiKey, env, deps, startedLocally }) => {
492
+ const sidecar = readFormalAiSidecarProvenance(env);
493
+ const probe = await (deps.probeBackendImpl || probeFormalAiBackend)({ baseUrl, apiKey, env });
494
+ const { version, memory } = assertSupportedFormalAiBackend(probe, { baseUrl, expectedVersion: sidecar?.version ?? null });
495
+ return {
496
+ baseUrl,
497
+ version,
498
+ memory,
499
+ image: sidecar?.image ?? null,
500
+ imageDigest: sidecar?.imageDigest ?? null,
501
+ imageSource: sidecar?.imageSource ?? null,
502
+ /** True when Hive Mind leased this endpoint from its own sidecar. */
503
+ leased: !!sidecar,
504
+ /** True when this process started the server it is now talking to. */
505
+ startedLocally,
506
+ probedAt: new Date().toISOString(),
507
+ };
508
+ };
509
+
324
510
  const runtimeCache = new Map();
325
511
  let exitHookInstalled = false;
326
512
 
@@ -357,16 +543,32 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
357
543
  const resolvedFormalAiPath = formalAiPath || env.HIVE_MIND_FORMAL_AI_PATH?.trim() || 'formal-ai';
358
544
  const cacheKey = `${tool}::${workdir}::${env.HIVE_MIND_FORMAL_AI_BASE_URL || ''}`;
359
545
  const cached = runtimeCache.get(cacheKey);
360
- if (cached) return cached.runtime;
546
+ if (cached) {
547
+ // Issue #2208: the cache key is the endpoint, not the release behind it. An
548
+ // external base URL can be re-pointed at a different container between
549
+ // tasks, so the cached provenance is re-checked instead of replayed.
550
+ const backend = await resolveFormalAiBackend({ baseUrl: cached.runtime.baseUrl, apiKey: resolveFormalAiApiKey(env), env, deps, startedLocally: cached.runtime.serverStarted });
551
+ if (backend.version !== cached.runtime.backend?.version || backend.imageDigest !== cached.runtime.backend?.imageDigest) {
552
+ await log(`🧠 Formal AI: serving backend changed to ${describeFormalAiBackend(backend)}`);
553
+ }
554
+ cached.runtime.backend = backend;
555
+ cached.runtime.formalAiVersion = backend.version;
556
+ return cached.runtime;
557
+ }
361
558
 
362
559
  installExitHook();
363
560
 
364
561
  // Issue #2146: `--no-tool-check` skipped the only version probe, allowing an
365
562
  // old Formal AI build to return the same unexecuted plan through all five
366
563
  // Claude/Codex restarts. Runtime safety cannot depend on preflight options.
367
- const formalAiVersion = await (deps.readVersionImpl || readFormalAiBinaryVersion)({ formalAiPath: resolvedFormalAiPath, env });
368
- assertSupportedFormalAiVersion(formalAiVersion);
369
- await log(`🧠 Formal AI: version ${formalAiVersion} (minimum ${FORMAL_AI_MINIMUM_VERSION})`);
564
+ //
565
+ // This is the *local wrapper*: the executable that starts the server and
566
+ // writes the client configuration. Issue #2208: it is not necessarily the
567
+ // release that answers the model requests, so it keeps its own name and its
568
+ // own compatibility check, and it is never reported as the serving version.
569
+ const formalAiWrapperVersion = await (deps.readVersionImpl || readFormalAiBinaryVersion)({ formalAiPath: resolvedFormalAiPath, env });
570
+ assertSupportedFormalAiVersion(formalAiWrapperVersion);
571
+ await log(`🧠 Formal AI: local wrapper version ${formalAiWrapperVersion} (minimum ${FORMAL_AI_MINIMUM_VERSION})`);
370
572
 
371
573
  const apiKey = resolveFormalAiApiKey(env);
372
574
  const externalBaseUrl = env.HIVE_MIND_FORMAL_AI_BASE_URL?.trim() || null;
@@ -386,6 +588,13 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
386
588
  await log(`🧠 Formal AI: using the configured server ${baseUrl}`, { verbose: true });
387
589
  }
388
590
 
591
+ // Before any client configuration is written, ask the endpoint who it is.
592
+ const backend = await resolveFormalAiBackend({ baseUrl, apiKey, env, deps, startedLocally: !!server });
593
+ await log(`🧠 Formal AI: serving backend ${describeFormalAiBackend(backend)}`);
594
+ if (backend.version !== formalAiWrapperVersion) {
595
+ await log(`🧠 Formal AI: local wrapper ${formalAiWrapperVersion} differs from the serving backend ${backend.version}; provenance records the backend`, { verbose: true });
596
+ }
597
+
389
598
  const clients = await (deps.loadRegistryImpl || loadFormalAiClientRegistry)({ formalAiPath: resolvedFormalAiPath, env });
390
599
  const client = findFormalAiClient(clients, tool);
391
600
  if (!client) throw new Error(`Formal AI does not list a client configuration for "${tool}"`);
@@ -429,7 +638,11 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
429
638
  client,
430
639
  notes,
431
640
  serverStarted: !!server,
432
- formalAiVersion,
641
+ /** The release that actually answers this task's model requests. */
642
+ formalAiVersion: backend.version,
643
+ /** The local executable that started the server and wrote the config. */
644
+ formalAiWrapperVersion,
645
+ backend,
433
646
  stop: async () => {
434
647
  runtimeCache.delete(cacheKey);
435
648
  await server?.stop?.();
@@ -46,7 +46,7 @@ import fs from 'node:fs';
46
46
  import { promisify } from 'node:util';
47
47
 
48
48
  import { FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast } from './formal-ai-version.lib.mjs';
49
- import { ensureFormalAiSidecarImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
49
+ import { ensureFormalAiSidecarImage, readAcceptedFormalAiImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
50
50
  import { isFormalAiModel } from './formal-ai-model.lib.mjs';
51
51
  import { getModelFromArgs } from './model-args.lib.mjs';
52
52
  import { withStateLock } from './state-lock.lib.mjs';
@@ -90,7 +90,7 @@ export const FORMAL_AI_MEMORY_PATH = `${FORMAL_AI_MEMORY_MOUNT}/memory.lino`;
90
90
  * issue #2154 taught us that "which image" and "is it actually pullable" are the
91
91
  * same question.
92
92
  */
93
- export { FORMAL_AI_IMAGE_REPOSITORY, resolveFormalAiSidecarImage, resolveFormalAiSidecarImageCandidates } from './formal-ai-image.lib.mjs';
93
+ export { FORMAL_AI_IMAGE_REPOSITORY, readAcceptedFormalAiImage, resolveAcceptedFormalAiImageCandidates, resolveFormalAiSidecarImage, resolveFormalAiSidecarImageCandidates } from './formal-ai-image.lib.mjs';
94
94
 
95
95
  /** Applied to the sidecar, its network and its volume so reconciliation can find them. */
96
96
  export const FORMAL_AI_SIDECAR_LABEL = 'com.link-assistant.hive-mind.formal-ai';
@@ -102,7 +102,13 @@ const SIDECAR_LOCK_NAME = 'formal-ai-sidecar';
102
102
  const DEFAULT_HEALTH_ATTEMPTS = 60;
103
103
  const DEFAULT_HEALTH_DELAY_MS = 1000;
104
104
 
105
- const EMPTY_STATE = Object.freeze({ version: 1, image: null, imageDigest: null, startedAt: null, leases: [], lastUpdate: null });
105
+ /**
106
+ * `lastUpdate` is the accepted-release record (issue #2207) and `serving` is the
107
+ * provenance of the container that last answered a lease (issue #2208). Neither
108
+ * is derivable from `image`/`imageDigest`, which are only a cache of whatever is
109
+ * running right now.
110
+ */
111
+ const EMPTY_STATE = Object.freeze({ version: 1, image: null, imageReference: null, imageDigest: null, startedAt: null, leases: [], lastUpdate: null, serving: null });
106
112
 
107
113
  /**
108
114
  * True when a task will be driven by Formal AI.
@@ -350,14 +356,21 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
350
356
  // #2154 the reference went straight into `docker run`, so a registry that
351
357
  // refused the pull surfaced as an unreadable `Command failed: docker run …`
352
358
  // dump and the task died even though a usable image sat on the host.
353
- const resolved = container.exists && container.image ? { image: container.image, source: 'running-sidecar', pulled: false } : await ensureFormalAiSidecarImage({ env, run, timeoutMs: imageTimeoutMs, log, verbose });
359
+ //
360
+ // `accepted` is what issue #2207 was missing: without it the candidate
361
+ // list was rebuilt from the bootstrap pin on every cold start, so an
362
+ // update that had already been pulled, migrated and verified was silently
363
+ // discarded by the very next task.
364
+ const accepted = readAcceptedFormalAiImage(state);
365
+ const resolved = container.exists && container.image ? { image: container.image, reference: container.image, source: 'running-sidecar', pulled: false } : await ensureFormalAiSidecarImage({ env, accepted, run, timeoutMs: imageTimeoutMs, log, verbose });
354
366
  const image = resolved.image;
367
+ const imageReference = resolved.reference || image;
355
368
 
356
369
  await ensureFormalAiNetwork({ run, timeoutMs, log, verbose });
357
370
  await ensureFormalAiMemoryVolume({ image, run, timeoutMs, log, verbose });
358
371
 
359
372
  if (!container.exists) {
360
- if (log) await log(`🧠 Starting the Formal AI sidecar (${image}, ${resolved.source}) on the internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
373
+ if (log) await log(`🧠 Starting the Formal AI sidecar (${imageReference}${imageReference === image ? '' : ` @ ${image}`}, ${resolved.source}) on the internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
361
374
  await dockerText(run, buildFormalAiSidecarRunArgs({ image, env }), { timeoutMs });
362
375
  container = await inspectDockerContainer(FORMAL_AI_SIDECAR_CONTAINER_NAME, { run, timeoutMs });
363
376
  }
@@ -375,15 +388,27 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
375
388
  // and then fail on the agent-mode API the tasks depend on.
376
389
  const reportedVersion = health.health?.version ?? null;
377
390
  if (reportedVersion && !isFormalAiVersionAtLeast(reportedVersion, FORMAL_AI_MINIMUM_VERSION)) {
378
- throw new Error(`Formal AI sidecar image ${image} (${resolved.source}) runs formal-ai ${reportedVersion}, but Hive Mind requires >= ${FORMAL_AI_MINIMUM_VERSION}. Rebuild or repin the image (HIVE_MIND_FORMAL_AI_IMAGE) before running Formal AI tasks.`);
391
+ throw new Error(`Formal AI sidecar image ${imageReference} (${resolved.source}) runs formal-ai ${reportedVersion}, but Hive Mind requires >= ${FORMAL_AI_MINIMUM_VERSION}. Rebuild or repin the image (HIVE_MIND_FORMAL_AI_IMAGE) before running Formal AI tasks.`);
379
392
  }
380
393
 
381
394
  const address = await readFormalAiSidecarAddress({ run, timeoutMs });
382
395
  const acquiredAt = now().toISOString();
383
396
  const nextLeases = [...leases.filter(lease => lease.sessionId !== sessionId), { sessionId, tool, model, acquiredAt }];
384
- writeFormalAiSidecarState({ ...state, image: container.image || image, imageDigest: container.imageDigest, startedAt: state.startedAt || acquiredAt, leases: nextLeases }, { env, fsImpl });
397
+ // The provenance of what is *actually serving* this lease, so a task's
398
+ // evidence can name the release that answered its requests rather than
399
+ // whatever binary happens to sit next to the wrapper (issue #2208).
400
+ const serving = {
401
+ image: imageReference,
402
+ imageDigest: container.imageDigest ?? resolved.digest ?? null,
403
+ imageSource: resolved.source,
404
+ version: reportedVersion,
405
+ memorySchemaVersion: health.health?.memory?.schema_version ?? null,
406
+ acceptedAt: accepted?.updatedAt ?? null,
407
+ observedAt: acquiredAt,
408
+ };
409
+ writeFormalAiSidecarState({ ...state, image: container.image || image, imageReference, imageDigest: container.imageDigest, startedAt: state.startedAt || acquiredAt, leases: nextLeases, serving }, { env, fsImpl });
385
410
 
386
- if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: lease '${sessionId}' acquired (${nextLeases.length} active), image=${container.image || image} (${resolved.source}), digest=${container.imageDigest ?? 'unknown'}, address=${address ?? 'unknown'}, formal-ai=${health.health?.version ?? 'unknown'}, memory schema=${health.health?.memory?.schema_version ?? 'unknown'}`);
411
+ if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: lease '${sessionId}' acquired (${nextLeases.length} active), image=${imageReference} (${resolved.source}), digest=${container.imageDigest ?? 'unknown'}, address=${address ?? 'unknown'}, formal-ai=${reportedVersion ?? 'unknown'}, memory schema=${serving.memorySchemaVersion ?? 'unknown'}`);
387
412
 
388
413
  return {
389
414
  address,
@@ -394,8 +419,14 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
394
419
  containerName: FORMAL_AI_SIDECAR_CONTAINER_NAME,
395
420
  memoryVolume: FORMAL_AI_MEMORY_VOLUME_NAME,
396
421
  image: container.image || image,
422
+ imageReference,
397
423
  imageSource: resolved.source,
398
- imageDigest: container.imageDigest,
424
+ imageDigest: serving.imageDigest,
425
+ acceptedImage: accepted?.image ?? null,
426
+ acceptedDigest: accepted?.digest ?? null,
427
+ acceptedVersion: accepted?.version ?? null,
428
+ servingVersion: reportedVersion,
429
+ memorySchemaVersion: serving.memorySchemaVersion,
399
430
  health: health.health,
400
431
  leaseCount: nextLeases.length,
401
432
  };
@@ -453,6 +484,7 @@ export default {
453
484
  inspectDockerContainer,
454
485
  isFormalAiSidecarEnabled,
455
486
  isFormalAiTask,
487
+ readAcceptedFormalAiImage,
456
488
  readDockerImageDigest,
457
489
  readFormalAiSidecarAddress,
458
490
  readFormalAiSidecarState,
@@ -92,6 +92,11 @@ export const resolveFormalAiToolExecution = async ({ tool, model, toolPath, work
92
92
  env: runtime.env,
93
93
  home: runtime.home,
94
94
  client: runtime.client,
95
+ // Issue #2208: provenance names the backend that answered, and the local
96
+ // wrapper under its own name — the two are not interchangeable.
97
+ backend: runtime.backend,
98
+ formalAiVersion: runtime.formalAiVersion,
99
+ formalAiWrapperVersion: runtime.formalAiWrapperVersion,
95
100
  stop: runtime.stop,
96
101
  };
97
102
  };
@@ -19,7 +19,7 @@ import fs from 'node:fs';
19
19
  import os from 'node:os';
20
20
  import path from 'node:path';
21
21
  import { isExecutingSessionStatus, isTerminalSessionStatus } from './session-status.lib.mjs';
22
- import { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, releaseFormalAiSidecarForTask } from './formal-ai-isolation.lib.mjs';
22
+ import { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, buildFormalAiTaskEnv, releaseFormalAiSidecarForTask } from './formal-ai-isolation.lib.mjs';
23
23
  // The image references live in their own module so the Formal AI sidecar can
24
24
  // resolve the locally present Hive Mind image (which bakes `formal-ai`) without
25
25
  // importing this runner and creating a cycle. Re-exported here because callers
@@ -399,7 +399,7 @@ export async function executeWithIsolation(command, args, options = {}) {
399
399
  await releaseFormalAiSidecarForTask({ sidecar, sessionId, env: hostEnv, verbose });
400
400
  return failLaunch(formalAiRoutingError);
401
401
  }
402
- const taskEnv = sidecar ? { ...hostEnv, HIVE_MIND_FORMAL_AI_BASE_URL: sidecar.baseUrl } : hostEnv;
402
+ const taskEnv = buildFormalAiTaskEnv({ sidecar, env: hostEnv });
403
403
  const effectiveOptions =
404
404
  backend === 'docker'
405
405
  ? {
@@ -48,20 +48,140 @@ import { quietProbe } from './quiet-probe.lib.mjs';
48
48
  const LARGE_DIFF_WARNING_BYTES = 8 * 1024 * 1024;
49
49
 
50
50
  /**
51
- * The solver's own scaffolding files, recognised by the content it writes into
52
- * them (`src/solve.auto-pr.lib.mjs`). A pull request whose whole diff is one of
53
- * these contains no solution: the placeholder exists only to give an empty
54
- * branch something to open a pull request from, and is reverted once the AI
55
- * commits real work.
56
- *
57
- * Matching on content, not on the file name, keeps a repository's own
58
- * `.gitkeep` or `CLAUDE.md` edits counted as the real changes they are.
51
+ * The solver's own scaffolding files (`src/solve.auto-pr.lib.mjs`). A pull
52
+ * request whose whole diff is one of these contains no solution: the
53
+ * placeholder exists only to give an empty branch something to open a pull
54
+ * request from, and is reverted once the AI commits real work.
55
+ *
56
+ * Issue #2211: asking "did the diff *add* the auto-generated header line?" only
57
+ * recognised the case where the solver created the file. When the repository
58
+ * already tracks a `.gitkeep` - the normal state of every repository generated
59
+ * from a template whose own solver run leaked one - `solve.auto-pr.lib.mjs`
60
+ * appends `# Updated: <timestamp>` instead, so the header is a context line, no
61
+ * pattern matched, and a pull request whose entire diff was that one timestamp
62
+ * measured as one changed file and was auto-merged:
63
+ *
64
+ * https://github.com/konard/audio-decomposer/pull/3
65
+ * .gitkeep | 2 +-
66
+ *
67
+ * The question asked here is therefore the one that actually decides it: once
68
+ * the solver's own generated lines are removed from both sides of the diff, is
69
+ * the file unchanged? That answers "created", "appended to" and "appended to
70
+ * again" with one rule, and it still counts a repository's own `.gitkeep` or
71
+ * `CLAUDE.md` edits as the real changes they are - a change to any line the
72
+ * solver did not write makes the two sides differ.
59
73
  */
60
- const PLACEHOLDER_CONTENT_PATTERNS = new Map([
61
- ['.gitkeep', [/^\+#\s*\.gitkeep file auto-generated at .+ for PR creation at branch /m]],
62
- ['CLAUDE.md', [/^\+Issue to solve: \S+/m, /^\+Your prepared branch: \S+/m]],
74
+
75
+ /** Lines `solve.auto-pr.lib.mjs` writes into `.gitkeep`. */
76
+ const GITKEEP_GENERATED_LINE_PATTERNS = [/^#\s*\.gitkeep file auto-generated at \S+ for PR creation at branch \S+ for issue \S+\s*$/, /^#\s*Updated: \d{4}-\d{2}-\d{2}T[\d:.]+Z?\s*$/];
77
+
78
+ /** The line that opens the task block `solve.auto-pr.lib.mjs` writes into `CLAUDE.md`. */
79
+ const CLAUDE_MD_TASK_BLOCK_HEAD = /^Issue to solve: \S+\s*$/;
80
+
81
+ /** Lines of that task block, including the ones only `--fork` runs emit. */
82
+ const CLAUDE_MD_GENERATED_LINE_PATTERNS = [CLAUDE_MD_TASK_BLOCK_HEAD, /^Your prepared branch: \S+\s*$/, /^Your prepared working directory: \S+\s*$/, /^Your forked repository: \S+\s*$/, /^Original repository \(upstream\): \S+\s*$/, /^Proceed\.\s*$/, /^Run timestamp: \d{4}-\d{2}-\d{2}T[\d:.]+Z?\s*$/];
83
+
84
+ const PLACEHOLDER_GENERATED_LINES = new Map([
85
+ ['.gitkeep', GITKEEP_GENERATED_LINE_PATTERNS],
86
+ ['CLAUDE.md', CLAUDE_MD_GENERATED_LINE_PATTERNS],
63
87
  ]);
64
88
 
89
+ /**
90
+ * Rebuild both sides of one file's unified-diff section.
91
+ *
92
+ * Only the hunks are read: everything before the first `@@` is git's own header
93
+ * (`index`, `new file mode`, `--- a/x`, `+++ b/x`) and belongs to neither side,
94
+ * and `` is a note about the previous line, not a
95
+ * line of the file.
96
+ *
97
+ * Context outside the hunks is missing from both sides equally, which is all the
98
+ * caller needs: it compares the two reconstructions against each other.
99
+ *
100
+ * @param {string} body - the section text, hunk headers included.
101
+ * @returns {{oldLines: string[], newLines: string[]}}
102
+ */
103
+ const reconstructSides = body => {
104
+ const oldLines = [];
105
+ const newLines = [];
106
+ let inHunk = false;
107
+ for (const line of body.split('\n')) {
108
+ if (line.startsWith('@@')) {
109
+ inHunk = true;
110
+ continue;
111
+ }
112
+ if (!inHunk || line.startsWith('\\')) continue;
113
+ const text = line.slice(1);
114
+ if (line[0] === '-') oldLines.push(text);
115
+ else if (line[0] === '+') newLines.push(text);
116
+ else {
117
+ // ' ' is a context line; a completely empty line is a context line whose
118
+ // content is empty (git omits the trailing space on some diffs).
119
+ oldLines.push(text);
120
+ newLines.push(text);
121
+ }
122
+ }
123
+ return { oldLines, newLines };
124
+ };
125
+
126
+ /**
127
+ * True when the first non-blank line at or after `from` opens the CLAUDE.md
128
+ * task block, i.e. the preceding `---` is the separator the solver writes
129
+ * rather than a horizontal rule a human wrote.
130
+ */
131
+ const opensClaudeTaskBlock = (lines, from) => {
132
+ for (let i = from; i < lines.length; i++) {
133
+ if (lines[i].trim() === '') continue;
134
+ return CLAUDE_MD_TASK_BLOCK_HEAD.test(lines[i]);
135
+ }
136
+ return false;
137
+ };
138
+
139
+ /**
140
+ * Drop every line the solver generated, leaving whatever the repository owns.
141
+ *
142
+ * @returns {{kept: string[], removed: number}}
143
+ */
144
+ const stripGeneratedLines = (path, lines) => {
145
+ const patterns = PLACEHOLDER_GENERATED_LINES.get(path) || [];
146
+ const kept = [];
147
+ let removed = 0;
148
+ for (let i = 0; i < lines.length; i++) {
149
+ const line = lines[i];
150
+ if (patterns.some(pattern => pattern.test(line))) {
151
+ removed += 1;
152
+ continue;
153
+ }
154
+ // The CLAUDE.md append path writes "\n\n---\n\n" ahead of the task block;
155
+ // that separator is generated too, but only in that position.
156
+ if (path === 'CLAUDE.md' && /^-{3,}\s*$/.test(line) && opensClaudeTaskBlock(lines, i + 1)) {
157
+ removed += 1;
158
+ continue;
159
+ }
160
+ kept.push(line);
161
+ }
162
+ return { kept, removed };
163
+ };
164
+
165
+ /**
166
+ * Is this section nothing but the solver's placeholder bookkeeping?
167
+ *
168
+ * @param {string|null} path - the file's path, or null when it is not a
169
+ * placeholder candidate.
170
+ * @param {string} body - the section text.
171
+ * @returns {boolean}
172
+ */
173
+ const isPlaceholderSection = (path, body) => {
174
+ if (!PLACEHOLDER_GENERATED_LINES.has(path)) return false;
175
+ const { oldLines, newLines } = reconstructSides(body);
176
+ const before = stripGeneratedLines(path, oldLines);
177
+ const after = stripGeneratedLines(path, newLines);
178
+ // Nothing generated on either side means this diff is not the solver's doing.
179
+ if (before.removed === 0 && after.removed === 0) return false;
180
+ // Trailing blank lines are what the append path leaves behind; they are not a
181
+ // change anyone made.
182
+ return before.kept.join('\n').trimEnd() === after.kept.join('\n').trimEnd();
183
+ };
184
+
65
185
  /**
66
186
  * Measure a unified diff in a single pass.
67
187
  *
@@ -96,8 +216,7 @@ const measureDiff = diff => {
96
216
 
97
217
  const closeSection = () => {
98
218
  if (!section) return;
99
- const isPlaceholder = Boolean(section.patterns) && section.patterns.every(pattern => pattern.test(section.body));
100
- if (isPlaceholder) placeholderSections += 1;
219
+ if (isPlaceholderSection(section.path, section.body)) placeholderSections += 1;
101
220
  else {
102
221
  filesChanged += 1;
103
222
  additions += section.additions;
@@ -116,14 +235,13 @@ const measureDiff = diff => {
116
235
  closeSection();
117
236
  const match = /^diff --git a\/(.+) b\/(.+)$/.exec(line);
118
237
  const path = match ? match[2] : '';
119
- const patterns = PLACEHOLDER_CONTENT_PATTERNS.get(path) || null;
120
- section = { patterns, body: '', additions: 0, deletions: 0 };
238
+ section = { path, candidate: PLACEHOLDER_GENERATED_LINES.has(path), body: '', additions: 0, deletions: 0 };
121
239
  continue;
122
240
  }
123
241
  if (!section) continue;
124
242
  // Only a placeholder candidate needs its text kept; every other file is
125
243
  // reduced to two counters as it streams past.
126
- if (section.patterns) section.body += `${line}\n`;
244
+ if (section.candidate) section.body += `${line}\n`;
127
245
  if (line.length > 1) {
128
246
  if (line[0] === '+' && line[1] !== '+') section.additions += 1;
129
247
  else if (line[0] === '-' && line[1] !== '-') section.deletions += 1;
@@ -147,7 +265,7 @@ const measureDiff = diff => {
147
265
  * @param {number} params.prNumber
148
266
  * @param {Function} params.$ command-stream tagged-template executor
149
267
  * @param {Function} [params.log] - optional logger for the size diagnostic
150
- * @returns {Promise<{hasChanges: boolean, filesChanged: number, additions: number, deletions: number, placeholderOnly: boolean, measured: boolean, diffBytes: number}>}
268
+ * @returns {Promise<{hasChanges: boolean, filesChanged: number, additions: number, deletions: number, placeholderOnly: boolean, placeholderSections: number, measured: boolean, diffBytes: number}>}
151
269
  * The counts cover the AI's own work: the solver's placeholder file is
152
270
  * excluded and reported through `placeholderOnly` instead. `measured` is
153
271
  * false when the diff could not be fetched, in which case callers must not
@@ -187,6 +305,11 @@ export const getPullRequestChangeStats = async ({ owner, repo, prNumber, $, log
187
305
  additions,
188
306
  deletions,
189
307
  placeholderOnly: filesChanged === 0 && placeholderSections > 0,
308
+ // Issue #2211: a pull request that has real changes *and* still carries the
309
+ // solver's placeholder is not empty, but it must not be merged with the
310
+ // placeholder in it either. Reported separately so the merge watcher can
311
+ // clean it up before merging instead of shipping it to the default branch.
312
+ placeholderSections,
190
313
  measured,
191
314
  diffBytes,
192
315
  };
@@ -233,4 +356,10 @@ export const EMPTY_PULL_REQUEST_BLOCKER = 'The pull request contains no changes
233
356
  */
234
357
  export const buildEmptyPullRequestBlocker = (stats = null) => (stats?.placeholderOnly ? 'The pull request contains only the placeholder file the solver commits to open a pull request, so there is nothing to merge' : EMPTY_PULL_REQUEST_BLOCKER);
235
358
 
236
- export default { getPullRequestChangeStats, formatChangeSummary, EMPTY_PULL_REQUEST_BLOCKER, buildEmptyPullRequestBlocker };
359
+ /**
360
+ * Exported for tests and for `experiments/issue-2211`: measuring a diff without
361
+ * a GitHub round trip is the only way to replay an archived pull request.
362
+ */
363
+ export const __measureDiffForTests = measureDiff;
364
+
365
+ export default { getPullRequestChangeStats, formatChangeSummary, EMPTY_PULL_REQUEST_BLOCKER, buildEmptyPullRequestBlocker, __measureDiffForTests: measureDiff };
@@ -150,3 +150,31 @@ export default {
150
150
  resolveDraftBlocker,
151
151
  resolveMergeFailure,
152
152
  };
153
+
154
+ /**
155
+ * Issue #2211: revert a solver placeholder that survived into the pull request diff.
156
+ *
157
+ * `src/solve.mjs` reverts the placeholder before this loop starts, so reaching
158
+ * here means the session that created it crashed, was resumed from a different
159
+ * working directory, or was restarted inside the loop. Merging anyway publishes
160
+ * hive-mind's own scaffolding to the default branch, which is what happened to
161
+ * https://github.com/konard/audio-decomposer/pull/3 - and, eight times over, to
162
+ * the `.gitkeep` on the default branch of the template repository it was
163
+ * generated from. See docs/case-studies/issue-2211.
164
+ *
165
+ * Failure is not fatal: the caller re-measures the diff, and a placeholder that
166
+ * could not be reverted still keeps the pull request from looking mergeable.
167
+ *
168
+ * @returns {Promise<boolean>} whether a cleanup was attempted
169
+ */
170
+ export const revertPlaceholderBeforeMerge = async ({ changeStats, tempDir, branchName, argv, log, formatAligned, cleanErrorMessage }) => {
171
+ if (!tempDir || !(changeStats?.placeholderSections > 0)) return false;
172
+ await log(formatAligned('🧹', 'Placeholder in diff:', 'reverting the solver placeholder file before it can be merged', 2), { level: 'warning' });
173
+ try {
174
+ const { cleanupClaudeFile } = await import('./solve.results.lib.mjs');
175
+ await cleanupClaudeFile(tempDir, branchName, null, argv);
176
+ } catch (error) {
177
+ await log(formatAligned('⚠️', 'Placeholder cleanup failed:', cleanErrorMessage(error), 2), { level: 'warning' });
178
+ }
179
+ return true;
180
+ };
@@ -34,7 +34,7 @@ const { mergePullRequest, getRepoVisibility, BILLING_LIMIT_ERROR_PATTERN, getDet
34
34
  // Issue #2182: guard rails for this loop (wall-clock ceiling, draft self-heal,
35
35
  // classified merge failures). See solve.auto-merge-guards.lib.mjs.
36
36
  const autoMergeGuards = await import('./solve.auto-merge-guards.lib.mjs');
37
- const { DRAFT_RECHECK_DELAY_MS, evaluateWatchTimeout, resolveDraftBlocker, resolveMergeFailure } = autoMergeGuards;
37
+ const { DRAFT_RECHECK_DELAY_MS, evaluateWatchTimeout, resolveDraftBlocker, resolveMergeFailure, revertPlaceholderBeforeMerge } = autoMergeGuards;
38
38
  // Re-exported so callers and tests keep a single entry point for the watch loop.
39
39
  export const { DEFAULT_WATCH_TIMEOUT_HOURS, normalizeWatchTimeoutHours } = autoMergeGuards;
40
40
  // Import GitHub functions for log attachment
@@ -131,6 +131,8 @@ export const watchUntilMergeable = async params => {
131
131
  // Issue #1503: Track consecutive "no workflow runs" checks per-SHA (reset on new push)
132
132
  let consecutiveNoRunsChecks = 0;
133
133
  let lastKnownHeadSha = null;
134
+ // Issue #2211: revert a leftover solver placeholder at most once per watch.
135
+ let placeholderCleanupAttempted = false;
134
136
  // Issue #1567: Initial cooldown to let CI register and solution logs post
135
137
  const INITIAL_COOLDOWN_SECONDS = MIN_CI_CHECK_INTERVAL_SECONDS;
136
138
  // Issue #2182: this loop used to be `while (true)` with no wall-clock ceiling
@@ -330,6 +332,16 @@ export const watchUntilMergeable = async params => {
330
332
  if (isEmptyPullRequest) {
331
333
  await log(formatAligned('⚠️', 'PR is empty:', changeStats.placeholderOnly ? 'only the solver placeholder file is in the diff - not treating it as mergeable' : 'net diff contains no files - not treating it as mergeable', 2), { level: 'warning' });
332
334
  }
335
+ // Issue #2211: defense in depth - never merge the solver's own placeholder.
336
+ if (changeStats.placeholderSections > 0 && !placeholderCleanupAttempted) {
337
+ placeholderCleanupAttempted = true;
338
+ if (await revertPlaceholderBeforeMerge({ changeStats, tempDir, branchName: prBranch || branchName, argv, log, formatAligned, cleanErrorMessage })) {
339
+ // Give the revert push a moment to register, then re-measure so the
340
+ // merge decision is made on the post-cleanup diff.
341
+ await interruptibleSleep(DRAFT_RECHECK_DELAY_MS);
342
+ continue;
343
+ }
344
+ }
333
345
  // If PR is mergeable, no blockers, no new comments, no issue metadata
334
346
  // edits, no uncommitted changes and it actually changes something
335
347
  if (blockers.length === 0 && !hasNewComments && !hasIssueMetadataChanges && !hasUncommittedChanges && !isEmptyPullRequest) {
package/src/solve.mjs CHANGED
@@ -1197,6 +1197,20 @@ try {
1197
1197
  logsAttached = true;
1198
1198
  }
1199
1199
  }
1200
+ // Issue #1516: Cleanup after all completion signals (it was before verifyResults, which
1201
+ // caused premature commits). Issue #2211: but strictly BEFORE the auto-merge watch loop.
1202
+ // It used to run after it, and `--auto-merge` therefore merged the placeholder into the
1203
+ // default branch and only then reverted it on a branch nobody would look at again:
1204
+ //
1205
+ // 19:20:07 Initial commit with task details (.gitkeep touched)
1206
+ // 19:28:09 Merge pull request #3 (.gitkeep leaked into main)
1207
+ // 19:28:13 Revert "Initial commit with task details" <- 4 seconds too late
1208
+ //
1209
+ // https://github.com/konard/audio-decomposer/pull/3, docs/case-studies/issue-2211.
1210
+ // Reverting first also lets the loop see the pull request as it really is: with the
1211
+ // placeholder gone, a pull request that implemented nothing has an empty diff and the
1212
+ // loop restarts the AI instead of merging an empty change.
1213
+ await cleanupClaudeFile(tempDir, branchName, claudeCommitHash, argv);
1200
1214
  // Issue #2182: the AI working session is over at this point — everything below is
1201
1215
  // monitoring and merging, not working. The pull request must therefore be back in
1202
1216
  // "ready for review" BEFORE the auto-merge watch loop starts, because that loop can
@@ -1237,8 +1251,6 @@ try {
1237
1251
  }
1238
1252
  // Issue #1952: Final --attach-logs safety net + logsAttached reconciliation. See attach-logs-guarantee.lib.mjs.
1239
1253
  logsAttached = (await attachFinalLogIfMissing({ shouldAttachLogs, prNumber, owner, repo, $, log, sanitizeLogContent, getLogFile, attachLogToGitHub, argv, sessionId, tempDir, anthropicTotalCostUSD, resultModelUsage })) || logsAttached;
1240
- // Issue #1516: Cleanup after all signals (was before verifyResults, caused premature commits)
1241
- await cleanupClaudeFile(tempDir, branchName, claudeCommitHash, argv);
1242
1254
  await finalizeDevelopmentLog(); // Issue #1596/#2048: idempotent no-op on the success path (already committed before readiness signal); still preserves late/error work.
1243
1255
  await endWorkSession({ isContinueMode, prNumber, argv, log, formatAligned, $, logsAttached });
1244
1256
  } catch (error) {