@link-assistant/hive-mind 2.19.0 → 2.19.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +28 -0
- package/package.json +1 -1
- package/src/formal-ai-image.lib.mjs +108 -10
- package/src/formal-ai-isolation.lib.mjs +15 -1
- package/src/formal-ai-runtime.lib.mjs +219 -6
- package/src/formal-ai-sidecar.lib.mjs +41 -9
- package/src/formal-ai.lib.mjs +5 -0
- package/src/isolation-runner.lib.mjs +2 -2
- package/src/pull-request-changes.lib.mjs +147 -18
- package/src/solve.auto-merge-guards.lib.mjs +28 -0
- package/src/solve.auto-merge.lib.mjs +13 -1
- package/src/solve.mjs +14 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,33 @@
|
|
|
1
1
|
# @link-assistant/hive-mind
|
|
2
2
|
|
|
3
|
+
## 2.19.2
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- 57b8258: Never merge hive-mind's own placeholder file, and restart when a pull request changes nothing.
|
|
8
|
+
|
|
9
|
+
`konard/audio-decomposer#1` was solved twice, both pull requests were auto-merged, and the complete diff of both was the `.gitkeep` hive-mind writes only so that an empty branch has something to open a pull request from. Two defects had to line up for that, and both are fixed here.
|
|
10
|
+
|
|
11
|
+
- **The placeholder is reverted before anything can merge it.** `cleanupClaudeFile()` ran after `startAutoRestartUntilMergeable()`, so with `--auto-merge` it was structurally guaranteed to be too late: on that pull request the revert commit is timestamped four seconds after the merge commit. It now runs before the watch loop and still after `verifyResults()`, which is the ordering issue #1516 actually asked for. As defense in depth, the watch loop reverts a placeholder that survived into its own diff — a crashed session, a resumed run — instead of merging it.
|
|
12
|
+
- **An appended placeholder is recognised as a placeholder.** The empty-pull-request detector matched an added `# .gitkeep file auto-generated at …` line, which the solver only writes when it _creates_ the file; when the file already exists it appends `# Updated: <timestamp>` instead, and that reads as an ordinary modification. The diff was counted as real work, so the pull request looked mergeable and the auto-restart from issue #2119 never fired. The measurement now reconstructs both sides of the file and compares them with hive-mind's own generated lines removed, so created, appended and re-appended placeholders are all caught — while a genuine edit to a `.gitkeep` or `CLAUDE.md` the repository owns still counts as work.
|
|
13
|
+
|
|
14
|
+
The leak was not a one-off: `.gitkeep` on the default branch of `link-foundation/rust-ai-driven-development-pipeline-template` had accumulated eight solver-generated lines from eight merged pull requests, one of which also carried real changes. Each miss makes the next one certain, because a surviving file forces the append path that the detector could not see. The full reconstruction is in `docs/case-studies/issue-2211`.
|
|
15
|
+
|
|
16
|
+
## 2.19.1
|
|
17
|
+
|
|
18
|
+
### Patch Changes
|
|
19
|
+
|
|
20
|
+
- 6cdf9e2: Keep the Formal AI release a verified update accepted, and record the backend that actually served each task rather than the local wrapper (issues #2207, #2208, delivered together for #2209).
|
|
21
|
+
|
|
22
|
+
Two defects sat on either side of the same handoff, and each one hid the other. The updater verified a new Formal AI image against the persisted memory volume, accepted it, and wrote it into `formal-ai-sidecar.json` — and then the next cold task threw it away, because `acquireFormalAiSidecar` resolved its image from the environment alone and re-selected the bootstrap release. Meanwhile the task's provenance reported `formal-ai --version` from the local wrapper, a binary that answers no requests, so the log could not have revealed the downgrade even when it happened.
|
|
23
|
+
|
|
24
|
+
- **The accepted image is what the next task boots.** `readAcceptedFormalAiImage` reads the acceptance back out of state and acquire passes it into image resolution. Precedence is an operator pin (exact and exclusive), then the accepted image (also exclusive), then the published bootstrap image only when nothing has been accepted yet, then the local Hive Mind image. The accepted candidates lead with the **immutable digest** and fall back to the tag only with `expectDigest` attached, so a tag that moved after acceptance is refused instead of booted — which is not hypothetical: `ghcr.io/link-assistant/formal-ai:latest` pointed at 0.346.0 when this was reported and points at 0.347.0 now. An accepted image that is no longer on the host is a refusal with the pin named in the message, never a silent downgrade onto memory that has already been migrated.
|
|
25
|
+
- **The serving backend is asked, not assumed.** `probeFormalAiBackend` queries `/health` on the configured endpoint before any client runs, within a bounded deadline, presenting the configured API key and returning immediately on 401/403 instead of retrying a credential that will still be wrong. `assertSupportedFormalAiBackend` gives each answer its own refusal — unreachable, unauthorized, non-JSON, JSON without a version, below `FORMAL_AI_MINIMUM_VERSION`, incompatible persisted memory — and never guesses a version.
|
|
26
|
+
- **The two versions have two names.** `runtime.formalAiVersion` is the release that answers the requests and is what the logs and provenance report; `runtime.formalAiWrapperVersion` is the local executable that started the server and wrote the config, keeping its own floor check and its own log line. `validateFormalAiToolConnection` is unchanged.
|
|
27
|
+
- **The halves are joined.** When Hive Mind manages the sidecar, the leased image, digest, source and verified version are published into the task environment, read back by the runtime, and required to agree with what `/health` reports; an endpoint serving something other than the accepted release is refused rather than recorded. The runtime cache is keyed by endpoint, not by the release behind it, so a cache hit re-probes and says so when the backend changed underneath a reused URL.
|
|
28
|
+
|
|
29
|
+
Three regression suites cover it: `tests/test-issue-2207-accepted-image-persistence.mjs` (candidate selection, update → stopped sidecar → acquire → release → acquire including a process restart against persisted state, a moved tag, a rollback, a pruned accepted image, an operator pin with a busy lease), `tests/test-issue-2208-serving-backend-provenance.mjs` (ten cases against real loopback HTTP backends), and `tests/test-issue-2209-verified-release-serving-provenance.mjs`, which runs the Docker simulator and a real serving process together so neither half can look correct while the pair fails. The same transition was then run against a real Docker daemon and the real published images; the evidence is in `docs/case-studies/issue-2209/`.
|
|
30
|
+
|
|
3
31
|
## 2.19.0
|
|
4
32
|
|
|
5
33
|
### Minor Changes
|
package/package.json
CHANGED
|
@@ -35,15 +35,20 @@
|
|
|
35
35
|
* - `HIVE_MIND_FORMAL_AI_IMAGE` is an **exact** operator pin: it is the only
|
|
36
36
|
* candidate, and if it cannot be resolved the acquire fails. An operator who
|
|
37
37
|
* names an image means that image.
|
|
38
|
-
* - Otherwise the
|
|
39
|
-
*
|
|
40
|
-
*
|
|
38
|
+
* - Otherwise the image accepted by the last verified idle update wins, by its
|
|
39
|
+
* immutable digest (issue #2207). Before that rule the bootstrap version was
|
|
40
|
+
* re-selected on every cold start, so an accepted update was discarded by the
|
|
41
|
+
* next task and the sidecar returned to the release Hive Mind ships with.
|
|
42
|
+
* - Otherwise the published bootstrap image is preferred, and the local Hive
|
|
43
|
+
* Mind image is the fallback. The fallback is only *used*; it is never pulled,
|
|
44
|
+
* because its whole point is that it is already on the host.
|
|
41
45
|
* - Nothing here downgrades a Formal AI task to another model. Issue #2146's
|
|
42
46
|
* fail-closed rule still holds: when no candidate resolves, the task is
|
|
43
47
|
* refused with an actionable message.
|
|
44
48
|
*
|
|
45
49
|
* @see https://github.com/link-assistant/hive-mind/issues/2154
|
|
46
50
|
* @see https://github.com/link-assistant/hive-mind/issues/2146
|
|
51
|
+
* @see https://github.com/link-assistant/hive-mind/issues/2207
|
|
47
52
|
*/
|
|
48
53
|
|
|
49
54
|
import { execFile } from 'node:child_process';
|
|
@@ -62,6 +67,7 @@ const DEFAULT_DOCKER_TIMEOUT_MS = 600_000;
|
|
|
62
67
|
/** Where each candidate came from, so logs and errors can explain themselves. */
|
|
63
68
|
export const FORMAL_AI_IMAGE_SOURCES = Object.freeze({
|
|
64
69
|
PINNED: 'operator-pin',
|
|
70
|
+
ACCEPTED: 'accepted-update',
|
|
65
71
|
PUBLISHED: 'published-image',
|
|
66
72
|
HIVE_MIND: 'hive-mind-image',
|
|
67
73
|
});
|
|
@@ -114,15 +120,82 @@ export const classifyDockerRegistryError = message => {
|
|
|
114
120
|
*/
|
|
115
121
|
export const resolveFormalAiFallbackImage = (env = process.env) => String(env.HIVE_MIND_FORMAL_AI_FALLBACK_IMAGE || '').trim() || getDockerIsolationImage({ env });
|
|
116
122
|
|
|
123
|
+
/**
|
|
124
|
+
* The last image an idle update actually accepted, read out of the durable
|
|
125
|
+
* sidecar record (issue #2207).
|
|
126
|
+
*
|
|
127
|
+
* `state.image`/`state.imageDigest` are a *cache of what is running* — every
|
|
128
|
+
* acquire overwrites them with the container it found — so they cannot answer
|
|
129
|
+
* "which image did we verify?". `state.lastUpdate` can: it is written only by
|
|
130
|
+
* `updateFormalAiSidecarWhenIdle` after the pull, the side-effect-free memory
|
|
131
|
+
* preflight, the backed-up migration and the `/health` memory-compatibility
|
|
132
|
+
* check have all succeeded, and it is left untouched by a rollback.
|
|
133
|
+
*
|
|
134
|
+
* @param {object|null} state - A `formal-ai-sidecar.json` record.
|
|
135
|
+
* @returns {{image: string|null, digest: string|null, version: string|null, memorySchemaVersion: (number|string|null), updatedAt: string|null}|null}
|
|
136
|
+
*/
|
|
137
|
+
export const readAcceptedFormalAiImage = state => {
|
|
138
|
+
const update = state?.lastUpdate;
|
|
139
|
+
if (!update || typeof update !== 'object') return null;
|
|
140
|
+
const image = String(update.image || '').trim() || null;
|
|
141
|
+
const digest = String(update.digest || '').trim() || null;
|
|
142
|
+
if (!image && !digest) return null;
|
|
143
|
+
return { image, digest, version: update.version ?? null, memorySchemaVersion: update.memorySchemaVersion ?? null, updatedAt: update.updatedAt ?? null };
|
|
144
|
+
};
|
|
145
|
+
|
|
146
|
+
/**
|
|
147
|
+
* Candidates that reproduce an accepted update, most trustworthy first.
|
|
148
|
+
*
|
|
149
|
+
* Two rules make this safe:
|
|
150
|
+
*
|
|
151
|
+
* - **The digest wins.** `docker image inspect --format {{.Id}}` returns the
|
|
152
|
+
* local content address, which Docker also accepts as a run reference. Using
|
|
153
|
+
* it means a tag that moved after acceptance (`:latest` is the update tag)
|
|
154
|
+
* cannot smuggle an unverified image past the memory verification that
|
|
155
|
+
* earned the acceptance.
|
|
156
|
+
* - **The tag is only a recovery path.** If the accepted image ID has been
|
|
157
|
+
* pruned, the recorded reference may be pulled again, but the result is
|
|
158
|
+
* accepted only when its digest still matches; otherwise the candidate is
|
|
159
|
+
* refused rather than booted.
|
|
160
|
+
*
|
|
161
|
+
* @param {object|null} accepted - Output of {@link readAcceptedFormalAiImage}.
|
|
162
|
+
* @returns {Array<object>}
|
|
163
|
+
*/
|
|
164
|
+
export const resolveAcceptedFormalAiImageCandidates = (accepted = null) => {
|
|
165
|
+
if (!accepted) return [];
|
|
166
|
+
const candidates = [];
|
|
167
|
+
if (accepted.digest) candidates.push({ image: accepted.digest, reference: accepted.image || accepted.digest, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: false, accepted: true });
|
|
168
|
+
if (accepted.image && accepted.digest) candidates.push({ image: accepted.image, reference: accepted.image, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: true, accepted: true, expectDigest: accepted.digest });
|
|
169
|
+
// An acceptance recorded before digests were captured: the reference is all
|
|
170
|
+
// there is, so it may be used but never re-pulled behind a moving tag.
|
|
171
|
+
if (accepted.image && !accepted.digest) candidates.push({ image: accepted.image, reference: accepted.image, source: FORMAL_AI_IMAGE_SOURCES.ACCEPTED, pullable: false, accepted: true });
|
|
172
|
+
return candidates;
|
|
173
|
+
};
|
|
174
|
+
|
|
117
175
|
/**
|
|
118
176
|
* Ordered list of images to try, most preferred first.
|
|
119
177
|
*
|
|
178
|
+
* Precedence, and why:
|
|
179
|
+
*
|
|
180
|
+
* 1. `HIVE_MIND_FORMAL_AI_IMAGE` — an operator who names an image means that
|
|
181
|
+
* image; the pin is exact and exclusive.
|
|
182
|
+
* 2. The last accepted, verified update (issue #2207). Also exclusive: once an
|
|
183
|
+
* image has been verified against the migrated persisted memory, quietly
|
|
184
|
+
* booting the older bootstrap release against that memory is exactly the
|
|
185
|
+
* silent downgrade the fail-closed rule forbids.
|
|
186
|
+
* 3. The bootstrap release, then the local Hive Mind image — the cold-start
|
|
187
|
+
* path for a host that has never completed an update.
|
|
188
|
+
*
|
|
120
189
|
* @param {object} [env]
|
|
190
|
+
* @param {object} [options]
|
|
191
|
+
* @param {object|null} [options.accepted] - Output of {@link readAcceptedFormalAiImage}.
|
|
121
192
|
* @returns {Array<{image: string, source: string, pullable: boolean}>}
|
|
122
193
|
*/
|
|
123
|
-
export const resolveFormalAiSidecarImageCandidates = (env = process.env) => {
|
|
194
|
+
export const resolveFormalAiSidecarImageCandidates = (env = process.env, { accepted = null } = {}) => {
|
|
124
195
|
const pinned = String(env.HIVE_MIND_FORMAL_AI_IMAGE || '').trim();
|
|
125
196
|
if (pinned) return [{ image: pinned, source: FORMAL_AI_IMAGE_SOURCES.PINNED, pullable: true }];
|
|
197
|
+
const acceptedCandidates = resolveAcceptedFormalAiImageCandidates(accepted);
|
|
198
|
+
if (acceptedCandidates.length > 0) return acceptedCandidates;
|
|
126
199
|
const candidates = [{ image: `${FORMAL_AI_IMAGE_REPOSITORY}:${FORMAL_AI_BOOTSTRAP_VERSION}`, source: FORMAL_AI_IMAGE_SOURCES.PUBLISHED, pullable: true }];
|
|
127
200
|
const fallback = resolveFormalAiFallbackImage(env);
|
|
128
201
|
// Never pulled: the fallback's value is that it is already on the host. A
|
|
@@ -132,7 +205,7 @@ export const resolveFormalAiSidecarImageCandidates = (env = process.env) => {
|
|
|
132
205
|
};
|
|
133
206
|
|
|
134
207
|
/** The preferred image, kept for callers and logs that only need the headline reference. */
|
|
135
|
-
export const resolveFormalAiSidecarImage = (env = process.env) => resolveFormalAiSidecarImageCandidates(env)[0].image;
|
|
208
|
+
export const resolveFormalAiSidecarImage = (env = process.env, options = {}) => resolveFormalAiSidecarImageCandidates(env, options)[0].image;
|
|
136
209
|
|
|
137
210
|
const inspectLocalImage = async (image, { run, timeoutMs }) => {
|
|
138
211
|
try {
|
|
@@ -148,7 +221,8 @@ const inspectLocalImage = async (image, { run, timeoutMs }) => {
|
|
|
148
221
|
* bot log knows the cause and the fix without opening a shell.
|
|
149
222
|
*/
|
|
150
223
|
const describeFailure = attempts => {
|
|
151
|
-
const
|
|
224
|
+
const accepted = attempts.some(attempt => attempt.source === FORMAL_AI_IMAGE_SOURCES.ACCEPTED);
|
|
225
|
+
const lines = [accepted ? 'The Formal AI image accepted by the last verified update could not be resolved, so the sidecar was not started.' : 'No Formal AI image could be resolved, so the sidecar was not started.'];
|
|
152
226
|
for (const attempt of attempts) {
|
|
153
227
|
if (attempt.source === FORMAL_AI_IMAGE_SOURCES.HIVE_MIND && attempt.kind === 'absent') {
|
|
154
228
|
lines.push(`• ${attempt.image} (local Hive Mind image, fallback): not present on this host — Docker-isolated tasks would have to pull it too.`);
|
|
@@ -157,6 +231,13 @@ const describeFailure = attempts => {
|
|
|
157
231
|
lines.push(`• ${attempt.image} (${attempt.source}): ${attempt.reason}${attempt.error ? ` — ${attempt.error}` : ''}`);
|
|
158
232
|
for (const step of attempt.remediation ?? []) lines.push(` → ${step}`);
|
|
159
233
|
}
|
|
234
|
+
if (accepted) {
|
|
235
|
+
// Falling back to the bootstrap release here would run an *older* binary
|
|
236
|
+
// against memory that the accepted release already migrated (issue #2207).
|
|
237
|
+
lines.push('Hive Mind will not fall back to an older Formal AI release against memory an accepted update already migrated.');
|
|
238
|
+
lines.push(' → restore the accepted image on this host (`docker pull` it, or copy it back), or');
|
|
239
|
+
lines.push(' → set HIVE_MIND_FORMAL_AI_IMAGE to the image you want this host to run, which overrides the accepted update.');
|
|
240
|
+
}
|
|
160
241
|
lines.push('Formal AI tasks fail closed by design (issue #2146): Hive Mind will not silently run them on another model.');
|
|
161
242
|
return lines.join('\n');
|
|
162
243
|
};
|
|
@@ -172,17 +253,27 @@ const describeFailure = attempts => {
|
|
|
172
253
|
* @param {Function|null} [params.log]
|
|
173
254
|
* @param {boolean} [params.verbose]
|
|
174
255
|
* @param {boolean} [params.pull] - Set false to accept only images already on the host.
|
|
175
|
-
* @
|
|
256
|
+
* @param {object|null} [params.accepted] - Output of {@link readAcceptedFormalAiImage}, used to build the default candidates.
|
|
257
|
+
* @returns {Promise<{image: string, reference: string, source: string, digest: string|null, pulled: boolean, attempts: object[]}>}
|
|
176
258
|
* @throws {Error} When no candidate resolves; the message names every attempt and its fix.
|
|
177
259
|
*/
|
|
178
|
-
export const ensureFormalAiSidecarImage = async ({ env = process.env, run = execFileAsync, timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS, log = null, verbose = false, pull = true, candidates = resolveFormalAiSidecarImageCandidates(env) } = {}) => {
|
|
260
|
+
export const ensureFormalAiSidecarImage = async ({ env = process.env, run = execFileAsync, timeoutMs = DEFAULT_DOCKER_TIMEOUT_MS, log = null, verbose = false, pull = true, accepted = null, candidates = resolveFormalAiSidecarImageCandidates(env, { accepted }) } = {}) => {
|
|
179
261
|
const attempts = [];
|
|
262
|
+
const reference = candidate => candidate.reference || candidate.image;
|
|
263
|
+
// A candidate that names an expected digest is a *recovery* path for an
|
|
264
|
+
// accepted update whose image ID was pruned: the reference may be re-fetched,
|
|
265
|
+
// but only the recorded content address is allowed to run (issue #2207).
|
|
266
|
+
const mismatch = (candidate, digest) => ({ ...candidate, kind: 'digest-mismatch', reason: `resolved to ${digest ?? 'no digest'} instead of the accepted ${candidate.expectDigest}`, error: null, remediation: ['the tag moved after the update was accepted; restore the accepted image or repin with HIVE_MIND_FORMAL_AI_IMAGE'] });
|
|
180
267
|
|
|
181
268
|
for (const candidate of candidates) {
|
|
182
269
|
const localDigest = await inspectLocalImage(candidate.image, { run, timeoutMs });
|
|
270
|
+
if (localDigest && candidate.expectDigest && localDigest !== candidate.expectDigest) {
|
|
271
|
+
attempts.push(mismatch(candidate, localDigest));
|
|
272
|
+
continue;
|
|
273
|
+
}
|
|
183
274
|
if (localDigest) {
|
|
184
275
|
if (verbose && log) await log(`[VERBOSE] formal-ai-image: using '${candidate.image}' (${candidate.source}) already present locally, digest=${localDigest}`);
|
|
185
|
-
return { image: candidate.image, source: candidate.source, digest: localDigest, pulled: false, attempts };
|
|
276
|
+
return { image: candidate.image, reference: reference(candidate), source: candidate.source, digest: localDigest, pulled: false, attempts };
|
|
186
277
|
}
|
|
187
278
|
|
|
188
279
|
if (!candidate.pullable || !pull) {
|
|
@@ -202,8 +293,13 @@ export const ensureFormalAiSidecarImage = async ({ env = process.env, run = exec
|
|
|
202
293
|
}
|
|
203
294
|
|
|
204
295
|
const digest = await inspectLocalImage(candidate.image, { run, timeoutMs });
|
|
296
|
+
if (candidate.expectDigest && digest !== candidate.expectDigest) {
|
|
297
|
+
attempts.push(mismatch(candidate, digest));
|
|
298
|
+
if (log) await log(`⚠️ ${candidate.image} no longer resolves to the accepted digest ${candidate.expectDigest}; refusing to boot it.`);
|
|
299
|
+
continue;
|
|
300
|
+
}
|
|
205
301
|
if (verbose && log) await log(`[VERBOSE] formal-ai-image: pulled '${candidate.image}' (${candidate.source}), digest=${digest ?? 'unknown'}`);
|
|
206
|
-
return { image: candidate.image, source: candidate.source, digest, pulled: true, attempts };
|
|
302
|
+
return { image: candidate.image, reference: reference(candidate), source: candidate.source, digest, pulled: true, attempts };
|
|
207
303
|
}
|
|
208
304
|
|
|
209
305
|
const error = new Error(describeFailure(attempts));
|
|
@@ -216,6 +312,8 @@ export default {
|
|
|
216
312
|
FORMAL_AI_IMAGE_SOURCES,
|
|
217
313
|
classifyDockerRegistryError,
|
|
218
314
|
ensureFormalAiSidecarImage,
|
|
315
|
+
readAcceptedFormalAiImage,
|
|
316
|
+
resolveAcceptedFormalAiImageCandidates,
|
|
219
317
|
resolveFormalAiFallbackImage,
|
|
220
318
|
resolveFormalAiSidecarImage,
|
|
221
319
|
resolveFormalAiSidecarImageCandidates,
|
|
@@ -16,6 +16,7 @@
|
|
|
16
16
|
* @see https://github.com/link-assistant/hive-mind/issues/2146
|
|
17
17
|
*/
|
|
18
18
|
|
|
19
|
+
import { buildFormalAiSidecarProvenanceEnv } from './formal-ai-runtime.lib.mjs';
|
|
19
20
|
import { acquireFormalAiSidecar, attachTaskToFormalAiNetwork, isFormalAiSidecarEnabled, isFormalAiTask, releaseFormalAiSidecar } from './formal-ai-sidecar.lib.mjs';
|
|
20
21
|
|
|
21
22
|
const logToConsole = message => console.log(message);
|
|
@@ -43,6 +44,19 @@ export const acquireFormalAiSidecarForTask = async ({ backend, args = [], model
|
|
|
43
44
|
}
|
|
44
45
|
};
|
|
45
46
|
|
|
47
|
+
/**
|
|
48
|
+
* The environment a Formal AI task is launched with.
|
|
49
|
+
*
|
|
50
|
+
* Besides the endpoint, this carries the identity of the image the lease pinned
|
|
51
|
+
* (issue #2208). The runtime inside the container still asks the endpoint which
|
|
52
|
+
* release is answering; these values let it refuse when the answer is not the
|
|
53
|
+
* accepted image Hive Mind leased, and let the task record the digest rather
|
|
54
|
+
* than a mutable tag.
|
|
55
|
+
*
|
|
56
|
+
* @returns {object} The task environment (`env` itself when Formal AI is not involved).
|
|
57
|
+
*/
|
|
58
|
+
export const buildFormalAiTaskEnv = ({ sidecar, env = process.env } = {}) => (sidecar ? { ...env, HIVE_MIND_FORMAL_AI_BASE_URL: sidecar.baseUrl, ...buildFormalAiSidecarProvenanceEnv(sidecar) } : env);
|
|
59
|
+
|
|
46
60
|
/**
|
|
47
61
|
* Add the internal Formal AI network to the freshly-created task container.
|
|
48
62
|
*
|
|
@@ -66,4 +80,4 @@ export const releaseFormalAiSidecarForTask = async ({ sidecar, sessionId, env =
|
|
|
66
80
|
}
|
|
67
81
|
};
|
|
68
82
|
|
|
69
|
-
export default { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, releaseFormalAiSidecarForTask };
|
|
83
|
+
export default { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, buildFormalAiTaskEnv, releaseFormalAiSidecarForTask };
|
|
@@ -53,7 +53,7 @@ import { homedir } from 'node:os';
|
|
|
53
53
|
import { dirname, join } from 'node:path';
|
|
54
54
|
import { promisify } from 'node:util';
|
|
55
55
|
|
|
56
|
-
import { assertSupportedFormalAiVersion, FORMAL_AI_MINIMUM_VERSION, readFormalAiBinaryVersion } from './formal-ai-version.lib.mjs';
|
|
56
|
+
import { assertSupportedFormalAiVersion, FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast, readFormalAiBinaryVersion } from './formal-ai-version.lib.mjs';
|
|
57
57
|
|
|
58
58
|
const execFileAsync = promisify(execFile);
|
|
59
59
|
|
|
@@ -65,6 +65,26 @@ export const FORMAL_AI_MODEL_NAME = 'formal-ai';
|
|
|
65
65
|
export const FORMAL_AI_DEFAULT_HOST = '127.0.0.1';
|
|
66
66
|
export const FORMAL_AI_SERVER_READY_TIMEOUT_MS = 90_000;
|
|
67
67
|
|
|
68
|
+
/** Where a Formal AI server reports its own version and memory compatibility. */
|
|
69
|
+
export const FORMAL_AI_HEALTH_PATH = '/health';
|
|
70
|
+
|
|
71
|
+
/**
|
|
72
|
+
* Bounded budget for the backend probe (issue #2208).
|
|
73
|
+
*
|
|
74
|
+
* Deliberately short: this runs immediately before the client executes, so a
|
|
75
|
+
* hanging endpoint must surface as a refusal rather than as a task that appears
|
|
76
|
+
* to be thinking. The server-start path has its own, much longer budget.
|
|
77
|
+
*/
|
|
78
|
+
export const FORMAL_AI_BACKEND_PROBE_TIMEOUT_MS = 15_000;
|
|
79
|
+
|
|
80
|
+
/** Environment carrying the Hive-Mind-managed sidecar's identity into the task container (issue #2207/#2208). */
|
|
81
|
+
export const FORMAL_AI_SIDECAR_PROVENANCE_ENV = Object.freeze({
|
|
82
|
+
image: 'HIVE_MIND_FORMAL_AI_SIDECAR_IMAGE',
|
|
83
|
+
imageDigest: 'HIVE_MIND_FORMAL_AI_SIDECAR_DIGEST',
|
|
84
|
+
version: 'HIVE_MIND_FORMAL_AI_SIDECAR_VERSION',
|
|
85
|
+
imageSource: 'HIVE_MIND_FORMAL_AI_SIDECAR_SOURCE',
|
|
86
|
+
});
|
|
87
|
+
|
|
68
88
|
export const resolveFormalAiApiKey = (env = process.env) => env.FORMAL_AI_API_KEY?.trim() || FORMAL_AI_DEFAULT_API_KEY;
|
|
69
89
|
|
|
70
90
|
/**
|
|
@@ -164,6 +184,109 @@ export const waitForFormalAiServerReady = async ({ baseUrl, probePath = '/api/op
|
|
|
164
184
|
return { ready: false, error: lastError || 'timed out' };
|
|
165
185
|
};
|
|
166
186
|
|
|
187
|
+
/**
|
|
188
|
+
* Ask the endpoint that will actually answer the model requests who it is
|
|
189
|
+
* (issue #2208).
|
|
190
|
+
*
|
|
191
|
+
* Until this existed, `prepareFormalAiRuntime` read `formal-ai --version` from
|
|
192
|
+
* the *local* executable and logged the answer as "the Formal AI version" even
|
|
193
|
+
* when `HIVE_MIND_FORMAL_AI_BASE_URL` pointed at a container running a
|
|
194
|
+
* completely different release. Every provenance record therefore named a binary
|
|
195
|
+
* that never saw the request.
|
|
196
|
+
*
|
|
197
|
+
* Properties that matter:
|
|
198
|
+
*
|
|
199
|
+
* - **Bounded.** The whole probe shares one deadline, and each request carries
|
|
200
|
+
* its own abort signal, so an endpoint that accepts the connection and then
|
|
201
|
+
* goes quiet cannot stall the task.
|
|
202
|
+
* - **Authenticated.** The configured key is presented the way the served API
|
|
203
|
+
* expects it. A 401/403 is reported as an authentication problem and is *not*
|
|
204
|
+
* retried — retrying a rejected credential only delays the failure.
|
|
205
|
+
* - **Honest about malformed answers.** A body that is not JSON, or that
|
|
206
|
+
* carries no version, is a distinct outcome from "unreachable"; guessing a
|
|
207
|
+
* version here would recreate the defect this function exists to fix.
|
|
208
|
+
*
|
|
209
|
+
* @returns {Promise<{ok: boolean, kind: string, status: number|null, version: string|null, memory: object|null, health: object|null, error: string|null}>}
|
|
210
|
+
*/
|
|
211
|
+
export const probeFormalAiBackend = async ({ baseUrl, apiKey = null, path = FORMAL_AI_HEALTH_PATH, timeoutMs = FORMAL_AI_BACKEND_PROBE_TIMEOUT_MS, fetchImpl = globalThis.fetch, intervalMs = 500, now = () => Date.now(), sleepImpl = delay } = {}) => {
|
|
212
|
+
if (!baseUrl) return { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: 'no base URL to probe' };
|
|
213
|
+
const url = `${String(baseUrl).replace(/\/+$/, '')}${path}`;
|
|
214
|
+
const headers = apiKey ? { Authorization: `Bearer ${apiKey}`, 'X-Api-Key': apiKey } : {};
|
|
215
|
+
const deadline = now() + timeoutMs;
|
|
216
|
+
let last = { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: 'not probed' };
|
|
217
|
+
|
|
218
|
+
for (;;) {
|
|
219
|
+
const remaining = deadline - now();
|
|
220
|
+
if (remaining <= 0) break;
|
|
221
|
+
try {
|
|
222
|
+
const response = await fetchImpl(url, { headers, signal: AbortSignal.timeout(Math.min(remaining, timeoutMs)) });
|
|
223
|
+
const status = response?.status ?? null;
|
|
224
|
+
if (status === 401 || status === 403) {
|
|
225
|
+
// Not retried: the credential is wrong, and it will still be wrong in 500ms.
|
|
226
|
+
return { ok: false, kind: 'unauthorized', status, version: null, memory: null, health: null, error: `the Formal AI endpoint rejected the configured credentials (HTTP ${status})` };
|
|
227
|
+
}
|
|
228
|
+
if (!response?.ok) {
|
|
229
|
+
last = { ok: false, kind: 'http-error', status, version: null, memory: null, health: null, error: `HTTP ${status}` };
|
|
230
|
+
} else {
|
|
231
|
+
const text = await response.text();
|
|
232
|
+
let health;
|
|
233
|
+
try {
|
|
234
|
+
health = JSON.parse(text);
|
|
235
|
+
} catch {
|
|
236
|
+
return { ok: false, kind: 'malformed', status, version: null, memory: null, health: null, error: `${path} did not return JSON: ${text.slice(0, 200)}` };
|
|
237
|
+
}
|
|
238
|
+
const version = typeof health?.version === 'string' ? health.version.trim() || null : null;
|
|
239
|
+
if (!version) return { ok: false, kind: 'no-version', status, version: null, memory: health?.memory ?? null, health, error: `${path} answered without a version field` };
|
|
240
|
+
return { ok: true, kind: 'ok', status, version, memory: health?.memory ?? null, health, error: null };
|
|
241
|
+
}
|
|
242
|
+
} catch (error) {
|
|
243
|
+
last = { ok: false, kind: 'unreachable', status: null, version: null, memory: null, health: null, error: error?.message || String(error) };
|
|
244
|
+
}
|
|
245
|
+
if (deadline - now() <= intervalMs) break;
|
|
246
|
+
await sleepImpl(intervalMs);
|
|
247
|
+
}
|
|
248
|
+
return last;
|
|
249
|
+
};
|
|
250
|
+
|
|
251
|
+
/**
|
|
252
|
+
* Turn a probe result into either the accepted backend description or an
|
|
253
|
+
* actionable refusal.
|
|
254
|
+
*
|
|
255
|
+
* Fail-closed (issue #2146) applies here too: a Formal AI task that cannot prove
|
|
256
|
+
* which release is serving it must stop, not proceed and record a guess.
|
|
257
|
+
*
|
|
258
|
+
* @param {object} probe - Output of {@link probeFormalAiBackend}.
|
|
259
|
+
* @param {object} context
|
|
260
|
+
* @param {string} context.baseUrl
|
|
261
|
+
* @param {string} [context.minimumVersion]
|
|
262
|
+
* @param {string|null} [context.expectedVersion] - Version the leased sidecar image was verified at.
|
|
263
|
+
* @returns {{version: string, memory: object|null}}
|
|
264
|
+
*/
|
|
265
|
+
export const assertSupportedFormalAiBackend = (probe, { baseUrl, minimumVersion = FORMAL_AI_MINIMUM_VERSION, expectedVersion = null } = {}) => {
|
|
266
|
+
const where = `the Formal AI endpoint ${baseUrl}`;
|
|
267
|
+
if (!probe?.ok) {
|
|
268
|
+
const detail = probe?.error ? `: ${probe.error}` : '';
|
|
269
|
+
if (probe?.kind === 'unauthorized') throw new Error(`${where} refused the configured credentials${detail}. Set FORMAL_AI_API_KEY to a key the server accepts.`);
|
|
270
|
+
if (probe?.kind === 'malformed') throw new Error(`${where} answered ${FORMAL_AI_HEALTH_PATH} with something other than JSON${detail}. Hive Mind will not guess which Formal AI release is serving this task.`);
|
|
271
|
+
if (probe?.kind === 'no-version') throw new Error(`${where} answered ${FORMAL_AI_HEALTH_PATH} without a version${detail}. Hive Mind requires a serving backend that reports its version (Formal AI >= ${minimumVersion}).`);
|
|
272
|
+
if (probe?.kind === 'http-error') throw new Error(`${where} did not serve ${FORMAL_AI_HEALTH_PATH}${detail}. Check that HIVE_MIND_FORMAL_AI_BASE_URL points at a Formal AI server >= ${minimumVersion}.`);
|
|
273
|
+
throw new Error(`${where} could not be reached${detail}. Check HIVE_MIND_FORMAL_AI_BASE_URL and that the Formal AI server is running.`);
|
|
274
|
+
}
|
|
275
|
+
if (!isFormalAiVersionAtLeast(probe.version, minimumVersion)) {
|
|
276
|
+
throw new Error(`${where} serves Formal AI ${probe.version}, but Hive Mind requires >= ${minimumVersion}. Upgrade the server; the local wrapper's version does not change what answers the requests.`);
|
|
277
|
+
}
|
|
278
|
+
if (probe.memory?.compatible === false) {
|
|
279
|
+
throw new Error(`${where} reports incompatible persisted memory (migration_state=${probe.memory?.migration_state ?? 'unknown'}). Refusing to run a task against memory the serving release cannot read.`);
|
|
280
|
+
}
|
|
281
|
+
if (expectedVersion && expectedVersion !== probe.version) {
|
|
282
|
+
// A lease pins the sidecar's image for the whole task, so the endpoint
|
|
283
|
+
// answering with a different release means it is not the container Hive
|
|
284
|
+
// Mind verified and leased.
|
|
285
|
+
throw new Error(`${where} serves Formal AI ${probe.version}, but the leased Hive Mind sidecar image was verified as ${expectedVersion}. Refusing to record provenance for a backend that is not the accepted release.`);
|
|
286
|
+
}
|
|
287
|
+
return { version: probe.version, memory: probe.memory ?? null };
|
|
288
|
+
};
|
|
289
|
+
|
|
167
290
|
/** Read the machine-readable client registry (`formal-ai clients --format json`). */
|
|
168
291
|
export const loadFormalAiClientRegistry = async ({ formalAiPath = 'formal-ai', run = execFileAsync, env = process.env, timeoutMs = 30_000 } = {}) => {
|
|
169
292
|
const result = await run(formalAiPath, ['clients', '--format', 'json'], { encoding: 'utf8', env: { ...process.env, ...env }, timeout: timeoutMs, maxBuffer: 32 * 1024 * 1024 });
|
|
@@ -321,6 +444,69 @@ export const startFormalAiServer = async ({ cwd, host = FORMAL_AI_DEFAULT_HOST,
|
|
|
321
444
|
};
|
|
322
445
|
};
|
|
323
446
|
|
|
447
|
+
/**
|
|
448
|
+
* The provenance Hive Mind's sidecar lifecycle publishes for the container it
|
|
449
|
+
* leased to this task (issues #2207, #2208).
|
|
450
|
+
*
|
|
451
|
+
* The lease pins one verified image for the whole task, so these values say
|
|
452
|
+
* which release *should* be answering. They are a cross-check, never a
|
|
453
|
+
* substitute for asking the endpoint itself.
|
|
454
|
+
*/
|
|
455
|
+
export const readFormalAiSidecarProvenance = (env = process.env) => {
|
|
456
|
+
const read = name => String(env[name] || '').trim() || null;
|
|
457
|
+
const image = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.image);
|
|
458
|
+
const imageDigest = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageDigest);
|
|
459
|
+
const version = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.version);
|
|
460
|
+
const imageSource = read(FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageSource);
|
|
461
|
+
if (!image && !imageDigest && !version) return null;
|
|
462
|
+
return { image, imageDigest, version, imageSource };
|
|
463
|
+
};
|
|
464
|
+
|
|
465
|
+
/**
|
|
466
|
+
* Publish the leased sidecar's identity into a task's environment so the runtime
|
|
467
|
+
* inside the container can cross-check the endpoint it is pointed at.
|
|
468
|
+
*
|
|
469
|
+
* @param {object|null} sidecar - An `acquireFormalAiSidecar` result.
|
|
470
|
+
* @returns {object} Environment entries (empty when nothing is known).
|
|
471
|
+
*/
|
|
472
|
+
export const buildFormalAiSidecarProvenanceEnv = (sidecar = null) => {
|
|
473
|
+
if (!sidecar) return {};
|
|
474
|
+
const entries = {
|
|
475
|
+
[FORMAL_AI_SIDECAR_PROVENANCE_ENV.image]: sidecar.imageReference || sidecar.image || null,
|
|
476
|
+
[FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageDigest]: sidecar.imageDigest || null,
|
|
477
|
+
[FORMAL_AI_SIDECAR_PROVENANCE_ENV.version]: sidecar.servingVersion || sidecar.health?.version || null,
|
|
478
|
+
[FORMAL_AI_SIDECAR_PROVENANCE_ENV.imageSource]: sidecar.imageSource || null,
|
|
479
|
+
};
|
|
480
|
+
return Object.fromEntries(Object.entries(entries).filter(([, value]) => value));
|
|
481
|
+
};
|
|
482
|
+
|
|
483
|
+
/** One-line description of the backend for logs and session provenance. */
|
|
484
|
+
const describeFormalAiBackend = backend => [`${backend.version} at ${backend.baseUrl}`, backend.image ? `image ${backend.image}` : null, backend.imageDigest ? `digest ${backend.imageDigest}` : null].filter(Boolean).join(', ');
|
|
485
|
+
|
|
486
|
+
/**
|
|
487
|
+
* Query the endpoint that will serve this task and build its provenance record.
|
|
488
|
+
*
|
|
489
|
+
* @returns {Promise<object>} `{ baseUrl, version, memory, image, imageDigest, imageSource, leased, startedLocally, probedAt }`
|
|
490
|
+
*/
|
|
491
|
+
const resolveFormalAiBackend = async ({ baseUrl, apiKey, env, deps, startedLocally }) => {
|
|
492
|
+
const sidecar = readFormalAiSidecarProvenance(env);
|
|
493
|
+
const probe = await (deps.probeBackendImpl || probeFormalAiBackend)({ baseUrl, apiKey, env });
|
|
494
|
+
const { version, memory } = assertSupportedFormalAiBackend(probe, { baseUrl, expectedVersion: sidecar?.version ?? null });
|
|
495
|
+
return {
|
|
496
|
+
baseUrl,
|
|
497
|
+
version,
|
|
498
|
+
memory,
|
|
499
|
+
image: sidecar?.image ?? null,
|
|
500
|
+
imageDigest: sidecar?.imageDigest ?? null,
|
|
501
|
+
imageSource: sidecar?.imageSource ?? null,
|
|
502
|
+
/** True when Hive Mind leased this endpoint from its own sidecar. */
|
|
503
|
+
leased: !!sidecar,
|
|
504
|
+
/** True when this process started the server it is now talking to. */
|
|
505
|
+
startedLocally,
|
|
506
|
+
probedAt: new Date().toISOString(),
|
|
507
|
+
};
|
|
508
|
+
};
|
|
509
|
+
|
|
324
510
|
const runtimeCache = new Map();
|
|
325
511
|
let exitHookInstalled = false;
|
|
326
512
|
|
|
@@ -357,16 +543,32 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
|
|
|
357
543
|
const resolvedFormalAiPath = formalAiPath || env.HIVE_MIND_FORMAL_AI_PATH?.trim() || 'formal-ai';
|
|
358
544
|
const cacheKey = `${tool}::${workdir}::${env.HIVE_MIND_FORMAL_AI_BASE_URL || ''}`;
|
|
359
545
|
const cached = runtimeCache.get(cacheKey);
|
|
360
|
-
if (cached)
|
|
546
|
+
if (cached) {
|
|
547
|
+
// Issue #2208: the cache key is the endpoint, not the release behind it. An
|
|
548
|
+
// external base URL can be re-pointed at a different container between
|
|
549
|
+
// tasks, so the cached provenance is re-checked instead of replayed.
|
|
550
|
+
const backend = await resolveFormalAiBackend({ baseUrl: cached.runtime.baseUrl, apiKey: resolveFormalAiApiKey(env), env, deps, startedLocally: cached.runtime.serverStarted });
|
|
551
|
+
if (backend.version !== cached.runtime.backend?.version || backend.imageDigest !== cached.runtime.backend?.imageDigest) {
|
|
552
|
+
await log(`🧠 Formal AI: serving backend changed to ${describeFormalAiBackend(backend)}`);
|
|
553
|
+
}
|
|
554
|
+
cached.runtime.backend = backend;
|
|
555
|
+
cached.runtime.formalAiVersion = backend.version;
|
|
556
|
+
return cached.runtime;
|
|
557
|
+
}
|
|
361
558
|
|
|
362
559
|
installExitHook();
|
|
363
560
|
|
|
364
561
|
// Issue #2146: `--no-tool-check` skipped the only version probe, allowing an
|
|
365
562
|
// old Formal AI build to return the same unexecuted plan through all five
|
|
366
563
|
// Claude/Codex restarts. Runtime safety cannot depend on preflight options.
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
564
|
+
//
|
|
565
|
+
// This is the *local wrapper*: the executable that starts the server and
|
|
566
|
+
// writes the client configuration. Issue #2208: it is not necessarily the
|
|
567
|
+
// release that answers the model requests, so it keeps its own name and its
|
|
568
|
+
// own compatibility check, and it is never reported as the serving version.
|
|
569
|
+
const formalAiWrapperVersion = await (deps.readVersionImpl || readFormalAiBinaryVersion)({ formalAiPath: resolvedFormalAiPath, env });
|
|
570
|
+
assertSupportedFormalAiVersion(formalAiWrapperVersion);
|
|
571
|
+
await log(`🧠 Formal AI: local wrapper version ${formalAiWrapperVersion} (minimum ${FORMAL_AI_MINIMUM_VERSION})`);
|
|
370
572
|
|
|
371
573
|
const apiKey = resolveFormalAiApiKey(env);
|
|
372
574
|
const externalBaseUrl = env.HIVE_MIND_FORMAL_AI_BASE_URL?.trim() || null;
|
|
@@ -386,6 +588,13 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
|
|
|
386
588
|
await log(`🧠 Formal AI: using the configured server ${baseUrl}`, { verbose: true });
|
|
387
589
|
}
|
|
388
590
|
|
|
591
|
+
// Before any client configuration is written, ask the endpoint who it is.
|
|
592
|
+
const backend = await resolveFormalAiBackend({ baseUrl, apiKey, env, deps, startedLocally: !!server });
|
|
593
|
+
await log(`🧠 Formal AI: serving backend ${describeFormalAiBackend(backend)}`);
|
|
594
|
+
if (backend.version !== formalAiWrapperVersion) {
|
|
595
|
+
await log(`🧠 Formal AI: local wrapper ${formalAiWrapperVersion} differs from the serving backend ${backend.version}; provenance records the backend`, { verbose: true });
|
|
596
|
+
}
|
|
597
|
+
|
|
389
598
|
const clients = await (deps.loadRegistryImpl || loadFormalAiClientRegistry)({ formalAiPath: resolvedFormalAiPath, env });
|
|
390
599
|
const client = findFormalAiClient(clients, tool);
|
|
391
600
|
if (!client) throw new Error(`Formal AI does not list a client configuration for "${tool}"`);
|
|
@@ -429,7 +638,11 @@ export const prepareFormalAiRuntime = async ({ tool, workdir, log = async () =>
|
|
|
429
638
|
client,
|
|
430
639
|
notes,
|
|
431
640
|
serverStarted: !!server,
|
|
432
|
-
|
|
641
|
+
/** The release that actually answers this task's model requests. */
|
|
642
|
+
formalAiVersion: backend.version,
|
|
643
|
+
/** The local executable that started the server and wrote the config. */
|
|
644
|
+
formalAiWrapperVersion,
|
|
645
|
+
backend,
|
|
433
646
|
stop: async () => {
|
|
434
647
|
runtimeCache.delete(cacheKey);
|
|
435
648
|
await server?.stop?.();
|
|
@@ -46,7 +46,7 @@ import fs from 'node:fs';
|
|
|
46
46
|
import { promisify } from 'node:util';
|
|
47
47
|
|
|
48
48
|
import { FORMAL_AI_MINIMUM_VERSION, isFormalAiVersionAtLeast } from './formal-ai-version.lib.mjs';
|
|
49
|
-
import { ensureFormalAiSidecarImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
|
|
49
|
+
import { ensureFormalAiSidecarImage, readAcceptedFormalAiImage, resolveFormalAiSidecarImage } from './formal-ai-image.lib.mjs';
|
|
50
50
|
import { isFormalAiModel } from './formal-ai-model.lib.mjs';
|
|
51
51
|
import { getModelFromArgs } from './model-args.lib.mjs';
|
|
52
52
|
import { withStateLock } from './state-lock.lib.mjs';
|
|
@@ -90,7 +90,7 @@ export const FORMAL_AI_MEMORY_PATH = `${FORMAL_AI_MEMORY_MOUNT}/memory.lino`;
|
|
|
90
90
|
* issue #2154 taught us that "which image" and "is it actually pullable" are the
|
|
91
91
|
* same question.
|
|
92
92
|
*/
|
|
93
|
-
export { FORMAL_AI_IMAGE_REPOSITORY, resolveFormalAiSidecarImage, resolveFormalAiSidecarImageCandidates } from './formal-ai-image.lib.mjs';
|
|
93
|
+
export { FORMAL_AI_IMAGE_REPOSITORY, readAcceptedFormalAiImage, resolveAcceptedFormalAiImageCandidates, resolveFormalAiSidecarImage, resolveFormalAiSidecarImageCandidates } from './formal-ai-image.lib.mjs';
|
|
94
94
|
|
|
95
95
|
/** Applied to the sidecar, its network and its volume so reconciliation can find them. */
|
|
96
96
|
export const FORMAL_AI_SIDECAR_LABEL = 'com.link-assistant.hive-mind.formal-ai';
|
|
@@ -102,7 +102,13 @@ const SIDECAR_LOCK_NAME = 'formal-ai-sidecar';
|
|
|
102
102
|
const DEFAULT_HEALTH_ATTEMPTS = 60;
|
|
103
103
|
const DEFAULT_HEALTH_DELAY_MS = 1000;
|
|
104
104
|
|
|
105
|
-
|
|
105
|
+
/**
|
|
106
|
+
* `lastUpdate` is the accepted-release record (issue #2207) and `serving` is the
|
|
107
|
+
* provenance of the container that last answered a lease (issue #2208). Neither
|
|
108
|
+
* is derivable from `image`/`imageDigest`, which are only a cache of whatever is
|
|
109
|
+
* running right now.
|
|
110
|
+
*/
|
|
111
|
+
const EMPTY_STATE = Object.freeze({ version: 1, image: null, imageReference: null, imageDigest: null, startedAt: null, leases: [], lastUpdate: null, serving: null });
|
|
106
112
|
|
|
107
113
|
/**
|
|
108
114
|
* True when a task will be driven by Formal AI.
|
|
@@ -350,14 +356,21 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
|
|
|
350
356
|
// #2154 the reference went straight into `docker run`, so a registry that
|
|
351
357
|
// refused the pull surfaced as an unreadable `Command failed: docker run …`
|
|
352
358
|
// dump and the task died even though a usable image sat on the host.
|
|
353
|
-
|
|
359
|
+
//
|
|
360
|
+
// `accepted` is what issue #2207 was missing: without it the candidate
|
|
361
|
+
// list was rebuilt from the bootstrap pin on every cold start, so an
|
|
362
|
+
// update that had already been pulled, migrated and verified was silently
|
|
363
|
+
// discarded by the very next task.
|
|
364
|
+
const accepted = readAcceptedFormalAiImage(state);
|
|
365
|
+
const resolved = container.exists && container.image ? { image: container.image, reference: container.image, source: 'running-sidecar', pulled: false } : await ensureFormalAiSidecarImage({ env, accepted, run, timeoutMs: imageTimeoutMs, log, verbose });
|
|
354
366
|
const image = resolved.image;
|
|
367
|
+
const imageReference = resolved.reference || image;
|
|
355
368
|
|
|
356
369
|
await ensureFormalAiNetwork({ run, timeoutMs, log, verbose });
|
|
357
370
|
await ensureFormalAiMemoryVolume({ image, run, timeoutMs, log, verbose });
|
|
358
371
|
|
|
359
372
|
if (!container.exists) {
|
|
360
|
-
if (log) await log(`🧠 Starting the Formal AI sidecar (${image}, ${resolved.source}) on the internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
|
|
373
|
+
if (log) await log(`🧠 Starting the Formal AI sidecar (${imageReference}${imageReference === image ? '' : ` @ ${image}`}, ${resolved.source}) on the internal network '${FORMAL_AI_SIDECAR_NETWORK_NAME}'`);
|
|
361
374
|
await dockerText(run, buildFormalAiSidecarRunArgs({ image, env }), { timeoutMs });
|
|
362
375
|
container = await inspectDockerContainer(FORMAL_AI_SIDECAR_CONTAINER_NAME, { run, timeoutMs });
|
|
363
376
|
}
|
|
@@ -375,15 +388,27 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
|
|
|
375
388
|
// and then fail on the agent-mode API the tasks depend on.
|
|
376
389
|
const reportedVersion = health.health?.version ?? null;
|
|
377
390
|
if (reportedVersion && !isFormalAiVersionAtLeast(reportedVersion, FORMAL_AI_MINIMUM_VERSION)) {
|
|
378
|
-
throw new Error(`Formal AI sidecar image ${
|
|
391
|
+
throw new Error(`Formal AI sidecar image ${imageReference} (${resolved.source}) runs formal-ai ${reportedVersion}, but Hive Mind requires >= ${FORMAL_AI_MINIMUM_VERSION}. Rebuild or repin the image (HIVE_MIND_FORMAL_AI_IMAGE) before running Formal AI tasks.`);
|
|
379
392
|
}
|
|
380
393
|
|
|
381
394
|
const address = await readFormalAiSidecarAddress({ run, timeoutMs });
|
|
382
395
|
const acquiredAt = now().toISOString();
|
|
383
396
|
const nextLeases = [...leases.filter(lease => lease.sessionId !== sessionId), { sessionId, tool, model, acquiredAt }];
|
|
384
|
-
|
|
397
|
+
// The provenance of what is *actually serving* this lease, so a task's
|
|
398
|
+
// evidence can name the release that answered its requests rather than
|
|
399
|
+
// whatever binary happens to sit next to the wrapper (issue #2208).
|
|
400
|
+
const serving = {
|
|
401
|
+
image: imageReference,
|
|
402
|
+
imageDigest: container.imageDigest ?? resolved.digest ?? null,
|
|
403
|
+
imageSource: resolved.source,
|
|
404
|
+
version: reportedVersion,
|
|
405
|
+
memorySchemaVersion: health.health?.memory?.schema_version ?? null,
|
|
406
|
+
acceptedAt: accepted?.updatedAt ?? null,
|
|
407
|
+
observedAt: acquiredAt,
|
|
408
|
+
};
|
|
409
|
+
writeFormalAiSidecarState({ ...state, image: container.image || image, imageReference, imageDigest: container.imageDigest, startedAt: state.startedAt || acquiredAt, leases: nextLeases, serving }, { env, fsImpl });
|
|
385
410
|
|
|
386
|
-
if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: lease '${sessionId}' acquired (${nextLeases.length} active), image=${
|
|
411
|
+
if (verbose && log) await log(`[VERBOSE] formal-ai-sidecar: lease '${sessionId}' acquired (${nextLeases.length} active), image=${imageReference} (${resolved.source}), digest=${container.imageDigest ?? 'unknown'}, address=${address ?? 'unknown'}, formal-ai=${reportedVersion ?? 'unknown'}, memory schema=${serving.memorySchemaVersion ?? 'unknown'}`);
|
|
387
412
|
|
|
388
413
|
return {
|
|
389
414
|
address,
|
|
@@ -394,8 +419,14 @@ export const acquireFormalAiSidecar = async ({ sessionId, tool = null, model = n
|
|
|
394
419
|
containerName: FORMAL_AI_SIDECAR_CONTAINER_NAME,
|
|
395
420
|
memoryVolume: FORMAL_AI_MEMORY_VOLUME_NAME,
|
|
396
421
|
image: container.image || image,
|
|
422
|
+
imageReference,
|
|
397
423
|
imageSource: resolved.source,
|
|
398
|
-
imageDigest:
|
|
424
|
+
imageDigest: serving.imageDigest,
|
|
425
|
+
acceptedImage: accepted?.image ?? null,
|
|
426
|
+
acceptedDigest: accepted?.digest ?? null,
|
|
427
|
+
acceptedVersion: accepted?.version ?? null,
|
|
428
|
+
servingVersion: reportedVersion,
|
|
429
|
+
memorySchemaVersion: serving.memorySchemaVersion,
|
|
399
430
|
health: health.health,
|
|
400
431
|
leaseCount: nextLeases.length,
|
|
401
432
|
};
|
|
@@ -453,6 +484,7 @@ export default {
|
|
|
453
484
|
inspectDockerContainer,
|
|
454
485
|
isFormalAiSidecarEnabled,
|
|
455
486
|
isFormalAiTask,
|
|
487
|
+
readAcceptedFormalAiImage,
|
|
456
488
|
readDockerImageDigest,
|
|
457
489
|
readFormalAiSidecarAddress,
|
|
458
490
|
readFormalAiSidecarState,
|
package/src/formal-ai.lib.mjs
CHANGED
|
@@ -92,6 +92,11 @@ export const resolveFormalAiToolExecution = async ({ tool, model, toolPath, work
|
|
|
92
92
|
env: runtime.env,
|
|
93
93
|
home: runtime.home,
|
|
94
94
|
client: runtime.client,
|
|
95
|
+
// Issue #2208: provenance names the backend that answered, and the local
|
|
96
|
+
// wrapper under its own name — the two are not interchangeable.
|
|
97
|
+
backend: runtime.backend,
|
|
98
|
+
formalAiVersion: runtime.formalAiVersion,
|
|
99
|
+
formalAiWrapperVersion: runtime.formalAiWrapperVersion,
|
|
95
100
|
stop: runtime.stop,
|
|
96
101
|
};
|
|
97
102
|
};
|
|
@@ -19,7 +19,7 @@ import fs from 'node:fs';
|
|
|
19
19
|
import os from 'node:os';
|
|
20
20
|
import path from 'node:path';
|
|
21
21
|
import { isExecutingSessionStatus, isTerminalSessionStatus } from './session-status.lib.mjs';
|
|
22
|
-
import { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, releaseFormalAiSidecarForTask } from './formal-ai-isolation.lib.mjs';
|
|
22
|
+
import { acquireFormalAiSidecarForTask, attachFormalAiTaskContainer, buildFormalAiTaskEnv, releaseFormalAiSidecarForTask } from './formal-ai-isolation.lib.mjs';
|
|
23
23
|
// The image references live in their own module so the Formal AI sidecar can
|
|
24
24
|
// resolve the locally present Hive Mind image (which bakes `formal-ai`) without
|
|
25
25
|
// importing this runner and creating a cycle. Re-exported here because callers
|
|
@@ -399,7 +399,7 @@ export async function executeWithIsolation(command, args, options = {}) {
|
|
|
399
399
|
await releaseFormalAiSidecarForTask({ sidecar, sessionId, env: hostEnv, verbose });
|
|
400
400
|
return failLaunch(formalAiRoutingError);
|
|
401
401
|
}
|
|
402
|
-
const taskEnv =
|
|
402
|
+
const taskEnv = buildFormalAiTaskEnv({ sidecar, env: hostEnv });
|
|
403
403
|
const effectiveOptions =
|
|
404
404
|
backend === 'docker'
|
|
405
405
|
? {
|
|
@@ -48,20 +48,140 @@ import { quietProbe } from './quiet-probe.lib.mjs';
|
|
|
48
48
|
const LARGE_DIFF_WARNING_BYTES = 8 * 1024 * 1024;
|
|
49
49
|
|
|
50
50
|
/**
|
|
51
|
-
* The solver's own scaffolding files
|
|
52
|
-
*
|
|
53
|
-
*
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
-
*
|
|
57
|
-
*
|
|
58
|
-
* `.gitkeep`
|
|
51
|
+
* The solver's own scaffolding files (`src/solve.auto-pr.lib.mjs`). A pull
|
|
52
|
+
* request whose whole diff is one of these contains no solution: the
|
|
53
|
+
* placeholder exists only to give an empty branch something to open a pull
|
|
54
|
+
* request from, and is reverted once the AI commits real work.
|
|
55
|
+
*
|
|
56
|
+
* Issue #2211: asking "did the diff *add* the auto-generated header line?" only
|
|
57
|
+
* recognised the case where the solver created the file. When the repository
|
|
58
|
+
* already tracks a `.gitkeep` - the normal state of every repository generated
|
|
59
|
+
* from a template whose own solver run leaked one - `solve.auto-pr.lib.mjs`
|
|
60
|
+
* appends `# Updated: <timestamp>` instead, so the header is a context line, no
|
|
61
|
+
* pattern matched, and a pull request whose entire diff was that one timestamp
|
|
62
|
+
* measured as one changed file and was auto-merged:
|
|
63
|
+
*
|
|
64
|
+
* https://github.com/konard/audio-decomposer/pull/3
|
|
65
|
+
* .gitkeep | 2 +-
|
|
66
|
+
*
|
|
67
|
+
* The question asked here is therefore the one that actually decides it: once
|
|
68
|
+
* the solver's own generated lines are removed from both sides of the diff, is
|
|
69
|
+
* the file unchanged? That answers "created", "appended to" and "appended to
|
|
70
|
+
* again" with one rule, and it still counts a repository's own `.gitkeep` or
|
|
71
|
+
* `CLAUDE.md` edits as the real changes they are - a change to any line the
|
|
72
|
+
* solver did not write makes the two sides differ.
|
|
59
73
|
*/
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
74
|
+
|
|
75
|
+
/** Lines `solve.auto-pr.lib.mjs` writes into `.gitkeep`. */
|
|
76
|
+
const GITKEEP_GENERATED_LINE_PATTERNS = [/^#\s*\.gitkeep file auto-generated at \S+ for PR creation at branch \S+ for issue \S+\s*$/, /^#\s*Updated: \d{4}-\d{2}-\d{2}T[\d:.]+Z?\s*$/];
|
|
77
|
+
|
|
78
|
+
/** The line that opens the task block `solve.auto-pr.lib.mjs` writes into `CLAUDE.md`. */
|
|
79
|
+
const CLAUDE_MD_TASK_BLOCK_HEAD = /^Issue to solve: \S+\s*$/;
|
|
80
|
+
|
|
81
|
+
/** Lines of that task block, including the ones only `--fork` runs emit. */
|
|
82
|
+
const CLAUDE_MD_GENERATED_LINE_PATTERNS = [CLAUDE_MD_TASK_BLOCK_HEAD, /^Your prepared branch: \S+\s*$/, /^Your prepared working directory: \S+\s*$/, /^Your forked repository: \S+\s*$/, /^Original repository \(upstream\): \S+\s*$/, /^Proceed\.\s*$/, /^Run timestamp: \d{4}-\d{2}-\d{2}T[\d:.]+Z?\s*$/];
|
|
83
|
+
|
|
84
|
+
const PLACEHOLDER_GENERATED_LINES = new Map([
|
|
85
|
+
['.gitkeep', GITKEEP_GENERATED_LINE_PATTERNS],
|
|
86
|
+
['CLAUDE.md', CLAUDE_MD_GENERATED_LINE_PATTERNS],
|
|
63
87
|
]);
|
|
64
88
|
|
|
89
|
+
/**
|
|
90
|
+
* Rebuild both sides of one file's unified-diff section.
|
|
91
|
+
*
|
|
92
|
+
* Only the hunks are read: everything before the first `@@` is git's own header
|
|
93
|
+
* (`index`, `new file mode`, `--- a/x`, `+++ b/x`) and belongs to neither side,
|
|
94
|
+
* and `` is a note about the previous line, not a
|
|
95
|
+
* line of the file.
|
|
96
|
+
*
|
|
97
|
+
* Context outside the hunks is missing from both sides equally, which is all the
|
|
98
|
+
* caller needs: it compares the two reconstructions against each other.
|
|
99
|
+
*
|
|
100
|
+
* @param {string} body - the section text, hunk headers included.
|
|
101
|
+
* @returns {{oldLines: string[], newLines: string[]}}
|
|
102
|
+
*/
|
|
103
|
+
const reconstructSides = body => {
|
|
104
|
+
const oldLines = [];
|
|
105
|
+
const newLines = [];
|
|
106
|
+
let inHunk = false;
|
|
107
|
+
for (const line of body.split('\n')) {
|
|
108
|
+
if (line.startsWith('@@')) {
|
|
109
|
+
inHunk = true;
|
|
110
|
+
continue;
|
|
111
|
+
}
|
|
112
|
+
if (!inHunk || line.startsWith('\\')) continue;
|
|
113
|
+
const text = line.slice(1);
|
|
114
|
+
if (line[0] === '-') oldLines.push(text);
|
|
115
|
+
else if (line[0] === '+') newLines.push(text);
|
|
116
|
+
else {
|
|
117
|
+
// ' ' is a context line; a completely empty line is a context line whose
|
|
118
|
+
// content is empty (git omits the trailing space on some diffs).
|
|
119
|
+
oldLines.push(text);
|
|
120
|
+
newLines.push(text);
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
return { oldLines, newLines };
|
|
124
|
+
};
|
|
125
|
+
|
|
126
|
+
/**
|
|
127
|
+
* True when the first non-blank line at or after `from` opens the CLAUDE.md
|
|
128
|
+
* task block, i.e. the preceding `---` is the separator the solver writes
|
|
129
|
+
* rather than a horizontal rule a human wrote.
|
|
130
|
+
*/
|
|
131
|
+
const opensClaudeTaskBlock = (lines, from) => {
|
|
132
|
+
for (let i = from; i < lines.length; i++) {
|
|
133
|
+
if (lines[i].trim() === '') continue;
|
|
134
|
+
return CLAUDE_MD_TASK_BLOCK_HEAD.test(lines[i]);
|
|
135
|
+
}
|
|
136
|
+
return false;
|
|
137
|
+
};
|
|
138
|
+
|
|
139
|
+
/**
|
|
140
|
+
* Drop every line the solver generated, leaving whatever the repository owns.
|
|
141
|
+
*
|
|
142
|
+
* @returns {{kept: string[], removed: number}}
|
|
143
|
+
*/
|
|
144
|
+
const stripGeneratedLines = (path, lines) => {
|
|
145
|
+
const patterns = PLACEHOLDER_GENERATED_LINES.get(path) || [];
|
|
146
|
+
const kept = [];
|
|
147
|
+
let removed = 0;
|
|
148
|
+
for (let i = 0; i < lines.length; i++) {
|
|
149
|
+
const line = lines[i];
|
|
150
|
+
if (patterns.some(pattern => pattern.test(line))) {
|
|
151
|
+
removed += 1;
|
|
152
|
+
continue;
|
|
153
|
+
}
|
|
154
|
+
// The CLAUDE.md append path writes "\n\n---\n\n" ahead of the task block;
|
|
155
|
+
// that separator is generated too, but only in that position.
|
|
156
|
+
if (path === 'CLAUDE.md' && /^-{3,}\s*$/.test(line) && opensClaudeTaskBlock(lines, i + 1)) {
|
|
157
|
+
removed += 1;
|
|
158
|
+
continue;
|
|
159
|
+
}
|
|
160
|
+
kept.push(line);
|
|
161
|
+
}
|
|
162
|
+
return { kept, removed };
|
|
163
|
+
};
|
|
164
|
+
|
|
165
|
+
/**
|
|
166
|
+
* Is this section nothing but the solver's placeholder bookkeeping?
|
|
167
|
+
*
|
|
168
|
+
* @param {string|null} path - the file's path, or null when it is not a
|
|
169
|
+
* placeholder candidate.
|
|
170
|
+
* @param {string} body - the section text.
|
|
171
|
+
* @returns {boolean}
|
|
172
|
+
*/
|
|
173
|
+
const isPlaceholderSection = (path, body) => {
|
|
174
|
+
if (!PLACEHOLDER_GENERATED_LINES.has(path)) return false;
|
|
175
|
+
const { oldLines, newLines } = reconstructSides(body);
|
|
176
|
+
const before = stripGeneratedLines(path, oldLines);
|
|
177
|
+
const after = stripGeneratedLines(path, newLines);
|
|
178
|
+
// Nothing generated on either side means this diff is not the solver's doing.
|
|
179
|
+
if (before.removed === 0 && after.removed === 0) return false;
|
|
180
|
+
// Trailing blank lines are what the append path leaves behind; they are not a
|
|
181
|
+
// change anyone made.
|
|
182
|
+
return before.kept.join('\n').trimEnd() === after.kept.join('\n').trimEnd();
|
|
183
|
+
};
|
|
184
|
+
|
|
65
185
|
/**
|
|
66
186
|
* Measure a unified diff in a single pass.
|
|
67
187
|
*
|
|
@@ -96,8 +216,7 @@ const measureDiff = diff => {
|
|
|
96
216
|
|
|
97
217
|
const closeSection = () => {
|
|
98
218
|
if (!section) return;
|
|
99
|
-
|
|
100
|
-
if (isPlaceholder) placeholderSections += 1;
|
|
219
|
+
if (isPlaceholderSection(section.path, section.body)) placeholderSections += 1;
|
|
101
220
|
else {
|
|
102
221
|
filesChanged += 1;
|
|
103
222
|
additions += section.additions;
|
|
@@ -116,14 +235,13 @@ const measureDiff = diff => {
|
|
|
116
235
|
closeSection();
|
|
117
236
|
const match = /^diff --git a\/(.+) b\/(.+)$/.exec(line);
|
|
118
237
|
const path = match ? match[2] : '';
|
|
119
|
-
|
|
120
|
-
section = { patterns, body: '', additions: 0, deletions: 0 };
|
|
238
|
+
section = { path, candidate: PLACEHOLDER_GENERATED_LINES.has(path), body: '', additions: 0, deletions: 0 };
|
|
121
239
|
continue;
|
|
122
240
|
}
|
|
123
241
|
if (!section) continue;
|
|
124
242
|
// Only a placeholder candidate needs its text kept; every other file is
|
|
125
243
|
// reduced to two counters as it streams past.
|
|
126
|
-
if (section.
|
|
244
|
+
if (section.candidate) section.body += `${line}\n`;
|
|
127
245
|
if (line.length > 1) {
|
|
128
246
|
if (line[0] === '+' && line[1] !== '+') section.additions += 1;
|
|
129
247
|
else if (line[0] === '-' && line[1] !== '-') section.deletions += 1;
|
|
@@ -147,7 +265,7 @@ const measureDiff = diff => {
|
|
|
147
265
|
* @param {number} params.prNumber
|
|
148
266
|
* @param {Function} params.$ command-stream tagged-template executor
|
|
149
267
|
* @param {Function} [params.log] - optional logger for the size diagnostic
|
|
150
|
-
* @returns {Promise<{hasChanges: boolean, filesChanged: number, additions: number, deletions: number, placeholderOnly: boolean, measured: boolean, diffBytes: number}>}
|
|
268
|
+
* @returns {Promise<{hasChanges: boolean, filesChanged: number, additions: number, deletions: number, placeholderOnly: boolean, placeholderSections: number, measured: boolean, diffBytes: number}>}
|
|
151
269
|
* The counts cover the AI's own work: the solver's placeholder file is
|
|
152
270
|
* excluded and reported through `placeholderOnly` instead. `measured` is
|
|
153
271
|
* false when the diff could not be fetched, in which case callers must not
|
|
@@ -187,6 +305,11 @@ export const getPullRequestChangeStats = async ({ owner, repo, prNumber, $, log
|
|
|
187
305
|
additions,
|
|
188
306
|
deletions,
|
|
189
307
|
placeholderOnly: filesChanged === 0 && placeholderSections > 0,
|
|
308
|
+
// Issue #2211: a pull request that has real changes *and* still carries the
|
|
309
|
+
// solver's placeholder is not empty, but it must not be merged with the
|
|
310
|
+
// placeholder in it either. Reported separately so the merge watcher can
|
|
311
|
+
// clean it up before merging instead of shipping it to the default branch.
|
|
312
|
+
placeholderSections,
|
|
190
313
|
measured,
|
|
191
314
|
diffBytes,
|
|
192
315
|
};
|
|
@@ -233,4 +356,10 @@ export const EMPTY_PULL_REQUEST_BLOCKER = 'The pull request contains no changes
|
|
|
233
356
|
*/
|
|
234
357
|
export const buildEmptyPullRequestBlocker = (stats = null) => (stats?.placeholderOnly ? 'The pull request contains only the placeholder file the solver commits to open a pull request, so there is nothing to merge' : EMPTY_PULL_REQUEST_BLOCKER);
|
|
235
358
|
|
|
236
|
-
|
|
359
|
+
/**
|
|
360
|
+
* Exported for tests and for `experiments/issue-2211`: measuring a diff without
|
|
361
|
+
* a GitHub round trip is the only way to replay an archived pull request.
|
|
362
|
+
*/
|
|
363
|
+
export const __measureDiffForTests = measureDiff;
|
|
364
|
+
|
|
365
|
+
export default { getPullRequestChangeStats, formatChangeSummary, EMPTY_PULL_REQUEST_BLOCKER, buildEmptyPullRequestBlocker, __measureDiffForTests: measureDiff };
|
|
@@ -150,3 +150,31 @@ export default {
|
|
|
150
150
|
resolveDraftBlocker,
|
|
151
151
|
resolveMergeFailure,
|
|
152
152
|
};
|
|
153
|
+
|
|
154
|
+
/**
|
|
155
|
+
* Issue #2211: revert a solver placeholder that survived into the pull request diff.
|
|
156
|
+
*
|
|
157
|
+
* `src/solve.mjs` reverts the placeholder before this loop starts, so reaching
|
|
158
|
+
* here means the session that created it crashed, was resumed from a different
|
|
159
|
+
* working directory, or was restarted inside the loop. Merging anyway publishes
|
|
160
|
+
* hive-mind's own scaffolding to the default branch, which is what happened to
|
|
161
|
+
* https://github.com/konard/audio-decomposer/pull/3 - and, eight times over, to
|
|
162
|
+
* the `.gitkeep` on the default branch of the template repository it was
|
|
163
|
+
* generated from. See docs/case-studies/issue-2211.
|
|
164
|
+
*
|
|
165
|
+
* Failure is not fatal: the caller re-measures the diff, and a placeholder that
|
|
166
|
+
* could not be reverted still keeps the pull request from looking mergeable.
|
|
167
|
+
*
|
|
168
|
+
* @returns {Promise<boolean>} whether a cleanup was attempted
|
|
169
|
+
*/
|
|
170
|
+
export const revertPlaceholderBeforeMerge = async ({ changeStats, tempDir, branchName, argv, log, formatAligned, cleanErrorMessage }) => {
|
|
171
|
+
if (!tempDir || !(changeStats?.placeholderSections > 0)) return false;
|
|
172
|
+
await log(formatAligned('🧹', 'Placeholder in diff:', 'reverting the solver placeholder file before it can be merged', 2), { level: 'warning' });
|
|
173
|
+
try {
|
|
174
|
+
const { cleanupClaudeFile } = await import('./solve.results.lib.mjs');
|
|
175
|
+
await cleanupClaudeFile(tempDir, branchName, null, argv);
|
|
176
|
+
} catch (error) {
|
|
177
|
+
await log(formatAligned('⚠️', 'Placeholder cleanup failed:', cleanErrorMessage(error), 2), { level: 'warning' });
|
|
178
|
+
}
|
|
179
|
+
return true;
|
|
180
|
+
};
|
|
@@ -34,7 +34,7 @@ const { mergePullRequest, getRepoVisibility, BILLING_LIMIT_ERROR_PATTERN, getDet
|
|
|
34
34
|
// Issue #2182: guard rails for this loop (wall-clock ceiling, draft self-heal,
|
|
35
35
|
// classified merge failures). See solve.auto-merge-guards.lib.mjs.
|
|
36
36
|
const autoMergeGuards = await import('./solve.auto-merge-guards.lib.mjs');
|
|
37
|
-
const { DRAFT_RECHECK_DELAY_MS, evaluateWatchTimeout, resolveDraftBlocker, resolveMergeFailure } = autoMergeGuards;
|
|
37
|
+
const { DRAFT_RECHECK_DELAY_MS, evaluateWatchTimeout, resolveDraftBlocker, resolveMergeFailure, revertPlaceholderBeforeMerge } = autoMergeGuards;
|
|
38
38
|
// Re-exported so callers and tests keep a single entry point for the watch loop.
|
|
39
39
|
export const { DEFAULT_WATCH_TIMEOUT_HOURS, normalizeWatchTimeoutHours } = autoMergeGuards;
|
|
40
40
|
// Import GitHub functions for log attachment
|
|
@@ -131,6 +131,8 @@ export const watchUntilMergeable = async params => {
|
|
|
131
131
|
// Issue #1503: Track consecutive "no workflow runs" checks per-SHA (reset on new push)
|
|
132
132
|
let consecutiveNoRunsChecks = 0;
|
|
133
133
|
let lastKnownHeadSha = null;
|
|
134
|
+
// Issue #2211: revert a leftover solver placeholder at most once per watch.
|
|
135
|
+
let placeholderCleanupAttempted = false;
|
|
134
136
|
// Issue #1567: Initial cooldown to let CI register and solution logs post
|
|
135
137
|
const INITIAL_COOLDOWN_SECONDS = MIN_CI_CHECK_INTERVAL_SECONDS;
|
|
136
138
|
// Issue #2182: this loop used to be `while (true)` with no wall-clock ceiling
|
|
@@ -330,6 +332,16 @@ export const watchUntilMergeable = async params => {
|
|
|
330
332
|
if (isEmptyPullRequest) {
|
|
331
333
|
await log(formatAligned('⚠️', 'PR is empty:', changeStats.placeholderOnly ? 'only the solver placeholder file is in the diff - not treating it as mergeable' : 'net diff contains no files - not treating it as mergeable', 2), { level: 'warning' });
|
|
332
334
|
}
|
|
335
|
+
// Issue #2211: defense in depth - never merge the solver's own placeholder.
|
|
336
|
+
if (changeStats.placeholderSections > 0 && !placeholderCleanupAttempted) {
|
|
337
|
+
placeholderCleanupAttempted = true;
|
|
338
|
+
if (await revertPlaceholderBeforeMerge({ changeStats, tempDir, branchName: prBranch || branchName, argv, log, formatAligned, cleanErrorMessage })) {
|
|
339
|
+
// Give the revert push a moment to register, then re-measure so the
|
|
340
|
+
// merge decision is made on the post-cleanup diff.
|
|
341
|
+
await interruptibleSleep(DRAFT_RECHECK_DELAY_MS);
|
|
342
|
+
continue;
|
|
343
|
+
}
|
|
344
|
+
}
|
|
333
345
|
// If PR is mergeable, no blockers, no new comments, no issue metadata
|
|
334
346
|
// edits, no uncommitted changes and it actually changes something
|
|
335
347
|
if (blockers.length === 0 && !hasNewComments && !hasIssueMetadataChanges && !hasUncommittedChanges && !isEmptyPullRequest) {
|
package/src/solve.mjs
CHANGED
|
@@ -1197,6 +1197,20 @@ try {
|
|
|
1197
1197
|
logsAttached = true;
|
|
1198
1198
|
}
|
|
1199
1199
|
}
|
|
1200
|
+
// Issue #1516: Cleanup after all completion signals (it was before verifyResults, which
|
|
1201
|
+
// caused premature commits). Issue #2211: but strictly BEFORE the auto-merge watch loop.
|
|
1202
|
+
// It used to run after it, and `--auto-merge` therefore merged the placeholder into the
|
|
1203
|
+
// default branch and only then reverted it on a branch nobody would look at again:
|
|
1204
|
+
//
|
|
1205
|
+
// 19:20:07 Initial commit with task details (.gitkeep touched)
|
|
1206
|
+
// 19:28:09 Merge pull request #3 (.gitkeep leaked into main)
|
|
1207
|
+
// 19:28:13 Revert "Initial commit with task details" <- 4 seconds too late
|
|
1208
|
+
//
|
|
1209
|
+
// https://github.com/konard/audio-decomposer/pull/3, docs/case-studies/issue-2211.
|
|
1210
|
+
// Reverting first also lets the loop see the pull request as it really is: with the
|
|
1211
|
+
// placeholder gone, a pull request that implemented nothing has an empty diff and the
|
|
1212
|
+
// loop restarts the AI instead of merging an empty change.
|
|
1213
|
+
await cleanupClaudeFile(tempDir, branchName, claudeCommitHash, argv);
|
|
1200
1214
|
// Issue #2182: the AI working session is over at this point — everything below is
|
|
1201
1215
|
// monitoring and merging, not working. The pull request must therefore be back in
|
|
1202
1216
|
// "ready for review" BEFORE the auto-merge watch loop starts, because that loop can
|
|
@@ -1237,8 +1251,6 @@ try {
|
|
|
1237
1251
|
}
|
|
1238
1252
|
// Issue #1952: Final --attach-logs safety net + logsAttached reconciliation. See attach-logs-guarantee.lib.mjs.
|
|
1239
1253
|
logsAttached = (await attachFinalLogIfMissing({ shouldAttachLogs, prNumber, owner, repo, $, log, sanitizeLogContent, getLogFile, attachLogToGitHub, argv, sessionId, tempDir, anthropicTotalCostUSD, resultModelUsage })) || logsAttached;
|
|
1240
|
-
// Issue #1516: Cleanup after all signals (was before verifyResults, caused premature commits)
|
|
1241
|
-
await cleanupClaudeFile(tempDir, branchName, claudeCommitHash, argv);
|
|
1242
1254
|
await finalizeDevelopmentLog(); // Issue #1596/#2048: idempotent no-op on the success path (already committed before readiness signal); still preserves late/error work.
|
|
1243
1255
|
await endWorkSession({ isContinueMode, prNumber, argv, log, formatAligned, $, logsAttached });
|
|
1244
1256
|
} catch (error) {
|