create-mercato-app 0.6.8-develop.7057.1.61440fc3bc → 0.6.8-develop.7063.1.664341cf65

package/agentic/shared/ai/harness/RELEASE.md

@@ -54,6 +54,8 @@ Each writable target is single-use because fixture preparation marks it disposab
 
 Routing cases carry no case-local duration budget, and that is a decision rather than an omission. `maxContextFiles` and the byte budgets measure the agent's context discipline, which is intrinsic to the case and portable between machines; duration measures the runner and the attempt count, which are not — the audited cohort spanned 71 s to 231 s for passing routing runs, one case measured 147 s and 132 s on two runs of the same model, and OMH-139 exhausted the evaluator's 300000 ms default outright. The operator budget carries that variance instead of the catalog. On this release path the lever is `--case-timeout` (default 600000 ms), which the release command passes on to the evaluator explicitly for every routing step; because that pass-through marks the timeout explicit, the evaluator's own runner-aware floors — 600000 ms for Claude, 900000 ms for a Codex `gpt-5.4-mini` high-effort run, 300000 ms for every other runner — apply only to direct `evaluate-agent-harness.mjs` invocations and never fire under `yarn harness:release`. Keeping that pass-through is deliberate rather than inherited: the routing step derives its own process budget from the same value, as slack plus the sum of the per-case ceilings it hands out, so letting the floors raise the inner ceiling while the outer budget still followed `--case-timeout` would kill an entire routing step instead of failing one slow case. Leaving it explicit also keeps the budget runner-independent, which is what lets the primary and portability lanes in one report be compared as models rather than as budgets. The default clears the slowest audited passing routing run — 231 s against 600000 ms, about 62% headroom — and matches both the Claude floor and the `timeoutMs` ceiling `cases.schema.json` enforces, so the gate carries one upper number instead of three; lower `--case-timeout` when a hung case should fail sooner, and note that the step's process budget drops with it — though at a lowered budget the declared writable ceilings keep their own slots, so the step's budget drops less than proportionally. That value was chosen from evaluator-path measurements — the audited cohort above and the live evidence recorded on #5068 — rather than from a driven `yarn harness:release` run, because the complete gate fails closed off the Linux-with-Bubblewrap host stated above; #5078 records the decision and that deviation, and #5433 carries the unmet measurement forward so the default is confirmed or corrected against the release path's own numbers. `--case-timeout` is one budget for three lanes, not a routing-only lever: the writable and review lanes resolve their own ceilings from the same value, so raising it raises theirs too. A declared writable `timeoutMs` is combined with whichever operator value applies as a maximum, so it raises the floor and never lowers it — but because the schema caps it at the shipped default, it can only raise a budget an operator has lowered; routing cases declare none, so there the operator value always stands alone. Writable cases keep their own `timeoutMs` because a writable one-shot's cost is dominated by the slice it must produce, which the case does define.
 
+`--case-timeout` governs the model lanes and only those: routing, writable, and the judge invocation that reviews a writable result. The two steps that invoke no model carry their own flat ceilings instead. Fixture preparation has always used 120000 ms, and the deterministic step now uses the exported `DETERMINISTIC_STEP_TIMEOUT_MS`, the same 120000 ms, rather than the per-model ceiling multiplied by catalog size it derived before. That earlier derivation handed a model-free step a budget that moved whenever an operator changed how patient the gate is with a language model, and it grew that step's ceiling by a factor of five when #5180 raised the `--case-timeout` default from 120000 ms to 600000 ms — roughly 39 hours over the shipped catalog for a pass that finishes in under a second. The replacement is measured rather than chosen freehand: timed on Linux x86_64, a complete 234-case deterministic pass finished in 768 ms to 998 ms over five runs, and narrowing the selection down to a single case measured 842 ms to 890 ms — inside the same spread, so the pass is dominated by process start and catalog load rather than by how many cases it validates. A ceiling roughly 120 times the slowest observed pass therefore bounds a hang without ever bounding a healthy run, and it has no reason to scale with the catalog. `packages/create-app/src/lib/agent-harness-release.test.ts` pins the deterministic argv together with that budget, so the model ceiling and the model-free one cannot silently converge again.
+
 UI-routed implementation reviews receive only the bounded backend UI guide and `om-backend-ui-design` design-system references. Non-UI reviews do not receive that extra context.
 
 The command writes a mode-`0600` `*-release-suite.json` artifact under `.ai/harness/results/`. Its `runnerPolicy` names the blocking primary runner and records either the explicitly requested portability runner or `null`. It stores no raw runner transcripts, target paths, environment values, URL credentials, or provider credentials. Deterministic and `generate`/`typecheck`/`lint`/`build` foundation commands receive the same minimal isolated environment as target validation, not the controller process environment. Exact values of sensitive inherited environment variables and every scalar string copied from Codex `auth.json` or Claude `.credentials.json` stay inside the isolated trusted-runner state, are unreachable through the MCP root/tool contract, and are redacted from all structured results and errors as defense in depth. The report contains exact coverage gaps, sanitized per-target command/test outcomes and actionable failure reasons, first-pass and correction rates, aggregate context-token measures, review verdict counts, and categorized misuse/violation rates.

package/agentic/shared/scripts/run-agent-harness-release.mjs

@@ -41,6 +41,14 @@ const SENSITIVE_AUTH_DATA_FILE = /^(?:auth|credentials?|secrets?|tokens?)(?:\.(?
 const SENSITIVE_ENV_KEY = /(?:^|_)(?:api_?key|auth|credential|credentials|password|passwd|private_?key|secret|token)(?:_|$)/i
 const GENERATED_YARN_CONFIG_PATH = '.yarnrc.yml'
 const GENERATED_YARN_CONFIG_LIMIT = 16_384
+// The deterministic step invokes no model, so its ceiling must not ride --case-timeout. Timed on
+// Linux x86_64, the complete 234-case catalog finished in 768-998 ms, and narrowing the selection
+// down to a single case measured 842-890 ms, inside the same spread, so the run is dominated by
+// fixed process and catalog load rather than by case count. A flat allowance is therefore the
+// honest shape, and 120000 ms is both about 120x the slowest observed run and the value this gate
+// already gives its other model-free step, fixture preparation. It bounds a hang; a healthy run
+// never approaches it.
+export const DETERMINISTIC_STEP_TIMEOUT_MS = 120_000
 
 function usage() {
   return `Run the complete standalone agent-harness release gate.
@@ -56,6 +64,7 @@ Options:
   --prepare-targets <absolute>   Clone this fresh scaffold once per writable case under an empty/new directory
   --writable-targets <absolute> JSON map of every writable case to a fresh disposable app
   --case-timeout <ms>           Per-model invocation timeout floor for the routing, writable, and review lanes (default: ${DEFAULT_CASE_TIMEOUT_MS})
+                                The model-free deterministic and fixture-preparation steps carry their own flat ceilings and do not read it.
   --validation-timeout <ms>     Timeout for each yarn validation (default: 1800000)
   --acknowledge-writes          Required: fixture preparation and validation commands write files
   --help                        Show this help
@@ -123,6 +132,10 @@ export function effectiveCaseTimeout(cases, caseId, fallback) {
   return Math.max(fallback, Number.isInteger(declared) ? declared : 0)
 }
 
+export function deterministicInvocation({ evaluator, root }) {
+  return { args: [evaluator, '--root', root, '--all'], timeout: DETERMINISTIC_STEP_TIMEOUT_MS }
+}
+
 export function routingInvocation({ evaluator, root, step, cases, caseTimeout }) {
   const args = [evaluator, '--root', root, '--runner', step.runner]
   if (step.lane === 'primary') args.push('--all')
@@ -1591,11 +1604,12 @@ export function main(argv = process.argv.slice(2)) {
   const { env: foundationEnv } = createMinimalValidationEnvironment(foundationTempRoot)
   let deterministicResult
   try {
+    const deterministic = deterministicInvocation({ evaluator, root })
     const deterministicExecution = execute(
       process.execPath,
-      [evaluator, '--root', root, '--all'],
+      deterministic.args,
       root,
-      options.caseTimeout * Math.max(1, plan.catalog.caseCount) + 60_000,
+      deterministic.timeout,
       foundationEnv,
     )
     const deterministicObserved = sortedUnique((deterministicExecution.stdout ?? '').match(/^PASS (OMH-[0-9]{3})/gm)?.map((entry) => entry.slice('PASS '.length)) ?? [])