@bjoernrochel/sevro 0.1.0-rc.2 → 0.1.0-rc.21

package/docs/evidence/codex-state-73/README.md

@@ -0,0 +1,204 @@
+# Candidate Codex state validation for #73
+
+Both requested Darrow cases have passed all four original task checks and activation
+in recorded single-trial invocations. [Sanitized results](results.json)
+retain every live invocation's identity and check statuses, plus hashes of the
+changed implementation inputs. Private responses, rollouts and diagnostics
+remain under the ignored `.quality/issue73/` directory.
+
+## Native cases
+
+Validation used macOS arm64, Codex CLI 0.162.1, the `app-server` entrypoint,
+requested candidate `gpt-6.1-sol/medium`, requested semantic grader
+`gpt-6-luna/medium`, passive owner evaluation, N=1 per invocation, jobs=1,
+threshold=1 and a 1,200,000 ms Codex deadline.
+
+Repair reruns initially kept that semantic route. After repeated false negatives
+on responses explicitly reporting effective capacity `8`, subsequent invocations
+use semantic grader `gpt-6.1-sol/medium` with the same cases and checks. The final
+matched pair selects candidate `gpt-6.1-sol/high` after medium-effort candidates
+repeatedly replaced the supplied dependency cache with an empty one.
+
+| Invocation     | Case                                        | Task checks | Activation  | Result                                                       |
+| -------------- | ------------------------------------------- | ----------- | ----------- | ------------------------------------------------------------ |
+| `doctor-01`    | `doctor-adaptive-goal-direct-codex`         | 2/4         | Passed      | Failed: missing offline `hatchling==1.32.3`                  |
+| `doctor-02`    | `doctor-adaptive-goal-direct-codex`         | 3/4         | Passed      | Failed: semantic false negative                              |
+| `doctor-03`    | `doctor-adaptive-goal-direct-codex`         | 4/4         | Passed      | Passed                                                       |
+| `authoring-01` | `create-agent-skill-refuse-undefined-skill` | 4/4         | Passed      | Passed                                                       |
+| `doctor-04`    | `doctor-adaptive-goal-direct-codex`         | 2/4         | Passed      | Failed: offline build dependency missing                     |
+| `authoring-02` | `create-agent-skill-refuse-undefined-skill` | 4/4         | Passed      | Passed after Linux command-skill grant repair                |
+| `doctor-05`    | `doctor-adaptive-goal-direct-codex`         | 2/4         | Passed      | Failed: semantic false negative; partial skill-read evidence |
+| `doctor-06`    | `doctor-adaptive-goal-direct-codex`         | 3/4         | Passed      | Failed: semantic false negative                              |
+| `doctor-07`    | `doctor-adaptive-goal-direct-codex`         | 4/4         | Passed      | Passed with semantic grader `gpt-6.1-sol/medium`             |
+| `authoring-03` | `create-agent-skill-refuse-undefined-skill` | 4/4         | Passed      | Passed with semantic grader `gpt-6.1-sol/medium`             |
+| `doctor-08`    | `doctor-adaptive-goal-direct-codex`         | 2/4         | Passed      | Failed: candidate replaced the seeded cache                  |
+| `authoring-04` | `create-agent-skill-refuse-undefined-skill` | 4/4         | Passed      | Passed after integrating main                                |
+| `doctor-09`    | `doctor-adaptive-goal-direct-codex`         | 1/4         | Passed      | Failed: bootstrap unavailable; partial skill-read evidence   |
+| `authoring-05` | `create-agent-skill-refuse-undefined-skill` | 4/4         | Passed      | Passed on final combined source                              |
+| `doctor-10`    | `doctor-adaptive-goal-direct-codex`         | 2/4         | Passed      | Failed: candidate replaced the seeded cache; high effort     |
+| `authoring-06` | `create-agent-skill-refuse-undefined-skill` | 4/4         | Passed      | Passed on final combined source; high effort                 |
+| `doctor-11`    | `doctor-adaptive-goal-direct-codex`         | Not run     | Unavailable | Exec route refused native-goal policy before execution       |
+| `authoring-07` | `create-agent-skill-refuse-undefined-skill` | Not run     | Unavailable | Exec route refused native-goal policy before execution       |
+
+The doctor retained the fixture's `.codex/config.toml` value `99`. Operator
+configuration selected concurrency `8`, which the private Codex home contained
+and the response diagnosed. Backend applicability stayed unknown when current
+evidence did not establish V1 or V2; concurrency and nesting remained distinct.
+The repository stayed unchanged and orchestration did not activate.
+
+The first failure reported an unavailable offline build dependency. Locked runtime
+dependencies were prepared before retrying. Later command inspection showed that
+`doctor-01`, `doctor-04`, `doctor-08` and `doctor-10` replaced the supplied seeded
+`UV_CACHE_DIR` with an empty `tmp/uv-cache` path. These were candidate cache
+overrides, not evidence that Sevro could not expose the selected configuration.
+In the second trial, the response
+explicitly reported effective capacity `8`, but the semantic grader said it
+reported only the recommendation `4`. The unchanged case and checks were rerun;
+the third trial passed. These observations do not establish a stability rate.
+
+Preparing only runtime dependencies can reuse an existing local environment
+without caching `hatchling`; preparation now includes the locked development
+group so a fresh isolated backend can build offline when it uses the supplied
+cache.
+
+`doctor-05` reported effective capacity `8`, but its semantic check missed that
+claim and the skill-read trace was partial. `doctor-06` again reported `8` and
+retained complete native evidence; the capacity check still produced a false
+negative. The original requirements were preserved throughout these invocations.
+
+The undefined-skill case kept its repository-preservation, no-commit,
+missing-contract and activation checks. Its user-skill check now requires all
+four original candidate skill roots rather than retired `.git/darrow-eval/state`
+or the grader's home. The absence oracle's six deterministic cases accept real
+empty state, detect creation in each root and refuse missing evidence. Darrow's
+37 integration tests passed against the Sevro checkout.
+
+## Downstream migration
+
+The [Darrow migration patch](darrow-migration.patch) comes from local commit
+`40603998` on `fix/73-sevro-codex-validation`, based on the ticket's local
+`fix/pr257-validation` checkpoint. It adds the runtime selection, migrates the
+absence oracle, tests positive and negative states, documents the selection and
+updates both authoring plugin manifests. Existing validation worktrees and their
+unpublished repairs were preserved. The capacity `8` override is local validation
+configuration and is excluded from that commit and patch.
+
+Apply the patch in that Darrow checkout, install its locked dependencies, and
+prepare the public dependency cache before native execution:
+
+```sh
+export SEVRO_CHECKOUT=/absolute/path/to/sevro-candidate
+export UV_PYTHON_INSTALL_DIR="$(uv python dir)"
+UV_DEFAULT_INDEX=https://pypi.org/simple UV_CACHE_DIR="$HOME/.darrow/cache/sevro-uv-cache" uv sync --locked --group dev --project plugins/orchestration/darrow-adaptive-goal/backend
+UV_DEFAULT_INDEX=https://pypi.org/simple UV_CACHE_DIR="$HOME/.darrow/cache/sevro-uv-cache" uv sync --locked --group dev --project plugins/foundation/darrow-skill-authoring/backend
+bun eval --harness codex --skill doctor-adaptive-goal --case doctor-adaptive-goal-direct-codex \
+  --codex-entrypoint app-server --model gpt-6.1-sol --effort high \
+  --semantic-check-model gpt-6.1-sol --semantic-check-effort medium \
+  --owner-evaluation passive --trials 1 --jobs 1 --threshold 1 \
+  --results-root "$PWD/evals/results/codex-state-doctor" -- \
+  --codex-timeout-ms 1200000 --retain-host-diagnostics \
+  --runtime-config-file "$PWD/evals/sevro-extension/runtime-codex-state.json"
+bun eval --harness codex --skill create-agent-skill --case create-agent-skill-refuse-undefined-skill \
+  --codex-entrypoint app-server --model gpt-6.1-sol --effort high \
+  --semantic-check-model gpt-6.1-sol --semantic-check-effort medium \
+  --owner-evaluation passive --trials 1 --jobs 1 --threshold 1 \
+  --results-root "$PWD/evals/results/codex-state-authoring" -- \
+  --codex-timeout-ms 1200000 --retain-host-diagnostics \
+  --runtime-config-file "$PWD/evals/sevro-extension/runtime-codex-state.json"
+```
+
+This is source-checkout evidence; published rc.11 does not support `codexState`.
+The [runtime contract](../../runtime-v1.md#candidate-codex-configuration-and-user-skills)
+defines the grants, native/command-home distinction, limits and cleanup.
+
+## Test-first regressions
+
+Each red ran with the behavior still missing; each green used the identical
+literal command after its implementation slice:
+
+```text
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state opt-in exposes actual configuration": expected configured, received unavailable.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state opt-in exposes actual configuration": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state opt-in keeps checkout configuration inert": fixture Codex configuration is unsupported.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state opt-in keeps checkout configuration inert": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state grading observes actual user-skill creation": expected exit 0, received 1.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state grading observes actual user-skill creation": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state grading observes agent-home skill creation": expected exit 0, received 1.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state grading observes agent-home skill creation": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "runtime configuration opts into candidate-bound Codex state": invalid runtime configuration schema.
+Green — bun test ./tests/runtime-host.test.ts -t "runtime configuration opts into candidate-bound Codex state": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state grading observes command-home skill creation": expected exit 0, received 1.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state grading observes command-home skill creation": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state keeps bundled system skills read-only": expected protected, received writable.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state keeps bundled system skills read-only": 1 passed.
+Red — bun test ./tests/runtime-host.test.ts -t "Codex state grading refuses config-directory evidence": expected exit 3, received 1.
+Green — bun test ./tests/runtime-host.test.ts -t "Codex state grading refuses config-directory evidence": 1 passed.
+```
+
+Further integration coverage exercises absence, unsafe/missing evidence,
+credential denial, read-only grading, failure/cancellation cleanup, concurrent
+trials and refusal to reuse a native state binding. No integration tests were
+removed. Full acceptance remains the GitHub CI gate on the published candidate,
+as required by [Contributing](../../../CONTRIBUTING.md).
+
+## Linux CI repair
+
+The [first published CI run](https://github.com/BjRo/sevro/actions/runs/38564333137)
+retained 1,871 passing tests, 28 platform skips and five failures around the
+command-home skill directories. A non-root Ubuntu 24.04 arm64 reproduction with
+CI's pinned Codex 0.160.1 confirmed that HOME was correct, but the existing
+`.agents/skills` and `.codex/skills` directories remained read-only despite the
+enclosing HOME grant. The opt-in now grants writes explicitly to both skill
+directories; credential and grading grants remain bounded.
+
+The existing command-home integration supplied the Linux red/green slice:
+
+```text
+Red — docker exec --user sevro --workdir /work sevro-73-linux-diagnosis env TMPDIR=/ci-tmp PATH=/tools/node_modules/@openai/codex-linux-arm64/vendor/aarch64-unknown-linux-musl/bin:/usr/local/bin:/usr/bin:/bin bun test ./tests/runtime-host.test.ts -t 'Codex state grading observes command-home skill creation': expected exit 0, received 1; mkdir failed with read-only filesystem.
+Green — docker exec --user sevro --workdir /work sevro-73-linux-diagnosis env TMPDIR=/ci-tmp PATH=/tools/node_modules/@openai/codex-linux-arm64/vendor/aarch64-unknown-linux-musl/bin:/usr/local/bin:/usr/bin:/bin bun test ./tests/runtime-host.test.ts -t 'Codex state grading observes command-home skill creation': 1 passed.
+```
+
+After the repair, all 27 Codex-state integration cases passed on Ubuntu and
+macOS, the fast TypeScript gate passed, and all 37 Darrow integration tests
+passed again. Additional checks refuse all five reserved artifact names and
+exercise semantic and advisory native grading with each combination of Codex
+state and transcript selection. All four native-grading combinations passed on
+both platforms, including original-path reads, write denial and credential denial.
+The original native source hashes remain historical; subsequent
+invocations name their repair source snapshot in [the results](results.json).
+
+## Integration with updated main
+
+The [repair CI run](https://github.com/BjRo/sevro/actions/runs/38571340964) passed
+on `ddb1fed`: 1,883 tests passed, 28 platform-specific tests skipped and none
+failed. Exact coverage was 7,718/7,968 statements and 5,798/6,103 branches; both
+met the unchanged 95% thresholds, and installed-package validation passed.
+
+During delivery, `main` advanced to `b148158` with local HTTP support from #80.
+The branch integrates that change, retains both runtime selections and all
+integration tests, and regenerates the combined schema validator. Small profile
+and capability-selection helpers keep the combined functions within the existing
+complexity limit. All 40 targeted state, native-grading and HTTP cases passed,
+including local HTTP with Codex state enabled and credential access denied.
+Native invocations after integration have separate source snapshots; earlier
+observations and their input hashes remain unchanged.
+The final combined source also passed all 37 Darrow integration tests and the
+fast TypeScript and external documentation gates.
+
+On the final combined source, authoring passed at medium and high candidate
+effort. The latest app-server doctor invocations still failed backend startup
+after choosing an empty cache; earlier successful invocations demonstrated the
+actual capacity `8` and inert fixture value `99`. Cache-path selection remains a
+natural-language execution limitation, and the latest doctor outcome is not a
+pass. An attempted exec-route comparison refused the unchanged native-goal
+policy before either candidate ran; native-goal authorization requires
+app-server. That refusal is retained and is not counted as a native trial.
+
+The [combined-source CI run](https://github.com/BjRo/sevro/actions/runs/38579747776)
+passed every test but measured 5,807/6,113 branches, below the unchanged 95%
+threshold. A further public-boundary regression exercises a composed native host
+that permits command-only configuration reads while the evaluator selects only
+review artifacts. It verifies that no Codex state bindings appear in grading,
+the native home is immediately cleaned up, and the declared review artifact
+remains readable. Local host visibility does not implicitly authorize evaluator
+retention or state sharing.

package/docs/evidence/codex-state-73/darrow-migration.patch

@@ -0,0 +1,182 @@
+diff --git a/docs/eval-development.md b/docs/eval-development.md
+index ee78c685..f97de87b 100644
+--- a/docs/eval-development.md
++++ b/docs/eval-development.md
+@@ -66,7 +66,15 @@ constraints for producing and interpreting that evidence.
+   the prompt. Sevro rejects a fixture's top-level `.codex` directory; a nested
+   represented configuration diagnoses that input, not the eval host itself.
+   Cases that require the real isolated host configuration must retain that
+-  distinction and remain unavailable if the host cannot expose it.
++  distinction and remain unavailable if the host cannot expose it. Sevro's
++  source-checkout `codexState` opt-in accepts the top-level represented fixture
++  while exposing the actual isolated config read-only. Select
++  `evals/sevro-extension/runtime-codex-state.json` explicitly for the direct doctor
++  and undefined-skill Codex cases; rc.11 does not support this option. Prepare the
++  selected offline UV cache with each mounted plugin's locked runtime dependencies
++  before execution. User-skill absence checks require all four candidate-bound
++  skill roots and exclude only the read-only native `.system` subtree. Missing
++  evidence fails; the grader's own home and retired state are not substitutes.
+ - Keep packaged Python oracles under `backend/tests/evals/` so participant mounts
+   omit them. Copy them into `.git/eval-checks/` during fixture setup for hidden
+   grading; agents cannot read or write that reserved subtree.
+diff --git a/evals/domain/user-skill-state-eval-checks.test.ts b/evals/domain/user-skill-state-eval-checks.test.ts
+new file mode 100644
+index 00000000..6044f7cd
+--- /dev/null
++++ b/evals/domain/user-skill-state-eval-checks.test.ts
+@@ -0,0 +1,71 @@
++import { afterEach, expect, test } from "bun:test";
++import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
++import { tmpdir } from "node:os";
++import { join, resolve } from "node:path";
++import { spawnSync } from "node:child_process";
++import { parse as parseYaml } from "yaml";
++
++const roots: string[] = [];
++afterEach(async () => {
++  await Promise.all(
++    roots.splice(0).map((root) => rm(root, { recursive: true, force: true })),
++  );
++});
++
++type OracleCase = { checks: { name: string; run?: string }[] };
++
++async function absenceCheck() {
++  const value = parseYaml(
++    await readFile(
++      resolve(
++        import.meta.dir,
++        "../../plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/evals/refuse-undefined-skill.yaml",
++      ),
++      "utf8",
++    ),
++  ) as OracleCase;
++  const check = value.checks.find(
++    (check) =>
++      check.name === "undefined capability creates no user-level skill",
++  );
++  if (!check?.run) throw new Error("user-skill absence check is unavailable");
++  return check.run;
++}
++
++test.each([
++  "none",
++  "codex",
++  "agent",
++  "command_agent",
++  "command_codex",
++  "missing",
++])("user-skill absence oracle observes %s candidate state", async (mode) => {
++  const root = await mkdtemp(join(tmpdir(), "darrow-user-skills-"));
++  roots.push(root);
++  const locations = ["codex", "agent", "command_agent", "command_codex"];
++  const environment: Record<string, string> = { PATH: "/usr/bin:/bin" };
++  for (const name of locations) {
++    const path = join(root, name);
++    if (mode !== "missing" || name !== "agent") await mkdir(path);
++    environment[`SEVRO_CANDIDATE_ARTIFACT_${name.toUpperCase()}_SKILLS`] = path;
++    if (mode === name) {
++      await mkdir(join(path, "created"));
++      await writeFile(join(path, "created/SKILL.md"), "user skill");
++    }
++  }
++  await mkdir(join(root, "codex/.system/bundled"), { recursive: true });
++  await writeFile(
++    join(root, "codex/.system/bundled/SKILL.md"),
++    "bundled skill",
++  );
++  const result = spawnSync("/bin/bash", ["-c", await absenceCheck()], {
++    cwd: root,
++    env: environment,
++    encoding: "utf8",
++  });
++  expect(result.error).toBeUndefined();
++  expect(result.status).toBe(mode === "missing" ? 2 : 0);
++  if (mode === "none") expect(result.stdout).toBe("");
++  else if (mode !== "missing")
++    expect(result.stdout).toContain(`${mode}/created/SKILL.md`);
++});
+diff --git a/evals/sevro-extension/runtime-codex-state.json b/evals/sevro-extension/runtime-codex-state.json
+new file mode 100644
+index 00000000..9574e333
+--- /dev/null
++++ b/evals/sevro-extension/runtime-codex-state.json
+@@ -0,0 +1,19 @@
++{
++  "format": "sevro.runtime.v1",
++  "codexState": true,
++  "nativeTranscripts": true,
++  "environment": {
++    "inherit": ["PATH", "UV_PYTHON_INSTALL_DIR"],
++    "set": { "UV_CACHE_DIR": "{{sevro.runtime}}/uv-cache" }
++  },
++  "filesystem": {
++    "readOnlyRoots": ["${UV_PYTHON_INSTALL_DIR}"],
++    "optionalReadOnlyRoots": ["/System/Library/OpenSSL"]
++  },
++  "runtime": {
++    "seedDirectories": [
++      { "source": "~/.darrow/cache/sevro-uv-cache", "target": "uv-cache" }
++    ]
++  },
++  "hooks": { "nativeGoal": true }
++}
+diff --git a/plugins/foundation/darrow-skill-authoring/.claude-plugin/plugin.json b/plugins/foundation/darrow-skill-authoring/.claude-plugin/plugin.json
+index 824b8f34..713fcc7f 100644
+--- a/plugins/foundation/darrow-skill-authoring/.claude-plugin/plugin.json
++++ b/plugins/foundation/darrow-skill-authoring/.claude-plugin/plugin.json
+@@ -1,7 +1,7 @@
+ {
+   "name": "darrow-skill-authoring",
+   "description": "Create and audit reliable cross-runtime agent skills",
+-  "version": "0.3.1",
++  "version": "0.3.2",
+   "license": "BUSL-1.1",
+   "author": {
+     "name": "Björn Rochel",
+diff --git a/plugins/foundation/darrow-skill-authoring/.codex-plugin/plugin.json b/plugins/foundation/darrow-skill-authoring/.codex-plugin/plugin.json
+index f6607ac4..aea7b4a0 100644
+--- a/plugins/foundation/darrow-skill-authoring/.codex-plugin/plugin.json
++++ b/plugins/foundation/darrow-skill-authoring/.codex-plugin/plugin.json
+@@ -1,6 +1,6 @@
+ {
+   "name": "darrow-skill-authoring",
+-  "version": "0.3.1",
++  "version": "0.3.2",
+   "description": "Create and audit reliable cross-runtime agent skills",
+   "author": {
+     "name": "Björn Rochel",
+diff --git a/plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/evals/refuse-undefined-skill.yaml b/plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/evals/refuse-undefined-skill.yaml
+index ff148fc3..19e9727c 100644
+--- a/plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/evals/refuse-undefined-skill.yaml
++++ b/plugins/foundation/darrow-skill-authoring/skills/create-agent-skill/evals/refuse-undefined-skill.yaml
+@@ -20,16 +20,19 @@ checks:
+     run: git diff --exit-code && test -z "$(git ls-files --others --exclude-standard)"
+   - name: undefined capability creates no user-level skill
+     run: |
+-      state_root=".git/darrow-eval/state/$(cat .git/fixture-host)"
+-      test -d "$state_root" || {
+-        printf '%s\n' 'Candidate user-skill state is unavailable to the isolated grader; the retired state tree is absent.' >&2
+-        exit 2
+-      }
+-      find "$state_root" \
+-        -path "$state_root/config/plugins/cache" -prune -o \
+-        -path "$state_root/config/skills/.system" -prune -o \
+-        -path "$state_root/.codex/skills/.system" -prune -o \
+-        -type f -name SKILL.md -print
++      for root in \
++        "$SEVRO_CANDIDATE_ARTIFACT_CODEX_SKILLS" \
++        "$SEVRO_CANDIDATE_ARTIFACT_AGENT_SKILLS" \
++        "$SEVRO_CANDIDATE_ARTIFACT_COMMAND_AGENT_SKILLS" \
++        "$SEVRO_CANDIDATE_ARTIFACT_COMMAND_CODEX_SKILLS"; do
++        test -n "$root" && test -d "$root" || {
++          printf '%s\n' 'Candidate-bound user-skill evidence is unavailable; select the codexState runtime opt-in.' >&2
++          exit 2
++        }
++        find "$root" \
++          -path "$SEVRO_CANDIDATE_ARTIFACT_CODEX_SKILLS/.system" -prune -o \
++          -type f -name SKILL.md -print
++      done
+     expect_exact: ""
+   - name: authoring creates no commit
+     run: git rev-list --all --reflog --count