@bjoernrochel/sevro 0.1.0-rc.2 → 0.1.0-rc.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +170 -0
- package/LICENSE +63 -136
- package/README.md +83 -66
- package/docs/README.md +44 -0
- package/docs/acknowledgements.md +14 -0
- package/docs/architecture.md +146 -0
- package/docs/assets/README.md +27 -0
- package/docs/assets/sevro-logo.png +0 -0
- package/docs/assets/sevro-symbol.svg +1 -0
- package/docs/assets/sevro-wordmark.svg +1 -0
- package/docs/builtin-graders-v1.md +41 -13
- package/docs/codex-host.md +184 -52
- package/docs/contracts.md +19 -0
- package/docs/creating-cases.md +139 -0
- package/docs/development-cli.md +190 -30
- package/docs/documentation-quality.md +61 -0
- package/docs/evaluation-workflow.md +110 -0
- package/docs/evidence/code-mode-controls-71.md +64 -0
- package/docs/evidence/code-mode-programs-85.md +64 -0
- package/docs/evidence/codex-state-73/README.md +204 -0
- package/docs/evidence/codex-state-73/darrow-migration.patch +182 -0
- package/docs/evidence/codex-state-73/results.json +661 -0
- package/docs/evidence/local-http-74.md +104 -0
- package/docs/evidence/runtime-startup-30/README.md +185 -0
- package/docs/evidence/runtime-startup-30/after.json +319 -0
- package/docs/evidence/runtime-startup-30/baseline-timeout.jsonl +2 -0
- package/docs/evidence/runtime-startup-30/before.json +403 -0
- package/docs/evidence/runtime-startup-30/consumer-smoke.json +63 -0
- package/docs/evidence/runtime-startup-30/full-policy-diagnostic.jsonl +3 -0
- package/docs/evidence/runtime-startup-30/minimal-policy-diagnostic.jsonl +3 -0
- package/docs/evidence/runtime-startup-30/plugin-template-assessment.json +20 -0
- package/docs/evidence/runtime-startup-30/seedless-timeout.jsonl +2 -0
- package/docs/evidence/skill-reads-70/results.json +823 -0
- package/docs/evidence.md +21 -0
- package/docs/extension-protocol-v1.md +50 -16
- package/docs/fixture-cleanup-validation.md +3 -0
- package/docs/getting-started.md +102 -0
- package/docs/guide-evaluation.md +89 -0
- package/docs/guide-verification.json +239 -0
- package/docs/identity-v1.md +35 -1
- package/docs/installing.md +88 -0
- package/docs/licenses/BUSL-1.1-0.1.0-rc.2.txt +159 -0
- package/docs/licensing.md +42 -0
- package/docs/native-controls.md +76 -5
- package/docs/native-hosts.md +228 -0
- package/docs/reading-results.md +141 -0
- package/docs/recipes.md +489 -0
- package/docs/releases.md +81 -40
- package/docs/report-v1.md +29 -1
- package/docs/results-v1.md +180 -0
- package/docs/run-settings-v1.md +147 -0
- package/docs/running-evaluations.md +31 -0
- package/docs/runtime-v1.md +614 -0
- package/docs/specs/repository-guide.md +58 -0
- package/docs/troubleshooting.md +30 -0
- package/docs/typescript-quality.md +234 -0
- package/docs/usability-study.md +114 -0
- package/examples/basic/host.ts +2 -2
- package/examples/recipes/candidate.ts +26 -0
- package/examples/recipes/cases/coding-change.json +21 -0
- package/examples/recipes/cases/host-tools.json +21 -0
- package/examples/recipes/cases/mise.json +21 -0
- package/examples/recipes/cases/node.json +21 -0
- package/examples/recipes/cases/python-uv.json +21 -0
- package/examples/recipes/cases/semantic.json +17 -0
- package/examples/recipes/coding-change/check.ts +23 -0
- package/examples/recipes/extension-artifacts.ts +65 -0
- package/examples/recipes/extension-methods.ts +174 -0
- package/examples/recipes/extension.ts +37 -0
- package/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
- package/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
- package/examples/recipes/repository-skill/README.md +3 -0
- package/examples/recipes/run-inputs.ts +188 -0
- package/examples/recipes/run-options.ts +131 -0
- package/examples/recipes/run.ts +63 -0
- package/examples/recipes/runtime/empty.json +1 -0
- package/examples/recipes/runtime/goals.json +1 -0
- package/examples/recipes/runtime/mise.json +6 -0
- package/examples/recipes/runtime/path.json +1 -0
- package/examples/recipes/runtime/plugins.json +4 -0
- package/examples/recipes/runtime/python-uv.json +10 -0
- package/examples/recipes/semantic-judge.ts +29 -0
- package/examples/templates/coding/adapter.ts +17 -0
- package/examples/templates/coding/case.json +15 -0
- package/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
- package/examples/templates/repository-skill/extension-data.ts +64 -0
- package/examples/templates/repository-skill/extension.ts +31 -0
- package/examples/templates/response/adapter.ts +12 -0
- package/examples/templates/response/case.json +13 -0
- package/package.json +39 -6
- package/schemas/report-v1.schema.json +9 -0
- package/schemas/run-evidence-v1.schema.json +2 -1
- package/schemas/runtime-v1.schema.json +193 -0
- package/src/advisory.ts +10 -5
- package/src/candidate-artifacts.ts +208 -0
- package/src/candidate-codex-state.ts +185 -0
- package/src/case-validation.ts +105 -0
- package/src/cli/cancellation.ts +20 -0
- package/src/cli/command.ts +81 -0
- package/src/cli/completion.ts +15 -0
- package/src/cli/demo.ts +59 -0
- package/src/cli/evaluation.ts +55 -0
- package/src/cli/extension.ts +79 -0
- package/src/cli/help.ts +222 -0
- package/src/cli/hosts.ts +59 -0
- package/src/cli/init-settings.ts +30 -0
- package/src/cli/init.ts +111 -0
- package/src/cli/invocation.ts +627 -0
- package/src/cli/loading.ts +151 -0
- package/src/cli/preparation.ts +163 -0
- package/src/cli/presentation.ts +91 -0
- package/src/cli/progress.ts +6 -0
- package/src/cli/run-manifest.ts +133 -0
- package/src/cli/run-options.ts +101 -0
- package/src/cli/run-overrides.ts +83 -0
- package/src/cli/settings.ts +160 -0
- package/src/cli/summary.ts +43 -0
- package/src/cli.ts +13 -843
- package/src/contract-types.ts +94 -0
- package/src/engine.ts +19 -2373
- package/src/evaluation/advisory.ts +150 -0
- package/src/evaluation/check-plan.ts +510 -0
- package/src/evaluation/collection.ts +88 -0
- package/src/evaluation/context.ts +10 -0
- package/src/evaluation/grading-host.ts +39 -0
- package/src/evaluation/grading.ts +258 -0
- package/src/evaluation/invocation-plan.ts +548 -0
- package/src/evaluation/plan.ts +736 -0
- package/src/evaluation/progress.ts +19 -0
- package/src/evaluation/resources.ts +241 -0
- package/src/evaluation/run.ts +206 -0
- package/src/{trial-scheduler.ts → evaluation/scheduler.ts} +3 -1
- package/src/evaluation/semantic.ts +312 -0
- package/src/evaluation/trial.ts +866 -0
- package/src/evaluation/types.ts +207 -0
- package/src/evidence/host-results.ts +92 -0
- package/src/evidence/retained-artifacts.ts +18 -0
- package/src/evidence/trial-writer.ts +203 -0
- package/src/evidence/usage.ts +17 -0
- package/src/extension-client.ts +228 -148
- package/src/extension-session.ts +326 -195
- package/src/fixtures/advisory.ts +338 -0
- package/src/fixtures/bin.ts +57 -0
- package/src/{fixture-cleanup.ts → fixtures/cleanup.ts} +13 -7
- package/src/fixtures/create.ts +187 -0
- package/src/fixtures/generated.ts +300 -0
- package/src/fixtures/git-hooks.ts +24 -0
- package/src/fixtures/preparation.ts +270 -0
- package/src/{repository-fixture.ts → fixtures/repository.ts} +207 -85
- package/src/fixtures/setup.ts +224 -0
- package/src/fixtures/tools.ts +88 -0
- package/src/fixtures/workspace-state.ts +212 -0
- package/src/generated/cli-result.cjs +2 -0
- package/src/generated/cli-result.d.cts +3 -0
- package/src/generated/extension.cjs +2 -0
- package/src/generated/extension.d.cts +3 -0
- package/src/generated/report.cjs +2 -0
- package/src/generated/report.d.cts +3 -0
- package/src/generated/run-evidence.cjs +2 -0
- package/src/generated/run-evidence.d.cts +3 -0
- package/src/generated/runtime.cjs +2 -0
- package/src/generated/runtime.d.cts +3 -0
- package/src/graders/artifact.ts +56 -23
- package/src/graders/git-head.ts +79 -75
- package/src/graders/output.ts +222 -140
- package/src/graders/semantic.ts +102 -43
- package/src/graders/shell.ts +379 -207
- package/src/host-diagnostic-redaction.ts +52 -0
- package/src/host-diagnostics.ts +319 -0
- package/src/hosts/claude-continuation.ts +63 -18
- package/src/hosts/claude-credential.ts +43 -62
- package/src/hosts/claude-events.ts +86 -51
- package/src/hosts/claude-keychain.ts +59 -0
- package/src/hosts/claude-native-goal.ts +57 -33
- package/src/hosts/claude-process.ts +111 -0
- package/src/hosts/claude-repository-invocation.ts +208 -117
- package/src/hosts/claude-settings.ts +240 -21
- package/src/hosts/claude-skill-activations.ts +166 -0
- package/src/hosts/claude-state.ts +403 -0
- package/src/hosts/claude-tool-calls.ts +171 -106
- package/src/hosts/claude.ts +561 -451
- package/src/hosts/codex-app-server.ts +326 -178
- package/src/hosts/codex-configuration.ts +44 -18
- package/src/hosts/codex-events.ts +97 -60
- package/src/hosts/codex-exec-controls.ts +516 -0
- package/src/hosts/codex-invocation.ts +70 -0
- package/src/hosts/codex-marketplace.ts +283 -0
- package/src/hosts/codex-native-calls.ts +546 -500
- package/src/hosts/codex-process.ts +202 -0
- package/src/hosts/codex-profile.ts +257 -28
- package/src/hosts/codex-session-files.ts +106 -0
- package/src/hosts/codex-state.ts +492 -0
- package/src/hosts/codex.ts +783 -832
- package/src/hosts/isolation-common.ts +94 -0
- package/src/hosts/isolation-roots.ts +54 -27
- package/src/hosts/isolation.ts +10 -0
- package/src/hosts/linux-sandbox.ts +159 -0
- package/src/hosts/mac-sandbox.ts +187 -61
- package/src/hosts/native-controls.ts +152 -62
- package/src/hosts/process-lifecycle.ts +37 -0
- package/src/hosts/protected-worktrees.ts +51 -19
- package/src/hosts/skill-hook-catalog.ts +131 -0
- package/src/hosts/skill-hook-handler.ts +374 -0
- package/src/hosts/skill-hooks.ts +580 -0
- package/src/hosts/workspace-fingerprint.ts +55 -35
- package/src/identity.ts +102 -48
- package/src/instrumentation.ts +109 -48
- package/src/native-transcript-state.ts +131 -0
- package/src/native-transcript-view.ts +90 -0
- package/src/native-transcripts.ts +286 -0
- package/src/process-lifecycle.ts +150 -0
- package/src/provenance.ts +161 -81
- package/src/prune.ts +215 -0
- package/src/report.ts +172 -140
- package/src/resolved-case.ts +33 -0
- package/src/results-lock.ts +29 -0
- package/src/results.ts +133 -86
- package/src/run-owner.ts +154 -63
- package/src/runtime/config.ts +456 -0
- package/src/runtime/discovery-apple.ts +90 -0
- package/src/runtime/discovery-providers.ts +63 -0
- package/src/runtime/discovery.ts +70 -0
- package/src/runtime/hook-processes.ts +97 -0
- package/src/runtime/hooks.ts +476 -0
- package/src/runtime/paths.ts +313 -0
- package/src/runtime/query.ts +85 -0
- package/src/runtime/seeds.ts +134 -0
- package/src/runtime/shared-cache.ts +143 -0
- package/src/runtime/state.ts +143 -0
- package/src/schema-types.ts +198 -0
- package/src/schema.ts +15 -19
- package/src/transcript-retention.ts +146 -0
- package/src/value-guards.ts +20 -0
- package/src/advisory-fixture.ts +0 -282
- package/src/fixture-bin.ts +0 -87
- package/src/fixture-setup.ts +0 -163
- package/src/generated-fixture.ts +0 -220
- package/src/git-hooks.ts +0 -63
- package/src/hosts/claude-nested-skills.ts +0 -218
- package/src/hosts/codex-command-output.ts +0 -399
- package/src/hosts/codex-executor-command.ts +0 -195
- package/src/hosts/codex-skill-reads.ts +0 -391
- package/src/preparation.ts +0 -211
|
@@ -0,0 +1,614 @@
|
|
|
1
|
+
# Runtime configuration v1
|
|
2
|
+
|
|
3
|
+
Configure access to existing host tools, private state, reusable caches, native goals,
|
|
4
|
+
and selected plugin hooks in `<project-root>/sevro.json`.
|
|
5
|
+
|
|
6
|
+
`--runtime-config-file /absolute/path/runtime.json` selects a replacement file.
|
|
7
|
+
Sevro reads exactly one file before execution, including dry runs. It does not
|
|
8
|
+
merge files, search parent directories, or read a candidate's copy. An absent
|
|
9
|
+
default file preserves the previous isolation defaults. An absent explicit
|
|
10
|
+
file, malformed configuration, unsupported field, or invalid selected input
|
|
11
|
+
fails with exit `64` before a candidate starts. Files must be regular,
|
|
12
|
+
non-symlink files of at most 64 KiB.
|
|
13
|
+
|
|
14
|
+
The [JSON schema](../schemas/runtime-v1.schema.json) has format
|
|
15
|
+
`sevro.runtime.v1`. All properties except `format` are optional.
|
|
16
|
+
Start with the smallest file:
|
|
17
|
+
|
|
18
|
+
```json
|
|
19
|
+
{ "format": "sevro.runtime.v1" }
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Then add `environment.inherit: ["PATH"]` for installed tools. Follow the
|
|
23
|
+
[runnable runtime recipes](recipes.md#add-host-tool-access-one-capability-at-a-time)
|
|
24
|
+
to add Node, Python/UV, mise, goals and selected plugin hooks separately.
|
|
25
|
+
The following composed example is for a workflow that already needs all these
|
|
26
|
+
capabilities:
|
|
27
|
+
|
|
28
|
+
```json
|
|
29
|
+
{
|
|
30
|
+
"format": "sevro.runtime.v1",
|
|
31
|
+
"environment": {
|
|
32
|
+
"inherit": ["PATH"],
|
|
33
|
+
"set": {
|
|
34
|
+
"UV_CACHE_DIR": "{{sevro.runtime}}/uv-cache",
|
|
35
|
+
"DARROW_CACHE_DIR": "{{sevro.runtime}}/darrow-cache"
|
|
36
|
+
}
|
|
37
|
+
},
|
|
38
|
+
"runtime": {
|
|
39
|
+
"sharedCacheDirectories": [
|
|
40
|
+
{
|
|
41
|
+
"source": "../public-uv-cache",
|
|
42
|
+
"target": "uv-cache",
|
|
43
|
+
"roles": ["candidate", "hooks"]
|
|
44
|
+
}
|
|
45
|
+
]
|
|
46
|
+
},
|
|
47
|
+
"hooks": {
|
|
48
|
+
"nativeGoal": true,
|
|
49
|
+
"plugins": ["darrow-review", "darrow-tdd"]
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Prepare the example's `public-uv-cache` directory outside the project, or
|
|
55
|
+
omit `runtime` when no prepared cache is needed. Selected plugin names must be
|
|
56
|
+
present in the case's prepared mounts; remove names that the case does not mount.
|
|
57
|
+
`DARROW_CACHE_DIR` is an ordinary explicit variable in this example, not a
|
|
58
|
+
Sevro-specific cache option.
|
|
59
|
+
|
|
60
|
+
## Environment and host tools
|
|
61
|
+
|
|
62
|
+
`environment.inherit` snapshots the named variables from the invoking process.
|
|
63
|
+
A named variable must exist. `environment.set` supplies explicit strings and
|
|
64
|
+
takes precedence over an inherited value. These values are retained in evidence:
|
|
65
|
+
declare only non-secret values. Credential names and runner-owned variables
|
|
66
|
+
such as `HOME`, temporary-directory variables, `CODEX_HOME`, `CLAUDE_CONFIG_DIR`,
|
|
67
|
+
shell startup overrides, subprocess injection flags, and `UV_OFFLINE` are refused.
|
|
68
|
+
|
|
69
|
+
An inherited or explicit `PATH` must contain nonempty absolute entries. For an
|
|
70
|
+
inherited `PATH`, Sevro removes entries whose directory, installation prefix, or
|
|
71
|
+
symlinked tools overlap credential roots, including `~/.codex` and `~/.claude`.
|
|
72
|
+
This lets evaluations invoked from a Codex CLI session use ordinary host tools
|
|
73
|
+
without manually removing injected paths. Lexical paths and canonical targets
|
|
74
|
+
are checked, including absent credential paths. The remaining entries keep their
|
|
75
|
+
original order; the effective `PATH` is retained in runtime evidence. A `PATH`
|
|
76
|
+
with no remaining entries fails rather than selecting fallback tools.
|
|
77
|
+
|
|
78
|
+
An explicit `environment.set.PATH` remains authoritative and conflicting grants
|
|
79
|
+
fail, even when `PATH` is also inherited. Source and evaluator overlaps, invalid
|
|
80
|
+
entries, and unreadable unprotected directories remain errors for either form.
|
|
81
|
+
Missing unprotected entries are inert. Sevro grants read-only access to existing directories,
|
|
82
|
+
resolved symlinked tools, and bounded installation prefixes above `bin`.
|
|
83
|
+
An inaccessible individual symlink target is inert; an unreadable PATH directory
|
|
84
|
+
still fails configuration. Explicitly declared read roots must remain readable.
|
|
85
|
+
It does not widen a `bin` grant to the whole home, `/usr`, `/usr/local`, or
|
|
86
|
+
`/opt/homebrew`. Direct versioned mise paths can reuse their existing Node or
|
|
87
|
+
Python installations. A mise shim may also need explicitly declared mise data
|
|
88
|
+
and configuration; inheriting `PATH` does not change the shim's own discovery
|
|
89
|
+
rules. Fixture binaries keep precedence over declared host tools.
|
|
90
|
+
|
|
91
|
+
When `PATH` is declared, Sevro also queries available installation metadata
|
|
92
|
+
before execution. Homebrew's `--prefix` and `--cellar` identify its existing
|
|
93
|
+
`Cellar`, `opt`, and `etc` support trees on macOS and Linux. The whole Homebrew
|
|
94
|
+
prefix is not granted. On macOS, selected Apple tool shims (including Git,
|
|
95
|
+
Python, and compiler tools) use `xcode-select --print-path` to locate their
|
|
96
|
+
developer support tree. A declared `DEVELOPER_DIR` participates in that query;
|
|
97
|
+
an unrelated ambient override does not. Installation locations are not assumed.
|
|
98
|
+
Each query has a three-second deadline and an 8 KiB combined output limit.
|
|
99
|
+
Absent providers and the known no-active-Apple-installation result add no roots.
|
|
100
|
+
Other query failures, invalid metadata, or a missing declared developer override
|
|
101
|
+
fail before execution. Diagnostics identify the provider and operation without
|
|
102
|
+
retaining arbitrary command output.
|
|
103
|
+
|
|
104
|
+
Declare supporting directories outside `PATH` with `filesystem.readOnlyRoots`:
|
|
105
|
+
|
|
106
|
+
```json
|
|
107
|
+
{
|
|
108
|
+
"format": "sevro.runtime.v1",
|
|
109
|
+
"environment": { "inherit": ["PATH", "UV_PYTHON_INSTALL_DIR"] },
|
|
110
|
+
"filesystem": { "readOnlyRoots": ["${UV_PYTHON_INSTALL_DIR}"] }
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Set `UV_PYTHON_INSTALL_DIR` in the invoking environment before using this example.
|
|
115
|
+
Filesystem declarations resolve relative to the selected file. `~/` names the
|
|
116
|
+
operator's real home; `${NAME}` expands only a declared environment variable.
|
|
117
|
+
Existing directory paths and symlink targets are canonicalized. Reads under
|
|
118
|
+
the real home are allowed only through bounded runtime grants. Source and
|
|
119
|
+
configuration worktrees, evaluator inputs, credentials, results, run ownership
|
|
120
|
+
state, and peer fixtures remain protected. A conflicting or broad grant fails.
|
|
121
|
+
Managed workspaces share a private parent that protects sibling workspaces,
|
|
122
|
+
including those created after launch. Independently prepared workspaces require
|
|
123
|
+
explicit protection for other private inputs. Claude hosts require Claude Code
|
|
124
|
+
2.1.257 or later for managed-workspace isolation; see [checked versions](typescript-quality.md#environment-and-ci).
|
|
125
|
+
|
|
126
|
+
Use `filesystem.optionalReadOnlyRoots` for supporting directories that may not
|
|
127
|
+
be installed on every host, with the same path rules as `readOnlyRoots`:
|
|
128
|
+
|
|
129
|
+
```json
|
|
130
|
+
{
|
|
131
|
+
"format": "sevro.runtime.v1",
|
|
132
|
+
"filesystem": {
|
|
133
|
+
"optionalReadOnlyRoots": [
|
|
134
|
+
"../public-python-installations",
|
|
135
|
+
"../public-tool-cache"
|
|
136
|
+
]
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
With `sevro.json` at the repository root, these paths name sibling support
|
|
142
|
+
directories outside the protected repository. Keep them bounded to public tool
|
|
143
|
+
data; directories inside the repository remain protected even when absent.
|
|
144
|
+
|
|
145
|
+
Only absent directories are skipped. Invalid variables or paths, unreadable
|
|
146
|
+
directories, regular files, and protected overlaps remain errors. Required roots
|
|
147
|
+
still fail when absent. Effective roots retain both lexical directory paths and
|
|
148
|
+
canonical targets, and both are checked against protected inputs. Native aliases
|
|
149
|
+
inside granted support trees (for example, `opt/package` pointing into `Cellar`)
|
|
150
|
+
remain usable. Codex may canonicalize a standalone symlink-root permission under
|
|
151
|
+
protected home, denying its lexical lookup; declare the intended bounded parent
|
|
152
|
+
support tree and its target rather than relying on that standalone alias.
|
|
153
|
+
|
|
154
|
+
On Linux, Codex starts candidate commands from an empty filesystem with bounded
|
|
155
|
+
read mounts. Sevro keeps a normal home ancestor denied by that empty-root policy
|
|
156
|
+
rather than adding a native deny mount that would hide its read-only descendants.
|
|
157
|
+
Protected source and credential descendants remain explicitly denied, and no
|
|
158
|
+
home ancestor receives content access. If the operator's home lies inside a
|
|
159
|
+
minimal system read tree such as `/usr` or `/etc`, Sevro retains its explicit deny;
|
|
160
|
+
the native preflight may refuse that overlapping setup rather than expose home
|
|
161
|
+
contents.
|
|
162
|
+
|
|
163
|
+
## Local HTTP test servers
|
|
164
|
+
|
|
165
|
+
Sevro supports local HTTP servers in isolated Codex candidate
|
|
166
|
+
commands and `sevro.shell` graders through an explicit operator selection:
|
|
167
|
+
|
|
168
|
+
```json
|
|
169
|
+
{
|
|
170
|
+
"format": "sevro.runtime.v1",
|
|
171
|
+
"network": { "localHttp": true }
|
|
172
|
+
}
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Add this property to the selected `sevro.json`, or select a replacement with
|
|
176
|
+
`--runtime-config-file`. Declare the existing Node/Python installations through
|
|
177
|
+
the environment and filesystem options above. The selection is snapshotted and
|
|
178
|
+
enters evaluation identity. Absent or `false` preserves existing permissions.
|
|
179
|
+
Hooks and semantic/advisory hosts keep their existing network permissions; this
|
|
180
|
+
option applies to Codex candidates and shell grading.
|
|
181
|
+
|
|
182
|
+
On macOS, shell isolation permits IP socket binding, loopback inbound traffic,
|
|
183
|
+
and outbound connections to loopback addresses. Wildcard listeners such as
|
|
184
|
+
Node's `server.listen(0)` work with loopback clients. Other outbound IP traffic
|
|
185
|
+
remains denied. On Linux, shell checks retain their private network namespace
|
|
186
|
+
with usable loopback and no external route.
|
|
187
|
+
|
|
188
|
+
Codex uses its managed command network proxy with `allow_local_binding = true`,
|
|
189
|
+
upstream proxy chaining disabled, and only `localhost`, `127.0.0.1`, and `::1`
|
|
190
|
+
allowlisted. Its native sandbox permits local binding and loopback traffic while
|
|
191
|
+
refusing direct outbound TCP. The native proxy may permit DNS queries; public
|
|
192
|
+
HTTP destinations are not allowlisted. The candidate can reach other services
|
|
193
|
+
on the available loopback interface, so choose this option for evaluations
|
|
194
|
+
that need that access. It does not provide a separate per-trial loopback service
|
|
195
|
+
namespace on macOS.
|
|
196
|
+
|
|
197
|
+
The supported host boundary is the native Codex sandbox and its experimental
|
|
198
|
+
`network_proxy` feature, documented in the [OpenAI permission reference](https://learn.chatgpt.com/docs/permissions).
|
|
199
|
+
Setting `allow_local_binding` while leaving command networking disabled still
|
|
200
|
+
denies `listen()` on macOS. Integration tests execute the installed native
|
|
201
|
+
Codex sandbox, a wildcard HTTP listener and loopback client, and an outbound TCP
|
|
202
|
+
denial probe. Host versions and actual upstream evaluation outcomes belong in
|
|
203
|
+
[validation evidence](evidence.md), rather than being inferred from a profile's
|
|
204
|
+
text. An unsupported host must not be interpreted as a passing behavior test.
|
|
205
|
+
|
|
206
|
+
## macOS ancestor metadata
|
|
207
|
+
|
|
208
|
+
Sevro's macOS outer sandbox for isolated shell checks and authorized command
|
|
209
|
+
hooks permits metadata reads on the exact ancestor paths of existing validated
|
|
210
|
+
grants. This includes declared read-only support trees, private writable
|
|
211
|
+
runtimes, and bounded transcript, candidate artifact, and managed workspace
|
|
212
|
+
access. The permissions are derived automatically from those grants; no
|
|
213
|
+
additional configuration is required. They let libc `realpath()` canonicalize
|
|
214
|
+
paths inside a permitted subtree.
|
|
215
|
+
|
|
216
|
+
Ancestor directory metadata, such as existence, type, ownership, permissions,
|
|
217
|
+
and timestamps, is visible. Directory listings, ancestor file contents, and
|
|
218
|
+
ungranted sibling metadata and contents remain denied under protected roots.
|
|
219
|
+
Writes remain limited by the existing writable grants; read-only grants stay
|
|
220
|
+
read-only. This metadata exception does not grant access to an ancestor subtree
|
|
221
|
+
or change network isolation.
|
|
222
|
+
|
|
223
|
+
## Private writable state
|
|
224
|
+
|
|
225
|
+
Codex candidate commands can write the isolated fixture's `.git` directory,
|
|
226
|
+
including its index, refs, objects, and task-owned admission locks. Sevro grants
|
|
227
|
+
that exact path explicitly because Codex's workspace permission otherwise keeps
|
|
228
|
+
Git metadata read-only. Real commits and publication to a fixture-local remote
|
|
229
|
+
need no additional runtime option. This grant does not authorize writes to
|
|
230
|
+
source repositories, grading inputs, host authentication or configuration, or
|
|
231
|
+
peer trials. Command network access remains disabled unless the operator selects
|
|
232
|
+
the [local HTTP option](#local-http-test-servers).
|
|
233
|
+
An existing fixture `.git` must be a physical directory. Symlinks and Git
|
|
234
|
+
directory pointer files are refused before launch because their metadata may
|
|
235
|
+
belong to a protected source repository rather than the isolated trial.
|
|
236
|
+
|
|
237
|
+
`runtime.seedDirectories` copies prepared public dependencies into a private
|
|
238
|
+
slot for each execution role. `source` uses the filesystem path rules above;
|
|
239
|
+
`target` is a single lowercase slot name, up to 64 characters, with letters,
|
|
240
|
+
digits, and hyphens. `home` and `tmp` are reserved. Targets must be unique.
|
|
241
|
+
An optional nonempty `roles` array selects any of `candidate`, `checks`, `hooks`,
|
|
242
|
+
`semantic`, and `advisory`. For example, `"roles": ["candidate", "checks", "hooks"]`
|
|
243
|
+
keeps dependency slots out of semantic and advisory runtimes. Without `roles`,
|
|
244
|
+
candidate, hook, artifact semantic, and advisory roles receive private copies.
|
|
245
|
+
Shell checks require explicit `"checks"` selection; a non-UV grader receives no
|
|
246
|
+
implicit UV/Python cache. Existing configurations whose shell checks need seeded
|
|
247
|
+
dependencies must add `"checks"` to `roles`. Response-only
|
|
248
|
+
semantic grading omits implicit seeds; include `"semantic"` explicitly in `roles`
|
|
249
|
+
when that route needs dependencies. Declared host tools remain available to shell checks.
|
|
250
|
+
|
|
251
|
+
Each source tree is bounded to 16,384 entries and 512 MiB. Escaping symlinks,
|
|
252
|
+
directory cycles, dangling links, and special files are refused. File content
|
|
253
|
+
and modes are fingerprinted at configuration load. At first use, the runner
|
|
254
|
+
captures the complete bounded tree in private memory and verifies it against that
|
|
255
|
+
digest. Each selected role materializes separate files from those verified bytes;
|
|
256
|
+
later roles and trials sharing the same policy reuse the captured bytes. Source
|
|
257
|
+
changes before first capture fail. Source changes after capture do not change
|
|
258
|
+
that evaluation's dependencies. A new policy snapshot starts a new capture.
|
|
259
|
+
No mutable hardlinks or writable filesystem snapshot are shared. The memory
|
|
260
|
+
snapshot is eligible for collection when its policy seed is no longer reachable;
|
|
261
|
+
retaining large policies can retain up to the configured per-seed byte bounds.
|
|
262
|
+
Source caches are never a writable runtime. Use prepared public caches rather
|
|
263
|
+
than credential-bearing homes or evaluator directories.
|
|
264
|
+
|
|
265
|
+
### Reuse a shared dependency cache
|
|
266
|
+
|
|
267
|
+
`runtime.sharedCacheDirectories` opts into persistent writable caches without
|
|
268
|
+
copying or hashing their contents for each trial. Each entry requires `source`,
|
|
269
|
+
`target`, and a nonempty `roles` array drawn from the same five roles above.
|
|
270
|
+
The source must already be an owned, readable, writable directory. Relative paths
|
|
271
|
+
resolve beside the runtime file. Sevro resolves declaration symlinks to their
|
|
272
|
+
canonical source and revalidates that directory before execution.
|
|
273
|
+
Targets use the seed slot rules and must be unique across seeds and shared caches.
|
|
274
|
+
Sources cannot overlap each other, private seed sources, declared read-only
|
|
275
|
+
inputs, evaluated project/source trees, results, credentials, or native private state.
|
|
276
|
+
|
|
277
|
+
Select only roles that actually use UV, for example `"roles": ["candidate", "hooks"]`.
|
|
278
|
+
If a grader also uses UV, add `"checks"` to that same entry; both roles use the
|
|
279
|
+
same cache. Each selected private slot aliases the original directory, with a
|
|
280
|
+
narrow sandbox write grant. Other roles receive no slot or source grant.
|
|
281
|
+
Shared content survives successful, failed, and cancelled trials and later runs.
|
|
282
|
+
Sevro removes only disposable role state, never the declared shared source.
|
|
283
|
+
|
|
284
|
+
Prepare UV packages once outside evaluation, point `UV_CACHE_DIR` at
|
|
285
|
+
`{{sevro.runtime}}/uv-cache`, and reuse the existing Python installation through
|
|
286
|
+
declared read-only tool roots. Set `UV_PYTHON_DOWNLOADS=never` to require that
|
|
287
|
+
installation. Sevro keeps `UV_OFFLINE=1`; a missing dependency fails rather than
|
|
288
|
+
downloading during a trial. Project virtual environments and temporary files
|
|
289
|
+
remain disposable; UV's install/link mode can still allocate files in a virtual
|
|
290
|
+
environment. The shared option removes the full cache copy per trial.
|
|
291
|
+
|
|
292
|
+
Shared caches are mutable across trials, roles, and runs. Use them only for public
|
|
293
|
+
dependencies and accept that a participant can change them for later participants.
|
|
294
|
+
Use isolated seeds when dependency independence is required. Configuration identity
|
|
295
|
+
binds the source, slot, and roles, but excludes mutable cache contents; equal
|
|
296
|
+
identities therefore do not prove identical shared-cache bytes. Sevro does not
|
|
297
|
+
age-delete shared caches. Manage their size between evaluations with the cache
|
|
298
|
+
tool, such as `uv cache prune --cache-dir /absolute/public-uv-cache`.
|
|
299
|
+
|
|
300
|
+
`{{sevro.runtime}}` in an environment value resolves to that role's writable
|
|
301
|
+
root. Codex commands receive a `TMPPREFIX` inside their private `TMPDIR`,
|
|
302
|
+
so zsh here-documents use their existing private temporary storage while shared
|
|
303
|
+
host temporary directories remain denied. `TMPPREFIX` is runner-owned and cannot
|
|
304
|
+
be inherited or set through runtime configuration.
|
|
305
|
+
Sevro supplies private `HOME` and temporary storage and sets `UV_OFFLINE=1`
|
|
306
|
+
and `PYTHONDONTWRITEBYTECODE=1`. Fixture setup uses the candidate commands'
|
|
307
|
+
private `HOME`, so `$HOME` paths initialize the same state the candidate will
|
|
308
|
+
use. Its temporary directory stays disposable.
|
|
309
|
+
Candidate, hook, checker, semantic, and advisory
|
|
310
|
+
state are separate. With a runtime policy, native candidate, semantic, and
|
|
311
|
+
advisory roots live in a unique runner-owned host directory outside the Git
|
|
312
|
+
fixture. The root contains `home`, `tmp`, private seed slots and selected shared-cache links; plugins can
|
|
313
|
+
use paths such as `~/.darrow/reviews` without creating repository-local review
|
|
314
|
+
state. Command runtime directories are separate from host authentication,
|
|
315
|
+
configuration, and live transcript sources. Checker and hook private state also lives
|
|
316
|
+
outside candidate-controlled workspace state. Shared sources remain at their declared locations. Private roots survive native
|
|
317
|
+
continuation turns and children, then are removed with their owning trial or
|
|
318
|
+
host state after owned hooks terminate. Checker home, temporary storage, and copied
|
|
319
|
+
seeds are removed after grading even when the fixture is retained after failure
|
|
320
|
+
or cancellation. Results and evidence are preserved. Opted-in candidate artifact runtimes
|
|
321
|
+
remain alive through grading as described below. Runs with neither fixture setup
|
|
322
|
+
nor a selected runtime policy
|
|
323
|
+
retain their previous writable-state locations.
|
|
324
|
+
|
|
325
|
+
Claude refuses managed workspace or runtime namespaces overlapping its
|
|
326
|
+
implicit `/tmp/claude` or `/private/tmp/claude` write roots, including canonical
|
|
327
|
+
symlink targets. A conflicting `TMPDIR` placement fails before native launch;
|
|
328
|
+
choose a temporary base outside those cache trees.
|
|
329
|
+
|
|
330
|
+
The same declared tool environment reaches native candidates, isolated shell
|
|
331
|
+
checks, and native semantic/advisory hosts. Candidate plugin-hook authority is
|
|
332
|
+
not forwarded to grading hosts. Injected adapters receive the policy as request
|
|
333
|
+
data and remain trusted code responsible for their own execution boundary.
|
|
334
|
+
|
|
335
|
+
## Candidate artifact access
|
|
336
|
+
|
|
337
|
+
Use `runtime.artifactDirectories` to let graders read selected artifacts in the
|
|
338
|
+
candidate's private home after execution.
|
|
339
|
+
|
|
340
|
+
```json
|
|
341
|
+
{
|
|
342
|
+
"format": "sevro.runtime.v1",
|
|
343
|
+
"runtime": {
|
|
344
|
+
"artifactDirectories": [{ "name": "reviews", "path": ".darrow/reviews" }]
|
|
345
|
+
}
|
|
346
|
+
}
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
The example permits only the `~/.darrow/reviews` subtree, which may be initialized
|
|
350
|
+
by trusted fixture setup or created by the candidate.
|
|
351
|
+
Its original absolute path reaches isolated shell checks and native
|
|
352
|
+
semantic/advisory commands as `SEVRO_CANDIDATE_ARTIFACT_REVIEWS`. Graders keep
|
|
353
|
+
their own private `HOME`; use the supplied variable to read candidate evidence.
|
|
354
|
+
For each declared name, the variable is `SEVRO_CANDIDATE_ARTIFACT_` followed by
|
|
355
|
+
the name in uppercase. Artifact content is untrusted candidate evidence.
|
|
356
|
+
|
|
357
|
+
At most 16 directories may be declared. Names are unique, contain lowercase
|
|
358
|
+
letters, digits and underscores, start with a letter, and are at most 64
|
|
359
|
+
characters. Paths are relative to candidate home, are at most 4,096 characters,
|
|
360
|
+
and cannot contain empty, `.` or `..` components, backslashes or control
|
|
361
|
+
characters. Whole-home and absolute grants are refused. Each subtree is bounded
|
|
362
|
+
to 16,384 entries, including its root. Missing directories, symlinked ancestors,
|
|
363
|
+
descendant symlinks, hardlinks and special files make grading an error; missing
|
|
364
|
+
evidence cannot produce a pass.
|
|
365
|
+
|
|
366
|
+
Initialize an empty directory or seed prior Review state in the extension's
|
|
367
|
+
trusted `fixtureSetup` command before the candidate starts. For example:
|
|
368
|
+
|
|
369
|
+
```sh
|
|
370
|
+
mkdir -p "$HOME/.darrow/reviews"
|
|
371
|
+
printf '%s\n' '{"verdict":"passed"}' > "$HOME/.darrow/reviews/prior.json"
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
Setup and candidate commands share this private trial home, including when no
|
|
375
|
+
artifact grants or runtime file are selected. Graders keep their own homes and
|
|
376
|
+
receive only declared read-only artifact grants. To test missing reports or
|
|
377
|
+
absence of new artifacts, initialize the directory without writing a report;
|
|
378
|
+
an empty safe directory reaches ordinary task checks. Missing required reports
|
|
379
|
+
can then fail those checks, while negative cases can assert the directory is
|
|
380
|
+
empty. Sevro does not automatically create declared artifact directories: if
|
|
381
|
+
setup and candidate both leave one absent, or replace it with an unsafe path,
|
|
382
|
+
grading still errors. Setup state is temporary and is removed with its trial,
|
|
383
|
+
including setup failure, cancellation, and dry runs.
|
|
384
|
+
|
|
385
|
+
The grant permits reads and denies writes, creation, deletion and renaming.
|
|
386
|
+
Credentials, other home contents and peer role/trial state stay protected.
|
|
387
|
+
Ordinary `filesystem.readOnlyRoots` cannot select this private-state exception.
|
|
388
|
+
Artifact grants and native transcript access are independent opt-ins.
|
|
389
|
+
|
|
390
|
+
The trial allocates the retained runtime before fixture setup and
|
|
391
|
+
removes it after grading, advisory review and evidence persistence, including
|
|
392
|
+
failure and cancellation paths. Review state remains outside the Git fixture.
|
|
393
|
+
These artifacts are temporary; declaring access does not copy their bytes into
|
|
394
|
+
retained run evidence. Failed-fixture retention does not retain this runtime.
|
|
395
|
+
Standalone bundled-host calls retain their immediate cleanup behavior.
|
|
396
|
+
|
|
397
|
+
Trusted injected adapters receive a runner-owned `candidateRuntimeRoot` on the
|
|
398
|
+
candidate request and a named `candidateArtifactRoots` path map on grading
|
|
399
|
+
requests. Candidate adapters opting into this feature must put their private
|
|
400
|
+
runtime at the supplied root. The lifetime binding is process-local; bundled
|
|
401
|
+
hosts validate it before use. Grading adapters must preserve the read-only
|
|
402
|
+
boundary. The directory declarations enter runtime configuration identity;
|
|
403
|
+
per-trial absolute paths do not.
|
|
404
|
+
|
|
405
|
+
## Candidate Codex configuration and user skills
|
|
406
|
+
|
|
407
|
+
Set top-level `"codexState": true` to expose the bundled Codex candidate's actual
|
|
408
|
+
isolated configuration and user-skill state. This opt-in is independent of
|
|
409
|
+
`nativeTranscripts` and ordinary
|
|
410
|
+
candidate-home artifact declarations:
|
|
411
|
+
|
|
412
|
+
```json
|
|
413
|
+
{
|
|
414
|
+
"format": "sevro.runtime.v1",
|
|
415
|
+
"codexState": true
|
|
416
|
+
}
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
Candidate commands receive the real isolated `CODEX_HOME`. They may read its
|
|
420
|
+
`config.toml` and write non-system skills below `CODEX_HOME/skills`. The native
|
|
421
|
+
process's separate user-skill directory is available as
|
|
422
|
+
`SEVRO_CODEX_AGENT_SKILLS`; it names that process's `HOME/.agents/skills`.
|
|
423
|
+
Bundled `CODEX_HOME/skills/.system` skills remain read-only. Authentication,
|
|
424
|
+
other private siblings, evaluator inputs and peer requests remain denied.
|
|
425
|
+
Configuration remains read-only, and the preflight still checks credential denial.
|
|
426
|
+
|
|
427
|
+
A physical fixture-local `.codex` directory is accepted only with this opt-in.
|
|
428
|
+
Sevro marks the isolated project untrusted in its generated Codex configuration,
|
|
429
|
+
so project-local configuration, hooks and rules remain represented inputs rather
|
|
430
|
+
than effective host settings. The fixture is preserved. Diagnose the absolute
|
|
431
|
+
`CODEX_HOME/config.toml`, not the checkout's `.codex/config.toml`. The runner's
|
|
432
|
+
selected concurrency limit is written into the private config; an omitted limit
|
|
433
|
+
still means the native host default, not a fabricated configured capacity.
|
|
434
|
+
CLI model/effort and vetted-hook overrides keep their existing precedence; the
|
|
435
|
+
file alone does not establish every effective native default or child route.
|
|
436
|
+
|
|
437
|
+
After execution, isolated shell and native semantic/advisory graders receive
|
|
438
|
+
read-only bindings to these original candidate paths:
|
|
439
|
+
|
|
440
|
+
| Environment variable | Original candidate path |
|
|
441
|
+
| ----------------------------------------------- | ------------------------------------------ |
|
|
442
|
+
| `SEVRO_CANDIDATE_ARTIFACT_CODEX_CONFIG` | Native `CODEX_HOME/config.toml` (one file) |
|
|
443
|
+
| `SEVRO_CANDIDATE_ARTIFACT_CODEX_SKILLS` | Native `CODEX_HOME/skills` |
|
|
444
|
+
| `SEVRO_CANDIDATE_ARTIFACT_AGENT_SKILLS` | Native process `HOME/.agents/skills` |
|
|
445
|
+
| `SEVRO_CANDIDATE_ARTIFACT_COMMAND_AGENT_SKILLS` | Command `HOME/.agents/skills` |
|
|
446
|
+
| `SEVRO_CANDIDATE_ARTIFACT_COMMAND_CODEX_SKILLS` | Command `HOME/.codex/skills` |
|
|
447
|
+
|
|
448
|
+
Native and command homes are separate. The runner initializes these actual
|
|
449
|
+
skill locations before the candidate starts and retains their resulting state;
|
|
450
|
+
it does not create replacement empty directories for grading. Graders keep their
|
|
451
|
+
own homes and must use these bindings. To assert absence, require every selected
|
|
452
|
+
directory to exist, then search for `SKILL.md`, excluding only the native Codex
|
|
453
|
+
root's read-only `.system` subtree. Missing evidence is an error. Do not search
|
|
454
|
+
the grader's home or retired fixture state.
|
|
455
|
+
|
|
456
|
+
The five artifact names are reserved. Each binding belongs to one active trial
|
|
457
|
+
and one native request; reuse by another host request is refused. The same
|
|
458
|
+
canonical ancestor, entry-limit, symlink, hardlink and special-file checks as
|
|
459
|
+
ordinary artifacts apply. Configuration cannot be replaced by a directory grant.
|
|
460
|
+
The native home is retained through grading and removed with its trial, including
|
|
461
|
+
failure and cancellation paths. Standalone host calls clean up immediately.
|
|
462
|
+
These grants do not persist native files in run evidence; private absolute paths
|
|
463
|
+
do not enter identity, while the `codexState` selection does. Other candidate
|
|
464
|
+
hosts cannot supply this Codex evidence and fail grading when it is requested.
|
|
465
|
+
|
|
466
|
+
For the Darrow doctor and undefined-skill cases, select this configuration with
|
|
467
|
+
`--runtime-config-file`, preserve the doctor fixture's checkout value `99`, and
|
|
468
|
+
replace the retired-state absence check with the four skill-directory bindings
|
|
469
|
+
above. Keep the original repository, no-commit, activation and semantic checks.
|
|
470
|
+
|
|
471
|
+
## Goals and plugin hooks
|
|
472
|
+
|
|
473
|
+
`hooks.nativeGoal` enables Claude's native goal Stop-hook path while preserving
|
|
474
|
+
subscription/OAuth and API-key authentication. For a Codex CLI candidate it
|
|
475
|
+
selects `app-server` when no entrypoint was supplied. Explicit `exec` conflicts
|
|
476
|
+
with a native-goal policy. Programmatic Codex callers must construct an
|
|
477
|
+
`app-server` host. Native goal observations report actual host status; enabling
|
|
478
|
+
the policy does not itself prove that a goal was created or completed.
|
|
479
|
+
|
|
480
|
+
`hooks.plugins` authorizes all command hooks from the named prepared plugins.
|
|
481
|
+
Sevro validates and copies mounts, filters unselected hooks, and wraps selected
|
|
482
|
+
handlers in a credential-free macOS or Linux sandbox. Dependency runtime and
|
|
483
|
+
hook isolation preparation occur only when an authorized command handler is
|
|
484
|
+
encountered. No plugins, unselected hooks, or an authorized plugin without
|
|
485
|
+
command handlers require hook dependency hashing or copying. Configuration-load
|
|
486
|
+
fingerprinting still binds all declared seed contents to identity. Plugin
|
|
487
|
+
validation, unsupported execution refusals, and filtering still apply without
|
|
488
|
+
hook runtime preparation. It covers default hook files,
|
|
489
|
+
manifest path/inline declarations in both host formats, and Markdown skill or
|
|
490
|
+
agent frontmatter. Unsupported handler types and exec-form arguments fail
|
|
491
|
+
instead of being skipped. The wrapper denies network access and protected
|
|
492
|
+
roots and owns asynchronous hook cleanup. Native timeout/cancellation stops
|
|
493
|
+
owned hook descendants before private state is removed.
|
|
494
|
+
|
|
495
|
+
Prepared native plugin projections live outside the fixture in the current
|
|
496
|
+
host request's private native state. Curated hook read grants validate the
|
|
497
|
+
canonical projection layout, ownership, private state permissions, and binding
|
|
498
|
+
to that request. Only the selected plugin subtree is readable; it stays
|
|
499
|
+
read-only, and credentials, private siblings, and peer requests remain denied.
|
|
500
|
+
Ordinary filesystem declarations cannot select this exception. Codex validates
|
|
501
|
+
and filters installed plugins before enabling its runner-owned skill collector,
|
|
502
|
+
including requests without a runtime policy. Plugin command handlers still
|
|
503
|
+
require explicit hook authority. Projections are removed with their owning host
|
|
504
|
+
request.
|
|
505
|
+
|
|
506
|
+
Auxiliary `.mcp.json` and `.lsp.json` files and plugin mods are unsupported under
|
|
507
|
+
hook authority. Manifest MCP/LSP declarations are removed from curated mounts.
|
|
508
|
+
|
|
509
|
+
Plugin-only settings are required when goals or custom hooks are enabled;
|
|
510
|
+
`--claude-project-settings` conflicts with that policy. Keep repository-skill
|
|
511
|
+
cases on their explicit project-settings route with hooks disabled. Unrelated
|
|
512
|
+
user settings and hooks are not imported. Sevro does not use Claude's `--bare`
|
|
513
|
+
mode, which would change authentication and behavior.
|
|
514
|
+
|
|
515
|
+
## Native transcript access
|
|
516
|
+
|
|
517
|
+
Set top-level `"nativeTranscripts": true` to enable live, read-only access to
|
|
518
|
+
the current role's full native root and subagent transcript tree. Absent or
|
|
519
|
+
`false` leaves this access and capture disabled. The selection is snapshotted
|
|
520
|
+
with the runtime policy and enters configuration identity; temporary native
|
|
521
|
+
homes and transcript views do not enter that identity. Codex `exec` persists
|
|
522
|
+
sessions when enabled, including runs without continuation.
|
|
523
|
+
|
|
524
|
+
Commands discover their own tree through `SEVRO_NATIVE_TRANSCRIPT_ROOT`.
|
|
525
|
+
Codex commands also receive runner-owned `CODEX_HOME`; native
|
|
526
|
+
`CODEX_THREAD_ID` selects `sessions/**/rollout-*-<thread>.jsonl`. Read the latest
|
|
527
|
+
native `turn_context` for that session's `model` and `effort`. Claude shell
|
|
528
|
+
commands receive a credential-free `CLAUDE_CONFIG_DIR` view whose `projects/`
|
|
529
|
+
tree contains `projects/*/<CLAUDE_CODE_SESSION_ID>.jsonl` and subagent files.
|
|
530
|
+
The latest native assistant `message.model` records the observed model.
|
|
531
|
+
Claude supplies `CLAUDE_CODE_SESSION_ID` and `CLAUDE_EFFORT` itself; Sevro does
|
|
532
|
+
not replace these with the root's requested route. Requested CLI effort is
|
|
533
|
+
configuration, and is not proof of an effective child effort. Missing native
|
|
534
|
+
route evidence stays unknown.
|
|
535
|
+
|
|
536
|
+
Observed with Claude Code 2.1.284: a native Agent's Bash may receive the parent's
|
|
537
|
+
`CLAUDE_CODE_SESSION_ID` and no `CLAUDE_EFFORT`. In that situation the environment
|
|
538
|
+
pair selects the parent transcript. Analyze the child's native JSONL and
|
|
539
|
+
`.meta.json` sidecar to establish its own model/session association; the
|
|
540
|
+
parent's model is not the child's observed model, and missing effort remains
|
|
541
|
+
unknown. Sevro preserves these native values without synthesizing a child
|
|
542
|
+
context or copying the root's requested effort into the child.
|
|
543
|
+
|
|
544
|
+
Native hosts keep their credential-bearing homes. Claude's host writes through
|
|
545
|
+
a runner-created, canonical-bound `projects` link into the isolated source;
|
|
546
|
+
runner-owned Bash and zsh startup files supply the shell compatibility view.
|
|
547
|
+
These sources and startup files sit outside candidate-writable state. Commands
|
|
548
|
+
cannot modify transcripts or read credential/settings siblings, evaluator
|
|
549
|
+
inputs, or peer role/trial state. A fixed private namespace is protected before
|
|
550
|
+
commands start, including peers created later. Pre-existing namespaces must be
|
|
551
|
+
owned private directories; links and unsafe permissions fail. Ordinary
|
|
552
|
+
filesystem declarations cannot authorize access to this namespace.
|
|
553
|
+
Namespace read roots and seed sources, including declaration symlinks into it,
|
|
554
|
+
are refused before private trees are hashed or copied.
|
|
555
|
+
On Linux, Codex keeps this namespace hidden through its empty-root policy and
|
|
556
|
+
mounts only runner-owned session/helper trees and retained views. It avoids an
|
|
557
|
+
ancestor deny mount that would mask these narrower read-only grants. Ordinary
|
|
558
|
+
declarations cannot select this exception; parent content, credentials, and
|
|
559
|
+
present or future peer state receive no grants.
|
|
560
|
+
|
|
561
|
+
After candidate execution, isolated shell checks and semantic/advisory hosts
|
|
562
|
+
receive `SEVRO_CANDIDATE_TRANSCRIPTS`, a separate read-only retained view with
|
|
563
|
+
`index.json` and `codex/` or `claude/` native relative file paths. This does not
|
|
564
|
+
replace a grading host's own native home or live tree. Trusted injected adapters
|
|
565
|
+
receive the same view through request `candidateTranscriptRoot`. The temporary
|
|
566
|
+
view is removed after grading; stable retained artifact references remain in
|
|
567
|
+
run evidence. Extensions and external benchmarks consume the
|
|
568
|
+
[native transcript bundles](results-v1.md#native-transcript-bundles).
|
|
569
|
+
|
|
570
|
+
## Evidence and compatibility
|
|
571
|
+
|
|
572
|
+
`configuration.redacted.runtimePolicy` retains the selected environment,
|
|
573
|
+
canonical read roots, seed digests, shared-cache declarations, and hook declaration. The engine snapshots
|
|
574
|
+
this input once; it participates in `configurationDigest` and evaluation
|
|
575
|
+
identity. Symbolic runtime placeholders remain symbolic in identity, while
|
|
576
|
+
execution uses per-role paths. This binds declared setup and seed contents;
|
|
577
|
+
it does not claim that every external installation file or OS dependency is
|
|
578
|
+
immutable.
|
|
579
|
+
|
|
580
|
+
The retained policy also records metadata providers, their selected executables
|
|
581
|
+
and effective support roots, plus skipped optional paths. These values and the
|
|
582
|
+
actual effective roots enter identity; private execution paths and external tool
|
|
583
|
+
file contents do not. The same discovery and skip evidence reaches
|
|
584
|
+
`sevro.host.runtime` observations.
|
|
585
|
+
|
|
586
|
+
Bundled hosts supply `sevro.host.runtime` with the applied policy digest, role,
|
|
587
|
+
environment names, read roots, seed slots, and selected shared-cache declarations.
|
|
588
|
+
Shared entries contain no content digest. `sevro.host.hooks` records source
|
|
589
|
+
and effective plugin digests plus observed executions and their exit codes.
|
|
590
|
+
An empty execution list does not prove hook activation. Partial lifecycle
|
|
591
|
+
evidence remains partial. Cases can require these observations; missing required
|
|
592
|
+
evidence cannot become a pass. Goal and hook success remain separate from the
|
|
593
|
+
task verdict.
|
|
594
|
+
|
|
595
|
+
`--toolchain-bin-dir` and `--claude-uv-cache-dir` are deprecated. They keep their
|
|
596
|
+
previous behavior with a warning when no runtime file is selected. Combining
|
|
597
|
+
them with a runtime file fails before execution. Migrate to `environment`,
|
|
598
|
+
`filesystem`, and either private seed slots or shared-cache declarations; removal
|
|
599
|
+
is no earlier than `0.2.0`. The deprecated UV option continues to copy privately.
|
|
600
|
+
See the [CLI reference](development-cli.md), [native hosts](native-hosts.md),
|
|
601
|
+
and [identity contract](identity-v1.md).
|
|
602
|
+
|
|
603
|
+
Programmatic consumers can import `RuntimeConfiguration`, `RuntimePolicy`, and
|
|
604
|
+
`loadRuntimeConfiguration` from `src/engine.ts`, then pass the resolved policy
|
|
605
|
+
to `runEvaluation` or a bundled host request. They must provide the same
|
|
606
|
+
protected-source declarations required by their execution route.
|
|
607
|
+
`loadRuntimeConfiguration` accepts those canonical protected roots as its third
|
|
608
|
+
argument, so provider queries themselves cannot run from protected source trees.
|
|
609
|
+
|
|
610
|
+
Derived grading policies retain omitted seed sources in `seedReadDenyRoots`.
|
|
611
|
+
Bundled hosts and checkers deny those roots even when no dependency slot is
|
|
612
|
+
materialized. Trusted injected adapters must preserve these source denials when
|
|
613
|
+
applying the request policy. This is derived programmatic policy metadata, not
|
|
614
|
+
a selectable `sevro.json` field.
|