@bjoernrochel/sevro 0.1.0-rc.12 → 0.1.0-rc.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +13 -2
- package/README.md +5 -0
- package/docs/README.md +4 -0
- package/docs/contracts.md +1 -0
- package/docs/creating-cases.md +9 -2
- package/docs/development-cli.md +93 -8
- package/docs/documentation-quality.md +12 -2
- package/docs/evaluation-workflow.md +107 -0
- package/docs/evidence/code-mode-programs-85.md +64 -0
- package/docs/evidence.md +1 -0
- package/docs/getting-started.md +23 -2
- package/docs/identity-v1.md +15 -1
- package/docs/installing.md +11 -11
- package/docs/native-controls.md +7 -3
- package/docs/native-hosts.md +17 -0
- package/docs/reading-results.md +13 -0
- package/docs/recipes.md +485 -0
- package/docs/releases.md +2 -1
- package/docs/report-v1.md +13 -1
- package/docs/results-v1.md +22 -0
- package/docs/run-settings-v1.md +147 -0
- package/docs/runtime-v1.md +21 -8
- package/docs/troubleshooting.md +12 -12
- package/docs/typescript-quality.md +38 -16
- package/docs/usability-study.md +114 -0
- package/examples/recipes/candidate.ts +26 -0
- package/examples/recipes/cases/coding-change.json +21 -0
- package/examples/recipes/cases/host-tools.json +21 -0
- package/examples/recipes/cases/mise.json +21 -0
- package/examples/recipes/cases/node.json +21 -0
- package/examples/recipes/cases/python-uv.json +21 -0
- package/examples/recipes/cases/semantic.json +17 -0
- package/examples/recipes/coding-change/check.ts +23 -0
- package/examples/recipes/extension-artifacts.ts +65 -0
- package/examples/recipes/extension-methods.ts +174 -0
- package/examples/recipes/extension.ts +37 -0
- package/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
- package/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
- package/examples/recipes/repository-skill/README.md +3 -0
- package/examples/recipes/run-inputs.ts +188 -0
- package/examples/recipes/run-options.ts +131 -0
- package/examples/recipes/run.ts +63 -0
- package/examples/recipes/runtime/empty.json +1 -0
- package/examples/recipes/runtime/goals.json +1 -0
- package/examples/recipes/runtime/mise.json +6 -0
- package/examples/recipes/runtime/path.json +1 -0
- package/examples/recipes/runtime/plugins.json +4 -0
- package/examples/recipes/runtime/python-uv.json +10 -0
- package/examples/recipes/semantic-judge.ts +29 -0
- package/examples/templates/coding/adapter.ts +17 -0
- package/examples/templates/coding/case.json +15 -0
- package/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
- package/examples/templates/repository-skill/extension-data.ts +64 -0
- package/examples/templates/repository-skill/extension.ts +31 -0
- package/examples/templates/response/adapter.ts +12 -0
- package/examples/templates/response/case.json +13 -0
- package/package.json +1 -1
- package/schemas/run-evidence-v1.schema.json +2 -1
- package/src/case-validation.ts +71 -48
- package/src/cli/command.ts +54 -3
- package/src/cli/completion.ts +12 -0
- package/src/cli/demo.ts +59 -0
- package/src/cli/evaluation.ts +9 -1
- package/src/cli/help.ts +187 -0
- package/src/cli/hosts.ts +21 -3
- package/src/cli/init-settings.ts +30 -0
- package/src/cli/init.ts +111 -0
- package/src/cli/invocation.ts +99 -78
- package/src/cli/loading.ts +16 -9
- package/src/cli/preparation.ts +2 -1
- package/src/cli/presentation.ts +23 -20
- package/src/cli/progress.ts +6 -0
- package/src/cli/run-manifest.ts +133 -0
- package/src/cli/run-options.ts +101 -0
- package/src/cli/run-overrides.ts +83 -0
- package/src/cli/settings.ts +160 -0
- package/src/cli/summary.ts +43 -0
- package/src/cli.ts +8 -1
- package/src/engine.ts +1 -0
- package/src/evaluation/plan.ts +8 -1
- package/src/evaluation/progress.ts +19 -0
- package/src/evaluation/run.ts +7 -0
- package/src/evaluation/trial.ts +22 -1
- package/src/evaluation/types.ts +12 -0
- package/src/generated/run-evidence.cjs +1 -1
- package/src/hosts/codex-command-output.ts +15 -2
- package/src/hosts/codex-exec-controls.ts +52 -4
- package/src/hosts/codex-native-calls.ts +42 -13
- package/src/hosts/codex-profile.ts +1 -0
- package/src/hosts/codex-skill-reads.ts +249 -26
- package/src/provenance.ts +24 -4
- package/src/report.ts +37 -2
- package/src/runtime/config.ts +1 -1
- package/src/schema-types.ts +1 -0
package/CONTRIBUTING.md
CHANGED
|
@@ -38,6 +38,13 @@ bun run test:docs-examples
|
|
|
38
38
|
```
|
|
39
39
|
|
|
40
40
|
The second command checks the tutorial in fresh consumer/contributor directories.
|
|
41
|
+
Its complete recipe checks also require UV, Python 3, mise, and an existing
|
|
42
|
+
`mise install node@24.13.0` installation; see [recipe prerequisites](docs/recipes.md).
|
|
43
|
+
CI explicitly provisions these tools. Missing prerequisites fail with a named
|
|
44
|
+
setup instruction rather than skipping a recipe.
|
|
45
|
+
To run recipe commands from this source checkout, set
|
|
46
|
+
`export SEVRO_RECIPES="$PWD/examples/recipes"` from the repository root, then
|
|
47
|
+
follow the [recipe commands](docs/recipes.md#prepare-a-recipe-project).
|
|
41
48
|
Native model trials need their own [credentials and prerequisites](docs/native-hosts.md).
|
|
42
49
|
|
|
43
50
|
## Quality contract and checks
|
|
@@ -68,6 +75,10 @@ Before committing, use `bun run check:typescript --fast` after installing the
|
|
|
68
75
|
locked dependencies. This runs inventory, schema, formatting, lint, typing and
|
|
69
76
|
documentation checks; it does not replace full acceptance. CI retains machine-readable coverage
|
|
70
77
|
reports under `.quality/` and exposes the stable `TypeScript quality` status.
|
|
78
|
+
The complete command's `static`, `docs`, `tests` and `package` phases run as
|
|
79
|
+
concurrent CI jobs. After frozen installation, use
|
|
80
|
+
`bun run check:typescript --phase <name>` to run one phase locally; it does not
|
|
81
|
+
replace acceptance across all four. See the [phase inventory](docs/typescript-quality.md).
|
|
71
82
|
|
|
72
83
|
Run `bun run check:docs:external` separately and [triage network failures](docs/documentation-quality.md#external-links).
|
|
73
84
|
Guide changes also need [native evals](docs/guide-evaluation.md).
|
|
@@ -84,8 +95,8 @@ repairing in-scope failures; it does not authorize merging or unrelated changes.
|
|
|
84
95
|
non-root user, and isolation prerequisites. Reproducing that environment locally
|
|
85
96
|
with Docker is optional for diagnosis and is not a publication prerequisite.
|
|
86
97
|
- After the local pre-commit checks, publish the authorized candidate to its PR
|
|
87
|
-
and require
|
|
88
|
-
delivery as complete. A local run, fast check, targeted suite, or historical
|
|
98
|
+
and require all four canonical `check:typescript` phases to succeed in GitHub CI
|
|
99
|
+
before reporting delivery as complete. A local run, fast check, targeted suite, or historical
|
|
89
100
|
result does not establish that the current Ubuntu gate passes.
|
|
90
101
|
- Keep the quality bar intact. Repair behavior or add meaningful coverage for
|
|
91
102
|
accepted behavior; do not lower thresholds, exempt authored sources, skip tests,
|
package/README.md
CHANGED
|
@@ -40,6 +40,11 @@ Continue with [your first evaluation](docs/getting-started.md): the exact comman
|
|
|
40
40
|
expected `passed` result, and retained evidence. Read [installation](docs/installing.md)
|
|
41
41
|
for updates, removal, and source checkouts.
|
|
42
42
|
|
|
43
|
+
The development checkout and freshly built packages provide a
|
|
44
|
+
[demo-to-iteration workflow](docs/evaluation-workflow.md): initialize real
|
|
45
|
+
assertions, prepare, execute, read progress and inspect the latest result.
|
|
46
|
+
The published `rc.11` package uses the installed tutorial above.
|
|
47
|
+
|
|
43
48
|
## Find your next task
|
|
44
49
|
|
|
45
50
|
The [documentation hub](docs/README.md) routes to running evaluations, authoring
|
package/docs/README.md
CHANGED
|
@@ -7,12 +7,15 @@ Start with the task you want to complete. These pages work without an agent.
|
|
|
7
7
|
| I want to… | Read |
|
|
8
8
|
| ---------------------------------------------- | -------------------------------------------------------- |
|
|
9
9
|
| Get a result without a model account | [Your first evaluation](getting-started.md) |
|
|
10
|
+
| Initialize, prepare, run and iterate | [Evaluation workflow](evaluation-workflow.md) |
|
|
10
11
|
| Install, update, remove, or use a checkout | [Installation](installing.md) |
|
|
11
12
|
| Run evaluations and select a host | [Running evaluations](running-evaluations.md) |
|
|
12
13
|
| Create cases and reusable policy | [Cases and extensions](creating-cases.md) |
|
|
14
|
+
| Copy a complete coding, skill or runtime setup | [Runnable recipes](recipes.md) |
|
|
13
15
|
| Interpret verdicts, evidence, and comparisons | [Reading results](reading-results.md) |
|
|
14
16
|
| Run Codex or Claude with isolation | [Native hosts](native-hosts.md) |
|
|
15
17
|
| Reuse host tools or enable goals and hooks | [Runtime configuration](runtime-v1.md) |
|
|
18
|
+
| Save recurring evaluation selections | [Run settings](run-settings-v1.md) |
|
|
16
19
|
| Resolve a symptom | [Troubleshooting](troubleshooting.md) |
|
|
17
20
|
| Trace runner lifecycle, ownership, and changes | [Architecture](architecture.md) |
|
|
18
21
|
| Change Sevro or its documentation | [Contributing](../CONTRIBUTING.md) |
|
|
@@ -35,6 +38,7 @@ There is no accepted Rust migration decision here;
|
|
|
35
38
|
[issue #2](https://github.com/BjRo/sevro/issues/2) tracks research.
|
|
36
39
|
|
|
37
40
|
Read [documentation quality](documentation-quality.md),
|
|
41
|
+
[the usability study protocol](usability-study.md),
|
|
38
42
|
[guide evaluation](guide-evaluation.md), [releases](releases.md), and
|
|
39
43
|
[acknowledgements](acknowledgements.md) for maintenance.
|
|
40
44
|
Return to the [Sevro introduction](../README.md).
|
package/docs/contracts.md
CHANGED
|
@@ -7,6 +7,7 @@ Read these human contracts and their schemas together:
|
|
|
7
7
|
- [Report v1](report-v1.md): summaries and comparison limitations.
|
|
8
8
|
- [Identity v1](identity-v1.md): versioned comparison dimensions and hashing.
|
|
9
9
|
- [Runtime configuration v1](runtime-v1.md): environment, tools, private state, goals, and hooks.
|
|
10
|
+
- [Run settings v1](run-settings-v1.md): recurring evaluation selections and CLI overrides.
|
|
10
11
|
- [Built-in graders v1](builtin-graders-v1.md): output, shell, Git HEAD, and semantic checks.
|
|
11
12
|
- [Schemas](../schemas/): machine-readable public forms.
|
|
12
13
|
|
package/docs/creating-cases.md
CHANGED
|
@@ -4,18 +4,25 @@ Start with a direct case for a prompt, fixture, and built-in success criteria.
|
|
|
4
4
|
Copy [the basic graded case](../examples/basic/graded.json), give it a stable
|
|
5
5
|
ID, and replace the prompt/check. Read [built-in graders](builtin-graders-v1.md)
|
|
6
6
|
for regex, JSON, and bounded schema assertions. No criteria means `not_assessed`.
|
|
7
|
+
For complete coding, repository-skill, runtime, semantic and extension projects,
|
|
8
|
+
follow the [runnable recipes](recipes.md), including their deliberate failures.
|
|
7
9
|
|
|
8
10
|
Run it with the [CLI](running-evaluations.md). Direct cases follow `ResolvedCase`
|
|
9
11
|
in [the engine](../src/engine.ts). Fixtures can provide inline files, generated
|
|
10
12
|
Git history, or a declared clean repository. The [CLI reference](development-cli.md)
|
|
11
13
|
owns required source roots/maps; an arbitrary returned path does not grant access.
|
|
12
14
|
|
|
15
|
+
Selecting `--project-root` identifies the project; it does not copy its contents
|
|
16
|
+
into a participant fixture. Declare the files or repository source the participant
|
|
17
|
+
should receive. The [run defaults reference](development-cli.md#run-defaults)
|
|
18
|
+
explains how project identity, fixtures and CLI path resolution interact.
|
|
19
|
+
|
|
13
20
|
## Evaluate your repository skills
|
|
14
21
|
|
|
15
22
|
Start with one representative task, your actual skill, and observable success
|
|
16
23
|
criteria. This example evaluates a documentation guide's answer with Codex.
|
|
17
24
|
Use [Sevro from `PATH`](installing.md#make-the-command-available).
|
|
18
|
-
Live runs require the [native host prerequisites](native-hosts.md),
|
|
25
|
+
Live runs require the [native host prerequisites](native-hosts.md), supported
|
|
19
26
|
isolation, an authenticated Codex installation, and your chosen model.
|
|
20
27
|
They consume that account's model quota.
|
|
21
28
|
|
|
@@ -109,7 +116,7 @@ prepares skill mounts and interprets complete native host evidence. The
|
|
|
109
116
|
[extension protocol](extension-protocol-v1.md) defines repository and plugin
|
|
110
117
|
invocations; the checkout's [guide evals](guide-evaluation.md) demonstrate the
|
|
111
118
|
approach. Check those capabilities against your exact installed release:
|
|
112
|
-
this checkout
|
|
119
|
+
this checkout can contain [changes beyond its published version](installing.md#published-package-and-checkout).
|
|
113
120
|
|
|
114
121
|
## Reusable project policy
|
|
115
122
|
|
package/docs/development-cli.md
CHANGED
|
@@ -11,11 +11,99 @@ grading, evidence, and extension lifecycle.
|
|
|
11
11
|
Examples assume [Sevro and Bun are on `PATH`](installing.md#make-the-command-available).
|
|
12
12
|
For a contributor checkout, use `bun src/cli.ts` in place of `sevro`.
|
|
13
13
|
|
|
14
|
+
Use `sevro --help`, `sevro run --help`, or `sevro report --help` for examples,
|
|
15
|
+
required options, and defaults without preparing a run. `sevro --version` prints
|
|
16
|
+
the installed package version. Invocation and configuration diagnostics go to
|
|
17
|
+
stderr in both output modes; `--json` keeps one `sevro.cli-result.v1` document
|
|
18
|
+
on stdout. Invalid case diagnostics name the input file and offending field.
|
|
19
|
+
|
|
20
|
+
<a id="run-defaults-in-this-checkout"></a>
|
|
21
|
+
|
|
22
|
+
## Run defaults
|
|
23
|
+
|
|
24
|
+
`sevro demo` owns a deterministic case, fixture and adapter. It needs no model
|
|
25
|
+
credentials and immediately prints an assessed verdict plus an evidence path in
|
|
26
|
+
`.sevro/results` beneath the invoking directory. Its temporary project and
|
|
27
|
+
configuration are independent of the caller's files and runtime policy.
|
|
28
|
+
|
|
29
|
+
An ordinary run needs a case source and candidate route. For example:
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
sevro run --case-file ./case.json --adapter-module ./host-adapter.ts
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
| Setting | Default |
|
|
36
|
+
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
|
|
37
|
+
| Condition | `passive` |
|
|
38
|
+
| Trial count | `1` |
|
|
39
|
+
| Pass threshold | `1`: all assessed trials must pass; any unassessed trial prevents a pass |
|
|
40
|
+
| Parallel trial limit (`--jobs`) | `3` |
|
|
41
|
+
| Project root | Invoking current directory |
|
|
42
|
+
| Results root | `<project-root>/.sevro/results` |
|
|
43
|
+
| Configuration root | Project root |
|
|
44
|
+
| Run-state root | Results root |
|
|
45
|
+
| Runtime policy | Discover `sevro.json` at project root; none when absent |
|
|
46
|
+
| Native binary | Resolve only the explicitly selected `codex` or `claude` from `PATH`; `--codex-bin` / `--claude-bin` override it |
|
|
47
|
+
| Native execution timeout | `600000` ms (10 minutes); Codex accepts `--codex-timeout-ms` |
|
|
48
|
+
| Codex entrypoint | `exec`; a native-goal runtime policy selects `app-server` |
|
|
49
|
+
| Codex agent concurrency | `agents.max_concurrent_threads_per_session` from `.codex/config.toml` at configuration root, otherwise the host default |
|
|
50
|
+
| Shell-check expected exit | `0` unless the check declares `expectedExitCode` |
|
|
51
|
+
| Shell-check timeout | `30000` ms unless the check declares `timeoutMs` (maximum `120000`) |
|
|
52
|
+
| Shell isolation, dry preparation, host diagnostic retention | Off |
|
|
53
|
+
|
|
54
|
+
Every run file/directory option accepts a relative path based on the invoking
|
|
55
|
+
current directory, including adapter modules, extension source files, native
|
|
56
|
+
binary overrides and protected roots. Sevro resolves them to canonical absolute
|
|
57
|
+
paths before constructing protection lists; missing output-directory suffixes
|
|
58
|
+
resolve through their nearest existing parent. A different `--project-root`
|
|
59
|
+
does not change that base. The default results directory is relative to the
|
|
60
|
+
selected project root. Paths inside runtime policy files retain their
|
|
61
|
+
[runtime-specific resolution rules](runtime-v1.md).
|
|
62
|
+
Extension command JSON still requires an absolute executable argv; arbitrary
|
|
63
|
+
argv entries are not treated as CLI path options.
|
|
64
|
+
|
|
65
|
+
Sevro prints a whitelist of effective run settings and routes on stderr before
|
|
66
|
+
candidate execution, including resolved storage paths, native executables,
|
|
67
|
+
timeouts and concurrency. JSON stdout remains one result document. The preview
|
|
68
|
+
does not print credential bytes, runtime environment values or native diagnostics.
|
|
69
|
+
The resolved executable is retained as `sevro.host.executable` in each used native
|
|
70
|
+
role's redacted host configuration; selecting a different path changes that
|
|
71
|
+
configuration's identity.
|
|
72
|
+
|
|
73
|
+
Use `sevro init` with `--template response`, `coding` or
|
|
74
|
+
`repository-skill` and `--directory <new-path>`. It refuses existing destinations
|
|
75
|
+
and writes runnable assertions, a manifest and template instructions. Follow the
|
|
76
|
+
[demo-to-iteration workflow](evaluation-workflow.md) for preparation, execution
|
|
77
|
+
and negative controls. Repository-skill requires explicit native model, effort
|
|
78
|
+
and authentication; dry preparation does not establish dispatch.
|
|
79
|
+
|
|
80
|
+
Actual preparation, candidate, grading and evidence-retention stage events use
|
|
81
|
+
stderr. Human completion includes each trial's separate states, failed-check
|
|
82
|
+
reasons, measured candidate time, known/unknown usage and retained paths. JSON
|
|
83
|
+
stdout remains one result. Each retained CLI run writes `result.json` beside run
|
|
84
|
+
evidence and `<results-root>/latest.json`. `sevro report --latest` reads the current
|
|
85
|
+
directory's default results root; add `--results-root <path>` for another root.
|
|
86
|
+
Dry and failed runs can be latest.
|
|
87
|
+
|
|
88
|
+
Project root identifies the evaluated project and extension discovery. It does
|
|
89
|
+
not populate the participant workspace: declare inline files, generated history
|
|
90
|
+
or a repository source in the case's fixture. See [case authoring](creating-cases.md).
|
|
91
|
+
Keep `.sevro/` out of Git. Results and run-state paths are excluded from project
|
|
92
|
+
snapshots and untracked Git identity; tracked source changes still count.
|
|
93
|
+
|
|
94
|
+
## Explicit run settings
|
|
95
|
+
|
|
96
|
+
To repeat these selections, use the versioned [saved run settings](run-settings-v1.md)
|
|
97
|
+
with `--run-manifest-file ./evaluation.json`. Built-in defaults are replaced by
|
|
98
|
+
saved settings, then by explicit CLI options. `sevro.json` keeps its separate
|
|
99
|
+
runtime scope. Saved file paths use the manifest's directory; explicit CLI
|
|
100
|
+
paths use the invoking directory.
|
|
101
|
+
|
|
14
102
|
An installed package records its name, version, and a digest of its packaged
|
|
15
103
|
runtime and public contract files without reading runner Git metadata. The CLI
|
|
16
104
|
derives the project digest from its revision and dirty patch, or from a bounded
|
|
17
105
|
content snapshot when the project has no Git revision. Result and run-state
|
|
18
|
-
directories are excluded from that snapshot. When running this source checkout
|
|
106
|
+
directories are excluded from that snapshot and untracked Git identity. When running this source checkout
|
|
19
107
|
for coordinated development, pass `--runner-checkout-root` with the absolute
|
|
20
108
|
path to this checkout. Sevro verifies that it is the code actually running and
|
|
21
109
|
records its revision and dirty-patch digest instead of package provenance.
|
|
@@ -67,8 +155,7 @@ Add `--retain-host-diagnostics` to a failing invocation to retain private
|
|
|
67
155
|
diagnostics from bundled Codex and Claude hosts. The switch defaults to off,
|
|
68
156
|
applies to every selected native role, and is independent of runtime
|
|
69
157
|
`nativeTranscripts`. It does not enable transcript access or change the
|
|
70
|
-
candidate's permissions.
|
|
71
|
-
the published `0.1.0-rc.8` does not accept it.
|
|
158
|
+
candidate's permissions.
|
|
72
159
|
|
|
73
160
|
For example, add the switch to the Codex command above:
|
|
74
161
|
|
|
@@ -239,7 +326,8 @@ For cases with `sevro.semantic` checks, also pass
|
|
|
239
326
|
explicit second `HostAdapter` for grading. Alternatively, select the bundled
|
|
240
327
|
Codex route with `--semantic-host codex --semantic-model <model>
|
|
241
328
|
--semantic-effort <effort>`. That route also requires `--codex-bin` and
|
|
242
|
-
`--codex-auth-file`, shared with a Codex candidate route when present. The
|
|
329
|
+
`--codex-auth-file`, shared with a Codex candidate route when present. The binary
|
|
330
|
+
defaults to the explicitly selected `codex` on `PATH`. The two
|
|
243
331
|
semantic route options are exclusive. The candidate and semantic routes are
|
|
244
332
|
recorded separately in evidence and evaluation identity. The semantic adapter
|
|
245
333
|
runs after a complete candidate response in its own empty workspace; missing
|
|
@@ -293,7 +381,7 @@ verdict and rejects unsolicited recommendations.
|
|
|
293
381
|
|
|
294
382
|
If the extension needs configuration, supply both
|
|
295
383
|
`--extension-configuration-file` and
|
|
296
|
-
`--extension-redacted-configuration-file` as
|
|
384
|
+
`--extension-redacted-configuration-file` as paths to JSON objects.
|
|
297
385
|
The latter replaces secret values for the retained configuration digest. Keep
|
|
298
386
|
secrets out of the command argv and the redacted file.
|
|
299
387
|
|
|
@@ -421,6 +509,3 @@ Claude also advertises `sevro.host.native-goal`. Its observer binds one persiste
|
|
|
421
509
|
transcript to the original successful native session and retains goal status
|
|
422
510
|
and objective length. Missing, ambiguous, malformed or oversized evidence is
|
|
423
511
|
unavailable; an unavailable observation cannot prove goal absence.
|
|
424
|
-
|
|
425
|
-
These additions require the `0.1.0-rc.2` candidate. The published `rc.1` lacks
|
|
426
|
-
the app-server selector and native goal observations.
|
|
@@ -14,6 +14,12 @@ Local checks need no network. Run `bun run test:docs-examples` for the exact
|
|
|
14
14
|
marked tutorial commands in fresh consumer/contributor directories, including
|
|
15
15
|
expected verdicts and saved evidence. Candidate and released-package runs are
|
|
16
16
|
reported separately. These commands create and clean temporary directories.
|
|
17
|
+
The [recipe verifier](../scripts/documentation-recipes.ts) runs the new packaged
|
|
18
|
+
projects separately from the current published version. It executes actual
|
|
19
|
+
isolated tool/check positive and negative controls and both native hosts' dry
|
|
20
|
+
preparation; authenticated model, plugin-hook and goal behavior remain
|
|
21
|
+
unexecuted by that check. Its [prerequisites](recipes.md#prepare-a-recipe-project)
|
|
22
|
+
are mandatory, including UV and mise's declared Node installation.
|
|
17
23
|
|
|
18
24
|
`package.json.version` is the authoritative current version. `bun run docs:sync`
|
|
19
25
|
updates only the marked [Current release](installing.md#current-release) section
|
|
@@ -25,6 +31,10 @@ stale output. Run synchronization after a version bump and include the resulting
|
|
|
25
31
|
documentation change. Link installation and update instructions to this section.
|
|
26
32
|
Keep historical versions, licensing transitions, and minimum supported versions
|
|
27
33
|
fixed and explain their purpose beside each reference.
|
|
34
|
+
Link to the current-release section instead of repeating the current Sevro
|
|
35
|
+
version elsewhere. Describe features for the latest version, using the installed
|
|
36
|
+
`sevro` command in user workflows. Keep source-checkout instructions in contributor
|
|
37
|
+
guidance and omit notes about feature unavailability in older releases.
|
|
28
38
|
|
|
29
39
|
## External links
|
|
30
40
|
|
|
@@ -38,8 +48,8 @@ make a network job green. External success does not establish a claim's truth.
|
|
|
38
48
|
|
|
39
49
|
- Can a new user reach installation, first success, results, troubleshooting,
|
|
40
50
|
architecture, and contributing without the guide?
|
|
41
|
-
- Are supported environments, credentials, side effects, and
|
|
42
|
-
|
|
51
|
+
- Are supported environments, credentials, side effects, and prerequisites
|
|
52
|
+
explicit beside the relevant commands?
|
|
43
53
|
- Do descriptive links, meaningful headings, source order, readable tables,
|
|
44
54
|
image alternatives, and words rather than color support accessibility?
|
|
45
55
|
- Are current contracts, accepted choices, research, and past evidence distinct?
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# From a demo to your next evaluation
|
|
2
|
+
|
|
3
|
+
Create an evaluation, prepare it, run its assertions, and inspect the latest
|
|
4
|
+
result with the installed `sevro` command.
|
|
5
|
+
|
|
6
|
+
## Try the demo and create a project
|
|
7
|
+
|
|
8
|
+
You need Bun 1.3.13. [Install Sevro](installing.md) and
|
|
9
|
+
[make its command available on `PATH`](installing.md#make-the-command-available),
|
|
10
|
+
then run:
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
sevro demo
|
|
14
|
+
sevro init --template response --directory ./response-evaluation
|
|
15
|
+
cd response-evaluation
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The demo returns an assessed `passed` task and evidence under `.sevro/results`,
|
|
19
|
+
using its own temporary project without credentials. The initializer refuses
|
|
20
|
+
existing destinations, including directories and symbolic links. It writes
|
|
21
|
+
real assertions, a deterministic adapter, `run.json` and template instructions.
|
|
22
|
+
[Saved settings](run-settings-v1.md) use their manifest's directory for paths;
|
|
23
|
+
`sevro.json` retains its separate runtime scope. Keep private `.sevro/` out of Git.
|
|
24
|
+
|
|
25
|
+
## Prepare before execution
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
sevro run --run-manifest-file run.json --dry
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Stderr shows the whitelisted resolved settings before candidate execution:
|
|
32
|
+
case/source selection, routes/model/effort, trials, threshold, concurrency,
|
|
33
|
+
runtime selection and canonical storage paths. Invalid input names its option,
|
|
34
|
+
file or field; missing executables name installation or explicit override paths.
|
|
35
|
+
Correct the named prerequisite and repeat preparation.
|
|
36
|
+
|
|
37
|
+
Dry preparation follows normal fixture, extension and check preparation without
|
|
38
|
+
calling the candidate or graders. Expect
|
|
39
|
+
`execution=not_run grading=not_requested task=not_assessed`. It does not establish
|
|
40
|
+
authentication, provider model availability, native skill dispatch or model success.
|
|
41
|
+
See [native prerequisites](native-hosts.md) and [troubleshooting](troubleshooting.md).
|
|
42
|
+
|
|
43
|
+
## Run, read and change one thing
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
sevro run --run-manifest-file run.json
|
|
47
|
+
sevro report --latest
|
|
48
|
+
SEVRO_TEMPLATE_FAIL=1 sevro run --run-manifest-file run.json
|
|
49
|
+
sevro report --latest
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
The first run's adapter returns exactly `ready`; its regex assertion must pass.
|
|
53
|
+
The negative control returns `wrong`: expect exit `1`, completed execution and
|
|
54
|
+
grading, task `failed`, and the named `ready` check's reason. Then edit `case.json`
|
|
55
|
+
and `adapter.ts`, prepare again and rerun. Explicit flags override saved settings;
|
|
56
|
+
`--trials 3` repeats the task without rewriting `run.json`.
|
|
57
|
+
|
|
58
|
+
Progress uses stderr at actual preparation, candidate, grading and evidence
|
|
59
|
+
retention boundaries. Trial numbers distinguish concurrent work, which may
|
|
60
|
+
interleave. Stage `completed` means the attempt finished; the result states whether
|
|
61
|
+
execution/grading succeeded. Dry runs explicitly skip candidate and grading;
|
|
62
|
+
failed candidates skip grading. Failed preparation never prints candidate start.
|
|
63
|
+
Evidence completion follows persistence. Optional synchronous engine observers
|
|
64
|
+
cannot change the result through a thrown exception.
|
|
65
|
+
|
|
66
|
+
Human completion shows per-trial states, failed/unavailable check reasons,
|
|
67
|
+
measured candidate time, known/unknown usage/cost and `result=`/`evidence=` paths.
|
|
68
|
+
Candidate time measures the host call, not total setup time. Absent metrics remain
|
|
69
|
+
`unknown`; dry duration is unknown. Prompt-only completion stays `not_assessed`.
|
|
70
|
+
Add `--json` to emit one `sevro.cli-result.v1` stdout document; progress remains on
|
|
71
|
+
stderr. See [reading results](reading-results.md).
|
|
72
|
+
|
|
73
|
+
Each retained CLI evaluation saves `result.json` beside its run evidence and
|
|
74
|
+
atomically publishes `<results-root>/latest.json`. Latest is the last retained CLI
|
|
75
|
+
result, including dry runs and failed assessments. Pre-run failures do not replace
|
|
76
|
+
it. Concurrent runs publish in completion order. Output files stay under excluded
|
|
77
|
+
results storage, preserving project identity. A latest-write failure cannot turn
|
|
78
|
+
an earlier failed assessment into success.
|
|
79
|
+
|
|
80
|
+
`report --latest` reads `<cwd>/.sevro/results/latest.json`. Use
|
|
81
|
+
`report --latest --results-root ./other-results` for another output root or working
|
|
82
|
+
directory. `--result-file ./path/to/result.json` selects older runs and accepts
|
|
83
|
+
relative paths. Reports validate referenced evidence; missing/moved evidence
|
|
84
|
+
produces a diagnostic. [Report v1](report-v1.md) defines fields and limits.
|
|
85
|
+
|
|
86
|
+
## Choose another starter
|
|
87
|
+
|
|
88
|
+
| Template | Real assertions | Prerequisites and interpretation |
|
|
89
|
+
| ------------------ | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
|
|
90
|
+
| `response` | Exact `ready` response; negative control returns `wrong` | Bun; no model credentials |
|
|
91
|
+
| `coding` | Edit `add.sh`; isolated checks test positive, negative and zero addition | Bun plus `sandbox-exec` on macOS or `bubblewrap`/`socat` on Linux; negative control leaves subtraction bug |
|
|
92
|
+
| `repository-skill` | Actual `.agents/skills/startup-guide/SKILL.md` mount; explicit invocation evidence; correct start command and port | Bun, Git, Codex, native isolation, explicit model/effort and readable auth file; dry is unassessed |
|
|
93
|
+
|
|
94
|
+
From a new parent directory, use `sevro init --template coding --directory ./coding-eval`
|
|
95
|
+
or `sevro init --template repository-skill --directory ./skill-eval`. Read its generated
|
|
96
|
+
README for controls and dry/live commands. Repository-skill `command.json` records
|
|
97
|
+
absolute Bun/extension argv; regenerate it after moving the project.
|
|
98
|
+
|
|
99
|
+
For model evaluations, replace a saved adapter route with explicit `--host`,
|
|
100
|
+
`--model`, `--effort` and host authentication options. Candidate, semantic and
|
|
101
|
+
advisory model choices remain independent. Deterministic adapters measure wiring;
|
|
102
|
+
they do not measure model ability. Native runs may incur provider charges.
|
|
103
|
+
Automated package checks run response/coding pass/fail and native skill preparation;
|
|
104
|
+
authenticated native behavior and human usability remain unmeasured. The
|
|
105
|
+
[study protocol](usability-study.md) records deferred human measurements.
|
|
106
|
+
[Runnable recipes](recipes.md) cover Claude, plugins/hooks, goals, semantic grading
|
|
107
|
+
and host tools.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# Template and batch inspection for issue 85
|
|
2
|
+
|
|
3
|
+
On 2026-10-15, the implementation for [issue #85](https://github.com/BjRo/sevro/issues/85)
|
|
4
|
+
replayed the two original Codex native transcript bundles through the changed
|
|
5
|
+
Sevro observers and Darrow's checkpoint `67f4fcdb` extension. Bundle and decoded
|
|
6
|
+
source SHA-256 values matched the issue before replay. Raw bundles and output
|
|
7
|
+
remain private and outside Git.
|
|
8
|
+
|
|
9
|
+
| Original case | Exec submissions | Complete receipts after replay | Regraded no-agent assertion |
|
|
10
|
+
| --------------------------------- | ---------------- | ------------------------------ | --------------------------- |
|
|
11
|
+
| goal-placement-direct-same-route | 11 | 11 | passed |
|
|
12
|
+
| goal-placement-coupled-sequential | 10 | 10 | passed |
|
|
13
|
+
|
|
14
|
+
The five exact [retained program fixtures](../../tests/fixtures/code-mode-85/README.md)
|
|
15
|
+
now preserve these native ordinals and ordered submission-local labels:
|
|
16
|
+
|
|
17
|
+
- Direct 45: three `exec_command` calls.
|
|
18
|
+
- Direct 69 and 77: `apply_patch`, then `exec_command`.
|
|
19
|
+
- Coupled 30: three `exec_command` calls.
|
|
20
|
+
- Coupled 68: two `exec_command` calls.
|
|
21
|
+
|
|
22
|
+
Regrading replaced only the native-call and portable native-control observations
|
|
23
|
+
with newly inspected observations and invoked the original case's extension
|
|
24
|
+
checks. Both `darrow.evals.transcript.1` results changed from unavailable to
|
|
25
|
+
passed. These are artifact rechecks, not new candidate runs or rewritten
|
|
26
|
+
historical task verdicts. Deterministic consumer coverage separately confirms
|
|
27
|
+
that a submitted `spawn_agent` fails the no-agent assertion even when the
|
|
28
|
+
accepted-agent count is zero, and partial receipts remain unavailable.
|
|
29
|
+
|
|
30
|
+
## Observed test-first slices
|
|
31
|
+
|
|
32
|
+
1. Red — `bun test tests/codex-native-calls.test.ts -t 'retained direct ordinal 45'`: the template-command receipt was partial.
|
|
33
|
+
Green — `bun test tests/codex-native-calls.test.ts -t 'retained direct ordinal 45'`: 1 passed, with three bound call labels.
|
|
34
|
+
2. Red — `bun test tests/codex-native-calls.test.ts -t 'retained coupled ordinal'`: both destructured batch receipts were partial.
|
|
35
|
+
Green — `bun test tests/codex-native-calls.test.ts -t 'retained coupled ordinal'`: 2 passed, with three and two bound call labels respectively.
|
|
36
|
+
|
|
37
|
+
Additional regressions preserve submitted spawns inside interpolated and
|
|
38
|
+
concatenated output, reject ambiguous bindings and composition, and keep
|
|
39
|
+
private source bytes out of observations. The focused native-call/control
|
|
40
|
+
suite passed 94 tests. These checks do not replace the full GitHub CI gate
|
|
41
|
+
on the PR's exact published commit.
|
|
42
|
+
|
|
43
|
+
## Fresh native trials
|
|
44
|
+
|
|
45
|
+
Fresh trials use Darrow checkpoint `67f4fcdb`, native Codex 0.162.1 app-server,
|
|
46
|
+
candidate and semantic grader `gpt-6-luna/medium`, passive ownership, N=1,
|
|
47
|
+
jobs=1, threshold=1, and a 1,200,000 ms timeout with retained diagnostics.
|
|
48
|
+
They explicitly select the changed Sevro source worktree through
|
|
49
|
+
`SEVRO_CHECKOUT`; they are development evidence, not published-package trials.
|
|
50
|
+
|
|
51
|
+
The direct trial, run `510f99df-882f-48fe-b893-d4b6fba8864d`, completed with
|
|
52
|
+
9/10 checks passing and the no-agent assertion unavailable. Its new submission
|
|
53
|
+
29 maps a callback that invokes tools over a command array; submission 74 uses
|
|
54
|
+
an output callback with a conditional expression. Both forms remain outside
|
|
55
|
+
the [bounded inspection contract](../native-controls.md) and correctly retain
|
|
56
|
+
partial receipts. The five original programs no longer cause that limitation.
|
|
57
|
+
|
|
58
|
+
The coupled trial, run `7f9f5d8a-641e-4744-a986-05298100541c`, completed with
|
|
59
|
+
10/11 checks passing and the no-agent assertion unavailable. Submissions 29,
|
|
60
|
+
45, and 68 each use a `for ... of` output loop with a conditional expression
|
|
61
|
+
and `String(...)` conversion. That output body is also outside the bounded
|
|
62
|
+
subset and remains partial. Neither fresh run is a passing placement trial;
|
|
63
|
+
their unavailable verdicts are preserved rather than treating zero accepted
|
|
64
|
+
agents as proof of absence.
|
package/docs/evidence.md
CHANGED
|
@@ -8,6 +8,7 @@ Keep their observations intact:
|
|
|
8
8
|
- [Claude continuation](claude-continuation-validation.md)
|
|
9
9
|
- [Concurrency](concurrency-validation.md)
|
|
10
10
|
- [Code Mode control observations](evidence/code-mode-controls-71.md)
|
|
11
|
+
- [Code Mode template and batch programs](evidence/code-mode-programs-85.md)
|
|
11
12
|
- [Candidate Codex configuration and user-skill state](evidence/codex-state-73/README.md)
|
|
12
13
|
- [Configuration roots](configuration-root-validation.md)
|
|
13
14
|
- [Fixture cleanup](fixture-cleanup-validation.md)
|
package/docs/getting-started.md
CHANGED
|
@@ -5,6 +5,27 @@ Darrow checkout, Codex/Claude login, or model credentials are required. The
|
|
|
5
5
|
adapter returns `ready`; Sevro grades that response against `^ready$`.
|
|
6
6
|
This teaches the runner lifecycle, not the quality of an actual model.
|
|
7
7
|
|
|
8
|
+
<a id="built-in-demo-in-this-checkout"></a>
|
|
9
|
+
|
|
10
|
+
## Built-in demo
|
|
11
|
+
|
|
12
|
+
With [Sevro installed and on `PATH`](installing.md#make-the-command-available), run:
|
|
13
|
+
|
|
14
|
+
```sh
|
|
15
|
+
sevro demo
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
It prints `execution=completed grading=completed task=passed` and an absolute
|
|
19
|
+
`evidence=` path under `.sevro/results` in your current directory. The demo owns
|
|
20
|
+
its packaged case and adapter, uses a separate temporary project, and ignores
|
|
21
|
+
your project's `sevro.json` and native configuration. No credentials are needed.
|
|
22
|
+
Effective run settings appear on stderr before evaluation. Add `--json` for one
|
|
23
|
+
versioned result on stdout.
|
|
24
|
+
|
|
25
|
+
Continue with [the initialized evaluation workflow](evaluation-workflow.md) to
|
|
26
|
+
create response, coding or repository-skill assertions, prepare them, inspect
|
|
27
|
+
progress and access the latest result without copying an evidence path.
|
|
28
|
+
|
|
8
29
|
## From an installed package
|
|
9
30
|
|
|
10
31
|
Prepare a new directory:
|
|
@@ -16,8 +37,8 @@ bun init -y
|
|
|
16
37
|
```
|
|
17
38
|
|
|
18
39
|
Install the [current release](installing.md#current-release) there, then return
|
|
19
|
-
here to run the example.
|
|
20
|
-
|
|
40
|
+
here to run the example. See [licensing](licensing.md) for the current terms and
|
|
41
|
+
the licenses retained by historical releases.
|
|
21
42
|
|
|
22
43
|
Make the local installation available on `PATH` for this shell, then call
|
|
23
44
|
`sevro` directly. Bun must also be on `PATH`.
|
package/docs/identity-v1.md
CHANGED
|
@@ -32,7 +32,10 @@ The CLI derives `runnerBuildDigest` from sorted packaged file paths and their
|
|
|
32
32
|
SHA-256 bytes under `src/`, `schemas/`, `docs/`, and `examples/`, plus `README.md` and
|
|
33
33
|
`package.json`. For a Git project, `projectDigest` binds the revision and dirty
|
|
34
34
|
patch digest without the checkout path. Without a revision it hashes a bounded
|
|
35
|
-
file snapshot, excluding `.git`, results, and run-state storage.
|
|
35
|
+
file snapshot, excluding `.git`, results, and run-state storage. Results and
|
|
36
|
+
run-state storage are also excluded from untracked Git identity, so repeated
|
|
37
|
+
project-local output does not change the project digest. Tracked source changes
|
|
38
|
+
remain included, even under a selected storage directory. A caller may
|
|
36
39
|
provide explicit 64-character digests for controlled development runs.
|
|
37
40
|
|
|
38
41
|
The prefix separates this digest from component digests and future identity
|
|
@@ -48,6 +51,17 @@ The effective positive trial job limit is retained as `configuration.redacted.jo
|
|
|
48
51
|
and enters `configurationDigest`. Changing parallelism changes evaluation identity,
|
|
49
52
|
even when the condition, host route, trial count, and assessment threshold match.
|
|
50
53
|
|
|
54
|
+
CLI [saved run settings](run-settings-v1.md) resolve before evaluation. Equivalent
|
|
55
|
+
manifest and explicit selections produce the same semantic inputs. CLI evidence
|
|
56
|
+
also retains `configuration.redacted.runSettings` for shell isolation, explicit
|
|
57
|
+
protected roots, deprecated runtime selections, selected native timeouts and
|
|
58
|
+
Claude project-settings authority. These values enter the
|
|
59
|
+
configuration digest. Optional `configuration.resolved` stores the whitelisted
|
|
60
|
+
operator preview outside that digest, including operational paths and the
|
|
61
|
+
selected manifest location. Relocating output storage alone does not alter
|
|
62
|
+
identity. A tracked manifest remains project content under the normal project
|
|
63
|
+
identity rules.
|
|
64
|
+
|
|
51
65
|
Extension runs retain the operator-supplied redacted extension configuration
|
|
52
66
|
under `configuration.redacted.extensionConfiguration`, alongside
|
|
53
67
|
`extensionConfigurationDigest`. The retained copy is the same snapshot used
|
package/docs/installing.md
CHANGED
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
# Install, update, or remove Sevro
|
|
2
2
|
|
|
3
3
|
Use Bun 1.3.13 for this TypeScript/Bun CLI. Installation does not make it a
|
|
4
|
-
Node-only executable. Full validation and native isolation target macOS
|
|
5
|
-
|
|
4
|
+
Node-only executable. Full validation and native isolation target macOS and
|
|
5
|
+
Ubuntu 24.04 x64. See the [bounded platform evidence](native-hosts.md#validated-platform-scope);
|
|
6
|
+
Windows remains unverified.
|
|
6
7
|
|
|
7
8
|
## Install locally
|
|
8
9
|
|
|
@@ -22,10 +23,10 @@ Local installation records the selected version in the project lockfile.
|
|
|
22
23
|
|
|
23
24
|
## Current release
|
|
24
25
|
|
|
25
|
-
The current release is `0.1.0-rc.
|
|
26
|
+
The current release is `0.1.0-rc.13`. Install this exact version:
|
|
26
27
|
|
|
27
28
|
```sh
|
|
28
|
-
bun add --exact @bjoernrochel/sevro@0.1.0-rc.
|
|
29
|
+
bun add --exact @bjoernrochel/sevro@0.1.0-rc.13
|
|
29
30
|
```
|
|
30
31
|
|
|
31
32
|
<!-- sevro-current-release:end -->
|
|
@@ -46,17 +47,16 @@ no adjustment is needed. Bun must also be on `PATH`: the executable uses
|
|
|
46
47
|
|
|
47
48
|
## Published package and checkout
|
|
48
49
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
[source-available licensing terms](licensing.md). Earlier `rc.1` and `rc.2`
|
|
52
|
-
packages retain BUSL-1.1. The license transition began with `0.1.0-rc.3`.
|
|
53
|
-
Source checkouts can include changes beyond their
|
|
54
|
-
package version; compare the exact release tag when establishing provenance.
|
|
50
|
+
See [licensing](licensing.md) for Sevro's source-available terms and historical
|
|
51
|
+
licensing transitions.
|
|
55
52
|
|
|
56
|
-
The
|
|
53
|
+
The npm package contains sources, schemas, examples, documentation,
|
|
57
54
|
LICENSE, and contribution guidance. It excludes tests, developer scripts, Git
|
|
58
55
|
metadata, and repository guide mounts. Use a source checkout for the guide.
|
|
59
56
|
|
|
57
|
+
Source checkouts can include changes beyond their package version; compare the
|
|
58
|
+
exact release tag when establishing provenance.
|
|
59
|
+
|
|
60
60
|
## Work from source
|
|
61
61
|
|
|
62
62
|
```sh
|
package/docs/native-controls.md
CHANGED
|
@@ -22,10 +22,14 @@ one receipt per native `exec` submission, with its native `ordinal`, inspection
|
|
|
22
22
|
The index is local to that submission; native ordinals are not synthesized.
|
|
23
23
|
The original direct-call list and total `submittedExecCalls` stay unchanged.
|
|
24
24
|
|
|
25
|
-
|
|
25
|
+
Sevro statically inspects bounded straight-line Code Mode
|
|
26
26
|
programs without executing them. It accepts literal tool calls, constant result
|
|
27
|
-
bindings,
|
|
28
|
-
|
|
27
|
+
bindings, template interpolation and `+` composition of inspected expressions,
|
|
28
|
+
direct `text`/`store` output, and literal `Promise.all`/`allSettled` batches.
|
|
29
|
+
Batch results may use flat constant array bindings or result-only
|
|
30
|
+
`forEach(text)` or `for ... of` output. Destructuring defaults, rest elements,
|
|
31
|
+
holes, nested bindings, duplicate names, and shadowed globals remain unsupported.
|
|
32
|
+
It requires one
|
|
29
33
|
unique, later native completion for the same call ID with an intact successful
|
|
30
34
|
script envelope. Tool argument strings, comments and printed tool-looking text
|
|
31
35
|
do not become call labels. A rejected tool request returned by a completed
|
package/docs/native-hosts.md
CHANGED
|
@@ -6,6 +6,23 @@ Claude Code), a supported native CLI, and model
|
|
|
6
6
|
authentication. Complete commands/rules live in the [CLI reference](development-cli.md)
|
|
7
7
|
and [Codex host reference](codex-host.md).
|
|
8
8
|
|
|
9
|
+
## Validated platform scope
|
|
10
|
+
|
|
11
|
+
The full source quality gate has passed on macOS arm64 and Ubuntu 24.04 x64.
|
|
12
|
+
Ubuntu evidence includes [the successful source CI run for `cd7542bf`](https://github.com/BjRo/sevro/actions/runs/38692971517).
|
|
13
|
+
That run exercises deterministic native-host protocols and real filesystem,
|
|
14
|
+
process, network and shell isolation with the pinned prerequisites in
|
|
15
|
+
[the quality contract](typescript-quality.md#environment-and-ci). It does not
|
|
16
|
+
establish authenticated model success for every host/model combination.
|
|
17
|
+
This is prior source evidence; changed candidates still need their own green CI.
|
|
18
|
+
Other Linux distributions and Windows are unverified.
|
|
19
|
+
|
|
20
|
+
On Ubuntu, install `bubblewrap`, `ripgrep`, and `socat`. Unprivileged user
|
|
21
|
+
namespaces must work; Ubuntu's AppArmor policy may need a `bwrap` profile.
|
|
22
|
+
[CI's setup script](../scripts/setup-ci-linux.sh) supplies and probes the
|
|
23
|
+
supported runner setup. Use it as an explicit development setup step where
|
|
24
|
+
appropriate; do not assume all Linux hosts have that configuration.
|
|
25
|
+
|
|
9
26
|
## Codex
|
|
10
27
|
|
|
11
28
|
Select `--host codex`, absolute `--codex-bin` and `--codex-auth-file` paths,
|