@bjoernrochel/sevro 0.1.0-rc.12 → 0.1.0-rc.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (94) hide show
  1. package/CONTRIBUTING.md +13 -2
  2. package/README.md +5 -0
  3. package/docs/README.md +4 -0
  4. package/docs/contracts.md +1 -0
  5. package/docs/creating-cases.md +9 -2
  6. package/docs/development-cli.md +93 -8
  7. package/docs/documentation-quality.md +12 -2
  8. package/docs/evaluation-workflow.md +107 -0
  9. package/docs/evidence/code-mode-programs-85.md +64 -0
  10. package/docs/evidence.md +1 -0
  11. package/docs/getting-started.md +23 -2
  12. package/docs/identity-v1.md +15 -1
  13. package/docs/installing.md +11 -11
  14. package/docs/native-controls.md +7 -3
  15. package/docs/native-hosts.md +17 -0
  16. package/docs/reading-results.md +13 -0
  17. package/docs/recipes.md +485 -0
  18. package/docs/releases.md +2 -1
  19. package/docs/report-v1.md +13 -1
  20. package/docs/results-v1.md +22 -0
  21. package/docs/run-settings-v1.md +147 -0
  22. package/docs/runtime-v1.md +21 -8
  23. package/docs/troubleshooting.md +12 -12
  24. package/docs/typescript-quality.md +38 -16
  25. package/docs/usability-study.md +114 -0
  26. package/examples/recipes/candidate.ts +26 -0
  27. package/examples/recipes/cases/coding-change.json +21 -0
  28. package/examples/recipes/cases/host-tools.json +21 -0
  29. package/examples/recipes/cases/mise.json +21 -0
  30. package/examples/recipes/cases/node.json +21 -0
  31. package/examples/recipes/cases/python-uv.json +21 -0
  32. package/examples/recipes/cases/semantic.json +17 -0
  33. package/examples/recipes/coding-change/check.ts +23 -0
  34. package/examples/recipes/extension-artifacts.ts +65 -0
  35. package/examples/recipes/extension-methods.ts +174 -0
  36. package/examples/recipes/extension.ts +37 -0
  37. package/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
  38. package/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
  39. package/examples/recipes/repository-skill/README.md +3 -0
  40. package/examples/recipes/run-inputs.ts +188 -0
  41. package/examples/recipes/run-options.ts +131 -0
  42. package/examples/recipes/run.ts +63 -0
  43. package/examples/recipes/runtime/empty.json +1 -0
  44. package/examples/recipes/runtime/goals.json +1 -0
  45. package/examples/recipes/runtime/mise.json +6 -0
  46. package/examples/recipes/runtime/path.json +1 -0
  47. package/examples/recipes/runtime/plugins.json +4 -0
  48. package/examples/recipes/runtime/python-uv.json +10 -0
  49. package/examples/recipes/semantic-judge.ts +29 -0
  50. package/examples/templates/coding/adapter.ts +17 -0
  51. package/examples/templates/coding/case.json +15 -0
  52. package/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
  53. package/examples/templates/repository-skill/extension-data.ts +64 -0
  54. package/examples/templates/repository-skill/extension.ts +31 -0
  55. package/examples/templates/response/adapter.ts +12 -0
  56. package/examples/templates/response/case.json +13 -0
  57. package/package.json +1 -1
  58. package/schemas/run-evidence-v1.schema.json +2 -1
  59. package/src/case-validation.ts +71 -48
  60. package/src/cli/command.ts +54 -3
  61. package/src/cli/completion.ts +12 -0
  62. package/src/cli/demo.ts +59 -0
  63. package/src/cli/evaluation.ts +9 -1
  64. package/src/cli/help.ts +187 -0
  65. package/src/cli/hosts.ts +21 -3
  66. package/src/cli/init-settings.ts +30 -0
  67. package/src/cli/init.ts +111 -0
  68. package/src/cli/invocation.ts +99 -78
  69. package/src/cli/loading.ts +16 -9
  70. package/src/cli/preparation.ts +2 -1
  71. package/src/cli/presentation.ts +23 -20
  72. package/src/cli/progress.ts +6 -0
  73. package/src/cli/run-manifest.ts +133 -0
  74. package/src/cli/run-options.ts +101 -0
  75. package/src/cli/run-overrides.ts +83 -0
  76. package/src/cli/settings.ts +160 -0
  77. package/src/cli/summary.ts +43 -0
  78. package/src/cli.ts +8 -1
  79. package/src/engine.ts +1 -0
  80. package/src/evaluation/plan.ts +8 -1
  81. package/src/evaluation/progress.ts +19 -0
  82. package/src/evaluation/run.ts +7 -0
  83. package/src/evaluation/trial.ts +22 -1
  84. package/src/evaluation/types.ts +12 -0
  85. package/src/generated/run-evidence.cjs +1 -1
  86. package/src/hosts/codex-command-output.ts +15 -2
  87. package/src/hosts/codex-exec-controls.ts +52 -4
  88. package/src/hosts/codex-native-calls.ts +42 -13
  89. package/src/hosts/codex-profile.ts +1 -0
  90. package/src/hosts/codex-skill-reads.ts +249 -26
  91. package/src/provenance.ts +24 -4
  92. package/src/report.ts +37 -2
  93. package/src/runtime/config.ts +1 -1
  94. package/src/schema-types.ts +1 -0
package/CONTRIBUTING.md CHANGED
@@ -38,6 +38,13 @@ bun run test:docs-examples
38
38
  ```
39
39
 
40
40
  The second command checks the tutorial in fresh consumer/contributor directories.
41
+ Its complete recipe checks also require UV, Python 3, mise, and an existing
42
+ `mise install node@24.13.0` installation; see [recipe prerequisites](docs/recipes.md).
43
+ CI explicitly provisions these tools. Missing prerequisites fail with a named
44
+ setup instruction rather than skipping a recipe.
45
+ To run recipe commands from this source checkout, set
46
+ `export SEVRO_RECIPES="$PWD/examples/recipes"` from the repository root, then
47
+ follow the [recipe commands](docs/recipes.md#prepare-a-recipe-project).
41
48
  Native model trials need their own [credentials and prerequisites](docs/native-hosts.md).
42
49
 
43
50
  ## Quality contract and checks
@@ -68,6 +75,10 @@ Before committing, use `bun run check:typescript --fast` after installing the
68
75
  locked dependencies. This runs inventory, schema, formatting, lint, typing and
69
76
  documentation checks; it does not replace full acceptance. CI retains machine-readable coverage
70
77
  reports under `.quality/` and exposes the stable `TypeScript quality` status.
78
+ The complete command's `static`, `docs`, `tests` and `package` phases run as
79
+ concurrent CI jobs. After frozen installation, use
80
+ `bun run check:typescript --phase <name>` to run one phase locally; it does not
81
+ replace acceptance across all four. See the [phase inventory](docs/typescript-quality.md).
71
82
 
72
83
  Run `bun run check:docs:external` separately and [triage network failures](docs/documentation-quality.md#external-links).
73
84
  Guide changes also need [native evals](docs/guide-evaluation.md).
@@ -84,8 +95,8 @@ repairing in-scope failures; it does not authorize merging or unrelated changes.
84
95
  non-root user, and isolation prerequisites. Reproducing that environment locally
85
96
  with Docker is optional for diagnosis and is not a publication prerequisite.
86
97
  - After the local pre-commit checks, publish the authorized candidate to its PR
87
- and require `bun run check:typescript` to succeed in GitHub CI before reporting
88
- delivery as complete. A local run, fast check, targeted suite, or historical
98
+ and require all four canonical `check:typescript` phases to succeed in GitHub CI
99
+ before reporting delivery as complete. A local run, fast check, targeted suite, or historical
89
100
  result does not establish that the current Ubuntu gate passes.
90
101
  - Keep the quality bar intact. Repair behavior or add meaningful coverage for
91
102
  accepted behavior; do not lower thresholds, exempt authored sources, skip tests,
package/README.md CHANGED
@@ -40,6 +40,11 @@ Continue with [your first evaluation](docs/getting-started.md): the exact comman
40
40
  expected `passed` result, and retained evidence. Read [installation](docs/installing.md)
41
41
  for updates, removal, and source checkouts.
42
42
 
43
+ The development checkout and freshly built packages provide a
44
+ [demo-to-iteration workflow](docs/evaluation-workflow.md): initialize real
45
+ assertions, prepare, execute, read progress and inspect the latest result.
46
+ The published `rc.11` package uses the installed tutorial above.
47
+
43
48
  ## Find your next task
44
49
 
45
50
  The [documentation hub](docs/README.md) routes to running evaluations, authoring
package/docs/README.md CHANGED
@@ -7,12 +7,15 @@ Start with the task you want to complete. These pages work without an agent.
7
7
  | I want to… | Read |
8
8
  | ---------------------------------------------- | -------------------------------------------------------- |
9
9
  | Get a result without a model account | [Your first evaluation](getting-started.md) |
10
+ | Initialize, prepare, run and iterate | [Evaluation workflow](evaluation-workflow.md) |
10
11
  | Install, update, remove, or use a checkout | [Installation](installing.md) |
11
12
  | Run evaluations and select a host | [Running evaluations](running-evaluations.md) |
12
13
  | Create cases and reusable policy | [Cases and extensions](creating-cases.md) |
14
+ | Copy a complete coding, skill or runtime setup | [Runnable recipes](recipes.md) |
13
15
  | Interpret verdicts, evidence, and comparisons | [Reading results](reading-results.md) |
14
16
  | Run Codex or Claude with isolation | [Native hosts](native-hosts.md) |
15
17
  | Reuse host tools or enable goals and hooks | [Runtime configuration](runtime-v1.md) |
18
+ | Save recurring evaluation selections | [Run settings](run-settings-v1.md) |
16
19
  | Resolve a symptom | [Troubleshooting](troubleshooting.md) |
17
20
  | Trace runner lifecycle, ownership, and changes | [Architecture](architecture.md) |
18
21
  | Change Sevro or its documentation | [Contributing](../CONTRIBUTING.md) |
@@ -35,6 +38,7 @@ There is no accepted Rust migration decision here;
35
38
  [issue #2](https://github.com/BjRo/sevro/issues/2) tracks research.
36
39
 
37
40
  Read [documentation quality](documentation-quality.md),
41
+ [the usability study protocol](usability-study.md),
38
42
  [guide evaluation](guide-evaluation.md), [releases](releases.md), and
39
43
  [acknowledgements](acknowledgements.md) for maintenance.
40
44
  Return to the [Sevro introduction](../README.md).
package/docs/contracts.md CHANGED
@@ -7,6 +7,7 @@ Read these human contracts and their schemas together:
7
7
  - [Report v1](report-v1.md): summaries and comparison limitations.
8
8
  - [Identity v1](identity-v1.md): versioned comparison dimensions and hashing.
9
9
  - [Runtime configuration v1](runtime-v1.md): environment, tools, private state, goals, and hooks.
10
+ - [Run settings v1](run-settings-v1.md): recurring evaluation selections and CLI overrides.
10
11
  - [Built-in graders v1](builtin-graders-v1.md): output, shell, Git HEAD, and semantic checks.
11
12
  - [Schemas](../schemas/): machine-readable public forms.
12
13
 
@@ -4,18 +4,25 @@ Start with a direct case for a prompt, fixture, and built-in success criteria.
4
4
  Copy [the basic graded case](../examples/basic/graded.json), give it a stable
5
5
  ID, and replace the prompt/check. Read [built-in graders](builtin-graders-v1.md)
6
6
  for regex, JSON, and bounded schema assertions. No criteria means `not_assessed`.
7
+ For complete coding, repository-skill, runtime, semantic and extension projects,
8
+ follow the [runnable recipes](recipes.md), including their deliberate failures.
7
9
 
8
10
  Run it with the [CLI](running-evaluations.md). Direct cases follow `ResolvedCase`
9
11
  in [the engine](../src/engine.ts). Fixtures can provide inline files, generated
10
12
  Git history, or a declared clean repository. The [CLI reference](development-cli.md)
11
13
  owns required source roots/maps; an arbitrary returned path does not grant access.
12
14
 
15
+ Selecting `--project-root` identifies the project; it does not copy its contents
16
+ into a participant fixture. Declare the files or repository source the participant
17
+ should receive. The [run defaults reference](development-cli.md#run-defaults)
18
+ explains how project identity, fixtures and CLI path resolution interact.
19
+
13
20
  ## Evaluate your repository skills
14
21
 
15
22
  Start with one representative task, your actual skill, and observable success
16
23
  criteria. This example evaluates a documentation guide's answer with Codex.
17
24
  Use [Sevro from `PATH`](installing.md#make-the-command-available).
18
- Live runs require the [native host prerequisites](native-hosts.md), macOS
25
+ Live runs require the [native host prerequisites](native-hosts.md), supported
19
26
  isolation, an authenticated Codex installation, and your chosen model.
20
27
  They consume that account's model quota.
21
28
 
@@ -109,7 +116,7 @@ prepares skill mounts and interprets complete native host evidence. The
109
116
  [extension protocol](extension-protocol-v1.md) defines repository and plugin
110
117
  invocations; the checkout's [guide evals](guide-evaluation.md) demonstrate the
111
118
  approach. Check those capabilities against your exact installed release:
112
- this checkout contains [changes beyond published `rc.2`](installing.md#published-package-and-checkout).
119
+ this checkout can contain [changes beyond its published version](installing.md#published-package-and-checkout).
113
120
 
114
121
  ## Reusable project policy
115
122
 
@@ -11,11 +11,99 @@ grading, evidence, and extension lifecycle.
11
11
  Examples assume [Sevro and Bun are on `PATH`](installing.md#make-the-command-available).
12
12
  For a contributor checkout, use `bun src/cli.ts` in place of `sevro`.
13
13
 
14
+ Use `sevro --help`, `sevro run --help`, or `sevro report --help` for examples,
15
+ required options, and defaults without preparing a run. `sevro --version` prints
16
+ the installed package version. Invocation and configuration diagnostics go to
17
+ stderr in both output modes; `--json` keeps one `sevro.cli-result.v1` document
18
+ on stdout. Invalid case diagnostics name the input file and offending field.
19
+
20
+ <a id="run-defaults-in-this-checkout"></a>
21
+
22
+ ## Run defaults
23
+
24
+ `sevro demo` owns a deterministic case, fixture and adapter. It needs no model
25
+ credentials and immediately prints an assessed verdict plus an evidence path in
26
+ `.sevro/results` beneath the invoking directory. Its temporary project and
27
+ configuration are independent of the caller's files and runtime policy.
28
+
29
+ An ordinary run needs a case source and candidate route. For example:
30
+
31
+ ```sh
32
+ sevro run --case-file ./case.json --adapter-module ./host-adapter.ts
33
+ ```
34
+
35
+ | Setting | Default |
36
+ | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
37
+ | Condition | `passive` |
38
+ | Trial count | `1` |
39
+ | Pass threshold | `1`: all assessed trials must pass; any unassessed trial prevents a pass |
40
+ | Parallel trial limit (`--jobs`) | `3` |
41
+ | Project root | Invoking current directory |
42
+ | Results root | `<project-root>/.sevro/results` |
43
+ | Configuration root | Project root |
44
+ | Run-state root | Results root |
45
+ | Runtime policy | Discover `sevro.json` at project root; none when absent |
46
+ | Native binary | Resolve only the explicitly selected `codex` or `claude` from `PATH`; `--codex-bin` / `--claude-bin` override it |
47
+ | Native execution timeout | `600000` ms (10 minutes); Codex accepts `--codex-timeout-ms` |
48
+ | Codex entrypoint | `exec`; a native-goal runtime policy selects `app-server` |
49
+ | Codex agent concurrency | `agents.max_concurrent_threads_per_session` from `.codex/config.toml` at configuration root, otherwise the host default |
50
+ | Shell-check expected exit | `0` unless the check declares `expectedExitCode` |
51
+ | Shell-check timeout | `30000` ms unless the check declares `timeoutMs` (maximum `120000`) |
52
+ | Shell isolation, dry preparation, host diagnostic retention | Off |
53
+
54
+ Every run file/directory option accepts a relative path based on the invoking
55
+ current directory, including adapter modules, extension source files, native
56
+ binary overrides and protected roots. Sevro resolves them to canonical absolute
57
+ paths before constructing protection lists; missing output-directory suffixes
58
+ resolve through their nearest existing parent. A different `--project-root`
59
+ does not change that base. The default results directory is relative to the
60
+ selected project root. Paths inside runtime policy files retain their
61
+ [runtime-specific resolution rules](runtime-v1.md).
62
+ Extension command JSON still requires an absolute executable argv; arbitrary
63
+ argv entries are not treated as CLI path options.
64
+
65
+ Sevro prints a whitelist of effective run settings and routes on stderr before
66
+ candidate execution, including resolved storage paths, native executables,
67
+ timeouts and concurrency. JSON stdout remains one result document. The preview
68
+ does not print credential bytes, runtime environment values or native diagnostics.
69
+ The resolved executable is retained as `sevro.host.executable` in each used native
70
+ role's redacted host configuration; selecting a different path changes that
71
+ configuration's identity.
72
+
73
+ Use `sevro init` with `--template response`, `coding` or
74
+ `repository-skill` and `--directory <new-path>`. It refuses existing destinations
75
+ and writes runnable assertions, a manifest and template instructions. Follow the
76
+ [demo-to-iteration workflow](evaluation-workflow.md) for preparation, execution
77
+ and negative controls. Repository-skill requires explicit native model, effort
78
+ and authentication; dry preparation does not establish dispatch.
79
+
80
+ Actual preparation, candidate, grading and evidence-retention stage events use
81
+ stderr. Human completion includes each trial's separate states, failed-check
82
+ reasons, measured candidate time, known/unknown usage and retained paths. JSON
83
+ stdout remains one result. Each retained CLI run writes `result.json` beside run
84
+ evidence and `<results-root>/latest.json`. `sevro report --latest` reads the current
85
+ directory's default results root; add `--results-root <path>` for another root.
86
+ Dry and failed runs can be latest.
87
+
88
+ Project root identifies the evaluated project and extension discovery. It does
89
+ not populate the participant workspace: declare inline files, generated history
90
+ or a repository source in the case's fixture. See [case authoring](creating-cases.md).
91
+ Keep `.sevro/` out of Git. Results and run-state paths are excluded from project
92
+ snapshots and untracked Git identity; tracked source changes still count.
93
+
94
+ ## Explicit run settings
95
+
96
+ To repeat these selections, use the versioned [saved run settings](run-settings-v1.md)
97
+ with `--run-manifest-file ./evaluation.json`. Built-in defaults are replaced by
98
+ saved settings, then by explicit CLI options. `sevro.json` keeps its separate
99
+ runtime scope. Saved file paths use the manifest's directory; explicit CLI
100
+ paths use the invoking directory.
101
+
14
102
  An installed package records its name, version, and a digest of its packaged
15
103
  runtime and public contract files without reading runner Git metadata. The CLI
16
104
  derives the project digest from its revision and dirty patch, or from a bounded
17
105
  content snapshot when the project has no Git revision. Result and run-state
18
- directories are excluded from that snapshot. When running this source checkout
106
+ directories are excluded from that snapshot and untracked Git identity. When running this source checkout
19
107
  for coordinated development, pass `--runner-checkout-root` with the absolute
20
108
  path to this checkout. Sevro verifies that it is the code actually running and
21
109
  records its revision and dirty-patch digest instead of package provenance.
@@ -67,8 +155,7 @@ Add `--retain-host-diagnostics` to a failing invocation to retain private
67
155
  diagnostics from bundled Codex and Claude hosts. The switch defaults to off,
68
156
  applies to every selected native role, and is independent of runtime
69
157
  `nativeTranscripts`. It does not enable transcript access or change the
70
- candidate's permissions. This switch is available in the development checkout;
71
- the published `0.1.0-rc.8` does not accept it.
158
+ candidate's permissions.
72
159
 
73
160
  For example, add the switch to the Codex command above:
74
161
 
@@ -239,7 +326,8 @@ For cases with `sevro.semantic` checks, also pass
239
326
  explicit second `HostAdapter` for grading. Alternatively, select the bundled
240
327
  Codex route with `--semantic-host codex --semantic-model <model>
241
328
  --semantic-effort <effort>`. That route also requires `--codex-bin` and
242
- `--codex-auth-file`, shared with a Codex candidate route when present. The two
329
+ `--codex-auth-file`, shared with a Codex candidate route when present. The binary
330
+ defaults to the explicitly selected `codex` on `PATH`. The two
243
331
  semantic route options are exclusive. The candidate and semantic routes are
244
332
  recorded separately in evidence and evaluation identity. The semantic adapter
245
333
  runs after a complete candidate response in its own empty workspace; missing
@@ -293,7 +381,7 @@ verdict and rejects unsolicited recommendations.
293
381
 
294
382
  If the extension needs configuration, supply both
295
383
  `--extension-configuration-file` and
296
- `--extension-redacted-configuration-file` as absolute paths to JSON objects.
384
+ `--extension-redacted-configuration-file` as paths to JSON objects.
297
385
  The latter replaces secret values for the retained configuration digest. Keep
298
386
  secrets out of the command argv and the redacted file.
299
387
 
@@ -421,6 +509,3 @@ Claude also advertises `sevro.host.native-goal`. Its observer binds one persiste
421
509
  transcript to the original successful native session and retains goal status
422
510
  and objective length. Missing, ambiguous, malformed or oversized evidence is
423
511
  unavailable; an unavailable observation cannot prove goal absence.
424
-
425
- These additions require the `0.1.0-rc.2` candidate. The published `rc.1` lacks
426
- the app-server selector and native goal observations.
@@ -14,6 +14,12 @@ Local checks need no network. Run `bun run test:docs-examples` for the exact
14
14
  marked tutorial commands in fresh consumer/contributor directories, including
15
15
  expected verdicts and saved evidence. Candidate and released-package runs are
16
16
  reported separately. These commands create and clean temporary directories.
17
+ The [recipe verifier](../scripts/documentation-recipes.ts) runs the new packaged
18
+ projects separately from the current published version. It executes actual
19
+ isolated tool/check positive and negative controls and both native hosts' dry
20
+ preparation; authenticated model, plugin-hook and goal behavior remain
21
+ unexecuted by that check. Its [prerequisites](recipes.md#prepare-a-recipe-project)
22
+ are mandatory, including UV and mise's declared Node installation.
17
23
 
18
24
  `package.json.version` is the authoritative current version. `bun run docs:sync`
19
25
  updates only the marked [Current release](installing.md#current-release) section
@@ -25,6 +31,10 @@ stale output. Run synchronization after a version bump and include the resulting
25
31
  documentation change. Link installation and update instructions to this section.
26
32
  Keep historical versions, licensing transitions, and minimum supported versions
27
33
  fixed and explain their purpose beside each reference.
34
+ Link to the current-release section instead of repeating the current Sevro
35
+ version elsewhere. Describe features for the latest version, using the installed
36
+ `sevro` command in user workflows. Keep source-checkout instructions in contributor
37
+ guidance and omit notes about feature unavailability in older releases.
28
38
 
29
39
  ## External links
30
40
 
@@ -38,8 +48,8 @@ make a network job green. External success does not establish a claim's truth.
38
48
 
39
49
  - Can a new user reach installation, first success, results, troubleshooting,
40
50
  architecture, and contributing without the guide?
41
- - Are supported environments, credentials, side effects, and current versus
42
- released behavior explicit beside the relevant commands?
51
+ - Are supported environments, credentials, side effects, and prerequisites
52
+ explicit beside the relevant commands?
43
53
  - Do descriptive links, meaningful headings, source order, readable tables,
44
54
  image alternatives, and words rather than color support accessibility?
45
55
  - Are current contracts, accepted choices, research, and past evidence distinct?
@@ -0,0 +1,107 @@
1
+ # From a demo to your next evaluation
2
+
3
+ Create an evaluation, prepare it, run its assertions, and inspect the latest
4
+ result with the installed `sevro` command.
5
+
6
+ ## Try the demo and create a project
7
+
8
+ You need Bun 1.3.13. [Install Sevro](installing.md) and
9
+ [make its command available on `PATH`](installing.md#make-the-command-available),
10
+ then run:
11
+
12
+ ```sh
13
+ sevro demo
14
+ sevro init --template response --directory ./response-evaluation
15
+ cd response-evaluation
16
+ ```
17
+
18
+ The demo returns an assessed `passed` task and evidence under `.sevro/results`,
19
+ using its own temporary project without credentials. The initializer refuses
20
+ existing destinations, including directories and symbolic links. It writes
21
+ real assertions, a deterministic adapter, `run.json` and template instructions.
22
+ [Saved settings](run-settings-v1.md) use their manifest's directory for paths;
23
+ `sevro.json` retains its separate runtime scope. Keep private `.sevro/` out of Git.
24
+
25
+ ## Prepare before execution
26
+
27
+ ```sh
28
+ sevro run --run-manifest-file run.json --dry
29
+ ```
30
+
31
+ Stderr shows the whitelisted resolved settings before candidate execution:
32
+ case/source selection, routes/model/effort, trials, threshold, concurrency,
33
+ runtime selection and canonical storage paths. Invalid input names its option,
34
+ file or field; missing executables name installation or explicit override paths.
35
+ Correct the named prerequisite and repeat preparation.
36
+
37
+ Dry preparation follows normal fixture, extension and check preparation without
38
+ calling the candidate or graders. Expect
39
+ `execution=not_run grading=not_requested task=not_assessed`. It does not establish
40
+ authentication, provider model availability, native skill dispatch or model success.
41
+ See [native prerequisites](native-hosts.md) and [troubleshooting](troubleshooting.md).
42
+
43
+ ## Run, read and change one thing
44
+
45
+ ```sh
46
+ sevro run --run-manifest-file run.json
47
+ sevro report --latest
48
+ SEVRO_TEMPLATE_FAIL=1 sevro run --run-manifest-file run.json
49
+ sevro report --latest
50
+ ```
51
+
52
+ The first run's adapter returns exactly `ready`; its regex assertion must pass.
53
+ The negative control returns `wrong`: expect exit `1`, completed execution and
54
+ grading, task `failed`, and the named `ready` check's reason. Then edit `case.json`
55
+ and `adapter.ts`, prepare again and rerun. Explicit flags override saved settings;
56
+ `--trials 3` repeats the task without rewriting `run.json`.
57
+
58
+ Progress uses stderr at actual preparation, candidate, grading and evidence
59
+ retention boundaries. Trial numbers distinguish concurrent work, which may
60
+ interleave. Stage `completed` means the attempt finished; the result states whether
61
+ execution/grading succeeded. Dry runs explicitly skip candidate and grading;
62
+ failed candidates skip grading. Failed preparation never prints candidate start.
63
+ Evidence completion follows persistence. Optional synchronous engine observers
64
+ cannot change the result through a thrown exception.
65
+
66
+ Human completion shows per-trial states, failed/unavailable check reasons,
67
+ measured candidate time, known/unknown usage/cost and `result=`/`evidence=` paths.
68
+ Candidate time measures the host call, not total setup time. Absent metrics remain
69
+ `unknown`; dry duration is unknown. Prompt-only completion stays `not_assessed`.
70
+ Add `--json` to emit one `sevro.cli-result.v1` stdout document; progress remains on
71
+ stderr. See [reading results](reading-results.md).
72
+
73
+ Each retained CLI evaluation saves `result.json` beside its run evidence and
74
+ atomically publishes `<results-root>/latest.json`. Latest is the last retained CLI
75
+ result, including dry runs and failed assessments. Pre-run failures do not replace
76
+ it. Concurrent runs publish in completion order. Output files stay under excluded
77
+ results storage, preserving project identity. A latest-write failure cannot turn
78
+ an earlier failed assessment into success.
79
+
80
+ `report --latest` reads `<cwd>/.sevro/results/latest.json`. Use
81
+ `report --latest --results-root ./other-results` for another output root or working
82
+ directory. `--result-file ./path/to/result.json` selects older runs and accepts
83
+ relative paths. Reports validate referenced evidence; missing/moved evidence
84
+ produces a diagnostic. [Report v1](report-v1.md) defines fields and limits.
85
+
86
+ ## Choose another starter
87
+
88
+ | Template | Real assertions | Prerequisites and interpretation |
89
+ | ------------------ | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
90
+ | `response` | Exact `ready` response; negative control returns `wrong` | Bun; no model credentials |
91
+ | `coding` | Edit `add.sh`; isolated checks test positive, negative and zero addition | Bun plus `sandbox-exec` on macOS or `bubblewrap`/`socat` on Linux; negative control leaves subtraction bug |
92
+ | `repository-skill` | Actual `.agents/skills/startup-guide/SKILL.md` mount; explicit invocation evidence; correct start command and port | Bun, Git, Codex, native isolation, explicit model/effort and readable auth file; dry is unassessed |
93
+
94
+ From a new parent directory, use `sevro init --template coding --directory ./coding-eval`
95
+ or `sevro init --template repository-skill --directory ./skill-eval`. Read its generated
96
+ README for controls and dry/live commands. Repository-skill `command.json` records
97
+ absolute Bun/extension argv; regenerate it after moving the project.
98
+
99
+ For model evaluations, replace a saved adapter route with explicit `--host`,
100
+ `--model`, `--effort` and host authentication options. Candidate, semantic and
101
+ advisory model choices remain independent. Deterministic adapters measure wiring;
102
+ they do not measure model ability. Native runs may incur provider charges.
103
+ Automated package checks run response/coding pass/fail and native skill preparation;
104
+ authenticated native behavior and human usability remain unmeasured. The
105
+ [study protocol](usability-study.md) records deferred human measurements.
106
+ [Runnable recipes](recipes.md) cover Claude, plugins/hooks, goals, semantic grading
107
+ and host tools.
@@ -0,0 +1,64 @@
1
+ # Template and batch inspection for issue 85
2
+
3
+ On 2026-10-15, the implementation for [issue #85](https://github.com/BjRo/sevro/issues/85)
4
+ replayed the two original Codex native transcript bundles through the changed
5
+ Sevro observers and Darrow's checkpoint `67f4fcdb` extension. Bundle and decoded
6
+ source SHA-256 values matched the issue before replay. Raw bundles and output
7
+ remain private and outside Git.
8
+
9
+ | Original case | Exec submissions | Complete receipts after replay | Regraded no-agent assertion |
10
+ | --------------------------------- | ---------------- | ------------------------------ | --------------------------- |
11
+ | goal-placement-direct-same-route | 11 | 11 | passed |
12
+ | goal-placement-coupled-sequential | 10 | 10 | passed |
13
+
14
+ The five exact [retained program fixtures](../../tests/fixtures/code-mode-85/README.md)
15
+ now preserve these native ordinals and ordered submission-local labels:
16
+
17
+ - Direct 45: three `exec_command` calls.
18
+ - Direct 69 and 77: `apply_patch`, then `exec_command`.
19
+ - Coupled 30: three `exec_command` calls.
20
+ - Coupled 68: two `exec_command` calls.
21
+
22
+ Regrading replaced only the native-call and portable native-control observations
23
+ with newly inspected observations and invoked the original case's extension
24
+ checks. Both `darrow.evals.transcript.1` results changed from unavailable to
25
+ passed. These are artifact rechecks, not new candidate runs or rewritten
26
+ historical task verdicts. Deterministic consumer coverage separately confirms
27
+ that a submitted `spawn_agent` fails the no-agent assertion even when the
28
+ accepted-agent count is zero, and partial receipts remain unavailable.
29
+
30
+ ## Observed test-first slices
31
+
32
+ 1. Red — `bun test tests/codex-native-calls.test.ts -t 'retained direct ordinal 45'`: the template-command receipt was partial.
33
+ Green — `bun test tests/codex-native-calls.test.ts -t 'retained direct ordinal 45'`: 1 passed, with three bound call labels.
34
+ 2. Red — `bun test tests/codex-native-calls.test.ts -t 'retained coupled ordinal'`: both destructured batch receipts were partial.
35
+ Green — `bun test tests/codex-native-calls.test.ts -t 'retained coupled ordinal'`: 2 passed, with three and two bound call labels respectively.
36
+
37
+ Additional regressions preserve submitted spawns inside interpolated and
38
+ concatenated output, reject ambiguous bindings and composition, and keep
39
+ private source bytes out of observations. The focused native-call/control
40
+ suite passed 94 tests. These checks do not replace the full GitHub CI gate
41
+ on the PR's exact published commit.
42
+
43
+ ## Fresh native trials
44
+
45
+ Fresh trials use Darrow checkpoint `67f4fcdb`, native Codex 0.162.1 app-server,
46
+ candidate and semantic grader `gpt-6-luna/medium`, passive ownership, N=1,
47
+ jobs=1, threshold=1, and a 1,200,000 ms timeout with retained diagnostics.
48
+ They explicitly select the changed Sevro source worktree through
49
+ `SEVRO_CHECKOUT`; they are development evidence, not published-package trials.
50
+
51
+ The direct trial, run `510f99df-882f-48fe-b893-d4b6fba8864d`, completed with
52
+ 9/10 checks passing and the no-agent assertion unavailable. Its new submission
53
+ 29 maps a callback that invokes tools over a command array; submission 74 uses
54
+ an output callback with a conditional expression. Both forms remain outside
55
+ the [bounded inspection contract](../native-controls.md) and correctly retain
56
+ partial receipts. The five original programs no longer cause that limitation.
57
+
58
+ The coupled trial, run `7f9f5d8a-641e-4744-a986-05298100541c`, completed with
59
+ 10/11 checks passing and the no-agent assertion unavailable. Submissions 29,
60
+ 45, and 68 each use a `for ... of` output loop with a conditional expression
61
+ and `String(...)` conversion. That output body is also outside the bounded
62
+ subset and remains partial. Neither fresh run is a passing placement trial;
63
+ their unavailable verdicts are preserved rather than treating zero accepted
64
+ agents as proof of absence.
package/docs/evidence.md CHANGED
@@ -8,6 +8,7 @@ Keep their observations intact:
8
8
  - [Claude continuation](claude-continuation-validation.md)
9
9
  - [Concurrency](concurrency-validation.md)
10
10
  - [Code Mode control observations](evidence/code-mode-controls-71.md)
11
+ - [Code Mode template and batch programs](evidence/code-mode-programs-85.md)
11
12
  - [Candidate Codex configuration and user-skill state](evidence/codex-state-73/README.md)
12
13
  - [Configuration roots](configuration-root-validation.md)
13
14
  - [Fixture cleanup](fixture-cleanup-validation.md)
@@ -5,6 +5,27 @@ Darrow checkout, Codex/Claude login, or model credentials are required. The
5
5
  adapter returns `ready`; Sevro grades that response against `^ready$`.
6
6
  This teaches the runner lifecycle, not the quality of an actual model.
7
7
 
8
+ <a id="built-in-demo-in-this-checkout"></a>
9
+
10
+ ## Built-in demo
11
+
12
+ With [Sevro installed and on `PATH`](installing.md#make-the-command-available), run:
13
+
14
+ ```sh
15
+ sevro demo
16
+ ```
17
+
18
+ It prints `execution=completed grading=completed task=passed` and an absolute
19
+ `evidence=` path under `.sevro/results` in your current directory. The demo owns
20
+ its packaged case and adapter, uses a separate temporary project, and ignores
21
+ your project's `sevro.json` and native configuration. No credentials are needed.
22
+ Effective run settings appear on stderr before evaluation. Add `--json` for one
23
+ versioned result on stdout.
24
+
25
+ Continue with [the initialized evaluation workflow](evaluation-workflow.md) to
26
+ create response, coding or repository-skill assertions, prepare them, inspect
27
+ progress and access the latest result without copying an evidence path.
28
+
8
29
  ## From an installed package
9
30
 
10
31
  Prepare a new directory:
@@ -16,8 +37,8 @@ bun init -y
16
37
  ```
17
38
 
18
39
  Install the [current release](installing.md#current-release) there, then return
19
- here to run the example. Read [version differences](installing.md#published-package-and-checkout)
20
- for the current docs, source-available license, and earlier BSL releases.
40
+ here to run the example. See [licensing](licensing.md) for the current terms and
41
+ the licenses retained by historical releases.
21
42
 
22
43
  Make the local installation available on `PATH` for this shell, then call
23
44
  `sevro` directly. Bun must also be on `PATH`.
@@ -32,7 +32,10 @@ The CLI derives `runnerBuildDigest` from sorted packaged file paths and their
32
32
  SHA-256 bytes under `src/`, `schemas/`, `docs/`, and `examples/`, plus `README.md` and
33
33
  `package.json`. For a Git project, `projectDigest` binds the revision and dirty
34
34
  patch digest without the checkout path. Without a revision it hashes a bounded
35
- file snapshot, excluding `.git`, results, and run-state storage. A caller may
35
+ file snapshot, excluding `.git`, results, and run-state storage. Results and
36
+ run-state storage are also excluded from untracked Git identity, so repeated
37
+ project-local output does not change the project digest. Tracked source changes
38
+ remain included, even under a selected storage directory. A caller may
36
39
  provide explicit 64-character digests for controlled development runs.
37
40
 
38
41
  The prefix separates this digest from component digests and future identity
@@ -48,6 +51,17 @@ The effective positive trial job limit is retained as `configuration.redacted.jo
48
51
  and enters `configurationDigest`. Changing parallelism changes evaluation identity,
49
52
  even when the condition, host route, trial count, and assessment threshold match.
50
53
 
54
+ CLI [saved run settings](run-settings-v1.md) resolve before evaluation. Equivalent
55
+ manifest and explicit selections produce the same semantic inputs. CLI evidence
56
+ also retains `configuration.redacted.runSettings` for shell isolation, explicit
57
+ protected roots, deprecated runtime selections, selected native timeouts and
58
+ Claude project-settings authority. These values enter the
59
+ configuration digest. Optional `configuration.resolved` stores the whitelisted
60
+ operator preview outside that digest, including operational paths and the
61
+ selected manifest location. Relocating output storage alone does not alter
62
+ identity. A tracked manifest remains project content under the normal project
63
+ identity rules.
64
+
51
65
  Extension runs retain the operator-supplied redacted extension configuration
52
66
  under `configuration.redacted.extensionConfiguration`, alongside
53
67
  `extensionConfigurationDigest`. The retained copy is the same snapshot used
@@ -1,8 +1,9 @@
1
1
  # Install, update, or remove Sevro
2
2
 
3
3
  Use Bun 1.3.13 for this TypeScript/Bun CLI. Installation does not make it a
4
- Node-only executable. Full validation and native isolation target macOS;
5
- Linux and Windows are unverified.
4
+ Node-only executable. Full validation and native isolation target macOS and
5
+ Ubuntu 24.04 x64. See the [bounded platform evidence](native-hosts.md#validated-platform-scope);
6
+ Windows remains unverified.
6
7
 
7
8
  ## Install locally
8
9
 
@@ -22,10 +23,10 @@ Local installation records the selected version in the project lockfile.
22
23
 
23
24
  ## Current release
24
25
 
25
- The current release is `0.1.0-rc.12`. Install this exact version:
26
+ The current release is `0.1.0-rc.13`. Install this exact version:
26
27
 
27
28
  ```sh
28
- bun add --exact @bjoernrochel/sevro@0.1.0-rc.12
29
+ bun add --exact @bjoernrochel/sevro@0.1.0-rc.13
29
30
  ```
30
31
 
31
32
  <!-- sevro-current-release:end -->
@@ -46,17 +47,16 @@ no adjustment is needed. Bun must also be on `PATH`: the executable uses
46
47
 
47
48
  ## Published package and checkout
48
49
 
49
- The [current release](#current-release) includes runtime
50
- configuration, native goals, selected plugin hooks, current onboarding, and
51
- [source-available licensing terms](licensing.md). Earlier `rc.1` and `rc.2`
52
- packages retain BUSL-1.1. The license transition began with `0.1.0-rc.3`.
53
- Source checkouts can include changes beyond their
54
- package version; compare the exact release tag when establishing provenance.
50
+ See [licensing](licensing.md) for Sevro's source-available terms and historical
51
+ licensing transitions.
55
52
 
56
- The candidate npm package contains sources, schemas, examples, documentation,
53
+ The npm package contains sources, schemas, examples, documentation,
57
54
  LICENSE, and contribution guidance. It excludes tests, developer scripts, Git
58
55
  metadata, and repository guide mounts. Use a source checkout for the guide.
59
56
 
57
+ Source checkouts can include changes beyond their package version; compare the
58
+ exact release tag when establishing provenance.
59
+
60
60
  ## Work from source
61
61
 
62
62
  ```sh
@@ -22,10 +22,14 @@ one receipt per native `exec` submission, with its native `ordinal`, inspection
22
22
  The index is local to that submission; native ordinals are not synthesized.
23
23
  The original direct-call list and total `submittedExecCalls` stay unchanged.
24
24
 
25
- The current source checkout statically inspects bounded straight-line Code Mode
25
+ Sevro statically inspects bounded straight-line Code Mode
26
26
  programs without executing them. It accepts literal tool calls, constant result
27
- bindings, direct `text`/`store` output, and literal `Promise.all`/`allSettled`
28
- batches with result-only `forEach(text)` or `for ... of` output. It requires one
27
+ bindings, template interpolation and `+` composition of inspected expressions,
28
+ direct `text`/`store` output, and literal `Promise.all`/`allSettled` batches.
29
+ Batch results may use flat constant array bindings or result-only
30
+ `forEach(text)` or `for ... of` output. Destructuring defaults, rest elements,
31
+ holes, nested bindings, duplicate names, and shadowed globals remain unsupported.
32
+ It requires one
29
33
  unique, later native completion for the same call ID with an intact successful
30
34
  script envelope. Tool argument strings, comments and printed tool-looking text
31
35
  do not become call labels. A rejected tool request returned by a completed
@@ -6,6 +6,23 @@ Claude Code), a supported native CLI, and model
6
6
  authentication. Complete commands/rules live in the [CLI reference](development-cli.md)
7
7
  and [Codex host reference](codex-host.md).
8
8
 
9
+ ## Validated platform scope
10
+
11
+ The full source quality gate has passed on macOS arm64 and Ubuntu 24.04 x64.
12
+ Ubuntu evidence includes [the successful source CI run for `cd7542bf`](https://github.com/BjRo/sevro/actions/runs/38692971517).
13
+ That run exercises deterministic native-host protocols and real filesystem,
14
+ process, network and shell isolation with the pinned prerequisites in
15
+ [the quality contract](typescript-quality.md#environment-and-ci). It does not
16
+ establish authenticated model success for every host/model combination.
17
+ This is prior source evidence; changed candidates still need their own green CI.
18
+ Other Linux distributions and Windows are unverified.
19
+
20
+ On Ubuntu, install `bubblewrap`, `ripgrep`, and `socat`. Unprivileged user
21
+ namespaces must work; Ubuntu's AppArmor policy may need a `bwrap` profile.
22
+ [CI's setup script](../scripts/setup-ci-linux.sh) supplies and probes the
23
+ supported runner setup. Use it as an explicit development setup step where
24
+ appropriate; do not assume all Linux hosts have that configuration.
25
+
9
26
  ## Codex
10
27
 
11
28
  Select `--host codex`, absolute `--codex-bin` and `--codex-auth-file` paths,