callva-harness-runner 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- callva_harness_runner-0.3.0/.gitignore +12 -0
- callva_harness_runner-0.3.0/DESIGN.md +107 -0
- callva_harness_runner-0.3.0/LICENSE +202 -0
- callva_harness_runner-0.3.0/PKG-INFO +192 -0
- callva_harness_runner-0.3.0/README.md +167 -0
- callva_harness_runner-0.3.0/callva/harness_runner/__init__.py +55 -0
- callva_harness_runner-0.3.0/callva/harness_runner/_launcher.py +48 -0
- callva_harness_runner-0.3.0/callva/harness_runner/binaries.py +81 -0
- callva_harness_runner-0.3.0/callva/harness_runner/classify.py +125 -0
- callva_harness_runner-0.3.0/callva/harness_runner/claude.py +157 -0
- callva_harness_runner-0.3.0/callva/harness_runner/codex.py +185 -0
- callva_harness_runner-0.3.0/callva/harness_runner/guard.py +78 -0
- callva_harness_runner-0.3.0/callva/harness_runner/launch.py +118 -0
- callva_harness_runner-0.3.0/callva/harness_runner/profile.py +371 -0
- callva_harness_runner-0.3.0/callva/harness_runner/profile.schema.json +38 -0
- callva_harness_runner-0.3.0/callva/harness_runner/renamed.py +32 -0
- callva_harness_runner-0.3.0/callva/harness_runner/result.py +90 -0
- callva_harness_runner-0.3.0/callva/harness_runner/run.py +291 -0
- callva_harness_runner-0.3.0/callva/harness_runner/tree.py +119 -0
- callva_harness_runner-0.3.0/callva/harness_runner/version.py +1 -0
- callva_harness_runner-0.3.0/pyproject.toml +49 -0
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# Design
|
|
2
|
+
|
|
3
|
+
`callva-harness-runner` runs one turn of an agent harness from a profile and hands back one result. It exists because the same fifty lines of launch-and-parse code had been written nine times across one owner's tools, each with a different bug, and a tenth was about to be written.
|
|
4
|
+
|
|
5
|
+
## 1. Scope
|
|
6
|
+
|
|
7
|
+
One turn: a prompt goes in, a harness runs to its end or its deadline, one `Result` comes out. The library runs the harness through its vendor's SDK against the CLI installed on the machine, configures it only from the profile's explicit knobs, keeps every process it starts under the deadline, reads the harness's report, and classifies a failure by what the harness said. That is the whole job.
|
|
8
|
+
|
|
9
|
+
### Non-goals
|
|
10
|
+
|
|
11
|
+
It does not schedule, queue, retry, or loop. It holds no ledger, no session store, no conversation memory. It renders no progress; it hands events to a callback and the caller renders. It does not know what a task, a message or a job is. It ships no presets and no fences: it never decides what a turn may do. Presets belong to consumers.
|
|
12
|
+
|
|
13
|
+
## 2. Names
|
|
14
|
+
|
|
15
|
+
The vocabulary: a harness is the vendor's agent runtime (Claude Code, the Codex CLI); a profile is one harness configuration; a run is one turn. The library holds no queue and no long-lived consumer of work; those belong to its callers.
|
|
16
|
+
|
|
17
|
+
The distribution is `callva-harness-runner` and the import path is `callva.harness_runner`: the distribution name is the import path with dashes, the same rule `callva-livekit` follows, and `callva` is a namespace package with no `__init__.py` so the two coexist in one environment. It was `callva-agentworker` (`callva.agentworker`) up to 0.2.0; the README maps each old name to its new one, and every old name is refused with a message naming the new one. The three nouns a caller uses are `Profile` (how to run), `Session` (which conversation), and `Result` (what came back). `run()` is the one verb. `probe()` answers whether a harness is present.
|
|
18
|
+
|
|
19
|
+
## 3. Harnesses
|
|
20
|
+
|
|
21
|
+
`claude` is Claude Code through `claude-agent-sdk`; `codex` is Codex through `openai-codex`, which drives `codex app-server`. Both are dependencies pinned exactly, because an SDK release changes the protocol it speaks and the flags it passes.
|
|
22
|
+
|
|
23
|
+
Both SDKs bundle a CLI, and the library never runs it: it always passes the installed CLI (`cli_path` on claude, the `app-server` launch on codex). The CLI is the profile's `cli_path`, or else the first `claude` or `codex` on the caller's `PATH`, then in the places a login shell would have added and a launchd job does not: `~/.local/bin`, fnm and nvm node directories, Homebrew. Codex is started at the file a symlinked install points to, because it finds its companion executables beside the path it was started from.
|
|
24
|
+
|
|
25
|
+
Claude always runs on the `claude_code` system-prompt preset. The SDK's own default is a minimal prompt that is not Claude Code's; the preset keeps the prompt a `claude -p` turn has. Neither harness's system prompt is ever replaced; `append_system_prompt` adds to claude's and is codex's developer instructions.
|
|
26
|
+
|
|
27
|
+
Codex's turn is read from the turn's notification stream, not from `thread.run()`, which raises on a failed turn and drops `codexErrorInfo`.
|
|
28
|
+
|
|
29
|
+
## 4. Profile
|
|
30
|
+
|
|
31
|
+
A `Profile` is a frozen, flat record of explicit knobs. There is no inheritance and no named fence. A knob at its default leaves the harness's own default in place. A knob one harness cannot honour is refused for that harness when the profile is built, with a `ProfileError` naming the knob; it is never ignored while the turn runs. Each knob's docstring on `Profile` is the home of what it does on each harness, and the README carries the same table for readers of the package page.
|
|
32
|
+
|
|
33
|
+
The knobs: `harness`, `cli_path`, `model`, `effort`, `timeout_seconds`, `budget_usd`; `tools`, `allowed_tools`, `disallowed_tools`, `permission_mode`, `setting_sources`; `settings`, `mcp_config`, `strict_mcp`; `add_dirs`, `append_system_prompt`, `output_schema`, `name`; `env_set`, `env_remove`, `strip_session_markers`; `codex_config`, `approval_policy`, `service_tier`, `bypass_hook_trust`; `claude_extra_args`, `codex_extra_args`.
|
|
34
|
+
|
|
35
|
+
`Profile.from_dict()` validates against `profile.schema.json`, which ships in the package so that every consumer's configuration file is checked by the same rule. Unknown keys are refused, and the 0.1.x keys (`fence`, `allow_tools`, `env`, `extra_args`) are refused with a message naming the knobs that replace them.
|
|
36
|
+
|
|
37
|
+
Hook trust on codex is a knob that is refused on both harnesses. Over `app-server` there is no way to run a hook the machine has not trusted: `bypass_hook_trust` is not a config key (`--strict-config` rejects it) and the CLI's `--dangerously-bypass-hook-trust`, accepted before `app-server`, does not make an untrusted hook run, although the same hook runs under `codex exec` with it. Hooks the machine trusts run.
|
|
38
|
+
|
|
39
|
+
### Environment
|
|
40
|
+
|
|
41
|
+
Both SDKs merge the profile's variables over the environment the Python process has, and neither can remove one. The library therefore puts a launcher of its own where the SDK expects the CLI: a two-line shell script that runs `_launcher.py` with `python -I -S`, which removes the variables the turn removes and `exec`s the real CLI. What reaches the launcher is a spec of names and paths; a variable's value only ever travels through the SDK's own env option.
|
|
42
|
+
|
|
43
|
+
The harness's environment is the caller's `environ` (this process's by default), with the profile's `env_remove` globs and, unless `strip_session_markers` is false, `SESSION_MARKERS` removed, and with `env_set` and the call's `extra_env` set on top. A variable in `env_set` is never removed, and a profile that both sets and removes one name is refused. On claude the variables the SDK itself sets for the CLI (its entrypoint, its version, its session-state flag, `PWD`) are the SDK's values and are not removed.
|
|
44
|
+
|
|
45
|
+
## 5. Session
|
|
46
|
+
|
|
47
|
+
`Session.fresh()` starts a new conversation. On claude the library generates the id before launch and pins it, so a turn that is killed is still addressable. On codex the id is the thread id the app-server gives, reported even when the turn is later killed. `Session.pinned(id)` lets the caller choose the id; it is claude-only and refused on codex. `Session.resume(id)` continues an existing conversation on either harness; on codex the thread is resumed with the profile's model, config, developer instructions, service tier and approval policy.
|
|
48
|
+
|
|
49
|
+
## 6. Harness guard
|
|
50
|
+
|
|
51
|
+
`TESTED_VERSIONS` declares, per harness, the inclusive range of CLI versions the release was tested with: the low end is the CLI installed where the release was tested, the high end is the CLI the pinned SDK bundles. Before every turn the installed CLI's `--version` is read (cached per binary file):
|
|
52
|
+
|
|
53
|
+
- below the range, or unreadable: the turn is refused with `incompatible_harness`, naming both versions;
|
|
54
|
+
- above the range: the turn runs and `Result.warnings` names both versions;
|
|
55
|
+
- no CLI: `binary_missing`, as it was.
|
|
56
|
+
|
|
57
|
+
## 7. Running
|
|
58
|
+
|
|
59
|
+
The turn runs on a thread of its own; the calling thread holds the deadline and the cancel. The launcher records the harness's pid and command line and makes the harness a session leader. When the deadline passes or the caller's `threading.Event` is set, the library first writes a stop file that the launcher checks before any later start, then finds the harness's tree three ways: by parentage from the harness, by the process groups of what it found, and by the sessions of what it found, the harness's own included. The harnesses give each tool command a process group of its own, and a command that backgrounds a child and exits leaves that child with its group and session but a new parent, so parentage alone would miss it. The tree gets SIGTERM, three seconds, then SIGKILL; the caller's own group and session are never touched. Then the SDK is closed.
|
|
60
|
+
|
|
61
|
+
A caller killed outright cannot do any of this, and the harness's tree outlives it.
|
|
62
|
+
|
|
63
|
+
## 8. Reading
|
|
64
|
+
|
|
65
|
+
Claude: the SDK's `ResultMessage` gives `result` as the answer (its `errors` when there is none), `structured_output`, `session_id`, `total_cost_usd`, `duration_ms`, `num_turns`, the four token counts from `usage`, the model that ran from `modelUsage`, `permission_denials`, and `subtype` with `is_error` as the verdict. The SDK raises after it hands over an error result; the result is the report and the raise is not. The last `api_retry` event is kept, because a turn that dies by the deadline while the harness retries a 401 or a 429 has already said why.
|
|
66
|
+
|
|
67
|
+
Codex: the turn's `item/completed` notifications give the answer (the last agent message), `thread/tokenUsage/updated` gives this turn's tokens (the thread's running total less what it held before the turn), and `turn/completed` gives the status, the duration and the error with its `codexErrorInfo`. Cost is `None`: codex reports none under ChatGPT authentication.
|
|
68
|
+
|
|
69
|
+
When `output_schema` is given and the harness returned no structured field, the answer text is parsed as JSON; if that fails, the turn is an `invalid_output` failure with the text kept.
|
|
70
|
+
|
|
71
|
+
## 9. Result
|
|
72
|
+
|
|
73
|
+
| Field | Meaning |
|
|
74
|
+
|---|---|
|
|
75
|
+
| `ok` | the harness confirmed a completed turn: claude `subtype == "success"` and `is_error` false; codex status `completed`, no error, and a non-empty answer. |
|
|
76
|
+
| `harness` | which harness ran. |
|
|
77
|
+
| `answer` | the final text, possibly empty on failure. |
|
|
78
|
+
| `structured` | the parsed structured answer, or `None`. |
|
|
79
|
+
| `session_id` | claude session id or codex thread id; set whenever it is known, including after a kill. |
|
|
80
|
+
| `model` | the model that actually ran when claude reports it, else the requested one, else `None`. |
|
|
81
|
+
| `cost_usd`, `duration_ms`, `num_turns`, `tokens` | as reported; `None` where the harness does not say. |
|
|
82
|
+
| `permission_denials` | claude's list, empty on codex. |
|
|
83
|
+
| `failure` | `None` when `ok`; otherwise a kind and the harness's own sentence. |
|
|
84
|
+
| `exit_code` | the exit code the claude SDK reported for a failed process, else `None`. |
|
|
85
|
+
| `stderr_tail` | the last lines of the harness's stderr, for a person. |
|
|
86
|
+
| `command` | the CLI command line the launcher started. |
|
|
87
|
+
| `raw` | claude's result message as a dict, or codex's `{"turn", "items", "errors"}`, for anything this table left out. |
|
|
88
|
+
| `harness_version` | the CLI version the guard read. |
|
|
89
|
+
| `warnings` | the guard's warning when the CLI is newer than tested. |
|
|
90
|
+
|
|
91
|
+
## 10. Failure kinds
|
|
92
|
+
|
|
93
|
+
`quota`: the subscription or rate limit is spent; the turn can be retried later without changing anything. `model_refused`: the harness will not run this model; retrying changes nothing. `not_authenticated`: no login or an invalid key. `timeout`: the deadline passed and the tree was killed. `cancelled`: the caller's event was set. `binary_missing`: no harness found. `max_turns`, `budget`: the harness ended the turn on its own limit. `invalid_output`: a schema was required and nothing parsed. `error`: the harness reported a failure this table does not name; the sentence is in `message`. `crash`: the harness ended without a report. `incompatible_harness`: the installed CLI is older than the tested range, or its version cannot be read.
|
|
94
|
+
|
|
95
|
+
Classification reads the harness's own verdict first (claude's `error_max_turns` and budget subtypes; codex's `codexErrorInfo`: `usageLimitExceeded` and `rateLimitExceeded` are `quota`, `unauthorized` is `not_authenticated`, `sessionBudgetExceeded` is `budget`), then an HTTP status the harness reports (claude's `api_error_status` or its last `api_retry` status, codex's `httpStatusCode`: 429 is `quota`, 401 is `not_authenticated`), then the sentence, in a fixed order: quota before model refusal, because a spent subscription on one model says both, and only one of them is a pause. Codex wraps a provider's HTTP error as JSON; the sentence inside it is what is read and kept.
|
|
96
|
+
|
|
97
|
+
## 11. Constraints worth knowing
|
|
98
|
+
|
|
99
|
+
The cost claude reports is its own estimate and includes subagents; a turn that dies reports no cost at all, so an accumulated total is a floor. Codex under ChatGPT authentication reports tokens but no cost. A profile with no knobs runs the harness exactly as the machine configures it: user settings, hooks and MCP servers included. Hooks on codex run only when the machine trusts them.
|
|
100
|
+
|
|
101
|
+
## 12. Versioning
|
|
102
|
+
|
|
103
|
+
SemVer with the 0.x rule: the minor is the breaking position. A changed field meaning, a removed knob or a changed default is a minor; an added result field, an added knob whose default leaves the harness alone, or a widened tested range is a patch. `callva/harness_runner/version.py` is the one home of the number.
|
|
104
|
+
|
|
105
|
+
## 13. Later
|
|
106
|
+
|
|
107
|
+
Progress rendering helpers, if two consumers end up writing the same one. Cost lookup for codex by token price table, if anyone needs a number rather than `None`. A parent watch in the launcher, if a crashed caller's orphaned tree becomes a real problem.
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
5
|
+
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
7
|
+
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright [yyyy] [name of copyright owner]
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: callva-harness-runner
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: One headless turn of a coding agent (Claude Code or Codex) through the vendor SDKs: explicit knobs, a deadline that kills the tree, one result.
|
|
5
|
+
Project-URL: Homepage, https://github.com/callva-io/harness-runner
|
|
6
|
+
Project-URL: Source, https://github.com/callva-io/harness-runner
|
|
7
|
+
Author: CallVA
|
|
8
|
+
License-Expression: Apache-2.0
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: agent,claude,claude-agent-sdk,codex,harness,headless,openai-codex
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
17
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
18
|
+
Requires-Python: >=3.11
|
|
19
|
+
Requires-Dist: claude-agent-sdk==0.2.161
|
|
20
|
+
Requires-Dist: openai-codex==0.159.0
|
|
21
|
+
Provides-Extra: dev
|
|
22
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
23
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
|
|
26
|
+
# callva-harness-runner
|
|
27
|
+
|
|
28
|
+
One run of an agent harness, as a Python library. A harness is the vendor's agent runtime: Claude Code, driven through Anthropic's `claude-agent-sdk`, or the Codex CLI, driven through OpenAI's `openai-codex`, always the CLI installed on the machine. A profile is one harness configuration, a flat set of explicit knobs. A run is one turn: a prompt goes in, the harness works to its end or its deadline, and one `Result` comes out, with the failure named. The deadline kills everything the harness started. The library ships no presets and no fences: what a run may do is whatever its profile says, and nothing else.
|
|
29
|
+
|
|
30
|
+
`DESIGN.md` is the contract. The source is at https://github.com/callva-io/harness-runner.
|
|
31
|
+
|
|
32
|
+
Formerly `callva-agentworker`, whose last release is 0.2.0. The 0.3.0 renames, with no aliases (each old name is refused with a message naming the new one):
|
|
33
|
+
|
|
34
|
+
| 0.2.0 | 0.3.0 |
|
|
35
|
+
|---|---|
|
|
36
|
+
| distribution `callva-agentworker` | `callva-harness-runner` |
|
|
37
|
+
| import `callva.agentworker` | `callva.harness_runner` |
|
|
38
|
+
| profile knob `engine` | `harness` |
|
|
39
|
+
| `Result.engine` | `Result.harness` |
|
|
40
|
+
| `Result.engine_version` | `Result.harness_version` |
|
|
41
|
+
| failure kind `incompatible_engine` (`FailureKind.INCOMPATIBLE_ENGINE`) | `incompatible_harness` (`FailureKind.INCOMPATIBLE_HARNESS`) |
|
|
42
|
+
| `ENGINES` | `HARNESSES` |
|
|
43
|
+
| `Probe.engine` | `Probe.harness` |
|
|
44
|
+
|
|
45
|
+
## Install
|
|
46
|
+
|
|
47
|
+
```
|
|
48
|
+
pip install callva-harness-runner
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
It depends on `claude-agent-sdk==0.2.161` and `openai-codex==0.159.0`, pinned exactly; together they add about 0.5 GB, most of it the CLIs both SDKs bundle. The bundled CLIs are never run: the library always runs the CLI installed on the machine.
|
|
52
|
+
|
|
53
|
+
In a PEP 723 script, pin the exact version so nothing outside the script's own release changes what it runs:
|
|
54
|
+
|
|
55
|
+
```python
|
|
56
|
+
# /// script
|
|
57
|
+
# dependencies = ["callva-harness-runner==0.3.0"]
|
|
58
|
+
# ///
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## Use
|
|
62
|
+
|
|
63
|
+
```python
|
|
64
|
+
from callva.harness_runner import Profile, Session, run
|
|
65
|
+
|
|
66
|
+
profile = Profile.from_dict(config["profile"]) # your configuration, your knobs
|
|
67
|
+
result = run("Summarise README.md in one line.", profile, cwd="/path/to/project")
|
|
68
|
+
if result.ok:
|
|
69
|
+
print(result.answer, result.cost_usd, result.session_id)
|
|
70
|
+
else:
|
|
71
|
+
print(result.failure.kind, result.failure.message)
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
`run()` never raises for anything the harness did. Every ending is a `Result`; `result.failure.kind` says what kind, and `result.failure.message` carries the harness's own sentence. It raises `ValueError` only for a request it cannot express, such as a pinned session on codex.
|
|
75
|
+
|
|
76
|
+
`run(prompt, profile, cwd, *, session=None, environ=None, extra_env=None, on_event=None, cancel=None, stderr_tail=40)`:
|
|
77
|
+
|
|
78
|
+
- `session`: `Session.fresh()` (the default), `Session.pinned(id)` or `Session.resume(id)`, below.
|
|
79
|
+
- `environ`: the environment the harness starts from; this process's by default. A variable absent from it is absent in the harness, even though both SDKs inherit this process's environment.
|
|
80
|
+
- `extra_env`: variables set on top of `environ` and the profile's `env_set`.
|
|
81
|
+
- `on_event`: called with one dict per harness event while the turn runs. On claude it is the SDK's message: a system message is the harness's own event (`subtype` `init` carries `apiKeySource`, `model` and `claude_code_version`), any other message is its fields plus `type`, the SDK class name. On codex it is `{"method", "params"}`, one app-server notification.
|
|
82
|
+
- `cancel`: a `threading.Event`; setting it ends the turn and kills its processes.
|
|
83
|
+
|
|
84
|
+
## Profile
|
|
85
|
+
|
|
86
|
+
A profile is a flat set of explicit knobs. There is no inheritance, no preset and no fence. A knob left at its default leaves the harness's own default in place. A knob one harness cannot honour is refused for that harness with a `ProfileError` when the profile is built, never ignored while it runs. Each knob's docstring on `Profile` says what it does on each harness; `PROFILE_SCHEMA` (shipped as `profile.schema.json`) validates a profile kept in configuration, and `Profile.from_dict` refuses unknown keys.
|
|
87
|
+
|
|
88
|
+
| Knob | claude | codex |
|
|
89
|
+
|---|---|---|
|
|
90
|
+
| `harness` | `claude` | `codex` |
|
|
91
|
+
| `cli_path` | the CLI the SDK launches; `None` finds the installed `claude` | the binary whose `app-server` the SDK talks to; `None` finds the installed `codex` |
|
|
92
|
+
| `model` | `--model` | the thread's model |
|
|
93
|
+
| `effort` | `--effort` | the turn's reasoning effort |
|
|
94
|
+
| `timeout_seconds` | the deadline; default 600 | the same |
|
|
95
|
+
| `budget_usd` | `--max-budget-usd` | refused: codex reports no cost |
|
|
96
|
+
| `tools` | `--tools`, the built-in tools that exist; `[]` removes them all | refused |
|
|
97
|
+
| `allowed_tools` | `--allowedTools`, patterns that run without a prompt | refused |
|
|
98
|
+
| `disallowed_tools` | `--disallowedTools` | refused |
|
|
99
|
+
| `permission_mode` | `--permission-mode` | refused: use `approval_policy` and `codex_config` |
|
|
100
|
+
| `setting_sources` | `--setting-sources`; `["project", "local"]` loads the target's CLAUDE.md, rules and hooks without the user's | refused |
|
|
101
|
+
| `settings` | `--settings`, an object or path layered above the settings files: sandbox, permissions | refused: use `codex_config` |
|
|
102
|
+
| `mcp_config` | `--mcp-config`, `{"mcpServers": {...}}` or a path | refused: `mcp_servers` in `codex_config` |
|
|
103
|
+
| `strict_mcp` | `--strict-mcp-config` | refused |
|
|
104
|
+
| `add_dirs` | `--add-dir` per entry | refused: give the directory access in `codex_config` permissions |
|
|
105
|
+
| `append_system_prompt` | appended to the Claude Code system prompt | the thread's developer instructions |
|
|
106
|
+
| `output_schema` | `--json-schema` | the turn's output schema |
|
|
107
|
+
| `name` | `--name` | the thread's name |
|
|
108
|
+
| `env_set` | set in the harness's environment | the same |
|
|
109
|
+
| `env_remove` | names or globs removed from what the harness inherits | the same |
|
|
110
|
+
| `strip_session_markers` | also remove `SESSION_MARKERS`; default true | the same |
|
|
111
|
+
| `codex_config` | refused: use `settings` | the thread's config, keyed as in `config.toml`: permission profiles, `sandbox_mode`, network, `mcp_servers` |
|
|
112
|
+
| `approval_policy` | refused: use `permission_mode` | `never` (escalations refused) or `auto_review` (codex's reviewer decides); `None` is the SDK default, `auto_review` |
|
|
113
|
+
| `service_tier` | refused | the thread's service tier |
|
|
114
|
+
| `bypass_hook_trust` | refused: claude has no hook trust | refused when true: the app-server has no hook-trust bypass, so hooks run only when the machine trusts them |
|
|
115
|
+
| `claude_extra_args` | extra CLI flags, `{"flag": "value"}` or `{"flag": None}` | refused |
|
|
116
|
+
| `codex_extra_args` | refused | arguments before `app-server`, such as `["-c", "key=value"]` |
|
|
117
|
+
|
|
118
|
+
Claude always runs on the `claude_code` system-prompt preset, so a turn has the same prompt `claude -p` has; the library never replaces either harness's system prompt, and `append_system_prompt` only adds to it.
|
|
119
|
+
|
|
120
|
+
### Environment
|
|
121
|
+
|
|
122
|
+
Both SDKs only add to the environment they inherit. To remove a variable for real, the library puts a small launcher of its own in front of the CLI: the SDK starts the launcher, the launcher removes what the profile removes and becomes the CLI. The launcher learns names, never values. `SESSION_MARKERS` are the variables a running Claude Code session leaves in its children (`CLAUDECODE`, `CLAUDE_CODE_SESSION_ID`, `CLAUDE_EFFORT`, `AI_AGENT` and the rest); a child harness that inherits them believes it runs inside that session, so they are removed unless `strip_session_markers` is false. On claude the values the SDK itself sets for the CLI, such as its entrypoint, stay.
|
|
123
|
+
|
|
124
|
+
### Deadline and cancel
|
|
125
|
+
|
|
126
|
+
The launcher makes the harness a session leader. When the deadline passes, or `cancel` is set, no further harness start is allowed, and the harness, every process it started, and everything those left behind in their process group or session are sent SIGTERM and then SIGKILL, grandchildren included. A caller that is itself killed cannot do this, and the harness's tree then outlives it.
|
|
127
|
+
|
|
128
|
+
## Harness guard
|
|
129
|
+
|
|
130
|
+
`TESTED_VERSIONS` holds, per harness, the range of CLI versions this release was tested with: claude 2.1.280 to 2.1.284, codex 0.159.0. Before a turn the library reads the installed CLI's `--version`:
|
|
131
|
+
|
|
132
|
+
- older than the range: the turn is refused with failure kind `incompatible_harness`, and the message names the installed version and the tested one;
|
|
133
|
+
- newer than the range: the turn runs, and `result.warnings` names both versions;
|
|
134
|
+
- no CLI: `binary_missing`, as before.
|
|
135
|
+
|
|
136
|
+
## Session
|
|
137
|
+
|
|
138
|
+
`Session.fresh()` starts a conversation; on claude the id is chosen before launch so a killed turn is still addressable, on codex it is the thread id the app-server gives. `Session.resume(id)` continues one on either harness. `Session.pinned(id)` chooses the id and is claude-only.
|
|
139
|
+
|
|
140
|
+
## Result
|
|
141
|
+
|
|
142
|
+
| Field | Meaning |
|
|
143
|
+
|---|---|
|
|
144
|
+
| `ok` | the harness confirmed a completed turn with an answer |
|
|
145
|
+
| `harness` | `claude` or `codex` |
|
|
146
|
+
| `answer` | the final text |
|
|
147
|
+
| `structured` | the parsed answer when `output_schema` was given |
|
|
148
|
+
| `session_id` | claude session id or codex thread id, set whenever known, including after a kill |
|
|
149
|
+
| `model` | the model that ran when claude reports it, else the requested one |
|
|
150
|
+
| `cost_usd` | claude's own estimate; `None` on codex |
|
|
151
|
+
| `duration_ms`, `num_turns` | as reported; `num_turns` is `None` on codex |
|
|
152
|
+
| `tokens` | `input`, `output`, `cache_read`, `cache_creation` for this turn |
|
|
153
|
+
| `permission_denials` | claude's list; empty on codex |
|
|
154
|
+
| `failure` | `None` when `ok`; else `kind` and the harness's own `message` |
|
|
155
|
+
| `exit_code` | the exit code the claude SDK reported for a failed process, else `None` |
|
|
156
|
+
| `stderr_tail` | the harness's last stderr lines, for a person |
|
|
157
|
+
| `command` | the CLI command line the launcher started |
|
|
158
|
+
| `raw` | claude's result message, or codex's `{"turn", "items", "errors"}` |
|
|
159
|
+
| `harness_version` | the installed CLI version the guard read |
|
|
160
|
+
| `warnings` | the guard's warning when the CLI is newer than tested |
|
|
161
|
+
|
|
162
|
+
### Failure kinds
|
|
163
|
+
|
|
164
|
+
`quota`, `model_refused`, `not_authenticated`, `timeout`, `cancelled`, `binary_missing`, `max_turns`, `budget`, `invalid_output`, `error`, `crash`, `incompatible_harness`. The harness's own verdict is read first (claude's result subtype, codex's `codexErrorInfo`), then an HTTP status it reports, then its sentence: a spent quota before a refused model, because a subscription spent on one model says both and only one of them is a pause. A claude turn that times out while the harness is retrying a 401 or a 429 is `not_authenticated` or `quota`. `failure.retryable` is true for the kinds where running the same turn again later can succeed.
|
|
165
|
+
|
|
166
|
+
## Changelog
|
|
167
|
+
|
|
168
|
+
### 0.3.0
|
|
169
|
+
|
|
170
|
+
The library is renamed and speaks of harnesses, profiles and runs; its behaviour is exactly 0.2.0's. What a 0.2.0 consumer must change: depend on `callva-harness-runner` and import `callva.harness_runner`; write `harness` where a profile said `engine`; read `Result.harness` and `Result.harness_version`; branch on `incompatible_harness`; import `HARNESSES`. The table at the top maps every old name.
|
|
171
|
+
|
|
172
|
+
### 0.2.0
|
|
173
|
+
|
|
174
|
+
The engines now run through the vendors' SDKs, and the profile is explicit knobs. What a 0.1.x consumer must change:
|
|
175
|
+
|
|
176
|
+
- `fence` is gone, and so is its default. A 0.1.x profile that named no fence ran fenced to `read`; the same profile now runs with the engine's own defaults. State what the turn may do with `tools`, `allowed_tools`, `disallowed_tools`, `permission_mode`, `setting_sources`, `settings`, `mcp_config` and `strict_mcp` on claude, and `codex_config` and `approval_policy` on codex. `Profile.from_dict` refuses a `fence` key with a message naming these knobs, and `Profile.fence` no longer exists.
|
|
177
|
+
- `allow_tools` becomes `tools` plus `allowed_tools`. `env` (`allow`, `deny`, `set`) becomes `env_set`, `env_remove` and `strip_session_markers`; there is no allow-list. `extra_args` becomes `claude_extra_args` (a mapping) or `codex_extra_args`. `EnvPolicy`, `DEFAULT_DENY` and `FENCES` are no longer exported.
|
|
178
|
+
- `run()` loses `schema` (now the profile's `output_schema`), `name` (now the profile's `name`) and `prompt_via` (the prompt always travels over the SDK's own channel).
|
|
179
|
+
- A knob the engine cannot honour is refused instead of ignored: `budget_usd` on codex, `service_tier` on claude.
|
|
180
|
+
- `on_event` receives SDK messages (claude) and app-server notifications (codex) instead of CLI JSON lines. `raw` changes shape the same way, `command` is the command line the launcher started, and `exit_code` is `None` unless the claude SDK reported one.
|
|
181
|
+
- A CLI older than the tested range is refused with the new kind `incompatible_engine`; `Result` gains `engine_version` and `warnings`.
|
|
182
|
+
- Two dependencies arrive, about 0.5 GB.
|
|
183
|
+
|
|
184
|
+
Unchanged: `Profile.from_dict`, `Session.fresh`, `Session.pinned`, `Session.resume`, `run(prompt, profile, cwd, session=, environ=, extra_env=, on_event=, cancel=)`, `FailureKind`, and every 0.1.x `Result` field.
|
|
185
|
+
|
|
186
|
+
## Versioning
|
|
187
|
+
|
|
188
|
+
SemVer with the 0.x rule: the minor is the breaking position. A changed field meaning, a removed knob or a changed default is a minor; an added result field, an added knob whose default leaves the harness alone, or a widened tested range is a patch.
|
|
189
|
+
|
|
190
|
+
## License
|
|
191
|
+
|
|
192
|
+
Apache-2.0.
|