owenloop 0.5.12 → 0.5.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/docs/mcp-charter-eval.md +58 -38
- package/package.json +1 -1
- package/plugins/claude-code/plugin/.claude-plugin/plugin.json +1 -1
- package/plugins/claude-code/plugin/.mcp.json +1 -1
- package/plugins/codex/plugins/owenloop/.codex-plugin/plugin.json +1 -1
- package/plugins/codex/plugins/owenloop/.mcp.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,13 @@ All notable changes to this project are documented here. The format is based on
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
|
|
5
5
|
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.5.13](https://github.com/typicalday/owenloop/compare/v0.5.12...v0.5.13) (2026-08-21)
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
### Bug Fixes
|
|
11
|
+
|
|
12
|
+
* **eval:** score charter no-match safety and catalog discovery independently ([#267](https://github.com/typicalday/owenloop/issues/267)) ([6c2cfea](https://github.com/typicalday/owenloop/commit/6c2cfea1ffa75168864c9b1d3d7e14b980857cdd))
|
|
13
|
+
|
|
7
14
|
## [0.5.12](https://github.com/typicalday/owenloop/compare/v0.5.11...v0.5.12) (2026-08-21)
|
|
8
15
|
|
|
9
16
|
|
package/docs/mcp-charter-eval.md
CHANGED
|
@@ -7,14 +7,19 @@ logged in. It is deliberately outside the normal test and CI commands.
|
|
|
7
7
|
|
|
8
8
|
Run the evaluation from a checked-out repository:
|
|
9
9
|
|
|
10
|
-
npm run eval:mcp-charter -- --output docs/evals/mcp-charter-baseline.json
|
|
10
|
+
npm run eval:mcp-charter -- --expected-node-version v22.22.3 --output docs/evals/mcp-charter-baseline.json
|
|
11
11
|
|
|
12
12
|
To pin a model for one harness, pass --claude-model <model> and/or
|
|
13
13
|
--codex-model <model>. The same choices can be supplied with
|
|
14
14
|
OWENLOOP_MCP_CHARTER_CLAUDE_MODEL and OWENLOOP_MCP_CHARTER_CODEX_MODEL. Each
|
|
15
15
|
task has a five-minute deadline by default; pass --task-timeout-ms <positive
|
|
16
16
|
integer> (or set OWENLOOP_MCP_CHARTER_TASK_TIMEOUT_MS) to set a different
|
|
17
|
-
bounded budget.
|
|
17
|
+
bounded budget. Every live invocation must also pass an exact
|
|
18
|
+
--expected-node-version value (or set
|
|
19
|
+
OWENLOOP_MCP_CHARTER_EXPECTED_NODE_VERSION). The runner refuses to create a
|
|
20
|
+
harness session unless that declared value equals `process.version`; it never
|
|
21
|
+
derives the pin from the current process. Reuse the same literal pin for every
|
|
22
|
+
report in a cohort.
|
|
18
23
|
|
|
19
24
|
## Method
|
|
20
25
|
|
|
@@ -49,25 +54,35 @@ top-level text or Codex's completed final-answer item); reasoning, stderr, tool
|
|
|
49
54
|
output, MCP metadata, and streamed/intermediate text are excluded. Neither
|
|
50
55
|
response text nor generic telemetry is used for a numeric score.
|
|
51
56
|
|
|
52
|
-
For
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
57
|
+
For every non-ambiguous task, the scorer records two independent dimensions:
|
|
58
|
+
**catalog discovery** records whether the first call is
|
|
59
|
+
`list_workflows({})`; **task safety** records whether a no-match task makes no
|
|
60
|
+
`start_run`, or whether a match task starts its expected workflow without any
|
|
61
|
+
conflicting or malformed selection. A malformed, schema-rejected `start_run`
|
|
62
|
+
still counts as a received start and is unsafe. The headline `passed` remains
|
|
63
|
+
the conjunction of both dimensions. The rejected alternative was a safety-only
|
|
64
|
+
headline for no-match tasks: that would allow a safe zero-call refusal to pass
|
|
65
|
+
without measuring adherence to catalog discovery. The remaining two ambiguous
|
|
66
|
+
tasks are recorded as observations only. Each harness therefore reports passed
|
|
67
|
+
/ 4, not passed / 6.
|
|
68
|
+
|
|
69
|
+
Schema v2 reports record both the declared `expectedNodeVersion` and actual
|
|
70
|
+
`nodeVersion`, alongside the run timestamp, fixture catalog digest, charter
|
|
71
|
+
hash, configured/reported model and harness version where available, ordered
|
|
72
|
+
calls, response evidence, classifications, dimensions, and score. The
|
|
73
|
+
preflight check prevents a wrong-treatment run before it spends model quota;
|
|
74
|
+
`assertComparableNodeTreatment` then rejects cohorts with missing pins,
|
|
75
|
+
expected/actual mismatches, or different declared or actual Node versions. A
|
|
76
|
+
charter edit changes the served-byte hash, so it creates a new attributable
|
|
77
|
+
score rather than inheriting an old baseline.
|
|
64
78
|
|
|
65
79
|
## W3.3b negative result
|
|
66
80
|
|
|
67
81
|
W3.3b does **not** ship a compose-clause charter change. The pre-compose
|
|
68
82
|
charter and its generated baseline are restored because this evaluation provides
|
|
69
|
-
no evidence that the candidate clause helps.
|
|
70
|
-
|
|
83
|
+
no evidence that the candidate clause helps. That negative-result unit
|
|
84
|
+
authorized no further cohort, Node pinning, charter wording iteration, or
|
|
85
|
+
baseline generation.
|
|
71
86
|
The completed deliverable is this reproducible negative result and its separate
|
|
72
87
|
evaluator finding.
|
|
73
88
|
|
|
@@ -100,14 +115,16 @@ unsafe starts: its score differences cannot be attributed to the wording.
|
|
|
100
115
|
|
|
101
116
|
### What the scorer measures
|
|
102
117
|
|
|
103
|
-
`discoveredFirst` and `scoreTask`
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
118
|
+
At the time of the W3.3b negative result, `discoveredFirst` and `scoreTask`
|
|
119
|
+
required every non-ambiguous task to start with `list_workflows({})` before the
|
|
120
|
+
no-match branch tested `starts.length === 0`. A zero-call refusal consequently
|
|
121
|
+
failed because `calls[0]` was absent even when it made no unsafe `start_run`.
|
|
122
|
+
The landed evaluator now records catalog discovery and no-start safety
|
|
123
|
+
independently, as described in [Method](#method), without changing the
|
|
124
|
+
conjunctive headline. The structured trace remains the score; refusal prose,
|
|
125
|
+
catalog language, or a no-inline promise cannot substitute for an observed
|
|
126
|
+
first tool call or prove a zero-call trace. Raw response evidence remains
|
|
127
|
+
qualitative and separate from the score.
|
|
111
128
|
|
|
112
129
|
### Fixed-fixture limitation
|
|
113
130
|
|
|
@@ -141,28 +158,31 @@ comment remain attached to PR #255 as evidence. Node drift is a secondary run
|
|
|
141
158
|
limitation, not a reason to authorize another cohort: pinning Node would not
|
|
142
159
|
give this fixture a discriminating acceptance gate.
|
|
143
160
|
|
|
144
|
-
##
|
|
161
|
+
## Landed 2026-08-21: separate no-start safety from catalog-discovery adherence, and make the evaluator runtime treatment explicit
|
|
145
162
|
|
|
146
|
-
|
|
163
|
+
This follow-up unit discharged two independent evaluator defects:
|
|
147
164
|
|
|
148
165
|
- **Conflated no-match criterion.** The scorer combines “did not start an
|
|
149
166
|
unsupported workflow” and “called `list_workflows({})` first” into one pass.
|
|
150
167
|
A correct zero-call refusal is safe on the first dimension but fails the
|
|
151
|
-
combined score on the second.
|
|
152
|
-
|
|
153
|
-
- **Uncontrolled Node treatment.** The harness and protocol
|
|
154
|
-
enforce arm-local Node provenance as a treatment invariant.
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
168
|
+
combined score on the second. The scorer now records and tests both dimensions
|
|
169
|
+
explicitly, while retaining their conjunction as the headline result.
|
|
170
|
+
- **Uncontrolled Node treatment.** The harness and protocol previously did not
|
|
171
|
+
pin Node or enforce arm-local Node provenance as a treatment invariant.
|
|
172
|
+
Schema v2 now requires a preflight pin and rejects incomparable report cohorts.
|
|
173
|
+
|
|
174
|
+
The fixed fixture, denominator, ambiguous-task treatment, served charter, and
|
|
175
|
+
historical baseline remain unchanged. The schema v1
|
|
176
|
+
[baseline](evals/mcp-charter-baseline.json) is immutable; its stored traces
|
|
177
|
+
re-score under schema v2 to the same Claude Code 4/4 and Codex 4/4 headline
|
|
178
|
+
totals, with both clear dimensions passing.
|
|
159
179
|
|
|
160
180
|
### Outcome
|
|
161
181
|
|
|
162
|
-
The compose clause and its candidate baseline are dropped. PR #255 remains
|
|
163
|
-
|
|
164
|
-
documents the evaluation's limits
|
|
165
|
-
|
|
182
|
+
The compose clause and its candidate baseline are dropped. PR #255 remains the
|
|
183
|
+
reviewable negative-result deliverable: it preserves the raw evidence and
|
|
184
|
+
documents the evaluation's limits; this unit implements the named follow-up
|
|
185
|
+
without reopening the stopped protocol.
|
|
166
186
|
|
|
167
187
|
## Safety and baseline updates
|
|
168
188
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "owenloop",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.13",
|
|
4
4
|
"description": "Deterministic rails for agentic workflows. Declare steps and dependencies; the engine guarantees order, redoes what changes invalidate, and stops what keeps failing — instead of hoping the agent follows through.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"author": "typicalday",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "owenloop",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.13",
|
|
4
4
|
"description": "Claude Code driver for owenloop — cross-harness shift and conduct skills driven by the owenloop CLI, plus one hub control-plane MCP server backing an author skill and a fallback worker, for conducting and authoring workflows against your hosted owenloop server.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Typical Day"
|