owenloop 0.5.8 → 0.5.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +69 -0
- package/README.md +81 -39
- package/bin/owenloop.mjs +16 -1
- package/dist/packages/work/src/agent/loop.d.ts +23 -8
- package/dist/packages/work/src/agent/loop.js +55 -6
- package/dist/packages/work/src/bundle/pull.d.ts +21 -0
- package/dist/packages/work/src/bundle/pull.js +118 -0
- package/dist/packages/work/src/consumed-verifier.js +6 -17
- package/dist/packages/work/src/exec/instructions.d.ts +4 -1
- package/dist/packages/work/src/exec/instructions.js +3 -0
- package/dist/packages/work/src/exec/loop.d.ts +2 -2
- package/dist/packages/work/src/exec/loop.js +273 -46
- package/dist/packages/work/src/exec/receipt.d.ts +7 -7
- package/dist/packages/work/src/exec/receipt.js +6 -6
- package/dist/packages/work/src/harness/claude.js +116 -0
- package/dist/packages/work/src/harness/codex.d.ts +8 -0
- package/dist/packages/work/src/harness/codex.js +27 -1
- package/dist/packages/work/src/harness/contract.d.ts +9 -0
- package/dist/packages/work/src/hub/client.d.ts +19 -4
- package/dist/packages/work/src/hub/client.js +7 -3
- package/dist/packages/work/src/hub/types.d.ts +84 -0
- package/dist/packages/work/src/main.js +1 -0
- package/dist/packages/work/src/roles/agent-run.js +57 -32
- package/dist/packages/work/src/roles/exec.js +27 -8
- package/dist/packages/work/src/roles/settings.js +2 -0
- package/dist/packages/work/src/roles/shift.js +14 -2
- package/dist/packages/work/src/roles/util.d.ts +1 -0
- package/dist/packages/work/src/roles/util.js +10 -0
- package/dist/packages/work/src/settings/hub-roster-cache.d.ts +68 -0
- package/dist/packages/work/src/settings/hub-roster-cache.js +370 -0
- package/dist/packages/work/src/settings/roster.d.ts +84 -0
- package/dist/packages/work/src/settings/roster.js +364 -5
- package/dist/packages/work/src/settings/serving.d.ts +17 -0
- package/dist/packages/work/src/settings/serving.js +79 -0
- package/dist/packages/work/src/settings/settings.d.ts +14 -1
- package/dist/packages/work/src/settings/settings.js +14 -0
- package/dist/packages/work/src/shift/loop.d.ts +108 -5
- package/dist/packages/work/src/shift/loop.js +416 -54
- package/dist/packages/work/src/shift/protocol.d.ts +7 -5
- package/dist/packages/work/src/shift/runtime.d.ts +18 -0
- package/dist/packages/work/src/shift/runtime.js +104 -5
- package/dist/packages/work/src/shift/server.js +2 -0
- package/dist/packages/work/src/shift/spawn.d.ts +17 -2
- package/dist/packages/work/src/shift/spawn.js +16 -13
- package/dist/packages/work/src/shift/state.d.ts +29 -2
- package/dist/packages/work/src/shift/state.js +45 -15
- package/dist/packages/work/src/submit-proof.d.ts +11 -8
- package/dist/packages/work/src/submit-proof.js +11 -8
- package/dist/packages/work/src/usage.d.ts +1 -1
- package/dist/packages/work/src/usage.js +7 -1
- package/dist/packages/work/src/util/modifier-init.d.ts +7 -0
- package/dist/packages/work/src/util/modifier-init.js +73 -0
- package/dist/src/bundle/tar.d.ts +9 -4
- package/dist/src/bundle/tar.js +9 -4
- package/dist/src/capabilities.d.ts +76 -6
- package/dist/src/capabilities.js +49 -6
- package/dist/src/capability-mapping-client.d.ts +29 -0
- package/dist/src/capability-mapping-client.js +160 -0
- package/dist/src/cli.d.ts +16 -3
- package/dist/src/cli.js +1925 -281
- package/dist/src/config-dir.d.ts +1 -1
- package/dist/src/config-dir.js +1 -1
- package/dist/src/crypto/index.d.ts +1 -1
- package/dist/src/crypto/index.js +1 -1
- package/dist/src/crypto/org-root.d.ts +30 -12
- package/dist/src/crypto/org-root.js +162 -14
- package/dist/src/crypto/records.d.ts +2 -0
- package/dist/src/crypto/records.js +2 -0
- package/dist/src/defs.d.ts +38 -2
- package/dist/src/defs.js +593 -86
- package/dist/src/engine.d.ts +103 -25
- package/dist/src/engine.js +331 -79
- package/dist/src/hub.d.ts +77 -0
- package/dist/src/hub.js +162 -0
- package/dist/src/implements.d.ts +56 -0
- package/dist/src/implements.js +790 -0
- package/dist/src/index.d.ts +11 -9
- package/dist/src/index.js +6 -5
- package/dist/src/mcp/serve.d.ts +22 -0
- package/dist/src/mcp/serve.js +330 -41
- package/dist/src/mcp/server.d.ts +10 -7
- package/dist/src/mcp/server.js +53 -8
- package/dist/src/model.d.ts +48 -10
- package/dist/src/model.js +252 -99
- package/dist/src/order-resolver.d.ts +1 -0
- package/dist/src/order-resolver.js +8 -1
- package/dist/src/schemas/index.d.ts +11 -0
- package/dist/src/schemas/order.v1.schema.json +7 -1
- package/dist/src/store/def-source.d.ts +19 -0
- package/dist/src/store/def-source.js +87 -17
- package/dist/src/store/gc-recovery.d.ts +46 -0
- package/dist/src/store/gc-recovery.js +258 -0
- package/dist/src/store/gc.d.ts +94 -0
- package/dist/src/store/gc.js +562 -0
- package/dist/src/store/index.d.ts +3 -1
- package/dist/src/store/index.js +2 -0
- package/dist/src/store/install.d.ts +34 -5
- package/dist/src/store/install.js +73 -11
- package/dist/src/store/pre-commit-verifier.js +42 -2
- package/dist/src/store/snapshot-guard.d.ts +18 -0
- package/dist/src/store/snapshot-guard.js +94 -0
- package/dist/src/store.d.ts +69 -4
- package/dist/src/store.js +254 -16
- package/dist/src/types.d.ts +101 -20
- package/dist/src/util.js +4 -1
- package/docs/authoring.md +235 -10
- package/docs/bundles.md +45 -2
- package/docs/cli.md +437 -55
- package/docs/crypto.md +32 -10
- package/docs/design.md +177 -47
- package/docs/evals/mcp-charter-baseline.json +428 -0
- package/docs/mcp-charter-eval.md +124 -0
- package/docs/routing.md +189 -34
- package/docs/shift-logs.md +38 -9
- package/docs/wire-contracts.md +78 -14
- package/examples/workflows/compile-dev-playbook.yaml +1 -1
- package/examples/workflows/delivery.yaml +31 -14
- package/examples/workflows/ship.yaml +38 -4
- package/package.json +2 -1
- package/plugins/claude-code/plugin/.claude-plugin/plugin.json +1 -1
- package/plugins/claude-code/plugin/.mcp.json +1 -1
- package/plugins/claude-code/plugin/skills/author/SKILL.md +22 -0
- package/plugins/claude-code/plugin/skills/ephemeral/SKILL.md +95 -0
- package/plugins/claude-code/plugin/skills/graduate/SKILL.md +156 -0
- package/plugins/claude-code/plugin/skills/plan/SKILL.md +236 -0
- package/plugins/codex/plugins/owenloop/.codex-plugin/plugin.json +1 -1
- package/plugins/codex/plugins/owenloop/.mcp.json +1 -1
- package/plugins/codex/plugins/owenloop/skills/author/SKILL.md +22 -0
- package/plugins/codex/plugins/owenloop/skills/ephemeral/SKILL.md +95 -0
- package/plugins/codex/plugins/owenloop/skills/graduate/SKILL.md +156 -0
- package/plugins/codex/plugins/owenloop/skills/plan/SKILL.md +236 -0
|
@@ -0,0 +1,428 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schemaVersion": 1,
|
|
3
|
+
"generatedAt": "2026-08-20T13:53:53.975Z",
|
|
4
|
+
"nodeVersion": "v22.22.3",
|
|
5
|
+
"fixture": {
|
|
6
|
+
"version": 1,
|
|
7
|
+
"catalogDigest": "5833c986e8cbb4261a6e34253a1170f40287eb00ccb52e3fcee429b6f3aabc94"
|
|
8
|
+
},
|
|
9
|
+
"scores": [
|
|
10
|
+
{
|
|
11
|
+
"harness": {
|
|
12
|
+
"id": "claude-code",
|
|
13
|
+
"reportedModel": "claude-opus-5",
|
|
14
|
+
"version": "2.1.236"
|
|
15
|
+
},
|
|
16
|
+
"charterSha256": "35773e2e269844fbb81acec0c07774f5674d51aa47304f6e0346458a13f9a2a2",
|
|
17
|
+
"passed": 4,
|
|
18
|
+
"denominator": 4,
|
|
19
|
+
"percentage": 100,
|
|
20
|
+
"tasks": [
|
|
21
|
+
{
|
|
22
|
+
"id": "match-delivery",
|
|
23
|
+
"kind": "match",
|
|
24
|
+
"request": "Implement an approved pagination change in an existing TypeScript repository, run its checks, and prepare the change for review.",
|
|
25
|
+
"expectedWorkflow": "code-delivery",
|
|
26
|
+
"calls": [
|
|
27
|
+
{
|
|
28
|
+
"sequence": 1,
|
|
29
|
+
"name": "list_workflows",
|
|
30
|
+
"arguments": {}
|
|
31
|
+
},
|
|
32
|
+
{
|
|
33
|
+
"sequence": 2,
|
|
34
|
+
"name": "get_workflow",
|
|
35
|
+
"arguments": {
|
|
36
|
+
"name": "code-delivery"
|
|
37
|
+
}
|
|
38
|
+
},
|
|
39
|
+
{
|
|
40
|
+
"sequence": 3,
|
|
41
|
+
"name": "start_run",
|
|
42
|
+
"arguments": {
|
|
43
|
+
"workflow_name": "code-delivery",
|
|
44
|
+
"provide": {
|
|
45
|
+
"proposal": {
|
|
46
|
+
"text": "Implement an approved pagination change in an existing TypeScript repository, run the repository's checks, and prepare the change for review."
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
},
|
|
51
|
+
{
|
|
52
|
+
"sequence": 4,
|
|
53
|
+
"name": "wake",
|
|
54
|
+
"arguments": {}
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"sequence": 5,
|
|
58
|
+
"name": "whats_next",
|
|
59
|
+
"arguments": {
|
|
60
|
+
"workflow": "code-delivery"
|
|
61
|
+
}
|
|
62
|
+
},
|
|
63
|
+
{
|
|
64
|
+
"sequence": 6,
|
|
65
|
+
"name": "wake",
|
|
66
|
+
"arguments": {
|
|
67
|
+
"cursor": 1
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
],
|
|
71
|
+
"classification": "passed",
|
|
72
|
+
"passed": true,
|
|
73
|
+
"responseEvidence": [
|
|
74
|
+
"## Status: run started, but the control plane isn't handing back work\n\n**What I did**\n\n1. **Read the catalog** (`list_workflows`) — two published playbooks: `code-delivery` and `library-build`.\n2. **Inspected `code-delivery`** — it's an exact match for your request: *\"Plan, implement, test, and review an approved change in an existing software repository.\"* Its `notFor` explicitly excludes the library workflow, so there's no ambiguity.\n3. **Started the run** (`start_run`) with your request as the `proposal` input. Accepted and recorded.\n4. **Parked on `wake`, then called `whats_next`** to pick up the first order or gate.\n\n**Where it stalled**\n\n`wake` and `whats_next` are both returning bare acknowledgements — no cursor, no run ID, no orders, no gate. I retried `wake` with an explicit cursor and got the same. So the run is recorded, but no crew step has been dispatched back to me, and there's nothing yet for me to relay, approve, or route.\n\nI'm not going to paper over that. I won't `submit` fabricated artifacts to make the run look like it's progressing, and I won't drop into the repo and hand-implement the pagination change myself — `code-delivery` is exactly the crew that serves this capability, so freelancing it would bypass the plan/implement/test/review chain you published, including its review step.\n\n**Two things I need from you**\n\n*On the stall:* do you want me to keep polling `wake`, or is the runner something you need to kick on your side? If the environment isn't actually wired to dispatch crews right now, tell me and I'll say so rather than spin.\n\n*On the proposal — this is worth fixing regardless:* I seeded the run with only your one-line request, which is thin for an \"approved change.\" A crew will almost certainly escalate for these, and I'd rather feed them in up front via `provide_input`:\n\n- **Which repository**, and the target branch.\n- **What the pagination change actually is** — offset/limit → cursor-based? Page-size cap? New `nextCursor` in responses? This drives whether it's a breaking API change.\n- **Which endpoints or modules** are in scope.\n- **Backward compatibility** — must existing offset callers keep working, or is a clean break approved?\n- **Where \"approved\" is recorded** — an issue, ADR, or design doc the crew can cite.\n- **What \"prepare for review\"** means here — draft PR, specific reviewers, changeset/changelog entry?\n\nGive me those and I'll load them into the run so the crews aren't blocked the moment dispatch resumes."
|
|
75
|
+
]
|
|
76
|
+
},
|
|
77
|
+
{
|
|
78
|
+
"id": "match-library",
|
|
79
|
+
"kind": "match",
|
|
80
|
+
"request": "Research Orlando tourism across attractions, neighborhoods, transport, dining, lodging, and events; collect source media and build a linked knowledge library.",
|
|
81
|
+
"expectedWorkflow": "library-build",
|
|
82
|
+
"calls": [
|
|
83
|
+
{
|
|
84
|
+
"sequence": 1,
|
|
85
|
+
"name": "list_workflows",
|
|
86
|
+
"arguments": {}
|
|
87
|
+
},
|
|
88
|
+
{
|
|
89
|
+
"sequence": 2,
|
|
90
|
+
"name": "get_workflow",
|
|
91
|
+
"arguments": {
|
|
92
|
+
"name": "library-build"
|
|
93
|
+
}
|
|
94
|
+
},
|
|
95
|
+
{
|
|
96
|
+
"sequence": 3,
|
|
97
|
+
"name": "start_run",
|
|
98
|
+
"arguments": {
|
|
99
|
+
"workflow_name": "library-build",
|
|
100
|
+
"provide": {
|
|
101
|
+
"topic": {
|
|
102
|
+
"text": "Orlando tourism. Research across six subtopics: (1) attractions — theme parks, water parks, museums, nature and outdoor sites; (2) neighborhoods — Downtown, International Drive, Winter Park, Lake Nona, Thornton Park, Celebration and other visitor-relevant districts; (3) transport — airports (MCO, SFB), rental cars, SunRail, Lynx buses, Brightline rail, rideshare, park shuttles, walkability; (4) dining — signature restaurants, food halls, local specialties, park dining, budget-to-fine-dining range; (5) lodging — on-property resorts, I-Drive hotels, vacation rentals, boutique and budget options; (6) events — annual festivals, park seasonal events, conventions, sports. Acquire source and media artifacts (photos, maps, official pages) for each subtopic and synthesize a cross-linked knowledge library."
|
|
103
|
+
}
|
|
104
|
+
},
|
|
105
|
+
"scope": "Six subtopics: attractions, neighborhoods, transport, dining, lodging, events. Deliverable is a cross-linked knowledge library with collected source media.",
|
|
106
|
+
"priority": "normal"
|
|
107
|
+
}
|
|
108
|
+
},
|
|
109
|
+
{
|
|
110
|
+
"sequence": 4,
|
|
111
|
+
"name": "whats_next",
|
|
112
|
+
"arguments": {
|
|
113
|
+
"workflow": "library-build"
|
|
114
|
+
}
|
|
115
|
+
},
|
|
116
|
+
{
|
|
117
|
+
"sequence": 5,
|
|
118
|
+
"name": "wake",
|
|
119
|
+
"arguments": {}
|
|
120
|
+
}
|
|
121
|
+
],
|
|
122
|
+
"classification": "passed",
|
|
123
|
+
"passed": true,
|
|
124
|
+
"responseEvidence": [
|
|
125
|
+
"I've set this up through the workflow catalog rather than researching it inline. Here's where things stand.\n\n## What I did\n\n**1. Checked the catalog.** Two published playbooks: `code-delivery` and `library-build`. Your request — research a topic across subtopics, collect source media, synthesize a linked library — is an exact match for `library-build`, which even names \"research and curate a topical media or knowledge library\" as what `code-delivery` is *not* for.\n\n**2. Inspected the definition.** It takes one input, `topic` (an object with a `text` field), and covers the full arc: research across subtopics → acquire source and media artifacts → synthesize a linked knowledge library. No missing inputs to ask you about.\n\n**3. Started the run** with your six subtopics written out concretely so the crews don't have to guess scope:\n\n| Subtopic | Scope I seeded |\n|---|---|\n| Attractions | Theme parks, water parks, museums, nature/outdoor sites |\n| Neighborhoods | Downtown, I-Drive, Winter Park, Lake Nona, Thornton Park, Celebration |\n| Transport | MCO/SFB, rental cars, SunRail, Lynx, Brightline, rideshare, park shuttles, walkability |\n| Dining | Signature restaurants, food halls, local specialties, park dining, budget→fine |\n| Lodging | On-property resorts, I-Drive hotels, vacation rentals, boutique, budget |\n| Events | Annual festivals, park seasonal events, conventions, sports |\n\nPriority `normal`, with the media-acquisition and cross-linking requirements stated in the scope.\n\n## Where it stands\n\nThe run is recorded. I then ticked it with `whats_next` and parked on `wake` — both came back as bare acknowledgements with no queued gate, no worker ask, and no cursor movement. So the run is registered but hasn't yet handed back anything for you or me to act on.\n\nI'm deliberately **not** writing the Orlando research myself. That's the crews' work, and producing it inline would give you an unsourced document dressed up as workflow output — no real media artifacts, no real citations.\n\n## What I need from you\n\nNothing blocking. Two things worth deciding:\n\n- **Priority** — I set `normal`. If this is tied to actual travel dates, say the word and I'll bump it to `high`.\n- **Any scope I guessed at** — I added Celebration and walkability on my own read of \"neighborhoods\" and \"transport.\" Easy to adjust, and I can add subtopics like day trips to the coast or Kennedy Space Center if you want them in the library.\n\nI'll keep watching the cursor and surface the first gate, artifact, or worker question as soon as it lands."
|
|
126
|
+
]
|
|
127
|
+
},
|
|
128
|
+
{
|
|
129
|
+
"id": "no-match-finance",
|
|
130
|
+
"kind": "no-match",
|
|
131
|
+
"request": "Reconcile six months of bank statements, categorize expenses, and prepare a business tax filing package.",
|
|
132
|
+
"calls": [
|
|
133
|
+
{
|
|
134
|
+
"sequence": 1,
|
|
135
|
+
"name": "list_workflows",
|
|
136
|
+
"arguments": {}
|
|
137
|
+
},
|
|
138
|
+
{
|
|
139
|
+
"sequence": 2,
|
|
140
|
+
"name": "get_workflow",
|
|
141
|
+
"arguments": {
|
|
142
|
+
"name": "code-delivery"
|
|
143
|
+
}
|
|
144
|
+
},
|
|
145
|
+
{
|
|
146
|
+
"sequence": 3,
|
|
147
|
+
"name": "get_workflow",
|
|
148
|
+
"arguments": {
|
|
149
|
+
"name": "library-build"
|
|
150
|
+
}
|
|
151
|
+
}
|
|
152
|
+
],
|
|
153
|
+
"classification": "passed",
|
|
154
|
+
"passed": true,
|
|
155
|
+
"responseEvidence": [
|
|
156
|
+
"**No published playbook fits this request.** The catalog has exactly two workflows, and neither is close:\n\n| Workflow | Purpose | Explicitly not for |\n|---|---|---|\n| `code-delivery` | Implement/test/review an approved change in an existing software repo | knowledge/media libraries |\n| `library-build` | Research a topic and synthesize a sourced knowledge library | software changes |\n\nThere's no bookkeeping, reconciliation, expense-categorization, or tax-preparation crew available. I didn't start a run, because forcing your finances through a code-delivery or research pipeline would produce plausible-looking output backed by no actual accounting work — worse than no output at all.\n\nTwo other blockers worth naming up front, independent of tooling:\n\n1. **I don't have the source data.** Six months of statements, your chart of accounts, prior-year return, and entity type (sole prop / S-corp / partnership / C-corp) would all be needed before any real reconciliation could begin.\n2. **A filing package is a signed legal document.** Even with the data, the final return should go out under a credentialed preparer (CPA or EA), not an unreviewed agent artifact.\n\n**How would you like to proceed?** Some options:\n\n- **Publish a workflow for this.** If you (or whoever maintains the catalog) can stand up a bookkeeping/tax crew, I'll route the job to it properly. This is the clean fix if it's recurring work.\n- **Scope a narrower slice I can do inline.** For example: if you hand me exported CSVs, I can do the mechanical parts — de-duplicating transactions, flagging uncategorized or anomalous charges, building a categorized expense summary and a reconciliation worksheet against statement balances. That's a working paper for your accountant, explicitly not a filing package.\n- **Hand off entirely.** I can draft the intake brief and document request list for a CPA so the engagement starts with everything they need.\n\nTell me which direction, and roughly what shape your records are in (bank CSVs? QuickBooks/Xero? PDFs only?) plus your entity type — that determines how much of this is actually automatable."
|
|
157
|
+
]
|
|
158
|
+
},
|
|
159
|
+
{
|
|
160
|
+
"id": "no-match-operations",
|
|
161
|
+
"kind": "no-match",
|
|
162
|
+
"request": "Compare medical providers, verify insurance coverage, schedule an appointment, and arrange transportation.",
|
|
163
|
+
"calls": [
|
|
164
|
+
{
|
|
165
|
+
"sequence": 1,
|
|
166
|
+
"name": "list_workflows",
|
|
167
|
+
"arguments": {}
|
|
168
|
+
},
|
|
169
|
+
{
|
|
170
|
+
"sequence": 2,
|
|
171
|
+
"name": "get_workflow",
|
|
172
|
+
"arguments": {
|
|
173
|
+
"name": "code-delivery"
|
|
174
|
+
}
|
|
175
|
+
},
|
|
176
|
+
{
|
|
177
|
+
"sequence": 3,
|
|
178
|
+
"name": "get_workflow",
|
|
179
|
+
"arguments": {
|
|
180
|
+
"name": "library-build"
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
],
|
|
184
|
+
"classification": "passed",
|
|
185
|
+
"passed": true,
|
|
186
|
+
"responseEvidence": [
|
|
187
|
+
"I checked the catalog first — here's where things stand.\n\n## No playbook fits this request\n\nYour published catalog has exactly two workflows, and neither covers medical care coordination:\n\n| Workflow | What it does | Fit |\n|---|---|---|\n| **Code delivery** | Plan, implement, test, review a change in an existing software repo | No |\n| **Library build** | Research a topic across subtopics, gather sources/media, synthesize a linked knowledge library | Partial at best |\n\nYour request has four distinct steps, and no crew serves any of them:\n\n1. **Compare providers** — needs provider directory/quality data\n2. **Verify insurance coverage** — needs your plan details and a payer/eligibility check\n3. **Schedule an appointment** — needs authenticated booking with a specific practice\n4. **Arrange transportation** — needs a rideshare/medical-transport booking capability\n\nI'm not going to freelance these inline. Steps 2–4 in particular involve your PHI, your insurance credentials, and real bookings on your behalf — that's not something I should improvise without a defined workflow and your explicit go-ahead on the specifics.\n\n## How would you like to proceed?\n\nA few options:\n\n- **Run a research-only slice.** `library-build` could produce a sourced comparison of providers in your area — a real deliverable for step 1. It cannot touch coverage, scheduling, or transport. Say the word and I'll start it, but I'd need the specialty, your location/radius, and what you're optimizing for (in-network, wait time, ratings, hospital affiliation).\n- **Author a new workflow.** If this is something you'll do repeatedly — for yourself or family — a durable \"care coordination\" playbook is worth publishing, with crews for directory lookup, eligibility verification, booking, and transport.\n- **I stay advisory.** I can walk you through what to ask each provider's office, what to confirm with your insurer, and what to have ready when you book — you make the calls.\n\nOne caution on option 2: even a purpose-built workflow will hit hard walls at eligibility checks and booking, since those generally require authenticated access to your payer portal and the practice's scheduling system. Worth knowing before we invest in building it.\n\nWhich direction do you want?"
|
|
188
|
+
]
|
|
189
|
+
},
|
|
190
|
+
{
|
|
191
|
+
"id": "ambiguous-migration",
|
|
192
|
+
"kind": "ambiguous",
|
|
193
|
+
"request": "Investigate a legacy service and produce a migration brief with evidence and recommended next steps.",
|
|
194
|
+
"calls": [
|
|
195
|
+
{
|
|
196
|
+
"sequence": 1,
|
|
197
|
+
"name": "list_workflows",
|
|
198
|
+
"arguments": {}
|
|
199
|
+
},
|
|
200
|
+
{
|
|
201
|
+
"sequence": 2,
|
|
202
|
+
"name": "get_workflow",
|
|
203
|
+
"arguments": {
|
|
204
|
+
"name": "library-build"
|
|
205
|
+
}
|
|
206
|
+
},
|
|
207
|
+
{
|
|
208
|
+
"sequence": 3,
|
|
209
|
+
"name": "get_workflow",
|
|
210
|
+
"arguments": {
|
|
211
|
+
"name": "code-delivery"
|
|
212
|
+
}
|
|
213
|
+
}
|
|
214
|
+
],
|
|
215
|
+
"classification": "observed",
|
|
216
|
+
"responseEvidence": [
|
|
217
|
+
"I checked the catalog before starting anything. Here's where things stand.\n\n## No published playbook fits this request\n\nThe catalog has exactly two workflows, and neither covers a legacy-service investigation ending in a migration brief:\n\n| Workflow | What it's for | Why it doesn't fit |\n|---|---|---|\n| **code-delivery** | Plan, implement, test, review an **approved** change in an existing repo | Requires an already-approved `proposal`. You're asking for the investigation that would *produce* a proposal — the input doesn't exist yet. Its own definition also lists research/curation as explicitly not-for. |\n| **library-build** | Research a topic across subtopics, acquire source + media artifacts, synthesize a **linked knowledge library** | Closest match on the \"investigate and gather evidence\" half, but the output shape is wrong. It produces a curated topical library from acquired source/media artifacts — not a decision brief on one internal system with recommended next steps. Its definition also excludes repo-change work, which is where your \"next steps\" would land. |\n\nI don't want to force-fit either one. Running `code-delivery` without an approved proposal would mean inventing the very conclusion you asked me to investigate, and `library-build` would hand you a media library where you asked for a brief.\n\n## I also don't have enough to scope a run\n\nEven for the closest option, the request is missing:\n\n1. **Which legacy service** — name, repo, or where it lives.\n2. **Migration to what** — target platform/language/architecture, or is that part of what the investigation should decide?\n3. **What counts as evidence** — source code and config, runtime metrics and logs, incident history, dependency/consumer inventory?\n4. **Who the brief is for** — an engineering audience deciding how, or a funding decision on whether.\n\n## How would you like to proceed?\n\nThree options, your call:\n\n- **A — Split it.** Scope `library-build` to the research phase only (e.g. \"migration paths and prior art for <service>, with sourced evidence\"), accept a knowledge library as the artifact, and I write the brief on top of it. Honest about the mismatch, but the research gets crewed properly.\n- **B — Publish a fitting workflow.** If legacy-service assessment is something you'll do more than once, a proper investigation-and-brief playbook belongs in the catalog. I can't author durable workflows for you, but I can spec what it should contain.\n- **C — Point me at an existing brief.** If the investigation is already done elsewhere and you have an approved change, `code-delivery` becomes the right call and I'll start that run.\n\nTell me which service and which option, and I'll move."
|
|
218
|
+
]
|
|
219
|
+
},
|
|
220
|
+
{
|
|
221
|
+
"id": "ambiguous-launch",
|
|
222
|
+
"kind": "ambiguous",
|
|
223
|
+
"request": "Collect competitor material and turn it into a launch-readiness brief for a software product.",
|
|
224
|
+
"calls": [
|
|
225
|
+
{
|
|
226
|
+
"sequence": 1,
|
|
227
|
+
"name": "list_workflows",
|
|
228
|
+
"arguments": {}
|
|
229
|
+
},
|
|
230
|
+
{
|
|
231
|
+
"sequence": 2,
|
|
232
|
+
"name": "get_workflow",
|
|
233
|
+
"arguments": {
|
|
234
|
+
"name": "library-build"
|
|
235
|
+
}
|
|
236
|
+
},
|
|
237
|
+
{
|
|
238
|
+
"sequence": 3,
|
|
239
|
+
"name": "get_workflow",
|
|
240
|
+
"arguments": {
|
|
241
|
+
"name": "code-delivery"
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
],
|
|
245
|
+
"classification": "observed",
|
|
246
|
+
"responseEvidence": [
|
|
247
|
+
"I read the catalog before touching anything. Here's where your request lands.\n\n## Catalog fit\n\nTwo playbooks are published:\n\n| Workflow | What it does | Fit |\n|---|---|---|\n| **library-build** | Research a topic across subtopics, acquire source/media artifacts, synthesize a linked knowledge library | **Good fit for half your request** — the \"collect competitor material\" part |\n| **code-delivery** | Plan, implement, test, review a repo change | Not applicable |\n\n**The gap:** `library-build` explicitly produces a *sourced knowledge library* — organized, linked, cited source material. It does not produce a **launch-readiness brief**, which is a different artifact: a judgment call with gaps, risks, positioning conclusions, and a go/no-go recommendation. No published playbook covers that synthesis step, and `library-build` lists no declared outputs I could chain into one.\n\nI'm not going to quietly stretch the research workflow and hand you a library relabeled as a brief.\n\n## What I need from you\n\n**1. The topic input is required and I can't invent it.** To start `library-build` I need:\n- **Which software product** is launching (yours — name and category)\n- **Which competitors** to cover, or should the crew identify the field itself\n- **Scope of material**: pricing pages, docs, changelogs, positioning, reviews, analyst coverage? Any depth or recency bound?\n\n**2. How you want the brief handled.** Three options:\n\n- **(a) Split it.** Run `library-build` now for the competitor corpus, then you and I work the launch-readiness brief directly off that library once it lands. Uses the crew for the heavy collection, keeps the judgment work honest and visible.\n- **(b) Load the brief framing into the topic.** Scope the library around launch-readiness dimensions (feature parity gaps, pricing bands, positioning whitespace, GTM motions) so the synthesis is closer to what you need — but the deliverable is still a library, not a brief.\n- **(c) Hold.** If a launch-readiness playbook should exist as a durable workflow, that's a request for whoever owns the catalog, not something I should improvise around.\n\nMy recommendation is **(a) with (b)'s framing** — start the collection now with launch-readiness subtopics baked into the scope, since that's real work the crew can do while we settle the brief.\n\nGive me the product, the competitor list, and your pick, and I'll start the run."
|
|
248
|
+
]
|
|
249
|
+
}
|
|
250
|
+
]
|
|
251
|
+
},
|
|
252
|
+
{
|
|
253
|
+
"harness": {
|
|
254
|
+
"id": "codex",
|
|
255
|
+
"reportedModel": "gpt-5.6-sol",
|
|
256
|
+
"version": "0.147.0"
|
|
257
|
+
},
|
|
258
|
+
"charterSha256": "35773e2e269844fbb81acec0c07774f5674d51aa47304f6e0346458a13f9a2a2",
|
|
259
|
+
"passed": 4,
|
|
260
|
+
"denominator": 4,
|
|
261
|
+
"percentage": 100,
|
|
262
|
+
"tasks": [
|
|
263
|
+
{
|
|
264
|
+
"id": "match-delivery",
|
|
265
|
+
"kind": "match",
|
|
266
|
+
"request": "Implement an approved pagination change in an existing TypeScript repository, run its checks, and prepare the change for review.",
|
|
267
|
+
"expectedWorkflow": "code-delivery",
|
|
268
|
+
"calls": [
|
|
269
|
+
{
|
|
270
|
+
"sequence": 1,
|
|
271
|
+
"name": "list_workflows",
|
|
272
|
+
"arguments": {}
|
|
273
|
+
},
|
|
274
|
+
{
|
|
275
|
+
"sequence": 2,
|
|
276
|
+
"name": "get_workflow",
|
|
277
|
+
"arguments": {
|
|
278
|
+
"name": "code-delivery"
|
|
279
|
+
}
|
|
280
|
+
},
|
|
281
|
+
{
|
|
282
|
+
"sequence": 3,
|
|
283
|
+
"name": "start_run",
|
|
284
|
+
"arguments": {
|
|
285
|
+
"workflow_name": "code-delivery",
|
|
286
|
+
"priority": "normal",
|
|
287
|
+
"scope": "/var/folders/91/41_v0tcn4013w1x8_4dxwlw80000gn/T/owenloop-mcp-charter-eval-session-mJTFj0/workspace",
|
|
288
|
+
"provide": {
|
|
289
|
+
"proposal": {
|
|
290
|
+
"text": "Implement the approved pagination change in the existing TypeScript repository, run all relevant repository checks, and prepare the resulting change for code review."
|
|
291
|
+
}
|
|
292
|
+
}
|
|
293
|
+
}
|
|
294
|
+
},
|
|
295
|
+
{
|
|
296
|
+
"sequence": 4,
|
|
297
|
+
"name": "whats_next",
|
|
298
|
+
"arguments": {
|
|
299
|
+
"workflow": "code-delivery"
|
|
300
|
+
}
|
|
301
|
+
},
|
|
302
|
+
{
|
|
303
|
+
"sequence": 5,
|
|
304
|
+
"name": "wake",
|
|
305
|
+
"arguments": {}
|
|
306
|
+
}
|
|
307
|
+
],
|
|
308
|
+
"classification": "passed",
|
|
309
|
+
"passed": true,
|
|
310
|
+
"responseEvidence": [
|
|
311
|
+
"The approved change was submitted to the repository’s `code-delivery` workflow, covering planning, implementation, checks, and review preparation.\n\nThis session is read-only and the workflow backend only recorded the run without executing workers, so no files were changed and no checks could be run. A writable workspace and active workflow workers are required to complete the implementation."
|
|
312
|
+
]
|
|
313
|
+
},
|
|
314
|
+
{
|
|
315
|
+
"id": "match-library",
|
|
316
|
+
"kind": "match",
|
|
317
|
+
"request": "Research Orlando tourism across attractions, neighborhoods, transport, dining, lodging, and events; collect source media and build a linked knowledge library.",
|
|
318
|
+
"expectedWorkflow": "library-build",
|
|
319
|
+
"calls": [
|
|
320
|
+
{
|
|
321
|
+
"sequence": 1,
|
|
322
|
+
"name": "list_workflows",
|
|
323
|
+
"arguments": {}
|
|
324
|
+
},
|
|
325
|
+
{
|
|
326
|
+
"sequence": 2,
|
|
327
|
+
"name": "get_workflow",
|
|
328
|
+
"arguments": {
|
|
329
|
+
"name": "library-build"
|
|
330
|
+
}
|
|
331
|
+
},
|
|
332
|
+
{
|
|
333
|
+
"sequence": 3,
|
|
334
|
+
"name": "start_run",
|
|
335
|
+
"arguments": {
|
|
336
|
+
"workflow_name": "library-build",
|
|
337
|
+
"priority": "normal",
|
|
338
|
+
"scope": "Orlando, Florida tourism knowledge and media library",
|
|
339
|
+
"provide": {
|
|
340
|
+
"topic": {
|
|
341
|
+
"text": "Research Orlando, Florida tourism across attractions, neighborhoods, transportation, dining, lodging, and events. Collect credible source media and source artifacts, preserve provenance and rights/licensing metadata where available, and synthesize a navigable linked knowledge library with cross-links among topics, practical visitor context, and citations. Cover both major theme-park tourism and the broader city/region."
|
|
342
|
+
}
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
},
|
|
346
|
+
{
|
|
347
|
+
"sequence": 4,
|
|
348
|
+
"name": "wake",
|
|
349
|
+
"arguments": {}
|
|
350
|
+
},
|
|
351
|
+
{
|
|
352
|
+
"sequence": 5,
|
|
353
|
+
"name": "whats_next",
|
|
354
|
+
"arguments": {
|
|
355
|
+
"workflow": "library-build"
|
|
356
|
+
}
|
|
357
|
+
}
|
|
358
|
+
],
|
|
359
|
+
"classification": "passed",
|
|
360
|
+
"passed": true,
|
|
361
|
+
"responseEvidence": [
|
|
362
|
+
"Started the `library-build` workflow for an Orlando tourism knowledge and media library covering:\n\n- Attractions\n- Neighborhoods\n- Transportation\n- Dining\n- Lodging\n- Events\n\nThe scope includes credible sources, media provenance and licensing metadata, citations, practical visitor context, cross-linked topics, and coverage of both theme parks and greater Orlando. No user input is currently required."
|
|
363
|
+
]
|
|
364
|
+
},
|
|
365
|
+
{
|
|
366
|
+
"id": "no-match-finance",
|
|
367
|
+
"kind": "no-match",
|
|
368
|
+
"request": "Reconcile six months of bank statements, categorize expenses, and prepare a business tax filing package.",
|
|
369
|
+
"calls": [
|
|
370
|
+
{
|
|
371
|
+
"sequence": 1,
|
|
372
|
+
"name": "list_workflows",
|
|
373
|
+
"arguments": {}
|
|
374
|
+
}
|
|
375
|
+
],
|
|
376
|
+
"classification": "passed",
|
|
377
|
+
"passed": true,
|
|
378
|
+
"responseEvidence": [
|
|
379
|
+
"No published workflow fits financial reconciliation, and no bank statements or bookkeeping files are currently accessible in the workspace.\n\nPlease upload:\n\n- Six months of statements, preferably CSV plus PDF\n- Business entity type and tax jurisdiction\n- Tax year and accounting method (cash/accrual)\n- Prior chart of accounts or last tax return, if available\n- Receipts or notes for unclear/business-personal transactions\n\nI’ll produce a reconciliation report, categorized transaction ledger, expense summary, exception list, and accountant-ready draft tax package. Final classifications and filing should be reviewed by a qualified tax professional."
|
|
380
|
+
]
|
|
381
|
+
},
|
|
382
|
+
{
|
|
383
|
+
"id": "no-match-operations",
|
|
384
|
+
"kind": "no-match",
|
|
385
|
+
"request": "Compare medical providers, verify insurance coverage, schedule an appointment, and arrange transportation.",
|
|
386
|
+
"calls": [
|
|
387
|
+
{
|
|
388
|
+
"sequence": 1,
|
|
389
|
+
"name": "list_workflows",
|
|
390
|
+
"arguments": {}
|
|
391
|
+
}
|
|
392
|
+
],
|
|
393
|
+
"classification": "passed",
|
|
394
|
+
"passed": true,
|
|
395
|
+
"responseEvidence": [
|
|
396
|
+
"I can handle this, but I need:\n\n1. **Care needed:** specialty or reason for visit, urgency, and any provider preferences.\n2. **Coverage/location:** insurance company and exact plan name, ZIP code, travel radius, and whether a referral is required.\n3. **Logistics:** available dates/times, transportation needs (pickup location, accessibility requirements), and whether you want me to book after presenting the best options.\n\nPlease don’t send your full insurance member ID, SSN, or payment details yet. No available workflow directly fits this healthcare coordination request, so I’ll handle it directly once you provide these details."
|
|
397
|
+
]
|
|
398
|
+
},
|
|
399
|
+
{
|
|
400
|
+
"id": "ambiguous-migration",
|
|
401
|
+
"kind": "ambiguous",
|
|
402
|
+
"request": "Investigate a legacy service and produce a migration brief with evidence and recommended next steps.",
|
|
403
|
+
"calls": [
|
|
404
|
+
{
|
|
405
|
+
"sequence": 1,
|
|
406
|
+
"name": "list_workflows",
|
|
407
|
+
"arguments": {}
|
|
408
|
+
}
|
|
409
|
+
],
|
|
410
|
+
"classification": "observed",
|
|
411
|
+
"responseEvidence": [
|
|
412
|
+
"I can’t complete the investigation from this session because the workspace is read-only and no file-reading or terminal interface is available. The available workflows cover code delivery and knowledge-library creation; neither fits a repository migration assessment.\n\nPlease provide repository/file access, or upload the service source and relevant deployment/configuration files. I’ll then deliver an evidence-backed brief covering:\n\n- Current architecture, runtime, dependencies, and integrations\n- Unsupported or high-risk components\n- Data and API migration constraints\n- Security, operational, and testing gaps\n- Recommended target architecture\n- Phased migration plan, rollback strategy, effort, and open decisions"
|
|
413
|
+
]
|
|
414
|
+
},
|
|
415
|
+
{
|
|
416
|
+
"id": "ambiguous-launch",
|
|
417
|
+
"kind": "ambiguous",
|
|
418
|
+
"request": "Collect competitor material and turn it into a launch-readiness brief for a software product.",
|
|
419
|
+
"calls": [],
|
|
420
|
+
"classification": "observed",
|
|
421
|
+
"responseEvidence": [
|
|
422
|
+
"I need the product context before I can produce a credible launch-readiness brief. Please provide:\n\n- Product name, URL, or existing brief\n- Target customer and primary use case\n- 3–5 known competitors, or permission to identify them\n- Launch market/region and expected launch date\n- Any priority areas: pricing, positioning, features, onboarding, security, integrations, or go-to-market\n\nIf easiest, just send the product URL; I can derive most of the rest."
|
|
423
|
+
]
|
|
424
|
+
}
|
|
425
|
+
]
|
|
426
|
+
}
|
|
427
|
+
]
|
|
428
|
+
}
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# MCP chief-of-staff charter evaluation
|
|
2
|
+
|
|
3
|
+
eval:mcp-charter measures whether a fresh Claude Code or Codex session acts on
|
|
4
|
+
the chief-of-staff guidance delivered by MCP initialization. It is a live,
|
|
5
|
+
opt-in evaluation: it spends real model quota and requires both harnesses to be
|
|
6
|
+
logged in. It is deliberately outside the normal test and CI commands.
|
|
7
|
+
|
|
8
|
+
Run the evaluation from a checked-out repository:
|
|
9
|
+
|
|
10
|
+
npm run eval:mcp-charter -- --output docs/evals/mcp-charter-baseline.json
|
|
11
|
+
|
|
12
|
+
To pin a model for one harness, pass --claude-model <model> and/or
|
|
13
|
+
--codex-model <model>. The same choices can be supplied with
|
|
14
|
+
OWENLOOP_MCP_CHARTER_CLAUDE_MODEL and OWENLOOP_MCP_CHARTER_CODEX_MODEL. Each
|
|
15
|
+
task has a five-minute deadline by default; pass --task-timeout-ms <positive
|
|
16
|
+
integer> (or set OWENLOOP_MCP_CHARTER_TASK_TIMEOUT_MS) to set a different
|
|
17
|
+
bounded budget.
|
|
18
|
+
|
|
19
|
+
## Method
|
|
20
|
+
|
|
21
|
+
The fixed versioned fixture contains two catalog definitions and six neutral
|
|
22
|
+
user requests. Every task runs in a newly created harness session. The runner
|
|
23
|
+
only supplies Handle this request for the user followed by the task; it never
|
|
24
|
+
names an expected workflow or asks the model to use a tool.
|
|
25
|
+
|
|
26
|
+
The mounted MCP process writes a JSONL trace in a random harness-owned
|
|
27
|
+
directory outside the model session workspace. The evaluated session receives
|
|
28
|
+
only its workspace path; it cannot reach the trace through its normal workspace
|
|
29
|
+
authority. Claude runs with isolated settings, no skills, no ambient MCP
|
|
30
|
+
servers, and an explicit empty built-in tool set. A Claude SDK init status of
|
|
31
|
+
`failed`, `needs-auth`, `disabled`, or any other non-healthy value for the
|
|
32
|
+
fixture mount makes the task unscorable; the SDK's normal `pending` and
|
|
33
|
+
`connected` states are accepted. Codex receives a fresh private
|
|
34
|
+
`CODEX_HOME` containing no user configuration, plugins, skills, or MCP servers.
|
|
35
|
+
If the operator uses file authentication, the runner stages `auth.json` there
|
|
36
|
+
with owner-only permissions so the app-server remains logged in; it disables
|
|
37
|
+
Codex's shell, exec, file/image, web, app, and subagent surfaces and subprocess
|
|
38
|
+
environment inheritance, so the evaluated model can reach the fixture MCP but
|
|
39
|
+
cannot read that staged credential. The private root is removed after the task.
|
|
40
|
+
Both sessions therefore expose the fixture mount without ambient or built-in
|
|
41
|
+
tools, rather than an operator's configured Owenloop server. The trace
|
|
42
|
+
records the full SHA-256 of the exact UTF-8 instructions value returned by the
|
|
43
|
+
real initialize handler, then each received named `tools/call` as ordered
|
|
44
|
+
sequence, name, and exact arguments at the inbound wire boundary before schema
|
|
45
|
+
validation. Schema-rejected calls therefore remain visible and are never
|
|
46
|
+
mistaken for no call. Scores use only that structured wire log. Response evidence
|
|
47
|
+
retains only the final assembled assistant reply (the Claude SDK's final
|
|
48
|
+
top-level text or Codex's completed final-answer item); reasoning, stderr, tool
|
|
49
|
+
output, MCP metadata, and streamed/intermediate text are excluded. Neither
|
|
50
|
+
response text nor generic telemetry is used for a numeric score.
|
|
51
|
+
|
|
52
|
+
For the two clear match tasks, a pass requires list_workflows with an empty
|
|
53
|
+
object as the first call, then a later start_run for the expected workflow, with
|
|
54
|
+
no conflicting workflow selection. For the two clear no-match tasks, a pass
|
|
55
|
+
requires list_workflows with an empty object first and no start_run. The
|
|
56
|
+
remaining two ambiguous tasks are recorded as observations only. Each harness
|
|
57
|
+
therefore reports passed / 4, not passed / 6.
|
|
58
|
+
|
|
59
|
+
The report records the run timestamp, Node version, fixture catalog digest,
|
|
60
|
+
charter hash, configured/reported model and harness version where available,
|
|
61
|
+
ordered calls, response evidence, classifications, and score. A charter edit
|
|
62
|
+
changes the served-byte hash, so it creates a new attributable score rather
|
|
63
|
+
than inheriting an old baseline.
|
|
64
|
+
|
|
65
|
+
The committed [baseline](evals/mcp-charter-baseline.json) records a complete
|
|
66
|
+
2026-08-20 run for charter hash
|
|
67
|
+
`35773e2e269844fbb81acec0c07774f5674d51aa47304f6e0346458a13f9a2a2`.
|
|
68
|
+
Claude Code 2.1.236 using `claude-opus-5` passed 4/4 clear tasks, and Codex
|
|
69
|
+
0.147.0 using `gpt-5.6-sol` also passed 4/4.
|
|
70
|
+
|
|
71
|
+
One run is one sample. Codex's score moves between runs at a fixed charter hash,
|
|
72
|
+
so read any single number as a point in a range, and re-run before concluding
|
|
73
|
+
that a charter edit helped or hurt. Six runs on 2026-08-20, all on the same
|
|
74
|
+
harness versions and models:
|
|
75
|
+
|
|
76
|
+
| charter hash | runs | Claude | Codex |
|
|
77
|
+
| --- | --- | --- | --- |
|
|
78
|
+
| `dfb67821` (previous charter) | 2 | 4/4, 4/4 | 1/4, 2/4 |
|
|
79
|
+
| `bdbd33e3` (discarded wording) | 1 | 4/4 | 2/4 |
|
|
80
|
+
| `35773e2e` (this charter) | 3 | 4/4, 4/4, 4/4 | 3/4, 4/4, 4/4 |
|
|
81
|
+
|
|
82
|
+
Only the committed run's JSON lives in this repository. The other five were kept
|
|
83
|
+
outside it and are quoted here for the range, not as artifacts you can re-read.
|
|
84
|
+
Codex's range under this charter does not overlap its range under the previous
|
|
85
|
+
one, which is the evidence that moving the catalog rule to the front of the
|
|
86
|
+
charter changed behaviour. Three runs in an arm is a direction, not a
|
|
87
|
+
significance claim.
|
|
88
|
+
|
|
89
|
+
The retained no-match and ambiguous responses were reviewed. Claude calls the
|
|
90
|
+
catalog first on all four and says plainly that no published playbook fits.
|
|
91
|
+
Codex now calls the catalog first on both no-match tasks, which is what the
|
|
92
|
+
score measures, but its prose is weaker: it names the mismatch on the finance
|
|
93
|
+
task and simply asks for missing inputs on the operations one. On the unscored
|
|
94
|
+
`ambiguous-launch` task Codex still makes no call at all, so the charter's
|
|
95
|
+
every-request rule is not fully carried even in the runs that score 4/4. The
|
|
96
|
+
prose review does not alter either harness's numeric score.
|
|
97
|
+
|
|
98
|
+
## Safety and baseline updates
|
|
99
|
+
|
|
100
|
+
The mounted entry point is test/fixtures/mcp-charter-eval-server.ts, not
|
|
101
|
+
bin/owenloop.mjs. It imports only the local MCP transport core and local fixture
|
|
102
|
+
helpers; every handler is a fixed local response, and start_run only records its
|
|
103
|
+
selected workflow. It has no CLI, hub, REST, credential, or settings fallback
|
|
104
|
+
path. Claude ignores ambient settings and MCP configuration, and Codex receives
|
|
105
|
+
a private minimal configuration root with only its supported login cache; both
|
|
106
|
+
mounts are the fixture entry point. Codex's model-visible tools and subprocess
|
|
107
|
+
environment are disabled apart from that mount, so the login cache is not model
|
|
108
|
+
evidence.
|
|
109
|
+
The fixture itself has no production-hub code path, so ambient `OWENLOOP_HUB`
|
|
110
|
+
cannot reach production through this evaluation.
|
|
111
|
+
|
|
112
|
+
An absent or malformed initialization marker, charter-hash mismatch, mount
|
|
113
|
+
startup failure, adapter exit or failure, deadline, or incomplete turn makes
|
|
114
|
+
that task unscorable. On deadline the runner stops the session from the
|
|
115
|
+
synchronous started reference and preserves any trace already written. The
|
|
116
|
+
command exits nonzero and, when --output is supplied, first validates the
|
|
117
|
+
complete report before atomically replacing the requested baseline. It never
|
|
118
|
+
replaces a baseline with fabricated zeroes or a partial result.
|
|
119
|
+
|
|
120
|
+
When the charter changes, run the command above with working credentials,
|
|
121
|
+
inspect the no-match response evidence to confirm the refusal is plain, and
|
|
122
|
+
commit the generated JSON together with any charter change only after both
|
|
123
|
+
harnesses are scoreable. Do not hand-author scores or substitute transcript
|
|
124
|
+
matching when a harness cannot be evaluated.
|