sortie-dogs 0.10.3 → 0.10.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +59 -19
- package/dist/core/operator-runtime.js +3 -1
- package/dist/plugin/profiled.js +60 -7
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -74,7 +74,7 @@ bounded implementation, canonical validation, and evidence-backed completion.
|
|
|
74
74
|
|
|
75
75
|
Guides: [日本語](docs/guide-ja.md) · [简体中文](docs/guide-zh-CN.md) · [テスト実行](docs/testing.md) · [CLI testing](docs/cli-testing.md)
|
|
76
76
|
|
|
77
|
-
Release: [v0.10.
|
|
77
|
+
Release: [v0.10.5](https://github.com/zufall-upon/Sortie-dogs/releases/tag/v0.10.5)
|
|
78
78
|
|
|
79
79
|
## Latest local benchmark case study
|
|
80
80
|
|
|
@@ -92,20 +92,20 @@ Run configuration was fixed per product configuration:
|
|
|
92
92
|
children on `openai/gpt-5.6-sol` / `medium`, with the pinned Sortie package and runtime assets.
|
|
93
93
|
No Luna, Astra, or Opus messages were observed in these trials.
|
|
94
94
|
|
|
95
|
-
| Metric · one frozen task | Bare OpenCode | Sortie v0.9.12 | Sortie v0.10.1 RC qualification |
|
|
96
|
-
| --- | ---: | ---: | ---: |
|
|
97
|
-
| Attempts needed | 3 | 5 | 1 |
|
|
98
|
-
| Completed runs compared | 3 | 3 | 1 |
|
|
99
|
-
| Verified PASS | 0/3 | 0/3 | 0/1 |
|
|
100
|
-
| Task checks · F2P | 11.1% · 3/27 | 85.2% · 23/27 | 88.9% · 8/9 |
|
|
101
|
-
| Retained checks · P2P | 282/282 | 282/282 | 94/94 |
|
|
102
|
-
| Median agent wall | 24.5 min | 25.7 min | 23.2 min · n=1 |
|
|
103
|
-
| Median model steps | 43 | 39 | 41 · CLI stream only |
|
|
104
|
-
| Implementation child sessions · total | 0 | 13 | 3 |
|
|
105
|
-
| Estimated API-equivalent cost · median completed run | $3.53 | $2.85 | **$4.48** · n=1 |
|
|
106
|
-
| Estimated cost · completed runs | $10.69 | $9.74 | **$4.48** |
|
|
107
|
-
| Additional interrupted-attempt cost · same fixed snapshot | $0 | $6.20 | $0 |
|
|
108
|
-
| Total cost to acquire completed runs · same fixed snapshot | $10.69 | $15.94 | **$4.48** |
|
|
95
|
+
| Metric · one frozen task | Bare OpenCode | Sortie v0.9.12 | Sortie v0.10.1 RC qualification | Sortie v0.10.3 qualification · Terra/xhigh | Sortie v0.10.4 qualification · Terra/xhigh | Sortie v0.10.5 candidate qualification · Terra/xhigh + Luna/max |
|
|
96
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
97
|
+
| Attempts needed | 3 | 5 | 1 | 1 | 2 | 1 recorded success |
|
|
98
|
+
| Completed runs compared | 3 | 3 | 1 | 1 | 2 | 1 recorded success |
|
|
99
|
+
| Verified PASS | 0/3 | 0/3 | 0/1 | 0/1 | 0/2 | **1/1** |
|
|
100
|
+
| Task checks · F2P | 11.1% · 3/27 | 85.2% · 23/27 | 88.9% · 8/9 | 77.8% · 7/9 | 72.2% · 5/9 and 8/9 | **100% · 9/9** |
|
|
101
|
+
| Retained checks · P2P | 282/282 | 282/282 | 94/94 | 94/94 | 94/94 | **94/94** |
|
|
102
|
+
| Median agent wall | 24.5 min | 25.7 min | 23.2 min · n=1 | 22.7 min · n=1 | 31.1 min · n=2 | 43.1 min · n=1 |
|
|
103
|
+
| Median model steps | 43 | 39 | 41 · CLI stream only | 36 · CLI stream only | 44 · CLI stream only | 25 · CLI stream only |
|
|
104
|
+
| Implementation child sessions · total | 0 | 13 | 3 | 5 | 10 | 3 |
|
|
105
|
+
| Estimated API-equivalent cost · median completed run | $3.53 | $2.85 | **$4.48** · n=1 | **$3.79** · n=1 | **$3.82** · n=2 | **$1.72** · n=1 |
|
|
106
|
+
| Estimated cost · completed runs | $10.69 | $9.74 | **$4.48** | **$3.79** | **$7.64** | **$1.72** |
|
|
107
|
+
| Additional interrupted-attempt cost · same fixed snapshot | $0 | $6.20 | $0 | $0 | $0 | Not reported |
|
|
108
|
+
| Total cost to acquire completed runs · same fixed snapshot | $10.69 | $15.94 | **$4.48** | **$3.79** | **$7.64** | Not reported |
|
|
109
109
|
|
|
110
110
|
All three Bare runs passed 1/9 task checks. The three completed Sortie runs passed 7/9, 8/9,
|
|
111
111
|
and 8/9. Every compared candidate retained 94/94 prior checks, but every official verifier still
|
|
@@ -128,12 +128,52 @@ into this RC's same-snapshot acquisition cost. Observed spend across both distin
|
|
|
128
128
|
**$8.755191**, which is development spend rather than this RC's acquisition cost. Historical Bare and
|
|
129
129
|
v0.9.12 costs retain their 2026-07-30 schedule; their cost cells are not same-rate comparisons with
|
|
130
130
|
this RC estimate.
|
|
131
|
-
|
|
132
|
-
|
|
131
|
+
|
|
132
|
+
The released **v0.10.3** Terra/xhigh treatment is a separate one-shot qualification, not a matched
|
|
133
|
+
comparison or leaderboard result. It reached `DONE` and the localized Docker-free official verifier,
|
|
134
|
+
which returned reward **0**, F2P **7/9**, and P2P **94/94**. Its full root-plus-eight-descendant audit
|
|
135
|
+
prices 93 assistant requests and 4,809,706 tokens at **$3.789190** API-equivalent cost: **$2.186540**
|
|
136
|
+
for Terra/xhigh and **$1.602650** for Sol/low. Pricing coverage is 100%, with no unpriced or pending
|
|
137
|
+
requests. This uses the same 2026-09-14 per-request schedule; the observed host cost of $0 is not used
|
|
138
|
+
as cost evidence, and the estimate is not an invoice or subscription charge.
|
|
139
|
+
|
|
140
|
+
The released **v0.10.4** Terra/xhigh treatment repeats that qualification twice on the same frozen
|
|
141
|
+
task, pins, and verifier. Both runs reached `DONE` with confirmed cleanup, and the localized
|
|
142
|
+
Docker-free official verifier returned reward **0** with P2P **94/94** each time. F2P was **5/9** and
|
|
143
|
+
**8/9**, so the single v0.10.3 observation of 7/9 sits inside this spread: neither release solves the
|
|
144
|
+
task, and the per-run difference is run-to-run variance at n=1 and n=2, not a measured quality change.
|
|
145
|
+
|
|
146
|
+
The **v0.10.5 candidate** switches the v0.10 implementation worker default from Sol/low to
|
|
147
|
+
Luna/max. One separate qualification on the same frozen task, pins, and localized Docker-free
|
|
148
|
+
official verifier reached `DONE` and returned reward **1**, F2P **9/9**, and P2P **94/94**. Agent
|
|
149
|
+
wall was **2,583,374 ms** (43.1 min), the CLI stream recorded 25 model steps, and the operation
|
|
150
|
+
created three implementation child sessions. Its full root-plus-five-descendant audit prices 106
|
|
151
|
+
assistant requests and 8,830,131 tokens at **$1.719999** API-equivalent cost: **$1.436725** for
|
|
152
|
+
Terra/xhigh and **$0.283274** for Luna/max. Pricing coverage was 100%, with no unpriced or pending
|
|
153
|
+
requests. The candidate package was built before the release version bump and reported package
|
|
154
|
+
metadata `0.10.4`; this is a source-snapshot reference, not a released-package measurement.
|
|
155
|
+
|
|
156
|
+
The v0.10.5 column deliberately reports this one verified success only. It is not a success-rate or
|
|
157
|
+
acquisition-cost aggregate, and its `Not reported` cells do not imply zero cost or zero failed
|
|
158
|
+
attempts outside this recorded success.
|
|
159
|
+
|
|
160
|
+
The release payload this qualification exists to check is a prompt-cache correctness fix, and that
|
|
161
|
+
effect is separable from task quality. Before the fix the proposal investigation child re-sent its
|
|
162
|
+
whole prompt uncached on every request, because a consumed-read counter sat in the system block and a
|
|
163
|
+
system element is an absolute prompt prefix. The audit now reports per-session prefix reuse, which
|
|
164
|
+
isolates exactly that failure: the v0.10.3 proposal child reused a median **0.065** of the previous
|
|
165
|
+
prompt and was the only flagged session in its run, while both v0.10.4 runs reuse **0.957** and
|
|
166
|
+
**0.986** and flag none. Its uncached input fell from **406,202** tokens to **213,115** and
|
|
167
|
+
**285,060**, and its estimated cost fell from **$0.943508** to **$0.574788** and **$0.802139**. Whole-run
|
|
168
|
+
cost is not a clean measure of the same fix, because each run performed a different amount of work:
|
|
169
|
+
the two v0.10.4 runs priced 97 requests at **$3.268415** and 119 requests at **$4.368122**, both at
|
|
170
|
+
100% pricing coverage on the 2026-09-14 per-request schedule.
|
|
171
|
+
|
|
172
|
+

|
|
133
173
|
|
|
134
174
|
Cost audits use deduplicated root and descendant session tokens, grouped by the model that produced
|
|
135
|
-
each message. Historical values retain their frozen 2026-07-30 short-context schedule; the
|
|
136
|
-
the product's 2026-09-14 per-request schedule described above. Completed-run cost shows execution
|
|
175
|
+
each message. Historical values retain their frozen 2026-07-30 short-context schedule; the v0.10
|
|
176
|
+
qualifications use the product's 2026-09-14 per-request schedule described above. Completed-run cost shows execution
|
|
137
177
|
efficiency; same-snapshot acquisition cost adds interrupted attempts. These are API-equivalent
|
|
138
178
|
estimates, not invoices.
|
|
139
179
|
|
|
@@ -1625,7 +1625,9 @@ export class OperatorRuntime {
|
|
|
1625
1625
|
committed_head: state.gitLifecycle.committedHead,
|
|
1626
1626
|
inherited: state.gitLifecycle.commitProvenance === "inherited-parent",
|
|
1627
1627
|
carried_uncommitted_paths: (state.gitLifecycle.carriedPaths ?? []).slice(0, 32) },
|
|
1628
|
-
|
|
1628
|
+
// Index-aligned with this packet's own `acceptance`. Restating each criterion here duplicated the
|
|
1629
|
+
// whole acceptance text inside a packet the root re-reads on every later turn of its session.
|
|
1630
|
+
requirements: state.acceptance.map((_criterion, index) => ({ index,
|
|
1629
1631
|
proof_ids: state.acceptanceProof[index], status: state.acceptanceProof[index].every(id => proved.has(id)) ? "observed-pass" : "unproven" })),
|
|
1630
1632
|
source_refs: state.sourceRefs, decision: state.decision,
|
|
1631
1633
|
contract_repair: state.contractRepair === null ? null : { code: state.contractRepair.code, unit_id: state.contractRepair.unit_id,
|
package/dist/plugin/profiled.js
CHANGED
|
@@ -16,7 +16,7 @@ const SERIAL_CAPABILITIES = new Set([
|
|
|
16
16
|
"sortie_bind_write_gate", "sortie_release_write_gate", "sortie_check_contract",
|
|
17
17
|
"sortie_compact_and_continue", "sortie_enable_backlog_drain",
|
|
18
18
|
]);
|
|
19
|
-
const PREVIEW_WORKER_ROUTE = Object.freeze({ model: "openai/gpt-5.6-
|
|
19
|
+
const PREVIEW_WORKER_ROUTE = Object.freeze({ model: "openai/gpt-5.6-luna", variant: "max" });
|
|
20
20
|
const PREVIEW_SCOUT_ROUTE = Object.freeze({ model: "openai/gpt-5.6-luna", variant: "xhigh" });
|
|
21
21
|
const PREVIEW_OPERATIONS_ROUTE = Object.freeze({ model: "openai/gpt-5.6-terra", variant: "xhigh" });
|
|
22
22
|
const PREVIEW_PRIMARY_ROUTE = Object.freeze({ model: "openai/gpt-5.6-sol", variant: "low" });
|
|
@@ -137,6 +137,27 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
137
137
|
}
|
|
138
138
|
return parentRoot;
|
|
139
139
|
}
|
|
140
|
+
/**
|
|
141
|
+
* Resolve the owning root of a proposal investigation child for prompt assembly only.
|
|
142
|
+
*
|
|
143
|
+
* `rootFor` stops resolving this child as soon as the proposal leaves `investigating`, which is the
|
|
144
|
+
* correct authorization answer: the child must not run another tool. It is the wrong answer for the
|
|
145
|
+
* system prefix, because losing the root also drops every profile element and changes the absolute
|
|
146
|
+
* prompt prefix, so the child's post-submit turn re-sent its whole investigation uncached. This
|
|
147
|
+
* resolver grants no tool authority and is never consulted on an execute path.
|
|
148
|
+
*/
|
|
149
|
+
async function proposalPromptRoot(id) {
|
|
150
|
+
if (retired.has(id))
|
|
151
|
+
return undefined;
|
|
152
|
+
const who = await identity(id);
|
|
153
|
+
if (who.role !== "dog-operator" || !who.parent)
|
|
154
|
+
return undefined;
|
|
155
|
+
const parentRoot = await rootFor(who.parent, 1);
|
|
156
|
+
if (!parentRoot)
|
|
157
|
+
return undefined;
|
|
158
|
+
const proposal = await proposals.read(parentRoot);
|
|
159
|
+
return proposal?.phase === "submitted" && proposal.proposal_session_id === id ? parentRoot : undefined;
|
|
160
|
+
}
|
|
140
161
|
function mapAgent(value, outward) {
|
|
141
162
|
if (outward) {
|
|
142
163
|
if (value.startsWith("foreign/"))
|
|
@@ -436,6 +457,19 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
436
457
|
}
|
|
437
458
|
return { ...packet, budget };
|
|
438
459
|
}
|
|
460
|
+
/**
|
|
461
|
+
* Proposal accounting without the submitted packet body.
|
|
462
|
+
*
|
|
463
|
+
* The full proposal carries the plan, every unit, and the acceptance text a further time. Once the
|
|
464
|
+
* root has compared it, approval froze it into the execution run, whose own packet already reports
|
|
465
|
+
* acceptance, units, and scopes. Re-emitting it on approval and on every later status call appended
|
|
466
|
+
* a redundant copy to the one session that re-reads its whole context on every turn. `content_hash`
|
|
467
|
+
* stays, so the exact submitted revision is still identifiable.
|
|
468
|
+
*/
|
|
469
|
+
function proposalIdentity(state) {
|
|
470
|
+
const { proposal: _packet, ...identity } = proposals.packet(state);
|
|
471
|
+
return identity;
|
|
472
|
+
}
|
|
439
473
|
tools[status] = { description: "Read the durable root-owned operator outcome and host budget counters (max_units, consumed_units, reserved_units, remaining_units) without claiming acceptance or retrying work. An investigating proposal returns its exact short Task reference only before a Task has been admitted; an existing admission never yields a redispatch Task.",
|
|
440
474
|
args: {}, execute: async (_args, context) => {
|
|
441
475
|
await requireRoot(context.sessionID);
|
|
@@ -452,7 +486,7 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
452
486
|
? "compare and approve the exact proposal; do not call operator_next before approval prepares a run"
|
|
453
487
|
: "approved proposal has no operator run; do not call operator_next; reconcile the approval or preparation failure";
|
|
454
488
|
return JSON.stringify(state ? { ...await operatorPacket(state), ...(draft ? { pending_draft: draft } : {}),
|
|
455
|
-
...(proposal ? { proposal:
|
|
489
|
+
...(proposal ? { proposal: proposalIdentity(proposal) } : {}) }
|
|
456
490
|
: draft ? { ...draft, ...(proposal ? { proposal: proposals.packet(proposal) } : {}) }
|
|
457
491
|
: proposal ? { profile: profile.id, proposal: proposals.packet(proposal),
|
|
458
492
|
...(proposal.phase === "investigating" && proposal.proposal_call_id === null ? { task: proposals.referenceTask(proposal),
|
|
@@ -701,7 +735,7 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
701
735
|
throw new Error("operator-proposal-approved-plan-invalid");
|
|
702
736
|
await registerPreparedGoal(context.sessionID, prepared.state);
|
|
703
737
|
const committed = await proposals.approve(context.sessionID, approval);
|
|
704
|
-
return JSON.stringify({ ...
|
|
738
|
+
return JSON.stringify({ ...proposalIdentity(committed), execution: JSON.parse(preparedTask(prepared.state)) });
|
|
705
739
|
});
|
|
706
740
|
} };
|
|
707
741
|
const ownTools = new Set([prepare, repair, next, status, cancel, complete, resume, resolveContractRepair,
|
|
@@ -1109,6 +1143,15 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
1109
1143
|
throw new Error("operator-contract-repair-validation-command-missing");
|
|
1110
1144
|
await operators.recordRepairValidationResult((await rootFor(id)), id, repairAccess.expected_command, exit);
|
|
1111
1145
|
}
|
|
1146
|
+
if (!ownership && request.tool.toLowerCase() === "read") {
|
|
1147
|
+
const proposal = await proposals.read((await rootFor(id)));
|
|
1148
|
+
if (proposal?.phase === "investigating" && proposal.proposal_session_id === id) {
|
|
1149
|
+
output.output = `${output.output ?? ""}\n\nSORTIE_PROPOSAL_BUDGET actual_reads=${proposal.read_count}; ` +
|
|
1150
|
+
`remaining_reads=${proposal.intent.proposal_budget.max_reads - proposal.read_count}; ` +
|
|
1151
|
+
`submissions=${proposal.submission_count}; ` +
|
|
1152
|
+
`remaining_submissions=${proposal.intent.proposal_budget.max_submissions - proposal.submission_count}.`;
|
|
1153
|
+
}
|
|
1154
|
+
}
|
|
1112
1155
|
if (ownership) {
|
|
1113
1156
|
const child = taskChildSessionID(output);
|
|
1114
1157
|
if (child)
|
|
@@ -1156,7 +1199,7 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
1156
1199
|
await core["permission.ask"]?.(request, output);
|
|
1157
1200
|
},
|
|
1158
1201
|
"experimental.chat.system.transform": async (request, output) => {
|
|
1159
|
-
const root = await rootFor(request.sessionID);
|
|
1202
|
+
const root = await rootFor(request.sessionID) ?? await proposalPromptRoot(request.sessionID);
|
|
1160
1203
|
if (!root)
|
|
1161
1204
|
return;
|
|
1162
1205
|
await core["experimental.chat.system.transform"]?.(request, output);
|
|
@@ -1164,12 +1207,22 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
1164
1207
|
`Shared MkII protocol role names are logical: ${protocolMap}. Use only ${profile.toolPrefix} tools for this profile. ` +
|
|
1165
1208
|
"Never rewrite user acceptance or evidence to rename protocol roles. Final acceptance belongs only to the root coordinator.");
|
|
1166
1209
|
const proposal = await proposals.read(root);
|
|
1167
|
-
if (proposal?.phase
|
|
1210
|
+
if (proposal?.phase !== "approved" && proposal?.proposal_session_id === request.sessionID) {
|
|
1211
|
+
// Only immutable identity and the frozen budget caps belong here. Consumed counters move with
|
|
1212
|
+
// every accounted read, and a system element is an absolute prompt prefix: restating them here
|
|
1213
|
+
// invalidated the whole cached prefix on every later request of the same investigation, so the
|
|
1214
|
+
// Task prompt and all accumulated reads were re-billed uncached. They are reported on the read
|
|
1215
|
+
// result instead, which is appended after the stable prefix.
|
|
1216
|
+
//
|
|
1217
|
+
// The phase is deliberately not part of this condition. Removing an element is the same absolute
|
|
1218
|
+
// prefix change as rewriting one: gating on `investigating` dropped this block the moment a
|
|
1219
|
+
// submission succeeded, so the child's final turn re-sent the entire accumulated investigation
|
|
1220
|
+
// uncached. The terminal submit result is appended after this prefix and is more recent.
|
|
1168
1221
|
(output.system ??= []).push(`SORTIE_PROPOSAL_PHASE investigating; intent=${proposal.intent_id}; root=${root}; child=${request.sessionID}. ` +
|
|
1169
1222
|
`This durable phase remains authoritative after compaction even when the latest message is a generic continuation. ` +
|
|
1170
1223
|
`Continue the admitted read-only investigation and submit through ${submitProposal}. Do not call ${next} or dispatch workers: no execution run exists yet. ` +
|
|
1171
|
-
`
|
|
1172
|
-
`
|
|
1224
|
+
`max_reads=${proposal.intent.proposal_budget.max_reads}; max_submissions=${proposal.intent.proposal_budget.max_submissions}. ` +
|
|
1225
|
+
`Consumed budget is reported as SORTIE_PROPOSAL_BUDGET on each read result and by ${submitProposal}; never infer it from this element.`);
|
|
1173
1226
|
}
|
|
1174
1227
|
},
|
|
1175
1228
|
"experimental.text.complete": async (request, output) => {
|