sortie-dogs 0.10.3 → 0.10.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -19
- package/dist/plugin/profiled.js +16 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -74,7 +74,7 @@ bounded implementation, canonical validation, and evidence-backed completion.
|
|
|
74
74
|
|
|
75
75
|
Guides: [日本語](docs/guide-ja.md) · [简体中文](docs/guide-zh-CN.md) · [テスト実行](docs/testing.md) · [CLI testing](docs/cli-testing.md)
|
|
76
76
|
|
|
77
|
-
Release: [v0.10.
|
|
77
|
+
Release: [v0.10.4](https://github.com/zufall-upon/Sortie-dogs/releases/tag/v0.10.4)
|
|
78
78
|
|
|
79
79
|
## Latest local benchmark case study
|
|
80
80
|
|
|
@@ -92,20 +92,20 @@ Run configuration was fixed per product configuration:
|
|
|
92
92
|
children on `openai/gpt-5.6-sol` / `medium`, with the pinned Sortie package and runtime assets.
|
|
93
93
|
No Luna, Astra, or Opus messages were observed in these trials.
|
|
94
94
|
|
|
95
|
-
| Metric · one frozen task | Bare OpenCode | Sortie v0.9.12 | Sortie v0.10.1 RC qualification |
|
|
96
|
-
| --- | ---: | ---: | ---: |
|
|
97
|
-
| Attempts needed | 3 | 5 | 1 |
|
|
98
|
-
| Completed runs compared | 3 | 3 | 1 |
|
|
99
|
-
| Verified PASS | 0/3 | 0/3 | 0/1 |
|
|
100
|
-
| Task checks · F2P | 11.1% · 3/27 | 85.2% · 23/27 | 88.9% · 8/9 |
|
|
101
|
-
| Retained checks · P2P | 282/282 | 282/282 | 94/94 |
|
|
102
|
-
| Median agent wall | 24.5 min | 25.7 min | 23.2 min · n=1 |
|
|
103
|
-
| Median model steps | 43 | 39 | 41 · CLI stream only |
|
|
104
|
-
| Implementation child sessions · total | 0 | 13 | 3 |
|
|
105
|
-
| Estimated API-equivalent cost · median completed run | $3.53 | $2.85 | **$4.48** · n=1 |
|
|
106
|
-
| Estimated cost · completed runs | $10.69 | $9.74 | **$4.48** |
|
|
107
|
-
| Additional interrupted-attempt cost · same fixed snapshot | $0 | $6.20 | $0 |
|
|
108
|
-
| Total cost to acquire completed runs · same fixed snapshot | $10.69 | $15.94 | **$4.48** |
|
|
95
|
+
| Metric · one frozen task | Bare OpenCode | Sortie v0.9.12 | Sortie v0.10.1 RC qualification | Sortie v0.10.3 qualification · Terra/xhigh | Sortie v0.10.4 qualification · Terra/xhigh |
|
|
96
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
97
|
+
| Attempts needed | 3 | 5 | 1 | 1 | 2 |
|
|
98
|
+
| Completed runs compared | 3 | 3 | 1 | 1 | 2 |
|
|
99
|
+
| Verified PASS | 0/3 | 0/3 | 0/1 | 0/1 | 0/2 |
|
|
100
|
+
| Task checks · F2P | 11.1% · 3/27 | 85.2% · 23/27 | 88.9% · 8/9 | 77.8% · 7/9 | 72.2% · 5/9 and 8/9 |
|
|
101
|
+
| Retained checks · P2P | 282/282 | 282/282 | 94/94 | 94/94 | 94/94 |
|
|
102
|
+
| Median agent wall | 24.5 min | 25.7 min | 23.2 min · n=1 | 22.7 min · n=1 | 31.1 min · n=2 |
|
|
103
|
+
| Median model steps | 43 | 39 | 41 · CLI stream only | 36 · CLI stream only | 44 · CLI stream only |
|
|
104
|
+
| Implementation child sessions · total | 0 | 13 | 3 | 5 | 10 |
|
|
105
|
+
| Estimated API-equivalent cost · median completed run | $3.53 | $2.85 | **$4.48** · n=1 | **$3.79** · n=1 | **$3.82** · n=2 |
|
|
106
|
+
| Estimated cost · completed runs | $10.69 | $9.74 | **$4.48** | **$3.79** | **$7.64** |
|
|
107
|
+
| Additional interrupted-attempt cost · same fixed snapshot | $0 | $6.20 | $0 | $0 | $0 |
|
|
108
|
+
| Total cost to acquire completed runs · same fixed snapshot | $10.69 | $15.94 | **$4.48** | **$3.79** | **$7.64** |
|
|
109
109
|
|
|
110
110
|
All three Bare runs passed 1/9 task checks. The three completed Sortie runs passed 7/9, 8/9,
|
|
111
111
|
and 8/9. Every compared candidate retained 94/94 prior checks, but every official verifier still
|
|
@@ -128,12 +128,38 @@ into this RC's same-snapshot acquisition cost. Observed spend across both distin
|
|
|
128
128
|
**$8.755191**, which is development spend rather than this RC's acquisition cost. Historical Bare and
|
|
129
129
|
v0.9.12 costs retain their 2026-07-30 schedule; their cost cells are not same-rate comparisons with
|
|
130
130
|
this RC estimate.
|
|
131
|
-
|
|
132
|
-
|
|
131
|
+
|
|
132
|
+
The released **v0.10.3** Terra/xhigh treatment is a separate one-shot qualification, not a matched
|
|
133
|
+
comparison or leaderboard result. It reached `DONE` and the localized Docker-free official verifier,
|
|
134
|
+
which returned reward **0**, F2P **7/9**, and P2P **94/94**. Its full root-plus-eight-descendant audit
|
|
135
|
+
prices 93 assistant requests and 4,809,706 tokens at **$3.789190** API-equivalent cost: **$2.186540**
|
|
136
|
+
for Terra/xhigh and **$1.602650** for Sol/low. Pricing coverage is 100%, with no unpriced or pending
|
|
137
|
+
requests. This uses the same 2026-09-14 per-request schedule; the observed host cost of $0 is not used
|
|
138
|
+
as cost evidence, and the estimate is not an invoice or subscription charge.
|
|
139
|
+
|
|
140
|
+
The released **v0.10.4** Terra/xhigh treatment repeats that qualification twice on the same frozen
|
|
141
|
+
task, pins, and verifier. Both runs reached `DONE` with confirmed cleanup, and the localized
|
|
142
|
+
Docker-free official verifier returned reward **0** with P2P **94/94** each time. F2P was **5/9** and
|
|
143
|
+
**8/9**, so the single v0.10.3 observation of 7/9 sits inside this spread: neither release solves the
|
|
144
|
+
task, and the per-run difference is run-to-run variance at n=1 and n=2, not a measured quality change.
|
|
145
|
+
|
|
146
|
+
The release payload this qualification exists to check is a prompt-cache correctness fix, and that
|
|
147
|
+
effect is separable from task quality. Before the fix the proposal investigation child re-sent its
|
|
148
|
+
whole prompt uncached on every request, because a consumed-read counter sat in the system block and a
|
|
149
|
+
system element is an absolute prompt prefix. The audit now reports per-session prefix reuse, which
|
|
150
|
+
isolates exactly that failure: the v0.10.3 proposal child reused a median **0.065** of the previous
|
|
151
|
+
prompt and was the only flagged session in its run, while both v0.10.4 runs reuse **0.957** and
|
|
152
|
+
**0.986** and flag none. Its uncached input fell from **406,202** tokens to **213,115** and
|
|
153
|
+
**285,060**, and its estimated cost fell from **$0.943508** to **$0.574788** and **$0.802139**. Whole-run
|
|
154
|
+
cost is not a clean measure of the same fix, because each run performed a different amount of work:
|
|
155
|
+
the two v0.10.4 runs priced 97 requests at **$3.268415** and 119 requests at **$4.368122**, both at
|
|
156
|
+
100% pricing coverage on the 2026-09-14 per-request schedule.
|
|
157
|
+
|
|
158
|
+

|
|
133
159
|
|
|
134
160
|
Cost audits use deduplicated root and descendant session tokens, grouped by the model that produced
|
|
135
|
-
each message. Historical values retain their frozen 2026-07-30 short-context schedule; the
|
|
136
|
-
the product's 2026-09-14 per-request schedule described above. Completed-run cost shows execution
|
|
161
|
+
each message. Historical values retain their frozen 2026-07-30 short-context schedule; the v0.10
|
|
162
|
+
qualifications use the product's 2026-09-14 per-request schedule described above. Completed-run cost shows execution
|
|
137
163
|
efficiency; same-snapshot acquisition cost adds interrupted attempts. These are API-equivalent
|
|
138
164
|
estimates, not invoices.
|
|
139
165
|
|
package/dist/plugin/profiled.js
CHANGED
|
@@ -1109,6 +1109,15 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
1109
1109
|
throw new Error("operator-contract-repair-validation-command-missing");
|
|
1110
1110
|
await operators.recordRepairValidationResult((await rootFor(id)), id, repairAccess.expected_command, exit);
|
|
1111
1111
|
}
|
|
1112
|
+
if (!ownership && request.tool.toLowerCase() === "read") {
|
|
1113
|
+
const proposal = await proposals.read((await rootFor(id)));
|
|
1114
|
+
if (proposal?.phase === "investigating" && proposal.proposal_session_id === id) {
|
|
1115
|
+
output.output = `${output.output ?? ""}\n\nSORTIE_PROPOSAL_BUDGET actual_reads=${proposal.read_count}; ` +
|
|
1116
|
+
`remaining_reads=${proposal.intent.proposal_budget.max_reads - proposal.read_count}; ` +
|
|
1117
|
+
`submissions=${proposal.submission_count}; ` +
|
|
1118
|
+
`remaining_submissions=${proposal.intent.proposal_budget.max_submissions - proposal.submission_count}.`;
|
|
1119
|
+
}
|
|
1120
|
+
}
|
|
1112
1121
|
if (ownership) {
|
|
1113
1122
|
const child = taskChildSessionID(output);
|
|
1114
1123
|
if (child)
|
|
@@ -1165,11 +1174,16 @@ export function createProfiledPlugin(profile, assetVersion) {
|
|
|
1165
1174
|
"Never rewrite user acceptance or evidence to rename protocol roles. Final acceptance belongs only to the root coordinator.");
|
|
1166
1175
|
const proposal = await proposals.read(root);
|
|
1167
1176
|
if (proposal?.phase === "investigating" && proposal.proposal_session_id === request.sessionID) {
|
|
1177
|
+
// Only immutable identity and the frozen budget caps belong here. Consumed counters move with
|
|
1178
|
+
// every accounted read, and a system element is an absolute prompt prefix: restating them here
|
|
1179
|
+
// invalidated the whole cached prefix on every later request of the same investigation, so the
|
|
1180
|
+
// Task prompt and all accumulated reads were re-billed uncached. They are reported on the read
|
|
1181
|
+
// result instead, which is appended after the stable prefix.
|
|
1168
1182
|
(output.system ??= []).push(`SORTIE_PROPOSAL_PHASE investigating; intent=${proposal.intent_id}; root=${root}; child=${request.sessionID}. ` +
|
|
1169
1183
|
`This durable phase remains authoritative after compaction even when the latest message is a generic continuation. ` +
|
|
1170
1184
|
`Continue the admitted read-only investigation and submit through ${submitProposal}. Do not call ${next} or dispatch workers: no execution run exists yet. ` +
|
|
1171
|
-
`
|
|
1172
|
-
`
|
|
1185
|
+
`max_reads=${proposal.intent.proposal_budget.max_reads}; max_submissions=${proposal.intent.proposal_budget.max_submissions}. ` +
|
|
1186
|
+
`Consumed budget is reported as SORTIE_PROPOSAL_BUDGET on each read result and by ${submitProposal}; never infer it from this element.`);
|
|
1173
1187
|
}
|
|
1174
1188
|
},
|
|
1175
1189
|
"experimental.text.complete": async (request, output) => {
|