ytdb-slate 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/context-budget.md +84 -48
- package/docs/design-principles.md +4 -4
- package/docs/model-routing.md +65 -31
- package/docs/thread-cache-cost.md +223 -0
- package/extension/episodes.ts +69 -24
- package/extension/index.ts +36 -21
- package/extension/mode.ts +14 -6
- package/extension/model-profiles.ts +168 -13
- package/extension/model-router.ts +128 -21
- package/extension/paths.ts +1 -0
- package/extension/render.ts +14 -6
- package/extension/route.ts +109 -1
- package/extension/state.ts +294 -8
- package/extension/thread-choice.ts +1207 -0
- package/extension/threads.ts +522 -32
- package/extension/tools.ts +107 -6
- package/extension/worker.ts +54 -0
- package/package.json +1 -1
package/docs/context-budget.md
CHANGED
|
@@ -142,13 +142,16 @@ schema. A tool description and a parameter description are therefore both
|
|
|
142
142
|
always loaded.
|
|
143
143
|
|
|
144
144
|
The current `thread` tool description is 1,386 UTF-8 bytes. Its serialized
|
|
145
|
-
parameter schema is
|
|
146
|
-
includes the 289-byte `type` parameter description
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
145
|
+
parameter schema is 2,005 bytes, measured as `JSON.stringify(parameters)`.
|
|
146
|
+
The schema includes the 289-byte `type` parameter description and the
|
|
147
|
+
`freshContext` argument.
|
|
148
|
+
|
|
149
|
+
The description and schema total 3,391 bytes before provider framing. The
|
|
150
|
+
`freshContext` schema entry adds 214 bytes to the previous combined figure.
|
|
151
|
+
The increase is roughly 54 tokens at four characters per token.
|
|
152
|
+
|
|
153
|
+
The figure excludes the prompt snippet, prompt guidelines, tool name,
|
|
154
|
+
provider framing, and serialization outside the parameter schema.
|
|
152
155
|
|
|
153
156
|
The always-loaded doctrine has a separate measurement after the compaction
|
|
154
157
|
policy.
|
|
@@ -174,14 +177,14 @@ you pay its input tokens every turn.
|
|
|
174
177
|
|
|
175
178
|
**Its size depends on where the package is installed.** That is not
|
|
176
179
|
obvious, and it catches anyone who measures: the doctrine cites the
|
|
177
|
-
shipped docs by ABSOLUTE path, so
|
|
178
|
-
are embedded in the block —
|
|
179
|
-
(`
|
|
180
|
-
plus `pr-publishing.md` when
|
|
181
|
-
`model-routing.md` when the routing
|
|
182
|
-
`writing-guidance.md` when `writing.check` is on.
|
|
183
|
-
character in the installed `docs/` directory therefore
|
|
184
|
-
characters of doctrine.
|
|
180
|
+
shipped docs by ABSOLUTE path, so four to seven full filesystem paths
|
|
181
|
+
are embedded in the block — four at minimum
|
|
182
|
+
(`thread-cache-cost.md`, `track-workflow.md`, `review-rules.md`,
|
|
183
|
+
`design-principles.md`), plus `pr-publishing.md` when
|
|
184
|
+
`workflow.draftPRs` is on, plus `model-routing.md` when the routing
|
|
185
|
+
rule renders, plus `writing-guidance.md` when `writing.check` is on.
|
|
186
|
+
Every additional character in the installed `docs/` directory therefore
|
|
187
|
+
costs 4–7 characters of doctrine.
|
|
185
188
|
|
|
186
189
|
The figures below are **portable characters**, defined exactly as
|
|
187
190
|
Slate's own automated `doctrine-budget` check defines them: the
|
|
@@ -204,18 +207,18 @@ must be named instead of presented as the same measurement:
|
|
|
204
207
|
|
|
205
208
|
| router | models | `draftPRs` | `writing.check` | paths | portable | lines |
|
|
206
209
|
| --- | --- | --- | --- | --- | --- | --- |
|
|
207
|
-
| off | — | off | off |
|
|
208
|
-
| off | — | off | on |
|
|
209
|
-
| off | — | on | off |
|
|
210
|
-
| off | — | on | on |
|
|
211
|
-
| on | `.pi/slate.json` six | off | off |
|
|
212
|
-
| on | `.pi/slate.json` six | off | on |
|
|
213
|
-
| on | `.pi/slate.json` six | on | off |
|
|
214
|
-
| on | `.pi/slate.json` six | on | on |
|
|
215
|
-
| on | all 9 shipped | off | off |
|
|
216
|
-
| on | all 9 shipped | off | on |
|
|
217
|
-
| on | all 9 shipped | on | off |
|
|
218
|
-
| on | all 9 shipped | on | on |
|
|
210
|
+
| off | — | off | off | 4 | 2,270 | 43 |
|
|
211
|
+
| off | — | off | on | 5 | 3,340 | 65 |
|
|
212
|
+
| off | — | on | off | 5 | 2,289 | 43 |
|
|
213
|
+
| off | — | on | on | 6 | 3,359 | 65 |
|
|
214
|
+
| on | `.pi/slate.json` six | off | off | 5 | 4,300 | 64 |
|
|
215
|
+
| on | `.pi/slate.json` six | off | on | 6 | 5,370 | 86 |
|
|
216
|
+
| on | `.pi/slate.json` six | on | off | 6 | 4,319 | 64 |
|
|
217
|
+
| on | `.pi/slate.json` six | on | on | 7 | 5,389 | 86 |
|
|
218
|
+
| on | all 9 shipped | off | off | 5 | 4,855 | 67 |
|
|
219
|
+
| on | all 9 shipped | off | on | 6 | 5,925 | 89 |
|
|
220
|
+
| on | all 9 shipped | on | off | 6 | 4,874 | 67 |
|
|
221
|
+
| on | all 9 shipped | on | on | 7 | 5,944 | 89 |
|
|
219
222
|
|
|
220
223
|
An untrusted project reads the first row whatever its `slate.json`
|
|
221
224
|
says: no project config is loaded, so no optional rule renders. Line
|
|
@@ -231,9 +234,9 @@ path:
|
|
|
231
234
|
|
|
232
235
|
- The routing rule is a live table with ONE ROW PER ROUTABLE MODEL,
|
|
233
236
|
so what renders is the models you CONFIGURE — not the nine Slate
|
|
234
|
-
ships profiles for. In this snapshot it is
|
|
235
|
-
characters and adds
|
|
236
|
-
for all nine it is 2,
|
|
237
|
+
ships profiles for. In this snapshot it is 2,030 portable
|
|
238
|
+
characters and adds 21 doctrine lines for six configured models;
|
|
239
|
+
for all nine it is 2,585 characters and adds 24 lines. The six
|
|
237
240
|
model rows are 146–181 characters; all nine are 146–183. The
|
|
238
241
|
legend adds a one-off clause per marker it has to explain, so
|
|
239
242
|
growth is not only the sum of new rows.
|
|
@@ -247,14 +250,14 @@ worker extensions and support verification decisions:
|
|
|
247
250
|
|
|
248
251
|
| basis | models | worker extensions | paths | portable | lines | rough tokens |
|
|
249
252
|
| --- | ---: | --- | ---: | ---: | ---: | ---: |
|
|
250
|
-
| current `.pi/slate.json` dogfood config | 6 |
|
|
251
|
-
| stable maximal fixture | 9 | synthetic 2 units / 4 tools, every rendered field at its cap |
|
|
253
|
+
| current `.pi/slate.json` dogfood config | 6 | pinned-package 2 units / 4 tools | 7 | 6,305 | 96 | ≈1,576 |
|
|
254
|
+
| stable maximal fixture | 9 | synthetic 2 units / 4 tools, every rendered field at its cap | 7 | 7,291 | 99 | ≈1,823 |
|
|
252
255
|
|
|
253
256
|
The dogfood row uses `workflow.draftPRs: true`, `writing.check: true`, and the
|
|
254
257
|
six models in `.pi/slate.json`. Its extension basis is
|
|
255
258
|
`pi-smart-fetch@0.3.12` plus `pi-web-search@1.3.1`. Those packages resolve to two
|
|
256
259
|
units and four tools. Their worker-extension rule is 916 portable characters /
|
|
257
|
-
11 lines. Raw size remains symbolic: `portable +
|
|
260
|
+
11 lines. Raw size remains symbolic: `portable + 7 × length(installed docs
|
|
258
261
|
directory)`. No maintainer checkout path belongs in this shipped document.
|
|
259
262
|
|
|
260
263
|
The stable maximal row uses all nine shipped profiles, draft PRs, and writing.
|
|
@@ -267,23 +270,54 @@ independent of installed extension labels, descriptions, versions and other
|
|
|
267
270
|
prose. The worker rule measures 1,347 characters against its 1,600-character
|
|
268
271
|
verification budget.
|
|
269
272
|
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
273
|
+
The reserve policy is: every doctrine upper bound must exceed its current
|
|
274
|
+
measurement by at least five percent, with a required character-bound raise
|
|
275
|
+
rounded up to the next hundred characters. Existing bounds stay in place when
|
|
276
|
+
they already satisfy the policy. Line bounds use the same five-percent rule and
|
|
277
|
+
the next whole line.
|
|
278
|
+
|
|
279
|
+
Two guards have different jobs. An exact pinned literal is the sensitive guard.
|
|
280
|
+
It fails for every size change in its rendered fixture, including one character.
|
|
281
|
+
The paired maximal fixtures cover rule 8 with draft pull requests enabled and
|
|
282
|
+
disabled. The bound is a coarse ceiling that stops unbounded growth over time.
|
|
283
|
+
A bound close to a fixture adds friction but no detection, because its exact
|
|
284
|
+
literal already detects the change. Real reserve therefore weakens nothing.
|
|
285
|
+
|
|
286
|
+
A doctrine change updates the exact literal and every published measurement. A
|
|
287
|
+
maintainer revisits a bound only when the reserve policy requires it. The check
|
|
288
|
+
applies the five-percent rule to every upper bound, so this decision is auditable.
|
|
289
|
+
|
|
290
|
+
| budget term | measured | old bound | current bound | current reserve |
|
|
291
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
292
|
+
| routing rule characters | 2,585 | 4,000 | 4,000 | 1,415 |
|
|
293
|
+
| routing rule lines | 25 | 34 | 34 | 9 |
|
|
294
|
+
| routing fixed prose | 1,110 | 1,500 | 1,500 | 390 |
|
|
295
|
+
| largest model row | 183 | 300 | 300 | 117 |
|
|
296
|
+
| router-on doctrine | 4,855 | 6,500 | 6,500 | 1,645 |
|
|
297
|
+
| writing-only doctrine | 3,340 | 5,600 | 5,600 | 2,260 |
|
|
298
|
+
| writing plus router | 5,925 | 6,300 | 6,500 | 575 |
|
|
299
|
+
| writing plus extensions | 3,595 | 6,000 | 6,000 | 2,405 |
|
|
300
|
+
| writing plus router and extensions | 6,180 | 6,600 | 6,800 | 620 |
|
|
301
|
+
| maximal doctrine, draft PRs enabled | 7,291 | 7,800 | 7,900 | 609 |
|
|
302
|
+
| maximal doctrine, draft PRs disabled | 7,272 | 7,800 | 7,900 | 628 |
|
|
303
|
+
| capped worker rule | 1,347 | 1,600 | 1,600 | 253 |
|
|
304
|
+
| writing rule characters | 1,070 | 1,150 | 1,150 | 80 |
|
|
305
|
+
| writing rule lines | 23 | 24 | 25 | 2 |
|
|
306
|
+
|
|
307
|
+
The positive control adds one capped tool and four copies of the largest
|
|
308
|
+
measured model row. It measures 8,239 portable characters and exceeds the new
|
|
309
|
+
7,900-character maximal bound by 339. That margin remains larger than the
|
|
310
|
+
184-character maximum model-row growth and the 212-character capped tool growth.
|
|
311
|
+
The raised bound does not blunt the positive control.
|
|
312
|
+
|
|
313
|
+
These figures are verification budgets, never runtime limits. Arbitrary user
|
|
314
|
+
extension rosters can exceed them. Keep the positive-control steps unchanged
|
|
315
|
+
unless the fixture design itself changes.
|
|
282
316
|
|
|
283
317
|
Against a 256,000-token context budget, these blocks remain small. At four
|
|
284
318
|
characters per token as a rough estimate, the shipped-rule table ranges from
|
|
285
|
-
about
|
|
286
|
-
tokens. The representative maximum is about 1,
|
|
319
|
+
about 568 tokens to about 1,486 tokens. The current dogfood basis is about 1,576
|
|
320
|
+
tokens. The representative maximum is about 1,823 tokens, or 0.72 percent of the
|
|
287
321
|
default budget.
|
|
288
322
|
|
|
289
323
|
No tokenizer was run, and tables are denser than prose. The block is re-sent on every request rather than paid once. These figures show how
|
|
@@ -381,7 +415,9 @@ long-context rates listed above.
|
|
|
381
415
|
|
|
382
416
|
With action-level routing on, this override also settles a warning
|
|
383
417
|
you will otherwise see once per session for each configured
|
|
384
|
-
`gpt-5.6-*` model
|
|
418
|
+
`gpt-5.6-*` model during resolution. Dispatch-time price divergence warnings
|
|
419
|
+
are conditional and can repeat on later dispatches. Slate's own profile for those
|
|
420
|
+
models records a
|
|
385
421
|
1,050,000-token window, pi's stock registry reports 272,000, and the
|
|
386
422
|
router reports that divergence without adjudicating it (routing uses
|
|
387
423
|
the registry figure). Raising the registry window removes the
|
|
@@ -316,10 +316,10 @@ the load-on-demand discipline the extension itself prescribes:
|
|
|
316
316
|
that ships with this package (`track-workflow.md` in this directory),
|
|
317
317
|
appended to the system prompt every turn while orchestrator mode is
|
|
318
318
|
on. Sizing it matters to the tiering argument, so it is measured
|
|
319
|
-
rather than asserted: with
|
|
320
|
-
`workflow.draftPRs` off, the block is **
|
|
321
|
-
|
|
322
|
-
|
|
319
|
+
rather than asserted: with every conditional tail rule off and
|
|
320
|
+
`workflow.draftPRs` off, the block is **2,270 portable characters /
|
|
321
|
+
43 lines**, and configuring the router for six models nearly
|
|
322
|
+
doubles it, to **4,300** on that same basis — the routing rule
|
|
323
323
|
renders one table row per routable model. (Portable characters, and
|
|
324
324
|
the rest of the configurations, are defined and tabulated in
|
|
325
325
|
`context-budget.md`; the raw count also depends on where the package
|
package/docs/model-routing.md
CHANGED
|
@@ -279,6 +279,7 @@ warning, but removal is not the ADD remedy in condition 2. Condition
|
|
|
279
279
|
| pi has no usable credentials configured for it | dropped | "has no usable credentials configured in pi. Slate drops it from routing." | configuration fault |
|
|
280
280
|
| every entry is dropped | router turns OFF | "survived validation. The warnings above name each dropped entry" | configuration fault |
|
|
281
281
|
| no usable input price exists for today's date | kept, ordered last | "Slate cannot compare its cost with the other models." | model data note |
|
|
282
|
+
| a profile price is negative or non-finite | kept, ordered last or unavailable | "has invalid input price data" or "has invalid output price data" | model data note |
|
|
282
283
|
| no usable effort ladder exists in the profile | kept | "Such a level passes through to pi, which clamps it" | model data note |
|
|
283
284
|
| profile context window differs from the registry window | kept, registry value used | "differs between two sources" | model data note |
|
|
284
285
|
| the first profile names unknown routing-critical fields | kept | "picks models from a research table shipped inside slate" | model data note |
|
|
@@ -288,20 +289,30 @@ warning, but removal is not the ADD remedy in condition 2. Condition
|
|
|
288
289
|
| every configured candidate is marked non-preferred | cheapest candidate becomes the base | "profiles mark every configured model as one it must never pick by itself" | configuration fault |
|
|
289
290
|
| resolution throws | router turns OFF | "routing is disabled. The router could not resolve its model list" | configuration fault |
|
|
290
291
|
|
|
291
|
-
|
|
292
|
-
resolution
|
|
293
|
-
|
|
294
|
-
channel
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
292
|
+
Each resolution warning is deduplicated by a condition key and retained in the
|
|
293
|
+
frozen resolution regardless of display filtering. With `router.showWarnings:
|
|
294
|
+
true`, every resolution warning reaches the normal UI notification or console
|
|
295
|
+
channel. With the default `false`, every configuration fault remains visible
|
|
296
|
+
and model data notes are hidden.
|
|
297
|
+
|
|
298
|
+
When resolution hides at least one note, Slate emits one discoverability line.
|
|
299
|
+
It gives the hidden warning count, names `router.showWarnings`, and says that a
|
|
300
|
+
hidden warning can affect model selection. The count is warnings, not physical
|
|
301
|
+
display lines. The line appears at most once per session.
|
|
302
|
+
|
|
303
|
+
Dispatch-time warnings are separate. Context substitution, effort evidence
|
|
304
|
+
gaps, long-context billing, failover window notices, and the model-visible
|
|
305
|
+
registry-price divergence warning are evaluated for each planned action. They
|
|
306
|
+
can repeat on later dispatches. The divergence warning is hardened and capped
|
|
307
|
+
through the router sanitizer. It omits exact rates and is not deduplicated.
|
|
308
|
+
|
|
309
|
+
The exact-rate companion uses the class-aware router sink as a model data note.
|
|
310
|
+
It is hidden by default and shown when `router.showWarnings` is true. Its
|
|
311
|
+
condition key includes the model, date, and differing rates, so identical live
|
|
312
|
+
evidence reports once while changed evidence reports again. It passes through
|
|
313
|
+
the same control stripping, citation cleanup, field caps, and whole-message cap
|
|
314
|
+
as resolution warnings. A hidden dispatch-time note can trigger the one-time
|
|
315
|
+
discoverability line only when resolution did not already trigger it.
|
|
305
316
|
|
|
306
317
|
Half a list is still a routing policy, so partial drops leave the
|
|
307
318
|
router ON. Nothing surviving is not a policy: the router turns OFF
|
|
@@ -485,17 +496,18 @@ model's ladder is unreadable.
|
|
|
485
496
|
`(unmeasured)` where that applies. `live=<model> (failover)` means
|
|
486
497
|
a held fallback currently overrides the nominal base for the live
|
|
487
498
|
session.
|
|
488
|
-
- **At the session level:**
|
|
489
|
-
|
|
499
|
+
- **At the session level:** configuration faults and model data notes enabled by
|
|
500
|
+
`router.showWarnings`, as UI notifications or console output. The default
|
|
501
|
+
instead shows one discoverability line when it hides notes.
|
|
490
502
|
- **In the orchestrator's own system prompt, every turn:** the
|
|
491
503
|
doctrine gains a routing rule — a table with one row per routable
|
|
492
504
|
model, plus the rules for reading it. This is the surface you do
|
|
493
|
-
not see, and it is the router's standing cost:
|
|
494
|
-
|
|
505
|
+
not see, and it is the router's standing cost: 2,030 characters /
|
|
506
|
+
21 added doctrine lines for six configured models, 2,585 / 24 for
|
|
495
507
|
all nine. In the current snapshot a model row costs 146–183
|
|
496
508
|
characters, plus a one-off legend clause for each marker it
|
|
497
509
|
introduces. Configuring the router roughly doubles Slate's
|
|
498
|
-
always-loaded block (
|
|
510
|
+
always-loaded block (2,270 characters with routing and writing
|
|
499
511
|
off). Those are PORTABLE characters — the
|
|
500
512
|
doctrine with each occurrence of the installed `docs/` directory
|
|
501
513
|
removed, filenames kept — because the doctrine embeds absolute doc
|
|
@@ -506,10 +518,12 @@ model's ladder is unreadable.
|
|
|
506
518
|
|
|
507
519
|
## Expected first-session warnings
|
|
508
520
|
|
|
509
|
-
The stock count below
|
|
510
|
-
`.pi/slate.json`, the shipped profile table, and
|
|
511
|
-
|
|
512
|
-
|
|
521
|
+
The stock count below comes from executing `resolveModelRouter` with this
|
|
522
|
+
repository's sanitized `.pi/slate.json`, the shipped profile table, and a real
|
|
523
|
+
`ModelRuntime` from the pinned pi 0.83.0 package. The runtime used
|
|
524
|
+
`modelsPath: null` and `allowModelNetwork: false`, so no local registry override
|
|
525
|
+
or network refresh could affect it. Dummy OpenAI and Anthropic credentials made
|
|
526
|
+
the configured-auth premise explicit. The packaged OpenAI data at
|
|
513
527
|
`@earendil-works/pi-coding-agent/node_modules/@earendil-works/pi-ai/dist/providers/data/openai.json`
|
|
514
528
|
reports a **272,000-token** context window for
|
|
515
529
|
`openai/gpt-5.6-luna`, `openai/gpt-5.6-terra` and
|
|
@@ -528,8 +542,8 @@ data notes**. The default `router.showWarnings: false` therefore
|
|
|
528
542
|
shows **0 of those warnings** and one discoverability line. That line
|
|
529
543
|
begins `slate: there are 11 hidden warnings in the model router.`
|
|
530
544
|
Enabling the option shows all 11 warnings and no discoverability
|
|
531
|
-
line. Each warning fires at most once per session, and the
|
|
532
|
-
is frozen after the first consultation.
|
|
545
|
+
line. Each resolution warning fires at most once per session, and the
|
|
546
|
+
resolution is frozen after the first consultation.
|
|
533
547
|
|
|
534
548
|
| warnings | class | condition keys | why | shown by default |
|
|
535
549
|
| --- | --- | --- | --- | --- |
|
|
@@ -576,9 +590,31 @@ A hidden warning can still alter which model runs an action. Context-
|
|
|
576
590
|
window divergence is the concrete case, and the stock registry emits
|
|
577
591
|
three such hidden notes for this list. Routing uses the registry
|
|
578
592
|
window, and that value drives the context-window substitution guard.
|
|
579
|
-
`router.showWarnings` changes display only
|
|
593
|
+
`router.showWarnings` changes display only. It does not change the
|
|
580
594
|
resolution or the guard input.
|
|
581
595
|
|
|
596
|
+
## Thread choice after routing
|
|
597
|
+
|
|
598
|
+
Routing resolves the model and effort for one action. Thread choice then uses that applied pair when it evaluates a continuation.
|
|
599
|
+
|
|
600
|
+
A model or effort change makes the existing prefix cold. Current cache-write prices are 12.5 times the matching cache-read prices.
|
|
601
|
+
|
|
602
|
+
The `thread` tool owns the economic comparison. The orchestrator still decides whether named episodes can replace the live transcript.
|
|
603
|
+
|
|
604
|
+
On a continuation with `threadChoice.act: true`, `freshContext` is required. With acting off, it is optional. Every supplied valid value reaches the planner on either continuation path. An empty array refuses a restart. A non-empty list of existing episode ids permits one and seeds the new thread.
|
|
605
|
+
|
|
606
|
+
Whenever `freshContext` is supplied, malformed input or an unknown episode is a tool error before creation or mutation. A valid value on a creation call is accepted and unused because creation has no planner.
|
|
607
|
+
|
|
608
|
+
The planner can return `continue`, `fresh`, `abstain`, or `refused`. Only `fresh` can trigger a restart when `threadChoice.act` is enabled.
|
|
609
|
+
|
|
610
|
+
Acting defaults off. The default `threadChoice.report` setting returns the verdict without moving the action.
|
|
611
|
+
|
|
612
|
+
The planner uses current price rows, recorded prefix size, episode size, and dated retention evidence. It abstains when required price or size evidence is missing.
|
|
613
|
+
|
|
614
|
+
A stale retention row grants no retention window. Missing retention or elapsed time does not prove expiry, so an unchanged route remains warm.
|
|
615
|
+
|
|
616
|
+
The complete mechanism and evidence limits are in [Thread cache cost](thread-cache-cost.md).
|
|
617
|
+
|
|
582
618
|
## What the router does NOT enforce
|
|
583
619
|
|
|
584
620
|
Two obligations carried by the shipped profile data and stated in
|
|
@@ -724,11 +760,9 @@ concern; a green suite says nothing about it.
|
|
|
724
760
|
the shipped profile for the plan target when pi's registry can serve
|
|
725
761
|
that model. The `model` argument itself is passed through
|
|
726
762
|
byte-for-byte, so a malformed spec still produces pi's own error.
|
|
727
|
-
- **A mid-thread
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
the warm one. The router compares base rates, not the cache state
|
|
731
|
-
of your thread.
|
|
763
|
+
- **A mid-thread route can cost the prompt cache.** A model or effort
|
|
764
|
+
change makes the prefix cold. The thread-choice planner prices that
|
|
765
|
+
cache state after routing. See [Thread choice after routing](#thread-choice-after-routing).
|
|
732
766
|
- **The window guard can move an action to a model you did not
|
|
733
767
|
choose** (it warns, and names what it replaced), and it never
|
|
734
768
|
blocks: if nothing listed is wider, the action runs and pi
|
|
@@ -0,0 +1,223 @@
|
|
|
1
|
+
# Thread cache cost
|
|
2
|
+
|
|
3
|
+
This document explains Slate's implemented choice between continuing a worker thread and starting a fresh one. It carries mechanism and evidence that would be too costly in Slate's doctrine.
|
|
4
|
+
|
|
5
|
+
## Evidence record
|
|
6
|
+
|
|
7
|
+
This document is the durable measurement summary. The deleted research log was folded into the description of [pull request 130](https://github.com/JetBrains/ytdb-slate/pull/130), so that description preserves the approved decisions and their reasoning. It does not contain raw measurements, per-arm tables, or transcripts, and no repository copy of those raw records survives. For more decision rationale and implementation review history, read the pull request. The numerical evidence needed to assess the rules is carried below. Dates are measurement dates, not guarantees about later provider behavior.
|
|
8
|
+
|
|
9
|
+
Every numerical figure carries one of these evidence labels:
|
|
10
|
+
|
|
11
|
+
- **Measured probe** reports an observed provider response or local run. Its date and sample size bound the claim. Re-run it under the matching refresh condition below. Slate accepts cache-retention probes for 90 days, then treats them as stale.
|
|
12
|
+
- **Documented provider price** reports a dated provider price row. The profile table records when a row was observed or took effect. No age check rejects an old price row, so maintainers must review prices under the refresh conditions below.
|
|
13
|
+
- **Modelled projection** computes a result from stated inputs. It stays current only while every input and assumption stays current. Regenerate it when any listed input changes.
|
|
14
|
+
- **Implemented constant** reports a code or configuration value. Re-check it whenever the named implementation changes.
|
|
15
|
+
|
|
16
|
+
## Terms
|
|
17
|
+
|
|
18
|
+
A **warm thread** can reuse the cached request prefix from its previous turn. That prefix includes the worker instructions, tools, conversation, and prior tool results accepted by the provider's cache.
|
|
19
|
+
|
|
20
|
+
A **cold thread** cannot reuse that prefix on its next request. The provider must process the prefix again. A cache hit remains a provider decision, so a warm classification predicts eligibility rather than guaranteeing billed cached tokens.
|
|
21
|
+
|
|
22
|
+
A **turn** is one worker model response and its associated tool activity. The planner prices a caller-supplied turn estimate.
|
|
23
|
+
|
|
24
|
+
## Total cold transitions
|
|
25
|
+
|
|
26
|
+
Changing an existing thread's model makes its next request cold. Changing its effort level also makes the request cold, even when the model stays the same. Treat either change as a rewrite of the whole prefix.
|
|
27
|
+
|
|
28
|
+
The two transition claims have different evidence.
|
|
29
|
+
|
|
30
|
+
**Measured probe, model-change method, 2026-08-06.** A transcript audit covered three real OpenAI worker threads and 71 assistant turns. It found one in-thread model transition, from `gpt-5.6-luna` to `gpt-5.6-sol`. The switch request read **0** cached tokens and wrote **53,763**; its next request recovered to a 0.528 cache-hit fraction. Thus the model-change sample size is **n=1**, on one provider family. It was observational rather than a controlled arm, and no Anthropic model-change sample was run.
|
|
31
|
+
|
|
32
|
+
**Measured probe, effort-change method, 2026-08-06.** A corrected controlled run gave each arm a distinct approximately 18,000-token system prefix and a unique tool. It changed only the live effort through the same `setThinkingLevel` route Slate uses. Each provider family had one unchanged control and three changed arms: step up, step down, and `off`.
|
|
33
|
+
|
|
34
|
+
| family and model | unchanged control | changed arms | changed-arm result |
|
|
35
|
+
| --- | ---: | ---: | --- |
|
|
36
|
+
| OpenAI `gpt-5.6-luna` | 18,113 written, then 18,113 read | 3 | each read 0 after an approximately 18,115-token write |
|
|
37
|
+
| Anthropic `claude-sonnet-5` | 18,555 written, then 18,555 read | 3 | each read 0 after an approximately 18,556-token write |
|
|
38
|
+
|
|
39
|
+
The effort-change sample is **n=6 transitions** across two model families, with **n=2 controls**. No changed arm showed partial survival. An earlier calibration run had cross-arm tool-definition reuse and was excluded; unique tools removed that independent cache breakpoint.
|
|
40
|
+
|
|
41
|
+
**Measured probe, effort-change replication, 2026-08-06.** A second controlled harness used a dedicated cache key for every probe, an approximately 6,000-token prefix, and a 60-second gap. The harness also asserted the effort value sent on every request. A low-to-high change produced zero cache reads in **3 of 3 probes** on OpenAI `gpt-5.6-luna` and **3 of 3 probes** on Anthropic `claude-sonnet-5`. Same-effort controls remained warm in **3 of 3 probes** on each model. The sample is three probes per changed or control cell. It tested only a low-to-high transition and only one model from each provider family.
|
|
42
|
+
|
|
43
|
+
Anthropic documents that a thinking or effort change invalidates the message portion of its cache, so the Anthropic result confirms documented behavior. OpenAI documents no cache effect for an effort change in either direction, so the OpenAI result is new evidence rather than a confirmation of a published guarantee.
|
|
44
|
+
|
|
45
|
+
The decision rule therefore treats the transitions as binary:
|
|
46
|
+
|
|
47
|
+
- the model and effort both stay unchanged, so the existing thread remains eligible for prefix reuse;
|
|
48
|
+
- either value changes, so none of the existing prefix receives warm-thread credit.
|
|
49
|
+
|
|
50
|
+
This rule is conservative rather than a provider guarantee. Provider caching is best effort. The same 2026-08-06 transcript audit found one unexplained OpenAI miss 571 seconds after the prior dispatch, inside the observed retention window and without a model change or known expiry. A zero read can therefore occur without a transition, and these samples cannot prove that every future model, API path, or provider release behaves the same way.
|
|
51
|
+
|
|
52
|
+
**Measured probe, unexplained-miss follow-up, 2026-08-06.** The 571-second miss did not reproduce. On the same OpenAI `gpt-5.6-sol` model, one dedicated key remained warm in **5 of 5 probes at 571 seconds** and **3 of 3 probes at 1,500 seconds**. A shared key under contention remained warm in **4 of 4 probes at 571 seconds**. An earlier OpenAI `gpt-5.6-luna` arm remained warm in **12 of 12 probes at 571 and 1,500 seconds** while four prefixes competed and a writer ran every 30 seconds.
|
|
53
|
+
|
|
54
|
+
The follow-up means the original miss is rare or environment-specific. It also means cache-key contention is not the explanation at this tested scale. It does not explain the original miss, which remains unresolved. The original observation stays in this record because failure to reproduce it does not erase the observation.
|
|
55
|
+
|
|
56
|
+
## Fresh-thread rediscovery
|
|
57
|
+
|
|
58
|
+
A fresh thread lacks the working context that an existing thread accumulated. Episodes can transfer durable results, but the new worker still has to orient itself to the task and repository.
|
|
59
|
+
|
|
60
|
+
**Measured probe, method and result, 2026-08-06.** One repository follow-up was given to a continuation and to a fresh thread with durable episode context. The fresh arm made **one more tool call** than the continuation and produced the better answer, with line numbers and a flagged error. The usable sample is **n=1 task pair**. It was not designed as a model-family comparison, and the surviving decision record does not identify the worker models, so no provider-family conclusion follows. A second question supplies no independent sample because its continuation arm was contaminated by prior self-correction.
|
|
61
|
+
|
|
62
|
+
Slate expresses the observed **+1 tool call** as an approximate one-turn rediscovery penalty. Charge the fresh-thread option one extra turn when comparing it with continuation. The raw transcript and absolute per-arm call counts were not retained; only the measured difference, answer comparison, and contamination finding survive. The sample cannot establish a distribution or a universal constant. A simple action may need no rediscovery, while a poorly compressed or unusual work stream may need more.
|
|
63
|
+
|
|
64
|
+
## Basis of the two-turn guard
|
|
65
|
+
|
|
66
|
+
**Modelled projection, method, 2026-08-06.** Six cost-model grids covered OpenAI and Anthropic paths. Each grid crossed context sizes **20k, 40k, 60k, 80k, 120k, and 160k** with **1, 3, 5, 10, and 20** future turns: 30 cells per grid and **180 modeled cells** in total, not 180 live dispatches. The cases were OpenAI warm and cold before shared seed reads, OpenAI warm and cold after shared seed reads, and Anthropic warm and cold.
|
|
67
|
+
|
|
68
|
+
The projection used **measured probe** seeds of **16,116 OpenAI tokens** and **26,481 Anthropic tokens**, each observed twice. Its model inputs were three injected episodes totaling **5,400 OpenAI tokens** or **7,200 Anthropic tokens**, plus **500 task tokens**, **3,500 growth tokens per turn**, and **600 output tokens per turn**. **Documented provider price:** the dated current rows produced a cache-write/cache-read ratio of **12.5**. Those inputs gave this modelled boundary:
|
|
69
|
+
|
|
70
|
+
`existing context > seed + episodes + 11.5 × written share / expected turns`
|
|
71
|
+
|
|
72
|
+
**Modelled projection:** the exact boundaries show why turn count matters.
|
|
73
|
+
|
|
74
|
+
| modelled case | 1 turn | 3 turns | 5 turns | 20 turns |
|
|
75
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
76
|
+
| OpenAI warm, seed written | 269k | 104k | 71k | 34k |
|
|
77
|
+
| OpenAI warm, seed read | 84k | 43k | 34k | 25k |
|
|
78
|
+
| OpenAI cold | 22k | 22k | 22k | 22k |
|
|
79
|
+
| Anthropic warm, seed read | 118k | 62k | 51k | 38k |
|
|
80
|
+
| Anthropic cold | 34k | 34k | 34k | 34k |
|
|
81
|
+
|
|
82
|
+
A 50k OpenAI threshold classified **22 of 30** warm, seed-written cells correctly and missed eight. Five misses were material: four at one turn and one at three turns. The guard therefore places one- and two-turn work on the continuation side. Two turns was inferred as the boundary between modeled one- and three-turn cases; no live two-turn break-even experiment was run.
|
|
83
|
+
|
|
84
|
+
The grids cannot establish a universal break-even point. Their result depends on the project's prompt seed, episode size, cache state, input and cache-write prices, expected output, and the accuracy of the orchestrator's turn estimate. They also optimize one dispatch rather than the permanent context growth of a whole work stream.
|
|
85
|
+
|
|
86
|
+
## Implemented choice mechanism
|
|
87
|
+
|
|
88
|
+
Unless a paragraph carries another label, every figure in this section is an **implemented constant**.
|
|
89
|
+
|
|
90
|
+
Slate divides thread choice between the orchestrator and the `thread` tool. The orchestrator retains every semantic decision.
|
|
91
|
+
|
|
92
|
+
The orchestrator chooses the work stream, thread, model, effort, task, and required context. It also decides whether episodes can replace the live transcript.
|
|
93
|
+
|
|
94
|
+
The tool makes one narrow economic judgement. It decides whether continuing the named thread costs more than a fresh thread seeded with named episodes.
|
|
95
|
+
|
|
96
|
+
### Permission through `freshContext`
|
|
97
|
+
|
|
98
|
+
The `freshContext` contract asks three separate questions:
|
|
99
|
+
|
|
100
|
+
1. **Presence:** a continuation requires the argument only when `threadChoice.act` is `true`.
|
|
101
|
+
2. **Validity:** every supplied value is validated on every call.
|
|
102
|
+
3. **Delivery:** every continuation delivers a supplied valid value to the planner. Creation has no planner, so it delivers nothing.
|
|
103
|
+
|
|
104
|
+
The twelve distinct combinations follow. Creation has one behavior for both acting settings, so its four input rows cover acting off and on.
|
|
105
|
+
|
|
106
|
+
| call | acting | input | result |
|
|
107
|
+
| --- | --- | --- | --- |
|
|
108
|
+
| create | off or on | omitted | accepted, with nothing delivered |
|
|
109
|
+
| create | off or on | malformed | tool error before creation or mutation |
|
|
110
|
+
| create | off or on | `[]` | accepted and unused |
|
|
111
|
+
| create | off or on | non-empty list | accepted and unused if every id names an existing episode, otherwise a tool error |
|
|
112
|
+
| continue | off | omitted | accepted, with no permission delivered to the planner |
|
|
113
|
+
| continue | off | malformed | tool error before mutation |
|
|
114
|
+
| continue | off | `[]` | accepted, with an explicit refusal delivered to the planner |
|
|
115
|
+
| continue | off | non-empty list | permission and seed episodes reach the planner if every id exists, otherwise a tool error |
|
|
116
|
+
| continue | on | omitted | tool error before mutation |
|
|
117
|
+
| continue | on | malformed | tool error before mutation |
|
|
118
|
+
| continue | on | `[]` | accepted, with an explicit refusal delivered to the planner |
|
|
119
|
+
| continue | on | non-empty list | permission and seed episodes reach the planner if every id exists, otherwise a tool error |
|
|
120
|
+
|
|
121
|
+
A valid array stays within the size limit and contains only strings. Slate never treats missing permission as consent. Slate also does not judge whether the named episodes preserve enough meaning. The orchestrator must make that judgement.
|
|
122
|
+
|
|
123
|
+
### Planner verdicts and order
|
|
124
|
+
|
|
125
|
+
The planner returns one of four verdicts:
|
|
126
|
+
|
|
127
|
+
- `continue` keeps the named thread.
|
|
128
|
+
- `fresh` says the fresh option costs less.
|
|
129
|
+
- `abstain` says the available inputs cannot support a choice.
|
|
130
|
+
- `refused` says a rule forbids a fresh thread.
|
|
131
|
+
|
|
132
|
+
Every verdict carries a machine code and a human-readable reason. Priced verdicts include both arm estimates and their warmth evidence.
|
|
133
|
+
|
|
134
|
+
The planner evaluates refusals before cache evidence or prices. Missing permission, unusable tools, and a failed last dispatch can forbid a fresh thread. The dispatch boundary rejects malformed or unknown episodes before the planner runs.
|
|
135
|
+
|
|
136
|
+
A dispatch without a source thread starts a new work stream. No comparison is needed because no existing prefix can be reused.
|
|
137
|
+
|
|
138
|
+
The pure planner keeps work of one or two turns on the named thread. The current dispatch caller estimates at least three turns from work-stream depth.
|
|
139
|
+
|
|
140
|
+
The doctrine therefore uses an empty `freshContext` for one-turn or two-turn work. It also uses an empty value when the live transcript is required.
|
|
141
|
+
|
|
142
|
+
For longer work, the planner prices both arms over every expected turn. It charges the fresh arm one extra rediscovery turn.
|
|
143
|
+
|
|
144
|
+
Input, cache-read, and cache-write buckets remain disjoint. Long-context billing applies independently to each arm at the exact threshold.
|
|
145
|
+
|
|
146
|
+
**Documented provider price:** the dated current rows produce a cache-write to cache-read ratio of **12.5**. A false cold estimate can therefore destroy a valuable warm prefix.
|
|
147
|
+
|
|
148
|
+
A tie continues the named thread. Uncertainty never becomes permission to destroy a warm prefix.
|
|
149
|
+
|
|
150
|
+
### Reporting, acting, and lineage
|
|
151
|
+
|
|
152
|
+
Configure the feature under `threadChoice` in the project's `.pi/slate.json` file. This example reports choices and enables automatic restarts:
|
|
153
|
+
|
|
154
|
+
```json
|
|
155
|
+
{
|
|
156
|
+
"threadChoice": {
|
|
157
|
+
"report": true,
|
|
158
|
+
"act": true
|
|
159
|
+
}
|
|
160
|
+
}
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
**Implemented constants:** `threadChoice.report` defaults to `true`, while `threadChoice.act` defaults to `false`. The default therefore reports the verdict without moving work. Only a `fresh` verdict can move work when acting is enabled.
|
|
164
|
+
|
|
165
|
+
Acting adds semantic refusals after the economic verdict. Reviewer and adversarial threads keep their live review transcripts.
|
|
166
|
+
|
|
167
|
+
A successor with no episode of its own cannot restart again. It must publish new evidence before another restart.
|
|
168
|
+
|
|
169
|
+
Slate writes the successor and restart lineage before the successor publishes an episode. The source record names the successor through `supersededBy`.
|
|
170
|
+
|
|
171
|
+
A later dispatch to the superseded source is a tool error. The error names the successor that must continue the work stream.
|
|
172
|
+
|
|
173
|
+
A restart carries the source thread's type, tools, and base route. The named `freshContext` episodes become the successor's injected context.
|
|
174
|
+
|
|
175
|
+
A restart has an explicit commit point. The commit point occurs when the successor's worker session starts executing its prompt.
|
|
176
|
+
|
|
177
|
+
Before the commit point, an abort or failure removes the successor and restores the source. A failure then runs the action on the source. An abort ends the action without running it again.
|
|
178
|
+
|
|
179
|
+
At or after the commit point, an abort or failure never rolls the restart back. The successor keeps its lineage, and Slate reports the action outcome on that successor. Slate never repeats the committed action on the source.
|
|
180
|
+
|
|
181
|
+
### Missing and stale evidence
|
|
182
|
+
|
|
183
|
+
The planner abstains rather than guessing when missing or stale evidence prevents honest pricing.
|
|
184
|
+
|
|
185
|
+
**Implemented constant:** the dispatch path accepts documented and measured retention evidence for 90 days. Older retention rows grant no retention window.
|
|
186
|
+
|
|
187
|
+
Documented retention comes from provider sources with retrieval dates. Measured retention comes from dated local probes listed in the model profiles.
|
|
188
|
+
|
|
189
|
+
Missing retention data does not prove expiry. The planner treats an unchanged route as warm when retention or elapsed time is unknown.
|
|
190
|
+
|
|
191
|
+
An unknown shared-seed cache state is priced as a write. This conservative assumption makes a fresh thread look no cheaper than the evidence supports.
|
|
192
|
+
|
|
193
|
+
The planner abstains when required prices or sizes are unavailable. Required sizes include the existing prefix, fresh seed, and named episodes.
|
|
194
|
+
|
|
195
|
+
A failed episode write makes the live session newer than its durable measurements. The next choice drops those stale measurements and abstains.
|
|
196
|
+
|
|
197
|
+
Do not preserve warmth by choosing a model or effort that cannot clear the action. Cache savings never justify lower required capability.
|
|
198
|
+
|
|
199
|
+
## Cache-key sharding experiment
|
|
200
|
+
|
|
201
|
+
**Measured probe, 2026-08-06.** A controlled experiment tested the implementation as shipped in this branch on the OpenAI `gpt-5.6-luna` path. Two arms used distinct long prefixes. Each arm created four worker threads sequentially in one persistent process. Treatment used the default two shards. Control set `cacheKeyEnabled: false`, so Slate installed no shared key.
|
|
202
|
+
|
|
203
|
+
| measured arm | threads | input tokens | cached tokens | cached/input | worker cost |
|
|
204
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
205
|
+
| two-shard treatment | 4 | 119,124 | 59,496 | 0.499 | $0.01612032 |
|
|
206
|
+
| disabled control | 4 | 119,116 | 0 | 0.000 | $0.02980240 |
|
|
207
|
+
|
|
208
|
+
**Measured probe:** worker cost fell 45.91%. All-in cost was $0.04249825 for treatment and $0.05489782 for control, or $0.09739607 across both arms.
|
|
209
|
+
|
|
210
|
+
The per-thread pattern is evidence that the provider path honors the key. The first thread on each treatment shard wrote its full prefix. The later thread on the same shard read 29,748 cached tokens. Threads with identical prefixes but different shard keys did not share cache. That pattern is inconsistent with a provider ignoring the key.
|
|
211
|
+
|
|
212
|
+
The experiment is thin: four threads per arm. Treatment ran before control. Traffic passed through a local proxy, so attribution to the upstream provider is indirect. The run covers only the first dispatch. It does not cover long conversations, retention expiry, compaction, other models, other providers, or high request rates. This result supersedes every earlier cache-key measurement in the pull request record.
|
|
213
|
+
|
|
214
|
+
## Limits and refresh conditions
|
|
215
|
+
|
|
216
|
+
Re-measure each claim when its inputs can have changed:
|
|
217
|
+
|
|
218
|
+
- **Cold transitions:** repeat controlled model and effort arms after a provider changes cache semantics, pi changes request construction, Slate changes route application, or a new model family becomes supported. A future model-change run should include controlled samples from both provider families because the current model evidence is one OpenAI event.
|
|
219
|
+
- **Rediscovery:** repeat controlled pairs across several task shapes, model families, and repository sizes. Replace the one-turn charge only after the distribution is stable enough to support another policy.
|
|
220
|
+
- **Two-turn guard and prices:** regenerate the grids after material changes to prompt-seed size, episode construction, provider prices, cache-write behavior, cache-key behavior, compaction, or supported model families. Validate the inferred two-turn boundary with live two-turn actions when possible. Price rows have no automatic age check, so review their provider sources whenever this projection is refreshed.
|
|
221
|
+
- **Cache-key sharding:** repeat with more threads and randomized arm order after changes to key derivation, shard assignment, the disable switch, the local proxy, the OpenAI API path, or provider cache policy. Add continuation, expiry, and compaction arms before applying the first-dispatch result to those cases.
|
|
222
|
+
|
|
223
|
+
This document describes decision inputs. The provider's reported usage remains the authority for the cost that actually occurred.
|