ytdb-slate 0.1.0 → 0.1.2

package/README.md

@@ -4,6 +4,49 @@ Slate is a thread-weaving orchestration extension for the [pi coding agent](http
 
 The orchestrator (your main pi session) dispatches **bounded actions** to persistent **worker threads**. Each completed action is compressed by an LLM into an **episode** — a durable, structured record (intent, actions, findings, artifacts, open issues, handoff notes) that the orchestrator composes into further dispatches instead of re-reading raw transcripts. On top of that, Slate injects a mandatory workflow doctrine: research before design, design review, adversarial review, and track review — with optional umbrella **draft-PR publishing** for tracks.
 
+## Why Slate?
+
+Long-horizon agentic work fails on context management, not model capability. Existing architectures each solve a piece of the problem and trade away the rest. Compaction is unpredictably lossy; naive subagents isolate context but hand back only a single response string; markdown plans go stale and get under-executed. Rigid task trees can't adapt to information discovered mid-task, and planner/executor stacks synchronize through compress-and-return boundaries that risk dropping critical state.
+
+Slate's answer is the **thread**: the orchestrator dispatches one bounded action at a time; the worker executes it and returns an **episode**. The orchestrator keeps the reactivity of a plain agent loop while gaining the context isolation, compaction, and parallelism that the single-context loop lacks.
+
+| Aspect | ReAct | Markdown plan | Task trees | RLM (Recursive Language Models) | Devin / Manus / Altera | Claude Code / Codex subagents | Slate |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Planning | implicit | file | explicit tree | REPL | planning agent | plan mode | implicit / adaptive — no upfront static plan |
+| Decomposition | none | none | direct tree | REPL functions | task-based | subagent delegation | implicit |
+| Synchronization | single thread | single thread | gated steps | REPL return | reduce & return | message passing | episodes |
+| Intermediate feedback | per step | per step | on task failure | on execution end | after compress | message passing | per episode |
+| Context isolation | none | none | per subtask | per subcall | subagent | subagent | per thread |
+| Context compaction | none | none | task-based | REPL slicing | subagent compress | compaction | episode compress |
+| Parallel execution | none | none | none | in REPL | Altera only | native | native |
+| Expressivity | high | high | low | high | medium | medium | high |
+| Adaptability | yes | if plan updated | no | limited — no mid-run course correction | yes | limited by message passing | yes |
+
+Characterizations of third-party systems reflect their publicly described designs at the time of writing. The taxonomy is adapted from the Random Labs technical report introducing the thread-weaving "Slate" architecture (its agent ships as the npm package `@randomlabs/slate`); ytdb-slate is an independent implementation of that architecture for pi.
+
+**Threads are not subagents:**
+
+- **Per-episode feedback.** Each thread executes one bounded action and hands control back. The orchestrator adapts after every episode — reactive like a ReAct loop — instead of firing off a subagent and hoping the result comes back usable.
+- **Compaction at a chosen boundary.** Compression happens at a predictable point — action completion — instead of mid-stream when context overflows. The compression itself is still LLM-performed and lossy, but the boundary is chosen, not forced.
+- **Episodes compose.** One thread's episode can seed another thread's context, so conclusions travel across the system by reference. Subagents pass back a single response string; episodes are durable, structured, reusable records.
+
+Full rationale: [`docs/design-principles.md`](docs/design-principles.md), shipped in the package — if this summary and that document disagree, the document wins.
+
+## Feature development workflow
+
+In orchestrator mode, Slate injects a track-based development workflow as doctrine — it is not optional; project configuration can extend it (`doctrineExtraPath`) but never replace it. The flow:
+
+1. **Research** — interactive exploration; a research log opens lazily when defined triggers fire (non-trivial decisions, surprises, risky invariants, session boundaries).
+2. **User design review** — the user approves the design direction before any machine review.
+3. **Adversarial review** — a fresh-context reviewer attacks the design pre-implementation; findings are triaged with the user.
+4. **Track split approval** — the user approves how the change splits into independently reviewable tracks.
+5. **Per-track loop** — implement → agent code review → fixes → mandatory user review → marker commit (an empty commit marking the track's boundary in history).
+6. **Delivery** — with draft-PR publishing enabled, the squash-merge is performed by the user; the agent never merges. With it disabled (the default), the change lands as a squashed commit that carries the workflow log's key content in its message.
+
+The workflow scales with change size: multi-track changes get the full flow, single-track changes skip the split and the marker commits, and trivial changes shrink the pre-implementation reviews to consent plus a micro-review (skippable with explicit user consent) — the agent code review and the mandatory user review still apply at every tier. Umbrella draft-PR publishing activates only when `workflow.draftPRs` is `true`.
+
+This summary is orientation only; the shipped docs listed below are normative.
+
 ## Install
 
 Install into a project (pinned, project-scoped — recorded in `.pi/settings.json`):
@@ -23,7 +66,7 @@ Optional config file: `slate.json` in the project's pi config dir (`.pi/slate.js
 | `orchestratorModeDefault` | boolean | `false` | Start fresh interactive sessions with orchestrator mode ON. |
 | `episodeModel` | string | newest available Anthropic Sonnet, else the worker's own model | Model (`provider/id`) used to compress a finished action into an episode. |
 | `workerTools` | string[] | `["read", "bash", "edit", "write", "grep", "find", "ls"]` | Tools available to worker threads (an empty list also falls back to the default). |
-| `maxConcurrent` | number | `4` | Maximum number of worker actions running concurrently. |
+| `maxConcurrent` | number | `4` | Maximum number of worker actions running concurrently (must be ≥ 1 — unenforced: a value of 0 or less silently hangs all dispatches). Excess dispatches wait in a queue; actions on the same thread always run in dispatch order. Default rationale: shipped `docs/design-principles.md` §5 (repo-local note). |
 | `pauseThresholdPercent` | number | `40` | Orchestrator context budget (percent, in `(0, 100]`) at which Slate auto-pauses and prepares a fresh-session handoff. |
 | `orchestratorPromptDocs` | string[] | `[]` | Project markdown files (paths relative to the project root) whose **contents** are appended to the orchestrator system prompt. |
 | `workerPromptDocs` | string[] | `[]` | Project markdown files whose **contents** are appended to every worker-thread system prompt. |

package/docs/design-principles.md

@@ -151,6 +151,19 @@ or two problems for the others:
 | P9 parallelism | `threads.ts` `maxConcurrent` queueing; doctrine rule 2 in `mode.ts` |
 | P10 context as RAM | `handoff.ts` context-budget auto-pause + fresh-session handoff |
 
+Repo-local note (not from the report): the `maxConcurrent` cap defaults
+to 4. Its failure modes are asymmetric: excess dispatches wait for a
+slot (actions on the same thread additionally serialize in dispatch
+order), so a low cap costs only latency, while a cap above the
+provider's effective rate limits turns rate-limit exhaustion into
+FAILED episodes and raises the unattended cost burn rate. Over its
+lifetime a slot covers a multi-turn worker conversation followed by its
+episode-compression call. The default is sized to cover typical
+parallel batches (recon fan-outs, a review wave of a few perspectives)
+while staying safe on common consumer API tiers; wider fan-outs only
+pay tail latency, and projects with higher-tier keys raise the cap in
+`slate.json`.
+
 ## 6. Runtime knowledge: what the orchestrator knows, and when
 
 The orchestrator's knowledge of these principles is two-tier, following

package/extension/state.ts

@@ -45,7 +45,7 @@ export interface SlateSnapshot {
 export interface SlateConfig {
 	episodeModel?: string; // "provider/id" for the episode compressor (D5)
 	workerTools?: string[];
-	maxConcurrent?: number;
+	maxConcurrent?: number; // global cap on concurrently running worker actions (default 4; must be ≥ 1 — unenforced, ≤ 0 silently hangs all dispatches; rationale: docs/design-principles.md §5 repo-local note)
 	pauseThresholdPercent?: number; // orchestrator context budget for auto-pause (default 40)
 	orchestratorModeDefault?: boolean; // seed orchestrator mode ON for fresh interactive sessions (unsaved until first real mutation)
 	orchestratorPromptDocs?: string[]; // role-guideline docs appended to the orchestrator prompt (cwd-relative paths, default none)

package/extension/threads.ts

@@ -83,7 +83,7 @@ export class ThreadManager {
 		private store: SlateStore,
 		private config: SlateConfig,
 	) {
-		this.semaphore = new Semaphore(config.maxConcurrent ?? 4);
+		this.semaphore = new Semaphore(config.maxConcurrent ?? 4); // default rationale: docs/design-principles.md §5 repo-local note
 	}
 
 	getConfig(): SlateConfig {

package/extension/worker.ts

@@ -14,12 +14,10 @@
 import { mkdirSync } from "node:fs";
 import { resolve } from "node:path";
 import {
-	AuthStorage,
 	CONFIG_DIR_NAME,
 	createAgentSession,
 	DefaultResourceLoader,
 	getAgentDir,
-	ModelRegistry,
 	SessionManager,
 	SettingsManager,
 	type ExtensionContext,
@@ -66,8 +64,6 @@ export async function openWorkerSession(opts: {
 	mkdirSync(dir, { recursive: true });
 
 	const agentDir = getAgentDir();
-	const authStorage = AuthStorage.create();
-	const modelRegistry = ModelRegistry.create(authStorage);
 
 	// Trust propagation (mirrors vanilla pi's runtime SettingsManager): when no
 	// settingsManager is passed, the SDK default-constructs one with
@@ -104,11 +100,16 @@ export async function openWorkerSession(opts: {
 		? SessionManager.open(opts.sessionFile)
 		: SessionManager.create(ctx.cwd, dir);
 
+	// No modelRuntime passed: createAgentSession (pi >= 0.80.8) defaults to a
+	// ModelRuntime replacing the AuthStorage + ModelRegistry setup this code
+	// hand-built before those SDK options were removed. Credential/config
+	// sources are unchanged — global agentDir auth.json + models.json, never
+	// project-local — but the default is a superset: it also reads/writes
+	// agentDir/models-store.json and may run a throttled (~4h-cached)
+	// create-time network catalog refresh (disabled by PI_OFFLINE).
 	const { session } = await createAgentSession({
 		cwd: ctx.cwd,
 		agentDir,
-		authStorage,
-		modelRegistry,
 		model: model ?? undefined,
 		tools: opts.tools && opts.tools.length > 0 ? opts.tools : DEFAULT_WORKER_TOOLS,
 		resourceLoader: loader,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ytdb-slate",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Slate: a thread-weaving orchestration extension for the pi coding agent — bounded worker actions, compressed episodes, mandatory research/design/adversarial/track review workflow, optional draft-PR publishing.",
   "license": "Apache-2.0",
   "keywords": [