@tokensapi/dsh-progressive-tools 0.1.0

package/docs/architecture.md

@@ -0,0 +1,225 @@
+# Architecture
+
+## Goals and invariants
+
+The default architecture has four invariants:
+
+1. The first AgentLoop request already carries the small surface.
+2. Discovery never changes the top-level tool list or generated SDK.
+3. Deferred calls still traverse the complete DSH execution pipeline.
+4. A caller cannot bypass discovery by naming a hidden tool directly.
+
+The plugin uses only public Harness APIs and extension points:
+
+- `ctx.systemPrompt.section()` for stable discovery guidance;
+- `system-prompt/assemble` for the authoritative request projection;
+- `ctx.tools.register()` for search and dispatch;
+- `ctx.tools.guard()` for monotonic routing enforcement;
+- `ctx.tools.execute()` for nested real-tool execution;
+- `tools/result` for authoritative discovery commits;
+- `tools/change` for in-process catalog invalidation;
+- `agent/session-start` and durable session events for state initialization;
+- `agent/disposed` and Cordis effects for cleanup.
+
+## The DSH lifecycle boundary
+
+DSH assembles prompt sections and tool schemas before `agent/pre-step`:
+
+```text
+claim inbox input
+      │
+      ▼
+systemPrompt.assemble()
+      │
+      ├── collect sections and tools
+      └── system-prompt/assemble waterfall
+                         │
+                         ▼
+                  agent/pre-step
+                         │
+                         ▼
+                    LLM request
+```
+
+Version 0.1.0 installed a restriction in `agent/pre-step`, so it could affect
+only the following assembly. Version 0.2.0 performs the authoritative
+projection in `system-prompt/assemble`, after all providers have contributed
+but before AgentLoop stores or sends the request.
+
+`agent/session-start` eagerly initializes the catalog for normal creation and
+resume. The assembly hook remains authoritative and also covers hot reload,
+late registration, and callers that assemble without a normal startup event.
+
+## Stable-proxy mode
+
+### Request projection
+
+On the first assembly for one agent, the plugin freezes that session's stable
+name set:
+
+- `tool_search`;
+- `tool_dispatch`;
+- registered names matching `alwaysVisible`;
+- the reserved `run_code` transport when DSH exposes it.
+
+The Tokens distribution defaults `alwaysVisible` to the common file, shell,
+Skill, task, interaction, and file-return tools used on nearly every product
+session. This keeps routine work direct while deferring plugin-specific and
+low-frequency schemas.
+
+The complete registry remains visible to in-process code. The assembly
+waterfall filters only the final `PromptAssembly.tools` projection. In Code
+Mode and `both` mode, the `tools:sdk` section is regenerated from the same
+stable names, preserving one coherent presentation.
+
+`deferToolGuidance` conservatively removes a section only when its name is an
+exact hidden-tool guidance slot (`tool:<name>` or `tool:<name>:...`). General
+guidance and sections that cannot be mapped safely remain untouched.
+
+The stable name set does not grow after first assembly. A later registry change
+refreshes only the searchable catalog, so installing a new deferred tool does
+not silently change an active session's cache prefix.
+
+### Discovery
+
+The catalog stores detached model-facing definitions in process memory. Search
+operates on individual tools, not whole families. A searchable document
+contains:
+
+- tool name and description;
+- nested parameter keys, descriptions, constants, and enums;
+- configured family ID, description, and aliases.
+
+Ranking combines exact-name and contained-label bonuses with a deterministic
+BM25-style lexical score. CJK text is additionally tokenized into character
+bigrams, so queries without space-delimited words can match definitions and
+family metadata without configured aliases. Search returns at most
+`maxResults` exact definitions; larger `max_results` requests are clamped.
+Those definitions enter the ordinary tool result and therefore extend history
+append-only.
+
+Each match also lists every member name of its family (`groupTools`), and the
+whole family becomes discovered in the same call. One query therefore opens a
+plugin's complete tool surface even when only its top-ranked members carry
+full schemas; the remaining siblings dispatch by name and validate against
+their original definitions, or can be schema-loaded first with one exact-name
+search.
+
+The `status` action lists every deferred family with its member tool names, so
+the model can browse the catalog when a search query has no lexical overlap.
+By default the listing is browse-only; `statusGrantsDiscovery` optionally
+turns it into a catalog-wide discovery grant.
+
+Successful `tools/result` observation commits the returned names to the
+agent's discovered set. Failed or invalid searches do not mutate live state.
+
+### Dispatch
+
+`tool_dispatch` accepts an exact discovered name and one JSON object of
+arguments. It then calls the ordinary registry:
+
+```text
+tool_dispatch root execution
+      │
+      ├── check catalog membership and discovery state
+      ├── preserve agent, rootCallId, signal, and arguments
+      └── ctx.tools.execute(real tool, parent = dispatcher token)
+                         │
+                         ├── pre-execute / approval
+                         ├── monotonic guards
+                         ├── timeout and retry wrappers
+                         ├── original schema validation and body
+                         ├── post-execute and finalization
+                         └── tools/result
+```
+
+Nested contexts and a successful turn-conclusion marker are ferried back to
+the outer result. The outer rendering uses the real tool's finalized content,
+including non-text blocks. A nested failure is rethrown with the real tool's
+structured error code preserved, and the dispatcher delegates its
+parallel-scheduling classification to the target tool's own declaration, so
+concurrency-safe deferred tools keep overlapping with sibling calls.
+
+### Routing guard
+
+Prompt filtering alone does not change registry lookup. Stable mode therefore
+registers a monotonic guard:
+
+- stable direct names are allowed;
+- deferred direct names are denied;
+- a nested execution whose parent token belongs to `tool_dispatch` is allowed;
+- descendants of that authorized nested execution inherit the authorization
+  for the duration of their execution tree.
+
+Tokens are registry-minted opaque identities, so a caller cannot manufacture
+the parent capability. Authorized tokens are removed on result and plugin
+cleanup. The guard prepares the agent's state on demand, so a call that
+arrives before the first assembly or session-start event is still classified
+against the deferred catalog instead of passing through unexamined.
+
+The guard is not an authorization boundary for the underlying capability. It
+enforces presentation-to-dispatch alignment while existing approval, sandbox,
+and policy layers continue to own security decisions.
+
+## Cache behavior
+
+For an unchanged session composition:
+
+```text
+request 1: stable tools + stable system + user history
+request 2: stable tools + stable system + prior history + search result
+request 3: stable tools + stable system + prior history + dispatch result
+```
+
+Only history grows. The tool/system prefix remains byte-identical. The
+AgentLoop integration test captures actual `GenerateOptions` values and asserts
+both the first request surface and post-search equality.
+
+A real composition change can still alter the prefix: changing plugin config,
+removing a stable tool, changing its definition, or replacing another system
+section is outside the discovery invariant.
+
+## Resume behavior
+
+Discovery state is recorded asymmetrically. The rendered search result — the
+text the model reads and re-reads in history — carries only the names newly
+discovered by that call, so conversation growth stays bounded no matter how
+much has been discovered. The result's presentation metadata, which never
+reaches the model, carries the cumulative discovered list and the action kind.
+
+On resume the plugin replays successful search results and skill bindings,
+preferring the cumulative metadata when present and unioning per-call
+increments otherwise. The metadata path restores full state from the latest
+surviving entry even when older events were compacted. Nested Code Mode calls
+have no top-level presentation metadata, so their standard
+`tool/code-dispatch` content increments are folded instead. A replayed
+`status` result also restores the catalog-listing grant used by
+`statusGrantsDiscovery`.
+
+The plugin introduces no custom session event vocabulary. Discovered names
+survive registry refreshes such as provider reconnects; dispatch validates
+catalog membership at call time, so a stale name fails the individual call
+without losing the rest of the discovery state.
+
+## Dynamic compatibility mode
+
+`mode: dynamic` retains the v0.1 family activation design for deployments that
+need provider-native definitions after search. Its lifecycle is corrected:
+
+- initial restriction is installed at `agent/session-start`;
+- turn expiry is reconciled at `agent/inbox/claimed`;
+- successful search and skill results reinstall the restriction immediately;
+- the assembly waterfall filters the already-collected current assembly and
+  regenerates Code Mode SDK text from the same visible set.
+
+This mode aligns presentation, lookup, and execution through
+`agent.ctx.tools.restrict()`, but changing families changes the tool prefix and
+can reduce cache reuse. It cannot hide tools registered in the exact agent
+scope because scoped restrictions intentionally preserve scope-local tools.
+
+## Cleanup
+
+Tool registrations, the prompt section, assembly listener, guard, and event
+listeners are Cordis-owned effects. Agent disposal removes its state and lifts
+any dynamic restriction. Plugin unload lifts all remaining restrictions and
+clears authorization tokens before its registered tools disappear.

package/docs/configuration.md

@@ -0,0 +1,238 @@
+# Configuration reference
+
+Configuration is validated when the plugin loads. Invalid modes, names,
+limits, duplicate family IDs, empty patterns, and bindings to unknown families
+fail loudly.
+
+## Options
+
+| Option | Type | Default | Meaning |
+| --- | --- | --- | --- |
+| `mode` | `stable-proxy` \| `dynamic` | `stable-proxy` | Cache-stable dispatch or changing native family activation. |
+| `toolName` | string | `tool_search` | Discovery tool name. |
+| `dispatchToolName` | string | `tool_dispatch` | Stable dispatcher name. Must differ from `toolName`. |
+| `alwaysVisible` | string[] | essential direct tools | Exact names or `*` patterns kept on the fixed direct surface. |
+| `groups` | group[] | built-in rules | Ordered search and dynamic activation families; first match wins. |
+| `skillBindings` | binding[] | `[]` | Successful Skill calls that discover or activate named families. |
+| `maxResults` | integer | `5` | Maximum exact definitions returned by stable search, or group matches in dynamic mode. Caller `max_results` values outside `1..maxResults` are clamped, not rejected. |
+| `requireDiscovery` | boolean | `true` | Require a search hit, a family-wide discovery, or a Skill binding before stable dispatch. |
+| `statusGrantsDiscovery` | boolean | `false` | Let one `status` listing make every cataloged name dispatchable. Off by default so dispatch always follows a seen schema. |
+| `deferToolGuidance` | boolean | `true` | Remove exact hidden `tool:<name>` prompt sections. |
+| `activationGroupLimit` | integer | `1` | Dynamic mode: highest-ranked families activated by one search. |
+| `maxActiveGroups` | integer | `3` | Dynamic mode: maximum retained active families. |
+| `maxActiveToolTokens` | integer | `6000` | Dynamic mode: approximate active-schema budget. |
+| `retentionTurns` | integer | `6` | Dynamic mode: inactive turns before expiry; `0` disables expiry. |
+| `charactersPerToken` | integer | `4` | Compact schema characters represented by one estimate token. |
+
+`activationGroupLimit` cannot exceed `maxActiveGroups`. Dynamic-only fields are
+still validated in stable mode so switching modes cannot reveal a latent bad
+configuration.
+
+## Stable direct tools
+
+The default patterns are:
+
+```yaml
+alwaysVisible:
+  - read
+  - write
+  - edit
+  - glob
+  - grep
+  - bash
+  - skill
+  - ask_user_question
+  - todo_write
+  - dsh_im_return_file
+  - report
+  - submit_*
+  - structured_output*
+```
+
+`tool_search` and `tool_dispatch` are added automatically. A matching tool must
+exist when the first assembly freezes the session surface. Tools registered
+later enter the deferred catalog even when their names match an
+`alwaysVisible` wildcard; this keeps the active session prefix stable. A new
+session sees the new composition.
+
+Keep this list small. Add only tools whose direct schema or completion role is
+worth paying on every request.
+
+## Wildcards
+
+`alwaysVisible`, `groups[].include`, and `groups[].exclude` use anchored `*`
+wildcards over complete tool names. Every other character is matched literally.
+Matching is case-insensitive.
+
+```yaml
+alwaysVisible:
+  - approval_*
+  - submit_*
+```
+
+## Family rules
+
+```yaml
+groups:
+  - id: browser
+    description: Browser navigation and page interaction
+    aliases: [browser, web page, 浏览器]
+    include: [browser_*]
+    exclude: [browser_experimental_*]
+```
+
+Rules are evaluated in list order. A tool belongs to at most one configured
+family. Unmatched tools are grouped by the first underscore-delimited prefix
+when at least two share it; otherwise each tool receives a singleton family.
+Generic verb prefixes (`get`, `set`, `list`, `create`, `delete`, `update`,
+`add`, `remove`, `cancel`, `run`, `start`, `stop`, `send`, `read`, `write`,
+`new`, `check`) never merge, because unrelated plugins routinely share them;
+such tools stay singleton families unless a configured rule claims them.
+
+In stable mode, family metadata improves exact-tool search, the result
+contains the highest-ranked individual definitions, and each match lists every
+member name of its family so siblings become dispatchable from one query. In
+dynamic mode, the highest-ranked whole family becomes natively visible.
+Because family membership now also widens discovery, a wrong rule no longer
+just skews ranking — it unlocks unrelated names. Keep custom rules precise.
+
+The built-in order covers browser, vision, image generation, filesystem,
+terminal, web, database, remote operations, memory, workbench, teams,
+subagents, workflows, and generated interfaces. Explicit rules are recommended
+when third-party packages use unrelated names for one capability.
+
+## Skill bindings
+
+Bindings inspect the `name` or `skill` argument of a successful tool named
+`skill`:
+
+```yaml
+skillBindings:
+  - skill: browser-automation
+    groups: [browser]
+  - skill: database-operations
+    groups: [database]
+```
+
+Stable mode marks bound tools as dispatchable. The Skill instructions should
+describe the tools and their arguments; otherwise call `tool_search` to load
+the exact definitions into history. Dynamic mode activates the bound native
+families under its normal cap, budget, and LRU rules.
+
+## Discovery gate
+
+With `requireDiscovery: true`, `tool_dispatch` accepts names that were matched
+by a successful search, listed as a family sibling of a match, or introduced
+through a Skill binding. This catches guessed or stale tool names before
+nested execution.
+
+A `status` listing is browse-only by default: it shows every family and member
+name but does not unlock dispatch, so the model must load a schema before
+calling. The rejection message points to the deterministic recovery — search
+the exact name once (exact matches always rank first) and dispatch. Set
+`statusGrantsDiscovery: true` to let one status call unlock the whole catalog;
+this trades away the seen-schema guarantee, so reserve it for deployments with
+approval layers or read-only tool surfaces.
+
+Set `requireDiscovery: false` only when another trusted catalog supplies exact
+names and schemas. Direct calls to deferred names remain denied; they must
+still pass through `tool_dispatch`.
+
+## Guidance deferral
+
+`deferToolGuidance: true` removes only sections whose names map exactly to a
+deferred tool:
+
+```text
+tool:<deferred-name>
+tool:<deferred-name>:<suffix>
+```
+
+It does not guess package ownership or delete general sections. This avoids
+silently removing unrelated policy text. A tool package that registers one
+shared family section should move large operational instructions into a Skill
+or keep the section always visible.
+
+## Onboarding an existing plugin ecosystem
+
+Three configuration moves cover most friction when this plugin fronts an
+already-installed tool ecosystem:
+
+1. Add high-frequency small tools to `alwaysVisible`. Anything the model calls
+   nearly every turn (todo updates, reporting) should not pay the
+   search-then-dispatch round trip.
+2. Bind plugin-shipped Skills with `skillBindings`. Loading a Skill should make
+   its tools dispatchable in the same turn; without a binding the model still
+   needs one search after the Skill call.
+3. Write explicit `groups` rules, with multilingual aliases, for packages whose
+   tool names do not follow common prefixes. This fixes automatic-group
+   scatter, improves family search recall, and keeps family-wide discovery
+   from unlocking strangers.
+
+## Token estimates
+
+The estimate for one tool is:
+
+```text
+ceil(JSON.stringify(schema).length / charactersPerToken)
+```
+
+It is deterministic diagnostic data, not provider billing. In stable mode,
+`estimatedSavedTokens` is the complete deferred catalog estimate because none
+of those definitions appears in the top-level request. Search results add only
+the matched definitions to history.
+
+## Dynamic-mode budgets
+
+When a dynamic activation exceeds `maxActiveGroups` or
+`maxActiveToolTokens`, the least recently used unprotected family is removed
+first. Families activated by the current search are protected for that
+operation. A requested family that alone exceeds the budget remains active and
+reports the over-budget estimate.
+
+Dynamic mode changes the request tool set on activation, eviction, and expiry.
+Prefer stable mode when context-cache reuse is important.
+
+## Full stable example
+
+```yaml
+- id: tokens-progressive-tools
+  config:
+    mode: stable-proxy
+    toolName: tool_search
+    dispatchToolName: tool_dispatch
+    maxResults: 5
+    requireDiscovery: true
+    statusGrantsDiscovery: false
+    deferToolGuidance: true
+    alwaysVisible: [read, write, edit, glob, grep, bash, skill, ask_user_question, todo_write, dsh_im_return_file, report, submit_*, structured_output*]
+    groups:
+      - id: browser
+        description: Browser navigation and page interaction
+        aliases: [browser, web page, 浏览器]
+        include: [browser_*]
+      - id: database
+        description: Database inspection and queries
+        aliases: [database, sql, 数据库]
+        include: [db_*, sql_*]
+    skillBindings:
+      - skill: browser-automation
+        groups: [browser]
+```
+
+## v0.1 migration
+
+Version 0.2 defaults to `stable-proxy`, so a prior configuration now returns
+exact definitions and uses `tool_dispatch` instead of exposing a native family.
+
+To retain v0.1 call semantics while taking the lifecycle fixes:
+
+```yaml
+- id: tokens-progressive-tools
+  config:
+    mode: dynamic
+```
+
+Dynamic mode now affects the first request and makes a successful search
+visible on the immediately following request, but it remains intentionally
+cache-hostile when the active family set changes.

package/docs/progressive-disclosure.md

@@ -0,0 +1,82 @@
+# Progressive disclosure model
+
+This plugin separates two related mechanisms that are often conflated:
+
+- Agent Skills progressively load instructions and resources.
+- Tool discovery progressively loads callable schemas.
+
+DeepSeek Harness already owns the first mechanism. This plugin supplies a
+cache-stable fallback for the second where the provider protocol has no native
+deferred-tool content blocks.
+
+## Skills layers
+
+The open Agent Skills format defines three disclosure levels:
+
+| Level | Session-visible material | Load boundary |
+| --- | --- | --- |
+| Metadata | `name` and `description` | Skill catalog publication |
+| Instructions | Complete `SKILL.md` body | `skill` activation |
+| Resources | Referenced scripts, files, and assets | Explicit need |
+
+DSH's Skills subsystem already publishes a compact name/description catalog,
+loads the complete body through the `skill` tool, and resolves resources only
+when needed. The plugin does not duplicate or replace that subsystem.
+
+References:
+
+- [Agent Skills specification](https://github.com/agentskills/agentskills/blob/main/docs/specification.mdx)
+- [DSH Skills subsystem](https://deepseek-harness.github.io/deepseek-harness/reference/subsystems/skills)
+
+## Tool layers
+
+Stable proxy mode implements an analogous three-level tool path:
+
+| Level | Session-visible material | Load boundary |
+| --- | --- | --- |
+| Discovery entry | `tool_search`, `tool_dispatch`, common direct tools | First request |
+| Exact definitions | Matching names, descriptions, and parameter schemas | `tool_search` result |
+| Execution | Original tool body and result | `tool_dispatch` nested call |
+
+The complete catalog and real executors stay in process memory. Only exact
+matches enter conversation history, but each match also names every sibling in
+its family, so one search opens a plugin's whole dispatchable surface. The
+`status` action browses the complete family catalog when no lexical query
+fits; by default it informs without unlocking dispatch.
+
+## Native deferred tools versus stable proxy
+
+A provider-native deferred-tool protocol can receive all definitions out of
+band, initially render only non-deferred tools, and later return typed tool
+references without changing its cache prefix. DSH's current generic
+`ToolSchema` contains only `name`, `description`, and `parameters`; the DeepSeek
+Chat Completions tool protocol also has no equivalent deferred-reference block.
+
+Stable proxy therefore keeps two generic schemas fixed and performs real-tool
+validation at dispatch time. This preserves prefix stability and ordinary DSH
+policy, with two explicit trade-offs:
+
+- the outer request has no provider-native grammar for a deferred tool;
+- discovery and execution are separate calls.
+
+When DSH adds provider capability negotiation and native deferred references,
+a future native mode can use those features without changing the stable proxy
+fallback.
+
+References:
+
+- [DSH tool subsystem](https://deepseek-harness.github.io/deepseek-harness/reference/subsystems/tools)
+- [DeepSeek Chat Completions](https://api-docs.deepseek.com/api/create-chat-completion/)
+
+## Cache contract
+
+The stable request prefix is:
+
+```text
+fixed tool schemas → fixed system sections → append-only message history
+```
+
+Discovery adds one tool result after the reusable prefix. It does not change
+the fixed schemas or the generated Code Mode SDK. This is the central
+difference from `dynamic` mode, whose native family activation intentionally
+changes the request header.

package/lib/catalog.d.ts

@@ -0,0 +1,12 @@
+import type { DeferredToolMatch, SearchMatch, ToolCatalog, ToolGroupConfig, ToolSchemaView } from './types.js';
+export declare function matchesToolName(name: string, patterns: readonly string[]): boolean;
+export declare function estimateSchemaTokens(schema: ToolSchemaView, charactersPerToken: number): number;
+export declare function buildCatalog(schemas: readonly ToolSchemaView[], configuredGroups: readonly ToolGroupConfig[], charactersPerToken: number, excludedNames?: ReadonlySet<string>): ToolCatalog;
+export declare function searchCatalog(catalog: ToolCatalog, query: string, limit: number): SearchMatch[];
+/**
+ * Rank individual tools with exact-name bonuses and a compact BM25-style score.
+ * Definitions, parameter descriptions, enums, and nested property names all
+ * participate in the searchable document.
+ */
+export declare function searchTools(catalog: ToolCatalog, query: string, limit: number): DeferredToolMatch[];
+export declare function matchingToolNames(catalog: ToolCatalog, patterns: readonly string[]): string[];