@theokit/sdk 2.15.1 → 2.18.0

package/CHANGELOG.md

@@ -1,5 +1,76 @@
 # Changelog
 
+## 2.18.0
+
+### Minor Changes
+
+- 3eca862: Resume is no longer lossy (#62). (1) **Session hydration** used to filter the rebuilt context to `user`/`assistant` only, silently dropping `tool_call`/`tool_result` turns — a resumed agent forgot every tool it had run. Tool turns are now folded into the hydrated context (as assistant-role context) so the tool history survives resume; legacy user/assistant JSONL loads unchanged. (Exact tool_use/tool_result LLM-block reconstruction for mid-call resume needs persisted tool-use ids — a schema change deferred.) (2) **Workflow resume** used to continue from the suspend point with only the resume payload, so earlier step outputs were lost; the resumed run now restores the snapshot's accumulated step outputs, so a post-suspend step can see prior results. (3) **Scoped session state:** `scopedConversationId(scope, id)` namespaces a conversation id by `app:`/`user:`/`temp:` scope (path-safe `__` separator), and `ConversationStorageAdapter.deleteScope(prefix)` prunes a whole scope (e.g. `temp` on logout) in one call — additive.
+- 4af68fc: Observability is now trustworthy (#64). (1) **Nested spans:** `startChildSpan` used to discard its parent and start a flat sibling, so a trace backend could not reconstruct the causal tree; it now links the child to the parent via an explicit OTel parent context (`llm.call` / `tool.call` nest under `agent.send`). (2) **EventBus fails loud:** a throwing subscriber used to vanish into an empty `catch {}`; `publish` now logs the error (event key + message) to stderr and increments an observable `handlerErrorCount`, while preserving the EC-2 contract that sibling handlers still fire. (3) **Metrics:** tool-call / LLM-call durations + LLM token throughput — previously measured but only attached as span attributes — are now emitted as dedicated metrics via the existing `recordHistogram` path.
+- b4cc298: A silent LLM token undercount is now observable (#66). When a provider omits `usage` on a finish, the SDK used to coerce the counts to `0` (`?? 0`), so budget consumption was under-reported without any signal. The loop now distinguishes "provider omitted usage" from "0 tokens used": it emits a `theokit_llm_usage_missing` metric and a stderr WARN so the gap is visible, instead of silently zeroing. Normal finishes emit the token throughput as a `theokit_llm_tokens` metric. No new dependency (no local tokenizer — the fix makes the silent gap loud + measurable rather than estimating). Also documents the artifacts scope decision (cloud-only/pre-release; a local ArtifactService is deferred) in docs.md.
+- 08539f0: Fix a cross-model semantic-cache false hit and add session revert (#67). (1) **Model-scoped cache:** the semantic-search path filtered eligible entries by embedder + namespace + dim + expiry but NOT `modelId`, so two models sharing an embedder could return each other's cached response; `semanticSearch` / `isEligibleForSearch` now require `modelId` equality (the composite KV key already included it). (2) **Session revert:** `ConversationStorageAdapter.truncateConversation(id, keepCount)` reverts a transcript back to its first `keepCount` messages ("undo the last turn(s)"), rewriting the JSONL atomically under the same cross-process lock as append/compaction; the FS + in-memory adapters implement it. `keepCount <= 0` empties, `keepCount >= length` is a no-op.
+
+### Patch Changes
+
+- 997ae59: An explicitly-passed `apiKey` now selects its provider, unblocking the Skills↔Harness seam (M4). A run created with `Agent.create({ apiKey: "sk-or-…", model: { id: "openai/gpt-4o-mini" } })` used to fail with a swallowed `ConfigurationError: No provider client could be resolved (primary=openai)` — the router inferred the provider from the model's `openai/` prefix and ignored the OpenRouter key, so the LLM was never called and the run ended `status: "error"` with zero stream events. Two fixes: (1) **provider selection** now consults the API key prefix (`sk-or-` → OpenRouter, `sk-ant-` → Anthropic) ABOVE the model-prefix inference — the key is the ground-truth credential of which endpoint is called — while an explicit `providers.routes[0].provider` still wins; an aggregator's model slug (`openai/gpt-4o-mini` under OpenRouter) is passed through unstripped. (2) the single `AgentOptions.apiKey` is now **threaded into the provider pool** for the resolved provider, so an explicitly-passed credential is used even when the matching env var is unset (an existing `providers.apiKeys` pool still wins; fixture / `local` sentinels are never threaded). The SDK's own `openrouter-stream` / `openrouter-tools` / `openrouter-structured` real-LLM tests now pass end-to-end. All stdlib — no new dependency.
+
+## 2.17.0
+
+### Minor Changes
+
+- beb1e9a: The stdio MCP client now reconnects after a transport drop (#59, completing the M0 timeout work). A server child that exits or closes mid-session used to leave pending requests hung forever (a second permanent-hang vector distinct from the request timeout). Now an unexpected exit of the active child rejects every pending request with a typed `NetworkError` (`code: "mcp_disconnected"`) and marks the client dropped; the next request re-spawns the server and re-runs the `initialize` handshake with a bounded full-jitter backoff (2 attempts) before failing with `mcp_disconnected`. A deliberate `close()` is not treated as a drop (no reconnect). The http transport is stateless — each request opens a fresh connection, so it reconnects inherently on the next call; its error-surfacing contract is unchanged. Elicitation, server notifications, and adopting the upstream MCP SDK remain out of scope (documented boundary). No new dependency.
+- 3765aed: The credential pool now backs off before retrying a rate-limited key and trips a circuit breaker when a provider is down (#60). On the first 429 the pool used to re-hit the same key immediately (a `continue` with no sleep), burning every retry in under a millisecond under a shared-quota storm; it now sleeps a full-jitter backoff (`computeBackoffMs` + `sleepWithAbort`, already in-tree, now wired) before the same-key retry. A consecutive-failure circuit breaker (relocated to a neutral `internal/resilience/` module and shared with Active Memory) guards each provider: after N consecutive whole-attempt failures the pool fails fast with a typed `NetworkError` (`code: "circuit_open"`) until a cooldown elapses, instead of re-running the whole select→retry→rotate dance against a provider that is down. All stdlib — no new dependency. Existing name-only behavior and the provider's `Retry-After` cooldown on the rotate path are unchanged.
+- a1d0f3d: Harden the streaming path against stalls, truncation, and malformed tool-call JSON (#61).
+
+  - **Idle timeout:** every SSE `reader.read()` is now raced against an idle timer (default 60s, `parseSseStream(body, signal, idleTimeoutMs)`; pass `0` to disable). An upstream that handshakes then goes silent no longer hangs the agent loop forever — it rejects a typed `NetworkError` (`code: "stream_idle_timeout"`) and the body socket is cancelled. "Idle" means _no bytes at all_ within the window, so a slow-but-alive stream is unaffected.
+  - **Truncation detection:** an OpenAI-compatible stream that ends with NEITHER a `finish_reason` NOR a `[DONE]` sentinel was truncated (dropped connection / proxy hiccup). It now throws a typed `NetworkError` (`code: "stream_truncated"`) instead of silently committing the partial turn as a clean `end_turn`, so retry/fallback can route it.
+  - **Tool-call JSON repair:** `parseToolArguments` now attempts `jsonrepair` (already an in-tree dependency) before the `{ raw }` fallback, so a slightly-malformed native tool call (trailing comma, unquoted key — the Kimi/K2 class) parses instead of bouncing to the model as an `invalid_request` round-trip. Genuinely unrepairable input still lands in `{ raw }`.
+
+  All stdlib + an existing dependency — no new runtime dependency.
+
+- daa71de: Conversation persistence is now batched, cross-process-safe, and paginated (#63).
+
+  - **Batch turn append:** `ConversationStorageAdapter.appendMessages(id, messages[])` writes a whole turn (user + assistant + N tool results) in ONE atomic write instead of N separate `mkdir` + `appendFile` cycles. It is a public adapter capability for consumers that persist a turn or bulk-import history; the SDK's own runtime still appends incrementally (one message per loop event), but every such append now funnels through the same lock-guarded write path below, so the cross-process hardening is live regardless. The single `appendMessage` delegates to the batch of one.
+  - **Cross-process atomicity:** FS append and compaction now hold the same `proper-lockfile` cross-process lock (falls back to an in-process mutex when `proper-lockfile` is absent). Two Node processes sharing a cwd (CLI + daemon, parallel workers) can no longer tear a >4KB JSONL line or drop a line in the compaction read→rename window.
+  - **Pagination:** `getMessages(id, { offset, limit })` returns a bounded window so a caller hydrating a long history need not materialize the whole log (omit `opts` for the previous full read — backward-compatible). The in-memory adapter reads a true bounded slice; the FS/JSONL adapter bounds the materialized result (a future SQLite backend would bound the read itself).
+
+  No new dependency (`proper-lockfile` was already declared).
+
+## 2.16.0
+
+### Minor Changes
+
+- f93bb9a: `PermissionEngine` now gates on tool **arguments**, not just the tool name, and defaults **fail-closed** (#55). A `PermissionRule` may declare `args?: Record<string, string | RegExp | (value) => boolean>`; `evaluate(toolName, args?)` matches a rule only when the tool name matches AND every declared argument predicate matches the corresponding call argument — so a single `shell` rule can deny `rm -rf` while letting `ls` fall through. A missing/undefined argument fails its predicate (the rule does not match; it never throws). Name-only rules are unchanged. `createPermissionPlugin(engine)` now forwards the tool arguments into `evaluate`, so argument-level gating works through the `pre_tool_call` flow automatically.
+
+  **BREAKING (behavior):** the action returned when NO rule matches is now `"ask"` (fail-closed), changed from the previous `"allow"` (fail-open). A permission engine that cannot positively allow must not silently allow. If you relied on the fail-open default, restore it explicitly with `new PermissionEngine(rules, { defaultAction: "allow" })`.
+
+- 16e24a3: Add an opt-in tool-result content guard against prompt injection and PII leakage (#57). Tool results are untrusted input to the model; the new guard runs at the `transform_tool_result` seam before results reach the LLM. Enable it per send via `SendOptions.toolResultGuard`: `{ delimit: true }` frames tool output in explicit `<untrusted-tool-output>` data boundaries ("spotlighting") so the model treats it as data rather than instructions — a forged closing boundary inside the content is neutralized so it cannot break out of the frame; `{ redactPii: true }` replaces email/phone PII with `[REDACTED]`. Both are opt-in and non-breaking (undefined = unchanged behavior). The `defineTool` / `CustomTool` handler type is also widened to accept the optional `ToolContext` 2nd argument (completing the #65 wiring): single-argument handlers are unaffected.
+- c004a3b: Cancellation now actually interrupts in-flight work, tools get a per-call timeout, and the job queue is bounded (#58).
+
+  - `JobQueue` runs each job under an `AbortController` whose signal is passed to the job fn, so `cancel(id)` interrupts a cooperative running job instead of only flipping a status flag. A new `maxConcurrency` option bounds how many jobs run at once (omit for the previous unbounded behavior; values < 1 clamp to 1). The job fn signature is now `(signal: AbortSignal) => Promise<T>` — existing `() => Promise<T>` callers are unaffected (the signal is simply ignored).
+  - Tool dispatch threads the run's `AbortSignal` into each tool handler and bounds each tool call with an optional per-tool timeout (`SendOptions.perToolTimeoutMs`) (via `AbortSignal.any([runSignal, AbortSignal.timeout(ms)])`), so cancelling a run interrupts a running tool and a hung tool rejects a typed timeout instead of wedging the loop; the loop also checks for cancellation between iterations. All stdlib (Node ≥22.12) — no new dependency.
+
+- 01b4edd: Wire the 7 previously-dead plugin hooks and add a `ToolContext` to tool handlers (#65). `HookName` declared 10 hooks but only 3 (`pre_tool_call`, `pre_user_send`, `post_assistant_reply`) were ever invoked — a plugin registering `post_tool_call`, `pre_llm_call`, `post_llm_call`, `on_session_start`, `on_session_end`, `transform_tool_result`, or `transform_llm_output` got a silent no-op. All 7 now fire at their real site in the agent loop: `on_session_start`/`on_session_end` at run start/end (even on error), `pre_llm_call`/`post_llm_call` around each LLM turn, `post_tool_call` after each tool completes, and the two `transform_*` hooks fold over their payload (a handler's return value replaces it) before it reaches the LLM — `transform_tool_result` is the seam for tool-result content defense. Per-handler errors are logged, never thrown (a throwing transform keeps the prior payload). Additionally, tool handlers defined via `defineTool` now receive an optional 2nd `ToolContext` argument carrying the run's `AbortSignal`, so a cooperative handler can stop early on cancellation; existing single-argument handlers are unaffected.
+
+## 2.15.3
+
+### Patch Changes
+
+- 5412d7a: Stop leaking host secrets into child processes and stop overclaiming sandbox isolation (#54). Every subprocess the SDK spawned (hook scripts via the hooks executor, the shell tool, and `LocalSandbox`) inherited the FULL `process.env`, so API keys, tokens and passwords were exposed to executed commands. A new stdlib env-policy helper (`resolveChildEnv`, modeled on codex's `ShellEnvironmentPolicy`) now scrubs secret-like variable names (`*KEY*`, `*SECRET*`, `*TOKEN*`, `*PASSWORD*`, `*_AUTH*`) from the child environment by default (`inherit-scrubbed`). Non-secret vars (including `PATH`/`HOME`) are preserved, and an explicit `env` override always wins, so this is non-breaking for legitimate use. Opt out with policy `"all"` or tighten with `"core"` via `SandboxConfig.env`. `LocalSandbox`'s documentation is corrected to state plainly that it provides NO OS/filesystem/network isolation — only a timeout, an output cap, and env scrubbing.
+
+  The scrub is also applied to the **MCP stdio server subprocess** (via a new `envPolicy` field on the stdio server config), which previously inherited the full `process.env` — the highest-risk path, since MCP servers are often third-party binaries launched via `npx`. The secret-name denylist covers `*KEY*`, `*SECRET*`, `*TOKEN*`, `*PASSWORD*`, `*PASSWD*`, `*PASSPHRASE*`, `*[_-]PWD*`, `*CREDENTIAL*` (incl. `GOOGLE_APPLICATION_CREDENTIALS`), `*PRIVATE*`, `*_AUTH*`; for untrusted children use policy `"core"` (allowlist), the only fail-closed model (a denylist cannot catch creds embedded in a value such as `DATABASE_URL`). **Behavior change:** hook scripts and the shell tool no longer inherit secret-named host vars by default — pass them explicitly via the tool's `env`, or use policy `"all"`.
+
+- 58f440d: Fix a cross-tenant active-recall cache leak (#56). Active Memory recall results were cached keyed only by `(queryMode, userText)`, so two callers in the same process issuing the same query text — but belonging to different tenants (namespace / userId / scope) — could receive each other's recall results. The cache key infrastructure already supported a tenant tuple; `runActiveMemory` now threads `{namespace, userId, scope}` into both `cache.get` and `cache.set`, so recall entries are isolated per tenant. Same-tenant cache hits are preserved (no over-keying). No public API change.
+- e4cc6e9: Bound every MCP request with a timeout so a non-responding server can no longer hang the agent loop (#59). The stdio transport's `request` returned a Promise that never resolved when the server read the request but never replied; the http transport's `fetch` had no timeout. Both now enforce a per-request `requestTimeoutMs` (default 30000, configurable per server): stdio races the pending request against a timer that rejects a typed `NetworkError` (`code: "mcp_timeout"`) and drops the pending map entry (a late reply after timeout is a no-op — never a double-settle); http passes `AbortSignal.timeout` and maps an abort to the same typed error while surfacing any other fetch failure unchanged. `close()` now also settles any in-flight requests (`code: "mcp_closed"`) instead of leaking their timers. A timed-out stdio server is torn down (SIGKILL) so it cannot linger as a zombie, and the stdout read buffer is capped (8 MB) so a hostile/broken server flooding stdout without a newline (`code: "mcp_buffer_overflow"`) cannot pin memory.
+- 98ac0d0: Fix a live security defect: the ACP `pre_tool_call` permission veto was never enforced (#68). `installPermissionPlugin` tried to register its veto hook via `pluginManager.register(...)`, but `PluginManager` exposed no `register()` method — only a single-shot `initialize()` that throws when called twice. The call fell through to `void mgr.initialize([plugin])`, whose "called twice" rejection was swallowed, so the permission hook was never aggregated and guarded tools ran **without** the permission check even under `permissionMode: "deny"`/`"ask"`.
+
+  `PluginManager` now exposes `register(plugin)` — a post-init, `general`-only registration that REPLACES a same-named plugin's hooks (idempotent for the per-prompt ACP re-install) instead of appending duplicates. Additionally, `installPermissionPlugin` is now **fail-closed**: when the runtime has no plugin manager (e.g. a CloudAgent) and the mode is `deny`/`ask`, it throws a `ConfigurationError` (`code: "permission_enforcement_unavailable"`) and the ACP prompt is refused — rather than letting tools run ungated while the operator believes they are gated. It is also now `async` and awaits registration, so the veto hook is guaranteed aggregated before the first tool dispatch (no fire-and-forget window).
+
+## 2.15.2
+
+### Patch Changes
+
+- 6336f81: Suppress the leaked-dialect tool-call from the visible stream (R7). When `extractToolCallsFromContent` is enabled and a model leaks a `<function=NAME>` tool call as assistant text, the OpenAI-compat streaming now HOLDS that text back at the stream boundary (a small suspicion-buffer FSM that reuses the request-scoped allowlist from R5) instead of emitting it as `text_delta` events — so the raw dialect no longer flashes by in the live stream or lands in the final assistant text. `finish()` still recovers the call (unchanged). Fail-open: a never-closing marker or un-suppressable input is flushed as visible text (never held forever). Flag-off streaming is byte-for-byte unchanged. Grounded in openclaw's stream-normalizer FSM.
+
 ## 2.15.1
 
 ### Patch Changes