@vellumai/assistant 0.8.10-dev.202606101324.2fc90b3 → 0.8.10-dev.202606101436.d73da44

package/docs/workspace-tools.md

@@ -0,0 +1,196 @@
+# Workspace Tool Overrides
+
+Workspace tool overrides let the operator replace a core assistant tool (or
+add a brand-new one) by dropping a single file under their
+`<workspaceDir>/tools/` directory. The override survives assistant restarts,
+takes effect during the same startup phase as core tool initialization,
+and is recoverable: removing the file restores the original core behavior.
+When the `workspace-tools-watcher` feature flag is enabled, overrides are
+also hot-reloaded by a filesystem watcher (no restart required after the
+initial boot).
+
+This page explains the file convention, lifecycle position, and the
+"single canonical source per name" invariant the design is built around.
+
+## File convention
+
+```
+<workspaceDir>/
+  tools/
+    skill_load.ts        ← name="skill_load" via .ts default export
+    my_tool.js           ← name="my_tool" via compiled .js
+    data_lookup.json     ← name="data_lookup" via JSON spec
+    host_bash.removed    ← name="host_bash" — strip from registry
+```
+
+Rules:
+
+- The **filename stem** is the registered tool name verbatim. There is no
+  derivation, no basename transformation, no directory layer.
+- The loader looks for `.js` first, then `.ts`, then `.json`. The
+  `.js`-preferred ordering matches compiled-binary semantics: a release
+  build that ships compiled artifacts next to source runs the compiled
+  file.
+- `.json` files load as declarative specs — they describe schema and
+  metadata but cannot supply an `execute` function. Calling a
+  JSON-sourced tool returns an `isError` result; they exist for stubs
+  and override placeholders.
+- A `<name>.removed` sentinel file (any contents, typically empty)
+  strips the same-named core tool from the registry without
+  substituting a replacement. Removing the sentinel file restores the
+  core tool on the next reconcile.
+- Filename stems must satisfy `isProviderSafeToolName`
+  (`/^[a-zA-Z0-9_-]{1,64}$/`). A stem with spaces or punctuation is
+  logged at error and skipped; the loader never silently rewrites the
+  name with a hash suffix that would be unfindable.
+- Stems starting with `.` are skipped (so `.gitignore`, `.DS_Store`
+  cannot accidentally register a tool).
+- If `<name>.ts` and `<name>.js` (or any other shadow combination)
+  coexist for the same stem, the winning extension is used and the
+  shadowed siblings are logged at warn.
+
+## Tool file shape
+
+The default export is a generic `ToolDefinition` — the same shape every
+tool surface in the assistant uses. All four normally-required fields
+fall back to documented defaults if omitted, so a misconfigured tool
+still loads cleanly and surfaces its problem at call time rather than
+blocking assistant boot.
+
+```ts
+// /workspace/tools/skill_load.ts
+import type {
+  ToolContext,
+  ToolDefinition,
+  ToolExecutionResult,
+} from "@vellumai/plugin-api";
+
+const tool: ToolDefinition = {
+  description: "Skill loader (custom workspace implementation)",
+  defaultRiskLevel: "low",
+  input_schema: {
+    type: "object",
+    properties: { skill_id: { type: "string" } },
+    required: ["skill_id"],
+  },
+  async execute(
+    input: { skill_id: string },
+    ctx: ToolContext,
+  ): Promise<ToolExecutionResult> {
+    return { content: `loaded ${input.skill_id}`, isError: false };
+  },
+};
+
+export default tool;
+```
+
+Defaults applied when fields are omitted:
+
+| Field              | Default                                                                                     |
+| ------------------ | ------------------------------------------------------------------------------------------- |
+| `description`      | `""`                                                                                        |
+| `defaultRiskLevel` | `"high"`                                                                                    |
+| `executionTarget`  | `"sandbox"`                                                                                 |
+| `input_schema`     | `{ type: "object", properties: {}, additionalProperties: false }`                           |
+| `execute`          | Returns `{ content: "workspace tool <name> has no execute implementation", isError: true }` |
+
+## Override semantics
+
+Workspace tools are the highest-priority origin in the tool registry:
+
+- **Same name as a core tool** → the original core tool is moved into an
+  internal stash (`getCoreToolOverride(name)`) and the workspace tool takes
+  its place. Removing the workspace file causes the watcher to restore
+  the original — workspace tools are not destructive to the core baseline.
+- **Brand-new name** → registers as a net-new entry. No stash.
+- **`<name>.removed` sentinel for a core tool** → the core tool is
+  stripped (stashed in the same map as override-style stashing) and no
+  replacement is registered. Deleting the sentinel restores the core
+  tool. Stripping a name that doesn't exist is a logged no-op.
+- **Collision with a skill / plugin / MCP tool of the same name** → the
+  workspace registration throws. Workspace tools must register before
+  other extension categories; seeing one of these owners during a
+  workspace registration means the assistant startup order regressed.
+
+Subsequent (post-workspace-load) registrations from plugins, skills, or
+MCP servers that try to claim a name a workspace tool already owns (or
+has stripped via `.removed`) are **warn-and-skipped**. Plugins, skills,
+and MCP servers cannot bypass this by registering before workspace
+tools load — the lifecycle pins workspace tools to register first.
+
+Skill / plugin / MCP collisions with a workspace tool name across the
+distinct namespacing those surfaces enforce are impossible — each runs
+in its own owner-scoped registry path, and the workspace registration's
+stamping is authoritative regardless of pre-existing ownership fields
+on the incoming tool.
+
+## Lifecycle position
+
+```
+initializeTools()             # core tools register
+  → loadWorkspaceTools()      # initial workspace tool scan
+    → MCP tool registration
+    → loadUserPlugins()
+    → bootstrapPlugins()
+
+# after providers-setup completes:
+DaemonServer.start()
+  → WorkspaceToolsWatcher.getInstance().start()   # hot register/unregister via fs.watch
+```
+
+Workspace tools register _after_ core tools and _before_ every other
+extension surface during the initial scan so that every subsequent
+registration sees the workspace tool as already-owned. The initial scan
+(`loadWorkspaceTools()`) always runs, so workspace tools load from disk
+at every boot regardless of the flag.
+
+The filesystem watcher is gated on the `workspace-tools-watcher` feature
+flag (default off). When enabled, it runs for the lifetime of the
+assistant, picking up add/change/delete events on `<workspaceDir>/tools/`
+and reconciling the registry without requiring a restart. When disabled,
+no watch loop is mounted and live edits to `<workspaceDir>/tools/` take
+effect only on the next daemon restart. The flag is read at startup, so
+toggling it takes effect on restart rather than mid-process.
+
+The watcher debounces per filename stem and reconciles by re-deriving
+the world from disk ("given what's on disk right now for `<stem>.*`,
+what registry state should the assistant be in?") rather than routing
+on `fs.watch`'s unreliable add/change/rename event types. This is the
+same eventual-consistency pattern the plugin source watcher uses.
+
+## Per-tool isolation
+
+A single broken workspace tool must never block assistant boot. The
+loader applies the same per-tool isolation contract as the user-plugin
+loader:
+
+- A `.ts` / `.js` whose module body throws during import is logged at
+  error and skipped — the loader proceeds to the next file.
+- A `.ts` / `.js` whose default export is missing or not an object is
+  logged at error and skipped.
+- A `.json` that doesn't parse, or doesn't decode to an object, is
+  logged at error and skipped.
+- A `.ts` / `.js` whose module evaluation hangs is bounded by a
+  10-second timeout. The abandoned import continues running in the
+  background but the loader has already moved on. Bun's `import()`
+  cannot be cancelled, so the registry guards against late-arriving
+  registrations by the time-bounded path.
+
+The registry's batch validation runs ahead of any mutation, so a single
+late-discovered conflict (e.g. duplicate name within the batch) aborts
+the whole call without partially populating the registry.
+
+## Unregistering
+
+Deleting `<workspaceDir>/tools/<name>.{ts,js,json}` triggers the file
+watcher's reconcile, which calls `unregisterWorkspaceTool(name)` and
+restores the stashed core tool when one exists, or simply deletes the
+entry when the workspace tool was net-new. No assistant restart is
+required.
+
+To strip a core tool without substituting a replacement, drop an empty
+`<workspaceDir>/tools/<name>.removed` file. Removing that sentinel
+restores the core tool on the next reconcile. The two states (override
+vs. strip) are mutually exclusive — placing both `<name>.ts` and
+`<name>.removed` for the same stem causes the watcher to tear down
+both states until the conflict is resolved on disk.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vellumai/assistant",
-  "version": "0.8.10-dev.202606101324.2fc90b3",
+  "version": "0.8.10-dev.202606101436.d73da44",
   "license": "MIT",
   "type": "module",
   "exports": {

package/src/__tests__/agent-loop.test.ts

@@ -6,6 +6,7 @@ import type {
   CheckpointInfo,
 } from "../agent/loop.js";
 import { AgentLoop } from "../agent/loop.js";
+import { REFUSAL_FALLBACK_TEXT } from "../plugins/defaults/empty-response/hooks/stop.js";
 import { resetPluginRegistryAndRegisterDefaults } from "../plugins/defaults/index.js";
 import type {
   ContentBlock,
@@ -2164,12 +2165,23 @@ describe("AgentLoop", () => {
     );
     expect(messageCompletes).toHaveLength(2);
 
-    // The last assistant message in history is the empty one
+    // An organic empty `end_turn` (not a refusal) is left as-is — the fallback
+    // is refusal-specific, so the exhausted empty turn stays empty.
     const lastAssistant = [...history]
       .reverse()
       .find((m) => m.role === "assistant");
     expect(lastAssistant).toBeDefined();
     expect(lastAssistant!.content).toEqual([]);
+
+    // AND no fallback text is streamed for a non-refusal empty turn.
+    const textDeltas = events.filter((e) => e.type === "text_delta");
+    expect(
+      textDeltas.some(
+        (e) =>
+          (e as { type: "text_delta"; text: string }).text ===
+          REFUSAL_FALLBACK_TEXT,
+      ),
+    ).toBe(false);
   });
 
   test("does not retry empty response on first turn (no prior tool use)", async () => {
@@ -2196,7 +2208,189 @@ describe("AgentLoop", () => {
 
     // Should NOT retry — this is the first turn with no tool use history
     expect(calls).toHaveLength(1);
-    expect(history).toHaveLength(2); // user + empty assistant
+    // user + assistant; an organic empty `end_turn` is left as-is (the fallback
+    // is refusal-specific), so the empty turn stays empty.
+    expect(history).toHaveLength(2);
+    expect(history[1].content).toEqual([]);
+  });
+
+  // Refusal stop boundary — the provider zeroed the response with
+  // `stopReason: "refusal"`. The default `stop` hook rewrites the empty turn
+  // into a user-facing fallback (no retry — a safety-classifier refusal
+  // re-fires on a re-query), and the loop persists + streams it rather than the
+  // empty assistant bubble the user would otherwise see.
+  test("rewrites a refusal into a user-facing fallback without retrying", async () => {
+    // GIVEN a provider that refuses (empty content, stopReason "refusal").
+    const refusalResponse: ProviderResponse = {
+      content: [],
+      model: "mock-model",
+      usage: { inputTokens: 10, outputTokens: 0 },
+      stopReason: "refusal",
+    };
+    const { provider, calls } = createMockProvider([
+      refusalResponse,
+      refusalResponse,
+    ]);
+    const loop = new AgentLoop({
+      provider: provider,
+      systemPrompt: "system",
+      conversationId: "test-conversation",
+    });
+
+    // WHEN the loop runs.
+    const events: AgentEvent[] = [];
+    const { history } = await loop.run({
+      messages: [userMessage],
+      onEvent: collectEvents(events),
+      trust: { sourceChannel: "vellum", trustClass: "unknown" },
+    });
+
+    // THEN the loop calls the provider once: the refusal is rewritten in place
+    // with no nudge-driven retry.
+    expect(calls).toHaveLength(1);
+
+    // AND the user-visible assistant turn is the fallback, not an empty bubble.
+    const lastAssistant = [...history]
+      .reverse()
+      .find((m) => m.role === "assistant");
+    expect(lastAssistant!.content).toEqual([
+      { type: "text", text: REFUSAL_FALLBACK_TEXT },
+    ]);
+
+    // AND the fallback is streamed so the client renders it.
+    const textDeltas = events.filter((e) => e.type === "text_delta");
+    expect(
+      textDeltas.some(
+        (e) =>
+          (e as { type: "text_delta"; text: string }).text ===
+          REFUSAL_FALLBACK_TEXT,
+      ),
+    ).toBe(true);
+  });
+
+  // The fallback must not pile a spurious apology beneath a real answer: when
+  // an earlier turn this run already produced visible text (text alongside a
+  // tool call), a refusal on the trailing turn after the tool result must not
+  // rewrite the turn even though a refusal would normally trigger it.
+  test("does not substitute a fallback when the run already produced visible text", async () => {
+    // GIVEN a first turn with text + a tool call, then a refusal after the
+    // tool result.
+    const textPlusToolUse: ProviderResponse = {
+      content: [
+        { type: "text", text: "Here is your answer." },
+        {
+          type: "tool_use",
+          id: "t1",
+          name: "read_file",
+          input: { path: "/a" },
+        },
+      ],
+      model: "mock-model",
+      usage: { inputTokens: 10, outputTokens: 5 },
+      stopReason: "tool_use",
+    };
+    const refusalResponse: ProviderResponse = {
+      content: [],
+      model: "mock-model",
+      usage: { inputTokens: 10, outputTokens: 0 },
+      stopReason: "refusal",
+    };
+    const { provider } = createMockProvider([textPlusToolUse, refusalResponse]);
+    const loop = new AgentLoop({
+      provider: provider,
+      systemPrompt: "system",
+      conversationId: "test-conversation",
+      tools: dummyTools,
+      toolExecutor: async () => ({ content: "data", isError: false }),
+    });
+
+    // WHEN the loop runs.
+    const events: AgentEvent[] = [];
+    const { history } = await loop.run({
+      messages: [userMessage],
+      onEvent: collectEvents(events),
+      trust: { sourceChannel: "vellum", trustClass: "unknown" },
+    });
+
+    // THEN no fallback text is emitted or persisted — the real answer stands
+    // alone and the trailing empty turn stays empty.
+    const fallbackEmitted = events.some(
+      (e) =>
+        e.type === "text_delta" &&
+        (e as { type: "text_delta"; text: string }).text ===
+          REFUSAL_FALLBACK_TEXT,
+    );
+    expect(fallbackEmitted).toBe(false);
+    const fallbackInHistory = history.some(
+      (m) =>
+        m.role === "assistant" &&
+        m.content.some(
+          (b) =>
+            b.type === "text" &&
+            "text" in b &&
+            (b as { text: string }).text === REFUSAL_FALLBACK_TEXT,
+        ),
+    );
+    expect(fallbackInHistory).toBe(false);
+  });
+
+  // A native web-search turn lands at the stop boundary with no `tool_use` and
+  // no visible text, but with `stopReason: "end_turn"` (not a refusal) and
+  // `server_tool_use`/`web_search_tool_result` blocks that render the search
+  // card. Because the fallback is refusal-specific, it must not fire here and
+  // the server-tool blocks must persist untouched.
+  test("does not substitute a fallback for a server-tool turn with no text", async () => {
+    // GIVEN a turn carrying only native web-search blocks (no text, no
+    // client-side tool_use).
+    const serverToolBlocks: ContentBlock[] = [
+      {
+        type: "server_tool_use",
+        id: "srv1",
+        name: "web_search",
+        input: { query: "weather" },
+      },
+      {
+        type: "web_search_tool_result",
+        tool_use_id: "srv1",
+        content: [{ title: "result" }],
+      },
+    ];
+    const webSearchResponse: ProviderResponse = {
+      content: serverToolBlocks,
+      model: "mock-model",
+      usage: { inputTokens: 10, outputTokens: 5 },
+      stopReason: "end_turn",
+    };
+    const { provider, calls } = createMockProvider([webSearchResponse]);
+    const loop = new AgentLoop({
+      provider: provider,
+      systemPrompt: "system",
+      conversationId: "test-conversation",
+    });
+
+    // WHEN the loop runs.
+    const events: AgentEvent[] = [];
+    const { history } = await loop.run({
+      messages: [userMessage],
+      onEvent: collectEvents(events),
+      trust: { sourceChannel: "vellum", trustClass: "unknown" },
+    });
+
+    // THEN the provider is called once (no nudge for a non-refusal turn) and
+    // the server-tool blocks are persisted untouched, with no fallback added.
+    expect(calls).toHaveLength(1);
+    const lastAssistant = [...history]
+      .reverse()
+      .find((m) => m.role === "assistant");
+    expect(lastAssistant!.content).toEqual(serverToolBlocks);
+
+    const fallbackEmitted = events.some(
+      (e) =>
+        e.type === "text_delta" &&
+        (e as { type: "text_delta"; text: string }).text ===
+          REFUSAL_FALLBACK_TEXT,
+    );
+    expect(fallbackEmitted).toBe(false);
   });
 
   // PR 6: callSite threading from AgentLoop.run() into provider config.

package/src/__tests__/empty-response-hook.test.ts

@@ -2,11 +2,12 @@
  * Tests for the default `empty-response` plugin's `stop` hook.
  *
  * Covers:
- * - The default hook's binary decision for the canonical cases: empty turn
- *   after a prior tool-use turn → continue (with the canonical nudge text);
- *   visible text → stop; prior turn already delivered visible text → stop;
- *   first model call with no prior turn → stop; provider refusal → continue
- *   (with the refusal-specific nudge text); refusal-but-recovered → stop.
+ * - The default hook's decision for the canonical cases: empty turn after a
+ *   prior tool-use turn → continue (with the canonical nudge text); visible
+ *   text → stop; prior turn already delivered visible text → stop; first model
+ *   call with no prior turn → stop; provider refusal → stop with the turn
+ *   rewritten to the user-facing fallback; refusal-but-recovered → stop;
+ *   refusal after a visible reply this run → stop with no rewrite.
  * - The hook scopes its prior-turn signals to the current response cycle (the
  *   messages after the last genuine user prompt), so visible text from the
  *   inbound conversation does not suppress the nudge.
@@ -28,7 +29,7 @@ import { HOOKS } from "../plugin-api/constants.js";
 import type { PluginLogger, StopContext } from "../plugin-api/types.js";
 import {
   NUDGE_TEXT,
-  REFUSAL_NUDGE_TEXT,
+  REFUSAL_FALLBACK_TEXT,
 } from "../plugins/defaults/empty-response/hooks/stop.js";
 import stop from "../plugins/defaults/empty-response/hooks/stop.js";
 import { defaultEmptyResponsePlugin } from "../plugins/defaults/index.js";
@@ -172,7 +173,7 @@ describe("empty-response stop hook — default decisions", () => {
 
   // ─── Refusal stop ──────────────────────────────────────────────────────────
 
-  test("refusal on the first call with no content → continue with refusal nudge", async () => {
+  test("refusal on the first call with no content → stop with rewritten fallback", async () => {
     // GIVEN the canonical failure mode: the provider's safety classifier zeros
     // the response on the very first model call, returning `stopReason:
     // "refusal"` and no visible text.
@@ -185,15 +186,16 @@ describe("empty-response stop hook — default decisions", () => {
     // WHEN the hook runs.
     await stop(ctx);
 
-    // THEN it continues with the refusal-specific nudge, even with no prior turn.
-    expect(ctx.decision).toBe("continue");
-    expect(ctx.messages.at(-1)).toEqual({
-      role: "user",
-      content: [{ type: "text", text: REFUSAL_NUDGE_TEXT }],
-    });
+    // THEN it lets the turn end and rewrites the turn content to the
+    // user-facing fallback, without nudging the model.
+    expect(ctx.decision).toBe("stop");
+    expect(ctx.responseContent).toEqual([
+      { type: "text", text: REFUSAL_FALLBACK_TEXT },
+    ]);
+    expect(ctx.messages).toEqual([]);
   });
 
-  test("refusal with a thinking-only block still continues", async () => {
+  test("refusal with a thinking-only block is still rewritten", async () => {
     // GIVEN a refusal whose only content is a thinking block — the user sees
     // nothing.
     const ctx = makeCtx({
@@ -207,15 +209,14 @@ describe("empty-response stop hook — default decisions", () => {
     // WHEN the hook runs.
     await stop(ctx);
 
-    // THEN it continues with the refusal-specific nudge.
-    expect(ctx.decision).toBe("continue");
-    expect(ctx.messages.at(-1)).toEqual({
-      role: "user",
-      content: [{ type: "text", text: REFUSAL_NUDGE_TEXT }],
-    });
+    // THEN it stops and rewrites the turn content to the fallback.
+    expect(ctx.decision).toBe("stop");
+    expect(ctx.responseContent).toEqual([
+      { type: "text", text: REFUSAL_FALLBACK_TEXT },
+    ]);
   });
 
-  test("refusal but visible text present → stop (model recovered)", async () => {
+  test("refusal but visible text present → stop, no rewrite (model recovered)", async () => {
     // GIVEN a refusal that still delivered some visible text before refusing —
     // the user has something to see.
     const ctx = makeCtx({
@@ -226,13 +227,35 @@ describe("empty-response stop hook — default decisions", () => {
     // WHEN the hook runs.
     await stop(ctx);
 
-    // THEN the decision stays at stop.
+    // THEN the decision stays at stop and the turn content is left untouched.
     expect(ctx.decision).toBe("stop");
+    expect(ctx.responseContent).toEqual([
+      { type: "text", text: "partial answer" },
+    ]);
   });
 
-  test("refusal beats the post-tool-empty nudge wording", async () => {
+  test("refusal after a visible reply this run → stop, no rewrite", async () => {
+    // GIVEN an earlier turn this run already delivered a real answer (text
+    // alongside a tool call), then the trailing turn refuses after the tool
+    // result. Rewriting here would stack an apology beneath the real answer.
+    const ctx = makeCtx({
+      messages: [userPrompt("do X"), priorVisibleTextTurn, priorToolUseTurn],
+      responseContent: [],
+      stopReason: "refusal",
+    });
+
+    // WHEN the hook runs.
+    await stop(ctx);
+
+    // THEN the refusal is left as-is — no fallback is stacked beneath the
+    // earlier reply.
+    expect(ctx.decision).toBe("stop");
+    expect(ctx.responseContent).toEqual([]);
+  });
+
+  test("refusal beats the post-tool-empty nudge", async () => {
     // GIVEN conditions that would trip both the refusal branch and the
-    // post-tool-empty branch.
+    // post-tool-empty branch (a prior tool-use turn with no visible text).
     const ctx = makeCtx({
       messages: [priorToolUseTurn],
       responseContent: [],
@@ -242,12 +265,13 @@ describe("empty-response stop hook — default decisions", () => {
     // WHEN the hook runs.
     await stop(ctx);
 
-    // THEN refusal wins because its wording is more accurate.
-    expect(ctx.decision).toBe("continue");
-    expect(ctx.messages.at(-1)).toEqual({
-      role: "user",
-      content: [{ type: "text", text: REFUSAL_NUDGE_TEXT }],
-    });
+    // THEN refusal wins: the turn content is rewritten to the fallback and the
+    // model is not nudged.
+    expect(ctx.decision).toBe("stop");
+    expect(ctx.responseContent).toEqual([
+      { type: "text", text: REFUSAL_FALLBACK_TEXT },
+    ]);
+    expect(ctx.messages).toEqual([priorToolUseTurn]);
   });
 });