dsh-py-codeact 0.2.0 → 0.2.2

package/README.md

@@ -58,7 +58,33 @@ Docstring: Read a file from the workspace. Results include line numbers…
 
 That is why the prompt block carries signatures only — the descriptions are one `?` away instead of resident in every request. Annotations are rendered host-side with dsh's own exported `jsonSchemaToPy`, so there is no second JSON-Schema mapper to drift.
 
-The **return** type comes from the tool's own `output` schema, by way of `ctx.tools.sdkSchemas(scope)` — the projection that carries it. It is the one annotation the model cannot recover by reading harder: a wrong argument fails loudly at the call, while an unknown return shape is only discoverable by calling once and printing the result, which costs a whole turn per tool. A tool that declares no output schema still renders `Any`; claiming a type nobody declared would be worse than admitting ignorance. The binding set is resent with every cell, because restrictions and mid-conversation tool changes can move a tool in or out between calls.
+The **return** type comes from the tool's own `output` schema, by way of `ctx.tools.sdkSchemas(scope)` — the projection that carries it. It is the one annotation the model cannot recover by reading harder: a wrong argument fails loudly at the call, while an unknown return shape is only discoverable by calling once and printing the result, which costs a whole turn per tool. A tool that declares no output schema still renders `Any`; claiming a type nobody declared would be worse than admitting ignorance.
+
+Each object in that schema whose keys — and whose own generated class name — Python can take is declared as a named `TypedDict` above the signatures. For a bridged MCP tool the schema described here is the payload, not the transport wrapper — see below; `dict[str, Any]` would say a dict arrives without saying which keys, which is the one thing a return annotation exists to say.
+
+```python
+class McpCalendarListEventsOutput(TypedDict):
+    result: str
+
+async def mcp__calendar__list_events(*, calendar_id: str) -> McpCalendarListEventsOutput: ...
+```
+
+`jsonSchemaToPy` cannot do this and says so — it is context-free, and naming a `TypedDict` needs the render context `renderToolsSdkPy` supplies. That renderer is not reusable here: it emits a whole document in Code Mode's own `tools.name(args)` contract. So only the object and array branches are handled locally, every leaf still going through `jsonSchemaToPy` — a place to hang the names, not a second JSON-Schema mapper. The binding set is resent with every cell, because restrictions and mid-conversation tool changes can move a tool in or out between calls.
+
+### The MCP wrapper does not reach the cell
+
+dsh's MCP client resolves a call to `{ content, structuredContent? }` — the protocol's block array plus the server's own payload — and declares exactly that as the tool's output. That wrapper is transport, not API: `content`'s text duplicates the payload, and an image inside it is re-attached to the conversation separately, so a cell can do nothing with it. Handed the wrapper, a model writes `r["structuredContent"]["result"]` — and spends a call discovering it has to.
+
+So the bridge unwraps, and the signature describes what the cell actually receives:
+
+```python
+async def mcp__review__search(*, q: str) -> McpReviewSearchOutput: ...   # the payload, not the wrapper
+async def mcp__email__ping() -> str: ...                                 # no declared payload: the text blocks, joined
+```
+
+A server that declares no output schema is the common case in the wild, and its result really is just text — one text block, in every one of the 4142 MCP results measured across six trials, which is why this is a `str` and not a `list[str]` that would cost an `r[0]` at every call site to preserve a boundary that never appears. A result carrying no text block at all resolves to the empty string: an image there is re-attached to the conversation and read on the next step, so there was never anything for the cell to receive. One divergence is possible and belongs to the server: a tool that declares an output schema but omits `structuredContent` on some call resolves to that call's text, where the signature promised the payload type.
+
+Unwrapping keys off the tool's declared output schema, not off the value that comes back: a tool whose own payload happens to be `{content: [...]}` is not wrapping anything, and replacing its value with joined text would change what that tool returns with nothing to say so. This deliberately differs from Code Mode, which hands the whole envelope to `tools.name(args)`.
 
 ### MCP tools work, with nothing special
 
@@ -161,7 +187,7 @@ A pure CPU loop (`while True: pass`) never reaches an await point. After `hardIn
 - **Native writes are not captured.** stdout/stderr are captured at the Python level, so `print` is captured but a subprocess writing to fd 1 is not. Use `subprocess.run(..., capture_output=True)`, or `%run`. (Anything that does reach fd 1/2 — including IPython's own colored traceback, deliberately routed there — is retained only for the crash message.)
 - **Scope is not optional.** The visible tool set comes from `ctx.tools.sdkSchemas(scope)` — the scope being the agent. Omitting it yields the *global* view, which in a preset composition holds only host-registered tools; the preset's own `read`/`bash`/`edit` live in the agent scope and vanish. The prompt section reads `assembly.scope`, and the kernel's name list is resent with every cell (restrictions and mid-conversation tool changes can move a tool in or out between calls).
 - **Pick a `toolName` nothing else answers to.** With an MCP IPython server also mounted, a model told to "use the python tool" reaches for `mcp__py__ipython_execute_code` — which has no `tools` binding — and then reports that your tool does not exist.
-- **MCP return types are shapeless.** An MCP tool's canonical value is the envelope (`content`, plus `structuredContent` when the server advertises an output schema), and `jsonSchemaToPy` collapses an object to `dict[str, Any]` — so the annotation says a dict arrives without saying which keys. dsh renders that shape as a `TypedDict` for Code Mode via `renderToolsSdkPy`, but that renderer emits a whole document in Code Mode's own `tools.name(args)` contract, not a line per signature. Since the envelope is uniform, a sentence of prose buys more here than a per-tool emitter that would duplicate dsh's mapper.
+- **A shape Python cannot name stays vague.** A field whose key is not a valid identifier degrades its own class back to `dict[str, Any]` rather than emitting a body that will not parse — the tool stays callable and only that one annotation goes quiet. The same guard covers the class name itself, which is not local damage: a tool named `123tool` would otherwise emit `class 123toolOutput`, a SyntaxError that takes the whole block with it. A schema with no declared properties still renders `Any`; claiming a type nobody declared would be worse.
 - **Cold start.** The PEP 723 environment is resolved on first use (`uv python find --script`). Warm afterwards; pass `python` to skip it.
 
   The interpreter is then spawned **directly**, never behind `uv run --script`. A wrapper stays in the process tree as the interpreter's parent: when it exits first, the interpreter is reparented to init, the handle the host holds reports an exit, and a perfectly live kernel looks dead — so the next cell respawns and the session's state vanishes with an `[the interpreter was restarted]` notice nothing actually caused. `alive` is likewise tracked from the exit event rather than read off `proc.killed`, which Node sets on any `kill()` call, including a signal the process survived.

package/lib/index.js

@@ -62,12 +62,38 @@ Your action space is Python. Each call runs one cell in a **persistent IPython s
 
 The available tools:`
 
+/**
+ * The wrapper dsh's MCP client puts around every result: `{ content, structuredContent? }`, with
+ * `content` the protocol's block array and `structuredContent` the server's own payload. Detected
+ * by shape rather than by an `mcp__` name, because the shape IS the declared contract — the client
+ * builds exactly this schema, `additionalProperties: false` and all.
+ *
+ * @returns the payload's schema when the server declares one, `null` when this is not an envelope,
+ *   and `undefined` for an envelope carrying only text.
+ */
+function mcpPayloadSchema(schema) {
+  if (schema?.type !== 'object' || schema.additionalProperties !== false) return null
+  const keys = Object.keys(schema.properties ?? {})
+  if (keys.length !== 2 || !keys.includes('content') || !keys.includes('structuredContent')) return null
+  if (schema.properties.content?.type !== 'array') return null
+  return (schema.required ?? []).includes('structuredContent') ? schema.properties.structuredContent : undefined
+}
+
+/** @internal exported for the suite. The same wrapper at run time. A value is only unwrapped when it carries nothing but the wrapper's own keys. */
+export function mcpPayload(value) {
+  if (value === null || typeof value !== 'object' || Array.isArray(value)) return undefined
+  const keys = Object.keys(value)
+  if (!keys.includes('content') || !Array.isArray(value.content)) return undefined
+  if (keys.some((k) => k !== 'content' && k !== 'structuredContent')) return undefined
+  return 'structuredContent' in value ? { value: value.structuredContent } : { value: contentText(value.content) }
+}
+
 /**
  * Project one tool schema onto the wire spec the kernel builds a binding from: dsh's description becomes the function's docstring, its parameters become a keyword-only signature. Annotations are rendered here with dsh's own `jsonSchemaToPy`, so the kernel needs no second JSON-Schema mapper.
  *
  * The return type comes from the tool's own `output` schema — the shape of the canonical value the call actually resolves to. It is the one annotation a model cannot recover by reading harder: a parameter it gets wrong fails loudly at the call, while an unknown return shape is only discoverable by calling once and printing the result, which is a whole extra turn per tool.
  */
-function toolSpec(schema) {
+function toolSpec(schema, declarations) {
   const parameters = schema.parameters ?? {}
   const properties = parameters.properties ?? {}
   const required = new Set(Array.isArray(parameters.required) ? parameters.required : [])
@@ -75,7 +101,10 @@ function toolSpec(schema) {
     name: schema.name,
     doc: schema.description,
     // `output` is absent only if the caller passed a `schemas()` projection; `sdkSchemas()` always carries it.
-    returns: schema.output === undefined ? 'Any' : jsonSchemaToPy(schema.output),
+    // An MCP tool is annotated by what the CELL receives, which is the payload the bridge unwraps
+    // to — never the transport wrapper. Describing the wrapper is accurate and useless: the model
+    // then hand-writes `r["structuredContent"]["result"]`, and calls once just to learn that.
+    returns: schema.output === undefined ? 'Any' : mcpReturn(schema, declarations),
     params: Object.entries(properties).map(([name, node]) => ({
       name,
       type: jsonSchemaToPy(node),
@@ -84,6 +113,33 @@ function toolSpec(schema) {
   }
 }
 
+/**
+ * Every visible tool's spec, plus the `TypedDict` bodies their return annotations name.
+ *
+ * Both consumers go through here so they cannot disagree: the prompt block declares the classes,
+ * and the kernel sends the same `returns` text on to `inspect.Signature`, so `read?` shows the
+ * name the block above it defines. It also keeps `toolSpec` off `.map`, where the callback's
+ * second argument is the INDEX — a number, so a default parameter never fires and the accumulator
+ * is silently a `0`.
+ */
+function toolSpecs(schemas) {
+  const declarations = new Map()
+  const sorted = [...schemas].sort(byName)
+  // Which tools DECLARE the wrapper. Unwrapping keys off this rather than the returned value:
+  // a tool whose own payload happens to be `{content: [...]}` is not wrapping anything, and
+  // replacing its value with the joined text would change what it returns with nothing to say so.
+  const envelopes = new Set(sorted.filter((schema) => mcpPayloadSchema(schema.output) !== null).map((schema) => schema.name))
+  return { specs: sorted.map((schema) => toolSpec(schema, declarations)), declarations, envelopes }
+}
+
+/** The annotation for one tool's output: an MCP payload if it is an envelope, else the schema as declared. */
+function mcpReturn(schema, declarations) {
+  const payload = mcpPayloadSchema(schema.output)
+  if (payload === null) return declareType(schema.output, `${pascal(schema.name)}Output`, declarations)
+  // An envelope with no declared payload resolves to the text blocks joined, so it really is a `str`.
+  return payload === undefined ? 'str' : declareType(payload, `${pascal(schema.name)}Output`, declarations)
+}
+
 /** Deterministic (lexicographic) order, so an unchanged tool set renders byte-identically. */
 const byName = (a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)
 
@@ -132,15 +188,55 @@ export function needsRestartNotice(session, toolName, plugin) {
 /** `keyword.kwlist + keyword.softkwlist` for the `requires-python = ">=3.12"` the kernel pins. */
 const PY_KEYWORDS = new Set(['False', 'None', 'True', '_', 'and', 'as', 'assert', 'async', 'await', 'break', 'case', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'match', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'type', 'while', 'with', 'yield'])
 
+/** A name this block can put in an `import`, a `def`, or a `TypedDict` field. */
+const isUsableName = (name) => /^[A-Za-z_][A-Za-z0-9_]*$/.test(name) && !PY_KEYWORDS.has(name)
+
+/** `mcp__calendar__list_events` -> `McpCalendarListEvents`, for naming that tool's declarations. */
+const pascal = (name) => name.split(/[_\-]+/).filter(Boolean).map((part) => part[0].toUpperCase() + part.slice(1)).join('')
+
+/** `dict[str, Any] | dict[str, Any]` is what a `oneOf` of two objects collapses to once both branches degrade to the same text. Rendering the duplicate says there is a choice to make where there is none. */
+const dedupeUnion = (expr) => [...new Set(expr.split(' | '))].join(' | ')
+
+/**
+ * Render one output schema, declaring a named `TypedDict` for every object shape it contains.
+ *
+ * `jsonSchemaToPy` alone cannot do this and says so: it is context-free, and "naming a `TypedDict` requires the render context that `renderToolsSdkPy` supplies" — with nowhere to put a class declaration it degrades every object to `dict[str, Any]`. That erases the one thing an MCP return needs to say. dsh's MCP client resolves a call to the ENVELOPE, `{ content, structuredContent }`, and declares exactly that as the tool's `output`; flattened to `dict[str, Any]` the model is not told the envelope exists, so it reaches for the payload directly, gets nothing, and spends a turn printing the result to find the wrapper — the very cost the return annotation is here to remove.
+ *
+ * Only the object and array branches are handled here; every leaf still goes through `jsonSchemaToPy`, so there is no second JSON-Schema mapper, only a place to hang the names. A field name Python cannot take degrades that one class back to `dict[str, Any]` rather than emitting a class body that will not parse.
+ */
+function declareType(schema, baseName, declarations) {
+  if (schema === null || typeof schema !== 'object') return jsonSchemaToPy(schema)
+  if (schema.type === 'array' && schema.items !== null && typeof schema.items === 'object') {
+    return `list[${declareType(schema.items, `${baseName}Item`, declarations)}]`
+  }
+  const properties = schema.type === 'object' ? (schema.properties ?? {}) : {}
+  const names = Object.keys(properties)
+  // `baseName` too, not just the fields: a tool name may start with a digit or carry a character
+  // `pascal` does not split on, and `123toolOutput` is a SyntaxError that takes the WHOLE block
+  // with it — including the tools that were fine. It bites hardest for a tool that is not even
+  // importable, whose class nothing would have referenced.
+  if (names.length === 0 || !isUsableName(baseName) || !names.every(isUsableName)) return dedupeUnion(jsonSchemaToPy(schema))
+  const required = new Set(Array.isArray(schema.required) ? schema.required : [])
+  const body = names.map((name) => {
+    const type = declareType(properties[name], `${baseName}${pascal(name)}`, declarations)
+    return `    ${name}: ${required.has(name) ? type : `NotRequired[${type}]`}`
+  })
+  // Two tools whose names differ only in separators pascal-case to the same string; keep the first and let the second carry a suffix rather than silently serving one tool the other's shape. Keyed by name, valued by body, so an identical shape reuses the class instead of declaring it twice.
+  const declared = body.join('\n')
+  let name = baseName
+  for (let n = 2; declarations.has(name) && declarations.get(name) !== declared; n++) name = `${baseName}${n}`
+  declarations.set(name, declared)
+  return name
+}
+
 /**
  * The prompt block. Signatures only, with the description carried as a real docstring the model can read with `read?` instead — so the resident prompt stays small and the detail is fetched on demand.
  */
 export function renderToolsSection(schemas) {
-  const specs = [...schemas].sort(byName).map(toolSpec)
+  const { specs, declarations } = toolSpecs(schemas)
   // MCP tools are named `mcp__<server>__<rawName>` over a `[A-Za-z0-9_-]` alphabet, so hyphens are routine — and legal nowhere in `import` or `def`. One such tool used to make the ENTIRE block a SyntaxError, so not a single line in it could be copied, with nothing saying why.
   //
   // A keyword is the same failure wearing a legal shape: `class` and `None` match the identifier pattern and then break `import` and `def` just as hard. `_` and the soft keywords (`match`, `case`, `type`) are in the list because they are legal identifiers everywhere EXCEPT where this block puts them.
-  const isUsableName = (name) => /^[A-Za-z_][A-Za-z0-9_]*$/.test(name) && !PY_KEYWORDS.has(name)
   const importable = specs.filter((spec) => isUsableName(spec.name))
   const awkward = specs.filter((spec) => !isUsableName(spec.name))
   const signature = (spec) => {
@@ -151,15 +247,29 @@ export function renderToolsSection(schemas) {
     return `async def ${spec.name}(${fields.length === 0 ? '' : `*, ${fields.join(', ')}`}) -> ${spec.returns}: ...`
   }
   // `ToolCallError` leads the import because the instructions tell the model to catch it; without it here the natural `except ToolCallError` NameErrors on the failure path, masking the tool failure it was meant to handle. It also keeps the line valid when no tool is importable.
+  // The declarations are stubs, not runtime objects — nothing constructs them — so the import line
+  // lists exactly what the render used. `Any` has always been reachable from a signature
+  // (`**kwargs: Any`, and any degraded annotation) and was never imported at all.
+  const signatures = importable.map(signature)
+  const classes = [...declarations].flatMap(([name, body]) => ['', `class ${name}(TypedDict):`, ...body.split('\n')])
+  const typing = [
+    // Whole identifiers: `AnyReportOutput` is a class name, not a use of `Any`, and a bare
+    // `includes` imports a symbol a tool can spell into existence without ever needing it.
+    signatures.concat(classes).some((line) => /\bAny\b/.test(line)) && 'Any',
+    classes.some((line) => line.includes('NotRequired[')) && 'NotRequired',
+    declarations.size > 0 && 'TypedDict',
+  ].filter(Boolean)
   const lines = [
     INSTRUCTIONS,
     '',
     renderEnvironment(),
     '',
     '```python',
+    ...(typing.length === 0 ? [] : [`from typing import ${typing.join(', ')}`]),
     `from __dsh__.tools import ToolCallError${importable.map((spec) => `, ${spec.name}`).join('')}`,
+    ...classes,
     '',
-    ...importable.map(signature),
+    ...signatures,
     '```',
   ]
   if (awkward.length > 0) {
@@ -315,9 +425,12 @@ export function apply(ctx, config = {}) {
         exec.deferContext(createUserMessage({ content: outcome.content, source: { kind: 'plugin', plugin: 'dsh-py-codeact' } }))
       }
       for (const context of outcome.additionalContexts ?? []) exec.deferContext(context)
+      // `=== undefined` rather than `??`: a payload that IS `null` is the server's answer, and
+      // falling back to the wrapper there would hand the cell the one shape it was promised not to see.
+      const unwrapped = entry.envelopes.get(from)?.has(name) ? mcpPayload(outcome.value) : undefined
       return outcome.isError
         ? { ok: false, message: contentText(outcome.content) || 'tool call failed' }
-        : { ok: true, value: outcome.value }
+        : { ok: true, value: unwrapped === undefined ? outcome.value : unwrapped.value }
     } catch (error) {
       session?.append('tool/code-dispatch', { ...trace, isError: true, content: [{ type: 'text', text: String(error?.message ?? error) }] })
       throw error
@@ -347,7 +460,7 @@ export function apply(ctx, config = {}) {
       } catch { /* it never came up — fall through and replace it */ }
     }
     // One entry backs every shell in the conversation tree, so `sent` is keyed BY SHELL. `calls` is the exception: it only has to make `subCallId` unique, so process-wide is right.
-    const entry = { kernel: undefined, current: new Map(), calls: 0, sent: new Map(), started: undefined }
+    const entry = { kernel: undefined, current: new Map(), calls: 0, sent: new Map(), envelopes: new Map(), started: undefined }
     entry.kernel = new PythonKernel({
       command: config.command ?? (config.python === undefined ? undefined : [config.python, KERNEL_PY]),
       cwd: exec.agent?.session?.header?.cwd ?? process.cwd(),
@@ -379,9 +492,10 @@ export function apply(ctx, config = {}) {
     },
     // No `isConcurrencySafe`: cells mutate the shell's namespace, so this agent's cells must stay exclusive ordering barriers. That is a per-AGENT constraint, not a per-process one — the kernel tracks one in-flight task per shell, so a subagent's cell runs alongside its parent's just fine.
     async execute(args, exec) {
-      const specs = visibleSchemas(exec.agent).sort(byName).map(toolSpec)
+      const { specs, envelopes } = toolSpecs(visibleSchemas(exec.agent))
       const { entry, shell, restarted: wasRestarted } = await kernelFor(exec, specs)
       entry.current.set(shell, exec)
+      entry.envelopes.set(shell, envelopes)
       try {
         // Only resend the bindings when the visible set actually moved — an unchanged catalogue is the norm, and it is 30+ schemas on the wire.
         const key = specsKey(specs)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-py-codeact",
-  "version": "0.2.0",
+  "version": "0.2.2",
   "description": "CodeAct agent loop for the DeepSeek Harness: a persistent IPython session as the model's action space, with harness tools bridged in as a virtual `__dsh__.tools` module.",
   "license": "MIT",
   "repository": {