dsh-py-codeact 0.1.2 → 0.2.1

package/README.md

@@ -52,11 +52,28 @@ A dunder-named package reads as harness-owned, survives `%reset`, and leaves the
 
 ```
 In [1]: read?
-Signature: read(*, file_path: 'str', offset: 'int' = Ellipsis, limit: 'int' = Ellipsis)
+Signature: read(*, file_path: 'str', offset: 'int' = Ellipsis, limit: 'int' = Ellipsis) -> 'str'
 Docstring: Read a file from the workspace. Results include line numbers…
 ```
 
-That is why the prompt block carries signatures only — the descriptions are one `?` away instead of resident in every request. Annotations are rendered host-side with dsh's own exported `jsonSchemaToPy`, so there is no second JSON-Schema mapper to drift. The binding set is resent with every cell, because restrictions and mid-conversation tool changes can move a tool in or out between calls.
+That is why the prompt block carries signatures only — the descriptions are one `?` away instead of resident in every request. Annotations are rendered host-side with dsh's own exported `jsonSchemaToPy`, so there is no second JSON-Schema mapper to drift.
+
+The **return** type comes from the tool's own `output` schema, by way of `ctx.tools.sdkSchemas(scope)` — the projection that carries it. It is the one annotation the model cannot recover by reading harder: a wrong argument fails loudly at the call, while an unknown return shape is only discoverable by calling once and printing the result, which costs a whole turn per tool. A tool that declares no output schema still renders `Any`; claiming a type nobody declared would be worse than admitting ignorance.
+
+Each object in that schema whose keys — and whose own generated class name — Python can take is declared as a named `TypedDict` above the signatures, because for an MCP tool the flat form is not merely vague — it is the whole message. dsh's MCP client resolves a call to the **envelope** `{ content, structuredContent }` and declares exactly that as the tool's output, so a model told only `dict[str, Any]` is not told the envelope exists: it reaches for the payload directly, gets nothing, and spends the turn printing the result to find the wrapper — the exact cost this annotation is here to remove.
+
+```python
+class McpCalendarListEventsOutputStructuredContent(TypedDict):
+    result: str
+
+class McpCalendarListEventsOutput(TypedDict):
+    content: list[Any]
+    structuredContent: McpCalendarListEventsOutputStructuredContent
+
+async def mcp__calendar__list_events(*, calendar_id: str) -> McpCalendarListEventsOutput: ...
+```
+
+`jsonSchemaToPy` cannot do this and says so — it is context-free, and naming a `TypedDict` needs the render context `renderToolsSdkPy` supplies. That renderer is not reusable here: it emits a whole document in Code Mode's own `tools.name(args)` contract. So only the object and array branches are handled locally, every leaf still going through `jsonSchemaToPy` — a place to hang the names, not a second JSON-Schema mapper. The binding set is resent with every cell, because restrictions and mid-conversation tool changes can move a tool in or out between calls.
 
 ### MCP tools work, with nothing special
 
@@ -157,9 +174,9 @@ A pure CPU loop (`while True: pass`) never reaches an await point. After `hardIn
 ## Known limitations
 
 - **Native writes are not captured.** stdout/stderr are captured at the Python level, so `print` is captured but a subprocess writing to fd 1 is not. Use `subprocess.run(..., capture_output=True)`, or `%run`. (Anything that does reach fd 1/2 — including IPython's own colored traceback, deliberately routed there — is retained only for the crash message.)
-- **Scope is not optional.** The visible tool set comes from `ctx.tools.schemas(scope)` — the scope being the agent. Omitting it yields the *global* view, which in a preset composition holds only host-registered tools; the preset's own `read`/`bash`/`edit` live in the agent scope and vanish. The prompt section reads `assembly.scope`, and the kernel's name list is resent with every cell (restrictions and mid-conversation tool changes can move a tool in or out between calls).
+- **Scope is not optional.** The visible tool set comes from `ctx.tools.sdkSchemas(scope)` — the scope being the agent. Omitting it yields the *global* view, which in a preset composition holds only host-registered tools; the preset's own `read`/`bash`/`edit` live in the agent scope and vanish. The prompt section reads `assembly.scope`, and the kernel's name list is resent with every cell (restrictions and mid-conversation tool changes can move a tool in or out between calls).
 - **Pick a `toolName` nothing else answers to.** With an MCP IPython server also mounted, a model told to "use the python tool" reaches for `mcp__py__ipython_execute_code` — which has no `tools` binding — and then reports that your tool does not exist.
-- **Return types render as `Any`.** The prompt section is generated from `ctx.tools.schemas()`, which whitelists name/description/parameters; the canonical *output* schemas sit behind the registry's private `sdkSchemas`. Argument annotations are complete; return annotations are not. A public accessor upstream would fix this — `renderToolsSdkPy` is already exported and takes exactly the `ToolSchema + output` shape it would provide.
+- **A shape Python cannot name stays vague.** A field whose key is not a valid identifier degrades its own class back to `dict[str, Any]` rather than emitting a body that will not parse — the tool stays callable and only that one annotation goes quiet. The same guard covers the class name itself, which is not local damage: a tool named `123tool` would otherwise emit `class 123toolOutput`, a SyntaxError that takes the whole block with it. A schema with no declared properties still renders `Any`; claiming a type nobody declared would be worse.
 - **Cold start.** The PEP 723 environment is resolved on first use (`uv python find --script`). Warm afterwards; pass `python` to skip it.
 
   The interpreter is then spawned **directly**, never behind `uv run --script`. A wrapper stays in the process tree as the interpreter's parent: when it exits first, the interpreter is reparented to init, the handle the host holds reports an exit, and a perfectly live kernel looks dead — so the next cell respawns and the session's state vanishes with an `[the interpreter was restarted]` notice nothing actually caused. `alive` is likewise tracked from the exit event rather than read off `proc.killed`, which Node sets on any `kill()` call, including a signal the process survived.

package/lib/index.js

@@ -64,14 +64,18 @@ The available tools:`
 
 /**
  * Project one tool schema onto the wire spec the kernel builds a binding from: dsh's description becomes the function's docstring, its parameters become a keyword-only signature. Annotations are rendered here with dsh's own `jsonSchemaToPy`, so the kernel needs no second JSON-Schema mapper.
+ *
+ * The return type comes from the tool's own `output` schema — the shape of the canonical value the call actually resolves to. It is the one annotation a model cannot recover by reading harder: a parameter it gets wrong fails loudly at the call, while an unknown return shape is only discoverable by calling once and printing the result, which is a whole extra turn per tool.
  */
-function toolSpec(schema) {
+function toolSpec(schema, declarations) {
   const parameters = schema.parameters ?? {}
   const properties = parameters.properties ?? {}
   const required = new Set(Array.isArray(parameters.required) ? parameters.required : [])
   return {
     name: schema.name,
     doc: schema.description,
+    // `output` is absent only if the caller passed a `schemas()` projection; `sdkSchemas()` always carries it.
+    returns: schema.output === undefined ? 'Any' : declareType(schema.output, `${pascal(schema.name)}Output`, declarations),
     params: Object.entries(properties).map(([name, node]) => ({
       name,
       type: jsonSchemaToPy(node),
@@ -80,6 +84,20 @@ function toolSpec(schema) {
   }
 }
 
+/**
+ * Every visible tool's spec, plus the `TypedDict` bodies their return annotations name.
+ *
+ * Both consumers go through here so they cannot disagree: the prompt block declares the classes,
+ * and the kernel sends the same `returns` text on to `inspect.Signature`, so `read?` shows the
+ * name the block above it defines. It also keeps `toolSpec` off `.map`, where the callback's
+ * second argument is the INDEX — a number, so a default parameter never fires and the accumulator
+ * is silently a `0`.
+ */
+function toolSpecs(schemas) {
+  const declarations = new Map()
+  return { specs: [...schemas].sort(byName).map((schema) => toolSpec(schema, declarations)), declarations }
+}
+
 /** Deterministic (lexicographic) order, so an unchanged tool set renders byte-identically. */
 const byName = (a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)
 
@@ -128,34 +146,88 @@ export function needsRestartNotice(session, toolName, plugin) {
 /** `keyword.kwlist + keyword.softkwlist` for the `requires-python = ">=3.12"` the kernel pins. */
 const PY_KEYWORDS = new Set(['False', 'None', 'True', '_', 'and', 'as', 'assert', 'async', 'await', 'break', 'case', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'match', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'type', 'while', 'with', 'yield'])
 
+/** A name this block can put in an `import`, a `def`, or a `TypedDict` field. */
+const isUsableName = (name) => /^[A-Za-z_][A-Za-z0-9_]*$/.test(name) && !PY_KEYWORDS.has(name)
+
+/** `mcp__calendar__list_events` -> `McpCalendarListEvents`, for naming that tool's declarations. */
+const pascal = (name) => name.split(/[_\-]+/).filter(Boolean).map((part) => part[0].toUpperCase() + part.slice(1)).join('')
+
+/** `dict[str, Any] | dict[str, Any]` is what a `oneOf` of two objects collapses to once both branches degrade to the same text. Rendering the duplicate says there is a choice to make where there is none. */
+const dedupeUnion = (expr) => [...new Set(expr.split(' | '))].join(' | ')
+
+/**
+ * Render one output schema, declaring a named `TypedDict` for every object shape it contains.
+ *
+ * `jsonSchemaToPy` alone cannot do this and says so: it is context-free, and "naming a `TypedDict` requires the render context that `renderToolsSdkPy` supplies" — with nowhere to put a class declaration it degrades every object to `dict[str, Any]`. That erases the one thing an MCP return needs to say. dsh's MCP client resolves a call to the ENVELOPE, `{ content, structuredContent }`, and declares exactly that as the tool's `output`; flattened to `dict[str, Any]` the model is not told the envelope exists, so it reaches for the payload directly, gets nothing, and spends a turn printing the result to find the wrapper — the very cost the return annotation is here to remove.
+ *
+ * Only the object and array branches are handled here; every leaf still goes through `jsonSchemaToPy`, so there is no second JSON-Schema mapper, only a place to hang the names. A field name Python cannot take degrades that one class back to `dict[str, Any]` rather than emitting a class body that will not parse.
+ */
+function declareType(schema, baseName, declarations) {
+  if (schema === null || typeof schema !== 'object') return jsonSchemaToPy(schema)
+  if (schema.type === 'array' && schema.items !== null && typeof schema.items === 'object') {
+    return `list[${declareType(schema.items, `${baseName}Item`, declarations)}]`
+  }
+  const properties = schema.type === 'object' ? (schema.properties ?? {}) : {}
+  const names = Object.keys(properties)
+  // `baseName` too, not just the fields: a tool name may start with a digit or carry a character
+  // `pascal` does not split on, and `123toolOutput` is a SyntaxError that takes the WHOLE block
+  // with it — including the tools that were fine. It bites hardest for a tool that is not even
+  // importable, whose class nothing would have referenced.
+  if (names.length === 0 || !isUsableName(baseName) || !names.every(isUsableName)) return dedupeUnion(jsonSchemaToPy(schema))
+  const required = new Set(Array.isArray(schema.required) ? schema.required : [])
+  const body = names.map((name) => {
+    const type = declareType(properties[name], `${baseName}${pascal(name)}`, declarations)
+    return `    ${name}: ${required.has(name) ? type : `NotRequired[${type}]`}`
+  })
+  // Two tools whose names differ only in separators pascal-case to the same string; keep the first and let the second carry a suffix rather than silently serving one tool the other's shape. Keyed by name, valued by body, so an identical shape reuses the class instead of declaring it twice.
+  const declared = body.join('\n')
+  let name = baseName
+  for (let n = 2; declarations.has(name) && declarations.get(name) !== declared; n++) name = `${baseName}${n}`
+  declarations.set(name, declared)
+  return name
+}
+
 /**
  * The prompt block. Signatures only, with the description carried as a real docstring the model can read with `read?` instead — so the resident prompt stays small and the detail is fetched on demand.
  */
 export function renderToolsSection(schemas) {
-  const specs = [...schemas].sort(byName).map(toolSpec)
+  const { specs, declarations } = toolSpecs(schemas)
   // MCP tools are named `mcp__<server>__<rawName>` over a `[A-Za-z0-9_-]` alphabet, so hyphens are routine — and legal nowhere in `import` or `def`. One such tool used to make the ENTIRE block a SyntaxError, so not a single line in it could be copied, with nothing saying why.
   //
   // A keyword is the same failure wearing a legal shape: `class` and `None` match the identifier pattern and then break `import` and `def` just as hard. `_` and the soft keywords (`match`, `case`, `type`) are in the list because they are legal identifiers everywhere EXCEPT where this block puts them.
-  const isUsableName = (name) => /^[A-Za-z_][A-Za-z0-9_]*$/.test(name) && !PY_KEYWORDS.has(name)
   const importable = specs.filter((spec) => isUsableName(spec.name))
   const awkward = specs.filter((spec) => !isUsableName(spec.name))
   const signature = (spec) => {
     // The same rule applies one level down, and used to not be applied at all: `file-path` is routine for MCP tools, and one such PARAMETER made every other tool's signature unusable too. The binding still takes it — the kernel folds unnameable parameters into `**kwargs` — so the tool stays importable and only its signature goes vague; `name?` still shows the real one.
-    if (!spec.params.every((p) => isUsableName(p.name))) return `async def ${spec.name}(**kwargs: Any) -> Any: ...  # parameter names are not all valid Python; see ${spec.name}?`
+    if (!spec.params.every((p) => isUsableName(p.name))) return `async def ${spec.name}(**kwargs: Any) -> ${spec.returns}: ...  # parameter names are not all valid Python; see ${spec.name}?`
     const fields = spec.params.map((p) => (p.required ? `${p.name}: ${p.type}` : `${p.name}: ${p.type} = ...`))
-    // `async def f(*, ) -> Any` is a SyntaxError: a `*` needs at least one name after it, and a parameterless tool takes no keywords at all.
-    return `async def ${spec.name}(${fields.length === 0 ? '' : `*, ${fields.join(', ')}`}) -> Any: ...`
+    // `async def f(*, ) -> T` is a SyntaxError: a `*` needs at least one name after it, and a parameterless tool takes no keywords at all.
+    return `async def ${spec.name}(${fields.length === 0 ? '' : `*, ${fields.join(', ')}`}) -> ${spec.returns}: ...`
   }
   // `ToolCallError` leads the import because the instructions tell the model to catch it; without it here the natural `except ToolCallError` NameErrors on the failure path, masking the tool failure it was meant to handle. It also keeps the line valid when no tool is importable.
+  // The declarations are stubs, not runtime objects — nothing constructs them — so the import line
+  // lists exactly what the render used. `Any` has always been reachable from a signature
+  // (`**kwargs: Any`, and any degraded annotation) and was never imported at all.
+  const signatures = importable.map(signature)
+  const classes = [...declarations].flatMap(([name, body]) => ['', `class ${name}(TypedDict):`, ...body.split('\n')])
+  const typing = [
+    // Whole identifiers: `AnyReportOutput` is a class name, not a use of `Any`, and a bare
+    // `includes` imports a symbol a tool can spell into existence without ever needing it.
+    signatures.concat(classes).some((line) => /\bAny\b/.test(line)) && 'Any',
+    classes.some((line) => line.includes('NotRequired[')) && 'NotRequired',
+    declarations.size > 0 && 'TypedDict',
+  ].filter(Boolean)
   const lines = [
     INSTRUCTIONS,
     '',
     renderEnvironment(),
     '',
     '```python',
+    ...(typing.length === 0 ? [] : [`from typing import ${typing.join(', ')}`]),
     `from __dsh__.tools import ToolCallError${importable.map((spec) => `, ${spec.name}`).join('')}`,
+    ...classes,
     '',
-    ...importable.map(signature),
+    ...signatures,
     '```',
   ]
   if (awkward.length > 0) {
@@ -237,7 +309,8 @@ export function apply(ctx, config = {}) {
   /**
    * The tools ONE agent can see. The scope argument is not optional in practice: omitting it yields the global view, which in a preset composition holds only host-registered tools — the preset's own `read`/`bash`/etc. live in the agent scope and would silently go missing.
    */
-  const visibleSchemas = (scope) => ctx.tools.schemas(scope).filter((schema) => schema.name !== toolName)
+  // `sdkSchemas`, not `schemas`: the latter is the native-function-calling projection and drops `output`, which left every signature returning `Any`. Code Mode uses this same projection for the same reason.
+  const visibleSchemas = (scope) => ctx.tools.sdkSchemas(scope).filter((schema) => schema.name !== toolName)
 
   ctx.systemPrompt.section({
     name: 'py-codeact:sdk',
@@ -374,7 +447,7 @@ export function apply(ctx, config = {}) {
     },
     // No `isConcurrencySafe`: cells mutate the shell's namespace, so this agent's cells must stay exclusive ordering barriers. That is a per-AGENT constraint, not a per-process one — the kernel tracks one in-flight task per shell, so a subagent's cell runs alongside its parent's just fine.
     async execute(args, exec) {
-      const specs = visibleSchemas(exec.agent).sort(byName).map(toolSpec)
+      const { specs } = toolSpecs(visibleSchemas(exec.agent))
       const { entry, shell, restarted: wasRestarted } = await kernelFor(exec, specs)
       entry.current.set(shell, exec)
       try {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-py-codeact",
-  "version": "0.1.2",
+  "version": "0.2.1",
   "description": "CodeAct agent loop for the DeepSeek Harness: a persistent IPython session as the model's action space, with harness tools bridged in as a virtual `__dsh__.tools` module.",
   "license": "MIT",
   "repository": {

package/py/kernel.py

@@ -18,7 +18,7 @@ Wire protocol: JSON-lines on fd 3. stdout/stderr stay free for native writes.
 
 Every frame but `result` and `shutdown` carries `shell` — the agent whose shell it addresses, defaulting to "main". One process holds one shell per agent, so a fan-out of subagents shares this interpreter, its event loop and its packages while keeping separate globals.
 
-  host  -> child  {"t":"init","shell":S,"tools":[{"name","doc","params":[...]}]}
+  host  -> child  {"t":"init","shell":S,"tools":[{"name","doc","returns","params":[...]}]}
                   {"t":"exec","id":N,"shell":S,"code":"...","tools":[...]}
                   {"t":"result","id":N,"ok":true,"value":<json>}
                   {"t":"result","id":N,"ok":false,"tool":"read","message":"..."}
@@ -233,7 +233,8 @@ def _make_binding(bridge: Bridge, spec):
     if dropped:
         params.append(inspect.Parameter("kwargs", inspect.Parameter.VAR_KEYWORD))
     with contextlib.suppress(ValueError, TypeError):
-        call.__signature__ = inspect.Signature(params)  # type: ignore
+        # The return annotation matters as much as the parameters here: it is what `read?` can tell the model that no amount of re-reading the call site can. Same source as the prompt block, so the two never disagree.
+        call.__signature__ = inspect.Signature(params, return_annotation=spec.get("returns") or "Any")  # type: ignore
     return call