nodmix 2026.5.25

package/docs/plugins/cli-backend-plugins.md

@@ -0,0 +1,310 @@
+---
+summary: "Build a plugin that registers a local AI CLI backend"
+title: "Building CLI backend plugins"
+sidebarTitle: "CLI backend plugins"
+read_when:
+  - You are building a local AI CLI backend plugin
+  - You want to register a backend for model refs such as acme-cli/model
+  - You need to map a third-party CLI into Nodmix's text fallback runner
+---
+
+CLI backend plugins let Nodmix call a local AI CLI as a text inference
+backend. The backend appears as a provider prefix in model refs:
+
+```text
+acme-cli/acme-large
+```
+
+Use a CLI backend when the upstream integration is already exposed as a local
+command, when the CLI owns local login state, or when the CLI is a useful
+fallback if API providers are unavailable.
+
+<Info>
+  If the upstream service exposes a normal HTTP model API, write a
+  [provider plugin](/plugins/sdk-provider-plugins) instead. If the upstream
+  runtime owns complete agent sessions, tool events, compaction, or background
+  task state, use an [agent harness](/plugins/sdk-agent-harness).
+</Info>
+
+## What the plugin owns
+
+A CLI backend plugin has three contracts:
+
+| Contract             | File                   | Purpose                                                   |
+| -------------------- | ---------------------- | --------------------------------------------------------- |
+| Package entry        | `package.json`         | Points Nodmix at the plugin runtime module              |
+| Manifest ownership   | `nodmix.plugin.json` | Declares the backend id before runtime loads              |
+| Runtime registration | `index.ts`             | Calls `api.registerCliBackend(...)` with command defaults |
+
+The manifest is discovery metadata. It does not execute the CLI and does not
+register runtime behavior. Runtime behavior starts when the plugin entry calls
+`api.registerCliBackend(...)`.
+
+## Minimal backend plugin
+
+<Steps>
+  <Step title="Create package metadata">
+    ```json package.json
+    {
+      "name": "@acme/nodmix-acme-cli",
+      "version": "1.0.0",
+      "type": "module",
+      "nodmix": {
+        "extensions": ["./index.ts"],
+        "compat": {
+          "pluginApi": ">=2026.3.24-beta.2",
+          "minGatewayVersion": "2026.3.24-beta.2"
+        },
+        "build": {
+          "nodmixVersion": "2026.3.24-beta.2",
+          "pluginSdkVersion": "2026.3.24-beta.2"
+        }
+      },
+      "dependencies": {
+        "nodmix": "^2026.3.24"
+      },
+      "devDependencies": {
+        "typescript": "^5.9.0"
+      }
+    }
+    ```
+
+    Published packages must ship built JavaScript runtime files. If your source
+    entry is `./src/index.ts`, add `nodmix.runtimeExtensions` that points at
+    the built JavaScript peer. See [Entry points](/plugins/sdk-entrypoints).
+
+  </Step>
+
+  <Step title="Declare backend ownership">
+    ```json nodmix.plugin.json
+    {
+      "id": "acme-cli",
+      "name": "Acme CLI",
+      "description": "Run Acme's local AI CLI through Nodmix",
+      "cliBackends": ["acme-cli"],
+      "setup": {
+        "cliBackends": ["acme-cli"],
+        "requiresRuntime": false
+      },
+      "activation": {
+        "onStartup": false
+      },
+      "configSchema": {
+        "type": "object",
+        "additionalProperties": false
+      }
+    }
+    ```
+
+    `cliBackends` is the runtime ownership list. It lets Nodmix auto-load the
+    plugin when config or model selection mentions `acme-cli/...`.
+
+    `setup.cliBackends` is the descriptor-first setup surface. Add it when
+    model discovery, onboarding, or status should recognize the backend without
+    loading plugin runtime. Use `requiresRuntime: false` only when those static
+    descriptors are enough for setup.
+
+  </Step>
+
+  <Step title="Register the backend">
+    ```typescript index.ts
+    import { definePluginEntry } from "nodmix/plugin-sdk/plugin-entry";
+    import {
+      CLI_FRESH_WATCHDOG_DEFAULTS,
+      CLI_RESUME_WATCHDOG_DEFAULTS,
+      type CliBackendPlugin,
+    } from "nodmix/plugin-sdk/cli-backend";
+
+    function buildAcmeCliBackend(): CliBackendPlugin {
+      return {
+        id: "acme-cli",
+        liveTest: {
+          defaultModelRef: "acme-cli/acme-large",
+          defaultImageProbe: false,
+          defaultMcpProbe: false,
+          docker: {
+            npmPackage: "@acme/acme-cli",
+            binaryName: "acme",
+          },
+        },
+        config: {
+          command: "acme",
+          args: ["chat", "--json"],
+          output: "json",
+          input: "stdin",
+          modelArg: "--model",
+          sessionArg: "--session",
+          sessionMode: "existing",
+          sessionIdFields: ["session_id", "conversation_id"],
+          systemPromptFileArg: "--system-file",
+          systemPromptWhen: "first",
+          imageArg: "--image",
+          imageMode: "repeat",
+          reliability: {
+            watchdog: {
+              fresh: { ...CLI_FRESH_WATCHDOG_DEFAULTS },
+              resume: { ...CLI_RESUME_WATCHDOG_DEFAULTS },
+            },
+          },
+          serialize: true,
+        },
+      };
+    }
+
+    export default definePluginEntry({
+      id: "acme-cli",
+      name: "Acme CLI",
+      description: "Run Acme's local AI CLI through Nodmix",
+      register(api) {
+        api.registerCliBackend(buildAcmeCliBackend());
+      },
+    });
+    ```
+
+    The backend id must match the manifest `cliBackends` entry. The registered
+    `config` is only the default; user config under
+    `agents.defaults.cliBackends.acme-cli` is merged over it at runtime.
+
+  </Step>
+</Steps>
+
+## Config shape
+
+`CliBackendConfig` describes how Nodmix should launch and parse the CLI:
+
+| Field                                     | Use                                                         |
+| ----------------------------------------- | ----------------------------------------------------------- |
+| `command`                                 | Binary name or absolute command path                        |
+| `args`                                    | Base argv for fresh runs                                    |
+| `resumeArgs`                              | Alternate argv for resumed sessions; supports `{sessionId}` |
+| `output` / `resumeOutput`                 | Parser: `json`, `jsonl`, or `text`                          |
+| `input`                                   | Prompt transport: `arg` or `stdin`                          |
+| `modelArg`                                | Flag used before the model id                               |
+| `modelAliases`                            | Map Nodmix model ids to CLI-native ids                    |
+| `sessionArg` / `sessionArgs`              | How to pass a session id                                    |
+| `sessionMode`                             | `always`, `existing`, or `none`                             |
+| `sessionIdFields`                         | JSON fields Nodmix reads from CLI output                  |
+| `systemPromptArg` / `systemPromptFileArg` | System prompt transport                                     |
+| `systemPromptWhen`                        | `first`, `always`, or `never`                               |
+| `imageArg` / `imageMode`                  | Image path support                                          |
+| `serialize`                               | Keep same-backend runs ordered                              |
+| `reliability.watchdog`                    | No-output timeout tuning                                    |
+
+Prefer the smallest static config that matches the CLI. Add plugin callbacks
+only for behavior that really belongs to the backend.
+
+## Advanced backend hooks
+
+`CliBackendPlugin` can also define:
+
+| Hook                               | Use                                                    |
+| ---------------------------------- | ------------------------------------------------------ |
+| `normalizeConfig(config, context)` | Rewrite legacy user config after merge                 |
+| `resolveExecutionArgs(ctx)`        | Add request-scoped flags such as thinking effort       |
+| `prepareExecution(ctx)`            | Create temporary auth or config bridges before launch  |
+| `transformSystemPrompt(ctx)`       | Apply a final CLI-specific system prompt transform     |
+| `textTransforms`                   | Bidirectional prompt/output replacements               |
+| `defaultAuthProfileId`             | Prefer a specific Nodmix auth profile                |
+| `authEpochMode`                    | Decide how auth changes invalidate stored CLI sessions |
+| `nativeToolMode`                   | Declare whether the CLI has always-on native tools     |
+| `bundleMcp` / `bundleMcpMode`      | Opt into Nodmix's loopback MCP tool bridge           |
+
+Keep these hooks provider-owned. Do not add CLI-specific branches to core when a
+backend hook can express the behavior.
+
+## MCP tool bridge
+
+CLI backends do not receive Nodmix tools by default. If the CLI can consume an
+MCP configuration, opt in explicitly:
+
+```typescript
+return {
+  id: "acme-cli",
+  bundleMcp: true,
+  bundleMcpMode: "codex-config-overrides",
+  config: {
+    command: "acme",
+    args: ["chat", "--json"],
+    output: "json",
+  },
+};
+```
+
+Supported bridge modes are:
+
+| Mode                     | Use                                                              |
+| ------------------------ | ---------------------------------------------------------------- |
+| `claude-config-file`     | CLIs that accept an MCP config file                              |
+| `codex-config-overrides` | CLIs that accept config overrides on argv                        |
+| `gemini-system-settings` | CLIs that read MCP settings from their system settings directory |
+
+Only enable the bridge when the CLI can actually consume it. If the CLI has its
+own built-in tool layer that cannot be disabled, set `nativeToolMode:
+"always-on"` so Nodmix can fail closed when a caller requires no native tools.
+
+## User configuration
+
+Users can override any backend default:
+
+```json5
+{
+  agents: {
+    defaults: {
+      cliBackends: {
+        "acme-cli": {
+          command: "/opt/acme/bin/acme",
+          args: ["chat", "--json", "--profile", "work"],
+          modelAliases: {
+            large: "acme-large-2026",
+          },
+        },
+      },
+      model: {
+        primary: "openai/gpt-5.5",
+        fallbacks: ["acme-cli/large"],
+      },
+    },
+  },
+}
+```
+
+Document the minimum override users are likely to need. Usually that is only
+`command` when the binary is outside `PATH`.
+
+## Verification
+
+For bundled plugins, add a focused test around the builder and setup
+registration, then run the plugin's targeted test lane:
+
+```bash
+pnpm test extensions/acme-cli
+```
+
+For local or installed plugins, verify discovery and one real model run:
+
+```bash
+nodmix plugins inspect acme-cli --runtime --json
+nodmix agent --message "reply exactly: backend ok" --model acme-cli/acme-large
+```
+
+If the backend supports images or MCP, add a live smoke that proves those paths
+with the real CLI. Do not rely on static inspection for prompt, image, MCP, or
+session-resume behavior.
+
+## Checklist
+
+<Check>`package.json` has `nodmix.extensions` and built runtime entries for published packages</Check>
+<Check>`nodmix.plugin.json` declares `cliBackends` and intentional `activation.onStartup`</Check>
+<Check>`setup.cliBackends` is present when setup/model discovery should see the backend cold</Check>
+<Check>`api.registerCliBackend(...)` uses the same backend id as the manifest</Check>
+<Check>User overrides under `agents.defaults.cliBackends.<id>` still win</Check>
+<Check>Session, system prompt, image, and output parser settings match the real CLI contract</Check>
+<Check>Targeted tests and at least one live CLI smoke prove the backend path</Check>
+
+## Related
+
+- [CLI backends](/gateway/cli-backends) - user configuration and runtime behavior
+- [Building plugins](/plugins/building-plugins) - package and manifest basics
+- [Plugin SDK overview](/plugins/sdk-overview) - registration API reference
+- [Plugin manifest](/plugins/manifest) - `cliBackends` and setup descriptors
+- [Agent harness](/plugins/sdk-agent-harness) - full external agent runtimes

package/docs/plugins/codex-computer-use.md

@@ -0,0 +1,293 @@
+---
+summary: "Set up Codex Computer Use for Codex-mode Nodmix agents"
+title: "Codex Computer Use"
+read_when:
+  - You want Codex-mode Nodmix agents to use Codex Computer Use
+  - You are deciding between Codex Computer Use, PeekabooBridge, and direct cua-driver MCP
+  - You are deciding between Codex Computer Use and a direct cua-driver MCP setup
+  - You are configuring computerUse for the bundled Codex plugin
+  - You are troubleshooting /codex computer-use status or install
+---
+
+Computer Use is a Codex-native MCP plugin for local desktop control. Nodmix
+does not vendor the desktop app, execute desktop actions itself, or bypass
+Codex permissions. The bundled `codex` plugin only prepares Codex app-server:
+it enables Codex plugin support, finds or installs the configured Codex
+Computer Use plugin, checks that the `computer-use` MCP server is available, and
+then lets Codex own the native MCP tool calls during Codex-mode turns.
+
+Use this page when Nodmix is already using the native Codex harness. For the
+runtime setup itself, see [Codex harness](/plugins/codex-harness).
+
+## Nodmix.app and Peekaboo
+
+Nodmix.app's Peekaboo integration is separate from Codex Computer Use. The
+macOS app can host a PeekabooBridge socket so the `peekaboo` CLI can reuse the
+app's local Accessibility and Screen Recording grants for Peekaboo's own
+automation tools. That bridge does not install or proxy Codex Computer Use, and
+Codex Computer Use does not call through the PeekabooBridge socket.
+
+Use [Peekaboo bridge](/platforms/mac/peekaboo) when you want Nodmix.app to be
+a permission-aware host for Peekaboo CLI automation. Use this page when a
+Codex-mode Nodmix agent should have Codex's native `computer-use` MCP plugin
+available before the turn starts.
+
+## iOS app
+
+The iOS app is separate from Codex Computer Use. It does not install or proxy
+the Codex `computer-use` MCP server and it is not a desktop-control backend.
+Instead, the iOS app connects as an Nodmix node and exposes mobile
+capabilities through node commands such as `canvas.*`, `camera.*`, `screen.*`,
+`location.*`, and `talk.*`.
+
+Use [iOS](/platforms/ios) when you want an agent to drive an iPhone node through
+the gateway. Use this page when a Codex-mode agent should control the local
+macOS desktop through Codex's native Computer Use plugin.
+
+## Direct cua-driver MCP
+
+Codex Computer Use is not the only way to expose desktop control. If you want
+Nodmix-managed runtimes to call TryCua's driver directly, use the upstream
+`cua-driver mcp` server through Nodmix's MCP registry instead of the
+Codex-specific marketplace flow.
+
+After installing `cua-driver`, either ask it for the Nodmix command:
+
+```bash
+cua-driver mcp-config --client nodmix
+```
+
+or register the stdio server yourself:
+
+```bash
+nodmix mcp set cua-driver '{"command":"cua-driver","args":["mcp"]}'
+```
+
+That path keeps the upstream MCP tool surface intact, including the driver
+schemas and structured MCP responses. Use it when you want the CUA driver
+available as a normal Nodmix MCP server. Use the Codex Computer Use setup on
+this page when Codex app-server should own plugin installation, MCP reloads,
+and native tool calls inside Codex-mode turns.
+
+CUA's driver is macOS-specific and still requires the local macOS permissions
+that its app prompts for, such as Accessibility and Screen Recording. Nodmix
+does not install `cua-driver`, grant those permissions, or bypass the upstream
+driver's safety model.
+
+## Quick setup
+
+Set `plugins.entries.codex.config.computerUse` when Codex-mode turns must have
+Computer Use available before a thread starts:
+
+```json5
+{
+  plugins: {
+    entries: {
+      codex: {
+        enabled: true,
+        config: {
+          computerUse: {
+            autoInstall: true,
+          },
+        },
+      },
+    },
+  },
+  agents: {
+    defaults: {
+      model: "openai/gpt-5.5",
+    },
+  },
+}
+```
+
+With this config, Nodmix checks Codex app-server before each Codex-mode turn.
+If Computer Use is missing but Codex app-server has already discovered an
+installable marketplace, Nodmix asks Codex app-server to install or re-enable
+the plugin and reload MCP servers. On macOS, when no matching marketplace is
+registered and the standard Codex app bundle exists, Nodmix also tries to
+register the bundled Codex marketplace from
+`/Applications/Codex.app/Contents/Resources/plugins/openai-bundled` before it
+fails. If setup still cannot make the MCP server available, the turn fails
+before the thread starts.
+
+After changing Computer Use config, use `/new` or `/reset` in the affected chat
+before testing if an existing Codex thread has already started.
+
+## Commands
+
+Use the `/codex computer-use` commands from any chat surface where the `codex`
+plugin command surface is available. These are Nodmix chat/runtime commands,
+not `nodmix codex ...` CLI subcommands:
+
+```text
+/codex computer-use status
+/codex computer-use install
+/codex computer-use install --source <marketplace-source>
+/codex computer-use install --marketplace-path <path>
+/codex computer-use install --marketplace <name>
+```
+
+`status` is read-only. It does not add marketplace sources, install plugins, or
+enable Codex plugin support.
+
+`install` enables Codex app-server plugin support, optionally adds a configured
+marketplace source, installs or re-enables the configured plugin through Codex
+app-server, reloads MCP servers, and verifies that the MCP server exposes tools.
+
+## Marketplace choices
+
+Nodmix uses the same app-server API that Codex itself exposes. The
+marketplace fields choose where Codex should find `computer-use`.
+
+| Field                | Use when                                                        | Install support                                          |
+| -------------------- | --------------------------------------------------------------- | -------------------------------------------------------- |
+| No marketplace field | You want Codex app-server to use marketplaces it already knows. | Yes, when app-server returns a local marketplace.        |
+| `marketplaceSource`  | You have a Codex marketplace source app-server can add.         | Yes, for explicit `/codex computer-use install`.         |
+| `marketplacePath`    | You already know the local marketplace file path on the host.   | Yes, for explicit install and turn-start auto-install.   |
+| `marketplaceName`    | You want to select one already registered marketplace by name.  | Yes only when the selected marketplace has a local path. |
+
+Fresh Codex homes may need a short moment to seed their official marketplaces.
+During install, Nodmix polls `plugin/list` for up to
+`marketplaceDiscoveryTimeoutMs` milliseconds. The default is 60 seconds.
+
+If multiple known marketplaces contain Computer Use, Nodmix prefers
+`openai-bundled`, then `openai-curated`, then `local`. Unknown ambiguous matches
+fail closed and ask you to set `marketplaceName` or `marketplacePath`.
+
+## Bundled macOS marketplace
+
+Recent Codex desktop builds bundle Computer Use here:
+
+```text
+/Applications/Codex.app/Contents/Resources/plugins/openai-bundled/plugins/computer-use
+```
+
+When `computerUse.autoInstall` is true and no marketplace containing
+`computer-use` is registered, Nodmix tries to add the standard bundled
+marketplace root automatically:
+
+```text
+/Applications/Codex.app/Contents/Resources/plugins/openai-bundled
+```
+
+You can also register it explicitly from a shell with Codex:
+
+```bash
+codex plugin marketplace add /Applications/Codex.app/Contents/Resources/plugins/openai-bundled
+```
+
+If you use a nonstandard Codex app path, set `computerUse.marketplacePath` to a
+local marketplace file path or run `/codex computer-use install --source
+<marketplace-source>` once.
+
+## Remote catalog limit
+
+Codex app-server can list and read remote-only catalog entries, but it does not
+currently support remote `plugin/install`. That means `marketplaceName` can
+select a remote-only marketplace for status checks, but installs and re-enables
+still need a local marketplace via `marketplaceSource` or `marketplacePath`.
+
+If status says the plugin is available in a remote Codex marketplace but remote
+install is unsupported, run install with a local source or path:
+
+```text
+/codex computer-use install --source <marketplace-source>
+/codex computer-use install --marketplace-path <path>
+```
+
+## Configuration reference
+
+| Field                           | Default        | Meaning                                                                        |
+| ------------------------------- | -------------- | ------------------------------------------------------------------------------ |
+| `enabled`                       | inferred       | Require Computer Use. Defaults to true when another Computer Use field is set. |
+| `autoInstall`                   | false          | Install or re-enable from already discovered marketplaces at turn start.       |
+| `marketplaceDiscoveryTimeoutMs` | 60000          | How long install waits for Codex app-server marketplace discovery.             |
+| `marketplaceSource`             | unset          | Source string passed to Codex app-server `marketplace/add`.                    |
+| `marketplacePath`               | unset          | Local Codex marketplace file path containing the plugin.                       |
+| `marketplaceName`               | unset          | Registered Codex marketplace name to select.                                   |
+| `pluginName`                    | `computer-use` | Codex marketplace plugin name.                                                 |
+| `mcpServerName`                 | `computer-use` | MCP server name exposed by the installed plugin.                               |
+
+Turn-start auto-install intentionally refuses configured `marketplaceSource`
+values. Adding a new source is an explicit setup operation, so use
+`/codex computer-use install --source <marketplace-source>` once, then let
+`autoInstall` handle future re-enables from discovered local marketplaces.
+Turn-start auto-install can use a configured `marketplacePath`, because that is
+already a local path on the host.
+
+## What Nodmix checks
+
+Nodmix reports a stable setup reason internally and formats the user-facing
+status for chat:
+
+| Reason                       | Meaning                                                | Next step                                     |
+| ---------------------------- | ------------------------------------------------------ | --------------------------------------------- |
+| `disabled`                   | `computerUse.enabled` resolved to false.               | Set `enabled` or another Computer Use field.  |
+| `marketplace_missing`        | No matching marketplace was available.                 | Configure source, path, or marketplace name.  |
+| `plugin_not_installed`       | Marketplace exists, but the plugin is not installed.   | Run install or enable `autoInstall`.          |
+| `plugin_disabled`            | Plugin is installed but disabled in Codex config.      | Run install to re-enable it.                  |
+| `remote_install_unsupported` | Selected marketplace is remote-only.                   | Use `marketplaceSource` or `marketplacePath`. |
+| `mcp_missing`                | Plugin is enabled, but the MCP server is unavailable.  | Check Codex Computer Use and OS permissions.  |
+| `ready`                      | Plugin and MCP tools are available.                    | Start the Codex-mode turn.                    |
+| `check_failed`               | A Codex app-server request failed during status check. | Check app-server connectivity and logs.       |
+| `auto_install_blocked`       | Turn-start setup would need to add a new source.       | Run explicit install first.                   |
+
+The chat output includes the plugin state, MCP server state, marketplace, tools
+when available, and the specific message for the failing setup step.
+
+## macOS permissions
+
+Computer Use is macOS-specific. The Codex-owned MCP server may need local OS
+permissions before it can inspect or control apps. If Nodmix says Computer Use
+is installed but the MCP server is unavailable, verify the Codex-side Computer
+Use setup first:
+
+- Codex app-server is running on the same host where desktop control should
+  happen.
+- The Computer Use plugin is enabled in Codex config.
+- The `computer-use` MCP server appears in Codex app-server MCP status.
+- macOS has granted the required permissions for the desktop-control app.
+- The current host session can access the desktop being controlled.
+
+Nodmix intentionally fails closed when `computerUse.enabled` is true. A
+Codex-mode turn should not silently proceed without the native desktop tools
+that the config required.
+
+## Troubleshooting
+
+**Status says not installed.** Run `/codex computer-use install`. If the
+marketplace is not discovered, pass `--source` or `--marketplace-path`.
+
+**Status says installed but disabled.** Run `/codex computer-use install` again.
+Codex app-server install writes the plugin config back to enabled.
+
+**Status says remote install is unsupported.** Use a local marketplace source or
+path. Remote-only catalog entries can be inspected but not installed through the
+current app-server API.
+
+**Status says the MCP server is unavailable.** Re-run install once so MCP
+servers reload. If it remains unavailable, fix the Codex Computer Use app,
+Codex app-server MCP status, or macOS permissions.
+
+**Status or a probe times out on `computer-use.list_apps`.** The plugin and MCP
+server are present, but the local Computer Use bridge did not answer. Quit or
+restart Codex Computer Use, relaunch Codex Desktop if needed, then retry in a
+fresh Nodmix session.
+
+**A Computer Use tool says `Native hook relay unavailable`.** The Codex-native
+tool hook could not reach an active Nodmix relay through the local bridge or
+Gateway fallback. Start a fresh Nodmix session with `/new` or `/reset`. If it
+keeps happening, restart the gateway so old app-server threads and hook
+registrations are dropped, then retry.
+
+**Turn-start auto-install refuses a source.** This is intentional. Add the
+source with explicit `/codex computer-use install --source <marketplace-source>`
+first, then future turn-start auto-install can use the discovered local
+marketplace.
+
+## Related
+
+- [Codex harness](/plugins/codex-harness)
+- [Peekaboo bridge](/platforms/mac/peekaboo)
+- [iOS app](/platforms/ios)