codex-computer-use-mcp 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +18 -2
- package/MIGRATION.md +22 -5
- package/PROOF.md +31 -4
- package/README.md +16 -12
- package/SECURITY.md +10 -8
- package/dist/audit.d.ts +1 -1
- package/dist/direct-broker.d.ts +17 -3
- package/dist/direct-broker.js +30 -16
- package/dist/direct-broker.js.map +1 -1
- package/dist/direct-service.d.ts +3 -1
- package/dist/direct-service.js +11 -6
- package/dist/direct-service.js.map +1 -1
- package/dist/mcp-elicitation.d.ts +8 -0
- package/dist/mcp-elicitation.js +47 -0
- package/dist/mcp-elicitation.js.map +1 -0
- package/dist/mcp-server.js +44 -42
- package/dist/mcp-server.js.map +1 -1
- package/dist/tools.d.ts +12 -6
- package/dist/tools.js +93 -23
- package/dist/tools.js.map +1 -1
- package/docs/adr/0001-standalone-code-mode-mcp.md +45 -0
- package/integrations/pi/README.md +16 -6
- package/integrations/pi/index.ts +156 -31
- package/integrations/pi/mcp.json.example +1 -1
- package/npm-shrinkwrap.json +24 -27
- package/package.json +5 -4
package/ARCHITECTURE.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# Direct architecture and boundary inventory
|
|
2
2
|
|
|
3
|
+
Code Mode remains outside this package. See [ADR 0001: Keep Code Mode in a standalone MCP process](docs/adr/0001-standalone-code-mode-mcp.md).
|
|
4
|
+
|
|
3
5
|
## Official API evidence
|
|
4
6
|
|
|
5
7
|
The architecture uses public source and the installed signed binaries; it does not speak a private Computer Use socket protocol.
|
|
@@ -12,6 +14,12 @@ At OpenAI Codex source commit [`5c19155cbd93bfa099016e7487259f61669823ff`](https
|
|
|
12
14
|
- the implementation loads the thread runtime and invokes `thread.call_mcp_tool(...)` directly ([processor lines 455–474](https://github.com/openai/codex/blob/5c19155cbd93bfa099016e7487259f61669823ff/codex-rs/app-server/src/request_processors/mcp_processor.rs#L455-L474));
|
|
13
15
|
- downstream structured elicitation is an explicit client capability rather than implicit approval ([README lines 89–94](https://github.com/openai/codex/blob/5c19155cbd93bfa099016e7487259f61669823ff/codex-rs/app-server/README.md#L89-L94)).
|
|
14
16
|
|
|
17
|
+
The Full access behavior is also public in the pinned Codex `rust-v0.144.2` source at commit [`a6645b6b8a656360fa16fb7e1c6721d0697d3d6a`](https://github.com/openai/codex/tree/a6645b6b8a656360fa16fb7e1c6721d0697d3d6a):
|
|
18
|
+
|
|
19
|
+
- legacy `danger-full-access` maps to `PermissionProfile::Disabled` ([models lines 289–297](https://github.com/openai/codex/blob/a6645b6b8a656360fa16fb7e1c6721d0697d3d6a/codex-rs/protocol/src/models.rs#L289-L297));
|
|
20
|
+
- MCP permission prompts are auto-approved when the approval policy is `Never` and the permission profile is disabled ([MCP policy lines 76–99](https://github.com/openai/codex/blob/a6645b6b8a656360fa16fb7e1c6721d0697d3d6a/codex-rs/codex-mcp/src/mcp/mod.rs#L76-L99));
|
|
21
|
+
- the elicitation manager applies that policy before client notification ([elicitation lines 196–221](https://github.com/openai/codex/blob/a6645b6b8a656360fa16fb7e1c6721d0697d3d6a/codex-rs/codex-mcp/src/elicitation.rs#L196-L221)), and limits the automatic response to empty-schema approval/confirmation forms ([lines 319–331](https://github.com/openai/codex/blob/a6645b6b8a656360fa16fb7e1c6721d0697d3d6a/codex-rs/codex-mcp/src/elicitation.rs#L319-L331)).
|
|
22
|
+
|
|
15
23
|
The installed CLI labels app-server experimental. This project therefore verifies the current method and upstream schemas on every call and fails closed on drift.
|
|
16
24
|
|
|
17
25
|
## Live architecture probe
|
|
@@ -103,7 +111,7 @@ nested prompts or result summaries
|
|
|
103
111
|
| Wrapper app/intent allowlists in full mode | **accidental** | absent |
|
|
104
112
|
| Safe/full wrapper modes | **compatibility-only** | removed; one durable no-permissions interface exposes all ten methods |
|
|
105
113
|
| Wrapper approval prompts/configuration | **compatibility-only** | removed; no command, config, environment, or per-call selector remains |
|
|
106
|
-
| Official first-party app access | **official-required** |
|
|
114
|
+
| Official first-party app access | **official-required** | official Codex Full access auto-accepts normal empty-schema app approval elicitations; any elicitation app-server emits is forwarded unchanged |
|
|
107
115
|
| macOS TCC | **official-required** | retained; never modified |
|
|
108
116
|
| Exact ten-tool inventory/schema | **security-essential** | retained and checked before each call |
|
|
109
117
|
| Canonical bundle identity | **security-essential** | retained before targeted dispatch |
|
|
@@ -121,7 +129,15 @@ nested prompts or result summaries
|
|
|
121
129
|
|
|
122
130
|
No-permissions is the sole interface and means unrestricted wrapper dispatch with no wrapper approval prompt. All ten methods are registered. No config file, environment value, slash command, CLI mode switch, or tool argument selects another route.
|
|
123
131
|
|
|
124
|
-
|
|
132
|
+
App-server is created with `approvalPolicy: "never"` and `sandbox: "danger-full-access"`. This is the official Codex Full access combination. The legacy sandbox field maps to `PermissionProfile::Disabled`; Codex then auto-accepts empty-schema MCP approval elicitations before client notification. Normal first-party Computer Use app approvals therefore do not prompt, and the bridge does not need to mutate the per-bundle approval store.
|
|
133
|
+
|
|
134
|
+
The broker still handles any `mcpServer/elicitation/request` that app-server emits. Pi advertises OpenAI-form support and renders emitted form, OpenAI-form, and URL requests. Generic stdio MCP forwards supported standard modes as `elicitation/create`. The response path preserves `accept`, `decline`, `cancel`, structured content, and response metadata; no callback or compatible UI yields `cancel`, never an invented decision.
|
|
135
|
+
|
|
136
|
+
## Pi tool disclosure boundary
|
|
137
|
+
|
|
138
|
+
The Pi extension registers all ten official definitions. On `session_start` it preserves active tools owned by Pi and other extensions, keeps `computer_use_list_apps` and `computer_use_get_app_state` active, and removes only the eight Computer Use interaction definitions from the initial active set. After a successful `get_app_state` result, it additively activates those eight definitions without removing any currently active tool.
|
|
139
|
+
|
|
140
|
+
This is context disclosure, not a wrapper permission gate. The schemas, execution path, and no-permissions policy do not change. Loader and interaction definitions remain registered throughout the session. The interaction definitions omit `promptSnippet` and `promptGuidelines`, allowing Pi 0.80.7's native deferred-loading representation to add schemas at the tool-result position without changing the system-prompt prefix. Providers without native support use Pi's next-request fallback.
|
|
125
141
|
|
|
126
142
|
## Output boundary
|
|
127
143
|
|
package/MIGRATION.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Migration and rollback
|
|
2
2
|
|
|
3
|
-
Version 0.2.0
|
|
3
|
+
Version 0.2.0 was the breaking direct-tool architecture change from version 0.1.0. Version 0.3.0 keeps that exact ten-tool contract and adds Pi 0.80.7 progressive disclosure. Use the relevant section below.
|
|
4
4
|
|
|
5
5
|
## What changes
|
|
6
6
|
|
|
@@ -35,7 +35,7 @@ CODEX_COMPUTER_USE_HOME="$(mktemp -d)" \
|
|
|
35
35
|
|
|
36
36
|
`-ne` suppresses auto-discovered extensions so the live 0.1 adapter is not loaded. This does not change live Pi configuration.
|
|
37
37
|
|
|
38
|
-
The source adapter starts only in no-permissions mode: all ten methods are available and the wrapper opens no
|
|
38
|
+
The source adapter starts only in no-permissions mode: all ten methods are available and the wrapper opens no permission UI of its own. The signed app-server runs with Codex Full access, so normal empty-schema app approval elicitations resolve without a prompt. Any elicitation app-server emits is still shown through Pi and answered only by the user. Use benign real apps for acceptance; do not use disposable harnesses as evidence.
|
|
39
39
|
|
|
40
40
|
## Migrate to version 0.2.0
|
|
41
41
|
|
|
@@ -47,7 +47,10 @@ The source adapter starts only in no-permissions mode: all ten methods are avail
|
|
|
47
47
|
6. Start a fresh Pi process; do not rely on hot-reloading a security-boundary change.
|
|
48
48
|
7. Verify `/computer-use-status` reports:
|
|
49
49
|
- `permissionMode: "no-permissions"`;
|
|
50
|
-
- `
|
|
50
|
+
- `wrapperPermissionPrompts: false`;
|
|
51
|
+
- `officialApprovalPolicy: "full-access"`;
|
|
52
|
+
- `officialAppApprovalHandling: "auto-approved-by-codex-full-access"`;
|
|
53
|
+
- `officialElicitationHandling: "forwarded-if-emitted"`;
|
|
51
54
|
- all ten `availableMethods`;
|
|
52
55
|
- `brokerVerified: true`;
|
|
53
56
|
- `nestedModel: false`;
|
|
@@ -60,11 +63,25 @@ Do not copy old safe/full configuration into the new state root. No-permissions
|
|
|
60
63
|
|
|
61
64
|
Version 0.2.0 does not support targeted local Computer Use while the Mac is locked. OpenAI reserves locked use for active trusted turns started from a connected device. See the [locked-screen limitation](README.md#locked-screen-limitation).
|
|
62
65
|
|
|
66
|
+
## Upgrade from version 0.2.0 to 0.3.0
|
|
67
|
+
|
|
68
|
+
The MCP contract and signed execution path do not change. The native Pi extension now registers all ten definitions but initially activates only `computer_use_list_apps` and `computer_use_get_app_state`.
|
|
69
|
+
|
|
70
|
+
1. Upgrade Pi to version 0.80.7 or newer.
|
|
71
|
+
2. Back up `~/.pi/agent/settings.json` and the installed 0.2 package directory.
|
|
72
|
+
3. Verify that npm resolves `codex-computer-use-mcp@0.3.0` exactly, then install that exact package.
|
|
73
|
+
4. Start a fresh Pi process.
|
|
74
|
+
5. Verify that the two inspection tools are initially active and the eight interaction tools remain registered but inactive.
|
|
75
|
+
6. Call `computer_use_get_app_state` against a benign app while the Mac is unlocked. A successful result should add all eight interaction tools without removing any active tool.
|
|
76
|
+
7. Exercise a benign interaction, inspect the private content-safe audit, and verify process cleanup.
|
|
77
|
+
|
|
78
|
+
The change is context disclosure, not a permission mode. Codex Full access, emitted elicitation forwarding, schemas, and the no-permissions policy remain unchanged.
|
|
79
|
+
|
|
63
80
|
## Rollback
|
|
64
81
|
|
|
65
82
|
1. Stop the current Pi process.
|
|
66
83
|
2. Confirm no direct app-server/client/focus/lock process remains.
|
|
67
|
-
3. Remove or disable the
|
|
84
|
+
3. Remove or disable the current package registration.
|
|
68
85
|
4. Restore the backed-up 0.1 package/config registration byte-for-byte.
|
|
69
86
|
5. Start a fresh Pi process.
|
|
70
87
|
6. Verify the old status command and installed hash.
|
|
@@ -74,4 +91,4 @@ Direct state can be removed only after rollback evidence is captured and no proc
|
|
|
74
91
|
|
|
75
92
|
## Generic MCP gateway
|
|
76
93
|
|
|
77
|
-
If Pi uses `mcp.json`, retain `directTools: false`. Use the exact `0.
|
|
94
|
+
If Pi uses `mcp.json`, retain `directTools: false`. Use the exact `0.3.0` package shown in `integrations/pi/mcp.json.example`. During source acceptance, use a distinct temporary server name and source path, then remove it. The direct Pi adapter is the primary live path; do not leave the version 0.1 aggregate server active after the switch.
|
package/PROOF.md
CHANGED
|
@@ -56,7 +56,7 @@ Current branch tests cover:
|
|
|
56
56
|
- production app-server arguments disable model transport, plugins, and remote control;
|
|
57
57
|
- durable no-permissions as the sole policy, with no config/environment/command/tool override;
|
|
58
58
|
- all ten methods available without wrapper prompts or mode gates;
|
|
59
|
-
-
|
|
59
|
+
- exact `thread/start` Full access parameters (`approvalPolicy: "never"`, `sandbox: "danger-full-access"`);
|
|
60
60
|
- fatal rejection of model-turn notifications, including a notification emitted during teardown;
|
|
61
61
|
- strict `ephemeral: true` / `path: null` / `turns: []` response attestation before dispatch;
|
|
62
62
|
- pre-buffer rejection of an oversized unterminated protocol line;
|
|
@@ -71,7 +71,21 @@ Current branch tests cover:
|
|
|
71
71
|
- global per-user same-app exclusion across different supported state roots, race behavior, crash release, private lock roots, and bounded lock filenames;
|
|
72
72
|
- focus-event ASN retry/drain behavior, including valid events at the start of a single large stdout chunk;
|
|
73
73
|
- stdio MCP all-ten inventory/status with no alternate mode route;
|
|
74
|
-
-
|
|
74
|
+
- signed app-server elicitation forwarding, exact user responses, headless cancellation, and cancellation while a UI callback is pending;
|
|
75
|
+
- standard form/URL forwarding across a real MCP SDK client/server transport;
|
|
76
|
+
- Pi source registration for every direct capability with no nested planner, permission command, or wrapper-generated approval UI;
|
|
77
|
+
- Pi session-start disclosure of only the two inspection definitions while preserving unrelated active tools;
|
|
78
|
+
- purely additive activation of all eight interaction definitions after successful `get_app_state`, with no lazy-tool prompt metadata;
|
|
79
|
+
- Pi form, opaque OpenAI-form, URL, decline, and headless-cancel elicitation handling.
|
|
80
|
+
|
|
81
|
+
## Official Full access approval probe
|
|
82
|
+
|
|
83
|
+
A live differential used Chess (`com.apple.Chess`), which was absent from the signed service's persistent per-bundle approval file before and after both calls:
|
|
84
|
+
|
|
85
|
+
1. `approvalPolicy: "never"` with `sandbox: "read-only"` returned `Computer Use approval denied`; app-server never emitted the service's empty-schema approval request to the bridge callback.
|
|
86
|
+
2. Changing only the thread sandbox to `danger-full-access` made the same `get_app_state` call succeed. The bridge callback count and broker `elicitationRequests` were both zero, the runtime remained zero-turn and ephemeral, cleanup verified, and a before/after hash confirmed the persistent approval file was unchanged.
|
|
87
|
+
|
|
88
|
+
This matches the pinned Codex source: `danger-full-access` maps to a disabled permission profile, and `Never` plus that profile auto-accepts empty-schema MCP confirmation elicitations inside the official host. It proves prompt-free request authorization without editing persistent app approvals or inventing a bridge response.
|
|
75
89
|
|
|
76
90
|
## Fresh Pi unlocked real-app acceptance
|
|
77
91
|
|
|
@@ -103,6 +117,19 @@ Evidence:
|
|
|
103
117
|
|
|
104
118
|
Earlier exploratory attempts exposed and fixed two acceptance-quality issues rather than being counted as proof: common `CMD+A` needed normalization to the official `Meta_L+a` key form, and TextEdit state restoration had to be reset before a fresh run. The final evidence above is from the corrected direct path.
|
|
105
119
|
|
|
120
|
+
## Pi 0.80.7 progressive-disclosure measurement
|
|
121
|
+
|
|
122
|
+
A fresh Pi 0.80.7 process loaded only the source Computer Use extension plus a read-only observer. All ten definitions were registered. The initial active Computer Use set was exactly:
|
|
123
|
+
|
|
124
|
+
- `computer_use_list_apps`
|
|
125
|
+
- `computer_use_get_app_state`
|
|
126
|
+
|
|
127
|
+
Serializing names, descriptions, schemas, and prompt guidelines measured 6,063 bytes for all ten definitions and 1,231 bytes for the initial two: 4,832 bytes removed from the initial request, a 79.7% reduction for this tool family.
|
|
128
|
+
|
|
129
|
+
A second fresh process exercised the exported activation path from inside a tool execution. Pi's tool result recorded the exact eight interaction names in `addedToolNames`; the before/after active sets retained both inspection tools and removed zero tools. The lazily activated definitions had no `promptSnippet` or `promptGuidelines`.
|
|
130
|
+
|
|
131
|
+
With `openai-codex/gpt-5.6-sol`, the first request reported 5,723 uncached input tokens. The immediately following request after additive activation reported 1,008 uncached input tokens and 5,632 cache-read tokens. This is live provider evidence that the initial prefix remained reusable while Pi inserted the eight definitions at the tool-result load point. The activation probe used the same `activateInteractionTools` function called after a successful production `get_app_state`; automated tests bind that successful-result branch and exact additive set.
|
|
132
|
+
|
|
106
133
|
## Review and release status
|
|
107
134
|
|
|
108
135
|
The direct implementation at commit `98d26f8040f8035a294caa8581d218a33c076990`, tree `01dd30c899863d182bcb0ac3256bd0cc11efc42b`, passed the immutable exact-head review and rollback-safe activation gates. It became the basis for the version 0.2.0 release.
|
|
@@ -119,10 +146,10 @@ Validation recorded for the reviewed direct implementation:
|
|
|
119
146
|
- public-source scrub: no secrets, private absolute paths, or machine identifiers found;
|
|
120
147
|
- fresh-Pi real-app acceptance: pass as above.
|
|
121
148
|
|
|
122
|
-
Independent reviews of earlier revisions found cleanup and coordination gaps: fail-open or partial descendant enumeration, early-exit orphan recovery, state-root-scoped same-app locking, false-success lease-release audit, unverified focus-listener exit, pre-response direct-call accounting, and large-chunk focus-event loss. The reviewed implementation retains those fixes while removing
|
|
149
|
+
Independent reviews of earlier revisions found cleanup and coordination gaps: fail-open or partial descendant enumeration, early-exit orphan recovery, state-root-scoped same-app locking, false-success lease-release audit, unverified focus-listener exit, pre-response direct-call accounting, and large-chunk focus-event loss. The reviewed implementation retains those fixes while removing wrapper-generated approval UI and safe or full configuration branches. The follow-up Full access change uses Codex's own policy to resolve normal empty-schema app approvals without prompts; any elicitation app-server emits remains a separate, faithfully forwarded user interaction.
|
|
123
150
|
|
|
124
151
|
Version 0.2.0 supports direct local calls only in an unlocked macOS session. Targeted calls failed with official error `-10005` during genuine locked-session acceptance. OpenAI limits locked Computer Use to active trusted turns started from a connected device, so locked local use remains follow-up work and is not part of this release.
|
|
125
152
|
|
|
126
153
|
## Non-goals
|
|
127
154
|
|
|
128
|
-
No private Sky protocol clone, browser-host integration, credential extraction, TCC automation, app injection, re-signing, sender-auth bypass,
|
|
155
|
+
No private Sky protocol clone, browser-host integration, credential extraction, TCC automation, app injection, re-signing, sender-auth bypass, wrapper-side approval fabrication, persistent approval-file mutation, nested model fallback, or tool-result persistence.
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Codex Computer Use MCP
|
|
2
2
|
|
|
3
|
-
Version 0.
|
|
3
|
+
Version 0.3.0 exposes the official signed macOS Computer Use capabilities as direct typed tools for Pi and MCP clients. The calling agent chooses every tool and argument itself.
|
|
4
4
|
|
|
5
5
|
The primary path has:
|
|
6
6
|
|
|
@@ -20,8 +20,8 @@ Pi registers namespaced tools to avoid collisions with Pi's built-ins. The MCP s
|
|
|
20
20
|
|
|
21
21
|
| Pi tool | MCP method | No-permissions | Purpose |
|
|
22
22
|
|---|---|---:|---|
|
|
23
|
-
| `computer_use_list_apps` | `list_apps` | yes | List
|
|
24
|
-
| `computer_use_get_app_state` | `get_app_state` | yes | Read
|
|
23
|
+
| `computer_use_list_apps` | `list_apps` | yes | List running and recently used apps |
|
|
24
|
+
| `computer_use_get_app_state` | `get_app_state` | yes | Read the key-window screenshot and accessibility tree |
|
|
25
25
|
| `computer_use_click` | `click` | yes | Click an element or screenshot coordinates |
|
|
26
26
|
| `computer_use_perform_secondary_action` | `perform_secondary_action` | yes | Invoke a named accessibility action |
|
|
27
27
|
| `computer_use_set_value` | `set_value` | yes | Assign an accessibility value |
|
|
@@ -31,17 +31,20 @@ Pi registers namespaced tools to avoid collisions with Pi's built-ins. The MCP s
|
|
|
31
31
|
| `computer_use_press_key` | `press_key` | yes | Send a key or key combination |
|
|
32
32
|
| `computer_use_type_text` | `type_text` | yes | Type literal text |
|
|
33
33
|
|
|
34
|
-
Pi
|
|
34
|
+
The MCP façade preserves the official signed helper's tool descriptions, input schemas, and annotations exactly. The Pi façade preserves its descriptions and input schemas; Pi's tool API does not expose MCP annotations. In particular, all ten MCP methods have `destructiveHint: false`; the wrapper does not conflate changing UI state with destructive behavior or add argument restrictions absent from the official schemas.
|
|
35
|
+
|
|
36
|
+
Pi 0.80.7 and newer registers all ten definitions but initially activates only `computer_use_list_apps` and `computer_use_get_app_state`. A successful `get_app_state` call additively activates the eight interaction definitions for the following model request. Other active Pi and extension tools remain active. The lazily activated tools omit `promptSnippet` and `promptGuidelines`, allowing Pi's native deferred-loading path to preserve the stable system-prompt prefix.
|
|
37
|
+
|
|
38
|
+
As the official contract specifies, Pi—not a nested planner—calls `computer_use_get_app_state` once per assistant turn before interacting with an app, then chooses and executes actions itself.
|
|
35
39
|
|
|
36
40
|
## Authorization policy: durable no-permissions
|
|
37
41
|
|
|
38
42
|
`no-permissions` has one precise meaning here: **the wrapper asks no permission questions and exposes all ten official actions**. It is the only mode and the durable default. There is no safe/full selector, config file, environment override, slash command, CLI switch, per-call elevation, app allowlist, intent classifier, task schema, per-action confirmation, special-case app policy, or method gate.
|
|
39
43
|
|
|
40
|
-
The app-server runtime
|
|
44
|
+
The app-server runtime uses the official Full access combination: `approvalPolicy: "never"` and `sandbox: "danger-full-access"`. In the pinned Codex host, that maps to a disabled permission profile and automatically accepts empty-schema MCP approval elicitations. Normal Computer Use app-access checks therefore proceed without a per-app prompt, just as they do in Codex Full access. The bridge does not edit the service's persistent per-bundle approval file. If app-server emits an elicitation instead of resolving it under Full access, the bridge still forwards it faithfully to the invoking client; unsupported clients return `cancel`, never a fabricated `accept` or `decline`.
|
|
41
45
|
|
|
42
46
|
No-permissions does **not** bypass:
|
|
43
47
|
|
|
44
|
-
- first-party OpenAI app approvals or sensitive-action prompts;
|
|
45
48
|
- macOS Screen Recording, Accessibility, or TCC controls;
|
|
46
49
|
- strict OpenAI Team ID and code-signature checks;
|
|
47
50
|
- exact upstream ten-tool schema verification;
|
|
@@ -64,23 +67,24 @@ See [`ARCHITECTURE.md`](ARCHITECTURE.md) for source links and the full restricti
|
|
|
64
67
|
|
|
65
68
|
- macOS
|
|
66
69
|
- Node.js 22 or newer
|
|
70
|
+
- Pi 0.80.7 or newer for native progressive tool disclosure
|
|
67
71
|
- official ChatGPT macOS app at `/Applications/ChatGPT.app`
|
|
68
|
-
- official Computer Use component installed
|
|
72
|
+
- official Computer Use component installed
|
|
69
73
|
|
|
70
74
|
The direct bridge starts app-server with a new private `CODEX_HOME` containing no account credentials and only one configured MCP server: official Computer Use. It does not inherit the user's Codex MCP servers, plugins, history, memories, API keys, or auth file. It selects a non-websocket dummy model provider bound to unreachable loopback, disables plugin/remote-control features, and never starts a turn; this prevents app-server model prewarm or Responses API traffic.
|
|
71
75
|
|
|
72
76
|
### Locked-screen limitation
|
|
73
77
|
|
|
74
|
-
Version 0.
|
|
78
|
+
Version 0.3.0 supports direct local calls in an unlocked macOS session. It does not support window or accessibility actions after the Mac locks.
|
|
75
79
|
|
|
76
|
-
OpenAI's [locked Computer Use](https://developers.openai.com/codex/app/computer-use#use-computer-use-while-your-mac-is-locked) is limited to active, trusted ChatGPT turns started from a connected device. It does not authorize other apps or local processes to unlock the Mac. This package uses local zero-turn dispatch, so targeted calls can fail with official error `-10005` while the Mac is locked. Support for locked local use remains a follow-up and is not part of version 0.
|
|
80
|
+
OpenAI's [locked Computer Use](https://developers.openai.com/codex/app/computer-use#use-computer-use-while-your-mac-is-locked) is limited to active, trusted ChatGPT turns started from a connected device. It does not authorize other apps or local processes to unlock the Mac. This package uses local zero-turn dispatch, so targeted calls can fail with official error `-10005` while the Mac is locked. Support for locked local use remains a follow-up and is not part of version 0.3.0.
|
|
77
81
|
|
|
78
82
|
## Pi integration
|
|
79
83
|
|
|
80
84
|
Install the exact release from npm:
|
|
81
85
|
|
|
82
86
|
```bash
|
|
83
|
-
pi install npm:codex-computer-use-mcp@0.
|
|
87
|
+
pi install npm:codex-computer-use-mcp@0.3.0
|
|
84
88
|
```
|
|
85
89
|
|
|
86
90
|
To evaluate a source checkout instead:
|
|
@@ -97,7 +101,7 @@ Command:
|
|
|
97
101
|
/computer-use-status
|
|
98
102
|
```
|
|
99
103
|
|
|
100
|
-
The native Pi adapter is the primary product path. It
|
|
104
|
+
The native Pi adapter is the primary product path. It registers all ten typed tools directly, starts each session with the two inspection tools active, and additively activates the eight interaction tools after successful app inspection. It exposes no mode-changing command or wrapper-generated approval UI. Official app-access approval elicitations are resolved by Codex Full access before they reach Pi. Any elicitation that app-server does emit is shown through Pi's UI, and only the user's choice is returned to the service.
|
|
101
105
|
|
|
102
106
|
## MCP server
|
|
103
107
|
|
|
@@ -123,7 +127,7 @@ For Pi's generic MCP gateway, keep `directTools: false` so this powerful surface
|
|
|
123
127
|
}
|
|
124
128
|
```
|
|
125
129
|
|
|
126
|
-
The generic MCP server exposes the same no-permissions behavior: no wrapper
|
|
130
|
+
The generic MCP server exposes the same no-permissions behavior: official Codex Full access, no wrapper permission gate, and all ten methods. App-access approvals are resolved inside the official host. Any standard form or URL elicitation that app-server emits is forwarded as an MCP `elicitation/create` request. The upstream client response is returned unchanged; unsupported or headless clients cancel rather than fabricate a decision.
|
|
127
131
|
|
|
128
132
|
## Security and privacy
|
|
129
133
|
|
package/SECURITY.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
## Reporting a vulnerability
|
|
4
4
|
|
|
5
|
-
Do not open a public issue for a vulnerability that could weaken signing,
|
|
5
|
+
Do not open a public issue for a vulnerability that could weaken signing, contract verification, no-permissions dispatch, first-party access handling, app identity, locking, focus telemetry, cleanup, or audit integrity.
|
|
6
6
|
|
|
7
7
|
Use GitHub private vulnerability reporting. Include the affected commit/version, the smallest non-sensitive reproduction, expected versus observed behavior, and whether an official action completed before failure.
|
|
8
8
|
|
|
@@ -20,10 +20,10 @@ This is direct tool dispatch—not model orchestration.
|
|
|
20
20
|
|
|
21
21
|
1. Only fixed app-bundled Codex and Computer Use client paths are allowed in production.
|
|
22
22
|
2. Both binaries pass strict code-signature verification and OpenAI Team ID `2DC432GLL2` checks before dispatch.
|
|
23
|
-
3. The helper's exact ten method names and input schemas match the pinned expected inventory before every call.
|
|
23
|
+
3. The helper's exact ten method names and input schemas match the pinned expected inventory before every call; release tests also verify its descriptions and annotations.
|
|
24
24
|
4. `no-permissions` is the only wrapper policy: all ten methods are exposed and no wrapper permission prompt is opened.
|
|
25
25
|
5. There is no config file, environment override, command, tool argument, per-call branch, or alternate safe/full route that an agent can select.
|
|
26
|
-
6. App-server uses `approvalPolicy: "never"
|
|
26
|
+
6. App-server uses the official Full access combination, `approvalPolicy: "never"` plus `sandbox: "danger-full-access"`. The pinned Codex host automatically accepts empty-schema MCP approval elicitations, so normal first-party app-access checks proceed without prompts. The wrapper does not synthesize this response or edit persistent per-app approvals. Any elicitation app-server emits is forwarded faithfully; an unavailable client cancels.
|
|
27
27
|
7. Target selectors resolve to canonical installed bundle IDs before dispatch.
|
|
28
28
|
8. Same-app work is excluded across native Pi, generic MCP, and custom state roots with one fixed per-user kernel `lockf` lease namespace.
|
|
29
29
|
9. Target focus events, periodic samples, watcher health, queued ASN resolution, and final state are checked. This is post-action detection, not a preventive OS sandbox.
|
|
@@ -40,20 +40,22 @@ This is direct tool dispatch—not model orchestration.
|
|
|
40
40
|
|
|
41
41
|
### No-permissions
|
|
42
42
|
|
|
43
|
-
All ten official methods and arbitrary resolvable app targets are available without wrapper permission prompts. The name means “ask no permission,” not “disable tools.” This is broad authorization. It does not imply that actions are reversible or that focus detection can prevent an already dispatched action.
|
|
43
|
+
All ten official methods and arbitrary resolvable app targets are available without wrapper permission prompts. The name means “ask no permission,” not “disable tools.” This is broad authorization. It does not imply that actions are reversible or that focus detection can prevent an already dispatched action.
|
|
44
44
|
|
|
45
45
|
The mode is compiled as the sole policy. Agent-writable audit state and legacy configuration files cannot change it, and there is no CLI/slash/tool/environment mode selector.
|
|
46
46
|
|
|
47
47
|
## Elicitations
|
|
48
48
|
|
|
49
|
-
|
|
49
|
+
The bridge starts app-server with `approvalPolicy: "never"` and `sandbox: "danger-full-access"`. This is Codex's Full access policy, not a wrapper-side approximation. Codex 0.144.2 maps the sandbox value to a disabled permission profile and automatically accepts empty-schema MCP approval elicitations before they are emitted to the client. That covers normal Computer Use app-access prompts without mutating the service's persistent approval file.
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+
If app-server emits a form, OpenAI-form, or URL elicitation, Pi renders it and stdio MCP forwards supported standard modes through `elicitation/create`. The wrapper preserves the user/client response. Missing UI or an unsupported client produces `cancel`, never a fabricated decision.
|
|
52
52
|
|
|
53
|
-
|
|
53
|
+
## Visible content
|
|
54
|
+
|
|
55
|
+
Computer Use returns the official app-state text and screenshots to the invoking client. The wrapper does not inspect, classify, redact, or use that content to narrow the actions authorized by the user's request, and it never writes the content to its audit.
|
|
54
56
|
|
|
55
57
|
## Supported versions
|
|
56
58
|
|
|
57
|
-
Only the latest approved release is supported. Version 0.
|
|
59
|
+
Only the latest approved release is supported. Version 0.3.0 supports direct local calls in an unlocked macOS session. Targeted local calls while the Mac is locked are not supported.
|
|
58
60
|
|
|
59
61
|
App-server is experimental and bundle paths or schemas can change. Drift fails closed until reviewed.
|
package/dist/audit.d.ts
CHANGED
|
@@ -16,7 +16,7 @@ export interface AuditRecord {
|
|
|
16
16
|
directCalls: number;
|
|
17
17
|
modelTurnsStarted: number;
|
|
18
18
|
ephemeralThread: boolean | null;
|
|
19
|
-
|
|
19
|
+
elicitationRequests: number;
|
|
20
20
|
backgroundPreserved: boolean | null;
|
|
21
21
|
brokerCleanupVerified: boolean;
|
|
22
22
|
appLeaseReleased: boolean;
|
package/dist/direct-broker.d.ts
CHANGED
|
@@ -2,6 +2,18 @@ import { type DirectMethod, type DirectToolArguments } from "./tools.ts";
|
|
|
2
2
|
export declare const CODEX_PATH = "/Applications/ChatGPT.app/Contents/Resources/codex";
|
|
3
3
|
export declare const COMPUTER_USE_PLUGIN_ROOT = "/Applications/ChatGPT.app/Contents/Resources/plugins/openai-bundled/plugins/computer-use";
|
|
4
4
|
export declare const COMPUTER_USE_CLIENT_PATH = "/Applications/ChatGPT.app/Contents/Resources/plugins/openai-bundled/plugins/computer-use/Codex Computer Use.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient";
|
|
5
|
+
export interface DirectBrokerElicitationRequest extends Record<string, unknown> {
|
|
6
|
+
mode?: string;
|
|
7
|
+
message?: string;
|
|
8
|
+
requestedSchema?: unknown;
|
|
9
|
+
url?: string;
|
|
10
|
+
elicitationId?: string;
|
|
11
|
+
}
|
|
12
|
+
export interface DirectBrokerElicitationResponse {
|
|
13
|
+
action: "accept" | "decline" | "cancel";
|
|
14
|
+
content?: unknown;
|
|
15
|
+
_meta?: unknown;
|
|
16
|
+
}
|
|
5
17
|
export interface DirectBrokerResult {
|
|
6
18
|
content: Array<Record<string, unknown>>;
|
|
7
19
|
structuredContent?: unknown;
|
|
@@ -9,7 +21,7 @@ export interface DirectBrokerResult {
|
|
|
9
21
|
brokerVersion: string;
|
|
10
22
|
clientBuild: string;
|
|
11
23
|
durationMs: number;
|
|
12
|
-
|
|
24
|
+
elicitationRequests: number;
|
|
13
25
|
modelTurnsStarted: 0;
|
|
14
26
|
ephemeralThread: true;
|
|
15
27
|
brokerCleanupVerified: true;
|
|
@@ -17,6 +29,8 @@ export interface DirectBrokerResult {
|
|
|
17
29
|
export interface DirectBrokerOptions {
|
|
18
30
|
timeoutMs?: number;
|
|
19
31
|
signal?: AbortSignal;
|
|
32
|
+
onElicitation?: (request: DirectBrokerElicitationRequest) => DirectBrokerElicitationResponse | Promise<DirectBrokerElicitationResponse>;
|
|
33
|
+
supportsOpenAiFormElicitation?: boolean;
|
|
20
34
|
/** Test-only executable override. Production callers never set this. */
|
|
21
35
|
appServerCommand?: string;
|
|
22
36
|
/** Test-only argument override. */
|
|
@@ -37,14 +51,14 @@ export declare class DirectBrokerCallError extends Error {
|
|
|
37
51
|
readonly directCalls: number;
|
|
38
52
|
readonly modelTurnsStarted: number;
|
|
39
53
|
readonly ephemeralThread: boolean;
|
|
40
|
-
readonly
|
|
54
|
+
readonly elicitationRequests: number;
|
|
41
55
|
readonly brokerVersion: string;
|
|
42
56
|
readonly clientBuild: string;
|
|
43
57
|
constructor(message: string, cleanupVerified: boolean, cause?: unknown, evidence?: {
|
|
44
58
|
directCalls?: number;
|
|
45
59
|
modelTurnsStarted?: number;
|
|
46
60
|
ephemeralThread?: boolean;
|
|
47
|
-
|
|
61
|
+
elicitationRequests?: number;
|
|
48
62
|
brokerVersion?: string;
|
|
49
63
|
clientBuild?: string;
|
|
50
64
|
});
|
package/dist/direct-broker.js
CHANGED
|
@@ -20,7 +20,7 @@ export class DirectBrokerCallError extends Error {
|
|
|
20
20
|
directCalls;
|
|
21
21
|
modelTurnsStarted;
|
|
22
22
|
ephemeralThread;
|
|
23
|
-
|
|
23
|
+
elicitationRequests;
|
|
24
24
|
brokerVersion;
|
|
25
25
|
clientBuild;
|
|
26
26
|
constructor(message, cleanupVerified, cause, evidence = {}) {
|
|
@@ -30,7 +30,7 @@ export class DirectBrokerCallError extends Error {
|
|
|
30
30
|
this.directCalls = evidence.directCalls ?? 0;
|
|
31
31
|
this.modelTurnsStarted = evidence.modelTurnsStarted ?? 0;
|
|
32
32
|
this.ephemeralThread = evidence.ephemeralThread ?? false;
|
|
33
|
-
this.
|
|
33
|
+
this.elicitationRequests = evidence.elicitationRequests ?? 0;
|
|
34
34
|
this.brokerVersion = evidence.brokerVersion ?? "unknown";
|
|
35
35
|
this.clientBuild = evidence.clientBuild ?? "unknown";
|
|
36
36
|
}
|
|
@@ -391,7 +391,7 @@ export async function callOfficialDirectTool(method, args, options = {}) {
|
|
|
391
391
|
let fatalError;
|
|
392
392
|
let stderr = "";
|
|
393
393
|
let nextId = 1;
|
|
394
|
-
let
|
|
394
|
+
let elicitationRequests = 0;
|
|
395
395
|
let modelTurnsStarted = 0;
|
|
396
396
|
let directCalls = 0;
|
|
397
397
|
let ephemeralThread = false;
|
|
@@ -500,14 +500,26 @@ export async function callOfficialDirectTool(method, args, options = {}) {
|
|
|
500
500
|
send({ id: message.id, error: { code: -32601, message: "Unsupported server request" } });
|
|
501
501
|
return;
|
|
502
502
|
}
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
503
|
+
elicitationRequests += 1;
|
|
504
|
+
const requestParams = message.params && typeof message.params === "object" && !Array.isArray(message.params)
|
|
505
|
+
? message.params
|
|
506
|
+
: {};
|
|
507
|
+
void (async () => {
|
|
508
|
+
let response = { action: "cancel" };
|
|
509
|
+
if (options.onElicitation) {
|
|
510
|
+
try {
|
|
511
|
+
const candidate = await options.onElicitation(requestParams);
|
|
512
|
+
if (candidate && ["accept", "decline", "cancel"].includes(candidate.action))
|
|
513
|
+
response = candidate;
|
|
514
|
+
}
|
|
515
|
+
catch {
|
|
516
|
+
response = { action: "cancel" };
|
|
517
|
+
}
|
|
518
|
+
}
|
|
519
|
+
if (fatalError || !proc?.stdin.writable)
|
|
520
|
+
return;
|
|
521
|
+
send({ id: message.id, result: response });
|
|
522
|
+
})().catch((error) => fail(error instanceof Error ? error : new Error(String(error))));
|
|
511
523
|
return;
|
|
512
524
|
}
|
|
513
525
|
}
|
|
@@ -555,11 +567,13 @@ export async function callOfficialDirectTool(method, args, options = {}) {
|
|
|
555
567
|
if (fatalError)
|
|
556
568
|
throw fatalError;
|
|
557
569
|
await request("initialize", {
|
|
558
|
-
clientInfo: { name: "pi_direct_computer_use", title: "Pi Direct Computer Use", version: "0.
|
|
559
|
-
capabilities: { mcpServerOpenaiFormElicitation:
|
|
570
|
+
clientInfo: { name: "pi_direct_computer_use", title: "Pi Direct Computer Use", version: "0.3.0" },
|
|
571
|
+
capabilities: { mcpServerOpenaiFormElicitation: options.supportsOpenAiFormElicitation === true && options.onElicitation !== undefined },
|
|
560
572
|
}, 15_000);
|
|
561
573
|
send({ method: "initialized" });
|
|
562
|
-
|
|
574
|
+
// Codex Full access requires both values: with `never` plus a restricted sandbox,
|
|
575
|
+
// core declines the signed service's app approval instead of prompting or accepting it.
|
|
576
|
+
const started = (await request("thread/start", { cwd: workDir, approvalPolicy: "never", sandbox: "danger-full-access", ephemeral: true, serviceName: "pi_direct_computer_use" }, 30_000));
|
|
563
577
|
const thread = started?.thread;
|
|
564
578
|
const threadId = thread?.id;
|
|
565
579
|
if (typeof threadId !== "string"
|
|
@@ -583,7 +597,7 @@ export async function callOfficialDirectTool(method, args, options = {}) {
|
|
|
583
597
|
brokerVersion: verification.brokerVersion,
|
|
584
598
|
clientBuild: verification.clientBuild,
|
|
585
599
|
durationMs: Date.now() - startedAt,
|
|
586
|
-
|
|
600
|
+
elicitationRequests,
|
|
587
601
|
modelTurnsStarted: 0,
|
|
588
602
|
ephemeralThread: true,
|
|
589
603
|
brokerCleanupVerified: true,
|
|
@@ -625,7 +639,7 @@ export async function callOfficialDirectTool(method, args, options = {}) {
|
|
|
625
639
|
directCalls,
|
|
626
640
|
modelTurnsStarted,
|
|
627
641
|
ephemeralThread,
|
|
628
|
-
|
|
642
|
+
elicitationRequests,
|
|
629
643
|
brokerVersion: verification.brokerVersion,
|
|
630
644
|
clientBuild: verification.clientBuild,
|
|
631
645
|
};
|