codex-chatgpt-control 0.1.0-alpha.1

package/references/backend-protocol.md

@@ -0,0 +1,238 @@
+# Backend Protocol
+
+The local backend is a long-lived language-neutral service. The initial implementation is Node/TypeScript, exposed over stdio NDJSON.
+
+This is the contract that keeps Node and Python deeply in sync:
+
+```text
+Node in-process SDK
+Node backend client
+Python SDK
+Future SDKs
+  -> backend protocol
+  -> compatible browser-control backend
+  -> browser bridge
+  -> chatgpt.com
+```
+
+The current live backend is Node-backed. A future Python-native backend can replace it by implementing this protocol and passing the same contract fixtures and smoke gates.
+
+## Stdio Transport
+
+Each request is one JSON line on stdin. Backend stdout is reserved for protocol JSON only; diagnostics go to stderr.
+
+```json
+{
+  "schemaVersion": "chatgpt.browser_control.backend_request.v1",
+  "requestId": "req_1",
+  "command": "backend.health",
+  "payload": {}
+}
+```
+
+Each non-streaming response is one JSON line:
+
+```json
+{
+  "schemaVersion": "chatgpt.browser_control.backend_response.v1",
+  "requestId": "req_1",
+  "ok": true,
+  "result": {
+    "ok": true,
+    "status": "ok"
+  }
+}
+```
+
+Protocol errors use the same response envelope:
+
+```json
+{
+  "schemaVersion": "chatgpt.browser_control.backend_response.v1",
+  "requestId": "req_1",
+  "ok": false,
+  "error": {
+    "code": "unknown_command",
+    "message": "Unknown backend command: runner.nope",
+    "recoverable": false
+  }
+}
+```
+
+Current protocol error codes are:
+
+- `invalid_request`
+- `unsupported_schema_version`
+- `unknown_command`
+
+Browser-control blockers are not protocol errors. They are normal command or runner results with `status: "blocked"`, `status: "partial"`, or `status: "needs_confirmation"` plus blocker/interruption details.
+
+## Streaming
+
+Streaming commands emit backend event lines until `completed` or `error`.
+
+```json
+{
+  "schemaVersion": "chatgpt.browser_control.backend_event.v1",
+  "requestId": "req_stream",
+  "type": "run_item_stream_event",
+  "name": "message_completed",
+  "item": {
+    "type": "message.completed"
+  }
+}
+```
+
+The final event contains a normal runner result:
+
+```json
+{
+  "schemaVersion": "chatgpt.browser_control.backend_event.v1",
+  "requestId": "req_stream",
+  "type": "completed",
+  "result": {
+    "status": "ok",
+    "output_text": "hi"
+  }
+}
+```
+
+Streaming is milestone streaming only. It does not promise token deltas or OpenAI API stream-event parity.
+
+## Required Backend Commands
+
+The backend must support:
+
+- lifecycle: `backend.version`, `backend.health`, `backend.capabilities`
+- runner: `runner.run`, `runner.plan`, `runner.stream`
+- Responses adapter: `responses.create`
+- workflows: `ask`, `askInThread`, `askWithFiles`, `askAndDownload`, `runMessages`, `openThread`, `runPlan`
+- reports: `createReport`, `reports.create`, `reports.redact`, `reports.summarize`
+- command discovery: `commands`, `describe`, `help`
+- primitives: `session.bootstrap`, `threads.*`, `messages.*`, `files.*`, `modes.set`, `tools.select`, `response.copy`
+
+`session.bootstrap` accepts `existingTab` for explicit reuse of a user-open Chrome tab before any read or prompt step. The wire shape is shared by TypeScript and Python:
+
+```json
+{
+  "existingTab": {
+    "target": { "type": "selected", "host": "chatgpt" },
+    "ifMissing": "block"
+  }
+}
+```
+
+Other supported targets are `{ "type": "url", "url": "https://chatgpt.com/c/..." }`, `{ "type": "conversationId", "conversationId": "..." }`, and `{ "type": "tabId", "tabId": "..." }`. Explicit existing-tab reuse blocks by default when no matching tab is open. `ifMissing: "open"` may open URL or conversation-id targets, but selected-tab and tab-id targets remain claim-only because there is no deterministic URL to create.
+
+`backend.capabilities` is the source of truth for supported commands, transports, and stream modes. The current backend advertises:
+
+```json
+{
+  "protocolVersion": "chatgpt.browser_control.backend_request.v1",
+  "transports": ["stdio"],
+  "streaming": {
+    "modes": ["ndjson"],
+    "tokenDeltas": false
+  }
+}
+```
+
+## HTTP/SSE Status
+
+HTTP/SSE is deferred in this phase. Stdio NDJSON is the default long-lived local transport because it has no port allocation, no browser-origin surface, and no local network security prompt. It also covers the current streaming requirement through backend event lines.
+
+Any future HTTP/SSE implementation must use the same request, response, event, capabilities, fixture, and conformance shapes. It should add `http` and `sse` capabilities only after transport-specific tests pass.
+
+## Runtime Boundary
+
+Python is a native SDK facade over the protocol. The current browser-control runtime is still Node-backed.
+
+An ordinary shell can launch:
+
+```bash
+node ../node/dist/codex-chatgpt-control-backend.mjs
+```
+
+That is enough to validate protocol shape, command dispatch, contracts, and blocker handling. It is not enough to guarantee live ChatGPT control, because a plain subprocess does not automatically inherit Codex's JavaScript `globalThis.agent` browser bridge.
+
+For live browser control, the backend process must have access to a compatible browser bridge through one of the backend runtime options:
+
+- Codex-hosted JavaScript runtime with `globalThis.agent`.
+- Explicit `RuntimeEnv.browser` or `RuntimeEnv.page` in a future embedding.
+- A future Python-native/native-host/CDP backend that implements this same protocol.
+
+Important: in Codex, `globalThis.agent` is not present until the Chrome plugin runtime is bootstrapped. Do not diagnose bridge availability by checking `globalThis.agent` in an ordinary shell or before calling the Chrome plugin's `setupBrowserRuntime({ globals: globalThis })`.
+
+The live Chrome bootstrap is:
+
+```js
+const { setupBrowserRuntime } = await import("/example/user/.codex/plugins/cache/openai-bundled/chrome/26.602.40724/scripts/browser-client.mjs");
+await setupBrowserRuntime({ globals: globalThis });
+globalThis.browser = await agent.browsers.get("extension");
+```
+
+## Ordinary-Shell Smoke
+
+Run from `packages/node`:
+
+```bash
+npm run bundle:backend
+```
+
+Run from `packages/python`:
+
+```bash
+python scripts/live_smoke.py --mode ordinary-shell
+```
+
+In an ordinary shell without Codex browser bridge access, browser-required commands must return a structured `browser_bridge_unavailable` blocker. This is a successful smoke result when the backend process stays alive and protocol calls such as `backend.health` and `commands` succeed.
+
+## Browser-Bridge Smoke
+
+Run only when intentionally operating a backend with live browser access:
+
+```bash
+python scripts/live_smoke.py --mode browser-bridge
+```
+
+Use `CHATGPT_BROWSER_BACKEND_COMMAND` or `--backend-command` when the bridge-enabled backend is not the default bundle:
+
+```bash
+CHATGPT_BROWSER_BACKEND_COMMAND="node /absolute/path/to/bridge-enabled-backend.mjs" \
+python scripts/live_smoke.py --mode browser-bridge
+```
+
+When the bridge-enabled backend is running inside an active Codex Chrome-plugin JS execution rather than a standalone process, run Python through the stdio relay:
+
+```bash
+CHATGPT_BROWSER_BACKEND_HTTP_URL=http://127.0.0.1:<relay-port> \
+python scripts/live_smoke.py \
+  --mode browser-bridge \
+  --backend-command "node scripts/http_stdio_relay.mjs"
+```
+
+The bridge-hosted JS execution must remain active for the duration of the Python smoke. The relay path is:
+
+```text
+Python SDK -> stdio relay -> bridge-hosted Node backend -> Codex Chrome bridge -> ChatGPT
+```
+
+The smoke covers `runner.run`, `runner.run_streamed`, `responses.create`, named `run_plan`, and redacted `reports.create`. It writes redacted JSON summaries and does not persist raw prompt/response content by default.
+
+## Contract Fixtures
+
+Shared fixtures live under:
+
+```text
+contracts/v1/
+```
+
+Required gates:
+
+```bash
+npm run contract:validate
+npm run parity:fixtures
+npm run test:backend-conformance
+```
+
+Python must also load and round-trip the same fixtures through Pydantic models. Any future backend implementation should pass these fixtures before claiming compatibility.

package/references/python-parity.md

@@ -0,0 +1,168 @@
+# Python Parity
+
+The Python package is a parity client over the TypeScript browser-control runtime. Keep browser automation owned by `packages/node/`; keep Python synchronized through the versioned wire contract in `contracts/v1/`.
+
+## Contract
+
+- Shared fixtures live in `contracts/v1/fixtures/`.
+- `npm run contract:validate` validates every fixture against JSON Schema.
+- `npm run parity:fixtures` enforces fixture shape, stream settlement, and wire-field casing.
+- `npm run parity:suite` validates `contracts/v1/parity-suite.json`, which ties every public backend command and fixture to TypeScript tests, Python tests, docs, and deterministic CI gates.
+- Python tests load the same manifest and round-trip every JSON fixture through Pydantic models.
+
+Wire fields stay TypeScript-compatible. Python exposes idiomatic aliases:
+
+| Wire | Python |
+| --- | --- |
+| `finalOutput` | `final_output` |
+| `newItems` | `new_items` |
+| `activeAgentName` | `active_agent_name` |
+| `lastAgentName` | `last_agent_name` |
+| `nextStepId` | `next_step_id` |
+
+## Sync Python
+
+```python
+from codex_chatgpt_control import ChatGPT, NodeSidecarTransport
+
+chatgpt = ChatGPT(
+    transport=NodeSidecarTransport(
+        command=["node", "dist/codex-chatgpt-control-backend.mjs"]
+    )
+)
+agent = chatgpt.agent(name="reviewer", instructions="Review deeply.")
+result = chatgpt.runner.run(agent, input="Reply with hi.")
+
+print(result.output_text)
+print(result.final_output)
+```
+
+## Async Python
+
+```python
+from codex_chatgpt_control import AsyncChatGPT
+
+chatgpt = AsyncChatGPT(transport=my_async_transport)
+agent = chatgpt.agent(name="reviewer", instructions="Review deeply.")
+result = await chatgpt.runner.run(agent, input="Reply with hi.")
+```
+
+## Responses Fixture Shape
+
+```python
+from codex_chatgpt_control import ChatGPTResponse
+
+response = ChatGPTResponse.from_wire(payload["response"])
+unsupported = response.unsupported_fields
+```
+
+Unsupported OpenAI API-only fields stay explicit in `browser_control.unsupported[]`; the Python adapter must not submit them silently.
+
+## Streaming
+
+`stream-*.ndjson` fixtures are milestone streams. They are not token streams. The final `completed` event contains a normal `ChatGPTRunResult` wire object, including blockers when the run cannot proceed.
+
+```python
+from codex_chatgpt_control import ChatGPTStreamEvent
+
+event = ChatGPTStreamEvent.from_wire(payload)
+if event.type == "completed" and event.result is not None:
+    print(event.result.status)
+```
+
+## Required Gates
+
+Run from `packages/node`:
+
+```bash
+npm run bundle:backend
+npm run contract:validate
+npm run parity:fixtures
+npm run parity:suite
+npm run test:backend-conformance
+npm test -- tests/unit/contract-fixtures.test.ts
+```
+
+Run from `packages/python`:
+
+```bash
+python -m pip install -e ".[dev]"
+python -m unittest discover -s tests
+python -m compileall -q src
+python -m pyright --pythonpath "$(which python)" src tests
+python scripts/live_smoke.py --mode ordinary-shell
+```
+
+## Backend Runtime
+
+Python is a native SDK facade over the local backend protocol. The initial browser-control runtime is still the TypeScript backend service:
+
+- `dist/codex-chatgpt-control-backend.mjs` is the stdio backend bundle.
+- `BackendClient` and `StdioBackendTransport` keep Python backend calls long-lived.
+- `NodeSidecarTransport.run(...)` remains as a compatibility wrapper over backend `runner.run`.
+- Ordinary-shell smoke passes when browser-required calls return structured `browser_bridge_unavailable`.
+- Browser-bridge runtime smoke remains explicitly gated because it can operate a real ChatGPT session.
+
+## Browser-Bridge Smoke
+
+Run this only when you intentionally want Python to drive a live backend with browser access:
+
+```bash
+python scripts/live_smoke.py --mode browser-bridge
+```
+
+The command covers:
+
+- `runner.run` new ask/read.
+- `runner.run_streamed` milestone streaming.
+- `responses.create` basic.
+- `run_plan` named `new-ask-read`.
+- `reports.create` redacted report output.
+
+The default backend command is the Node stdio bundle:
+
+```text
+../node/dist/codex-chatgpt-control-backend.mjs
+```
+
+That default is enough for protocol validation and structured blockers, but a plain subprocess cannot inherit Codex's JavaScript `globalThis.agent` browser bridge. For a true live browser pass, point Python at a stdio backend command that already runs in a bridge-enabled host:
+
+```bash
+CHATGPT_BROWSER_BACKEND_COMMAND="node /absolute/path/to/bridge-enabled-backend.mjs" \
+python scripts/live_smoke.py --mode browser-bridge
+```
+
+### Codex Chrome Plugin Relay
+
+When the live backend is hosted inside the Codex Chrome plugin runtime, do not test bridge availability from a normal shell or an unbootstrapped Node REPL. First initialize the Chrome runtime:
+
+```js
+const { setupBrowserRuntime } = await import("/example/user/.codex/plugins/cache/openai-bundled/chrome/26.602.40724/scripts/browser-client.mjs");
+await setupBrowserRuntime({ globals: globalThis });
+globalThis.browser = await agent.browsers.get("extension");
+```
+
+Then run the backend server inside that active JS execution context and point Python at the stdio-to-HTTP relay:
+
+```bash
+CHATGPT_BROWSER_BACKEND_HTTP_URL=http://127.0.0.1:<relay-port> \
+python scripts/live_smoke.py \
+  --mode browser-bridge \
+  --backend-command "node scripts/http_stdio_relay.mjs"
+```
+
+Keep the bridge-hosted JS execution active while Python runs. If that JS execution returns first, the browser client no longer has an active execution context and operations can fail with `node_repl exec context not found`.
+
+This is the intended live test chain:
+
+```text
+Python SDK -> scripts/http_stdio_relay.mjs -> bridge-hosted Node backend -> Codex Chrome bridge -> ChatGPT
+```
+
+Smoke output is a redacted JSON summary. It reports output matches and lengths, not raw prompts or raw responses. Exit codes are:
+
+| Code | Meaning |
+| --- | --- |
+| `0` | All browser-bridge scenarios passed. |
+| `1` | At least one scenario failed unexpectedly. |
+| `2` | Scenarios recorded documented blockers such as `browser_bridge_unavailable`, `login_required`, or `selector_drift`. |

package/references/responses-adapter.md

@@ -0,0 +1,28 @@
+# Responses Adapter
+
+`chatgpt.responses.create()` is a narrow convenience wrapper around visible ChatGPT browser control. It returns a Responses-shaped object with `object: "chatgpt.browser.response"`, but it is not the OpenAI Responses API and does not support hidden model controls.
+
+Accepted fields:
+
+- `input`
+- `thread`
+- `attachments`
+- `mode`
+- `tools`
+- `text.format`
+- `stream: false`
+- `report`
+- `instructions` only with `instructionsMode: "visible_prefix"`
+
+Rejected API-only fields return `status: "unsupported"` before any prompt is submitted. The response includes `browser_control.unsupported[]` with `path`, `reason`, and `alternative`.
+
+```ts
+const response = await chatgpt.responses.create({
+  input: "Summarize the latest plan.",
+  thread: { type: "conversationId", conversationId: "abc-123" },
+  text: { format: "markdown" },
+  stream: false
+});
+```
+
+Use `chatgpt.runner.run()` for lower-level browser-control workflows, multi-step command planning, attachments, downloads, reports, and explicit interruption handling.

package/references/safety.md

@@ -0,0 +1,14 @@
+# Safety Rules
+
+- Treat ChatGPT conversation content as private unless the user explicitly says otherwise.
+- Do not save raw ChatGPT conversation content to memory.
+- Do not send ChatGPT content to external model tools without explicit user approval.
+- Do not create shared links automatically.
+- Do not delete, archive, move, or share threads without exact confirmation.
+- Do not connect or disconnect ChatGPT apps/connectors automatically.
+- Do not change account settings, memory, custom instructions, data controls, or workspace state automatically.
+- Do not read browser cookies, localStorage, sessionStorage, auth tokens, hidden headers, or private network request internals.
+- Prefer DOM/UI automation and official APIs over private endpoint replay.
+- Runner `instructions` are visible prompt text unless `instructionsMode: "metadata_only"` keeps them local. Do not imply hidden system-message semantics.
+- `chatgpt.responses.create()` is not the OpenAI Responses API. Reject API-only model controls before submitting a prompt.
+- Runner streaming is milestone-only. Do not claim token-level streaming or API stream event parity.

package/references/streaming.md

@@ -0,0 +1,15 @@
+# Runner Streaming
+
+`chatgpt.runner.run(agent, input, { stream: true })` returns an async iterable of runner milestone events plus a `completed` promise.
+
+```ts
+const stream = chatgpt.runner.run(agent, "Reply with hi", { stream: true });
+
+for await (const event of stream) {
+  console.log(event.name, event.item.type);
+}
+
+const result = await stream.completed;
+```
+
+This is not token streaming. Events are emitted for browser-control milestones such as `message_submitted`, `message_completed`, `file_attached`, and `run_blocked`. Do not expect token deltas or OpenAI API stream event parity.

package/references/troubleshooting.md

@@ -0,0 +1,91 @@
+# Troubleshooting
+
+## `browser_bridge_unavailable`
+
+The backend process does not have access to a browser bridge. This is expected when a live-smoke command runs from an ordinary shell: it proves the protocol stayed alive and surfaced a structured blocker.
+
+The structured blocker should include `code: "codex_chrome_bridge_unavailable"` plus `blocker.remediation[]`. Agents should read those remediation steps before asking the user to restart Chrome or change permissions.
+
+Do not conclude that Chrome or the extension is broken from a plain shell result, or from checking `globalThis.agent` before the Chrome plugin runtime is initialized. For a true Codex Chrome-plugin live run, bootstrap the runtime first:
+
+```js
+const { setupBrowserRuntime } = await import("/example/user/.codex/plugins/cache/openai-bundled/chrome/26.602.40724/scripts/browser-client.mjs");
+await setupBrowserRuntime({ globals: globalThis });
+globalThis.browser = await agent.browsers.get("extension");
+```
+
+If the command was intentionally running in a bridge-enabled host and still returns this blocker, verify that the Codex Chrome extension is installed and enabled, then restart Chrome or Codex if the backend is still unavailable. Do not keep retrying the same attach path indefinitely.
+
+For Python live bridge smokes, a plain Python-spawned Node subprocess does not inherit Codex's in-process bridge. Use the relay into an active bridge-hosted backend:
+
+```bash
+CHATGPT_BROWSER_BACKEND_HTTP_URL=http://127.0.0.1:<relay-port> \
+python scripts/live_smoke.py \
+  --mode browser-bridge \
+  --backend-command "node scripts/http_stdio_relay.mjs"
+```
+
+Keep the bridge-hosted JS execution active while Python runs. If that call returns first, browser operations can fail with `node_repl exec context not found`.
+
+## `login_required`
+
+The user needs to sign in to ChatGPT in Chrome. Stop and ask the user to complete login.
+
+## `captcha`
+
+Stop. Do not attempt bypass.
+
+## `rate_limit`
+
+Return the visible limit text and stop unless the user asks to wait or try a different path.
+
+## `selector_drift`
+
+The ChatGPT UI changed or the page loaded an unexpected surface. Return visible menu/button candidates and a screenshot/DOM summary if available.
+
+Runner results surface this as `interruptions[0].type === "selector_drift"` with `blocker.candidates` when visible candidates were available. Do not retry automatically; ask the user or update selectors.
+
+## File Upload Permission
+
+File uploads have two separate permission gates:
+
+1. Codex app settings for Chrome uploads must allow `chatgpt.com`, or uploads must be set to always allow.
+2. Chrome's extension details page for the Codex extension may also need `Allow access to file URLs`.
+
+If `fileChooser.setFiles` returns `Not allowed`, the ChatGPT chooser was reached but Chrome refused the local file handoff. Check both gates before retrying.
+
+Agent-facing remediation text should name both settings:
+
+> File upload is blocked by Chrome/Codex permissions. Ask the user to enable both: Codex Settings > Computer Use > Chrome > Permissions > Uploads, and Chrome chrome://extensions > Codex extension > Details > Allow access to file URLs. Then retry.
+
+## Clipboard Unavailable
+
+`response.copy` falls back to DOM text extraction when the macOS system clipboard does not change.
+
+## Flattened Or Unreadable Response Capture
+
+Use `readLatest({ format: "markdown" })`, `copyLatest()`, or the default SDK `read: true` path for human-readable responses. Use `format: "normalized_text"` only for exact-string assertions or polling. Check `data.source`, `data.fidelity`, and `data.warnings`: clipboard output is highest fidelity, while DOM Markdown is semantic reconstruction. If Markdown capture degrades, return the command warning and save the structured `blocks` or diagnostic `html` representation instead of silently writing flattened text as a Markdown report.
+
+## Response Branch Ambiguity
+
+When ChatGPT exposes previous/next response controls, `readLatest` and `copyLatest` include `branch.current`, `branch.total`, `actions`, and `thoughtDurationText` when visible. If the branch state is missing but the user expects a rerun or edited-message branch, reload the thread and read again before claiming to have captured the latest answer.
+
+## Download Unavailable
+
+The command only downloads visible files with a download affordance. If no download control exists, ask ChatGPT to create or expose the file again.
+
+## Redacted Reports
+
+`createReport`, `chatgpt.reports.*`, SDK workflow reports, and live-smoke reports redact raw prompt/response/file content by default. Use `includeContent: true` only when the user explicitly asks to persist raw content.
+
+## Responses Adapter Unsupported Fields
+
+`chatgpt.responses.create()` rejects OpenAI API-only fields such as `model`, `temperature`, `logprobs`, `previous_response_id`, `store`, and `max_output_tokens` before submitting a prompt. Inspect `browser_control.unsupported[]` for `path`, `reason`, and `alternative`.
+
+## Runner Milestone Streaming
+
+`chatgpt.runner.run(agent, input, { stream: true })` returns milestone events and `stream.completed`. It does not stream token deltas. If an agent expects API stream events, use `event.name` and `event.item.type` instead.
+
+## Doctor Preflight
+
+Run `doctor({ check: ["bridge", "login", "upload", "download", "clipboard"] })` before long workflows when the browser state or permissions are uncertain. Upload readiness may remain `unknown` until a live attach attempt, but the remediation should still name both upload permission gates.