@gemus/mcp-proxy 0.1.6 → 0.1.8

package/README.md

@@ -1,9 +1,10 @@
 # @gemus/mcp-proxy (Issues #1751, #1756, #1917)
 
 Local stdio↔HTTP MCP proxy that gives **Codex desktop (direct-connect / Mode B)** users canvas
-image write-back. Bundled into the Codex plugin (`.codex-plugin/`), declared as a **command-based**
-MCP server so codex spawns it locally; the proxy owns upstream auth to the remote gemus `/api/mcp`
-(Codex does no OAuth for stdio servers). Full design: `docs/plans/1751-codex-desktop-companion.md`
+image write-back. The Codex plugin declares it as a **default-disabled, command-based** MCP server
+so Codex can spawn it locally after explicit enablement; the proxy owns upstream auth to the remote
+Gemus `/api/mcp` (Codex does no OAuth for stdio servers). Full design:
+`docs/plans/1751-codex-desktop-companion.md`
 and `docs/architecture/agent architecture/codex-desktop-companion.md`.
 
 ## Status: implemented
@@ -27,24 +28,55 @@ server blueprint policy.
 - **(c) turn-end + retrieval** — `Stop` hook fires post-flush carrying `transcript_path` → exact
   rollout; imagegen writes `generated_images/<session>/<sanitized-call-id>.png` (historically `ig_*`,
   currently `call_*`) + rollout `saved_path`+base64.
-- **Session cap** — `KEY_ONLY_SESSION_CAP=3`/user is real; the proxy MUST `DELETE` its upstream
-  session on close (implemented in `shutdown()`) to avoid leaking slots.
+- **Session cap** — the key-only session cap defaults to 10 concurrent sessions per user
+  (`MCP_KEY_ONLY_SESSION_CAP`); the proxy MUST `DELETE` its upstream session on close (implemented
+  in `cleanup()`) to avoid leaking slots.
 
-## Install (Codex desktop)
+## Companion setup and migration (Codex desktop)
 
-The key is supplied as a **literal** in the user's *local* codex config — codex does NOT interpolate
-`${VAR}` in an MCP server's `env`, and a public plugin manifest must never carry a key (Step-0.4
-findings). So the keyed MCP server is registered locally, not via the plugin manifest:
+The plugin owns the non-secret process contract: exact `@gemus/mcp-proxy@0.1.8`,
+`startup_timeout_sec = 60`, `tool_timeout_sec = 300`, and a default-disabled rollout. The user
+environment owns `GEMUS_KEY` and optional `GEMUS_URL`; the public plugin and generated commands
+never contain their values.
 
-```bash
-# 1. Register the proxy as codex's gemus MCP server (writes a literal key into ~/.codex/config.toml).
-codex mcp add gemus --env GEMUS_KEY=mak_<your key> -- npx -y @gemus/mcp-proxy
-#    (get a mak_ key from gemus.ai → Settings → MCP Keys; GEMUS_URL defaults to https://gemus.ai/api/mcp)
-
-# 2. Install the plugin (Stop hook + skills) from the gemus marketplace, then TRUST the hook:
-codex plugin add gemus@<marketplace>
-#    In interactive codex run `/hooks` and trust the gemus Stop hook — REQUIRED for canvas backfill.
-#    Without a trusted hook, images still land on the canvas (orphan node), just not the planned node.
+Environment ownership and lifetime:
+
+- **Windows / PowerShell:** persist the key in the Windows user environment and refresh the current
+  process.
+- **macOS:** If changing login context, sign out and back in first; rerun the `launchctl setenv`
+  setup in the new login session; then fully quit and restart Codex and open a new task.
+- **Linux:** export the key and launch Codex from the same terminal; no universal desktop-session
+  inheritance is assumed.
+
+Companion and direct modes are mutually exclusive. The canonical new-user and legacy migration flow
+is:
+
+1. Set `GEMUS_KEY` in the user environment.
+2. Remove any legacy global server (`codex mcp remove gemus`; not-found is harmless).
+3. Install with `codex plugin marketplace add Gemus-AI/gemus-codex-plugin`, then
+   `codex plugin add gemus@gemus`. Existing users refresh the marketplace snapshot and replace the
+   installed cache:
+
+   ```bash
+   codex plugin marketplace upgrade gemus
+   codex plugin remove gemus@gemus
+   codex plugin add gemus@gemus
+   ```
+4. Explicitly enable the plugin-scoped server:
+
+   ```toml
+   [plugins."gemus@gemus".mcp_servers.gemus]
+   enabled = true
+   ```
+
+5. Restart Codex and open a new task, then use `/hooks` to trust the Stop hook separately.
+
+Intentional direct HTTP/OAuth users keep their global direct server and leave the plugin Companion
+disabled:
+
+```toml
+[plugins."gemus@gemus".mcp_servers.gemus]
+enabled = false
 ```
 
 ## Local dev
@@ -52,7 +84,6 @@ codex plugin add gemus@<marketplace>
 ```bash
 pnpm --filter @gemus/codex-backfill-core build          # proxy imports the built core at runtime
 GEMUS_KEY=mak_... GEMUS_URL=http://localhost:3000/api/mcp PROXY_LOG=/tmp/proxy.log node src/proxy.mjs
-# or register against dev:  codex mcp add gemus --env GEMUS_KEY=mak_... GEMUS_URL=http://localhost:3000/api/mcp -- node <abs>/src/proxy.mjs
 ```
 
 Env: `GEMUS_KEY` (required), `GEMUS_URL` (default `https://gemus.ai/api/mcp`), `PROXY_LOG` (optional

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gemus/mcp-proxy",
-  "version": "0.1.6",
+  "version": "0.1.8",
   "description": "Local stdio<->HTTP MCP proxy for Codex desktop: transparent passthrough + execute tap + imagegen backfill (Issue #1751, P1).",
   "type": "module",
   "bin": {

package/src/__tests__/failFast.test.ts

@@ -0,0 +1,254 @@
+/**
+ * @gemus/mcp-proxy fail-fast (#2068) — single-pass upstream-disconnect correlation.
+ *
+ * Pure logic; the socket is an injected `realFetch` returning controllable `Response`s. Cases that
+ * involve the final-safety timer use fake timers with advanceTimersByTimeAsync (lesson: fake timers
+ * + real reader.read() promises need the async advance to flush continuations).
+ */
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
+import { parseRequestIds, isCodexOriginated, createSseIdScanner, createFailFast, isJsonRpcResponse } from '../failFast.mjs'
+
+const enc = new TextEncoder()
+
+/** A response body that emits the given SSE frames then closes. */
+function bodyOf(...frames: string[]): ReadableStream<Uint8Array> {
+  return new ReadableStream<Uint8Array>({
+    start(c) {
+      for (const f of frames) c.enqueue(enc.encode(f))
+      c.close()
+    },
+  })
+}
+/** A response body that never emits and never closes (zombie stream). */
+function neverBody(): ReadableStream<Uint8Array> {
+  return new ReadableStream<Uint8Array>({ start() {} })
+}
+function sseRes(body: ReadableStream<Uint8Array> | null, init?: { status?: number; contentType?: string }) {
+  const status = init?.status ?? 200
+  return new Response(body, { status, headers: { 'content-type': init?.contentType ?? 'text/event-stream' } })
+}
+function resultFrame(id: number | string): string {
+  return `event: message\ndata: ${JSON.stringify({ jsonrpc: '2.0', id, result: { ok: true } })}\n\n`
+}
+function post(id: number | string, method = 'tools/call') {
+  return { method: 'POST', body: JSON.stringify({ jsonrpc: '2.0', id, method, params: {} }) }
+}
+async function drain(res: Response): Promise<void> {
+  const r = res.body!.getReader()
+  for (;;) { const { done } = await r.read(); if (done) break }
+}
+
+describe('parseRequestIds', () => {
+  it('extracts the id from a POST JSON-RPC request', () => {
+    expect(parseRequestIds(post(42))).toEqual([42])
+  })
+  it('extracts all ids from a batch', () => {
+    const body = JSON.stringify([
+      { jsonrpc: '2.0', id: 1, method: 'a' },
+      { jsonrpc: '2.0', id: 2, method: 'b' },
+    ])
+    expect(parseRequestIds({ method: 'POST', body })).toEqual([1, 2])
+  })
+  it('returns [] for GET / DELETE / bodyless requests', () => {
+    expect(parseRequestIds({ method: 'GET' })).toEqual([])
+    expect(parseRequestIds({ method: 'DELETE' })).toEqual([])
+    expect(parseRequestIds({ method: 'POST' })).toEqual([])
+  })
+  it('returns [] for non-JSON body and for notifications (no id)', () => {
+    expect(parseRequestIds({ method: 'POST', body: 'not json' })).toEqual([])
+    expect(parseRequestIds({ method: 'POST', body: JSON.stringify({ jsonrpc: '2.0', method: 'x' }) })).toEqual([])
+  })
+})
+
+describe('isCodexOriginated', () => {
+  it.each([
+    [42, true],
+    ['abc', true],
+    ['gemus-bf-3', false],
+    ['gemus-ping-9', false],
+    [null, false],
+    [undefined, false],
+  ])('id=%s → %s', (id, expected) => {
+    expect(isCodexOriginated(id as never)).toBe(expected)
+  })
+})
+
+describe('isJsonRpcResponse', () => {
+  it('is true for a result/error response, false for a request or notification', () => {
+    expect(isJsonRpcResponse({ id: 1, result: {} })).toBe(true)
+    expect(isJsonRpcResponse({ id: 1, error: {} })).toBe(true)
+    expect(isJsonRpcResponse({ id: 1, method: 'elicitation/create', params: {} })).toBe(false) // server→client request
+    expect(isJsonRpcResponse({ method: 'notifications/x' })).toBe(false)
+    expect(isJsonRpcResponse(null)).toBe(false)
+  })
+})
+
+describe('createSseIdScanner', () => {
+  it('yields the id of a complete RESPONSE frame; ignores comment lines', () => {
+    const s = createSseIdScanner()
+    expect(s.push(enc.encode(': keepalive\n\n'))).toEqual([])
+    expect(s.push(enc.encode(resultFrame(7)))).toEqual([7])
+  })
+  it('does NOT yield a server→client REQUEST id (has a method) — avoids id-collision premature settle', () => {
+    const s = createSseIdScanner()
+    const reqFrame = `event: message\ndata: ${JSON.stringify({ jsonrpc: '2.0', id: 3, method: 'elicitation/create', params: {} })}\n\n`
+    expect(s.push(enc.encode(reqFrame))).toEqual([])
+  })
+  it('buffers a half-frame split across chunks until its boundary arrives', () => {
+    const s = createSseIdScanner()
+    const frame = resultFrame(5)
+    const mid = Math.floor(frame.length / 2)
+    expect(s.push(enc.encode(frame.slice(0, mid)))).toEqual([])
+    expect(s.push(enc.encode(frame.slice(mid)))).toEqual([5])
+  })
+  it('handles CRLF line endings (normalizes to LF)', () => {
+    const s = createSseIdScanner()
+    const crlf = `event: message\r\ndata: ${JSON.stringify({ jsonrpc: '2.0', id: 8, result: {} })}\r\n\r\n`
+    expect(s.push(enc.encode(crlf))).toEqual([8])
+  })
+  it('ignores non-JSON data payloads', () => {
+    const s = createSseIdScanner()
+    expect(s.push(enc.encode('data: not-json\n\n'))).toEqual([])
+  })
+})
+
+describe('createFailFast.fetch', () => {
+  it('injects a JSON-RPC error when the stream closes before the result', async () => {
+    const onLost = vi.fn()
+    const realFetch = vi.fn(async () => sseRes(bodyOf(': keepalive\n\n'))) // no result frame → lost
+    const ff = createFailFast({ onLost, realFetch })
+
+    const res = await ff.fetch('u', post(42))
+    await drain(res)
+
+    expect(onLost).toHaveBeenCalledTimes(1)
+    expect(onLost.mock.calls[0][0]).toBe(42)
+    expect(ff.failedIds.has('42')).toBe(true)
+  })
+
+  it('does NOT inject an error when the result crosses the wire before close', async () => {
+    const onLost = vi.fn()
+    const realFetch = vi.fn(async () => sseRes(bodyOf(resultFrame(42))))
+    const ff = createFailFast({ onLost, realFetch })
+
+    const res = await ff.fetch('u', post(42))
+    await drain(res)
+
+    expect(onLost).not.toHaveBeenCalled()
+    expect(ff.failedIds.has('42')).toBe(false)
+  })
+
+  it('fails fast immediately when realFetch throws before any response', async () => {
+    const onLost = vi.fn()
+    const realFetch = vi.fn(async () => { throw new Error('ECONNRESET') })
+    const ff = createFailFast({ onLost, realFetch })
+
+    await expect(ff.fetch('u', post(7))).rejects.toThrow('ECONNRESET')
+    expect(onLost).toHaveBeenCalledWith(7, expect.stringContaining('upstream fetch failed'))
+    expect(realFetch).toHaveBeenCalledTimes(1) // must NOT retry — a 2nd fetch could execute the tool
+  })
+
+  it('fails fast on a non-ok HTTP response', async () => {
+    const onLost = vi.fn()
+    const realFetch = vi.fn(async () => sseRes(null, { status: 404, contentType: 'application/json' }))
+    const ff = createFailFast({ onLost, realFetch })
+
+    await ff.fetch('u', post(7))
+    expect(onLost).toHaveBeenCalledWith(7, 'upstream HTTP 404')
+  })
+
+  it('passes GET and DELETE through untouched (no body → no monitoring, no onLost)', async () => {
+    const onLost = vi.fn()
+    const getRes = sseRes(bodyOf())
+    const realFetch = vi.fn(async () => getRes)
+    const ff = createFailFast({ onLost, realFetch })
+
+    const g = await ff.fetch('u', { method: 'GET' })
+    const d = await ff.fetch('u', { method: 'DELETE' })
+    expect(g).toBe(getRes)
+    expect(d).toBe(getRes)
+    expect(onLost).not.toHaveBeenCalled()
+  })
+
+  it('does not track internal gemus-bf-* / gemus-ping-* ids', async () => {
+    const onLost = vi.fn()
+    const realFetch = vi.fn(async () => sseRes(bodyOf(': keepalive\n\n'))) // closes with no result
+    const ff = createFailFast({ onLost, realFetch })
+
+    await drain(await ff.fetch('u', post('gemus-bf-1')))
+    await drain(await ff.fetch('u', post('gemus-ping-2')))
+    expect(onLost).not.toHaveBeenCalled()
+  })
+
+  it('degrades to plain passthrough if internal logic throws', async () => {
+    const onLost = vi.fn()
+    const badRes = { ok: true, status: 200, headers: { get() { throw new Error('boom') } }, body: null } as unknown as Response
+    const realFetch = vi.fn(async () => badRes)
+    const ff = createFailFast({ onLost, realFetch })
+
+    const res = await ff.fetch('u', post(1))
+    expect(res).toBe(badRes)              // returns the response we ALREADY have — no re-fetch
+    expect(realFetch).toHaveBeenCalledTimes(1)
+    expect(onLost).not.toHaveBeenCalled()
+    ff.dispose()
+  })
+
+  it('isolates concurrent requests — disconnecting one does not affect the other', async () => {
+    const onLost = vi.fn()
+    const realFetch = vi
+      .fn()
+      .mockResolvedValueOnce(sseRes(bodyOf(': keepalive\n\n')))   // id 1 → dies with no result
+      .mockResolvedValueOnce(sseRes(bodyOf(resultFrame(2))))      // id 2 → delivers
+    const ff = createFailFast({ onLost, realFetch })
+
+    await drain(await ff.fetch('u', post(1)))
+    await drain(await ff.fetch('u', post(2)))
+
+    expect(onLost).toHaveBeenCalledTimes(1)
+    expect(onLost.mock.calls[0][0]).toBe(1)
+  })
+
+  describe('idle safety timeout', () => {
+    beforeEach(() => vi.useFakeTimers())
+    afterEach(() => vi.useRealTimers())
+
+    it('injects an error for a zombie connection that never emits a byte', async () => {
+      const onLost = vi.fn()
+      const realFetch = vi.fn(async () => sseRes(neverBody()))
+      const ff = createFailFast({ onLost, realFetch, idleTimeoutMs: 120_000 })
+
+      await ff.fetch('u', post(99)) // never emits → idle timer is never reset
+      expect(onLost).not.toHaveBeenCalled()
+      await vi.advanceTimersByTimeAsync(120_000)
+      expect(onLost).toHaveBeenCalledWith(99, expect.stringContaining('idle'))
+      ff.dispose()
+    })
+
+    it('does NOT kill a healthy-but-slow stream — heartbeat bytes reset the idle timer', async () => {
+      const onLost = vi.fn()
+      let ctl!: ReadableStreamDefaultController<Uint8Array>
+      const body = new ReadableStream<Uint8Array>({ start(c) { ctl = c } })
+      const realFetch = vi.fn(async () => sseRes(body))
+      const ff = createFailFast({ onLost, realFetch, idleTimeoutMs: 100 })
+
+      const res = await ff.fetch('u', post(1))
+      const reader = res.body!.getReader()
+
+      // Keepalive every 60ms (< 100ms idle) for 300ms total — each read touches → never fires.
+      for (let i = 0; i < 5; i++) {
+        ctl.enqueue(enc.encode(': keepalive\n\n'))
+        await reader.read()
+        await vi.advanceTimersByTimeAsync(60)
+      }
+      expect(onLost).not.toHaveBeenCalled() // survived 300ms > idleTimeout because bytes kept flowing
+
+      // Now the result arrives and the stream closes → still no fail-fast.
+      ctl.enqueue(enc.encode(resultFrame(1)))
+      await reader.read()
+      ctl.close()
+      await reader.read()
+      expect(onLost).not.toHaveBeenCalled()
+      ff.dispose()
+    })
+  })
+})

package/src/__tests__/fixtures/proxy-runtime-loader.mjs

@@ -0,0 +1,136 @@
+function moduleUrl(source) {
+  return `data:text/javascript,${encodeURIComponent(source)}`
+}
+
+const markerPrelude = `
+  import fs from 'node:fs'
+  function mark(name) {
+    const file = process.env.PROXY_TEST_CONSTRUCTION_LOG
+    if (file) fs.appendFileSync(file, name + '\\n')
+  }
+`
+
+const modules = new Map([
+  ['@gemus/codex-backfill-core', `
+    export const CLIENT_CAPABILITIES = []
+    export const CLIENT_CAPABILITIES_HEADER = 'X-Gemus-Client-Capabilities'
+    export function serializeClientCapabilities(value) { return JSON.stringify(value) }
+    export function unwrapMcpContent(value) { return value }
+  `],
+  ['@modelcontextprotocol/sdk/server/stdio.js', `
+    ${markerPrelude}
+    export class StdioServerTransport {
+      constructor() { mark('stdio-transport') }
+      async start() {
+        if (process.env.PROXY_TEST_STDIO_START_FAILURE === '1') {
+          this.onclose?.()
+          throw new Error(
+            'stdio failed Authorization: Bearer '
+            + process.env.GEMUS_KEY
+            + ' at '
+            + process.env.GEMUS_URL
+          )
+        }
+        if (process.env.PROXY_TEST_FORWARD_STDIN === '1') {
+          let input = ''
+          process.stdin.setEncoding('utf8')
+          process.stdin.on('data', (chunk) => {
+            input += chunk
+            for (;;) {
+              const newline = input.indexOf('\\n')
+              if (newline < 0) break
+              const line = input.slice(0, newline).trim()
+              input = input.slice(newline + 1)
+              if (line) this.onmessage?.(JSON.parse(line))
+            }
+          })
+        }
+        process.stdin.resume()
+        process.stdin.once('end', () => this.onclose?.())
+      }
+      async send() {}
+    }
+  `],
+  ['@modelcontextprotocol/sdk/client/streamableHttp.js', `
+    ${markerPrelude}
+    export class StreamableHTTPClientTransport {
+      constructor() { mark('upstream-transport') }
+      async start() {}
+      async send(message) {
+        if (
+          process.env.PROXY_TEST_UPSTREAM_INITIALIZE_FAILURE === '1'
+          && message?.method === 'initialize'
+        ) {
+          throw new Error(
+            'initialize fetch failed Authorization: Bearer '
+            + process.env.GEMUS_KEY
+            + ' at '
+            + process.env.GEMUS_URL
+          )
+        }
+      }
+      async terminateSession() {}
+      async close() {}
+    }
+  `],
+  ['node:http', `
+    ${markerPrelude}
+    function hookError() {
+      return new Error(
+        'hook listen failed Authorization: Bearer '
+        + process.env.GEMUS_KEY
+        + ' at '
+        + process.env.GEMUS_URL
+      )
+    }
+    const http = {
+      createServer() {
+        mark('hook-server')
+        const listeners = new Map()
+        return {
+          on(event, handler) { listeners.set(event, handler); return this },
+          listen(_port, _host, callback) {
+            if (process.env.PROXY_TEST_HOOK_LISTEN_FAILURE === '1') {
+              queueMicrotask(() => listeners.get('error')?.(hookError()))
+            } else {
+              queueMicrotask(callback)
+            }
+            return this
+          },
+          address() { return { port: 43114 } },
+          unref() {},
+          close() {
+            mark('hook-server-close')
+            if (process.env.PROXY_TEST_HOOK_ERROR_AFTER_CLOSE === '1') {
+              queueMicrotask(() => listeners.get('error')?.(hookError()))
+            }
+          },
+        }
+      },
+    }
+    export default http
+  `],
+  ['./backfill.mjs', `
+    ${markerPrelude}
+    export function createBackfiller() {
+      mark('backfiller')
+      return {
+        observeCall() {},
+        observeResult() {},
+        async backfillTurn() {},
+      }
+    }
+    export function injectSteering() { return false }
+  `],
+])
+
+export async function resolve(specifier, context, nextResolve) {
+  const source = modules.get(specifier)
+  if (
+    source !== undefined
+    && (specifier !== './backfill.mjs' || context.parentURL?.endsWith('/proxy.mjs'))
+  ) {
+    return { url: moduleUrl(source), shortCircuit: true }
+  }
+  return nextResolve(specifier, context)
+}