switchboard-fyi 0.1.0

package/docs/smoke-test.md

@@ -0,0 +1,190 @@
+# Switchboard Gateway Smoke Test
+
+This smoke test checks whether Claude Code and Codex can send model requests
+through a local Switchboard gateway while preserving their existing
+authentication path.
+
+The wrapper starts the gateway in forwarding mode for normal use. Standalone
+gateway smoke tests can run with or without `--forward`; forwarding mode proxies
+requests upstream while preserving the existing auth path.
+
+## Start the gateway
+
+```bash
+npm run gateway
+```
+
+The event log is written to:
+
+```text
+~/.switchboard/harnesses/<harness>/events.jsonl
+```
+
+## Forwarding test
+
+Forwarding mode proxies the request upstream using the auth headers supplied by
+Claude Code or Codex:
+
+```bash
+node ./bin/switchboard-gateway.mjs start --port 8787 --forward
+```
+
+Current result from local testing:
+
+- Claude Code subscription OAuth successfully forwards to Anthropic through the gateway.
+- Codex ChatGPT bearer auth reaches the gateway but is rejected by the public OpenAI Responses API with `401 Missing scopes: api.responses.write`.
+
+That means Codex subscription auth must not be forwarded to
+`https://api.openai.com/v1/responses`.
+Switchboard v1 also rejects OpenAI API-key Codex auth locally before routing,
+so API-billed requests do not spend Switchboard request credits or fall through
+to Platform billing.
+
+For Codex subscription routing, use the Codex-specific ChatGPT backend:
+
+```bash
+node ./bin/switchboard-gateway.mjs start \
+  --port 8787 \
+  --forward \
+  --codex-chatgpt-upstream
+```
+
+This maps local Codex `/v1/responses` traffic to:
+
+```text
+https://chatgpt.com/backend-api/codex/responses
+```
+
+## Codex subscription provider test
+
+Run this in another terminal while the gateway is running:
+
+```bash
+codex exec \
+  -c 'model_provider="switchboard"' \
+  -c 'model="gpt-5.5"' \
+  -c 'model_providers.switchboard.name="Switchboard"' \
+  -c 'model_providers.switchboard.base_url="http://127.0.0.1:8787/v1"' \
+  -c 'model_providers.switchboard.wire_api="responses"' \
+  -c 'model_providers.switchboard.requires_openai_auth=true' \
+  "Reply with one short sentence."
+```
+
+Then inspect the latest event:
+
+```bash
+tail -n 5 ~/.switchboard/harnesses/codex/events.jsonl
+```
+
+What matters:
+
+- `harness` should be `codex`
+- `auth.authorization.present` should be `true`
+- `auth.apiKey.present` should usually be `false` if Codex is using ChatGPT sign-in
+- `requestedModel` should be `gpt-5.5`, unless you intentionally override it
+- `forward_attempt.path` should be `/backend-api/codex/responses` when
+  `--codex-chatgpt-upstream` is enabled
+- `forward_result.status` should be `200`
+- `upstream_response.ok` should be `true`; when the stream exposes it, this
+  event also includes response ids, usage, event counts, and an output preview
+
+## Codex per-internal-call proof
+
+Use a task that forces tool use and a follow-up model call:
+
+```bash
+tmpdir=$(mktemp -d)
+cat > "$tmpdir/math.mjs" <<'EOF'
+export function add(a, b) {
+  return a - b;
+}
+EOF
+cat > "$tmpdir/math.test.mjs" <<'EOF'
+import assert from "node:assert/strict";
+import { add } from "./math.mjs";
+assert.equal(add(2, 3), 5);
+console.log("ok");
+EOF
+
+codex exec \
+  --skip-git-repo-check \
+  --dangerously-bypass-approvals-and-sandbox \
+  -C "$tmpdir" \
+  -c 'model_provider="switchboard"' \
+  -c 'model="gpt-5.5"' \
+  -c 'model_providers.switchboard.name="Switchboard"' \
+  -c 'model_providers.switchboard.base_url="http://127.0.0.1:8787/v1"' \
+  -c 'model_providers.switchboard.wire_api="responses"' \
+  -c 'model_providers.switchboard.requires_openai_auth=true' \
+  -c 'model_providers.switchboard.request_max_retries=0' \
+  -c 'model_providers.switchboard.stream_max_retries=0' \
+  "Run the test in this directory, fix the bug, run the test again, and then reply with exactly: fixed-subscription-probe"
+```
+
+Local result on 2026-05-18:
+
+- One user prompt produced 6 `model_request` events for Codex.
+- All 6 forwarded to `/backend-api/codex/responses`.
+- All 6 returned `status: 200`.
+- A forced forwarded model change from `gpt-5.5` to `gpt-5.4-mini` also returned
+  `status: 200`.
+
+## Routing API proof
+
+Run Codex through the wrapper with the Switchboard API classifier enabled:
+
+```bash
+codex \
+  --port 8916 \
+  exec --skip-git-repo-check --dangerously-bypass-approvals-and-sandbox \
+  "Reply with exactly: full-router-ok"
+```
+
+Then inspect:
+
+```bash
+tail -n 20 ~/.switchboard/harnesses/codex/events.jsonl
+switchboard dashboard codex
+```
+
+What matters:
+
+- `classification_api_payload` contains the compact packet sent to the Switchboard API.
+- `classification_api_result` contains the API route, reason code, and dashboard
+  reason mapped by the API.
+- `route_applied.route.source` should be `switchboard`.
+- `route_applied.route.targetTier` should be `lower`, `mid`, or `best`.
+- `forward_attempt.path` should still be `/backend-api/codex/responses` for
+  Codex subscription routing.
+- `forward_result.status` should be `200`.
+- `upstream_response.ok` should be `true` and should carry the same
+  `decisionId` as the route, so the inspector can tie the result back to the
+  model choice.
+
+## Claude subscription-auth test
+
+Run this in another terminal while the gateway is running:
+
+```bash
+ANTHROPIC_BASE_URL=http://127.0.0.1:8787 claude -p "Reply with one short sentence." --model sonnet
+```
+
+Then inspect the latest event:
+
+```bash
+tail -n 5 ~/.switchboard/harnesses/claude/events.jsonl
+```
+
+What matters:
+
+- `harness` should be `claude`
+- `auth.authorization.present` should be `true` for subscription OAuth
+- `auth.apiKey.present` should be `false` unless `ANTHROPIC_API_KEY` is set
+- `requestedModel` should be the Claude model/alias sent by Claude Code
+
+## Safety notes
+
+The log stores only redacted auth fingerprints, not token values.
+
+If either tool sends an API key instead of subscription auth, unset the relevant
+API key environment variable and retry.