@cuylabs/agent-physical-capx 5.0.2

package/examples/README.md

@@ -0,0 +1,387 @@
+# Examples
+
+Runnable examples for `@cuylabs/agent-physical-capx`.
+
+These examples are the `agents-ts` client path. `capx-agent-runtime` remains
+the harness-neutral Python service; any other agent can use the same CaP-X
+runtime by calling its HTTP API directly.
+
+## Main Examples
+
+`01-capx-runtime-solver.ts` is the default single-turn bring-your-own-agent
+example. It creates an `agent-core` agent, wires in the CaP-X physical tools,
+and gives the agent one user turn. Inside that turn, `agent-core` may still run
+multiple model/tool steps, but the example prompt asks for one useful
+Code-as-Policy action and a summary:
+
+1. observe the CaP-X task and rendered simulator state,
+2. inspect runtime turn history and the CaP-X skill library when available,
+3. write one Python Code-as-Policy step,
+4. execute it through `capx-agent-runtime`,
+5. observe again and summarize reward, stdout/stderr, and task completion.
+
+`02-capx-runtime-autosolve.ts` keeps the same agent session open across several
+user turns. After each turn, the script observes the runtime and stops when
+CaP-X reports task completion or when `CAPX_MAX_SOLVER_TURNS` is reached. Use
+this when you want the harness to continue attempting the task instead of
+exiting after one scripted turn.
+
+The example enables the packaged `capx-code-as-policy` agent-core skill by
+default. That skill teaches the model how to use the CaP-X tools. It is not the
+same as CaP-X's runtime-side Python skill library, which is exposed dynamically
+through observation `codeContext` and programmatic runtime APIs.
+
+## Service-First Setup
+
+The normal path is to start the runtime service first, usually on a Linux GPU
+workstation, then run the TypeScript agent from your local machine or another
+client.
+
+Follow the runtime workstation setup first:
+
+[capx-agent-runtime workstation setup](../../../repos/capx-agent-runtime/docs/workstation-setup.md)
+
+The runtime server is typically started from the CaP-X checkout like this:
+
+```bash
+cd /path/to/cap-x
+uv run --no-sync --active capx-agent-runtime serve \
+  --repo-path "$(pwd)" \
+  --config-path env_configs/cube_stack/franka_robosuite_cube_stack.yaml \
+  --host 127.0.0.1 \
+  --port 8210
+```
+
+That command starts the CaP-X runtime around the selected YAML config. For the
+cube-stack config, the runtime is a Robosuite simulation. The TypeScript example
+connects to this service and acts as the external solver agent.
+
+You can point the runtime service at another compatible CaP-X config, for
+example:
+
+```bash
+--config-path env_configs/cube_stack/franka_robosuite_cube_stack_multiturn.yaml
+--config-path env_configs/cube_stack/franka_robosuite_cube_stack_multiturn_vf.yaml
+--config-path env_configs/cube_stack/franka_robosuite_cube_stack_multiturn_vdm.yaml
+```
+
+The `02-capx-runtime-autosolve.ts` example can run multiple external-agent
+turns against any of those configs. The `multiturn` configs add CaP-X
+continuation prompt text. The `vf` and `vdm` configs expose visual-feedback or
+visual-differencing intent; in this bring-your-own-agent path, the agent should
+call `capx_observe` with `includeImages=true` and do the comparison in the host
+model/harness.
+
+If the runtime is remote, open an SSH tunnel so your local machine can reach
+the service:
+
+```bash
+ssh -L 8210:127.0.0.1:8210 <user>@<gpu-host>
+```
+
+## Local Workspace Setup
+
+These examples are meant to run from a local `agents-ts` checkout. The
+`@cuylabs/agent-physical` and `@cuylabs/agent-physical-capx` packages are
+workspace-linked here, so source changes can be tested before publishing.
+
+From the repo root, install dependencies and build this package plus its local
+workspace dependencies:
+
+```bash
+cd /path/to/agents-ts
+pnpm install
+pnpm --filter @cuylabs/agent-physical-capx... build
+```
+
+Use the `pnpm` already available on your machine. If `pnpm` is missing and your
+Node install includes Corepack, you can enable it with `corepack enable`; if
+`corepack` is not available, install `pnpm` directly with your normal Node
+package-manager setup.
+
+The trailing `...` is intentional. It includes the dependency chain, so
+`@cuylabs/agent-core`, `@cuylabs/agent-physical`, and
+`@cuylabs/agent-physical-capx` are built together.
+
+Then configure the example environment:
+
+```bash
+cd packages/agent-physical-capx
+cp examples/.env.example examples/.env
+```
+
+Both examples import `examples/_setup.ts`. You do not run `_setup.ts`
+directly; it loads `examples/.env` and creates the OpenAI-compatible provider
+used by `agent-core`.
+
+Set the required values in `examples/.env`:
+
+```bash
+OPENAI_API_KEY=...
+OPENAI_MODEL=gpt-4o-mini
+
+CAPX_RUNTIME_SERVER_URL=http://127.0.0.1:8210
+```
+
+`OPENAI_BASE_URL` is optional. Leave it unset for the default OpenAI endpoint.
+Set it only when using an OpenAI-compatible provider, for example a local
+gateway or hosted inference endpoint.
+
+## Run Modes
+
+By default, the examples are observe-only. The agent can inspect the runtime
+state, propose policy code, and summarize what it would do, but it cannot call
+`capx_run_policy_code`:
+
+```bash
+pnpm exec tsx examples/01-capx-runtime-solver.ts
+```
+
+Allow the single-turn solver to execute one Python Code-as-Policy action in
+simulation:
+
+```bash
+CAPX_ALLOW_DESTRUCTIVE=1 \
+pnpm exec tsx examples/01-capx-runtime-solver.ts
+```
+
+The startup line should include `approval=policy-code-enabled`. If it still
+shows `approval=observe-only`, the environment variable did not reach the Node
+process. In that case, use a single-line command:
+
+```bash
+env CAPX_ALLOW_DESTRUCTIVE=1 pnpm exec tsx examples/01-capx-runtime-solver.ts
+```
+
+When execution is enabled, the log should also show an approval request for
+`capx_run_policy_code` followed by an approval resolution. If the startup line
+shows `approval=policy-code-enabled` but the tool result still says
+`Approval denied for capx_run_policy_code`, rebuild and rerun the local
+workspace; older example code used a hard default deny policy before the
+example callback could approve the tool.
+
+Allow execution and force video recording for that policy-code turn:
+
+```bash
+CAPX_ALLOW_DESTRUCTIVE=1 \
+CAPX_POLICY_EXECUTION_RECORD_VIDEO=1 \
+pnpm exec tsx examples/01-capx-runtime-solver.ts
+```
+
+Run the multi-turn solver in observe-only mode:
+
+```bash
+CAPX_MAX_SOLVER_TURNS=6 pnpm exec tsx examples/02-capx-runtime-autosolve.ts
+```
+
+Allow the multi-turn solver to execute policy code:
+
+```bash
+CAPX_ALLOW_DESTRUCTIVE=1 \
+CAPX_MAX_SOLVER_TURNS=6 \
+pnpm exec tsx examples/02-capx-runtime-autosolve.ts
+```
+
+Allow multi-turn execution and force video recording for each policy-code turn:
+
+```bash
+CAPX_ALLOW_DESTRUCTIVE=1 \
+CAPX_POLICY_EXECUTION_RECORD_VIDEO=1 \
+CAPX_MAX_SOLVER_TURNS=6 \
+CAPX_RECOVER_ON_RUNTIME_ERROR=reset \
+CAPX_MAX_RUNTIME_RESETS=1 \
+CAPX_STOP_ON_EXIT=1 \
+pnpm exec tsx examples/02-capx-runtime-autosolve.ts
+```
+
+For the video mode, the startup line should include
+`approval=policy-code-enabled` and `recordVideo=1`.
+`CAPX_STOP_ON_EXIT=1` stops the runtime session at the end of the example so
+`capx-agent-runtime` can flush the combined session video artifact. Stopped
+sessions still keep their artifacts available through the console and HTTP API;
+the session may remain listed there, but the live simulator environment has
+been stopped. This does not shut down the top-level `capx-agent-runtime serve`
+process.
+
+For the default Franka cube-stack config, a healthy run usually finishes after
+one useful policy-code turn. The exact sampled poses and artifact URLs vary by
+trial, but the important terminal lines look like this:
+
+```text
+executionOk=true, taskCompleted=true, reward=1
+terminated=true, truncated=false
+sandboxRc=0
+CaP-X reported completion state: taskCompleted=true terminated=true truncated=false sandboxRc=0 reward=1
+```
+
+The server log should show the same lifecycle:
+
+```text
+POST /sessions ... 200 OK
+POST /sessions/<id>/execute-code ... 200 OK
+Saved interaction video to .../video_1.000_turn_00.mp4
+Saved interaction video to .../video_session_combined.mp4
+POST /sessions/<id>/stop ... 200 OK
+```
+
+The `video_..._turn_00.mp4` file is the per-policy-turn recording. The
+`video_session_combined.mp4` file is written when the session stops, which is
+why `CAPX_STOP_ON_EXIT=1` is recommended for video examples. In the console,
+the combined session video is shown at the top of the artifact list; per-turn
+videos remain linked as individual artifact files.
+
+By default, each example run writes to a unique remote CaP-X output directory
+under `outputs/capx-agent-runtime/<agent-session-id>`. That keeps artifacts
+from separate runs from being mixed together in the console. Set
+`CAPX_OUTPUT_DIR` only when you intentionally want a specific remote output
+directory.
+
+The autosolver also stops early if CaP-X reports a persistent observation or
+depth-rendering failure. This is different from ordinary Python policy-code
+failure. CaP-X can ask an agent to regenerate code when `env.step(code)`
+returns a normal `info_step` with `stderr`. But if the Robosuite observation
+pipeline raises while collecting camera/depth observations, `env.step(code)`
+does not return a normal result. In that state even `pass` can fail before user
+policy code runs, so continuing to submit more code in the same session is not
+useful.
+
+When `CAPX_RECOVER_ON_RUNTIME_ERROR=reset` is set, the autosolver resets only
+for runtime-level `env.step(...)` failures where CaP-X cannot return a normal
+step result. It does not reset for ordinary policy-code `stderr`; those are
+left to the next agent turn so the model can inspect the error and try a better
+policy. Runtime resets use the next CaP-X trial/seed. The default reset budget
+is one reset; set `CAPX_MAX_RUNTIME_RESETS` to change that. If
+`CAPX_POLICY_EXECUTION_TRIAL` is unset, the first session uses trial `1`, the
+first recovery reset uses trial `2`, and so on. `CAPX_STOP_ON_EXIT` is separate
+from this recovery behavior: recovery reset happens during the solver loop,
+while stop-on-exit runs once the example is done, fails, or exhausts its reset
+budget.
+
+If the reset budget is exhausted, or if you are running without automatic
+recovery, do the cleanup first, then retry with a fresh session.
+
+If you used `CAPX_STOP_ON_EXIT=1`, the example asks the server to stop the
+runtime session before exiting and flushes the combined session video. You can
+then rerun the example directly.
+
+If the session is still running, find the runtime session id and stop it:
+
+```bash
+curl -sS http://127.0.0.1:8210/sessions
+curl -X POST http://127.0.0.1:8210/sessions/<session-id>/stop
+```
+
+You can also reset an existing session, but for observation/depth assertion
+failures a fresh session is usually clearer:
+
+```bash
+curl -X POST \
+  -H 'content-type: application/json' \
+  -d '{}' \
+  http://127.0.0.1:8210/sessions/<session-id>/reset
+```
+
+Then run the autosolver again:
+
+```bash
+CAPX_ALLOW_DESTRUCTIVE=1 \
+CAPX_POLICY_EXECUTION_RECORD_VIDEO=1 \
+CAPX_MAX_SOLVER_TURNS=6 \
+CAPX_RECOVER_ON_RUNTIME_ERROR=reset \
+CAPX_MAX_RUNTIME_RESETS=1 \
+CAPX_STOP_ON_EXIT=1 \
+pnpm exec tsx examples/02-capx-runtime-autosolve.ts
+```
+
+If the depth assertion repeats immediately on a clean session, restart the
+`capx-agent-runtime serve` process too. That recreates the Python environment
+and the child API services instead of reusing the same process state.
+
+If you want to isolate the TypeScript adapter and `agent-core` loop from the
+vision/depth stack, start `capx-agent-runtime` with a privileged cube-stack
+config when available:
+
+```bash
+uv run --no-sync --active capx-agent-runtime serve \
+  --repo-path "$(pwd)" \
+  --config-path env_configs/cube_stack/franka_robosuite_cube_stack_privileged.yaml \
+  --host 127.0.0.1 \
+  --port 8210
+```
+
+That path avoids some vision-derived object-pose calls and is useful for
+checking that HTTP tools, approvals, artifacts, videos, and the external agent
+loop are wired correctly before debugging the Robosuite camera/depth pipeline.
+
+`CAPX_ALLOW_DESTRUCTIVE=1` means "allow side-effecting CaP-X tools" in this
+example harness. It is required for `capx_run_policy_code` because that tool
+executes model-authored Python inside the live CaP-X runtime. For hardware
+configs, policy execution is still blocked unless
+`CAPX_ALLOW_HARDWARE_POLICY_EXECUTION=1` is also set.
+
+The example always uses the live runtime path: `mode: "runtime"`,
+`startSession: true`, `enablePolicyCodeExecution: true`,
+and `policyExecutionMode: "live-runtime"`. The adapter does not accept
+`repoPath` or `configPath`; those belong to the runtime server startup command.
+That keeps the example aligned with the production architecture: the Python
+runtime service owns the CaP-X repo/config/simulator setup, and `agent-core`
+owns the external agent loop.
+
+The adapter defaults to `toolExecutionMode: "plan"`. In `agent-core`, "plan"
+means framework-owned tool dispatch, not "only produce a written plan." The
+model still reasons normally and emits tool calls; `agent-core` executes those
+tool calls after applying approval and scheduling policy, then records the tool
+results before the next model step. The console output lines such as
+`capx_status({ ... })` and `capx_run_policy_code({ ... })` are the visible
+planned tool calls.
+
+Both examples use `agent-core`'s `createEventPrinter` to render progress:
+steps, tool calls, tool results, approval events, text output, and completion.
+For CaP-X, those logs are the easiest way to see the external agent loop:
+status, observe, optional policy-code execution, observe again, then final
+summary. Set `CAPX_TOOL_RESULT_MAX_CHARS` if you want the terminal to print
+longer tool-result previews while debugging.
+
+## Environment Model
+
+`OPENAI_API_KEY` configures the `agent-core` model provider. `OPENAI_MODEL`
+defaults to `gpt-4o-mini` through `examples/_setup.ts` if you omit it.
+`OPENAI_BASE_URL` is only needed for non-default OpenAI-compatible endpoints.
+
+`CAPX_RUNTIME_SERVER_URL` points to the `capx-agent-runtime` service. When
+the example creates a session, it lets the server's startup arguments define
+the CaP-X repo, YAML config, output directory, and simulator context. This
+matches the workstation setup command above.
+
+`CAPX_ALLOW_DESTRUCTIVE=1` lets the example approval policy allow
+`capx_run_policy_code`. Without it, the agent can observe and propose code but
+will not execute policy code.
+
+`CAPX_MAX_SOLVER_TURNS` controls the outer loop in
+`02-capx-runtime-autosolve.ts`. The same `agent-core` session id is reused for
+each turn so the agent keeps conversation and tool history.
+
+`CAPX_RECOVER_ON_RUNTIME_ERROR=reset` lets
+`02-capx-runtime-autosolve.ts` reset the live CaP-X runtime session to the next
+trial/seed and continue when CaP-X reports an observation/depth failure. This
+is session-level recovery for failures where `env.step(code)` cannot return a
+normal multi-turn result. `CAPX_MAX_RUNTIME_RESETS` controls the reset budget
+and defaults to `1` when recovery is enabled.
+
+`CAPX_POLICY_EXECUTION_RECORD_VIDEO` is optional. Leave it unset to use the
+selected CaP-X YAML's `record_video` setting. Set it to `1` or `0` only when
+you want the TypeScript example to override the runtime server/YAML value.
+
+## Prompt Context
+
+This package does not copy CaP-X prompt templates into TypeScript. In runtime
+mode, `capx-agent-runtime` loads the selected CaP-X YAML config and trial. Then
+`capx_observe` returns the task prompt, full prompt, observations, API
+descriptions, rendered frame when available, and last-step result to the
+`agent-core` agent.
+
+The external agent reads that CaP-X-provided context and acts by calling
+`capx_run_policy_code`.
+
+This is the clean bring-your-own-agent reference: start the runtime service
+first, then connect an external `agent-core` agent to it.

package/examples/_setup.ts

@@ -0,0 +1,61 @@
+/**
+ * Shared example setup — loads `.env` from the examples directory.
+ */
+
+import { createOpenAI } from "@ai-sdk/openai";
+import { config } from "dotenv";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+export const examplesDir = dirname(fileURLToPath(import.meta.url));
+
+config({ path: join(examplesDir, ".env"), quiet: true });
+
+const DEFAULT_OPENAI_MODEL = "gpt-4o-mini";
+
+function firstEnv(names: string[]): string | undefined {
+  for (const name of names) {
+    const value = process.env[name];
+    if (value && value.trim()) {
+      return value.trim();
+    }
+  }
+  return undefined;
+}
+
+export function getExampleOpenAIModelId(
+  fallback = DEFAULT_OPENAI_MODEL,
+): string {
+  return (
+    firstEnv([
+      "OPENAI_MODEL",
+      "OPENAI_MODEL_ID",
+      "openai_model",
+      "openai_model_id",
+    ]) ?? fallback
+  );
+}
+
+export function getExampleOpenAIBaseURL(): string | undefined {
+  return firstEnv([
+    "OPENAI_BASE_URL",
+    "OPENAI_API_BASE_URL",
+    "OPENAI_BASEURL",
+    "openai_base_url",
+    "openai_api_base_url",
+  ]);
+}
+
+export function createExampleOpenAIProvider() {
+  const apiKey = firstEnv(["OPENAI_API_KEY", "openai_api_key"]);
+  const baseURL = getExampleOpenAIBaseURL();
+
+  return createOpenAI({
+    ...(apiKey ? { apiKey } : {}),
+    ...(baseURL ? { baseURL } : {}),
+  });
+}
+
+export function exampleOpenAIModel(modelId = getExampleOpenAIModelId()) {
+  return createExampleOpenAIProvider()(modelId);
+}

package/package.json

@@ -0,0 +1,76 @@
+{
+  "name": "@cuylabs/agent-physical-capx",
+  "version": "5.0.2",
+  "description": "CaP-X adapter for @cuylabs/agent-physical sessions and tools",
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "default": "./dist/index.js"
+    },
+    "./agent": {
+      "types": "./dist/agent.d.ts",
+      "import": "./dist/agent.js",
+      "default": "./dist/agent.js"
+    },
+    "./session": {
+      "types": "./dist/session.d.ts",
+      "import": "./dist/session.js",
+      "default": "./dist/session.js"
+    }
+  },
+  "files": [
+    "dist",
+    "docs",
+    "examples/*.ts",
+    "examples/.env.example",
+    "examples/README.md",
+    "skills",
+    "README.md"
+  ],
+  "dependencies": {
+    "zod": "^3.25.76 || ^4.1.8",
+    "@cuylabs/agent-core": "^5.0.2",
+    "@cuylabs/agent-physical": "^5.0.2"
+  },
+  "devDependencies": {
+    "@ai-sdk/openai": "4.0.0-beta.38",
+    "@types/node": "^22.0.0",
+    "dotenv": "^17.2.3",
+    "tsup": "^8.0.0",
+    "tsx": "^4.21.0",
+    "typescript": "^5.7.0",
+    "vitest": "^4.0.18"
+  },
+  "keywords": [
+    "agent",
+    "physical-ai",
+    "robotics",
+    "capx",
+    "code-as-policy"
+  ],
+  "author": "cuylabs",
+  "license": "Apache-2.0",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/cuylabs-ai/agents-ts.git",
+    "directory": "packages/agent-physical-capx"
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "tsup --config tsup.config.ts",
+    "dev": "tsup --config tsup.config.ts --watch",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "clean": "rm -rf dist"
+  }
+}

package/skills/capx-code-as-policy/SKILL.md

@@ -0,0 +1,22 @@
+---
+name: capx-code-as-policy
+description: Use this when controlling a CaP-X Code-as-Policy runtime through capx_* tools. It explains the observe, render, policy-code, turn-history, and artifact loop for an external agent harness.
+version: 1.0.0
+tags: [capx, robotics, physical-ai, code-as-policy]
+---
+
+# CaP-X Code-as-Policy Agent Skill
+
+Use CaP-X as the Python robotics runtime. The external agent owns the reasoning loop. First inspect the session with `capx_status` and `capx_observe`. Treat the CaP-X task prompt, full prompt, `codeContext`, policy-code context, reset metadata, rendered frames, and last-step result as the source of truth.
+
+One CaP-X runtime session represents one live environment. Keep using that same session across observe/run/observe turns. Do not ask for a new session, reset, or stop unless the user asks, the current trial needs a deliberate retry, or continuing would be unsafe.
+
+When `capx_run_policy_code` is available, write concise Python policy code using APIs exposed by the observed CaP-X prompt and `codeContext`. Prefer one purposeful code step at a time, then observe again.
+
+Use `capx_observe` with image observations when visual state matters. Compare the new observation, frame, stdout, stderr, reward, and task-completion status against the previous turn before deciding whether another policy-code step is needed.
+
+Use `capx_turn_history` to inspect prior submitted code and results. Use `capx_artifacts` when you need saved code, logs, summaries, images, or videos.
+
+Use policy-code context returned by `capx_observe` for reusable CaP-X Python helper hints. These are runtime-side Python functions, not agent-core skills or separate robot tools. Submit policy code through `capx_run_policy_code`.
+
+For ensemble reasoning, generate and compare candidate plans in the agent harness, then submit only the selected Python code through `capx_run_policy_code`. Keep execution approval and hardware safety policy in the host application.