agenticros 0.3.4 → 0.4.0

package/runtime/README.md

@@ -33,6 +33,29 @@ With AgenticROS, your robot can describe what it sees, follow intent ("go check
 
 AgenticROS is built so that new adapters (LangGraph, OpenAI, local models, voice stacks, etc.) can be added without touching the ROS 2 layer. The core transport and tool contract are platform-agnostic; adapters are thin shims that surface those tools to each agent runtime.
 
+## Local VLM (Ollama) — no cloud API required
+
+Control your robot with a **local vision-language model** instead of OpenAI or other cloud APIs. AgenticROS robot tools work the same; you point **OpenClaw** or **Hermes** at Ollama on your machine.
+
+```bash
+ollama pull qwen3-vl:8b-instruct   # recommended: chat + tools + camera vision
+# or: ollama pull qwen3-vl:2b     # smaller hardware
+
+npx agenticros init               # skip the OpenAI key step when prompted
+# Configure OpenClaw to use Ollama as primary model (see guide below)
+agenticros up sim-amr             # or: agenticros up real
+```
+
+Then chat in OpenClaw web UI or Hermes: *"List ROS topics"*, *"drive forward slowly"*, *"what do you see?"*
+
+| Path | Best for |
+|------|----------|
+| **OpenClaw + Ollama** | Web chat, teleop, messaging channels, skills |
+| **Hermes + Ollama** | Terminal agent with any Ollama model |
+| **NemoClaw + Ollama** | Sandboxed Jetson / edge (see [docs/nemoclaw.md](docs/nemoclaw.md)) |
+
+**Full setup, model picks, multimodal catalog patches, describer, Follow Me VLM, and troubleshooting:** **[docs/local-vlm.md](docs/local-vlm.md)**
+
 ## Architecture
 
 ![AgenticROS architecture: agent platforms → adapter packages → shared TypeScript runtime → transports → ROS 2 workspace and robot](docs/images/agenticros-architecture.png)
@@ -103,6 +126,8 @@ On the ROS side, every robot publishes a 1 Hz heartbeat on `<ns>/agenticros/robo
 
 `mission_cancel({ mission_id })` flips an in-process cancellation token; the runner stops at the next step boundary and marks remaining steps `cancelled`. When the [shared memory backend](#memory-optional) is on, every step is also written to `mission:<id>` in long-term memory, tagged with `step:<status>` and `capability:<id>`. A different agent — different process, different vendor — can `memory_recall` later and reconstruct exactly what happened, so two agents can collaborate on or hand off a mission.
 
+**How-to:** step-by-step fleet setup, declarative plans, NL goals, cancel, and handoff — **[docs/missions.md](docs/missions.md)**. Runnable walkthrough: **[examples/find-and-approach/README.md](examples/find-and-approach/README.md)**.
+
 Full architecture + design trade-offs: **[docs/strategy-ai-agents-plus-ros.md](docs/strategy-ai-agents-plus-ros.md)**.
 
 ## Repository layout
@@ -112,7 +137,7 @@ Full architecture + design trade-offs: **[docs/strategy-ai-agents-plus-ros.md](d
 - `**packages/agenticros-claude-code**` — MCP server for Claude Code + Claude desktop / Dispatch (tools only; no config UI).
 - `**packages/agenticros-gemini**` — Gemini CLI (function calling; no MCP).
 - `**ros2_ws/**` — ROS2 workspace: `agenticros_msgs`, `agenticros_bringup` (Gazebo + RViz + rosbridge launches), `agenticros_discovery`, `agenticros_agent`, `agenticros_follow_me`.
-- `**docs/**` — Architecture, skills, robot setup, Zenoh, teleop.
+- `**docs/**` — Architecture, skills, robot setup, Zenoh, teleop, **[local VLM / Ollama](docs/local-vlm.md)**.
 - `**scripts/**` — Workspace setup, gateway plugin config, run demos.
 - `**docker/**` — Docker Compose and Dockerfiles for ROS2 + plugin images.
 - `**examples/**` — Example projects.
@@ -138,7 +163,7 @@ The first run launches the interactive menu:
 ? What would you like to do?
   Launch with real robot
 ❯ Launch with simulation
-  First-time setup (workspace + OpenClaw plugin + Codex MCP + API key)
+  First-time setup (workspace + OpenClaw plugin + Codex MCP + optional API key)
   Manage skills (2 registered, 0 available, 0 broken)
   Stop everything
   Doctor (health check)
@@ -146,8 +171,8 @@ The first run launches the interactive menu:
   Tail logs
 ```
 
-Pick **First-time setup** once (workspace + OpenClaw plugin + optional Codex MCP + API key, all
-idempotent), then choose how you want to run:
+Pick **First-time setup** once (workspace + OpenClaw plugin + optional Codex MCP + optional API key, all
+idempotent). **Using local Ollama instead of OpenAI?** Skip the API key step — see **[Local VLM (Ollama)](#local-vlm-ollama--no-cloud-api-required)**. Then choose how you want to run:
 
 | You want to … | Pick |
 |---|---|
@@ -163,7 +188,7 @@ so **Stop everything** cleanly tears the demo down.
 Prefer scripted invocations? Every menu item maps to a direct command:
 
 ```bash
-npx agenticros init             # one-time workspace + plugin + Codex/Hermes MCP + API key
+npx agenticros init             # one-time workspace + plugin + Codex/Hermes MCP (+ optional API key)
 agenticros codex setup          # register AgenticROS MCP for OpenAI Codex CLI
 agenticros codex doctor         # validate ~/.codex/config.toml
 agenticros hermes setup         # register AgenticROS MCP for Hermes Agent
@@ -212,7 +237,7 @@ needing token auth: run `node scripts/agenticros-proxy.cjs 18790` and open
 See **[docs/openclaw-releases-and-plugin-routes.md](docs/openclaw-releases-and-plugin-routes.md)**
 and **[docs/teleop.md](docs/teleop.md)**.
 
-See **`docs/`** for robot setup, skills, teleop, simulation internals, and Docker.
+See **`docs/`** for robot setup, **[missions](docs/missions.md)**, skills, teleop, simulation internals, and Docker.
 
 ## RViz2 and Gazebo (TurtleBot3 + rosbridge)
 

package/runtime/docs/cli.md

@@ -64,7 +64,7 @@ skipped (with a checkmark) when already done:
 3. ROS 2 workspace build (`colcon build --symlink-install`)
 4. OpenClaw plugin install (via `scripts/setup_gateway_plugin.sh`)
 5. Robot config (writes `~/.agenticros/config.json`)
-6. OpenAI API key (paste once → `scripts/configure_agenticros.sh`)
+6. OpenAI API key (optional — skip when using local Ollama; see [local-vlm.md](local-vlm.md))
 7. MCP client config (optional — `agenticros mcp setup` for Codex, Hermes, and Claude)
 8. Final `agenticros doctor` summary
 

package/runtime/docs/robot-setup.md

@@ -296,7 +296,7 @@ The message comes from **OpenAI's API limits**, not from AgenticROS. The gateway
 1. **Confirm it's OpenAI** — Check gateway logs:  
    `journalctl --user -u openclaw-gateway.service -n 100 --no-pager | grep -i "429\|rate\|limit\|error"`
 2. **Check your OpenAI account** — [Usage](https://platform.openai.com/usage), [API keys](https://platform.openai.com/api-keys). Free/low tiers have strict RPM/TPM and can stay limited for a while.
-3. **What helps** — Wait (RPM/TPM reset in minutes; daily limits at UTC midnight). Or in OpenClaw switch to a model with higher limits if available.
+3. **What helps** — Wait (RPM/TPM reset in minutes; daily limits at UTC midnight). Or switch OpenClaw to a **local Ollama VLM** — see [local-vlm.md](local-vlm.md). Or in OpenClaw switch to a model with higher limits if available.
 4. **AgenticROS** — We only trim context/tool output to use fewer tokens; we can't remove OpenAI's limits.
 
 ### 404 errors when installing rosbridge_suite

package/runtime/packages/agenticros/openclaw.plugin.json

@@ -21,6 +21,8 @@
       "ros2_find_robots_for",
       "run_mission",
       "mission_cancel",
+      "mission_pause",
+      "mission_resume",
       "ros2_camera_snapshot",
       "ros2_depth_distance",
       "memory_remember",
@@ -30,7 +32,8 @@
       "follow_robot",
       "follow_me_see",
       "ollama_status",
-      "find_object"
+      "find_object",
+      "scan_surroundings"
     ]
   },
   "configSchema": {
@@ -178,6 +181,149 @@
           }
         }
       },
+      "robots": {
+        "type": "array",
+        "description": "Multi-robot fleet list. When non-empty, used by ros2_list_robots and per-tool robot_id. When empty, the legacy config.robot entry is used. Manage via `agenticros robots` CLI.",
+        "default": [],
+        "items": {
+          "type": "object",
+          "required": [
+            "id"
+          ],
+          "properties": {
+            "id": {
+              "type": "string",
+              "description": "Stable identifier referenced by robot_id tool arguments"
+            },
+            "name": {
+              "type": "string",
+              "default": "Robot"
+            },
+            "namespace": {
+              "type": "string",
+              "default": ""
+            },
+            "cameraTopic": {
+              "type": "string",
+              "default": ""
+            },
+            "default": {
+              "type": "boolean",
+              "description": "When true, this robot is active even if not first in the list"
+            },
+            "kind": {
+              "type": "string",
+              "description": "Robot kind for ros2_find_robots_for (e.g. amr, arm, drone, rover)",
+              "default": "amr"
+            },
+            "sensors": {
+              "type": "object",
+              "properties": {
+                "has_realsense": {
+                  "type": "boolean",
+                  "default": false
+                },
+                "has_lidar": {
+                  "type": "boolean",
+                  "default": false
+                },
+                "has_arm": {
+                  "type": "boolean",
+                  "default": false
+                }
+              }
+            },
+            "capabilities": {
+              "type": "array",
+              "description": "Optional per-robot capability allowlist for ros2_find_robots_for",
+              "items": {
+                "type": "string"
+              }
+            },
+            "transport": {
+              "type": "object",
+              "description": "Per-robot transport override (inherits global transport when omitted)",
+              "properties": {
+                "mode": {
+                  "type": "string",
+                  "enum": [
+                    "rosbridge",
+                    "local",
+                    "webrtc",
+                    "zenoh"
+                  ]
+                },
+                "zenoh": {
+                  "type": "object",
+                  "properties": {
+                    "routerEndpoint": {
+                      "type": "string"
+                    },
+                    "domainId": {
+                      "type": "number"
+                    },
+                    "keyFormat": {
+                      "type": "string",
+                      "enum": [
+                        "ros2dds",
+                        "rmw_zenoh"
+                      ]
+                    },
+                    "bridgeNamespace": {
+                      "type": "string"
+                    }
+                  }
+                },
+                "rosbridge": {
+                  "type": "object",
+                  "properties": {
+                    "url": {
+                      "type": "string"
+                    },
+                    "reconnect": {
+                      "type": "boolean"
+                    },
+                    "reconnectInterval": {
+                      "type": "number"
+                    }
+                  }
+                },
+                "local": {
+                  "type": "object",
+                  "properties": {
+                    "domainId": {
+                      "type": "number"
+                    }
+                  }
+                },
+                "webrtc": {
+                  "type": "object",
+                  "properties": {
+                    "signalingUrl": {
+                      "type": "string"
+                    },
+                    "apiUrl": {
+                      "type": "string"
+                    },
+                    "robotId": {
+                      "type": "string"
+                    },
+                    "robotKey": {
+                      "type": "string"
+                    },
+                    "iceServers": {
+                      "type": "array",
+                      "items": {
+                        "type": "object"
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          }
+        }
+      },
       "teleop": {
         "type": "object",
         "description": "Teleop web app: camera and twist controls",

package/runtime/packages/agenticros/src/__tests__/capabilities-plugin.test.ts

@@ -131,6 +131,8 @@ test("plugin: register() is synchronous and produces the base 15-tool surface",
     "run_mission",
     // Phase 1.f — cancel an in-flight mission by id.
     "mission_cancel",
+    "mission_pause",
+    "mission_resume",
   ];
   for (const name of expected) {
     assert.ok(toolNames.includes(name), `Expected base tool ${name} in registered set`);

package/runtime/packages/agenticros/src/index.ts

@@ -1,5 +1,5 @@
 import type { OpenClawPluginApi } from "./plugin-api.js";
-import { parseConfig, isCdrTypeSupported, agenticROSBannerLines } from "@agenticros/core";
+import { parseConfig, isCdrTypeSupported, agenticROSBannerLines, applyCachedSkillRefs } from "@agenticros/core";
 import { readAgenticROSConfigFromFile } from "./config-file.js";
 import { registerService } from "./service.js";
 import { registerTools } from "./tools/index.js";
@@ -36,6 +36,13 @@ export default {
       api.logger.warn("AgenticROS: could not read config from file: " + msg + " — using gateway pluginConfig.");
       config = parseConfig(api.pluginConfig ?? {});
     }
+    // Merge skillRefs that are already cached (no network in sync register).
+    config = applyCachedSkillRefs(config);
+    if ((config.skillRefs?.length ?? 0) > 0) {
+      api.logger.info(
+        `AgenticROS: skillRefs=${config.skillRefs!.length} (cached paths merged into skillPaths; run skills install to warm cache)`,
+      );
+    }
     const mode = config.transport?.mode ?? "local";
     const zenohEndpoint = config.zenoh?.routerEndpoint ?? "";
     api.logger.info(`AgenticROS: transport mode=${mode}${mode === "zenoh" && zenohEndpoint ? ` endpoint=${zenohEndpoint}` : ""}`);

package/runtime/packages/agenticros/src/routes.ts

@@ -1,6 +1,6 @@
 import type { OpenClawPluginApi, HttpRouteHandler, HttpRouteResponse } from "./plugin-api.js";
 import type { AgenticROSConfig } from "@agenticros/core";
-import { parseConfig } from "@agenticros/core";
+import { parseConfig, prepareConfigForPersistence } from "@agenticros/core";
 import {
   readOpenClawConfig,
   writeAgenticROSConfig,
@@ -172,8 +172,9 @@ export function registerRoutes(api: OpenClawPluginApi, config: AgenticROSConfig)
         api.logger.warn("AgenticROS config read failed: " + readMsg);
         return { success: false, error: readMsg, statusCode: 500 };
       }
+      let combined: Record<string, unknown>;
       try {
-        const combined =
+        combined =
           existingAgenticROS && typeof existingAgenticROS === "object"
             ? mergeIncomingAgenticROSConfig(existingAgenticROS, body)
             : body;
@@ -192,7 +193,7 @@ export function registerRoutes(api: OpenClawPluginApi, config: AgenticROSConfig)
         merged.webrtc.robotKey = existingKey;
       }
       try {
-        writeAgenticROSConfig(merged as unknown as Record<string, unknown>);
+        writeAgenticROSConfig(prepareConfigForPersistence(merged, combined));
       } catch (err) {
         const msg = err instanceof ConfigFileError ? err.message : (err instanceof Error ? err.message : "Failed to write config");
         api.logger.warn("AgenticROS config write failed: " + msg);

package/runtime/packages/agenticros/src/tools/index.ts

@@ -14,6 +14,8 @@ import { registerDiscoverRobotsTool } from "./ros2-discover.js";
 import { registerFindRobotsForTool } from "./ros2-find-robots-for.js";
 import { registerMissionTool, type ToolRegistry } from "./ros2-mission.js";
 import { registerMissionCancelTool } from "./mission-cancel.js";
+import { registerMissionPauseTool } from "./mission-pause.js";
+import { registerMissionResumeTool } from "./mission-resume.js";
 
 /**
  * Wrap the OpenClaw API so every registerTool() call is also recorded in
@@ -21,11 +23,8 @@ import { registerMissionCancelTool } from "./mission-cancel.js";
  * dispatch sub-tool calls by name (e.g. capability "drive_base" routes
  * to the registered "ros2_publish" tool's execute()).
  *
- * We keep this internal — skills that register their own tools via
- * `api.registerTool` won't appear in the registry, which is fine for
- * v1: today the mission runner only supports the eight intrinsic
- * capability bindings declared in ros2-mission.ts. Phase 1.d will
- * extend this to capture skill-declared tools as well.
+ * Skill tools registered via `api.registerTool` are captured too, so
+ * `buildMissionBindings` can resolve `ros2_<capability_id>` when present.
  */
 function wrapApiWithToolCapture(api: OpenClawPluginApi): {
   wrappedApi: OpenClawPluginApi;
@@ -63,4 +62,6 @@ export function registerTools(api: OpenClawPluginApi, config: AgenticROSConfig):
   registerFindRobotsForTool(wrappedApi, config);
   registerMissionTool(wrappedApi, config, registry);
   registerMissionCancelTool(wrappedApi);
+  registerMissionPauseTool(wrappedApi);
+  registerMissionResumeTool(wrappedApi);
 }

package/runtime/packages/agenticros/src/tools/mission-pause.ts

@@ -0,0 +1,49 @@
+/**
+ * Tool: mission_pause — pause a running mission at the next step boundary.
+ */
+
+import { Type } from "@sinclair/typebox";
+import type { OpenClawPluginApi } from "../plugin-api.js";
+import { getMissionRegistry } from "../mission-registry.js";
+
+export function registerMissionPauseTool(api: OpenClawPluginApi): void {
+  api.registerTool({
+    name: "mission_pause",
+    label: "Pause Mission",
+    description:
+      "Pause a mission that's currently running in this OpenClaw plugin process. " +
+      "Pass the mission_id returned by run_mission. The runner waits at the next step " +
+      "boundary until mission_resume (or mission_cancel). Idempotent if already paused.",
+    parameters: Type.Object({
+      mission_id: Type.String({
+        description: "The mission_id echoed by run_mission. Required.",
+      }),
+      reason: Type.Optional(
+        Type.String({
+          description: "Optional free-text reason — surfaced in the paused transcript entry.",
+        }),
+      ),
+    }),
+
+    async execute(_toolCallId, params) {
+      const missionId = typeof params["mission_id"] === "string" ? (params["mission_id"] as string).trim() : "";
+      if (!missionId) {
+        const text = "mission_pause requires 'mission_id' (a non-empty string returned by run_mission).";
+        return { content: [{ type: "text", text }], details: { success: false, error: text } };
+      }
+      const reason = typeof params["reason"] === "string" ? (params["reason"] as string) : undefined;
+      const outcome = getMissionRegistry().pause(missionId, reason);
+      const details = {
+        success: true,
+        mission_id: missionId,
+        found: outcome.found,
+        already_paused: outcome.alreadyPaused,
+        reason: reason ?? null,
+      };
+      return {
+        content: [{ type: "text", text: JSON.stringify(details) }],
+        details,
+      };
+    },
+  });
+}

package/runtime/packages/agenticros/src/tools/mission-resume.ts

@@ -0,0 +1,41 @@
+/**
+ * Tool: mission_resume — resume a paused mission.
+ */
+
+import { Type } from "@sinclair/typebox";
+import type { OpenClawPluginApi } from "../plugin-api.js";
+import { getMissionRegistry } from "../mission-registry.js";
+
+export function registerMissionResumeTool(api: OpenClawPluginApi): void {
+  api.registerTool({
+    name: "mission_resume",
+    label: "Resume Mission",
+    description:
+      "Resume a mission previously paused with mission_pause. Pass the mission_id " +
+      "returned by run_mission. Idempotent if the mission is not paused.",
+    parameters: Type.Object({
+      mission_id: Type.String({
+        description: "The mission_id echoed by run_mission. Required.",
+      }),
+    }),
+
+    async execute(_toolCallId, params) {
+      const missionId = typeof params["mission_id"] === "string" ? (params["mission_id"] as string).trim() : "";
+      if (!missionId) {
+        const text = "mission_resume requires 'mission_id' (a non-empty string returned by run_mission).";
+        return { content: [{ type: "text", text }], details: { success: false, error: text } };
+      }
+      const outcome = getMissionRegistry().resume(missionId);
+      const details = {
+        success: true,
+        mission_id: missionId,
+        found: outcome.found,
+        was_paused: outcome.wasPaused,
+      };
+      return {
+        content: [{ type: "text", text: JSON.stringify(details) }],
+        details,
+      };
+    },
+  });
+}

package/runtime/packages/agenticros/src/tools/ros2-capabilities.ts

@@ -16,7 +16,7 @@
 import { Type } from "@sinclair/typebox";
 import type { OpenClawPluginApi } from "../plugin-api.js";
 import type { AgenticROSConfig, Capability } from "@agenticros/core";
-import { listAllCapabilities } from "@agenticros/core";
+import { listCapabilitiesWithDiscoverable } from "@agenticros/core";
 import { ROBOT_ID_SCHEMA, resolveRobotForTool } from "./_robot-helpers.js";
 
 export function registerCapabilitiesTool(
@@ -29,29 +29,26 @@ export function registerCapabilitiesTool(
     description:
       "List the high-level capabilities (named verbs) this robot can perform — built-in verbs like " +
       "drive_base / take_snapshot / measure_depth plus every capability declared by installed " +
-      "AgenticROS skills (e.g. follow_person, find_object). PREFER this over ros2_list_topics for " +
-      "high-level planning: capabilities are agent-meaningful verbs with typed inputs/outputs, " +
-      "not raw topic names. Returns one structured response listing every capability the robot " +
-      "supports right now. Pass robot_id (from ros2_list_robots) to scope to a specific robot; " +
-      "today every robot exposes the same capabilities, but the API is in place for per-robot " +
-      "capability declarations.",
+      "AgenticROS skills (e.g. follow_person, find_object). Also includes discoverable marketplace " +
+      "capabilities (discoverable:true, install_ref) that are not installed yet so you can propose " +
+      "`agenticros skills install <install_ref>`. PREFER this over ros2_list_topics for high-level " +
+      "planning. Pass robot_id to scope to a specific robot when using a multi-robot fleet.",
     parameters: Type.Object({ ...ROBOT_ID_SCHEMA }),
 
     async execute(_toolCallId, params) {
-      // robot_id is validated even though the response doesn't depend on
-      // it yet — unknown ids surface as a tool error, matching every
-      // other tool in the suite.
       const resolved = resolveRobotForTool(config, params);
       if ("error" in resolved) return resolved.error;
 
-      const caps: Capability[] = listAllCapabilities(config);
+      const caps = await listCapabilitiesWithDiscoverable(config);
       const intrinsic = caps.filter((c) => c.source?.kind === "builtin").length;
-      const skill = caps.filter((c) => c.source?.kind === "skill").length;
+      const skill = caps.filter((c) => c.installed !== false && c.source?.kind === "skill").length;
+      const discoverable = caps.filter((c) => c.discoverable === true).length;
       const result = {
         success: true,
         total: caps.length,
         intrinsic_count: intrinsic,
         skill_count: skill,
+        discoverable_count: discoverable,
         capabilities: caps,
       };
       return {