stable-harness 0.0.8 → 0.0.10

package/docs/granite-tool-calling-comparison.zh.md

@@ -0,0 +1,206 @@
+# Self-hosted Tool Calling Comparison
+
+本诊断用于比较 self-hosted 模型在不同 backend 下的工具调用表现。当前低显存默认选择是 Ollama `qwen3.5:0.8b`。
+
+## Scope
+
+- Default Ollama model id: `qwen3.5:0.8b`
+- Default Ollama options: `num_ctx=2048`, `num_predict=256`
+- Granite baseline model id: `granite4.1:3b`
+- Hugging Face model id: `ibm-granite/granite-4.1-3b`
+- Ollama native endpoint: `/api/chat`
+- vLLM/SGLang endpoint: OpenAI-compatible `/v1/chat/completions`
+
+这个脚本只做诊断，不改变 stable-harness 默认 runtime 行为。
+
+## Commands
+
+Run local Ollama only:
+
+```bash
+TOOLCALL_RUN_OPENAI=false npm run compare:tool-calling
+```
+
+Run remote Ollama only:
+
+```bash
+TOOLCALL_OLLAMA_BASE_URL=http://192.168.0.33:11434 \
+TOOLCALL_RUN_OPENAI=false \
+npm run compare:tool-calling
+```
+
+Run an OpenAI-compatible backend:
+
+```bash
+TOOLCALL_RUN_OLLAMA=false \
+TOOLCALL_OPENAI_BASE_URL=http://127.0.0.1:8000/v1 \
+TOOLCALL_OPENAI_MODEL=ibm-granite/granite-4.1-3b \
+npm run compare:tool-calling
+```
+
+Use named function mode to test argument generation after runtime-selected tool choice:
+
+```bash
+TOOLCALL_RUN_OLLAMA=false \
+TOOLCALL_OPENAI_TOOL_CHOICE=named \
+TOOLCALL_OPENAI_BASE_URL=http://127.0.0.1:8000/v1 \
+TOOLCALL_OPENAI_MODEL=ibm-granite/granite-4.1-3b \
+npm run compare:tool-calling
+```
+
+## Backend Startup
+
+vLLM on a CUDA host:
+
+```bash
+vllm serve ibm-granite/granite-4.1-3b \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --enable-auto-tool-choice \
+  --tool-call-parser hermes
+```
+
+SGLang on a CUDA host:
+
+```bash
+python -m sglang.launch_server \
+  --model-path ibm-granite/granite-4.1-3b \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --grammar-backend xgrammar
+```
+
+Parser support varies by backend and model template. Keep parser selection as backend config, not stable-harness runtime logic.
+
+## 2026-05-05 Baseline
+
+Validated in this checkout:
+
+- local Ollama `http://127.0.0.1:11434`, model `granite4.1:3b`: 3/3 passed.
+- remote Ollama `https://ollama-rtx-4070.easynet.world`, model `granite4.1:3b`: 3/3 passed.
+- local Ollama through OpenAI-compatible `/v1`, `tool_choice=required`: 3/3 passed.
+- local Ollama through OpenAI-compatible `/v1`, `tool_choice=named`: 3/3 passed.
+- CUDA host Ollama `http://192.168.0.201:11434`, model `granite4.1:3b`: 3/3 passed.
+- CUDA host Ollama through OpenAI-compatible `/v1`, `tool_choice=required`: 3/3 passed.
+- CUDA host Ollama through OpenAI-compatible `/v1`, `tool_choice=named`: 3/3 passed.
+- CUDA host vLLM `http://192.168.0.201:8000/v1`, model `ibm-granite/granite-4.1-3b`, `tool_choice=auto`: 3/3 passed.
+- CUDA host vLLM `http://192.168.0.201:8000/v1`, model `ibm-granite/granite-4.1-3b`, `tool_choice=required`: 3/3 passed.
+- CUDA host vLLM `http://192.168.0.201:8000/v1`, model `ibm-granite/granite-4.1-3b`, `tool_choice=named`: 3/3 passed.
+
+## 2026-05-05 Deep Matrix
+
+Expanded matrix: 11 cases across current weather, forecast vs current-weather, create vs search ticket, shell vs file read, Chinese Kubernetes, Chinese stock quote, general-news vs finance, exact shell command, and enum priority.
+
+Results:
+
+| Backend | Mode | Result | Notes |
+| --- | --- | ---: | --- |
+| Ollama `192.168.0.201:11434` | native auto tools | 11/11 | News query contained unnatural Chinese but still matched the relaxed semantic check. |
+| Ollama `192.168.0.201:11434/v1` | auto | 11/11 | Same issue: news query contained unnatural Chinese. |
+| Ollama `192.168.0.201:11434/v1` | required | 11/11 | Same issue: news query contained unnatural Chinese. |
+| Ollama `192.168.0.201:11434/v1` | named | 10/11 | Failed news-query semantic check: `不止 定期 AI 环境 新雪`. |
+| vLLM `192.168.0.201:8000/v1` | auto | 10/11 | Tool selection was correct, but news-query args were semantically bad. |
+| vLLM `192.168.0.201:8000/v1` | required | 10/11 | News case failed with vLLM `400 Bad Request` from invalid JSON emitted by the tool parser. |
+| vLLM `192.168.0.201:8000/v1` | named | 11/11 | Best result in this matrix. Runtime-selected tool plus backend-generated args was most stable. |
+
+Interpretation:
+
+- Both backends can select the right tool on this matrix.
+- The failure mode is argument semantics, especially Chinese query generation.
+- vLLM `required` is not automatically safer than `auto`; parser-level strictness can surface as hard 400 errors.
+- `named` mode best matches the stable-harness two-step strategy: runtime chooses the tool from typed inventory, backend fills schema-bound arguments, gateway validates before execution.
+
+Not yet validated here:
+
+- SGLang with `ibm-granite/granite-4.1-3b`.
+
+vLLM startup details:
+
+- Host: `boqiang@192.168.0.201`
+- GPU: RTX 4070 Ti SUPER 16GB
+- Driver: `550.163.01`
+- vLLM image: `vllm/vllm-openai:v0.8.5`
+- `latest` image was not usable on this host because it required CUDA 13.
+- Container: `stable-granite-vllm`
+
+SGLang remains unvalidated because no SGLang server was listening on `192.168.0.201:30000`.
+
+## 2026-05-05 CUDA 13 Upgrade And GPT-OSS
+
+CUDA host `192.168.0.201` was upgraded from NVIDIA driver `550.163.01` to `580.142`.
+
+Validation:
+
+- Host `nvidia-smi`: driver `580.142`, CUDA `13.0`.
+- CUDA 13 container smoke: `nvidia/cuda:13.0.0-base-ubuntu24.04` ran `nvidia-smi` successfully.
+- `vllm/vllm-openai:latest` now starts on this host.
+
+APT note:
+
+- `apt update` was blocked by a stale Warp signing key and a removed Kubernetes `apt.kubernetes.io kubernetes-xenial` source.
+- Those two source entries were disabled with `# codex-disabled`; backups were written beside them as `*.codex-bak`.
+
+GPT-OSS run:
+
+- Container: `stable-gpt-oss-vllm`
+- Image: `vllm/vllm-openai:latest`
+- vLLM version from logs: `0.20.1`
+- Endpoint: `http://192.168.0.201:8001/v1`
+- Model: `openai/gpt-oss-20b`
+- Working startup flags: `--max-model-len 2048 --gpu-memory-utilization 0.97 --enforce-eager --no-enable-prefix-caching`
+
+The initial `4096` context attempt failed before serving:
+
+- Model loaded: `13.72 GiB`.
+- Error: no available memory for KV cache blocks.
+
+Tool matrix:
+
+| Backend | Mode | Result | Notes |
+| --- | --- | ---: | --- |
+| vLLM latest GPT-OSS 20B | auto | 10/11 | Tool selection correct; `read_file` path lost leading `/`. |
+| vLLM latest GPT-OSS 20B | named | 10/11 | Same path failure. Runtime-selected tool does not fix path precision. |
+
+Interpretation:
+
+- GPT-OSS 20B can run on this 16GB GPU through vLLM latest after the driver upgrade, but only with conservative memory settings.
+- It is faster than Granite in this matrix, but the absolute-path error shows that schema/tool parsing still needs runtime semantic validation before execution.
+
+## 2026-05-05 Model Matrix
+
+Expanded matrix: 15 cases. The additional cases cover closed-ticket search, Hong Kong stock lookup, monthly web freshness, and absolute paths containing spaces.
+
+Ollama models currently visible on `192.168.0.201` include:
+
+- Small/medium chat candidates: `qwen3.5:0.8b`, `qwen3:0.6b`, `lfm2.5-thinking:latest`, `granite4.1:3b`, `qwen3.5:2b`, `qwen3.5:4b`, `qwen3:latest`, `qwen3.5:9b`, `qwen2.5:7b-instruct`, `gemma4:e2b`, `gemma4:e4b`, `gpt-oss:latest`.
+- Very large or slower candidates not included in the fast matrix: `qwen3.6:27b`, `qwen3.6:35b`, `gemma4:26b`, `qwen3.5:27b`, `nemotron-cascade-2:30b`, `lfm2:latest`.
+- Non-chat/specialized models not included: `nomic-embed-text:latest`, `glm-ocr:q8_0`.
+
+Fast matrix results:
+
+| Backend | Model | Mode | Result | Main failure |
+| --- | --- | --- | ---: | --- |
+| Ollama | `qwen3.5:2b` | native auto tools | 15/15 | none |
+| Ollama | `qwen3.5:4b` | native auto tools | 15/15 | none |
+| Ollama | `qwen3.5:9b` | native auto tools | 15/15 | none |
+| Ollama | `qwen3.5:0.8b` | native auto tools | 14/15 | `freshness` day instead of month |
+| Ollama | `granite4.1:3b` | native auto tools | 14/15 | Chinese news query became unnatural text |
+| Ollama | `qwen3:latest` | native auto tools | 14/15 | Workday ticker became `WORK` |
+| Ollama | `qwen2.5:7b-instruct` | native auto tools | 14/15 | path with space was collapsed |
+| Ollama | `gemma4:e2b` | native auto tools | 14/15 | missed Chinese Workday stock tool call |
+| Ollama | `gemma4:e4b` | native auto tools | 14/15 | missed Chinese Workday stock tool call |
+| Ollama | `lfm2.5-thinking:latest` | native auto tools | 13/15 | namespace typo and HK market error |
+| Ollama | `qwen3:0.6b` | native auto tools | 12/15 | one timeout, weaker exact title/query handling |
+| Ollama | `gpt-oss:latest` | native auto tools | 12/15 | path and freshness enum errors |
+| vLLM latest | `ibm-granite/granite-4.1-3b` | auto | 0/15 | no tool calls emitted |
+| vLLM latest | `ibm-granite/granite-4.1-3b` | required | 13/15 | Chinese query corruption and HK ticker normalization |
+| vLLM latest | `ibm-granite/granite-4.1-3b` | named | 14/15 | HK ticker normalized as `00700` |
+| vLLM latest | `openai/gpt-oss-20b` | auto | 14/15 | absolute path lost leading `/` |
+| vLLM latest | `openai/gpt-oss-20b` | named | 14/15 | same path failure |
+
+Current recommendation:
+
+1. Stable default for this host: Ollama `qwen3.5:4b`.
+2. Best runtime-controlled strategy: vLLM named tools, but only after the runtime has selected the tool and semantic validators are active.
+3. Do not use vLLM Granite auto tool calling as a default on `vllm/vllm-openai:latest`; it emitted no tool calls in this matrix.
+4. Do not trust any backend without semantic validation. Passing JSON schema still missed cases such as absolute path preservation, ticker normalization, freshness enum semantics, namespace spelling, and Chinese query quality.

package/docs/guides/getting-started.md

@@ -0,0 +1,126 @@
+# Getting Started
+
+This guide gets a new application from zero to a runnable Stable Harness
+workspace without cloning the Stable Harness repository.
+
+## Requirements
+
+- Node.js `>=24 <25`
+- An OpenAI-compatible model endpoint, or environment variables that point at
+  the model provider you want to use
+
+## Install
+
+```bash
+npm install stable-harness
+```
+
+For a one-off trial:
+
+```bash
+npx stable-harness init ./my-agent-app
+```
+
+## Initialize A Workspace
+
+```bash
+stable-harness init ./my-agent-app
+cd ./my-agent-app
+```
+
+The scaffold creates:
+
+```text
+config/
+  agents/orchestra.yaml
+  catalogs/models.yaml
+  runtime/workspace.yaml
+resources/
+  tools/echo_tool.mjs
+```
+
+It does not overwrite existing scaffold files. If the target already contains a
+generated file, Stable Harness stops so you can decide what to keep.
+
+## Configure The Model
+
+The default scaffold expects an OpenAI-compatible endpoint:
+
+```bash
+export OPENAI_API_KEY=...
+export STABLE_HARNESS_MODEL=gpt-4.1-mini
+export STABLE_HARNESS_OPENAI_BASE_URL=https://api.openai.com/v1
+```
+
+For a local or private OpenAI-compatible endpoint, keep the same model catalog
+shape and change the environment variables:
+
+```bash
+export STABLE_HARNESS_MODEL=granite4.1:3b
+export STABLE_HARNESS_OPENAI_BASE_URL=http://127.0.0.1:11434/v1
+export OPENAI_API_KEY=local
+```
+
+## Run The Workspace
+
+Show the loaded workspace:
+
+```bash
+stable-harness -w .
+```
+
+Invoke the scaffolded tool through the runtime gateway:
+
+```bash
+stable-harness -w . --agent orchestra --tool echo_tool --tool-args-json '{"value":"hello"}'
+```
+
+Send a natural-language request to the default agent:
+
+```bash
+stable-harness -w . --agent orchestra "Summarize what this workspace can do."
+```
+
+Print runtime trace lines:
+
+```bash
+stable-harness -w . --agent orchestra --trace "Call the echo tool with hello."
+```
+
+## Start The OpenAI-Compatible Facade
+
+```bash
+stable-harness start -w . --port 8642
+```
+
+Clients can point at:
+
+```text
+http://127.0.0.1:8642/v1
+```
+
+The facade is a protocol adapter around the same Stable Harness runtime. It is
+not a separate execution path.
+
+## Develop The Framework
+
+Clone the repository only when you are changing Stable Harness itself:
+
+```bash
+git clone git@github.com:botbotgo/stable-harness.git
+cd stable-harness
+npm install
+npm run build
+npm run check:rules
+npm test
+npm run example:minimal
+```
+
+## What To Do Next
+
+- Add real tools under `resources/tools`.
+- Let local module tools be auto-discovered, or declare non-module tools in
+  `config/catalogs/tools.yaml`.
+- Add specialist agents under `config/agents`.
+- Enable memory, approvals, or protocol exposure in `config/runtime/workspace.yaml`.
+- Embed the runtime in your service when the CLI path is stable.

package/docs/guides/index.md

@@ -0,0 +1,40 @@
+# Stable Harness Documentation
+
+Stable Harness is a stable application runtime and operator control plane for
+agent workspaces. These docs are organized for teams that want to adopt it,
+embed it, operate it, or explain why it exists.
+
+## Start Here
+
+- [Getting started](getting-started.md): install the package, initialize a
+  workspace, run a tool, and start the OpenAI-compatible facade.
+- [Workspace authoring](workspace-authoring.md): define agents, models, tools,
+  workflows, memory, and protocol exposure in YAML.
+- [Integration guide](integration-guide.md): embed the runtime inside an app,
+  run it through CLI, or expose it through HTTP/OpenAI-compatible clients.
+- [Operator runbook](operator-runbook.md): validate a workspace, inspect
+  events, run smoke tests, and keep the runtime operable.
+
+## Product And Adoption
+
+- [Adoption playbook](../product/adoption-playbook.md): practical paths for getting a
+  team, a product, or a community to try Stable Harness.
+- [Market positioning](../product/market-positioning.md): how to explain Stable Harness
+  relative to agent frameworks, protocol gateways, and orchestration systems.
+- [Product boundary](../product-boundary.md): the guardrails that keep the project
+  framework-generic and passthrough-first.
+- [Compatibility matrix](../compatibility-matrix.md): current backend and protocol
+  surfaces.
+
+## Engineering References
+
+- [Implementation blueprint](../implementation-blueprint.md)
+- [Adapter contract](../adapter-contract.md)
+- [Engineering rules](../engineering-rules.md)
+- [OpenAI-compatible protocol](../protocols/openai-compatible.md)
+- [LangGraph-compatible protocol](../protocols/langgraph-compatible.md)
+- [HTTP runtime protocol](../protocols/http-runtime.md)
+
+The short version: keep execution semantics upstream-native, define product
+inventory in YAML, and use Stable Harness for lifecycle, governance,
+observability, recovery, tool reliability, protocol access, and operator control.

package/docs/guides/integration-guide.md

@@ -0,0 +1,126 @@
+# Integration Guide
+
+Stable Harness can be used three ways: as a CLI, as an embedded runtime, or as a
+protocol facade. All three paths use the same workspace inventory and runtime
+boundary.
+
+## CLI Integration
+
+Use the CLI while developing or validating a workspace:
+
+```bash
+stable-harness -w ./workspace
+stable-harness -w ./workspace --agent orchestra "Review the latest release evidence."
+stable-harness -w ./workspace --agent orchestra --tool echo_tool --tool-args-json '{"value":"hello"}'
+```
+
+Render inventory without executing a request:
+
+```bash
+stable-harness agent render orchestra -w ./workspace
+stable-harness workflow render review-shell -w ./workspace
+stable-harness workflow inspect review-shell -w ./workspace
+```
+
+The CLI is intentionally thin. It loads the workspace, creates the tool gateway,
+starts memory services, registers adapters, and calls the core runtime.
+
+## Embedded Runtime
+
+Use the SDK when Stable Harness runs inside your service:
+
+```ts
+import { createStableHarnessRuntime } from "stable-harness";
+
+const runtime = await createStableHarnessRuntime("/srv/my-agent-workspace");
+
+const response = await runtime.request({
+  input: "Review the current release evidence.",
+  agentId: "orchestra",
+  sessionId: "customer-123",
+});
+
+console.log(response.output);
+```
+
+The runtime exposes operator surfaces for applications that need more than a
+single final answer:
+
+- `subscribe`: stream runtime events
+- `inspect`: inspect runtime inventory and state
+- `getRun`: read one run record
+- `listRequests`: list historical or active requests
+- `listSessions`: inspect session state
+- `inspectRequest`: inspect one request lifecycle
+- `cancel`: stop active work
+- `stop`: close the runtime
+
+Use those surfaces to build product UI around requests, approvals, event traces,
+artifacts, and memory lifecycle.
+
+## OpenAI-Compatible Facade
+
+Start the server:
+
+```bash
+stable-harness start -w ./workspace --host 127.0.0.1 --port 8642
+```
+
+Point compatible clients at:
+
+```text
+http://127.0.0.1:8642/v1
+```
+
+Use this path when an existing product or evaluation harness already speaks
+OpenAI-compatible chat completions. Stable Harness still owns workspace loading,
+runtime events, tool-gateway policy, memory lifecycle, and adapter selection.
+
+## HTTP Runtime Protocol
+
+Use the HTTP runtime protocol when the caller needs Stable Harness concepts
+directly: request IDs, sessions, traces, approvals, cancellation, and runtime
+inspection.
+
+See [HTTP runtime protocol](protocols/http-runtime.md).
+
+## Backend Adapters
+
+Adapters translate Stable Harness runtime requests into upstream backend calls.
+
+Current public adapter surfaces:
+
+- DeepAgents: primary agent backend
+- LangGraph: workflow and graph topology adapter
+- Custom adapters: implement the runtime adapter contract
+
+Adapter rule: preserve upstream execution semantics first. Add wrappers only for
+runtime lifecycle, governance, observability, persistence, recovery, protocol
+access, memory lifecycle, or tool-gateway control.
+
+## Tool Gateway
+
+The tool gateway is the boundary where Stable Harness can validate, repair,
+authorize, execute, and observe tool calls.
+
+The default CLI path configures BetterCall repair mode for registered tools.
+This helps small models recover malformed arguments, missing wrappers, or
+near-miss tool-call shapes without allowing arbitrary execution.
+
+Repair remains constrained:
+
+- registered tool inventory is authoritative
+- schemas and semantic validators are authoritative
+- approval and sandbox policy can block execution
+- repaired calls remain visible through runtime events
+
+## Production Embedding Checklist
+
+- Keep a workspace directory next to the service deployment.
+- Keep secrets in environment variables or the service secret manager.
+- Start one runtime per workspace or tenant boundary.
+- Subscribe to events and persist the run IDs your product exposes.
+- Treat request cancellation and timeout as product-level controls.
+- Run a registry/package smoke test before publishing a service image.
+- Validate OpenAI-compatible behavior only through the facade when that is the
+  public contract your callers use.

package/docs/guides/operator-runbook.md

@@ -0,0 +1,153 @@
+# Operator Runbook
+
+This runbook is for teams operating a Stable Harness workspace in development,
+CI, or production-like environments.
+
+## Validate The Package
+
+For framework development:
+
+```bash
+npm run check:rules
+npm run check
+npm test
+```
+
+Before publishing:
+
+```bash
+npm run release:pack
+npm run release:smoke
+```
+
+`release:smoke` installs the packed package into an isolated temporary project,
+initializes a workspace, and runs a CLI tool call. It catches missing published
+dependencies that monorepo tests can hide.
+
+## Validate A Workspace
+
+Show workspace status:
+
+```bash
+stable-harness -w ./workspace
+```
+
+Render agent inventory:
+
+```bash
+stable-harness agent render orchestra -w ./workspace
+```
+
+Render or inspect workflow topology:
+
+```bash
+stable-harness workflow render review-shell -w ./workspace
+stable-harness workflow inspect review-shell -w ./workspace
+```
+
+Run a direct tool smoke:
+
+```bash
+stable-harness -w ./workspace --agent orchestra --tool echo_tool --tool-args-json '{"value":"operator-smoke"}'
+```
+
+Run a model-backed request:
+
+```bash
+stable-harness -w ./workspace --agent orchestra --trace "Use the echo tool with operator-smoke."
+```
+
+## Start And Stop Protocol Serving
+
+Start:
+
+```bash
+stable-harness start -w ./workspace --host 127.0.0.1 --port 8642
+```
+
+Stop:
+
+```bash
+stable-harness stop -w ./workspace
+```
+
+Use `STABLE_HARNESS_OPENAI_HOST`, `STABLE_HARNESS_OPENAI_PORT`, and
+`STABLE_HARNESS_OPENAI_API_KEY` when the server is managed by an environment
+wrapper.
+
+## Read Runtime Evidence
+
+Prefer structured runtime evidence over final prose:
+
+- request ID
+- session ID
+- runtime event stream
+- tool invocation events
+- repaired tool-call diagnostics
+- approval decisions
+- memory lifecycle events
+- artifacts and exported traces
+
+Final answers are user-facing presentation. Runtime events are the operator
+record.
+
+## Common Failures
+
+### The CLI cannot find a package after publishing
+
+Run:
+
+```bash
+npm run release:smoke
+```
+
+If the smoke fails only after isolated install, add the missing dependency to
+the root package that is published to npm, not only to a workspace child package.
+
+### A tool exists in code but is unavailable to the agent
+
+Check both surfaces:
+
+- the tool module under `resources/tools`
+- the `kind: Tool` inventory document
+
+The runtime only exposes registered inventory.
+
+### A model can answer but does not call tools reliably
+
+Check the tool schema and the gateway events first. BetterCall repair can fix
+malformed calls, but it cannot authorize unknown tools or infer product intent
+from prose.
+
+### A workflow runs the wrong topology
+
+Render it:
+
+```bash
+stable-harness workflow render <workflow-id> -w ./workspace
+```
+
+Workflow edges are static control-plane topology. They should not depend on user
+phrasing or benchmark-specific cases.
+
+### Output looks correct but production behavior is uncertain
+
+Run the exact public path your users call:
+
+- CLI if the CLI is the product surface
+- OpenAI-compatible facade if clients use `/v1`
+- embedded runtime if your service calls the SDK
+
+Do not treat a nearby internal smoke as proof of a different public contract.
+
+## Release Evidence
+
+A release is ready when the evidence includes:
+
+- project rules pass
+- TypeScript check passes
+- full tests pass
+- package dry-run passes
+- isolated npm install smoke passes
+- the published version can initialize and run a real workspace
+- GitHub release and code-quality checks are green