nonoka-cli 0.2.5__tar.gz → 0.2.10__tar.gz

{nonoka_cli-0.2.5 → nonoka_cli-0.2.10}/.gitignore

@@ -61,4 +61,12 @@ verdaccio-storage/
 
 # Private planning documents
 DESIGN.md
-REFACTOR.md
+REFACTOR.md
+/SWE_BENCH_FOLLOW_UP.md
+/TODO.md
+/nonoka.yaml
+
+# Runtime nonoka project artifacts
+.nonoka/repo_map.jsonl
+.nonoka/tasks/
+.nonoka/eval/

{nonoka_cli-0.2.5 → nonoka_cli-0.2.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nonoka-cli
-Version: 0.2.5
+Version: 0.2.10
 Summary: OpenCode backend for the Nonoka Agent framework
 License: MIT
 License-File: LICENSE
@@ -14,7 +14,7 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.10
-Requires-Dist: nonoka>=1.3.4
+Requires-Dist: nonoka>=1.3.7
 Requires-Dist: pydantic>=2.0
 Requires-Dist: python-dotenv>=1.0
 Requires-Dist: pyyaml>=6.0
@@ -29,6 +29,8 @@ Description-Content-Type: text/markdown
 
 # nonoka-cli
 
+English | [简体中文](README.zh-CN.md)
+
 OpenCode backend for the [Nonoka](https://pypi.org/project/nonoka/) Agent framework.
 
 `nonoka-cli` runs as a stdio NDJSON bridge server (`nonoka-cli --server`) that
@@ -99,7 +101,7 @@ For scripted setups, use the non-interactive mode (you'll still need to set the
 API key via `.env` or `export`):
 
 ```bash
-nonoka-cli config init --yes --model deepseek-chat
+nonoka-cli config init --yes --model deepseek/deepseek-v4-pro
 ```
 
 2. Generate an OpenCode config in the current project or globally:
@@ -130,10 +132,11 @@ Example output:
 
 ```
 nonoka-cli doctor
-✓ nonoka-cli 0.2.5
+✓ nonoka-cli 0.2.10
 ✓ Python 3.11
 ✓ opencode 1.18.2
-✓ provider nonoka-opencode-provider@0.2.12
+✓ provider nonoka-opencode-provider@0.2.16
+✓ nonoka framework 1.3.7
 ✓ config ~/.config/nonoka/config.yaml
 ✓ API key DEEPSEEK_API_KEY set
 ✓ OpenCode provider config in /home/user/.config/opencode/opencode.json
@@ -146,12 +149,196 @@ key with a real (small) call, use:
 nonoka-cli doctor --check-llm
 ```
 
+Verify the Docker-backed command sandbox separately:
+
+```bash
+nonoka-cli doctor --check-sandbox
+```
+
+## Execution Observability
+
+Every local runner session writes credential-redacted structured events to
+`~/.local/share/nonoka/events.db`. Events include LLM prompts/responses,
+tool I/O, errors, and LiteLLM token/cost usage. Inspect them without opening
+the database directly:
+
+```bash
+nonoka-cli sessions list
+nonoka-cli sessions show <session-id>
+nonoka-cli logs --session-id <session-id>
+nonoka-cli logs --json
+```
+
+The framework exposes a provider-neutral `TelemetryExporter` protocol and
+`ObservabilityPipeline`; downstream applications can add Langfuse, OTLP, or
+other exporters without coupling `Runner` to a vendor SDK. Export failures are
+best-effort and never interrupt an agent run.
+
+## Service Deployment
+
+`nonoka-agent` includes an authenticated FastAPI application with `/run`,
+`/chat`, `/tasks`, `/health`, and `/metrics`. Set a bearer token before
+starting it:
+
+```bash
+export NONOKA_API_TOKEN="replace-with-a-long-random-token"
+uv run uvicorn nonoka.server.app:create_app --factory --host 0.0.0.0 --port 8000
+```
+
+For a containerized deployment, copy `.env.example` to `.env`, set
+`NONOKA_API_TOKEN`, then run `docker compose up --build`. Compose defaults to
+PostgreSQL for persisted events; local development stays on SQLite. The service
+is non-root, read-only, drops Linux capabilities, and does not mount the host
+Docker socket.
+
+## Agent evaluation
+
+`nonoka-cli eval` is a thin frontend for the framework-owned benchmark engine. Its
+scored built-ins are the open HumanEval and MBPP datasets. The bundled
+`tool_use` suite is deliberately labelled as deterministic smoke/regression
+coverage; it is not presented as a substitute for an open benchmark. Results
+are stored under `.nonoka/eval/` in the current project and can be compared
+locally:
+
+```bash
+nonoka-cli eval list
+nonoka-cli eval run --dataset humaneval --model deepseek/deepseek-v4-pro --limit 20
+nonoka-cli eval run --dataset mbpp --model deepseek/deepseek-v4-pro --limit 20
+nonoka-cli eval leaderboard
+```
+
+Each built-in run records a normal Nonoka agent and a same-model direct
+baseline, including pass@1, turns, tool calls, token usage, wall time, and the
+agent lift. Live model calls are opt-in and are never part of the default test
+suite.
+
+For a release comparison, create a manifest before incurring any model cost.
+It pins the model policy and includes HumanEval, MBPP sanitized, EvalPlus,
+τ³ retail/airline, and Terminal-Bench. EvalPlus runs in a separate environment
+because it owns the official strengthened verifier:
+
+```bash
+nonoka-cli eval matrix plan --model deepseek/deepseek-v4-pro --output .nonoka/eval/release-matrix.json
+export NONOKA_EVALPLUS_PYTHON=/path/to/evalplus-python
+nonoka-cli eval matrix run --manifest .nonoka/eval/release-matrix.json --include evalplus-humaneval
+```
+
+For complex agent tasks, the framework delegates to official harnesses instead
+of reimplementing their verifiers. τ³-bench (the `tau2-bench` package) provides
+multi-turn customer-service tasks with policies, simulated users, environment
+tools, and action-level reward; it runs from an isolated Python 3.12
+environment because of its dependency pins:
+
+```bash
+export NONOKA_TAU2_PYTHON=/path/to/tau2-python
+nonoka-cli eval external run --benchmark tau2-bench --model deepseek/deepseek-v4-pro --domain retail --limit 10
+```
+
+For framework-only terminal-agent tasks, Terminal-Bench 2 delegates Docker
+lifecycle and verification to Harbor. The OpenCode bridge has its own
+reproducible benchmark command, so a framework score is never presented as a
+CLI bridge score. Install Harbor plus the local framework/CLI checkouts in an
+isolated environment, then verify prerequisites before starting a live run:
+
+```bash
+uv venv .venv-bench --python 3.13
+uv pip install --python .venv-bench/bin/python -e ../nonoka-agent -e . harbor
+export NONOKA_HARBOR_BIN="$PWD/.venv-bench/bin/harbor"
+nonoka-cli doctor --check-benchmarks
+nonoka-cli benchmark smoke --model deepseek/deepseek-v4-pro
+nonoka-cli benchmark terminal-bench --model deepseek/deepseek-v4-pro
+```
+
+The Terminal-Bench command builds fresh local wheels, stages the built OpenCode
+provider plus its dependencies, and copies verified host OpenCode and Python
+3.13 runtime artifacts into every Harbor task container. The runtime is also
+registered in uv's conventional managed-Python directory, so both the non-root
+agent and official verifier scripts can reuse it without downloading another
+interpreter. This ensures the
+official verifier observes OpenCode using the current nonoka bridge against the task
+filesystem—not a host-side shell. To verify provisioning before spending model
+tokens, run one pinned task first:
+
+```bash
+nonoka-cli benchmark terminal-bench --task regex-log --install-only
+```
+
+The adapter also exposes the staged `uv` as `/root/.local/bin/uv` and `uvx`.
+Verifier scripts that replace that binary still discover the registered Python
+runtime. This does not preinstall task-specific test packages, data, or solution
+artifacts.
+
+Live benchmark runs do not impose cumulative model-turn, tool-call, or
+per-model-call limits by default. The one-hour process watchdog only protects
+against a lost or permanently stuck agent process. Use `--max-turns`,
+`--tool-budget`, or `--timeout` when a deliberately bounded profile is needed.
+
+Harbor receives the model credential through its `${DEEPSEEK_API_KEY}`
+environment template. The value is never placed in the benchmark manifest or
+artifact directory.
+
+Each bridge run writes a credential-redacted manifest, OpenCode JSON events,
+provider/bridge traces, and a reference to the official Harbor job directory
+under `.nonoka/eval/opencode/`. Docker access is required.
+
+SWE-bench Lite uses the official verifier and keeps its bridge artifacts
+separate from Terminal-Bench. It requires the official `swebench` package,
+Docker, at least 120 GiB free disk, and 16 GiB RAM for a full Lite run. A
+single explicit instance may be generated and verified on a constrained host:
+
+```bash
+nonoka-cli benchmark swe-bench --instance-id django__django-10914 \
+  --model deepseek/deepseek-v4-flash \
+  --swebench-python /path/to/swebench-venv/bin/python \
+  --max-workers 1 \
+  --artifact-dir .nonoka/eval/swe-flash-django-10914
+```
+
+To verify a previously generated prediction without another model call, pass
+its official `predictions.jsonl` instead:
+
+```bash
+nonoka-cli benchmark swe-bench --instance-id <instance-id> \
+  --predictions /path/to/predictions.jsonl \
+  --artifact-dir .nonoka/eval/swe-lite-<instance-id>
+```
+
+The command writes a verifier command, stdout/stderr, `diagnosis.json`, and a
+human-readable diagnosis. It classifies infrastructure, bridge/provider,
+agent-loop, and verifier failures separately; use the same instance for an
+explicit Aider or native OpenCode comparison after a healthy bridge run fails.
+
+### Verified benchmark and regression results
+
+The current verification-contract and bounded multi-agent implementation has passed the following checks against the local sibling `nonoka-agent` checkout:
+
+- `543` deterministic nonoka-agent tests passed (`45` opt-in live tests were deselected).
+- `280` nonoka-cli unit, integration, and bridge tests passed.
+- `62` OpenCode provider tests passed, followed by a successful TypeScript build.
+- A clean OpenCode TUI multi-agent run used `agent__spawn` for both planning and review, recovered two intentionally partial file observations with bounded follow-up reads, completed a real workspace change, and passed all `16` focused acceptance tests. Its final response turn exposed no tools and made no host tool calls.
+- Official SWE-bench Lite verification for eight distinct instances: `astropy__astropy-12907`, `django__django-10914`, `django__django-10924`, `django__django-11001`, `django__django-11099`, `pytest-dev__pytest-11143`, `pallets__flask-4045`, and `sympy__sympy-11400`.
+
+The pinned `swe-flash-selected10-v1` regression sample resolved 6 of 10 instances with `deepseek/deepseek-v4-flash`. Subsequent `deepseek/deepseek-v4-pro` runs independently resolved several Django and cross-project instances and, after the verification-contract remediation, resolved the previously failing `pallets__flask-4045` and `sympy__sympy-11400` instances. All reported results come from the official SWE-bench verifier rather than model-authored assertions. They are targeted engineering samples, not a claim of a full SWE-bench Lite score.
+
+Reference end-to-end validation with `deepseek/deepseek-v4-pro` against the
+pinned Terminal-Bench 2 revision (`69671fba`) has earned official Harbor reward
+`1` on three distinct tasks: `sanitize-git-repo`, `configure-git-webserver`, and
+`break-filter-js-from-html`. The latest bridge-hardening rerun of
+`break-filter-js-from-html` completed without an exception in 10m21s. These are
+single-trial engineering checks, not a statistically representative leaderboard
+score.
+
+`benchmark smoke` pins OpenCode to the checkout's built provider with a local
+`file:` dependency and temporarily writes an isolated `opencode.json` in the
+benchmark workspace. Use a clean `--cwd` (or pass `--provider-source`) so it
+never overrides an existing project OpenCode configuration.
+
 ## Configuration
 
 ### `nonoka-cli config init`
 
 Interactive wizard that writes `~/.config/nonoka/config.yaml`. It asks for a
-model identifier (e.g. `deepseek-chat`, `openai/gpt-4o`, `ollama/llama3.3`), a
+model identifier (e.g. `deepseek/deepseek-v4-pro`, `openai/gpt-4o`, `ollama/llama3.3`), a
 masked API key, and whether to save it to `~/.config/nonoka/.env` (recommended),
 directly in `config.yaml`, or skip saving. It also asks for a system prompt and
 whether to auto-approve all tool calls.
@@ -176,6 +363,46 @@ nonoka-cli config set hitl.dangerous_tools '["write_file", "execute_command"]'
 
 Print the resolved configuration and its file path.
 
+### Cost controls and response cache
+
+The runner keeps a local SQLite exact-response cache by default. It only stores
+complete responses without tool calls, and its key includes the model, complete
+message history, tool schema, generation settings, and workspace namespace.
+
+Semantic reuse is opt-in because it has a stricter correctness contract. It
+uses an OpenAI-compatible embedding endpoint only for deterministic, tool-free,
+single-turn requests in a Git worktree. The cache scope is recomputed before
+each completion from `HEAD`, tracked/untracked changes, the repo-map index,
+system prompt, and workspace path. A write made earlier in the same OpenCode
+session therefore invalidates semantic candidates immediately. Non-Git
+workspaces, tool calls, and multi-turn conversations fall back to the model.
+
+```yaml
+cache:
+  enabled: true
+  path: ~/.cache/nonoka/llm-cache.sqlite3
+  ttl_seconds: 604800
+  semantic_enabled: true
+  embedding_model: qwen3.7-text-embedding
+  embedding_api_base: https://your-endpoint/compatible-mode/v1
+  embedding_api_key_env: DASHSCOPE_API_KEY
+  embedding_dimensions: 1024
+  semantic_threshold: 0.92
+
+budget:
+  max_total_tokens: 120000
+  max_cost_usd: 5.0
+  fail_on_unknown_cost: true
+```
+
+Keep the embedding credential in the named environment variable or
+`~/.config/nonoka/.env`, never in `config.yaml`. A semantic hit is recorded as
+saved usage rather than actual spend; the event store exposes cache source and
+similarity score without storing raw cache queries. `max_total_tokens` and
+`max_cost_usd` are hard limits persisted with the task session; when price data
+is unavailable, `fail_on_unknown_cost: true` terminates the task rather than
+silently exceeding the cost budget.
+
 ### `nonoka-cli opencode init`
 
 Generate or merge an `opencode.json` in the current directory and create
@@ -207,7 +434,7 @@ A typical generated `opencode.json` looks like:
         "configPath": "~/.config/nonoka/config.yaml"
       },
       "models": {
-        "default": { "name": "Nonoka deepseek-chat" }
+        "default": { "name": "Nonoka deepseek-v4-pro" }
       }
     }
   },
@@ -292,7 +519,7 @@ nonoka-cli can merge MCP tools and lazy-loaded skills alongside OpenCode's
 native tools. Configure them in `~/.config/nonoka/config.yaml`:
 
 ```yaml
-model: deepseek-chat
+model: deepseek/deepseek-v4-pro
 
 mcp_servers:
   filesystem:
@@ -305,13 +532,12 @@ skills:
   - nextjs-best-practices
 ```
 
+Store each skill at `.agents/skills/<name>/SKILL.md` in the project or at `~/.agents/skills/<name>/SKILL.md` for user-wide use. Project definitions take precedence. Legacy flat `skills/<name>.md` files are still recognized.
+
 - **MCP tools** are executed locally by nonoka-cli and are exposed with a
   `mcp__<server>__<tool>` namespace prefix so they do not collide with OpenCode
   native tools.
-- **Skills** use nonoka-agent's lazy `SkillRegistry`. Only names and
-  descriptions are injected into the system prompt; full guidance is loaded
-  on-demand via the `load_skill` tool. Skill tools are prefixed with
-  `skill__<skill>__<tool>` in external-tools mode.
+- **Skills** use nonoka-agent's lazy `SkillRegistry`. Discovery reads names and descriptions without importing skill tools; enabled skill tools are resolved when the runtime catalog is built, while full guidance, its root directory, and bundled resource paths are loaded on-demand via `load_skill` and protected from normal context compaction. Skill tools are prefixed with `skill__<skill>__<tool>` in external-tools mode.
 
 Both MCP tools and skill tools remain available in standalone mode without any
 prefixing.
@@ -330,10 +556,7 @@ tools:
 When loaded, use the say_hello tool to greet the user by name.
 ```
 
-`greet_tool:say_hello` is resolved by Python's normal import machinery, so the
-module must be importable from the project working directory (or from a directory
-on `PYTHONPATH`). Place the tool module next to your skill file or add the skill
-source directory to `PYTHONPATH` if you use a nested layout.
+`greet_tool:say_hello` is resolved by Python's normal import machinery, so the module must be importable from the project working directory (or from a directory on `PYTHONPATH`). For a tool file bundled with the skill, use a path relative to `SKILL.md`, for example `file: scripts/greet_tool.py:say_hello`.
 
 ### Avoiding OpenCode's native `skill` tool
 
@@ -376,55 +599,58 @@ query the index on demand.
 
 ## Sub-agent workflow
 
-`nonoka-cli` can optionally expose two sub-agent tools that the main agent can
-call for complex tasks:
-
-- `plan_task` — delegates planning to a dedicated **planner** agent and returns a
-  numbered, file-level execution plan.
-- `review_changes` — delegates final review to a dedicated **reviewer** agent and
-  returns a structured review with issues, suggestions, and an approval flag.
-
-Both are disabled by default. Enable them by setting their model in `nonoka.yaml`:
-
-```yaml
-model: deepseek-chat
-max_turns: 20
+Projects may declare bounded advisory roles in `.nonoka/plugin.json`. Each valid role becomes a local tool named `agent__<role>`; the main Agent decides whether to delegate to it.
 
-agents:
-  planner:
-    model: deepseek-chat
-    system_prompt: "You are a planning agent..."
-  reviewer:
-    model: deepseek-chat
-    system_prompt: "You are a senior code reviewer..."
+```json
+{
+  "schema_version": "1.0",
+  "name": "project-agents",
+  "agents": [
+    {
+      "name": "planner",
+      "description": "Produce a concise implementation plan.",
+      "model": "deepseek/deepseek-v4-pro",
+      "system_prompt": "Return a numbered plan, risks, and focused checks.",
+      "max_turns": 2,
+      "max_invocations": 1,
+      "allowed_tools": []
+    },
+    {
+      "name": "reviewer",
+      "description": "Review a proposed change for blocking defects.",
+      "model": "deepseek/deepseek-v4-pro",
+      "system_prompt": "Return blocking issues, missing requirements, and an approval decision.",
+      "output_contract": "review",
+      "max_turns": 3,
+      "max_invocations": 2,
+      "allowed_tools": []
+    }
+  ],
+  "dynamic_agent": {
+    "enabled": true,
+    "model": "deepseek/deepseek-v4-pro",
+    "base_system_prompt": "You are a temporary advisory sub-agent. Return concise, actionable findings to the parent.",
+    "max_turns": 2,
+    "max_invocations": 2,
+    "max_instruction_chars": 2000,
+    "max_context_chars": 16000
+  }
+}
 ```
 
-`max_turns` at the top level controls the main executor agent. The
-`planner`/`reviewer` roles each have their own `max_turns` inside
-`agents.<role>`.
+The tool input is `{"task": "...", "context": "..."}`. Child Agents use isolated memory, one-level delegation, bounded turns, and no tools. They cannot read or modify the workspace, so the parent must include all relevant evidence in `context`; their output is advisory and cannot replace editing or verification.
 
-When enabled, these tools are injected into the main agent's tool list
-alongside OpenCode's native tools. The main agent decides when to call them;
-the planner/reviewer run inside their own short-lived `nonoka` agent invocation
-and return their results as tool output.
+When `dynamic_agent.enabled` is true, the main Agent also receives `agent__spawn`. It may choose a bounded `role`, `instructions`, `task`, and `context`, but the project policy fixes the model, base prompt, turn budget, invocation budget, and input sizes. Dynamically created children are still tool-free and cannot create further agents. The tool deliberately accepts no `model`, `tools`, permission, or budget arguments.
 
-`review_changes` accepts an optional `files` argument. When the main agent
-passes file paths, the reviewer reads those files and prepends their contents
-to the review context automatically:
+After the configured mutation and verification evidence is satisfied, Nonoka uses a tool-free finalization turn. This prevents optional cleanup or repeated checks from consuming the remaining turn budget after a task is already done. For evidence-gated runs, `maxTurns` therefore counts work turns and the runtime reserves one additional model call solely to produce the final response.
 
-```yaml
-# In the conversation the model can call:
-# review_changes({
-#   "task": "Review the changes against the goal: add logging",
-#   "context": "<diff or summary>",
-#   "files": ["src/main.py", "src/utils.py"]
-# })
+Validate the effective role configuration with:
+
+```bash
+nonoka-cli plugin validate --manifest .nonoka/plugin.json
 ```
 
-> **Note:** `plan_task` and `review_changes` used to live in `nonoka-agent`. They
-> have been moved to `nonoka-cli` so that sub-agent configuration (model, system
-> prompt, max turns) is controlled by the CLI config and can use different
-> models from the main agent.
+Set `NONOKA_DISABLE_PROJECT_AGENTS=1` to disable both static project roles and dynamic spawning. Benchmark profiles set this automatically so existing single-Agent scores remain comparable.
 
 ## Plugin manifest
 
@@ -432,13 +658,12 @@ Projects can declare their own Nonoka plugins via `.nonoka/plugin.json`:
 
 ```json
 {
+  "schema_version": "1.0",
   "name": "my-plugin",
-  "skills": ["code-review"],
-  "agents": {
-    "planner": { "system_prompt": "..." }
-  },
-  "mcpServers": {},
-  "allowedTools": ["read", "edit", "bash"]
+  "skills": [{"name": "code-review"}],
+  "agents": [],
+  "mcp_servers": {},
+  "allowed_tools": ["read", "edit", "bash"]
 }
 ```
 
@@ -526,7 +751,7 @@ src/nonoka_cli/
 ├── config/          # YAML config loading and Pydantic models
 ├── core/            # Orchestrator, RunnerService, SessionService, ToolService,
 │                    # MCPService, AgentFactory, prompt/context/task-state/output pruning,
-│                    # git safety net, repo map, planning, and plugin manifest
+│                    # git safety net, repo map, project agents, and plugin manifest
 │                    #   agent_factory.py              # Build nonoka Agent from CLI config
 │                    #   prompt_builder.py             # System prompt assembly for OpenCode mode
 │                    #   context_trimmer.py            # Turn-based context window trimming
@@ -534,9 +759,8 @@ src/nonoka_cli/
 │                    #   tool_output_policy.py         # Tool output pruning / spill policy
 │                    #   git_service.py                # Git checkpoint / rollback helpers
 │                    #   repo_map_service.py           # Symbol index generation and search
-│                    #   planning_service.py           # Planner sub-agent (AgentTool)
-│                    #   review_service.py             # Reviewer sub-agent (AgentTool)
 │                    #   plugin_manifest.py            # .nonoka/plugin.json loader
+│                    #   project_agents.py             # Compile bounded manifest roles
 │                    #   plugin_manifest_converter.py  # OpenCode skill/permission conversion
 ├── mcp/             # MCP server lifecycle manager (thin wrapper around nonoka-agent)
 ├── sessions/        # Session metadata persistence