ultra-builder-pro-cli 0.5.0

package/skills/ultra-verify/evals/evals.json

@@ -0,0 +1,51 @@
+{
+  "skill_name": "ultra-verify",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "/ultra-verify decision 'Should we use PostgreSQL or MongoDB for our new microservice?'",
+      "expected_output": "Claude answers first, then launches Gemini and Codex in parallel, collects all three analyses, and presents a synthesis with consensus scoring",
+      "assertions": [
+        "Claude writes its own analysis BEFORE reading external AI output",
+        "Launches Gemini and Codex in parallel (run_in_background)",
+        "Creates session directory under .ultra/collab/ with verify-decision in name",
+        "Reads all three output files with Read tool",
+        "Presents confidence level (consensus/majority/no consensus)",
+        "Includes per-AI recommendation summary and trade-off matrix"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "/ultra-verify diagnose 'API returns 500 intermittently under load, logs show connection pool exhaustion'",
+      "expected_output": "Three sets of hypotheses collected and ranked by consensus count",
+      "assertions": [
+        "Each AI provides top-3 hypotheses",
+        "Hypotheses ranked by consensus count (3/3 > 2/3 > 1/3)",
+        "Each hypothesis includes evidence and verification steps",
+        "Recommended investigation order based on consensus"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "/ultra-verify audit",
+      "expected_output": "Three-way code audit with findings graded by consensus (3=critical, 2=high, 1=investigate)",
+      "assertions": [
+        "Sends same audit prompt to all three AIs",
+        "Findings graded by how many AIs found them",
+        "3/3 findings marked Critical, 2/3 marked High, 1/3 marked Investigate",
+        "Action items prioritized by consensus count"
+      ]
+    },
+    {
+      "id": 4,
+      "prompt": "/ultra-verify estimate 'Implement OAuth2 login with Google and GitHub providers'",
+      "expected_output": "Three effort estimates compared for convergence",
+      "assertions": [
+        "Each AI provides estimate with breakdown and assumptions",
+        "Convergence analysis (within 20% = high confidence)",
+        "Outlier estimates investigated",
+        "Final recommended estimate with confidence level"
+      ]
+    }
+  ]
+}

package/skills/ultra-verify/references/confidence-system.md

@@ -0,0 +1,38 @@
+# Confidence System
+
+Confidence scoring based on AI consensus for cross-verify results.
+
+## Levels
+
+### Consensus (3/3 Agree)
+- **Confidence**: Highest
+- **Action**: Strongly recommended — proceed with high confidence
+- **Display**: `[CONSENSUS 3/3]`
+
+### Majority (2/3 Agree)
+- **Confidence**: High
+- **Action**: Recommended — investigate the dissenting view for edge cases or alternative perspectives
+- **Display**: `[MAJORITY 2/3]` + note which AI dissented and why
+
+### No Consensus (All Differ)
+- **Confidence**: Low
+- **Action**: Decompose the problem into smaller questions, gather more data, or accept that the answer is genuinely ambiguous
+- **Display**: `[NO CONSENSUS]` + summary of each position
+
+## Application by Mode
+
+| Mode | Consensus | Majority | No Consensus |
+|------|-----------|----------|--------------|
+| decision | Strong recommendation | Recommended + dissent analysis | Present all options, suggest decomposition |
+| diagnose | Investigate first | Investigate second | Check all, may need more data |
+| audit | Critical — fix now | High — likely real | Investigate — may be false positive |
+| estimate | Use average, high confidence | Investigate outlier | Decompose task further |
+
+## Degraded Confidence
+
+When only 2 AIs respond (one failed):
+
+- **2/2 Agree**: Equivalent to Majority (not Consensus — missing perspective)
+- **2/2 Differ**: Low confidence — note missing third opinion
+
+When only 1 AI responds: Claude-only analysis — explicitly mark as single-source, no consensus scoring.

package/skills/ultra-verify/references/cross-verify-modes.md

@@ -0,0 +1,73 @@
+# Ultra Verify Modes
+
+Four modes for three-way AI verification. All modes follow the safe orchestration contract in the parent `SKILL.md`.
+
+## 1. Decision (`decision`)
+
+For architecture decisions, technology choices, or design trade-offs.
+
+**Flow:**
+1. Present the decision context, constraints, and options to all three AIs independently
+2. Each AI provides: recommendation, pros/cons, risks, trade-offs
+3. Synthesis ranks options by consensus:
+   - 3/3 recommend same → **Strong recommendation**
+   - 2/3 agree → **Recommended** with dissenting view analyzed
+   - All different → Present all three with trade-off matrix
+
+**Output structure:**
+- Per-AI recommendation summary
+- Consensus matrix (which AI recommends what)
+- Trade-off analysis
+- Final recommendation with confidence level
+
+## 2. Diagnose (`diagnose`)
+
+For bug diagnosis, root cause analysis, or troubleshooting.
+
+**Flow:**
+1. Provide symptoms, error messages, relevant code, and recent changes to all three AIs
+2. Each AI provides its top-3 hypotheses with evidence and verification steps
+3. Synthesis:
+   - Hypotheses appearing in 3/3 lists → **Most likely** (investigate first)
+   - Hypotheses in 2/3 lists → **Probable** (investigate second)
+   - Unique hypotheses → **Worth checking** (may catch edge cases)
+
+**Output structure:**
+- Ranked hypothesis list (by consensus count)
+- Per-hypothesis: description, evidence, verification steps, suggested fix
+- Recommended investigation order
+
+## 3. Audit (`audit`)
+
+For code review, security audit, or quality assessment.
+
+**Flow:**
+1. Provide the code scope to all three AIs with the same audit prompt
+2. Each AI reports findings with severity and location
+3. Synthesis grades findings by consensus:
+   - Found by 3/3 → **Critical** — fix immediately
+   - Found by 2/3 → **High** — likely real, investigate
+   - Found by 1/3 → **Investigate** — may be false positive or edge case
+
+**Output structure:**
+- Findings table: description, severity, consensus count, which AIs found it
+- Grouped by consensus level (Critical → High → Investigate)
+- Action items prioritized by consensus
+
+## 4. Estimate (`estimate`)
+
+For effort estimation, complexity assessment, or timeline planning.
+
+**Flow:**
+1. Describe the task, requirements, and constraints to all three AIs
+2. Each AI provides: estimate (time/effort), breakdown, assumptions, risks
+3. Synthesis evaluates convergence:
+   - All within 20% of each other → **High confidence** estimate (use average)
+   - One outlier → **Moderate confidence** — investigate the outlier's reasoning
+   - All significantly different → **Low confidence** — task needs decomposition
+
+**Output structure:**
+- Per-AI estimate with breakdown and assumptions
+- Convergence analysis
+- Recommended estimate with confidence level
+- Risk factors that could shift the estimate

package/skills/ultra-verify/scripts/verify_wait.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+"""Verify Wait - File-based completion waiter for /ultra-verify pipeline.
+
+Polls the session directory for expected AI output files.
+Blocks until both AIs produce output OR timeout. Returns structured JSON on stdout.
+
+Two exit conditions only:
+    1. Both AIs have output (non-empty + stable size) → exit 0, status="complete"
+    2. Timeout → exit 0, status="timeout" (JSON status field tells the story)
+
+Expected files:
+    - gemini-output.md  (Gemini)
+    - codex-output.md   (Codex)
+
+Usage:
+    python3 verify_wait.py <session_path> [--timeout SECONDS]
+
+Exit codes:
+    0 - Always (result expressed via JSON status field)
+    2 - Invalid arguments (missing path, bad --timeout)
+"""
+
+import json
+import sys
+import time
+from pathlib import Path
+
+POLL_INTERVAL = 3  # seconds
+DEFAULT_TIMEOUT = 1200  # 20 min — runs via run_in_background (no Bash 600s limit)
+
+# Each AI produces exactly one output file
+EXPECTED = {
+    "gemini": "gemini-output.md",
+    "codex": "codex-output.md",
+}
+
+
+def _file_size(path: Path) -> int:
+    """Return file size in bytes, or -1 if not found."""
+    try:
+        return path.stat().st_size
+    except (FileNotFoundError, OSError):
+        return -1
+
+
+def _check_output(session_path: Path, name: str) -> dict:
+    """Check if an AI has produced non-empty output."""
+    output = session_path / EXPECTED[name]
+    if _file_size(output) > 0:
+        return {"name": name, "status": "complete", "file": str(output)}
+    return {"name": name, "status": "pending", "file": None}
+
+
+def _timeout_status(session_path: Path, name: str) -> dict:
+    """Determine final status at timeout. Check error logs and empty files."""
+    output = session_path / EXPECTED[name]
+    error = session_path / f"{name}-error.log"
+    output_size = _file_size(output)
+    if output_size > 0:
+        return {"name": name, "status": "complete", "file": str(output)}
+    if _file_size(error) > 0:
+        return {"name": name, "status": "failed", "file": str(error)}
+    if output_size == 0:
+        return {"name": name, "status": "empty", "file": str(output)}
+    return {"name": name, "status": "pending", "file": None}
+
+
+def main():
+    if len(sys.argv) < 2:
+        print(__doc__, file=sys.stderr)
+        sys.exit(2)
+
+    session_path = Path(sys.argv[1])
+    timeout = DEFAULT_TIMEOUT
+
+    if "--timeout" in sys.argv:
+        idx = sys.argv.index("--timeout")
+        if idx + 1 < len(sys.argv):
+            try:
+                timeout = int(sys.argv[idx + 1])
+            except ValueError:
+                print(f"Error: --timeout must be integer: {sys.argv[idx + 1]}", file=sys.stderr)
+                sys.exit(2)
+
+    if not session_path.is_dir():
+        print(f"Error: session directory not found: {session_path}", file=sys.stderr)
+        sys.exit(2)
+
+    deadline = time.monotonic() + timeout
+    prev_sizes = {name: _file_size(session_path / f) for name, f in EXPECTED.items()}
+
+    # Poll loop — exit condition 1: both outputs ready + stable
+    while time.monotonic() < deadline:
+        cur_sizes = {name: _file_size(session_path / f) for name, f in EXPECTED.items()}
+        gemini = _check_output(session_path, "gemini")
+        codex = _check_output(session_path, "codex")
+
+        gemini_done = gemini["status"] == "complete" and cur_sizes["gemini"] == prev_sizes["gemini"]
+        codex_done = codex["status"] == "complete" and cur_sizes["codex"] == prev_sizes["codex"]
+
+        if gemini_done and codex_done:
+            elapsed = int(timeout - (deadline - time.monotonic()))
+            print(json.dumps({"status": "complete", "gemini": gemini, "codex": codex, "elapsed_seconds": elapsed}))
+            sys.exit(0)
+
+        prev_sizes = cur_sizes
+
+        # Progress display
+        remaining = int(deadline - time.monotonic())
+        parts = []
+        for name, done, result in [("gemini", gemini_done, gemini), ("codex", codex_done, codex)]:
+            if done:
+                parts.append(f"{name}:complete")
+            elif result["status"] == "complete":
+                parts.append(f"{name}:stabilizing")
+            else:
+                parts.append(f"{name}:waiting")
+        done_count = int(gemini_done) + int(codex_done)
+        sys.stderr.write(f"\r  [{done_count}/2] {' | '.join(parts)} ({remaining}s remaining)  ")
+        sys.stderr.flush()
+        time.sleep(POLL_INTERVAL)
+
+    # Exit condition 2: timeout — check error logs NOW for final status
+    gemini = _timeout_status(session_path, "gemini")
+    codex = _timeout_status(session_path, "codex")
+    print(json.dumps({"status": "timeout", "gemini": gemini, "codex": codex, "elapsed_seconds": timeout}))
+    sys.exit(0)
+
+
+if __name__ == "__main__":
+    main()

package/spec/cli-protocol.md

@@ -0,0 +1,118 @@
+# `ultra-tools` CLI protocol
+
+`ultra-tools` is the shell fallback for the live MCP state contract
+(PLAN §4, D12). Every tool in `spec/mcp-tools.yaml` has a matching
+`ultra-tools <subcommand>`; host-native review, impact discovery, skill
+resolution, and human interaction are intentionally outside this protocol.
+
+Trace: PLAN §6 Phase 1.4, decisions D12 / D29 / D33 / D37.
+
+## 1. Invocation
+
+```
+ultra-tools <family> <verb> [--flag value] [positional ...]
+```
+
+- `family` and `verb` together correspond to a `tool.name` in
+  `spec/mcp-tools.yaml` (e.g. `task create`, `session admission`).
+- Flags map 1-to-1 to the `input_schema.properties` of the tool.
+- Positional args are reserved for the **id-like primary key** of the tool,
+  if the tool has one (`task get <id>`, `session get <sid>`). All other
+  fields must be flags so machine callers do not depend on argument order.
+- Boolean flags: `--takeover` for true; `--no-takeover` for false. Default
+  is whatever the schema declares.
+- JSON values: pass with `--json '<inline json>'` or `--json-file <path>`
+  when an input field is itself an object/array.
+
+## 2. Output
+
+The CLI may print human-readable lines to stdout while it runs. The **last
+line of stdout MUST be a single JSON object** matching the tool's
+`output_schema` on success, or an error envelope on failure:
+
+```json
+// success
+{ "ok": true, "data": { ... output_schema ... } }
+```
+
+```json
+// failure
+{ "ok": false, "error": { "code": "<ERROR_CODE>", "message": "...", "retriable": false } }
+```
+
+`error.code` MUST be one of the `errors[].code` values declared for that
+tool in `spec/mcp-tools.yaml`. Hooks parse this last line; any extra noise
+must come **before** it.
+
+Streaming subcommands (e.g. `task subscribe`) emit one JSON object per
+line during the stream, and finish with the standard envelope on stdout's
+last line.
+
+## 3. Exit codes
+
+| Code | Meaning                                           |
+|-----:|---------------------------------------------------|
+| `0`  | success — `data` envelope is on stdout            |
+| `1`  | user error (bad flags, missing required input)    |
+| `2`  | system error (DB unavailable, IO failure)         |
+| `3`  | lease conflict — see `error.code = ADMISSION_DENIED` (D33) |
+| `4`  | schema mismatch (input or output failed validation) |
+| `5`  | timeout — operation exceeded its budget           |
+| `124`| reserved for SIGTERM / killed by orchestrator     |
+
+Exit code is **the** ground truth for hooks; the JSON envelope is
+informational. Never rely on stderr for status.
+
+## 4. Stdin / stdout / stderr discipline
+
+- **stdin**: closed by default. Tools that accept large input
+  (`task parse-prd`) read from `--prd-file <path>` rather than stdin to
+  keep streaming pipelines deterministic.
+- **stdout**: human-readable lines + the trailing JSON envelope.
+- **stderr**: log lines only (structured JSON; `level / message / context`).
+  Never carry success data on stderr.
+
+## 5. Tool ↔ CLI mapping
+
+The mapping below is generated from `spec/mcp-tools.yaml` by
+`spec/scripts/check-cli-mapping.cjs` and asserted at gate time. This table
+is the authoritative reference; do not duplicate it in command md files.
+
+| MCP tool                  | CLI subcommand              | Phase | Writer  |
+|---------------------------|-----------------------------|------:|---------|
+| `task.create`             | `task create`               | 2     | mcp     |
+| `task.update`             | `task update`               | 2     | mcp     |
+| `task.list`               | `task list`                 | 2     | any     |
+| `task.get`                | `task get`                  | 2     | any     |
+| `task.delete`             | `task delete`               | 2     | mcp     |
+| `task.switch_tag`         | `task switch-tag`           | 7.2   | mcp     |
+| `task.init_project`       | `task init-project`         | 3     | mcp     |
+| `task.expand`             | `task expand`               | 8a    | mcp     |
+| `task.parse_prd`          | `task parse-prd`            | 8a    | mcp     |
+| `task.dependency_topo`    | `task topo`                 | 8a    | mcp     |
+| `task.append_event`       | `task append-event`         | 2     | any     |
+| `task.subscribe_events`   | `task subscribe`            | 2     | any     |
+| `session.spawn`           | `session spawn`             | 4.5   | mcp     |
+| `session.close`           | `session close`             | 4.5   | mcp     |
+| `session.get`             | `session get`               | 4.5   | any     |
+| `session.list`            | `session list`              | 4.5   | any     |
+| `session.admission_check` | `session admission`         | 4.5   | any     |
+| `session.heartbeat`       | `session heartbeat`         | 4.5   | mcp     |
+| `session.subscribe_events`| `session subscribe`         | 4.5   | any     |
+| `plan.export`             | `plan export`               | 8a    | mcp     |
+| `plan.get`                | `plan get`                  | 8a    | any     |
+
+`Writer = mcp` rows go through the MCP server's single-writer connection;
+the CLI shells out to the running MCP server over stdio rather than
+touching `.ultra/state.db` directly. `Writer = any` rows are safe to
+execute from any process — read-only queries and `events` appends fall
+into this bucket because `events.id AUTOINCREMENT` makes concurrent
+INSERTs collision-free under WAL + `busy_timeout`. The full policy lives
+in `docs/STATE-DB-ACCESS-POLICY.md` (Phase 2.2 / R25).
+
+## 6. Versioning
+
+The CLI follows the `version` field of `spec/mcp-tools.yaml`. Adding a new
+flag is a minor bump; renaming or removing a flag, or changing exit-code
+semantics, is a major bump. Major bumps require an entry in
+`docs/PLAN.zh-CN.md §14 Decision log`.