pilot-workers 0.2.0__tar.gz → 0.4.0__tar.gz

pilot_workers-0.4.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: pilot-workers
+Version: 0.4.0
+Summary: Dispatch bounded tasks to isolated LLM workers (GLM, Kimi, and more)
+License-Expression: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0; extra == "yaml"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Dynamic: license-file
+
+# pilot-workers
+
+Dispatch bounded tasks to isolated LLM workers. Your main AI agent (Claude, Codex, or any planner) stays in control of requirements, planning, and verification — the worker only executes what it's told.
+
+## What it does
+
+- **Provider isolation**: each model (GLM, Kimi, DeepSeek, or your own) gets its own credentials, XDG directories, logs, and session storage. No cross-contamination.
+- **Fixed routing**: provider, model, and endpoint are locked per YAML config. Tasks cannot override them.
+- **Security by default**: API keys never appear in CLI args, environment variables, task contracts, or logs. Output is auto-redacted.
+- **Five modes**: `code` (edit), `explore` (read-only), `test` (run tests), `review` (read-only audit), `resume` (continue a prior code session).
+- **Pluggable runners**: the runner adapter layer (`Runner` ABC) abstracts engine-specific details. Currently ships with OpenCode; designed for future alternatives.
+- **Observable**: two-line JSON contract (`started` + `verdict`) for AI planners; human-readable `latest.log` for `tail -f`.
+- **Per-provider installs**: `install glm on claude` deploys only GLM's integration to Claude Code. Mix and match providers × hosts.
+
+## Install
+
+```bash
+pip install pilot-workers
+```
+
+## Quick start
+
+```bash
+# 1. Install the worker runtime
+pilot-workers install runner opencode
+
+# 2. Configure credentials (interactive, key never displayed)
+pilot-workers credentials glm
+pilot-workers credentials kimi-k3
+pilot-workers credentials ds
+
+# 3. Deploy integrations to your host
+pilot-workers install all on claude    # Claude Code: 12 agents + 8 slash commands
+pilot-workers install all on codex     # Codex: $glm / $kimi / $ds skills
+pilot-workers install glm on claude    # or just one provider on one host
+
+# 4. Check everything is ready
+pilot-workers status
+
+# 5. Verify with a dry-run
+pilot-workers run --provider glm --mode explore --workdir . --task "hello" --dry-run
+
+# 6. Run a real task
+pilot-workers template code > /tmp/task.md    # generate a structured task template
+# fill in the template, then:
+pilot-workers dispatch --provider glm --mode code --workdir /path/to/project --task-file /tmp/task.md
+# dispatch stdout = exactly two JSON lines: worker_runner.started + worker_runner.verdict
+```
+
+## CLI reference
+
+```
+pilot-workers <subcommand> [args]
+
+  run              Dispatch a task (streaming output).
+  dispatch         Deterministic wrapper around run (two-line JSON: started + verdict).
+  fanout           Dispatch several jobs concurrently; stdout = one JSON array of verdicts.
+  template         Print the task template for a mode (code|explore|test|review).
+  install          Install integrations or runner.
+                     install <provider|all> on <host|all>
+                     install runner <name>
+  uninstall        Remove integrations or runner.
+                     uninstall <provider|all> on <host|all>
+                     uninstall runner <name>
+  status           Show provider credentials, host installs, and runner state.
+                     status [--json]
+                     status <provider> on <host>
+  credentials      Configure isolated worker credentials.
+  maintain         Worker log and worktree lifecycle tools.
+```
+
+## Adding a new provider
+
+Drop a YAML file in `data/providers/` (inside the package):
+
+```yaml
+key: my-model
+provider_id: my-worker
+model_id: my-model-v1
+base_url: https://api.example.com/v1
+display_name: My Model Worker
+context_tokens: 128000
+output_tokens: 8192
+# runner: opencode          # optional, default opencode
+# permissions: relaxed      # optional, reference a permission profile
+# asset_prefix: my-model    # optional, default = key; used for integration file naming
+```
+
+Then `pilot-workers credentials my-model` and `pilot-workers install my-model on claude`.
+
+Reserved keys (cannot be used as provider key): `runner`, `all`, `on`, `claude`, `codex`.
+
+## Host integration
+
+The **host** is whatever AI agent acts as the planner. `integrations/` has ready-made configs:
+- **`claude-host/`**: 12 agents (glm/kimi/ds × coder/explorer/reviewer/tester) + 8 slash commands
+- **`codex-host/`**: `$glm` / `$kimi` / `$ds` skill entry points
+
+Adding a new host: create `integrations/<name>-host/`, put whatever config your host needs, point it at `pilot-workers dispatch`. See `integrations/README.md`.
+
+## Architecture
+
+See [CLAUDE.md](CLAUDE.md) for the current architecture, module reference, and conventions. See [docs/architecture.md](docs/architecture.md) for the detailed contract and security model.
+
+## Development
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
+.venv/bin/pytest    # 130 tests, all offline
+```
+
+## License
+
+MIT

pilot_workers-0.4.0/README.md

@@ -0,0 +1,114 @@
+# pilot-workers
+
+Dispatch bounded tasks to isolated LLM workers. Your main AI agent (Claude, Codex, or any planner) stays in control of requirements, planning, and verification — the worker only executes what it's told.
+
+## What it does
+
+- **Provider isolation**: each model (GLM, Kimi, DeepSeek, or your own) gets its own credentials, XDG directories, logs, and session storage. No cross-contamination.
+- **Fixed routing**: provider, model, and endpoint are locked per YAML config. Tasks cannot override them.
+- **Security by default**: API keys never appear in CLI args, environment variables, task contracts, or logs. Output is auto-redacted.
+- **Five modes**: `code` (edit), `explore` (read-only), `test` (run tests), `review` (read-only audit), `resume` (continue a prior code session).
+- **Pluggable runners**: the runner adapter layer (`Runner` ABC) abstracts engine-specific details. Currently ships with OpenCode; designed for future alternatives.
+- **Observable**: two-line JSON contract (`started` + `verdict`) for AI planners; human-readable `latest.log` for `tail -f`.
+- **Per-provider installs**: `install glm on claude` deploys only GLM's integration to Claude Code. Mix and match providers × hosts.
+
+## Install
+
+```bash
+pip install pilot-workers
+```
+
+## Quick start
+
+```bash
+# 1. Install the worker runtime
+pilot-workers install runner opencode
+
+# 2. Configure credentials (interactive, key never displayed)
+pilot-workers credentials glm
+pilot-workers credentials kimi-k3
+pilot-workers credentials ds
+
+# 3. Deploy integrations to your host
+pilot-workers install all on claude    # Claude Code: 12 agents + 8 slash commands
+pilot-workers install all on codex     # Codex: $glm / $kimi / $ds skills
+pilot-workers install glm on claude    # or just one provider on one host
+
+# 4. Check everything is ready
+pilot-workers status
+
+# 5. Verify with a dry-run
+pilot-workers run --provider glm --mode explore --workdir . --task "hello" --dry-run
+
+# 6. Run a real task
+pilot-workers template code > /tmp/task.md    # generate a structured task template
+# fill in the template, then:
+pilot-workers dispatch --provider glm --mode code --workdir /path/to/project --task-file /tmp/task.md
+# dispatch stdout = exactly two JSON lines: worker_runner.started + worker_runner.verdict
+```
+
+## CLI reference
+
+```
+pilot-workers <subcommand> [args]
+
+  run              Dispatch a task (streaming output).
+  dispatch         Deterministic wrapper around run (two-line JSON: started + verdict).
+  fanout           Dispatch several jobs concurrently; stdout = one JSON array of verdicts.
+  template         Print the task template for a mode (code|explore|test|review).
+  install          Install integrations or runner.
+                     install <provider|all> on <host|all>
+                     install runner <name>
+  uninstall        Remove integrations or runner.
+                     uninstall <provider|all> on <host|all>
+                     uninstall runner <name>
+  status           Show provider credentials, host installs, and runner state.
+                     status [--json]
+                     status <provider> on <host>
+  credentials      Configure isolated worker credentials.
+  maintain         Worker log and worktree lifecycle tools.
+```
+
+## Adding a new provider
+
+Drop a YAML file in `data/providers/` (inside the package):
+
+```yaml
+key: my-model
+provider_id: my-worker
+model_id: my-model-v1
+base_url: https://api.example.com/v1
+display_name: My Model Worker
+context_tokens: 128000
+output_tokens: 8192
+# runner: opencode          # optional, default opencode
+# permissions: relaxed      # optional, reference a permission profile
+# asset_prefix: my-model    # optional, default = key; used for integration file naming
+```
+
+Then `pilot-workers credentials my-model` and `pilot-workers install my-model on claude`.
+
+Reserved keys (cannot be used as provider key): `runner`, `all`, `on`, `claude`, `codex`.
+
+## Host integration
+
+The **host** is whatever AI agent acts as the planner. `integrations/` has ready-made configs:
+- **`claude-host/`**: 12 agents (glm/kimi/ds × coder/explorer/reviewer/tester) + 8 slash commands
+- **`codex-host/`**: `$glm` / `$kimi` / `$ds` skill entry points
+
+Adding a new host: create `integrations/<name>-host/`, put whatever config your host needs, point it at `pilot-workers dispatch`. See `integrations/README.md`.
+
+## Architecture
+
+See [CLAUDE.md](CLAUDE.md) for the current architecture, module reference, and conventions. See [docs/architecture.md](docs/architecture.md) for the detailed contract and security model.
+
+## Development
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
+.venv/bin/pytest    # 130 tests, all offline
+```
+
+## License
+
+MIT

{pilot_workers-0.2.0 → pilot_workers-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pilot-workers"
-version = "0.2.0"
+version = "0.4.0"
 description = "Dispatch bounded tasks to isolated LLM workers (GLM, Kimi, and more)"
 readme = "README.md"
 license = "MIT"

pilot_workers-0.4.0/src/pilot_workers/cli/fanout.py

@@ -0,0 +1,436 @@
+#!/usr/bin/env python3
+"""Concurrent multi-job dispatch with aggregated verdicts.
+
+Takes N pre-written task files, dispatches them concurrently (each via a
+``pilot_workers.cli.dispatch`` child process), re-emits each child's
+``worker_runner.started`` event on stderr, and prints ONE JSON array of
+verdicts on stdout -- one object per job, in the original job order.
+
+Log note: same-provider concurrent jobs interleave that provider's
+``latest.log`` (human log only; per-run jsonl/verdict.json files are still
+separate).
+
+Stdout contract (callers depend on it): exactly one JSON array, nothing
+else. Started events and warnings go to stderr only.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from dataclasses import dataclass
+from typing import Any
+
+from pilot_workers import policy, providers, runtime
+from pilot_workers.runners import get_runner
+
+
+FANOUT_ERROR_EXIT = 2
+DEFAULT_TIMEOUT_S = 3600
+DEFAULT_IDLE_TIMEOUT_S = 900
+JOBS_FILE_FIELDS = {"provider", "mode", "task_file", "worktree"}
+SUCCESS_VERDICTS = ("completed", "step_capped_partial")
+
+
+@dataclass
+class Job:
+    provider: str
+    mode: str
+    task_file: str
+    worktree: bool = False
+
+    @property
+    def label(self) -> str:
+        return f"{self.provider}:{self.mode}"
+
+
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Dispatch several jobs concurrently; stdout is one JSON array "
+            "of verdicts in the original job order."
+        )
+    )
+    parser.add_argument(
+        "--workdir",
+        default=None,
+        help="Existing project directory passed to every worker.",
+    )
+    parser.add_argument(
+        "--job",
+        action="append",
+        default=[],
+        metavar="PROVIDER:MODE:TASK_FILE",
+        help="One job spec; repeatable. Parsed with a left split (paths may "
+             "contain colons).",
+    )
+    parser.add_argument(
+        "--jobs-file",
+        default=None,
+        metavar="JSON",
+        help="JSON array of job objects: provider, mode, task_file, worktree.",
+    )
+    parser.add_argument(
+        "--providers",
+        default=None,
+        metavar="KEYS",
+        help="Comma-separated provider keys (shorthand; requires --mode and "
+             "--task-file).",
+    )
+    parser.add_argument(
+        "--mode",
+        default=None,
+        help="Worker mode for the --providers shorthand.",
+    )
+    parser.add_argument(
+        "--task-file",
+        default=None,
+        help="Shared task file for the --providers shorthand.",
+    )
+    parser.add_argument(
+        "--max-parallel",
+        type=int,
+        default=None,
+        metavar="N",
+        help="Maximum concurrent jobs (default: number of jobs).",
+    )
+    parser.add_argument(
+        "--timeout",
+        type=int,
+        default=DEFAULT_TIMEOUT_S,
+        help=f"Per-job wall-clock limit in seconds (default {DEFAULT_TIMEOUT_S}).",
+    )
+    parser.add_argument(
+        "--idle-timeout",
+        type=int,
+        default=DEFAULT_IDLE_TIMEOUT_S,
+        help=f"Per-job idle limit in seconds (default {DEFAULT_IDLE_TIMEOUT_S}).",
+    )
+    return parser.parse_args(argv)
+
+
+# ---------------------------------------------------------------------------
+# Job list construction / validation
+# ---------------------------------------------------------------------------
+
+
+class _SpecError(RuntimeError):
+    pass
+
+
+def _validate_job(job: Job) -> None:
+    if job.provider not in providers.PROVIDERS:
+        raise _SpecError(f"unknown provider: {job.provider}")
+    if job.mode not in policy.MODE_TO_AGENT:
+        raise _SpecError(f"unknown mode: {job.mode}")
+    if job.mode == "resume":
+        raise _SpecError("resume is not supported in fanout")
+    if not Path(job.task_file).is_file():
+        raise _SpecError(f"task file not found: {job.task_file}")
+
+
+def _parse_job_spec(spec: str) -> Job:
+    parts = spec.split(":", 2)
+    if len(parts) != 3 or not all(parts):
+        raise _SpecError(f"invalid --job spec (expected PROVIDER:MODE:TASK_FILE): {spec}")
+    return Job(provider=parts[0], mode=parts[1], task_file=parts[2])
+
+
+def _load_jobs_file(path_arg: str) -> list[Job]:
+    try:
+        with open(path_arg, "r", encoding="utf-8") as handle:
+            data = json.load(handle)
+    except (OSError, json.JSONDecodeError) as exc:
+        raise _SpecError(f"cannot read jobs file: {exc}")
+    if not isinstance(data, list):
+        raise _SpecError("jobs file must be a JSON array of job objects")
+    jobs: list[Job] = []
+    for entry in data:
+        if not isinstance(entry, dict):
+            raise _SpecError("jobs file entries must be objects")
+        unknown = set(entry) - JOBS_FILE_FIELDS
+        if unknown:
+            raise _SpecError(
+                f"unknown jobs file fields: {', '.join(sorted(unknown))}"
+            )
+        missing = {"provider", "mode", "task_file"} - set(entry)
+        if missing:
+            raise _SpecError(
+                f"jobs file entry missing fields: {', '.join(sorted(missing))}"
+            )
+        jobs.append(
+            Job(
+                provider=str(entry["provider"]),
+                mode=str(entry["mode"]),
+                task_file=str(entry["task_file"]),
+                worktree=bool(entry.get("worktree", False)),
+            )
+        )
+    return jobs
+
+
+def _build_jobs(args: argparse.Namespace) -> list[Job]:
+    shorthand_used = any([args.providers, args.mode, args.task_file])
+    explicit_used = bool(args.job) or args.jobs_file is not None
+    if args.job and args.jobs_file is not None:
+        raise _SpecError("--job and --jobs-file are mutually exclusive")
+    if shorthand_used and explicit_used:
+        raise _SpecError(
+            "--providers/--mode/--task-file are mutually exclusive with "
+            "--job and --jobs-file"
+        )
+    if shorthand_used:
+        if not (args.providers and args.mode and args.task_file):
+            raise _SpecError("--providers requires --mode and --task-file")
+        jobs = [
+            Job(provider=key.strip(), mode=args.mode, task_file=args.task_file)
+            for key in args.providers.split(",")
+            if key.strip()
+        ]
+    elif args.jobs_file is not None:
+        jobs = _load_jobs_file(args.jobs_file)
+    else:
+        jobs = [_parse_job_spec(spec) for spec in args.job]
+    if not jobs:
+        raise _SpecError("at least one job is required")
+    for job in jobs:
+        _validate_job(job)
+    return jobs
+
+
+# ---------------------------------------------------------------------------
+# Safety guards
+# ---------------------------------------------------------------------------
+
+
+def _credential_preflight(jobs: list[Job]) -> None:
+    seen: set[str] = set()
+    for job in jobs:
+        if job.provider in seen:
+            continue
+        seen.add(job.provider)
+        provider = providers.PROVIDERS[job.provider]
+        meta = runtime.credential_metadata(provider, get_runner(provider.runner))
+        if not meta.get("configured"):
+            raise _SpecError(
+                f"credential missing for {job.provider}; "
+                f"run: pilot-workers credentials {job.provider}"
+            )
+
+
+def _workdir_collision_guard(jobs: list[Job]) -> None:
+    code_jobs = [job for job in jobs if job.mode == "code"]
+    if len(code_jobs) >= 2 and not all(job.worktree for job in code_jobs):
+        raise _SpecError(
+            "multiple code jobs share one workdir; use --jobs-file with "
+            'per-job "worktree": true'
+        )
+    test_jobs = [job for job in jobs if job.mode == "test"]
+    if len(test_jobs) >= 2:
+        print(
+            "note: concurrent test jobs share a workdir; caches may collide",
+            file=sys.stderr,
+        )
+
+
+# ---------------------------------------------------------------------------
+# Execution
+# ---------------------------------------------------------------------------
+
+
+def _build_dispatch_command(
+    job: Job, workdir: str, timeout: int, idle_timeout: int
+) -> list[str]:
+    cmd: list[str] = [
+        sys.executable,
+        "-m", "pilot_workers.cli.dispatch",
+        "--provider",
+        job.provider,
+        "--mode",
+        job.mode,
+        "--workdir",
+        workdir,
+        "--task-file",
+        job.task_file,
+    ]
+    if job.worktree:
+        cmd.append("--worktree")
+    cmd.extend(["--timeout", str(timeout)])
+    cmd.extend(["--idle-timeout", str(idle_timeout)])
+    return cmd
+
+
+def _synthesized_verdict(
+    job: Job, exit_code: Any, stderr_tail: str, interrupted: bool = False
+) -> dict[str, Any]:
+    verdict: dict[str, Any] = {
+        "job": job.label,
+        "type": "worker_runner.verdict",
+        "verdict": "error",
+        "synthesized": True,
+        "exit_code": exit_code,
+        "stderr_tail": stderr_tail,
+    }
+    if interrupted:
+        verdict["interrupted"] = True
+    return verdict
+
+
+def _run_job(
+    index: int,
+    job: Job,
+    cmd: list[str],
+    results: list[dict[str, Any] | None],
+    procs: list[Any],
+    stderr_lock: threading.Lock,
+) -> None:
+    """Run one dispatch child; never raises out -- always records a verdict."""
+    proc = None
+    try:
+        proc = subprocess.Popen(
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            text=True,
+            bufsize=1,
+        )
+        procs[index] = proc
+
+        stderr_chunks: list[str] = []
+
+        def _drain_stderr() -> None:
+            try:
+                assert proc.stderr is not None
+                stderr_chunks.append(proc.stderr.read())
+            except Exception:
+                pass
+
+        drain = threading.Thread(target=_drain_stderr, daemon=True)
+        drain.start()
+
+        started_seen = False
+        last_line = ""
+        assert proc.stdout is not None
+        for raw_line in proc.stdout:
+            line = raw_line.rstrip("\n")
+            if not line:
+                continue
+            if not started_seen:
+                started_seen = True
+                try:
+                    event = json.loads(line)
+                except json.JSONDecodeError:
+                    event = None
+                if isinstance(event, dict):
+                    with stderr_lock:
+                        sys.stderr.write(
+                            json.dumps({"job": job.label, **event},
+                                       ensure_ascii=False)
+                            + "\n"
+                        )
+                        sys.stderr.flush()
+            last_line = line
+        proc.wait()
+        drain.join(timeout=5)
+        stderr_text = "".join(stderr_chunks)
+
+        verdict: dict[str, Any] | None = None
+        if last_line:
+            try:
+                parsed = json.loads(last_line)
+            except json.JSONDecodeError:
+                parsed = None
+            if isinstance(parsed, dict):
+                parsed["job"] = job.label
+                verdict = parsed
+        if verdict is None:
+            verdict = _synthesized_verdict(
+                job, proc.returncode, stderr_text[-500:]
+            )
+        results[index] = verdict
+    except Exception as exc:  # the array must always be printed
+        results[index] = _synthesized_verdict(
+            job, getattr(proc, "returncode", None), str(exc)[-500:]
+        )
+
+
+def run_fanout(args: argparse.Namespace) -> int:
+    if not args.workdir:
+        print("error: --workdir is required", file=sys.stderr)
+        return FANOUT_ERROR_EXIT
+    try:
+        jobs = _build_jobs(args)
+        _credential_preflight(jobs)
+        _workdir_collision_guard(jobs)
+    except _SpecError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return FANOUT_ERROR_EXIT
+
+    max_workers = args.max_parallel or len(jobs)
+    results: list[dict[str, Any] | None] = [None] * len(jobs)
+    procs: list[Any] = [None] * len(jobs)
+    stderr_lock = threading.Lock()
+    interrupted = False
+
+    try:
+        with ThreadPoolExecutor(max_workers=max_workers) as pool:
+            futures = [
+                pool.submit(
+                    _run_job,
+                    index,
+                    job,
+                    _build_dispatch_command(
+                        job, args.workdir, args.timeout, args.idle_timeout
+                    ),
+                    results,
+                    procs,
+                    stderr_lock,
+                )
+                for index, job in enumerate(jobs)
+            ]
+            for future in futures:
+                future.result()
+    except KeyboardInterrupt:
+        interrupted = True
+        # Children received SIGINT via the process group and self-terminate;
+        # give them a bounded grace period to reap.
+        deadline = time.monotonic() + runtime.TERMINATE_GRACE_SECONDS + 5
+        for proc in procs:
+            if proc is None:
+                continue
+            remaining = deadline - time.monotonic()
+            if remaining <= 0:
+                break
+            try:
+                proc.wait(timeout=remaining)
+            except Exception:
+                pass
+
+    for index, job in enumerate(jobs):
+        if results[index] is None:
+            results[index] = _synthesized_verdict(job, None, "", interrupted)
+
+    final: list[dict[str, Any]] = [r for r in results if r is not None]
+    sys.stdout.write(json.dumps(final, ensure_ascii=False) + "\n")
+    sys.stdout.flush()
+
+    if interrupted:
+        return 1
+    if all(r.get("verdict") in SUCCESS_VERDICTS for r in final):
+        return 0
+    return 1
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = parse_args(argv)
+    return run_fanout(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())