kylon-cli 0.1.0-next.4

package/README.md

@@ -0,0 +1,560 @@
+# kylon-cli
+
+Gateway CLI for connecting local agent providers to a P2 workspace.
+
+Requires Node.js 22 or newer.
+
+## Install
+
+The current release train ships a single self-contained `.mjs` bundle
+as a GitHub Release asset on the `fre-so/p2` repo. The Web UI
+generates two separate commands — an **install** step and a **run**
+step — each with its own Copy button. Operators paste the install
+once per host (and any time they want to upgrade) and the run
+whenever they want to start the daemon. npm is on the roadmap (see
+[Planned: npm install](#planned-npm-install) below) but not wired yet.
+
+### Step 1 — Install kylon (once per host, re-paste to upgrade)
+
+```bash
+gh release download --repo fre-so/p2 --pattern 'kylon.mjs' --output /tmp/kylon.new \
+  && chmod +x /tmp/kylon.new \
+  && sudo mv /tmp/kylon.new /usr/local/bin/kylon
+```
+
+Downloads the latest bundle to an unprivileged scratch file, flips
+the executable bit, and `sudo mv`'s it into `/usr/local/bin/kylon`
+atomically. The repo is private, so `gh` (authenticated via
+`gh auth login`) is required instead of plain `curl`.
+
+Re-paste to upgrade: the `sudo mv` overwrites the binary in place
+with whatever the latest release publishes as `kylon.mjs`. If you
+want a specific build instead of latest, download the matching
+`kylon-vX.Y.Z.mjs` from that release's page — every release
+publishes both the stable-name `kylon.mjs` and the versioned copy.
+
+Requirements: Node.js 22+, [GitHub CLI](https://cli.github.com/)
+(`gh auth login`), `sudo`. Linux / macOS only (Windows operators
+should use WSL).
+
+### Step 2 — Start the gateway daemon
+
+```bash
+kylon gateway run \
+  --server-url https://<origin>/api \
+  --provider codex \
+  --api-key <agent-api-key>
+```
+
+`gateway run` registers the session (writing `~/.kylon/gateway-session.json`
+with mode `0600`) and starts the daemon in one step. On the next
+invocation, if a session already exists, plain `kylon gateway run`
+without flags picks it up and jumps straight to start. See
+[Usage → Run](#run) for the full decision table. The daemon runs
+in the foreground until you stop it with `Ctrl+C`; hosts that want a
+supervised daemon typically wrap the same command in
+`launchd` / `systemd` / `supervisord`.
+
+### Local development (contributors)
+
+```bash
+pnpm install
+pnpm --filter kylon-cli build
+```
+
+See [Development → Exposing `kylon` to provider subprocesses](#exposing-kylon-to-provider-subprocesses)
+for the debugger-friendly tsx-based dev loop.
+
+### Planned: npm install
+
+The npm distribution path is designed (see
+[`docs/journal_docs/04_13_cli_npx_gateway_run_release_plan.md`](../../docs/journal_docs/04_13_cli_npx_gateway_run_release_plan.md))
+and this package is already publish-ready (`packages/cli/package.json`
+has the metadata and `prepack` runs `bundle:release`). Publishing is
+held pending the rollout-config side of that plan (platform-served
+`recommendedCliVersion`), at which point the Web UI will drop the
+Install step entirely and flip the Run step to
+`npx -y kylon-cli@<recommended> gateway run …`. Don't `npm publish`
+ad-hoc before that — the first live release should come out through
+the paired CI workflow so the token path is exercised end-to-end.
+
+## Usage
+
+### Run
+
+`gateway run` is the recommended entrypoint for external agent operators.
+It composes `connect` + `start` into a single command.
+
+```bash
+kylon gateway run \
+  --server-url https://api.p2.ai \
+  --provider codex \
+  --api-key pak_xxxxx
+```
+
+Decision table:
+
+| Saved session? | Flags passed? | What `run` does |
+|---|---|---|
+| no | `--server-url` + `--api-key` + `--provider` | connect, persist session, start daemon |
+| no | any field missing | error — lists the missing flag |
+| yes | none | start the daemon from the saved session |
+| yes | any | reconnect with the overrides (falls back to saved values for fields you didn't pass), persist the refreshed session, start the daemon |
+
+Only CLI flags count as overrides. Setting `KYLON_API_KEY` in the
+environment does **not** trigger a reconnect on a saved session —
+it still feeds the normal API-key resolution inside `gateway start`.
+
+### Connect
+
+Register this machine as the gateway client for an external agent.
+Most operators should prefer `gateway run` above. Use `connect` on its
+own when scripting or when you need to register a session without
+immediately starting the daemon.
+
+```bash
+kylon gateway connect \
+  --server-url https://api.p2.ai \
+  --api-key pak_xxxxx \
+  --provider codex
+```
+
+The agent's API key identifies which agent the daemon will serve;
+channels are bound separately via the "invite agent into channel" flow
+in the web UI.
+
+### Bind
+
+Create or update a logical session binding for an agent. Run from the
+target working directory:
+
+```bash
+kylon gateway bind \
+  --agent agent_123 \
+  --provider codex
+```
+
+Switch an existing binding to a different directory:
+
+```bash
+cd /path/to/other/repo
+kylon gateway bind --agent agent_123 --workdir .
+```
+
+Bindings are keyed by `(gateway session, agent)`. The same agent
+behaves the same way regardless of which channel an assignment
+arrives on — see
+[`docs/journal_docs/05_27_gateway_routing_simplification.md`](../../docs/journal_docs/05_27_gateway_routing_simplification.md).
+
+### Commands
+
+| Command | Description |
+|---|---|
+| `kylon gateway run` | **Recommended.** Connect + start in one step; idempotent when a session already exists. |
+| `kylon gateway connect` | Register this machine as the gateway client for an agent (API key identifies which), without starting the daemon. |
+| `kylon gateway bind` | Create or update a logical session binding. |
+| `kylon gateway start` | Start the gateway daemon from a saved session — opens the SSE stream and executes assignments. Reads the API key from the saved session. |
+
+### Run Options
+
+| Flag | Description |
+|---|---|
+| `--server-url <url>` | P2 server URL (required for the first run; override otherwise) |
+| `--api-key <key>` | Agent API key, e.g. `pak_xxx` (required for first run; override otherwise. `KYLON_API_KEY` env var satisfies the first-run requirement but does not count as an override on subsequent runs) |
+| `--provider <name>` | Provider CLI: `codex`, `claude-code`, `hermes`, `openclaw`, `generic` (required for first run; override otherwise) |
+
+### Connect Options
+
+| Flag | Description |
+|---|---|
+| `--server-url <url>` | P2 server URL (required) |
+| `--api-key <key>` | Agent API key, e.g. `pak_xxx` (required, or set `KYLON_API_KEY`) |
+| `--provider <name>` | Provider CLI: `codex`, `claude-code`, `hermes`, `openclaw`, `generic` (required) |
+
+### Bind Options
+
+| Flag | Description |
+|---|---|
+| `--agent <id>` | Agent ID (required) |
+| `--provider <name>` | Provider CLI (required for new, optional for update) |
+| `--workdir <path>` | Working directory (default: current directory) |
+
+### Supported Providers
+
+- `codex` — OpenAI Codex CLI
+- `claude-code` — Anthropic Claude Code CLI
+- `hermes` — Hermes CLI using `hermes -z <prompt>`
+- `openclaw` — OpenClaw CLI using `openclaw agent --message <prompt> --json`
+- `generic` - provider-neutral wrapper. Runs a `kylon-provider`
+  executable on `PATH` and expects newline-delimited JSON events matching
+  `docs/journal_docs/05_25_external_agent_provider_adapter_contract.md`.
+
+## State Model
+
+The CLI uses a three-layer state model:
+
+- **GatewaySession** — authenticated connection to the P2 server (one per machine)
+- **LogicalSessionState** — per-`(gateway session, agent)` binding holding the current workdir and provider
+- **ProviderRuntimeEntry** — per-conversation provider resume cache, keyed by `(gateway session, channel, agent, scope, provider, workdir)` (disposable)
+
+Switching workdir updates the logical session without creating a new one. Provider runtimes are cached per workdir + conversation scope — switching back resumes the old runtime.
+
+State is persisted to `~/.kylon/` (or `$XDG_CONFIG_HOME/kylon/`).
+
+## Development
+
+> This section assumes you cloned `fre-so/p2` and ran `pnpm install` at the
+> repo root. All commands are run from anywhere in the monorepo unless
+> otherwise noted.
+
+### Inner loop
+
+```bash
+# type-check without emitting
+pnpm --filter kylon-cli typecheck
+
+# tsc build — emits dist/bin/kylon.js and other .js files
+pnpm --filter kylon-cli build
+
+# unit tests (no network, no DB)
+pnpm --filter kylon-cli test
+
+# run the locally built CLI
+node packages/cli/dist/bin/kylon.js --help
+```
+
+The Web agent-settings "Local dev" block generates the same
+`node packages/cli/dist/bin/kylon.js …` invocation from the logged-in
+agent's API key. If you change `src/bin/kylon.ts` or anything it
+imports, rerun `pnpm --filter kylon-cli build` before re-executing.
+
+### Exposing `kylon` to provider subprocesses
+
+`node packages/cli/dist/bin/kylon.js gateway run …` starts the daemon
+but leaves **no `kylon` binary on PATH**. When the provider subprocess
+(claude-code / codex) then tries `kylon workspace …` via its Bash tool,
+the shell fails with `command not found`. Two ways to fix this:
+
+**Option 1 — `pnpm link` (persistent).** Symlink kylon's published
+`bin.kylon` into pnpm's global bin dir:
+
+```bash
+pnpm --filter kylon-cli bundle            # or bundle:minify for a prod-shaped build
+pnpm --filter kylon-cli link --global
+
+which kylon
+# → ~/Library/pnpm/kylon (or similar) → packages/cli/dist/kylon-bundle.mjs
+```
+
+From here on, start the daemon via the linked binary instead of the
+raw `node dist/bin/kylon.js`:
+
+```bash
+kylon gateway run --server-url http://localhost:5173/api --provider codex --api-key pak_…
+```
+
+Every provider call to `kylon workspace …` resolves to the linked
+binary. Edit source → `pnpm --filter kylon-cli bundle` → next provider
+call picks it up (each workspace invocation is a fresh process;
+restart the daemon only for *daemon*-side edits). Clean up with
+`pnpm --filter kylon-cli unlink --global`.
+
+**Option 2 — `--dev-cli-shim` (ephemeral, IDE-friendly).** The daemon
+can install a temporary bash shim that execs the TS source through
+tsx. Pass `--dev-cli-shim <abs-path-to-p2-repo>` on `gateway run` or
+`gateway start`:
+
+```bash
+node packages/cli/dist/bin/kylon.js gateway run \
+  --server-url http://localhost:5173/api \
+  --provider codex \
+  --api-key pak_… \
+  --dev-cli-shim "$(pwd)"
+```
+
+The daemon prints the shim location on startup:
+
+```text
+[dev] kylon shim: /tmp/kylon-dev-shim-abc123/kylon
+[dev] provider calls will exec: node --import tsx /abs/path/packages/cli/src/bin/kylon.ts
+```
+
+The tmpdir is prepended to the daemon's `PATH`, so the provider
+subprocess (and the bash shell it spawns) resolves `kylon` to the
+shim. The shim execs `node --import tsx <src>`, so:
+
+- Every `kylon workspace …` invocation reads the current `src/*.ts` —
+  no bundle rebuild needed between edits.
+- A Node debugger attached to the daemon is inherited by each shim
+  invocation (they exec `node`, so `NODE_OPTIONS` and VS Code's
+  Auto-Attach loader pass through).
+- On SIGINT/SIGTERM the tmpdir is wiped.
+
+This flag is **only compiled into local (non-minified) builds**. The
+npm package, the GitHub Release bundle, and any `bundle:release`
+output reject `--dev-cli-shim` as an unknown argument and omit it from
+`--help`, so it can never accidentally ship.
+
+### Debugging both the daemon and `kylon workspace` calls
+
+Combine `--dev-cli-shim` with a Node debugger to step through the
+full chain — daemon → provider subprocess → `kylon workspace …` — in
+one IDE session.
+
+**VS Code**:
+
+1. Enable `Debug: Toggle Auto Attach → Always` (or `Only With Flag`).
+   VS Code prepends its `js-debug` bootloader to `NODE_OPTIONS`, which
+   every child Node process — including the ones launched by the
+   shim — inherits.
+2. Open an integrated terminal and run:
+
+   ```bash
+   node packages/cli/dist/bin/kylon.js gateway run \
+     --server-url http://localhost:5173/api \
+     --provider codex \
+     --api-key pak_… \
+     --dev-cli-shim "$(pwd)"
+   ```
+
+3. Set breakpoints in both `packages/cli/src/commands/gateway-start.ts`
+   (daemon) and `packages/cli/src/commands/workspace/*.ts` (child
+   commands). Both fire the next time the provider issues a workspace
+   call.
+
+**JetBrains / others**: export `NODE_OPTIONS=--inspect=0.0.0.0:0`
+before launching the daemon. Every subsequent Node process — daemon
+and every `kylon workspace …` invocation — opens its own inspector
+port. Attach your IDE to the process list.
+
+No debugger attach? The shim still works — source edits are picked up
+on the next provider call, but breakpoints just don't fire.
+
+### Bundles
+
+The release artifact is a single-file ESM bundle produced by
+`scripts/bundle.mjs`. Three variants:
+
+| Script | Output | Passes to `bundle.mjs` | Use when |
+|---|---|---|---|
+| `pnpm --filter kylon-cli bundle` | `dist/kylon-bundle.mjs` | _(none)_ | debugging the bundled shape while keeping readable names — not shipped |
+| `pnpm --filter kylon-cli bundle:minify` | `dist/kylon-bundle.mjs` | `--minify` | reproducing the pre-obfuscation size and behavior for a bisect |
+| `pnpm --filter kylon-cli bundle:release` | `dist/kylon-bundle.mjs` | `--minify --obfuscate` | what ships on npm and in GitHub Releases |
+
+The `bundle:release` path runs esbuild with `--minify`, then passes the
+output through `javascript-obfuscator`. It's also what the `prepack` hook
+runs, so `pnpm pack` / `npm publish` always produce the obfuscated shape
+even if you forget to call `bundle:release` explicitly.
+
+Run the bundle directly to sanity-check it:
+
+```bash
+pnpm --filter kylon-cli bundle:release
+node packages/cli/dist/kylon-bundle.mjs --help
+```
+
+### Testing
+
+Unit tests run against Node's built-in test runner (`node:test`) plus
+`tsx`. They touch the filesystem inside temp dirs but never the
+network or a database, so `pnpm --filter kylon-cli test` is safe to
+run anywhere.
+
+E2E tests drive the CLI as a subprocess against a configurable mock
+or live server:
+
+```bash
+# headless e2e (mock provider processes spawned from scripts/mock-*.mjs)
+pnpm --filter kylon-cli test:e2e
+
+# live e2e against a real P2 environment — requires doppler secrets
+doppler run --project p2 --config prd -- pnpm --filter kylon-cli test:e2e:live
+```
+
+Live E2E tests provision a throwaway workspace via the REST API, so
+expect them to take several minutes and to leave audit trail rows in
+the target environment. Do not point them at production casually.
+
+Before shipping a release, also smoke-test the packaged artifact
+exactly as operators will receive it:
+
+```bash
+# 1. produce the release bundle + tarball
+pnpm --filter kylon-cli bundle:release
+cd packages/cli && pnpm pack --pack-destination /tmp/kylon-out
+
+# 2. install the tarball in a clean directory
+WORK=$(mktemp -d) && cd "$WORK"
+npm init -y >/dev/null
+npm install /tmp/kylon-out/kylon-cli-*.tgz
+
+# 3. the published shape should have only three entries
+ls node_modules/kylon-cli               # dist/  package.json  README.md
+ls node_modules/kylon-cli/dist          # kylon-bundle.mjs
+
+# 4. verify the binary is runnable
+node_modules/.bin/kylon --help
+
+# 5. confirm the bundle is actually obfuscated
+head -c 200 node_modules/kylon-cli/dist/kylon-bundle.mjs
+# expect `#!/usr/bin/env node` followed by hexadecimal identifier soup,
+# not recognizable function names or source strings
+```
+
+### Obfuscation
+
+`bundle:release` and `prepack` run `javascript-obfuscator` with a
+conservative preset chosen for runtime safety:
+
+- **On:** `compact`, `identifierNamesGenerator: "hexadecimal"`,
+  `stringArray` with `base64` encoding + rotate + shuffle, two
+  wrapper function layers.
+- **Off:** `controlFlowFlattening`, `deadCodeInjection`,
+  `selfDefending`, `debugProtection`, `unicodeEscapeSequence`. These
+  trade correctness and startup latency for marginal protection — do
+  not turn them on without measuring startup and rerunning the full
+  test:e2e suite.
+- `renameGlobals` stays off so Node built-ins keep their names.
+
+Startup stays under 100 ms on modern hardware; bundle grows from
+~85 KB (minify only) to ~230 KB (obfuscated). Obfuscation is a
+tampering and casual-reading deterrent, not a security control — the
+API server is the security boundary.
+
+## Release
+
+Today there is **one** active release channel: a bundled `.mjs`
+uploaded as a GitHub Release asset. The npm channel is designed and
+publish-ready but intentionally held pending the
+`recommendedCliVersion` rollout config described in
+[`docs/journal_docs/04_13_cli_npx_gateway_run_release_plan.md`](../../docs/journal_docs/04_13_cli_npx_gateway_run_release_plan.md).
+See [Planned: npm publish](#planned-npm-publish) below.
+
+### Versioning
+
+Use explicit semver in `packages/cli/package.json`:
+
+| Bump | When |
+|---|---|
+| patch (`0.1.0 → 0.1.1`) | bug fix, no new flags, no behavior change |
+| minor (`0.1.x → 0.2.0`) | backward-compatible capability (new command, new flag) |
+| major (`0.x.x → 1.0.0`) | breaking change to CLI contract or runtime behavior |
+
+The GitHub Release tag still uses the historical
+`cli/v0.0.${github.run_number}` scheme, which is fine as an opaque
+build id while there is no consumer that pins to it. The moment we
+flip the npm plan on, bump `packages/cli/package.json` to explicit
+semver and let the publish workflow own version selection.
+
+### Active channel — GitHub Release asset
+
+Triggered automatically by `.github/workflows/release-cli.yml` on any
+push to `main` that touches `packages/cli/**`,
+`packages/workspace-cli-core/**`, or `packages/types/**`. The workflow:
+
+1. Runs `pnpm --filter kylon-cli bundle:release` (obfuscated output).
+2. Smoke-tests `node packages/cli/dist/kylon-bundle.mjs --help`.
+3. Copies the bundle to **two** asset names under the release:
+   - `kylon-v0.0.${run_number}.mjs` — versioned, useful for pinning.
+   - `kylon.mjs` — stable name, downloadable via
+     `gh release download --repo fre-so/p2 --pattern 'kylon.mjs'`.
+4. Creates a GitHub Release tagged `cli/v0.0.${run_number}` with both
+   `.mjs` files attached.
+
+The repo is private, so downloads require `gh` (authenticated) rather
+than plain `curl`. This is the shape the Web UI generates:
+
+```bash
+gh release download --repo fre-so/p2 --pattern 'kylon.mjs' --output /tmp/kylon
+chmod +x /tmp/kylon
+sudo mv /tmp/kylon /usr/local/bin/kylon
+```
+
+Pin to a specific build by downloading the versioned asset from its
+release page instead.
+
+No npm token required; the workflow only needs `contents: write`.
+
+### Rollback (GitHub channel)
+
+If the most recent push produced a broken CLI:
+
+1. Identify the last known good release under
+   <https://github.com/fre-so/p2/releases>.
+2. Manually re-upload that release's bundle under the name
+   `kylon.mjs` on the most recent release — that's what the Web UI's
+   stable URL resolves against. (Alternatively, revert the bad commit
+   and let `release-cli.yml` cut a new release.)
+3. Hosts with persistent installs re-run `curl -fsSL … -o
+   /usr/local/bin/kylon` to pull the rolled-back bundle.
+
+There is no equivalent of `npm deprecate` on GitHub Releases, so a
+broken bundle is "fixed" only by publishing a newer one.
+
+### Planned: npm publish
+
+This package is already wired for `npm publish` (metadata, `files`,
+`prepack`, `publishConfig.access: public`). The only things missing
+are the publish CI workflow and the owning npm account. The plan:
+
+1. Someone claims `kylon-cli` on npm (the name is currently 404).
+2. A granular publish token (Read + Write on `kylon-cli`) is stored
+   as the `NPM_TOKEN` GitHub Actions secret on `fre-so/p2`.
+3. Split the release automation:
+   - **`cli-verify.yml`** — runs on PRs and pushes touching
+     `packages/cli/**`. Runs typecheck, lint, test, `bundle:release`,
+     and `npm pack`. Does not publish.
+   - **`cli-publish.yml`** — runs on push of tag `cli/vX.Y.Z`.
+     Verifies the tag matches `packages/cli/package.json#version`,
+     then `pnpm --filter kylon-cli publish --access public
+     --no-git-checks` with
+     `NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}`.
+4. The API exposes `recommendedCliVersion`; the Web UI flips from
+   the `curl` shape above to `npx -y kylon-cli@<recommended> gateway
+   run …`.
+
+Until (2) + (3) land, do **not** `npm publish` ad-hoc — the first
+live release should come out through the paired CI workflow so the
+token path is exercised end-to-end. Once the workflow exists, the
+release ritual becomes:
+
+```bash
+# 1. On a release branch
+pnpm --filter kylon-cli version 0.1.1   # bumps packages/cli/package.json only
+
+# 2. Merge to main via PR, verify CI is green
+
+# 3. Tag the commit on main
+git pull
+git tag cli/v0.1.1
+git push origin cli/v0.1.1
+
+# 4. GitHub Actions publishes; verify
+npm view kylon-cli@0.1.1 dist-tags
+```
+
+Rollback once npm is live: flip `recommendedCliVersion` in the API
+back to the last known good version (new operators get that version
+in their install command; in-flight daemons keep their current CLI
+until they restart). If the pushed version is actively broken, also
+`npm deprecate kylon-cli@<bad> "…"` so operators that ignore the
+recommendation see a warning. Do not `npm unpublish`.
+
+### Pre-release channel (npm-era, also planned)
+
+Once npm is live, risky changes can go out under a `next` dist-tag so
+the Web UI's recommended version is untouched:
+
+```bash
+# 1. Bump to a prerelease version
+pnpm --filter kylon-cli version 0.2.0-next.0
+
+# 2. Publish under the `next` dist-tag
+cd packages/cli && pnpm publish --tag next --access public --no-git-checks
+
+# 3. Operators can opt in explicitly
+npx -y kylon-cli@next gateway run …
+
+# 4. Promote to `latest` once validated
+npm dist-tag add kylon-cli@0.2.0-next.0 latest
+```