akm-cli 0.10.261016-beta → 0.10.261017-beta

package/docs/architecture/internals/fresh-host-rebuild-runbook.md

@@ -1,41 +1,39 @@
 # Fresh-Host Rebuild Runbook
 
-Rebuild a working akm install on a new machine. Assumes your bundle lives in a
-git remote and that a versioned `config.json` copy is committed to the bundle
-git (the 08-F1 recovery pattern — see step 3).
+Rebuild a working akm install on a new machine. It assumes your bundle lives in
+a git remote and that you kept a copy of `config.json` somewhere you can
+restore it from.
 
 1. **Install the CLI.** Prebuilt binary (no runtime needed):
    `curl -fsSL https://github.com/itlackey/akm/releases/latest/download/install.sh | bash`
-   — or npm package (Node.js >= 22 required): `npm install -g akm-cli`.
-   If a working Bun >= 1.0 is also on `PATH`, the package prefers it after Node
-   bootstrap; old, unusable, or absent Bun installations fall back to Node.js.
-   The standalone binary is runtime-free.
-   (Windows: `irm https://github.com/itlackey/akm/releases/latest/download/install.ps1 | iex`.)
+   (Windows: `irm https://github.com/itlackey/akm/releases/latest/download/install.ps1 | iex`).
+   Or the npm package, which needs Node.js 22 or later: `npm install -g akm-cli`.
+   A working Bun 1.0 or later on `PATH` is optional and preferred over Node.
 2. **Confirm the binary resolves:** `akm --version`.
-3. **Restore `config.json`.** akm reads a single user config at
-   `~/.config/akm/config.json` (`%APPDATA%\akm\config.json` on Windows;
-   override with `AKM_CONFIG_DIR`). This file is gitignored on the host, so the
-   recovery source is the ONE versioned copy committed to the bundle git
-   (08-F1 pattern). Copy that file into place; it carries the `connections`,
-   `harnesses`, and cron-load-bearing improve
-   strategies that are otherwise unrecoverable.
-4. **Provide secrets/env.** `config.json` references API keys via `$VAR`
-   placeholders; export those environment variables (or restore
-   `credentials.json` from your own backup; it is never part of `config.json`)
-   so the connections and the embedding endpoint resolve. Never commit the
-   resolved values.
-5. **Restore the working bundle.** Clone your bundle git remote to the path
-   named by your working bundle's entry under `bundles` in the restored
-   config (`defaultBundle` selects which one; default path `~/akm`). There is
-   no top-level `stashDir` config key in 0.9 — it is retired and rejected at
-   load. If starting clean instead, run `akm setup --yes` to scaffold it.
-6. **Re-add non-git sources.** For each managed source not carried in the bundle
-   git (websites, npm packages, GitHub repos), run `akm bundle add <ref>`. Verify the
-   full set with `akm bundle list`.
-7. **Rebuild the index:** `akm index --full` (forces a complete reindex rather
-   than an incremental update).
-8. **Verify health:** `akm health` (checks runtime, artifacts, and improve
-   metrics — exits non-zero if `state.db` is missing) and `akm info` for index
-   stats and effective config.
-9. **Smoke-test discovery:** `akm search <term>` returns results, confirming the
-   index and bundle are wired correctly.
+3. **Restore `config.json`.** akm reads one user config at
+   `~/.config/akm/config.json` (`%APPDATA%\akm\config.json` on Windows; set
+   `AKM_CONFIG_DIR` to move it). It holds your `connections`, `harnesses`, and
+   your improve strategies.
+4. **Provide secrets.** `config.json` refers to API keys as `$VAR` or
+   `${VAR}`. Export those environment variables, or restore `credentials.json`
+   from your own backup (it is never part of `config.json`), so the
+   connections and the embedding endpoint resolve.
+5. **Restore the working bundle.** Clone your bundle's git remote to the path
+   of the working bundle's entry under `bundles` in the restored config
+   (`defaultBundle` selects which one; the default path is `~/akm`). To start
+   clean instead, run `akm setup --yes`, which scaffolds the bundle.
+6. **Check the other bundles.** `akm bundle list` shows the bundles the
+   restored config defines. Re-add any that are missing with
+   `akm bundle add <ref>`.
+7. **Rebuild the index:** `akm index --full`. `index.db` is derived from your
+   bundles and is rebuilt from scratch.
+8. **Verify health:** `akm health`, then `akm info` for index stats and the
+   effective config. `akm health` creates `state.db` if it is missing and
+   applies its migrations. It exits 0 when every check passes, 4 on warnings,
+   and 1 when a hard check fails. See [Health Checks](health-advisories.md) for
+   what each check means.
+9. **Smoke-test discovery:** `akm search <term>` returns results.
+
+`$DATA/state.db` holds proposals, events, and task history, and is not rebuilt
+from your bundles. Copy it from the old host before step 8 if you want that
+history.

package/docs/architecture/internals/health-advisories.md

@@ -1,43 +1,70 @@
-# `akm health` advisory → action map
+# `akm health` checks
 
-Run `akm health` (JSON by default; `--format text` for the table view). The authoritative result
-combines hard failures with deterministic warnings across both output
-channels:
+`akm health` prints JSON by default (`--format text` for the table view) and
+emits the checks below. Each result has a `name`, and lands in `hardChecks` (the
+state.db, `task-history-read`, `active-runs`, connection and harness, and
+semantic-search checks) or `advisories` (the rest). The list comes from
+`HEALTH_CHECKS` in `src/commands/health/checks.ts`, `akmHealth` in
+`src/commands/health.ts`, and the collectors in `src/commands/health/`. The
+checks from `index-state-generation` down to `opencode-plugin-version` appear
+only when they have something to report, and `state-db-readable` only in place
+of every other check; the others are always present.
 
-- Deterministic hard-check and advisory warnings set the overall status to `warn` and exit code 4.
-- A hard check `fail` sets the overall status to `fail` and exit code 1.
-- A heuristic (non-deterministic) advisory `warn` does not change or gate the overall status.
-- A heuristic (non-deterministic) advisory `warn` does not change or gate the exit code.
+## Status and exit code
 
-This table is the interpretive key for a second operator: what each named
-advisory measures and whether to act. Some `warn`s below are *adjudicated,
-expected* states — treat them as "no action" until the referenced condition
-changes.
+Each check is `pass`, `warn`, `fail`, or `unknown`, and `deterministic` or
+`heuristic`.
 
-| Advisory (name in output) | What it means | Action |
+- Any `fail` in `hardChecks` makes the overall status `fail` and the exit code 1.
+- Otherwise, any `warn` from a `deterministic` check (hard or advisory) makes it `warn` and the exit code 4.
+- `heuristic` warns (`session-extraction`, `auto-accept-validation`) and `unknown` results do not change the overall status.
+
+`--since` sets the window the task and improve checks read. The default is 24 hours.
+
+## Network and processes
+
+Probing is on by default; `--no-probe` turns it off. With probing on, health sends a
+`GET` to `<endpoint>/models` of each configured connection (3 second timeout; any HTTP
+response counts as reachable, so a cold local server is never asked to load a model; one
+request per distinct endpoint per invocation, with no cross-run cache), runs each configured
+harness's `--version`, looks up the latest akm release on GitHub for `cli-version`
+(5 second timeout, no retries), and runs the akm binary recorded by the scheduler with
+`--version` for `scheduler-binary`. `plugin-version` runs `git ls-remote --symref origin`
+(it lists refs and fetches nothing) against the Claude plugin marketplace clone whenever a
+Claude plugin is installed. With
+`--no-probe` none of these runs: the connection rows carry no `reachable` field and the
+message says reachability was not probed, and `plugin-version` reports `unknown` (its
+`AKM_VERSION_RANGE` check is local and still runs). The `plugin-version` and `cli-version`
+lookups are best-effort and silent on failure; see `src/commands/health/plugin-staleness.ts`
+and `src/commands/health/version-drift.ts`.
+
+## Checks
+
+| Check | Fires when | Remedy |
 |---|---|---|
-| `state-db-schema` | state.db is missing required tables. | Re-run `akm bundle create`; a fresh/older DB was opened. |
-| `state-db-round-trip` | Append/read probe against state.db failed. | Check disk/permissions on the state.db path; the store is unwritable. |
-| `state-db-integrity` | state.db failed `PRAGMA integrity_check` (which, unlike `quick_check`, also compares every index with its table), or more than half its pages are free (reclaimable by VACUUM). | Back up state.db, then run `sqlite3 -readonly state.db ".dump" \| sqlite3 state.new.db`, verify `state.new.db` passes `integrity_check`, and swap it in; the freelist warning clears on its own on the next `akm improve` retention purge, which VACUUMs state.db above the same 50% threshold. |
-| `state-db-readable` | `state.db` is missing or cannot be read, or is behind on migrations that a read-only run must not apply. Every other check is skipped, because none of them can read it. | Check the owner and mode of the data directory (or point `AKM_DATA_DIR` / `XDG_DATA_HOME` at one you own); run any akm command that records state, such as `akm health`, to create it. |
-| `state-db-migrations` | Reports what `akm health`'s own open of `state.db` applied (`evidence.applied`, and the copy it made first in `evidence.backupPath`); `fail` only when a pending migration could not be applied. | Run `akm migrate apply` when it names a pending migration. |
-| `task-history-read` | How many `task_history` rows were read for the window (informational; it passes). | None. |
-| `active-runs` | A task run has exceeded the stale threshold (>15 min). | Inspect with `akm task history`; a lane is likely wedged — kill/re-run it. |
+| `state-db-schema` | `fail`: state.db is missing one of `events`, `proposals`, `schema_migrations`, `task_history`. | The message lists the missing tables. |
+| `state-db-round-trip` | `fail`: appending and reading back a probe event failed. The probe row is deleted again. | The message gives the error. |
+| `state-db-integrity` | `fail`: `PRAGMA integrity_check` failed, or reading the freelist failed. `warn`: more than 50% of the pages are free. | For a failed check, follow the repair in the message: back up state.db, run `sqlite3 -readonly state.db ".recover" \| sqlite3 state.new.db`, confirm `state.new.db` passes `integrity_check`, stop every akm process, delete `state.db-wal` and `state.db-shm`, then swap `state.new.db` in as state.db. The free-page warning clears when `akm improve`'s retention purge VACUUMs state.db. |
+| `state-db-migrations` | Always `pass` in a normal run; the message lists the migrations this open applied and the pre-migration copy (`evidence.applied`, `evidence.backupPath`). `fail` when the open could not apply a pending migration, or a read-only run (the `akm ui` health view) finds one pending, which it must not apply; this row is then the whole report. | Run `akm migrate apply`. |
+| `task-history-read` | Always `pass`; reports the number of `task_history` rows read in the window. | None. |
+| `active-runs` | `warn`: a task run has been active for more than 15 minutes. The message names the tasks and their age. | Inspect with `akm task history`. |
 | `connections-and-harnesses` | One table of every configured connection (endpoint, whether its key is held — a `$VAR` set in this shell, a `credentials.json` entry, or none — and whether the endpoint answers) and harness (platform, whether its binary is on `PATH`, and with a probe its `--version`), then the work that runs on them: improve, judges, agents, extract, enrich, index, embedding and curate rerank, each naming the connection or harness and the model it resolves to through the same resolver the calls use (`evidence.rows`, typed `connection`, `harness` or `process`). `warn` when a key variable is unset, an endpoint does not answer, a harness is not installed, several connections (or harnesses) are configured and none is called `default` (said once, with the fix: name one `default`), or a work row cannot resolve because of the config; `unknown` when neither a connection nor a harness is configured; otherwise `pass`. Endpoint reachability is one bounded `GET /models` and the version is the harness's own `--version`; with `--no-probe` neither runs and the message says so. | Correct the named connection, key or harness; run `akm setup` when nothing is configured. A work row whose parts run somewhere other than the row's own target lists them in `overrides`, each with the kind (connection or harness), name and model it runs on: that is a per-process pin, working as set. |
-| `task-fail-rate` | ≥5% of scheduled task runs failed in the window (exit 143/70 recurring), or a single task_id fails at/above the same threshold. (#943) For `command`-kind (agent/LLM dispatch) failures specifically, `evidence.agentFailureReasonCounts` always carries a `detail.reason` breakdown (`timeout`, `non_zero_exit`, `spawn_failed`, …), and the warn message names the dominant reason — e.g. `(timeout-dominant: 9/12 command-task failures)` — when one reason is at least half of the counted command-task failures. | Triage as a bug: `akm task doctor`, inspect failing lane logs. For a native shell/script task, exit-143 = killed/timeout, exit-70 = internal error. For a `command`-kind (agent/LLM) task, read `detail.reason` from `akm task history` instead of the exit code — `timeout` means the dispatch's own deadline (not an outer process kill) expired; `non_zero_exit`/`spawn_failed`/`aborted`/etc. name the other failure shapes. |
-| `data-dir-usage` | The data directory is more than 3× the live databases (`state.db` + `index.db` + `logs.db`), or one top-level subdirectory holds more than half of it; the message names that subdirectory with its size and share. | Look at the named subdirectory. `backups/tasks/` holds one snapshot per `akm migrate apply` run that rewrote a task file; everything under `backups/` is yours to prune. |
 | `semantic-search` | What the last `akm index` recorded about semantic search, with the index's entry and vector counts read now; no model load, no request. `pass` when every entry has a vector (local: on `onnxruntime-node`) or `semanticSearchMode` is `off`; `warn` when vectors lag entries, local embeddings ran on another runtime, or akm offers no local embeddings on this platform (`unavailable: platform`, where keyword search or an endpoint is the default); `fail` when the last index found it `blocked` (endpoint or model download failed) or `unavailable: load` (the runtime could not be downloaded or did not load, for example from a `noexec` cache directory), with the reason; `unknown` before the first index. A hard check, so a `fail` fails `akm health`. | `fail`: fix the reason named, then run `akm index`; the check reads the next index's record. `warn` for the platform: `akm setup --embedding off` for keyword search, or `akm setup --embedding <model> --embedding-endpoint <url>` for a remote endpoint. |
-| `stash-git-exposure` | `env/` or `secrets/` assets are git-tracked **and** a remote is set — `git push` can leak keys. | `git rm --cached` the files, add `env/`+`secrets/` to `.gitignore` (a rule alone does not untrack). |
-| `session-extraction` | (#914) Derived from the `extract_sessions_seen` ledger for the last 7 days — not `improve_runs`, which the hook-driven `akm proposal extract --session-id ...` never writes. `unknown` when nothing was recorded; `warn` when every session in the window was skipped for an infrastructure reason (`llm_unavailable`, `read_failed`, `exception`, `locked_concurrent`), naming the reason and engine; otherwise `pass` with per-outcome counts. | An `unknown` machine may be deliberately not configured for extraction, or may have stopped harvesting — check for recent sessions. A `warn` names the broken engine/reason directly. |
-| `auto-accept-validation` | Proposals passed the confidence gate but failed validation (bad frontmatter, truncation). | Review the affected pending proposals via `akm proposal list`; they were held, not lost. |
+| `task-fail-rate` | `warn`: at least 5% of task runs in the window failed, or one task with at least 5 runs fails at that rate. The message names the dominant `detail.reason` (`timeout`, `non_zero_exit`, `spawn_failed`, ...) when one reason is at least half of the command-task failures. | Inspect failed runs with `akm task history` and `akm task doctor`. `evidence.agentFailureReasonCounts` breaks command-task failures down by reason. |
+| `session-extraction` | `heuristic`. Reads the last 7 days of the extract ledger. `unknown`: nothing recorded. `warn`: every session was skipped or failed for an infrastructure reason (`llm_unavailable`, `read_failed`, `exception`, `locked_concurrent`); the message names the reason and the connection or harness that ran extraction. | Fix the named connection or harness. `unknown` is normal on a machine that does not extract sessions. |
+| `auto-accept-validation` | `heuristic`. Reads `triage.promoted` and `gateAutoAcceptFailedCount` (only rows an older release wrote carry it) from the improve runs in the window. `warn`: proposals passed the confidence gate but failed validation (truncated description, invalid frontmatter); they stay pending. | Review them with `akm proposal list`. |
+| `cli-version` | `warn`: a newer akm release exists on GitHub. `unknown`: `--no-probe`, or the release lookup failed. | Run `akm upgrade`. |
+| `scheduler-binary` | `warn`: the akm binary recorded in the scheduler reports a different version than the running CLI. `unknown`: `--no-probe`, no scheduled task installed, or the recorded binary cannot be run. | Run `akm task sync` to rebind the schedule. |
+| `index-state-generation` | `warn`: usage links in state.db point at index entries that no longer match (an index and state.db from different generations). | Stop concurrent writers and run `akm index --full`. |
+| `stash-git-exposure` | `warn`: `env/` or `secrets/` files are tracked by git in the working bundle and the bundle has a remote. Skipped when the bundle has no `.git`. | Run `git rm --cached` on the listed files and add `env/` and `secrets/` to `.gitignore`. A `.gitignore` rule alone does not untrack files. |
+| `binary-config-skew` | `warn`: `config.json` has a `configVersion` newer than this binary knows. | Upgrade this install. |
 | `egress-endpoints` | Every remote destination in the effective config, as one informational entry: enabled registries, git and website bundles, each connection's endpoint and the embedding endpoint (credentials in a URL are masked). Silent when nothing remote is configured. | Eyeball it against what you expect; a destination you did not configure came from an `extends` base or a shared bundle. |
-| `binary-config-skew` | `config.json` carries a `configVersion` newer than this binary knows. A newer akm wrote the shared config, so this install is stale. | Upgrade this install; do not keep a stale binary against the shared config and databases. |
-| `memory-cleanup-archive` | Size of a bundle's memory-cleanup archive. `pass` for a bundle with no git history (the purge sweep never touches it); `warn` when part of a git-backed bundle's archive cannot be purged (untracked, modified or unverifiable in git). Silent when nothing needs saying. | Commit the named files so the purge sweep can remove them once they age out. |
-| `type-directory-disagreement` | An indexed asset's resolved type disagrees with the type its directory declares. Always `warn`, never a gate; each row names the `winner` signal and marks the known-good overrides (a `knowledge/` file that is a command, an `agents/` file with an `agent:` key). | Check the unexplained rows; a deliberate override reads as such. |
-| `index-state-generation` | Durable usage links in `state.db` disagree with the current searchable index generation, which an interrupted cross-database update leaves behind. | Stop concurrent writers, then run `akm index --full` to relink state to the current index. |
-| `cli-version` | (#950) Compares the installed akm-cli version against the latest GitHub release (the same source `akm upgrade` trusts). `--probe`-gated: `unknown` "not probed" with `--no-probe`; offline/rate-limited also degrades to `unknown`, never a false `warn`. | Run `akm upgrade` when a newer release is reported. An `unknown` result carries no staleness claim either way — check manually if it matters. |
-| `scheduler-binary` | (#953) Reads the scheduler's recorded akm invocation for the installed tasks (the same binding `task sync`/`task doctor` already read — no crontab/launchd/schtasks text parsing) and runs it with `--version`, compared against the running CLI's version. `--probe`-gated: `unknown` "not probed" with `--no-probe`. `unknown` when no scheduled task is installed or the recorded binary cannot be executed. | A `warn` names both versions — upgrading akm through a different installer than the one active at the last `task sync` left the schedule pointed at the old binary; run `akm task sync` to rebind it. |
-| `plugin-version` / `opencode-plugin-version` | (itlackey/akm#832,) `plugin-version` reports each installed Claude Code harness plugin's version and its declared `AKM_VERSION_RANGE` admission of the running CLI. `opencode-plugin-version` does the same for the OpenCode plugin's own bundled `akm-cli` (read from its `package.json`, never executed) — it runs in-process and shares the host's databases, so its lag matters the same way. OpenCode 1 caches each spec in its own folder, so `opencode-plugin-version` reads the one folder of the spec the global OpenCode config names under `plugin` (`akm-opencode`, read as `akm-opencode@latest`, or `akm-opencode@next`), and for `akm-opencode@<version>` reports the plugin as pinned with no staleness verdict; with no entry, or no folder for it, it reads `packages/akm-opencode`, and it never scans the other cached specs. The newest version is read on the channel the marketplace clone follows: on the remote's default branch the newest stable tag, on any other branch (such as `next`) the release tag on that branch's head, and on a clone pinned to a tag that tag. `plugin-version`'s newest-tag lookup is `--probe`-gated: with `--no-probe` its row is `unknown` ("not probed") and only the local range check still runs. `opencode-plugin-version` reads local files only and is never gated. | A `warn` names the stale/out-of-range plugin and the update command (`claude plugin update <ref>` for the Claude plugin; update the OpenCode plugin for the bundled copy). |
+| `type-directory-disagreement` | `warn`: an indexed asset's resolved type differs from the type its directory declares. Each line gives the file, both types, and the classifier signal that won (`winner`). | Review the listed files. Intended overrides (for example a `knowledge/` file containing `$ARGUMENTS`) are marked `known-good override`. |
+| `memory-cleanup-archive` | `pass` with size and file count when the working bundle has `.akm/memory-cleanup/archive` content and no git history (the purge does not run there). `warn` when a git-backed bundle has archived files the purge cannot remove (untracked, modified, or unverifiable in git). | Commit the archived files so the purge can remove them once they age out. |
+| `data-dir-usage` | `warn`: the data directory is over 1 GB and either more than 3 times the size of `state.db`, `index.db`, and `logs.db` (with their WAL and SHM files), or one top-level subdirectory holds more than half of it. The message names that subdirectory and its size. | Inspect the named subdirectory. `backups/tasks/` holds task-file snapshots written by `akm migrate apply`. |
+| `plugin-version` | One row per installed Claude Code harness plugin: its version and whether its declared `AKM_VERSION_RANGE` admits the running CLI. The newest version is read on the channel the marketplace clone follows: on the remote's default branch the newest stable tag, on any other branch (such as `next`) the release tag on that branch's head, and on a clone pinned to a tag that tag. The newest-tag lookup is `--probe`-gated: with `--no-probe` the row is `unknown` ("not probed") and only the local range check still runs. `warn`: the plugin is stale, or its range does not admit the running CLI. | Run the `claude plugin update <ref>` command in the message. |
+| `opencode-plugin-version` | The same check for the OpenCode plugin's own bundled `akm-cli` (read from its `package.json`, never executed): it runs in-process and shares the host's databases, so its lag matters the same way. OpenCode 1 caches each spec in its own folder, so it reads the one folder of the spec the global OpenCode config names under `plugin` (`akm-opencode`, read as `akm-opencode@latest`, or `akm-opencode@next`), and for `akm-opencode@<version>` reports the plugin as pinned with no staleness verdict; with no entry, or no folder for it, it reads `packages/akm-opencode`, and it never scans the other cached specs. It reads local files only and is never `--probe`-gated. `warn`: the bundled version differs from the running CLI. Omitted when the OpenCode plugin is not installed. | Update the OpenCode plugin. |
+| `state-db-readable` | `fail`: `state.db` cannot be read, or is missing in a read-only run (which does not create it). Every other check is skipped, because none of them can read it. | Check the owner and mode of the data directory (or point `AKM_DATA_DIR` / `XDG_DATA_HOME` at one you own); run any akm command that records state, such as `akm health`, to create it. |
 
 ## Connections, harnesses and the work on them
 
@@ -52,20 +79,6 @@ The work rows are resolved by the resolvers the calls themselves use, so the
 table never disagrees with a call. An unreachable endpoint is a `warn` like
 every other finding here, never a `fail`.
 
-Reachability is a real, bounded (3 s) probe (#914) — one `GET` against the
-endpoint's `/models` route; any HTTP response counts, so a cold local server is
-never asked to load a model — made once per distinct endpoint per invocation
-(no cross-run cache, so a stale "reachable" verdict never lingers). Pass
-`--no-probe` to skip it (offline or air-gapped hosts): the rows then carry no
-`reachable` field and the message notes that reachability was not probed.
-
 The table's evidence names endpoints so an operator can tell which connection
 was checked; it never includes credential values or the names of the variables
 that hold them, and an endpoint's embedded credentials are masked.
-
-`akm health` makes at most two network calls besides the endpoint probe, both
-best-effort and silent on failure and both gated behind `--probe`/`--no-probe`,
-the same flag as endpoint reachability: `plugin-version`'s `git ls-remote` and `cli-version`'s
-GitHub-release lookup. See
-`src/commands/health/plugin-staleness.ts` and
-`src/commands/health/version-drift.ts` for the discipline each follows.