akm-cli 0.9.0-rc.8 → 0.9.0

package/docs/migration/v0.9.0-troubleshooting.md

@@ -0,0 +1,561 @@
+# AKM 0.9.0 Migration Troubleshooting
+
+This guide covers recovery actions for a 0.9.0 installation that did not
+complete cleanly. It is written for operators repairing their own AKM
+installation; no source-code changes are required.
+
+## Where AKM Stores Data
+
+Recovery steps below reference these locations. Each default can be
+overridden with the matching `AKM_*_DIR` environment variable; check the
+environment before assuming the defaults.
+
+| Variable | Default | Contents |
+|---|---|---|
+| `$CONFIG` | `~/.config/akm` | `config.json` |
+| `$DATA` | `~/.local/share/akm` | `state.db`, `index.db`, `logs.db`, `txn/`, `backups/` |
+| `$CACHE` | `~/.cache/akm` | task logs under `tasks/logs/`, index logs under `logs/` |
+| `$BUNDLE` | `~/akm` | working bundle: assets, tasks, scripts, workflows, env files |
+
+`akm config path --all` prints the resolved config, bundle, cache, and index
+paths for the installation you are repairing.
+
+## Before You Retry
+
+Stop scheduled AKM jobs and any running `akm improve`, `akm proposal extract`, or
+workflow processes. Do not delete databases, migration sentinels, WAL files, or
+backup directories while a migration is incomplete.
+
+Confirm that the command you are invoking is the intended 0.9 binary:
+
+```sh
+command -v akm
+akm --version
+akm upgrade --check
+```
+
+If the shell resolves an older installation, invoke the new package-manager
+launcher or staged binary explicitly for every migration command.
+
+## Pin the Runtime the Scheduler Uses
+
+`akm task doctor` reports the runtime `kind` and whether it is `eligible`
+for scheduling. Scheduled jobs must be bound to an installed release, never
+to a development checkout:
+
+- If doctor reports `kind: checkout` with `eligible: false`, the scheduler
+  is invoking a mutable source tree. Rebuilding that tree changes task
+  behavior mid-schedule, which shows up as failures flipping to successes
+  (or the reverse) within the same day's logs with no configuration change.
+- A launcher on `PATH` can be a thin wrapper back into a checkout. Verify
+  with `command -v akm` and `realpath` before trusting the version string.
+- The version string itself can lie: an uncommitted local `package.json`
+  edit produces a version that corresponds to no reproducible artifact.
+
+Also audit the task definitions and their helper scripts. Any command that
+invokes bare `akm` resolves through the scheduler's `PATH` at run time and
+can silently pick up a different binary than the one you validated. Pin
+nested invocations to the absolute launcher path — including in currently
+disabled tasks, so re-enabling one later cannot fall back to a stale
+binary — then resynchronize the bindings:
+
+```sh
+akm task doctor
+akm task sync --rebind
+akm task doctor
+```
+
+## Migration Status
+
+Start with the read-only status command:
+
+```sh
+akm migrate status
+```
+
+If you prepared a separate target configuration, pass it again:
+
+```sh
+akm migrate status --config ./prepared-0.9.json
+```
+
+The status output identifies any incomplete operation, source configuration,
+target configuration, and whether recovery is required.
+
+## Common Failures
+
+### WAL or database is busy
+
+If migration reports that another process holds a database, lock, or workflow
+claim, stop all schedulers and close every AKM process. Check for remaining
+processes, then retry:
+
+```sh
+akm migrate apply --config ./prepared-0.9.json
+```
+
+Do not remove `state.db-wal` or `state.db-shm` manually. SQLite owns those files.
+
+### Migration was interrupted
+
+After a killed process, host crash, or power loss, run status and retry apply:
+
+```sh
+akm migrate status
+akm migrate apply
+```
+
+The coordinator reuses the original backup and phase-free incomplete sentinel,
+including its retained target config and path base, then reruns the idempotent
+schema, data, and asset transforms. Do not edit or delete files under the AKM
+data directory to force progress. Preserve the sentinel and backup if retry
+reports malformed control data.
+
+### `akm migrate status` stays "old" for one surface
+
+If status reports every surface current except one, look for an obsolete
+database file left behind by an earlier version — for example an empty
+`workflow.db` in `$DATA` from a pre-0.9 install. Its mere presence can make
+that surface report as unmigrated. Stop all writers, move the file into a
+dated backup directory, and re-run `akm migrate status`. Do not delete the
+file; a later diagnosis may need it.
+
+### Configuration still uses retired keys
+
+Errors naming `stashDir`, `sources`, `installed`, `wikiName`, or another
+pre-0.9 key mean the current configuration has not crossed the bundle cutover.
+Prepare a 0.9 configuration with `bundles` and `defaultBundle`, then run the
+migration with that file:
+
+```sh
+akm migrate status --config ./prepared-0.9.json
+akm migrate apply --config ./prepared-0.9.json --dry-run
+akm migrate apply --config ./prepared-0.9.json
+```
+
+Do not run a 0.8 binary with `akm migrate`; that command exists only in the
+0.9 runtime.
+
+Retired keys also linger inside task commands and helper scripts. A common
+0.8 pattern was `akm config get stashDir`; it now fails with an unknown-key
+error on every scheduled run. Replace it with the bundle path from
+`akm config path --all` or `akm info`. Use `akm task history` to spot
+tasks failing on every interval: an unbroken run of failures starting at
+the migration date almost always means a retired key or reference inside
+the task definition, not a scheduler fault.
+
+### Old references no longer resolve
+
+The normal 0.9 grammar is `[bundle//]conceptId`, such as:
+
+```text
+skills/code-review
+memories/vpn-note
+knowledge/api-guide
+env/production
+```
+
+Replace old `type:name` references in your own prompts, task files, scripts,
+and documentation. The migration rewrites durable state and eligible asset
+content, but it cannot safely infer every reference embedded in operator-owned
+text.
+
+Check scheduled task YAML and workflow documents first: `env:name`,
+`workflow:name`, `knowledge:name`, and `skill:name` references inside a
+task definition fail on every run until rewritten as `env/name`,
+`workflows/name`, and so on.
+
+Workflow runs that were already in flight before the upgrade keep their
+frozen pre-0.9 plan. If a driver task resumes such a run, it executes the
+old plan — including retired commands — even after you rewrite the workflow
+document. Let the stale run reach a terminal state or abandon it, then
+start a fresh run so the rewritten plan is what executes.
+
+### Renamed or moved assets lose their ranking signal
+
+`akm mv` was removed in 0.9. A rename is now a plain filesystem move followed
+by `akm index` and `akm lint`, and the destination gets a **fresh identity**.
+Everything the old ref had earned — feedback events, usage events, salience and
+outcome history — stays keyed to the ref that no longer exists. The symptom is a
+long-serving asset that suddenly ranks like a brand-new one after you renamed
+or reorganized it, and orphaned rows accumulating in `state.db`.
+
+Re-key the rows onto the new ref from a source clone:
+
+```sh
+# Preview the counts it would move
+bun scripts/rekey-asset-ref.ts memories/old-note memories/new-note --dry-run
+
+# Apply
+bun scripts/rekey-asset-ref.ts memories/old-note memories/new-note
+```
+
+It re-keys the index `entries` row **in place** (preserving the row id, and
+with it the utility/embedding history keyed to that id), the `asset_salience`
+and `asset_outcome` rows, and `usage_events.entry_ref` — then appends one
+`rekey` event. It is idempotent: a second run reports zero changed rows.
+
+Two constraints follow from the identity model, and the script refuses rather
+than guess: refs must name the same bundle and the same asset type
+(cross-bundle and cross-type movement is copy/import plus delete), and the old
+file must be gone — if both files exist that is a copy, not a rename.
+
+Run it **before** `akm index` when you can. Afterwards still works and carries
+the `state.db` signal, but `akm index` will already have dropped the old
+`entries` row and minted a fresh one, so the utility/embedding history attached
+to that row id is gone. The improve maintenance pass marks and clears orphaned
+salience/outcome state. With `improve.stateGc.collect: true`, it collects rows
+that remain orphaned after the seven-day grace period and emits an
+`asset_state_gc` event. Re-key before indexing when you need to preserve that
+history rather than collect it.
+
+### `akm wiki` commands fail
+
+The `akm wiki` command family was removed in 0.9. Use ordinary bundle and
+knowledge commands instead:
+
+```sh
+akm index
+akm search "your query"
+akm show knowledge/your-document
+akm lint --type knowledge
+```
+
+For URL snapshots, use `akm import URL --path articles`, then index and lint
+the destination.
+
+Scheduled pipelines built on `akm wiki` must be rewritten, not renamed:
+
+1. Acquire URLs with `akm import <url> --path <subdir>` into an ordinary
+   knowledge subtree (for example `knowledge/articles`).
+2. Deduplicate by source URL before importing, and normalize alias hosts
+   (for example `twitter.com` vs `x.com`) so one article is not imported
+   twice under two URLs.
+3. Verify with `akm index`, `akm search`, and `akm show`, and lint the
+   destination subtree rather than the whole bundle.
+4. Advance any incremental cursor (channel position, feed offset) only
+   after the whole task run succeeds, so a failed run is retried instead
+   of silently skipped.
+
+### Tasks do not run after migration
+
+Task files are strict YAML v2 in 0.9. Inspect and resynchronize the installed
+scheduler bindings:
+
+```sh
+akm task doctor
+akm task sync --rebind
+akm task doctor
+```
+
+Use `akm task history` to distinguish a disabled task, a failed command, and a
+task that was never installed. A disabled task can still be run explicitly for
+testing:
+
+```sh
+akm task run <task-id>
+```
+
+If a periodic task fails only when it coincides with a running improve
+cycle, add `--skip-if-locked` to its improve invocation so the overlap is
+skipped instead of recorded as a failure.
+
+`akm task history` can also show rows stuck in an active state from runs
+interrupted by a crash or power loss. Confirm no AKM process is actually
+running before treating them as abandoned. Such rows are historical
+records only, but they distort health fail-rate statistics until resolved.
+
+### Improve aborts on legacy pending proposals
+
+Proposals created before 0.9 can lack metadata the 0.9 lifecycle requires —
+most often the proposed target and per-change paths. Symptoms:
+
+- `akm improve` exits with code 70 on every cycle while processing a
+  specific proposal.
+- `akm proposal reject` fails on the same row because archival cannot
+  serialize the missing metadata.
+
+Inspect the queue read-only before changing anything:
+
+```sh
+akm proposal list --status pending --format json
+sqlite3 -readonly "$DATA/state.db" \
+  "SELECT COUNT(*) FROM proposals WHERE status='pending' \
+   AND json_extract(metadata_json,'$.proposedTarget') IS NULL;"
+```
+
+Reject duplicates and stale drafts through the normal proposal commands
+wherever they still work; that preserves the audit trail. If rejection
+itself fails on a malformed row, stop all writers, take a fresh
+`sqlite3 ".backup"` copy of `state.db`, add only the missing metadata
+fields to the affected rows, verify `PRAGMA quick_check` returns `ok`, and
+then retry the CLI rejection. Never delete proposal rows directly.
+
+The 0.9 pre-publish lint gate keeps structurally valid but lint-invalid
+legacy drafts pending instead of publishing them. They are harmless, but
+each is re-evaluated every cycle; reject drafts that can never promote so
+they stop consuming the promotion budget.
+
+### Stale transaction journals under `$DATA/txn`
+
+An interrupted run can leave a journal directory under
+`$DATA/txn/<installation>/<transaction-id>/`. Read its `journal.json`
+before acting:
+
+- A journal in an applying phase with zero completed operations recorded
+  no file changes. After stopping all writers, move the whole transaction
+  directory into a dated backup location. Do not delete it.
+- A journal whose operations already published files needs per-proposal
+  reconciliation. If a published file matches the proposal content,
+  finalize the acceptance through the proposal commands. If the file has
+  since diverged (for example because you fixed lint defects in it), keep
+  the newer file, quarantine the journal, and reject the superseded
+  proposal.
+
+A leftover journal can also block `akm proposal reject` for the proposal
+it references. Resolve the journal first, then retry the rejection.
+
+A journal whose `kind` no longer exists in 0.9 — notably `"mv"`, left by an
+rc-era `akm mv` run — needs no action. It records no recoverable work, and any
+recovery scan that meets it removes it once it is older than five minutes
+rather than failing.
+
+### Index or search results look incomplete
+
+The index is regenerable. First verify the bundle and configuration, then
+rebuild and inspect lint findings:
+
+```sh
+akm index
+akm health
+akm lint --type knowledge
+akm search "a known document title"
+```
+
+Do not restore an old index database over a current one unless you have a
+separate, verified recovery procedure. Rebuilding the index preserves the
+source assets and avoids mixing database generations.
+
+### `source "<name>" was not scanned completely`
+
+Indexing prints this warning and preserves the source's last-known-good
+rows when a file listed by the source cannot be read back during the walk.
+On a Git-backed bundle the usual cause is benign: a tracked file was deleted
+in the working tree and the deletion is not yet committed. The final 0.9.0
+runtime handles that state; if you still see the warning, commit or restore
+the deletion in the bundle repository and re-index:
+
+```sh
+git -C <bundle-dir> status --short
+akm index
+```
+
+Because last-known-good rows are preserved, search keeps working while the
+warning is active — but deleted assets do not disappear from the index
+until a complete scan succeeds.
+
+### Local LLM features fail after upgrade
+
+Check the configured engine and endpoint without exposing credentials:
+
+```sh
+akm config get engines
+akm health
+```
+
+Confirm that the local model server is running, reachable from the host, and
+serves the configured model. Keep the endpoint and model under the named
+`engines` configuration; the retired top-level `llm` configuration is not a
+0.9 setting.
+
+Verify the endpoint directly before changing AKM configuration:
+
+```sh
+curl -sS -o /dev/null -w '%{http_code}\n' http://HOST:PORT/v1/models
+```
+
+A 200 without an `Authorization` header means no token is needed — do not
+wire one in. Old 401 entries in task logs can be stale evidence of a
+transient server state, not a configuration rule; trust the live probe.
+If a chat or classification feature fails while embeddings work, confirm
+the exact configured model is loaded on the host you are pointing at: a
+reachable server without the model loaded fails in ways that resemble
+authentication or endpoint errors, and the fix may simply be pointing the
+configuration at the host that actually serves the model.
+
+### Derived memories contain placeholder text
+
+A small local model can echo a prompt's example structure instead of
+producing real content. Affected derived memories contain literal template
+text — a description like "one sentence summary", tags like `tag1`, a
+generic template body — and often omit the `updated` field. The final
+0.9.0 runtime rejects this output pattern and uses a new inference cache
+namespace, so upgrade first; regeneration under an old binary can
+reproduce the same placeholders from cache.
+
+Then clean up in this order:
+
+1. Search the bundle for the literal template phrases and remove only the
+   placeholder derived assets. Do not edit dates onto them; that conceals
+   the bad automation without fixing it.
+2. Re-run indexing and the improvement task with the upgraded runtime.
+3. Confirm the regenerated memories contain real content and an `updated`
+   date before re-enabling any schedule that consumes them.
+
+### Improve does less than it did on 0.8
+
+0.9 gates autonomous maintenance lanes (memory inference, memory cleanup,
+automatic triage promotion) behind an explicit opt-in. While
+`experimental.improveAutonomy` is false, those lanes are skipped or queue
+their work for review instead of applying it. That is configuration, not
+damage. Check the current value and enable the opt-in only after reviewing
+what the lanes may write:
+
+```sh
+akm config get experimental
+```
+
+Read health fail-rate advisories with the same care: the aggregate combines
+improve results with every scheduled task, so one misconfigured legacy task
+failing on a tight schedule can dominate the percentage while improve
+itself is healthy. Distinguish deterministic check failures (integrity,
+index, scheduler) from historical-rate advisories before treating health
+as degraded.
+
+### State databases are very large after migration
+
+Improve telemetry accumulates in `state.db` and can dominate its size;
+multi-gigabyte stored improve-run results are the usual cause. Set a
+retention window, let the next quick improve run purge expired rows, then
+compact offline:
+
+```sh
+akm config set improve.eventRetentionDays 30
+akm task run <quick-improve-task-id>
+```
+
+Check the run log for the purge counts, stop all writers, then compact and
+verify each database:
+
+```sh
+sqlite3 "$DATA/state.db" "VACUUM; PRAGMA quick_check;"
+sqlite3 "$DATA/logs.db" "VACUUM; PRAGMA quick_check;"
+```
+
+Also review `$DATA/backups`: migration backups are large by design. Keep at
+least the most recent verified pre-cutover backup, and do not prune backup
+directories while any migration or recovery question is open.
+
+### Proposal promotion is rejected by lint
+
+0.9 rejects proposal promotion when a proposal has critical `unquoted-colon`,
+`missing-ref`, or `stale-path` findings. Inspect the proposal, fix the content
+or add an intentional lint suppression, then retry:
+
+```sh
+akm proposal show <proposal-id>
+akm proposal diff <proposal-id>
+akm proposal accept <proposal-id>
+```
+
+Rejected promotion leaves the proposal pending and does not publish the bad
+asset. Do not force-copy the proposal into the bundle; fix the proposal through
+the proposal command so its audit trail remains intact.
+
+### Read-only bundles still contain v1 tasks
+
+The migration deliberately never rewrites a read-only bundle. Affected
+bundles are reported per bundle (a `readOnlyLegacyTasks` warning naming the
+bundle and files), and those tasks remain non-executable warnings until the
+upstream source ships 0.9-format tasks. Do not mark a lock-materialized
+cache `writable: true` to silence the warning — the next source update
+overwrites the cache and your edit with it. Update the upstream source, or
+replace it with a writable local source if you must run those tasks now.
+
+## Migrations Performed With a Pre-Release Build
+
+If the migration originally ran under a 0.9.0 release candidate, re-verify
+it with the final binary before trusting the environment:
+
+1. Run `akm migrate status` with the final 0.9.0 binary.
+2. Read
+   `$DATA/backups/migrations/<installation-id>/content-migration-report.json`
+   and confirm every installed bundle you expected was imported. Early
+   builds could skip an installed tree whose lock still used a prefixed
+   locator (for example `github:owner/repo`), leaving its legacy metadata
+   and filesystem proposals unimported.
+3. Check whether quarantined legacy rows were preserved in full. The final
+   migration stores them in a `legacy_state_rows` table inside `state.db`;
+   some early builds recorded only per-surface summary counts (a
+   `legacy_state` table with counts but no `legacy_state_rows`). In that
+   case the full data exists only in the pre-cutover backup — preserve that
+   backup indefinitely, or restore and re-migrate with the final binary if
+   the quarantined history matters to you.
+
+```sh
+sqlite3 -readonly "$DATA/state.db" \
+  "SELECT name FROM sqlite_master \
+   WHERE name IN ('legacy_state','legacy_state_rows');"
+```
+
+## Credential Hygiene After Failures
+
+Runs under pre-final builds could write full webhook URLs into task logs
+when an unhandled network error printed the failing request. Before
+archiving or sharing logs, scan for embedded credentials and rotate
+anything you find — deleting the log copy does not un-expose a credential:
+
+```sh
+grep -rl 'discord.com/api/webhooks/' "$CACHE/tasks/logs" || true
+```
+
+Keep scheduler environment files (`env/*.env` in the bundle) at mode `600`
+and out of version control. Scheduled task environments frequently hold
+plaintext credentials; never paste their contents into logs, reports, or
+issue trackers while diagnosing a failure.
+
+## Recovery and Downgrade
+
+Migration recovery runs are stored below:
+
+```text
+$DATA/backups/migrations/<installation-id>/<run-id>/
+```
+
+If you must restore, stop all writers first and use a verified recovery run:
+
+```sh
+akm-migrate restore --for 0.9.0 --run <run-id> --confirm
+```
+
+Only install an older AKM binary after restore completes. A 0.8 binary must
+not run against a 0.9 configuration or migrated databases. If no verified
+pre-cutover backup exists, preserve the current installation and reconstruct a
+separate older-version data root instead of downgrading in place.
+
+## Final Verification
+
+After recovery, run these checks with the same binary used for migration:
+
+```sh
+akm --version
+akm migrate status
+akm task doctor
+akm index
+akm health
+akm lint --type knowledge
+akm proposal list --status pending
+```
+
+Before restarting schedules, also confirm the environment is quiescent:
+
+- `$DATA/txn` contains no leftover transaction journals.
+- No pending proposal is missing its target metadata (see the legacy
+  pending proposals section above).
+- `akm task history` shows no rows stuck in an active state.
+- Task logs contain no embedded credentials.
+
+Restart scheduled jobs only after status is committed, the scheduler is bound
+to the current launcher, and the index and lint checks are clean. Then watch
+the first full scheduled cycle: one completed run of each task family under
+cron is the real proof of recovery, not a successful manual invocation.

package/docs/reference/README.md

@@ -0,0 +1,12 @@
+# Reference
+
+Authoritative reference documentation for the akm CLI and its data.
+
+- [CLI](cli.md) -- All `akm` commands and flags
+- [Configuration](configuration.md) -- Engines, strategies, bundles, and settings
+- [Workflows](workflows.md) -- Unified Markdown workflow schema, run state, and native orchestration engine
+- [Registry](https://github.com/itlackey/akm/blob/main/docs/reference/registry.md) -- Registries, search, hosting, and managing sources
+- [Wiki Snapshot Fetchers](https://github.com/itlackey/akm/blob/main/docs/reference/wiki-snapshot-fetchers.md) -- The pluggable fetcher API for URL-based knowledge reads
+- [Data & Telemetry](data-and-telemetry.md) -- Exactly what akm reads and writes on your machine (no remote telemetry)
+- [akm-eval](https://github.com/itlackey/akm/blob/main/docs/reference/akm-eval.md) -- Standalone toolkit for measuring whether `akm improve` is working
+- [Roadmap](https://github.com/itlackey/akm/blob/main/docs/reference/roadmap.md) -- High-level focus for the 0.9 and 1.0 releases