akm-cli 0.9.15 → 0.9.16-alpha.1

package/docs/migration/release-notes/0.9.16.md

@@ -0,0 +1,110 @@
+Migration notes for akm v0.9.16
+
+The index is redesigned end to end (`docs/plans/index-redesign.md`): `akm
+index` is now reconcile-then-drain instead of a walk/clean/embed/finalize
+phase pipeline, everything derived from a file is keyed by the hash of what
+it was derived from, and there is exactly one text table
+(`unit_texts`/`units_fts`) and one vector table (`units`/`units_vec`) instead
+of the old entry-level and fragment-level duplicates. The derived `index.db`
+generation changes from v23 to v24.
+
+**First run after upgrade.** On the first normal read (or an explicit `akm
+index`), akm detects the v23 cache cannot serve queries under v24 and
+rebuilds the derived generation: `entries` and every unit are re-derived from
+your files (a stat walk plus a parse of everything, since the old generation
+has no `files` stat cache to diff against — this is CPU-bound and fast, not
+network-bound). The new unit vector store starts empty: the old
+`embeddings`/`entries_vec` tables it replaces are not carried forward, so
+every unit is embedded once at your configured provider, the same as a fresh
+install. On a large corpus this is a real, paid cost, once per embedding
+model, and it is never repeated for unchanged text again:
+not on a rename, not on the next `akm index --full`, not on a future
+generation bump, only on a genuine content or model change. Search works
+lexically from the first minute of the rebuild and gains semantic results as
+the embedding queue drains in the background. `akm index status` reports the
+queue's progress (units with a vector for the active identity, versus
+still-pending) without triggering any writes. Do not run an older akm against
+an index this release has already rebuilt: upgrade that binary instead.
+
+**Config keys removed.** `embedding.maxInputTokens`, `embedding.maxTokens`,
+`embedding.batchSize`, and `embedding.contextLength` are gone from the
+schema entirely (0.9.15 had already stopped reading them for packing;
+0.9.16 removes them outright). `akm index` packs every embedding request
+against the provider's own probed context window and slot count — this is
+unchanged from 0.9.15, only what it packs (units instead of whole entries)
+moved. `search.minScore` is also gone: reciprocal-rank fusion was tried and
+measured worse (0.918 mean vs. a 0.933 pre-redesign baseline, full table in
+`docs/plans/index-redesign.md`) and rejected. What ships instead scores
+lexical evidence by magnitude through the calibrated BM25 transform
+(`stableFtsScore`) and fuses it with cosine similarity at a 0.7/0.3 split,
+with the exact/prefix/relaxed lexical tier a hit came from ranking ahead of
+that fused score — the transform's realized output band turned out to be
+only 0.021 wide after weighting, too narrow on its own to separate a strong
+match from a weak one. No separate floor is applied over any of it, which
+is why `minScore` has nothing left to tune. A config file that still sets any of these five keys
+loads without error — they are simply ignored, like `embedding.chunkSize`
+before them — see [Retired
+Configuration](../../reference/configuration.md#retired-configuration).
+
+**`--skip-if-locked` is deprecated and does nothing.** Index runs no longer
+take a rebuild lock at all — every write is a short, idempotent,
+content-addressed transaction, and two concurrent `akm index` runs converge
+on the same end state instead of contending — so there is nothing left to
+skip around. Passing the flag prints one deprecation warning and the run
+proceeds exactly as an ordinary `akm index` would; it is kept only so an
+existing script or scheduled task does not fail on an unknown flag.
+
+**`--enrich` and `--re-enrich` are removed.** Plain `akm index` now always
+performs metadata enrichment when an LLM engine is configured (the same
+behavior `--enrich` used to opt into); re-enrichment of index-time LLM
+passes is not exposed in this slice. Both flags now fail with a `UsageError`
+(exit 2) naming the replacement, rather than silently doing nothing.
+
+**`--clean` and `--dry-run` are removed.** Every `akm index` run already
+removes stale entries as part of reconcile — the same work `--clean` used to
+opt into — so there is no longer a distinct pass for `--dry-run` to preview.
+Both flags now fail with a `UsageError` (exit 2) instead of being silently
+accepted.
+
+**`akm index`'s result envelope renames two fields and drops one.**
+`generatedMetadata` is now `entriesUpserted`: it always counted the files
+reconcile added or changed, never LLM-generated metadata, so a script reading
+it as enrichment coverage was being misled. `directoriesScanned` is now
+`sourcesScanned`, because reconcile is a flat per-file stat walk with no
+directory granularity left to count, and it reports configured source roots.
+`directoriesSkipped` is gone: it could only ever be `0` under the new walk.
+The text output's summary line follows the same change. A script reading the
+old names gets `undefined` and should switch to the new ones.
+
+**`akm index status` is new.** A cheap, read-only snapshot — files tracked,
+entries, distinct units, how many have a vector for the active embedding
+identity, the identity itself, and the last reconcile/build times — with no
+writes. It mirrors `akm info`'s handling of a missing or unreadable index: an
+absent index reads as the ordinary first-run state, and one that exists but
+cannot be read is reported, never silently presented as empty.
+
+**Exit codes are unchanged.** `INDEX_DB_CONTENDED` (exit 75) already existed
+and still means the same thing — a second connection held a write
+transaction long enough to exhaust SQLite's own busy timeout — but it now
+covers genuinely rare contention instead of also standing in for the old
+rebuild lock's own failure modes, which no longer exist because the rebuild
+lock itself is gone.
+
+**The optional cross-encoder rerank pass moves from `akm curate` to `akm
+search`.** It was always meant for search; `search.curateRerank` (0.9.15,
+#951) is renamed `search.rerank` with no compatibility alias — the old key
+shipped hours earlier and default-off, so nobody has it meaningfully set.
+`akm curate` no longer reranks at all.
+
+**`"ready-js"` is retired from `akm info`'s `semanticSearch.status` and from
+`akm index`'s verification status.** It named a pure-JS cosine-similarity
+fallback for when the `sqlite-vec` extension was unavailable; the new unit
+vector store has no BLOB-table fallback to fall back to, so nothing produces
+that value any more. The two surfaces now diverge in what replaces it:
+`akm info`'s `semanticSearch.status` reports `"pending"` until the
+embedding queue finishes draining and `"ready-vec"` once it has — a host
+without the extension can never reach `"ready-vec"` (`vecAvailable` false
+rules it out) and stays `"pending"` indefinitely; that status has no
+`"blocked"` value at all. `akm index`'s own verification status reports the
+same `"pending"`/`"ready-vec"`, plus `"blocked"` when a run makes no
+embedding progress at all.

package/docs/migration/release-notes/README.md

@@ -7,6 +7,11 @@ live one level up in `docs/migration/`.
 
 ## Available notes
 
+- [0.9.16](0.9.16.md) — the index redesign: `akm index` is now
+  reconcile-then-drain over one content-addressed text table and one vector
+  table, index generation v23-to-v24, five retired config keys, `--enrich`/
+  `--re-enrich`/`--clean`/`--dry-run` removed, `--skip-if-locked` now a
+  no-op, and the new `akm index status`
 - [0.9.15](0.9.15.md) — exit-code 75 for lease/state.db contention,
   `--require-engines` scheduled task templates, `--no-probe` cli-version
   skip, thinking-control wire forms, embedding re-embed safety and

package/docs/reference/cli.md

@@ -218,97 +218,87 @@ and builds the search index.
 Build or refresh the search index.
 
 ```sh
-akm index            # Incremental (only changed directories)
-akm index --full     # Full rebuild (reuses unchanged embeddings — see below)
+akm index            # Reconcile: diff every configured root against the index, drain the embedding queue
+akm index --full     # Force every file to be re-derived, ignoring the unchanged-file shortcut, reconciling in place
 akm index --verbose  # Print phase progress to stderr
-akm index --clean    # Normal index + remove stale entries from the DB
-akm index --clean --dry-run # Report stale entries without deleting
-akm index --reembed  # Force re-embedding of every entry
-akm index --skip-if-locked  # for scheduled/opportunistic runs: skip (exit 0) if a run is already in progress
+akm index --reembed  # Drop the active embedding identity's vectors, then re-embed every unit from scratch
+akm index status     # Report file/entry/unit counts, active identity, and last-reconcile time — no writes
 ```
 
-Returns stats: `totalEntries`, `generatedMetadata`, `directoriesScanned`,
-`directoriesSkipped`, `verification`, optional `warnings`, and `timing`
-breakdown in milliseconds. Use `--verbose` to print the indexing mode,
+Returns stats: `totalEntries`, `entriesUpserted`, `sourcesScanned`,
+`verification`, optional `warnings`, and `timing` breakdown in milliseconds.
+Use `--verbose` to print the indexing mode,
 semantic-search settings, and phase-by-phase progress to stderr while the
 index is being built. Malformed workflow assets are skipped with file-path
 warnings instead of aborting the full run.
 
-**Progress in non-verbose JSON mode (default output format, #954):** even
-without `--verbose`, phase-start messages and the embedding heartbeat
-(`Still generating embeddings: X/N stored, F failed; waiting on embedding
-provider.`) are now written to stderr, and a failed embedding batch logs at
-the default level instead of `--verbose`-only — a long-running index build
-against a slow or unresponsive provider is no longer silent until the whole
-run finishes. Text-mode output keeps its spinner instead (no stderr line
-growth); JSON stdout output is unaffected either way. The high-frequency
-per-batch `Embedded N/M entries.` line stays out of non-verbose stderr (it
-fires after every committed batch) — pass `--verbose` for that level of
-detail.
-
-**`--clean` flag:** After indexing completes, verifies every indexed entry's source
-file still exists on disk. Removes any entries whose file is missing (for local
-bundle sources only; remote entries are skipped). Returns a `clean` block in the
-JSON result with `checked`, `removed`, `removedRefs` arrays, and `dryRun` flag.
-Use `--clean` to resolve the edge case where a deleted file in an unchanged
-directory lingers in the index across incremental runs. With `--dry-run`, reports
-which entries would be removed without modifying the database.
-
-**`--full` no longer re-embeds unchanged content (#955):** a full rebuild
-(and an index-generation bump on first open under a new binary) used to
-delete every embedding unconditionally, forcing a full re-embed of the
-whole corpus even when nothing changed. Vectors about to be discarded are
-now salvaged (keyed by a hash of their content plus the fingerprint they
-were generated under) and handed straight back to unchanged entries at the
-start of the next embedding pass, with zero provider calls for them — a
-progress line reports the split (`Reused N embeddings from the previous
-generation; embedding M new.`). Content that changed even by one byte, or
-a fingerprint that no longer matches, still goes through the provider
-normally. `--reembed` is the way to force a full re-embed regardless.
-
-**`--reembed` flag:** Forces a full purge and re-embed of every entry,
-independent of the embedding-model-rename compatibility check described
-below. Ordinary indexing already tells a config-only rename of
-`embedding.model` (e.g. a gateway that changes how it names the same model)
-apart from a genuine model change, and keeps the stored vectors when they
-are still compatible; `--reembed` skips that check and forces a rebuild
-regardless of what it would have decided.
-
-**`--skip-if-locked` flag:** Every explicit `akm index` run acquires an
-opt-in, PID-liveness-only rebuild lock and releases it on exit — this is
-advisory, never the blocking lock #872 removed (see
-[Locks](https://github.com/itlackey/akm/blob/main/docs/architecture/internals/indexing.md#locks)). A human-typed
-`akm index` with no flag is never gated by it: if another run already holds
-the lock, it warns and proceeds anyway, contending with the existing run. If
-that contention makes index.db genuinely busy (SQLite `database is locked`)
-long enough to exhaust the driver's retry window, the run now fails with
-exit 75 (`TransientError`, code `INDEX_DB_CONTENDED`) instead of the raw
-driver error at exit 70 — the same retry-shortly contract as
-`STATE_DB_CONTENDED`, so a scheduler can branch on it instead of alerting.
-The rebuild lock itself is registered through a brief internal barrier
-(`getMaintenanceBarrierPath()`) shared with every other akm lock/lease; two
-`akm index` runs launched close enough together to collide on that
-registration step retry briefly and then, if it is still busy, also exit 75
-(code `MAINTENANCE_BARRIER_BUSY`) rather than the config-error exit 78 a
-2026-09-10 field report found — a busy registration barrier is ordinary
-contention between two legitimate runs, never a broken config file.
-`--skip-if-locked` changes that only for the invocation that passes it: if
-the lock is already held by a live process, it skips gracefully (exit 0,
-`{ ok: true, skipped: { reason: "lock-held", pid, launcherPid, startedAt } }`
-— `launcherPid` is the holder's launcher pid when known, `null` otherwise,
-#956) instead of contending. `akm index` and `akm curate` are both safe to call frequently —
-`curate` never blocks on a rebuild in progress ([read-path indexing stays
-non-blocking](#curate)) — but a hook, cron job, or scheduled task that
-invokes `akm index` directly should pass `--skip-if-locked` so it steps
-aside instead of piling up behind a longer rebuild (the shipped
-`index-refresh` task does this).
-
-`akm index` always rebuilds the search index and keeps metadata in the index.
-When a selected named LLM engine (`defaults.llmEngine` or an indexing-pass
-override) is configured and the per-pass gate allows it, metadata
-enhancement runs during indexing. In text mode, the default CLI UI shows a
-spinner with processed-versus-total source counts; structured output modes
-(`json`, `yaml`, `jsonl`) stay clean and machine-readable.
+**Progress in non-verbose JSON/yaml/jsonl mode:** phase-start messages and
+each phase's summary line — including `[embed] endpoint ...` and `[drain]
+done: ...` — reach stderr regardless of `--verbose`: a long-running index
+build against a slow or unresponsive provider is no longer silent until the
+whole run finishes. Two high-frequency, one-line-per-unit-of-work lines are
+the exception and stay suppressed without `--verbose`: drain's per-batch
+commit line (`[drain] batch N: ...`) and reconcile's per-root line
+(`Reconciled "<path>": N files scanned.`) — printing one of those per batch
+or per root would be spam, not a heartbeat. Pass `--verbose` to print those
+two as well, alongside the same summary lines. Text mode is where
+`--verbose` also matters for everything else: without it, progress updates a
+single spinner line in place; with it, every line is printed as it arrives
+instead. JSON stdout output is unaffected either way.
+
+**Reconcile, not a walk-and-rebuild pipeline:** `akm index` diffs every
+configured root's files against the index (stat cache: unchanged files are
+skipped without a re-parse), upserts what changed, removes what is genuinely
+gone, then drains the content-addressed embedding queue. Every write is a
+short, idempotent transaction — there is no rebuild lock, no writer lock on
+the index path, and no per-command background reindex spawn: two concurrent
+`akm index` runs simply converge on the same end state instead of
+contending. `akm index status` (no writes) reports file/entry/unit counts,
+the active embedding identity, and the last-reconcile time.
+
+**`--full`:** does NOT drop or wipe anything first. It reconciles with every
+walked file treated as needing re-derivation — the stat-hint "unchanged"
+shortcut is skipped, so every file is re-parsed — but each file's existing
+row (keyed by its stable `item_ref`) is updated in place, not deleted and
+reinserted: the row keeps its id, its embeddings, and its learned utility
+scores. Content-addressed units (`units`/`units_vec`, keyed by content hash,
+never by entry id) are never at risk from a reindex at all, `--full`
+included — there is nothing to re-embed for unchanged content. Gone-path
+detection (files genuinely removed) is unaffected by `--full`; a root whose
+walk could not be trusted this run (an unreachable path, a mid-walk stat
+failure) has its entire existing snapshot preserved rather than partially
+wiped over a transient scan failure.
+
+**`--reembed`:** drops the active embedding identity's vectors, then the
+drain re-embeds every unit from scratch under that identity. Ordinary
+indexing does not need a separate rename-compatibility check: the identity a
+unit's vector is keyed under is exactly what the provider's response
+reported (model id and vector width), so a config-only rename of
+`embedding.model` that still resolves to the same server-reported model
+keeps the same identity — and its stored vectors — automatically, while a
+genuine model or dimension change lands under a different identity and its
+units are simply "missing" until the next drain.
+
+**`--skip-if-locked`:** deprecated, no effect — index runs no longer take a
+rebuild lock, so there is nothing left to skip around. Passing it prints one
+deprecation warning; the run proceeds exactly as an ordinary `akm index`
+would. Kept only so an existing script or scheduled task does not fail on an
+unknown flag. If index.db is genuinely busy (a second connection holds a
+write transaction) long enough to exhaust the driver's own SQLite
+`busy_timeout`, the run fails with exit 75 (`TransientError`, code
+`INDEX_DB_CONTENDED`) instead of the raw driver error at exit 70 — the same
+retry-shortly contract as `STATE_DB_CONTENDED`, so a scheduler can branch on
+it instead of alerting.
+Locks that do still exist (`akm improve`, `akm workflow run`) are registered through a brief
+internal barrier shared by every akm lock and lease; two runs launched close enough together to
+collide on that registration step retry briefly and then, if it is still busy, exit 75 (code
+`MAINTENANCE_BARRIER_BUSY`) rather than the config-error exit 78 a 2026-09-10 field report
+found — a busy registration barrier is ordinary contention between two legitimate runs, never a
+broken config file.
+
+In text mode, the default CLI UI shows a spinner with processed-versus-total
+source counts; structured output modes (`json`, `yaml`, `jsonl`) stay clean
+and machine-readable.
 
 ### info
 
@@ -337,12 +327,16 @@ Returns a JSON object with:
 | `indexStats` | Index stats: `entryCount`, `byType` (per-asset-type breakdown), `lastBuiltAt`, `hasEmbeddings`, `vecAvailable` |
 
 `semanticSearch.status` values:
-- `"ready-vec"` — native sqlite-vec extension active (fastest)
-- `"ready-js"` — pure JS fallback active (correct but slower at scale)
+- `"ready-vec"` — embeddings present, `sqlite-vec` active
 - `"pending"` — not yet initialized (run `akm index` to set up)
-- `"blocked"` — setup failed (see `reason` and `message` fields)
 - `"disabled"` — semantic search is turned off in config
 
+(`"ready-js"`, a pure-JS cosine fallback for when the extension was
+unavailable, is retired — the unit vector store has no BLOB fallback to fall
+back to. A degraded/failed embedding run is reflected here as `"pending"`,
+not a separate `"blocked"` value; see `akm index`'s own `verification.semanticStatus`,
+which does distinguish `"blocked"`, for that detail.)
+
 Use `akm info` to verify that semantic search is working after setup.
 
 Scripts that need akm's resolved paths (for example a health check that
@@ -499,6 +493,18 @@ query. The last case also adds one sanitized, endpoint-naming entry to
 preserved by `--shape agent` so machine consumers can lower their confidence
 instead of treating keyword fallback as healthy semantic ranking.
 
+An optional cross-encoder rerank pass (#951, `search.rerank.*` —
+`docs/reference/configuration.md`) can run over local hits before `--from
+local`/`--from all` diverge; disabled by default, and registry hits (`--from
+registry`, and the registry half of `--from all`) are never reranked. When it
+runs, it changes hit **ORDER** only — each hit's `score` stays the retrieval
+score `akm search` computed, not the reranker's own relevance value. A
+consumer that wants the reranked ranking must read hits in ARRAY ORDER, not
+by re-sorting on `score`: `score` is a fixed `[0,1]` contract (see the
+callout below) that a reranker's own scale — provider-defined, not
+calibrated to `[0,1]` — would break if it overwrote it, and a re-sort would
+silently undo the rerank for exactly the consumers it exists to serve.
+
 | Flag | Values | Default | Description |
 | --- | --- | --- | --- |
 | `--type` | `skill`, `command`, `agent`, `knowledge`, `instruction`, `workflow`, `script`, `memory`, `env`, `secret`, `lesson`, `task`, `session`, `fact`, `any` | `any` | Filter by asset type. Free-form and unvalidated — an unknown type returns no hits. Also accepts any adapter-defined type (e.g. `website`) — see [Bundle Types](bundle-types.md) for the open types each adapter emits. |