akm-cli 0.9.29 → 0.9.31

package/docs/architecture/specs/0.9.0-close-out-plan.md

@@ -0,0 +1,361 @@
+# akm 0.9.0 — milestone close-out plan
+
+**Status:** PLANNED (2026-07-30). Owner ruling applied: all four open issues
+ship in 0.9.0.
+
+The release sits at `0.9.0-rc.11` with the CLI overhaul (#734) merged and the
+[open-items register](0.9.0-open-items-register.md) recording its close-out
+state. This document is the implementation plan for the four issues remaining
+in the 0.9.0 milestone, in landing order.
+
+## Milestone triage
+
+All five open issues across 0.9.0 and 0.9.1 were reviewed. An initial
+recommendation deferred #733 and #730 to 0.9.1; the owner ruled both stay in
+0.9.0. Final state:
+
+| Issue | Milestone | Scope in one line |
+| --- | --- | --- |
+| #692 salience boost | 0.9.0 | Remove the R2 contributor from default ranking; the removal also deletes a confirmed 5 s hot-path stall. |
+| #672 repositories + guard | 0.9.0 | Move salience/outcome SQL behind `src/storage/repositories/`; add a raw-SQL lint rule. |
+| #733 orphan-GC | 0.9.0 | One maintenance pass, one additive migration, one event type, one config gate. Lean by design — see Workstream C. |
+| #730 OKF v0.2 | 0.9.0 | Read v0.2 frontmatter; stamp `generated`/`verified` on AKM-native writes via proposals; conformance updated. The `okf` adapter stays consumer-only. |
+| #652 write provenance | 0.9.1 | Unchanged — already re-scoped by the rc-5 audit to attribution/concurrency polish. |
+
+Sequencing note: A and B are small and touch the same salience surface — land
+them first (A deletes a raw-SQL site B would otherwise relocate). C and D are
+independent of each other and of A/B, and can proceed in parallel after.
+
+## Workstream A — #692: remove the salience boost from default ranking
+
+**Decision: fix option 1 (removal), the issue's stated preference.** The R2
+contributor multiplies search/curate scores by up to 1.2× from a signal that is
+retrieval-dominated (`w_r = 0.60`, `src/commands/improve/salience.ts:356-360`),
+warm-starts non-zero with no outcome evidence (`salience.ts:293-297`), has zero
+pack coverage, and measures as noise on live data (max ×1.071). No measured
+curate-bench delta justifies it, and `STABILITY.md:308` classifies ranking as a
+tuning target — removal in 0.9.0 is contractually clean.
+
+**No config gate is added.** A config key for a term being removed would be
+dead surface that the 1.0 freeze would then have to carry. If a future
+curate-golden-bench delta proves the signal out, reintroduction behind a
+default-off gate happens then (the contributor stays exported for exactly that
+experiment). `rank_score` itself is untouched — it remains the improve-internal
+selection signal (proactive/high-salience lanes, forgetting safety).
+
+The removal also deletes the hot-path defect confirmed in the rc-5 audit
+as a side effect rather than tuning around it: today every default search
+acquires the maintenance activity synchronously — up to a 5 s spin
+(`src/core/maintenance-barrier.ts:91-107`) plus a lock-file create — before the
+250 ms SQLite busy-timeout even applies (`src/indexer/search/ranking.ts:102`).
+
+### Changes
+
+1. `src/indexer/search/ranking-contributors.ts:551-554` — drop
+   `salienceRankingContributor` from `defaultUtilityRankingContributors`
+   (the list becomes `[utilityRankingContributor]`). The contributor and its
+   constants stay exported.
+2. `src/indexer/search/ranking.ts` — delete `loadSalienceRankScores`
+   (`:96-139`) and the `:263` default fallback. Its only production caller is
+   that fallback; tests are the only other importers. New semantics for
+   `RankEntriesOptions.salienceRankScores`: a `Map` applies the contributor
+   explicitly (tests/evals); `null`/`undefined` mean off. Update the option's
+   doc comment (`:58-64`) accordingly.
+3. With the loader gone, the search path holds no maintenance-barrier
+   acquisition and opens no state.db handle. `db-search.ts:502-511` needs no
+   edit beyond whatever the option-type change forces.
+4. Remove the now-dead barrier import from `ranking.ts:6` and confirm the
+   search layer has no remaining `maintenance-barrier` dependency.
+
+### Tests
+
+- `tests/integration/ranking-salience-boost.test.ts` — keep the contributor
+  math block (boost exactness at rank 1.0 / 0.5, no-op cases) exercised via
+  explicit injection. Replace the state.db read-path block: the held-barrier
+  test (`:111-131`, currently budgeted 10 s) inverts into the release's
+  regression guard — *search with the maintenance barrier held completes
+  without waiting*. Keep the "search never creates the state.db file"
+  assertion (`:101-109`) in its new form: default search never touches
+  state.db at all.
+- `tests/integration/ranking-contributor-ablation.test.ts:52-53` — update for
+  the one-entry default list; ablation of `"salience-ranking"` against an
+  explicitly-supplied list still proves the mechanism.
+- `tests/integration/curate-golden-eval.test.ts` and
+  `tests/integration/ranking-regression.test.ts` — expected neutral (live
+  effect measured as noise); run and record the deltas in the PR description.
+
+### Acceptance
+
+- Bare `akm search` under a held maintenance barrier returns promptly (no 5 s
+  stall) — covered by the inverted regression test.
+- Default search/curate perform zero state.db reads and create no lock files.
+- Golden evals within tolerance; CHANGELOG entry under Changed noting removal,
+  the retained improve-internal role of `rank_score`, and the latency fix.
+- Close #692 citing option 1 and the reintroduction path.
+
+## Workstream B — #672: salience/outcome repositories + widened SQL guard
+
+Part 1 of the issue (prep-stage accumulator decomposition) shipped at
+`028cc3de`. This lands the remaining part 2, with two corrections to the issue
+text discovered against the tree:
+
+- **Location correction.** The issue proposes `src/core/state/salience-repo.ts`
+  / `outcome-repo.ts`; the tree's actual convention is
+  `src/storage/repositories/*-repository.ts` (22 files, including the state.db
+  precedents `proposals-repository.ts`, `improve-runs-repository.ts`,
+  `events-repository.ts`, whose shared pattern is "extracted verbatim …
+  re-exported so existing importers resolve"). Follow the tree.
+- **Guard correction.** The issue calls the widening "a one-line allowlist
+  change", but `scripts/lint-repository-sql.ts` enforces *DB-owner imports and
+  direct opens* (`RULES` `:55-71`), not raw SQL. `salience.ts` and
+  `outcome-loop.ts` take `db` parameters and import no owners — they would
+  pass a widened prefix list today with their SQL intact, and commands
+  legitimately import owners like `withStateDb`, so prefix-widening alone
+  cannot express the intended boundary. The faithful implementation is a new
+  rule.
+
+### Changes
+
+1. `src/storage/repositories/salience-repository.ts` — verbatim moves from
+   `src/commands/improve/salience.ts`: `AssetSalienceRow` (`:380-393`),
+   `upsertAssetSalience` (`:425-458`), `getAssetSalience` (`:463-473`),
+   `getAllRankScores` (`:484-494`), `recordNoOp` (`:509-514`),
+   `resetConsecutiveNoOps` (`:520-525`), `getConsecutiveNoOps` (`:531-535`).
+   Add `getTopRetrievalSalience(db, limit)` absorbing the health read
+   (`src/commands/health/metrics.ts:246-251`).
+2. `src/storage/repositories/outcome-repository.ts` — verbatim moves from
+   `src/commands/improve/outcome-loop.ts`: the upsert SQL inside
+   `updateAssetOutcome` (`:245-268`; the domain function stays put and calls
+   the repository), `getAssetOutcome` (`:278-288`), `getAllAssetOutcomes`
+   (`:293-302`), `getOutcomeScoresByRef` (`:308-323`).
+3. Re-export the moved functions from `improve/salience.ts` and
+   `improve/outcome-loop.ts` per the established precedent, so the six improve
+   importers and eleven test files do not churn in this PR.
+4. `src/commands/health/metrics.ts` — replace the inline
+   `SELECT retrieval_salience …` with the repository call; the `withStateDb`
+   handle it already holds is passed through.
+5. The fifth raw-SQL site (`ranking.ts:122`) is deleted by Workstream A; if A
+   has not merged first, its chunked select moves behind
+   `salience-repository.ts` instead.
+6. `scripts/lint-repository-sql.ts` — add a third rule, `state-table-sql`:
+   flag raw SQL naming `asset_salience` or `asset_outcome` anywhere under
+   `src/` except `src/storage/repositories/` and
+   `src/core/state/migrations.ts`. Per-rule scoping (the existing two rules
+   keep their current `GUARDED_PREFIXES`; the new rule carries its own
+   scope/exclusions). Workstream C's pass consumes the repositories, so the
+   rule holds at zero from day one.
+
+### Tests
+
+- `tests/integration/lint-repository-sql.test.ts` — the live-tree-is-zero test
+  now also proves no raw salience/outcome SQL survives outside repositories.
+  Add fixtures: raw `asset_salience` SQL under `src/commands/improve/` fires;
+  the identical string inside `src/storage/repositories/` does not; prose
+  mentions still do not (comment-stripping already covered). The `:48-53`
+  assertion that `src/commands/improve/preparation.ts` is unguarded is updated
+  to reflect the new rule's coverage.
+- The improve-side suites
+  (`tests/integration/commands/improve/{salience,salience-wiring,outcome-loop,outcome-loop-wiring,outcome-invariance}.test.ts`,
+  `tests/integration/health-ws5-observability.test.ts`) must pass unchanged —
+  verbatim moves, pure refactor, no behavior delta.
+
+### Acceptance
+
+- `bun run lint` green with the new rule at zero violations.
+- No test-behavior changes outside the lint meta-test.
+- Close #672 with a comment recording the two corrections above.
+
+## Workstream C — #733: orphan-GC pass (lean)
+
+Deliberately minimal: **one maintenance pass, one additive migration, one
+event type, one config gate.** No quarantine archive, no circuit breaker, no
+health-advisory plumbing, no new tables. The issue's design constraints are
+each satisfied by something that already exists or by the smallest possible
+addition:
+
+- **Absent ≠ deleted** is inherited from the indexer, not re-implemented: an
+  incomplete or failed scan preserves a source's last-known-good `entries`
+  rows (`src/indexer/indexer.ts:1195-1199`), and mass-wipe is already blocked
+  upstream (`preserveExistingIndex` + the `fullDelete && scanComplete` gate).
+  So "ref not present in `entries.item_ref`" *is* the authoritative-deletion
+  predicate — a temporarily unreachable source never surfaces candidates.
+- **Grace window** needs a clock rows don't have (`updated_at` is bumped by
+  improve itself and `mtime` doesn't exist for rows), so: one additive
+  migration (`021`, mirroring the 6-line migration-011 precedent) adds
+  `missing_since INTEGER DEFAULT NULL` to `asset_salience` and
+  `asset_outcome`. Grace duration is a named constant
+  (`STATE_GC_GRACE_MS`, 7 days), mirroring `TXN_SWEEP_GRACE_MS`
+  (`src/core/fs-txn.ts:298`) — not a config knob.
+- **Prove before deleting**: the pass always runs and always reports; actual
+  row deletion is behind a single boolean, `improve.stateGc.collect`
+  (default **false** for 0.9.0). Every run emits the counts either way, so
+  live data accumulates proof; flipping the default later is a one-line
+  change plus a CHANGELOG note.
+- **Events**: one new documented `EventType` literal, `asset_state_gc`, with
+  the sentinel ref `asset_state/_gc` (same convention as
+  `proposals/_orphan-purge`), emitted only when there is something to say
+  (the rekey script's no-op-stays-silent precedent).
+
+**Scope cut: `usage_events` is excluded.** Attached rows (`entry_id` set)
+already cascade on entry deletion (`deleteUsageEventsByEntryIds`,
+`src/storage/repositories/index-entries-repository.ts:607-618`), and the
+90-day retention purge (`purgeOldUsageEvents`,
+`src/indexer/usage/usage-events.ts:141`) bounds detached remnants. If live
+data shows that purge isn't reaching them, extending the pass is a follow-up.
+
+### Changes
+
+1. Migration `021-asset-state-missing-since` in
+   `src/core/state/migrations.ts` — the two `ALTER TABLE … ADD COLUMN`
+   statements, nothing else.
+2. `runOrphanStateGcPass(ctx)` in `src/commands/improve/loop-stages.ts`,
+   registered in the maintenance sequence at `:1005-1036` next to
+   `runOrphanProposalPurgePass` (`:1303`), which it copies structurally
+   (same `{count, warnings}` return, same borrowed-connection handling via
+   `withStateDb(…, { borrowed })` — the #585 pattern at `:1400`). Per run:
+   a. Collect distinct refs from both tables (repository calls — Workstream
+      B's modules), resolve them against `entries.item_ref` with the same
+      chunked-probe shape the rekey script uses
+      (`scripts/rekey-asset-ref.ts:278`), reusing the existing
+      legacy-spelling normalization from `preparation.ts:2153-2163` so a
+      live asset keyed under an old spelling is never a candidate.
+   b. Unresolved and `missing_since IS NULL` → stamp now. Resolved and
+      `missing_since` set → clear it (the asset came back).
+   c. If `improve.stateGc.collect` is true: delete rows whose
+      `missing_since` is older than the grace constant, per-table, in one
+      transaction each.
+   d. Emit `asset_state_gc` with `{pending, collected, byTable}` when either
+      is nonzero; surface the same counts in the pass summary line like the
+      neighboring purge passes do.
+3. Config: `stateGc: { collect: boolean }` added to
+   `ImproveConfigSchema` (`src/core/config/schema/improve.ts`), default
+   false; JSON schema regenerated via `scripts/gen-config-schema.ts`.
+
+### Tests
+
+One integration suite, `tests/integration/commands/improve/state-gc.test.ts`:
+
+- Orphaned ref is stamped, not deleted, on first sighting; deleted only after
+  the grace elapses *and* `collect` is on (fake clock via the stamped value).
+- Default config: nothing is ever deleted; counts still reported.
+- A live asset whose state row carries a legacy ref spelling is not a
+  candidate (the normalization case).
+- A ref that resolves again after being stamped has `missing_since` cleared.
+- The event appears in `akm log` with the expected metadata; a run with
+  nothing pending emits no event.
+- Rekey-race guard: rows stamped but inside grace are untouched, and
+  `rekey-asset-ref.ts` still moves them intact.
+
+### Acceptance
+
+- Field-failure class reproduced in a fixture (state rows under a retired
+  source root) is reported immediately and collected only with
+  `collect: true` after grace.
+- No behavior change for any source that fails to scan (entries preserved →
+  zero candidates).
+- CHANGELOG entry under Added; #733 closes with a note that the
+  `collect` default flips after live runs prove the report clean.
+
+## Workstream D — #730: OKF v0.2
+
+Three parts: read-side adapter support, write-side provenance stamping through
+the proposal path, and the conformance/spec updates. One scope decision up
+front: **the `okf` adapter stays consumer-only.** AKM markdown is already an
+OKF-compatible superset (`okf-support.md` progressive-enhancement contract),
+so the new provenance fields are *written* only to AKM-native assets;
+runbook §10's write-rejection contract and the tests at
+`okf-conformance.test.ts:832-936` are unchanged.
+
+### D1 — read side (`src/core/adapter/adapters/okf-adapter.ts`)
+
+Today `recognize` (`:125-171`) reads exactly five keys (`type`, `title`,
+`description`, `tags`, `timestamp`); everything else folds into opaque
+`documentJson`. Add, staying lenient (never reject):
+
+1. `updated` ← `generated.at`, falling back to legacy `timestamp` (the v0.2
+   breaking change, with the fallback v0.2 itself allows).
+2. Parse the v0.2 families into new optional `IndexDocument` fields
+   (`src/core/adapter/types.ts:141-296`): `generated` (`by`/`at`), `verified`
+   (list; tolerate the single-mapping form), v0.2 `sources` (object list),
+   `status`, `stale_after`, and `okf_version` (currently read nowhere in
+   `src/`).
+3. **Collision handling** — three AKM fields already occupy adjacent names:
+   `sources?: string[]` (wiki citations, silently drops non-strings at
+   `metadata.ts:486-489`), `generation?: number` (consolidation depth), and
+   `quality: "generated"`. The v0.2 families therefore land under *new,
+   distinct* field names (recommendation: a single namespaced
+   `provenance?: { sources, generatedBy, generatedAt, verified }` plus flat
+   `lifecycleStatus` / `staleAfter`), leaving the existing fields untouched.
+   Final naming is an owner call at PR review; the plan only fixes that no
+   existing field is overloaded.
+
+### D2 — write side (AKM-native assets via proposals)
+
+The proposals system already tracks exactly what v0.2 wants on disk —
+`source`/`sourceRun` (PROV-DM modeled, `proposal-types.ts:82-120`),
+`gateDecision`, `review` — it just never leaves state.db. Project it at
+promotion time:
+
+1. `promoteProposal` stamps `generated: { by, at }` into the written asset's
+   frontmatter, actor per the v0.2 convention (`<producer>/<version>` →
+   `akm/<pkg-version>` for automated lanes; `human:<id>` when a human
+   accepted the proposal).
+2. A proposal that passed its gate or review appends a
+   `verified: [{ by, at }]` entry alongside.
+3. Where a proposal carries `evidenceSources` (`types.ts:265-268`), project
+   them as v0.2 `sources` entries (`{ resource }` minimum).
+4. `DOCUMENT_JSON_CARRIED_FIELDS` (`akm-adapter.ts:192-213`) gains the new
+   keys so AKM's own adapter rereads what it wrote; `akm show` surfaces them.
+
+Deeper consumers — `stale_after`-driven re-verification, trust-tier ranking —
+are explicitly *not* 0.9.0 scope; they are the 0.9.x improve-tuning track.
+
+### D3 — conformance and spec docs
+
+1. Rewrite `docs/architecture/testing/okf-v0.1-conformance-runbook.md` for
+   v0.2 and bump the pinned `KC_REF` (`:57`), inspecting the upstream
+   `okf/SPEC.md` diff first as the runbook itself instructs (`:61-62`).
+2. Update the §0.1 OKF↔AKM field-mapping table in
+   `akm-0.9.0-bundle-adapter-spec.md` (`:22-35`) and the frozen-reference
+   paragraph (`:480`); add a v0.2 note to `okf-support.md`.
+3. Add the missing `tests/fixtures/format-family-goldens/okf/` golden — OKF
+   is the only registered adapter without one.
+
+### Tests
+
+- `tests/core/adapter/okf-adapter.test.ts` — update the `timestamp`
+  assertions (`:142`, `:291`, `:300`) to cover `generated.at` precedence and
+  legacy fallback; add parsing cases for each new family, including the
+  `verified` single-mapping form and non-string v0.2 `sources` surviving
+  intact.
+- `tests/integration/okf-conformance.test.ts` — v0.2 fixture bundle alongside
+  the v0.1 one; both must index; write-rejection block unchanged.
+- Promotion-path test: an accepted automated proposal produces frontmatter
+  with `generated` (and `verified` when gated), round-tripping through
+  `asset-serialize.ts` and re-indexing losslessly.
+
+### Acceptance
+
+- v0.1 and v0.2 bundles both index correctly (v0.1 fixtures byte-identical in
+  behavior to today).
+- AKM-authored assets written through promotion carry `generated`, plus
+  `verified` when a gate/review confirmed them.
+- Conformance runbook passes end to end at the new `KC_REF`; consumer-only
+  status intact.
+- CHANGELOG entries under Added (v0.2 read support; provenance stamping);
+  #730 closes.
+
+## Sequencing and release
+
+1. **PR 1 — Workstream A** (behavior change; eval evidence + CHANGELOG).
+2. **PR 2 — Workstream B** (pure refactor + guard rule; rebases trivially).
+3. **PR 3 — Workstream C** and **PR 4 — Workstream D** in parallel (disjoint
+   surfaces: state.db/maintenance vs adapter/proposal serialization). D is the
+   largest; its one open naming decision (D1.3) should be settled in review,
+   not up front.
+4. Cut the next rc with all four merged, soak, ship **0.9.0**.
+5. On release: revisit milestone due dates (0.9.0's 2026-07-06 has passed;
+   0.9.1 — now carrying only #652 — is dated 2026-08-08), and schedule the
+   two deliberate follow-ups: flipping `improve.stateGc.collect` on once live
+   reports prove clean, and the deeper v0.2 consumers on the improve-tuning
+   track.