okf 2.0.0 → 2.1.0

data/lib/okf/skill/reference/cli/registry.md

@@ -0,0 +1,70 @@
+# `registry` — naming bundles once
+
+Kind: reference. Answers: which file a registry op writes and how okf finds it,
+which verbs key on a path and which on a slug, what a group is and which two
+verbs consume one, and why the default is a position rather than a stored slug.
+
+The *persistent registry* is a plain JSON file under `$OKF_HOME` (default
+`~/.okf`), managed by the `okf registry` umbrella — like git's `remote` family,
+and split by what each verb keys on. It is what every `@slug` resolves through
+([cli.md](../cli.md)) and what a bundle-less `okf server` hosts
+([serve.md](serve.md)).
+
+**`okf registry init`** creates a *project-local* registry instead: a
+`.okf-registry.json` in the current directory, which okf discovers by walking up
+from the working directory and uses in place of the global one while you are
+inside its tree (the nearest wins, so nested registries resolve nearest-first).
+Every registry op — and every `@slug` — then resolves through it, so a bare
+`okf server` inside a repo serves that repo's bundles with no `$OKF_HOME` setup;
+`okf registry list` names the local file it found. `OKF_NO_DISCOVERY=1` forces
+the global registry — the escape hatch for a fixed-cwd caller (CI, a tool). A
+local registry stores **portable** paths: a bundle inside its tree is written
+relative to the `.okf-registry.json`, so committing the file lets it travel with
+the repo (a checkout elsewhere, a container mounting it) and resolve unchanged;
+a bundle outside the tree stays absolute, since it cannot travel. Paths still read
+back absolute wherever the CLI reports them.
+**Entry verbs** take a path: `okf registry set <dir>` adds it
+(slug from the basename, or `--as`, which errors on a collision; `--default`
+puts it first), and because the entry is keyed by path, `set` on an
+already-registered dir updates it in place — refreshing its title, and renaming
+it when `--as` is given. `okf registry del <dir|@slug>` removes a bundle *or* a group — by name, so an
+entry whose directory is already gone still deletes, and removing a bundle
+**cascade-drops** it from every group that named it (a group emptied that way is
+deleted). Slug *or* dir, never both readings at
+once: an argument with a `/` in it names a location and only a location, so
+`del ./notes` refuses when no entry points there rather than stripping to the
+slug `notes` and deleting a bundle somewhere else entirely.
+<!-- rule:okf-registry-del-path-or-slug -->
+**Slug verbs** take the name — bare, or as an `@slug`: `okf registry default <@slug>`
+chooses which bundle `/` opens **by moving that entry to the front** (a group is
+refused — the default is one bundle), and
+`okf registry rename <@slug> <new>` renames a bundle *or* group slug (mount path
+and switcher name), **cascading** the new name into every group's member list —
+`<new>` is a name being minted, so it is never a ref. The registry is ordered and **the first entry still on disk is the
+default** — that is the whole rule, so the first bundle you register is the
+default until you move another one, a rename keeps its position, and a `del`
+promotes whatever is next. A vanished directory is stepped over (the server
+cannot open one, so starring it would name a bundle `/` never serves), and
+`registry default @slug` refuses one outright — the same refusal `registry set`
+gives a directory that is not there. The file is hand-editable and reorders
+visibly, which is the point: there is no stored slug that can dangle.
+<!-- rule:okf-registry-default-position -->
+**Group verbs** name a *set* of bundles under one slug — a durable subset for the
+two verbs that take several bundles. `okf registry group <slug> <@member…>`
+creates a group, or adds members to one (a union); members are bundle *or* group
+slugs, so groups nest. A group shares the slug namespace with bundles (a slug
+names one *or* the other, never both), `all` stays reserved, and a member set that
+would make the group reach itself is refused. `okf registry ungroup <slug>
+<@member…>` removes members; emptying a group deletes it. A group resolves,
+recursively and path-deduped, to its bundle leaves — which **only `okf search`
+and `okf server` consume**: every single-bundle verb (`lint`, `index`, …) refuses
+a `@group` with exit 2, the same rule that refuses a second bundle. `@all` is
+unchanged — it still names every registered *bundle*, groups being named subsets
+of that.
+`okf registry list` (or a bare
+`okf registry`) stars the default and flags vanished dirs `(missing)` — the
+server skips those with a note — and lists any groups with their members and
+resolved leaf count; `--json` answers
+`{ registry: <file>, count, bundles: [{ slug, title, dir, mount, default,
+missing }], groups: [{ slug, members, resolved }] }`, naming the file it read so a
+`$OKF_HOME` mismatch is visible.

data/lib/okf/skill/reference/cli/search.md

@@ -0,0 +1,130 @@
+# `search` — ranked text retrieval (metadata + body)
+
+Kind: reference. Answers: how terms are matched and by which engine, what the
+index buys and what it silently loses, how one query spans several bundles, and
+which of the two JSON envelopes comes back.
+
+The shared contract — `@slug` refs, exit codes, `--json`, the filters this verb
+also takes — is in [cli.md](../cli.md) and is not repeated here.
+
+The browser page's search brought to the CLI and extended to bodies, so "which
+concept covers X?" costs rows, not body reads. `okf search <dir> <term…>`:
+terms AND together — every term must hit at least one searched field, not
+necessarily the same one — matched **literally against raw text** by default, or
+as Ruby regular expressions with `--regexp`/`-e` (an invalid pattern is a usage
+error, exit 2). `--fuzzy` forgives typos; pairing it with `-e` is a usage error,
+since a pattern is matched literally rather than by edit distance.
+`--in a,b` restricts the searched fields (title, id, tags, type, description,
+sources, body — `sources` is each entry's title and resource joined, so a
+migrated bundle keeps the recall its `# Citations` body text used to give it);
+the shared `--type/--dir/--tag/--status/--trust` filters narrow the candidates
+*first*, so a search scoped by what `index` taught you stays surgical.
+
+**The default is exact, so an exact query means what it looks like.** A phrase in
+one argument (`"dedup key"`), a dotted version (`7.2.0`), an underscored
+identifier (`customer_id`), a mid-word fragment (`ustomer`) and a word written in
+`backticks` all match literally. This is what the scan engine buys, and it is the
+default precisely because those queries are the common ones and the alternative
+loses them silently. <!-- rule:okf-search-exact-identifiers -->
+
+**`--engine index` is the other engine, and the one to reach for when ranking
+matters more than exactness.** The engine is normally chosen by what the query
+needs — `--fuzzy` routes to the index, anything else stays on the default scan —
+and nothing is printed about the choice. `--engine NAME` overrides that for the
+case the flags cannot express: a matching *model* requires no capability, so no
+flag selects one. Under the index, terms match whole tokens and their prefixes
+(`dedup` finds `deduplication`), rows rank by BM25+, and it is the engine the
+browser page runs — so name it when reconciling a CLI answer with the page. The
+cost is real: its tokenizer splits on punctuation, so identifiers shatter
+(`customer_id` → `customer` + `id`), an infix finds nothing, and a backtick is
+never split off at all, so a word inside a code span is unfindable — a large
+silent loss, since technical prose is full of them. **Do not count on ranking to
+rescue it** — BM25 normalizes by field length, so a short concept dense in `7`,
+`2` and `0` can outrank the one that actually says `7.2.0`. Naming an engine that
+cannot do what you also asked (`--engine index -e`) is a usage error naming one
+that can. <!-- rule:okf-search-engine-choice -->
+
+**The capabilities, and which engine has them.** An engine is selected by what
+the query *requires*; only a matching model has to be named, because requiring
+nothing is not something a flag can express:
+
+| Flag | Capability | Engine | What it does |
+|---|---|---|---|
+| *(none)* | — | scan | literal substring over raw text; scores by summed field weight |
+| `-e` / `--regexp` | `regexp` | scan | each term is a Ruby regexp, case-insensitive; invalid → exit 2 |
+| `--fuzzy` | `fuzzy` | **index** | edit distance 0.2 × term length — and switches engine |
+| `--engine index` | — | index | whole-token + prefix matching, BM25+ ranking, browser parity |
+| `--engine scan` | — | scan | the default, spelled out |
+
+Two consequences worth holding. **`--fuzzy` is an engine switch, not a mode**: it
+carries the whole index with it, so a run that wanted one typo forgiven also gets
+shattered identifiers and unfindable code spans — fix the spelling and stay on
+the default when you can. And **`-e` moves nothing** now, because the default
+engine already offers `regexp`; it changes how a term is *read* (pattern rather
+than literal), not where it is matched. <!-- rule:okf-search-fuzzy-is-a-switch -->
+
+`prefix` is a capability the index declares but no flag selects — it is always on
+there. **It is not a reason to reach for the index**: a substring match already
+covers every prefix and then some, so `dedup` finds `deduplication` under both
+engines, while `duplication` and `uplicat` find it under the default only. Prefix
+is what the index needs to catch up to raw text, not a capability it adds on top.
+The index's real advantages over the default are exactly three — relevance
+ranking, typo tolerance, and page parity.
+
+**Search spans bundles.** Leading @refs pick several registered bundles
+(`okf search @handbook @notes auth`); **`@all`** is the ref that means every one.
+Rows from different bundles are ranked together and comparable, and each row
+carries its bundle's slug. Under `--engine index` the bundles go into **one
+corpus** — BM25 prices a term by how rare it is, so separately-ranked lists would
+not compare — which makes a score relative to the whole answer: the same concept
+scores lower searched beside others than searched alone. The default scan needs
+no such trick — its score is absolute, so a row is worth the same either way.
+This is the
+cross-bundle retrieval the in-page search does not have: one question, every
+bundle you keep. <!-- rule:okf-search-all -->
+
+`@all` is a ref, not a flag, which is what keeps the grammar single: slot 1 is
+always a bundle identity, so a directory there is a directory and nothing can
+flip it into a term. Being a ref, it is normalized like one — `@ALL` and `@All`
+name every bundle just as `@One` names the bundle registered from dir `One`. It composes accordingly — `@all @docs` expands and dedupes
+(all ⊇ docs), needing no diagnostic. **Asking for everything tolerates gaps;
+naming one bundle demands it**: `@all` skips a bundle whose directory has
+vanished with a note on stderr, while `@docs` fails hard. `@all` is only
+`search`'s: every other verb answers about one bundle, so it refuses `@all` by
+name rather than letting the answer depend on how many bundles you happen to
+have registered. `all` is reserved as a slug — a directory named `all/` registers
+as `all-2`, `--as all` is refused, and an `all` row already in the registry file
+(hand-typed, or written before the name was reserved) is read as `all-2` rather
+than taken as grounds to reject the file — so `@all` is never ambiguous, and the
+reservation never strands a registry it inherited. **The read normalizes every
+slug** the same way registration would, so a hand-typed `"slug": "My Docs"` lists
+and resolves as `my-docs`; an entry the listing shows is always an entry `@slug`,
+`rename`, and `default` can name.
+
+`--fields` projects the shape the mode actually emits: `slug` is available in
+registry mode, and a usage error naming the real fields on a path-named search,
+which has no slug to give. Two sharp edges: every *leading* @-arg is taken as a ref, so a literal @-term
+(`@babel/core`, a Ruby `@ivar`) needs a non-@ term before it or `-e '\@term'` —
+the CLI notes both traps on stderr — and any ref, even one, switches the JSON
+envelope (next paragraph).
+
+Rows rank by where they hit — title 5, id 4, tags 3, type/description 2, body 1 —
+summed as an absolute score by the default scan, and carried as per-field boost
+into **BM25+** under `--engine index`. Each row carries one bounded context
+snippet from the strongest match that needs context (description or body). Every row still names the fields that hit (`matched`), so a result stays
+citable rather than being a bare relevance number. Exact by default: the
+consuming agent is the fuzzy layer — when terms miss, learn the bundle's
+vocabulary from `tags`/`types` and re-ask in its own words, rather than
+hammering synonyms or reaching for `--fuzzy` before you have looked. Advisory read: **exit 0 even with zero matches**.
+JSON, plain-dir mode: `{ bundle, query, count, matches: [{ id, title, type,
+dir, top_dir, tags, matched, score, snippet }] }`. Registry mode — any leading @ref,
+`@all` or a `@group` among them (a group fans out to its member bundles) — swaps the envelope: `{ bundles: [{ slug, dir }, …],
+query, count, matches: [{ slug, id, … }] }`; a parser must branch on which form
+it called. The head maps each slug to its dir once, so a row resolves to
+`<dir>/<id>.md` without a second lookup and without repeating a path per row.
+Both are projectable with `--fields/--except`, and projection is literal — when
+merging bundles, put `slug` in your `--fields` list or the row label drops and
+same-id concepts from different bundles become indistinguishable. The retrieval procedure that puts this verb in sequence —
+map first, finder second, bodies last — is the
+[search playbook](../../playbooks/search.md).
+

data/lib/okf/skill/reference/cli/serve.md

@@ -0,0 +1,83 @@
+# Serving the graph — `server` and `render`
+
+Kind: reference. Answers: what the interactive page shows and what it fetches
+live, how several bundles mount behind one hub, what the static export trades
+away, and the trust boundary both modes share.
+
+The shared contract — `@slug` refs, exit codes, `--json` — is in
+[cli.md](../cli.md) and is not repeated here.
+
+## server — interactive graph server
+
+Starts a local HTTP server (`okf server <dir>`; `-p`/`--port`, default 8808, and
+`--bind`) and prints its URL — stop it with Ctrl-C. The page boots from a lean
+payload (nodes carry only `id` and `title`, plus compact type/tag indexes) and
+fetches each concept's markdown body **live from disk** as you click it, so the
+initial load stays small and edits show without a restart. Mermaid code blocks
+in a body render as diagrams, and a click (or tap) opens the diagram full
+screen with drag-to-pan and wheel/pinch zoom. Concepts render as nodes
+coloured by `type` and sized by degree, links as edges, with a detail panel
+(rendered markdown, "Links to" / "Linked from" backlinks), layout switching,
+type/dir/tag filters on every view (the dir chips take a directory *and* its
+subtree, the same rule `--dir` uses), and search. Cluster mode groups the
+concepts into one box per directory, nested to a depth picked beside the layout
+select — depth 1 is the flat view, and a flat bundle is offered no control. The authored layer is in the
+UI too: the Files view carries **Files | Indexes** tabs — the Indexes tab
+lists the log first (the chronological index), then every `index.md` — and
+folder nodes in file-tree mode and directory boxes in cluster mode open a
+directory's §8 map in the inspector (authored, or synthesized when none
+exists). Links to an `index.md`, `log.md`, or bare directory navigate instead
+of dead-ending, and the log is fetched fresh on every read, so a
+just-appended entry shows without a restart. `?view=index` jumps straight to
+the Indexes tab. It is a Rack app, so the same server can be mounted in a
+host app (e.g. Rails).
+
+**Hosting many bundles (the hub).** `okf server` takes zero or more dirs.
+One dir is the classic single bundle at `/`. Two or more mounts each under
+`/b/<slug>/` behind a hub, `/` redirects to the default, and `/b/` is a
+self-contained **bundle index** (every hosted bundle, concept counts, default
+marked — the browser counterpart of `okf registry`). An unknown slug 404s as a
+page listing the hosted bundles, so a stale bookmark after a rename gets a way
+home. With **no** dir it serves the *persistent registry*. The hub roster is a
+**boot-time snapshot**: restart `okf server` after registry changes. Behind a
+hub the page gains a **bundle switcher** (⌘/Ctrl-K, or the rail button with its
+bundle-count badge): the current bundle is pinned, the default chipped; ⏎
+opens, ⌘/Ctrl-⏎ opens a new tab, and the current view carries over. Switching
+is a server-only affordance — a static `render` file has no siblings and shows
+none.
+
+**Bundle-less run.** Register bundles once, then `okf server` (no dir) hosts
+them all with the registry's first entry still on disk at `/` — the way to keep
+several bundles a keystroke apart without re-passing paths.
+`okf server @a @b` serves a registry subset, each mounted under its registered
+slug — but as with any dirs-given run, the *first argument* lands at `/`; the
+registry's own order applies only to the bundle-less run. A `@group` argument
+fans out to its member bundles in the same way (`okf server @backend`), its first
+member landing at `/`; `okf search @group <term…>` merges the group's members
+into one ranking, exactly as naming them individually would.
+The registry itself — the file it reads, how okf finds it, and the verbs that
+write it — is in [registry.md](registry.md).
+
+**Trust boundary:** the page renders each fetched markdown body through
+DOMPurify and escapes everything it inlines (every `<` in the graph data is
+escaped, so it cannot break out of its `<script>`), but it still loads its
+viewer libraries (Cytoscape, marked, DOMPurify — plus Mermaid and Panzoom,
+lazy-loaded on first use) from a CDN and renders whatever
+links the bundle carries — so only serve bundles you trust.
+## render — static graph export
+
+Writes the same interactive page as one static, self-contained HTML file
+(`okf render <dir>`), so the graph hosts where there is no server — GitHub Pages,
+an object store, an attachment. Prints to stdout (`okf render <dir> > graph.html`)
+or writes `-o FILE`; `--title`/`--link`/`--layout` mirror `server`. It is the same
+template `server` renders, one switch apart: rather than fetching each body,
+description, catalog, index, and log live, `render` bakes the whole bundle into
+the page and the browser reads from that embedded payload — no server, no build
+step. The trade-off is weight (every body is inlined), so `server` stays the
+choice for a bundle too large to ship whole.
+
+**Trust boundary:** the same two guards as `server` — every inlined body is
+`</script>`-escaped like the graph data and still sanitized by DOMPurify when
+rendered — so a static file is no laxer than the live server. Only render bundles
+you trust.
+

data/lib/okf/skill/reference/cli/views.md

@@ -0,0 +1,59 @@
+# The server views as text — `catalog`, `files`, `tags`, `types`, `stats`
+
+Kind: reference. Answers: which of the five reads a question wants, what each
+row carries, and the JSON each emits.
+
+The filters all five narrow with — `--type`, `--dir`, `--tag`, `--status`,
+`--trust` — are in [cli.md](../cli.md) with the rest of the shared contract,
+since `search` takes them too.
+
+The browser server ([serve.md](serve.md)) has Catalog, Files, Tags and Stats
+panels; these
+verbs reproduce them on the CLI so an agent can read a bundle without a browser.
+All are advisory reads (exit 0) sharing one data source (per-concept metadata plus
+in/out link degree). Add `--json` to any for a machine substrate.
+
+- **`catalog`** — every concept with its metadata (type, status, trust, tags,
+  provenance, in/out link degree, description), grouped by top-level dir (`dir`
+  on every row carries the full path, `top_dir` the first segment). The "what's
+  here, in detail" view. JSON: `{ bundle, count, concepts: [{ id, title, type,
+  description, tags, generated_at, generated_by, generated, trust, status,
+  stale_after, sources, backlog_ref, dir, top_dir, links_out, links_in }] }`.
+  Four of those deserve a sentence: `generated` is the raw boolean ("does the
+  document *declare* a generated mapping"), which is what tells hand-written
+  apart from v0.1-with-timestamp — `generated_at` alone conflates them, because
+  §13.1 lifts a legacy `timestamp` into it. `trust` is the derived §5.3 tier as
+  a hyphenated literal (`unverified` | `machine-confirmed` | `human-reviewed`) —
+  compare against exactly those. `status` is the *declared* value, `null` when
+  absent (the row never fabricates frontmatter; the `--status` filter is what
+  applies the §5.4 default). `sources` is a count. Temporal fields render
+  ISO 8601 (`stale_after` as `YYYY-MM-DD`). The `timestamp` column is retired —
+  `--fields timestamp` is a usage error naming the valid fields.
+- **`files`** — the folder tree: each concept's filename + title, grouped by
+  directory. The "how it's organised" view. JSON: `{ bundle, count, files: [{ path,
+  id, dir, type, title, description }] }`.
+- **`tags`** — every tag with the concepts that carry it, ordered by count
+  descending. The "what themes dominate" view. JSON: `{ bundle, count, tags: [{ tag,
+  count, concepts: [id, …] }] }`. `--by type|dir` regroups the list per concept
+  dimension with **within-group** counts (a tag spanning groups appears in each);
+  each row also carries the tag's **total** across the narrowed set, printed
+  `count/total` when they differ — so a tag's locality reads per row (a plain
+  count = wholly local; `2/7` = a cross-cutting spread). The substrate for tag
+  curation and for [refine](../../playbooks/refine.md)'s domain-vs-concern read;
+  the judgment recipes live in the [maintain playbook](../../playbooks/maintain.md)
+  and the [refine playbook](../../playbooks/refine.md). JSON: `{ bundle, count, by,
+  groups: [{ <dim>, count, tags: [{ tag, count, total, concepts }] }] }`.
+- **`types`** — every type with the concepts that carry it, ordered by count
+  descending. The "what kinds of knowledge" view. JSON: `{ bundle, count, types:
+  [{ type, count, concepts: [id, …] }] }`.
+- **`stats`** — bundle rollups: concept / dir / type / cross-link / distinct-tag
+  totals plus per-type and per-dir breakdowns. The "shape at a glance" view. JSON:
+  `{ bundle, concepts, dirs, top_dirs, concept_types, cross_links, distinct_tags,
+  by_type, by_dir, by_top_dir }` (`top_dirs`/`by_top_dir` are the first-segment
+  rollup). `dirs`/`by_dir` cover every directory `okf dirs`
+  lists — counts are direct, so a directory holding nothing itself is present at
+  `0` rather than missing, and `by_dir.keys` is a complete list of what `--dir`
+  can address.
+
+Reach for `stats` first to size a bundle, `catalog`/`files` to enumerate it, `tags`
+to find thematic clusters — all without standing up the server.