okf 2.1.0 → 2.2.0

data/.okf/design/search-engines.md

@@ -0,0 +1,154 @@
+---
+type: Constraint
+title: Search engines are adapters, and the facade owns the row
+description: One facade over N retrieval engines — the scan by default, the index when a query needs it or names it — with a shared conformance suite standing in for the oracle rule that multiple engines made impossible.
+resource: gems/okf/lib/okf/bundle/search.rb
+tags: [architecture, search, extensibility, testing]
+generated:
+  by: human:maintainer
+  at: 2026-07-24T12:00:00Z
+sources:
+  - title: gems/okf/lib/okf/bundle/search.rb
+    resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/bundle/search.rb
+  - title: gems/okf/test/unit/bundle/search/engine_conformance.rb
+    resource: https://github.com/serradura/okf/blob/main/gems/okf/test/unit/bundle/search/engine_conformance.rb
+  - title: gems/okf/test/unit/bundle/search/accepted_losses_test.rb
+    resource: https://github.com/serradura/okf/blob/main/gems/okf/test/unit/bundle/search/accepted_losses_test.rb
+---
+
+# Overview
+
+[`OKF::Bundle::Search`](../capabilities/search.md) is a **facade**, not an
+implementation. It owns everything that defines what a result *is*, and delegates
+only the retrieval question to an engine:
+
+| The facade owns | An engine answers |
+|---|---|
+| entry points (`.call`, `.across`) | which documents match |
+| document assembly and the `"<slug>\0<id>"` key | how well each matches |
+| the row hash and its key order | which fields the terms hit |
+| the snippet window, `top_dir`, the final sort | which matcher to point the snippet at |
+
+The split exists because an engine that built its own rows could disagree about
+what a match *means* — two engines, two answer shapes, and a merged ranking that
+looks sorted and compares nothing. Putting the row in the facade makes that
+unrepresentable rather than merely discouraged.
+
+# Selection is by capability, then by name
+
+An engine declares three things — `id`, `capabilities`, `available?` — and the
+router picks the first available engine offering **every** capability the query
+requires, default first, then registration order:
+
+```
+--fuzzy       requires :fuzzy   →  Search::Index
+-e            requires :regexp  →  the default already offers it
+neither                         →  the default (:scan)
+nothing qualifies               →  UnsupportedQuery, which the CLI makes exit 2
+```
+
+Capability routing came first and was, for one release, the *only* selector —
+"the flags a user already reaches for are the selector" — on the reasoning that a
+second vocabulary would have to be kept consistent with the first. That was
+wrong in one specific way, and the gap is worth recording: **a capability flag can
+only express what a query needs, not which matching model answers it.** Raw-text
+matching requires nothing; BM25 ranking requires nothing. Neither can be asked for
+by requiring something, so under capability routing alone the non-default engine
+was unreachable at any price.
+
+`--engine NAME` is the override that closes it, and naming an engine is an
+override rather than a hint: an engine that cannot do what was *also* asked is an
+error (`UnsupportedQuery` naming the engine), never a silent fallback, since
+falling back would answer a different question than the one posed. An unknown
+name lists what is registered.
+
+# The default is the scan
+
+The default engine is `Search::Scan`, and the reason is that a CLI process builds
+an index, asks one question and exits — a build with a single query to amortize
+it over. End to end: 3.00 s against 0.24 s at 1,000 concepts, 0.83 s against
+0.18 s at 250. The build is ~95% of the index path's cost at every size.
+
+Recall settles it. Raw-text matching has no tokenizer, so it has none of the
+tokenizer's holes: MiniSearch splits on `\p{Z}\p{P}`, and a backtick is `Sk`
+while `$` is `Sc`, so a word inside a code span is stored as the token
+`` `minifts` `` and the query `minifts` does not match it. On this bundle that
+was 2 hits where the scan found 5. A loss that returns plausible rows while
+omitting most of the answer is the worst kind to have on a path nobody opted
+into.
+
+What the index gives back — BM25+ ranking, prefix matching, `--fuzzy`, and
+parity with the browser page — is now reached by asking. That the *page* still
+runs MiniSearch means the CLI and the page rank identically only under
+`--engine index`; the claim used to be unconditional and is not any more.
+
+This has now been revisited on the server side. An engine may expose `prepare`,
+which builds whatever it would otherwise build per call and hands it back; a
+`Search::Corpus` holds one per engine id and passes it as `prepared:`. The scan
+declares no `prepare` and is handed none — the seam is opt-in, so adding it broke
+no engine and required nothing of an addon. The arithmetic above still holds for
+the CLI, which has one query to amortize over; it does not for a server, which
+has every keystroke after boot. See
+[the search capability](../capabilities/search.md).
+
+`available?` is not decorative. The base gem's two engines are always available —
+`minifts` is a hard dependency with no native extension — but an addon backed by
+SQLite can fail to build, and the rule it inherits is that a broken build
+*degrades* rather than crashes. An engine whose store is missing answers
+`available? == false`; it never raises at query time.
+
+# `Search.register` is the second extension point
+
+```ruby
+OKF::Bundle::Search.register(engine)  # append-only, idempotent by id
+OKF::Bundle::Search.engines           # frozen snapshot, preference order
+```
+
+Append-only and idempotent by `id`, so a double `require` cannot double the
+registry and an addon cannot quietly displace a built-in. Capabilities outside
+the declared vocabulary are refused at registration — a typo like `:regex` would
+otherwise present as "my engine is silently never chosen".
+
+The built-ins register at *their* load, an addon at *its*, so `require "okf"`
+yields exactly `[Index, Scan]`. A clean-subprocess probe pins that — the same
+guard that keeps the [CLI shell out of the library load](core-shell-split.md) —
+because a `gem install` that silently changed what `okf search` answers would be
+a surprise nobody opted into.
+
+# The conformance suite replaced the oracle rule
+
+The earlier plan for addon search was that `Bundle::Search` defines semantics and
+every backend must return "the same match set, modulo ranking order". **That rule
+cannot survive more than one engine.** The index and the scan disagree about match
+sets by design — a phrase, an infix, a dotted identifier — so naming either the
+oracle would make the other a bug.
+
+What replaced it is narrower and actually holds: a shared conformance suite,
+included by one test class per engine, asserting only what must be true
+regardless of engine — the row shape and key order, ANDed terms, what `fields:`
+restricts, the ordering, the empty answers, and that `across` keeps same-id
+concepts distinct. Capability-gated blocks run only for engines declaring the
+capability, so an engine earns its own semantics by declaring them rather than by
+being excused from a rule.
+
+A registered engine with no conformance class is itself a test failure. That is
+the property the oracle rule was reaching for and could not express.
+
+**What the conformance suite cannot do, and what covers it.** It asserts that
+engines agree about the *shape* of an answer — row keys, ordering, ANDed terms,
+what `fields:` restricts. It cannot notice that every engine is missing a third
+of the matches, because consistency is not correctness. That blind spot is not
+hypothetical: the index swap made every word inside a code span unfindable, and
+the suite stayed green throughout.
+
+`recall_test.rb` covers it by running real queries over a corpus of tokenization
+hazards and measuring the index against the scan. The scan earns the oracle role
+there and *only* there — for recall it is sound, because raw-text matching finds
+any word that is present; for ranking and match sets the two engines disagree by
+design. The test pins the known holes and fails when the set changes in either
+direction, so a new hazard is named rather than discovered later in use.
+
+This follows the same discipline as the [core/shell split](core-shell-split.md):
+a boundary is only real when a test fails on crossing it, and it is checked the
+same way [integration first](integration-first.md) checks the CLI — against what
+a caller actually gets, not what an internal returns.

data/.okf/design/server-trust-boundary.md

@@ -0,0 +1,139 @@
+---
+type: Constraint
+title: The server trust boundary
+description: The trust boundary for serving a bundle you may not fully trust — both XSS paths into the page are closed, the registry write routes carry their own locks, and every read is realpath-contained so a symlinked file cannot escape the bundle root.
+resource: gems/okf/lib/okf/render/graph/template.html.erb
+tags: [security, server, xss, containment]
+generated:
+  by: human:maintainer
+  at: 2026-08-13T12:00:00Z
+sources:
+  - title: README.md — Server trust boundary
+    resource: https://github.com/serradura/okf/blob/main/README.md
+  - title: gems/okf/lib/okf/render/graph/template.html.erb
+    resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/render/graph/template.html.erb
+  - title: gems/okf/lib/okf/safe_read.rb
+    resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/safe_read.rb
+---
+
+# Overview
+
+The [graph server](../capabilities/graph-server.md) renders whatever bundle you
+point it at, and a bundle is just files, so the page has to assume a body might
+carry active content. Two defenses handle that — one for each path into the page.
+
+# Where the boundary sits
+
+There are two data paths into the page, and each carries its own guard:
+
+| Path                                               | Handling                                                                                     | Safe?                                                        |
+| -------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
+| Graph data **inlined** into the page               | through `json_for_script`, which escapes `<`                                                 | yes — it cannot break out of its `<script>`                  |
+| Concept bodies **fetched** on demand (`/node?id=`) | `marked` renders the Markdown, then `DOMPurify.sanitize` scrubs it before it reaches the DOM | yes — scripts, handlers, and `javascript:` URLs are stripped |
+
+The description and the §5 trust line shown in the inspector take a third path,
+and it moved when `/node/meta` became JSON. The server no longer sends a
+fragment to escape — it sends values (`description`, and the null-stripped
+`trust` mapping) — so the guard is that `renderMeta` puts every one of them into
+the DOM as **text**: a text node for the description, `textContent` for each
+chip. Nothing on this path is ever parsed as markup, which is why no sanitizer
+stands on it. A render path that reached for `innerHTML` here would need one.
+
+# The static render carries both guards
+
+[`okf render`](../capabilities/render.md) bakes every body into the page
+instead of fetching it, so an embedded body takes the *inlined* path **and** the
+rendered one: `json_for_script` escapes it at inject time (a `</script>` inside a
+body cannot break out of its `<script>`), and it is still
+`DOMPurify.sanitize(marked.parse(...))`'d when the getter hands it to the DOM. The
+same two defenses, now both on the one path — a static file is no laxer than the
+server, and the embedded description and trust line go through the same
+text-only `renderMeta` as above — composed client-side in both modes, from
+`/node/meta` when served and from the baked catalog row when not.
+
+# Both guards are asserted, against a bundle that attacks them
+
+`okf/test/browser/specs/sanitization.spec.js` drives
+`okf/test/browser/fixtures/hostile` — a conformant OKF bundle whose content is
+trying to execute script in the page rendering it — in both render modes. The
+payloads set flags on `window`, so the assertion is not "the markup looks
+clean" but *the script did not run*.
+<!-- rule:okf-verify-the-sanitizer -->
+
+This was a gap the browser suite's coverage review turned up: for a long time
+the only checks were that the string `DOMPurify` appeared in the emitted page
+and that it was a function at boot, both of which a render path skipping the
+sanitizer passes cleanly. The table above described intent, not a contract.
+
+Each guard was then mutation-checked, because a security test that cannot fail
+is worse than none:
+
+| Mutation | Result |
+|---|---|
+| `DOMPurify.sanitize(marked.parse(…))` → `marked.parse(…)` | 4 body specs red; `__xssImg` **fired** — real code execution |
+| `esc()` back to `&<>` only (pre-c2cedb6) | the tag breakout spec red; a live `onmouseover` in the DOM |
+| `json_for_script` without its `<` escape | all 14 red — the `</script>` in a title closes the block and the page never boots |
+
+The first of those carries a lesson for anyone extending the fixture: with the
+sanitizer removed, the `<script>` payload did **not** fire, because `innerHTML`
+does not execute script tags. Only the `<img onerror>` did. A fixture carrying
+script tags alone would have gone green against a page with no sanitizer at
+all — proving the defense while the hole stood open.
+
+# A second boundary: the server can now be asked to change something
+
+Everything above is about content coming *in* to the page. The
+[registry routes](../capabilities/bundles-manager.md) opened the other
+direction — four `POST` routes that write the [registry](../registry.md) — and it
+carries its own three locks rather than borrowing these: writable-at-all (loopback
+by default, declined with `--read-only`, refused outright anywhere else), a registry to write to, and same-origin
+plus a per-boot token. Sanitizing has nothing to say about a well-formed request
+that should not have been honoured, which is why that gate is described where it
+lives instead of being folded in here.
+
+# A third boundary: a file may not be where its name says
+
+Everything above trusts that a file inside the bundle *is* inside the bundle. A
+symlink breaks that: its name sits under the root, but its target need not, and
+`File.expand_path` — the lexical guard every read used — resolves the name, not
+the link. So a bundle carrying `services/billing.md → /etc/passwd`, or an
+`index.md` symlinked outside the root, had its target read and served: verbatim
+over the [graph server](../capabilities/graph-server.md) and `okf render`, and —
+the reason it finally mattered — over okf-mcp's tool set (`@okf-mcp design/the-tool-set`)
+`--http`, to any host that could reach the port. Two code comments and a test
+name asserted a containment that was never enforced; none tested it.
+
+The guard is the target's real path now, not its name. Every read a bundle
+serves — the Reader's bulk load, the single-concept handle, the live `log.md`
+re-read, and the MCP concept and index reads — goes through one shell primitive
+(`OKF::SafeRead`) that resolves `File.realpath`, refuses a result outside the
+real root (`Path.under?` is the pure decision, the I/O stays in the shell), and
+reads from the *resolved* path so a swap cannot slip between the check and the
+open. An escaping file is quarantined like any unreadable one, so one planted
+link cannot take a whole bundle down, and an escaping or vanished concept reads
+as a plain not-found rather than an error naming the target.
+<!-- rule:okf-realpath-contains-reads -->
+
+The durable half is the rule the old comments claimed and did not hold: **a
+lexical path check guards the name; a symlink escapes by its target, and only
+`realpath` sees where a name actually leads.** The boundary is drawn where it
+can be, not oversold past it — the same discipline the
+extension seam (`@okf-eco design/extension-points`) applies to its prefix: a hardlink shares
+its target's inode and keeps an in-root path that `realpath` cannot distinguish,
+and neither a hardlink nor a symlink survives a `git clone`, a copy or a tarball.
+So the portable, adversarial bundle is a symlink's to plant and `realpath`'s to
+close; the hardlink case is a local, non-portable one, named rather than claimed
+covered, because the obvious guard (reject `st_nlink > 1`) would break a bundle
+on a deduplicating filesystem.
+
+# What sanitizing does not cover
+
+DOMPurify removes the code, not the content. The page still fetches and shows the
+links, images, and Mermaid diagrams a body names (Mermaid runs in its `strict`
+mode), and it runs third-party code from a CDN — Cytoscape, marked and DOMPurify
+at boot, with Mermaid, Panzoom, the extra layout engines and
+[MiniSearch](../capabilities/graph-server.md) lazily on first use. Each of those
+is trust extended to the CDN as much as to the bundle; MiniSearch alone is pinned
+to an exact version (`7.2.0`), because it has to *agree* with the Ruby port rather
+than merely work. So the rule is no longer _only serve bundles you trust_ — it is
+the ordinary care you would give any document from a source you do not know.

data/.okf/index.md

@@ -0,0 +1,40 @@
+---
+okf_version: "0.2"
+---
+
+# okf knowledge bundle
+
+Everything about **okf**, the kernel of this ecosystem: what it does, how the
+code is arranged, why each rule is a rule, and how a change is proven. It is
+written to be read *before* opening `lib/`, so an agent about to add a verb, a
+check or a search engine does not re-derive the layering and does not rebuild
+something the model already answers.
+
+`AGENTS.md` beside this bundle is the gem's maintainer guide. It keeps the
+contract, the commands and the release steps, and routes here for everything
+else; the repository root's guide is the ecosystem's and states no contract of
+this gem's at all.
+
+`structure/` is pinned: `test/unit/bundle_catalog_test.rb` fails when a file
+under `lib/` is named by no concept, when a concept names a file that is gone,
+or when the verb table in [cli](cli.md) disagrees with `OKF::CLI.builtins`. The
+code is the truth and this bundle is the claim, so the two cannot drift quietly.
+
+One thing is deliberately elsewhere. The **format** — what a citation is, what
+§5 declares, what a bundle must contain — is the repository's own bundle
+(`@okf-eco`), because the skill, the plugin, the three sibling gems and any future
+non-Ruby implementation all speak it, and a reader asking about §5.1 is not
+asking about a Ruby gem. A concept cannot link out of its own bundle, so
+references to it name it in prose: `@okf-eco format/frontmatter`.
+
+* [Overview](overview.md) - The gem at a glance: the seven capabilities and the design ethos behind them.
+* [Command line](cli.md) - The `okf` executable — the one layer that parses argv, prints, and exits.
+* [Bundle registry](registry.md) - The list of bundles a machine or a project knows, addressable as `@slug`.
+
+# Areas
+
+* [Structure](structure/) - Every file under `lib/`, grouped by the layer that owns it — pinned to the tree by a test.
+* [The model](model/) - The pure in-memory data structures: concept, bundle, graph, skeleton.
+* [Capabilities](capabilities/) - The things the gem does: validate, lint, search, serve, render, the library, the skill.
+* [Design](design/) - The enforced boundaries that keep the gem light and honest.
+* [Testing](testing/) - How a change is proven, and the walk a new verb owes.