okf 1.8.0 → 1.10.0

data/README.md

@@ -35,7 +35,7 @@ The package, end to end:
 <p align="center">
   <picture>
     <source media="(prefers-color-scheme: dark)" srcset=".github/overview-dark.png">
-    <img src=".github/overview-light.png" width="760" alt="The package: the Agent Skill (your coding agent authors and curates, you stay the editor) writes and maintains the bundle, a folder of Markdown + YAML in your repo where one concept is one file and links between files are the knowledge graph. The bundle is read by the CLI/Lib (validate: legal OKF per section 9; lint: well-curated and fresh; search: ranked retrieval; require okf for Ruby objects) and by the Graph, in four modes: okf server (a live local server), okf render (the same page exported as one static, self-contained HTML file you can host anywhere), okf registry (every registered bundle behind one hub), and OKF::Server::App (the Rack app mounted in a Rails route). One gem, 100% local, Ruby 2.4 or newer, only rack and webrick as dependencies.">
+    <img src=".github/overview-light.png" width="760" alt="The package: the Agent Skill (your coding agent authors and curates, you stay the editor) writes and maintains the bundle, a folder of Markdown + YAML in your repo where one concept is one file and links between files are the knowledge graph. The bundle is read by the CLI/Lib (validate: legal OKF per section 9; lint: well-curated and fresh; search: ranked retrieval; require okf for Ruby objects) and by the Graph, in four modes: okf server (a live local server), okf render (the same page exported as one static, self-contained HTML file you can host anywhere), okf registry (every registered bundle behind one hub), and OKF::Server::App (the Rack app mounted in a Rails route). One gem, 100% local, Ruby 2.4 or newer, only rack, webrick and minifts as dependencies.">
   </picture>
 </p>
 
@@ -72,10 +72,13 @@ It is deliberately light so it runs on the Ruby your OS already ships:
 
 - works on every Ruby since 2.4, the same floor as [rack](https://github.com/rack/rack),
   its core dependency;
-- only two runtime dependencies: `rack` (the server is a mountable Rack app)
-  and `webrick` (unbundled from Ruby in 3.0);
-- no ActiveSupport, no build step, no JavaScript toolchain — the
-  [design constraints](.okf/design/) that hold this line are enforced by tests.
+- only three runtime dependencies: `rack` (the server is a mountable Rack app),
+  `webrick` (unbundled from Ruby in 3.0), and
+  [`minifts`](https://github.com/serradura/minifts) (the search engine — pure
+  Ruby, no dependencies of its own, same 2.4 floor);
+- no ActiveSupport, no native extension, no build step, no JavaScript
+  toolchain — the [design constraints](.okf/design/) that hold this line are
+  enforced by tests.
 
 That range is not aspirational: CI runs the full test suite and RuboCop on every
 one of these on each push.
@@ -222,10 +225,15 @@ docker run --rm -v "$PWD:/data" ghcr.io/serradura/okf validate .
 docker run --rm -v "$PWD:/data" -p 8808:8808 ghcr.io/serradura/okf server . --bind 0.0.0.0
 ```
 
+A container has to bind `0.0.0.0` for the host to reach it at all, which is
+exactly the bind that makes the registry read-only. Reading the graph is
+unaffected, and there is no flag that opens the registry on that bind — it is a
+per-user file, managed from the machine that owns it or from `okf registry`.
+
 Then open <http://127.0.0.1:8808>. Images are published for `linux/amd64` and
 `linux/arm64` on
 [ghcr.io](https://github.com/serradura/okf-gem/pkgs/container/okf): `:latest`
-tracks the newest release, or pin a version like `:1.8.0`.
+tracks the newest release, or pin a version like `:1.10.0`.
 
 Tired of the long line? Install a Docker-backed [`okf` command](https://docker.okfgem.com),
 so every verb drops the `docker run` prefix and reads exactly like the native CLI
@@ -243,13 +251,20 @@ instead: `irm https://docker.okfgem.com/install.ps1 | iex`.
 
 ## Command line
 
+These verbs are written to be read by an **agent first and a person second** —
+that is what the skill drives, with no wrapper in between. Every read verb takes
+`--json`, `index`/`catalog`/`files` project down to the fields you ask for
+(`--fields`/`--except`), so nothing pays for output it will not read, and the
+exit codes are stable enough to branch on in CI. The same commands render as
+scannable plain text when a human is the one looking.
+
 ```bash
 okf validate  <dir|@slug> [--json]                         # check OKF v0.1 conformance (§9)
 okf lint      <dir|@slug> [--json] [--fail-on warn] [...]  # report curation-quality issues
 okf loose     <dir|@slug> [--json]                         # list files with no graph links, by folder
-okf search    <dir|@slug…|@all> <term…> [-e] [...]         # ranked retrieval; @slugs or @all span bundles
+okf search    <dir|@slug…|@all> <term…> [--regexp|--fuzzy] # ranked retrieval; @slugs or @all span bundles
 okf index     <dir|@slug> [--json] [--area A] [--no-body]  # progressive-disclosure map (§6): bodies, rollups, listings
-okf server    [DIR|@slug…] [-p PORT] [--bind ADDR] [...]   # serve one bundle, or many behind a hub (⌘K to switch)
+okf server    [DIR|@slug…] [-p PORT] [--bind ADDR] [...]   # serve one bundle, or many behind a hub (⌘K searches every one)
 okf render    <dir|@slug> [-o FILE] [--layout NAME] [...]  # export the graph as one static, self-contained HTML file
 okf registry  list [--json]                                # list registered bundles (* marks the default)
 okf registry  set <dir|@slug> [--as SLUG] [--default]      # add or update a bundle (a bare `server` serves it)
@@ -266,8 +281,11 @@ okf registry  default <@slug> | rename <@slug> <new>       # move a bundle to th
 # set`, or bare @ for the default. Anywhere a <dir> goes, an @slug goes:
 # okf lint @handbook works from anywhere. Set $OKF_HOME to point every verb
 # at another registry.
-okf graph     <dir|@slug> [--json] [--minimal] [--no-body]        # print the knowledge graph
+okf graph     <dir|@slug> [--json] [--minimal] [--hubs]           # print the knowledge graph (--hubs ranks by inbound links)
 okf catalog | files | tags | types | stats  <dir|@slug> [--json]  # the browser views, on the CLI
+# `tags --by type|area` regroups the tag index per concept dimension; each row
+# shows count/total, so a tag confined to one area (a domain) reads differently
+# from one spread across several (a cross-cutting concern).
 okf skill     <dest> [--here] [--force]                 # install the companion agent skill
 okf --version
 ```
@@ -300,12 +318,20 @@ _The graph server on this repo's own [`.okf`](.okf) bundle, with the
 
 The page is one template from a phone to a desktop: on small screens the
 navigation rail becomes a drawer, the toolbar folds into a `⚙` sheet, and the
-panels go full-bleed — rotate a tablet and the layout re-evaluates. It is
-keyboard-first too: `⌘/Ctrl-K` opens a command palette in every mode (views
+panels go full-bleed — rotate a tablet and the layout re-evaluates. On a touch
+screen a tap opens a preview card at the bottom edge rather than a panel over
+the whole viewport, so the graph stays on screen and live while you read: drag
+the card up for the neighbourhood, tap a link in it and it walks there in place.
+It is keyboard-first too: `⌘/Ctrl-K` opens a command palette in every mode (views
 always; bundles too when a [hub](#one-registry-many-bundles) is serving), `/`
 jumps to the current view's search, and `?` answers with a sheet of every
 shortcut.
 
+The search box says what it is doing while it does it: a live `7/8` count of
+what the filter kept, and — when a word matches nothing here — a panel naming
+the bundle and the query, offering to search every other bundle instead. The
+query carries over, so nothing is typed twice.
+
 To skip the server entirely, **`okf render <dir>`** writes that same page as one
 self-contained HTML file, the whole bundle baked in, so you can publish the
 graph on GitHub Pages or any static host.
@@ -326,10 +352,32 @@ okf server                              # no args: the whole registry behind one
 
 The first entry still on disk is the **default** — the bundle a bare `okf
 server` opens at `/` and a bare `@` names; `okf registry default @slug` moves an
-entry to the front. Behind the hub each bundle mounts at `/b/<slug>/`, `/b/` is
-a browsable index, and the `⌘/Ctrl-K` palette switches bundles without leaving
-the page. The hub reads the registry at boot, so restart it after registry
-changes; set `$OKF_HOME` to point every verb at a different registry.
+entry to the front. Behind the hub each bundle mounts at `/b/<slug>/`, and the
+`⌘/Ctrl-K` palette both switches bundles and **searches every one of them at
+once** — type a few words and the matching concepts appear with their bundle and
+a snippet, from wherever you are.
+
+The ⚙ in the rail opens **Bundles** — the registry on the graph page itself, so
+switching the default, renaming an entry or dropping one no longer means finding
+`/b/` first. Each row carries its size, its health as a *word*, and which one `/`
+opens. There is no Add: registering a bundle means naming a filesystem path,
+which a browser cannot hand over and an agent can (`okf registry set <dir>`), and
+the panel says so instead of leaving the absence to be noticed.
+
+`/b/` is the **bundles list**: every bundle with its size, its health
+(conformance plus curation, so a warning is a warning and not a failure), the
+default marked, and any entry whose folder has gone missing shown rather than
+quietly dropped.
+
+Managing the registry without a terminal happens on the graph page instead, in
+the ⚙ **Bundles** panel — make default, rename, remove, where you are already
+reading. Those controls are the one thing that does not follow you onto a
+network: bind anywhere but loopback and they are refused outright, with no flag
+that says otherwise, since `--bind 0.0.0.0` is how a personal tool becomes a
+public one. `--read-only` declines them on loopback too, and is the only switch
+there is. Writes rebuild the hub's set as they go, so a rename takes effect on
+the next click; a registry change made *elsewhere* while it runs still wants a
+restart to be served. Set `$OKF_HOME` to point every verb at a different registry.
 
 `graph` and `server` are best-effort (§9): a file with invalid frontmatter is
 skipped (and noted on stderr), not fatal, so one bad file never breaks the rest.
@@ -359,6 +407,22 @@ OKF lint — docs
   ⚠ 3 warn, 31 info
 ```
 
+That `hubs:` line is the bundle's centre of gravity — the concepts the most links
+point *at*, with their inbound count. **`okf graph <dir> --hubs`** expands it into
+the full ranking and, for each hub, the areas those links come *from*:
+
+```bash
+$ okf graph docs --hubs
+Hubs — docs (38 of 40 concepts with inbound links)
+
+  storage/reconcile   ×16   storage 6, shape 4, exploration 3, (root) 1, …
+```
+
+That second half is what decides whether a concept sits where it belongs: a hub
+pulled in mostly from outside its own folder is usually homed by history rather
+than by meaning. Together with `okf tags <dir> --by area`, it is the evidence the
+skill's [`refine`](#agent-skill) verb reads before proposing anything.
+
 `loose` lists the files that float in the graph: concepts with no cross-links
 in or out (graph degree 0), grouped by folder. It is a curation lens over
 `lint`'s `unlinked` check, distinct from `orphan`. An `index.md` listing makes a
@@ -388,11 +452,21 @@ The skill routes a small set of verbs. In Claude Code they run as `/okf:gem
 | `produce`        | Create or extend a bundle from code, docs, or knowledge in people's heads                                   |
 | `migrate`        | Adopt existing Markdown docs in place: frontmatter and reserved files added, bodies kept verbatim           |
 | `maintain`       | Sync the bundle's content with reality after the code or docs change                                        |
+| `refine`         | Restructure it for retrieval: evidence-first, cohesion over balance — proposes, never applies             |
 | `consume`        | Use the bundle as context for a task, writing back what you learn                                           |
 | `curate`         | Structural upkeep as it stands: `validate` + `lint` + `loose`                                               |
 | `doctor`         | Install and verify the CLI, then doctor the bundle                                                          |
 | `<okf-cli-verb>` | Run any CLI verb (`validate`, `lint`, `search`, `index`, `server`, the read views) and interpret its output |
 
+Three of those look alike and are not, which is the distinction worth learning
+first: **`curate`** keeps the bundle *sound* (the structure as it stands),
+**`maintain`** keeps it *true* (the code changed, so the content must catch up),
+and **`refine`** changes *where knowledge lives* — the folder a concept sits in,
+a fact re-explained in three overviews. Reach for `refine` when nothing is wrong
+and everything is hard to find: a folder grown to twenty concepts, a hub homed by
+history, tags that neither connect nor mark. It reads the evidence, then hands
+you a proposal — it never rearranges your bundle on its own.
+
 Point it at your agent's config directory (or its skills directory) and the tree
 settles in its own `skills/okf/` folder, so a shared skills directory never gets
 the files loose:
@@ -510,6 +584,26 @@ Two loop concerns from the format's own guidance, _contradictions_ and _semantic
 staleness, need to understand meaning and are not computed here; `lint --json`
 is the structured input an agent consumes to reason about those.
 
+## Extending okf
+
+This is the other kind of plugin, and it is worth separating from the
+[Claude Code plugin](#claude-code-plugin) below: that one teaches an *agent* to
+use okf, while this one adds behavior to the `okf` command everyone runs.
+
+Publish a gem named `okf-*` that carries an `okf/plugin.rb`, and installing it is
+the whole installation: your verb answers to `okf`, appears in `okf help` under
+`installed extensions:`, and behaves like a built-in — nothing to register,
+nothing to configure, no list of known addons in this gem. The same seam takes
+retrieval backends, which is what `okf search --engine NAME` chooses between.
+
+Two promises make that safe to install: nothing an addon registers can displace a
+built-in, and a broken addon is skipped and reported rather than taking the CLI
+down with it.
+
+The [extension points](.okf/design/extension-points.md) concept has the contract
+and the threat model; `test/integration/cli/cli_plugin_test.rb` is a working
+example to copy.
+
 ## Server trust boundary
 
 > [!NOTE]

data/lib/okf/bundle/folder.rb

@@ -46,10 +46,24 @@ module OKF
         @bundle.catalog
       end
 
+      def hubs
+        @bundle.hubs
+      end
+
       def directory_index
         @bundle.directory_index
       end
 
+      # Every log.md with its content, root scope first — read live from disk so a
+      # just-appended entry shows without a reload; the reserved snapshot is the
+      # fallback if the file has since vanished. Shared by `okf render`'s bake
+      # (OKF::Render::Graph.payload) and OKF::Server::App's /log endpoint.
+      def log_entries
+        @bundle.log_files.sort_by { |path| [ path == "log.md" ? 0 : 1, path ] }.map do |path|
+          { path: path, dir: File.dirname(path), content: log_content(path) }
+        end
+      end
+
       # Human-readable "parent/dir" name — the default HTML title.
       # The bundle's display label, "parent/dir" — path arithmetic, no disk. It
       # is a class method so a caller that only wants the label (the registry
@@ -96,6 +110,12 @@ module OKF
                .select { |entry| File.basename(entry.path) == basename }
                .each_with_object({}) { |entry, hash| hash[entry.path] = entry.content }
       end
+
+      def log_content(path)
+        File.read(File.join(@root, path), encoding: "UTF-8")
+      rescue SystemCallError
+        @bundle.reserved_content(path)
+      end
     end
   end
 end

data/lib/okf/bundle/search/index.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+require "minifts"
+require "okf/bundle/search"
+
+module OKF
+  class Bundle
+    class Search
+      # The default engine: a MiniFTS full-text index — the same engine, and the
+      # same BM25+ arithmetic, the browser page already runs as MiniSearch, so a
+      # Ruby-built index and the page's rank identically.
+      #
+      # Matching is by *token*: a term matches a whole word or a word it prefixes
+      # ("dedup" reaches "deduplication"), and `fuzzy:` opts into typo tolerance.
+      # The index is built per call — see .okf/capabilities/search.md for why that
+      # ceiling stands and what lifts it.
+      module Index
+        CAPABILITIES = %i[fuzzy prefix].freeze
+
+        class << self
+          def id
+            :index
+          end
+
+          def capabilities
+            CAPABILITIES
+          end
+
+          # minifts is a hard runtime dependency with no native extension, so it
+          # is here whenever the gem is. An addon backed by a native build is the
+          # case this predicate exists for.
+          def available?
+            true
+          end
+
+          # `fields:` narrows where a term may hit, so a field the caller excluded
+          # can neither match nor be credited. The hit's `terms` are MiniFTS's
+          # matched *document* terms — already lowercased, and present in the text
+          # verbatim even when the query only prefixed them.
+          def call(documents, terms, fields:, fuzzy: false, **_options)
+            index = MiniFTS.new(fields: FIELDS, id_field: "key")
+            index.add_all(documents)
+
+            options = { combine_with: "AND", prefix: true, boost: WEIGHTS, fields: fields }
+            options[:fuzzy] = FUZZY_DISTANCE if fuzzy
+
+            index.search(terms.join(" "), options).map do |hit|
+              { key: hit[:id], matched: matched_in(hit), score: hit[:score], terms: hit[:terms] }
+            end
+          end
+
+          private
+
+          # The union of fields any term hit, in WEIGHTS order. MiniFTS reports it
+          # per query term as { term => [field, …] }.
+          def matched_in(hit)
+            FIELDS.select { |field| hit[:match].any? { |_term, found| found.include?(field) } }
+          end
+        end
+
+        Search.register(self)
+      end
+    end
+  end
+end

data/lib/okf/bundle/search/scan.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+require "okf/bundle/search"
+
+module OKF
+  class Bundle
+    class Search
+      # The linear engine: terms matched against raw field text, one document at a
+      # time. No index, so nothing is tokenized and nothing is normalized — a
+      # phrase stays a phrase, `7.2.0` stays one string, and an infix matches.
+      # That is the exactness a token index gives up, and the reason this engine
+      # survived the swap.
+      #
+      # Two readings of a term, and the engine is the *raw text* half of the split
+      # rather than the regexp half:
+      #
+      #   `regexp: false` — literal substring, which is what this engine did
+      #                     before the index landed, and what `--engine scan`
+      #                     restores. Terms are escaped, so `7.2.0` does not
+      #                     match `7x2y0` and `[draft]` is not a character class.
+      #   `regexp: true`  — the term is a pattern, opted into with `-e`.
+      #
+      # Conflating the two would make choosing the engine silently change what the
+      # terms mean, and turn an ordinary term like `review (pending` into exit 2.
+      #
+      # Scoring is the summed weight of the fields that matched: absolute, and so
+      # comparable across bundles without a corpus to normalize against.
+      module Scan
+        CAPABILITIES = %i[regexp].freeze
+
+        class << self
+          def id
+            :scan
+          end
+
+          def capabilities
+            CAPABILITIES
+          end
+
+          # No backing store to fail: the engine is Regexp and Enumerable.
+          def available?
+            true
+          end
+
+          # Raises RegexpError on an invalid pattern under `regexp: true` — the
+          # caller owns turning that into a usage error. A literal term cannot
+          # raise, because it is escaped before it is compiled. The hit's `terms`
+          # are the compiled patterns, which is what the facade points its snippet
+          # window at.
+          def call(documents, terms, fields:, regexp: false, **_options)
+            patterns = terms.map do |term|
+              Regexp.new(regexp ? term : Regexp.escape(term), Regexp::IGNORECASE)
+            end
+
+            hits = []
+            documents.each do |document|
+              matched = matched_fields(document, patterns, fields)
+              next if matched.nil?
+
+              hits << {
+                key: document["key"],
+                matched: matched,
+                score: matched.map { |field| WEIGHTS[field] }.reduce(0, :+),
+                terms: patterns
+              }
+            end
+            hits
+          end
+
+          private
+
+          # The union of fields any pattern hit, in WEIGHTS order — or nil when
+          # some pattern hit nothing (terms are ANDed).
+          def matched_fields(document, patterns, fields)
+            hits = patterns.map do |pattern|
+              found = fields.select { |field| pattern.match?(document[field]) }
+              return nil if found.empty?
+
+              found
+            end
+            FIELDS.select { |field| hits.any? { |found| found.include?(field) } }
+          end
+        end
+
+        Search.register(self)
+      end
+    end
+  end
+end