okf 2.1.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.okf/capabilities/agent-skill.md +112 -0
- data/.okf/capabilities/bundles-manager.md +144 -0
- data/.okf/capabilities/graph-server.md +678 -0
- data/.okf/capabilities/index.md +26 -0
- data/.okf/capabilities/library-api.md +82 -0
- data/.okf/capabilities/linter.md +83 -0
- data/.okf/capabilities/read-views.md +228 -0
- data/.okf/capabilities/render.md +66 -0
- data/.okf/capabilities/search.md +297 -0
- data/.okf/capabilities/validator.md +60 -0
- data/.okf/cli.md +214 -0
- data/.okf/design/browser-tests.md +211 -0
- data/.okf/design/core-shell-split.md +73 -0
- data/.okf/design/index.md +17 -0
- data/.okf/design/integration-first.md +140 -0
- data/.okf/design/packaging.md +65 -0
- data/.okf/design/ruby-floor.md +53 -0
- data/.okf/design/runtime-dependencies.md +82 -0
- data/.okf/design/search-engines.md +154 -0
- data/.okf/design/server-trust-boundary.md +139 -0
- data/.okf/index.md +40 -0
- data/.okf/log.md +724 -0
- data/.okf/model/bundle.md +47 -0
- data/.okf/model/concept.md +75 -0
- data/.okf/model/graph.md +59 -0
- data/.okf/model/index.md +9 -0
- data/.okf/model/skeleton.md +76 -0
- data/.okf/overview.md +87 -0
- data/.okf/registry.md +432 -0
- data/.okf/structure/format-layer.md +59 -0
- data/.okf/structure/index.md +22 -0
- data/.okf/structure/search.md +53 -0
- data/.okf/structure/the-analysers.md +60 -0
- data/.okf/structure/the-cli.md +99 -0
- data/.okf/structure/the-disk-shell.md +76 -0
- data/.okf/structure/the-model.md +81 -0
- data/.okf/structure/the-server.md +74 -0
- data/.okf/structure/the-skill.md +52 -0
- data/.okf/testing/adding-a-verb.md +76 -0
- data/.okf/testing/index.md +12 -0
- data/.okf/testing/the-harness.md +45 -0
- data/CHANGELOG.md +161 -16
- data/README.md +226 -17
- data/lib/okf/cli/command.rb +8 -3
- data/lib/okf/cli/registry.rb +306 -47
- data/lib/okf/cli.rb +1 -1
- data/lib/okf/registry.rb +448 -22
- data/lib/okf/render/graph/template.html.erb +6 -2
- data/lib/okf/server/hub.rb +6 -2
- data/lib/okf/skill/reference/cli/registry.md +49 -6
- data/lib/okf/skill/reference/cli.md +1 -1
- data/lib/okf/version.rb +1 -1
- metadata +46 -5
|
@@ -0,0 +1,297 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Capability
|
|
3
|
+
title: Ranked text search (search)
|
|
4
|
+
description: Full-text retrieval over concept metadata and bodies — raw-text matching by default, BM25+ token ranking on request, and explainable row by row either way.
|
|
5
|
+
resource: gems/okf/lib/okf/bundle/search.rb
|
|
6
|
+
tags: [read, cli, json, registry, search]
|
|
7
|
+
generated:
|
|
8
|
+
by: human:maintainer
|
|
9
|
+
at: 2026-08-13T12:00:00Z
|
|
10
|
+
sources:
|
|
11
|
+
- title: gems/okf/lib/okf/bundle/search.rb
|
|
12
|
+
resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/bundle/search.rb
|
|
13
|
+
- title: gems/okf/test/integration/cli/by_dir/cli_search_test.rb
|
|
14
|
+
resource: https://github.com/serradura/okf/blob/main/gems/okf/test/integration/cli/by_dir/cli_search_test.rb
|
|
15
|
+
- title: gems/okf/test/unit/bundle/search/recall_test.rb
|
|
16
|
+
resource: https://github.com/serradura/okf/blob/main/gems/okf/test/unit/bundle/search/recall_test.rb
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# Overview
|
|
20
|
+
|
|
21
|
+
`okf search <dir> <term…>` answers "which concept covers X?" for the price of a
|
|
22
|
+
few rows instead of a body read. The [browser page](graph-server.md) already had
|
|
23
|
+
search; this brings it to the [CLI](../cli.md) — the agent's eyes — and goes
|
|
24
|
+
further by searching bodies too. The core is `OKF::Bundle::Search`, a pure class
|
|
25
|
+
(guarded by the [core/shell boundary](../design/core-shell-split.md)) over the
|
|
26
|
+
in-memory [bundle](../model/bundle.md), so the CLI and any embedding app share
|
|
27
|
+
it: `OKF::Bundle::Search.call(bundle, [ "dedup", "key" ])`.
|
|
28
|
+
|
|
29
|
+
It is a **facade over two engines**. The default is a linear scan over raw text;
|
|
30
|
+
`--engine index` (and `--fuzzy`, which implies it) reaches a
|
|
31
|
+
[`minifts`](../design/runtime-dependencies.md) full-text index — the pure-Ruby
|
|
32
|
+
port of the MiniSearch build the browser loads, so a Ruby-built index and the
|
|
33
|
+
page's rank identically by construction rather than by two implementations
|
|
34
|
+
agreeing for as long as someone maintains both.
|
|
35
|
+
|
|
36
|
+
# Why the scan is the default
|
|
37
|
+
|
|
38
|
+
A CLI process loads the bundle, asks one question, and exits. An index build has
|
|
39
|
+
exactly one query to amortize it over, and it is not close:
|
|
40
|
+
|
|
41
|
+
| concepts | `--engine index` | default (scan) |
|
|
42
|
+
|---|---|---|
|
|
43
|
+
| 24 | 0.16 s | 0.10 s |
|
|
44
|
+
| 250 | 0.83 s | 0.18 s |
|
|
45
|
+
| 1,000 | 3.00 s | 0.24 s |
|
|
46
|
+
|
|
47
|
+
End to end through the CLI, 2026-07-18, Ruby 4.0.5. The build is ~95% of the
|
|
48
|
+
index path's cost at every size, and the gap widens with the bundle.
|
|
49
|
+
|
|
50
|
+
The headline number points the other way — `minifts` sustains
|
|
51
|
+
[~44–56× the query throughput](https://github.com/serradura/minifts) of a scan —
|
|
52
|
+
and that is the right measure for a **long-lived** index (a browser page, a
|
|
53
|
+
server) and the wrong one for a one-shot process. So the arithmetic above governs
|
|
54
|
+
the CLI only, and the server takes the other branch.
|
|
55
|
+
|
|
56
|
+
# The server holds the index, the CLI cannot
|
|
57
|
+
|
|
58
|
+
`Search.prepare` builds a **Corpus** — the documents, the key → concept map, and
|
|
59
|
+
the built index — and `Search.with` queries it without rebuilding. That is the
|
|
60
|
+
whole difference between the two callers: `okf server` prepares one at boot
|
|
61
|
+
(`warm_search`), so the build lands in startup where it is attributable, and
|
|
62
|
+
every search after it is a query alone.
|
|
63
|
+
|
|
64
|
+
| 414 concepts, served | before | after |
|
|
65
|
+
|---|---|---|
|
|
66
|
+
| boot | — | 1.39 s (once) |
|
|
67
|
+
| each search | 1.45 s | 0.016 – 0.052 s |
|
|
68
|
+
|
|
69
|
+
Measured 2026-07-22 through the Rack app, Ruby 4.0.5. Flat before, because each
|
|
70
|
+
request rebuilt the corpus it had just thrown away.
|
|
71
|
+
|
|
72
|
+
The cost is staleness: a corpus is a **snapshot**, so a body edited after it was
|
|
73
|
+
built is searchable only once the holder drops it — the hub does exactly that
|
|
74
|
+
when a registry write changes the served set. The graph is memoized on the same
|
|
75
|
+
terms, so this is the boundary the server already had, not a new one.
|
|
76
|
+
|
|
77
|
+
The CLI still builds per call and still defaults to the scan: a one-shot process
|
|
78
|
+
has nothing to amortize over, which is the asymmetry this whole section is about.
|
|
79
|
+
|
|
80
|
+
Speed is not the only reason. Raw-text matching has **no tokenizer**, so it has
|
|
81
|
+
no tokenizer-shaped recall holes — see below.
|
|
82
|
+
|
|
83
|
+
# Matching and ranking
|
|
84
|
+
|
|
85
|
+
Terms **AND** together: every term must hit at least one searched field, though
|
|
86
|
+
not necessarily the same one. Field weights are shared by both engines:
|
|
87
|
+
|
|
88
|
+
| Field | Weight |
|
|
89
|
+
|-------|-------|
|
|
90
|
+
| `title` | 5 |
|
|
91
|
+
| `id` | 4 |
|
|
92
|
+
| `tags` | 3 |
|
|
93
|
+
| `type`, `description` | 2 |
|
|
94
|
+
| `sources`, `body` | 1 |
|
|
95
|
+
|
|
96
|
+
The **scan** matches a term as a literal substring anywhere in a field and scores
|
|
97
|
+
by summing the weights of the fields that matched — an absolute number, small and
|
|
98
|
+
integral. The **index** matches a whole token or a token it prefixes (`dedup`
|
|
99
|
+
reaches `deduplication`), and scores BM25+ with those weights riding as per-field
|
|
100
|
+
boost — a float, relative to the corpus.
|
|
101
|
+
|
|
102
|
+
Rows order by score descending, then slug, then id. A match in `description`,
|
|
103
|
+
`body` or `sources` carries one bounded context snippet (~44 characters each
|
|
104
|
+
side of the first matched term); the other fields need none because they
|
|
105
|
+
already appear whole on the row. `sources` is each entry's title and resource
|
|
106
|
+
joined, at the weight the `# Citations` body text carried in v0.1, so a
|
|
107
|
+
migrated bundle keeps its recall and a source-only hit keeps a snippet — the
|
|
108
|
+
snippet *moves* from body text to source text rather than vanishing.
|
|
109
|
+
|
|
110
|
+
The pages share the fields with one pinned asymmetry: the **static** page bakes
|
|
111
|
+
each concept's body and source text and indexes both offline; the **served**
|
|
112
|
+
page indexes metadata only — its catalog row carries a source *count*, and
|
|
113
|
+
spending body-sized bytes on every fetch to serve one view is the trade the
|
|
114
|
+
payload already refuses for `body`. Pre-existing, deliberate, and stated at
|
|
115
|
+
`FT_FIELDS` in the template rather than discovered.
|
|
116
|
+
|
|
117
|
+
**The row still says which fields hit.** A relevance number alone would be a
|
|
118
|
+
verdict an agent cannot check, so every row carries its `matched` list — read off
|
|
119
|
+
the engine's own per-term field record, not recomputed beside it.
|
|
120
|
+
|
|
121
|
+
# What the index gives up, and why it is opt-in
|
|
122
|
+
|
|
123
|
+
Everything a token index cannot represent is something the tokenizer already
|
|
124
|
+
split or normalized away. These are the reasons the scan leads:
|
|
125
|
+
|
|
126
|
+
| Query | Default (scan) | `--engine index` |
|
|
127
|
+
|---|---|---|
|
|
128
|
+
| `"dedup key"` as one argument | contiguous only | two tokens ANDed — matches words paragraphs apart |
|
|
129
|
+
| `7.2.0` | one string | tokens `7`, `2`, `0` — matches "0 downtime, 7 regions, 2 zones" |
|
|
130
|
+
| `customer_id` | the identifier, whole | `customer` + `id` — matches "the customer table has an id column" |
|
|
131
|
+
| `ustomer` | finds Customers | nothing: an infix is not a token |
|
|
132
|
+
| `minifts` | **5** concepts | 2 — three write it only as `` `minifts` `` |
|
|
133
|
+
| `json_for_script` | **4** | 1 |
|
|
134
|
+
|
|
135
|
+
Measured on this bundle, 2026-07-18. The tokenizer splits on whitespace **and
|
|
136
|
+
punctuation**, which is why a dot and an underscore both shatter an identifier.
|
|
137
|
+
The last two rows are a different fault: a backtick is Unicode `Sk` and `$` is
|
|
138
|
+
`Sc`, neither of which is `P`, so **neither is ever split off**. A word inside a
|
|
139
|
+
code span is stored as the token `` `minifts` ``, which the query `minifts` does
|
|
140
|
+
not match — 409 such tokens on this bundle, 1,013 occurrences.
|
|
141
|
+
|
|
142
|
+
That class of loss is invisible: the search succeeds, returns plausible rows, and
|
|
143
|
+
silently omits most of the answer. Making raw text the default is what removed it
|
|
144
|
+
from the path nobody opted into.
|
|
145
|
+
|
|
146
|
+
**Ranking does not contain the loss.** This capability once claimed the true hit
|
|
147
|
+
still ranks first, so the cost was only extra rows below the answer. That is
|
|
148
|
+
false, and the pinning tests found it: BM25 normalizes by field length, so a short
|
|
149
|
+
body dense in `7`, `2` and `0` outscores the concept that actually says `7.2.0`.
|
|
150
|
+
On this bundle, `okf search .okf 7.2.0 --engine index` ranks
|
|
151
|
+
[the Ruby floor](../design/ruby-floor.md) — a page full of `2.4`, `2.6`, `3.x` —
|
|
152
|
+
**above** [the graph server](graph-server.md), the one concept naming the version.
|
|
153
|
+
|
|
154
|
+
# What the index buys
|
|
155
|
+
|
|
156
|
+
Reaching for it is a real choice, not a legacy path:
|
|
157
|
+
|
|
158
|
+
- **BM25+ ranking.** Corpus-relative relevance, which absolute field weights only
|
|
159
|
+
approximate. On a large or uneven bundle this is the better ordering.
|
|
160
|
+
- **Fuzzy matching.** `--fuzzy` is only available here — the scan has no notion
|
|
161
|
+
of edit distance, so asking for it routes automatically.
|
|
162
|
+
- **Parity with the browser page**, which runs the same MiniSearch build. If you
|
|
163
|
+
are reconciling a CLI answer with what the page shows, name the index.
|
|
164
|
+
|
|
165
|
+
That is the whole list — three things. The `prefix` capability is conspicuously
|
|
166
|
+
**not** a fourth, though the index declares it: a substring match already reaches
|
|
167
|
+
every prefix, so `dedup` finds `deduplication` under either engine while
|
|
168
|
+
`duplication` and `uplicat` find it under the scan alone. Prefix is what a token
|
|
169
|
+
index needs to catch up to raw text, not something it adds on top. Worth stating
|
|
170
|
+
plainly, because "prefix matching" reads like a feature the default lacks.
|
|
171
|
+
|
|
172
|
+
# Engines are adapters, chosen by what the query needs
|
|
173
|
+
|
|
174
|
+
`OKF::Bundle::Search` is a facade over N engines, not one implementation with a
|
|
175
|
+
branch. It owns everything that defines what a *result* is — documents, the row
|
|
176
|
+
and its key order, the snippet window, the final sort — and delegates only "which
|
|
177
|
+
documents match, how well, and where":
|
|
178
|
+
|
|
179
|
+
| Engine | Capabilities | Scoring |
|
|
180
|
+
|---|---|---|
|
|
181
|
+
| `Search::Scan` (default) | `regexp` | summed field weights, absolute |
|
|
182
|
+
| `Search::Index` | `fuzzy`, `prefix` | BM25+, corpus-relative |
|
|
183
|
+
|
|
184
|
+
Selection happens two ways, and they answer different questions.
|
|
185
|
+
|
|
186
|
+
**By capability**, when the query requires something: `--fuzzy` requires `:fuzzy`,
|
|
187
|
+
which only the index offers, so it routes there without naming it. `-e` requires
|
|
188
|
+
`:regexp`, which the default already provides, so it moves nothing. A query
|
|
189
|
+
requiring nothing gets the default. Routing is **silent** — no note on stderr,
|
|
190
|
+
nothing in the header, nothing in the JSON envelope.
|
|
191
|
+
|
|
192
|
+
**By name**, with `--engine`, when the query requires nothing but the *matching
|
|
193
|
+
model* matters. This is the case capability flags cannot express: BM25 ranking
|
|
194
|
+
requires no capability, so there is nothing to route on. A named engine that
|
|
195
|
+
cannot do what was *also* asked is an error, never a silent fallback:
|
|
196
|
+
|
|
197
|
+
```
|
|
198
|
+
okf search . --engine index -e 'err_[a-z]+' # error: --engine index does not
|
|
199
|
+
# support --regexp (try --engine scan)
|
|
200
|
+
okf search . --engine fts5 auth # error: unknown search engine: fts5
|
|
201
|
+
# (available: index, scan)
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
The two readings of a term stay separate from the engine that reads them: the
|
|
205
|
+
scan matches **literally** and `-e` opts into the pattern reading, so `7.2.0`
|
|
206
|
+
does not match `7x2y0` and `[draft]` is not a character class unless you said so.
|
|
207
|
+
Note the edge: `-e` is a *pattern* language, so the literal wants `-e '7\.2\.0'`.
|
|
208
|
+
|
|
209
|
+
`Search.register` is the seam an addon plugs into — the second base-gem extension
|
|
210
|
+
point, deliberately shaped like the linter's — and
|
|
211
|
+
[the engine contract](../design/search-engines.md) is what keeps a registered
|
|
212
|
+
engine from redefining what a match is. `--engine` reads the registry at parse
|
|
213
|
+
time, so an addon appears in `okf search --help` without the CLI knowing it exists.
|
|
214
|
+
|
|
215
|
+
# It composes with the shared CLI surface
|
|
216
|
+
|
|
217
|
+
`--in FIELDS` restricts the searched fields; the `--type`/`--dir`/`--tag`
|
|
218
|
+
filters and `--fields`/`--except` projections shared with the
|
|
219
|
+
[read views](read-views.md) apply unchanged. It is an advisory read: exit `0`
|
|
220
|
+
even with zero matches — only an invalid `--regexp` pattern, `-e` paired with
|
|
221
|
+
`--fuzzy`, or an `--engine` that cannot honour a flag, is a usage error (exit `2`).
|
|
222
|
+
|
|
223
|
+
# One question, every bundle you keep
|
|
224
|
+
|
|
225
|
+
Knowledge rarely lives in one bundle, so search is the one verb that spans the
|
|
226
|
+
[registry](../registry.md): leading @slugs pick bundles explicitly
|
|
227
|
+
(`okf search @handbook @notes auth`), and `@all` is the ref that means every
|
|
228
|
+
registered one — including the bundles a [link](../registry.md) folds in, since
|
|
229
|
+
those are registered here in every sense that matters to a reader looking for an
|
|
230
|
+
answer. Every row is labeled by its bundle's slug.
|
|
231
|
+
|
|
232
|
+
**Merged rows are comparable by construction, and each engine earns that
|
|
233
|
+
differently.** The scan's score is absolute — summed field weights, with no
|
|
234
|
+
corpus term to move — so a row is worth exactly the same alone or beside two
|
|
235
|
+
other bundles. The index has no such luxury: BM25 prices a term by how rare it
|
|
236
|
+
is, so the bundles go into **one** index rather than N. A per-bundle index would
|
|
237
|
+
score the same match differently depending on where it came from, and
|
|
238
|
+
interleaving those lists would produce a ranking that looks sorted and compares
|
|
239
|
+
nothing. The visible consequence, under `--engine index` only, is that a score is
|
|
240
|
+
relative to the whole answer: the same concept is worth less searched beside two
|
|
241
|
+
other bundles than alone, because the term got commoner.
|
|
242
|
+
|
|
243
|
+
The graph stays per-bundle on purpose — cross-links are bundle-relative, so a
|
|
244
|
+
merged graph would be disconnected components — which makes search the one
|
|
245
|
+
cross-bundle question the CLI can answer honestly, and (for now) a capability the
|
|
246
|
+
[hub](graph-server.md) does not mirror.
|
|
247
|
+
|
|
248
|
+
**Asking for everything tolerates gaps; naming one bundle demands it.** `@all`
|
|
249
|
+
skips a registered bundle whose directory has vanished, with a note — the same
|
|
250
|
+
forgiveness the hub shows a stale entry — while `@handbook` fails hard, because
|
|
251
|
+
an explicit ask that silently answered about less than it named would be a
|
|
252
|
+
confident wrong answer. `@all @handbook` needs no diagnostic at all: all ⊇
|
|
253
|
+
handbook, so it expands, dedupes by resolved path, and answers.
|
|
254
|
+
|
|
255
|
+
That "every bundle" is a **ref rather than a flag** is what keeps the grammar
|
|
256
|
+
single, and it was not always so. A `--all` flag *reinterpreted the
|
|
257
|
+
positionals* — `okf search .okf home` read `.okf` as the bundle, `okf search
|
|
258
|
+
--all .okf` read it as a term — the same slot meaning opposite things, decided
|
|
259
|
+
by a flag optparse accepts anywhere in argv. Every diagnostic around it existed
|
|
260
|
+
to explain that flip. As a ref, slot 1 is always a bundle identity: a directory
|
|
261
|
+
there is a directory, a term after it is a term, and the explanations have
|
|
262
|
+
nothing left to explain. Being a ref also means being normalized like one:
|
|
263
|
+
`@ALL` reaches `@all` through the same `Registry.normalize` that makes `@One`
|
|
264
|
+
find dir `One`, because a ref exempt from the grammar's one normalization is a
|
|
265
|
+
trapdoor. Only `search` expands `@all`, since it is the only verb
|
|
266
|
+
that merges; see [the CLI](../cli.md) for why the others refuse it by name.
|
|
267
|
+
|
|
268
|
+
Three edges of the grammar, all deliberate. Any leading @-arg — even one —
|
|
269
|
+
switches the JSON envelope from `{ bundle, slug, … }` to
|
|
270
|
+
`{ bundles: [{ slug, dir }, …], …, matches: [{ slug, id, … }] }`, so a consumer
|
|
271
|
+
branches on the form it called; the head maps each slug to its dir once, which
|
|
272
|
+
is what lets a row resolve to `<dir>/<id>.md` without a second lookup while
|
|
273
|
+
keeping long paths off every row. Projection is literal: when merging, put
|
|
274
|
+
`slug` in `--fields` or the row label drops and same-id concepts from
|
|
275
|
+
different bundles become indistinguishable. And every leading @-arg is taken
|
|
276
|
+
as a ref, so a literal @-term (`@babel/core`) needs a non-@ term before it or
|
|
277
|
+
`-e '\@term'` — the CLI notes each of these traps on stderr when it sees one.
|
|
278
|
+
|
|
279
|
+
# Exact by default, fuzzy on request
|
|
280
|
+
|
|
281
|
+
No stemming, no synonyms, and no typo distance **unless asked**: `--fuzzy` turns
|
|
282
|
+
on an edit distance of `0.2 × term length`, the same tolerance the browser page
|
|
283
|
+
passes. The default stays exact because determinism is what keeps a result
|
|
284
|
+
citable — the consuming agent is the fuzzy layer, since synonyms and vocabulary
|
|
285
|
+
drift are judgment over the index map, not string distance.
|
|
286
|
+
|
|
287
|
+
Worth knowing when you reach for it: `--fuzzy` is not merely a mode, it is an
|
|
288
|
+
**engine switch**. It routes to the index, so everything on this page about token
|
|
289
|
+
matching applies to that run — including the recall holes. A typo forgiven and an
|
|
290
|
+
identifier shattered arrive together.
|
|
291
|
+
|
|
292
|
+
# The retrieval eval keeps the economics honest
|
|
293
|
+
|
|
294
|
+
The suite plants a fact in a fixture bundle and asserts that the progressive
|
|
295
|
+
path — index skeleton, one search, one body — answers it in **under 25% of the
|
|
296
|
+
bytes** of the full graph dump. The [companion skill](agent-skill.md)'s search
|
|
297
|
+
playbook rides that path, so its economics stay true by construction.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Capability
|
|
3
|
+
title: Conformance validator (validate)
|
|
4
|
+
description: Implements the spec's §11 conformance definition exactly — three hard conditions, everything else a machine-readable warning.
|
|
5
|
+
resource: gems/okf/lib/okf/bundle/validator.rb
|
|
6
|
+
tags: [conformance, cli]
|
|
7
|
+
generated:
|
|
8
|
+
by: human:maintainer
|
|
9
|
+
at: 2026-08-13T12:00:00Z
|
|
10
|
+
sources:
|
|
11
|
+
- title: gems/okf/lib/okf/bundle/validator.rb
|
|
12
|
+
resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/bundle/validator.rb
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Overview
|
|
16
|
+
|
|
17
|
+
`okf validate` answers one question: *is this a legal OKF (`@okf-eco format/okf-format`)
|
|
18
|
+
bundle?* `OKF::Bundle::Validator` implements §11 exactly and is the **only**
|
|
19
|
+
capability that can fail a bundle — exit `1` on any hard error, `0` otherwise.
|
|
20
|
+
|
|
21
|
+
# The three hard conditions (errors)
|
|
22
|
+
|
|
23
|
+
| Rule | Condition |
|
|
24
|
+
|------|-----------|
|
|
25
|
+
| §11 cond. 1 | every non-reserved file can be **read** and has a parseable frontmatter (`@okf-eco format/frontmatter`) block |
|
|
26
|
+
| §11 cond. 2 | every such block has a **non-empty `type`** |
|
|
27
|
+
| §11 cond. 3 | every `index.md` / `log.md` present is well-formed (nested index has no frontmatter, root index carries only `okf_version`, log dates are real ISO calendar days — `2026-02-30` matches the shape and is refused) |
|
|
28
|
+
|
|
29
|
+
A file that will not **open** fails condition 1 too, not only one whose frontmatter will
|
|
30
|
+
not parse: the reader keeps it in [`bundle.unparseable`](../model/bundle.md) with
|
|
31
|
+
its errno rather than letting one locked file abort the read, and `validate`
|
|
32
|
+
reports it there — one unusable file counted as one, named with why.
|
|
33
|
+
|
|
34
|
+
# Everything else is a warning
|
|
35
|
+
|
|
36
|
+
The validator is **forbidden by §11** from rejecting a bundle for soft issues, so
|
|
37
|
+
these are warnings that never change conformance:
|
|
38
|
+
|
|
39
|
+
- missing recommended fields, non-list `tags`, an unparseable `timestamp`;
|
|
40
|
+
- the shape of every §5/§10 family — `generated` not a mapping or missing `by`,
|
|
41
|
+
a non-integer `usage_count`, a `stale_after` that is not `YYYY-MM-DD`, a
|
|
42
|
+
missing `runtime` on an Attested Computation — read off the raw keys, never
|
|
43
|
+
the fallback-carrying accessors, so a pure v0.1 bundle validates silently;
|
|
44
|
+
- an `okf_version` the gem does not know (read best-effort under §12, compared
|
|
45
|
+
after `to_s.strip` because an unquoted `0.2` is a Psych Float);
|
|
46
|
+
- **broken cross-links (`@okf-eco format/cross-links`)** (§6.1) — consumers MUST
|
|
47
|
+
tolerate them.
|
|
48
|
+
|
|
49
|
+
Judging those is the [linter](linter.md)'s job, and keeping the two apart is a
|
|
50
|
+
[hard design contract](../design/core-shell-split.md). The
|
|
51
|
+
[writer](library-api.md) runs this validator *before* publishing, so a saved
|
|
52
|
+
bundle is never written non-conformant.
|
|
53
|
+
|
|
54
|
+
# Warnings are machine-readable
|
|
55
|
+
|
|
56
|
+
Every warning carries `check:` (a stable id) and `source:` — `:spec` when the
|
|
57
|
+
SPEC's own words state the rule, `:convention` for the shapes this gem asks for
|
|
58
|
+
beyond them (a tested constant pins the set). A consumer that wants only the
|
|
59
|
+
spec-normative warnings filters on `source` instead of string-matching
|
|
60
|
+
messages; errors keep their exact two-key `{ path, message }` shape.
|
data/.okf/cli.md
ADDED
|
@@ -0,0 +1,214 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Component
|
|
3
|
+
title: The okf command-line front end
|
|
4
|
+
description: The only layer that parses argv, prints, writes files, and decides exit codes.
|
|
5
|
+
resource: gems/okf/lib/okf/cli.rb
|
|
6
|
+
tags: [cli, shell, registry]
|
|
7
|
+
generated:
|
|
8
|
+
by: human:maintainer
|
|
9
|
+
at: 2026-08-13T12:00:00Z
|
|
10
|
+
sources:
|
|
11
|
+
- title: gems/okf/lib/okf/cli.rb
|
|
12
|
+
resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/cli.rb
|
|
13
|
+
- title: gems/okf/lib/okf/cli/command.rb
|
|
14
|
+
resource: https://github.com/serradura/okf/blob/main/gems/okf/lib/okf/cli/command.rb
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# Overview
|
|
18
|
+
|
|
19
|
+
`OKF::CLI` is the executable's front end and the single place where the gem
|
|
20
|
+
touches the outside world for a command: it parses `argv`, prints, writes files,
|
|
21
|
+
and chooses the exit code. Every library class beneath it just returns data — the
|
|
22
|
+
CLI is the [shell half](design/core-shell-split.md) of the architecture. Output
|
|
23
|
+
streams are injected (`out:`/`err:`) so the whole surface is driven in tests
|
|
24
|
+
without a real terminal or socket — which is what lets this layer, the product a
|
|
25
|
+
user actually touches, be [proven end to end](design/integration-first.md) rather
|
|
26
|
+
than by proxy. Even `--help` keeps that contract: rather than the `exit`
|
|
27
|
+
OptionParser's own handler would call — printing past those streams to the
|
|
28
|
+
process's stdout — each parser writes its banner to `out:` and throws `:help`
|
|
29
|
+
back to `run`, which returns the caught status, so a command's help is driven
|
|
30
|
+
and asserted like the command itself.
|
|
31
|
+
|
|
32
|
+
# Subcommands
|
|
33
|
+
|
|
34
|
+
Dispatch goes through a **registry**. Each verb is a `CLI::Command` subclass in
|
|
35
|
+
its own file under `okf/lib/okf/cli/`, registering itself at load; `cli.rb` looks the
|
|
36
|
+
name up and calls it. The require block at the bottom of `cli.rb` *is* the order
|
|
37
|
+
`okf help` lists them in, and a test pins that so the coupling cannot drift
|
|
38
|
+
unnoticed.
|
|
39
|
+
|
|
40
|
+
That registry is also the CLI's extension point (`@okf-eco design/extension-points`): a
|
|
41
|
+
gem shipping `okf/plugin.rb` adds a verb with no edit here, and it appears in the
|
|
42
|
+
map under `installed extensions:`. Discovery is lazy — a built-in never scans —
|
|
43
|
+
and a broken addon is reported on stderr rather than being fatal.
|
|
44
|
+
|
|
45
|
+
A verb's group is now something it *declares* — `.group`, one of the four
|
|
46
|
+
questions a command answers about itself — and `CLI::GROUPS` fixes the order they
|
|
47
|
+
print in. So this is the map `okf help` builds, not an editorial arrangement of
|
|
48
|
+
it; a group here that no command returns is a group nobody sees.
|
|
49
|
+
|
|
50
|
+
| Group | Verbs | Notes |
|
|
51
|
+
|-------|-------|-------|
|
|
52
|
+
| `:act` | `skill`, `server`, `render` | boot the [graph server](capabilities/graph-server.md) or write it as a [static file](capabilities/render.md); install the [agent skill](capabilities/agent-skill.md). |
|
|
53
|
+
| `:registry` | `registry` | one umbrella verb over its subcommands — curate the [bundle registry](registry.md), global or [project-local](registry.md#global-by-default-project-local-by-discovery). |
|
|
54
|
+
| `:judge` | `lint`, `loose`, `validate` | [validate](capabilities/validator.md) and [lint](capabilities/linter.md) answer different questions and stay separate. |
|
|
55
|
+
| `:read` | `search`, `index`, `dirs`, `stats`, `types`, `tags`, `files`, `references`, `catalog` | the [browser views as text](capabilities/read-views.md), plus the `index` map and [ranked search](capabilities/search.md). |
|
|
56
|
+
| `:graph` | `graph` | its own group because it is the whole model at once, not a view onto part of it. |
|
|
57
|
+
| `:extension` | *(whatever is installed)* | the only group with a printed heading — "where did this come from?" is a question only an addon raises. |
|
|
58
|
+
|
|
59
|
+
Plus `version` / `--version` / `-v` and `help` / `--help` / `-h`.
|
|
60
|
+
|
|
61
|
+
# `server` reads its mode from how many dirs you give it
|
|
62
|
+
|
|
63
|
+
One verb covers three intentions, and the argument count is the whole interface —
|
|
64
|
+
no `--hub` flag, no second verb:
|
|
65
|
+
|
|
66
|
+
| Invocation | Serves |
|
|
67
|
+
|------------|--------|
|
|
68
|
+
| `okf server <dir>` | that bundle at `/` — the classic single server |
|
|
69
|
+
| `okf server <dir> <dir>…` | those bundles behind a [hub](capabilities/graph-server.md), ephemerally (the first is the default); nothing is registered |
|
|
70
|
+
| `okf server` | the [registry](registry.md), its first entry still on disk at `/` |
|
|
71
|
+
|
|
72
|
+
Passing dirs never writes to the registry: an ad-hoc look at two bundles side by
|
|
73
|
+
side should not enrol them in the user's durable list. Registering is always the
|
|
74
|
+
explicit act of `okf registry set`.
|
|
75
|
+
|
|
76
|
+
`registry` is an umbrella verb — `init`, `list`, `set`, `del`, `default`,
|
|
77
|
+
`rename`, `group`, `ungroup`, `link`, `unlink`, `import` — over one persistent
|
|
78
|
+
file, and it
|
|
79
|
+
is **one row in the map**. It carried ten for a while, which made a third of `okf
|
|
80
|
+
help` about registry management when registry management is not a third of what
|
|
81
|
+
okf does; the row now points at `okf registry --help`, which prints the ten with
|
|
82
|
+
their grammar and the `-g` they share. The map's job is to name the verbs. A verb
|
|
83
|
+
whose own help is the manual is the same progressive disclosure the bundles are
|
|
84
|
+
built on, applied to the CLI's front page — and the row has to *say* the help
|
|
85
|
+
exists, since nothing else tells a reader an umbrella has subcommands at all. `$OKF_HOME` points every
|
|
86
|
+
one of them at a different global registry, which is what keeps the tests off the
|
|
87
|
+
real `~/.okf`; `registry init` and discovery add a [project-local](registry.md#global-by-default-project-local-by-discovery)
|
|
88
|
+
one that replaces it while you stand in its tree, with `OKF_NO_DISCOVERY=1` to
|
|
89
|
+
force the global one.
|
|
90
|
+
|
|
91
|
+
## One lever, not two
|
|
92
|
+
|
|
93
|
+
A *cross-cutting* lever is an env var, never a flag. An earlier design carried a
|
|
94
|
+
`--home DIR` flag, which had to be remembered on the three verbs that offered it
|
|
95
|
+
and forgotten on every other one — a flag whose whole job was to name a location
|
|
96
|
+
the env var already named. `$OKF_HOME` composes where a flag cannot: it reaches
|
|
97
|
+
every verb at once without being typed, it survives into a subprocess, and if the
|
|
98
|
+
directory ever holds more than `registry.json` it keeps meaning the same thing.
|
|
99
|
+
`OKF_NO_DISCOVERY` earns its keep the same way, and this is still why there is no
|
|
100
|
+
`--global` on `lint`, on `search`, or on the eleven others: a per-verb flag for
|
|
101
|
+
resolution would be fourteen places to thread it and fourteen to forget.
|
|
102
|
+
|
|
103
|
+
The `registry` umbrella is the exception, and the line is *subject*, not
|
|
104
|
+
convenience. Every other verb takes a bundle and merely happens to resolve a name
|
|
105
|
+
through a registry; `registry`'s argument **is** a registry file. Saying which one
|
|
106
|
+
is therefore an argument to that verb in the way `--as` is, not a lever bolted
|
|
107
|
+
across the CLI — so `-g`/`--global` lives on its subcommands (all but `init`,
|
|
108
|
+
whose whole job is to create a *local* file) and nowhere else. The env var is
|
|
109
|
+
still the right answer for a whole session or a CI job; the flag is the right
|
|
110
|
+
answer for one command, and for the plain fact that a capability reachable only
|
|
111
|
+
through an env var is one most people never discover. Both name the same
|
|
112
|
+
behavior, so neither can drift from the other.
|
|
113
|
+
|
|
114
|
+
# Every output names its bundle, in the identity the caller used
|
|
115
|
+
|
|
116
|
+
Two keys, one meaning each: `bundle` is always a directory, `slug` always a
|
|
117
|
+
registry slug. Name a bundle by [@slug](registry.md) and the answer comes back in
|
|
118
|
+
that identity — `OKF lint — @handbook (/path/to/one)`, and
|
|
119
|
+
`{ "bundle": "/path/to/one", "slug": "handbook", … }`. The point is an agent
|
|
120
|
+
holding several bundles at once: an output that names only a path forces it to
|
|
121
|
+
remember which invocation produced which answer, and remembering is where a
|
|
122
|
+
model confabulates. [Cross-bundle search](capabilities/search.md) goes further
|
|
123
|
+
and maps every slug to its dir in the head, so a hit resolves to a file with no
|
|
124
|
+
second call.
|
|
125
|
+
|
|
126
|
+
A bundle named by *path* carries no slug, deliberately. It may not have one, and
|
|
127
|
+
inventing a name it was never given would imply a registration that does not
|
|
128
|
+
exist — while looking one up would cost a registry read on every plain-dir run,
|
|
129
|
+
which is the laziness that keeps unregistered use free. The identity the caller
|
|
130
|
+
used is the identity they get back.
|
|
131
|
+
|
|
132
|
+
# @slug: the registry names bundles for every verb
|
|
133
|
+
|
|
134
|
+
Wherever a `<dir>` goes, `@slug` resolves a [registered bundle](registry.md)
|
|
135
|
+
and bare `@` the registry's default — one resolution seam (`resolve_ref`, shared
|
|
136
|
+
by the positional parsers and search's ref list), inherited by all verbs at once,
|
|
137
|
+
so `okf lint @handbook` works from any directory. They read the
|
|
138
|
+
[active registry](registry.md#global-by-default-project-local-by-discovery) — a
|
|
139
|
+
project-local one discovered from the working directory, else `$OKF_HOME` — which
|
|
140
|
+
is why the not-registered error names the registry file it consulted, so a
|
|
141
|
+
mismatch self-diagnoses rather than reading as "never registered".
|
|
142
|
+
|
|
143
|
+
A leading `@` always means the registry (`./@name` keeps an odd directory
|
|
144
|
+
reachable), the registry file loads only when a ref appears, and an explicit ask
|
|
145
|
+
fails hard: an unknown slug, a registered-but-gone directory, or a malformed
|
|
146
|
+
registry is a usage error whose message names the next move, never a silent skip.
|
|
147
|
+
The normalization is the subtle part — a slug is normalized exactly as
|
|
148
|
+
registration normalized it, so `@One` finds the bundle from dir `One`, but
|
|
149
|
+
*without* the placeholder that minting a slug from a basename needs: a name has
|
|
150
|
+
to come out of `slugify("!!!")`, and nothing may come out of a lookup, or `@***`
|
|
151
|
+
would quietly resolve to whatever bundle is slugged `bundle`.
|
|
152
|
+
|
|
153
|
+
A `server` `@slug` carries its registered slug to the mount, and reserves it before any
|
|
154
|
+
plain dir's basename is deduped — otherwise `server ./two @two` would hand `/b/two/`
|
|
155
|
+
to the *unregistered* directory and a bookmark would open the wrong graph. (The
|
|
156
|
+
first argument still lands at `/`; the registry's own order applies only to a
|
|
157
|
+
bundle-less run.) [`search`](capabilities/search.md) is the one verb that *merges*
|
|
158
|
+
several bundles into one answer — several @slugs, or `@all`.
|
|
159
|
+
|
|
160
|
+
`@all` is a ref, not a flag, and only `search` expands it. That restraint is the
|
|
161
|
+
point: through the shared seam, `okf lint @all` would resolve to one bundle when
|
|
162
|
+
one is registered (and lint it) and to two when two are (exit 2 by the
|
|
163
|
+
[second-bundle rule](#exit-codes)) — the same command's meaning tracking the size
|
|
164
|
+
of the registry, which is the silent-wrong-answer shape the rule exists to stop.
|
|
165
|
+
So every other verb refuses `@all` by name instead, and `all` is reserved as a
|
|
166
|
+
slug so the refusal can never be wrong.
|
|
167
|
+
|
|
168
|
+
# Exit codes
|
|
169
|
+
|
|
170
|
+
The contract every verb keeps:
|
|
171
|
+
|
|
172
|
+
| Code | Meaning |
|
|
173
|
+
|------|---------|
|
|
174
|
+
| `0` | success — including a bundle with lint findings (`lint` is advisory) |
|
|
175
|
+
| `1` | a non-conformant bundle (`validate`) or a `lint --fail-on warn` threshold crossed |
|
|
176
|
+
| `2` | usage error — unknown command, missing directory, bad flag, a bad `-o` path, or a *second* positional |
|
|
177
|
+
|
|
178
|
+
That last one is the subtle member: only [`search`](capabilities/search.md) merges
|
|
179
|
+
several bundles and only `server` mounts them, so a second bundle handed to any
|
|
180
|
+
other verb is a question it cannot answer. Reading the first and dropping the rest
|
|
181
|
+
would answer confidently about a bundle nobody asked about, which is why it is a
|
|
182
|
+
usage error rather than a convenience — the same reason a bad `-o` path is exit 2
|
|
183
|
+
and not a backtrace.
|
|
184
|
+
|
|
185
|
+
It is a rule about the **positional**, not about bundles, and stating it the
|
|
186
|
+
narrow way is how it came to be broken. `skill` takes a destination rather than a
|
|
187
|
+
`<dir>`, so it read as outside the rule, hand-rolled its own argument shift, and
|
|
188
|
+
accepted a second destination — installing into the first, ignoring the second,
|
|
189
|
+
exiting 0. It goes through the same shared pair as every other single-positional
|
|
190
|
+
verb now (`positional` for the value, `no_extras?` for what must not follow it),
|
|
191
|
+
because a guarantee that lives in a helper is only as wide as the verbs that
|
|
192
|
+
call it.
|
|
193
|
+
|
|
194
|
+
# Best-effort reads
|
|
195
|
+
|
|
196
|
+
`graph`, `server`, `render`, and the read views are best-effort under §11: a file
|
|
197
|
+
the reader cannot use is kept in `bundle.unparseable`, skipped, and *noted on
|
|
198
|
+
stderr* (so JSON on stdout stays clean) rather than aborting the whole command.
|
|
199
|
+
One bad file never breaks the rest. Run [validate](capabilities/validator.md) for
|
|
200
|
+
the details of what was skipped — the note counts, `validate` names each file and
|
|
201
|
+
why.
|
|
202
|
+
|
|
203
|
+
Two causes reach that bucket, and the tolerance has to cover both or it is not a
|
|
204
|
+
posture but a coincidence: frontmatter that will not **parse**, and a file that
|
|
205
|
+
will not **open** at all. The second was the gap — an unreadable file threw its
|
|
206
|
+
errno out of the reader, and since the read is the one path every verb shares, a
|
|
207
|
+
single locked file took the whole bundle down through all of them, as a backtrace
|
|
208
|
+
under an exit code that claims *non-conformant*. It is one unusable file. It
|
|
209
|
+
reports as one, under §11 condition 1, naming the file and the errno.
|
|
210
|
+
<!-- rule:okf-read-best-effort -->
|
|
211
|
+
|
|
212
|
+
The boundary: `Path.join_under!` still raises. A path leaving the bundle root is
|
|
213
|
+
not a bad file — it is a bundle lying about its shape, and best-effort is
|
|
214
|
+
tolerance for damage, never for a claim.
|