textus 0.4.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (95) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +147 -2
  3. data/README.md +38 -28
  4. data/SPEC.md +84 -147
  5. data/docs/architecture.md +82 -28
  6. data/lib/textus/builder/pipeline.rb +56 -0
  7. data/lib/textus/builder/renderer/json.rb +42 -0
  8. data/lib/textus/builder/renderer/markdown.rb +22 -0
  9. data/lib/textus/builder/renderer/text.rb +14 -0
  10. data/lib/textus/builder/renderer/yaml.rb +42 -0
  11. data/lib/textus/builder/renderer.rb +17 -0
  12. data/lib/textus/builder.rb +9 -114
  13. data/lib/textus/cli/group/hook.rb +11 -0
  14. data/lib/textus/cli/group/key.rb +12 -0
  15. data/lib/textus/cli/group/schema.rb +13 -0
  16. data/lib/textus/cli/group.rb +51 -0
  17. data/lib/textus/cli/verb/accept.rb +15 -0
  18. data/lib/textus/cli/verb/build.rb +13 -0
  19. data/lib/textus/cli/verb/delete.rb +16 -0
  20. data/lib/textus/cli/verb/deps.rb +12 -0
  21. data/lib/textus/cli/verb/doctor.rb +15 -0
  22. data/lib/textus/cli/verb/get.rb +12 -0
  23. data/lib/textus/cli/verb/hook_run.rb +48 -0
  24. data/lib/textus/cli/verb/hooks.rb +50 -0
  25. data/lib/textus/cli/verb/init.rb +14 -0
  26. data/lib/textus/cli/verb/intro.rb +11 -0
  27. data/lib/textus/cli/verb/list.rb +14 -0
  28. data/lib/textus/cli/verb/migrate_keys.rb +16 -0
  29. data/lib/textus/cli/verb/mv.rb +17 -0
  30. data/lib/textus/cli/verb/published.rb +11 -0
  31. data/lib/textus/cli/verb/put.rb +50 -0
  32. data/lib/textus/cli/verb/rdeps.rb +12 -0
  33. data/lib/textus/cli/verb/refresh.rb +15 -0
  34. data/lib/textus/cli/verb/schema.rb +12 -0
  35. data/lib/textus/cli/verb/schema_diff.rb +12 -0
  36. data/lib/textus/cli/verb/schema_init.rb +16 -0
  37. data/lib/textus/cli/verb/schema_migrate.rb +16 -0
  38. data/lib/textus/cli/verb/stale.rb +14 -0
  39. data/lib/textus/cli/verb/uid.rb +12 -0
  40. data/lib/textus/cli/verb/where.rb +12 -0
  41. data/lib/textus/cli/verb.rb +62 -0
  42. data/lib/textus/cli.rb +44 -385
  43. data/lib/textus/doctor/check/audit_log.rb +50 -0
  44. data/lib/textus/doctor/check/hooks.rb +29 -0
  45. data/lib/textus/doctor/check/illegal_keys.rb +49 -0
  46. data/lib/textus/doctor/check/manifest_files.rb +38 -0
  47. data/lib/textus/doctor/check/schema_violations.rb +22 -0
  48. data/lib/textus/doctor/check/schemas.rb +26 -0
  49. data/lib/textus/doctor/check/sentinels.rb +57 -0
  50. data/lib/textus/doctor/check/templates.rb +26 -0
  51. data/lib/textus/doctor/check/unowned_schema_fields.rb +34 -0
  52. data/lib/textus/doctor/check.rb +30 -0
  53. data/lib/textus/doctor.rb +29 -264
  54. data/lib/textus/entry/base.rb +30 -0
  55. data/lib/textus/entry/json.rb +11 -5
  56. data/lib/textus/entry/markdown.rb +5 -5
  57. data/lib/textus/entry/text.rb +4 -4
  58. data/lib/textus/entry/yaml.rb +11 -5
  59. data/lib/textus/entry.rb +2 -7
  60. data/lib/textus/envelope.rb +30 -0
  61. data/lib/textus/errors.rb +2 -2
  62. data/lib/textus/hooks/builtin.rb +70 -0
  63. data/lib/textus/hooks/dispatcher.rb +49 -0
  64. data/lib/textus/hooks/loader.rb +26 -0
  65. data/lib/textus/hooks/registry.rb +73 -0
  66. data/lib/textus/init.rb +14 -11
  67. data/lib/textus/intro.rb +16 -18
  68. data/lib/textus/key/distance.rb +55 -0
  69. data/lib/textus/key/grammar.rb +33 -0
  70. data/lib/textus/key/path.rb +17 -0
  71. data/lib/textus/manifest/entry.rb +199 -0
  72. data/lib/textus/manifest.rb +20 -254
  73. data/lib/textus/migrate_keys.rb +1 -1
  74. data/lib/textus/projection.rb +6 -5
  75. data/lib/textus/proposal.rb +4 -4
  76. data/lib/textus/refresh.rb +17 -17
  77. data/lib/textus/schema/tools.rb +89 -0
  78. data/lib/textus/store/audit_log.rb +71 -0
  79. data/lib/textus/store/mover.rb +121 -0
  80. data/lib/textus/store/reader.rb +67 -0
  81. data/lib/textus/store/staleness.rb +133 -0
  82. data/lib/textus/store/validator.rb +56 -0
  83. data/lib/textus/store/view.rb +29 -0
  84. data/lib/textus/store/writer.rb +132 -0
  85. data/lib/textus/store.rb +26 -527
  86. data/lib/textus/version.rb +2 -2
  87. data/lib/textus.rb +14 -29
  88. metadata +78 -8
  89. data/lib/textus/audit_log.rb +0 -32
  90. data/lib/textus/builtin_actions.rb +0 -68
  91. data/lib/textus/extension_registry.rb +0 -61
  92. data/lib/textus/extensions.rb +0 -33
  93. data/lib/textus/key_distance.rb +0 -53
  94. data/lib/textus/schema_tools.rb +0 -87
  95. data/lib/textus/store_view.rb +0 -27
data/SPEC.md CHANGED
@@ -1,7 +1,7 @@
1
- # textus/1 — Specification
1
+ # textus/2 — Specification
2
2
 
3
- **Status:** Draft v1.0 (2026-05-19)
4
- **Protocol identifier:** `textus/1`
3
+ **Status:** Draft v2.0 (2026-05-19)
4
+ **Protocol identifier:** `textus/2`
5
5
  **Reference implementation:** Ruby gem `textus`
6
6
 
7
7
  > *textus* — Latin for "the fabric a text is woven from," same root as *context* (from *con-texere*, "to weave together"). This spec defines a storage shape and wire protocol for that fabric.
@@ -31,7 +31,7 @@ textus is organized as five composable layers. Each layer has a single responsib
31
31
  ## 2. Goals and non-goals
32
32
 
33
33
  **Goals**
34
- - Stable wire format (`textus/1`) any language can speak.
34
+ - Stable wire format (`textus/2`) any language can speak.
35
35
  - Deterministic read/write of structured Markdown via a CLI returning JSON.
36
36
  - Schema-validated frontmatter using YAML schemas as data.
37
37
  - Role-based write gates (humans, scripts, AI, build runners get different permissions per zone).
@@ -91,7 +91,7 @@ The manifest declares: (a) which zones exist and which roles may write to each,
91
91
 
92
92
  ```yaml
93
93
  # .textus/manifest.yaml
94
- version: textus/1
94
+ version: textus/2
95
95
 
96
96
  zones:
97
97
  - name: canon
@@ -139,7 +139,7 @@ zones:
139
139
 
140
140
  Old manifests written against textus/1 draft v0.1 therefore parse without modification, and any tooling expecting `fixed`/`state`/`derived` continues to work.
141
141
 
142
- **Key grammar (enforced from v1.2):** dotted segments matching `/^[a-z0-9][a-z0-9-]*$/`. Segments are joined by `.`. A key has at most 8 segments; each segment is at most 64 characters. Segments MUST NOT contain dots, slashes, uppercase letters, or underscores. Example: `working.projects.acme.dashboard`. Enforcement points: manifest load (rejects illegal `key:` declarations and illegal nested file/directory names), `put` (rejects illegal keys before any write), `enumerate` (filters and warns on illegal filenames so existing trees still load with a clear migration message). Run-once migration: `textus migrate-keys --dry-run` then `--write` (see §audit).
142
+ **Key grammar (enforced from v1.2):** dotted segments matching `/^[a-z0-9][a-z0-9-]*$/`. Segments are joined by `.`. A key has at most 8 segments; each segment is at most 64 characters. Segments MUST NOT contain dots, slashes, uppercase letters, or underscores. Example: `working.projects.acme.dashboard`. Enforcement points: manifest load (rejects illegal `key:` declarations and illegal nested file/directory names), `put` (rejects illegal keys before any write), `enumerate` (filters and warns on illegal filenames so existing trees still load with a clear migration message). Run-once migration: `textus key migrate --dry-run` then `--write` (see §audit).
143
143
 
144
144
  **Per-entry `format:` (enforced from v1.2):** an entry MAY declare `format:` to be one of `markdown` (default), `json`, `yaml`, or `text`. The `format` controls the on-disk shape and which path extension is required:
145
145
 
@@ -250,38 +250,36 @@ A sentinel is written for each published file at `<store_root>/sentinels/<target
250
250
 
251
251
  **Per-leaf publishing.** A nested entry MAY declare `publish_each:` instead of `publish_to:` (see §4). When the build runs, every leaf reachable under the nested entry is byte-copied to the path produced by substituting `{leaf}` / `{basename}` / `{key}` / `{ext}` in the template, with a sentinel written under `<store_root>/sentinels/` at the mirrored target path. The build envelope grows a `published_leaves` array — one row per leaf, with `key`, `source`, and `target` — alongside the existing `built` array. Targets that would resolve outside the repo root are refused.
252
252
 
253
- ### 5.4 Intake (declared, refreshed via registered action)
253
+ ### 5.4 Intake (declared, refreshed via registered fetch hook)
254
254
 
255
- Intake entries declare an external source by naming an **action** — a registered, named function that pulls data into the entry. textus itself still makes no implicit network calls: an action only runs in intake mode when explicitly invoked by `textus refresh KEY --as=script`. The declaration is data only:
255
+ Intake entries declare an external source by naming a **fetch hook** — a registered, named function that pulls data into the entry. textus itself still makes no implicit network calls: a fetch hook only runs in intake mode when explicitly invoked by `textus refresh KEY --as=script`. The declaration is data only:
256
256
 
257
257
  ```yaml
258
258
  - key: intake.calendar.events
259
259
  zone: intake
260
260
  source:
261
- action: ical-events
261
+ fetch: ical-events
262
262
  config:
263
263
  url: "https://calendar.google.com/.../basic.ics"
264
264
  ttl: 6h
265
265
  ```
266
266
 
267
- `action` names a registered action; `config` is an opaque hash handed to the action; `ttl` is the staleness budget. Implementations MUST reject legacy `source.from`, `source.parse`, and `source.fetcher` with a clear usage error.
267
+ `fetch` names a registered `:fetch` hook (see §5.10 for the hook contract); `config` is an opaque hash handed to the hook; `ttl` is the staleness budget. Implementations MUST reject legacy `source.from`, `source.parse`, `source.fetcher`, and `source.action` with a clear usage error.
268
268
 
269
- **Action contract (intake mode).** An action is registered via `Textus.action(:name) do |config:, store:, args:| ... end`. In intake mode the action MUST return one of three shapes, all normalized by the store into its internal `{frontmatter, body, content}` representation (§5.12):
269
+ In intake mode the hook MUST return one of three shapes, all normalized by the store into its internal `{_meta, body, content}` representation (§5.12):
270
270
 
271
- - `{ frontmatter:, body: }` — markdown-friendly.
271
+ - `{ _meta:, body: }` — markdown-friendly; `_meta` becomes the entry's parsed metadata hash.
272
272
  - `{ content: }` — for `format: json|yaml` entries; the parsed object becomes the entry's content.
273
273
  - `{ body: }` — raw bytes for `text` or for any format that prefers verbatim writes; the store re-parses and validates per `format:`.
274
274
 
275
- The `store:` argument is a writable `Textus::StoreView` (§5.11) bound to the calling role; the `args:` argument is `{}` in intake mode (it carries CLI flags in verb mode§5.11). Every action call is wrapped in `Timeout.timeout(2)`; exceptions and timeouts surface as `usage` errors that abort the refresh.
276
-
277
- **Built-in actions.** `json`, `csv`, `markdown-links`, `ical-events`, `rss` are always available. They expect raw bytes in `config["bytes"]` and produce structured frontmatter/body. Built-ins do not perform I/O themselves — the caller (or an outer action) is responsible for supplying bytes.
275
+ **Built-in fetch hooks.** `json`, `csv`, `markdown-links`, `ical-events`, `rss` are always available. They expect raw bytes in `config["bytes"]` and produce structured `_meta`/body. Built-ins do not perform I/O themselvesthe caller (or an outer hook) is responsible for supplying bytes.
278
276
 
279
277
  **Refresh paths.** Two are supported:
280
278
 
281
- 1. **In-process** — `textus refresh KEY --as=script` resolves the entry's `source.action`, invokes it with `(config:, store:, args: {})`, and writes the result under role `script`.
279
+ 1. **In-process** — `textus refresh KEY --as=script` resolves the entry's `source.fetch`, invokes the registered `:fetch` hook with `(config:, store:, args: {})`, and writes the result under role `script`.
282
280
  2. **External runner** — a cron job or agent harness reads `textus list --zone=intake --stale --format=json`, fetches the source out of band, and pipes bytes back through `textus put KEY --as=script --stdin`.
283
281
 
284
- Both paths share the same role gate, audit-log entry, and `:refresh` event (§5.10). User-supplied actions live in `.textus/extensions/*.rb` and auto-load at `Store#initialize` (§5.11).
282
+ Both paths share the same role gate, audit-log entry, and `:refresh` event. User-supplied hooks live in `.textus/hooks/*.rb` and auto-load at `Store#initialize` — see §5.10 for the full hook contract.
285
283
 
286
284
  ### 5.5 Pending / accept workflow
287
285
 
@@ -306,15 +304,19 @@ Proposed body content.
306
304
 
307
305
  ### 5.6 Audit log
308
306
 
309
- Every successful write appends one line to an append-only TSV file at `.textus/audit.log`. The file is opened with `flock(LOCK_EX)` for the duration of each append so concurrent writers serialize cleanly.
307
+ Every successful write appends one compact JSON object (NDJSON) to `.textus/audit.log`. The file is opened with `flock(LOCK_EX)` for the duration of each append so concurrent writers serialize cleanly.
310
308
 
311
- Schema (tab-separated, one record per line):
309
+ Schema (one JSON object per line, no interior whitespace):
312
310
 
311
+ ```json
312
+ {"ts":"<iso8601-utc>","role":"<role>","verb":"<verb>","key":"<key>","etag_before":<etag-or-null>,"etag_after":<etag-or-null>}
313
313
  ```
314
- <iso8601-utc>\t<role>\t<verb>\t<key>\t<etag-before-or-NULL>\t<etag-after-or-NULL>
315
- ```
316
314
 
317
- `<iso8601-utc>` is the wall-clock timestamp in UTC with second (or finer) precision. `<role>` is the resolved role for the invocation. `<verb>` is the CLI verb (`put`, `delete`, `accept`, `compute`, `migrate-keys`, `mv`, ...). `<key>` is the affected entry key. For `mv`, `<key>` is the **new** key, and an extras JSON column carries `from_key`, `to_key`, `from_path`, `to_path`, and `uid`. `<etag-before>` and `<etag-after>` are the entry etags before and after the write, or the literal string `NULL` when not applicable (e.g. create has no before-etag, delete has no after-etag). `migrate-keys --write` emits one line per renamed file using the new key as `<key>` and the file's pre- and post-rename etags.
315
+ `ts` is the wall-clock timestamp in UTC with second precision. `role` is the resolved role for the invocation. `verb` is the audit-log payload string identifying the operation (`put`, `delete`, `accept`, `compute`, `migrate-keys`, `mv`, ...). Note that `migrate-keys` here is the on-disk payload key the CLI surface is `textus key migrate`; the payload string is retained for log stability. `key` is the affected entry key. `etag_before` and `etag_after` are the entry etags before and after the write, or JSON `null` when not applicable (e.g. create has no before-etag, delete has no after-etag). `key migrate --write` emits one line per renamed file (with payload `verb: "migrate-keys"`) using the new key as `key` and the file's pre- and post-rename etags.
316
+
317
+ For `mv`, the structural fields `from_key`, `to_key`, and `uid` appear at the top level of the JSON object. Remaining verb-specific data (e.g. `from_path`, `to_path`) is nested under an `extras` key. The `extras` key is omitted entirely when empty.
318
+
319
+ **Backward compatibility (v0.5):** files written by v0.4 and earlier contain TSV rows. Readers MUST accept mixed-format files: lines starting with `{` are parsed as JSON; other lines are treated as legacy TSV (`ts\trole\tverb\tkey\tetag_before\tetag_after[\tjson_extras]`). TSV write support is removed in v0.6.
318
320
 
319
321
  ### 5.7 Security bounds
320
322
 
@@ -351,118 +353,51 @@ evolution:
351
353
  OLD_FIELD: NEW_FIELD
352
354
  ```
353
355
 
354
- `textus schema-migrate NAME` consults `evolution.migrate_from` when invoked without `--rename=OLD:NEW`, applying every declared rename across affected entries in one pass. An explicit `--rename` flag overrides the schema-declared map for that invocation.
356
+ `textus schema migrate NAME` consults `evolution.migrate_from` when invoked without `--rename=OLD:NEW`, applying every declared rename across affected entries in one pass. An explicit `--rename` flag overrides the schema-declared map for that invocation.
355
357
 
356
358
  **Backwards compat:** v1.0 schemas (no `fields:`, no `evolution:`) continue to parse and behave identically. `schema.maintained_by(field)` returns `nil` for every field; `schema.evolution` returns `{}`.
357
359
 
358
- **Override rule:** the role `human` is permitted to write any `maintained_by` field, regardless of declared owner. This preserves human authority over AI/script-managed data — humans curating canon over AI-written embeddings is a feature, not a bug. All other role mismatches are reported by `validate-all` with code `role_authority`, including fields `key`, `field`, `expected`, and `last_writer`.
360
+ **Override rule:** the role `human` is permitted to write any `maintained_by` field, regardless of declared owner. This preserves human authority over AI/script-managed data — humans curating canon over AI-written embeddings is a feature, not a bug. All other role mismatches are reported by `doctor --check=schema_violations` with code `role_authority`, including fields `key`, `field`, `expected`, and `last_writer`.
359
361
 
360
362
  ### 5.9 Reducers (v1.2)
361
363
 
362
- Reducers are pure, named functions that shape projection rows into projection rows. Registered via the module-level DSL:
364
+ Reducers are RPC hooks on the `:reduce` event. See §5.10.
363
365
 
364
- ```ruby
365
- Textus.reducer(:rank_by_recency) do |rows:, config:|
366
- rows.sort_by { |r| r["updated_at"].to_s }.reverse
367
- end
368
- ```
369
-
370
- **Declaration.** A projection opts into a reducer via `projection.reducer`, with optional `projection.reducer_config`:
366
+ ### 5.10 Hooks
371
367
 
372
- ```yaml
373
- projection:
374
- select: [working.projects]
375
- pluck: [name, status, updated_at]
376
- reducer: rank_by_recency
377
- reducer_config: { tiebreak: name }
378
- sort_by: updated_at
379
- limit: 50
380
- ```
368
+ textus has a single extension verb: `Textus.hook(event, name, **opts) { ... }`. The EVENTS table below defines every extension point. Files in `.textus/hooks/*.rb` are `load`ed at `Store#initialize` in lexical order.
381
369
 
382
- The reducer runs **between pluck and sort**. `config:` receives the manifest's `reducer_config` hash (or `{}`). Rows in, rows out.
370
+ | Event | Mode | Args | Return | Failure |
371
+ |----------|---------|-----------------------------------|---------------|---------|
372
+ | :fetch | rpc | store:, config:, args: | {_meta:, body:} | aborts op |
373
+ | :reduce | rpc | store:, rows:, config: | rows array | aborts op |
374
+ | :check | rpc | store: | issues array | aborts doctor |
375
+ | :put | pubsub | store:, key:, envelope: | (discarded) | logged |
376
+ | :delete | pubsub | store:, key: | (discarded) | logged |
377
+ | :refresh | pubsub | store:, key:, envelope:, change: | (discarded) | logged |
378
+ | :build | pubsub | store:, key:, envelope:, sources: | (discarded) | logged |
379
+ | :accept | pubsub | store:, key:, target_key: | (discarded) | logged |
383
380
 
384
- **Purity.** A reducer MUST NOT perform I/O or mutate the store; no `store:` kwarg is passed.
381
+ **Signature invariant** every hook receives `store:` as its first keyword argument. Event-specific kwargs follow in stable left-to-right order. The primary entity is always `key:` (for `:accept`, `key:` is the pending key being accepted and `target_key:` is the destination).
385
382
 
386
- **Timeout.** Each invocation is wrapped in `Timeout.timeout(Textus::Projection::REDUCER_TIMEOUT_SECONDS)` (2s). Timeouts, exceptions, and unknown names raise `usage` errors and abort the build.
383
+ **RPC mode** exactly one handler per (event, name). The manifest references the handler by name (`source.fetch: NAME`, `projection.reduce: NAME`). Failure or timeout aborts the calling operation.
387
384
 
388
- **Auto-load.** Reducers register from `.textus/extensions/*.rb`, loaded at `Store#initialize` in lexical order (§5.11). The registry is per-Store; reducers do not share state across `Store` instances.
385
+ **Pub-sub mode** zero or more handlers per event. All matching handlers fire. The `keys:` option restricts a handler to keys matching one of the given globs (`File.fnmatch?` rules). Absence of `keys:` fires on every event of that type. Handler failures and 2s timeouts are logged to `audit.log` as `event_error` rows; they NEVER abort the triggering operation.
389
386
 
390
- ### 5.10 Events (v1.2)
387
+ The `store:` argument is always a read-only store proxy. Write attempts raise `UsageError`.
391
388
 
392
- Lifecycle events fire in-process. Subscribers register via `Textus.hook(:event, :name) do |**kwargs| ... end`. Hooks are fire-and-forget: return values are discarded.
393
-
394
- **Event set and kwargs:**
395
-
396
- | Event | Fired by | Kwargs |
397
- |-----------|-------------------------|--------------------------------------------------------------|
398
- | `:put` | `Store#put` | `key:, envelope:, store:` |
399
- | `:delete` | `Store#delete` | `key:, store:` |
400
- | `:refresh`| `Refresh.call` | `key:, envelope:, store:, change:` (`:created` or `:updated`)|
401
- | `:build` | `Builder#materialize` | `key:, envelope:, store:, sources:` |
402
- | `:accept` | `Proposal.accept` | `pending_key:, target_key:, store:` |
403
-
404
- `:refresh` with `change: :unchanged` does NOT fire — only `:created` and `:updated` are emitted. The `store:` kwarg is always a `Textus::StoreView` (§5.11).
405
-
406
- **Timeout and isolation.** Each hook runs under `Timeout.timeout(2)`. Hook errors and timeouts are recorded as `event_error` rows in `.textus/audit.log` (column 7, JSON-encoded extras with `event`, `hook`, `error`) but do NOT abort the triggering operation. The store write that fired the event is already committed by the time hooks run.
407
-
408
- **Manifest declarations.** A manifest entry MAY declare external-runner hooks under an `events:` block, keyed by event name:
409
-
410
- ```yaml
411
- events:
412
- refresh:
413
- - { exec: scripts/reindex.sh, as: script }
414
- build:
415
- - { exec: scripts/rebuild-index.sh, as: build }
416
- ```
417
-
418
- Textus does NOT invoke these — they surface only through `textus extensions list --kind=hook` for orchestrators (lefthook, cron, CI) to consume. Each entry has `exec` (opaque runner-resolvable string) and `as` (role to claim, defaults to `script`).
419
-
420
- **Removed.** The v1.1 `on_stale` event is removed in 0.2. Staleness is a poll, surfaced by `textus stale`. The `on_`-prefix convention from v1.1 is gone; events are bare symbols.
421
-
422
- ### 5.11 Extension surface (v1.3)
423
-
424
- Four DSL verbs cover all user-supplied code:
425
-
426
- ```
427
- Textus.action(:name) do |config:, store:, args:| ... end # intake mode: returns content; verb mode: writes via store.put
428
- Textus.reducer(:name) do |rows:, config:| ... end # returns rows
429
- Textus.hook(:event, :name) do |**kwargs| ... end # side effects; return ignored
430
- Textus.doctor_check(:name) do |store:| ... end # returns array of issue hashes
431
- ```
432
-
433
- Files in `.textus/extensions/*.rb` are loaded at `Store#initialize`, in lexical order, with the registry installed as the current registry for that store. Registries are per-Store: two Store instances in the same process do not share state.
434
-
435
- **Action invocation modes.**
436
-
437
- | Mode | Invoked by | `config:` | `store:` | `args:` | Return |
438
- |---------|---------------------------|------------------------|------------------------|----------------------|------------------------------------------|
439
- | intake | `textus refresh KEY` | manifest `source.config` | writable view (role from `--as`) | `{}` | required; normalized into entry write |
440
- | verb | `textus action NAME ...` | `{}` | writable view (role from `--as`) | parsed CLI kv hash | ignored |
441
- | put-fetch | `textus put K --action=N --stdin` | `{ "bytes" => stdin }` | read-only view | `{}` | required; merged into the put payload |
442
-
443
- **Failure modes:**
444
-
445
- | Surface | Timeout | Exception | Bad return |
446
- |-----------------|------------|---------------------------------------------|------------|
447
- | action | aborts op | aborts op (wrapped as `UsageError`) | aborts op |
448
- | reducer | aborts op | aborts op | aborts op |
449
- | hook | logged | logged (audit `event_error` row) | n/a |
450
- | doctor_check | reported as `doctor_check.timeout` issue | reported as `doctor_check.failed` issue | reported as `doctor_check.bad_return` |
451
-
452
- Actions and reducers are pure transforms in modes where their return matters; their return values flow into the store. Hooks and doctor_checks shape side outputs (event chain, doctor report) — only doctor_check return values are merged.
453
-
454
- The `store:` argument is always a `Textus::StoreView`. In intake and verb modes it is writable and bound to the calling role; in put-fetch mode and inside doctor_checks it is read-only. Write attempts on a read-only view raise `Textus::UsageError`.
389
+ Each handler runs under `Timeout.timeout(2)`.
455
390
 
456
391
  ### 5.12 Storage formats (v1.2)
457
392
 
458
- An entry's `format:` selects a storage strategy. All strategies expose the same `parse(bytes) → {frontmatter, body, content}` and `serialize(frontmatter:, body:, content:) → bytes` contract. The store, audit, etag, and projection layers operate on the parsed shape; only (de)serialization differs.
393
+ An entry's `format:` selects a storage strategy. All strategies expose the same `parse(bytes) → {_meta, body, content}` and `serialize(meta:, body:, content:) → bytes` contract. The store, audit, etag, and projection layers operate on the parsed shape; only (de)serialization differs.
459
394
 
460
- - **markdown** — YAML frontmatter between `---` fences, free-form body. Parse: Psych `safe_load` on the front matter; body is the remainder. Serialize: emit `---\n<yaml>\n---\n<body>`. `content` is always `nil`.
461
- - **json** — entire file is a JSON document. Parse: `JSON.parse`. Serialize: `JSON.pretty_generate(content)` + trailing newline. `frontmatter` is populated from a top-level `_meta` hash (if present, else `{}`); `body` is the raw bytes; `content` is the parsed object.
462
- - **yaml** — entire file is a YAML mapping. Parse: `YAML.safe_load(bytes, permitted_classes: [Date, Time], aliases: false)`; anchors/aliases rejected. Serialize: `YAML.dump(content).sub(/\A---\n/, "")`. Same `_meta` / `frontmatter` / `body` / `content` rules as JSON.
463
- - **text** — raw UTF-8 bytes. Parse: body is the file verbatim, `frontmatter` is `{}`, `content` is `nil`. Serialize: write `body` bytes (with trailing newline if missing).
395
+ - **markdown** — YAML frontmatter between `---` fences, free-form body. Parse: Psych `safe_load` on the frontmatter block; body is the remainder. Serialize: emit `---\n<yaml>\n---\n<body>`. `content` is always `nil`. `_meta` holds the parsed frontmatter hash.
396
+ - **json** — entire file is a JSON document. Parse: `JSON.parse`. Serialize: `JSON.pretty_generate(content)` + trailing newline. `_meta` is populated from the top-level `_meta` hash (if present, else `{}`); `body` is the raw bytes; `content` is the parsed object with `_meta` stripped.
397
+ - **yaml** — entire file is a YAML mapping. Parse: `YAML.safe_load(bytes, permitted_classes: [Date, Time], aliases: false)`; anchors/aliases rejected. Serialize: `YAML.dump(content).sub(/\A---\n/, "")`. Same `_meta` / `body` / `content` rules as JSON.
398
+ - **text** — raw UTF-8 bytes. Parse: body is the file verbatim, `_meta` is `{}`, `content` is `nil`. Serialize: write `body` bytes (with trailing newline if missing).
464
399
 
465
- **Envelope shape.** Every envelope carries `format:` (always present, defaults to `markdown` for back-compat). For `json|yaml`, the envelope additionally carries `content:` (parsed object). `body` is always the raw on-disk bytes. `frontmatter` always exists, and for `json|yaml` mirrors the `_meta` block (`{}` if absent). `text` always has `frontmatter: {}` and no `content`.
400
+ **Envelope shape.** Every envelope carries `format:` (always present, defaults to `markdown` for back-compat). For `json|yaml`, the envelope additionally carries `content:` (parsed object). `body` is always the raw on-disk bytes. `_meta` always exists in the envelope: for `markdown` it holds the parsed YAML frontmatter; for `json|yaml` it mirrors the top-level `_meta` block (`{}` if absent); for `text` it is `{}`.
466
401
 
467
402
  **`_meta` convention.** Derived structured entries (json, yaml) embed a `_meta` hash as the first top-level key. Builder-injected keys appear in a fixed order for etag stability:
468
403
 
@@ -529,13 +464,13 @@ Every successful CLI response (`--format=json`) is a single JSON envelope:
529
464
 
530
465
  ```json
531
466
  {
532
- "protocol": "textus/1",
467
+ "protocol": "textus/2",
533
468
  "key": "working.network.org.jane",
534
469
  "zone": "working",
535
470
  "owner": "textus:network",
536
471
  "path": "/absolute/path/to/.textus/zones/working/network/org/jane.md",
537
472
  "format": "markdown",
538
- "frontmatter": { "name": "jane", "relationship": "peer", "org": "acme" },
473
+ "_meta": { "name": "jane", "relationship": "peer", "org": "acme" },
539
474
  "body": "Short body in Markdown.\n",
540
475
  "etag": "sha256:8f3c…",
541
476
  "schema_ref": "person",
@@ -544,13 +479,13 @@ Every successful CLI response (`--format=json`) is a single JSON envelope:
544
479
  ```
545
480
 
546
481
  **Field rules:**
547
- - `protocol` MUST be the exact string `textus/1`.
482
+ - `protocol` MUST be the exact string `textus/2`.
548
483
  - `key` MUST be the canonical resolved key.
549
484
  - `zone` MUST be one of the zones declared in the manifest (`canon`, `working`, `intake`, `pending`, `derived` for the default v1.0 model; legacy v0.1 manifests synthesize `fixed`, `state`, `derived` per §4).
550
485
  - `path` MUST be an absolute filesystem path.
551
486
  - `format` MUST be one of `markdown`, `json`, `yaml`, `text` (§5.12). Absent envelopes are treated as `markdown` for back-compat.
552
487
  - `body` is the raw on-disk bytes as a UTF-8 string for every format.
553
- - `content` is present only when `format` is `json` or `yaml`; equals the parsed object. For `json|yaml`, `frontmatter` mirrors the top-level `_meta` (or `{}` if absent).
488
+ - `content` is present only when `format` is `json` or `yaml`; equals the parsed object. For `json|yaml`, `_meta` mirrors the top-level `_meta` block (or `{}` if absent). For `markdown`, `_meta` holds the parsed YAML frontmatter. For `text`, `_meta` is `{}`.
554
489
  - `etag` MUST be `sha256:<hex>` of the raw file bytes, computed identically for every format.
555
490
  - `schema_ref` MAY be `null` for entries in subtrees with `schema: null`.
556
491
  - `uid` is the stable Textus UID (§7) if the entry carries one, else `null`. Always present in the envelope.
@@ -559,7 +494,7 @@ Errors use a distinct envelope:
559
494
 
560
495
  ```json
561
496
  {
562
- "protocol": "textus/1",
497
+ "protocol": "textus/2",
563
498
  "ok": false,
564
499
  "code": "write_forbidden",
565
500
  "message": "zone 'canon' is not writable by role 'ai' for key 'canon.identity'",
@@ -590,29 +525,29 @@ All verbs accept `--format=json` and emit a canonical envelope (success or error
590
525
  | `list [--prefix=K] [--zone=Z] [--stale]` | read | any |
591
526
  | `where K` | read | any |
592
527
  | `get K` | read | any |
593
- | `schema K` | read | any |
528
+ | `schema show K` | read | any |
594
529
  | `stale [--prefix=K] [--strict]` | read | any |
595
530
  | `deps K` / `rdeps K` | read | any |
596
531
  | `published` | read | any |
597
- | `validate-all` | read | any |
598
- | `put K --stdin --as=R [--action=NAME]` | write | per zone |
532
+ | `hook list` | read | any |
533
+ | `doctor [--check=NAME[,NAME]] [--format=json]` | read | any |
534
+ | `intro [--format=json]` | read | any |
535
+ | `put K --stdin --as=R [--fetch=NAME]` | write | per zone |
599
536
  | `delete K --if-etag=E --as=R` | write | per zone |
600
537
  | `refresh K --as=script` | write | per zone (typically `script`) |
601
538
  | `build [--prefix=K] [--dry-run]` | write | `build` (default) |
602
539
  | `accept K --as=human` | write | `human` |
603
540
  | `init` | write | `human` |
604
- | `schema-init NAME` / `schema-diff NAME` / `schema-migrate NAME --rename=OLD:NEW` | write | `human` |
605
- | `migrate-keys [--dry-run\|--write]` | write (with `--write`) | `human` |
606
- | `mv OLD NEW [--as=R] [--dry-run]` | write | per zone (same-zone only) |
607
- | `uid K` | read | any |
608
- | `extensions list [--kind=action\|reducer\|hook]` | read | any |
609
- | `doctor [--format=json]` | read | any |
610
- | `intro [--format=json]` | read | any |
541
+ | `schema init NAME` / `schema diff NAME` / `schema migrate NAME [--rename=OLD:NEW]` | write | `human` |
542
+ | `key migrate [--dry-run\|--write]` | write (with `--write`) | `human` |
543
+ | `key mv OLD NEW [--as=R] [--dry-run]` | write | per zone (same-zone only) |
544
+ | `key uid K` | read | any |
545
+ | `hook run NAME` | write | any |
611
546
 
612
547
  **`put` input** (read from stdin when `--stdin` is given):
613
548
 
614
549
  ```json
615
- { "frontmatter": { "name": "jane", "relationship": "peer", "org": "acme" },
550
+ { "_meta": { "name": "jane", "relationship": "peer", "org": "acme" },
616
551
  "body": "Short body.\n",
617
552
  "if_etag": "sha256:8f3c…" }
618
553
  ```
@@ -637,7 +572,7 @@ All verbs accept `--format=json` and emit a canonical envelope (success or error
637
572
 
638
573
  `textus init` scaffolds a fresh `.textus/` tree (manifest, zones, schemas, audit log) under the current directory with a default manifest. Customize by editing `.textus/manifest.yaml` after init.
639
574
 
640
- `textus schema-init NAME` writes a stub schema. `schema-diff NAME` compares the on-disk schema against entries that claim it and prints the deltas. `schema-migrate NAME --rename=OLD:NEW` rewrites the frontmatter key `OLD` to `NEW` across every entry that uses the named schema, in a single transactional sweep that logs each touched file.
575
+ `textus schema show K` prints the schema for entry `K`. `textus schema init NAME` writes a stub schema. `textus schema diff NAME` compares the on-disk schema against entries that claim it and prints the deltas. `textus schema migrate NAME --rename=OLD:NEW` rewrites the `_meta` key `OLD` to `NEW` across every entry that uses the named schema, in a single transactional sweep that logs each touched file.
641
576
 
642
577
  ## 10. ETag semantics
643
578
 
@@ -649,16 +584,17 @@ Every `Textus::Error` exposes `code`, `message`, and an optional `hint:`. The hi
649
584
 
650
585
  ## 10.2 `textus doctor`
651
586
 
652
- `textus doctor` returns a health-check envelope: `{ "protocol": "textus/1", "ok": bool, "issues": [...], "summary": {error, warning, info} }`. Each issue carries `code`, `level` (`error|warning|info`), `subject`, `message`, and optionally `fix`. `ok` is true iff no error-level issues are present; warnings and info do not flip the bit. Checks include manifest sanity, missing schemas/templates, extension load failures, illegal nested keys (with proposed normalisation), sentinel drift/orphans, and audit-log line corruption. Exit code is 0 on `ok`, 1 otherwise.
587
+ `textus doctor` returns a health-check envelope: `{ "protocol": "textus/2", "ok": bool, "issues": [...], "summary": {error, warning, info} }`. Each issue carries `code`, `level` (`error|warning|info`), `subject`, `message`, and optionally `fix`. `ok` is true iff no error-level issues are present; warnings and info do not flip the bit. Builtin checks: `manifest_files`, `schemas`, `templates`, `hooks`, `illegal_keys`, `sentinels`, `audit_log`, `unowned_schema_fields`, `schema_violations`. Additional registered `:check` hooks (§5.10) run after the builtin set. Exit code is 0 on `ok`, 1 otherwise.
653
588
 
654
589
  ## 11. Versioning
655
590
 
656
- - The wire string `textus/1` is the protocol version.
657
- - Backward-compatible additions (new fields, new error codes, new schema types) MAY be made under `textus/1`.
658
- - Breaking changes (renamed/removed fields, zone semantics, key grammar) require a new wire string `textus/2`.
591
+ - The current wire string is `textus/2`. It was introduced in gem v0.5, which unified the `_meta` block across all storage formats (markdown, json, yaml, text) and replaced the legacy TSV audit-log write path with NDJSON.
592
+ - `textus/1` was the original protocol (gem v0.4). Manifests declaring `version: textus/1` are still accepted for backward compatibility (§4).
593
+ - Backward-compatible additions (new fields, new error codes, new schema types) MAY be made under `textus/2`.
594
+ - Breaking changes (renamed/removed envelope fields, zone semantics, key grammar) require a new wire string `textus/3`.
659
595
  - Implementations MUST reject envelopes whose `protocol` they do not recognize.
660
596
 
661
- The reference Ruby gem follows semver independently. Gem 1.x speaks `textus/1`.
597
+ The reference Ruby gem follows semver independently. The current gem version is `0.8.0`, which speaks `textus/2`.
662
598
 
663
599
  ## 12. Conformance fixtures
664
600
 
@@ -686,7 +622,7 @@ Given a derived entry with a `template` clause referencing a `.mustache` file an
686
622
  Given a manifest entry with `publish_to: <path>`, a successful `textus build` for that entry leaves a plain file at `<path>` whose contents are byte-identical to the in-store artifact at `.textus/zones/<...>`, accompanied by a sentinel at `.textus/sentinels/<path>.textus-managed.json` recording `source`, `target`, `sha256`, and `mode: "copy"`. Re-running `build` is idempotent.
687
623
 
688
624
  **Fixture H — Audit log format:**
689
- Every successful write verb (`put`, `delete`, `build`, `accept`, `schema-migrate`) appends exactly one line per affected key to the audit log, in the canonical format defined in §audit (timestamp, actor role, verb, key, etag-before, etag-after). No write produces zero or multiple lines per key.
625
+ Every successful write verb (`put`, `delete`, `build`, `accept`, `schema migrate`) appends exactly one line per affected key to the audit log, in the canonical format defined in §audit (timestamp, actor role, verb, key, etag-before, etag-after). No write produces zero or multiple lines per key.
690
626
 
691
627
  **Fixture I — Pending → accept:**
692
628
  Given a pending entry `pending.canon.identity.patch` proposing a change to `canon.identity`, `textus accept canon.identity --as=human` copies the patch body into `canon.identity`, deletes the pending entry, and appends two audit lines (one for the canon write, one for the pending delete) in that order.
@@ -695,19 +631,19 @@ Given a pending entry `pending.canon.identity.patch` proposing a change to `cano
695
631
 
696
632
  - **Why not MCP?** MCP is a transport; textus is a data model. The two compose: a 50-line MCP server can wrap `textus get/put` as tools. textus exists because the *shape* of agent-readable project memory deserves a standalone spec, separate from how it's served.
697
633
 
698
- - **Why doesn't textus execute generator commands itself?** textus is a dataflow oracle, not a build runner. The moment a spec includes process execution, it inherits shell-injection surface, OS-portability concerns, and signal-handling semantics — and ends up duplicating whatever build system the consumer already runs (make, rake, just, lefthook, CI). Keeping execution external means a Python or TypeScript port of `textus/1` only has to parse YAML and emit JSON; it doesn't have to spawn processes safely. Build runners stay the executor; textus stays a data tool.
634
+ - **Why doesn't textus execute generator commands itself?** textus is a dataflow oracle, not a build runner. The moment a spec includes process execution, it inherits shell-injection surface, OS-portability concerns, and signal-handling semantics — and ends up duplicating whatever build system the consumer already runs (make, rake, just, lefthook, CI). Keeping execution external means a Python or TypeScript port of `textus/2` only has to parse YAML and emit JSON; it doesn't have to spawn processes safely. Build runners stay the executor; textus stays a data tool.
699
635
 
700
636
  - **Why not plain Markdown vaults (Obsidian / Foam)?** No schema enforcement, no write-gating, no addressable wire format. Fine for human notes; underspecified for agents that must act on the contents deterministically.
701
637
 
702
638
  - **Why not Notion / Coda?** Closed, hosted, lossy export. textus is local-first, plain-files, diffable in git.
703
639
 
704
- - **Why not JSON Schema for the schemas?** Considered. Bespoke YAML chosen for v1: simpler implementation, lighter dependency footprint, matches the reference impl's house language. JSON Schema MAY be added as an alternate schema-language adapter in a future minor revision without breaking `textus/1`.
640
+ - **Why not JSON Schema for the schemas?** Considered. Bespoke YAML chosen for v1: simpler implementation, lighter dependency footprint, matches the reference impl's house language. JSON Schema MAY be added as an alternate schema-language adapter in a future minor revision without breaking `textus/2`.
705
641
 
706
642
  - **Why not a database (SQLite, kv store)?** textus's whole point is that the storage is plain files agents and humans both read. A binary store loses git-diff, grep, and editor support.
707
643
 
708
644
  - **Why not vector embeddings?** Different problem. textus is for facts agents act on deterministically; embeddings are for fuzzy retrieval. They compose — index a textus tree into a vector store if you need both.
709
645
 
710
- ## 14. Open questions (v1.x scope)
646
+ ## 14. Open questions (v2.x scope)
711
647
 
712
648
  - **Locking on `put`:** the reference impl uses sha256 etags. Should the spec also define a file-lock fallback for systems where read-before-write is racy?
713
649
  - **Schema imports:** can one schema reference another (`type: $ref: person`)? Defer to v1.1.
@@ -716,21 +652,22 @@ Given a pending entry `pending.canon.identity.patch` proposing a change to `cano
716
652
 
717
653
  ## 15. Implementation checklist
718
654
 
719
- A v1 implementation MUST:
655
+ A `textus/2` implementation MUST:
720
656
 
721
- - [ ] Parse `.textus/manifest.yaml` and validate the `version: textus/1` declaration.
657
+ - [ ] Parse `.textus/manifest.yaml` and accept `version: textus/2` (and `textus/1` for backward compat per §11).
722
658
  - [ ] Resolve keys via longest-prefix match against manifest entries.
723
- - [ ] Read frontmatter + body from `.md` files; validate against the named schema.
659
+ - [ ] Read `_meta` + body from `.md` files; validate against the named schema.
660
+ - [ ] Read `_meta` from the top-level `_meta` hash in `.json` / `.yaml` files; validate against the named schema.
724
661
  - [ ] Compute `sha256:<hex>` etags over raw file bytes.
725
662
  - [ ] Refuse writes whose resolved role is not in the target zone's `writable_by` list with `write_forbidden`.
726
- - [ ] Return envelopes matching the shape in §8 exactly.
663
+ - [ ] Return envelopes matching the shape in §8 exactly (with `_meta`, not `frontmatter`).
727
664
  - [ ] Use the error codes in §8 and the exit-code table.
728
665
  - [ ] Implement `textus stale` per §5.1 and §9, comparing each derived entry's `generator.sources` against its `generated.at` timestamp without invoking any commands.
729
666
  - [ ] Pass the conformance fixtures A–I in §12.
730
667
 
731
- A v1 implementation MAY:
668
+ A `textus/2` implementation MAY:
732
669
 
733
- - Add additional CLI verbs (e.g. `move`, vendor-specific reporters) beyond the v1.0 set in §9.
670
+ - Add additional CLI verbs (e.g. `move`, vendor-specific reporters) beyond the current set in §9.
734
671
  - Provide alternate output formats (`--format=yaml`, `--format=table`) for human use.
735
672
  - Support additional schema field types beyond §6, marked as `vendor:<name>` extensions.
736
673
 
data/docs/architecture.md CHANGED
@@ -2,56 +2,110 @@
2
2
 
3
3
  How the reference Ruby implementation is organized. The wire protocol itself lives in [`../SPEC.md`](../SPEC.md); this document covers *how* the gem implements that spec.
4
4
 
5
- ## Layering
5
+ The codebase is a flat graph of small modules under one CLI dispatcher, not a strict pyramid. The clusters below describe what each module exists for and which other modules it talks to.
6
+
7
+ ## At a glance
6
8
 
7
9
  ```
8
- ┌──────────────────────────────────────────────┐
9
- exe/textus │ thin shim: load lib, call CLI.run
10
- ├──────────────────────────────────────────────┤
11
- Textus::CLI │ argv parsing, JSON I/O, exit codes
12
- ├──────────────────────────────────────────────┤
13
- Textus::Store │ verb implementations (get/put/list/…)
14
- ├───────────────┬───────────────┬──────────────┤
15
- │ Manifest │ Schema │ Entry │ parse/resolve, validate, (de)serialize
16
- ├───────────────┴───────────────┴──────────────┤
17
- Etag · Errors · version │ primitives
18
- └──────────────────────────────────────────────┘
10
+ exe/textus → Textus::CLI ──┬──► Store (facade — delegates to Reader/Writer/Mover)
11
+ ├──► Store::Reader (get/list/where/uid/deps/published/stale/validate_all)
12
+ │ ├──► Store::Writer (put/delete/accept)
13
+ ├──► Store::Mover (mv)
14
+ │ └──► Hooks::Dispatcher (lifecycle publish/subscribe)
15
+ ├──► Builder (build verb Pipeline + per-format renderers)
16
+ ├──► Refresh (refresh verb)
17
+ ├──► Doctor (doctor verb)
18
+ ├──► Init (init verb)
19
+ ├──► Intro (intro verb)
20
+ ├──► MigrateKeys (key migrate, key mv verbs)
21
+ ├──► Schema::Tools (schema init/diff/migrate verbs)
22
+ ├──► Store::View (read-only projection over Store::Reader)
23
+ └──► Role (role gate)
19
24
  ```
20
25
 
21
- Each layer talks only to the layer below it. `Store` is the only class that touches the filesystem for read/write; `Manifest` and `Schema` read at load time and are otherwise pure.
26
+ CLI is the single entry point. It parses argv and dispatches each verb to whichever module owns that capability there is no single mediator below CLI.
27
+
28
+ ## Module clusters
29
+
30
+ ### 1. Request path — core read/write verbs
31
+
32
+ `Store` is a thin facade (~110 LOC) that holds `Manifest`, `Hooks::Registry`, `Hooks::Dispatcher`, and the lazy `Store::AuditLog`, then delegates verbs to a small set of focused collaborators:
22
33
 
23
- ## Key resolution
34
+ - **`Store::Reader`** — owns `get`, `list`, `where`, `uid`, `deps`, `rdeps`, `published`, `schema_envelope`, `stale`, `validate_all`. The only module that reads working-store entry files.
35
+ - **`Store::Writer`** — owns `put`, `delete`, `accept`. Handles serialization, uid minting, etag check, role gate, audit append, and event publication. The only module that writes working-store entry files.
36
+ - **`Store::Mover`** — owns `mv` (same-zone rename) with uid preservation and one audit row.
37
+ - **`Store::Validator`** / **`Store::Staleness`** — back the `validate_all` / `stale` reads. Take explicit collaborators (`reader:`, `manifest:`, `audit_log:`, `schema_for:`) instead of the full store.
24
38
 
25
- `Manifest#resolve(key)` does **longest-prefix match** against `entries[].key`. The matched entry's `path:` is the base; if `nested: true`, remaining dotted segments become `/`-joined subdirectories under that path and a `.md` is appended.
39
+ Shared value modules and primitives consumed by Reader/Writer/Mover:
26
40
 
27
- Resolution is **path-only** it does not check whether the file exists. Existence is the verb's concern: `get` raises `unknown_key` when the file is missing; `put` happily creates new files in nested entries.
41
+ - **`Textus::Key::Path`**`Key::Path.resolve(manifest, mentry)` returns the absolute leaf path for a manifest entry. Single source for zone-path construction; used by `Manifest`, `Staleness`, `Builder`, and Writer.
42
+ - **`Textus::Envelope`** — `Envelope.build(...)` returns the canonical envelope hash (protocol, key, zone, owner, path, format, `_meta`, body, etag, schema_ref, uid, optional content). Single source for envelope shape across `get` and `put`.
43
+ - **`Manifest`** — parses `.textus/manifest.yaml`; resolves a dotted key to a path via longest-prefix match. `nested: true` entries treat unmatched suffix segments as `/`-joined subdirs, with `.md` appended. Resolution is path-only; existence is the verb's concern.
44
+ - **`Schema`** — loads YAML schema files; validates frontmatter shape and surfaces unknown-key warnings (the §6 forward-compat rule).
45
+ - **`Entry`** + format adapters (`entry/markdown.rb`, `entry/text.rb`, `entry/json.rb`, `entry/yaml.rb`) — splits raw bytes on `---\n`, feeds the YAML chunk to `YAML.safe_load` (no aliases, restricted classes). The frontmatter `name:` field is enforced against the file basename in Reader/Writer (on read and on write) — mismatch raises `bad_frontmatter`.
46
+ - **`Etag`** — `sha256:<hex>` over raw file bytes. `put` accepts optional `if_etag:`; mismatch raises `etag_mismatch`. No locking, no temp-file-and-rename — v1 leaves stronger guarantees to v1.x.
47
+ - **`Role`** — agent-vs-human gate. Writer checks `Manifest::Entry#zone_writers` before doing anything else; otherwise raises `write_forbidden`.
48
+ - **`Store::AuditLog`** — append-only NDJSON; every successful write emits one line.
49
+ - **`Proposal`** — `accept` verb flow for promoting a pending entry into its target zone.
50
+ - **`Dependencies`** — `deps`/`rdeps`/`published` verb backing; walks manifest declarations.
28
51
 
29
- ## Frontmatter parsing
52
+ ### 2. Build / publish pipeline
53
+
54
+ Separate from the request path. Owns derived-entry materialization and byte-copy publish. `Builder` orchestrates per-entry materialization through `Builder::Pipeline`, which runs an ordered step list and dispatches the rendering step to one of four format-specific renderers. Adding a new output format is a single-file change under `lib/textus/builder/renderer/`.
55
+
56
+ ```
57
+ Builder ──► Pipeline ──► LoadSources ──► Project ──► Render (per-format) ──► Write ──► Publisher ──► (sentinel)
58
+
59
+ └──► Renderer::{Markdown, Text, Json, Yaml}
60
+ ```
30
61
 
31
- `Entry.parse` splits raw bytes on `---\n` boundaries and feeds the YAML chunk to `YAML.safe_load` (no aliases, restricted classes). Unknown top-level frontmatter keys are not rejected here — they are surfaced as warnings by `Schema#validate!`. This is the forward-compat rule from §6 of the spec.
62
+ - **`Builder`** iterates `zone: derived` entries, hands each to `Pipeline.run`, then handles `Publisher` copy-out and fires the `:build` event. Holds no format-specific logic.
63
+ - **`Builder::Pipeline`** — `Pipeline.run(store:, mentry:, template_loader:)` is the orchestrator: runs the projection, merges `intro` if `inject_intro: true`, dispatches to the matching renderer, writes the bytes to the derived path.
64
+ - **`Builder::InjectMeta`** — builds the `_meta` block (`generated_at`, `from`, `template`, `reduce`) and threads it onto JSON/YAML content as the first key per SPEC §6 ordering.
65
+ - **`Builder::Renderer::{Markdown,Text,Json,Yaml}`** — one class per format, inheriting `Builder::Renderer`. Receives a template-loader lambda and `(mentry:, data:)`; returns rendered bytes. Markdown/Text always require a template; JSON/YAML optionally accept one (otherwise default-shape the projection rows).
66
+ - **`Projection`** — collects rows from manifest-declared source keys, applies optional reducer, sorts and positions. Pure data shaping.
67
+ - **`Mustache`** — minimal mustache renderer for templates in `.textus/templates/`.
68
+ - **`Publisher`** — byte-copy from store path to external target path. Refuses to overwrite unmanaged targets; writes a sentinel in `.textus/sentinels/` to track managed targets.
32
69
 
33
- The frontmatter `name:` field is enforced against the file basename inside `Store` (both on read and on write) so a misnamed file or a mistyped `name:` in `put` payload fails fast with `bad_frontmatter`.
70
+ ### 3. Extension surface
34
71
 
35
- ## Zone enforcement
72
+ Declared in the manifest, loaded on demand, dispatched by `Store` and `Refresh`.
36
73
 
37
- Three zones (`fixed`, `state`, `derived`) declared per-entry in the manifest. `Store#put` checks `ManifestEntry#agent_writable?` (true only for `state`) before doing anything else and raises `write_forbidden` otherwise. Zone semantics live in the manifest, not directory names a project can rename `state/` to whatever it wants.
74
+ - **`Hooks::Registry`** loads one `.rb` per hook from `.textus/hooks/`, registers callables under their `(event, name)`. Single source of truth via the `EVENTS` table (rpc vs pubsub, arg shape, failure semantics). For pub-sub events it also forwards registrations to the `Hooks::Dispatcher`.
75
+ - **`Hooks::Dispatcher`** — first-class pub/sub for lifecycle events (`:put`, `:delete`, `:refresh`, `:build`, `:accept`). Owns the 2-second per-handler timeout and the audit-on-failure middleware (raising handlers do not abort the write; they produce an `event_error` audit row). Embedded callers can `store.dispatcher.subscribe(:put, :name) { ... }` outside `.textus/hooks/`.
76
+ - **`Hooks::Builtin`** — ships built-in `:fetch` hooks (e.g. json, csv, ical-events, rss) available without user-supplied hooks.
77
+ - **`Refresh`** — `refresh` verb: looks up the `:fetch` hook for a key, invokes it, normalizes the result by declared format, writes through `Store::Writer` with an etag check.
38
78
 
39
- ## ETag and concurrency
79
+ ### 4. Operational tooling
40
80
 
41
- `Etag.for_bytes` returns `sha256:<hex>` over the raw file bytes. `put` accepts an optional `if_etag:` — if provided and the on-disk file's etag differs, `etag_mismatch` is raised. No locking, no temp-file-and-rename the v1 spec leaves stronger guarantees to v1.x (§14 open question).
81
+ First-class CLI verbs that don't fit the read/write/build axes. Read-mostly; side modules off CLI.
42
82
 
43
- ## Staleness (the dataflow oracle)
83
+ - **`Doctor`** `doctor` verb: orchestrator that runs 9 builtin checks under `Doctor::Check::*`. Talks to Manifest/Schema/Entry/Hooks::Registry directly.
84
+ - **`Doctor::Check`** — explicit base class for doctor checks. Each of the 9 builtin checks is its own file under `lib/textus/doctor/check/`.
85
+ - **`MigrateKeys`** — `key migrate` and `key mv` verbs; computes renames against the manifest.
86
+ - **`Schema::Tools`** — `schema init`, `schema diff`, `schema migrate` verbs.
87
+ - **`Init`** — `init` verb: scaffolds `.textus/` with the five zone directories, baseline schemas, empty audit log, starter manifest.
88
+ - **`Intro`** — `intro` verb: emits the human/agent-facing onboarding payload.
89
+ - **`Store::View`** — read-only projection over `Store::Reader` for hook code that should not mutate.
90
+ - **`Key::Distance`** — Levenshtein-ish suggestion for `did-you-mean` on unknown keys.
44
91
 
45
- `Store#stale` walks every `zone: derived` entry that declares a `generator:` block, reads its `generated.at` frontmatter timestamp, and compares against each `generator.sources` entry's current mtime. Returns the offenders **and their declared `command`**; it does **not** execute anything. This is the core "dataflow oracle, not executor" boundary from §5.1 of the spec.
92
+ ### 5. Primitives
46
93
 
47
- Sources are heuristically classified: a string matching the textus key grammar with no `/` is treated as a textus key (and enumerated via the manifest); anything else is treated as a repo-relative path.
94
+ - **`Errors`** `Textus::Error` subclasses, each carrying a stable `code`, a `details` hash, and an `exit_code`. `CLI` catches them at the top level and emits the §8 error envelope on stdout. In `--format=json` mode, errors are **never** written to stderr — agents read stdout.
95
+ - **`version`** — gem semver string (independent of the wire protocol `textus/2`).
48
96
 
49
- ## Errors → envelopes
97
+ ## Invariants
50
98
 
51
- All `Textus::Error` subclasses carry a stable error `code`, a `details` hash, and an `exit_code`. `CLI` catches them at the top level and emits the §8 error envelope on stdout; the exit code matches the §8 table. Errors are never written to stderr in `--format=json` mode — agents read stdout.
99
+ - **CLI is the only entry point.** No public API surface guarantees outside the verbs CLI exposes.
100
+ - **Manifest is pure.** Reads at load, no mutation.
101
+ - **Store::Writer is the only module that writes to working-store entry files.** Reader reads them; Mover moves them within a zone. Init, MigrateKeys, Publisher, Builder, AuditLog write to **other** parts of `.textus/` (scaffolding, sentinels, audit log, derived targets) — they do not edit existing entry files behind the Store facade's back.
102
+ - **`name:` frontmatter matches file basename.** Enforced on read and write.
103
+ - **Zone semantics live in the manifest, not in directory names.** A project may rename `state/` to anything; the manifest declares which zone each entry belongs to.
104
+ - **`stale` does not execute anything.** It walks `zone: derived` entries with a `generator:` block, compares `generated.at` against source mtimes, and returns offenders **plus their declared `command`**. Build runners execute. This is the §5.1 "dataflow oracle, not executor" boundary.
52
105
 
53
106
  ## What this implementation deliberately leaves out
54
107
 
55
108
  - **No process spawning.** Even `stale` does not execute. Build runners do that.
56
109
  - **No transport.** No HTTP server, no socket, no MCP server in this gem. Those are downstream wrappers (see [`./conventions.md`](./conventions.md)).
57
110
  - **No indexes.** Listing walks the filesystem each time. Premature optimisation for v1.
111
+ - **No locking.** Etag is advisory; concurrent writers can still race. Left to v1.x (§14 open question).