textus 0.2.0

data/docs/architecture.md

@@ -0,0 +1,57 @@
+# Architecture
+
+How the reference Ruby implementation is organized. The wire protocol itself lives in [`../SPEC.md`](../SPEC.md); this document covers *how* the gem implements that spec.
+
+## Layering
+
+```
+┌──────────────────────────────────────────────┐
+│ exe/textus                                   │  thin shim: load lib, call CLI.run
+├──────────────────────────────────────────────┤
+│ Textus::CLI                                  │  argv parsing, JSON I/O, exit codes
+├──────────────────────────────────────────────┤
+│ Textus::Store                                │  verb implementations (get/put/list/…)
+├───────────────┬───────────────┬──────────────┤
+│ Manifest      │ Schema        │ Entry        │  parse/resolve, validate, (de)serialize
+├───────────────┴───────────────┴──────────────┤
+│ Etag · Errors · version                      │  primitives
+└──────────────────────────────────────────────┘
+```
+
+Each layer talks only to the layer below it. `Store` is the only class that touches the filesystem for read/write; `Manifest` and `Schema` read at load time and are otherwise pure.
+
+## Key resolution
+
+`Manifest#resolve(key)` does **longest-prefix match** against `entries[].key`. The matched entry's `path:` is the base; if `nested: true`, remaining dotted segments become `/`-joined subdirectories under that path and a `.md` is appended.
+
+Resolution is **path-only** — it does not check whether the file exists. Existence is the verb's concern: `get` raises `unknown_key` when the file is missing; `put` happily creates new files in nested entries.
+
+## Frontmatter parsing
+
+`Entry.parse` splits raw bytes on `---\n` boundaries and feeds the YAML chunk to `YAML.safe_load` (no aliases, restricted classes). Unknown top-level frontmatter keys are not rejected here — they are surfaced as warnings by `Schema#validate!`. This is the forward-compat rule from §6 of the spec.
+
+The frontmatter `name:` field is enforced against the file basename inside `Store` (both on read and on write) so a misnamed file or a mistyped `name:` in `put` payload fails fast with `bad_frontmatter`.
+
+## Zone enforcement
+
+Three zones (`fixed`, `state`, `derived`) declared per-entry in the manifest. `Store#put` checks `ManifestEntry#agent_writable?` (true only for `state`) before doing anything else and raises `write_forbidden` otherwise. Zone semantics live in the manifest, not directory names — a project can rename `state/` to whatever it wants.
+
+## ETag and concurrency
+
+`Etag.for_bytes` returns `sha256:<hex>` over the raw file bytes. `put` accepts an optional `if_etag:` — if provided and the on-disk file's etag differs, `etag_mismatch` is raised. No locking, no temp-file-and-rename — the v1 spec leaves stronger guarantees to v1.x (§14 open question).
+
+## Staleness (the dataflow oracle)
+
+`Store#stale` walks every `zone: derived` entry that declares a `generator:` block, reads its `generated.at` frontmatter timestamp, and compares against each `generator.sources` entry's current mtime. Returns the offenders **and their declared `command`**; it does **not** execute anything. This is the core "dataflow oracle, not executor" boundary from §5.1 of the spec.
+
+Sources are heuristically classified: a string matching the textus key grammar with no `/` is treated as a textus key (and enumerated via the manifest); anything else is treated as a repo-relative path.
+
+## Errors → envelopes
+
+All `Textus::Error` subclasses carry a stable error `code`, a `details` hash, and an `exit_code`. `CLI` catches them at the top level and emits the §8 error envelope on stdout; the exit code matches the §8 table. Errors are never written to stderr in `--format=json` mode — agents read stdout.
+
+## What this implementation deliberately leaves out
+
+- **No process spawning.** Even `stale` does not execute. Build runners do that.
+- **No transport.** No HTTP server, no socket, no MCP server in this gem. Those are downstream wrappers (see [`./conventions.md`](./conventions.md)).
+- **No indexes.** Listing walks the filesystem each time. Premature optimisation for v1.

data/docs/conventions.md

@@ -0,0 +1,85 @@
+# Conventions
+
+Guidelines for shaping a `.textus/` tree, naming keys, organising schemas, and integrating with build runners. The spec ([`../SPEC.md`](../SPEC.md)) defines what's enforceable; this document captures what's *idiomatic*.
+
+## Key naming
+
+- **Segments are lowercase, kebab- or snake-case.** The grammar `^[a-z0-9](?:[a-z0-9_-]*[a-z0-9])?$` is the hard limit. Prefer `acme-dashboard` over `acmedashboard` when there's a natural word break.
+- **Lead with the zone in the key path.** `state.projects.acme.dashboard`, not `projects.acme.dashboard`. The zone prefix makes it obvious from the key alone whether a write will be accepted.
+- **Mirror the directory structure.** If `state.projects.acme.dashboard` resolves to `state/projects/acme/dashboard.md`, do not invent shortcuts that diverge.
+- **Don't pluralise the leaf.** `state.network.org.jane`, not `state.network.org.janes`. Pluralise the container, not the entry.
+
+## Zone layout
+
+Recommended top-level layout — the spec allows alternatives, but this is what tooling will default to:
+
+```
+.textus/
+  manifest.yaml
+  schemas/        # YAML schema definitions
+  fixed/          # identity, voice, canon — humans only
+  state/          # agent-writable working memory
+  derived/        # generated by build runners — never edit by hand
+```
+
+Inside `state/`, group by **domain** (people, projects, decisions, runbooks), not by file type or date. Inside `derived/`, group by **producer** (`derived/catalogs/`, `derived/indexes/`) so it's clear which build job owns what.
+
+## Schema design
+
+- **One schema per entry type, not per directory.** `person.yaml`, `project.yaml`, `decision.yaml` — applied across multiple subtrees if the shape matches.
+- **Required = "this entry is meaningless without it."** Everything else is `optional`. Resist the urge to mark organisational metadata (like `tags`) required.
+- **Prefer `enum` over free-text** for low-cardinality fields (relationship type, status, severity). Agents are far better at picking from a list than at producing exact strings.
+- **Cap string lengths** with `max:` where the field has a natural bound (names, summaries). Skip for prose body — bodies are not schema-validated, only frontmatter is.
+
+## Owner strings
+
+The `owner:` field in the manifest is **advisory metadata**, not an ACL. Use it to label *who's expected to write here*:
+
+- `textus:network` — humans curate
+- `agent:planner` — a specific named agent
+- `build:catalog-skills` — a specific build job
+
+Tooling around `git blame` or audit logs may filter on owner; the gem itself only echoes it back in envelopes.
+
+## Derived entries and build runners
+
+**Always** declare `generator:` on derived entries that participate in any build pipeline. Without it, `textus stale` cannot help — the entry is just an opaque file.
+
+```yaml
+- key: derived.catalogs.skills
+  path: derived/catalogs/skills
+  zone: derived
+  schema: null
+  owner: build:catalog-skills
+  generator:
+    command: "rake catalog:skills"
+    sources:
+      - state.projects
+      - state.network
+```
+
+**The build runner is responsible for writing the `generated:` frontmatter block** when it regenerates. The gem will never synthesize it. A typical lefthook / rake / just integration looks like:
+
+```sh
+textus stale --format=json | jq -r '.[] | .generator.command' | sort -u | while read cmd; do
+  eval "$cmd"
+done
+```
+
+`generated.from` SHOULD match `generator.sources` from the manifest — they're the same list, recorded in two places so a diffable file proves what was actually consumed.
+
+## Body content
+
+- **Bodies are Markdown.** Headings, lists, code fences — whatever a human or agent finds useful.
+- **The schema does not validate the body.** If a field belongs in structured data, put it in frontmatter, not the body.
+- **Keep entries short.** If a project entry hits 500 lines, it probably wants to be split into sub-entries (e.g. `state.projects.acme.dashboard` + `state.projects.acme.api`) rather than one mega-document.
+
+## Concurrency
+
+For multi-writer environments, **always pass `if_etag`** on `put`. The gem treats etag-less writes as last-writer-wins on purpose (single-writer scripts, fresh-file creation), but anything resembling a daemon or a long-running agent should round-trip the etag.
+
+## Pairing with other tools
+
+- **MCP servers**: a thin server that exposes `textus get` and `textus put` as tools is the recommended way to give Claude/agents access. Don't bake MCP into this gem.
+- **Vector stores**: index `body` content into a vector store if you want fuzzy retrieval. `frontmatter` stays in textus as the source of truth for deterministic facts.
+- **CI**: run `textus stale` (or `textus list` + schema validation) in CI to catch drift between derived entries and their sources.

data/exe/textus

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+require "textus"
+exit Textus::CLI.run(ARGV)

data/lib/textus/audit_log.rb

@@ -0,0 +1,32 @@
+require "json"
+require "time"
+
+module Textus
+  class AuditLog
+    def initialize(root)
+      @path = File.join(root, "audit.log")
+    end
+
+    def last_writer_for(key)
+      return nil unless File.exist?(@path)
+
+      File.foreach(@path).map { |l| l.chomp.split("\t") }
+                         .select { |row| row[3] == key && %w[put delete].include?(row[2]) }
+                         .last&.fetch(1)
+    end
+
+    def append(role:, verb:, key:, etag_before:, etag_after:, extras: nil)
+      fields = [
+        Time.now.utc.iso8601, role, verb, key,
+        etag_before || "NULL",
+        etag_after  || "NULL"
+      ]
+      fields << JSON.generate(extras) if extras && !extras.empty?
+      line = fields.join("\t") + "\n"
+      File.open(@path, File::WRONLY | File::APPEND | File::CREAT, 0o644) do |f|
+        f.flock(File::LOCK_EX)
+        f.write(line)
+      end
+    end
+  end
+end

data/lib/textus/builder.rb

@@ -0,0 +1,191 @@
+require "fileutils"
+require "json"
+require "time"
+require "yaml"
+
+module Textus
+  class Builder
+    def initialize(store)
+      @store = store
+      @manifest = store.manifest
+      @root = store.root
+    end
+
+    def build(prefix: nil)
+      built = []
+      @manifest.entries.each do |mentry|
+        next unless derived_zone?(mentry)
+        next unless mentry.projection || mentry.template
+        next if prefix && !mentry.key.start_with?(prefix)
+
+        result = materialize(mentry)
+        built << result
+      end
+      published_leaves = publish_leaves(prefix: prefix)
+      { "protocol" => Textus::PROTOCOL, "built" => built, "published_leaves" => published_leaves }
+    end
+
+    private
+
+    def publish_leaves(prefix: nil)
+      repo_root = File.dirname(@root)
+      out = []
+      @manifest.entries.each do |mentry|
+        next unless mentry.nested && mentry.publish_each
+        next if prefix && !mentry.key.start_with?(prefix) && !prefix.start_with?("#{mentry.key}.")
+
+        @manifest.enumerate(prefix: mentry.key).each do |row|
+          next unless row[:manifest_entry].equal?(mentry)
+          next if prefix && !row[:key].start_with?(prefix) && row[:key] != prefix
+
+          target_rel = mentry.publish_target_for(row[:key])
+          target_abs = File.expand_path(File.join(repo_root, target_rel))
+          unless target_abs.start_with?(File.expand_path(repo_root) + File::SEPARATOR)
+            raise PublishError.new(
+              "entry '#{mentry.key}': publish_each target '#{target_rel}' for key '#{row[:key]}' escapes repo root",
+            )
+          end
+
+          Publisher.publish(source: row[:path], target: target_abs, store_root: @root)
+          out << { "key" => row[:key], "source" => row[:path], "target" => target_abs }
+        end
+      end
+      out
+    end
+
+    def derived_zone?(mentry)
+      writers = @manifest.zone_writers(mentry.zone)
+      writers.include?("build")
+    end
+
+    def materialize(mentry)
+      data =
+        if mentry.projection
+          Projection.new(@store, mentry.projection).run
+        else
+          { "entries" => [], "count" => 0, "generated_at" => Time.now.utc.iso8601 }
+        end
+
+      bytes =
+        case mentry.format
+        when "markdown" then build_markdown(mentry, data)
+        when "text"     then build_text(mentry, data)
+        when "json"     then build_structured(mentry, data, "json")
+        when "yaml"     then build_structured(mentry, data, "yaml")
+        else raise UsageError.new("builder: unsupported format #{mentry.format.inspect} for '#{mentry.key}'")
+        end
+
+      target_path = File.join(@root, "zones", mentry.path)
+      FileUtils.mkdir_p(File.dirname(target_path))
+      File.binwrite(target_path, bytes)
+
+      publish_and_fire(mentry, target_path)
+      { "key" => mentry.key, "path" => target_path, "published_to" => mentry.publish_to }
+    end
+
+    # Markdown: projection -> template -> markdown.serialize(frontmatter, body).
+    # Frontmatter carries the legacy `generated:` bookkeeping block. Per plan-1.2 §6,
+    # `_meta` ordering applies to structured formats only; markdown keeps existing shape
+    # for backward compat with consumers reading frontmatter["generated"]["at"].
+    def build_markdown(mentry, data)
+      data = data.merge("intro" => Intro.run(@store)) if mentry.inject_intro
+      body = render_template!(mentry, data)
+      frontmatter = {
+        "generated" => {
+          "at" => Time.now.utc.iso8601,
+          "from" => Array(mentry.projection&.fetch("select", nil)).compact,
+        },
+      }
+      Entry.for_format("markdown").serialize(frontmatter: frontmatter, body: body)
+    end
+
+    # Text: projection -> template -> text.serialize(body). No frontmatter, no _meta.
+    def build_text(mentry, data)
+      data = data.merge("intro" => Intro.run(@store)) if mentry.inject_intro
+      body = render_template!(mentry, data)
+      Entry.for_format("text").serialize(frontmatter: {}, body: body)
+    end
+
+    # JSON / YAML pipeline. Templateless = default; template = escape hatch.
+    def build_structured(mentry, data, format)
+      strategy = Entry.for_format(format)
+
+      content =
+        if mentry.template
+          parse_rendered_template!(mentry, data, format)
+        else
+          # Default rule: if the reducer returned a Hash (it replaced `rows`), use it as-is.
+          # Otherwise wrap the entries list as { "entries" => [...] } so the top level is a Hash
+          # (required to carry _meta).
+          if mentry.projection && mentry.projection["reducer"] && data.is_a?(Hash) && !data.key?("entries")
+            data
+          elsif data.is_a?(Hash) && data["entries"].is_a?(Array)
+            { "entries" => data["entries"] }
+          else
+            data.is_a?(Hash) ? data : { "entries" => Array(data) }
+          end
+        end
+
+      final = inject_meta(content, mentry)
+      strategy.serialize(frontmatter: {}, body: "", content: final)
+    end
+
+    def render_template!(mentry, data)
+      raise TemplateError.new("entry '#{mentry.key}': #{mentry.format} build requires a template") unless mentry.template
+
+      tpl_path = File.join(@root, "templates", mentry.template)
+      raise TemplateError.new("template not found: #{tpl_path}", template_name: mentry.template) unless File.exist?(tpl_path)
+
+      Mustache.render(File.read(tpl_path), data)
+    end
+
+    def parse_rendered_template!(mentry, data, format)
+      tpl_path = File.join(@root, "templates", mentry.template)
+      raise TemplateError.new("template not found: #{tpl_path}", template_name: mentry.template) unless File.exist?(tpl_path)
+
+      rendered = Mustache.render(File.read(tpl_path), data)
+      begin
+        parsed =
+          case format
+          when "json" then ::JSON.parse(rendered)
+          when "yaml" then ::YAML.safe_load(rendered, permitted_classes: [Date, Time], aliases: false)
+          end
+      rescue ::JSON::ParserError, Psych::SyntaxError, Psych::DisallowedClass, Psych::AliasesNotEnabled => e
+        raise BadRender.new("entry '#{mentry.key}': template did not render valid #{format}: #{e.message}", format: format)
+      end
+      unless parsed.is_a?(Hash)
+        raise BadRender.new("entry '#{mentry.key}': template must render a top-level object/mapping",
+                            format: format)
+      end
+
+      parsed
+    end
+
+    # Builds the _meta block per §6 ordering and inserts it as the first top-level key.
+    def inject_meta(content_hash, mentry)
+      meta = {}
+      meta["generated_at"] = Time.now.utc.iso8601
+      from = Array(mentry.projection&.fetch("select", nil)).compact
+      meta["from"] = from unless from.empty?
+      meta["template"] = mentry.template if mentry.template
+      reducer = mentry.projection&.dig("reducer")
+      meta["reducer"] = reducer if reducer
+
+      # Rebuild so _meta appears first; user content follows.
+      out = { "_meta" => meta }
+      content_hash.each { |k, v| out[k] = v unless k == "_meta" }
+      out
+    end
+
+    def publish_and_fire(mentry, target_path)
+      mentry.publish_to.each do |rel|
+        repo_root = File.dirname(@root)
+        Publisher.publish(source: target_path, target: File.join(repo_root, rel), store_root: @root)
+      end
+
+      envelope = @store.get(mentry.key)
+      @store.fire_event(:build, key: mentry.key, envelope: envelope,
+                                sources: Array(mentry.projection&.fetch("select", nil)).compact)
+    end
+  end
+end

data/lib/textus/builtin_fetchers.rb

@@ -0,0 +1,63 @@
+require "json"
+require "csv"
+require "yaml"
+require "rexml/document"
+
+module Textus
+  module BuiltinFetchers
+    # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
+    def self.register_all
+      Textus.fetcher(:json) do |config:, store:|
+        _ = store
+        data = JSON.parse(config["bytes"].to_s)
+        { frontmatter: {}, body: YAML.dump(data) }
+      end
+
+      Textus.fetcher(:csv) do |config:, store:|
+        _ = store
+        rows = CSV.parse(config["bytes"].to_s, headers: true).map(&:to_h)
+        { frontmatter: {}, body: YAML.dump(rows) }
+      end
+
+      Textus.fetcher(:"markdown-links") do |config:, store:|
+        _ = store
+        links = config["bytes"].to_s.scan(%r{\[([^\]]+)\]\((https?://[^)\s]+)\)}).map do |text, href|
+          { "text" => text, "href" => href }
+        end
+        { frontmatter: {}, body: YAML.dump(links) }
+      end
+
+      Textus.fetcher(:"ical-events") do |config:, store:|
+        _ = store
+        events = []
+        current = nil
+        config["bytes"].to_s.each_line do |line|
+          line = line.strip
+          case line
+          when "BEGIN:VEVENT" then current = {}
+          when "END:VEVENT"
+            events << current if current
+            current = nil
+          when /\A(SUMMARY|DTSTART|DTEND|UID|LOCATION|DESCRIPTION):(.*)\z/
+            current[Regexp.last_match(1).downcase] = Regexp.last_match(2) if current
+          end
+        end
+        { frontmatter: {}, body: YAML.dump(events) }
+      end
+
+      Textus.fetcher(:rss) do |config:, store:|
+        _ = store
+        doc = REXML::Document.new(config["bytes"].to_s)
+        items = doc.elements.to_a("//item").map do |item|
+          {
+            "title" => item.elements["title"]&.text,
+            "link" => item.elements["link"]&.text,
+            "pubDate" => item.elements["pubDate"]&.text,
+          }
+        end
+        { frontmatter: {}, body: YAML.dump(items) }
+      end
+    end
+    # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
+  end
+end