textus 0.54.1 → 0.54.2

data/docs/reference/conventions.md

@@ -8,11 +8,11 @@ Guidelines for shaping a `.textus/` tree, naming keys, organising schemas, and i
 ## Key naming
 
 - **Segments are lowercase, kebab- or snake-case.** The grammar `^[a-z0-9](?:[a-z0-9_-]*[a-z0-9])?$` is the hard limit. Prefer `acme-dashboard` over `acmedashboard` when there's a natural word break.
-- **Lead with the zone in the key path.** `working.projects.acme.dashboard`, not `projects.acme.dashboard`. The zone prefix makes it obvious from the key alone whether a write will be accepted.
+- **Lead with the lane in the key path.** `working.projects.acme.dashboard`, not `projects.acme.dashboard`. The lane prefix makes it obvious from the key alone whether a write will be accepted.
 - **Mirror the directory structure.** If `working.projects.acme.dashboard` resolves to `working/projects/acme/dashboard.md`, do not invent shortcuts that diverge.
 - **Don't pluralise the leaf.** `working.network.org.jane`, not `working.network.org.janes`. Pluralise the container, not the entry.
 
-## Zone layout
+## Lane layout
 
 Recommended top-level layout — the spec allows alternatives, but this is what tooling will default to:
 
@@ -20,12 +20,14 @@ Recommended top-level layout — the spec allows alternatives, but this is what
 .textus/
   manifest.yaml
   schemas/        # YAML schema definitions
-  zones/
+  workflows/      # Textus.workflow DSL files for produced entry acquisition
+  templates/      # ERB templates for publish rendering
+  data/
     knowledge/    # authored truth: identity, voice, decisions — author-holders write
     notebook/     # agent's own durable lane (workspace) — keep-holders write
-    feeds/        # automation-fed external inputs (fetch)
     proposals/    # AI proposals awaiting accept (propose)
-    artifacts/    # generated by build automation — never edit by hand
+    artifacts/    # computed outputs produced by drain — never edit by hand
+    raw/          # write-once external source material (ingest)
 ```
 
 Inside `knowledge/`, group by **domain** (identity, people, projects, decisions, runbooks), not by file type or date. `knowledge.identity.*` is the convention for slow-changing identity facts. Inside `artifacts/`, group by **producer** (`artifacts/catalogs/`, `artifacts/indexes/`) so it's clear which build job owns what.
@@ -47,75 +49,57 @@ The `owner:` field in the manifest is **advisory metadata**, not an ACL. Use it
 
 Tooling around `git blame` or audit logs may filter on owner; the gem itself only echoes it back in envelopes.
 
-## Derived entries
+## Produced entries
 
-A derived entry declares a `source:` block with a `from:` discriminator (ADR 0093/0094). A `source:` acquires **data** — it never renders; rendering is a publish concern (below). Two `from:` values for derived entries:
-
-**`source: { from: project }`** — textus computes the entry's data on `textus drain` from other store entries. Declarative; nothing shells out. Projection fields are flat under `source:`.
+A produced entry declares `source: { from: external, command: "true", sources: [] }` and a matching `Textus.workflow` block in `.textus/workflows/**/*.rb`. The `source:` acquires **data** — it never renders; rendering is a publish concern (below).
 
 ```yaml
-- key: artifacts.derived.people
-  path: artifacts/derived/people.md
-  zone: artifacts
-  schema: null
-  owner: automation:catalog-people
-  source:
-    from: project
-    select: knowledge.network.org   # prefix or list of prefixes
-    pluck:  [name, relationship, org]
-    sort_by: name
-    format: json                    # the stored form is data
+- key: artifacts.feeds.skills
+  lane: artifacts
+  kind: produced
+  format: json
+  source: { from: external, command: "true", sources: [] }
   publish:
-    - { to: docs/people.md, template: people.mustache }   # render the data through a template
-```
-
-**`source: { from: command }`** — an external build tool (rake, just, a shell script) produces the file. textus never executes the `command:`; it only tracks `sources:` so `doctor`'s `generator_drift` check can compare source mtimes against the file's `_meta.generated.at`. The role running the build must hold `converge` (default: `automation`).
-
-```yaml
-- key: artifacts.derived.skills
-  path: artifacts/derived/skills.md
-  zone: artifacts
-  owner: automation:catalog-skills
-  source:
-    from: command
-    command: "rake catalog:skills"   # informational; the automation invokes it
-    sources: [knowledge.projects, knowledge.network]
+    - { to: docs/reference/skills.md, template: feeds/skills.erb }
 ```
 
-The build automation is responsible for writing the `generated:` frontmatter block (`by`, `at`, `from`) when it produces the file. `generated.from` SHOULD match `source.sources` — same list, recorded twice so a diff proves what was consumed.
-
-Full contract for both shapes is in [`../../SPEC.md` §5.2.1 and §5.2.2](../../SPEC.md). Data transforms (`source.transform:`) and publishing (a `publish:` list of `{ to, template? }` / `{ tree }` targets — render or copy the data) are covered in §5.2–§5.3.
+The matching workflow block:
 
-## Intake and freshness
+```ruby
+# .textus/workflows/feeds/agentskills.rb
+Textus.workflow "agentskills" do
+  match "artifacts.feeds.skills"
 
-External inputs land via `:resolve_handler` hooks, not shell commands. Each intake entry declares `source: { from: handler, handler: <name>, ttl: <dur> }`; re-pull is system-pushed via `drain` (scheduled sweep) and `hook run` (event push) — a `get` never refreshes (ADR 0089):
+  step :fetch do |_, _ctx|
+    { "content" => { "skills" => [...], "count" => 1 } }
+  end
 
-```sh
-textus pulse --output=json                             # `stale` lists expired entries; `next_due_at` is the soonest deadline
-textus drain --as=automation                       # re-pulls every intake entry past its source.ttl
+  publish
+end
 ```
 
-The re-pull cadence is the entry's own `source.ttl`. Age-based garbage collection is the orthogonal `retention:` rule slot in the top-level `rules:` block, matched by glob (`{ ttl, action: drop | archive }`); the two compose — re-pull hourly *and* archive at 90 days:
+`drain` discovers workflows, matches entries, runs the steps, and writes the result back to the entry's data path. Publishing then copies or renders each `publish:` target.
 
-```yaml
-- key: artifacts.feeds.notion.events
-  kind: produced                # produce-method (intake) read from source.from: handler
-  zone: artifacts
-  source: { from: handler, handler: notion, config: { ... }, ttl: 6h }
+Age-based GC uses a `retention:` rule in the top-level `rules:` block:
 
+```yaml
 rules:
-  - match: artifacts.feeds.notion.**
+  - match: artifacts.feeds.**
     retention: { ttl: 90d, action: archive }
 ```
 
-A typical scheduled integration runs `drain` on a cron to re-pull every
-expired feed:
+Full contract in [`../../SPEC.md` §5.2](../../SPEC.md).
+
+## Freshness
+
+A `get` is always a pure read (ADR 0089) — it never triggers a re-produce:
 
 ```sh
-textus drain --as=automation   # in cron / CI — re-pulls all stale feeds
+textus pulse --output=json       # `stale` lists entries past their ttl
+textus drain --as=automation     # re-produces stale entries
 ```
 
-See [`./zones.md` §6](zones.md) for the full intake contract and [`../how-to/writing-hooks.md`](../how-to/writing-hooks.md) for writing custom handlers.
+See [`./lanes.md`](lanes.md) for lane semantics and [`../how-to/configuring-lanes.md`](../how-to/configuring-lanes.md) for setting up produced entries and workflows.
 
 ### Read vs. refresh
 
@@ -141,11 +125,11 @@ For multi-writer environments, **always pass `if_etag`** on `put`. The gem treat
 
 The application layer is organised around three shapes — `Manifest` as a composition record, a single `Container` capability record handed to every use case, and a split envelope reader/writer. See [ADR 0018](../architecture/decisions/0018-manifest-carving.md), [ADR 0017](../architecture/decisions/0017-envelope-io-split.md), [ADR 0022](../architecture/decisions/0022-container-call-dispatcher.md), and [ADR 0023](../architecture/decisions/0023-uniform-use-case-shape.md).
 
-- **`Manifest` is a composition record** (`Data.define(:data, :resolver, :policy, :rules)`). Reach individual concerns through the field accessors: `manifest.data.entries`, `manifest.policy.permission_for(zone)`, `manifest.resolver.resolve(key)`, `manifest.rules.for(key)`.
+- **`Manifest` is a composition record** (`Data.define(:data, :resolver, :policy, :rules)`). Reach individual concerns through the field accessors: `manifest.data.entries`, `manifest.policy.permission_for(lane)`, `manifest.resolver.resolve(key)`, `manifest.rules.for(key)`.
 - **Use cases are plain `(container:, call:)` classes.** Each is a single class under `lib/textus/{read,write,maintenance}/` with `def initialize(container:, call:)` and a `#call(...)` method; verbs are looked up in the static `Textus::Dispatcher::VERBS` table. `Container` is a `Data.define` record bundling the wired ports + manifest (`manifest`, `file_store`, `schemas`, `root`, `audit_log`, `events`, `rpc`); `Call` is the immutable per-invocation value (`role`, `correlation_id`, `now`, `dry_run`). A use case that emits events derives its `Hooks::Context` from `(container, call)` — nothing is injected. Use cases pull only what they need into ivars; nobody passes the raw `Store` around the application layer. `store.as(role)` returns a `RoleScope` that forwards verbs to the dispatcher.
 - **Write path is split**: `Envelope::IO::Reader` owns read/parse (existing-uid lookup, raw read, parse), and `Envelope::IO::Writer` owns put/delete/move + the audit-append invariant (every public method's final action is `@audit_log.append(...)`).
 
-The user-facing CLI surface, the wire envelope shape, and the protocol version (`textus/3`) are unchanged.
+The user-facing CLI surface, the wire envelope shape, and the protocol version (`textus/4`) are unchanged.
 
 ## Pairing with other tools
 

data/lib/textus/action/ingest.rb

@@ -2,6 +2,7 @@
 
 require "fileutils"
 require "date"
+require "digest"
 
 module Textus
   module Action
@@ -22,6 +23,8 @@ module Textus
       view { |env| { "key" => env.key, "uid" => env.uid, "etag" => env.etag } }
 
       SOURCE_KINDS = %w[url file asset].freeze
+      CONTENT_HASH_ALGO = "sha256"
+      TOMBSTONE_RETAIN = %w[ingested_at].freeze
 
       def initialize(kind:, slug:, url: nil, path: nil, zone: nil, label: nil)
         super()
@@ -43,7 +46,22 @@ module Textus
           action: :ingest, actor: call.role, key: key,
         )
 
-        write_raw_entry(key, now, container, call)
+        content_hash = compute_content_hash
+        writer = Textus::Envelope::Writer.from(container: container, call: call)
+        mentry = container.manifest.resolver.resolve(key).entry
+        ts = now.iso8601
+        structured = build_structured(ts, container, now, content_hash)
+
+        index = Textus::Ports::RawIndex.new(root: container.root)
+        duplicate_key = find_duplicate(index, content_hash)
+
+        if duplicate_key && duplicate_key != key
+          supersede_entry(duplicate_key, key, structured, container, call, index)
+        else
+          env = write_raw_entry(key, structured, mentry, writer)
+          index.upsert(content_hash: content_hash, url: @url, key: key)
+          env
+        end
       end
 
       private
@@ -70,50 +88,100 @@ module Textus
         "raw.#{date}.#{@kind}-#{@slug}"
       end
 
-      def write_raw_entry(key, now, container, call)
-        ts     = now.iso8601
-        mentry = container.manifest.resolver.resolve(key).entry
-        writer = Textus::Envelope::Writer.from(container: container, call: call)
+      def compute_content_hash
+        digest = Digest::SHA256.new
+        case @kind
+        when "url"
+          digest.update(@url)
+        when "file", "asset"
+          digest.file(@path)
+        end
+        "#{CONTENT_HASH_ALGO}:#{digest.hexdigest}"
+      end
 
+      def build_structured(timestamp, container, now, content_hash)
+        base = { "ingested_at" => timestamp, "content_hash" => content_hash }
         case @kind
         when "url"
-          structured = {
-            "ingested_at" => ts,
-            "source" => { "kind" => "url", "url" => @url, "label" => @label || @url },
-            "body" => nil,
-          }
-          writer.put(key, mentry: mentry,
-                          payload: Textus::Envelope::Writer::Payload.new(
-                            meta: nil, body: nil, content: structured,
-                          ))
+          base.merge("source" => { "kind" => "url", "url" => @url, "label" => @label || @url },
+                     "body" => nil)
         when "file"
           body_content = File.read(@path)
-          structured = {
-            "ingested_at" => ts,
-            "source" => { "kind" => "file", "path" => @path,
-                          "label" => @label || File.basename(@path) },
-            "body" => body_content,
-          }
-          writer.put(key, mentry: mentry,
-                          payload: Textus::Envelope::Writer::Payload.new(
-                            meta: nil, body: nil, content: structured,
-                          ))
+          base.merge("source" => { "kind" => "file", "path" => @path,
+                                   "label" => @label || File.basename(@path) },
+                     "body" => body_content)
         when "asset"
           asset_rel = copy_asset_file(container, now)
-          structured = {
-            "ingested_at" => ts,
-            "source" => { "kind" => "asset",
-                          "label" => @label || File.basename(@path) },
-            "asset" => asset_rel,
-            "body" => nil,
-          }
-          writer.put(key, mentry: mentry,
-                          payload: Textus::Envelope::Writer::Payload.new(
-                            meta: nil, body: nil, content: structured,
-                          ))
+          base.merge("source" => { "kind" => "asset",
+                                   "label" => @label || File.basename(@path) },
+                     "asset" => asset_rel,
+                     "body" => nil)
         end
       end
 
+      def write_raw_entry(key, structured, mentry, writer)
+        writer.put(key, mentry: mentry,
+                        payload: Textus::Envelope::Writer::Payload.new(
+                          meta: nil, body: nil, content: structured,
+                        ))
+      end
+
+      def find_duplicate(index, content_hash)
+        dup = index.find_by_hash(content_hash)
+        return dup if dup
+
+        return unless @kind == "url"
+
+        index.find_by_url(@url)
+      end
+
+      def supersede_entry(old_key, new_key, structured, container, call, index)
+        old_mentry = container.manifest.resolver.resolve(old_key).entry
+        writer = Textus::Envelope::Writer.from(container: container, call: call)
+
+        reader = Textus::Envelope::Reader.from(container: container)
+        old_env = reader.read(old_key)
+        old_content = old_env&.content || {}
+        tombstone = {}
+        TOMBSTONE_RETAIN.each do |k|
+          tombstone[k] = old_content[k] if old_content.key?(k)
+        end
+        source_kind = old_content.dig("source", "kind")
+        tombstone["source"] = { "kind" => source_kind } if source_kind
+        tombstone["superseded_by"] = new_key
+
+        writer.put(old_key, mentry: old_mentry,
+                            payload: Textus::Envelope::Writer::Payload.new(
+                              meta: nil, body: nil, content: tombstone,
+                            ))
+
+        structured["supersedes"] = old_key
+        env = write_raw_entry(new_key, structured, container.manifest.resolver.resolve(new_key).entry, writer)
+
+        move_asset_file(container, old_content["asset"]) if @kind == "asset" && old_content["asset"]
+
+        index.upsert(content_hash: structured["content_hash"], url: @url, key: new_key)
+        env
+      end
+
+      def move_asset_file(container, old_asset_rel)
+        old_path = File.join(container.root, "assets", old_asset_rel)
+        return unless File.exist?(old_path)
+
+        now = Time.now.utc
+        date_path = now.strftime("%Y/%m/%d")
+        filename = File.basename(old_path)
+        new_dir = File.join(container.root, "assets", "raw", date_path, @zone)
+        new_path = File.join(new_dir, filename)
+
+        return if old_path == new_path
+
+        FileUtils.mkdir_p(new_dir)
+        FileUtils.mv(old_path, new_path)
+      rescue Errno::ENOENT, Errno::EACCES => e
+        warn "[textus ingest] could not move asset #{old_asset_rel}: #{e.message}"
+      end
+
       def copy_asset_file(container, now)
         date_path = now.strftime("%Y/%m/%d")
         filename  = File.basename(@path)

data/lib/textus/action/pulse.rb

@@ -46,7 +46,7 @@ module Textus
       end
 
       def index_etag(container)
-        path = container.manifest.resolver.resolve("artifacts.index").path
+        path = container.manifest.resolver.resolve("artifacts.system.index").path
         File.exist?(path) ? container.file_store.etag(path) : nil
       rescue Textus::Error
         nil

data/lib/textus/boot.rb

@@ -158,7 +158,7 @@ module Textus
         "contract_etag" => etag,
         "lanes" => lanes_for(manifest),
         "agent_quickstart" => agent_quickstart(manifest, container.audit_log),
-        "orientation" => read_artifact_content(container, "artifacts.orientation"),
+        "orientation" => read_artifact_content(container, "artifacts.config.orientation"),
         "context" => read_boot_context(container),
         "index_key" => index_key_if_present(container),
         "agent_protocol" => agent_protocol(manifest),
@@ -189,8 +189,8 @@ module Textus
     end
 
     def self.index_key_if_present(container)
-      res = container.manifest.resolver.resolve("artifacts.index")
-      res.path && File.exist?(res.path) ? "artifacts.index" : nil
+      res = container.manifest.resolver.resolve("artifacts.system.index")
+      res.path && File.exist?(res.path) ? "artifacts.system.index" : nil
     rescue Textus::Error
       nil
     end

data/lib/textus/cursor_store.rb

@@ -1,7 +1,7 @@
 require "fileutils"
 
 module Textus
-  # Per-role cursor cache under <root>/.run/state/cursor.<role>. A convenience so
+  # Per-role cursor cache under <root>/.state/cursors/<role>. A convenience so
   # `textus pulse` (no --since) means "since I last looked". Gitignored;
   # losing it just re-emits recent deltas, never corrupts the store. ADR 0036/0038.
   class CursorStore

data/lib/textus/init.rb

@@ -93,7 +93,7 @@ module Textus
 
     def self.setup_state_dirs(target_root)
       FileUtils.mkdir_p(Textus::Layout.audit_dir(target_root))
-      FileUtils.mkdir_p(Textus::Layout.state(target_root))
+      FileUtils.mkdir_p(Textus::Layout.cursors(target_root))
       FileUtils.mkdir_p(Textus::Layout.locks(target_root))
     end
 

data/lib/textus/layout.rb

@@ -1,9 +1,9 @@
 module Textus
   # Single source of truth for every path textus owns under a store root.
-  # All disposable runtime state nests under <root>/.run/ so the
+  # All disposable runtime state nests under <root>/.state/ so the
   # tracked/disposable boundary is a directory boundary. ADR 0038.
   module Layout
-    RUN = ".run"
+    RUN = ".state"
     DATA = "data"
 
     def self.data(root)
@@ -18,12 +18,12 @@ module Textus
       File.join(root, RUN)
     end
 
-    def self.state(root)
-      File.join(run(root), "state")
+    def self.cursors(root)
+      File.join(run(root), "cursors")
     end
 
     def self.cursor(root, role)
-      File.join(state(root), "cursor.#{role}")
+      File.join(cursors(root), role.to_s)
     end
 
     def self.locks(root)
@@ -51,19 +51,27 @@ module Textus
     end
 
     # Sentinels are machine-generated (the published target's sha), not authored
-    # source, so they live on the runtime side under `.run/` — git-ignored,
+    # source, so they live on the runtime side under `.state/` — git-ignored,
     # regenerated by the next build via content-identical adoption (ADR 0070,
     # superseding ADR 0038's `:config` classification).
     def self.sentinels(root)
       File.join(run(root), "sentinels")
     end
 
+    def self.indexes(root)
+      File.join(run(root), "indexes")
+    end
+
+    def self.raw_index(root)
+      File.join(indexes(root), "raw.yaml")
+    end
+
     def self.audit_log(root)
       File.join(audit_dir(root), "audit.log")
     end
 
     # The store's `.gitignore` body. Always ignores the runtime subtree
-    # (`.run/`, ADR 0038); when given untracked entry paths (entries marked
+    # (`.state/`, ADR 0038); when given untracked entry paths (entries marked
     # `tracked: false`), it also lists those so they stay protocol-readable but
     # uncommitted (ADR 0043, refining 0038). Generated, never hand-kept — no
     # drift between the manifest and the ignore file.

data/lib/textus/manifest/entry/publish/to_paths.rb

@@ -45,11 +45,8 @@ module Textus
 
           private
 
-          # A structured-data entry that textus owns: its `_meta` stays in the
-          # store, so the published file is the re-serialized meta-free content.
-          # An external (command) entry is opaque — never parse/re-serialize it.
           def strip_meta?(entry)
-            !entry.external? && %w[json yaml].include?(entry.format.to_s)
+            %w[json yaml].include?(entry.format.to_s)
           end
 
           def render_bytes(target, content, renderer, pctx)

data/lib/textus/ports/raw_index.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "yaml"
+require "fileutils"
+
+module Textus
+  module Ports
+    # Content-addressable index for the raw lane. Maps content hashes and URLs
+    # to their current canonical key. Lives under <root>/.state/indexes/raw.yaml
+    # (gitignored, regenerable — truth is in the on-disk raw entries).
+    class RawIndex
+      def initialize(root:)
+        @root = root
+        @path = Layout.raw_index(root)
+      end
+
+      attr_reader :path
+
+      def load
+        return empty_index unless File.exist?(@path)
+
+        YAML.safe_load_file(@path) || empty_index
+      rescue StandardError
+        empty_index
+      end
+
+      def save(index)
+        FileUtils.mkdir_p(File.dirname(@path))
+        File.write(@path, YAML.dump(index))
+        index
+      end
+
+      def find_by_hash(content_hash)
+        index = load
+        index["hashes"]&.fetch(content_hash, nil)
+      end
+
+      def find_by_url(url)
+        return nil unless url
+
+        index = load
+        index["urls"]&.fetch(url, nil)
+      end
+
+      def upsert(content_hash:, url:, key:)
+        index = load
+        index["hashes"] ||= {}
+        index["urls"] ||= {}
+        index["hashes"][content_hash] = key
+        index["urls"][url] = key if url
+        save(index)
+      end
+
+      private
+
+      def empty_index
+        { "hashes" => {}, "urls" => {} }
+      end
+    end
+  end
+end

data/lib/textus/version.rb

@@ -1,4 +1,4 @@
 module Textus
-  VERSION = "0.54.1"
+  VERSION = "0.54.2"
   PROTOCOL = "textus/4"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: textus
 version: !ruby/object:Gem::Version
-  version: 0.54.1
+  version: 0.54.2
 platform: ruby
 authors:
 - Patrick
@@ -291,6 +291,7 @@ files:
 - lib/textus/ports/job_store.rb
 - lib/textus/ports/job_store/job.rb
 - lib/textus/ports/publisher.rb
+- lib/textus/ports/raw_index.rb
 - lib/textus/ports/sentinel_store.rb
 - lib/textus/ports/storage/file_stat.rb
 - lib/textus/ports/storage/file_store.rb