textus 0.14.4 → 0.15.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d1e2a12b234bfc677ce44ec7b49b3b04b4ac69abbc15f13496911c82d89549c8
-  data.tar.gz: 6b83336a05cb3cee509d3e0b90ce2048ed57fdeb5fbc603f22303e10c2d5c27c
+  metadata.gz: 54b9e7266b017d02abba0f6daeba977c580c07f1bdab798ebc7a9b943be555cd
+  data.tar.gz: 47ebf2a56523dbf33058fe95c45163899a6f7b91500a2f749ce3d731e60ab502
 SHA512:
-  metadata.gz: f126d898385e0d0236a1818a741923f8ee9f88f1f13cab8ef37bbb810d14145904186084f72d0fbc5a4d6bfdf53b5e7e68699f56db4f9aa02d22772019a11316
-  data.tar.gz: 054ea4fea78cf26d570319d49f2a0ffbd4023e6355b56b8830650e7d59b1b57ae2de7b158ef0611b465369136e11b292ba9f968113cd425ecd1cfed1bf96633d
+  metadata.gz: 76abb1c22c3f519574dfd310a7ff2162b023bbbc8667b2e4dc2c32edbb94d432bf507620116dbcbc1f7ce3328964515ee8eded62abe1c833249b3b2bb1bd9fce
+  data.tar.gz: a405b5d81159c8dd0638e9ca6f4fca419358792c7a3a29f64cb0b19386553f37aa091f913c65ad078a039205ad6ac0368af681eba455fcb4c1787bcfb213ac05

data/CHANGELOG.md

@@ -9,6 +9,71 @@ The **gem version** (`0.x.y`) is distinct from the **protocol version**
 bump is a breaking change that requires a store migration; the gem version
 tracks both additive improvements and breaking protocol bumps independently.
 
+## 0.15.0 — 2026-05-26
+
+### Breaking
+
+- `Application::Reads::Get#call` is now a **pure read**: it returns the
+  on-disk envelope annotated with a freshness verdict, and never
+  triggers refresh. `Reads::Get.new` no longer accepts `orchestrator:`.
+  Callers that relied on refresh-on-read should switch to
+  `Application::Reads::GetOrRefresh` (new), accessible via
+  `ops.reads.get_or_refresh`. The CLI verb `textus get` is migrated
+  internally; users of `textus get` see no behavior change.
+- `Application::Context#bypass_freshness?` and the `bypass_freshness:`
+  kwarg on `Context.new` / `Operations.for` are **removed**. They
+  shipped in 0.14.4 as a workaround; with `Reads::Get` now pure by
+  default, the flag is dead code. Callers passing `bypass_freshness:`
+  will see `ArgumentError`.
+- `Projection.new` signature is **breaking**: now
+  `Projection.new(reader:, spec:, lister:, transform_resolver:, transform_context:)`.
+  `Projection` no longer constructs its own `Operations` chain. Callers
+  inject collaborators. `Builder::Pipeline` is migrated internally.
+- Intake handlers now receive `args: { trigger_key:, leaf_segments: }`
+  instead of `args: {}`. Handlers that destructure `args` should
+  expect the new keys. Handlers that pass `args` through unchanged
+  are unaffected.
+
+### Fixed
+
+- **Bug 1 / single-flight.** `Refresh::Orchestrator#run_timed` now
+  probes the per-leaf lock before forking the detached refresh
+  worker. If the lock is held (by a sibling process or earlier fork),
+  the orchestrator returns `Outcome::Detached` without spawning a
+  redundant worker. Prevents wasted forks when the same key is read
+  concurrently across processes.
+- **Bug 2 / leaf-aware intake.** `Refresh::Worker` now keeps the
+  `remaining` segments from `Manifest#resolve(key)` and passes them
+  to the intake handler as `args: { trigger_key:, leaf_segments: }`.
+  Handlers can scope to one leaf instead of re-processing the full
+  parent `intake_config` for every leaf refresh.
+- **`textus refresh stale` now exits 0 on success.** Previously the
+  verb fell off the end returning `nil`, which propagated up through
+  `CLI.run` to `exe/textus:4`'s `exit nil` and raised `TypeError`,
+  exit-coding 1 on every successful refresh. Fixed by returning an
+  explicit Integer from the verb. The verb return-value contract is
+  now codified: every verb's `#call` returns Integer (or `nil` →
+  treated as 0); `CLI.run` coerces. (#61)
+
+### Added
+
+- `Application::Reads::GetOrRefresh` — explicit composition of pure
+  `Reads::Get` with the refresh orchestrator. Use for interactive
+  reads that want freshest-obtainable envelopes.
+- `ops.reads.get_or_refresh` accessor.
+
+### Migration
+
+If your code (or hook / handler / extension) called:
+
+| Old | New |
+|---|---|
+| `ops.reads.get.call(key)` to get the freshest envelope | `ops.reads.get_or_refresh.call(key)` |
+| `ops.reads.get.call(key)` for pure read | unchanged; now also pure semantics |
+| `Operations.for(store, bypass_freshness: true)` | `Operations.for(store)` |
+| `Projection.new(store, spec)` | `Projection.new(reader:, spec:, lister:, transform_resolver:, transform_context:)` — see `builder/pipeline.rb` for canonical wiring |
+| Handler signature `lambda { \|store:, config:, args:\| ... }` with `args == {}` | unchanged — `args` is now populated with `:trigger_key` and `:leaf_segments`; handlers that ignore them keep working |
+
 ## 0.14.4 — 2026-05-26
 
 ### Fixed

data/docs/conventions.md

@@ -105,6 +105,17 @@ textus refresh-stale --zone=intake --as=runner   # in cron / CI
 
 See [`./zones.md` §6](zones.md) for the full intake contract and [`./events.md`](events.md) for writing custom handlers.
 
+### Read vs. refresh
+
+There are two read operations, and the difference matters in custom code:
+
+| Operation | Triggers refresh? | Use for |
+|-----------|-------------------|---------|
+| `ops.reads.get` | No — pure read of on-disk envelope + freshness verdict | build / materialization paths, schema tooling, any context where a side-effecting read would surprise the caller |
+| `ops.reads.get_or_refresh` | Yes — composes `get` with the orchestrator per the rule's `on_stale` policy | interactive reads (`textus get`, dashboards) where the caller wants the freshest envelope obtainable |
+
+Build always uses the pure path; injecting refresh into materialization caused the cascading-staleness incident behind issue #59. Pick `get_or_refresh` only when you genuinely want side effects on read.
+
 ## Body content
 
 - **Bodies are Markdown.** Headings, lists, code fences — whatever a human or agent finds useful.

data/lib/textus/application/context.rb

@@ -9,13 +9,12 @@ module Textus
         new(store: store, role: "human")
       end
 
-      def initialize(store:, role:, correlation_id: nil, clock: Time, dry_run: false, bypass_freshness: false)
+      def initialize(store:, role:, correlation_id: nil, clock: Time, dry_run: false)
         @store             = store
         @role              = role.to_s
         @correlation_id    = correlation_id || SecureRandom.uuid
         @clock             = clock
         @dry_run           = dry_run
-        @bypass_freshness  = bypass_freshness
         @now               = nil
       end
 
@@ -27,10 +26,6 @@ module Textus
         @dry_run
       end
 
-      def bypass_freshness?
-        @bypass_freshness
-      end
-
       def can_write?(zone)
         store.manifest.permission_for(zone.to_s).allows_write?(role)
       end
@@ -46,7 +41,6 @@ module Textus
           correlation_id: @correlation_id,
           clock: @clock,
           dry_run: @dry_run,
-          bypass_freshness: @bypass_freshness,
         )
       end
     end

data/lib/textus/application/reads/get.rb

@@ -1,17 +1,20 @@
 module Textus
   module Application
     module Reads
+      # Pure read: returns the on-disk envelope annotated with a freshness
+      # verdict. Never triggers refresh; never invokes the orchestrator.
+      #
+      # For interactive reads that want refresh-on-stale, use
+      # `Reads::GetOrRefresh`, which composes this with the orchestrator.
       class Get
-        def initialize(ctx:, orchestrator:, evaluator: Textus::Domain::Freshness::Evaluator)
-          @ctx          = ctx
-          @orchestrator = orchestrator
-          @evaluator    = evaluator
+        def initialize(ctx:, evaluator: Textus::Domain::Freshness::Evaluator)
+          @ctx       = ctx
+          @evaluator = evaluator
         end
 
         def call(key)
           envelope = @ctx.store.reader.read_raw_envelope(key)
           return nil if envelope.nil?
-          return annotate_fresh(envelope) if @ctx.bypass_freshness?
 
           policy_set = @ctx.store.manifest.rules_for(key)
           refresh_policy = policy_set.refresh
@@ -20,37 +23,15 @@ module Textus
           policy = refresh_policy.to_freshness_policy
           verdict = @evaluator.call(policy, envelope, now: @ctx.now)
 
-          return annotate(envelope, verdict, refreshing: false) if verdict.fresh?
-
-          action = policy.decide(verdict)
-          outcome = @orchestrator.execute(action, key: key)
-
-          case outcome
-          when Textus::Domain::Outcome::Skipped
-            annotate(envelope, verdict, refreshing: false)
-          when Textus::Domain::Outcome::Refreshed
-            fresh_verdict = @evaluator.call(policy, outcome.envelope, now: @ctx.now)
-            annotate(outcome.envelope, fresh_verdict, refreshing: false)
-          when Textus::Domain::Outcome::Detached
-            annotate(envelope, verdict, refreshing: true)
-          when Textus::Domain::Outcome::Failed
-            annotate(envelope, verdict, refreshing: false, refresh_error: outcome.error.message)
-          end
+          envelope.with(freshness: {
+                          "stale" => verdict.stale?,
+                          "stale_reason" => verdict.reason,
+                          "refreshing" => false,
+                        })
         end
 
         private
 
-        def annotate(envelope, verdict, refreshing:, refresh_error: nil)
-          fresh = {
-            "stale" => verdict.stale?,
-            "stale_reason" => verdict.reason,
-            "refreshing" => refreshing,
-          }
-          fresh["refresh_error"] = refresh_error if refresh_error
-          envelope.with(freshness: fresh)
-        end
-
-        # No refresh policy applies to this key — treat as fresh, skip evaluation/orchestration.
         def annotate_fresh(envelope)
           envelope.with(freshness: { "stale" => false, "stale_reason" => nil, "refreshing" => false })
         end

data/lib/textus/application/reads/get_or_refresh.rb

@@ -0,0 +1,51 @@
+module Textus
+  module Application
+    module Reads
+      # Composes pure `Reads::Get` with the refresh orchestrator: runs Get
+      # to obtain the envelope and freshness verdict, then if the verdict
+      # is stale and the rule's `on_stale` policy demands action, hands
+      # off to the orchestrator. Use for interactive reads where the
+      # caller wants the freshest obtainable envelope.
+      #
+      # Pure reads (build, projection, schema tooling) should use
+      # `Reads::Get` directly; it has no orchestrator dependency.
+      class GetOrRefresh
+        def initialize(ctx:, get:, orchestrator:)
+          @ctx          = ctx
+          @get          = get
+          @orchestrator = orchestrator
+        end
+
+        def call(key)
+          envelope = @get.call(key)
+          return nil if envelope.nil?
+          return envelope unless envelope.freshness["stale"]
+
+          policy_set = @ctx.store.manifest.rules_for(key)
+          refresh_policy = policy_set.refresh
+          return envelope if refresh_policy.nil?
+
+          policy = refresh_policy.to_freshness_policy
+          verdict = Textus::Domain::Freshness::Verdict.stale(envelope.freshness["stale_reason"])
+          action = policy.decide(verdict)
+          outcome = @orchestrator.execute(action, key: key)
+
+          case outcome
+          when Textus::Domain::Outcome::Skipped
+            envelope
+          when Textus::Domain::Outcome::Refreshed
+            outcome.envelope.with(
+              freshness: { "stale" => false, "stale_reason" => nil, "refreshing" => false },
+            )
+          when Textus::Domain::Outcome::Detached
+            envelope.with(freshness: envelope.freshness.merge("refreshing" => true))
+          when Textus::Domain::Outcome::Failed
+            envelope.with(
+              freshness: envelope.freshness.merge("refresh_error" => outcome.error.message),
+            )
+          end
+        end
+      end
+    end
+  end
+end

data/lib/textus/application/refresh/orchestrator.rb

@@ -47,6 +47,15 @@ module Textus
 
           if thread.alive?
             thread.kill
+
+            # Single-flight: if a sibling process / earlier fork holds the
+            # per-leaf lock, don't fork another worker — they're already
+            # doing this work.
+            probe = Textus::Infra::Refresh::Lock.new(root: @store_root, key: key)
+            return Textus::Domain::Outcome::Detached.new unless probe.try_acquire
+
+            probe.release
+
             store_view = @store ? Textus::Application::Context.new(store: @store, role: @role) : nil
             payload = { key: key, started_at: Time.now.utc.iso8601, budget_ms: budget_ms }
             payload[:store] = store_view if store_view

data/lib/textus/application/refresh/worker.rb

@@ -12,11 +12,11 @@ module Textus
         end
 
         def run(key)
-          mentry, path, = @ctx.store.manifest.resolve(key)
+          mentry, path, remaining = @ctx.store.manifest.resolve(key)
           raise UsageError.new("no intake declared for '#{key}'") unless mentry.intake_handler
 
           before_etag = File.exist?(path) ? Etag.for_file(path) : nil
-          result = fetch_with_bus(key, mentry)
+          result = fetch_with_bus(key, mentry, remaining)
           persist_and_notify(key, mentry, result, before_etag)
         end
 
@@ -31,17 +31,21 @@ module Textus
           rule&.refresh&.fetch_timeout_seconds || FETCH_TIMEOUT_SECONDS
         end
 
-        def fetch_with_bus(key, mentry)
+        def fetch_with_bus(key, mentry, remaining)
           callable = @ctx.store.registry.rpc_callable(:resolve_intake, mentry.intake_handler)
           @bus.publish(:refresh_started, store: read_view, key: key, mode: :sync,
                                          correlation_id: @ctx.correlation_id)
-          call_intake(key, mentry, callable)
+          call_intake(key, mentry, callable, remaining)
         end
 
-        def call_intake(key, mentry, callable)
+        def call_intake(key, mentry, callable, remaining)
           timeout = fetch_timeout_for(key)
           Timeout.timeout(timeout) do
-            callable.call(store: @ctx, config: mentry.intake_config, args: {})
+            callable.call(
+              store: @ctx,
+              config: mentry.intake_config,
+              args: { trigger_key: key, leaf_segments: remaining || [] },
+            )
           end
         rescue Timeout::Error
           @bus.publish(:refresh_failed, store: read_view, key: key, error_class: "Timeout::Error",

data/lib/textus/builder/pipeline.rb

@@ -62,7 +62,14 @@ module Textus
         # 1. Load sources + project + reduce
         data =
           if mentry.projection
-            Projection.new(store, mentry.projection, bypass_freshness: true).run
+            ops = Operations.for(store)
+            Projection.new(
+              reader: ops.reads.get.method(:call),
+              spec: mentry.projection,
+              lister: ops.reads.list.method(:call),
+              transform_resolver: ->(name) { store.registry.rpc_callable(:transform_rows, name) },
+              transform_context: Application::Context.system(store),
+            ).run
           else
             { "entries" => [], "count" => 0, "generated_at" => Time.now.utc.iso8601 }
           end

data/lib/textus/cli/verb/get.rb

@@ -6,7 +6,7 @@ module Textus
 
         def call(store)
           key = positional.shift or raise UsageError.new("get requires a key")
-          result = operations_for(store).reads.get.call(key)
+          result = operations_for(store).reads.get_or_refresh.call(key)
           raise Textus::UnknownKey.new(key, suggestions: store.manifest.suggestions_for(key)) if result.nil?
 
           emit(result.to_h_for_wire)

data/lib/textus/cli/verb/refresh_stale.rb

@@ -10,7 +10,7 @@ module Textus
           ctx = context_for(store)
           result = Textus::Application::Refresh::All.call(ctx, prefix: prefix, zone: zone)
           emit(result)
-          exit(1) unless result["ok"]
+          result["ok"] ? 0 : 1
         end
       end
     end

data/lib/textus/cli.rb

@@ -47,21 +47,35 @@ module Textus
       verb = argv.shift
       raise UsageError.new("missing verb") if verb.nil?
 
-      case verb
-      when "--version", "-v" then @stdout.puts(VERSION)
-                                  0
-      when "--help", "-h"    then print_help
-                                  0
-      else
-        klass = VERBS[verb] or raise UsageError.new("unknown verb: #{verb}")
-        dispatch(klass, argv)
-      end
+      result =
+        case verb
+        when "--version", "-v" then @stdout.puts(VERSION)
+                                    0
+        when "--help", "-h"    then print_help
+                                    0
+        else
+          klass = VERBS[verb] or raise UsageError.new("unknown verb: #{verb}")
+          dispatch(klass, argv)
+        end
+
+      coerce_exit_code(result)
     rescue Textus::Error => e
       emit_error(e)
     end
 
     private
 
+    def coerce_exit_code(value)
+      case value
+      when Integer then value
+      when true, nil then 0
+      when false then 1
+      else
+        @stderr.puts("warning: verb returned non-Integer #{value.class}; treating as 0")
+        0
+      end
+    end
+
     def store
       @store ||= Store.discover(@cwd, root: @root_arg)
     end

data/lib/textus/operations/reads.rb

@@ -1,12 +1,29 @@
 module Textus
   class Operations
     class Reads
+      # `get`              — pure read; returns envelope + freshness verdict;
+      #                      never triggers refresh; no orchestrator dependency.
+      # `get_or_refresh`   — composes `get` with the refresh orchestrator; runs
+      #                      refresh per policy when the verdict says stale.
+      #                      Use this for interactive reads where the caller
+      #                      wants the freshest envelope obtainable.
+      #
+      # Pick `get` for materialization paths (build, projection, schema tooling).
+      # Pick `get_or_refresh` for interactive `textus get` and equivalent.
       def initialize(ctx)
         @ctx = ctx
       end
 
       def get
-        Application::Reads::Get.new(ctx: @ctx, orchestrator: orchestrator)
+        Application::Reads::Get.new(ctx: @ctx)
+      end
+
+      def get_or_refresh # rubocop:disable Naming/AccessorMethodName
+        Application::Reads::GetOrRefresh.new(
+          ctx: @ctx,
+          get: get,
+          orchestrator: orchestrator,
+        )
       end
 
       def freshness      = Application::Reads::Freshness.new(ctx: @ctx)

data/lib/textus/operations.rb

@@ -4,18 +4,18 @@ module Textus
   #
   #   ops = Textus::Operations.for(store, role: "agent")
   #   ops.writes.put.call(key, body: "...")
-  #   ops.reads.get.call(key)
+  #   ops.reads.get.call(key)               # pure read
+  #   ops.reads.get_or_refresh.call(key)    # read + refresh-on-stale
   #   ops.refresh.worker.call(key)
   #
   # Replaces the prior `Textus::Composition` module (deleted in v0.12.2).
   class Operations
-    def self.for(store, role: Role::DEFAULT, correlation_id: nil, dry_run: false, bypass_freshness: false)
+    def self.for(store, role: Role::DEFAULT, correlation_id: nil, dry_run: false)
       ctx = Application::Context.new(
         store: store,
         role: role,
         correlation_id: correlation_id,
         dry_run: dry_run,
-        bypass_freshness: bypass_freshness,
       )
       new(ctx)
     end

data/lib/textus/projection.rb

@@ -6,10 +6,18 @@ module Textus
     MAX_LIMIT = 1000
     REDUCER_TIMEOUT_SECONDS = 2
 
-    def initialize(store, spec, bypass_freshness: false)
-      @store = store
-      @spec = spec || {}
-      @bypass_freshness = bypass_freshness
+    # `reader` — a callable `->(key) { envelope_or_nil }`. Caller picks
+    #   semantics: pure read (`ops.reads.get`) for materialization paths;
+    #   `ops.reads.get_or_refresh` if you want refresh-on-stale.
+    # `lister` — a callable `->(prefix:) { [ { "key" => ... }, ... ] }`.
+    # `transform_resolver` — a callable `->(name) { callable_or_raise }`.
+    # `transform_context` — `Application::Context` handed to the transform reducer.
+    def initialize(reader:, spec:, lister:, transform_resolver:, transform_context:)
+      @reader             = reader
+      @spec               = spec || {}
+      @lister             = lister
+      @transform_resolver = transform_resolver
+      @transform_context  = transform_context
       @limit = (@spec["limit"] || MAX_LIMIT).to_i
       raise InvalidProjection.new("limit #{@limit} exceeds max #{MAX_LIMIT}") if @limit > MAX_LIMIT
     end
@@ -17,9 +25,8 @@ module Textus
     def run
       keys = collect_keys
       explicit_pluck = !@spec["pluck"].nil? && @spec["pluck"] != "*"
-      ops = Operations.for(@store, bypass_freshness: @bypass_freshness)
       rows = keys.map do |key|
-        env = ops.reads.get.call(key)
+        env = @reader.call(key)
         row = pluck(env.meta, env.body)
         explicit_pluck ? row : row.merge("_key" => key)
       end
@@ -41,10 +48,9 @@ module Textus
 
     def apply_reducer(rows)
       name = @spec["transform"] or return rows
-      callable = @store.registry.rpc_callable(:transform_rows, name)
-      view = Application::Context.system(@store)
+      callable = @transform_resolver.call(name)
       Timeout.timeout(REDUCER_TIMEOUT_SECONDS) do
-        callable.call(store: view, rows: rows, config: @spec["transform_config"] || {})
+        callable.call(store: @transform_context, rows: rows, config: @spec["transform_config"] || {})
       end
     rescue Timeout::Error
       raise UsageError.new("transform_rows '#{name}' exceeded #{REDUCER_TIMEOUT_SECONDS}s timeout")
@@ -52,8 +58,7 @@ module Textus
 
     def collect_keys
       prefixes = Array(@spec["select"])
-      ops = Operations.for(@store, bypass_freshness: @bypass_freshness)
-      prefixes.flat_map { |p| ops.reads.list.call(prefix: p).map { |row| row["key"] } }.uniq
+      prefixes.flat_map { |p| @lister.call(prefix: p).map { |row| row["key"] } }.uniq
     end
 
     def pluck(frontmatter, _body)

data/lib/textus/version.rb

@@ -1,4 +1,4 @@
 module Textus
-  VERSION = "0.14.4"
+  VERSION = "0.15.0"
   PROTOCOL = "textus/3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: textus
 version: !ruby/object:Gem::Version
-  version: 0.14.4
+  version: 0.15.0
 platform: ruby
 authors:
 - Patrick
@@ -115,6 +115,7 @@ files:
 - lib/textus/application/reads/deps.rb
 - lib/textus/application/reads/freshness.rb
 - lib/textus/application/reads/get.rb
+- lib/textus/application/reads/get_or_refresh.rb
 - lib/textus/application/reads/list.rb
 - lib/textus/application/reads/policy_explain.rb
 - lib/textus/application/reads/published.rb