igniter 0.4.3 → 0.5.0

data/docs/CONTENT_ADDRESSING_V1.md

@@ -0,0 +1,221 @@
+# Content-Addressed Computation — v1
+
+Content addressing gives `pure` executors a universal cache key derived from their logic
+(fingerprint) and their input values. The same computation — regardless of which
+contract, which execution, or which process produced it — always returns the cached result.
+
+This is the Nix/Merkle model applied to contract nodes: **identical inputs → identical
+output, fetched from cache**.
+
+## Quick Start
+
+```ruby
+require "igniter/extensions/content_addressing"
+
+class TaxCalculator < Igniter::Executor
+  pure                        # marks executor as side-effect-free
+  fingerprint "tax_calc_v1"   # optional: bumps the cache key when logic changes
+
+  def call(country:, amount:)
+    TAX_RATES[country] * amount
+  end
+end
+
+class InvoiceContract < Igniter::Contract
+  define do
+    input :country
+    input :amount, type: :numeric
+
+    compute :tax, depends_on: %i[country amount], call: TaxCalculator
+
+    output :tax
+  end
+end
+
+# First execution — computes and caches the result
+c1 = InvoiceContract.new(country: "UA", amount: 1000)
+c1.result.tax  # => 220.0  (computed)
+
+# Second execution with identical inputs — served from the content cache
+c2 = InvoiceContract.new(country: "UA", amount: 1000)
+c2.result.tax  # => 220.0  (cache hit — TaxCalculator was never called)
+```
+
+## How It Works
+
+For every `pure` executor, the resolver computes a **content key** before calling
+the executor:
+
+```
+key = SHA-256( fingerprint + "\x00" + stable_serialize(dep_values) )[0..23]
+```
+
+- **fingerprint** — the executor class name (or explicit `fingerprint "v1"` string).
+- **stable_serialize** — deterministic, order-independent serialization of all dependency
+  values: Hash keys are sorted, Array elements are serialized recursively, primitives use
+  `inspect`.
+
+The resolver looks up the key in the global `ContentAddressing.cache` before calling the
+executor. On a hit it uses the cached value and emits a `:node_content_cache_hit` event.
+On a miss it computes the result, stores it, and continues normally.
+
+## Executor DSL
+
+### `pure`
+
+Marks the executor as having no side effects. Enables content-addressed caching.
+Shorthand for `capabilities(:pure)`.
+
+```ruby
+class MyExecutor < Igniter::Executor
+  pure
+end
+```
+
+### `fingerprint "v1"`
+
+Sets an explicit version string used as the first component of the content key.
+Bump the fingerprint whenever the executor logic changes to immediately invalidate
+all cached results for this executor.
+
+```ruby
+class TaxCalculator < Igniter::Executor
+  pure
+  fingerprint "tax_calc_v2"   # bumped — old v1 cache entries are ignored
+end
+```
+
+If `fingerprint` is not set, the executor's class name is used. Anonymous executors
+use `"anonymous_executor"`.
+
+## Content Key
+
+`Igniter::ContentAddressing::ContentKey` is an immutable value object:
+
+```ruby
+key = Igniter::ContentAddressing::ContentKey.compute(TaxCalculator, { country: "UA", amount: 1000 })
+
+key.hex    # => "8f47805dc6dd7926"  (24-hex digest prefix)
+key.to_s   # => "ca:8f47805dc6dd7926"
+key == key # => true  (equality by hex, not object identity)
+key.frozen? # => true
+```
+
+Keys are equal if their hex values match — two keys computed from the same executor and
+the same dep values will always be equal, even if produced in separate processes.
+
+## Content Cache
+
+The global cache is a thread-safe in-process Hash by default:
+
+```ruby
+Igniter::ContentAddressing.cache          # => #<Igniter::ContentAddressing::Cache ...>
+Igniter::ContentAddressing.cache.stats    # => { size: 3, hits: 12, misses: 4 }
+Igniter::ContentAddressing.cache.size     # => 3
+Igniter::ContentAddressing.cache.clear    # clears entries and resets counters
+```
+
+### Distributed Cache (Redis)
+
+Replace the default cache with any object implementing `#fetch(key)` and `#store(key, value)`:
+
+```ruby
+class RedisContentCache
+  def initialize(redis, ttl: 3600)
+    @redis = redis
+    @ttl   = ttl
+  end
+
+  def fetch(key)
+    val = @redis.get(key.to_s)
+    val ? Marshal.load(val) : nil
+  end
+
+  def store(key, value)
+    @redis.setex(key.to_s, @ttl, Marshal.dump(value))
+  end
+end
+
+Igniter::ContentAddressing.cache = RedisContentCache.new(Redis.new)
+```
+
+With a shared Redis cache, results are reused across deployments, canary instances,
+and background workers — any process that computes `TaxCalculator(country: "UA", amount: 1000)`
+once populates the cache for all others.
+
+## Runtime Events
+
+The resolver emits `:node_content_cache_hit` when a cached result is used:
+
+```ruby
+contract.execution.events.select { |e| e.type == :node_content_cache_hit }
+# => [#<Event type=:node_content_cache_hit node=:tax ...>]
+```
+
+## Loading
+
+```ruby
+require "igniter/extensions/content_addressing"
+```
+
+This single require:
+1. Loads `lib/igniter/content_addressing.rb` (ContentKey, Cache, module-level cache accessor).
+2. Activates the resolver hooks via `lib/igniter/runtime/resolver.rb`.
+
+Non-pure executors are completely unaffected — no overhead, no behavior change.
+
+## Combining with Temporal Contracts
+
+When used with [temporal contracts](TEMPORAL_V1.md), the `as_of` value is part of the
+dependency hash and therefore part of the content key. Historical and current timestamps
+produce distinct cache entries, and identical timestamps produce cache hits:
+
+```ruby
+require "igniter/temporal"
+require "igniter/extensions/content_addressing"
+
+class TaxRateExecutor < Igniter::Temporal::Executor
+  pure
+  fingerprint "tax_rate_v1"
+
+  def call(country:, as_of:)
+    RATES.dig(country, as_of.year) || 0.0
+  end
+end
+```
+
+Replaying a historical execution with the same `as_of` will hit the cache — the executor
+is never called twice for the same (country, year) pair.
+
+## Fingerprint Invalidation Pattern
+
+When you deploy a fix to a `pure` executor, bump the fingerprint so the old cached
+results are ignored immediately:
+
+```ruby
+# Before fix
+class DiscountCalculator < Igniter::Executor
+  pure
+  fingerprint "discount_v1"
+  def call(amount:, code:) = amount * 0.9  # bug: off-by-one in rate
+end
+
+# After fix
+class DiscountCalculator < Igniter::Executor
+  pure
+  fingerprint "discount_v2"  # <-- bumped; v1 cache entries are silently ignored
+  def call(amount:, code:) = amount * 0.85
+end
+```
+
+Old cache entries with `"discount_v1"` prefix in the key are never read again.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `lib/igniter/content_addressing.rb` | `ContentKey`, `Cache`, module-level `cache` accessor |
+| `lib/igniter/extensions/content_addressing.rb` | Entry point (`require "igniter/content_addressing"`) |
+| `lib/igniter/executor.rb` | `pure`, `fingerprint`, `content_fingerprint`, `pure?` class DSL |
+| `lib/igniter/runtime/resolver.rb` | `build_content_key` + cache fetch/store hooks in `resolve_compute` |
+| `spec/igniter/content_addressing_spec.rb` | 19 examples |

data/docs/DATAFLOW_V1.md

@@ -0,0 +1,274 @@
+# Incremental Dataflow — `mode: :incremental`
+
+> **Status**: v1 shipped (2026-04)
+> **Require**: `require "igniter/extensions/dataflow"`
+
+---
+
+## Overview
+
+Incremental dataflow adds **O(change)** execution to `collection` nodes.
+Instead of re-running every child contract on every `resolve_all`, the runtime:
+
+1. Computes a **Diff** — which item keys were added, removed, or changed since the last call.
+2. **Skips** child contracts for unchanged items (reuses cached results).
+3. **Retracts** removed items from the result automatically.
+4. **Applies sliding-window filtering** before the diff so memory stays bounded.
+
+This is inspired by differential dataflow (Frank McSherry / Materialize) adapted to
+Igniter's contract-graph model. It makes sensor pipelines, live analytics, and
+event-driven workflows dramatically more efficient without any API changes to the
+child contracts.
+
+---
+
+## Quick Start
+
+```ruby
+require "igniter/extensions/dataflow"
+
+class SensorAnalysis < Igniter::Contract
+  define do
+    input  :sensor_id
+    input  :value, type: :numeric
+    compute :status, depends_on: :value do |value:|
+      value > 75 ? :critical : value > 25 ? :warning : :normal
+    end
+    output :status
+  end
+end
+
+class SensorPipeline < Igniter::Contract
+  define do
+    input :readings, type: :array
+
+    collection :processed,
+               with: :readings,
+               each: SensorAnalysis,
+               key:  :sensor_id,
+               mode: :incremental,
+               window: { last: 1000 }  # optional
+
+    output :processed
+  end
+end
+
+pipeline = SensorPipeline.new(readings: initial_batch)
+pipeline.resolve_all
+
+# Push a diff — only changed sensors re-run
+pipeline.feed_diff(:readings,
+  add:    [{ sensor_id: "new-1", value: 10 }],
+  update: [{ sensor_id: "tmp-2", value: 90 }],
+  remove: ["hum-1"]
+)
+pipeline.resolve_all
+
+diff = pipeline.collection_diff(:processed)
+diff.added      # => ["new-1"]
+diff.changed    # => ["tmp-2"]
+diff.removed    # => ["hum-1"]
+diff.unchanged  # => ["tmp-1", ...]
+diff.processed_count  # => 2  (new-1 + tmp-2)
+```
+
+---
+
+## DSL: `collection` with `mode: :incremental`
+
+```ruby
+collection :name,
+           with: :input_name,   # source input (must be Array)
+           each: ChildContract,  # child contract class
+           key:  :field_name,    # unique key field in each item Hash
+           mode: :incremental,   # enables differential execution
+           window: { last: N }   # optional: keep last N items
+           # window: { seconds: 60, field: :ts }  # optional: time window
+```
+
+### Parameters
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `with:` | Symbol | yes | Input name that holds the Array |
+| `each:` | Class | yes | Child contract class |
+| `key:` | Symbol | yes | Hash field used as unique item identifier |
+| `mode:` | Symbol | yes | `:incremental` to enable differential execution |
+| `window:` | Hash | no | Sliding window filter (see below) |
+
+---
+
+## Sliding Window
+
+The window is applied **before** diff computation, so items outside the window
+are treated as if they were never present. This bounds both memory and latency.
+
+### `{ last: N }`
+
+```ruby
+window: { last: 500 }
+```
+
+Keeps the **last N items** from the input array.
+
+### `{ seconds: N, field: :sym }`
+
+```ruby
+window: { seconds: 60, field: :ts }
+```
+
+Keeps only items where `item[:ts] >= Time.now - 60`. The `:ts` field must
+hold a `Time` object (or something that responds to `>=` for comparison with
+`Time.now - N`).
+
+---
+
+## `#feed_diff` — event-style push
+
+Instead of replacing the full input array, push only the delta:
+
+```ruby
+contract.feed_diff(:input_name,
+  add:    [{ sensor_id: "x", value: 5 }],      # new items
+  remove: ["old-sensor"],                        # keys to remove
+  update: [{ sensor_id: "tmp-2", value: 90 }]  # replace by key
+)
+```
+
+`remove:` accepts either **keys** (scalars) or **full Hash items** (the key is
+extracted automatically using the collection node's `key_name`).
+
+Returns `self` for chaining.
+
+**Raises** `ArgumentError` when no incremental collection node uses the given input name.
+
+---
+
+## `#collection_diff` — inspect what changed
+
+```ruby
+diff = contract.collection_diff(:collection_node_name)
+```
+
+Returns `nil` before the first `resolve_all`, or an `Igniter::Dataflow::Diff`:
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `added` | `Array` | Keys of items added in the last resolve |
+| `removed` | `Array` | Keys of items removed in the last resolve |
+| `changed` | `Array` | Keys of items whose content changed |
+| `unchanged` | `Array` | Keys of items that were identical — no child contract re-run |
+| `any_changes?` | Bool | `true` if anything was added, changed, or removed |
+| `processed_count` | Int | `added.size + changed.size` — number of child contracts that ran |
+| `explain` | String | Human-readable summary |
+| `to_h` | Hash | Serialisable representation |
+
+---
+
+## `IncrementalCollectionResult`
+
+The `output` value for an incremental collection is an `IncrementalCollectionResult`
+(inherits from `CollectionResult`), which adds:
+
+```ruby
+result = contract.result.collection_name  # => IncrementalCollectionResult
+result.diff         # => Igniter::Dataflow::Diff
+result.summary      # extends base summary with :added/:removed/:changed/:unchanged counts
+result.as_json      # extends base JSON with :diff key
+```
+
+All existing `CollectionResult` methods work as before:
+`result["key"]`, `result.keys`, `result.successes`, `result.failures`, etc.
+
+---
+
+## Internal Architecture
+
+```
+resolve_incremental_collection(node)
+  │
+  ├─ Resolve source items (same as :collect)
+  ├─ Apply WindowFilter (if node.window present)
+  ├─ DiffState#compute_diff → Diff
+  │     ├─ partition items into added / changed / unchanged
+  │     └─ identify removed keys (in @snapshots but not in current)
+  │
+  ├─ For each UNCHANGED key → reuse @cached_items[key]
+  ├─ For each REMOVED  key → DiffState#retract!(key)
+  ├─ For each ADDED/CHANGED key → run child contract → DiffState#update!(key, ...)
+  │
+  └─ Return NodeState(IncrementalCollectionResult)
+```
+
+### `DiffState`
+
+One `DiffState` instance per collection node, stored on `Execution#diff_states`.
+Persists across `update_inputs` calls for the lifetime of the contract execution.
+
+```
+@snapshots    Hash{ key => fingerprint }   change detection
+@cached_items Hash{ key => Item }          cached child results
+```
+
+The **fingerprint** is content-based (order-independent for Hash items), so
+reordering keys in an item Hash does not trigger a re-run.
+
+---
+
+## Compiler Validation
+
+The compiler validates `window:` options at definition time:
+
+```ruby
+# Accepted
+window: { last: 100 }
+window: { seconds: 60, field: :ts }
+
+# Rejected at compile time (raises CompileError)
+window: { bogus: 1 }           # unknown key
+window: { seconds: 60 }        # missing :field
+window: { last: -1 }           # non-positive integer
+```
+
+---
+
+## Load Guard
+
+If `mode: :incremental` is used without requiring the extension:
+
+```
+Igniter::ResolutionError: Incremental dataflow requires the dataflow extension.
+  Add: require 'igniter/extensions/dataflow'
+```
+
+---
+
+## Performance Characteristics
+
+| Scenario | Child contracts run |
+|----------|-------------------|
+| First resolve (N items) | N |
+| All items unchanged | 0 |
+| K items changed | K |
+| K items added | K |
+| K items removed | 0 (retraction only) |
+| Mixed (K changed + M added) | K + M |
+
+The `window:` filter caps the maximum work per round at `window_size`, not total dataset size.
+
+---
+
+## File Reference
+
+| File | Purpose |
+|------|---------|
+| `lib/igniter/dataflow.rb` | Entry point |
+| `lib/igniter/dataflow/diff.rb` | `Diff` struct |
+| `lib/igniter/dataflow/diff_state.rb` | Per-node mutable state |
+| `lib/igniter/dataflow/window_filter.rb` | `WindowFilter` — `last:` and `seconds:` |
+| `lib/igniter/dataflow/incremental_collection_result.rb` | Result type with `.diff` |
+| `lib/igniter/extensions/dataflow.rb` | Extension — patches `Contract` with `feed_diff`, `collection_diff` |
+| `lib/igniter/runtime/resolver.rb` | `resolve_incremental_collection` |
+| `lib/igniter/runtime/execution.rb` | `diff_state_for(node_name)` |
+| `spec/igniter/dataflow_spec.rb` | 33 examples |
+| `examples/dataflow.rb` | Sensor pipeline demo |