shifty 0.5.0 → 0.6.0

data/docs/use_cases.md

@@ -0,0 +1,59 @@
+# Shifty Framework: Use Cases
+
+## Introduction
+
+Shifty is a Ruby framework designed for building data processing pipelines. It utilizes a system of cooperatively multitasking "workers" that operate in a single thread. Each worker performs a specific task and can pass data to the next worker in a chain, allowing for the creation of complex data flows in a manageable and sequential manner.
+
+## When to Use Shifty?
+
+Shifty is well-suited for a variety of scenarios where data needs to be processed in a step-by-step fashion. Consider using Shifty for:
+
+*   **Building Data Processing Pipelines:** When your task can be broken down into a series of sequential steps, where each step transforms or enriches the data. Shifty allows you to encapsulate each step within a dedicated worker.
+*   **ETL-like Workflows:** For scenarios that involve extracting data from a source, transforming it according to certain rules, and then loading it elsewhere. While Shifty is primarily focused on the in-application transformation aspects, it can be a valuable part of a larger ETL process within a Ruby application.
+*   **Cooperative Multitasking Scenarios:** When you have multiple tasks that can effectively "take turns" executing. This is particularly useful if tasks involve waiting for non-blocking operations (though the current version of Shifty operates synchronously) or involve complex stateful interactions that are simpler to manage cooperatively rather than with preemptive threading.
+*   **Simplifying Complex Sequential Logic:** If you have a long, monolithic process with many stages, Shifty can help by breaking it down into a chain of smaller, focused, and more understandable workers. This improves modularity and maintainability.
+*   **Creating Reusable Components:** Shifty encourages the design of workers that perform specific, well-defined tasks. These workers can then be reused and recombined in different pipelines for various purposes, promoting code reuse.
+
+## Example Scenarios
+
+Here are a few conceptual examples of how Shifty could be applied:
+
+*   **Log Processing:**
+    Imagine a pipeline for processing application logs:
+    1.  A `FileReaderWorker` reads log lines from a file.
+    2.  A `LogParserWorker` parses each line into a structured format (e.g., timestamp, level, message).
+    3.  A `ErrorFilterWorker` checks the parsed log data and only passes on entries marked as "ERROR" or "CRITICAL".
+    4.  A `ReportFormatterWorker` formats these error entries into a human-readable report.
+    5.  A `FileWriterWorker` or `EmailNotifierWorker` outputs the report.
+
+*   **Data Transformation Chain:**
+    A simple pipeline demonstrating data manipulation:
+    1.  A `NumberSourceWorker` generates a sequence of numbers (e.g., 1, 2, 3, ...).
+    2.  A `MultiplierWorker` receives each number and multiplies it by 2.
+    3.  A `StringFormatterWorker` converts the multiplied number into a string, perhaps adding a prefix (e.g., "RESULT: 4").
+    4.  A `ConsoleOutputWorker` prints the final formatted string to the console.
+
+*   **Batch Processing:**
+    Accumulating data into batches before further processing:
+    1.  An `ItemStreamWorker` produces individual items (e.g., from a database query or an incoming data stream).
+    2.  A `BatchWorker` (Shifty provides a `batch_worker` for this purpose) accumulates these items. It passes the accumulated batch to the next worker once a certain number of items are collected or a timeout occurs.
+    3.  A `BatchProcessorWorker` then processes the entire batch of items at once (e.g., bulk database insert, writing to a file).
+
+## A Note on Values Passed Between Workers
+
+The examples above pass data from one worker to the next. Because Shifty runs
+each value through every worker before starting the next value, workers must
+treat a handed-off value as read-only and express changes as new values
+(`arr + [x]`, `hash.merge(...)`, `value.with(...)`) rather than mutating in
+place (`arr <<`, `hash[k] =`, `map!`). As of 0.6.0 this is enforced: values
+are deeply frozen at every handoff by default, and a task that mutates its
+input raises `Shifty::PolicyViolation` at the offending worker. Workers that
+genuinely need a private scratch copy can declare `policy: :isolated`; workers
+that need a shared mutable reference can declare `policy: :shared`. See the
+wiki's [Handoff Policies](https://github.com/joelhelbling/shifty/wiki/Handoff-Policies)
+and [Coding Idioms Under :frozen](https://github.com/joelhelbling/shifty/wiki/Coding-Idioms-Under-Frozen)
+pages, and the [Migration Guide](https://github.com/joelhelbling/shifty/wiki/Migration-Guide-0.6).
+
+## Conclusion
+
+Shifty aims to provide an intuitive and straightforward way to construct data processing systems within Ruby applications. By breaking down complex tasks into manageable, cooperatively multitasking workers, it helps in building modular, maintainable, and easy-to-understand data pipelines.

data/docs/wiki/Coding-Idioms-Under-Frozen.md

@@ -0,0 +1,74 @@
+# Coding Idioms Under `:frozen`
+
+Under the 0.6.0 default policy, every value a worker receives is deeply frozen (see [[Handoff-Policies]]). That sounds restrictive; in practice it asks for one habit: express change as *new values* rather than in-place edits. This page collects the idioms that make `:frozen` pleasant — copy-on-write equivalents, `Data.define` as the value envelope, the "mutable within, immutable between" rule for stateful workers, and the snapshot rule for builders.
+
+## Copy-on-write instead of mutation
+
+Most migrations are a one-character diff. The left column raises `PolicyViolation` under `:frozen`; the right column does the same job non-destructively:
+
+| Mutating (raises under `:frozen`) | Non-destructive equivalent |
+|---|---|
+| `str << "x"`, `str.upcase!` | `str + "x"`, `str.upcase` |
+| `arr << x`, `arr.map!` | `arr + [x]`, `arr.map` |
+| `hash[k] = v`, `hash.merge!` | `hash.merge(k => v)` |
+| `obj.attr = v` | `obj.with(attr: v)` (see below) |
+
+There's a quiet synergy here: copy-on-write creates only delta-sized new structure, and `:frozen`'s freeze traversal only visits objects created since the last handoff. Each half makes the other cheap — see [[Performance]] for the numbers.
+
+## `Data.define` as the value envelope
+
+Ruby 3.2+ `Data` classes are immutable by construction and ship copy-on-update via `#with`:
+
+```ruby
+Token = Data.define(:payload, :meta)
+
+# in a worker task:
+enricher = relay_worker { |v| v.with(payload: enrich(v.payload)) }
+```
+
+`#with` allocates one new outer object and copies member *references*; unchanged members are structurally shared between old and new value. Structural sharing is only safe because the shared substructure is frozen — which under `:frozen` it always is. Freeze and `#with` aren't two features that happen to coexist; they're a matched set.
+
+For plain objects without `#with`, prefer converting them to `Data`; failing that, `clone(freeze: true)` plus constructor patterns work. Shifty deliberately ships no value mixin of its own — stay unopinionated, bring your own envelope.
+
+## Mutable within, immutable between
+
+Closure state remains fully mutable — that is Shifty's design and its selling point for stateful workers, and nothing about it changed in 0.6.0. Workers that *build* values (sources; batch or trailing workers accumulating in closure scope) may mutate freely during construction. The freeze happens at handoff. The framework thereby enforces the functional-core boundary exactly where Shifty always drew it conceptually:
+
+```ruby
+reader = source_worker do
+  buffer = []                 # closure state: mutable, private, fine
+  while (line = io.gets)
+    buffer << line            # construction: mutate freely
+    if buffer.size == 10
+      handoff buffer.join     # handoff: frozen from here on
+      buffer = []
+    end
+  end
+end
+```
+
+Note what got handed off: `buffer.join` — a fresh string, not the buffer itself. Which brings us to:
+
+## The snapshot rule for builders
+
+A builder that hands off an object and *keeps a reference to it* will raise on its next append under `:frozen`. The handoff freezes the live object; the builder's next `<<` hits a frozen array. This is correct — it is exactly the aliasing bug the policy exists to catch — but it means builders that intend to keep building must **hand off a snapshot**: `buffer.dup`, `buffer.join`, `value.with(...)`.
+
+Shifty's own `trailing_worker` is the in-repo example. It accumulates a rolling window in a closure array and keeps mutating it across calls, so it hands off `trail.dup` — a snapshot — rather than the live array, which a downstream `:frozen` intake would otherwise freeze in place:
+
+```ruby
+# from lib/shifty/dsl.rb
+# Hand off a snapshot: the builder keeps mutating `trail` across
+# calls, and a downstream :frozen intake would freeze the live
+# closure array in place.
+trail.dup
+```
+
+If you write custom accumulating workers, follow the same rule.
+
+## Heavy accumulation: persistent data structures
+
+For genuinely large collections under frequent update, `arr + [x]`'s O(n) pointer copy can pinch. The [immutable-ruby](https://github.com/immutable-ruby/immutable-ruby) gem's persistent `Vector` and `Hash` offer O(log n) structurally-shared updates and play nicely with frozen handoffs. This is a documented user-side optimization, **not** a Shifty dependency — reach for it when profiling says so, not before.
+
+## When the idioms don't fit
+
+Some tasks are legitimately mutation-shaped — a third-party proc you don't control, a gnarly legacy transform not worth rewriting. Don't contort them; declare a policy on that worker instead: `policy: :isolated` for a private scratch copy, or `policy: :shared` when downstream really should observe the mutation. See [[Handoff-Policies]] for the trade-offs and [[Migration-Guide-0.6]] for choosing among them.

data/docs/wiki/Handoff-Policies.md

@@ -0,0 +1,167 @@
+# Handoff Policies
+
+A handoff policy governs how a value crosses a worker boundary. As of 0.6.0 there are three: **`:frozen`** (the default — the value is deeply frozen and passed by reference), **`:isolated`** (the task gets a private mutable deep copy), and **`:shared`** (the raw reference passes through, as in classic Shifty). Policy is applied at *intake* — the moment a worker pulls a value across its boundary — and can be declared per worker, per pipeline/Gang, or globally. This page covers each policy's contract, how declarations compose, the error classes, and the companion `#freeze!` topology lock.
+
+## The three policies
+
+### `:frozen` — the default
+
+The value is deeply frozen in place via `Ractor.make_shareable(value)` and passed **by reference**. Zero copies. Any attempt to mutate the value — or anything reachable from it — raises immediately, at the worker that misbehaved.
+
+- **What the task receives:** a frozen shared reference.
+- **Guarantee:** no worker can mutate a value observed by any other worker.
+- **Cost:** freeze traversal only; no copying, no allocations. Amortized cost is proportional to *newly created* objects, not graph size — at steady state it's a flat ~71ns per handoff regardless of value size (see [[Performance]]).
+- **Developer contract:** tasks must be non-destructive. "Apparent mutation" is expressed as copy-on-write: `value.with(...)`, `arr + [x]`, `hash.merge(...)` (see [[Coding-Idioms-Under-Frozen]]).
+- **Failure modes:** a mutating task raises `Shifty::PolicyViolation`; an unfreezable value raises `Shifty::UnshareableValue` at the handoff. IO values are rejected **proactively** — see below.
+
+### `:isolated` — the scratch-copy mode
+
+The task receives a **deep, mutable copy** of the value, produced by a `Marshal.load(Marshal.dump(value))` round-trip. The task may mutate its copy freely — `<<` and friends work exactly as in classic Shifty code — but nothing leaks back upstream. Whatever the task returns is what flows downstream, mutations included.
+
+Why Marshal and not `Ractor.make_shareable(copy: true)`? Because `make_shareable(copy: true)` returns a *frozen* copy, which cannot satisfy `:isolated`'s contract of a mutable scratch value.
+
+- **What the task receives:** a private mutable deep copy.
+- **Guarantee:** upstream references are protected; task-local mutation is invisible outside the worker.
+- **Cost:** a full deep copy of the value graph per worker, per value, plus the corresponding GC pressure. This is why it is not the default.
+- **Use cases:** mutation-heavy legacy task code not worth rewriting; side workers (this is the old `:hardened` semantics, generalized); untrusted or third-party task procs.
+- **Side workers:** a side worker's return value is discarded, so under `:isolated` its mutations simply evaporate — the behavior the documentation always wished for.
+- **Failure modes:** values Marshal cannot dump raise `Shifty::UnshareableValue`: Procs, lazy enumerators, File/IO handles, StringIO, objects with singleton methods.
+
+### `:shared` — the escape hatch
+
+The raw object reference passes through untouched. This is the pre-0.6.0 default behavior, renamed to say what it actually is. Two legitimate reasons to reach for it:
+
+1. **Intentional in-place mutation** of a value downstream workers are meant to observe.
+2. **Uncopyable / unfreezable values** — IO handles, sockets, procs, lazy enumerators. Sometimes you aren't asking permission to mutate; you're asking for pass-by-reference.
+
+- **Guarantee:** none. The framework provides no protection on this boundary.
+- **Cost:** zero.
+- **Failure mode of incorrect task code:** silent corruption downstream — the very thing 0.6.0 exists to prevent. Reach for `:shared` deliberately, not reflexively.
+
+### Comparison
+
+| | `:frozen` (default) | `:isolated` | `:shared` |
+|---|---|---|---|
+| What the task receives | frozen shared reference | private mutable deep copy | raw shared reference |
+| Upstream protected from mutation | yes (mutation raises) | yes (mutation is local) | no |
+| Mutation in task code | raises `PolicyViolation` | works, stays local | works, leaks |
+| Copying per handoff | none | full graph (Marshal round-trip) | none |
+| Freeze traversal per handoff | delta only (amortized) | none | none |
+| GC pressure | none | high | none |
+| Handles unshareable values (IO, procs, …) | no — `UnshareableValue` | no — `UnshareableValue` (Marshal's failure set) | yes |
+| Failure mode of incorrect task code | loud, local, immediate | silent no-op upstream | silent corruption downstream |
+
+## Declaring policy, and who wins
+
+Policy may be declared at three levels, narrowest to widest. **Precedence: worker beats pipeline beats global.**
+
+```ruby
+# 1. Worker level — part of the worker's contract; always wins
+worker = Shifty::Worker.new(policy: :isolated) { |v| v << transform(v) }
+side_worker(policy: :shared) { |v| logger.info(v) }
+
+# 2. Pipeline level — a default applied to workers that didn't declare their own
+pipeline = (source | parser | enricher | sink).with_policy(:frozen)
+
+# Gangs take a policy at construction or via with_policy; either fans out
+# to the roster, and workers appended later inherit it:
+gang = Shifty::Gang.new([a, b], policy: :isolated)
+Shifty::Gang[a, b].with_policy(:shared)
+
+# 3. Global default — :frozen out of the box
+Shifty.configure { |c| c.default_policy = :frozen }
+```
+
+Details worth knowing:
+
+- `with_policy` walks **upstream** through the supply chain from the node you call it on, so call it on the pipeline's tail (the thing you'd call `shift` on). It returns its receiver, so it chains.
+- A worker's own `policy:` declaration is its **contract** and is never overridden by a pipeline default. This is what makes testing reliable (see [[Testing-Workers]]): any harness that runs the worker through the framework automatically exercises production semantics.
+- Policy names are validated **eagerly**, at declaration time — `Worker.new(policy: :bogus)`, `Gang.new(..., policy: :bogus)`, `with_policy(:bogus)`, and `c.default_policy = :bogus` all raise `ArgumentError` where the typo was written, not at first shift.
+- `:hardened` is accepted as a deprecated alias for `:isolated` (with a warning) until 1.0.0 — see [[Migration-Guide-0.6]].
+- A worker's resolved policy is inspectable as `worker.effective_policy`.
+
+### Where policy is applied
+
+Policy is applied at **intake**: the single seam every value crosses when a worker pulls it from its supply. This includes:
+
+- values pulled mid-task — a task's second argument is a policy-governed supply proxy that responds only to `#shift`, so when a filter, batch, or trailing worker pulls extra values itself, each one crosses the boundary under the same policy;
+- values that bypass a worker's task because its `criteria` said no — the value still crosses the boundary, so it is still governed;
+- each part a splitter yields — every part arrives frozen downstream.
+
+### The terminal-output caveat
+
+Policies govern **worker-to-worker** boundaries only. What the *last* worker returns to your calling code has not crossed another intake, so it is not policy-governed: under `:frozen` it may or may not already be frozen, depending on what the final task did. If your calling code needs a guarantee about the terminal value, establish it yourself (freeze it, copy it, or wrap the pipeline's tail with one more pass-through worker).
+
+## Error reference
+
+Both errors inherit from `Shifty::PolicyError` (< `Shifty::Error` < `StandardError`) and expose `#worker`, `#policy`, and `#value`. Give workers a `name:` and they'll introduce themselves properly in the message.
+
+### `Shifty::PolicyViolation`
+
+Raised when a task mutates a value it received under a policy that forbids mutation. The framework catches the raw `FrozenError` at the task call site and re-raises it wrapped — never masked; the original is available as `#cause`.
+
+Attributes: `worker`, `policy`, `receiver` (the object the task tried to mutate, from `FrozenError#receiver`), `value` (the handed-off value), `cause`.
+
+The message locates the receiver relative to the handed-off value using a bounded reachability walk, producing one of three descriptions:
+
+1. *"…the handed-off value itself"* — the task mutated exactly what it was handed.
+2. *"…reachable from the handed-off value"* — the task mutated something nested inside it (e.g. `v[:items] << 3`).
+3. *"…which may be unrelated to the handed-off value … inspect both to judge"* — the `FrozenError` came from some other frozen object in the task's own code (a frozen string literal, a constant); the policy machinery reports honestly rather than misattributing. (The walk gives up past 50,000 nodes and falls back to this message rather than risk masking the violation.)
+
+Every message ends with the two documented exits: make the task non-destructive (`map` instead of `map!`, `value.with(...)`, `arr + [x]`, `hash.merge(...)`), or declare `policy: :isolated` / `policy: :shared` on that worker.
+
+**Recovery semantics:** a rescued `PolicyViolation` does not kill the pipeline. The raising Fiber is terminated and can never be resumed, so the worker discards it; the next `shift` builds a fresh Fiber and continues with the next value. Closure and context state survive — only the loop restarts. **Exception:** a `#freeze!`-d pipeline cannot rebuild its Fiber, so a violation there ends the pipeline — the topology guarantee wins.
+
+```ruby
+source  = source_worker [[:bad], [:good]]
+mutator = Shifty::Worker.new { |v| (v == [:bad]) ? v << :x : v }
+pipeline = source | mutator
+
+begin
+  pipeline.shift
+rescue Shifty::PolicyViolation => e
+  e.worker   #=> the mutator
+  e.policy   #=> :frozen
+  e.receiver #=> [:bad]
+  e.cause    #=> the original FrozenError
+end
+
+pipeline.shift #=> [:good]  — the pipeline lives on
+```
+
+### `Shifty::UnshareableValue`
+
+Raised at the handoff itself when a value cannot cross the boundary under the effective policy. The failure sets differ per policy:
+
+- **Under `:frozen`:** anything `Ractor.make_shareable` rejects (Procs, lazy enumerators, …), wrapped from the underlying `Ractor::Error` — **plus IO values, rejected proactively**. `make_shareable` does *not* reject an IO; it would silently freeze the live handle in place, a process-wide side effect on shared resources like loggers or `$stdout`. Shifty checks for top-level IO before calling it, and the handle is left untouched and usable.
+- **Under `:isolated`:** anything Marshal cannot dump — Procs, lazy enumerators, File/IO, StringIO, objects with singleton methods.
+
+The message names the value's class and offers the exits: declare `policy: :shared` on this worker (raw pass-by-reference), or restructure the value.
+
+**Caveat:** only a *top-level* IO is detected under `:frozen`. An IO nested inside a container (`{log: $stdout}`) will be silently frozen in place by `make_shareable`. Declare `:shared` on workers handling such values.
+
+### The silent-loosening asymmetry
+
+Error wrapping only catches motion in the *strict* direction. Moving **looser** — `:frozen`/`:isolated` → `:shared` — never raises: a task that was harmlessly mutating its private copy (or that would have raised) now mutates the live shared value, silently. No exception will ever fire. Before loosening any worker's policy, run the mutation detector — `Shifty::Testing.mutates_input?(worker, input)` — to learn whether the task mutates its input at all (see [[Testing-Workers]]).
+
+## `#freeze!` — locking the topology
+
+The same intentionality argument applies to the pipeline itself: `supply=` and Gang appends allow a fully assembled pipeline to be rewired mid-stream. `#freeze!` makes "the pipeline you composed is the pipeline that runs" a guarantee rather than a convention.
+
+```ruby
+pipeline = (source_worker([1, 2, 3]) | relay_worker { |v| v * 2 }).freeze!
+
+pipeline.shift                 #=> 2      — a frozen pipeline still runs
+pipeline.supply = other_source #=> raises FrozenError
+```
+
+What you need to know:
+
+- `#freeze!` locks the node you call it on **and everything upstream** via the supply chain — so call it on the pipeline's **tail**. Freezing a mid-chain node leaves downstream nodes mutable.
+- Worker closure and context state stay mutable — **only the topology freezes**. Stateful workers (batch, trailing, custom accumulators) keep working.
+- Freezing a `Gang` locks its roster membership as well: `append` (and any roster mutation) raises `FrozenError` afterward. The freeze walk continues past the gang to upstream workers.
+- Each worker materializes its lazy state (default task, Fiber) before freezing, so freezing can't surprise it mid-shift. Which is also why you should call `#freeze!` and **not** plain `Object#freeze` — bare `freeze` skips that materialization and will break the worker at its next shift.
+- As noted above, a `#freeze!`-d pipeline cannot recover from a `PolicyViolation` (no Fiber rebuild).
+- `#freeze!` returns its receiver, so it chains: `(a | b | c).with_policy(:frozen).freeze!`.
+
+See also: [[Coding-Idioms-Under-Frozen]] for writing tasks that thrive under the default, and [[Migration-Guide-0.6]] for moving existing pipelines over.

data/docs/wiki/Home.md

@@ -0,0 +1,59 @@
+# The Shifty Framework
+
+Shifty is a Ruby framework for building data processing pipelines out of small, cooperatively multitasking "workers." Each worker is a Fiber-backed unit of work with private state held in closure scope, connected to its neighbors only by the values it hands off. Pipelines are composed with a vertical pipe (`source | transform | sink`), and each value travels through the *entire* pipeline before the next value starts — the escalator, not the elevator.
+
+## What's new in 0.6.0: handoff immutability policies
+
+Shifty workers have always been easy to reason about because each one is isolated; the values flowing *between* them were, until now, the un-governed part of the system. As of 0.6.0, **values are deeply frozen at every handoff by default**. Workers that need a private scratch copy can declare `policy: :isolated`; workers that genuinely need shared mutable references can declare `policy: :shared`. Mutation bugs that used to surface as mysterious downstream corruption now either cannot happen or raise immediately at the worker responsible — with an error message that tells you exactly what to do about it.
+
+0.6.0 also ships:
+
+- **`#freeze!`** — lock a pipeline's topology so the pipeline you composed is the pipeline that runs ([[Handoff-Policies]])
+- **`Shifty::Testing`** and RSpec helpers — unit-test workers under the same policy production will use ([[Testing-Workers]])
+- **Worker `name:`** — so error messages can tell you *which* of nine anonymous workers misbehaved
+- **Benchmarks** demonstrating that the `:frozen` default is essentially free at steady state ([[Performance]])
+- **Deprecation of `side_worker`'s `mode: :hardened`** in favor of `policy: :isolated` ([[Migration-Guide-0.6]])
+
+## Quick start
+
+```ruby
+require "shifty"
+include Shifty::DSL
+
+source  = source_worker (0..3)
+squarer = relay_worker { |n| n ** 2 }
+
+pipeline = source | squarer
+
+pipeline.shift #=> 0
+pipeline.shift #=> 1
+pipeline.shift #=> 4
+pipeline.shift #=> 9
+pipeline.shift #=> nil
+```
+
+Every value the squarer receives arrives deeply frozen. Since the squarer never mutates its input, nothing changes for it — and if some future task tries a sneaky `<<`, it raises a `Shifty::PolicyViolation` right there, naming the worker and the object. Cold cases become caught-in-the-act.
+
+## The pages
+
+| Page | What's in it |
+|---|---|
+| [[Handoff-Policies]] | The three policies (`:frozen`, `:isolated`, `:shared`) in depth: guarantees, costs, failure modes, declaration and precedence, the error reference, and `#freeze!` |
+| [[Coding-Idioms-Under-Frozen]] | Copy-on-write patterns, `Data.define` as the value envelope, and the "mutable within, immutable between" rule |
+| [[Migration-Guide-0.6]] | What breaks, the four migration paths, and the `:hardened` deprecation timeline |
+| [[Testing-Workers]] | `Shifty::Testing.run`, the mutation detector, and the RSpec matcher and shared example |
+| [[Worker-Types]] | The full DSL reference: source, relay, side, filter, batch, splitter, trailing workers, raw `Worker.new`, and `Gang` |
+| [[Performance]] | Benchmark results, why `:frozen` amortizes to ~nothing, and when `:isolated`'s cost matters |
+
+## Ruby version support
+
+| Shifty version | Ruby requirement | Notes |
+|---|---|---|
+| **0.6.0** | Ruby >= 3.2 | Handoff policies, `Data`-based idioms, `#freeze!`, testing harness |
+| 0.5.0 | older Rubies | Last release with the previous defaults (`:shared`-style handoffs, `side_worker mode: :hardened`); remains available for older Rubies and legacy patterns |
+
+The 3.2 floor aligns the gem with `Data.define` (the recommended value envelope — see [[Coding-Idioms-Under-Frozen]]) and mature `Ractor.make_shareable` behavior, which powers the `:frozen` policy.
+
+## Concurrency model, in one paragraph
+
+Shifty uses Ruby Fibers for cooperative multitasking: all workers run in a single OS thread and explicitly yield to one another, which frees you from *preemptive* hazards (races, mutexes) within a pipeline. Single-threading never made the *data* between workers safe, though — that is exactly what handoff policies now govern. The frozen, Ractor-shareable values that `:frozen` produces are also deliberately compatible with a possible future Ractor-backed worker type.

data/docs/wiki/Migration-Guide-0.6.md

@@ -0,0 +1,96 @@
+# Migrating to 0.6.0
+
+Shifty 0.6.0 changes the default handoff behavior: values now arrive at each worker **deeply frozen** instead of as raw shared references (see [[Handoff-Policies]]). Most pipelines — anything already written in a non-destructive style — upgrade with no changes at all. Pipelines whose tasks mutate their inputs will raise loudly and immediately, with an error message that names the worker, the object, and the fix. This page covers what breaks, the four migration paths in order of preference, and the deprecation timeline.
+
+## What breaks
+
+1. **The default flips from `:shared` to `:frozen`.** Any task that mutates its input (`<<`, `map!`, `merge!`, attribute assignment) raises `Shifty::PolicyViolation` on first contact — at the offending worker, not downstream. The message tells you which worker, which object, and the two exits.
+
+2. **Unshareable values can't cross a default-policy boundary.** IO handles, sockets, procs, lazy enumerators, and (under `:isolated`) objects with singleton methods raise `Shifty::UnshareableValue` at the handoff. Workers passing such values along must declare `policy: :shared`. Note that IO is rejected proactively under `:frozen` — but only *top-level* IO; an IO nested inside a container is silently frozen in place by `Ractor.make_shareable`, so declare `:shared` on workers handling those too.
+
+3. **`side_worker`'s `mode:` option is deprecated.** `mode: :hardened` maps to `policy: :isolated` with a deprecation warning; any other `mode:` value warns and is ignored. Note that `side_worker` now takes an options hash, so the old *positional* form `side_worker(:hardened) { ... }` must become `side_worker(mode: :hardened)` (transitional, warns) or better, `side_worker(policy: :isolated)`. The `mode:` option (and the `:hardened` policy alias) will be **removed in 1.0.0**.
+
+4. **The task's second argument is now a policy-governed supply proxy.** Tasks that accept `(value, supply)` — filters, batchers, anything pulling extra values mid-task — now receive a proxy responding only to `#shift`, not the raw upstream worker. Each pulled value crosses the boundary under the worker's policy, exactly like the primary intake. If your task called anything other than `#shift` on its supply argument, that code needs rework.
+
+5. **Ruby floor is 3.2** (`required_ruby_version` in the gemspec). Shifty 0.5.0 remains available for older Rubies and the old behavior.
+
+## Migration paths, in order of preference
+
+### 1. Rewrite tasks non-destructively
+
+Usually a small diff — `map!` → `map`, `<<` → `+`, `merge!` → `merge` — and the `PolicyViolation` message names the object and the worker for you. This is the best destination: zero policy declarations, zero copies, full protection. See [[Coding-Idioms-Under-Frozen]] for the full pattern table.
+
+```ruby
+# before                                  # after
+relay_worker { |v| v << compute(v) }      relay_worker { |v| v + [compute(v)] }
+```
+
+### 2. Declare `:isolated` on mutation-heavy workers
+
+When a task isn't worth rewriting, give it a private scratch copy. Correctness is preserved; the deep-copy cost is localized to exactly those workers.
+
+```ruby
+legacy = Shifty::Worker.new(policy: :isolated) { |v| gnarly_in_place_transform(v) }
+```
+
+### 3. Declare `:shared` where pass-by-reference is genuinely required
+
+Uncopyable values, or intentional shared mutation observed downstream. This restores the pre-0.6.0 semantics for that worker only — along with its lack of protection, so use deliberately.
+
+```ruby
+log_forwarder = side_worker(policy: :shared) { |io| io.flush }
+```
+
+### 4. Blanket opt-out (large legacy codebases)
+
+```ruby
+Shifty.configure { |c| c.default_policy = :shared }
+```
+
+This reproduces 0.5.0 behavior globally, letting a team migrate worker-by-worker at its own pace. Treat it as scaffolding, not a destination — while it's in place you have none of 0.6.0's protection anywhere a worker hasn't declared its own policy.
+
+## Inventory first: the mutation detector
+
+Before upgrading (or before loosening any policy), let the framework tell you which workers actually mutate their inputs:
+
+```ruby
+require "shifty/testing"
+
+Shifty::Testing.mutates_input?(worker, representative_input)
+#=> true  — needs path 1, 2, or 3
+#=> false — this worker upgrades clean
+```
+
+Run it across your workers with representative inputs and you have your migration worklist. This matters doubly because loosening is *silent*: moving a worker to `:shared` never raises, even if its task mutates — the detector is the only thing that will tell you. See [[Testing-Workers]].
+
+## `:hardened` → `:isolated`: what's preserved, what changed
+
+The old `side_worker(:hardened)` handed the block a `Marshal.load(Marshal.dump(value))` deep copy. `:isolated` uses **the same Marshal round-trip**, so both semantics and failure behavior are preserved: a value that raised under `:hardened` (a proc, an IO, a singleton-methoded object — anything Marshal rejects) still raises under `:isolated`, now as a `Shifty::UnshareableValue` with a diagnostic message instead of a bare `TypeError`. No value that used to fail will start silently succeeding, and vice versa.
+
+(An earlier design draft had `:isolated` using `Ractor.make_shareable(copy: true)`, which would have shifted the failure set slightly — that mechanism returns a frozen copy and was rejected precisely because `:isolated` promises a *mutable* scratch copy.)
+
+## Deprecation timeline
+
+| Version | Status |
+|---|---|
+| 0.6.0 | `mode: :hardened` and `policy: :hardened` work, emit deprecation warnings, map to `:isolated` |
+| 1.0.0 | `mode:` option and `:hardened` alias removed |
+
+## A worked upgrade
+
+```ruby
+# 0.5.0 code
+source = source_worker [[:foo], [:bar]]
+stasher = side_worker(:hardened) { |v| v << :boo }   # old positional mode
+# NOTE: on 0.6.0 this positional form raises TypeError (side_worker now
+# takes an options hash) — it does NOT get the friendly deprecation
+# warning. Only the keyword forms (mode:/policy:) warn.
+
+# 0.6.0 code — same behavior, no warning
+stasher = side_worker(policy: :isolated) { |v| v << :boo }
+
+pipeline = source | stasher
+pipeline.shift #=> [:foo]   — mutations evaporate, value passes through pristine
+```
+
+And remember: policy names are validated eagerly, so typos in `policy:` declarations fail at the declaration site, not three workers downstream at 2 a.m.

data/docs/wiki/Performance.md

@@ -0,0 +1,59 @@
+# Performance
+
+Shifty 0.6.0's `:frozen` default was chosen on a claim: deep-freezing at every handoff amortizes to approximately nothing, while deep-copying (`:isolated`) does not. The benchmark suite in `benchmark/` verifies it. Headline: once a value is shareable, `:frozen` costs a flat **~71ns per handoff with zero allocations, independent of value size** — at steady state it is the *cheapest* of the three policies, not just the safest. Numbers below are from Apple Silicon; re-run on your own hardware — the relative ordering is what matters.
+
+## Method note (read first)
+
+Rows labeled *fresh value* construct a new value inside each iteration, so they include construction cost — that keeps `:frozen`'s freeze bit honest (freezing is once-per-object; a pre-built value would be frozen after the first iteration). The *steady state* column uses a pre-built, already-shareable value and measures the policy call alone. Compare fresh columns against the `:shared` fresh column (same construction overhead); read the steady-state column as the per-handoff cost once a value is shareable.
+
+`:hardened` (legacy) has no separate row: its mechanism — a Marshal round-trip — is exactly what `:isolated` uses, so the `:isolated` columns are its numbers.
+
+## Per-handoff throughput (i/s; higher is better)
+
+| Shape | `:shared` (fresh) | `:frozen` first handoff (fresh) | `:frozen` steady state | `:isolated` (fresh) | `:isolated` on already-frozen |
+|---|---|---|---|---|---|
+| small string token | 10.56M | 5.02M | **14.04M** | 1.44M | 1.57M |
+| array, 100 strings | 140.8k | 84.3k | **14.12M** | 34.5k | 46.6k |
+| hash, 100 pairs | 48.6k | 38.4k | **13.87M** | 14.1k | 20.4k |
+| nested document, ~5k nodes | 9.9k | 5.3k | **13.74M** | 1.9k | 2.4k |
+| deep nesting, 500 levels | 49.0k | 21.3k | **13.99M** | 8.5k | 10.8k |
+
+## Allocations per handoff (GC pressure)
+
+| Shape | `:shared` | `:frozen` (steady) | `:isolated` |
+|---|---|---|---|
+| small string token | 0 | 0 | 5 |
+| array, 100 strings | 0 | 0 | 105 |
+| hash, 100 pairs | 0 | 0 | 205 |
+| nested document | 0 | 0 | 1,711 |
+| deep nesting, 500 levels | 0 | 0 | 504 |
+
+## Findings
+
+1. **The amortization claim holds.** Once a value is shareable, `:frozen`'s per-handoff cost is a flat ~71ns **independent of value size** (13.7–14.1M i/s across every shape) with zero allocations. In an N-worker pipeline, only the first boundary pays the freeze traversal; boundaries 2..N are effectively free. Combined with copy-on-write task code, per-boundary cost is proportional to the *change*, not the value.
+2. **First-handoff freeze costs roughly 0.6–1.9× the value's own construction cost** (e.g. ~4.8µs extra for a 100-string array that costs ~7.1µs to build) — paid once per object graph, not per hop.
+3. **`:isolated` is the expensive policy, as designed**: 3–7× slower than `:shared` per boundary *per hop*, and the only policy that allocates — a full copy of the graph per boundary, 1,711 objects per handoff for the ~5k-node document. This is why it is not the default.
+4. **No `shareable?` fast path for `:isolated`.** An already-frozen input can't be mutated by anyone, so `:isolated` could skip the copy — but the fast path would hand the task a frozen object where its contract promises a mutable scratch copy. Marshal only gains ~25–35% on already-frozen values (it re-serializes regardless), so contract simplicity wins at this gem's scale. Reopen trigger: a real workload where `:isolated` boundaries dominate and its inputs are typically already shareable.
+5. **The `:frozen` default is vindicated**: it is the only policy that is simultaneously safe and, at steady state, the cheapest of the three. Freeze once, shift forever.
+
+## Why `:frozen` amortizes
+
+MRI marks objects shareable once `Ractor.make_shareable` has blessed them, and the traversal short-circuits on already-shareable subgraphs. So in a `:frozen` pipeline, the *first* handoff pays a full traversal of the value; every subsequent handoff — through however many downstream workers — traverses only objects created since the previous one. Write your tasks copy-on-write ([[Coding-Idioms-Under-Frozen]]) and each worker creates only delta-sized new structure, so each boundary freezes only that delta. The value's bulk rides through the whole pipeline on that flat ~71ns already-shareable check, allocating nothing.
+
+## When `:isolated`'s cost matters
+
+`:isolated` copies the **entire value graph at every boundary it governs** — per worker, per value. Rules of thumb:
+
+- **Token-sized values, a few `:isolated` workers:** noise. Don't think about it.
+- **Large values (parsed documents, big collections) through `:isolated` boundaries:** a serious tax, in both CPU (3–7× vs `:shared`) and GC pressure (thousands of allocations per handoff). Often the GC cost outstrips the copy CPU itself.
+- **Many `:isolated` workers in one pipeline:** costs multiply — nine `:isolated` workers means nine full copies and nine graphs of garbage per value.
+
+The remedy is targeted: keep `:isolated` on exactly the workers that need a scratch copy, and migrate hot ones to non-destructive tasks under `:frozen` (see [[Migration-Guide-0.6]]). `Shifty::Testing.mutates_input?` tells you which workers actually still need it ([[Testing-Workers]]).
+
+## Re-running the benchmarks
+
+```
+bundle exec ruby benchmark/handoff_policies.rb
+```
+
+Results are written up in [`benchmark/RESULTS.md`](https://github.com/joelhelbling/shifty/blob/main/benchmark/RESULTS.md). Absolute numbers vary by hardware and Ruby version; the shape of the results — flat steady-state `:frozen`, graph-proportional `:isolated` — should not.