shifty 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/CHANGELOG.md +79 -0
- data/README.md +79 -35
- data/benchmark/RESULTS.md +69 -0
- data/benchmark/handoff_policies.rb +82 -0
- data/docs/planning/artifacts/.discovery-notes.md +40 -0
- data/docs/planning/artifacts/.round1-specialist-outputs.md +196 -0
- data/docs/planning/artifacts/implementation-decision-log.md +231 -0
- data/docs/planning/artifacts/implementation-iteration-history.md +73 -0
- data/docs/planning/feature-implementation-plan.md +191 -0
- data/docs/planning/handoff-immutability-policies.md +590 -0
- data/docs/shifty/worker.md +8 -19
- data/docs/use_cases.md +59 -0
- data/docs/wiki/Coding-Idioms-Under-Frozen.md +74 -0
- data/docs/wiki/Handoff-Policies.md +167 -0
- data/docs/wiki/Home.md +59 -0
- data/docs/wiki/Migration-Guide-0.6.md +96 -0
- data/docs/wiki/Performance.md +59 -0
- data/docs/wiki/Testing-Workers.md +127 -0
- data/docs/wiki/Worker-Types.md +171 -0
- data/docs/wiki/_Sidebar.md +13 -0
- data/lib/shifty/configuration.rb +27 -0
- data/lib/shifty/dsl.rb +35 -14
- data/lib/shifty/errors.rb +121 -0
- data/lib/shifty/gang.rb +25 -2
- data/lib/shifty/policy.rb +124 -0
- data/lib/shifty/rspec.rb +37 -0
- data/lib/shifty/testing.rb +82 -0
- data/lib/shifty/version.rb +1 -1
- data/lib/shifty/worker.rb +108 -5
- data/lib/shifty.rb +3 -0
- data/shifty.gemspec +6 -1
- metadata +56 -8
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# Testing Workers
|
|
2
|
+
|
|
3
|
+
Shifty 0.6.0 ships an opt-in test harness (`require "shifty/testing"`) and RSpec sugar (`require "shifty/rspec"`) so your unit tests exercise workers under the **same handoff policy the production pipeline will** (see [[Handoff-Policies]]). Neither is loaded by `require "shifty"` — they're deliberately opt-in. This page covers the parity problem they solve, `Shifty::Testing.run`, the mutation detector, the RSpec matcher and shared example, and the "test at the ceiling" guidance.
|
|
4
|
+
|
|
5
|
+
## The parity problem
|
|
6
|
+
|
|
7
|
+
A pipeline runs `:frozen`; a unit test exercises the worker's raw proc with an ordinary mutable object:
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
worker.task.call(input) # bypasses the framework — and the policy
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The test passes; the pipeline raises. The gap closes from two directions:
|
|
14
|
+
|
|
15
|
+
1. **Worker-level policy declarations win** over pipeline defaults, so the policy travels *with* the worker. Any test that runs the worker through the framework automatically exercises production semantics.
|
|
16
|
+
2. **The framework path is the convenient path** — that's `Shifty::Testing.run`.
|
|
17
|
+
|
|
18
|
+
With `:frozen` as the global default, the common case requires no test configuration at all.
|
|
19
|
+
|
|
20
|
+
## `Shifty::Testing.run`
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
require "shifty/testing"
|
|
24
|
+
|
|
25
|
+
Shifty::Testing.run(worker, inputs:, policy: nil, max_shifts: 10_000)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Feeds the inputs to the worker via a synthetic source and collects its outputs until the end-of-stream sentinel (`nil`). The worker's declared/effective policy governs each handoff, exactly as in production; the harness restores the worker's policy, supply, and Fiber afterward, so it never leaves a mark on the worker under test.
|
|
29
|
+
|
|
30
|
+
```ruby
|
|
31
|
+
worker = relay_worker { |v| v.upcase }
|
|
32
|
+
Shifty::Testing.run(worker, inputs: ["a", "b", "c"]) #=> ["A", "B", "C"]
|
|
33
|
+
|
|
34
|
+
# Parity with production — a mutating task raises under the default policy:
|
|
35
|
+
mutator = Shifty::Worker.new { |v| v && v << :x }
|
|
36
|
+
Shifty::Testing.run(mutator, inputs: [[:a]]) # raises PolicyViolation
|
|
37
|
+
|
|
38
|
+
# A worker that declared :isolated runs under :isolated:
|
|
39
|
+
scratch = Shifty::Worker.new(policy: :isolated) { |v| v && v << :x }
|
|
40
|
+
Shifty::Testing.run(scratch, inputs: [[:a]]) #=> [[:a, :x]]
|
|
41
|
+
|
|
42
|
+
# Explicit policy: override beats the worker's declaration (policy-matrix testing);
|
|
43
|
+
# the worker's real declaration is untouched afterward:
|
|
44
|
+
loose = Shifty::Worker.new(policy: :shared) { |v| v && v << :x }
|
|
45
|
+
Shifty::Testing.run(loose, inputs: [[:a]], policy: :frozen) # raises PolicyViolation
|
|
46
|
+
loose.effective_policy #=> :shared
|
|
47
|
+
|
|
48
|
+
# Non-1:1 output streams collect naturally:
|
|
49
|
+
evens = filter_worker { |v| v.even? }
|
|
50
|
+
Shifty::Testing.run(evens, inputs: [1, 2, 3, 4]) #=> [2, 4]
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Sentinel semantics
|
|
54
|
+
|
|
55
|
+
`nil` is Shifty's end-of-stream sentinel; collection stops at the first `nil` output. **`false` is a legitimate payload**, not end-of-stream — it flows through and gets collected:
|
|
56
|
+
|
|
57
|
+
```ruby
|
|
58
|
+
flipper = relay_worker { |v| !v }
|
|
59
|
+
Shifty::Testing.run(flipper, inputs: [true, false, true])
|
|
60
|
+
#=> [false, false, false]
|
|
61
|
+
# (relay_worker's `value &&` guard passes the false input through untouched.)
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
If a worker's task converts `nil` into something non-nil, the run would never terminate — so after `max_shifts` (default 10,000) the harness raises a diagnostic `Shifty::Error` telling you to let `nil` pass through (e.g. `value && ...`) or raise `max_shifts:`.
|
|
65
|
+
|
|
66
|
+
## The mutation detector: `mutates_input?`
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
Shifty::Testing.mutates_input?(worker, input) #=> true / false
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Hands the task a private mutable deep copy of `input`, runs it through the framework under `:shared`, and compares the copy before and after (via `Marshal.dump`). It surfaces mutation **even when the worker's current policy permits or hides it**:
|
|
73
|
+
|
|
74
|
+
```ruby
|
|
75
|
+
sneaky = side_worker(policy: :isolated) { |v| v << :boo }
|
|
76
|
+
Shifty::Testing.mutates_input?(sneaky, [:a]) #=> true — the :isolated copy hid it in production
|
|
77
|
+
|
|
78
|
+
clean = relay_worker { |v| v + [:x] }
|
|
79
|
+
Shifty::Testing.mutates_input?(clean, [:a]) #=> false
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
This is exactly the information you need **before** loosening a policy — moving to `:shared` never raises, so the detector is your only warning (the silent-loosening asymmetry; see [[Handoff-Policies]]). It's also the pre-upgrade inventory tool for [[Migration-Guide-0.6]]. The input must be Marshal-dumpable; anything else raises a descriptive `Shifty::Error`.
|
|
83
|
+
|
|
84
|
+
## RSpec sugar
|
|
85
|
+
|
|
86
|
+
```ruby
|
|
87
|
+
# spec_helper.rb
|
|
88
|
+
require "shifty/rspec" # pulls in shifty/testing; assumes RSpec is loaded
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### The `mutate_input` matcher
|
|
92
|
+
|
|
93
|
+
Takes the input as an argument and delegates to the mutation detector:
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
expect(worker).not_to mutate_input([:a])
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
The negated failure message spells out the consequences: a mutating task is only correct under `:isolated` (mutation stays local) or `:shared` (mutation is intentional); it will raise under `:frozen`.
|
|
100
|
+
|
|
101
|
+
### The `"a policy-safe worker"` shared example
|
|
102
|
+
|
|
103
|
+
Expects `worker` and `safe_input` to be defined with `let`:
|
|
104
|
+
|
|
105
|
+
```ruby
|
|
106
|
+
RSpec.describe "my enricher" do
|
|
107
|
+
let(:worker) { relay_worker { |v| v.merge(enriched: true) } }
|
|
108
|
+
let(:safe_input) { {id: 1} }
|
|
109
|
+
|
|
110
|
+
it_behaves_like "a policy-safe worker"
|
|
111
|
+
end
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
It runs two checks: the worker processes a **deeply frozen** input through the framework without raising (`Testing.run(..., policy: :frozen)`), and it does not mutate its input.
|
|
115
|
+
|
|
116
|
+
## Test at the ceiling
|
|
117
|
+
|
|
118
|
+
Policy safety is (almost) a one-way ratchet: a task correct on deeply frozen input is correct under every policy; the reverse does not hold. So the guidance is to **test under `:frozen`** — the ceiling — unless the worker explicitly declares mutation as part of its design. Consequences:
|
|
119
|
+
|
|
120
|
+
- Loosening a pipeline's policy can never break a ceiling-tested worker.
|
|
121
|
+
- Tightening is the only breaking direction, and tightening fails loudly with `PolicyViolation` diagnostics.
|
|
122
|
+
|
|
123
|
+
**The "almost":** `:isolated` hands the task a *copy*, so a worker depending on object **identity** (rare, but conceivable with identity-keyed caches) behaves differently there. This is the one spot where strict-passing does not imply lax-passing. If your worker cares about `equal?`, test it under `:isolated` explicitly:
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
Shifty::Testing.run(worker, inputs: [thing], policy: :isolated)
|
|
127
|
+
```
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
# Worker Types
|
|
2
|
+
|
|
3
|
+
Shifty's DSL (`include Shifty::DSL`) provides shorthand constructors for the common worker shapes; underneath them all is `Shifty::Worker`, and above them sits `Shifty::Gang` for treating a chain as one unit. This page is the per-type reference for 0.6.0, including how each type behaves under the default `:frozen` handoff policy (see [[Handoff-Policies]]). All DSL constructors except `source_worker` and `trailing_worker` accept an options hash that passes through to `Worker.new` — so `policy:`, `name:`, `tags:` work everywhere.
|
|
4
|
+
|
|
5
|
+
## `source_worker`
|
|
6
|
+
|
|
7
|
+
At the headwater of every pipeline: a worker that generates its own work items. Its block takes no arguments; `nil` signals end-of-stream (and once a source returns `nil`, it returns `nil` henceforth).
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
worker = source_worker { "Number 9" } # generates forever
|
|
11
|
+
|
|
12
|
+
counter = 0
|
|
13
|
+
finite = source_worker do # generates until it returns nil
|
|
14
|
+
if counter < 3
|
|
15
|
+
counter += 1
|
|
16
|
+
counter + 1000
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
w1 = source_worker [0, 1, 2] # or hand it a series
|
|
21
|
+
w2 = source_worker (0..2) # ranges work too
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
A series-based source yields each element and then `nil` forever. Strings are split into characters; a bare scalar becomes a one-element series. You can also combine a series with a block, which acts as a transform: `source_worker([1,2]) { |v| v * 10 }`.
|
|
25
|
+
|
|
26
|
+
Sources are where "mutable within, immutable between" starts: build values freely in closure scope, and hand off snapshots (see [[Coding-Idioms-Under-Frozen]]).
|
|
27
|
+
|
|
28
|
+
## `relay_worker`
|
|
29
|
+
|
|
30
|
+
The most "normal" worker: accepts a value, returns a transformation of it. `nil` passes through untouched (the end-of-stream sentinel survives).
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
squarer = relay_worker { |number| number ** 2 }
|
|
34
|
+
pipeline = source_worker(0..3) | squarer
|
|
35
|
+
pipeline.shift #=> 0, 1, 4, 9, nil
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Under `:frozen`, relay tasks must be non-destructive — `v.merge(...)`, `v + [...]`, `v.with(...)`. A `v <<` raises `PolicyViolation` at this worker, by name if you gave it one.
|
|
39
|
+
|
|
40
|
+
## `side_worker`
|
|
41
|
+
|
|
42
|
+
Passes through the value it received while performing a side effect — logging, metrics, stashing. Its purpose is *intentionality*: side effects live in named, removable workers, so pulling one out of the pipeline never changes the pipeline's output.
|
|
43
|
+
|
|
44
|
+
```ruby
|
|
45
|
+
evens = []
|
|
46
|
+
even_stasher = side_worker { |value| evens << value if value.even? }
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Policy behavior is where 0.6.0 makes the side worker's contract real:
|
|
50
|
+
|
|
51
|
+
- **Under `:frozen` (default):** a block that mutates the passed value raises `PolicyViolation`. Observation is enforced.
|
|
52
|
+
- **Under `:isolated`:** the block observes a private scratch copy; since a side worker's return value is discarded, its mutations simply **evaporate** and the untouched value flows on:
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
source = source_worker [[:foo], [:bar]]
|
|
56
|
+
unsafe = side_worker(policy: :isolated) { |v| v << :boo }
|
|
57
|
+
pipeline = source | unsafe
|
|
58
|
+
pipeline.shift #=> [:foo] — shenanigans contained
|
|
59
|
+
pipeline.shift #=> [:bar]
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
- **`mode:` is deprecated:** `mode: :hardened` maps to `policy: :isolated` with a warning; any other `mode:` value warns and is ignored. Removed in 1.0.0 — see [[Migration-Guide-0.6]].
|
|
63
|
+
|
|
64
|
+
## `filter_worker`
|
|
65
|
+
|
|
66
|
+
Passes through only values for which the block is truthy; falsy values are discarded (the worker pulls from its supply until something passes).
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
evens_only = filter_worker { |value| value % 2 == 0 }
|
|
70
|
+
pipeline = source_worker(0..5) | evens_only
|
|
71
|
+
pipeline.shift #=> 0, 2, 4, nil
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Every value the filter pulls mid-task crosses the boundary under the worker's policy — including the ones it discards.
|
|
75
|
+
|
|
76
|
+
## `batch_worker`
|
|
77
|
+
|
|
78
|
+
Gathers values into batches. Either a fixed size:
|
|
79
|
+
|
|
80
|
+
```ruby
|
|
81
|
+
batch = batch_worker gathering: 3
|
|
82
|
+
pipeline = source_worker(0..7) | batch
|
|
83
|
+
pipeline.shift #=> [0, 1, 2], then [3, 4, 5], then [6, 7], then nil
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
(the final batch may be short), or a condition — batch until the block is truthy:
|
|
87
|
+
|
|
88
|
+
```ruby
|
|
89
|
+
line_reader = batch_worker { |value| value.end_with?("\n") }
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## `splitter_worker`
|
|
93
|
+
|
|
94
|
+
Accepts one value, produces an array from it, and hands off each element successively before asking its supply for more.
|
|
95
|
+
|
|
96
|
+
```ruby
|
|
97
|
+
splitter = splitter_worker { |value| value.split(" ") }
|
|
98
|
+
pipeline = source_worker(["A bold", "move westward"]) | splitter
|
|
99
|
+
pipeline.shift #=> "A", "bold", "move", "westward", nil
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Every part a splitter yields is policy-governed as it crosses into the next worker — under `:frozen`, each part arrives frozen downstream.
|
|
103
|
+
|
|
104
|
+
## `trailing_worker`
|
|
105
|
+
|
|
106
|
+
Returns an array of the last *n* values — useful for rolling averages. Nothing is returned until *n* values have accumulated; new values are unshifted into position zero.
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
trailer = trailing_worker 4
|
|
110
|
+
pipeline = source_worker(0..5) | trailer
|
|
111
|
+
pipeline.shift #=> [3, 2, 1, 0], then [4, 3, 2, 1], then [5, 4, 3, 2], then nil
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
New in 0.6.0: the trailing worker **hands off a snapshot** (`trail.dup`) rather than its live closure array. It keeps mutating that array across calls, and a downstream `:frozen` intake would otherwise freeze the live array in place. This is the canonical example of the builder snapshot rule — see [[Coding-Idioms-Under-Frozen]].
|
|
115
|
+
|
|
116
|
+
(Signature note: `trailing_worker(trail_length = 2)` takes the length positionally and no options hash.)
|
|
117
|
+
|
|
118
|
+
## Raw `Shifty::Worker.new`
|
|
119
|
+
|
|
120
|
+
When the DSL shapes don't fit, build a worker directly:
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
worker = Shifty::Worker.new(
|
|
124
|
+
supply: upstream, # or wire later: worker.supply = upstream
|
|
125
|
+
policy: :isolated, # this worker's contract; validated eagerly
|
|
126
|
+
name: "enricher", # used in PolicyViolation / UnshareableValue messages
|
|
127
|
+
tags: [:etl], # also reported in diagnostics
|
|
128
|
+
criteria: ->(w) { ... }, # when falsy, the task is bypassed (value still policy-governed)
|
|
129
|
+
task: some_callable # or pass a block
|
|
130
|
+
) { |value, supply, context| ... }
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
The task's arity matters:
|
|
134
|
+
|
|
135
|
+
- **Arity 0** — a source; it cannot accept a supply (`supply=` raises). Use `handoff(value)` (which is `Fiber.yield`) to emit from inside loops.
|
|
136
|
+
- **Arity 1+** — `|value|` receives the policy-governed intake value.
|
|
137
|
+
- **Second argument** — a policy-governed supply proxy responding only to `#shift`, for tasks that pull additional values themselves (this is how filter/batch/trailing work). It is *not* the raw upstream worker.
|
|
138
|
+
- **Third argument** — the worker's `context`, an `OpenStruct` by default or whatever you pass as `context:`; per-worker mutable state that survives across shifts (and survives a rescued `PolicyViolation`).
|
|
139
|
+
|
|
140
|
+
A worker with no task gets a default pass-through task. `worker.effective_policy` reports the resolved policy (own declaration, else pipeline default, else global default).
|
|
141
|
+
|
|
142
|
+
## Composition: `|`, `with_policy`, `freeze!`
|
|
143
|
+
|
|
144
|
+
```ruby
|
|
145
|
+
pipeline = source | filter | transform | sink # tail-returned; call shift on it
|
|
146
|
+
|
|
147
|
+
pipeline = (source | filter | transform | sink)
|
|
148
|
+
.with_policy(:frozen) # pipeline default for workers with no declaration
|
|
149
|
+
.freeze! # lock the topology; upstream walk from the tail
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Both `with_policy` and `freeze!` walk **upstream** through the supply chain from their receiver and return it, so they chain — and both should be called on the pipeline's tail. `freeze!` locks topology only (supply wiring, Gang rosters); closure and context state stay mutable. Use `#freeze!`, never bare `#freeze`. Details in [[Handoff-Policies]].
|
|
153
|
+
|
|
154
|
+
## `Shifty::Gang`
|
|
155
|
+
|
|
156
|
+
A Gang wraps an ordered roster of workers so a whole chain acts like one worker: it has a `supply`, a `shift`, and composes with `|` like anything else.
|
|
157
|
+
|
|
158
|
+
```ruby
|
|
159
|
+
gang = Shifty::Gang[parser, enricher] # or Gang.new([parser, enricher])
|
|
160
|
+
pipeline = reader | gang | writer
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
Policy features:
|
|
164
|
+
|
|
165
|
+
- **Construction fanout:** `Gang.new(workers, policy: :isolated)` sets the pipeline default on every roster member.
|
|
166
|
+
- **`with_policy` fanout:** `gang.with_policy(:shared)` does the same, chainably.
|
|
167
|
+
- **Append inheritance:** the gang persists its declared policy, so workers appended *after* the declaration inherit it too.
|
|
168
|
+
- **Member contracts win:** a roster member's own `policy:` declaration beats the gang's, per the usual precedence.
|
|
169
|
+
- **Chains walk through:** `(upstream | gang | tail).with_policy(:isolated)` reaches the gang's roster *and* workers upstream of it; `freeze!` walks the same path.
|
|
170
|
+
- **Freezing:** a frozen gang still runs, but its roster membership is locked — `append` raises `FrozenError`.
|
|
171
|
+
- **Criteria bypass is still governed:** when a gang's `criteria` bypasses its workers, the value crossing the gang boundary is still policy-governed at the entry worker.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
module Shifty
|
|
2
|
+
class Configuration
|
|
3
|
+
attr_reader :default_policy
|
|
4
|
+
|
|
5
|
+
def initialize
|
|
6
|
+
@default_policy = :frozen
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
def default_policy=(policy_name)
|
|
10
|
+
@default_policy = Policy.validate!(policy_name)
|
|
11
|
+
end
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
class << self
|
|
15
|
+
def config
|
|
16
|
+
@config ||= Configuration.new
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def configure
|
|
20
|
+
yield config
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def reset_configuration!
|
|
24
|
+
@config = Configuration.new
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
data/lib/shifty/dsl.rb
CHANGED
|
@@ -1,6 +1,4 @@
|
|
|
1
1
|
module Shifty
|
|
2
|
-
class WorkerInitializationError < StandardError; end
|
|
3
|
-
|
|
4
2
|
module DSL
|
|
5
3
|
def source_worker(argument = nil, &block)
|
|
6
4
|
ensure_correct_arity_for!(argument, block)
|
|
@@ -22,7 +20,7 @@ module Shifty
|
|
|
22
20
|
def relay_worker(options = {}, &block)
|
|
23
21
|
options[:tags] ||= []
|
|
24
22
|
options[:tags] << :relay
|
|
25
|
-
ensure_regular_arity(block)
|
|
23
|
+
ensure_regular_arity!(block)
|
|
26
24
|
|
|
27
25
|
Worker.new(options) do |value|
|
|
28
26
|
value && block.call(value)
|
|
@@ -32,17 +30,24 @@ module Shifty
|
|
|
32
30
|
def side_worker(options = {}, &block)
|
|
33
31
|
options[:tags] ||= []
|
|
34
32
|
options[:tags] << :side_effect
|
|
35
|
-
|
|
36
|
-
ensure_regular_arity(block)
|
|
33
|
+
deprecate_mode_option!(options)
|
|
34
|
+
ensure_regular_arity!(block)
|
|
37
35
|
|
|
38
|
-
|
|
36
|
+
# The block must ask the worker for its policy at shift time, so the
|
|
37
|
+
# local is captured by the closure before it is assigned. Not redundant.
|
|
38
|
+
worker = nil
|
|
39
|
+
worker = Worker.new(options) do |value| # standard:disable Style/RedundantAssignment
|
|
39
40
|
value.tap do |v|
|
|
40
|
-
|
|
41
|
-
|
|
41
|
+
next unless v
|
|
42
|
+
# Under :isolated the block observes a private scratch copy, so
|
|
43
|
+
# its mutations evaporate and the untouched value flows on.
|
|
44
|
+
used_value = (worker.effective_policy == :isolated) ?
|
|
45
|
+
Policy::Isolated.call(v, worker: worker) : v
|
|
42
46
|
|
|
43
|
-
|
|
47
|
+
block.call(used_value)
|
|
44
48
|
end
|
|
45
49
|
end
|
|
50
|
+
worker
|
|
46
51
|
end
|
|
47
52
|
|
|
48
53
|
def filter_worker(options = {}, &block)
|
|
@@ -70,7 +75,7 @@ module Shifty
|
|
|
70
75
|
options[:tags] << :batch
|
|
71
76
|
options[:gathering] ||= 1
|
|
72
77
|
|
|
73
|
-
ensure_regular_arity(block) if block
|
|
78
|
+
ensure_regular_arity!(block) if block
|
|
74
79
|
batch_full = block ||
|
|
75
80
|
proc { |_, batch| batch.size >= options[:gathering] }
|
|
76
81
|
|
|
@@ -93,7 +98,7 @@ module Shifty
|
|
|
93
98
|
def splitter_worker(options = {}, &block)
|
|
94
99
|
options[:tags] ||= []
|
|
95
100
|
options[:tags] << :splitter
|
|
96
|
-
ensure_regular_arity(block)
|
|
101
|
+
ensure_regular_arity!(block)
|
|
97
102
|
|
|
98
103
|
Worker.new(options) do |value|
|
|
99
104
|
if value.nil?
|
|
@@ -122,7 +127,10 @@ module Shifty
|
|
|
122
127
|
trail.unshift supply.shift
|
|
123
128
|
end
|
|
124
129
|
|
|
125
|
-
trail
|
|
130
|
+
# Hand off a snapshot: the builder keeps mutating `trail` across
|
|
131
|
+
# calls, and a downstream :frozen intake would freeze the live
|
|
132
|
+
# closure array in place.
|
|
133
|
+
trail.dup
|
|
126
134
|
else
|
|
127
135
|
value # hint: it's nil!
|
|
128
136
|
end
|
|
@@ -135,6 +143,19 @@ module Shifty
|
|
|
135
143
|
|
|
136
144
|
private
|
|
137
145
|
|
|
146
|
+
def deprecate_mode_option!(options)
|
|
147
|
+
return unless options.key?(:mode)
|
|
148
|
+
mode = options.delete(:mode)
|
|
149
|
+
if mode == :hardened
|
|
150
|
+
warn "[shifty] side_worker mode: :hardened is deprecated and will be " \
|
|
151
|
+
"removed in 1.0.0; use policy: :isolated instead."
|
|
152
|
+
options[:policy] ||= :isolated
|
|
153
|
+
else
|
|
154
|
+
warn "[shifty] side_worker's mode: option is deprecated and ignored " \
|
|
155
|
+
"(received mode: #{mode.inspect}); declare a policy: instead."
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
138
159
|
def throw_with(*msg)
|
|
139
160
|
raise WorkerInitializationError.new([msg].flatten.join(" "))
|
|
140
161
|
end
|
|
@@ -145,7 +166,7 @@ module Shifty
|
|
|
145
166
|
end
|
|
146
167
|
end
|
|
147
168
|
|
|
148
|
-
def ensure_regular_arity(block)
|
|
169
|
+
def ensure_regular_arity!(block)
|
|
149
170
|
if block.arity != 1
|
|
150
171
|
throw_with \
|
|
151
172
|
"Worker must accept exactly one argument (arity == 1)"
|
|
@@ -156,7 +177,7 @@ module Shifty
|
|
|
156
177
|
def ensure_correct_arity_for!(argument, block)
|
|
157
178
|
return unless block
|
|
158
179
|
if argument
|
|
159
|
-
ensure_regular_arity(block)
|
|
180
|
+
ensure_regular_arity!(block)
|
|
160
181
|
elsif block.arity > 0
|
|
161
182
|
throw_with \
|
|
162
183
|
"Source worker cannot accept any arguments (arity == 0)"
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
module Shifty
|
|
2
|
+
class Error < StandardError; end
|
|
3
|
+
|
|
4
|
+
class WorkerError < Error; end
|
|
5
|
+
|
|
6
|
+
class WorkerInitializationError < Error; end
|
|
7
|
+
|
|
8
|
+
class PolicyError < Error
|
|
9
|
+
attr_reader :worker, :policy, :value
|
|
10
|
+
|
|
11
|
+
def cause
|
|
12
|
+
@wrapped_cause || super
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
private
|
|
16
|
+
|
|
17
|
+
def worker_label
|
|
18
|
+
label = (worker.respond_to?(:name) && worker.name) ? "`#{worker.name}`" : "(unnamed)"
|
|
19
|
+
tags = worker.respond_to?(:tags) ? worker.tags : []
|
|
20
|
+
tags&.any? ? "#{label} (tags: #{tags.inspect})" : label
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
# Raised when a task mutates a value it received under a policy that
|
|
25
|
+
# forbids mutation. Wraps the original FrozenError (never masks it) and
|
|
26
|
+
# names the worker, the effective policy, and the object the task tried
|
|
27
|
+
# to mutate, with a heuristic locating that object relative to the
|
|
28
|
+
# handed-off value.
|
|
29
|
+
class PolicyViolation < PolicyError
|
|
30
|
+
attr_reader :receiver
|
|
31
|
+
|
|
32
|
+
def initialize(worker:, policy:, receiver:, value:, cause:)
|
|
33
|
+
@worker = worker
|
|
34
|
+
@policy = policy
|
|
35
|
+
@receiver = receiver
|
|
36
|
+
@value = value
|
|
37
|
+
@wrapped_cause = cause
|
|
38
|
+
super(build_message)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
private
|
|
42
|
+
|
|
43
|
+
def build_message
|
|
44
|
+
<<~MSG
|
|
45
|
+
Worker #{worker_label} received its value under the #{policy.inspect} handoff \
|
|
46
|
+
policy, and its task attempted to mutate #{receiver_description}.
|
|
47
|
+
|
|
48
|
+
Either make the task non-destructive — e.g. `map` instead of `map!`, \
|
|
49
|
+
`value.with(...)`, `arr + [x]`, `hash.merge(...)` — or declare a different \
|
|
50
|
+
policy on this worker: `policy: :isolated` (task works on a private scratch \
|
|
51
|
+
copy) or `policy: :shared` (raw reference; no protection). \
|
|
52
|
+
More: https://github.com/joelhelbling/shifty/wiki/Handoff-Policies
|
|
53
|
+
MSG
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def receiver_description
|
|
57
|
+
if receiver.equal?(value)
|
|
58
|
+
"an instance of #{receiver.class} — the handed-off value itself"
|
|
59
|
+
elsif reachable_from_value?
|
|
60
|
+
"an instance of #{receiver.class} reachable from the handed-off value"
|
|
61
|
+
else
|
|
62
|
+
"an instance of #{receiver.class} (#{receiver.inspect[0, 80]}), which may be " \
|
|
63
|
+
"unrelated to the handed-off value (an instance of #{value.class}); " \
|
|
64
|
+
"inspect both to judge"
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Bounds the diagnostic graph walk; past this the heuristic gives up
|
|
69
|
+
# and reports the honest "inspect both to judge" fallback rather than
|
|
70
|
+
# risking a SystemStackError that would mask the violation itself.
|
|
71
|
+
MAX_REACHABILITY_NODES = 50_000
|
|
72
|
+
|
|
73
|
+
def reachable_from_value?
|
|
74
|
+
seen = {}.compare_by_identity
|
|
75
|
+
stack = [value]
|
|
76
|
+
until stack.empty?
|
|
77
|
+
node = stack.pop
|
|
78
|
+
next if node.nil? || seen[node]
|
|
79
|
+
return false if seen.size >= MAX_REACHABILITY_NODES
|
|
80
|
+
seen[node] = true
|
|
81
|
+
return true if node.equal?(receiver)
|
|
82
|
+
stack.concat(children_of(node))
|
|
83
|
+
end
|
|
84
|
+
false
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def children_of(node)
|
|
88
|
+
case node
|
|
89
|
+
when Array then node
|
|
90
|
+
when Hash then node.keys + node.values
|
|
91
|
+
when Struct then node.to_a
|
|
92
|
+
else
|
|
93
|
+
members = (node.respond_to?(:to_h) && node.is_a?(Data)) ? node.to_h.values : []
|
|
94
|
+
members + node.instance_variables.map { |iv| node.instance_variable_get(iv) }
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# Raised at the handoff itself when a value cannot cross the boundary
|
|
100
|
+
# under the effective policy (cannot be frozen or deep-copied).
|
|
101
|
+
class UnshareableValue < PolicyError
|
|
102
|
+
def initialize(worker:, policy:, value:, cause: nil)
|
|
103
|
+
@worker = worker
|
|
104
|
+
@policy = policy
|
|
105
|
+
@value = value
|
|
106
|
+
@wrapped_cause = cause
|
|
107
|
+
super(build_message)
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
private
|
|
111
|
+
|
|
112
|
+
def build_message
|
|
113
|
+
"Worker #{worker_label} received a value (an instance of #{value.class}) " \
|
|
114
|
+
"that cannot cross the handoff boundary under the #{policy.inspect} policy: " \
|
|
115
|
+
"it cannot be #{(policy == :isolated) ? "deep-copied" : "frozen"}. " \
|
|
116
|
+
"Declare `policy: :shared` on this worker (raw pass-by-reference), " \
|
|
117
|
+
"or restructure the value. " \
|
|
118
|
+
"More: https://github.com/joelhelbling/shifty/wiki/Handoff-Policies"
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
end
|
data/lib/shifty/gang.rb
CHANGED
|
@@ -6,18 +6,31 @@ module Shifty
|
|
|
6
6
|
attr_reader :roster, :tags
|
|
7
7
|
|
|
8
8
|
include Taggable
|
|
9
|
+
include PolicyDeclarable
|
|
9
10
|
|
|
10
11
|
def initialize(workers = [], p = {})
|
|
11
12
|
@roster = Roster.new(workers)
|
|
12
13
|
self.criteria = p[:criteria]
|
|
13
14
|
self.tags = p[:tags]
|
|
15
|
+
self.pipeline_policy = Policy.validate!(p[:policy]) if p[:policy]
|
|
14
16
|
end
|
|
15
17
|
|
|
18
|
+
def pipeline_policy=(policy_name)
|
|
19
|
+
# Persisted so workers appended after the declaration inherit it.
|
|
20
|
+
@pipeline_policy = policy_name
|
|
21
|
+
roster.workers.each { |w| w.pipeline_policy = policy_name }
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
attr_reader :pipeline_policy
|
|
25
|
+
|
|
16
26
|
def shift
|
|
17
27
|
if criteria_passes?
|
|
18
28
|
roster.last.shift
|
|
19
29
|
else
|
|
20
|
-
|
|
30
|
+
# Even when the gang's criteria bypasses its workers, the value
|
|
31
|
+
# still crosses the gang's boundary — govern it with the entry
|
|
32
|
+
# worker's policy, matching Worker#shift's bypass behavior.
|
|
33
|
+
roster.first.intake(roster.first.supply.shift)
|
|
21
34
|
end
|
|
22
35
|
end
|
|
23
36
|
|
|
@@ -26,7 +39,7 @@ module Shifty
|
|
|
26
39
|
end
|
|
27
40
|
|
|
28
41
|
def supply
|
|
29
|
-
roster.first
|
|
42
|
+
roster.first&.supply
|
|
30
43
|
end
|
|
31
44
|
|
|
32
45
|
def supply=(supplier)
|
|
@@ -41,6 +54,16 @@ module Shifty
|
|
|
41
54
|
|
|
42
55
|
def append(worker)
|
|
43
56
|
roster << worker
|
|
57
|
+
worker.pipeline_policy = pipeline_policy if pipeline_policy
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
# Freezes every roster member plus the roster's own membership, so
|
|
61
|
+
# append/push/pop/shift/unshift all raise FrozenError afterward.
|
|
62
|
+
def freeze_topology_node!
|
|
63
|
+
roster.workers.each(&:freeze_topology_node!)
|
|
64
|
+
roster.workers.freeze
|
|
65
|
+
roster.freeze
|
|
66
|
+
freeze
|
|
44
67
|
end
|
|
45
68
|
|
|
46
69
|
class << self
|