shifty 0.4.2 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/CHANGELOG.md +79 -0
- data/README.md +79 -35
- data/benchmark/RESULTS.md +69 -0
- data/benchmark/handoff_policies.rb +82 -0
- data/docs/planning/artifacts/.discovery-notes.md +40 -0
- data/docs/planning/artifacts/.round1-specialist-outputs.md +196 -0
- data/docs/planning/artifacts/implementation-decision-log.md +231 -0
- data/docs/planning/artifacts/implementation-iteration-history.md +73 -0
- data/docs/planning/feature-implementation-plan.md +191 -0
- data/docs/planning/handoff-immutability-policies.md +590 -0
- data/docs/shifty/worker.md +8 -19
- data/docs/use_cases.md +59 -0
- data/docs/wiki/Coding-Idioms-Under-Frozen.md +74 -0
- data/docs/wiki/Handoff-Policies.md +167 -0
- data/docs/wiki/Home.md +59 -0
- data/docs/wiki/Migration-Guide-0.6.md +96 -0
- data/docs/wiki/Performance.md +59 -0
- data/docs/wiki/Testing-Workers.md +127 -0
- data/docs/wiki/Worker-Types.md +171 -0
- data/docs/wiki/_Sidebar.md +13 -0
- data/lib/shifty/configuration.rb +27 -0
- data/lib/shifty/dsl.rb +64 -32
- data/lib/shifty/errors.rb +121 -0
- data/lib/shifty/gang.rb +25 -2
- data/lib/shifty/policy.rb +124 -0
- data/lib/shifty/rspec.rb +37 -0
- data/lib/shifty/taggable.rb +1 -1
- data/lib/shifty/testing.rb +82 -0
- data/lib/shifty/version.rb +1 -1
- data/lib/shifty/worker.rb +108 -5
- data/lib/shifty.rb +3 -0
- data/shifty.gemspec +6 -1
- metadata +56 -8
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# Testing Workers
|
|
2
|
+
|
|
3
|
+
Shifty 0.6.0 ships an opt-in test harness (`require "shifty/testing"`) and RSpec sugar (`require "shifty/rspec"`) so your unit tests exercise workers under the **same handoff policy the production pipeline will** (see [[Handoff-Policies]]). Neither is loaded by `require "shifty"` — they're deliberately opt-in. This page covers the parity problem they solve, `Shifty::Testing.run`, the mutation detector, the RSpec matcher and shared example, and the "test at the ceiling" guidance.
|
|
4
|
+
|
|
5
|
+
## The parity problem
|
|
6
|
+
|
|
7
|
+
A pipeline runs `:frozen`; a unit test exercises the worker's raw proc with an ordinary mutable object:
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
worker.task.call(input) # bypasses the framework — and the policy
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The test passes; the pipeline raises. The gap closes from two directions:
|
|
14
|
+
|
|
15
|
+
1. **Worker-level policy declarations win** over pipeline defaults, so the policy travels *with* the worker. Any test that runs the worker through the framework automatically exercises production semantics.
|
|
16
|
+
2. **The framework path is the convenient path** — that's `Shifty::Testing.run`.
|
|
17
|
+
|
|
18
|
+
With `:frozen` as the global default, the common case requires no test configuration at all.
|
|
19
|
+
|
|
20
|
+
## `Shifty::Testing.run`
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
require "shifty/testing"
|
|
24
|
+
|
|
25
|
+
Shifty::Testing.run(worker, inputs:, policy: nil, max_shifts: 10_000)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Feeds the inputs to the worker via a synthetic source and collects its outputs until the end-of-stream sentinel (`nil`). The worker's declared/effective policy governs each handoff, exactly as in production; the harness restores the worker's policy, supply, and Fiber afterward, so it never leaves a mark on the worker under test.
|
|
29
|
+
|
|
30
|
+
```ruby
|
|
31
|
+
worker = relay_worker { |v| v.upcase }
|
|
32
|
+
Shifty::Testing.run(worker, inputs: ["a", "b", "c"]) #=> ["A", "B", "C"]
|
|
33
|
+
|
|
34
|
+
# Parity with production — a mutating task raises under the default policy:
|
|
35
|
+
mutator = Shifty::Worker.new { |v| v && v << :x }
|
|
36
|
+
Shifty::Testing.run(mutator, inputs: [[:a]]) # raises PolicyViolation
|
|
37
|
+
|
|
38
|
+
# A worker that declared :isolated runs under :isolated:
|
|
39
|
+
scratch = Shifty::Worker.new(policy: :isolated) { |v| v && v << :x }
|
|
40
|
+
Shifty::Testing.run(scratch, inputs: [[:a]]) #=> [[:a, :x]]
|
|
41
|
+
|
|
42
|
+
# Explicit policy: override beats the worker's declaration (policy-matrix testing);
|
|
43
|
+
# the worker's real declaration is untouched afterward:
|
|
44
|
+
loose = Shifty::Worker.new(policy: :shared) { |v| v && v << :x }
|
|
45
|
+
Shifty::Testing.run(loose, inputs: [[:a]], policy: :frozen) # raises PolicyViolation
|
|
46
|
+
loose.effective_policy #=> :shared
|
|
47
|
+
|
|
48
|
+
# Non-1:1 output streams collect naturally:
|
|
49
|
+
evens = filter_worker { |v| v.even? }
|
|
50
|
+
Shifty::Testing.run(evens, inputs: [1, 2, 3, 4]) #=> [2, 4]
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Sentinel semantics
|
|
54
|
+
|
|
55
|
+
`nil` is Shifty's end-of-stream sentinel; collection stops at the first `nil` output. **`false` is a legitimate payload**, not end-of-stream — it flows through and gets collected:
|
|
56
|
+
|
|
57
|
+
```ruby
|
|
58
|
+
flipper = relay_worker { |v| !v }
|
|
59
|
+
Shifty::Testing.run(flipper, inputs: [true, false, true])
|
|
60
|
+
#=> [false, false, false]
|
|
61
|
+
# (relay_worker's `value &&` guard passes the false input through untouched.)
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
If a worker's task converts `nil` into something non-nil, the run would never terminate — so after `max_shifts` (default 10,000) the harness raises a diagnostic `Shifty::Error` telling you to let `nil` pass through (e.g. `value && ...`) or raise `max_shifts:`.
|
|
65
|
+
|
|
66
|
+
## The mutation detector: `mutates_input?`
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
Shifty::Testing.mutates_input?(worker, input) #=> true / false
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Hands the task a private mutable deep copy of `input`, runs it through the framework under `:shared`, and compares the copy before and after (via `Marshal.dump`). It surfaces mutation **even when the worker's current policy permits or hides it**:
|
|
73
|
+
|
|
74
|
+
```ruby
|
|
75
|
+
sneaky = side_worker(policy: :isolated) { |v| v << :boo }
|
|
76
|
+
Shifty::Testing.mutates_input?(sneaky, [:a]) #=> true — the :isolated copy hid it in production
|
|
77
|
+
|
|
78
|
+
clean = relay_worker { |v| v + [:x] }
|
|
79
|
+
Shifty::Testing.mutates_input?(clean, [:a]) #=> false
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
This is exactly the information you need **before** loosening a policy — moving to `:shared` never raises, so the detector is your only warning (the silent-loosening asymmetry; see [[Handoff-Policies]]). It's also the pre-upgrade inventory tool for [[Migration-Guide-0.6]]. The input must be Marshal-dumpable; anything else raises a descriptive `Shifty::Error`.
|
|
83
|
+
|
|
84
|
+
## RSpec sugar
|
|
85
|
+
|
|
86
|
+
```ruby
|
|
87
|
+
# spec_helper.rb
|
|
88
|
+
require "shifty/rspec" # pulls in shifty/testing; assumes RSpec is loaded
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### The `mutate_input` matcher
|
|
92
|
+
|
|
93
|
+
Takes the input as an argument and delegates to the mutation detector:
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
expect(worker).not_to mutate_input([:a])
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
The negated failure message spells out the consequences: a mutating task is only correct under `:isolated` (mutation stays local) or `:shared` (mutation is intentional); it will raise under `:frozen`.
|
|
100
|
+
|
|
101
|
+
### The `"a policy-safe worker"` shared example
|
|
102
|
+
|
|
103
|
+
Expects `worker` and `safe_input` to be defined with `let`:
|
|
104
|
+
|
|
105
|
+
```ruby
|
|
106
|
+
RSpec.describe "my enricher" do
|
|
107
|
+
let(:worker) { relay_worker { |v| v.merge(enriched: true) } }
|
|
108
|
+
let(:safe_input) { {id: 1} }
|
|
109
|
+
|
|
110
|
+
it_behaves_like "a policy-safe worker"
|
|
111
|
+
end
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
It runs two checks: the worker processes a **deeply frozen** input through the framework without raising (`Testing.run(..., policy: :frozen)`), and it does not mutate its input.
|
|
115
|
+
|
|
116
|
+
## Test at the ceiling
|
|
117
|
+
|
|
118
|
+
Policy safety is (almost) a one-way ratchet: a task correct on deeply frozen input is correct under every policy; the reverse does not hold. So the guidance is to **test under `:frozen`** — the ceiling — unless the worker explicitly declares mutation as part of its design. Consequences:
|
|
119
|
+
|
|
120
|
+
- Loosening a pipeline's policy can never break a ceiling-tested worker.
|
|
121
|
+
- Tightening is the only breaking direction, and tightening fails loudly with `PolicyViolation` diagnostics.
|
|
122
|
+
|
|
123
|
+
**The "almost":** `:isolated` hands the task a *copy*, so a worker depending on object **identity** (rare, but conceivable with identity-keyed caches) behaves differently there. This is the one spot where strict-passing does not imply lax-passing. If your worker cares about `equal?`, test it under `:isolated` explicitly:
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
Shifty::Testing.run(worker, inputs: [thing], policy: :isolated)
|
|
127
|
+
```
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
# Worker Types
|
|
2
|
+
|
|
3
|
+
Shifty's DSL (`include Shifty::DSL`) provides shorthand constructors for the common worker shapes; underneath them all is `Shifty::Worker`, and above them sits `Shifty::Gang` for treating a chain as one unit. This page is the per-type reference for 0.6.0, including how each type behaves under the default `:frozen` handoff policy (see [[Handoff-Policies]]). All DSL constructors except `source_worker` and `trailing_worker` accept an options hash that passes through to `Worker.new` — so `policy:`, `name:`, `tags:` work everywhere.
|
|
4
|
+
|
|
5
|
+
## `source_worker`
|
|
6
|
+
|
|
7
|
+
At the headwater of every pipeline: a worker that generates its own work items. Its block takes no arguments; `nil` signals end-of-stream (and once a source returns `nil`, it returns `nil` henceforth).
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
worker = source_worker { "Number 9" } # generates forever
|
|
11
|
+
|
|
12
|
+
counter = 0
|
|
13
|
+
finite = source_worker do # generates until it returns nil
|
|
14
|
+
if counter < 3
|
|
15
|
+
counter += 1
|
|
16
|
+
counter + 1000
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
w1 = source_worker [0, 1, 2] # or hand it a series
|
|
21
|
+
w2 = source_worker (0..2) # ranges work too
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
A series-based source yields each element and then `nil` forever. Strings are split into characters; a bare scalar becomes a one-element series. You can also combine a series with a block, which acts as a transform: `source_worker([1,2]) { |v| v * 10 }`.
|
|
25
|
+
|
|
26
|
+
Sources are where "mutable within, immutable between" starts: build values freely in closure scope, and hand off snapshots (see [[Coding-Idioms-Under-Frozen]]).
|
|
27
|
+
|
|
28
|
+
## `relay_worker`
|
|
29
|
+
|
|
30
|
+
The most "normal" worker: accepts a value, returns a transformation of it. `nil` passes through untouched (the end-of-stream sentinel survives).
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
squarer = relay_worker { |number| number ** 2 }
|
|
34
|
+
pipeline = source_worker(0..3) | squarer
|
|
35
|
+
pipeline.shift #=> 0, 1, 4, 9, nil
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Under `:frozen`, relay tasks must be non-destructive — `v.merge(...)`, `v + [...]`, `v.with(...)`. A `v <<` raises `PolicyViolation` at this worker, by name if you gave it one.
|
|
39
|
+
|
|
40
|
+
## `side_worker`
|
|
41
|
+
|
|
42
|
+
Passes through the value it received while performing a side effect — logging, metrics, stashing. Its purpose is *intentionality*: side effects live in named, removable workers, so pulling one out of the pipeline never changes the pipeline's output.
|
|
43
|
+
|
|
44
|
+
```ruby
|
|
45
|
+
evens = []
|
|
46
|
+
even_stasher = side_worker { |value| evens << value if value.even? }
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Policy behavior is where 0.6.0 makes the side worker's contract real:
|
|
50
|
+
|
|
51
|
+
- **Under `:frozen` (default):** a block that mutates the passed value raises `PolicyViolation`. Observation is enforced.
|
|
52
|
+
- **Under `:isolated`:** the block observes a private scratch copy; since a side worker's return value is discarded, its mutations simply **evaporate** and the untouched value flows on:
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
source = source_worker [[:foo], [:bar]]
|
|
56
|
+
unsafe = side_worker(policy: :isolated) { |v| v << :boo }
|
|
57
|
+
pipeline = source | unsafe
|
|
58
|
+
pipeline.shift #=> [:foo] — shenanigans contained
|
|
59
|
+
pipeline.shift #=> [:bar]
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
- **`mode:` is deprecated:** `mode: :hardened` maps to `policy: :isolated` with a warning; any other `mode:` value warns and is ignored. Removed in 1.0.0 — see [[Migration-Guide-0.6]].
|
|
63
|
+
|
|
64
|
+
## `filter_worker`
|
|
65
|
+
|
|
66
|
+
Passes through only values for which the block is truthy; falsy values are discarded (the worker pulls from its supply until something passes).
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
evens_only = filter_worker { |value| value % 2 == 0 }
|
|
70
|
+
pipeline = source_worker(0..5) | evens_only
|
|
71
|
+
pipeline.shift #=> 0, 2, 4, nil
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Every value the filter pulls mid-task crosses the boundary under the worker's policy — including the ones it discards.
|
|
75
|
+
|
|
76
|
+
## `batch_worker`
|
|
77
|
+
|
|
78
|
+
Gathers values into batches. Either a fixed size:
|
|
79
|
+
|
|
80
|
+
```ruby
|
|
81
|
+
batch = batch_worker gathering: 3
|
|
82
|
+
pipeline = source_worker(0..7) | batch
|
|
83
|
+
pipeline.shift #=> [0, 1, 2], then [3, 4, 5], then [6, 7], then nil
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
(the final batch may be short), or a condition — batch until the block is truthy:
|
|
87
|
+
|
|
88
|
+
```ruby
|
|
89
|
+
line_reader = batch_worker { |value| value.end_with?("\n") }
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## `splitter_worker`
|
|
93
|
+
|
|
94
|
+
Accepts one value, produces an array from it, and hands off each element successively before asking its supply for more.
|
|
95
|
+
|
|
96
|
+
```ruby
|
|
97
|
+
splitter = splitter_worker { |value| value.split(" ") }
|
|
98
|
+
pipeline = source_worker(["A bold", "move westward"]) | splitter
|
|
99
|
+
pipeline.shift #=> "A", "bold", "move", "westward", nil
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Every part a splitter yields is policy-governed as it crosses into the next worker — under `:frozen`, each part arrives frozen downstream.
|
|
103
|
+
|
|
104
|
+
## `trailing_worker`
|
|
105
|
+
|
|
106
|
+
Returns an array of the last *n* values — useful for rolling averages. Nothing is returned until *n* values have accumulated; new values are unshifted into position zero.
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
trailer = trailing_worker 4
|
|
110
|
+
pipeline = source_worker(0..5) | trailer
|
|
111
|
+
pipeline.shift #=> [3, 2, 1, 0], then [4, 3, 2, 1], then [5, 4, 3, 2], then nil
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
New in 0.6.0: the trailing worker **hands off a snapshot** (`trail.dup`) rather than its live closure array. It keeps mutating that array across calls, and a downstream `:frozen` intake would otherwise freeze the live array in place. This is the canonical example of the builder snapshot rule — see [[Coding-Idioms-Under-Frozen]].
|
|
115
|
+
|
|
116
|
+
(Signature note: `trailing_worker(trail_length = 2)` takes the length positionally and no options hash.)
|
|
117
|
+
|
|
118
|
+
## Raw `Shifty::Worker.new`
|
|
119
|
+
|
|
120
|
+
When the DSL shapes don't fit, build a worker directly:
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
worker = Shifty::Worker.new(
|
|
124
|
+
supply: upstream, # or wire later: worker.supply = upstream
|
|
125
|
+
policy: :isolated, # this worker's contract; validated eagerly
|
|
126
|
+
name: "enricher", # used in PolicyViolation / UnshareableValue messages
|
|
127
|
+
tags: [:etl], # also reported in diagnostics
|
|
128
|
+
criteria: ->(w) { ... }, # when falsy, the task is bypassed (value still policy-governed)
|
|
129
|
+
task: some_callable # or pass a block
|
|
130
|
+
) { |value, supply, context| ... }
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
The task's arity matters:
|
|
134
|
+
|
|
135
|
+
- **Arity 0** — a source; it cannot accept a supply (`supply=` raises). Use `handoff(value)` (which is `Fiber.yield`) to emit from inside loops.
|
|
136
|
+
- **Arity 1+** — `|value|` receives the policy-governed intake value.
|
|
137
|
+
- **Second argument** — a policy-governed supply proxy responding only to `#shift`, for tasks that pull additional values themselves (this is how filter/batch/trailing work). It is *not* the raw upstream worker.
|
|
138
|
+
- **Third argument** — the worker's `context`, an `OpenStruct` by default or whatever you pass as `context:`; per-worker mutable state that survives across shifts (and survives a rescued `PolicyViolation`).
|
|
139
|
+
|
|
140
|
+
A worker with no task gets a default pass-through task. `worker.effective_policy` reports the resolved policy (own declaration, else pipeline default, else global default).
|
|
141
|
+
|
|
142
|
+
## Composition: `|`, `with_policy`, `freeze!`
|
|
143
|
+
|
|
144
|
+
```ruby
|
|
145
|
+
pipeline = source | filter | transform | sink # tail-returned; call shift on it
|
|
146
|
+
|
|
147
|
+
pipeline = (source | filter | transform | sink)
|
|
148
|
+
.with_policy(:frozen) # pipeline default for workers with no declaration
|
|
149
|
+
.freeze! # lock the topology; upstream walk from the tail
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Both `with_policy` and `freeze!` walk **upstream** through the supply chain from their receiver and return it, so they chain — and both should be called on the pipeline's tail. `freeze!` locks topology only (supply wiring, Gang rosters); closure and context state stay mutable. Use `#freeze!`, never bare `#freeze`. Details in [[Handoff-Policies]].
|
|
153
|
+
|
|
154
|
+
## `Shifty::Gang`
|
|
155
|
+
|
|
156
|
+
A Gang wraps an ordered roster of workers so a whole chain acts like one worker: it has a `supply`, a `shift`, and composes with `|` like anything else.
|
|
157
|
+
|
|
158
|
+
```ruby
|
|
159
|
+
gang = Shifty::Gang[parser, enricher] # or Gang.new([parser, enricher])
|
|
160
|
+
pipeline = reader | gang | writer
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
Policy features:
|
|
164
|
+
|
|
165
|
+
- **Construction fanout:** `Gang.new(workers, policy: :isolated)` sets the pipeline default on every roster member.
|
|
166
|
+
- **`with_policy` fanout:** `gang.with_policy(:shared)` does the same, chainably.
|
|
167
|
+
- **Append inheritance:** the gang persists its declared policy, so workers appended *after* the declaration inherit it too.
|
|
168
|
+
- **Member contracts win:** a roster member's own `policy:` declaration beats the gang's, per the usual precedence.
|
|
169
|
+
- **Chains walk through:** `(upstream | gang | tail).with_policy(:isolated)` reaches the gang's roster *and* workers upstream of it; `freeze!` walks the same path.
|
|
170
|
+
- **Freezing:** a frozen gang still runs, but its roster membership is locked — `append` raises `FrozenError`.
|
|
171
|
+
- **Criteria bypass is still governed:** when a gang's `criteria` bypasses its workers, the value crossing the gang boundary is still policy-governed at the entry worker.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
module Shifty
|
|
2
|
+
class Configuration
|
|
3
|
+
attr_reader :default_policy
|
|
4
|
+
|
|
5
|
+
def initialize
|
|
6
|
+
@default_policy = :frozen
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
def default_policy=(policy_name)
|
|
10
|
+
@default_policy = Policy.validate!(policy_name)
|
|
11
|
+
end
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
class << self
|
|
15
|
+
def config
|
|
16
|
+
@config ||= Configuration.new
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def configure
|
|
20
|
+
yield config
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def reset_configuration!
|
|
24
|
+
@config = Configuration.new
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
data/lib/shifty/dsl.rb
CHANGED
|
@@ -1,6 +1,4 @@
|
|
|
1
1
|
module Shifty
|
|
2
|
-
class WorkerInitializationError < StandardError; end
|
|
3
|
-
|
|
4
2
|
module DSL
|
|
5
3
|
def source_worker(argument = nil, &block)
|
|
6
4
|
ensure_correct_arity_for!(argument, block)
|
|
@@ -19,36 +17,46 @@ module Shifty
|
|
|
19
17
|
end
|
|
20
18
|
end
|
|
21
19
|
|
|
22
|
-
def relay_worker(&block)
|
|
23
|
-
|
|
20
|
+
def relay_worker(options = {}, &block)
|
|
21
|
+
options[:tags] ||= []
|
|
22
|
+
options[:tags] << :relay
|
|
23
|
+
ensure_regular_arity!(block)
|
|
24
24
|
|
|
25
|
-
Worker.new(
|
|
25
|
+
Worker.new(options) do |value|
|
|
26
26
|
value && block.call(value)
|
|
27
27
|
end
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
-
def side_worker(
|
|
31
|
-
|
|
30
|
+
def side_worker(options = {}, &block)
|
|
31
|
+
options[:tags] ||= []
|
|
32
|
+
options[:tags] << :side_effect
|
|
33
|
+
deprecate_mode_option!(options)
|
|
34
|
+
ensure_regular_arity!(block)
|
|
32
35
|
|
|
33
|
-
|
|
36
|
+
# The block must ask the worker for its policy at shift time, so the
|
|
37
|
+
# local is captured by the closure before it is assigned. Not redundant.
|
|
38
|
+
worker = nil
|
|
39
|
+
worker = Worker.new(options) do |value| # standard:disable Style/RedundantAssignment
|
|
34
40
|
value.tap do |v|
|
|
35
|
-
|
|
36
|
-
|
|
41
|
+
next unless v
|
|
42
|
+
# Under :isolated the block observes a private scratch copy, so
|
|
43
|
+
# its mutations evaporate and the untouched value flows on.
|
|
44
|
+
used_value = (worker.effective_policy == :isolated) ?
|
|
45
|
+
Policy::Isolated.call(v, worker: worker) : v
|
|
37
46
|
|
|
38
|
-
|
|
47
|
+
block.call(used_value)
|
|
39
48
|
end
|
|
40
49
|
end
|
|
50
|
+
worker
|
|
41
51
|
end
|
|
42
52
|
|
|
43
|
-
def filter_worker(
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
callable = argument.respond_to?(:call) ? argument : block
|
|
48
|
-
ensure_callable(callable)
|
|
53
|
+
def filter_worker(options = {}, &block)
|
|
54
|
+
options[:tags] ||= []
|
|
55
|
+
options[:tags] << :filter
|
|
56
|
+
ensure_callable!(block)
|
|
49
57
|
|
|
50
|
-
Worker.new(
|
|
51
|
-
while value && !
|
|
58
|
+
Worker.new(options) do |value, supply|
|
|
59
|
+
while value && !block.call(value)
|
|
52
60
|
value = supply.shift
|
|
53
61
|
end
|
|
54
62
|
value
|
|
@@ -62,14 +70,18 @@ module Shifty
|
|
|
62
70
|
end
|
|
63
71
|
end
|
|
64
72
|
|
|
65
|
-
def batch_worker(options = {
|
|
66
|
-
|
|
73
|
+
def batch_worker(options = {}, &block)
|
|
74
|
+
options[:tags] ||= []
|
|
75
|
+
options[:tags] << :batch
|
|
76
|
+
options[:gathering] ||= 1
|
|
77
|
+
|
|
78
|
+
ensure_regular_arity!(block) if block
|
|
67
79
|
batch_full = block ||
|
|
68
80
|
proc { |_, batch| batch.size >= options[:gathering] }
|
|
69
81
|
|
|
70
|
-
|
|
82
|
+
options[:context] = BatchContext.new({batch_full: batch_full})
|
|
71
83
|
|
|
72
|
-
Worker.new(
|
|
84
|
+
Worker.new(options) do |value, supply, context|
|
|
73
85
|
if value
|
|
74
86
|
context.collection = [value]
|
|
75
87
|
until context.batch_complete?(
|
|
@@ -83,10 +95,12 @@ module Shifty
|
|
|
83
95
|
end
|
|
84
96
|
end
|
|
85
97
|
|
|
86
|
-
def splitter_worker(&block)
|
|
87
|
-
|
|
98
|
+
def splitter_worker(options = {}, &block)
|
|
99
|
+
options[:tags] ||= []
|
|
100
|
+
options[:tags] << :splitter
|
|
101
|
+
ensure_regular_arity!(block)
|
|
88
102
|
|
|
89
|
-
Worker.new(
|
|
103
|
+
Worker.new(options) do |value|
|
|
90
104
|
if value.nil?
|
|
91
105
|
value
|
|
92
106
|
else
|
|
@@ -99,9 +113,11 @@ module Shifty
|
|
|
99
113
|
end
|
|
100
114
|
end
|
|
101
115
|
|
|
116
|
+
# don't like that this is a second exception to accepting options..
|
|
102
117
|
def trailing_worker(trail_length = 2)
|
|
118
|
+
options = {tags: [:trailing]}
|
|
103
119
|
trail = []
|
|
104
|
-
Worker.new(
|
|
120
|
+
Worker.new(options) do |value, supply|
|
|
105
121
|
if value
|
|
106
122
|
trail.unshift value
|
|
107
123
|
if trail.size >= trail_length
|
|
@@ -111,9 +127,12 @@ module Shifty
|
|
|
111
127
|
trail.unshift supply.shift
|
|
112
128
|
end
|
|
113
129
|
|
|
114
|
-
trail
|
|
130
|
+
# Hand off a snapshot: the builder keeps mutating `trail` across
|
|
131
|
+
# calls, and a downstream :frozen intake would freeze the live
|
|
132
|
+
# closure array in place.
|
|
133
|
+
trail.dup
|
|
115
134
|
else
|
|
116
|
-
value
|
|
135
|
+
value # hint: it's nil!
|
|
117
136
|
end
|
|
118
137
|
end
|
|
119
138
|
end
|
|
@@ -124,17 +143,30 @@ module Shifty
|
|
|
124
143
|
|
|
125
144
|
private
|
|
126
145
|
|
|
146
|
+
def deprecate_mode_option!(options)
|
|
147
|
+
return unless options.key?(:mode)
|
|
148
|
+
mode = options.delete(:mode)
|
|
149
|
+
if mode == :hardened
|
|
150
|
+
warn "[shifty] side_worker mode: :hardened is deprecated and will be " \
|
|
151
|
+
"removed in 1.0.0; use policy: :isolated instead."
|
|
152
|
+
options[:policy] ||= :isolated
|
|
153
|
+
else
|
|
154
|
+
warn "[shifty] side_worker's mode: option is deprecated and ignored " \
|
|
155
|
+
"(received mode: #{mode.inspect}); declare a policy: instead."
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
127
159
|
def throw_with(*msg)
|
|
128
160
|
raise WorkerInitializationError.new([msg].flatten.join(" "))
|
|
129
161
|
end
|
|
130
162
|
|
|
131
|
-
def ensure_callable(callable)
|
|
163
|
+
def ensure_callable!(callable)
|
|
132
164
|
unless callable&.respond_to?(:call)
|
|
133
165
|
throw_with "You must supply a callable"
|
|
134
166
|
end
|
|
135
167
|
end
|
|
136
168
|
|
|
137
|
-
def ensure_regular_arity(block)
|
|
169
|
+
def ensure_regular_arity!(block)
|
|
138
170
|
if block.arity != 1
|
|
139
171
|
throw_with \
|
|
140
172
|
"Worker must accept exactly one argument (arity == 1)"
|
|
@@ -145,7 +177,7 @@ module Shifty
|
|
|
145
177
|
def ensure_correct_arity_for!(argument, block)
|
|
146
178
|
return unless block
|
|
147
179
|
if argument
|
|
148
|
-
ensure_regular_arity(block)
|
|
180
|
+
ensure_regular_arity!(block)
|
|
149
181
|
elsif block.arity > 0
|
|
150
182
|
throw_with \
|
|
151
183
|
"Source worker cannot accept any arguments (arity == 0)"
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
module Shifty
|
|
2
|
+
class Error < StandardError; end
|
|
3
|
+
|
|
4
|
+
class WorkerError < Error; end
|
|
5
|
+
|
|
6
|
+
class WorkerInitializationError < Error; end
|
|
7
|
+
|
|
8
|
+
class PolicyError < Error
|
|
9
|
+
attr_reader :worker, :policy, :value
|
|
10
|
+
|
|
11
|
+
def cause
|
|
12
|
+
@wrapped_cause || super
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
private
|
|
16
|
+
|
|
17
|
+
def worker_label
|
|
18
|
+
label = (worker.respond_to?(:name) && worker.name) ? "`#{worker.name}`" : "(unnamed)"
|
|
19
|
+
tags = worker.respond_to?(:tags) ? worker.tags : []
|
|
20
|
+
tags&.any? ? "#{label} (tags: #{tags.inspect})" : label
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
# Raised when a task mutates a value it received under a policy that
|
|
25
|
+
# forbids mutation. Wraps the original FrozenError (never masks it) and
|
|
26
|
+
# names the worker, the effective policy, and the object the task tried
|
|
27
|
+
# to mutate, with a heuristic locating that object relative to the
|
|
28
|
+
# handed-off value.
|
|
29
|
+
class PolicyViolation < PolicyError
|
|
30
|
+
attr_reader :receiver
|
|
31
|
+
|
|
32
|
+
def initialize(worker:, policy:, receiver:, value:, cause:)
|
|
33
|
+
@worker = worker
|
|
34
|
+
@policy = policy
|
|
35
|
+
@receiver = receiver
|
|
36
|
+
@value = value
|
|
37
|
+
@wrapped_cause = cause
|
|
38
|
+
super(build_message)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
private
|
|
42
|
+
|
|
43
|
+
def build_message
|
|
44
|
+
<<~MSG
|
|
45
|
+
Worker #{worker_label} received its value under the #{policy.inspect} handoff \
|
|
46
|
+
policy, and its task attempted to mutate #{receiver_description}.
|
|
47
|
+
|
|
48
|
+
Either make the task non-destructive — e.g. `map` instead of `map!`, \
|
|
49
|
+
`value.with(...)`, `arr + [x]`, `hash.merge(...)` — or declare a different \
|
|
50
|
+
policy on this worker: `policy: :isolated` (task works on a private scratch \
|
|
51
|
+
copy) or `policy: :shared` (raw reference; no protection). \
|
|
52
|
+
More: https://github.com/joelhelbling/shifty/wiki/Handoff-Policies
|
|
53
|
+
MSG
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def receiver_description
|
|
57
|
+
if receiver.equal?(value)
|
|
58
|
+
"an instance of #{receiver.class} — the handed-off value itself"
|
|
59
|
+
elsif reachable_from_value?
|
|
60
|
+
"an instance of #{receiver.class} reachable from the handed-off value"
|
|
61
|
+
else
|
|
62
|
+
"an instance of #{receiver.class} (#{receiver.inspect[0, 80]}), which may be " \
|
|
63
|
+
"unrelated to the handed-off value (an instance of #{value.class}); " \
|
|
64
|
+
"inspect both to judge"
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Bounds the diagnostic graph walk; past this the heuristic gives up
|
|
69
|
+
# and reports the honest "inspect both to judge" fallback rather than
|
|
70
|
+
# risking a SystemStackError that would mask the violation itself.
|
|
71
|
+
MAX_REACHABILITY_NODES = 50_000
|
|
72
|
+
|
|
73
|
+
def reachable_from_value?
|
|
74
|
+
seen = {}.compare_by_identity
|
|
75
|
+
stack = [value]
|
|
76
|
+
until stack.empty?
|
|
77
|
+
node = stack.pop
|
|
78
|
+
next if node.nil? || seen[node]
|
|
79
|
+
return false if seen.size >= MAX_REACHABILITY_NODES
|
|
80
|
+
seen[node] = true
|
|
81
|
+
return true if node.equal?(receiver)
|
|
82
|
+
stack.concat(children_of(node))
|
|
83
|
+
end
|
|
84
|
+
false
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def children_of(node)
|
|
88
|
+
case node
|
|
89
|
+
when Array then node
|
|
90
|
+
when Hash then node.keys + node.values
|
|
91
|
+
when Struct then node.to_a
|
|
92
|
+
else
|
|
93
|
+
members = (node.respond_to?(:to_h) && node.is_a?(Data)) ? node.to_h.values : []
|
|
94
|
+
members + node.instance_variables.map { |iv| node.instance_variable_get(iv) }
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# Raised at the handoff itself when a value cannot cross the boundary
|
|
100
|
+
# under the effective policy (cannot be frozen or deep-copied).
|
|
101
|
+
class UnshareableValue < PolicyError
|
|
102
|
+
def initialize(worker:, policy:, value:, cause: nil)
|
|
103
|
+
@worker = worker
|
|
104
|
+
@policy = policy
|
|
105
|
+
@value = value
|
|
106
|
+
@wrapped_cause = cause
|
|
107
|
+
super(build_message)
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
private
|
|
111
|
+
|
|
112
|
+
def build_message
|
|
113
|
+
"Worker #{worker_label} received a value (an instance of #{value.class}) " \
|
|
114
|
+
"that cannot cross the handoff boundary under the #{policy.inspect} policy: " \
|
|
115
|
+
"it cannot be #{(policy == :isolated) ? "deep-copied" : "frozen"}. " \
|
|
116
|
+
"Declare `policy: :shared` on this worker (raw pass-by-reference), " \
|
|
117
|
+
"or restructure the value. " \
|
|
118
|
+
"More: https://github.com/joelhelbling/shifty/wiki/Handoff-Policies"
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
end
|