shifty 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/CHANGELOG.md +79 -0
- data/README.md +79 -35
- data/benchmark/RESULTS.md +69 -0
- data/benchmark/handoff_policies.rb +82 -0
- data/docs/planning/artifacts/.discovery-notes.md +40 -0
- data/docs/planning/artifacts/.round1-specialist-outputs.md +196 -0
- data/docs/planning/artifacts/implementation-decision-log.md +231 -0
- data/docs/planning/artifacts/implementation-iteration-history.md +73 -0
- data/docs/planning/feature-implementation-plan.md +191 -0
- data/docs/planning/handoff-immutability-policies.md +590 -0
- data/docs/shifty/worker.md +8 -19
- data/docs/use_cases.md +59 -0
- data/docs/wiki/Coding-Idioms-Under-Frozen.md +74 -0
- data/docs/wiki/Handoff-Policies.md +167 -0
- data/docs/wiki/Home.md +59 -0
- data/docs/wiki/Migration-Guide-0.6.md +96 -0
- data/docs/wiki/Performance.md +59 -0
- data/docs/wiki/Testing-Workers.md +127 -0
- data/docs/wiki/Worker-Types.md +171 -0
- data/docs/wiki/_Sidebar.md +13 -0
- data/lib/shifty/configuration.rb +27 -0
- data/lib/shifty/dsl.rb +35 -14
- data/lib/shifty/errors.rb +121 -0
- data/lib/shifty/gang.rb +25 -2
- data/lib/shifty/policy.rb +124 -0
- data/lib/shifty/rspec.rb +37 -0
- data/lib/shifty/testing.rb +82 -0
- data/lib/shifty/version.rb +1 -1
- data/lib/shifty/worker.rb +108 -5
- data/lib/shifty.rb +3 -0
- data/shifty.gemspec +6 -1
- metadata +56 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8e7ad29ca6b77f1216ed40460e252d48743f6b6786a8adbd1e277fc673115829
|
|
4
|
+
data.tar.gz: ab8d5fced324f9b7d1834ac9c1741428122de18acd03f02d1e7d550d8f190879
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f7884214ed282f0d200d18391989c04af01493f2f17477e2b0f4fe69ebe782e1401fcaed5dcd40561f23f4e05831b79dd09b50e8b09d422262de7af1310e5ec3
|
|
7
|
+
data.tar.gz: 67c9354307d51baf7628b4079cf0f86dd9d80e7444d6c286fc65d6dd3df6316c9f24d62190281ba127700d2f64562d716cedec5fa5f605d337d81639e1cd37f8
|
data/.gitignore
CHANGED
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.6.0
|
|
4
|
+
|
|
5
|
+
The handoff immutability release. Values crossing worker boundaries are now
|
|
6
|
+
governed by a **handoff policy**, and the default changed from raw shared
|
|
7
|
+
references to deeply frozen values. This is a breaking change — see the
|
|
8
|
+
[Migration Guide](https://github.com/joelhelbling/shifty/wiki/Migration-Guide-0.6).
|
|
9
|
+
|
|
10
|
+
### Breaking changes
|
|
11
|
+
|
|
12
|
+
- **`:frozen` is the new default handoff policy.** Every value a worker
|
|
13
|
+
receives is deeply frozen (`Ractor.make_shareable`) at intake. A task that
|
|
14
|
+
mutates its input now raises `Shifty::PolicyViolation` at the offending
|
|
15
|
+
worker — immediately and with diagnostics — instead of silently corrupting
|
|
16
|
+
what downstream workers observe. Restore the old behavior per worker with
|
|
17
|
+
`policy: :shared`, per pipeline with `.with_policy(:shared)`, or globally
|
|
18
|
+
with `Shifty.configure { |c| c.default_policy = :shared }`.
|
|
19
|
+
- **Unshareable values can no longer cross a default-policy boundary.**
|
|
20
|
+
IO handles, procs, lazy enumerators, and (under `:isolated`) anything
|
|
21
|
+
Marshal rejects raise `Shifty::UnshareableValue` with guidance. Declare
|
|
22
|
+
`policy: :shared` on workers that pass such values. Note: an IO *nested
|
|
23
|
+
inside* a container is not detected — declare `:shared` for those workers.
|
|
24
|
+
- **The task's second argument is now a policy-governed supply proxy** that
|
|
25
|
+
responds to `#shift` only, no longer the raw upstream worker object.
|
|
26
|
+
- **Ruby floor is 3.2** (`required_ruby_version >= 3.2`). Older Rubies stay
|
|
27
|
+
on shifty 0.5.0.
|
|
28
|
+
- `Shifty::WorkerError` and `Shifty::WorkerInitializationError` are
|
|
29
|
+
reparented under a new `Shifty::Error` base (still `StandardError`s).
|
|
30
|
+
|
|
31
|
+
### Deprecations
|
|
32
|
+
|
|
33
|
+
- **`side_worker mode: :hardened` is deprecated** (removed in 1.0.0); it maps
|
|
34
|
+
to `policy: :isolated` with a warning. Semantics preserved — both use a
|
|
35
|
+
Marshal deep copy. Any other `mode:` value now warns and is ignored.
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
|
|
39
|
+
- **Handoff policies**: `:frozen` (default), `:isolated` (private mutable
|
|
40
|
+
deep copy; task mutations stay local), `:shared` (raw reference — the
|
|
41
|
+
escape hatch). Declarable per worker (`policy:` kwarg), per pipeline
|
|
42
|
+
(`.with_policy(...)` on a chain or Gang, `policy:` kwarg on Gang), and
|
|
43
|
+
globally. Precedence: worker > pipeline > global.
|
|
44
|
+
- **Rich diagnostics**: `Shifty::PolicyViolation` (worker, policy, receiver,
|
|
45
|
+
value, cause; a heuristic reports whether the mutated object is the
|
|
46
|
+
handed-off value, reachable from it, or possibly unrelated) and
|
|
47
|
+
`Shifty::UnshareableValue`. A rescued `PolicyViolation` does not kill the
|
|
48
|
+
pipeline — the next `shift` continues with the next value.
|
|
49
|
+
- **Worker `name:` kwarg** for diagnostics.
|
|
50
|
+
- **`Shifty::Testing`** (opt-in `require "shifty/testing"`):
|
|
51
|
+
`Testing.run(worker, inputs:, policy: nil)` runs a worker through the
|
|
52
|
+
framework under its production policy; `Testing.mutates_input?` is the
|
|
53
|
+
mutation detector — the pre-upgrade migration inventory tool.
|
|
54
|
+
- **RSpec sugar** (opt-in `require "shifty/rspec"`): the `mutate_input`
|
|
55
|
+
matcher and the `"a policy-safe worker"` shared example.
|
|
56
|
+
- **`#freeze!`** locks an assembled pipeline's topology (call it on the
|
|
57
|
+
tail). Rewiring — `supply=`, Gang `append`, roster mutation — raises
|
|
58
|
+
`FrozenError`. Worker closure/context state stays mutable.
|
|
59
|
+
- **Benchmark suite** (`benchmark/handoff_policies.rb`, manual-run) with
|
|
60
|
+
results in `benchmark/RESULTS.md` and on the wiki: steady-state `:frozen`
|
|
61
|
+
costs ~71ns per handoff regardless of value size, with zero allocations.
|
|
62
|
+
- **In-depth documentation** on the
|
|
63
|
+
[GitHub wiki](https://github.com/joelhelbling/shifty/wiki): policies,
|
|
64
|
+
coding idioms under `:frozen`, migration, testing, worker types,
|
|
65
|
+
performance.
|
|
66
|
+
|
|
67
|
+
### Fixed
|
|
68
|
+
|
|
69
|
+
- `trailing_worker` handed off its live internal array; under `:frozen` this
|
|
70
|
+
is the exact aliasing bug the policies exist to catch. It now hands off a
|
|
71
|
+
snapshot.
|
|
72
|
+
- `ostruct` is declared as a runtime dependency (no longer a default gem as
|
|
73
|
+
of Ruby 4).
|
|
74
|
+
- CI now tests Ruby 3.2/3.3/3.4 (was 2.6/2.7/3.0); dropped the abandoned
|
|
75
|
+
codeclimate-test-reporter (which pinned simplecov to a 2016 release).
|
|
76
|
+
|
|
77
|
+
## 0.5.0 and earlier
|
|
78
|
+
|
|
79
|
+
See the [release history](https://github.com/joelhelbling/shifty/releases).
|
data/README.md
CHANGED
|
@@ -1,12 +1,65 @@
|
|
|
1
1
|
[](https://badge.fury.io/rb/shifty)
|
|
2
2
|
[](https://github.com/joelhelbling/shifty/actions/workflows/ruby.yml)
|
|
3
|
-
[](https://codeclimate.com/github/joelhelbling/shifty/maintainability)
|
|
4
|
-
[](https://codeclimate.com/github/joelhelbling/shifty/test_coverage)
|
|
5
3
|
|
|
6
4
|
# The Shifty Framework
|
|
7
5
|
|
|
8
6
|
_"How many Ruby fibers does it take to screw in a lightbulb?"_
|
|
9
7
|
|
|
8
|
+
Shifty requires **Ruby 3.2 or newer** as of version 0.6.0. On older Rubies,
|
|
9
|
+
use shifty 0.5.0 (which also retains the classic mutable-handoff behavior).
|
|
10
|
+
|
|
11
|
+
## Concurrency Model
|
|
12
|
+
|
|
13
|
+
Shifty utilizes Ruby Fibers for cooperative multitasking. This means that all tasks (or "workers") run within a single operating system thread and explicitly yield control to one another. This model is intentionally chosen for its simplicity, which makes it easier to reason about and build sequential data processing pipelines.
|
|
14
|
+
|
|
15
|
+
Single-threading frees you from *preemptive* concurrency hazards (races, mutexes) within a pipeline. It does not, by itself, make the data passing between workers safe — a worker that mutates a value it was handed could corrupt what later workers see. As of 0.6.0, those handoffs are governed by [handoff immutability policies](#handoff-policies-new-in-060). The Fiber model is also deliberately compatible with a future Ractor-backed worker type, rather than a rejection of parallelism — and frozen, shareable handoff values are exactly what Ractor boundaries want.
|
|
16
|
+
|
|
17
|
+
For a more detailed explanation of Shifty's design and typical use cases, please see the [Use Cases Document](docs/use_cases.md).
|
|
18
|
+
|
|
19
|
+
## Handoff Policies (new in 0.6.0)
|
|
20
|
+
|
|
21
|
+
Shifty workers are easy to reason about because each one is isolated; the
|
|
22
|
+
values flowing *between* them were, until now, the un-governed part of the
|
|
23
|
+
system. As of 0.6.0, values are **deeply frozen at every handoff by
|
|
24
|
+
default**. Workers that need a private scratch copy can declare
|
|
25
|
+
`policy: :isolated`; workers that genuinely need shared mutable references
|
|
26
|
+
can declare `policy: :shared`. Mutation bugs that used to surface as
|
|
27
|
+
mysterious downstream corruption now either cannot happen or raise
|
|
28
|
+
immediately at the worker responsible — with an error message that tells
|
|
29
|
+
you exactly what to do about it.
|
|
30
|
+
|
|
31
|
+
```ruby
|
|
32
|
+
# Per worker — part of the worker's contract:
|
|
33
|
+
scratch = Shifty::Worker.new(policy: :isolated) { |v| v << transform(v) }
|
|
34
|
+
|
|
35
|
+
# Per pipeline — the default for workers that don't declare their own:
|
|
36
|
+
pipeline = (source | parser | sink).with_policy(:shared)
|
|
37
|
+
|
|
38
|
+
# Globally (the built-in default is :frozen):
|
|
39
|
+
Shifty.configure { |c| c.default_policy = :shared }
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
| | `:frozen` (default) | `:isolated` | `:shared` |
|
|
43
|
+
|---|---|---|---|
|
|
44
|
+
| The task receives | frozen reference | private mutable deep copy | the raw reference |
|
|
45
|
+
| Mutation in the task | raises `Shifty::PolicyViolation` | works, stays local | works, leaks |
|
|
46
|
+
| Copying per handoff | none | full graph | none |
|
|
47
|
+
|
|
48
|
+
### Upgrading from 0.5.0?
|
|
49
|
+
|
|
50
|
+
Start with the [Migration Guide](https://github.com/joelhelbling/shifty/wiki/Migration-Guide-0.6) —
|
|
51
|
+
it names everything that breaks and gives four migration paths in order of
|
|
52
|
+
preference. `Shifty::Testing.mutates_input?` will inventory which of your
|
|
53
|
+
workers mutate their input *before* you flip the switch.
|
|
54
|
+
|
|
55
|
+
There's much more in the wiki: [Handoff Policies](https://github.com/joelhelbling/shifty/wiki/Handoff-Policies) in depth,
|
|
56
|
+
[coding idioms under :frozen](https://github.com/joelhelbling/shifty/wiki/Coding-Idioms-Under-Frozen),
|
|
57
|
+
the [0.6 migration guide](https://github.com/joelhelbling/shifty/wiki/Migration-Guide-0.6),
|
|
58
|
+
[testing workers](https://github.com/joelhelbling/shifty/wiki/Testing-Workers), and
|
|
59
|
+
[performance numbers](https://github.com/joelhelbling/shifty/wiki/Performance)
|
|
60
|
+
(spoiler: at steady state, `:frozen` costs about 71 nanoseconds per handoff
|
|
61
|
+
no matter how big the value is).
|
|
62
|
+
|
|
10
63
|
## Quick Start
|
|
11
64
|
|
|
12
65
|
Add this line to your application's Gemfile:
|
|
@@ -134,7 +187,7 @@ work.
|
|
|
134
187
|
source = source_worker (0..3)
|
|
135
188
|
|
|
136
189
|
evens = []
|
|
137
|
-
even_stasher =
|
|
190
|
+
even_stasher = side_worker do |value|
|
|
138
191
|
if value % 2 == 0
|
|
139
192
|
evens << value
|
|
140
193
|
end
|
|
@@ -163,10 +216,10 @@ it is, there really isn't a way to prevent the implementation of any
|
|
|
163
216
|
worker from creating side effects.
|
|
164
217
|
|
|
165
218
|
_And still wait,_ you'll also be wanting to say, _isn't it possible
|
|
166
|
-
that a side worker could mutate the value as it's passed through?_
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
219
|
+
that a side worker could mutate the value as it's passed through?_
|
|
220
|
+
As of 0.6.0: no, not by accident. A side worker's contract is
|
|
221
|
+
"observe, don't modify," and the default `:frozen` handoff policy
|
|
222
|
+
finally enforces it.
|
|
170
223
|
|
|
171
224
|
The side effect worker's purpose is to provide _intentionality_ and
|
|
172
225
|
clarity. When you're creating a side effect, let it be very clear.
|
|
@@ -178,55 +231,40 @@ same token, if there is a problem with a side effect, troubleshooting
|
|
|
178
231
|
it will be much simpler if the side effects are already isolated and
|
|
179
232
|
named.
|
|
180
233
|
|
|
181
|
-
Another measure for preventing side workers from creating unwanted
|
|
182
|
-
side-effects is to use `:hardened` mode. That's right, `side_worker`
|
|
183
|
-
has a mode (which defaults to `:normal`). When set to `:hardened`,
|
|
184
|
-
each value is `Marshal.dump`ed and `Marshal.load`ed before being
|
|
185
|
-
passed to the side worker's callable. This helps to ensure that
|
|
186
|
-
the original value will be passed through to the next worker in the
|
|
187
|
-
queue in a pristine state (unmodified), and that no future or
|
|
188
|
-
subsequent modification can take place.
|
|
189
|
-
|
|
190
234
|
Given a source...
|
|
191
235
|
|
|
192
236
|
```ruby
|
|
193
237
|
source = source_worker [[:foo], [:bar]]
|
|
194
238
|
```
|
|
195
239
|
|
|
196
|
-
|
|
240
|
+
...consider this mutating side worker:
|
|
197
241
|
|
|
198
242
|
```ruby
|
|
199
243
|
unsafe = side_worker { |v| v << :boo }
|
|
200
|
-
```
|
|
201
|
-
|
|
202
|
-
This produces unwanted mutations, because of the `<<`.
|
|
203
244
|
|
|
204
|
-
```ruby
|
|
205
245
|
pipeline = source | unsafe
|
|
206
246
|
|
|
207
|
-
pipeline.shift #=>
|
|
208
|
-
pipeline.shift #=> [:bar, :boo] <-- Disaster!
|
|
209
|
-
pipeline.shift #=> nil
|
|
247
|
+
pipeline.shift #=> raises Shifty::PolicyViolation, naming this worker
|
|
210
248
|
```
|
|
211
249
|
|
|
212
|
-
|
|
250
|
+
Under the default `:frozen` policy that mutation raises immediately,
|
|
251
|
+
at the offending worker. If the side worker's mutations are harmless
|
|
252
|
+
scratch work you'd rather keep, give it a private copy instead:
|
|
213
253
|
|
|
214
254
|
```ruby
|
|
215
|
-
unsafe = side_worker(:
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
The `:hardened` side worker doesn't have access to the original
|
|
219
|
-
value, and therefore its shenanigans are moot with respect to
|
|
220
|
-
the main workflow:
|
|
255
|
+
unsafe = side_worker(policy: :isolated) { |v| v << :boo }
|
|
221
256
|
|
|
222
|
-
```ruby
|
|
223
257
|
pipeline = source | unsafe
|
|
224
258
|
|
|
225
|
-
pipeline.shift #=> [:foo] <--
|
|
226
|
-
pipeline.shift #=> [:bar] <--
|
|
259
|
+
pipeline.shift #=> [:foo] <-- mutations evaporate
|
|
260
|
+
pipeline.shift #=> [:bar] <-- the original flows on, pristine
|
|
227
261
|
pipeline.shift #=> nil
|
|
228
262
|
```
|
|
229
263
|
|
|
264
|
+
(Old-timers: `side_worker mode: :hardened` still works, mapped to
|
|
265
|
+
`policy: :isolated` with a deprecation warning; it will be removed in
|
|
266
|
+
1.0.0.)
|
|
267
|
+
|
|
230
268
|
### Filter Worker
|
|
231
269
|
|
|
232
270
|
The filter worker simply passes through the values which are given
|
|
@@ -453,7 +491,13 @@ example).
|
|
|
453
491
|
## Shifty::Worker
|
|
454
492
|
|
|
455
493
|
The Shifty::Worker documentation (which was the old README) is now
|
|
456
|
-
[here](docs/shifty/worker.md)
|
|
494
|
+
[here](docs/shifty/worker.md), and the
|
|
495
|
+
[wiki](https://github.com/joelhelbling/shifty/wiki) goes deeper on every
|
|
496
|
+
worker type, handoff policies, testing, migration, and performance.
|
|
457
497
|
|
|
458
498
|
## Roadmap
|
|
459
499
|
|
|
500
|
+
- **1.0.0** — remove the deprecated `:hardened` alias; stabilize the API.
|
|
501
|
+
- **Someday** — a Ractor-backed worker type for true parallelism; the
|
|
502
|
+
`:frozen` policy's shareable values are deliberately the groundwork.
|
|
503
|
+
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Handoff policy benchmark results
|
|
2
|
+
|
|
3
|
+
Run: `bundle exec ruby benchmark/handoff_policies.rb`
|
|
4
|
+
Environment: Ruby 4.0.5 (arm64-darwin25), Apple Silicon. 2026-07-10.
|
|
5
|
+
(Re-run on your own hardware; relative ordering is what matters.)
|
|
6
|
+
|
|
7
|
+
## Method note (read first)
|
|
8
|
+
|
|
9
|
+
Rows labeled *fresh value* construct a new value inside each iteration, so
|
|
10
|
+
they include construction cost — that keeps `:frozen`'s freeze bit honest
|
|
11
|
+
(freezing is once-per-object; a pre-built value would be frozen after the
|
|
12
|
+
first iteration). The *steady state* row uses a pre-built, already-shareable
|
|
13
|
+
value and measures the policy call alone. Compare fresh rows against the
|
|
14
|
+
`:shared` fresh row (same construction overhead); read the steady-state row
|
|
15
|
+
as the per-handoff cost once a value is shareable.
|
|
16
|
+
|
|
17
|
+
`:hardened` (legacy) has no separate row: its mechanism — a Marshal
|
|
18
|
+
round-trip — is exactly what `:isolated` uses, so the `:isolated` rows are
|
|
19
|
+
its numbers.
|
|
20
|
+
|
|
21
|
+
## Per-handoff throughput (i/s; higher is better)
|
|
22
|
+
|
|
23
|
+
| Shape | `:shared` (fresh) | `:frozen` first handoff (fresh) | `:frozen` steady state | `:isolated` (fresh) | `:isolated` on already-frozen (§8.3) |
|
|
24
|
+
|---|---|---|---|---|---|
|
|
25
|
+
| small string token | 10.56M | 5.02M | **14.04M** | 1.44M | 1.57M |
|
|
26
|
+
| array, 100 strings | 140.8k | 84.3k | **14.12M** | 34.5k | 46.6k |
|
|
27
|
+
| hash, 100 pairs | 48.6k | 38.4k | **13.87M** | 14.1k | 20.4k |
|
|
28
|
+
| nested document, ~5k nodes | 9.9k | 5.3k | **13.74M** | 1.9k | 2.4k |
|
|
29
|
+
| deep nesting, 500 levels | 49.0k | 21.3k | **13.99M** | 8.5k | 10.8k |
|
|
30
|
+
|
|
31
|
+
## Allocations per handoff (GC pressure)
|
|
32
|
+
|
|
33
|
+
| Shape | `:shared` | `:frozen` (steady) | `:isolated` |
|
|
34
|
+
|---|---|---|---|
|
|
35
|
+
| small string token | 0 | 0 | 5 |
|
|
36
|
+
| array, 100 strings | 0 | 0 | 105 |
|
|
37
|
+
| hash, 100 pairs | 0 | 0 | 205 |
|
|
38
|
+
| nested document | 0 | 0 | 1,711 |
|
|
39
|
+
| deep nesting, 500 levels | 0 | 0 | 504 |
|
|
40
|
+
|
|
41
|
+
## Findings
|
|
42
|
+
|
|
43
|
+
1. **The §8.2 amortization claim holds.** Once a value is shareable,
|
|
44
|
+
`:frozen`'s per-handoff cost is a flat ~71ns **independent of value
|
|
45
|
+
size** (13.7–14.1M i/s across every shape) with zero allocations.
|
|
46
|
+
In an N-worker pipeline, only the first boundary pays the freeze
|
|
47
|
+
traversal; boundaries 2..N are effectively free. Combined with
|
|
48
|
+
copy-on-write task code, per-boundary cost is proportional to the
|
|
49
|
+
*change*, not the value.
|
|
50
|
+
2. **First-handoff freeze costs roughly 0.6–1.9× the value's own
|
|
51
|
+
construction cost** (e.g. ~4.8µs extra for a 100-string array that
|
|
52
|
+
costs ~7.1µs to build) — paid once per object graph, not per hop.
|
|
53
|
+
3. **`:isolated` is the expensive policy, as designed**: 3–7× slower than
|
|
54
|
+
`:shared` per boundary *per hop*, and the only policy that allocates
|
|
55
|
+
(a full copy of the graph per boundary — 1,711 objects per handoff for
|
|
56
|
+
the ~5k-node document). This is why it is not the default.
|
|
57
|
+
4. **§8.3 fast-path decision: keep option (b) — no fast path.** The
|
|
58
|
+
"already-frozen" column is the pure Marshal cost on a pre-built value
|
|
59
|
+
(no construction overhead): 92µs for the deep-nested shape, 21µs for
|
|
60
|
+
the mid array, versus ~71ns if a `Ractor.shareable?` check skipped
|
|
61
|
+
the copy — so a fast path would make those handoffs roughly three
|
|
62
|
+
orders of magnitude cheaper *when inputs are already shareable*.
|
|
63
|
+
It still loses: it would hand the task a frozen object where the
|
|
64
|
+
`:isolated` contract promises a mutable scratch copy. Contract
|
|
65
|
+
simplicity wins at this gem's scale. Reopen trigger: a real workload
|
|
66
|
+
where `:isolated` boundaries dominate and their inputs are typically
|
|
67
|
+
already shareable.
|
|
68
|
+
5. **The `:frozen` default is vindicated**: it is the only policy that is
|
|
69
|
+
simultaneously safe and, at steady state, the cheapest of the three.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Handoff policy benchmarks (spec §8.4). Manual-run, not CI:
|
|
2
|
+
#
|
|
3
|
+
# bundle exec ruby benchmark/handoff_policies.rb
|
|
4
|
+
#
|
|
5
|
+
# Measures, per policy and value shape:
|
|
6
|
+
# 1. per-handoff cost of :shared / :frozen / :isolated / legacy :hardened
|
|
7
|
+
# (Marshal), including :frozen's first-handoff vs steady-state cost
|
|
8
|
+
# (the §8.2 amortization claim), and
|
|
9
|
+
# 2. the pure Marshal cost :isolated pays on an already-shareable
|
|
10
|
+
# value — the copy a Ractor.shareable? fast path would skip
|
|
11
|
+
# entirely (the §8.3 open question), and
|
|
12
|
+
# 3. allocation counts per policy (GC pressure).
|
|
13
|
+
#
|
|
14
|
+
# Results feed the wiki Performance page.
|
|
15
|
+
|
|
16
|
+
require "bundler/setup"
|
|
17
|
+
require "shifty"
|
|
18
|
+
require "benchmark/ips"
|
|
19
|
+
|
|
20
|
+
SHAPES = {
|
|
21
|
+
"small token (string)" => -> { +"a token" },
|
|
22
|
+
"mid array (100 strings)" => -> { Array.new(100) { |i| "item-#{i}" } },
|
|
23
|
+
"mid hash (100 pairs)" => -> { Array.new(100) { |i| ["key-#{i}", "value-#{i}"] }.to_h },
|
|
24
|
+
"large document (nested, ~5k nodes)" => -> {
|
|
25
|
+
Array.new(50) do |i|
|
|
26
|
+
{"id" => i, "name" => "record-#{i}",
|
|
27
|
+
"tags" => Array.new(10) { |t| "tag-#{t}" },
|
|
28
|
+
"children" => Array.new(10) { |c| {"idx" => c, "payload" => "data-#{i}-#{c}"} }}
|
|
29
|
+
end
|
|
30
|
+
},
|
|
31
|
+
"deep nesting (500 levels)" => -> {
|
|
32
|
+
(1..500).reduce([]) { |acc, i| [i, acc] }
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
null_worker = Object.new
|
|
37
|
+
def null_worker.name = "bench"
|
|
38
|
+
|
|
39
|
+
def null_worker.tags = []
|
|
40
|
+
|
|
41
|
+
puts "Ruby #{RUBY_VERSION} — #{RUBY_PLATFORM}"
|
|
42
|
+
puts
|
|
43
|
+
|
|
44
|
+
SHAPES.each do |label, build|
|
|
45
|
+
puts "=" * 72
|
|
46
|
+
puts "SHAPE: #{label}"
|
|
47
|
+
puts "=" * 72
|
|
48
|
+
|
|
49
|
+
Benchmark.ips do |x|
|
|
50
|
+
x.report(":shared") do
|
|
51
|
+
Shifty::Policy::Shared.call(build.call, worker: null_worker)
|
|
52
|
+
end
|
|
53
|
+
x.report(":frozen (first handoff — fresh value each time)") do
|
|
54
|
+
Shifty::Policy::Frozen.call(build.call, worker: null_worker)
|
|
55
|
+
end
|
|
56
|
+
frozen_value = Ractor.make_shareable(build.call)
|
|
57
|
+
x.report(":frozen (steady state — already shareable)") do
|
|
58
|
+
Shifty::Policy::Frozen.call(frozen_value, worker: null_worker)
|
|
59
|
+
end
|
|
60
|
+
x.report(":isolated (Marshal deep copy)") do
|
|
61
|
+
Shifty::Policy::Isolated.call(build.call, worker: null_worker)
|
|
62
|
+
end
|
|
63
|
+
x.report(":isolated on already-frozen value (§8.3 fast-path question)") do
|
|
64
|
+
Shifty::Policy::Isolated.call(frozen_value, worker: null_worker)
|
|
65
|
+
end
|
|
66
|
+
x.compare!
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# Allocation pressure per single handoff
|
|
70
|
+
value = build.call
|
|
71
|
+
frozen_value = Ractor.make_shareable(build.call)
|
|
72
|
+
{":shared" => [Shifty::Policy::Shared, value],
|
|
73
|
+
":frozen (steady)" => [Shifty::Policy::Frozen, frozen_value],
|
|
74
|
+
":isolated" => [Shifty::Policy::Isolated, value]}.each do |name, (policy, v)|
|
|
75
|
+
GC.start
|
|
76
|
+
before = GC.stat(:total_allocated_objects)
|
|
77
|
+
100.times { policy.call(v, worker: null_worker) }
|
|
78
|
+
allocated = GC.stat(:total_allocated_objects) - before
|
|
79
|
+
puts format("allocations per handoff %-20s %8.1f", name, allocated / 100.0)
|
|
80
|
+
end
|
|
81
|
+
puts
|
|
82
|
+
end
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Discovery Notes — Handoff Immutability Policies implementation planning
|
|
2
|
+
|
|
3
|
+
## Tech stack
|
|
4
|
+
- Ruby gem "shifty", current version 0.5.0 (`lib/shifty/version.rb`). No `required_ruby_version` in gemspec yet (decision: add `>= 3.2`).
|
|
5
|
+
- Local Ruby: 4.0.5 (arm64-darwin). Runtime deps: none (stdlib only; `ostruct`, `forwardable`, `fiber`).
|
|
6
|
+
- Tests: RSpec 3.x + rspec-given (Given/When/Then style), simplecov; specs in `spec/shifty/{worker,gang,roster,dsl}_spec.rb`; `.rspec_status` persistence; monkey-patching disabled.
|
|
7
|
+
- Lint: standardrb (`.standard.yml` ignores Layout/ExtraSpacing). Rakefile standard bundler/gem_tasks (build via `rake build` / `gem build`).
|
|
8
|
+
- CI: `.github/` exists (workflows not yet examined by specialists — check if Ruby matrix needs updating for the 3.2 floor).
|
|
9
|
+
|
|
10
|
+
## Source spec
|
|
11
|
+
- `docs/planning/handoff-immutability-policies.md` — full design doc, 13 sections + appendix. Treat as ground truth for WHAT. No plan-a-feature artifacts (no decision-log/team-findings/feature-technical-notes).
|
|
12
|
+
|
|
13
|
+
## Code touch points
|
|
14
|
+
- `lib/shifty/worker.rb` — Worker: `initialize(p = {}, &block)` takes `:supply, :task, :context, :criteria, :tags`. Fiber-based `workflow` loop at line ~62: `Fiber.yield @task.call(value, supply, @context)` — THE handoff site for task output; `value = supply&.shift` is the intake site. Has `supply=` writer (no `task=` writer despite spec §5.4 mentioning one — only `@task ||= default_task` lazily). Extensive concurrency-model comment block (lines 90–130) added by PR #26; forward-references this plan and must be updated from "planned" to shipped.
|
|
15
|
+
- `lib/shifty/dsl.rb` — DSL worker constructors: `source_worker`, `relay_worker`, `side_worker` (has `mode: :hardened` → `Marshal.load(Marshal.dump(v))` at lines 40–41 — the feature to deprecate/subsume), `filter_worker`, `batch_worker` (BatchContext < OpenStruct), `splitter_worker` (mutates `parts` array internally via `parts.shift` — internal, not the handed-off value), `trailing_worker` (accumulates `trail` array in closure AND hands off `trail` itself — a live aliasing case under `:frozen`; hands off the same array it keeps mutating via unshift/pop!). `handoff` = `Fiber.yield`.
|
|
16
|
+
- `lib/shifty/gang.rb` — Gang wraps Roster; `initialize(workers = [], p = {})`; `shift` delegates to `roster.last.shift`; `Gang[*workers]` constructor; `|`/`supplies` composition same as Worker.
|
|
17
|
+
- `lib/shifty/roster.rb` — push/pop/shift/unshift rewires `supply` — mutation of topology, relevant to §5.4 `#freeze!`.
|
|
18
|
+
- `lib/shifty/taggable.rb` — tags/criteria mixin shared by Worker and Gang.
|
|
19
|
+
- `lib/shifty.rb` — requires all; no `Shifty.configure` exists yet (global config is new).
|
|
20
|
+
- No `Shifty::Testing`, no RSpec helpers shipped by the gem today.
|
|
21
|
+
|
|
22
|
+
## Important behavioral details for policy implementation
|
|
23
|
+
- Composition (`|`) links workers by setting `supply`; there is NO chain/pipeline object for plain worker chains — a "pipeline" is just the last worker; Gang is the only reifying container. `.with_policy` on a composed chain must therefore live on Worker (and Gang) and propagate upstream via the supply chain.
|
|
24
|
+
- `filter_worker` and `batch_worker` tasks call `supply.shift` themselves mid-task — values obtained that way ALSO cross a boundary (policy must apply at intake, not just at task-output yield, or these secondary intakes leak).
|
|
25
|
+
- `splitter_worker` and `source_worker` use `handoff` (Fiber.yield) directly from inside the task — multiple handoffs per task call; policy application at the single `Fiber.yield @task.call(...)` site won't see these. Handoff site analysis is a core design task.
|
|
26
|
+
- `trailing_worker` hands off its live closure array — will break under `:frozen` default; the spec (§7.3) says this is exactly the bug class the policy should catch; the shipped worker itself must be fixed (hand off a snapshot).
|
|
27
|
+
- Worker's nil sentinel: nil marks end-of-stream; `relay_worker`/`side_worker` guard `value &&`. nil is already frozen/shareable — no policy cost.
|
|
28
|
+
- `@context` (OpenStruct) is closure-ish per-worker state passed to tasks — NOT a handoff; stays mutable per spec §12.
|
|
29
|
+
|
|
30
|
+
## ADRs / coding standards
|
|
31
|
+
- None found (no docs/adr, no coding standards docs). Style = standardrb defaults. Doc precedent: `docs/use_cases.md`, `docs/shifty/worker.md`, README.md (rewritten by PR #26 with "Concurrency Model" section).
|
|
32
|
+
|
|
33
|
+
## Recent activity (90 days)
|
|
34
|
+
- 1dc5433 planning doc (this feature); 847ee06/d14823a PR #26 concurrency docs (README.md, docs/use_cases.md, lib/shifty/worker.rb comments); 43d5948 concurrency analysis; 5e8ca16 arity fix in dsl.rb.
|
|
35
|
+
|
|
36
|
+
## Gaps searched and not found
|
|
37
|
+
- No CLAUDE.md/AGENTS.md in repo. No CI examination yet. No benchmark tooling (no benchmark-ips in Gemfile). No existing implementation-plan format precedent in this repo.
|
|
38
|
+
|
|
39
|
+
## User-committed decisions (from Joel, this session)
|
|
40
|
+
1. Version 0.6.0, all four spec phases ship in it. 2. required_ruby_version >= 3.2. 3. API: worker `policy:` kwarg; chainable `.with_policy(:p)` on Worker AND Gang; `policy:` kwarg on Gang; `Shifty.configure { |c| c.default_policy = ... }`; built-in default `:frozen`; precedence worker > pipeline > global. 4. One PR per phase. 5. Wiki content drafted under `docs/wiki/`, pushed to GitHub wiki after merge. 6. `:hardened` deprecated → mapped to `:isolated` with warning. 7. §8.3 option (b) — no shareable? fast path initially; worker declarations authoritative, pipeline policy default-only. 8. Joel publishes the gem himself (2FA); plan ends at built gem + release prep.
|