fiber_stream 0.5.0 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 504a7400182e09bb66a5a07981bbb2cfab350397d09c23ea036e856f29afe1d2
4
- data.tar.gz: 67c7ddead203d42bf869dcba7d840e7d09514d888b90b84b77d97a8cedc19e1e
3
+ metadata.gz: b729811e8b39efdfa4203d0e6167709105d524ec098f018fdc120986c0b73b14
4
+ data.tar.gz: 162a3bf8a893fe13ae994ec77c8a2df7c8c44f9c5ab11895ee3b3b0ddfe31192
5
5
  SHA512:
6
- metadata.gz: f9c8d80faca53d9c6e059326bc9e6ffe7497aa8a6cbcad3b42eefe0ef3a3d8d29f08a3afbec8fc2b02b2064a745bb00ea644e816f913aa624a2ad11f1b087b99
7
- data.tar.gz: 9e6f87b74cbfe2ccaaa9c32990f9786af3ee3ee3d7f2c3dd605a88efa8f3e436e4a1051d65c1eb1664859af212a2fecc82c66dc585cf9333dd799917c2a9035e
6
+ metadata.gz: edd97c825b99250b4169443764cb384330fd8c3176e4086c6a1a93834c67fd288b5f0583fce50b68277936de1fe9f303162063ff182523beed3af935b568d2e0
7
+ data.tar.gz: dcb33221af26887b3a914f428c0b58e6d8ecf8511c384c0282b889e8b74b1d17fecd6e5ba86b465dfc466252bef0bb960f9bcfd66229c9e6ccd0b12cf7b3a768
data/CHANGELOG.md CHANGED
@@ -1,5 +1,28 @@
1
1
  # Changelog
2
2
 
3
+ ## Unreleased
4
+
5
+ ## 0.6.0 - 2026-07-27
6
+
7
+ ### Added
8
+
9
+ - `Flow.ractor_unordered_map(workers:)` and `Source#ractor_unordered_map` for
10
+ Ractor-backed CPU-bound mapping that emits results in completion order.
11
+ - `Sink.find { |element| ... }` for predicate-based terminal search with
12
+ early upstream completion.
13
+ - `Sink.any? { |element| ... }` for predicate-based existential checks with
14
+ early upstream completion.
15
+ - `Sink.all? { |element| ... }` for predicate-based universal checks with
16
+ early upstream completion.
17
+ - Example and benchmark coverage for unordered Ractor-backed mapping.
18
+
19
+ ### Changed
20
+
21
+ - Added a rate-limiting tutorial covering local, shared in-process,
22
+ database-backed, and Redis-backed quota policies.
23
+ - Centralized the local and release verification gates and added documentation
24
+ index validation.
25
+
3
26
  ## 0.5.0 - 2026-06-21
4
27
 
5
28
  ### Added
data/README.md CHANGED
@@ -37,8 +37,9 @@ Implemented capabilities:
37
37
  observation, one-to-many expansion, limiting, predicate-based limiting and
38
38
  dropping, fixed-prefix dropping, fixed-size grouping, line splitting,
39
39
  buffering, async boundaries, throttling, ordered and unordered parallel
40
- mapping, and ordered Ractor-backed mapping
41
- - array, first-element, count, fold, foreach, and IO sinks
40
+ mapping, and ordered and unordered Ractor-backed mapping
41
+ - array, first-element, predicate search/check, count, fold, foreach, and IO
42
+ sinks
42
43
  - reusable flow composition and runnable pipelines
43
44
  - foreground and scheduler-backed background pipeline execution
44
45
  - public RBS signatures
@@ -309,8 +310,10 @@ responses =
309
310
  responses
310
311
  ```
311
312
 
312
- Use `ractor_map` for ordered CPU-bound mapping in Ractor workers. The mapper
313
- must be shareable, usually by creating it with `Ractor.shareable_proc`.
313
+ Use `ractor_map` for ordered CPU-bound mapping in Ractor workers, and
314
+ `ractor_unordered_map` when later completed CPU-bound results should not wait
315
+ behind a slower earlier input. The mapper must be shareable, usually by
316
+ creating it with `Ractor.shareable_proc`.
314
317
 
315
318
  ```ruby
316
319
  require "digest"
@@ -343,6 +346,19 @@ digests =
343
346
  or `output_transfer: :move` only when the moved object will not be reused by
344
347
  the sender.
345
348
 
349
+ `ractor_unordered_map` uses the same Ractor transfer and shareability rules,
350
+ but emits values in worker completion order:
351
+
352
+ ```ruby
353
+ fastest_first =
354
+ FiberStream::Source.each(records)
355
+ .ractor_unordered_map(workers: 2, input_transfer: :move, &HASH_RECORD)
356
+ .run_with(FiberStream::Sink.to_a)
357
+
358
+ # Results are in completion order, not necessarily input order.
359
+ fastest_first
360
+ ```
361
+
346
362
  ### Sinks
347
363
 
348
364
  A `Sink` consumes the stream and returns a materialized value.
@@ -361,6 +377,30 @@ FiberStream::Source.each([1, 2, 3])
361
377
  # => 3
362
378
  ```
363
379
 
380
+ Use `Sink.find` when the stream should stop after the first matching element:
381
+
382
+ ```ruby
383
+ FiberStream::Source.each([1, 2, 3, 4])
384
+ .run_with(FiberStream::Sink.find(&:even?))
385
+ # => 2
386
+ ```
387
+
388
+ Use `Sink.any?` when only the existence of a matching element matters:
389
+
390
+ ```ruby
391
+ FiberStream::Source.each([1, 2, 3, 4])
392
+ .run_with(FiberStream::Sink.any?(&:even?))
393
+ # => true
394
+ ```
395
+
396
+ Use `Sink.all?` when every element must satisfy a predicate:
397
+
398
+ ```ruby
399
+ FiberStream::Source.each([2, 4, 6])
400
+ .run_with(FiberStream::Sink.all?(&:even?))
401
+ # => true
402
+ ```
403
+
364
404
  Use `Sink.foreach` when the terminal operation is a side effect and the stream
365
405
  values should not be accumulated:
366
406
 
@@ -575,6 +615,7 @@ Source convenience methods:
575
615
  - `Source#parallel_map(concurrency:) { |element| ... }`
576
616
  - `Source#parallel_unordered_map(concurrency:) { |element| ... }`
577
617
  - `Source#ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
618
+ - `Source#ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
578
619
  - `Source#select { |element| ... }`
579
620
  - `Source#reject { |element| ... }`
580
621
  - `Source#take(count)`
@@ -602,6 +643,7 @@ Flows:
602
643
  - `FiberStream::Flow.parallel_map(concurrency:) { |element| ... }`
603
644
  - `FiberStream::Flow.parallel_unordered_map(concurrency:) { |element| ... }`
604
645
  - `FiberStream::Flow.ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
646
+ - `FiberStream::Flow.ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
605
647
  - `FiberStream::Flow.select { |element| ... }`
606
648
  - `FiberStream::Flow.reject { |element| ... }`
607
649
  - `FiberStream::Flow.take(count)`
@@ -629,6 +671,9 @@ Sinks:
629
671
 
630
672
  - `FiberStream::Sink.to_a`
631
673
  - `FiberStream::Sink.first`
674
+ - `FiberStream::Sink.find { |element| truthy_or_falsey }`
675
+ - `FiberStream::Sink.any? { |element| truthy_or_falsey }`
676
+ - `FiberStream::Sink.all? { |element| truthy_or_falsey }`
632
677
  - `FiberStream::Sink.count`
633
678
  - `FiberStream::Sink.fold(initial) { |accumulator, element| ... }`
634
679
  - `FiberStream::Sink.foreach { |element| ... }`
@@ -655,6 +700,7 @@ bundle exec ruby examples/file_copy.rb
655
700
  bundle exec ruby examples/backpressure_buffer.rb
656
701
  bundle exec ruby examples/background_execution.rb
657
702
  bundle exec ruby examples/ractor_map_hashing.rb
703
+ bundle exec ruby examples/ractor_unordered_map_hashing.rb
658
704
  bundle exec ruby examples/ractor_producer_sources.rb
659
705
  bundle exec ruby examples/ractor_port_source.rb
660
706
  bundle exec ruby examples/ractor_merge_ports_and_map.rb
@@ -668,6 +714,9 @@ events so the difference between direct demand and bounded prefetch is visible.
668
714
  `examples/ractor_map_hashing.rb` demonstrates ordered Ractor-backed hashing
669
715
  with a shareable mapper proc and `input_transfer: :move`.
670
716
 
717
+ `examples/ractor_unordered_map_hashing.rb` demonstrates completion-order
718
+ Ractor-backed hashing for uneven CPU-bound work.
719
+
671
720
  `examples/ractor_producer_sources.rb` demonstrates high-level owned producer
672
721
  Ractors with `Source.ractor_producer` and `Source.ractor_merge_producers`.
673
722
 
@@ -705,6 +754,14 @@ Install dependencies:
705
754
  bundle install
706
755
  ```
707
756
 
757
+ Install website dependencies when building documentation locally:
758
+
759
+ ```sh
760
+ cd website
761
+ npm ci
762
+ cd ..
763
+ ```
764
+
708
765
  Run the test suite:
709
766
 
710
767
  ```sh
@@ -729,6 +786,18 @@ Run all default checks:
729
786
  bundle exec rake
730
787
  ```
731
788
 
789
+ Run the default local verification gate explicitly:
790
+
791
+ ```sh
792
+ bundle exec rake verify
793
+ ```
794
+
795
+ Run the full release-readiness gate, including the website build:
796
+
797
+ ```sh
798
+ bundle exec rake verify:full
799
+ ```
800
+
732
801
  Build the gem:
733
802
 
734
803
  ```sh
data/examples/README.md CHANGED
@@ -10,6 +10,7 @@ bundle exec ruby examples/file_copy.rb
10
10
  bundle exec ruby examples/backpressure_buffer.rb
11
11
  bundle exec ruby examples/background_execution.rb
12
12
  bundle exec ruby examples/ractor_map_hashing.rb
13
+ bundle exec ruby examples/ractor_unordered_map_hashing.rb
13
14
  bundle exec ruby examples/ractor_port_source.rb
14
15
  bundle exec ruby examples/ractor_producer_sources.rb
15
16
  bundle exec ruby examples/ractor_merge_ports_and_map.rb
@@ -44,6 +45,10 @@ the foreground fiber keeps doing scheduler-managed work.
44
45
  `input_transfer: :move` because the input records are not reused after the
45
46
  pipeline runs.
46
47
 
48
+ `ractor_unordered_map_hashing.rb` hashes independent payloads in Ractor workers
49
+ and emits results as workers finish. It shows how a slower earlier input no
50
+ longer holds back later completed CPU-bound work.
51
+
47
52
  `ractor_port_source.rb` demonstrates a producer Ractor connected to
48
53
  `Source.ractor_port`. The producer creates its acknowledgment port, waits for
49
54
  `RactorPort::Ack`, and sends one typed `RactorPort::Element` per downstream
@@ -0,0 +1,62 @@
1
+ # frozen_string_literal: true
2
+
3
+ $LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
4
+
5
+ require "digest"
6
+ require "fiber_stream"
7
+
8
+ records = [
9
+ { name: "slow-alpha.bin", payload: +"A" * 160_000, rounds: 1_200 },
10
+ { name: "fast-bravo.bin", payload: +"B" * 120_000, rounds: 120 },
11
+ { name: "fast-charlie.bin", payload: +"C" * 140_000, rounds: 120 },
12
+ { name: "medium-delta.bin", payload: +"D" * 100_000, rounds: 450 }
13
+ ]
14
+
15
+ HASH_RECORD =
16
+ Ractor.shareable_proc do |record|
17
+ payload = record.fetch(:payload)
18
+ digest = Digest::SHA256.hexdigest(payload)
19
+
20
+ record.fetch(:rounds).times do
21
+ digest = Digest::SHA256.hexdigest(digest)
22
+ end
23
+
24
+ {
25
+ name: record.fetch(:name),
26
+ bytes: payload.bytesize,
27
+ rounds: record.fetch(:rounds),
28
+ sha256: digest
29
+ }
30
+ end
31
+
32
+ puts "Input order"
33
+ records.each.with_index(1) do |record, index|
34
+ puts format(
35
+ "%<index>2d. %-16<name>s %4<rounds>d rounds",
36
+ index: index,
37
+ name: record.fetch(:name),
38
+ rounds: record.fetch(:rounds)
39
+ )
40
+ end
41
+
42
+ digests =
43
+ FiberStream::Source.each(records)
44
+ .ractor_unordered_map(workers: 2, input_transfer: :move, &HASH_RECORD)
45
+ .run_with(FiberStream::Sink.to_a)
46
+
47
+ puts
48
+ puts "Completion order"
49
+ digests.each.with_index(1) do |digest, index|
50
+ puts format(
51
+ "%<index>2d. %-16<name>s %7<bytes>d bytes %4<rounds>d rounds %<sha256>s",
52
+ index: index,
53
+ name: digest.fetch(:name),
54
+ bytes: digest.fetch(:bytes),
55
+ rounds: digest.fetch(:rounds),
56
+ sha256: digest.fetch(:sha256)
57
+ )
58
+ end
59
+
60
+ puts
61
+ puts "Results are emitted as Ractor workers finish, not by input position."
62
+ puts "input_transfer: :move is safe here because the input records are not reused."
@@ -111,6 +111,26 @@ module FiberStream
111
111
  new { |upstream| Pull.ractor_map(upstream, workers, input_transfer, output_transfer, block) }
112
112
  end
113
113
 
114
+ # Creates an unordered Ractor-backed mapping flow.
115
+ #
116
+ # The mapper runs inside worker ractors and must be shareable, typically
117
+ # created with `Ractor.shareable_proc`. Results are emitted in worker
118
+ # completion order, and at most `workers` upstream elements are pulled but
119
+ # not yet emitted. `input_transfer` and `output_transfer` must be `:copy`
120
+ # or `:move` and are passed to Ractor message sends for element and result
121
+ # transfer.
122
+ def self.ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy, &block)
123
+ raise ArgumentError, "missing block" unless block
124
+ raise TypeError, "workers must be an Integer" unless workers.is_a?(Integer)
125
+ raise ArgumentError, "workers must be positive" unless workers.positive?
126
+
127
+ Internal::RactorTransferPolicy.validate!(:input_transfer, input_transfer)
128
+ Internal::RactorTransferPolicy.validate!(:output_transfer, output_transfer)
129
+ raise TypeError, "block must be shareable" unless Ractor.shareable?(block)
130
+
131
+ new { |upstream| Pull.ractor_unordered_map(upstream, workers, input_transfer, output_transfer, block) }
132
+ end
133
+
114
134
  # Creates a filtering flow.
115
135
  #
116
136
  # The block is called for upstream elements until it returns a truthy value
@@ -0,0 +1,521 @@
1
+ # frozen_string_literal: true
2
+
3
+ module FiberStream
4
+ module Pull
5
+ # Unordered Ractor-backed worker boundary for
6
+ # `Flow.ractor_unordered_map`.
7
+ #
8
+ # Upstream is pulled by the downstream caller, while blocking waits for
9
+ # Ractor worker messages are isolated in a coordinator thread. Downstream
10
+ # emits worker results in completion order and admission stays bounded by
11
+ # the number of workers.
12
+ class RactorUnorderedMapBoundary
13
+ Job = ::Data.define(:sequence, :value)
14
+ Shutdown = ::Data.define
15
+ Ready = ::Data.define(:worker_id)
16
+ WorkerValue = ::Data.define(:worker_id, :sequence, :value)
17
+ WorkerFailure = ::Data.define(:worker_id, :sequence, :kind, :cause_class_name, :cause_message)
18
+ Stopped = ::Data.define(:worker_id)
19
+ ResultValue = ::Data.define(:sequence, :value)
20
+ ResultDone = ::Data.define
21
+ ResultCloseError = ::Data.define(:sequence, :error)
22
+ ResultError = ::Data.define(:sequence, :error)
23
+
24
+ private_constant :Job, :Shutdown, :Ready, :WorkerValue, :WorkerFailure, :Stopped
25
+ private_constant :ResultValue, :ResultDone, :ResultCloseError, :ResultError
26
+
27
+ def initialize(upstream, workers, input_transfer, output_transfer, transform)
28
+ @upstream = upstream
29
+ @workers_count = workers
30
+ @input_transfer = input_transfer
31
+ @output_transfer = output_transfer
32
+ @transform = transform
33
+ @result_port = nil
34
+ @ready_workers = Thread::SizedQueue.new(workers)
35
+ @results = Thread::SizedQueue.new(workers)
36
+ @workers = []
37
+ @active_sequences = {}
38
+ @worker_state_mutex = Mutex.new
39
+ @coordinator = nil
40
+ @next_sequence = 0
41
+ @outstanding_jobs = 0
42
+ @terminal_message = nil
43
+ @started = false
44
+ @closed = false
45
+ @done = false
46
+ @admission_closed = false
47
+ @worker_shutdown_sent = false
48
+ @upstream_closed = false
49
+ @upstream_close_error = nil
50
+ end
51
+
52
+ def next
53
+ return DONE if @closed || @done
54
+
55
+ start
56
+ next_message
57
+ end
58
+
59
+ def close
60
+ return if @closed
61
+
62
+ @closed = true
63
+ @done = true
64
+ close_error = close_upstream
65
+ close_admission(close_upstream: false)
66
+ close_ready_queue
67
+ close_result_queue
68
+ request_worker_shutdown
69
+ wait_for_workers
70
+ close_error ||= @upstream_close_error
71
+ raise close_error if close_error
72
+ end
73
+
74
+ private
75
+
76
+ def start
77
+ return if @started
78
+
79
+ @started = true
80
+ @result_port = Ractor::Port.new
81
+ @workers_count.times do |worker_id|
82
+ @workers << self.class.spawn_worker(
83
+ worker_id,
84
+ @result_port,
85
+ @transform,
86
+ @output_transfer
87
+ )
88
+ end
89
+ @coordinator = Thread.new { run_coordinator }
90
+ end
91
+
92
+ def next_message
93
+ return emit_terminal(@terminal_message) if terminal_ready?
94
+
95
+ ready = take_result(block: false)
96
+ return emit(ready) if ready
97
+
98
+ ready = fill_capacity
99
+ return emit(ready) if ready
100
+
101
+ return emit_terminal(@terminal_message) if terminal_ready?
102
+
103
+ message = take_result(block: true)
104
+ return complete unless message
105
+
106
+ emit(message)
107
+ end
108
+
109
+ def fill_capacity
110
+ return if @admission_closed
111
+
112
+ while @outstanding_jobs < @workers_count
113
+ ready = take_result(block: false)
114
+ return ready if ready
115
+
116
+ worker = take_ready_worker(block: @outstanding_jobs.zero? && @terminal_message.nil?)
117
+ break unless worker
118
+
119
+ message = pull_job_message
120
+ if message.is_a?(Job)
121
+ @outstanding_jobs += 1
122
+ break unless deliver_job(worker, message)
123
+ elsif upstream_failure?(message)
124
+ fail_with_error(message.sequence, message.error)
125
+ else
126
+ close_admission(close_upstream: false)
127
+ @terminal_message = message
128
+ break
129
+ end
130
+ end
131
+
132
+ nil
133
+ end
134
+
135
+ def pull_job_message
136
+ value = @upstream.next
137
+ return terminal_done_message if Pull.done?(value)
138
+
139
+ sequence = @next_sequence
140
+ @next_sequence += 1
141
+ Job.new(sequence, value)
142
+ rescue StandardError => error
143
+ close_upstream(record_error: false)
144
+ ResultError.new(sequence: @next_sequence, error:)
145
+ end
146
+
147
+ def terminal_done_message
148
+ close_error = close_upstream
149
+ if close_error
150
+ ResultCloseError.new(sequence: @next_sequence, error: close_error)
151
+ else
152
+ ResultDone.new
153
+ end
154
+ end
155
+
156
+ def deliver_job(worker, message)
157
+ sequence = message.sequence
158
+ track_worker_job(worker, sequence)
159
+
160
+ if @input_transfer == :move
161
+ worker.send(message, move: true)
162
+ else
163
+ worker.send(message)
164
+ end
165
+ true
166
+ rescue StandardError => error
167
+ clear_worker_job(worker)
168
+ deliver_result(ResultError.new(sequence:, error: build_ractor_map_error(sequence, :input_transfer, error)))
169
+ close_admission
170
+ request_worker_shutdown
171
+ false
172
+ end
173
+
174
+ def take_ready_worker(block:)
175
+ if block
176
+ loop do
177
+ worker = @ready_workers.pop
178
+ return worker if worker || @closed || @admission_closed || @ready_workers.closed?
179
+ end
180
+ else
181
+ @ready_workers.pop(true)
182
+ end
183
+ rescue ThreadError, ClosedQueueError
184
+ nil
185
+ end
186
+
187
+ def take_result(block:)
188
+ block ? @results.pop : @results.pop(true)
189
+ rescue ThreadError, ClosedQueueError
190
+ nil
191
+ end
192
+
193
+ def emit(message)
194
+ case message
195
+ in ResultValue[_sequence, value]
196
+ emit_value(value)
197
+ in ResultError[sequence:, error:]
198
+ fail_with_error(sequence, error)
199
+ end
200
+ end
201
+
202
+ def emit_value(value)
203
+ @outstanding_jobs -= 1 if @outstanding_jobs.positive?
204
+ value
205
+ end
206
+
207
+ def terminal_ready?
208
+ @terminal_message && @outstanding_jobs.zero?
209
+ end
210
+
211
+ def emit_terminal(message)
212
+ case message
213
+ in ResultDone
214
+ complete
215
+ in ResultCloseError[sequence:, error:]
216
+ fail_with_error(sequence, error, close_admission: false)
217
+ in ResultError[sequence:, error:]
218
+ fail_with_error(sequence, error, close_admission: false)
219
+ end
220
+ end
221
+
222
+ def upstream_failure?(message)
223
+ message.is_a?(ResultError)
224
+ end
225
+
226
+ def fail_with_error(_sequence, error, close_admission: true)
227
+ @done = true
228
+ close_admission() if close_admission
229
+ close_result_queue
230
+ request_worker_shutdown
231
+ if error.is_a?(RactorMapError) && error.original_cause
232
+ raise error, cause: error.original_cause
233
+ end
234
+
235
+ raise error
236
+ end
237
+
238
+ def complete
239
+ @done = true
240
+ request_worker_shutdown
241
+ DONE
242
+ end
243
+
244
+ def run_coordinator
245
+ stopped = 0
246
+ live_workers = @workers.dup
247
+
248
+ until stopped == @workers_count
249
+ selected, message = select_worker_message(live_workers)
250
+ if selected == :worker_remote_error
251
+ stopped += 1
252
+ elsif selected == @result_port
253
+ stopped += handle_worker_message(message, live_workers)
254
+ else
255
+ live_workers.delete(selected)
256
+ handle_worker_termination(selected)
257
+ stopped += 1
258
+ end
259
+ end
260
+ ensure
261
+ close_ready_queue
262
+ close_result_queue if @closed
263
+ end
264
+
265
+ def select_worker_message(live_workers)
266
+ Ractor.select(@result_port, *live_workers)
267
+ rescue Ractor::RemoteError => error
268
+ worker = remote_error_worker(error, live_workers) || failed_worker_for_remote_error(live_workers)
269
+ live_workers.delete(worker) if worker
270
+ handle_worker_remote_error(worker, error)
271
+ [:worker_remote_error, nil]
272
+ end
273
+
274
+ def remote_error_worker(error, live_workers)
275
+ return unless error.respond_to?(:ractor)
276
+
277
+ worker = error.ractor
278
+ live_workers.include?(worker) ? worker : nil
279
+ end
280
+
281
+ def failed_worker_for_remote_error(live_workers)
282
+ @worker_state_mutex.synchronize do
283
+ live_workers
284
+ .select { |worker| @active_sequences.key?(worker) }
285
+ .min_by { |worker| @active_sequences.fetch(worker) }
286
+ end || live_workers.first
287
+ end
288
+
289
+ def handle_worker_remote_error(worker, error)
290
+ sequence = worker ? clear_worker_job(worker) : nil
291
+ sequence ||= @next_sequence
292
+ return if @closed || @worker_shutdown_sent
293
+
294
+ deliver_worker_termination_error(worker, sequence, cause: error)
295
+ end
296
+
297
+ def handle_worker_message(message, live_workers)
298
+ case message
299
+ in Ready[worker_id]
300
+ deliver_ready_worker(worker_id)
301
+ 0
302
+ in WorkerValue
303
+ handle_worker_value_message(message)
304
+ 0
305
+ in WorkerFailure
306
+ handle_worker_error_message(message)
307
+ 0
308
+ in Stopped
309
+ handle_worker_stopped_message(message, live_workers)
310
+ end
311
+ end
312
+
313
+ def handle_worker_value_message(message)
314
+ worker = worker_for_id(message.worker_id)
315
+
316
+ clear_worker_job(worker)
317
+ deliver_result(ResultValue.new(sequence: message.sequence, value: message.value))
318
+ end
319
+
320
+ def handle_worker_error_message(message)
321
+ worker = worker_for_id(message.worker_id)
322
+
323
+ clear_worker_job(worker)
324
+ deliver_result(normalize_worker_error_message(message))
325
+ end
326
+
327
+ def handle_worker_stopped_message(message, live_workers)
328
+ worker = worker_for_id(message.worker_id)
329
+ live_workers.delete(worker)
330
+ sequence = clear_worker_job(worker)
331
+ deliver_worker_termination_error(worker, sequence) if sequence && !@closed && !@worker_shutdown_sent
332
+ 1
333
+ end
334
+
335
+ def handle_worker_termination(worker)
336
+ sequence = clear_worker_job(worker) || @next_sequence
337
+ return if @closed || @worker_shutdown_sent
338
+
339
+ deliver_worker_termination_error(worker, sequence)
340
+ end
341
+
342
+ def deliver_worker_termination_error(worker, sequence, cause: nil)
343
+ close_ready_queue
344
+ error =
345
+ RactorMapError.new(
346
+ sequence: sequence,
347
+ kind: :worker_termination,
348
+ cause_class_name: cause&.class&.name || worker.class.name,
349
+ cause_message: cause&.message || "worker terminated without a lifecycle message",
350
+ cause: cause
351
+ )
352
+
353
+ deliver_result(ResultError.new(sequence:, error:))
354
+ end
355
+
356
+ def deliver_ready_worker(worker_id)
357
+ return if @closed
358
+
359
+ push_until_delivered_or_closed(@ready_workers, worker_for_id(worker_id))
360
+ end
361
+
362
+ def deliver_result(message)
363
+ return if @closed
364
+
365
+ push_until_delivered_or_closed(@results, message)
366
+ end
367
+
368
+ def push_until_delivered_or_closed(queue, message)
369
+ return if @closed
370
+
371
+ queue.push(message)
372
+ rescue ThreadError, ClosedQueueError
373
+ nil
374
+ end
375
+
376
+ def normalize_worker_error_message(message)
377
+ sequence = message.sequence
378
+ error =
379
+ RactorMapError.new(
380
+ sequence: sequence,
381
+ kind: message.kind,
382
+ cause_class_name: message.cause_class_name,
383
+ cause_message: message.cause_message
384
+ )
385
+
386
+ ResultError.new(sequence:, error:)
387
+ end
388
+
389
+ def worker_for_id(worker_id)
390
+ @workers.fetch(worker_id)
391
+ end
392
+
393
+ def track_worker_job(worker, sequence)
394
+ @worker_state_mutex.synchronize do
395
+ @active_sequences[worker] = sequence
396
+ end
397
+ end
398
+
399
+ def clear_worker_job(worker)
400
+ @worker_state_mutex.synchronize do
401
+ @active_sequences.delete(worker)
402
+ end
403
+ end
404
+
405
+ def close_admission(close_upstream: true)
406
+ return if @admission_closed
407
+
408
+ @admission_closed = true
409
+ close_upstream(record_error: false) if close_upstream
410
+ end
411
+
412
+ def request_worker_shutdown
413
+ return unless @started
414
+ return if @worker_shutdown_sent
415
+
416
+ @worker_shutdown_sent = true
417
+ @workers.each do |worker|
418
+ worker.send(Shutdown.new)
419
+ rescue StandardError
420
+ nil
421
+ end
422
+ end
423
+
424
+ def wait_for_workers
425
+ return unless @coordinator
426
+
427
+ @coordinator.join
428
+ end
429
+
430
+ def close_upstream(record_error: true)
431
+ return nil if @upstream_closed
432
+
433
+ @upstream_closed = true
434
+ @upstream.close
435
+ nil
436
+ rescue StandardError => error
437
+ @upstream_close_error ||= error if record_error
438
+ error
439
+ end
440
+
441
+ def close_ready_queue
442
+ @ready_workers.close
443
+ end
444
+
445
+ def close_result_queue
446
+ @results.close
447
+ end
448
+
449
+ def build_ractor_map_error(sequence, kind, error)
450
+ RactorMapError.new(
451
+ sequence: sequence,
452
+ kind: kind,
453
+ cause_class_name: error.class.name,
454
+ cause_message: error.message,
455
+ cause: error
456
+ )
457
+ end
458
+
459
+ class << self
460
+ def spawn_worker(worker_id, result_port, transform, output_transfer) # :nodoc:
461
+ Ractor.new(worker_id, result_port, transform, output_transfer) do |id, port, mapper, transfer|
462
+ current_sequence = nil
463
+ send_control =
464
+ lambda do |message|
465
+ port.send(message)
466
+ true
467
+ rescue Exception # rubocop:disable Lint/RescueException
468
+ false
469
+ end
470
+ send_failure =
471
+ lambda do |sequence, kind, error|
472
+ send_control.call(WorkerFailure.new(id, sequence, kind, error.class.name, error.message))
473
+ rescue Exception # rubocop:disable Lint/RescueException
474
+ false
475
+ end
476
+
477
+ begin
478
+ if send_control.call(Ready.new(id))
479
+ loop do
480
+ message = Ractor.receive
481
+ case message
482
+ in Shutdown
483
+ break
484
+ in Job[sequence, value]
485
+ current_sequence = sequence
486
+ else
487
+ raise TypeError, "invalid ractor_unordered_map worker message: #{message.class}"
488
+ end
489
+
490
+ begin
491
+ mapped_value = mapper.call(value)
492
+ rescue Exception => error # rubocop:disable Lint/RescueException
493
+ break unless send_failure.call(current_sequence, :worker, error)
494
+ else
495
+ begin
496
+ if transfer == :move
497
+ port.send(WorkerValue.new(id, current_sequence, mapped_value), move: true)
498
+ else
499
+ port.send(WorkerValue.new(id, current_sequence, mapped_value))
500
+ end
501
+ rescue Exception => error # rubocop:disable Lint/RescueException
502
+ break unless send_failure.call(current_sequence, :output_transfer, error)
503
+ end
504
+ end
505
+
506
+ current_sequence = nil
507
+ break unless send_control.call(Ready.new(id))
508
+ end
509
+ end
510
+ rescue Exception => error # rubocop:disable Lint/RescueException
511
+ sequence = current_sequence || -1
512
+ send_failure.call(sequence, :worker_termination, error)
513
+ ensure
514
+ send_control.call(Stopped.new(id))
515
+ end
516
+ end
517
+ end
518
+ end
519
+ end
520
+ end
521
+ end
@@ -113,6 +113,10 @@ module FiberStream
113
113
  RactorMapBoundary.new(upstream, workers, input_transfer, output_transfer, transform)
114
114
  end
115
115
 
116
+ def self.ractor_unordered_map(upstream, workers, input_transfer, output_transfer, transform)
117
+ RactorUnorderedMapBoundary.new(upstream, workers, input_transfer, output_transfer, transform)
118
+ end
119
+
116
120
  def self.select(upstream, predicate)
117
121
  Select.new(upstream, predicate)
118
122
  end
@@ -198,12 +202,14 @@ require_relative "pull/throttle"
198
202
  require_relative "pull/parallel_map_boundary"
199
203
  require_relative "pull/parallel_unordered_map_boundary"
200
204
  require_relative "pull/ractor_map_boundary"
205
+ require_relative "pull/ractor_unordered_map_boundary"
201
206
 
202
207
  module FiberStream
203
208
  module Pull
204
209
  private_constant :Each, :IOSource, :RactorPortSource, :RactorMergePortsSource, :RactorProducerSource, :Concat,
205
210
  :Zip, :Merge, :Map, :FilterMap, :Compact, :MapConcat, :Tap, :Select, :Reject, :Take, :Drop,
206
211
  :Grouped, :Scan, :TakeWhile, :DropWhile, :Lines, :Split, :AsyncBoundary, :BufferBoundary,
207
- :Throttle, :ParallelMapBoundary, :ParallelUnorderedMapBoundary, :RactorMapBoundary
212
+ :Throttle, :ParallelMapBoundary, :ParallelUnorderedMapBoundary, :RactorMapBoundary,
213
+ :RactorUnorderedMapBoundary
208
214
  end
209
215
  end
@@ -29,6 +29,58 @@ module FiberStream
29
29
  end
30
30
  end
31
31
 
32
+ # Creates a sink that returns the first element matching a predicate.
33
+ #
34
+ # The sink pulls upstream until the block returns a truthy value or
35
+ # upstream completes. It returns the original matching element, or `nil`
36
+ # when no element matches. Matching `nil` elements are returned as `nil`,
37
+ # following Ruby's `Enumerable#find` ambiguity.
38
+ def self.find(&block)
39
+ raise ArgumentError, "missing block" unless block
40
+
41
+ new do |stream|
42
+ loop do
43
+ value = stream.next
44
+ break nil if Pull.done?(value)
45
+ break value if block.call(value)
46
+ end
47
+ end
48
+ end
49
+
50
+ # Creates a sink that returns whether any element matches a predicate.
51
+ #
52
+ # The sink pulls upstream until the block returns a truthy value or upstream
53
+ # completes. It returns `true` after a truthy predicate result and `false`
54
+ # when no element matches.
55
+ def self.any?(&block)
56
+ raise ArgumentError, "missing block" unless block
57
+
58
+ new do |stream|
59
+ loop do
60
+ value = stream.next
61
+ break false if Pull.done?(value)
62
+ break true if block.call(value)
63
+ end
64
+ end
65
+ end
66
+
67
+ # Creates a sink that returns whether all elements match a predicate.
68
+ #
69
+ # The sink pulls upstream until the block returns false or nil, or upstream
70
+ # completes. It returns `false` after a falsey predicate result and `true`
71
+ # when every predicate result is truthy, including for empty upstream.
72
+ def self.all?(&block)
73
+ raise ArgumentError, "missing block" unless block
74
+
75
+ new do |stream|
76
+ loop do
77
+ value = stream.next
78
+ break true if Pull.done?(value)
79
+ break false unless block.call(value)
80
+ end
81
+ end
82
+ end
83
+
32
84
  # Creates a sink that counts all stream elements.
33
85
  #
34
86
  # The sink consumes upstream until normal completion and returns the number
@@ -246,6 +246,25 @@ module FiberStream
246
246
  )
247
247
  end
248
248
 
249
+ # Returns a new source definition that maps elements in Ractor workers and
250
+ # emits mapped values as workers complete.
251
+ #
252
+ # This is a convenience wrapper around
253
+ # `via(FiberStream::Flow.ractor_unordered_map(workers:) { ... })` and
254
+ # preserves the same shareable mapper requirement, unordered delivery,
255
+ # transfer policy, bounded upstream run-ahead, and cooperative worker
256
+ # shutdown behavior.
257
+ def ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy, &block)
258
+ via(
259
+ Flow.ractor_unordered_map(
260
+ workers: workers,
261
+ input_transfer: input_transfer,
262
+ output_transfer: output_transfer,
263
+ &block
264
+ )
265
+ )
266
+ end
267
+
249
268
  # Returns a new source definition that keeps elements matching `block`.
250
269
  #
251
270
  # This is a convenience wrapper around
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module FiberStream
4
- VERSION = "0.5.0"
4
+ VERSION = "0.6.0"
5
5
  end
data/sig/fiber_stream.rbs CHANGED
@@ -87,6 +87,7 @@ module FiberStream
87
87
  def parallel_map: [Out] (concurrency: Integer) { (Elem) -> Out } -> Source[Out]
88
88
  def parallel_unordered_map: [Out] (concurrency: Integer) { (Elem) -> Out } -> Source[Out]
89
89
  def ractor_map: [Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (Elem) -> Out } -> Source[Out]
90
+ def ractor_unordered_map: [Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (Elem) -> Out } -> Source[Out]
90
91
  def filter_map: [Out] () { (Elem) -> (Out | false | nil) } -> Source[Out]
91
92
  def compact: () -> Source[Elem]
92
93
  def map_concat: [Out] () { (Elem) -> Enumerable[Out] } -> Source[Out]
@@ -113,6 +114,7 @@ module FiberStream
113
114
  def self.parallel_map: [In, Out] (concurrency: Integer) { (In) -> Out } -> Flow[In, Out]
114
115
  def self.parallel_unordered_map: [In, Out] (concurrency: Integer) { (In) -> Out } -> Flow[In, Out]
115
116
  def self.ractor_map: [In, Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (In) -> Out } -> Flow[In, Out]
117
+ def self.ractor_unordered_map: [In, Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (In) -> Out } -> Flow[In, Out]
116
118
  def self.filter_map: [In, Out] () { (In) -> (Out | false | nil) } -> Flow[In, Out]
117
119
  def self.compact: [Elem] () -> Flow[Elem, Elem]
118
120
  def self.map_concat: [In, Out] () { (In) -> Enumerable[Out] } -> Flow[In, Out]
@@ -136,6 +138,9 @@ module FiberStream
136
138
  class Sink[In, Mat]
137
139
  def self.to_a: [Elem] () -> Sink[Elem, Array[Elem]]
138
140
  def self.first: [Elem] () -> Sink[Elem, Elem?]
141
+ def self.find: [Elem] () { (Elem) -> boolish } -> Sink[Elem, Elem?]
142
+ def self.any?: [Elem] () { (Elem) -> boolish } -> Sink[Elem, bool]
143
+ def self.all?: [Elem] () { (Elem) -> boolish } -> Sink[Elem, bool]
139
144
  def self.count: [Elem] () -> Sink[Elem, Integer]
140
145
  def self.fold: [Elem, Acc] (Acc initial) { (Acc, Elem) -> Acc } -> Sink[Elem, Acc]
141
146
  def self.foreach: [Elem] () { (Elem) -> void } -> Sink[Elem, Integer]
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fiber_stream
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.0
4
+ version: 0.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Dai Akatsuka
@@ -120,6 +120,7 @@ files:
120
120
  - examples/ractor_merge_ports_and_map.rb
121
121
  - examples/ractor_port_source.rb
122
122
  - examples/ractor_producer_sources.rb
123
+ - examples/ractor_unordered_map_hashing.rb
123
124
  - lib/fiber_stream.rb
124
125
  - lib/fiber_stream/errors.rb
125
126
  - lib/fiber_stream/flow.rb
@@ -146,6 +147,7 @@ files:
146
147
  - lib/fiber_stream/pull/ractor_merge_ports_source.rb
147
148
  - lib/fiber_stream/pull/ractor_port_source.rb
148
149
  - lib/fiber_stream/pull/ractor_producer_source.rb
150
+ - lib/fiber_stream/pull/ractor_unordered_map_boundary.rb
149
151
  - lib/fiber_stream/pull/reject.rb
150
152
  - lib/fiber_stream/pull/scan.rb
151
153
  - lib/fiber_stream/pull/select.rb
@@ -169,7 +171,7 @@ licenses:
169
171
  metadata:
170
172
  allowed_push_host: https://rubygems.org
171
173
  homepage_uri: https://github.com/dakatsuka/fiber_stream
172
- source_code_uri: https://github.com/dakatsuka/fiber_stream/tree/v0.5.0
174
+ source_code_uri: https://github.com/dakatsuka/fiber_stream/tree/v0.6.0
173
175
  changelog_uri: https://github.com/dakatsuka/fiber_stream/blob/main/CHANGELOG.md
174
176
  rubygems_mfa_required: 'true'
175
177
  rdoc_options: []