fiber_stream 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +23 -0
- data/README.md +73 -4
- data/examples/README.md +5 -0
- data/examples/ractor_unordered_map_hashing.rb +62 -0
- data/lib/fiber_stream/flow.rb +20 -0
- data/lib/fiber_stream/pull/ractor_unordered_map_boundary.rb +521 -0
- data/lib/fiber_stream/pull.rb +7 -1
- data/lib/fiber_stream/sink.rb +52 -0
- data/lib/fiber_stream/source.rb +19 -0
- data/lib/fiber_stream/version.rb +1 -1
- data/sig/fiber_stream.rbs +5 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b729811e8b39efdfa4203d0e6167709105d524ec098f018fdc120986c0b73b14
|
|
4
|
+
data.tar.gz: 162a3bf8a893fe13ae994ec77c8a2df7c8c44f9c5ab11895ee3b3b0ddfe31192
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: edd97c825b99250b4169443764cb384330fd8c3176e4086c6a1a93834c67fd288b5f0583fce50b68277936de1fe9f303162063ff182523beed3af935b568d2e0
|
|
7
|
+
data.tar.gz: dcb33221af26887b3a914f428c0b58e6d8ecf8511c384c0282b889e8b74b1d17fecd6e5ba86b465dfc466252bef0bb960f9bcfd66229c9e6ccd0b12cf7b3a768
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,28 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
## 0.6.0 - 2026-07-27
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- `Flow.ractor_unordered_map(workers:)` and `Source#ractor_unordered_map` for
|
|
10
|
+
Ractor-backed CPU-bound mapping that emits results in completion order.
|
|
11
|
+
- `Sink.find { |element| ... }` for predicate-based terminal search with
|
|
12
|
+
early upstream completion.
|
|
13
|
+
- `Sink.any? { |element| ... }` for predicate-based existential checks with
|
|
14
|
+
early upstream completion.
|
|
15
|
+
- `Sink.all? { |element| ... }` for predicate-based universal checks with
|
|
16
|
+
early upstream completion.
|
|
17
|
+
- Example and benchmark coverage for unordered Ractor-backed mapping.
|
|
18
|
+
|
|
19
|
+
### Changed
|
|
20
|
+
|
|
21
|
+
- Added a rate-limiting tutorial covering local, shared in-process,
|
|
22
|
+
database-backed, and Redis-backed quota policies.
|
|
23
|
+
- Centralized the local and release verification gates and added documentation
|
|
24
|
+
index validation.
|
|
25
|
+
|
|
3
26
|
## 0.5.0 - 2026-06-21
|
|
4
27
|
|
|
5
28
|
### Added
|
data/README.md
CHANGED
|
@@ -37,8 +37,9 @@ Implemented capabilities:
|
|
|
37
37
|
observation, one-to-many expansion, limiting, predicate-based limiting and
|
|
38
38
|
dropping, fixed-prefix dropping, fixed-size grouping, line splitting,
|
|
39
39
|
buffering, async boundaries, throttling, ordered and unordered parallel
|
|
40
|
-
mapping, and ordered Ractor-backed mapping
|
|
41
|
-
- array, first-element, count, fold, foreach, and IO
|
|
40
|
+
mapping, and ordered and unordered Ractor-backed mapping
|
|
41
|
+
- array, first-element, predicate search/check, count, fold, foreach, and IO
|
|
42
|
+
sinks
|
|
42
43
|
- reusable flow composition and runnable pipelines
|
|
43
44
|
- foreground and scheduler-backed background pipeline execution
|
|
44
45
|
- public RBS signatures
|
|
@@ -309,8 +310,10 @@ responses =
|
|
|
309
310
|
responses
|
|
310
311
|
```
|
|
311
312
|
|
|
312
|
-
Use `ractor_map` for ordered CPU-bound mapping in Ractor workers
|
|
313
|
-
|
|
313
|
+
Use `ractor_map` for ordered CPU-bound mapping in Ractor workers, and
|
|
314
|
+
`ractor_unordered_map` when later completed CPU-bound results should not wait
|
|
315
|
+
behind a slower earlier input. The mapper must be shareable, usually by
|
|
316
|
+
creating it with `Ractor.shareable_proc`.
|
|
314
317
|
|
|
315
318
|
```ruby
|
|
316
319
|
require "digest"
|
|
@@ -343,6 +346,19 @@ digests =
|
|
|
343
346
|
or `output_transfer: :move` only when the moved object will not be reused by
|
|
344
347
|
the sender.
|
|
345
348
|
|
|
349
|
+
`ractor_unordered_map` uses the same Ractor transfer and shareability rules,
|
|
350
|
+
but emits values in worker completion order:
|
|
351
|
+
|
|
352
|
+
```ruby
|
|
353
|
+
fastest_first =
|
|
354
|
+
FiberStream::Source.each(records)
|
|
355
|
+
.ractor_unordered_map(workers: 2, input_transfer: :move, &HASH_RECORD)
|
|
356
|
+
.run_with(FiberStream::Sink.to_a)
|
|
357
|
+
|
|
358
|
+
# Results are in completion order, not necessarily input order.
|
|
359
|
+
fastest_first
|
|
360
|
+
```
|
|
361
|
+
|
|
346
362
|
### Sinks
|
|
347
363
|
|
|
348
364
|
A `Sink` consumes the stream and returns a materialized value.
|
|
@@ -361,6 +377,30 @@ FiberStream::Source.each([1, 2, 3])
|
|
|
361
377
|
# => 3
|
|
362
378
|
```
|
|
363
379
|
|
|
380
|
+
Use `Sink.find` when the stream should stop after the first matching element:
|
|
381
|
+
|
|
382
|
+
```ruby
|
|
383
|
+
FiberStream::Source.each([1, 2, 3, 4])
|
|
384
|
+
.run_with(FiberStream::Sink.find(&:even?))
|
|
385
|
+
# => 2
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
Use `Sink.any?` when only the existence of a matching element matters:
|
|
389
|
+
|
|
390
|
+
```ruby
|
|
391
|
+
FiberStream::Source.each([1, 2, 3, 4])
|
|
392
|
+
.run_with(FiberStream::Sink.any?(&:even?))
|
|
393
|
+
# => true
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
Use `Sink.all?` when every element must satisfy a predicate:
|
|
397
|
+
|
|
398
|
+
```ruby
|
|
399
|
+
FiberStream::Source.each([2, 4, 6])
|
|
400
|
+
.run_with(FiberStream::Sink.all?(&:even?))
|
|
401
|
+
# => true
|
|
402
|
+
```
|
|
403
|
+
|
|
364
404
|
Use `Sink.foreach` when the terminal operation is a side effect and the stream
|
|
365
405
|
values should not be accumulated:
|
|
366
406
|
|
|
@@ -575,6 +615,7 @@ Source convenience methods:
|
|
|
575
615
|
- `Source#parallel_map(concurrency:) { |element| ... }`
|
|
576
616
|
- `Source#parallel_unordered_map(concurrency:) { |element| ... }`
|
|
577
617
|
- `Source#ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
|
|
618
|
+
- `Source#ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
|
|
578
619
|
- `Source#select { |element| ... }`
|
|
579
620
|
- `Source#reject { |element| ... }`
|
|
580
621
|
- `Source#take(count)`
|
|
@@ -602,6 +643,7 @@ Flows:
|
|
|
602
643
|
- `FiberStream::Flow.parallel_map(concurrency:) { |element| ... }`
|
|
603
644
|
- `FiberStream::Flow.parallel_unordered_map(concurrency:) { |element| ... }`
|
|
604
645
|
- `FiberStream::Flow.ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
|
|
646
|
+
- `FiberStream::Flow.ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
|
|
605
647
|
- `FiberStream::Flow.select { |element| ... }`
|
|
606
648
|
- `FiberStream::Flow.reject { |element| ... }`
|
|
607
649
|
- `FiberStream::Flow.take(count)`
|
|
@@ -629,6 +671,9 @@ Sinks:
|
|
|
629
671
|
|
|
630
672
|
- `FiberStream::Sink.to_a`
|
|
631
673
|
- `FiberStream::Sink.first`
|
|
674
|
+
- `FiberStream::Sink.find { |element| truthy_or_falsey }`
|
|
675
|
+
- `FiberStream::Sink.any? { |element| truthy_or_falsey }`
|
|
676
|
+
- `FiberStream::Sink.all? { |element| truthy_or_falsey }`
|
|
632
677
|
- `FiberStream::Sink.count`
|
|
633
678
|
- `FiberStream::Sink.fold(initial) { |accumulator, element| ... }`
|
|
634
679
|
- `FiberStream::Sink.foreach { |element| ... }`
|
|
@@ -655,6 +700,7 @@ bundle exec ruby examples/file_copy.rb
|
|
|
655
700
|
bundle exec ruby examples/backpressure_buffer.rb
|
|
656
701
|
bundle exec ruby examples/background_execution.rb
|
|
657
702
|
bundle exec ruby examples/ractor_map_hashing.rb
|
|
703
|
+
bundle exec ruby examples/ractor_unordered_map_hashing.rb
|
|
658
704
|
bundle exec ruby examples/ractor_producer_sources.rb
|
|
659
705
|
bundle exec ruby examples/ractor_port_source.rb
|
|
660
706
|
bundle exec ruby examples/ractor_merge_ports_and_map.rb
|
|
@@ -668,6 +714,9 @@ events so the difference between direct demand and bounded prefetch is visible.
|
|
|
668
714
|
`examples/ractor_map_hashing.rb` demonstrates ordered Ractor-backed hashing
|
|
669
715
|
with a shareable mapper proc and `input_transfer: :move`.
|
|
670
716
|
|
|
717
|
+
`examples/ractor_unordered_map_hashing.rb` demonstrates completion-order
|
|
718
|
+
Ractor-backed hashing for uneven CPU-bound work.
|
|
719
|
+
|
|
671
720
|
`examples/ractor_producer_sources.rb` demonstrates high-level owned producer
|
|
672
721
|
Ractors with `Source.ractor_producer` and `Source.ractor_merge_producers`.
|
|
673
722
|
|
|
@@ -705,6 +754,14 @@ Install dependencies:
|
|
|
705
754
|
bundle install
|
|
706
755
|
```
|
|
707
756
|
|
|
757
|
+
Install website dependencies when building documentation locally:
|
|
758
|
+
|
|
759
|
+
```sh
|
|
760
|
+
cd website
|
|
761
|
+
npm ci
|
|
762
|
+
cd ..
|
|
763
|
+
```
|
|
764
|
+
|
|
708
765
|
Run the test suite:
|
|
709
766
|
|
|
710
767
|
```sh
|
|
@@ -729,6 +786,18 @@ Run all default checks:
|
|
|
729
786
|
bundle exec rake
|
|
730
787
|
```
|
|
731
788
|
|
|
789
|
+
Run the default local verification gate explicitly:
|
|
790
|
+
|
|
791
|
+
```sh
|
|
792
|
+
bundle exec rake verify
|
|
793
|
+
```
|
|
794
|
+
|
|
795
|
+
Run the full release-readiness gate, including the website build:
|
|
796
|
+
|
|
797
|
+
```sh
|
|
798
|
+
bundle exec rake verify:full
|
|
799
|
+
```
|
|
800
|
+
|
|
732
801
|
Build the gem:
|
|
733
802
|
|
|
734
803
|
```sh
|
data/examples/README.md
CHANGED
|
@@ -10,6 +10,7 @@ bundle exec ruby examples/file_copy.rb
|
|
|
10
10
|
bundle exec ruby examples/backpressure_buffer.rb
|
|
11
11
|
bundle exec ruby examples/background_execution.rb
|
|
12
12
|
bundle exec ruby examples/ractor_map_hashing.rb
|
|
13
|
+
bundle exec ruby examples/ractor_unordered_map_hashing.rb
|
|
13
14
|
bundle exec ruby examples/ractor_port_source.rb
|
|
14
15
|
bundle exec ruby examples/ractor_producer_sources.rb
|
|
15
16
|
bundle exec ruby examples/ractor_merge_ports_and_map.rb
|
|
@@ -44,6 +45,10 @@ the foreground fiber keeps doing scheduler-managed work.
|
|
|
44
45
|
`input_transfer: :move` because the input records are not reused after the
|
|
45
46
|
pipeline runs.
|
|
46
47
|
|
|
48
|
+
`ractor_unordered_map_hashing.rb` hashes independent payloads in Ractor workers
|
|
49
|
+
and emits results as workers finish. It shows how a slower earlier input no
|
|
50
|
+
longer holds back later completed CPU-bound work.
|
|
51
|
+
|
|
47
52
|
`ractor_port_source.rb` demonstrates a producer Ractor connected to
|
|
48
53
|
`Source.ractor_port`. The producer creates its acknowledgment port, waits for
|
|
49
54
|
`RactorPort::Ack`, and sends one typed `RactorPort::Element` per downstream
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
|
|
4
|
+
|
|
5
|
+
require "digest"
|
|
6
|
+
require "fiber_stream"
|
|
7
|
+
|
|
8
|
+
records = [
|
|
9
|
+
{ name: "slow-alpha.bin", payload: +"A" * 160_000, rounds: 1_200 },
|
|
10
|
+
{ name: "fast-bravo.bin", payload: +"B" * 120_000, rounds: 120 },
|
|
11
|
+
{ name: "fast-charlie.bin", payload: +"C" * 140_000, rounds: 120 },
|
|
12
|
+
{ name: "medium-delta.bin", payload: +"D" * 100_000, rounds: 450 }
|
|
13
|
+
]
|
|
14
|
+
|
|
15
|
+
HASH_RECORD =
|
|
16
|
+
Ractor.shareable_proc do |record|
|
|
17
|
+
payload = record.fetch(:payload)
|
|
18
|
+
digest = Digest::SHA256.hexdigest(payload)
|
|
19
|
+
|
|
20
|
+
record.fetch(:rounds).times do
|
|
21
|
+
digest = Digest::SHA256.hexdigest(digest)
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
{
|
|
25
|
+
name: record.fetch(:name),
|
|
26
|
+
bytes: payload.bytesize,
|
|
27
|
+
rounds: record.fetch(:rounds),
|
|
28
|
+
sha256: digest
|
|
29
|
+
}
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
puts "Input order"
|
|
33
|
+
records.each.with_index(1) do |record, index|
|
|
34
|
+
puts format(
|
|
35
|
+
"%<index>2d. %-16<name>s %4<rounds>d rounds",
|
|
36
|
+
index: index,
|
|
37
|
+
name: record.fetch(:name),
|
|
38
|
+
rounds: record.fetch(:rounds)
|
|
39
|
+
)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
digests =
|
|
43
|
+
FiberStream::Source.each(records)
|
|
44
|
+
.ractor_unordered_map(workers: 2, input_transfer: :move, &HASH_RECORD)
|
|
45
|
+
.run_with(FiberStream::Sink.to_a)
|
|
46
|
+
|
|
47
|
+
puts
|
|
48
|
+
puts "Completion order"
|
|
49
|
+
digests.each.with_index(1) do |digest, index|
|
|
50
|
+
puts format(
|
|
51
|
+
"%<index>2d. %-16<name>s %7<bytes>d bytes %4<rounds>d rounds %<sha256>s",
|
|
52
|
+
index: index,
|
|
53
|
+
name: digest.fetch(:name),
|
|
54
|
+
bytes: digest.fetch(:bytes),
|
|
55
|
+
rounds: digest.fetch(:rounds),
|
|
56
|
+
sha256: digest.fetch(:sha256)
|
|
57
|
+
)
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
puts
|
|
61
|
+
puts "Results are emitted as Ractor workers finish, not by input position."
|
|
62
|
+
puts "input_transfer: :move is safe here because the input records are not reused."
|
data/lib/fiber_stream/flow.rb
CHANGED
|
@@ -111,6 +111,26 @@ module FiberStream
|
|
|
111
111
|
new { |upstream| Pull.ractor_map(upstream, workers, input_transfer, output_transfer, block) }
|
|
112
112
|
end
|
|
113
113
|
|
|
114
|
+
# Creates an unordered Ractor-backed mapping flow.
|
|
115
|
+
#
|
|
116
|
+
# The mapper runs inside worker ractors and must be shareable, typically
|
|
117
|
+
# created with `Ractor.shareable_proc`. Results are emitted in worker
|
|
118
|
+
# completion order, and at most `workers` upstream elements are pulled but
|
|
119
|
+
# not yet emitted. `input_transfer` and `output_transfer` must be `:copy`
|
|
120
|
+
# or `:move` and are passed to Ractor message sends for element and result
|
|
121
|
+
# transfer.
|
|
122
|
+
def self.ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy, &block)
|
|
123
|
+
raise ArgumentError, "missing block" unless block
|
|
124
|
+
raise TypeError, "workers must be an Integer" unless workers.is_a?(Integer)
|
|
125
|
+
raise ArgumentError, "workers must be positive" unless workers.positive?
|
|
126
|
+
|
|
127
|
+
Internal::RactorTransferPolicy.validate!(:input_transfer, input_transfer)
|
|
128
|
+
Internal::RactorTransferPolicy.validate!(:output_transfer, output_transfer)
|
|
129
|
+
raise TypeError, "block must be shareable" unless Ractor.shareable?(block)
|
|
130
|
+
|
|
131
|
+
new { |upstream| Pull.ractor_unordered_map(upstream, workers, input_transfer, output_transfer, block) }
|
|
132
|
+
end
|
|
133
|
+
|
|
114
134
|
# Creates a filtering flow.
|
|
115
135
|
#
|
|
116
136
|
# The block is called for upstream elements until it returns a truthy value
|
|
@@ -0,0 +1,521 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module FiberStream
|
|
4
|
+
module Pull
|
|
5
|
+
# Unordered Ractor-backed worker boundary for
|
|
6
|
+
# `Flow.ractor_unordered_map`.
|
|
7
|
+
#
|
|
8
|
+
# Upstream is pulled by the downstream caller, while blocking waits for
|
|
9
|
+
# Ractor worker messages are isolated in a coordinator thread. Downstream
|
|
10
|
+
# emits worker results in completion order and admission stays bounded by
|
|
11
|
+
# the number of workers.
|
|
12
|
+
class RactorUnorderedMapBoundary
|
|
13
|
+
Job = ::Data.define(:sequence, :value)
|
|
14
|
+
Shutdown = ::Data.define
|
|
15
|
+
Ready = ::Data.define(:worker_id)
|
|
16
|
+
WorkerValue = ::Data.define(:worker_id, :sequence, :value)
|
|
17
|
+
WorkerFailure = ::Data.define(:worker_id, :sequence, :kind, :cause_class_name, :cause_message)
|
|
18
|
+
Stopped = ::Data.define(:worker_id)
|
|
19
|
+
ResultValue = ::Data.define(:sequence, :value)
|
|
20
|
+
ResultDone = ::Data.define
|
|
21
|
+
ResultCloseError = ::Data.define(:sequence, :error)
|
|
22
|
+
ResultError = ::Data.define(:sequence, :error)
|
|
23
|
+
|
|
24
|
+
private_constant :Job, :Shutdown, :Ready, :WorkerValue, :WorkerFailure, :Stopped
|
|
25
|
+
private_constant :ResultValue, :ResultDone, :ResultCloseError, :ResultError
|
|
26
|
+
|
|
27
|
+
def initialize(upstream, workers, input_transfer, output_transfer, transform)
|
|
28
|
+
@upstream = upstream
|
|
29
|
+
@workers_count = workers
|
|
30
|
+
@input_transfer = input_transfer
|
|
31
|
+
@output_transfer = output_transfer
|
|
32
|
+
@transform = transform
|
|
33
|
+
@result_port = nil
|
|
34
|
+
@ready_workers = Thread::SizedQueue.new(workers)
|
|
35
|
+
@results = Thread::SizedQueue.new(workers)
|
|
36
|
+
@workers = []
|
|
37
|
+
@active_sequences = {}
|
|
38
|
+
@worker_state_mutex = Mutex.new
|
|
39
|
+
@coordinator = nil
|
|
40
|
+
@next_sequence = 0
|
|
41
|
+
@outstanding_jobs = 0
|
|
42
|
+
@terminal_message = nil
|
|
43
|
+
@started = false
|
|
44
|
+
@closed = false
|
|
45
|
+
@done = false
|
|
46
|
+
@admission_closed = false
|
|
47
|
+
@worker_shutdown_sent = false
|
|
48
|
+
@upstream_closed = false
|
|
49
|
+
@upstream_close_error = nil
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def next
|
|
53
|
+
return DONE if @closed || @done
|
|
54
|
+
|
|
55
|
+
start
|
|
56
|
+
next_message
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def close
|
|
60
|
+
return if @closed
|
|
61
|
+
|
|
62
|
+
@closed = true
|
|
63
|
+
@done = true
|
|
64
|
+
close_error = close_upstream
|
|
65
|
+
close_admission(close_upstream: false)
|
|
66
|
+
close_ready_queue
|
|
67
|
+
close_result_queue
|
|
68
|
+
request_worker_shutdown
|
|
69
|
+
wait_for_workers
|
|
70
|
+
close_error ||= @upstream_close_error
|
|
71
|
+
raise close_error if close_error
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
private
|
|
75
|
+
|
|
76
|
+
def start
|
|
77
|
+
return if @started
|
|
78
|
+
|
|
79
|
+
@started = true
|
|
80
|
+
@result_port = Ractor::Port.new
|
|
81
|
+
@workers_count.times do |worker_id|
|
|
82
|
+
@workers << self.class.spawn_worker(
|
|
83
|
+
worker_id,
|
|
84
|
+
@result_port,
|
|
85
|
+
@transform,
|
|
86
|
+
@output_transfer
|
|
87
|
+
)
|
|
88
|
+
end
|
|
89
|
+
@coordinator = Thread.new { run_coordinator }
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def next_message
|
|
93
|
+
return emit_terminal(@terminal_message) if terminal_ready?
|
|
94
|
+
|
|
95
|
+
ready = take_result(block: false)
|
|
96
|
+
return emit(ready) if ready
|
|
97
|
+
|
|
98
|
+
ready = fill_capacity
|
|
99
|
+
return emit(ready) if ready
|
|
100
|
+
|
|
101
|
+
return emit_terminal(@terminal_message) if terminal_ready?
|
|
102
|
+
|
|
103
|
+
message = take_result(block: true)
|
|
104
|
+
return complete unless message
|
|
105
|
+
|
|
106
|
+
emit(message)
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def fill_capacity
|
|
110
|
+
return if @admission_closed
|
|
111
|
+
|
|
112
|
+
while @outstanding_jobs < @workers_count
|
|
113
|
+
ready = take_result(block: false)
|
|
114
|
+
return ready if ready
|
|
115
|
+
|
|
116
|
+
worker = take_ready_worker(block: @outstanding_jobs.zero? && @terminal_message.nil?)
|
|
117
|
+
break unless worker
|
|
118
|
+
|
|
119
|
+
message = pull_job_message
|
|
120
|
+
if message.is_a?(Job)
|
|
121
|
+
@outstanding_jobs += 1
|
|
122
|
+
break unless deliver_job(worker, message)
|
|
123
|
+
elsif upstream_failure?(message)
|
|
124
|
+
fail_with_error(message.sequence, message.error)
|
|
125
|
+
else
|
|
126
|
+
close_admission(close_upstream: false)
|
|
127
|
+
@terminal_message = message
|
|
128
|
+
break
|
|
129
|
+
end
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
nil
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
def pull_job_message
|
|
136
|
+
value = @upstream.next
|
|
137
|
+
return terminal_done_message if Pull.done?(value)
|
|
138
|
+
|
|
139
|
+
sequence = @next_sequence
|
|
140
|
+
@next_sequence += 1
|
|
141
|
+
Job.new(sequence, value)
|
|
142
|
+
rescue StandardError => error
|
|
143
|
+
close_upstream(record_error: false)
|
|
144
|
+
ResultError.new(sequence: @next_sequence, error:)
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
def terminal_done_message
|
|
148
|
+
close_error = close_upstream
|
|
149
|
+
if close_error
|
|
150
|
+
ResultCloseError.new(sequence: @next_sequence, error: close_error)
|
|
151
|
+
else
|
|
152
|
+
ResultDone.new
|
|
153
|
+
end
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
def deliver_job(worker, message)
|
|
157
|
+
sequence = message.sequence
|
|
158
|
+
track_worker_job(worker, sequence)
|
|
159
|
+
|
|
160
|
+
if @input_transfer == :move
|
|
161
|
+
worker.send(message, move: true)
|
|
162
|
+
else
|
|
163
|
+
worker.send(message)
|
|
164
|
+
end
|
|
165
|
+
true
|
|
166
|
+
rescue StandardError => error
|
|
167
|
+
clear_worker_job(worker)
|
|
168
|
+
deliver_result(ResultError.new(sequence:, error: build_ractor_map_error(sequence, :input_transfer, error)))
|
|
169
|
+
close_admission
|
|
170
|
+
request_worker_shutdown
|
|
171
|
+
false
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
def take_ready_worker(block:)
|
|
175
|
+
if block
|
|
176
|
+
loop do
|
|
177
|
+
worker = @ready_workers.pop
|
|
178
|
+
return worker if worker || @closed || @admission_closed || @ready_workers.closed?
|
|
179
|
+
end
|
|
180
|
+
else
|
|
181
|
+
@ready_workers.pop(true)
|
|
182
|
+
end
|
|
183
|
+
rescue ThreadError, ClosedQueueError
|
|
184
|
+
nil
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
def take_result(block:)
|
|
188
|
+
block ? @results.pop : @results.pop(true)
|
|
189
|
+
rescue ThreadError, ClosedQueueError
|
|
190
|
+
nil
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
def emit(message)
|
|
194
|
+
case message
|
|
195
|
+
in ResultValue[_sequence, value]
|
|
196
|
+
emit_value(value)
|
|
197
|
+
in ResultError[sequence:, error:]
|
|
198
|
+
fail_with_error(sequence, error)
|
|
199
|
+
end
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def emit_value(value)
|
|
203
|
+
@outstanding_jobs -= 1 if @outstanding_jobs.positive?
|
|
204
|
+
value
|
|
205
|
+
end
|
|
206
|
+
|
|
207
|
+
def terminal_ready?
|
|
208
|
+
@terminal_message && @outstanding_jobs.zero?
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
def emit_terminal(message)
|
|
212
|
+
case message
|
|
213
|
+
in ResultDone
|
|
214
|
+
complete
|
|
215
|
+
in ResultCloseError[sequence:, error:]
|
|
216
|
+
fail_with_error(sequence, error, close_admission: false)
|
|
217
|
+
in ResultError[sequence:, error:]
|
|
218
|
+
fail_with_error(sequence, error, close_admission: false)
|
|
219
|
+
end
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def upstream_failure?(message)
|
|
223
|
+
message.is_a?(ResultError)
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
def fail_with_error(_sequence, error, close_admission: true)
|
|
227
|
+
@done = true
|
|
228
|
+
close_admission() if close_admission
|
|
229
|
+
close_result_queue
|
|
230
|
+
request_worker_shutdown
|
|
231
|
+
if error.is_a?(RactorMapError) && error.original_cause
|
|
232
|
+
raise error, cause: error.original_cause
|
|
233
|
+
end
|
|
234
|
+
|
|
235
|
+
raise error
|
|
236
|
+
end
|
|
237
|
+
|
|
238
|
+
def complete
|
|
239
|
+
@done = true
|
|
240
|
+
request_worker_shutdown
|
|
241
|
+
DONE
|
|
242
|
+
end
|
|
243
|
+
|
|
244
|
+
def run_coordinator
|
|
245
|
+
stopped = 0
|
|
246
|
+
live_workers = @workers.dup
|
|
247
|
+
|
|
248
|
+
until stopped == @workers_count
|
|
249
|
+
selected, message = select_worker_message(live_workers)
|
|
250
|
+
if selected == :worker_remote_error
|
|
251
|
+
stopped += 1
|
|
252
|
+
elsif selected == @result_port
|
|
253
|
+
stopped += handle_worker_message(message, live_workers)
|
|
254
|
+
else
|
|
255
|
+
live_workers.delete(selected)
|
|
256
|
+
handle_worker_termination(selected)
|
|
257
|
+
stopped += 1
|
|
258
|
+
end
|
|
259
|
+
end
|
|
260
|
+
ensure
|
|
261
|
+
close_ready_queue
|
|
262
|
+
close_result_queue if @closed
|
|
263
|
+
end
|
|
264
|
+
|
|
265
|
+
def select_worker_message(live_workers)
|
|
266
|
+
Ractor.select(@result_port, *live_workers)
|
|
267
|
+
rescue Ractor::RemoteError => error
|
|
268
|
+
worker = remote_error_worker(error, live_workers) || failed_worker_for_remote_error(live_workers)
|
|
269
|
+
live_workers.delete(worker) if worker
|
|
270
|
+
handle_worker_remote_error(worker, error)
|
|
271
|
+
[:worker_remote_error, nil]
|
|
272
|
+
end
|
|
273
|
+
|
|
274
|
+
def remote_error_worker(error, live_workers)
|
|
275
|
+
return unless error.respond_to?(:ractor)
|
|
276
|
+
|
|
277
|
+
worker = error.ractor
|
|
278
|
+
live_workers.include?(worker) ? worker : nil
|
|
279
|
+
end
|
|
280
|
+
|
|
281
|
+
def failed_worker_for_remote_error(live_workers)
|
|
282
|
+
@worker_state_mutex.synchronize do
|
|
283
|
+
live_workers
|
|
284
|
+
.select { |worker| @active_sequences.key?(worker) }
|
|
285
|
+
.min_by { |worker| @active_sequences.fetch(worker) }
|
|
286
|
+
end || live_workers.first
|
|
287
|
+
end
|
|
288
|
+
|
|
289
|
+
def handle_worker_remote_error(worker, error)
|
|
290
|
+
sequence = worker ? clear_worker_job(worker) : nil
|
|
291
|
+
sequence ||= @next_sequence
|
|
292
|
+
return if @closed || @worker_shutdown_sent
|
|
293
|
+
|
|
294
|
+
deliver_worker_termination_error(worker, sequence, cause: error)
|
|
295
|
+
end
|
|
296
|
+
|
|
297
|
+
def handle_worker_message(message, live_workers)
|
|
298
|
+
case message
|
|
299
|
+
in Ready[worker_id]
|
|
300
|
+
deliver_ready_worker(worker_id)
|
|
301
|
+
0
|
|
302
|
+
in WorkerValue
|
|
303
|
+
handle_worker_value_message(message)
|
|
304
|
+
0
|
|
305
|
+
in WorkerFailure
|
|
306
|
+
handle_worker_error_message(message)
|
|
307
|
+
0
|
|
308
|
+
in Stopped
|
|
309
|
+
handle_worker_stopped_message(message, live_workers)
|
|
310
|
+
end
|
|
311
|
+
end
|
|
312
|
+
|
|
313
|
+
def handle_worker_value_message(message)
|
|
314
|
+
worker = worker_for_id(message.worker_id)
|
|
315
|
+
|
|
316
|
+
clear_worker_job(worker)
|
|
317
|
+
deliver_result(ResultValue.new(sequence: message.sequence, value: message.value))
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
def handle_worker_error_message(message)
|
|
321
|
+
worker = worker_for_id(message.worker_id)
|
|
322
|
+
|
|
323
|
+
clear_worker_job(worker)
|
|
324
|
+
deliver_result(normalize_worker_error_message(message))
|
|
325
|
+
end
|
|
326
|
+
|
|
327
|
+
def handle_worker_stopped_message(message, live_workers)
|
|
328
|
+
worker = worker_for_id(message.worker_id)
|
|
329
|
+
live_workers.delete(worker)
|
|
330
|
+
sequence = clear_worker_job(worker)
|
|
331
|
+
deliver_worker_termination_error(worker, sequence) if sequence && !@closed && !@worker_shutdown_sent
|
|
332
|
+
1
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
def handle_worker_termination(worker)
|
|
336
|
+
sequence = clear_worker_job(worker) || @next_sequence
|
|
337
|
+
return if @closed || @worker_shutdown_sent
|
|
338
|
+
|
|
339
|
+
deliver_worker_termination_error(worker, sequence)
|
|
340
|
+
end
|
|
341
|
+
|
|
342
|
+
def deliver_worker_termination_error(worker, sequence, cause: nil)
|
|
343
|
+
close_ready_queue
|
|
344
|
+
error =
|
|
345
|
+
RactorMapError.new(
|
|
346
|
+
sequence: sequence,
|
|
347
|
+
kind: :worker_termination,
|
|
348
|
+
cause_class_name: cause&.class&.name || worker.class.name,
|
|
349
|
+
cause_message: cause&.message || "worker terminated without a lifecycle message",
|
|
350
|
+
cause: cause
|
|
351
|
+
)
|
|
352
|
+
|
|
353
|
+
deliver_result(ResultError.new(sequence:, error:))
|
|
354
|
+
end
|
|
355
|
+
|
|
356
|
+
def deliver_ready_worker(worker_id)
|
|
357
|
+
return if @closed
|
|
358
|
+
|
|
359
|
+
push_until_delivered_or_closed(@ready_workers, worker_for_id(worker_id))
|
|
360
|
+
end
|
|
361
|
+
|
|
362
|
+
def deliver_result(message)
|
|
363
|
+
return if @closed
|
|
364
|
+
|
|
365
|
+
push_until_delivered_or_closed(@results, message)
|
|
366
|
+
end
|
|
367
|
+
|
|
368
|
+
def push_until_delivered_or_closed(queue, message)
|
|
369
|
+
return if @closed
|
|
370
|
+
|
|
371
|
+
queue.push(message)
|
|
372
|
+
rescue ThreadError, ClosedQueueError
|
|
373
|
+
nil
|
|
374
|
+
end
|
|
375
|
+
|
|
376
|
+
def normalize_worker_error_message(message)
|
|
377
|
+
sequence = message.sequence
|
|
378
|
+
error =
|
|
379
|
+
RactorMapError.new(
|
|
380
|
+
sequence: sequence,
|
|
381
|
+
kind: message.kind,
|
|
382
|
+
cause_class_name: message.cause_class_name,
|
|
383
|
+
cause_message: message.cause_message
|
|
384
|
+
)
|
|
385
|
+
|
|
386
|
+
ResultError.new(sequence:, error:)
|
|
387
|
+
end
|
|
388
|
+
|
|
389
|
+
def worker_for_id(worker_id)
|
|
390
|
+
@workers.fetch(worker_id)
|
|
391
|
+
end
|
|
392
|
+
|
|
393
|
+
def track_worker_job(worker, sequence)
|
|
394
|
+
@worker_state_mutex.synchronize do
|
|
395
|
+
@active_sequences[worker] = sequence
|
|
396
|
+
end
|
|
397
|
+
end
|
|
398
|
+
|
|
399
|
+
def clear_worker_job(worker)
|
|
400
|
+
@worker_state_mutex.synchronize do
|
|
401
|
+
@active_sequences.delete(worker)
|
|
402
|
+
end
|
|
403
|
+
end
|
|
404
|
+
|
|
405
|
+
def close_admission(close_upstream: true)
|
|
406
|
+
return if @admission_closed
|
|
407
|
+
|
|
408
|
+
@admission_closed = true
|
|
409
|
+
close_upstream(record_error: false) if close_upstream
|
|
410
|
+
end
|
|
411
|
+
|
|
412
|
+
def request_worker_shutdown
|
|
413
|
+
return unless @started
|
|
414
|
+
return if @worker_shutdown_sent
|
|
415
|
+
|
|
416
|
+
@worker_shutdown_sent = true
|
|
417
|
+
@workers.each do |worker|
|
|
418
|
+
worker.send(Shutdown.new)
|
|
419
|
+
rescue StandardError
|
|
420
|
+
nil
|
|
421
|
+
end
|
|
422
|
+
end
|
|
423
|
+
|
|
424
|
+
def wait_for_workers
|
|
425
|
+
return unless @coordinator
|
|
426
|
+
|
|
427
|
+
@coordinator.join
|
|
428
|
+
end
|
|
429
|
+
|
|
430
|
+
def close_upstream(record_error: true)
|
|
431
|
+
return nil if @upstream_closed
|
|
432
|
+
|
|
433
|
+
@upstream_closed = true
|
|
434
|
+
@upstream.close
|
|
435
|
+
nil
|
|
436
|
+
rescue StandardError => error
|
|
437
|
+
@upstream_close_error ||= error if record_error
|
|
438
|
+
error
|
|
439
|
+
end
|
|
440
|
+
|
|
441
|
+
def close_ready_queue
|
|
442
|
+
@ready_workers.close
|
|
443
|
+
end
|
|
444
|
+
|
|
445
|
+
def close_result_queue
|
|
446
|
+
@results.close
|
|
447
|
+
end
|
|
448
|
+
|
|
449
|
+
def build_ractor_map_error(sequence, kind, error)
|
|
450
|
+
RactorMapError.new(
|
|
451
|
+
sequence: sequence,
|
|
452
|
+
kind: kind,
|
|
453
|
+
cause_class_name: error.class.name,
|
|
454
|
+
cause_message: error.message,
|
|
455
|
+
cause: error
|
|
456
|
+
)
|
|
457
|
+
end
|
|
458
|
+
|
|
459
|
+
class << self
|
|
460
|
+
def spawn_worker(worker_id, result_port, transform, output_transfer) # :nodoc:
|
|
461
|
+
Ractor.new(worker_id, result_port, transform, output_transfer) do |id, port, mapper, transfer|
|
|
462
|
+
current_sequence = nil
|
|
463
|
+
send_control =
|
|
464
|
+
lambda do |message|
|
|
465
|
+
port.send(message)
|
|
466
|
+
true
|
|
467
|
+
rescue Exception # rubocop:disable Lint/RescueException
|
|
468
|
+
false
|
|
469
|
+
end
|
|
470
|
+
send_failure =
|
|
471
|
+
lambda do |sequence, kind, error|
|
|
472
|
+
send_control.call(WorkerFailure.new(id, sequence, kind, error.class.name, error.message))
|
|
473
|
+
rescue Exception # rubocop:disable Lint/RescueException
|
|
474
|
+
false
|
|
475
|
+
end
|
|
476
|
+
|
|
477
|
+
begin
|
|
478
|
+
if send_control.call(Ready.new(id))
|
|
479
|
+
loop do
|
|
480
|
+
message = Ractor.receive
|
|
481
|
+
case message
|
|
482
|
+
in Shutdown
|
|
483
|
+
break
|
|
484
|
+
in Job[sequence, value]
|
|
485
|
+
current_sequence = sequence
|
|
486
|
+
else
|
|
487
|
+
raise TypeError, "invalid ractor_unordered_map worker message: #{message.class}"
|
|
488
|
+
end
|
|
489
|
+
|
|
490
|
+
begin
|
|
491
|
+
mapped_value = mapper.call(value)
|
|
492
|
+
rescue Exception => error # rubocop:disable Lint/RescueException
|
|
493
|
+
break unless send_failure.call(current_sequence, :worker, error)
|
|
494
|
+
else
|
|
495
|
+
begin
|
|
496
|
+
if transfer == :move
|
|
497
|
+
port.send(WorkerValue.new(id, current_sequence, mapped_value), move: true)
|
|
498
|
+
else
|
|
499
|
+
port.send(WorkerValue.new(id, current_sequence, mapped_value))
|
|
500
|
+
end
|
|
501
|
+
rescue Exception => error # rubocop:disable Lint/RescueException
|
|
502
|
+
break unless send_failure.call(current_sequence, :output_transfer, error)
|
|
503
|
+
end
|
|
504
|
+
end
|
|
505
|
+
|
|
506
|
+
current_sequence = nil
|
|
507
|
+
break unless send_control.call(Ready.new(id))
|
|
508
|
+
end
|
|
509
|
+
end
|
|
510
|
+
rescue Exception => error # rubocop:disable Lint/RescueException
|
|
511
|
+
sequence = current_sequence || -1
|
|
512
|
+
send_failure.call(sequence, :worker_termination, error)
|
|
513
|
+
ensure
|
|
514
|
+
send_control.call(Stopped.new(id))
|
|
515
|
+
end
|
|
516
|
+
end
|
|
517
|
+
end
|
|
518
|
+
end
|
|
519
|
+
end
|
|
520
|
+
end
|
|
521
|
+
end
|
data/lib/fiber_stream/pull.rb
CHANGED
|
@@ -113,6 +113,10 @@ module FiberStream
|
|
|
113
113
|
RactorMapBoundary.new(upstream, workers, input_transfer, output_transfer, transform)
|
|
114
114
|
end
|
|
115
115
|
|
|
116
|
+
def self.ractor_unordered_map(upstream, workers, input_transfer, output_transfer, transform)
|
|
117
|
+
RactorUnorderedMapBoundary.new(upstream, workers, input_transfer, output_transfer, transform)
|
|
118
|
+
end
|
|
119
|
+
|
|
116
120
|
def self.select(upstream, predicate)
|
|
117
121
|
Select.new(upstream, predicate)
|
|
118
122
|
end
|
|
@@ -198,12 +202,14 @@ require_relative "pull/throttle"
|
|
|
198
202
|
require_relative "pull/parallel_map_boundary"
|
|
199
203
|
require_relative "pull/parallel_unordered_map_boundary"
|
|
200
204
|
require_relative "pull/ractor_map_boundary"
|
|
205
|
+
require_relative "pull/ractor_unordered_map_boundary"
|
|
201
206
|
|
|
202
207
|
module FiberStream
|
|
203
208
|
module Pull
|
|
204
209
|
private_constant :Each, :IOSource, :RactorPortSource, :RactorMergePortsSource, :RactorProducerSource, :Concat,
|
|
205
210
|
:Zip, :Merge, :Map, :FilterMap, :Compact, :MapConcat, :Tap, :Select, :Reject, :Take, :Drop,
|
|
206
211
|
:Grouped, :Scan, :TakeWhile, :DropWhile, :Lines, :Split, :AsyncBoundary, :BufferBoundary,
|
|
207
|
-
:Throttle, :ParallelMapBoundary, :ParallelUnorderedMapBoundary, :RactorMapBoundary
|
|
212
|
+
:Throttle, :ParallelMapBoundary, :ParallelUnorderedMapBoundary, :RactorMapBoundary,
|
|
213
|
+
:RactorUnorderedMapBoundary
|
|
208
214
|
end
|
|
209
215
|
end
|
data/lib/fiber_stream/sink.rb
CHANGED
|
@@ -29,6 +29,58 @@ module FiberStream
|
|
|
29
29
|
end
|
|
30
30
|
end
|
|
31
31
|
|
|
32
|
+
# Creates a sink that returns the first element matching a predicate.
|
|
33
|
+
#
|
|
34
|
+
# The sink pulls upstream until the block returns a truthy value or
|
|
35
|
+
# upstream completes. It returns the original matching element, or `nil`
|
|
36
|
+
# when no element matches. Matching `nil` elements are returned as `nil`,
|
|
37
|
+
# following Ruby's `Enumerable#find` ambiguity.
|
|
38
|
+
def self.find(&block)
|
|
39
|
+
raise ArgumentError, "missing block" unless block
|
|
40
|
+
|
|
41
|
+
new do |stream|
|
|
42
|
+
loop do
|
|
43
|
+
value = stream.next
|
|
44
|
+
break nil if Pull.done?(value)
|
|
45
|
+
break value if block.call(value)
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
# Creates a sink that returns whether any element matches a predicate.
|
|
51
|
+
#
|
|
52
|
+
# The sink pulls upstream until the block returns a truthy value or upstream
|
|
53
|
+
# completes. It returns `true` after a truthy predicate result and `false`
|
|
54
|
+
# when no element matches.
|
|
55
|
+
def self.any?(&block)
|
|
56
|
+
raise ArgumentError, "missing block" unless block
|
|
57
|
+
|
|
58
|
+
new do |stream|
|
|
59
|
+
loop do
|
|
60
|
+
value = stream.next
|
|
61
|
+
break false if Pull.done?(value)
|
|
62
|
+
break true if block.call(value)
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
# Creates a sink that returns whether all elements match a predicate.
|
|
68
|
+
#
|
|
69
|
+
# The sink pulls upstream until the block returns false or nil, or upstream
|
|
70
|
+
# completes. It returns `false` after a falsey predicate result and `true`
|
|
71
|
+
# when every predicate result is truthy, including for empty upstream.
|
|
72
|
+
def self.all?(&block)
|
|
73
|
+
raise ArgumentError, "missing block" unless block
|
|
74
|
+
|
|
75
|
+
new do |stream|
|
|
76
|
+
loop do
|
|
77
|
+
value = stream.next
|
|
78
|
+
break true if Pull.done?(value)
|
|
79
|
+
break false unless block.call(value)
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
32
84
|
# Creates a sink that counts all stream elements.
|
|
33
85
|
#
|
|
34
86
|
# The sink consumes upstream until normal completion and returns the number
|
data/lib/fiber_stream/source.rb
CHANGED
|
@@ -246,6 +246,25 @@ module FiberStream
|
|
|
246
246
|
)
|
|
247
247
|
end
|
|
248
248
|
|
|
249
|
+
# Returns a new source definition that maps elements in Ractor workers and
|
|
250
|
+
# emits mapped values as workers complete.
|
|
251
|
+
#
|
|
252
|
+
# This is a convenience wrapper around
|
|
253
|
+
# `via(FiberStream::Flow.ractor_unordered_map(workers:) { ... })` and
|
|
254
|
+
# preserves the same shareable mapper requirement, unordered delivery,
|
|
255
|
+
# transfer policy, bounded upstream run-ahead, and cooperative worker
|
|
256
|
+
# shutdown behavior.
|
|
257
|
+
def ractor_unordered_map(workers:, input_transfer: :copy, output_transfer: :copy, &block)
|
|
258
|
+
via(
|
|
259
|
+
Flow.ractor_unordered_map(
|
|
260
|
+
workers: workers,
|
|
261
|
+
input_transfer: input_transfer,
|
|
262
|
+
output_transfer: output_transfer,
|
|
263
|
+
&block
|
|
264
|
+
)
|
|
265
|
+
)
|
|
266
|
+
end
|
|
267
|
+
|
|
249
268
|
# Returns a new source definition that keeps elements matching `block`.
|
|
250
269
|
#
|
|
251
270
|
# This is a convenience wrapper around
|
data/lib/fiber_stream/version.rb
CHANGED
data/sig/fiber_stream.rbs
CHANGED
|
@@ -87,6 +87,7 @@ module FiberStream
|
|
|
87
87
|
def parallel_map: [Out] (concurrency: Integer) { (Elem) -> Out } -> Source[Out]
|
|
88
88
|
def parallel_unordered_map: [Out] (concurrency: Integer) { (Elem) -> Out } -> Source[Out]
|
|
89
89
|
def ractor_map: [Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (Elem) -> Out } -> Source[Out]
|
|
90
|
+
def ractor_unordered_map: [Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (Elem) -> Out } -> Source[Out]
|
|
90
91
|
def filter_map: [Out] () { (Elem) -> (Out | false | nil) } -> Source[Out]
|
|
91
92
|
def compact: () -> Source[Elem]
|
|
92
93
|
def map_concat: [Out] () { (Elem) -> Enumerable[Out] } -> Source[Out]
|
|
@@ -113,6 +114,7 @@ module FiberStream
|
|
|
113
114
|
def self.parallel_map: [In, Out] (concurrency: Integer) { (In) -> Out } -> Flow[In, Out]
|
|
114
115
|
def self.parallel_unordered_map: [In, Out] (concurrency: Integer) { (In) -> Out } -> Flow[In, Out]
|
|
115
116
|
def self.ractor_map: [In, Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (In) -> Out } -> Flow[In, Out]
|
|
117
|
+
def self.ractor_unordered_map: [In, Out] (workers: Integer, ?input_transfer: ractor_transfer_policy, ?output_transfer: ractor_transfer_policy) { (In) -> Out } -> Flow[In, Out]
|
|
116
118
|
def self.filter_map: [In, Out] () { (In) -> (Out | false | nil) } -> Flow[In, Out]
|
|
117
119
|
def self.compact: [Elem] () -> Flow[Elem, Elem]
|
|
118
120
|
def self.map_concat: [In, Out] () { (In) -> Enumerable[Out] } -> Flow[In, Out]
|
|
@@ -136,6 +138,9 @@ module FiberStream
|
|
|
136
138
|
class Sink[In, Mat]
|
|
137
139
|
def self.to_a: [Elem] () -> Sink[Elem, Array[Elem]]
|
|
138
140
|
def self.first: [Elem] () -> Sink[Elem, Elem?]
|
|
141
|
+
def self.find: [Elem] () { (Elem) -> boolish } -> Sink[Elem, Elem?]
|
|
142
|
+
def self.any?: [Elem] () { (Elem) -> boolish } -> Sink[Elem, bool]
|
|
143
|
+
def self.all?: [Elem] () { (Elem) -> boolish } -> Sink[Elem, bool]
|
|
139
144
|
def self.count: [Elem] () -> Sink[Elem, Integer]
|
|
140
145
|
def self.fold: [Elem, Acc] (Acc initial) { (Acc, Elem) -> Acc } -> Sink[Elem, Acc]
|
|
141
146
|
def self.foreach: [Elem] () { (Elem) -> void } -> Sink[Elem, Integer]
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fiber_stream
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Dai Akatsuka
|
|
@@ -120,6 +120,7 @@ files:
|
|
|
120
120
|
- examples/ractor_merge_ports_and_map.rb
|
|
121
121
|
- examples/ractor_port_source.rb
|
|
122
122
|
- examples/ractor_producer_sources.rb
|
|
123
|
+
- examples/ractor_unordered_map_hashing.rb
|
|
123
124
|
- lib/fiber_stream.rb
|
|
124
125
|
- lib/fiber_stream/errors.rb
|
|
125
126
|
- lib/fiber_stream/flow.rb
|
|
@@ -146,6 +147,7 @@ files:
|
|
|
146
147
|
- lib/fiber_stream/pull/ractor_merge_ports_source.rb
|
|
147
148
|
- lib/fiber_stream/pull/ractor_port_source.rb
|
|
148
149
|
- lib/fiber_stream/pull/ractor_producer_source.rb
|
|
150
|
+
- lib/fiber_stream/pull/ractor_unordered_map_boundary.rb
|
|
149
151
|
- lib/fiber_stream/pull/reject.rb
|
|
150
152
|
- lib/fiber_stream/pull/scan.rb
|
|
151
153
|
- lib/fiber_stream/pull/select.rb
|
|
@@ -169,7 +171,7 @@ licenses:
|
|
|
169
171
|
metadata:
|
|
170
172
|
allowed_push_host: https://rubygems.org
|
|
171
173
|
homepage_uri: https://github.com/dakatsuka/fiber_stream
|
|
172
|
-
source_code_uri: https://github.com/dakatsuka/fiber_stream/tree/v0.
|
|
174
|
+
source_code_uri: https://github.com/dakatsuka/fiber_stream/tree/v0.6.0
|
|
173
175
|
changelog_uri: https://github.com/dakatsuka/fiber_stream/blob/main/CHANGELOG.md
|
|
174
176
|
rubygems_mfa_required: 'true'
|
|
175
177
|
rdoc_options: []
|