fiber_stream 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2401496eff99cd4792deda8fa927688fc3ba0e97e8a35637db4395895ab04cd9
-  data.tar.gz: 9f9674e2bc7dc9ce49b899727a02b9158ce6c9629c744ec6d626b16923e14160
+  metadata.gz: 504a7400182e09bb66a5a07981bbb2cfab350397d09c23ea036e856f29afe1d2
+  data.tar.gz: 67c7ddead203d42bf869dcba7d840e7d09514d888b90b84b77d97a8cedc19e1e
 SHA512:
-  metadata.gz: e7237c7c15b66105b09cdccf1e52980f3c030c74acbd2dc7cc3b5cfb720a716836925b5801bdc59351131dcd7ecab72ebdedb6b50593f5258a1cff2d0ebb39a9
-  data.tar.gz: d9bc7c85992c344bef4a36b0f234840c154c06274c450ef0696246d3042a017f8764c22ab1a2df43e3059420c52af7827852e63b4a5e91d668f4c9bb6a191d57
+  metadata.gz: f9c8d80faca53d9c6e059326bc9e6ffe7497aa8a6cbcad3b42eefe0ef3a3d8d29f08a3afbec8fc2b02b2064a745bb00ea644e816f913aa624a2ad11f1b087b99
+  data.tar.gz: 9e6f87b74cbfe2ccaaa9c32990f9786af3ee3ee3d7f2c3dd605a88efa8f3e436e4a1051d65c1eb1664859af212a2fecc82c66dc585cf9333dd799917c2a9035e

data/CHANGELOG.md

@@ -1,5 +1,53 @@
 # Changelog
 
+## 0.5.0 - 2026-06-21
+
+### Added
+
+- `Flow.tap { |element| ... }` for lazy pass-through observation without
+  changing emitted elements.
+- `Flow.filter_map { |element| ... }` and `Source#filter_map` for combined
+  transformation and falsey-value dropping.
+- `Flow.reject { |element| ... }` and `Source#reject` for complement
+  predicate filtering.
+- `Flow.compact` and `Source#compact` for nil-only filtering while preserving
+  `false`.
+- `Flow.map_concat { |element| enumerable }` and `Source#map_concat` for
+  one-to-many element expansion.
+- `Flow.throttle(...)`, `Source#throttle(...)`, and `RateLimiter` for
+  pull-driven rate limiting with optional shared quota state.
+- `Sink.count` for counting stream elements without accumulating them.
+
+### Changed
+
+- Expanded README and website reference coverage for the flow operators and
+  rate limiter added in 0.5.0.
+- Promoted completed flow product specs and design docs from draft to accepted
+  status.
+
+## 0.4.0 - 2026-06-09
+
+### Added
+
+- `Flow.parallel_unordered_map(concurrency:)` and
+  `Source#parallel_unordered_map(concurrency:)` for scheduler-backed mapping
+  that emits results in completion order instead of preserving input order.
+- `Source.ractor_producer` for FiberStream-owned single producer Ractors with
+  one-outstanding-ack backpressure and cooperative cleanup.
+- `Source.ractor_merge_producers` for ready-order fan-in from multiple
+  FiberStream-owned producer Ractors without requiring a `Fiber.scheduler`.
+- `Flow.scan(initial)` and `Source#scan(initial)` for lazy running
+  accumulators using `Sink.fold`-style reducer semantics.
+
+### Changed
+
+- Updated README and website reference coverage for owned Ractor producers,
+  unordered parallel mapping, and scan.
+- Prefer high-level owned Ractor producer examples in user-facing
+  documentation while keeping low-level port APIs documented for externally
+  owned producers.
+- Updated the project Ruby pin to 4.0.5.
+
 ## 0.3.0 - 2026-06-06
 
 ### Added

data/README.md

@@ -1,12 +1,15 @@
 # FiberStream
 
-FiberStream is an early-stage Ruby library for linear, pull-based stream
-processing with backpressure.
+FiberStream is a Ruby library for linear stream processing with pull-based
+backpressure.
 
-Build a lazy `Source`, transform it with `Flow` stages, and materialize it with
-a `Sink`.
+It builds lazy `Source` definitions, transforms values with `Flow` stages, and
+materializes results with `Sink` objects.
+
+[![Gem Version](https://badge.fury.io/rb/fiber_stream.svg)](https://badge.fury.io/rb/fiber_stream)
 
 ## Quick Start
+Please see the project [documentation](https://dakatsuka.github.io/fiber_stream/) for more details.
 
 ```ruby
 require "fiber_stream"
@@ -27,12 +30,15 @@ FiberStream currently supports linear pipelines only.
 
 Implemented capabilities:
 
-- in-memory, IO, backpressure-aware Ractor port, and Ractor port merge sources
+- in-memory, IO, FiberStream-owned Ractor producer, backpressure-aware Ractor
+  port, and Ractor port merge sources
 - lazy source concatenation, zipping, and scheduler-backed merging
-- mapping, filtering, limiting, predicate-based limiting and dropping,
-  fixed-prefix dropping, fixed-size grouping, line splitting, buffering, async
-  boundaries, ordered parallel mapping, and ordered Ractor-backed mapping
-- array, first-element, fold, foreach, and IO sinks
+- mapping, filtering, transform-and-filter, nil compaction, side-effect
+  observation, one-to-many expansion, limiting, predicate-based limiting and
+  dropping, fixed-prefix dropping, fixed-size grouping, line splitting,
+  buffering, async boundaries, throttling, ordered and unordered parallel
+  mapping, and ordered Ractor-backed mapping
+- array, first-element, count, fold, foreach, and IO sinks
 - reusable flow composition and runnable pipelines
 - foreground and scheduler-backed background pipeline execution
 - public RBS signatures
@@ -96,64 +102,51 @@ chunks =
   end.wait
 ```
 
-Ractor port sources connect a producer Ractor with an explicit ack handshake.
-The producer creates its acknowledgment port, waits for `RactorPort::Ack`, and
-then sends one typed message back to the FiberStream data port:
+Owned Ractor producer sources run producer blocks in FiberStream-managed
+Ractors. The producer block receives a `RactorProducer` context and emits one
+value per downstream demand:
 
 ```ruby
-data_port = Ractor::Port.new
-setup_port = Ractor::Port.new
-
-producer =
-  Ractor.new(data_port, setup_port) do |outbox, setup|
-    ack_port = Ractor::Port.new
-    setup.send(ack_port)
-
-    values = [1, 2, 3].to_enum
-
-    loop do
-      case ack_port.receive
-      in FiberStream::RactorPort::Ack
-        begin
-          outbox.send(FiberStream::RactorPort::Element.new(values.next))
-        rescue StopIteration
-          outbox.send(FiberStream::RactorPort::Complete.new)
-          break
-        end
-      in FiberStream::RactorPort::Cancel
-        break
-      end
+PRODUCE_VALUES =
+  Ractor.shareable_proc do |producer, values|
+    values.each do |value|
+      break unless producer.emit(value)
     end
   end
 
-ack_port = setup_port.receive
-
-FiberStream::Source.ractor_port(data_port, ack_port: ack_port)
+FiberStream::Source.ractor_producer([1, 2, 3], &PRODUCE_VALUES)
   .run_with(FiberStream::Sink.to_a)
 # => [1, 2, 3]
-
-producer.value
 ```
 
-`RactorPort::Failure` cause metadata is producer-provided and is surfaced on
-`RactorPortSourceError`. Redact internal paths, secrets, tenant data, or other
-sensitive details before sending failures across trust boundaries.
-
-Multiple producer Ractors can be merged directly without a scheduler-backed
-`Source#merge`. Each producer still receives at most one outstanding ack:
+Multiple owned producer Ractors can be merged directly without a
+scheduler-backed `Source#merge`. Each producer still receives at most one
+outstanding ack:
 
 ```ruby
+PRODUCE_TAGGED_VALUES =
+  Ractor.shareable_proc do |producer, tag, values|
+    values.each do |value|
+      break unless producer.emit([tag, value])
+    end
+  end
+
 source =
-  FiberStream::Source.ractor_merge_ports(
-    [
-      { port: data_port_a, ack_port: ack_port_a },
-      { port: data_port_b, ack_port: ack_port_b }
-    ]
-  )
+  FiberStream::Source.ractor_merge_producers do |group|
+    group.producer(:a, [1, 2], &PRODUCE_TAGGED_VALUES)
+    group.producer(:b, [3, 4], &PRODUCE_TAGGED_VALUES)
+  end
 
-values = source.run_with(FiberStream::Sink.to_a)
+source.run_with(FiberStream::Sink.to_a)
+# Example result: [[:a, 1], [:b, 3], [:a, 2], [:b, 4]]
 ```
 
+Use the lower-level `Source.ractor_port` and `Source.ractor_merge_ports` APIs
+when producer Ractors are owned outside FiberStream or need custom lifecycle
+handling. `RactorPort::Failure` cause metadata is producer-provided and is
+surfaced on `RactorPortSourceError`; redact sensitive details before sending
+failures across trust boundaries.
+
 Streaming HTTP response bodies that implement `#each`, such as
 `async-http` response bodies, can be used with `Source.each` without buffering
 the full body first. Use the HTTP client's block form or an explicit `ensure`
@@ -216,6 +209,62 @@ FiberStream::Source.each([" a ", "", " b "])
 # => ["a", "b"]
 ```
 
+Use `reject` when the predicate names values to drop. Truthy predicate results
+drop the original element; `false` and `nil` results pass it through unchanged:
+
+```ruby
+result =
+  FiberStream::Source.each([1, 2, 3, 4])
+    .reject(&:even?)
+    .run_with(FiberStream::Sink.to_a)
+
+result # => [1, 3]
+```
+
+Use `filter_map` when filtering and transformation are one decision. Truthy
+block results are emitted as transformed values; `false` and `nil` are
+dropped:
+
+```ruby
+ids =
+  FiberStream::Source.each([{ id: 1 }, {}, { id: 3 }])
+    .filter_map { |record| record[:id] }
+    .run_with(FiberStream::Sink.to_a)
+
+ids # => [1, 3]
+```
+
+Use `compact` to drop only `nil` while keeping `false`, and `map_concat` to
+expand one upstream element into zero or more downstream elements:
+
+```ruby
+tokens =
+  FiberStream::Source.each(["alpha beta", nil, "gamma"])
+    .compact
+    .map_concat { |line| line.split }
+    .run_with(FiberStream::Sink.to_a)
+
+tokens # => ["alpha", "beta", "gamma"]
+```
+
+Use `Flow.tap` for observation inside a reusable flow without changing the
+element:
+
+```ruby
+seen = []
+
+observed =
+  FiberStream::Flow.tap { |value| seen << value }
+    .via(FiberStream::Flow.map { |value| value * 10 })
+
+FiberStream::Source.each([1, 2])
+  .via(observed)
+  .run_with(FiberStream::Sink.to_a)
+# => [10, 20]
+
+seen # => [1, 2]
+```
+
 Use `parallel_map` for ordered scheduler-backed mapping when each element
 waits on non-blocking IO. It preserves input order while allowing up to
 `concurrency` mapping operations to be in flight:
@@ -240,6 +289,26 @@ profiles =
 profiles.map { |profile| profile.fetch(:id) } # => [1, 2, 3, 4]
 ```
 
+Use `parallel_unordered_map` when every result can be handled independently
+and lower head-of-line blocking matters more than input order. It still limits
+in-flight mapping work to `concurrency`, but emits values as mapping jobs
+finish:
+
+```ruby
+require "async"
+require "fiber_stream"
+
+responses =
+  Sync do
+    FiberStream::Source.each(["/a", "/slow", "/b"])
+      .parallel_unordered_map(concurrency: 3) { |path| fetch_path(path) }
+      .run_with(FiberStream::Sink.to_a)
+  end
+
+# Results are in completion order, not necessarily input order.
+responses
+```
+
 Use `ractor_map` for ordered CPU-bound mapping in Ractor workers. The mapper
 must be shareable, usually by creating it with `Ractor.shareable_proc`.
 
@@ -284,6 +353,14 @@ FiberStream::Source.each([1, 2, 3])
 # => 6
 ```
 
+Use `Sink.count` when only the number of elements matters:
+
+```ruby
+FiberStream::Source.each([1, 2, 3])
+  .run_with(FiberStream::Sink.count)
+# => 3
+```
+
 Use `Sink.foreach` when the terminal operation is a side effect and the stream
 values should not be accumulated:
 
@@ -384,6 +461,17 @@ batches =
 batches # => [[1, 2], [3, 4], [5]]
 ```
 
+`Flow.scan` emits the updated accumulator for each upstream element:
+
+```ruby
+running_totals =
+  FiberStream::Source.each([1, 2, 3, 4])
+    .scan(0) { |sum, number| sum + number }
+    .run_with(FiberStream::Sink.to_a)
+
+running_totals # => [1, 3, 6, 10]
+```
+
 `Flow.take_while` emits the leading prefix while a predicate is truthy, then
 closes upstream at the first false or nil result:
 
@@ -450,14 +538,18 @@ merged =
 
 `merge` does not make scheduler-unaware blocking source work non-blocking and
 does not provide CPU parallelism. Use producer ractors with
-`Source.ractor_port` or `Source.ractor_merge_ports` when producer work needs
-true isolation.
-
-`Flow.buffer(count)` allows bounded prefetch. `Flow.async`, `Flow.buffer`,
-`Flow.parallel_map`, `Source.io`, `Source#merge`, `Sink.io`, and
-`Pipeline#run_async` require an installed `Fiber.scheduler` and a non-blocking
-current fiber when demanded or started. FiberStream does not install a
-scheduler and does not depend on Async at runtime.
+`Source.ractor_producer` or `Source.ractor_merge_producers` when producer work
+needs true isolation.
+
+`Flow.buffer(count)` allows bounded prefetch. `Flow.throttle(rate:, per:)`
+paces elements before downstream side effects. `Flow.async`, `Flow.buffer`,
+`Flow.parallel_map`, `Flow.parallel_unordered_map`, `Source.io`,
+`Source#merge`, `Sink.io`, and `Pipeline#run_async` require an installed
+`Fiber.scheduler` and a non-blocking current fiber when demanded or started.
+`Flow.throttle` requires that scheduler context only when it needs to wait.
+Pass `throttle(limiter:)` with a `FiberStream::RateLimiter` when multiple
+pipelines or repeated runs should share quota state. FiberStream does not
+install a scheduler and does not depend on Async at runtime.
 
 ## API Surface
 
@@ -465,6 +557,8 @@ Sources:
 
 - `FiberStream::Source.each(enumerable)`
 - `FiberStream::Source.io(io, chunk_size: 16 * 1024, close: false)`
+- `FiberStream::Source.ractor_producer(*args, transfer: :copy, ack_transfer: :copy) { |producer, *args| ... }`
+- `FiberStream::Source.ractor_merge_producers(transfer: :copy, ack_transfer: :copy) { |group| ... }`
 - `FiberStream::Source.ractor_port(port, ack_port:, ack_transfer: :copy, cancel: true)`
 - `FiberStream::Source.ractor_merge_ports(ports, ack_transfer: :copy, cancel: true)`
 
@@ -475,16 +569,24 @@ Source convenience methods:
 - `Source#zip(source)`
 - `Source#merge(source)`
 - `Source#map { |element| ... }`
+- `Source#filter_map { |element| ... }`
+- `Source#compact`
+- `Source#map_concat { |element| enumerable }`
 - `Source#parallel_map(concurrency:) { |element| ... }`
+- `Source#parallel_unordered_map(concurrency:) { |element| ... }`
 - `Source#ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
 - `Source#select { |element| ... }`
+- `Source#reject { |element| ... }`
 - `Source#take(count)`
 - `Source#drop(count)`
 - `Source#grouped(count)`
+- `Source#scan(initial) { |accumulator, element| ... }`
 - `Source#take_while { |element| ... }`
 - `Source#drop_while { |element| ... }`
 - `Source#async`
 - `Source#buffer(count)`
+- `Source#throttle(rate:, per: 1, burst: nil)`
+- `Source#throttle(limiter:)`
 - `Source#lines(chomp: true, max_length: nil)`
 - `Source#split(separator, keep_separator: false, max_length: nil)`
 - `Source#to(sink)`
@@ -493,20 +595,31 @@ Source convenience methods:
 Flows:
 
 - `FiberStream::Flow.map { |element| ... }`
+- `FiberStream::Flow.filter_map { |element| ... }`
+- `FiberStream::Flow.compact`
+- `FiberStream::Flow.map_concat { |element| enumerable }`
+- `FiberStream::Flow.tap { |element| ... }`
 - `FiberStream::Flow.parallel_map(concurrency:) { |element| ... }`
+- `FiberStream::Flow.parallel_unordered_map(concurrency:) { |element| ... }`
 - `FiberStream::Flow.ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
 - `FiberStream::Flow.select { |element| ... }`
+- `FiberStream::Flow.reject { |element| ... }`
 - `FiberStream::Flow.take(count)`
 - `FiberStream::Flow.drop(count)`
 - `FiberStream::Flow.grouped(count)`
+- `FiberStream::Flow.scan(initial) { |accumulator, element| ... }`
 - `FiberStream::Flow.take_while { |element| ... }`
 - `FiberStream::Flow.drop_while { |element| ... }`
 - `FiberStream::Flow.async`
 - `FiberStream::Flow.buffer(count)`
+- `FiberStream::Flow.throttle(rate:, per: 1, burst: nil)`
+- `FiberStream::Flow.throttle(limiter:)`
 - `FiberStream::Flow.lines(chomp: true, max_length: nil)`
 - `FiberStream::Flow.split(separator, keep_separator: false, max_length: nil)`
 - `Flow#via(flow)`
 - `Flow#to(sink)`
+- `FiberStream::RateLimiter.new(rate:, per: 1, burst: nil)`
+- `FiberStream::RateLimiter#acquire(permits: 1)`
 
 `lines` and `split` default to `max_length: nil`, which allows one
 unterminated line or frame to buffer without bound. Set a positive
@@ -516,6 +629,7 @@ Sinks:
 
 - `FiberStream::Sink.to_a`
 - `FiberStream::Sink.first`
+- `FiberStream::Sink.count`
 - `FiberStream::Sink.fold(initial) { |accumulator, element| ... }`
 - `FiberStream::Sink.foreach { |element| ... }`
 - `FiberStream::Sink.io(io, close: false, flush: false)`
@@ -541,6 +655,7 @@ bundle exec ruby examples/file_copy.rb
 bundle exec ruby examples/backpressure_buffer.rb
 bundle exec ruby examples/background_execution.rb
 bundle exec ruby examples/ractor_map_hashing.rb
+bundle exec ruby examples/ractor_producer_sources.rb
 bundle exec ruby examples/ractor_port_source.rb
 bundle exec ruby examples/ractor_merge_ports_and_map.rb
 bundle exec ruby examples/async_http_requests.rb
@@ -553,6 +668,9 @@ events so the difference between direct demand and bounded prefetch is visible.
 `examples/ractor_map_hashing.rb` demonstrates ordered Ractor-backed hashing
 with a shareable mapper proc and `input_transfer: :move`.
 
+`examples/ractor_producer_sources.rb` demonstrates high-level owned producer
+Ractors with `Source.ractor_producer` and `Source.ractor_merge_producers`.
+
 `examples/ractor_port_source.rb` demonstrates a producer Ractor that waits for
 `RactorPort::Ack` before sending each `RactorPort::Element`.
 
@@ -578,7 +696,7 @@ bundle exec ruby benchmarks/heavy_cpu_map.rb
 
 ## Development
 
-This project targets Ruby 4.x. The repository currently pins Ruby 4.0.3 in
+This project targets Ruby 4.x. The repository currently pins Ruby 4.0.5 in
 `mise.toml`.
 
 Install dependencies:

data/examples/README.md

@@ -11,6 +11,7 @@ bundle exec ruby examples/backpressure_buffer.rb
 bundle exec ruby examples/background_execution.rb
 bundle exec ruby examples/ractor_map_hashing.rb
 bundle exec ruby examples/ractor_port_source.rb
+bundle exec ruby examples/ractor_producer_sources.rb
 bundle exec ruby examples/ractor_merge_ports_and_map.rb
 bundle exec ruby examples/async_http_requests.rb
 bundle exec ruby examples/async_http_streaming_body.rb
@@ -48,6 +49,11 @@ pipeline runs.
 `RactorPort::Ack`, and sends one typed `RactorPort::Element` per downstream
 demand.
 
+`ractor_producer_sources.rb` demonstrates the high-level owned-producer APIs:
+`Source.ractor_producer` for one producer and `Source.ractor_merge_producers`
+for ready-order fan-in from multiple producers. FiberStream creates the ports,
+producer Ractors, and cooperative cleanup path.
+
 `ractor_merge_ports_and_map.rb` runs CPU-bound work in multiple producer
 Ractors, merges their port outputs with `Source.ractor_merge_ports`, then runs
 another CPU-bound verification stage with `ractor_map`.

data/examples/ractor_producer_sources.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+
+require "fiber_stream"
+
+EMIT_NUMBERS =
+  Ractor.shareable_proc do |producer, range|
+    range.each do |number|
+      break unless producer.emit(number)
+    end
+  end
+
+EMIT_TAGGED_NUMBERS =
+  Ractor.shareable_proc do |producer, tag, range|
+    range.each do |number|
+      break unless producer.emit([tag, number])
+    end
+  end
+
+squares =
+  FiberStream::Source.ractor_producer(1..5, &EMIT_NUMBERS)
+    .map { |number| number * number }
+    .run_with(FiberStream::Sink.to_a)
+
+puts "Squares from one FiberStream-owned producer Ractor:"
+puts squares.join(", ")
+
+merged =
+  FiberStream::Source.ractor_merge_producers do |group|
+    group.producer(:low, 1..3, &EMIT_TAGGED_NUMBERS)
+    group.producer(:high, 10..12, &EMIT_TAGGED_NUMBERS)
+  end.run_with(FiberStream::Sink.to_a)
+
+puts
+puts "Merged values from two owned producer Ractors:"
+merged.each do |tag, number|
+  puts "- #{tag}: #{number}"
+end
+
+puts
+puts "Producer blocks use RactorProducer#emit and stop when it returns false."
+puts "FiberStream creates the data ports, ack ports, producer Ractors, and cleanup path."