fiber_stream 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ab4449558105eb57805e40970a28971e32a37e748176df0875de75816bf53d2c
-  data.tar.gz: 6d1998ce477a4602a6f23d60817f2f49b9acc6efde64b18ee90ebd7084905bd5
+  metadata.gz: 2401496eff99cd4792deda8fa927688fc3ba0e97e8a35637db4395895ab04cd9
+  data.tar.gz: 9f9674e2bc7dc9ce49b899727a02b9158ce6c9629c744ec6d626b16923e14160
 SHA512:
-  metadata.gz: 4242c44baa5c1db6f7cb6c39ac217729603838dba31876180f4fcc2382e72308cad4d5197d8687b522317dfc9ad964bcd6f85b9449f580f88324098536b8906c
-  data.tar.gz: c2cc8091ec14b27eef4a60a82c1c22503118eee9da0a1a53e24a3e137058e1dd8c5e1536e89c5ba13c7f725cdc0ad241d48ade86cbe2162f6ac5ae2d9ce8f96f
+  metadata.gz: e7237c7c15b66105b09cdccf1e52980f3c030c74acbd2dc7cc3b5cfb720a716836925b5801bdc59351131dcd7ecab72ebdedb6b50593f5258a1cff2d0ebb39a9
+  data.tar.gz: d9bc7c85992c344bef4a36b0f234840c154c06274c450ef0696246d3042a017f8764c22ab1a2df43e3059420c52af7827852e63b4a5e91d668f4c9bb6a191d57

data/CHANGELOG.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## 0.3.0 - 2026-06-06
+
+### Added
+
+- `Flow.grouped(count)` and `Source#grouped(count)` for fixed-size batches
+  with final partial-group emission.
+- `Source#merge(source)` for scheduler-backed ready-order merging of two
+  sources while preserving each input source's own order.
+- `Source.ractor_merge_ports(ports)` for backpressure-aware merging of
+  multiple producer Ractor ports without requiring a `Fiber.scheduler`.
+- `Flow.split(separator)` and `Source#split(separator)` for delimiter-based
+  framing with optional separator retention and per-frame length limits.
+- Benchmarks and examples for async IO fanout, stream lifecycle probes, and
+  Ractor port merge workflows.
+
+### Changed
+
+- Reworked flow operator tests into focused per-operator test files.
+- Expanded README and repository documentation for source merging, Ractor port
+  merging, split framing, grouped batches, and runtime safety guidance.
+- Clarified that `Flow.lines(max_length: nil)` and
+  `Flow.split(max_length: nil)` may buffer one unterminated frame without
+  bound, and documented explicit `max_length` usage for untrusted streams.
+- Clarified `Source.io` `chunk_size` allocation behavior and Ractor failure
+  metadata exposure.
+
+### Fixed
+
+- Deferred `Source#concat` receiver materialization until downstream demand
+  reaches the concatenated source.
+- Cancelled async and buffer producers when downstream closes early.
+- Removed polling from Ractor map enqueue and cleanup paths.
+- Re-raised background pipeline process-control exceptions instead of treating
+  them as ordinary stream failures.
+- Hardened Ractor map worker teardown notifications so secondary send failures
+  do not cascade during shutdown.
+
+## 0.2.0 - 2026-06-05
+
+### Added
+
+- `Source#zip(source)` for element-wise pairing of two sources with
+  demand-driven materialization and shortest-source completion.
+- `Source#concat(source)` for lazy source concatenation.
+- `Sink.foreach { |element| ... }` for side-effecting stream consumption
+  without accumulating elements.
+- `Flow.drop(count)` and `Source#drop(count)` for fixed-prefix dropping.
+- `Flow.take_while { |element| ... }` and `Source#take_while { |element| ... }`
+  for predicate-based prefix limiting.
+- `Flow.drop_while { |element| ... }` and
+  `Source#drop_while { |element| ... }` for predicate-based prefix dropping.
+
+### Changed
+
+- Clarified documentation around FiberStream's linear roadmap and Ractor port
+  cancellation contract.
+
 ## 0.1.0 - 2026-06-03
 
 Initial release.

data/README.md

@@ -27,17 +27,18 @@ FiberStream currently supports linear pipelines only.
 
 Implemented capabilities:
 
-- in-memory, IO, and backpressure-aware Ractor port sources
-- mapping, filtering, limiting, line splitting, buffering, async boundaries,
-  ordered parallel mapping, and ordered Ractor-backed mapping
-- array, first-element, fold, and IO sinks
+- in-memory, IO, backpressure-aware Ractor port, and Ractor port merge sources
+- lazy source concatenation, zipping, and scheduler-backed merging
+- mapping, filtering, limiting, predicate-based limiting and dropping,
+  fixed-prefix dropping, fixed-size grouping, line splitting, buffering, async
+  boundaries, ordered parallel mapping, and ordered Ractor-backed mapping
+- array, first-element, fold, foreach, and IO sinks
 - reusable flow composition and runnable pipelines
 - foreground and scheduler-backed background pipeline execution
 - public RBS signatures
 
-Not yet implemented:
-
-- graph DSLs
+FiberStream intentionally keeps the public model linear: one source, an
+ordered chain of flows, and one sink.
 
 ## Core Concepts
 
@@ -52,8 +53,35 @@ source = FiberStream::Source.each([1, 2, 3])
 source.run_with(FiberStream::Sink.to_a) # => [1, 2, 3]
 ```
 
+Sources can be concatenated without materializing the appended source until the
+first source completes:
+
+```ruby
+result =
+  FiberStream::Source.each([1, 2])
+    .concat(FiberStream::Source.each([3, 4]))
+    .run_with(FiberStream::Sink.to_a)
+
+result # => [1, 2, 3, 4]
+```
+
+Sources can also be zipped element-by-element. The zipped source emits pairs
+and completes when either input source completes:
+
+```ruby
+result =
+  FiberStream::Source.each([1, 2, 3])
+    .zip(FiberStream::Source.each(["a", "b"]))
+    .run_with(FiberStream::Sink.to_a)
+
+result # => [[1, "a"], [2, "b"]]
+```
+
 IO sources read chunks on demand and require a scheduler-backed non-blocking
-fiber:
+fiber. The `chunk_size` option is the maximum byte count passed to
+`readpartial` for one downstream pull; very large values may cause the IO
+implementation to attempt large allocations, so choose a bounded value
+appropriate for the workload:
 
 ```ruby
 require "async"
@@ -107,6 +135,58 @@ FiberStream::Source.ractor_port(data_port, ack_port: ack_port)
 producer.value
 ```
 
+`RactorPort::Failure` cause metadata is producer-provided and is surfaced on
+`RactorPortSourceError`. Redact internal paths, secrets, tenant data, or other
+sensitive details before sending failures across trust boundaries.
+
+Multiple producer Ractors can be merged directly without a scheduler-backed
+`Source#merge`. Each producer still receives at most one outstanding ack:
+
+```ruby
+source =
+  FiberStream::Source.ractor_merge_ports(
+    [
+      { port: data_port_a, ack_port: ack_port_a },
+      { port: data_port_b, ack_port: ack_port_b }
+    ]
+  )
+
+values = source.run_with(FiberStream::Sink.to_a)
+```
+
+Streaming HTTP response bodies that implement `#each`, such as
+`async-http` response bodies, can be used with `Source.each` without buffering
+the full body first. Use the HTTP client's block form or an explicit `ensure`
+close because `Source.each` does not own the response body:
+
+```ruby
+require "async"
+require "async/http/internet/instance"
+require "fiber_stream"
+
+url = "https://raw.githubusercontent.com/elastic/examples/master/" \
+  "Common%20Data%20Formats/nginx_logs/nginx_logs"
+
+status_counts = Hash.new(0)
+
+processed =
+  Sync do
+    Async::HTTP::Internet.get(url) do |response|
+      raise "unexpected status #{response.status}" unless response.status == 200
+
+      FiberStream::Source.each(response.body)
+        .lines(max_length: 16 * 1024)
+        .map { |line| line.split.fetch(8, nil) }
+        .select { |status| status&.match?(/\A\d{3}\z/) }
+        .run_with(
+          FiberStream::Sink.foreach do |status|
+            status_counts[status] += 1
+          end
+        )
+    end
+  end
+```
+
 ### Flows
 
 Flows transform a stream lazily. Convenience methods on `Source` delegate to
@@ -136,6 +216,64 @@ FiberStream::Source.each([" a ", "", " b "])
 # => ["a", "b"]
 ```
 
+Use `parallel_map` for ordered scheduler-backed mapping when each element
+waits on non-blocking IO. It preserves input order while allowing up to
+`concurrency` mapping operations to be in flight:
+
+```ruby
+require "async"
+require "fiber_stream"
+
+def fetch_profile(user_id)
+  # Example: perform scheduler-aware HTTP, database, or socket IO here.
+  sleep 0.05
+  { id: user_id, name: "user-#{user_id}" }
+end
+
+profiles =
+  Sync do
+    FiberStream::Source.each([1, 2, 3, 4])
+      .parallel_map(concurrency: 4) { |user_id| fetch_profile(user_id) }
+      .run_with(FiberStream::Sink.to_a)
+  end
+
+profiles.map { |profile| profile.fetch(:id) } # => [1, 2, 3, 4]
+```
+
+Use `ractor_map` for ordered CPU-bound mapping in Ractor workers. The mapper
+must be shareable, usually by creating it with `Ractor.shareable_proc`.
+
+```ruby
+require "digest"
+require "fiber_stream"
+
+records = [
+  { name: "alpha.bin", payload: +"A" * 200_000 },
+  { name: "bravo.bin", payload: +"B" * 120_000 }
+]
+
+HASH_RECORD =
+  Ractor.shareable_proc do |record|
+    payload = record.fetch(:payload)
+
+    {
+      name: record.fetch(:name),
+      bytes: payload.bytesize,
+      sha256: Digest::SHA256.hexdigest(payload)
+    }
+  end
+
+digests =
+  FiberStream::Source.each(records)
+    .ractor_map(workers: 2, input_transfer: :move, &HASH_RECORD)
+    .run_with(FiberStream::Sink.to_a)
+```
+
+`ractor_map` preserves input order, limits pulled-but-unemitted work to
+`workers`, and does not require `Fiber.scheduler`. Use `input_transfer: :move`
+or `output_transfer: :move` only when the moved object will not be reused by
+the sender.
+
 ### Sinks
 
 A `Sink` consumes the stream and returns a materialized value.
@@ -146,6 +284,17 @@ FiberStream::Source.each([1, 2, 3])
 # => 6
 ```
 
+Use `Sink.foreach` when the terminal operation is a side effect and the stream
+values should not be accumulated:
+
+```ruby
+count =
+  FiberStream::Source.each(["a", "b", "c"])
+    .run_with(FiberStream::Sink.foreach { |value| puts value })
+
+count # => 3
+```
+
 ### Pipelines
 
 `Source#to(sink)` creates a reusable runnable pipeline.
@@ -212,11 +361,103 @@ limited =
 limited # => [1, 2]
 ```
 
+`Flow.drop` skips a fixed prefix and then passes later elements through:
+
+```ruby
+tail =
+  FiberStream::Source.each([1, 2, 3, 4])
+    .drop(2)
+    .run_with(FiberStream::Sink.to_a)
+
+tail # => [3, 4]
+```
+
+`Flow.grouped` batches adjacent elements into arrays and emits the final
+partial group:
+
+```ruby
+batches =
+  FiberStream::Source.each([1, 2, 3, 4, 5])
+    .grouped(2)
+    .run_with(FiberStream::Sink.to_a)
+
+batches # => [[1, 2], [3, 4], [5]]
+```
+
+`Flow.take_while` emits the leading prefix while a predicate is truthy, then
+closes upstream at the first false or nil result:
+
+```ruby
+prefix =
+  FiberStream::Source.each([1, 2, 3, 1])
+    .take_while { |number| number < 3 }
+    .run_with(FiberStream::Sink.to_a)
+
+prefix # => [1, 2]
+```
+
+`Flow.drop_while` skips the leading prefix while a predicate is truthy, then
+passes the first false or nil result and all later elements through:
+
+```ruby
+tail =
+  FiberStream::Source.each([1, 2, 3, 1])
+    .drop_while { |number| number < 3 }
+    .run_with(FiberStream::Sink.to_a)
+
+tail # => [3, 1]
+```
+
+`Source#concat` preserves pull-driven demand across source boundaries. The
+appended source is not materialized while the first source can still satisfy
+downstream demand:
+
+```ruby
+first =
+  FiberStream::Source.each([1])
+    .concat(FiberStream::Source.each([2]))
+    .run_with(FiberStream::Sink.first)
+
+first # => 1
+```
+
+`Source#zip` keeps input source materialization behind downstream demand. The
+other source is not materialized until the receiver has produced an element for
+a pair:
+
+```ruby
+first =
+  FiberStream::Source.each([1])
+    .zip(FiberStream::Source.each([2]))
+    .run_with(FiberStream::Sink.first)
+
+first # => [1, 2]
+```
+
+`Source#merge` emits values from either input source in scheduler-observed
+ready order while preserving each input's own order:
+
+```ruby
+merged =
+  Sync do
+    FiberStream::Source.each([1, 2])
+      .merge(FiberStream::Source.each(["a", "b"]))
+      .run_with(FiberStream::Sink.to_a)
+  end
+
+# Example result: [1, "a", 2, "b"]
+```
+
+`merge` does not make scheduler-unaware blocking source work non-blocking and
+does not provide CPU parallelism. Use producer ractors with
+`Source.ractor_port` or `Source.ractor_merge_ports` when producer work needs
+true isolation.
+
 `Flow.buffer(count)` allows bounded prefetch. `Flow.async`, `Flow.buffer`,
-`Flow.parallel_map`, `Source.io`, `Sink.io`, and `Pipeline#run_async` require an
-installed `Fiber.scheduler` and a non-blocking current fiber when demanded or
-started. FiberStream does not install a scheduler and does not depend on Async
-at runtime.
+`Flow.parallel_map`, `Source.io`, `Source#merge`, `Sink.io`, and
+`Pipeline#run_async` require an installed `Fiber.scheduler` and a non-blocking
+current fiber when demanded or started. FiberStream does not install a
+scheduler and does not depend on Async at runtime.
 
 ## API Surface
 
@@ -225,18 +466,27 @@ Sources:
 - `FiberStream::Source.each(enumerable)`
 - `FiberStream::Source.io(io, chunk_size: 16 * 1024, close: false)`
 - `FiberStream::Source.ractor_port(port, ack_port:, ack_transfer: :copy, cancel: true)`
+- `FiberStream::Source.ractor_merge_ports(ports, ack_transfer: :copy, cancel: true)`
 
 Source convenience methods:
 
 - `Source#via(flow)`
+- `Source#concat(source)`
+- `Source#zip(source)`
+- `Source#merge(source)`
 - `Source#map { |element| ... }`
 - `Source#parallel_map(concurrency:) { |element| ... }`
 - `Source#ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
 - `Source#select { |element| ... }`
 - `Source#take(count)`
+- `Source#drop(count)`
+- `Source#grouped(count)`
+- `Source#take_while { |element| ... }`
+- `Source#drop_while { |element| ... }`
 - `Source#async`
 - `Source#buffer(count)`
 - `Source#lines(chomp: true, max_length: nil)`
+- `Source#split(separator, keep_separator: false, max_length: nil)`
 - `Source#to(sink)`
 - `Source#run_with(sink)`
 
@@ -247,17 +497,27 @@ Flows:
 - `FiberStream::Flow.ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
 - `FiberStream::Flow.select { |element| ... }`
 - `FiberStream::Flow.take(count)`
+- `FiberStream::Flow.drop(count)`
+- `FiberStream::Flow.grouped(count)`
+- `FiberStream::Flow.take_while { |element| ... }`
+- `FiberStream::Flow.drop_while { |element| ... }`
 - `FiberStream::Flow.async`
 - `FiberStream::Flow.buffer(count)`
 - `FiberStream::Flow.lines(chomp: true, max_length: nil)`
+- `FiberStream::Flow.split(separator, keep_separator: false, max_length: nil)`
 - `Flow#via(flow)`
 - `Flow#to(sink)`
 
+`lines` and `split` default to `max_length: nil`, which allows one
+unterminated line or frame to buffer without bound. Set a positive
+`max_length` for untrusted, network-facing, or otherwise unbounded inputs.
+
 Sinks:
 
 - `FiberStream::Sink.to_a`
 - `FiberStream::Sink.first`
 - `FiberStream::Sink.fold(initial) { |accumulator, element| ... }`
+- `FiberStream::Sink.foreach { |element| ... }`
 - `FiberStream::Sink.io(io, close: false, flush: false)`
 
 Pipelines:
@@ -282,7 +542,9 @@ bundle exec ruby examples/backpressure_buffer.rb
 bundle exec ruby examples/background_execution.rb
 bundle exec ruby examples/ractor_map_hashing.rb
 bundle exec ruby examples/ractor_port_source.rb
+bundle exec ruby examples/ractor_merge_ports_and_map.rb
 bundle exec ruby examples/async_http_requests.rb
+bundle exec ruby examples/async_http_streaming_body.rb
 ```
 
 `examples/backpressure_buffer.rb` prints timestamped producer and consumer
@@ -294,14 +556,23 @@ with a shareable mapper proc and `input_transfer: :move`.
 `examples/ractor_port_source.rb` demonstrates a producer Ractor that waits for
 `RactorPort::Ack` before sending each `RactorPort::Element`.
 
+`examples/ractor_merge_ports_and_map.rb` demonstrates CPU-bound producer
+Ractors merged with `Source.ractor_merge_ports`, followed by CPU-bound
+verification in `ractor_map` workers.
+
 `examples/async_http_requests.rb` starts a local HTTP server and shows
 FiberStream overlapping independent HTTP request waits with `parallel_map`.
 
+`examples/async_http_streaming_body.rb` streams a public nginx access log with
+`async-http`, feeds the response body through `Source.each(response.body)`, and
+aggregates lines without storing the full body.
+
 Benchmark scripts live under `benchmarks/`.
 
 ```sh
 bundle exec ruby benchmarks/stream_transform.rb
 bundle exec ruby benchmarks/latency_overlap.rb
+bundle exec ruby benchmarks/async_io_fanout.rb
 bundle exec ruby benchmarks/heavy_cpu_map.rb
 ```
 

data/examples/README.md

@@ -11,7 +11,9 @@ bundle exec ruby examples/backpressure_buffer.rb
 bundle exec ruby examples/background_execution.rb
 bundle exec ruby examples/ractor_map_hashing.rb
 bundle exec ruby examples/ractor_port_source.rb
+bundle exec ruby examples/ractor_merge_ports_and_map.rb
 bundle exec ruby examples/async_http_requests.rb
+bundle exec ruby examples/async_http_streaming_body.rb
 ```
 
 `basic_pipeline.rb` uses only in-memory values and does not require an async
@@ -46,6 +48,14 @@ pipeline runs.
 `RactorPort::Ack`, and sends one typed `RactorPort::Element` per downstream
 demand.
 
+`ractor_merge_ports_and_map.rb` runs CPU-bound work in multiple producer
+Ractors, merges their port outputs with `Source.ractor_merge_ports`, then runs
+another CPU-bound verification stage with `ractor_map`.
+
 `async_http_requests.rb` starts a local HTTP server and compares serial
 requests with FiberStream `parallel_map` requests. It keeps responses ordered
 while overlapping independent network waits.
+
+`async_http_streaming_body.rb` downloads a public nginx access log with
+`async-http` and streams `response.body` through `Source.each`, `Flow.lines`,
+and `Sink.foreach` so the full HTTP body is not buffered in memory.

data/examples/async_http_streaming_body.rb

@@ -0,0 +1,115 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+
+require "async"
+require "async/http/internet/instance"
+require "fiber_stream"
+
+DEFAULT_URL =
+  "https://raw.githubusercontent.com/elastic/examples/master/" \
+  "Common%20Data%20Formats/nginx_logs/nginx_logs"
+
+URL = ENV.fetch("FIBER_STREAM_HTTP_LOG_URL", DEFAULT_URL)
+PROGRESS_EVERY = Integer(ENV.fetch("FIBER_STREAM_HTTP_PROGRESS_EVERY", "10_000"))
+
+LOG_LINE =
+  /
+    \A
+    (?<remote_addr>\S+)\s+\S+\s+\S+\s+
+    \[[^\]]+\]\s+
+    "[^"]+"\s+
+    (?<status>\d{3})\s+
+    (?<bytes>\d+|-)\s
+  /x
+
+def monotonic_time
+  Process.clock_gettime(Process::CLOCK_MONOTONIC)
+end
+
+def parse_access_log(line)
+  match = LOG_LINE.match(line)
+  return nil unless match
+
+  {
+    remote_addr: match[:remote_addr],
+    status: match[:status],
+    bytes: match[:bytes] == "-" ? 0 : match[:bytes].to_i
+  }
+end
+
+def empty_stats
+  {
+    lines: 0,
+    parsed: 0,
+    payload_bytes: 0,
+    statuses: Hash.new(0),
+    remote_addrs: Hash.new(0),
+    started_at: monotonic_time
+  }
+end
+
+def record_entry(stats, entry)
+  stats[:lines] += 1
+
+  if entry
+    stats[:parsed] += 1
+    stats[:payload_bytes] += entry.fetch(:bytes)
+    stats[:statuses][entry.fetch(:status)] += 1
+    stats[:remote_addrs][entry.fetch(:remote_addr)] += 1
+  end
+
+  if (stats[:lines] % PROGRESS_EVERY).zero?
+    elapsed = monotonic_time - stats.fetch(:started_at)
+    puts format(
+      "processed %<lines>d lines in %<elapsed>.2fs",
+      lines: stats.fetch(:lines),
+      elapsed: elapsed
+    )
+  end
+
+  stats
+end
+
+def print_summary(stats)
+  elapsed = monotonic_time - stats.fetch(:started_at)
+  mib = stats.fetch(:payload_bytes).fdiv(1024 * 1024)
+
+  puts
+  puts "Streaming HTTP body summary"
+  puts "URL: #{URL}"
+  puts format("lines parsed: %<parsed>d/%<lines>d", stats)
+  puts format("logged payload bytes: %<mib>.2f MiB", mib: mib)
+  puts format("unique remote addresses: %<count>d", count: stats.fetch(:remote_addrs).length)
+  puts format("elapsed: %<elapsed>.2fs", elapsed: elapsed)
+
+  puts
+  puts "HTTP status counts"
+  stats.fetch(:statuses).sort.each do |status, count|
+    puts format("- %<status>s: %<count>d", status: status, count: count)
+  end
+end
+
+stats = empty_stats
+
+processed =
+  Sync do
+    Async::HTTP::Internet.get(URL) do |response|
+      unless response.status == 200
+        raise "unexpected HTTP status #{response.status} for #{URL}"
+      end
+
+      FiberStream::Source.each(response.body)
+        .lines(max_length: 16 * 1024)
+        .map { |line| parse_access_log(line) }
+        .run_with(
+          FiberStream::Sink.foreach do |entry|
+            record_entry(stats, entry)
+          end
+        )
+    end
+  end
+
+raise "processed count mismatch" unless processed == stats.fetch(:lines)
+
+print_summary(stats)