fiber_stream 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ab4449558105eb57805e40970a28971e32a37e748176df0875de75816bf53d2c
+  data.tar.gz: 6d1998ce477a4602a6f23d60817f2f49b9acc6efde64b18ee90ebd7084905bd5
+SHA512:
+  metadata.gz: 4242c44baa5c1db6f7cb6c39ac217729603838dba31876180f4fcc2382e72308cad4d5197d8687b522317dfc9ad964bcd6f85b9449f580f88324098536b8906c
+  data.tar.gz: c2cc8091ec14b27eef4a60a82c1c22503118eee9da0a1a53e24a3e137058e1dd8c5e1536e89c5ba13c7f725cdc0ad241d48ade86cbe2162f6ac5ae2d9ce8f96f

data/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+## 0.1.0 - 2026-06-03
+
+Initial release.
+
+### Added
+
+- Lazy linear `Source`, `Flow`, `Sink`, and `Pipeline` APIs.
+- Pull-based backpressure with `Source.each`, `Flow.map`, `Flow.select`,
+  `Flow.take`, `Sink.to_a`, `Sink.first`, and `Sink.fold`.
+- Scheduler-aware IO source and sink support.
+- Line framing with `Flow.lines`.
+- Scheduler-backed async and bounded buffer boundaries.
+- Ordered `Flow.parallel_map` and `Flow.ractor_map`.
+- Backpressure-aware `Source.ractor_port` with typed Ractor protocol envelopes.
+- Background pipeline execution with cancellation support.
+- Public RBS signatures.
+
+### Known Limitations
+
+- Only linear pipelines are supported.
+- IO and scheduler-backed stages require the caller to provide a Ruby
+  `Fiber.scheduler`; FiberStream does not install one.
+- Ractor APIs are experimental in Ruby and may change in future Ruby releases.

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2026 Dai Akatsuka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,361 @@
+# FiberStream
+
+FiberStream is an early-stage Ruby library for linear, pull-based stream
+processing with backpressure.
+
+Build a lazy `Source`, transform it with `Flow` stages, and materialize it with
+a `Sink`.
+
+## Quick Start
+
+```ruby
+require "fiber_stream"
+
+result =
+  FiberStream::Source.each([1, 2, 3, 4])
+    .map { |number| number * 2 }
+    .select(&:even?)
+    .take(2)
+    .run_with(FiberStream::Sink.to_a)
+
+result # => [2, 4]
+```
+
+## Status
+
+FiberStream currently supports linear pipelines only.
+
+Implemented capabilities:
+
+- in-memory, IO, and backpressure-aware Ractor port sources
+- mapping, filtering, limiting, line splitting, buffering, async boundaries,
+  ordered parallel mapping, and ordered Ractor-backed mapping
+- array, first-element, fold, and IO sinks
+- reusable flow composition and runnable pipelines
+- foreground and scheduler-backed background pipeline execution
+- public RBS signatures
+
+Not yet implemented:
+
+- graph DSLs
+
+## Core Concepts
+
+### Sources
+
+A `Source` is a lazy stream definition. It is not consumed until the source is
+run with a sink.
+
+```ruby
+source = FiberStream::Source.each([1, 2, 3])
+
+source.run_with(FiberStream::Sink.to_a) # => [1, 2, 3]
+```
+
+IO sources read chunks on demand and require a scheduler-backed non-blocking
+fiber:
+
+```ruby
+require "async"
+require "fiber_stream"
+
+chunks =
+  Async do
+    File.open("input.txt", "rb") do |file|
+      FiberStream::Source.io(file)
+        .run_with(FiberStream::Sink.to_a)
+    end
+  end.wait
+```
+
+Ractor port sources connect a producer Ractor with an explicit ack handshake.
+The producer creates its acknowledgment port, waits for `RactorPort::Ack`, and
+then sends one typed message back to the FiberStream data port:
+
+```ruby
+data_port = Ractor::Port.new
+setup_port = Ractor::Port.new
+
+producer =
+  Ractor.new(data_port, setup_port) do |outbox, setup|
+    ack_port = Ractor::Port.new
+    setup.send(ack_port)
+
+    values = [1, 2, 3].to_enum
+
+    loop do
+      case ack_port.receive
+      in FiberStream::RactorPort::Ack
+        begin
+          outbox.send(FiberStream::RactorPort::Element.new(values.next))
+        rescue StopIteration
+          outbox.send(FiberStream::RactorPort::Complete.new)
+          break
+        end
+      in FiberStream::RactorPort::Cancel
+        break
+      end
+    end
+  end
+
+ack_port = setup_port.receive
+
+FiberStream::Source.ractor_port(data_port, ack_port: ack_port)
+  .run_with(FiberStream::Sink.to_a)
+# => [1, 2, 3]
+
+producer.value
+```
+
+### Flows
+
+Flows transform a stream lazily. Convenience methods on `Source` delegate to
+the matching `FiberStream::Flow` constructors.
+
+```ruby
+result =
+  FiberStream::Source.each(["a\nb", "\nc"])
+    .lines
+    .select { |line| line != "b" }
+    .map(&:upcase)
+    .run_with(FiberStream::Sink.to_a)
+
+result # => ["A", "C"]
+```
+
+Reusable flows can be composed with `Flow#via`:
+
+```ruby
+normalize =
+  FiberStream::Flow.map(&:strip)
+    .via(FiberStream::Flow.select { |line| !line.empty? })
+
+FiberStream::Source.each([" a ", "", " b "])
+  .via(normalize)
+  .run_with(FiberStream::Sink.to_a)
+# => ["a", "b"]
+```
+
+### Sinks
+
+A `Sink` consumes the stream and returns a materialized value.
+
+```ruby
+FiberStream::Source.each([1, 2, 3])
+  .run_with(FiberStream::Sink.fold(0) { |sum, value| sum + value })
+# => 6
+```
+
+### Pipelines
+
+`Source#to(sink)` creates a reusable runnable pipeline.
+
+```ruby
+pipeline =
+  FiberStream::Source.each([1, 2, 3])
+    .map { |number| number * 2 }
+    .to(FiberStream::Sink.to_a)
+
+pipeline.run # => [2, 4, 6]
+```
+
+`Pipeline#run_async` starts a pipeline in a scheduler-backed background fiber
+and returns a handle:
+
+```ruby
+require "async"
+require "fiber_stream"
+
+result =
+  Async do
+    running =
+      FiberStream::Source.each([1, 2, 3])
+        .map { |number| number * 2 }
+        .to(FiberStream::Sink.to_a)
+        .run_async
+
+    # Foreground scheduler-managed work can continue here.
+
+    running.wait
+  end.wait
+
+result # => [2, 4, 6]
+```
+
+The handle supports `wait`, `cancel`, `done?`, and `cancel_requested?`.
+
+## Backpressure
+
+The initial runtime is pull-based. A sink asks for one element, each flow pulls
+only what it needs from upstream, and the source advances only when downstream
+demands a value.
+
+`Sink.first` demonstrates sink-side early completion:
+
+```ruby
+first =
+  FiberStream::Source.each([1, 2, 3])
+    .run_with(FiberStream::Sink.first)
+
+first # => 1
+```
+
+`Flow.take` demonstrates flow-side early completion and closes upstream after
+the requested number of elements:
+
+```ruby
+limited =
+  FiberStream::Source.each([1, 2, 3])
+    .take(2)
+    .run_with(FiberStream::Sink.to_a)
+
+limited # => [1, 2]
+```
+
+`Flow.buffer(count)` allows bounded prefetch. `Flow.async`, `Flow.buffer`,
+`Flow.parallel_map`, `Source.io`, `Sink.io`, and `Pipeline#run_async` require an
+installed `Fiber.scheduler` and a non-blocking current fiber when demanded or
+started. FiberStream does not install a scheduler and does not depend on Async
+at runtime.
+
+## API Surface
+
+Sources:
+
+- `FiberStream::Source.each(enumerable)`
+- `FiberStream::Source.io(io, chunk_size: 16 * 1024, close: false)`
+- `FiberStream::Source.ractor_port(port, ack_port:, ack_transfer: :copy, cancel: true)`
+
+Source convenience methods:
+
+- `Source#via(flow)`
+- `Source#map { |element| ... }`
+- `Source#parallel_map(concurrency:) { |element| ... }`
+- `Source#ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
+- `Source#select { |element| ... }`
+- `Source#take(count)`
+- `Source#async`
+- `Source#buffer(count)`
+- `Source#lines(chomp: true, max_length: nil)`
+- `Source#to(sink)`
+- `Source#run_with(sink)`
+
+Flows:
+
+- `FiberStream::Flow.map { |element| ... }`
+- `FiberStream::Flow.parallel_map(concurrency:) { |element| ... }`
+- `FiberStream::Flow.ractor_map(workers:, input_transfer: :copy, output_transfer: :copy) { |element| ... }`
+- `FiberStream::Flow.select { |element| ... }`
+- `FiberStream::Flow.take(count)`
+- `FiberStream::Flow.async`
+- `FiberStream::Flow.buffer(count)`
+- `FiberStream::Flow.lines(chomp: true, max_length: nil)`
+- `Flow#via(flow)`
+- `Flow#to(sink)`
+
+Sinks:
+
+- `FiberStream::Sink.to_a`
+- `FiberStream::Sink.first`
+- `FiberStream::Sink.fold(initial) { |accumulator, element| ... }`
+- `FiberStream::Sink.io(io, close: false, flush: false)`
+
+Pipelines:
+
+- `FiberStream::Pipeline#run`
+- `FiberStream::Pipeline#run_async`
+- `FiberStream::RunningPipeline#wait`
+- `FiberStream::RunningPipeline#cancel`
+- `FiberStream::RunningPipeline#done?`
+- `FiberStream::RunningPipeline#cancel_requested?`
+
+## Examples
+
+Runnable examples live under `examples/`.
+
+```sh
+bundle exec ruby examples/basic_pipeline.rb
+bundle exec ruby examples/composable_pipeline.rb
+bundle exec ruby examples/line_processing.rb
+bundle exec ruby examples/file_copy.rb
+bundle exec ruby examples/backpressure_buffer.rb
+bundle exec ruby examples/background_execution.rb
+bundle exec ruby examples/ractor_map_hashing.rb
+bundle exec ruby examples/ractor_port_source.rb
+bundle exec ruby examples/async_http_requests.rb
+```
+
+`examples/backpressure_buffer.rb` prints timestamped producer and consumer
+events so the difference between direct demand and bounded prefetch is visible.
+
+`examples/ractor_map_hashing.rb` demonstrates ordered Ractor-backed hashing
+with a shareable mapper proc and `input_transfer: :move`.
+
+`examples/ractor_port_source.rb` demonstrates a producer Ractor that waits for
+`RactorPort::Ack` before sending each `RactorPort::Element`.
+
+`examples/async_http_requests.rb` starts a local HTTP server and shows
+FiberStream overlapping independent HTTP request waits with `parallel_map`.
+
+Benchmark scripts live under `benchmarks/`.
+
+```sh
+bundle exec ruby benchmarks/stream_transform.rb
+bundle exec ruby benchmarks/latency_overlap.rb
+bundle exec ruby benchmarks/heavy_cpu_map.rb
+```
+
+## Development
+
+This project targets Ruby 4.x. The repository currently pins Ruby 4.0.3 in
+`mise.toml`.
+
+Install dependencies:
+
+```sh
+bundle install
+```
+
+Run the test suite:
+
+```sh
+bundle exec rake test
+```
+
+Run RBS validation:
+
+```sh
+bundle exec rbs validate
+```
+
+Run RuboCop:
+
+```sh
+bundle exec rubocop
+```
+
+Run all default checks:
+
+```sh
+bundle exec rake
+```
+
+Build the gem:
+
+```sh
+bundle exec gem build fiber_stream.gemspec
+```
+
+Release uses RubyGems Trusted Publishing from the `Release` GitHub Actions
+workflow. Configure a pending trusted publisher for the `fiber_stream` gem with
+workflow filename `release.yml` and environment `release`, then publish by
+pushing a version tag such as `v0.1.0`.
+
+## Documentation
+
+Design and planning documents live under `docs/`:
+
+- `docs/product-specs/`
+- `docs/design-docs/`
+- `docs/exec-plans/`
+- `docs/references/`

data/examples/README.md

@@ -0,0 +1,51 @@
+# FiberStream Examples
+
+Run examples from the repository root with Bundler:
+
+```sh
+bundle exec ruby examples/basic_pipeline.rb
+bundle exec ruby examples/composable_pipeline.rb
+bundle exec ruby examples/line_processing.rb
+bundle exec ruby examples/file_copy.rb
+bundle exec ruby examples/backpressure_buffer.rb
+bundle exec ruby examples/background_execution.rb
+bundle exec ruby examples/ractor_map_hashing.rb
+bundle exec ruby examples/ractor_port_source.rb
+bundle exec ruby examples/async_http_requests.rb
+```
+
+`basic_pipeline.rb` uses only in-memory values and does not require an async
+scheduler.
+
+`composable_pipeline.rb` demonstrates reusable flow pipelines, sink
+composition, and runnable `Source#to(...).run` pipelines.
+
+`line_processing.rb` demonstrates `Source#lines` over arbitrary String chunks.
+
+`file_copy.rb` uses Ruby core `File` objects with `Source.io` and `Sink.io`.
+IO examples require a scheduler-backed non-blocking fiber, so they run inside
+an `Async do ... end.wait` block provided by the `async` gem.
+
+`backpressure_buffer.rb` prints timestamped producer and consumer events. The
+unbuffered run stays demand-driven, while the buffered run allows bounded
+prefetch. The `produced_ahead` counter includes queued values plus in-flight
+producer and consumer work, so it can be larger than the configured queue size
+without becoming unbounded.
+
+`background_execution.rb` starts a runnable pipeline with `Pipeline#run_async`
+and uses the returned handle to wait for the background materialized value while
+the foreground fiber keeps doing scheduler-managed work.
+
+`ractor_map_hashing.rb` hashes independent payloads in Ractor workers. It uses
+`Ractor.shareable_proc`, preserves input order, and opts into
+`input_transfer: :move` because the input records are not reused after the
+pipeline runs.
+
+`ractor_port_source.rb` demonstrates a producer Ractor connected to
+`Source.ractor_port`. The producer creates its acknowledgment port, waits for
+`RactorPort::Ack`, and sends one typed `RactorPort::Element` per downstream
+demand.
+
+`async_http_requests.rb` starts a local HTTP server and compares serial
+requests with FiberStream `parallel_map` requests. It keeps responses ordered
+while overlapping independent network waits.

data/examples/async_http_requests.rb

@@ -0,0 +1,132 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+
+require "async"
+require "fiber_stream"
+require "socket"
+
+ROUTES = {
+  "/profile" => [0.18, "profile loaded"],
+  "/orders" => [0.12, "orders loaded"],
+  "/recommendations" => [0.16, "recommendations loaded"]
+}.freeze
+
+Endpoint = Struct.new(:path, :label, keyword_init: true)
+
+def monotonic_time
+  Process.clock_gettime(Process::CLOCK_MONOTONIC)
+end
+
+def run_server(server)
+  loop do
+    socket = server.accept
+    Async { handle_connection(socket) }
+  rescue IOError, Errno::EBADF
+    break
+  end
+end
+
+def handle_connection(socket)
+  request = socket.readpartial(1024)
+  path = request[/GET\s+(\S+)/, 1] || "/"
+  delay, body = ROUTES.fetch(path, [0.02, "not found"])
+  status = ROUTES.key?(path) ? "200 OK" : "404 Not Found"
+
+  sleep delay
+
+  socket.write(
+    "HTTP/1.1 #{status}\r\n" \
+    "Content-Type: text/plain\r\n" \
+    "Content-Length: #{body.bytesize}\r\n" \
+    "Connection: close\r\n\r\n" \
+    "#{body}"
+  )
+ensure
+  socket&.close
+end
+
+def http_get(host, port, endpoint)
+  response = +""
+  started_at = monotonic_time
+  socket = TCPSocket.new(host, port)
+
+  socket.write(
+    "GET #{endpoint.path} HTTP/1.1\r\n" \
+    "Host: #{host}:#{port}\r\n" \
+    "Connection: close\r\n\r\n"
+  )
+
+  loop do
+    response << socket.readpartial(1024)
+  rescue EOFError
+    break
+  end
+
+  headers, body = response.split("\r\n\r\n", 2)
+  status = headers.lines.first.split.fetch(1)
+  elapsed = monotonic_time - started_at
+
+  {
+    label: endpoint.label,
+    status: status,
+    body: body,
+    elapsed: elapsed
+  }
+ensure
+  socket&.close
+end
+
+def measure(label)
+  started_at = monotonic_time
+  result = yield
+  elapsed = monotonic_time - started_at
+  [label, elapsed, result]
+end
+
+def print_run(label, elapsed, responses)
+  puts "#{label}: #{format('%.3f', elapsed)}s"
+  responses.each do |response|
+    puts format(
+      "- %-15<label>s status=%<status>s request=%<elapsed>.3fs body=%<body>s",
+      label: response.fetch(:label),
+      status: response.fetch(:status),
+      elapsed: response.fetch(:elapsed),
+      body: response.fetch(:body)
+    )
+  end
+  puts
+end
+
+endpoints = [
+  Endpoint.new(path: "/profile", label: "profile"),
+  Endpoint.new(path: "/orders", label: "orders"),
+  Endpoint.new(path: "/recommendations", label: "recommendations")
+]
+
+Sync do
+  server = TCPServer.new("127.0.0.1", 0)
+  host = "127.0.0.1"
+  port = server.addr.fetch(1)
+  server_task = Async { run_server(server) }
+
+  begin
+    serial = measure("Serial HTTP requests") do
+      endpoints.map { |endpoint| http_get(host, port, endpoint) }
+    end
+
+    parallel = measure("FiberStream parallel HTTP requests") do
+      FiberStream::Source.each(endpoints)
+        .parallel_map(concurrency: endpoints.length) { |endpoint| http_get(host, port, endpoint) }
+        .run_with(FiberStream::Sink.to_a)
+    end
+
+    print_run(*serial)
+    print_run(*parallel)
+
+    puts "Serial waits add up. FiberStream starts all requests together and keeps the responses ordered."
+  ensure
+    server.close
+    server_task.stop
+  end
+end

data/examples/background_execution.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+
+require "async"
+require "fiber_stream"
+
+jobs = [
+  { id: "import", delay: 0.12 },
+  { id: "validate", delay: 0.05 },
+  { id: "publish", delay: 0.08 }
+]
+
+Async do
+  running =
+    FiberStream::Source.each(jobs)
+      .parallel_map(concurrency: 2) do |job|
+        sleep job.fetch(:delay)
+        "#{job.fetch(:id)} complete"
+      end
+      .to(FiberStream::Sink.to_a)
+      .run_async
+
+  3.times do |tick|
+    sleep 0.03
+    puts "foreground tick #{tick + 1}"
+  end
+
+  puts "Background result"
+  running.wait.each { |line| puts "- #{line}" }
+end.wait

data/examples/backpressure_buffer.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+
+require "async"
+require "fiber_stream"
+
+ITEM_COUNT = 8
+BUFFER_SIZE = 3
+PRODUCER_DELAY = 0.05
+CONSUMER_DELAY = 0.20
+
+def run_pipeline(label, buffer_size: nil)
+  started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  produced = 0
+  consumed = 0
+  max_produced_ahead = 0
+
+  log = lambda do |event, item|
+    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
+    produced_ahead = produced - consumed
+    max_produced_ahead = [max_produced_ahead, produced_ahead].max
+
+    puts format(
+      "%<elapsed>6.2fs  %-10<event>s item=%<item>2d produced_ahead=%<ahead>d",
+      elapsed: elapsed,
+      event: event,
+      item: item,
+      ahead: produced_ahead
+    )
+  end
+
+  source =
+    FiberStream::Source.each(1..ITEM_COUNT)
+      .map do |item|
+        produced += 1
+        log.call("produce", item)
+        sleep PRODUCER_DELAY
+        item
+      end
+
+  source = source.buffer(buffer_size) if buffer_size
+
+  result =
+    source.run_with(
+      FiberStream::Sink.fold([]) do |items, item|
+        log.call("consume", item)
+        sleep CONSUMER_DELAY
+        consumed += 1
+        items << item
+      end
+    )
+
+  puts "#{label}: result=#{result.inspect}"
+  puts "#{label}: max produced ahead=#{max_produced_ahead}"
+  puts
+end
+
+Async do
+  puts "Unbuffered: downstream demand gates upstream one item at a time."
+  run_pipeline("unbuffered")
+
+  puts "buffer(#{BUFFER_SIZE}): upstream can prefetch, but backpressure keeps it bounded."
+  puts "produced_ahead includes queued values plus producer/consumer in-flight work."
+  run_pipeline("buffered", buffer_size: BUFFER_SIZE)
+end.wait