durable_streams 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8b7669dbd968851bfab918dd9de9e7f31cc9a936ad5088d4920cc2543adf5737
+  data.tar.gz: c58558467f45deb44f6497c48e119d8e4ec2a7842e6f5dba07c7dc32294594af
+SHA512:
+  metadata.gz: 5420a19b41a0333e82da75a559f1f6f24b475c0367d1cfddf3ccfe6197c23948b577b77fa793fa7c9d59e4c834cefcd499c075023c015c65c49d6720a02a6950
+  data.tar.gz: 9453e5324920d5cfee9c5b0548e4a212ebb9b2490bb3573a4a6fafc69ebc45ecf5e5b33286a7b56acb4ebeb1f8ae2cc674cabd2ccccff18dcde6ff56364148ff

data/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## 0.1.0
+
+- Initial release based on upstream [durable-streams/packages/client-rb](https://github.com/durable-streams/durable-streams/tree/main/packages/client-rb) at commit TBD
+- All source code manually reviewed
+- Published to RubyGems for straightforward installation

data/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) Durable Streams contributors (https://github.com/durable-streams/durable-streams)
+Copyright (c) tokimonki (https://github.com/tokimonki/durable_streams)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,276 @@
+# Durable Streams Ruby Client
+
+A maintained Ruby client for [Durable Streams](https://github.com/durable-streams/durable-streams) — the open protocol for persistent, resumable event streams over HTTP.
+
+Based on the [upstream reference client](https://github.com/durable-streams/durable-streams/tree/main/packages/client-rb) with manual code review, testing, and ongoing maintenance. See [UPSTREAM.md](UPSTREAM.md) for the full sync policy.
+
+## Why this gem exists
+
+The Durable Streams project maintains clients in 10+ languages from a monorepo. The Ruby client is not yet published to RubyGems, which means installing it requires a `git:` + `glob:` directive in your Gemfile pointing into a monorepo subdirectory.
+
+We maintain this standalone copy so that:
+
+- **Installation is simple** — `gem "durable_streams"` in your Gemfile
+- **The code is reviewed** — every line has been manually examined
+- **Upstream changes are tracked** — we sync protocol and bug fixes, documented in the [CHANGELOG](CHANGELOG.md)
+
+If the upstream team publishes the Ruby client to RubyGems, we will coordinate with them to consolidate.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "durable_streams"
+```
+
+For edge (latest from GitHub):
+
+```ruby
+gem "durable_streams", github: "tokimonki/durable_streams"
+```
+
+**Requirements:** Ruby 3.1+
+
+## Quick Start
+
+```ruby
+require "durable_streams"
+
+# Configure once at startup
+DurableStreams.configure do |config|
+  config.base_url = "https://streams.example.com"
+  config.timeout = 30
+  config.default_headers = {
+    "Authorization" => -> { "Bearer #{current_token}" }
+  }
+end
+
+# Create a stream
+stream = DurableStreams.create("/my-stream", content_type: :json)
+
+# Write
+stream << { event: "user.created", user_id: 123 }
+
+# Read
+stream.read.each { |msg| puts msg }
+```
+
+## Writing
+
+```ruby
+stream = DurableStreams.create("/events/orders", content_type: :json)
+
+stream << { order_id: 1, status: "created" }
+stream << { order_id: 2, status: "shipped" }
+
+# With sequence numbers for ordering
+stream.append({ order_id: 3 }, seq: "123")
+```
+
+## Reading
+
+```ruby
+stream = DurableStreams.stream("/events/orders")
+
+# Catch-up: read all existing messages
+stream.each { |event| process(event) }
+
+# With checkpointing
+stream.read(offset: saved_offset).each_batch do |batch|
+  batch.items.each { |event| process(event) }
+  save_offset(batch.next_offset)
+end
+```
+
+## Live Streaming
+
+```ruby
+stream = DurableStreams.stream("/events")
+
+# Long-poll mode
+stream.read(live: :long_poll).each { |msg| process(msg) }
+
+# SSE mode (for JSON/text streams)
+stream.read(live: :sse).each { |msg| puts msg }
+
+# Lazy enumeration
+stream.read(live: :sse).each.lazy.take(10).to_a
+```
+
+## Exactly-Once Writes with Producer
+
+```ruby
+DurableStreams::Producer.open(
+  url: "https://streams.example.com/events",
+  producer_id: "order-service-#{Process.pid}",
+  epoch: 0,
+  auto_claim: true
+) do |producer|
+  1000.times { |i| producer << { order_id: i, status: "created" } }
+end
+```
+
+## Configuration
+
+```ruby
+DurableStreams.configure do |config|
+  config.base_url = ENV["DURABLE_STREAMS_URL"]
+  config.default_content_type = :json
+  config.timeout = 30
+  config.retry_policy = DurableStreams::RetryPolicy.new(
+    max_retries: 5,
+    initial_delay: 0.1,
+    max_delay: 30.0
+  )
+  config.default_headers = {
+    "Authorization" => -> { "Bearer #{refresh_token}" }
+  }
+end
+
+# Reset to defaults (useful in tests)
+DurableStreams.reset_configuration!
+```
+
+### Isolated contexts
+
+For multi-tenant or testing scenarios:
+
+```ruby
+staging = DurableStreams.new_context do |config|
+  config.base_url = "https://staging.example.com"
+end
+
+stream = DurableStreams::Stream.new("/events", context: staging)
+```
+
+## API Reference
+
+### Module methods
+
+```ruby
+DurableStreams.configure { |config| ... }
+DurableStreams.reset_configuration!
+DurableStreams.new_context { |config| ... }
+DurableStreams.stream("/path")
+DurableStreams.create("/path", content_type: :json)
+DurableStreams.append("/path", data)
+DurableStreams.read("/path", offset: "-1")
+```
+
+### Stream
+
+```ruby
+stream = DurableStreams.stream("/events")
+
+# Metadata
+stream.head         # => HeadResult
+stream.exists?      # => true/false
+stream.json?        # => true/false
+
+# Writing
+stream.append(data, seq: nil)   # => AppendResult
+stream << data                  # => self (chainable)
+
+# Reading
+stream.each { |msg| ... }
+stream.read(offset: "-1", live: false, format: :auto)  # => Reader
+stream.read_all(offset: "-1")                           # => Array
+
+# Lifecycle
+stream.create_stream(content_type:, ttl_seconds: nil, expires_at: nil)
+stream.delete
+stream.close
+```
+
+### Read options
+
+| `live` mode  | Behavior |
+|---|---|
+| `false` | Return when caught up (catch-up only) |
+| `:long_poll` | Wait for new data or timeout |
+| `:sse` | Server-Sent Events with automatic reconnection |
+
+| `format` | Behavior |
+|---|---|
+| `:auto` | Detect from Content-Type header |
+| `:json` | Force JSON parsing (JsonReader) |
+| `:bytes` | Force raw bytes (ByteReader) |
+
+### Producer
+
+```ruby
+DurableStreams::Producer.open(url: "...", producer_id: "...") do |producer|
+  producer << data
+end
+
+# Manual form
+producer = DurableStreams::Producer.new(
+  url: "...",
+  producer_id: "unique-id",
+  epoch: 0,
+  auto_claim: false,
+  max_batch_bytes: 1_048_576,
+  linger_ms: 5,
+  max_in_flight: 5
+)
+
+producer.append(data)     # Fire-and-forget (batched)
+producer << data          # Same as append
+producer.append!(data)    # Wait for acknowledgment => ProducerResult
+producer.flush
+producer.close
+```
+
+### Errors
+
+All errors inherit from `DurableStreams::Error`:
+
+| Error | HTTP Status | Meaning |
+|---|---|---|
+| `StreamNotFoundError` | 404 | Stream doesn't exist |
+| `StreamExistsError` | 409 | Already exists with different config |
+| `SeqConflictError` | 409 | Sequence number conflict |
+| `ContentTypeMismatchError` | 409 | Wrong content type |
+| `StaleEpochError` | 403 | Producer epoch is stale |
+| `SequenceGapError` | 409 | Producer sequence gap |
+| `RateLimitedError` | 429 | Rate limited |
+| `BadRequestError` | 400 | Invalid request |
+| `ConnectionError` | — | Network error |
+| `TimeoutError` | — | Request timeout |
+
+## Testing
+
+```ruby
+require "durable_streams/testing"
+
+# Setup
+DurableStreams::Testing.install!
+DurableStreams::Testing.clear!
+
+# Seed and use
+DurableStreams::Testing.mock_transport.seed_stream("/events", [])
+stream = DurableStreams.create("/events", content_type: :json)
+stream.append({ event: "test" })
+
+# Verify
+messages = DurableStreams::Testing.messages_for("/events")
+
+# Teardown
+DurableStreams::Testing.reset!
+```
+
+## Thread safety
+
+- **Configuration** — thread-safe after freeze (configure at startup)
+- **Stream** — create one per thread for concurrent reads
+- **Producer** — thread-safe, uses mutex internally
+- **Readers** — single-threaded (create new reader per thread)
+
+## Upstream
+
+This gem is based on the reference Ruby client from the [Durable Streams](https://github.com/durable-streams/durable-streams) project (`packages/client-rb/`). See [UPSTREAM.md](UPSTREAM.md) for the sync policy and process.
+
+## License
+
+MIT — see [LICENSE](LICENSE)

data/UPSTREAM.md

@@ -0,0 +1,85 @@
+# Upstream Sync
+
+This gem is based on the Ruby client from the [Durable Streams](https://github.com/durable-streams/durable-streams) project. The upstream code lives at `packages/client-rb/` in their monorepo.
+
+We publish this as a standalone gem because the upstream team maintains clients in 10+ languages and has not yet published the Ruby client to RubyGems. Since we depend on this client for `durable_streams-rails`, we maintain a reviewed, tested copy here.
+
+## Relationship with upstream
+
+- **We coordinate, not compete.** If upstream publishes `durable_streams` to RubyGems, we will work with them to consolidate — either transferring gem ownership or deprecating this fork.
+- **We attribute.** The LICENSE includes the original copyright. Every CHANGELOG entry that pulls upstream changes links to the specific commits.
+- **We contribute back.** Bugs found here get reported upstream. Fixes that apply to the reference client get submitted as PRs to the upstream repo.
+
+## Sync process
+
+### Checking for upstream changes
+
+```bash
+# Add upstream as a remote (one-time setup)
+git remote add upstream-ref https://github.com/durable-streams/durable-streams.git
+
+# Fetch latest
+git fetch upstream-ref main
+
+# Compare the Ruby client directory against our lib/
+git diff HEAD upstream-ref/main -- packages/client-rb/lib/
+```
+
+Alternatively, review the upstream commit log filtered to the Ruby client:
+
+```bash
+git log upstream-ref/main -- packages/client-rb/
+```
+
+### Applying upstream changes
+
+1. **Review each upstream commit individually.** Don't blindly merge. The upstream client may include changes we've already fixed differently, or patterns we've deliberately diverged from.
+
+2. **For each relevant change, create a branch:**
+   ```bash
+   git checkout -b sync/upstream-YYYY-MM-DD
+   ```
+
+3. **Cherry-pick or manually apply** the changes. Prefer manual application — it forces you to understand what changed.
+
+4. **Run the full test suite** before merging.
+
+5. **Document in CHANGELOG.md** with a link to the upstream commit:
+   ```
+   ## x.y.z
+
+   - Sync: applied upstream fix for SSE reconnection (durable-streams/durable-streams@abc1234)
+   ```
+
+### What to sync
+
+- **Bug fixes** — always pull these in after review
+- **Protocol compliance changes** — required to stay compatible with the server
+- **New features** — evaluate case by case; only pull in what we need or what improves the client
+- **Refactors** — evaluate carefully; our codebase may have diverged structurally
+
+### What NOT to sync
+
+- **Conformance adapter** (`conformance_adapter.rb`) — this is upstream's test harness, not part of the client
+- **Design docs** (`design.md`) — internal upstream documentation
+- **Style changes** — we follow our own conventions where they diverge
+
+## Sync schedule
+
+Check upstream monthly, or whenever a new protocol version is released. Pin the upstream commit hash in CHANGELOG.md so we always know our baseline.
+
+## RubyGems coordination
+
+The upstream team has an open issue to publish the Ruby client to RubyGems:
+https://github.com/durable-streams/durable-streams/issues/166
+
+As of February 2026, this issue has no activity beyond the initial filing. If upstream
+publishes their own `durable_streams` gem, we will coordinate with them — either
+transferring ownership of this gem or deprecating it in favor of theirs.
+
+## Current baseline
+
+- **Upstream repo:** https://github.com/durable-streams/durable-streams
+- **Upstream path:** `packages/client-rb/`
+- **Baseline commit:** TBD (fill in when publishing)
+- **Baseline date:** TBD

data/lib/durable_streams/byte_reader.rb

@@ -0,0 +1,185 @@
+# frozen_string_literal: true
+
+require "base64"
+
+module DurableStreams
+  # Reader for byte streams - yields raw chunks
+  class ByteReader
+    attr_reader :next_offset, :cursor, :up_to_date, :status
+
+    # @param stream [Stream] Parent stream handle
+    # @param offset [String] Starting offset
+    # @param live [Symbol, false] Live mode (:long_poll, :sse, false)
+    # @param cursor [String, nil] Initial cursor
+    def initialize(stream, offset: "-1", live: false, cursor: nil)
+      @stream = stream
+      @offset = DurableStreams.normalize_offset(offset)
+      @live = live
+      @next_offset = @offset
+      @cursor = cursor
+      @up_to_date = false
+      @closed = false
+      @status = nil
+      @sse_reader = nil
+    end
+
+    # Iterate over byte chunks
+    # @yield [ByteChunk] Each chunk with data, next_offset, cursor, up_to_date
+    def each(&block)
+      return enum_for(:each) unless block_given?
+
+      # Handle SSE mode
+      if @live == :sse
+        each_sse(&block)
+        return
+      end
+
+      loop do
+        break if @closed
+
+        chunk = fetch_next_chunk
+        break if chunk.nil?
+
+        @next_offset = chunk.next_offset
+        @cursor = chunk.cursor
+        @up_to_date = chunk.up_to_date
+
+        yield chunk
+
+        # Break for non-live modes when up_to_date
+        break if @live == false && @up_to_date
+        # Break for long-poll on 204 timeout (up_to_date with empty data = no new data)
+        break if @live == :long_poll && @up_to_date && chunk.data.empty? && @status == 204
+        break if @closed
+      end
+    end
+
+    # Accumulate all bytes until up_to_date
+    # @return [String]
+    def body
+      chunks = []
+      each { |chunk| chunks << chunk.data }
+      chunks.join
+    end
+
+    # Get as text
+    # @return [String]
+    def text
+      body.encode("UTF-8")
+    end
+
+    # Collect chunks as array
+    def to_a
+      result = []
+      each { |chunk| result << chunk }
+      result
+    end
+
+    # Cancel/close the reader
+    def close
+      @closed = true
+      @sse_reader&.close
+    end
+
+    def closed?
+      @closed
+    end
+
+    def up_to_date?
+      @up_to_date
+    end
+
+    private
+
+    def each_sse(&block)
+      @sse_reader = SSEReader.new(
+        @stream,
+        offset: @next_offset,
+        cursor: @cursor
+      )
+
+      @sse_reader.each_event do |event|
+        break if @closed
+
+        @next_offset = event[:next_offset] if event[:next_offset]
+        @cursor = event[:cursor] if event[:cursor]
+        @up_to_date = event[:up_to_date]
+        @status = 200
+
+        # Only yield if there's data
+        if event[:data] && !event[:data].empty?
+          chunk = ByteChunk.new(
+            data: event[:data],
+            next_offset: @next_offset,
+            cursor: @cursor,
+            up_to_date: @up_to_date
+          )
+          yield chunk
+        elsif event[:up_to_date]
+          # Yield empty chunk on control event with up_to_date
+          chunk = ByteChunk.new(
+            data: "",
+            next_offset: @next_offset,
+            cursor: @cursor,
+            up_to_date: @up_to_date
+          )
+          yield chunk
+        end
+      end
+    ensure
+      @sse_reader&.close
+    end
+
+    def fetch_next_chunk
+      params = { offset: @next_offset }
+      params[:cursor] = @cursor if @cursor
+
+      # Add live mode parameter
+      case @live
+      when :long_poll
+        params[:live] = "long-poll"
+      when :sse
+        # SSE is handled separately via each_sse
+        return nil
+      when false
+        # No live param for catch-up only
+      end
+
+      request_url = HTTP.build_url(@stream.url, params)
+      headers = @stream.resolved_headers
+
+      response = @stream.transport.request(:get, request_url, headers: headers)
+      @status = response.status
+
+      if response.status == 404
+        raise StreamNotFoundError.new(url: @stream.url)
+      end
+
+      # Handle 204 No Content (long-poll timeout)
+      # Still parse headers as they contain offset info
+      if response.status == 204
+        @next_offset = response[STREAM_NEXT_OFFSET_HEADER] || @next_offset
+        @cursor = response[STREAM_CURSOR_HEADER] || @cursor
+        @up_to_date = true
+        return ByteChunk.new(data: "", next_offset: @next_offset, cursor: @cursor, up_to_date: true)
+      end
+
+      unless response.success?
+        raise DurableStreams.error_from_status(response.status, url: @stream.url, body: response.body,
+                                               headers: response.headers)
+      end
+
+      headers = DurableStreams.parse_stream_headers(response, next_offset: @next_offset, cursor: @cursor)
+      @next_offset = headers[:next_offset]
+      @cursor = headers[:cursor]
+      @up_to_date = headers[:up_to_date]
+
+      ByteChunk.new(
+        data: response.body || "",
+        next_offset: @next_offset,
+        cursor: @cursor,
+        up_to_date: @up_to_date
+      )
+    end
+  end
+end

data/lib/durable_streams/client.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module DurableStreams
+  # Client manages HTTP connections and provides stream handles.
+  # Creates a Context internally for configuration isolation.
+  # Thread-safe for concurrent use.
+  class Client
+    attr_reader :context
+
+    # Open a client with block form for automatic cleanup
+    # @example
+    #   Client.open(base_url: "https://...") do |client|
+    #     stream = client.stream("/events")
+    #     # ...
+    #   end # auto-closes
+    # @yield [Client] The client instance
+    # @return [Object] The block's return value
+    def self.open(**options, &block)
+      client = new(**options)
+      return client unless block_given?
+
+      begin
+        yield client
+      ensure
+        client.close
+      end
+    end
+
+    # @param base_url [String, nil] Optional base URL for relative paths
+    # @param headers [Hash] Default headers (values can be strings or callables)
+    # @param timeout [Numeric] Request timeout in seconds
+    # @param retry_policy [RetryPolicy] Custom retry configuration
+    def initialize(base_url: nil, headers: {}, timeout: 30, retry_policy: nil)
+      @context = Context.new do |config|
+        config.base_url = base_url
+        config.default_headers = headers || {}
+        config.timeout = timeout
+        config.retry_policy = retry_policy if retry_policy
+      end
+    end
+
+    # Get a Stream handle for the given URL
+    # @param url [String] Full URL or path (if base_url set)
+    # @return [Stream]
+    def stream(url, **options)
+      Stream.new(url, context: @context, **options)
+    end
+
+    # Shortcut: connect to existing stream
+    # @param url [String] Stream URL or path
+    # @return [Stream]
+    def connect(url, **options)
+      Stream.connect(url, context: @context, **options)
+    end
+
+    # Shortcut: create new stream on server
+    # @param url [String] Stream URL or path
+    # @param content_type [String] Content type for the stream
+    # @return [Stream]
+    def create(url, content_type:, **options)
+      Stream.create(url, content_type: content_type, context: @context, **options)
+    end
+
+    # No-op close (Context doesn't hold resources)
+    def close
+    end
+  end
+end

data/lib/durable_streams/configuration.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module DurableStreams
+  # Configuration holds all settings for DurableStreams.
+  # Thread-safe when frozen (after configure block completes).
+  class Configuration
+    attr_accessor :base_url, :default_content_type, :default_headers,
+                  :timeout, :retry_policy
+
+    def initialize
+      @base_url = nil
+      @default_content_type = :json
+      @default_headers = {}
+      @timeout = 30
+      @retry_policy = RetryPolicy.default
+    end
+
+    # Deep dup nested mutable objects when copying.
+    # Called automatically by dup/clone.
+    def initialize_copy(other)
+      super
+      @default_headers = other.default_headers.dup
+      @retry_policy = other.retry_policy.dup if other.retry_policy.respond_to?(:dup) && !other.retry_policy.frozen?
+    end
+  end
+end