omq-qos 0.3.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b322c352350486881a5e55890321584ab7ada43df692853909dab67ecc54459e
+  data.tar.gz: b6c35d6dc9230d79f0478cd73502c4a0a7591b291b21d27205ebe0c2f728b5e8
+SHA512:
+  metadata.gz: 447a1787214ede18698fb24c91714dcd8b08baf2ffe28e0e1ba43cfa2722f9f4146415a5c32b8ef4648db86a43049d21cf8b2a888b5294269efff6b216eddf31
+  data.tar.gz: aa9c899206ca43d8791110cb2571d6ee7ba6e14981e06e643852c640939061ab78efd1a997a65ecd913e60e2f14d8df791f227bb87d9680929e602dfc28b6f4f

data/LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2026, Patrik Wenger
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

data/README.md

@@ -0,0 +1,101 @@
+# OMQ::QoS -- Delivery Guarantees for OMQ
+
+[![CI](https://github.com/zeromq/omq.rb/actions/workflows/ci.yml/badge.svg)](https://github.com/zeromq/omq.rb/actions/workflows/ci.yml)
+[![Gem Version](https://img.shields.io/gem/v/omq-qos?color=e9573f)](https://rubygems.org/gems/omq-qos)
+[![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+
+Per-hop delivery guarantees for [OMQ](https://github.com/zeromq/omq.rb),
+inspired by MQTT QoS levels. Adds ACK-based at-least-once delivery using
+xxHash message identification.
+
+Experimental. Wire details and retry semantics may change before stable
+release. Use only when you can tolerate protocol churn.
+
+```ruby
+require "omq"
+require "omq/qos"
+
+push = OMQ::PUSH.new(nil, qos: 1)
+push.connect("tcp://worker-1:5555")
+push.connect("tcp://worker-2:5555")
+push << "reliably delivered"
+# If worker-1 dies, unacked messages retry on worker-2.
+```
+
+## QoS Levels
+
+| Level | Name            | Behavior                                      |
+|-------|-----------------|-----------------------------------------------|
+| 0     | Fire-and-forget | Default ZMQ behavior (no overhead)             |
+| 1     | At-least-once   | Receiver ACKs, sender retries on connection loss |
+
+## How it works
+
+`require "omq/qos"` prepends onto OMQ routing strategies. No
+monkey-patching of core send/receive paths. The prepends activate only
+when `qos >= 1`:
+
+- **Sender** (PUSH, SCATTER): tracks sent messages in a pending store
+  keyed by xxHash digest. An ACK listener reads ACK command frames from
+  each peer. On disconnect, unacked messages are re-enqueued for
+  delivery to the next peer.
+- **Receiver** (PULL, GATHER): sends an ACK command frame back to the
+  sender after each message is received.
+- **REQ/REP**: the reply IS the ACK. At QoS 1, if the connection drops
+  before a reply arrives, the request is transparently re-sent to the
+  next REP.
+
+Fan-out patterns (PUB/SUB, XPUB/XSUB, RADIO/DISH) are deliberately out
+of scope. See the RFC for the rationale.
+
+### ACK protocol
+
+ACKs are ZMTP command frames (invisible to applications):
+
+```
+Name: "ACK"   Data: 'x' + XXH64(wire_bytes)    # 9 bytes total
+```
+
+The hash covers raw ZMTP wire bytes (frame headers + bodies), so
+different framings of the same payload produce different digests.
+
+### Backpressure
+
+Pending (un-ACK'd) messages count toward `send_hwm`. When the pending
+store is full, `send` blocks in the fiber until an ACK arrives or the
+connection drops (which re-enqueues the stuck messages). A misbehaving
+peer that never ACKs will stall the sender rather than grow the store
+unboundedly.
+
+### Linger and pending messages
+
+`Socket#close` with `linger: 0` discards anything that hasn't yet been
+ACK'd. This is correct but worth calling out: with QoS 1, messages you
+sent just before closing, even successfully written on the wire, can
+still be lost if the ACKs hadn't come back yet. Set `linger` to a
+non-zero value (or `Float::INFINITY`) if you need the close to wait
+for outstanding ACKs.
+
+### Zero overhead at QoS 0
+
+At QoS 0 (the default), no pending store is created, no ACK commands are
+sent, and no xxHash is computed. The prepended methods check
+`engine.options.qos` and fall through to the original behavior.
+
+## Supported socket types
+
+| Sender         | Receiver        | ACK mechanism       |
+|----------------|-----------------|---------------------|
+| PUSH / SCATTER | PULL / GATHER   | ACK command frame   |
+| REQ            | REP             | Reply = ACK         |
+
+## Requirements
+
+- Ruby >= 3.3
+- [omq](https://github.com/zeromq/omq.rb) >= 0.12
+- [xxhash](https://rubygems.org/gems/xxhash) (C extension)
+
+## RFC
+
+See [rfc/zmtp-qos.md](rfc/zmtp-qos.md) for the full specification.

data/lib/omq/qos/connection_ext.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module OMQ
+  class QoS
+    # Prepended onto Protocol::ZMTP::Connection so that command frames
+    # received during a normal {Connection#receive_message} loop are
+    # dispatched to a per-connection QoS handler. Without this, ACK
+    # commands sent by the peer would be silently dropped by the recv
+    # pump (which skips command frames).
+    #
+    module ConnectionExt
+      attr_accessor :qos_on_command
+
+
+      # Re-reads +@qos_on_command+ on each command frame rather than
+      # capturing it once at call-start. The PUSH reaper fires
+      # +receive_message+ immediately after handshake — before the
+      # routing layer has wired up the QoS handler — so a one-time
+      # capture would pin the handler to +nil+ for the lifetime of the
+      # connection.
+      def receive_message(&block)
+        super() do |frame|
+          if (handler = @qos_on_command)
+            cmd = Protocol::ZMTP::Codec::Command.from_body(frame.body)
+            handler.call(cmd)
+          end
+          block&.call(frame)
+        end
+      end
+
+    end
+  end
+end
+
+
+Protocol::ZMTP::Connection.prepend(OMQ::QoS::ConnectionExt)

data/lib/omq/qos/dead_letter.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module OMQ
+  class QoS
+    # Builds a {DeadLetter} for +entry+, resolves the entry's Promise
+    # with it, and releases one slot on +semaphore+ so a waiting sender
+    # can proceed.
+    #
+    # No-op if the Promise is already resolved (late ACK after
+    # dead-letter, or shutdown racing a sweep).
+    #
+    # @param entry [PeerRegistry::Entry]
+    # @param peer_info [Protocol::ZMTP::PeerInfo]
+    # @param reason [Symbol] +:peer_timeout+, +:terminal_nack+,
+    #   +:retry_exhausted+, or +:socket_closed+
+    # @param semaphore [Async::Semaphore, nil] the send-slot semaphore
+    #   to release; pass +nil+ on shutdown-drain where the whole
+    #   semaphore is being torn down anyway
+    # @param error [Object, nil] NackInfo at QoS 3; nil at QoS 2
+    # @return [DeadLetter, nil] the built DeadLetter, or nil if the
+    #   Promise was already resolved
+    def self.dead_letter(entry, peer_info:, reason:, semaphore: nil, error: nil)
+      return nil if entry.promise.resolved?
+
+      dl = DeadLetter.new(
+        parts:     entry.parts,
+        reason:    reason,
+        peer_info: peer_info,
+        error:     error,
+      )
+      entry.promise.resolve(dl)
+      semaphore&.release
+      dl
+    end
+  end
+end

data/lib/omq/qos/dedup_set.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require "async/clock"
+
+module OMQ
+  class QoS
+    # Per-receiving-connection dedup set for QoS >= 2.
+    #
+    # An ordered map of +{digest => added_at}+ keyed by the 8-byte XXH64 /
+    # SHA-1 digest. Hash insertion order gives us oldest-first eviction
+    # for free.
+    #
+    # Entries are added on message delivery, removed eagerly on CLR from
+    # the sender, and swept lazily on TTL expiry via {#sweep}. When a new
+    # entry would exceed +capacity+ the oldest entry is evicted — the
+    # sender will not retransmit past +dead_letter_timeout+ anyway, so an
+    # eviction older than that is safe.
+    #
+    class DedupSet
+      # @param capacity [Integer] typically recv_hwm; maximum entries
+      #   before oldest-first eviction kicks in
+      def initialize(capacity:)
+        @capacity = capacity
+        @entries  = {}
+      end
+
+
+      # @param digest [String] 8-byte binary
+      def seen?(digest)
+        @entries.key?(digest)
+      end
+
+
+      # Adds +digest+ and stamps it with the current monotonic clock.
+      # Evicts the oldest entry if capacity would be exceeded.
+      def add(digest)
+        if @entries.size >= @capacity && !@entries.key?(digest)
+          @entries.shift
+        end
+        @entries[digest] = Async::Clock.now
+      end
+
+
+      # Removes an entry (called on CLR from the sender).
+      def remove(digest)
+        @entries.delete(digest)
+      end
+
+
+      # Drops entries older than +ttl+ seconds at +now+.
+      def sweep(now, ttl)
+        cutoff = now - ttl
+        @entries.delete_if { |_digest, added_at| added_at < cutoff }
+      end
+
+
+      # @return [Integer]
+      def size
+        @entries.size
+      end
+
+
+      # @return [Boolean]
+      def empty?
+        @entries.empty?
+      end
+    end
+  end
+end

data/lib/omq/qos/error_codes.rb

@@ -0,0 +1,102 @@
+# frozen_string_literal: true
+
+module OMQ
+  class QoS
+    # NACK error codes and exception mapping for QoS 3.
+    #
+    # Receiver side: applications raise one of these exceptions from the
+    # block passed to +#receive+; the mapped error code travels back to
+    # the sender in a NACK command.
+    #
+    # Sender side: NACK payloads are decoded into {NackInfo} (attached to
+    # {DeadLetter#error} when the NACK is terminal or retries are
+    # exhausted).
+    #
+    # The high bit (+0x80+) of the error code byte is the *retryable*
+    # flag (see RFC §163–196). Senders MUST respect this flag for unknown
+    # codes.
+    #
+    module ErrorCodes
+      CODE_TIMEOUT    = 0x81  # retryable
+      CODE_BAD_INPUT  = 0x02  # terminal
+      CODE_INTERNAL   = 0x83  # retryable
+      CODE_OVERLOADED = 0x84  # retryable
+      CODE_REJECTED   = 0x05  # terminal
+
+      RETRYABLE_BIT = 0x80
+      MAX_MSG_BYTES = 65_535
+
+
+      # @param code [Integer]
+      # @return [Boolean]
+      def self.retryable?(code)
+        (code & RETRYABLE_BIT) != 0
+      end
+
+
+      # Maps an exception to a NACK code + truncated UTF-8 message
+      # suitable for the wire.
+      #
+      # @param exc [Exception]
+      # @return [Array(Integer, String)] [code, message bytes]
+      def self.exception_to_payload(exc)
+        code =
+          case exc
+          when TimeoutError    then CODE_TIMEOUT
+          when BadInputError   then CODE_BAD_INPUT
+          when OverloadedError then CODE_OVERLOADED
+          when RejectedError   then CODE_REJECTED
+          else                      CODE_INTERNAL
+          end
+
+        msg = (exc.message || "").b
+        msg = msg.byteslice(0, MAX_MSG_BYTES) if msg.bytesize > MAX_MSG_BYTES
+        [code, msg]
+      end
+    end
+
+
+    # Raised by a QoS 3 receive block when processing exceeds
+    # +processing_timeout+. Retryable (code +0x81+).
+    class TimeoutError < StandardError
+    end
+
+
+    # Raised by the application to indicate malformed or invalid input.
+    # Terminal (code +0x02+).
+    class BadInputError < StandardError
+    end
+
+
+    # Raised by the application to indicate capacity pressure. Retryable
+    # (code +0x84+).
+    class OverloadedError < StandardError
+    end
+
+
+    # Raised by the application for explicit rejection. Terminal
+    # (code +0x05+).
+    class RejectedError < StandardError
+    end
+
+
+    # Raised to a QoS 3 REQ sender's fiber when the REP peer returns a
+    # NACK. Carries the wire-level code + message so the caller can
+    # inspect them.
+    class NackError < StandardError
+      attr_reader :code
+
+
+      def initialize(code, message)
+        super(message)
+        @code = code
+      end
+
+    end
+
+
+    # Decoded NACK payload. Attached to {DeadLetter#error} at QoS 3 when
+    # the failure is terminal or retries are exhausted.
+    NackInfo = Data.define(:code, :message)
+  end
+end

data/lib/omq/qos/hasher.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+require "xxhash"
+require "digest/sha1"
+
+module OMQ
+  class QoS
+    # Supported hash algorithms in default preference order.
+    # x = XXH64 (8 bytes) — fast, requires xxhash library
+    # s = SHA-1 truncated to 64 bits (8 bytes) — available in any standard library
+    #
+    # All digests are 8 bytes (64 bits). Collision probability within
+    # the in-flight window (typically ≤ HWM) is negligible: ~N²/2⁶⁵
+    # where N is the number of un-ACK'd messages.
+    SUPPORTED_HASH_ALGOS = "xs".freeze
+    DEFAULT_HASH_ALGO    = "x".freeze
+    HASH_SIZE            = 8
+
+
+    # Computes an 8-byte digest over raw ZMTP wire bytes.
+    #
+    # @param parts [Array<String>] message frames
+    # @param algorithm [String] "x" (XXH64) or "s" (SHA-1 truncated)
+    # @return [String] 8-byte binary digest
+    #
+    def self.digest(parts, algorithm: DEFAULT_HASH_ALGO)
+      wire = Protocol::ZMTP::Codec::Frame.encode_message(parts)
+      case algorithm
+      when "x"
+        [XXhash.xxh64(wire)].pack("Q<")
+      when "s"
+        Digest::SHA1.digest(wire).byteslice(0, 8)
+      else
+        raise ArgumentError, "unsupported QoS hash algorithm: #{algorithm.inspect}"
+      end
+    end
+
+
+    # Negotiates the hash algorithm for a connection.
+    # Returns the first algo in our preference list that the peer supports.
+    #
+    # @param peer_hash [String] peer's supported algos (e.g. "sx")
+    # @return [String, nil] single-char algorithm, or nil if no overlap
+    #
+    def self.negotiate_hash(peer_hash)
+      return DEFAULT_HASH_ALGO if peer_hash.empty?
+      SUPPORTED_HASH_ALGOS.each_char do |algo|
+        return algo if peer_hash.include?(algo)
+      end
+      nil
+    end
+
+
+    # Builds an ACK command for the given message.
+    #
+    # @param parts [Array<String>] message frames
+    # @param algorithm [String] "x" (XXH64) or "s" (SHA-1 truncated)
+    # @return [Protocol::ZMTP::Codec::Command]
+    #
+    def self.ack_command(parts, algorithm: DEFAULT_HASH_ALGO)
+      Protocol::ZMTP::Codec::Command.ack(digest(parts, algorithm: algorithm), algorithm: algorithm)
+    end
+  end
+end

data/lib/omq/qos/lifecycle_ext.rb

@@ -0,0 +1,106 @@
+# frozen_string_literal: true
+
+module OMQ
+  class QoS
+    # Prepended onto OMQ::Engine::ConnectionLifecycle so that the
+    # Protocol::ZMTP::Connection is built with this socket's QoS
+    # metadata, and the peer's QoS properties are validated after the
+    # handshake.
+    #
+    module LifecycleExt
+      def handshake!(io, as_server:)
+        transition!(:handshaking)
+        conn_class =
+          if @transport.respond_to?(:connection_class)
+            @transport.connection_class
+          else
+            Protocol::ZMTP::Connection
+          end
+        conn = conn_class.new io,
+          socket_type:      @engine.socket_type.to_s,
+          identity:         @engine.options.identity,
+          as_server:        as_server,
+          mechanism:        @engine.options.mechanism&.dup,
+          max_message_size: @engine.options.max_message_size,
+          **QoS.handshake_metadata(@engine.options)
+
+        Async::Task.current.with_timeout(handshake_timeout) do
+          conn.handshake!
+        end
+
+        QoS.validate_handshake!(@engine.options, conn)
+
+        OMQ::Engine::Heartbeat.start(@barrier, conn, @engine.options)
+        ready!(conn)
+        @conn
+      rescue Protocol::ZMTP::Error, *OMQ::CONNECTION_LOST, Async::TimeoutError => error
+        @engine.emit_monitor_event :handshake_failed,
+          endpoint: @endpoint, detail: { error: error }
+
+        conn&.close
+
+        tear_down!(reconnect: true)
+        raise
+      end
+
+    end
+
+
+    # Builds the READY-property metadata hash this socket should
+    # advertise based on its QoS configuration. Empty hash at QoS 0 so
+    # the Connection sees no extras.
+    #
+    # @param options [OMQ::Options]
+    # @return [Hash{String => String}]
+    def self.handshake_metadata(options)
+      qos = options.qos
+      return {} if qos.nil?
+      meta = { "X-QoS" => qos.level.to_s }
+      meta["X-QoS-Hash"] = qos.hash_algos unless qos.hash_algos.empty?
+      meta
+    end
+
+
+    # Validates the peer's READY properties against the local QoS
+    # configuration. At QoS >= 2 also asserts that the connection has a
+    # stable per-peer identity (CURVE pubkey or non-empty ZMQ_IDENTITY)
+    # — without one the {PeerRegistry} cannot pin pending messages
+    # across reconnects.
+    def self.validate_handshake!(options, conn)
+      props     = conn.peer_properties || {}
+      qos       = options.qos
+      local_lvl = qos&.level || 0
+      peer_lvl  = (props["X-QoS"] || "0").to_i
+
+      if local_lvl != peer_lvl
+        raise Protocol::ZMTP::Error,
+              "QoS mismatch: local=#{local_lvl} peer=#{peer_lvl}"
+      end
+
+      return if local_lvl == 0
+
+      local_hash = qos.hash_algos
+      peer_hash  = props["X-QoS-Hash"] || ""
+
+      unless local_hash.empty? || peer_hash.empty?
+        algo = local_hash.each_char.find { |c| peer_hash.include?(c) }
+        unless algo
+          raise Protocol::ZMTP::Error,
+                "QoS hash algorithm mismatch: local=#{local_hash.inspect} peer=#{peer_hash.inspect}"
+        end
+      end
+
+      return if local_lvl < 2
+
+      info = conn.peer_info
+      if info.nil? || (info.public_key.nil? && info.identity.empty?)
+        raise Protocol::ZMTP::Error,
+              "QoS #{local_lvl} requires a stable peer anchor " \
+              "(CURVE public key or non-empty ZMQ_IDENTITY)"
+      end
+    end
+  end
+end
+
+
+OMQ::Engine::ConnectionLifecycle.prepend(OMQ::QoS::LifecycleExt)

data/lib/omq/qos/options_ext.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module OMQ
+  class QoS
+    # Prepended onto {OMQ::Options} once omq-qos is loaded. Changes the
+    # semantics of +Options#qos+ from "an Integer level" to "nil or an
+    # {OMQ::QoS} instance":
+    #
+    #   options.qos        # => nil     (QoS 0, default)
+    #   options.qos        # => <OMQ::QoS level=2 ...>
+    #   options.qos_level  # => 0, 1, 2, or 3 (convenience Integer)
+    #
+    # The Integer form is still accepted by callers that never loaded
+    # omq-qos — core omq keeps the Integer default. Once this extension
+    # is installed the default resets to +nil+.
+    #
+    module OptionsExt
+      def initialize(**kwargs)
+        super
+        @qos = nil
+      end
+
+
+      # Convenience: 0 / 1 / 2 / 3. nil-safe.
+      #
+      # @return [Integer]
+      def qos_level
+        @qos&.level || 0
+      end
+    end
+  end
+end
+
+
+OMQ::Options.prepend(OMQ::QoS::OptionsExt)