milvaion-worker 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 690ac8cdb6818d2ee65b491eee4415b102d57b8ad48a90b03655f68cde194b90
-  data.tar.gz: 66e2b832f777e3dec76371caedcbbb9cedcdf25e854cacd52c815836b8af32e5
+  metadata.gz: 1033916061253c1a73d22565a6f32702505cd8f623a219e07954e9107a685fcf
+  data.tar.gz: b89bafd296550ad1f2b25465af782d87be5b4979b18dabfa601ee918770ee8e3
 SHA512:
-  metadata.gz: 793e35c5741b0f1be61f4154dd3796bf2554ba229c9ea0baafc85b2b5d54b9a3198dfa8c4fd7bebe19bd7439a6e82e4e0218ff6dbc4b95cae592a94c7854e67f
-  data.tar.gz: aeaccc8157b3a95d6237a1db1561a5b9c61f1e2f52bc635a761f27889a7e860f4d15a1056b676ded1c6ce6828266531be92ba2a0a98d436c68d2f08f0b4346c6
+  metadata.gz: 685939781a8206aae1350bc5a316979d07378b4980c009504bd90e0fbc9e13fe5d25722be5fa415fa7ff77fbfc7d7e250fa977807c656839bf69f3772337a313
+  data.tar.gz: ecf1afed8544856409d617e37020ef5f627e809256c35df35bb70916724535026c6063b11126d3f0b5e13df0545822ba5ed1172c6c24a2f42f689222079392b4

data/README.md

@@ -9,10 +9,10 @@ gem install milvaion-worker
 Or add it to your Gemfile:
 
 ```ruby
-gem "milvaion-worker", "~> 1.0"
+gem "milvaion-worker", "~> 1.1"
 ```
 
-It requires Ruby 3.1 or later, and connects to the same RabbitMQ and Redis instances as your Milvaion API. It uses [bunny](https://github.com/ruby-amqp/bunny) for RabbitMQ and [redis-rb](https://github.com/redis/redis-rb) for Redis.
+It requires Ruby 3.1 or later, and connects to the same RabbitMQ and Redis instances as your Milvaion API. It uses [bunny](https://github.com/ruby-amqp/bunny) for RabbitMQ and [redis-rb](https://github.com/redis/redis-rb) for Redis. Apache Kafka can replace RabbitMQ; see [Kafka](#kafka).
 
 ## Quick start
 
@@ -146,6 +146,8 @@ Pass these to `Milvaion::Worker.new(...)` or `Milvaion::WorkerOptions.new(...)`:
 | `version` | `1.0.0` | Shown in the dashboard |
 | `instance_id` | `{worker_id}-{8 hex}` | Must be unique per process |
 | `logger` | a `Logger` on `$stdout` at INFO | |
+| `transport` | `rabbitmq` | `rabbitmq` or `kafka`, see [Kafka](#kafka) |
+| `kafka` | none | Kafka settings, a Hash, see [Kafka](#kafka) |
 
 For containers, `Milvaion::Worker.new(Milvaion::WorkerOptions.from_env)` reads the following variables. Keyword arguments to `from_env` take precedence.
 
@@ -157,6 +159,62 @@ For containers, `Milvaion::Worker.new(Milvaion::WorkerOptions.from_env)` reads t
 - `MILVAION_MAX_PARALLEL_JOBS`
 - `MILVAION_EXECUTION_TIMEOUT_SECONDS`
 - `MILVAION_HEARTBEAT_INTERVAL_SECONDS`
+- `MILVAION_TRANSPORT` and the `MILVAION_KAFKA_*` variables listed under [Kafka](#kafka)
+
+## Kafka
+
+RabbitMQ is the default broker. A deployment can use Apache Kafka instead. The API and every worker of a deployment must use the same broker, so switch the API to Kafka too (`MilvaionConfig:Transport:Type`). Redis still carries cancellation requests, as on RabbitMQ.
+
+The Kafka client is the [rdkafka](https://github.com/karafka/rdkafka-ruby) gem (librdkafka). It is not a dependency of `milvaion-worker`, so add it to your Gemfile next to the SDK. The worker loads it only when the transport is `kafka`, and `start` raises an error that says so if it is missing.
+
+```ruby
+gem "milvaion-worker", "~> 1.1"
+gem "rdkafka", ">= 0.19"
+```
+
+rdkafka ships precompiled builds for Linux (glibc and musl) and macOS; on other platforms it compiles librdkafka when installed, which needs a C toolchain. Ruby 3.1 gets rdkafka 0.22, newer Rubies get later releases.
+
+```ruby
+worker = Milvaion::Worker.new(
+  worker_id: "mail-worker",
+  transport: "kafka",
+  kafka: {
+    brokers: ["kafka-1:9092", "kafka-2:9092"],
+    topic_prefix: "prod." # must match the API's
+  },
+  redis_url: "redis://localhost:6379"
+)
+```
+
+Handlers do not change. Jobs, outcomes, retries, logs and cancellation behave as on RabbitMQ.
+
+| `kafka:` key | Default | Description |
+|---|---|---|
+| `brokers` | `localhost:9092` | Bootstrap servers, an Array or a comma separated String |
+| `tls` | `false` | Turns TLS on. Setting any of the three files below turns it on too. |
+| `tls_ca_file` | system roots | PEM bundle used to verify the brokers |
+| `tls_cert_file`, `tls_key_file` | none | PEM client certificate and key for mutual TLS |
+| `sasl_mechanism` | none | `plain`, `scram-sha-256` or `scram-sha-512`, with `sasl_username` and `sasl_password` |
+| `topic_prefix` | empty | Put in front of every topic and consumer group name. Only `a-z A-Z 0-9 . _ -`. Must match the API's. |
+| `auto_create_topics` | `true` | Create the worker's topics, and the shared topics it writes to, at startup |
+| `num_partitions` | 12 | Partitions of the topics the worker creates |
+| `replication_factor` | -1 | Replication of the topics the worker creates. -1 uses the broker's default. |
+| `max_in_flight_per_partition` | worker `max_parallel_jobs` | Jobs of one partition that may run at the same time |
+
+`WorkerOptions.from_env` reads them from `MILVAION_TRANSPORT` (`kafka`), `MILVAION_KAFKA_BOOTSTRAP_SERVERS` (or `KAFKA_BOOTSTRAP_SERVERS`), `MILVAION_KAFKA_TLS` (`true`), `MILVAION_KAFKA_TLS_CA_FILE`, `MILVAION_KAFKA_TLS_CERT_FILE`, `MILVAION_KAFKA_TLS_KEY_FILE`, `MILVAION_KAFKA_SASL_MECHANISM`, `MILVAION_KAFKA_SASL_USERNAME`, `MILVAION_KAFKA_SASL_PASSWORD`, `MILVAION_KAFKA_TOPIC_PREFIX`, `MILVAION_KAFKA_AUTO_CREATE_TOPICS` (`false` disables it), `MILVAION_KAFKA_NUM_PARTITIONS` (or `MILVAION_KAFKA_PARTITIONS`), `MILVAION_KAFKA_REPLICATION_FACTOR` and `MILVAION_KAFKA_MAX_IN_FLIGHT_PER_PARTITION`. Keys of a `kafka:` Hash passed to `from_env` take precedence.
+
+How it works on Kafka:
+
+- **Topics.** The worker reads `milvaion.jobs.{worker}` and `milvaion.jobs.{worker}.retry`, where `{worker}` is the worker id in lower case with every character outside `a-z 0-9 . _ -` replaced by `_`. It writes to `milvaion.status`, `milvaion.logs`, `milvaion.heartbeats`, `milvaion.registrations` and `milvaion.dlq`. All names get the topic prefix. It picks the handler by the job name in the record, so routing patterns do not decide where a job goes.
+- **Instances.** Every instance of a worker joins the consumer group `milvaion.worker.{worker}`. Partitions are shared between the instances, so a topic with 12 partitions serves at most 12 instances. Raise `num_partitions` before you scale out further.
+- **Parallelism.** An instance runs up to `max_parallel_jobs` jobs at once, also several from one partition, up to `max_in_flight_per_partition`. The per job `max_parallel_jobs` applies to RabbitMQ only. When the instance is full, it pauses the partitions and keeps polling, so it stays in the group.
+- **Commits.** A job record is committed only after its outcome has been reported. Kafka commits a position, not a single record, so a partition is committed up to its oldest unfinished job. If an instance crashes, every record after that one runs again, including jobs that had already finished. Make handlers idempotent; `ctx.occurrence_id` helps with that.
+- **Retries.** A failed attempt with retries left is written to the retry topic with a time before which it must not run (`base × 2^n`). The instance slot is free during the wait. The instance that reads the retry holds it until it is due, and the occurrence keeps sending job heartbeats meanwhile.
+- **Dead letters.** A job that fails for good goes to `milvaion.dlq` as on RabbitMQ. A record that is not a valid job is forwarded there unchanged, with the headers `x-original-topic` and `x-dlq-reason`.
+- **Shutdown.** `stop` stops taking records, waits up to `shutdown_timeout`, writes the jobs that are still running back to their topic, commits and leaves the group.
+- **Priority** is best effort: records of a partition are read in the order the API wrote them.
+
+The full wire contract is in [KAFKA-TRANSPORT.md](https://github.com/Milvasoft/milvaion/blob/master/docs/githubdocs/KAFKA-TRANSPORT.md).
 
 ## Reliability
 
@@ -170,7 +228,7 @@ For containers, `Milvaion::Worker.new(Milvaion::WorkerOptions.from_env)` reads t
 
 ## How it works
 
-The SDK implements the language-neutral [worker protocol](https://github.com/Milvasoft/milvaion/blob/master/docs/githubdocs/WORKER-PROTOCOL.md): the RabbitMQ queues and message formats the .NET SDK uses.
+The SDK implements the language-neutral [worker protocol](https://github.com/Milvasoft/milvaion/blob/master/docs/githubdocs/WORKER-PROTOCOL.md): the RabbitMQ queues and message formats the .NET SDK uses. On Kafka it follows [KAFKA-TRANSPORT.md](https://github.com/Milvasoft/milvaion/blob/master/docs/githubdocs/KAFKA-TRANSPORT.md).
 
 ## Development
 
@@ -188,3 +246,10 @@ docker run -d --name rabbit -p 5672:5672 rabbitmq:4
 docker run -d --name redis -p 6379:6379 redis:7
 MILVAION_TEST_RABBITMQ_URL=amqp://guest:guest@localhost:5672 MILVAION_TEST_REDIS_URL=redis://localhost:6379 bundle exec rspec
 ```
+
+The Kafka integration specs run against a real broker and are skipped unless `MILVAION_KAFKA_BROKERS` is set. The Gemfile adds rdkafka for development and tests.
+
+```bash
+docker run -d --name kafka -p 9092:9092 apache/kafka:3.9.0
+MILVAION_KAFKA_BROKERS=localhost:9092 bundle exec rspec
+```

data/lib/milvaion/kafka.rb

@@ -0,0 +1,299 @@
+# frozen_string_literal: true
+
+module Milvaion
+  # Kafka wire contract shared with the Milvaion API and the other SDKs.
+  # See docs/githubdocs/KAFKA-TRANSPORT.md in the repository. Nothing in this file needs the
+  # rdkafka gem; the transport itself (kafka_transport.rb) loads it only when it is used.
+  module Kafka
+    HEADER_OCCURRENCE_ID = "OccurrenceId"
+    HEADER_CORRELATION_ID = "CorrelationId"
+    HEADER_RETRY_COUNT = "x-retry-count"
+    HEADER_MAX_RETRIES = "MaxRetries"
+    HEADER_NOT_BEFORE = "x-not-before"
+    HEADER_ORIGINAL_TOPIC = "x-original-topic"
+    HEADER_DLQ_REASON = "x-dlq-reason"
+
+    TOPIC_PREFIX_PATTERN = /\A[A-Za-z0-9._-]*\z/.freeze
+    SASL_MECHANISMS = { "plain" => "PLAIN", "scramsha256" => "SCRAM-SHA-256", "scramsha512" => "SCRAM-SHA-512" }.freeze
+
+    # Topic and consumer group names of one worker.
+    Topics = Struct.new(:jobs, :retry, :status, :logs, :heartbeats, :registrations, :dlq, :group, keyword_init: true)
+
+    # The worker id as it appears in topic and group names: lower case, with every character
+    # outside a-z 0-9 . _ - replaced by "_".
+    def self.worker_segment(worker_id)
+      worker_id.to_s.each_char.map do |char|
+        lower = char.downcase
+        lower.match?(/\A[a-z0-9._-]\z/) ? lower : "_"
+      end.join
+    end
+
+    def self.topics(prefix, worker_id)
+      worker = worker_segment(worker_id)
+
+      Topics.new(
+        jobs: "#{prefix}milvaion.jobs.#{worker}",
+        retry: "#{prefix}milvaion.jobs.#{worker}.retry",
+        status: "#{prefix}milvaion.status",
+        logs: "#{prefix}milvaion.logs",
+        heartbeats: "#{prefix}milvaion.heartbeats",
+        registrations: "#{prefix}milvaion.registrations",
+        dlq: "#{prefix}milvaion.dlq",
+        group: "#{prefix}milvaion.worker.#{worker}"
+      )
+    end
+
+    # Record headers as the processor reads them: string keys and string values, the last of
+    # repeated headers winning (rdkafka returns an Array for a repeated header).
+    def self.decode_headers(headers)
+      (headers || {}).to_h do |key, value|
+        value = value.last if value.is_a?(Array)
+        [key.to_s, value.nil? ? nil : value.to_s]
+      end
+    end
+
+    # Headers as UTF-8 strings, numbers as decimal strings, in a stable order.
+    def self.encode_headers(headers)
+      (headers || {}).sort_by { |key, _| key.to_s }.to_h do |key, value|
+        [key.to_s, header_value(value)]
+      end
+    end
+
+    def self.header_value(value)
+      case value
+      when nil then ""
+      when String then value.dup.force_encoding(Encoding::UTF_8)
+      else value.to_s
+      end
+    end
+
+    # x-not-before as a Time, or nil when the header is missing or invalid.
+    def self.not_before(headers)
+      raw = (headers || {})[HEADER_NOT_BEFORE].to_s.strip
+      return nil unless raw.match?(/\A-?\d+\z/)
+
+      Time.at(Integer(raw, 10) / 1000r)
+    end
+
+    def self.unix_ms(time)
+      (time.to_r * 1000).floor
+    end
+
+    # Kafka settings with every default applied, validated. Built by WorkerOptions.
+    class Settings
+      KEYS = %i[brokers tls tls_ca_file tls_cert_file tls_key_file sasl_mechanism sasl_username sasl_password
+                topic_prefix auto_create_topics num_partitions replication_factor max_in_flight_per_partition].freeze
+
+      attr_reader(*KEYS)
+
+      def initialize(options, max_parallel_jobs)
+        options = (options || {}).to_h.transform_keys(&:to_sym)
+        unknown = options.keys - KEYS
+        raise ArgumentError, "Unknown Kafka option(s): #{unknown.join(', ')}" unless unknown.empty?
+
+        @brokers = Array(options[:brokers]).flat_map { |entry| entry.to_s.split(",") }.map(&:strip).reject(&:empty?)
+        @brokers = ["localhost:9092"] if @brokers.empty?
+
+        @topic_prefix = options[:topic_prefix].to_s
+        unless TOPIC_PREFIX_PATTERN.match?(@topic_prefix)
+          raise ArgumentError, "Kafka topic_prefix may only contain a-z A-Z 0-9 . _ -, got #{@topic_prefix.inspect}"
+        end
+
+        @auto_create_topics = options.fetch(:auto_create_topics, true) != false
+        @num_partitions = integer(options[:num_partitions], 12, "num_partitions")
+        raise ArgumentError, "Kafka num_partitions must be positive, got #{@num_partitions}" unless @num_partitions.between?(1, 1 << 20)
+
+        @replication_factor = integer(options[:replication_factor], -1, "replication_factor")
+        unless @replication_factor == -1 || @replication_factor.between?(1, 32_767)
+          raise ArgumentError, "Kafka replication_factor must be -1 or positive, got #{@replication_factor}"
+        end
+
+        @max_in_flight_per_partition = integer(options[:max_in_flight_per_partition], max_parallel_jobs, "max_in_flight_per_partition")
+        unless @max_in_flight_per_partition.positive?
+          raise ArgumentError, "Kafka max_in_flight_per_partition must be positive, got #{@max_in_flight_per_partition}"
+        end
+
+        resolve_tls(options)
+        resolve_sasl(options)
+      end
+
+      def tls?
+        @tls
+      end
+
+      def sasl?
+        !@sasl_mechanism.nil?
+      end
+
+      # librdkafka settings shared by the producer, the admin client and the consumer.
+      def client_config(client_id)
+        config = {
+          "bootstrap.servers": brokers.join(","),
+          "client.id": client_id,
+          "security.protocol": security_protocol
+        }
+
+        config[:"ssl.ca.location"] = tls_ca_file if tls_ca_file
+        config[:"ssl.certificate.location"] = tls_cert_file if tls_cert_file
+        config[:"ssl.key.location"] = tls_key_file if tls_key_file
+
+        if sasl?
+          config[:"sasl.mechanisms"] = sasl_mechanism
+          config[:"sasl.username"] = sasl_username
+          config[:"sasl.password"] = sasl_password.to_s
+        end
+
+        config
+      end
+
+      def security_protocol
+        if sasl?
+          tls? ? "sasl_ssl" : "sasl_plaintext"
+        else
+          tls? ? "ssl" : "plaintext"
+        end
+      end
+
+      private
+
+      def integer(value, default, name)
+        return default if value.nil? || value.to_s.strip.empty? || value == 0
+
+        Integer(value)
+      rescue ArgumentError, TypeError
+        raise ArgumentError, "Kafka #{name} must be an integer, got #{value.inspect}"
+      end
+
+      def resolve_tls(options)
+        @tls_ca_file = presence(options[:tls_ca_file])
+        @tls_cert_file = presence(options[:tls_cert_file])
+        @tls_key_file = presence(options[:tls_key_file])
+        @tls = (options[:tls] ? true : false) || !(@tls_ca_file || @tls_cert_file || @tls_key_file).nil?
+
+        if @tls_cert_file.nil? != @tls_key_file.nil?
+          raise ArgumentError, "Kafka tls_cert_file and tls_key_file must be set together"
+        end
+
+        { tls_ca_file: @tls_ca_file, tls_cert_file: @tls_cert_file, tls_key_file: @tls_key_file }.each do |name, path|
+          raise ArgumentError, "Kafka #{name} #{path.inspect} cannot be read" if path && !File.readable?(path)
+        end
+      end
+
+      def resolve_sasl(options)
+        raw = options[:sasl_mechanism].to_s.strip
+        @sasl_username = presence(options[:sasl_username])
+        @sasl_password = options[:sasl_password]
+        @sasl_mechanism = nil
+        return if raw.empty?
+
+        @sasl_mechanism = SASL_MECHANISMS[raw.downcase.delete("-_")]
+
+        if @sasl_mechanism.nil?
+          raise ArgumentError, %(Kafka sasl_mechanism must be "plain", "scram-sha-256" or "scram-sha-512", got #{raw.inspect})
+        end
+
+        raise ArgumentError, "Kafka sasl_username is required with sasl_mechanism" if @sasl_username.nil?
+      end
+
+      def presence(value)
+        value = value.to_s.strip
+        value.empty? ? nil : value
+      end
+    end
+
+    ENV_STRINGS = {
+      "MILVAION_KAFKA_TLS_CA_FILE" => :tls_ca_file,
+      "MILVAION_KAFKA_TLS_CERT_FILE" => :tls_cert_file,
+      "MILVAION_KAFKA_TLS_KEY_FILE" => :tls_key_file,
+      "MILVAION_KAFKA_SASL_MECHANISM" => :sasl_mechanism,
+      "MILVAION_KAFKA_SASL_USERNAME" => :sasl_username,
+      "MILVAION_KAFKA_SASL_PASSWORD" => :sasl_password,
+      "MILVAION_KAFKA_TOPIC_PREFIX" => :topic_prefix
+    }.freeze
+
+    ENV_INTEGERS = {
+      "MILVAION_KAFKA_NUM_PARTITIONS" => :num_partitions,
+      "MILVAION_KAFKA_PARTITIONS" => :num_partitions,
+      "MILVAION_KAFKA_REPLICATION_FACTOR" => :replication_factor,
+      "MILVAION_KAFKA_MAX_IN_FLIGHT_PER_PARTITION" => :max_in_flight_per_partition
+    }.freeze
+
+    # The Kafka options found in MILVAION_KAFKA_* variables, as a Hash for WorkerOptions' kafka:.
+    def self.options_from_env(env)
+      read = ->(key) { env[key].to_s.strip }
+      values = {}
+
+      brokers = read.call("MILVAION_KAFKA_BOOTSTRAP_SERVERS")
+      brokers = read.call("KAFKA_BOOTSTRAP_SERVERS") if brokers.empty?
+      values[:brokers] = brokers unless brokers.empty?
+
+      values[:tls] = true if read.call("MILVAION_KAFKA_TLS").casecmp?("true")
+
+      ENV_STRINGS.each do |key, name|
+        value = read.call(key)
+        values[name] = value unless value.empty?
+      end
+
+      values[:auto_create_topics] = false if read.call("MILVAION_KAFKA_AUTO_CREATE_TOPICS").casecmp?("false")
+
+      ENV_INTEGERS.each do |key, name|
+        raw = read.call(key)
+        next if raw.empty? || values.key?(name)
+
+        begin
+          values[name] = Integer(raw, 10)
+        rescue ArgumentError
+          raise ArgumentError, "#{key} must be a number, got #{raw.inspect}"
+        end
+      end
+
+      values
+    end
+
+    # Follows the records of one partition that this instance has taken, in offset order, and knows
+    # how far the partition may be committed: up to the first record that is not finished, because
+    # a Kafka commit covers every record before it.
+    class OffsetTracker
+      def initialize
+        @pending = [] # [offset, done] pairs in offset order
+        @last = -1
+        @position = -1
+      end
+
+      # Registers a record taken from the partition. A repeated or older offset returns false.
+      def add(offset)
+        return false if offset <= @last
+
+        @last = offset
+        @pending << [offset, false]
+        true
+      end
+
+      # Marks a record finished. Returns false for an offset the tracker does not hold.
+      def done(offset)
+        index = @pending.bsearch_index { |(o, _)| o >= offset }
+        return false if index.nil? || @pending[index][0] != offset
+
+        @pending[index][1] = true
+
+        while (head = @pending.first) && head[1]
+          @position = head[0] + 1
+          @pending.shift
+        end
+
+        true
+      end
+
+      # The offset to commit, one past the end of the contiguous finished range, or nil until the
+      # first record has finished.
+      def commit_position
+        @position.negative? ? nil : @position
+      end
+
+      # Records taken that are not ready to commit yet.
+      def outstanding
+        @pending.size
+      end
+    end
+  end
+end