milvaion-worker 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 690ac8cdb6818d2ee65b491eee4415b102d57b8ad48a90b03655f68cde194b90
+  data.tar.gz: 66e2b832f777e3dec76371caedcbbb9cedcdf25e854cacd52c815836b8af32e5
+SHA512:
+  metadata.gz: 793e35c5741b0f1be61f4154dd3796bf2554ba229c9ea0baafc85b2b5d54b9a3198dfa8c4fd7bebe19bd7439a6e82e4e0218ff6dbc4b95cae592a94c7854e67f
+  data.tar.gz: aeaccc8157b3a95d6237a1db1561a5b9c61f1e2f52bc635a761f27889a7e860f4d15a1056b676ded1c6ce6828266531be92ba2a0a98d436c68d2f08f0b4346c6

data/README.md

@@ -0,0 +1,190 @@
+# Milvaion Ruby Worker SDK
+
+Run [Milvaion](https://github.com/Milvasoft/milvaion) jobs in Ruby. A Ruby worker shows up in the dashboard like a .NET worker. The scheduler dispatches jobs to it, and the dashboard shows its live logs, status, retries, cancellation and timeouts.
+
+```bash
+gem install milvaion-worker
+```
+
+Or add it to your Gemfile:
+
+```ruby
+gem "milvaion-worker", "~> 1.0"
+```
+
+It requires Ruby 3.1 or later, and connects to the same RabbitMQ and Redis instances as your Milvaion API. It uses [bunny](https://github.com/ruby-amqp/bunny) for RabbitMQ and [redis-rb](https://github.com/redis/redis-rb) for Redis.
+
+## Quick start
+
+```ruby
+require "milvaion-worker"
+
+worker = Milvaion::Worker.new(
+  worker_id: "report-worker",
+  rabbitmq_url: "amqp://guest:guest@localhost:5672/",
+  redis_url: "redis://localhost:6379"
+)
+
+worker.job("BuildReportJob", max_retries: 3, execution_timeout: 600) do |ctx|
+  data = ctx.data || {}
+  raise Milvaion::PermanentJobError, 'Job data needs "month".' unless data["month"] # no retries, straight to the DLQ
+
+  ctx.log("Building the report for #{data['month']}")
+
+  rows = fetch_rows(data["month"])
+
+  rows.each_slice(500) do |batch|
+    ctx.raise_if_aborted # stop early if the job was cancelled or timed out
+    write_report(batch)
+  end
+
+  { rows: rows.size } # shown as the occurrence result
+end
+
+worker.run # blocks; SIGINT/SIGTERM shut down gracefully
+```
+
+Then create a job in the dashboard: pick the worker `report-worker` and the job `BuildReportJob`, and set a schedule.
+
+To run the worker inside your own process, call `worker.start` and `worker.stop` instead of `run`.
+
+## Jobs
+
+There are two ways to register a job. Either way, names are unique within a worker and matched case-insensitively.
+
+**With a block** (or any object that responds to `call(ctx)`):
+
+```ruby
+worker.job("SendEmailJob", max_retries: 5) { |ctx| Mailer.deliver(ctx.data) }
+worker.job("CleanupJob", Cleanup.method(:run))
+```
+
+**With a class.** Subclass `Milvaion::Job` and implement `perform(ctx)`. Each execution gets a new instance. The job name defaults to the class name without its namespace.
+
+```ruby
+class SendEmailJob < Milvaion::Job
+  job_options max_retries: 5, data_schema: { type: "object", properties: { to: { type: "string" } } }
+
+  def perform(ctx)
+    ctx.log("Sending to #{ctx.data['to']}")
+    { sent: true }
+  end
+end
+
+worker.job(SendEmailJob)
+worker.job(SendEmailJob, max_parallel_jobs: 2) # options here override job_options
+```
+
+Use `job_name "OtherName"` in the class to register it under a different name.
+
+The handler receives a `Milvaion::JobContext`:
+
+| Member | Description |
+|---|---|
+| `ctx.data` | Parsed job data with string keys, or `nil`. Data that is not valid JSON fails the job permanently. |
+| `ctx.raw_data` | Job data as the raw JSON string. |
+| `ctx.occurrence_id` | Id of this execution. It stays the same across retries and redeliveries, so you can use it as an idempotency key. |
+| `ctx.retry_count` | `0` on the first attempt, then `1`, `2` and so on. |
+| `ctx.aborted?`, `ctx.abort_reason`, `ctx.raise_if_aborted` | Whether the job was cancelled, timed out or interrupted by shutdown. `abort_reason` is `:cancelled`, `:timed_out`, `:shutdown` or `nil`. |
+| `ctx.sleep(seconds)` | Like `Kernel#sleep`, but wakes up and raises `Milvaion::JobAborted` as soon as the job is aborted. |
+| `ctx.log` / `info` / `debug` / `warn(message, data = nil)` | Write to the occurrence log in the dashboard. |
+| `ctx.error(message, error = nil, data = nil)` | Write an error entry, with the exception's type and stack trace when given. |
+| `ctx.job` | The scheduled job as the API sent it, a Hash with string keys. |
+
+How the handler ends decides the outcome:
+
+| The handler... | Outcome |
+|---|---|
+| returns a value | `Completed`. A string is stored as-is; anything else is stored as JSON. `nil` stores `Job {name} completed successfully`. |
+| raises | Retried with exponential backoff. After `max_retries`, the occurrence is `Failed` and the job goes to the dead letter queue. |
+| raises `Milvaion::PermanentJobError` | `Failed` without retrying, and the job goes to the dead letter queue. |
+| runs past its timeout | `TimedOut` |
+| is cancelled from the dashboard | `Cancelled` |
+| is still running when the worker stops | The job goes back to the queue and runs again. |
+
+### Cancellation and timeouts in Ruby threads
+
+Jobs run on a thread pool, so blocking code does not stall other jobs. Ruby has no safe way to stop a thread from outside: `Thread#raise` and `Thread#kill` can interrupt a job in the middle of an `ensure` block or while it holds a lock. So cancellation and timeouts are **cooperative** by default:
+
+- When a job is cancelled or times out, the worker reports the occurrence as `Cancelled` or `TimedOut` and settles the message right away. It does not wait for the handler.
+- The handler's thread keeps running until it returns. Until then it still uses CPU, memory and connections, and it is not counted in the worker's capacity any more.
+- To stop early, check `ctx.aborted?` or call `ctx.raise_if_aborted` in long loops, and use `ctx.sleep` instead of `sleep`.
+
+If a job cannot check the context, for example because it spends its time in one long blocking call, set `interrupt_on_abort: true` on the job. The worker then raises `Milvaion::JobAborted` inside the handler's thread when the job is aborted. This is the same mechanism as Ruby's `Timeout` module, with the same risks, so only use it for code that is safe to interrupt at any point.
+
+### Job options
+
+| Option | Default | Description |
+|---|---|---|
+| `max_parallel_jobs` | min(10, worker `max_parallel_jobs`) | Jobs of this type that may run at once on this instance |
+| `execution_timeout` | worker default (3600 s) | Seconds. `0` disables the timeout. A timeout set on the job in the dashboard takes precedence. |
+| `max_retries` | 3 | Retries after the first failure |
+| `base_retry_delay` | 5 | Retry *n* waits `base × 2^(n-1)` seconds |
+| `data_schema` | none | JSON Schema of the job data, as a Hash or a JSON string. The dashboard builds the job data form from it. |
+| `result_schema` | none | JSON Schema of the result |
+| `routing_pattern` | `{worker_id}.{name without "Job"}.*` | RabbitMQ routing pattern |
+| `consumer_id` | `{name without "Job"}-consumer` | Shown in the dashboard |
+| `interrupt_on_abort` | `false` | Raise `JobAborted` inside the handler thread on cancel or timeout (see above) |
+
+## Worker options
+
+Pass these to `Milvaion::Worker.new(...)` or `Milvaion::WorkerOptions.new(...)`:
+
+| Option | Default | Description |
+|---|---|---|
+| `worker_id` | required | Identifies the worker application. Use the same value on every replica. |
+| `rabbitmq_url` | `amqp://guest:guest@localhost:5672/` | |
+| `queue_type` | `classic` | Must match the API's `MilvaionConfig:RabbitMQ:QueueType` |
+| `redis_url` | `redis://localhost:6379` | Redis only carries cancellation requests. `nil` disables them. |
+| `redis_key_prefix` | `Milvaion:JobScheduler:` | Must match the API's `MilvaionConfig:Redis:KeyPrefix` |
+| `max_parallel_jobs` | CPU count × 2 | Size of the job thread pool, reported as this instance's capacity |
+| `execution_timeout` | 3600 | Default job timeout in seconds |
+| `heartbeat_interval` | 30 | |
+| `job_heartbeat_interval` | 60 | Keeps long jobs from being flagged as zombies |
+| `shutdown_timeout` | 30 | How long `stop` lets running jobs finish |
+| `startup_timeout` | 30 | How long `start` waits for RabbitMQ and Redis |
+| `version` | `1.0.0` | Shown in the dashboard |
+| `instance_id` | `{worker_id}-{8 hex}` | Must be unique per process |
+| `logger` | a `Logger` on `$stdout` at INFO | |
+
+For containers, `Milvaion::Worker.new(Milvaion::WorkerOptions.from_env)` reads the following variables. Keyword arguments to `from_env` take precedence.
+
+- `MILVAION_WORKER_ID`
+- `MILVAION_RABBITMQ_URL`
+- `MILVAION_RABBITMQ_QUEUE_TYPE`
+- `MILVAION_REDIS_URL` (`false` disables Redis)
+- `MILVAION_REDIS_KEY_PREFIX`
+- `MILVAION_MAX_PARALLEL_JOBS`
+- `MILVAION_EXECUTION_TIMEOUT_SECONDS`
+- `MILVAION_HEARTBEAT_INTERVAL_SECONDS`
+
+## Reliability
+
+- **Reconnects.** If RabbitMQ goes away, bunny's automatic recovery restores the connection with its queues and consumers, and the worker registers again. The Redis listener reconnects and subscribes again.
+- **At-least-once delivery.** A message is acknowledged only after its outcome has been reported, so a crash makes the job run again. Make handlers idempotent; `ctx.occurrence_id` helps with that.
+- **Graceful shutdown.** `stop` does the following in order:
+  1. Stops taking new jobs, and returns jobs that were waiting for a free thread to the queue.
+  2. Waits up to `shutdown_timeout` for running jobs to finish.
+  3. Returns the jobs that are still running to the queue.
+  4. Flushes logs and tells the API the instance has left.
+
+## How it works
+
+The SDK implements the language-neutral [worker protocol](https://github.com/Milvasoft/milvaion/blob/master/docs/githubdocs/WORKER-PROTOCOL.md): the RabbitMQ queues and message formats the .NET SDK uses.
+
+## Development
+
+```bash
+bundle install
+bundle exec rspec
+bundle exec ruby examples/basic_worker.rb   # configure with MILVAION_* variables
+gem build milvaion-worker.gemspec
+```
+
+The integration specs run a worker against a real RabbitMQ and Redis. They are skipped unless `MILVAION_TEST_RABBITMQ_URL` is set:
+
+```bash
+docker run -d --name rabbit -p 5672:5672 rabbitmq:4
+docker run -d --name redis -p 6379:6379 redis:7
+MILVAION_TEST_RABBITMQ_URL=amqp://guest:guest@localhost:5672 MILVAION_TEST_REDIS_URL=redis://localhost:6379 bundle exec rspec
+```

data/lib/milvaion/cancellation_listener.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require "json"
+require "redis"
+
+module Milvaion
+  # Receives the cancellation requests the API publishes on Redis, on a thread of its own.
+  # If the connection drops, it reconnects and subscribes again.
+  class CancellationListener
+    # A subscription that sees no message for this long is renewed, which also detects a dead connection.
+    IDLE_RESUBSCRIBE_SECONDS = 60
+    RECONNECT_DELAY_SECONDS = 2
+
+    def initialize(url, channel, registry, logger, redis_factory: nil)
+      @url = url
+      @channel = channel
+      @registry = registry
+      @logger = logger
+      @redis_factory = redis_factory || ->(redis_url) { Redis.new(url: redis_url, connect_timeout: 5, reconnect_attempts: 0) }
+      @mutex = Mutex.new
+      @client = nil
+      @thread = nil
+      @stopped = false
+    end
+
+    # Checks that Redis is reachable, then starts listening. Raises if it is not.
+    def start(timeout)
+      deadline = monotonic + timeout
+
+      begin
+        probe = @redis_factory.call(@url)
+        probe.ping
+        probe.close
+      rescue StandardError => e
+        raise ConnectionError, "Could not connect to Redis within #{timeout}s: #{e.message}" if monotonic + RECONNECT_DELAY_SECONDS > deadline
+
+        @logger.warn("Could not connect to Redis: #{e.message}")
+        Kernel.sleep(RECONNECT_DELAY_SECONDS)
+        retry
+      end
+
+      ready = Queue.new
+
+      @thread = Thread.new do
+        Thread.current.name = "milvaion-cancel"
+        listen(ready)
+      end
+
+      # Wait for the first subscription so a cancellation sent right after start is not missed.
+      Thread.new { Kernel.sleep(timeout) && ready << :timeout }
+      @logger.warn("Subscribing to #{@channel} is taking long; continuing.") if ready.pop == :timeout
+
+      @logger.info("Listening for cancellations on #{@channel}")
+      self
+    end
+
+    def stop
+      @mutex.synchronize do
+        @stopped = true
+        client = @client
+        @client = nil
+
+        begin
+          client&.close
+        rescue StandardError
+          nil
+        end
+      end
+
+      @thread&.kill
+      @thread&.join(2)
+      @thread = nil
+    end
+
+    # Handles one message from the channel. Public for tests.
+    def on_message(payload)
+      request = JSON.parse(payload.to_s)
+      return unless request.is_a?(Hash)
+
+      occurrence_id = request["CorrelationId"] || request["OccurrenceId"]
+      reason = request["Reason"]
+
+      return unless occurrence_id && @registry.cancel(occurrence_id, reason)
+
+      @logger.info("Cancelling occurrence #{occurrence_id}: #{reason.to_s.empty? ? 'no reason given' : reason}")
+    rescue JSON::ParserError
+      @logger.debug("Ignored a cancellation message that is not JSON.")
+    end
+
+    private
+
+    def listen(ready)
+      failed = false
+
+      until stopped?
+        begin
+          client = @redis_factory.call(@url)
+          @mutex.synchronize { @client = client }
+          break if stopped?
+
+          client.subscribe_with_timeout(IDLE_RESUBSCRIBE_SECONDS, @channel) do |on|
+            on.subscribe do
+              ready << :ok
+              @logger.info("Reconnected to Redis.") if failed
+              failed = false
+            end
+
+            on.message { |_channel, payload| on_message(payload) }
+          end
+        rescue Redis::TimeoutError
+          # No message for a while: renew the subscription.
+        rescue StandardError => e
+          break if stopped?
+
+          @logger.warn("Redis cancellation listener error: #{e.message}. Reconnecting...") unless failed
+          failed = true
+          Kernel.sleep(RECONNECT_DELAY_SECONDS)
+        ensure
+          begin
+            client&.close
+          rescue StandardError
+            nil
+          end
+        end
+      end
+    end
+
+    def stopped?
+      @mutex.synchronize { @stopped }
+    end
+
+    def monotonic
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+  end
+end

data/lib/milvaion/context.rb

@@ -0,0 +1,191 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Milvaion
+  # Abort state of one execution. Safe to read and wait on from any thread.
+  class AbortSignal
+    attr_reader :reason, :message
+
+    def initialize
+      @mutex = Mutex.new
+      @changed = ConditionVariable.new
+      @reason = nil
+      @message = ""
+      @listeners = []
+    end
+
+    def aborted?
+      !@reason.nil?
+    end
+
+    # Sets the reason once; later calls return false.
+    def trigger(reason, message)
+      listeners = @mutex.synchronize do
+        return false unless @reason.nil?
+
+        @reason = reason
+        @message = message
+        @changed.broadcast
+        @listeners.dup
+      end
+
+      listeners.each(&:call)
+      true
+    end
+
+    # Calls the block once when the signal is triggered, right away if it already was.
+    def on_abort(&listener)
+      @mutex.synchronize do
+        if @reason.nil?
+          @listeners << listener
+          return
+        end
+      end
+
+      listener.call
+    end
+
+    # Sleeps up to seconds, waking early on abort. Returns true if aborted.
+    def wait(seconds)
+      deadline = monotonic + seconds.to_f
+
+      @mutex.synchronize do
+        while @reason.nil?
+          remaining = deadline - monotonic
+          break if remaining <= 0
+
+          @changed.wait(@mutex, remaining)
+        end
+      end
+
+      aborted?
+    end
+
+    private
+
+    def monotonic
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+  end
+
+  # What a job handler receives.
+  class JobContext
+    # Id of this execution. Stable across retries and redeliveries, so it works as an idempotency key.
+    attr_reader :occurrence_id
+    # 0 on the first attempt, then 1, 2, ...
+    attr_reader :retry_count
+    # Instance id of this worker process.
+    attr_reader :instance_id
+    # The scheduled job as the API sent it, a Hash with string keys.
+    attr_reader :job
+
+    def initialize(occurrence_id:, job:, retry_count:, instance_id:, abort:, sink:)
+      @occurrence_id = occurrence_id
+      @job = job
+      @retry_count = retry_count
+      @instance_id = instance_id
+      @abort = abort
+      @sink = sink
+      @data_mutex = Mutex.new
+      @parsed = false
+      @data = nil
+    end
+
+    def job_id
+      job["id"]
+    end
+
+    def job_name
+      job["jobNameInWorker"]
+    end
+
+    # Job data as the raw JSON string, or nil.
+    def raw_data
+      job["jobData"]
+    end
+
+    # Parsed job data (string keys), or nil when the job has none. Invalid JSON fails the job permanently.
+    def data
+      @data_mutex.synchronize do
+        unless @parsed
+          raw = raw_data.to_s.strip
+
+          begin
+            @data = raw.empty? ? nil : JSON.parse(raw)
+          rescue JSON::ParserError
+            raise PermanentJobError, "Job data is not valid JSON."
+          end
+
+          @parsed = true
+        end
+      end
+
+      @data
+    end
+
+    # True once the job has been cancelled, has timed out or the worker is shutting down.
+    # Ruby threads cannot be stopped safely from outside, so check this in long loops.
+    def aborted?
+      @abort.aborted?
+    end
+
+    # :cancelled, :timed_out, :shutdown or nil.
+    def abort_reason
+      @abort.reason
+    end
+
+    # Raises JobAborted if the job has been aborted.
+    def raise_if_aborted
+      raise JobAborted.new(@abort.reason, @abort.message) if @abort.aborted?
+    end
+
+    # Sleeps up to seconds, and raises JobAborted as soon as the job is aborted. Use it instead of Kernel#sleep.
+    def sleep(seconds)
+      @abort.wait(seconds)
+      raise_if_aborted
+    end
+
+    # Writes an Information entry to the occurrence log.
+    def log(message, data = nil)
+      write("Information", message, data)
+    end
+    alias info log
+
+    def debug(message, data = nil)
+      write("Debug", message, data)
+    end
+
+    def warn(message, data = nil)
+      write("Warning", message, data)
+    end
+
+    # Writes an Error entry, with the details of error when given.
+    def error(message, error = nil, data = nil)
+      details = (data || {}).dup
+      exception_type = nil
+
+      if error
+        exception_type = Milvaion.error_type_name(error)
+        details["ExceptionType"] = exception_type
+        details["StackTrace"] = Milvaion.format_error(error)
+        details["InnerException"] = error.cause.message if error.cause
+      end
+
+      write("Error", message, details, "UserDefined", exception_type)
+    end
+
+    private
+
+    def write(level, message, data, category = "UserCode", exception_type = nil)
+      @sink.call({
+        "timestamp" => Milvaion.utc_now,
+        "level" => level,
+        "message" => message.to_s,
+        "data" => data,
+        "category" => category,
+        "exceptionType" => exception_type
+      })
+    end
+  end
+end

data/lib/milvaion/errors.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module Milvaion
+  # Base class of the errors this SDK raises.
+  class Error < StandardError; end
+
+  # A failure that retrying cannot fix, such as invalid job data.
+  # A job that raises it is not retried and goes straight to the dead letter queue.
+  class PermanentJobError < Error; end
+
+  # RabbitMQ or Redis could not be reached, or a message was not confirmed by the broker.
+  class ConnectionError < Error; end
+
+  # Why a running job was aborted.
+  module AbortReason
+    # Cancelled from the dashboard.
+    CANCELLED = :cancelled
+    # Ran past its execution timeout.
+    TIMED_OUT = :timed_out
+    # The worker is shutting down; the job goes back to the queue.
+    SHUTDOWN = :shutdown
+  end
+
+  # Raised by JobContext#raise_if_aborted and JobContext#sleep once the job has been aborted.
+  class JobAborted < Error
+    attr_reader :reason
+
+    def initialize(reason, message = nil)
+      super(message || "Job was aborted (#{reason}).")
+      @reason = reason
+    end
+  end
+
+  # Renders an exception the way the .NET SDK does, so the dashboard shows both alike.
+  def self.format_error(error)
+    formatted = +"Type: #{error_type_name(error)}\nMessage: #{error.message}\nStackTrace: #{(error.backtrace || []).join("\n")}\n"
+    formatted << "\nInner Exception:\n" << format_error(error.cause) if error.cause
+
+    formatted
+  end
+
+  def self.error_type_name(error)
+    error.class.name || error.class.to_s
+  end
+end

data/lib/milvaion/job.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module Milvaion
+  # Base class for class-based jobs. Each execution gets a new instance.
+  #
+  #   class SendEmailJob < Milvaion::Job
+  #     job_options max_retries: 5, data_schema: { type: "object" }
+  #
+  #     def perform(ctx)
+  #       ctx.log("Sending to #{ctx.data['to']}")
+  #       { sent: true }
+  #     end
+  #   end
+  #
+  #   worker.job(SendEmailJob)
+  class Job
+    class << self
+      # The name the dashboard dispatches the job by. Defaults to the class name without its namespace.
+      def job_name(name = nil)
+        @job_name = name.to_s unless name.nil?
+        @job_name || self.name.to_s.split("::").last
+      end
+
+      # Job options (see JobOptions). Merged with those of the parent class.
+      def job_options(**options)
+        @job_options = own_job_options.merge(options) unless options.empty?
+        inherited = superclass.respond_to?(:job_options) ? superclass.job_options : {}
+        inherited.merge(own_job_options)
+      end
+
+      private
+
+      def own_job_options
+        @job_options || {}
+      end
+    end
+
+    # Runs the job. The return value is stored as the occurrence result.
+    def perform(_ctx)
+      raise NotImplementedError, "#{self.class} must implement perform(ctx)"
+    end
+  end
+end