stampeded 0.3.1-x86_64-darwin

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ad19088f24b0b655f7688915b34ca674f1299a905c5c9e55bb445c71363556b7
+  data.tar.gz: ea4f514ad4803f51af5c5a7b56d5b9b44d5d10ac70e109084d054a9dc0a278bd
+SHA512:
+  metadata.gz: 9ca24b62e3df42aa0da4b64775d056a2472fafa4f9218cfca9e0a641037b216f0cfca2e62be98cf24824eb68c7188a71a629c341b115df98ce06a92ac6056113
+  data.tar.gz: d59ab4e715d9d56eff4ea08ad9070c7c9ff5b5bc360987903c5d405ead0057e16d11b72cb53de685d353addef6a9e07fe5f205ec574166b919d433e8391d69a7

data/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Abdelkader Boudih
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,206 @@
+# stampeded (Ruby)
+
+Rails integration for [Stampeded](https://github.com/seuros/stampeded): an
+ActiveJob adapter, a worker, and a launcher for the daemon.
+
+```ruby
+gem "stampeded"
+```
+
+```ruby
+# config/application.rb
+config.active_job.queue_adapter = :stampeded
+```
+
+```bash
+bin/rails stampeded:install   # apply the schema (Postgres 18+)
+bin/rails stampeded:work      # run your jobs
+bin/rails stampeded:daemon    # scheduler, reaper, pruner, dashboard
+```
+
+## Two processes
+
+`stampeded:work` runs your jobs. `stampeded:daemon` runs the Rust daemon:
+scheduler, reaper, pruner, expirer, health checks, HTTP API and dashboard. It
+never runs a job itself. Without the daemon, jobs still run, but there is no
+cron, no recovery of jobs from dead workers, no retention and no dashboard.
+
+## Transactional enqueue
+
+```ruby
+Order.transaction do
+  order.save!
+  ShipOrderJob.perform_later(order)   # same transaction
+end                                    # rolled back? the job is gone too
+```
+
+The enqueue is an `INSERT` on your own connection. This only holds when the
+jobs live in the same database as your data; with a separate database
+(`config.connection_class`), the job commits on its own.
+
+## Jobs other runtimes can run
+
+```ruby
+class ParseInvoiceJob < ApplicationJob
+  queue_as :parsing
+  stampeded task: "invoice.parse"
+
+  def perform(document_id:) = ...
+end
+
+ParseInvoiceJob.perform_later(document_id: document.id)
+# task "invoice.parse", args {"document_id": "..."}
+```
+
+Any worker that handles `invoice.parse` can run it: Rust, Go, or this gem's
+worker. Its arguments must be keywords with plain JSON values (no records, times
+or symbols); anything else raises `Stampeded::PortableArgumentsError` at
+enqueue. A job another runtime enqueues as `invoice.parse` runs
+`ParseInvoiceJob` here, with `args` as its keywords.
+
+Every other job works as before. Keyword calls with plain values are stored the
+same standard way; anything else (records, positional arguments) keeps
+ActiveJob's own format, which only a Ruby worker can run.
+
+## Retries
+
+ActiveJob's `retry_on` raises `Stampeded::RetryConflict` when the job class
+loads: it re-enqueues retries as new jobs, while stampeded counts attempts on
+the job itself. Declare stampeded's options instead:
+
+```ruby
+class ChargeCard < ApplicationJob
+  stampeded max_attempts: 5,
+            backoff_base_ms: 1_000, backoff_max_ms: 300_000,
+            dead_letter_queue: "payments_dlq",
+            expire_seconds: 120
+end
+```
+
+- `raise Stampeded::RetryIn.new(seconds)` picks the next delay, e.g. a
+  server's `Retry-After`.
+- `stampeded_attempt` is the attempt number. ActiveJob's `executions` is always
+  1 here.
+- `discard_on` works as usual.
+- Jobs inside installed gems (Rails' `ActiveStorage::AnalyzeJob`, for one) may
+  keep `retry_on`; they get a warning at load instead.
+
+Other options: `unique:` / `unique_period:` (one waiting job per key) and
+`destination:`.
+
+## Continuations
+
+Rails 8.1 continuations work as written:
+
+```ruby
+class ImportJob < ApplicationJob
+  include ActiveJob::Continuable
+
+  def perform(import_id)
+    step :rows, start: 0 do |rows|
+      Row.where(import_id:).find_each(start: rows.cursor) do |row|
+        row.process!
+        rows.advance!(from: row.id)
+      end
+    end
+  end
+end
+```
+
+- Progress is kept on the job's row, so a crashed worker's job resumes too.
+- A deploy interrupts the job at its next checkpoint; it goes back to the queue
+  without spending an attempt and resumes on the same row.
+- An error after progress is an attempt: `max_attempts` and backoff apply, and
+  the next attempt resumes from the last checkpoint.
+
+## Recurring jobs
+
+```ruby
+Stampeded::Schedule.recurring(
+  key:  "nightly-report",
+  job:  ReportJob,
+  cron: "0 3 * * *",
+  args: [Date.current],
+  timezone: "America/New_York"
+)
+```
+
+The daemon fires it; a new schedule first runs at its next occurrence. An
+expression that names its hours runs once on a DST fall-back day; one using `*`
+or a step runs on real elapsed time. `Stampeded::Schedule.pause`, `.remove` and
+`.all` manage them.
+
+## Workers
+
+The pool is threads, which suits IO-bound jobs; for CPU-bound jobs run more
+processes.
+
+```bash
+QUEUE=mail CONCURRENCY=10 bin/rails stampeded:work
+QUEUES=default,mail,reports bin/rails stampeded:work    # one pool, several queues
+QUEUES='*' QUEUES_EXCEPT=payments_dlq bin/rails stampeded:work
+```
+
+Or configure it once:
+
+```ruby
+config.queues = %w[default mail reports]   # or :all
+config.except_queues = %w[payments_dlq]    # with :all
+```
+
+- With named queues, name every queue your jobs use: a job in a queue nobody
+  polls waits forever.
+- With `:all`, the worker serves every queue that has work. Leave out
+  dead-letter queues and queues other languages' workers own: this worker runs
+  everything as ActiveJob and discards a job whose class it cannot find.
+- `SIGTERM` lets in-flight jobs finish before exiting. Jobs still running after
+  `shutdown_timeout` are reclaimed by the daemon.
+- The worker runs on `libstampeded_ffi`, the same core every runtime's worker
+  uses; see [The binaries](#the-binaries). `STAMPEDED_LOG=1` logs each job;
+  `STAMPEDED_LOG=debug` (or `info`, `warn`, ...) also sets the core's level.
+
+## Queue policies
+
+```ruby
+Stampeded::QueuePolicy.set("extraction", max_running: 2)  # at most two at a time
+Stampeded::QueuePolicy.set("mail", max_wait: 60)          # first in line once its next job waits a minute
+```
+
+The daemon turns these into a verdict per queue every second; the worker claims
+rushed queues first and skips held ones. `max_running` also limits what one
+worker process holds from that queue, with or without the daemon.
+
+## Without Rails
+
+```ruby
+Stampeded.config.database_url = ENV["DATABASE_URL"]
+
+Stampeded.enqueue(queue: "mail", task: "welcome", args: { user_id: 42 })
+
+worker = Stampeded::Worker.new(queue: "mail")
+worker.register("welcome") { |job| Mailer.welcome(job.args["user_id"]).deliver_now }
+worker.run
+```
+
+## The binaries
+
+Platform gems include the daemon and the worker's library, `libstampeded_ffi`.
+Otherwise (a platform without them, or the gem from git), build both and point
+at them:
+
+```bash
+cargo build --release -p stampeded -p stampeded-ffi --bin stampeded --lib
+export STAMPEDED_INSTALL_DIR=/path/to/stampeded/target/release
+```
+
+The worker, schedules, steps and job state checks load the library; a process
+that only enqueues needs neither.
+
+## Tests
+
+```bash
+createdb stampeded_test
+STAMPEDED_TEST_DATABASE_URL=postgres://localhost/stampeded_test bundle exec rake test
+```
+
+`rake test` builds the library first, unless `STAMPEDED_INSTALL_DIR` names one.

data/exe/stampeded

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "stampeded/version"
+require "stampeded/errors"
+require "stampeded/daemon"
+
+begin
+  Stampeded::Daemon.exec!(*ARGV)
+rescue Stampeded::DaemonNotFound => e
+  warn e.message
+  exit 1
+end

data/lib/active_job/queue_adapters/stampeded_adapter.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+require "active_job"
+require "stampeded"
+
+module ActiveJob
+  module QueueAdapters
+    class StampededAdapter < AbstractAdapter
+      def initialize(**options)
+        super()
+        @options = options
+      end
+
+      def enqueue(job)
+        job.provider_job_id = insert(job, nil)
+      end
+
+      def enqueue_at(job, timestamp)
+        job.provider_job_id = insert(job, Time.at(timestamp).utc)
+      end
+
+      def enqueue_all(jobs)
+        jobs.count do |job|
+          id = insert(job, job.scheduled_at && Time.at(job.scheduled_at).utc)
+          job.provider_job_id = id
+          !id.nil?
+        end
+      end
+
+      def stopping? = Stampeded.stopping?
+
+      private
+
+      def insert(job, scheduled_at)
+        options = job.class.try(:stampeded_options) || {}
+
+        task, args, meta = Stampeded::Envelope.from_active_job(job)
+
+        Stampeded::Client.enqueue(
+          queue: job.queue_name.to_s,
+          task: task,
+          args: args,
+          meta: meta,
+          scheduled_at: scheduled_at,
+          priority: job.priority,
+          max_attempts: options[:max_attempts],
+          destination: options[:destination],
+          dead_letter_queue: options[:dead_letter_queue],
+          expire_seconds: options[:expire_seconds],
+          unique: options[:unique]&.then { |u| u.respond_to?(:call) ? u.call(job) : u },
+          unique_period: options[:unique_period]&.to_s,
+          backoff_base_ms: options[:backoff_base_ms],
+          backoff_max_ms: options[:backoff_max_ms]
+        )
+      end
+    end
+  end
+end

data/lib/generators/stampeded/install/install_generator.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module Stampeded
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+
+      class_option :queue, type: :string, default: "default",
+                           desc: "Queue to provision a partition for"
+
+      def create_initializer
+        template "stampeded.rb", "config/initializers/stampeded.rb"
+      end
+
+      def install_schema
+        if Stampeded::Schema.installed?
+          say_status :schema, "already installed, skipping", :blue
+        else
+          say_status :schema, "applying stampeded schema (Postgres 18+)"
+          Stampeded::Schema.install!
+        end
+
+        Stampeded.create_queue(options[:queue])
+      rescue StandardError => e
+        say_status :error, e.message, :red
+        say <<~MSG
+          The schema was not applied. Check that the database exists and runs
+          Postgres 18 or newer, then: bin/rails stampeded:install
+        MSG
+      end
+
+      def show_next_steps
+        say <<~MSG
+
+          Stampeded installed.
+
+            config.active_job.queue_adapter = :stampeded    # in config/application.rb
+
+            bin/rails stampeded:work      # runs your jobs
+            bin/rails stampeded:daemon    # scheduler, reaper, pruner, dashboard
+        MSG
+      end
+    end
+  end
+end

data/lib/generators/stampeded/install/templates/stampeded.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+Stampeded.configure do |config|
+  config.concurrency = 5
+
+  config.batch_size = 5
+
+  config.poll_interval = 1.0
+
+  config.heartbeat_interval = 30.0
+
+  config.shutdown_timeout = 25.0
+end

data/lib/stampeded/active_job_ext.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+require_relative "continuable_ext"
+
+module Stampeded
+  module ActiveJobExt
+    def self.included(base)
+      base.extend(ClassMethods)
+    end
+
+    def self.installed_gem?(file)
+      Gem.path.any? { |dir| file.start_with?(File.join(dir, "")) }
+    end
+
+    def stampeded_checkpoint
+      Stampeded::Current.context&.checkpoint
+    end
+
+    def stampeded_checkpoint!(state = nil, conn: nil, **rest)
+      ctx = Stampeded::Current.context
+      raise Stampeded::Error, "no Stampeded job context: is this running under a Stampeded worker?" if ctx.nil?
+
+      ctx.checkpoint!(state || rest, conn: conn)
+    end
+
+    def stampeded_step(name, cursor: nil, &)
+      ctx = Stampeded::Current.context
+      raise Stampeded::Error, "no Stampeded job context: is this running under a Stampeded worker?" if ctx.nil?
+
+      ctx.step(name, cursor: cursor, &)
+    end
+
+    def stampeded_resumed?
+      !Stampeded::Current.context&.checkpoint.nil?
+    end
+
+    def stampeded_attempt
+      Stampeded::Current.context&.attempt
+    end
+
+    module ClassMethods
+      def stampeded_task = @stampeded_task
+
+      def stampeded_options
+        @stampeded_options ||= superclass.respond_to?(:stampeded_options) ? superclass.stampeded_options.dup : {}
+      end
+
+      def stampeded(**options)
+        unknown = options.keys - %i[task max_attempts destination dead_letter_queue
+                                    expire_seconds unique unique_period
+                                    backoff_base_ms backoff_max_ms]
+        raise ArgumentError, "unknown stampeded options: #{unknown.join(", ")}" if unknown.any?
+
+        if (task = options.delete(:task))
+          Stampeded::Envelope.register(task, self)
+          @stampeded_task = task.to_s
+        end
+
+        base = options[:backoff_base_ms]
+        if base && base.to_i > 32_767
+          raise ArgumentError,
+                "backoff_base_ms is at most 32767 (smallint on the row), got #{base}. " \
+                "It is the FIRST retry delay, which doubles from there; the ceiling " \
+                "you want is probably backoff_max_ms, which is an integer."
+        end
+
+        stampeded_options.merge!(options)
+      end
+
+      def retry_on(*, **)
+        if stampeded_adapter?
+          site = caller_locations(1, 1).first&.path.to_s
+          raise Stampeded::RetryConflict, name || "This job" unless Stampeded::ActiveJobExt.installed_gem?(site)
+
+          Kernel.warn "[stampeded] #{name} (#{site}) declares retry_on; its retries are not counted as attempts"
+        end
+
+        super
+      end
+
+      def stampeded_adapter?
+        adapter = respond_to?(:queue_adapter) ? queue_adapter : nil
+        adapter.class.name.to_s.include?("Stampeded")
+      rescue StandardError
+        false
+      end
+    end
+  end
+end

data/lib/stampeded/active_job_worker.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require "stampeded"
+
+module Stampeded
+  class ActiveJobWorker < Worker
+    def initialize(**kwargs)
+      super
+      @handlers.default_proc = proc { |_h, _task| method(:execute_active_job) }
+    end
+
+    private
+
+    def execute_active_job(job)
+      payload = Envelope.to_active_job_payload(id: job.id, queue: job.queue, task: job.task,
+                                               args: job.args, meta: job.meta, checkpoint: job.checkpoint)
+
+      klass = payload["job_class"]
+      begin
+        Object.const_get(klass)
+      rescue NameError
+        raise UnknownTask, "#{klass} is not defined in this worker"
+      end
+
+      Current.with(job) { ActiveJob::Base.execute(payload) }
+    end
+  end
+end

data/lib/stampeded/client.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Stampeded
+  module Client
+    module_function
+
+    def enqueue(queue:, task:, args: {}, meta: {}, scheduled_at: nil, priority: nil,
+                max_attempts: nil, tenant_id: nil, unique: nil, unique_period: nil,
+                destination: nil, dead_letter_queue: nil, expire_seconds: nil,
+                backoff_base_ms: nil, backoff_max_ms: nil)
+      sql = <<~SQL
+        INSERT INTO stampeded_jobs
+          (queue, task, args, tenant_id, priority, max_attempts, scheduled_at,
+           dedup_scope, dedup_period, destination, dead_letter_queue, expire_seconds,
+           backoff_base_ms, backoff_max_ms, meta)
+        VALUES ($1, $2, $3, $4, coalesce($5, 0), coalesce($6, 25), coalesce($7, now()),
+                $8, $9, $10, $11, coalesce($12, 3600),
+                coalesce($13, 1000), coalesce($14, 3600000), $15)
+        ON CONFLICT DO NOTHING
+        RETURNING id
+      SQL
+
+      params = [
+        queue, task, JSON.generate(args.nil? ? {} : args), tenant_id, priority, max_attempts,
+        scheduled_at&.then { |t| t.utc.iso8601(6) },
+        unique, unique_period, destination, dead_letter_queue, expire_seconds,
+        backoff_base_ms, backoff_max_ms, JSON.generate(meta.nil? ? {} : meta)
+      ]
+
+      Connection.translating_errors do
+        Connection.with { |c| c.exec_params(sql, params) }
+                  .then { |r| r.ntuples.zero? ? nil : r[0]["id"] }
+      end
+    end
+
+    def fetch(queue, id)
+      Connection.translating_errors do
+        Connection.with do |c|
+          r = c.exec_params("SELECT * FROM stampeded_jobs WHERE queue = $1 AND id = $2", [queue, id])
+          r.ntuples.zero? ? nil : Job.new(r[0])
+        end
+      end
+    end
+  end
+end

data/lib/stampeded/configuration.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Stampeded
+  class Configuration
+    attr_accessor :database_url
+
+    attr_accessor :default_queue
+
+    attr_accessor :queues
+
+    attr_accessor :except_queues
+
+    attr_accessor :concurrency
+
+    attr_accessor :batch_size
+
+    attr_accessor :poll_interval
+
+    attr_accessor :heartbeat_interval
+
+    attr_accessor :shutdown_timeout
+
+    attr_accessor :connection_class
+
+    attr_accessor :job_timeout
+
+    def initialize
+      @database_url       = ENV.fetch("STAMPEDED_DATABASE_URL", nil) || ENV.fetch("DATABASE_URL", nil)
+      @default_queue      = "default"
+      @queues             = nil
+      @except_queues      = nil
+      @concurrency        = 5
+      @batch_size         = 5
+      @poll_interval      = 1.0
+      @heartbeat_interval = 30.0
+      @shutdown_timeout   = 25.0
+      @connection_class   = nil
+      @job_timeout        = 900.0
+    end
+  end
+end

data/lib/stampeded/connection.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+require "pg"
+require "uri"
+
+module Stampeded
+  module Connection
+    module_function
+
+    def with(&block)
+      if active_record?
+        ar_class.connection.raw_connection.then(&block)
+      else
+        own(&block)
+      end
+    end
+
+    def own
+      conn = PG.connect(connection_params)
+      begin
+        yield conn
+      ensure
+        conn.close
+      end
+    end
+
+    def connection_params
+      explicit = Stampeded.config.database_url
+
+      if active_record?
+        cfg = ar_class.connection_db_config.configuration_hash
+        return {
+          host: cfg[:host] || "localhost",
+          port: cfg[:port],
+          dbname: cfg[:database],
+          user: cfg[:username],
+          password: cfg[:password],
+          **cfg.slice(*TLS_KEYS)
+        }.compact
+      end
+
+      explicit or raise Error, <<~MSG
+        No database configured.
+
+        Set Stampeded.config.database_url, or STAMPEDED_DATABASE_URL, or run
+        inside a Rails app where ActiveRecord is configured.
+      MSG
+    end
+
+    TLS_KEYS = %i[sslmode sslrootcert sslcert sslkey].freeze
+
+    def url(params = connection_params)
+      return params if params.is_a?(String)
+
+      query = params.filter_map { |k, v| "#{k}=#{URI.encode_www_form_component(v.to_s)}" unless v.nil? }
+      "postgres://?#{query.join("&")}"
+    end
+
+    def translating_errors
+      yield
+    rescue PG::UndefinedTable, PG::UndefinedFunction => e
+      raise SchemaMissing, e if e.message.include?("stampeded_")
+
+      raise
+    end
+
+    def ar_class
+      name = Stampeded.config.connection_class
+      return ActiveRecord::Base if name.nil?
+
+      Object.const_get(name.to_s)
+    rescue NameError
+      raise Error, "Stampeded.config.connection_class is #{name.inspect}, which does not exist"
+    end
+
+    def active_record?
+      defined?(ActiveRecord::Base) && ActiveRecord::Base.respond_to?(:connection_db_config)
+    rescue StandardError
+      false
+    end
+  end
+end