belt-jobs 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3acaa084dabde8ed9d6cb0dd4ef975f2be00e7c81232d15e2c62c27a01f87be6
+  data.tar.gz: ed45b7ecf991df5f91b846dff0d453d8f6b8a09b1f18f0164dfee9bf83e2425d
+SHA512:
+  metadata.gz: 0704ada74c04c57dc117d139ee457b5d061f6837ba878ab781e48ca6371ec5a1b99c683f853dc4ee1501d38f675d480fe416a047a601df54029eed73f38c2b87
+  data.tar.gz: 757b0bd862efe1a9d83babcaa23818d901a34221a1700cac8dd4c5371ed66bd3020979e8d4d55670a15884709a4f1d71f70d87a34d6ef5e921cfca97e8483f95

data/AGENTS.md

@@ -0,0 +1,53 @@
+# AGENTS.md — belt-jobs
+
+This is the Belt Active Job plugin. Host apps own their job classes and generated Terraform; runtime behavior stays in this gem.
+
+## Public contract
+
+- `ApplicationJob < ActiveJob::Base`
+- `perform_now`, `perform_later`, `set(wait:)`, `set(wait_until:)`
+- `retry_on`, `discard_on`, callbacks, and Active Job serialization
+- `belt g jobs`, `belt d jobs`, `belt g job NAME`, `belt d job NAME`
+
+Immediate and <=900-second delays use SQS. Longer delays use one-time EventBridge Scheduler schedules targeting the same queue. Recurring Terraform schedules also target SQS. Execution is at least once.
+
+## Layout
+
+| Path | Responsibility |
+|---|---|
+| `lib/belt/jobs/queue_adapter.rb` | Active Job enqueue and delay routing |
+| `lib/belt/jobs/worker.rb` | allowlisted execution + partial batch failures |
+| `lib/belt/jobs/lambda_handler.rb` | Lambda Loadout lifecycle |
+| `lib/belt/jobs/schedule.rb` | friendly schedule compiler |
+| `lib/belt/jobs/recurring_config.rb` | recurring YAML validation |
+| `lib/belt/generators/jobs_generator.rb` | shared infra installer/destroyer |
+| `lib/belt/generators/job_generator.rb` | singular class/schedule generator |
+| `lib/belt/jobs/templates/` | host-owned Terraform/config/Lambda files |
+
+Belt discovers both generator files by path and class name; do not add a separate registry.
+
+## Development
+
+```sh
+bundle install
+bundle exec rspec
+bundle exec rubocop
+bundle exec rake build
+```
+
+For generator smoke testing, create a temporary standard Belt app module containing `infrastructure/modules/app/main.tf` and `lambda/config/environment.rb`, invoke the generators from that root, run `terraform fmt -check`, then exercise both destroy paths.
+
+## Invariants
+
+1. Never exceed the configured 256 KiB serialized payload cap.
+2. Keep the exact 900-second SQS/Scheduler routing boundary.
+3. Validate a job class name against loaded Active Job descendants before constant resolution.
+4. Return only failed SQS message IDs in `batchItemFailures`.
+5. Scheduler targets SQS, never Lambda directly.
+6. Recurring config accepts job classes and JSON-like args only—never eval or arbitrary commands.
+7. Generator-owned host edits must be marked and reversible.
+8. Preserve `retry_on` / `discard_on` by executing through Active Job.
+
+## Deferred beyond 0.0.1
+
+FIFO queues, numeric priority, distributed keyed concurrency, workflows/chains, dynamic schedule CRUD/UI, and automatic DLQ redrive are intentionally out of scope.

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## 0.0.1 - 2026-09-30
+
+### Added
+
+- Active Job queue adapter backed by SQS for immediate and <=15-minute delayed jobs.
+- One-time EventBridge Scheduler delivery for longer `wait` / `wait_until` jobs.
+- Lambda worker with loaded-class allowlisting and per-record `batchItemFailures`.
+- Belt Lambda Loadout logging and `JobEnqueued`, `JobSucceeded`, and `JobFailed` metrics.
+- `belt g jobs` installer for SQS, execution and Scheduler DLQs, IAM, alarms, Lambda config, and reversible host wiring.
+- `belt g job NAME` generator with queue, friendly/native schedule, and timezone options.
+- Validated `config/recurring.yml` definitions compiled to EventBridge Scheduler resources.
+- 256 KiB payload limit and documented at-least-once/idempotency contract.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Stowzilla
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,136 @@
+# belt-jobs
+
+Active Job for Belt applications on AWS Lambda: SQS-backed execution, delayed jobs, recurring EventBridge schedules, DLQs, alarms, and Rails-style generators.
+
+## Installation
+
+```ruby
+# Gemfile
+gem "belt-jobs", "0.0.1"
+```
+
+```sh
+bundle install
+belt g jobs
+belt g job nightly_cleanup
+```
+
+`belt g jobs` installs and wires the queue, worker Lambda, Scheduler IAM, recurring schedules, DLQs, alarms, and `ApplicationJob`. `belt g job` auto-installs that shared infrastructure if needed.
+
+## Active Job API
+
+Generated jobs are ordinary Active Job classes:
+
+```ruby
+class NightlyCleanupJob < ApplicationJob
+  queue_as :default
+  retry_on Net::ReadTimeout, wait: :polynomially_longer, attempts: 5
+
+  def perform(account_id)
+    # Keep jobs idempotent: SQS and Lambda provide at-least-once delivery.
+  end
+end
+```
+
+Use the standard API from controllers, models, the console, or other jobs:
+
+```ruby
+NightlyCleanupJob.perform_now(account.id)
+NightlyCleanupJob.perform_later(account.id)
+NightlyCleanupJob.set(wait: 10.minutes).perform_later(account.id)
+NightlyCleanupJob.set(wait_until: tomorrow.noon).perform_later(account.id)
+```
+
+Immediate jobs and delays up to 15 minutes use SQS directly. Longer delays create a one-time EventBridge Scheduler schedule that deletes itself after delivery. Every path targets the same SQS queue and worker Lambda.
+
+## Recurring jobs
+
+Generate and schedule in one command:
+
+```sh
+belt g job daily_digest \
+  --schedule "every day at 9am" \
+  --timezone America/New_York
+```
+
+Supported friendly schedules are `every N minutes|hours|days` and `every day at 9am` / `daily 09:00`. Native AWS `cron(...)` and `rate(...)` expressions are accepted directly. The generator writes `config/recurring.yml`:
+
+```yaml
+daily_digest:
+  class: DailyDigestJob
+  args: []
+  schedule: cron(0 9 * * ? *)
+  timezone: America/New_York
+  queue: default
+  enabled: true
+```
+
+Recurring arguments must be YAML/JSON values. There is deliberately no arbitrary `command:` evaluation.
+
+## Generated files
+
+```text
+infrastructure/modules/jobs/       SQS, DLQs, Scheduler, IAM, alarms
+config/lambda/jobs.yml             15-minute worker + partial batch failures
+config/recurring.yml               recurring definitions
+lambda/jobs.rb                     Lambda entrypoint
+lambda/jobs/application_job.rb     app-owned Active Job base
+lambda/jobs/*_job.rb               generated jobs
+```
+
+The installer updates the standard `infrastructure/modules/app/main.tf` and `lambda/config/environment.rb`. It is idempotent, supports `--force`, and `belt d jobs` removes its files and marked wiring. `belt d job NAME` removes one class and its recurring entry.
+
+## Configuration
+
+Terraform supplies production settings through environment variables. Tests and unusual deployments can configure the runtime directly:
+
+```ruby
+Belt::Jobs.configure do |config|
+  config.queue_url = "https://sqs.us-east-1.amazonaws.com/123/jobs"
+  config.queue_arn = "arn:aws:sqs:us-east-1:123:jobs"
+  config.scheduler_role_arn = "arn:aws:iam::123:role/jobs-scheduler"
+  config.allowed_job_classes = [NightlyCleanupJob]
+end
+Belt::Jobs.configure_active_job!
+```
+
+| Environment variable | Purpose |
+|---|---|
+| `BELT_JOBS_QUEUE_URL` | SQS enqueue URL |
+| `BELT_JOBS_QUEUE_ARN` | Scheduler target ARN |
+| `BELT_JOBS_SCHEDULER_ROLE_ARN` | Role Scheduler assumes to send to SQS |
+| `BELT_JOBS_SCHEDULER_DLQ_ARN` | Failed Scheduler delivery queue |
+| `BELT_JOBS_SCHEDULE_GROUP_NAME` | Group for one-time and recurring schedules |
+
+By default, the worker allowlists the loaded concrete `ApplicationJob` descendants before resolving a class name. Set `allowed_job_classes` to tighten this further.
+
+## Reliability contract
+
+- Delivery is **at least once**. Job code must be idempotent.
+- `retry_on`, `discard_on`, callbacks, serialization, and `perform_now` are Active Job behavior.
+- Handled `retry_on` errors enqueue a replacement and acknowledge the current SQS record.
+- Unhandled, malformed, or disallowed records are returned in `batchItemFailures`; SQS retries them and then moves them to the execution DLQ.
+- Serialized payloads are capped at 256 KiB. Prefer IDs over large object graphs.
+- Jobs have Lambda's 900-second execution ceiling.
+- Long and recurring schedules have EventBridge Scheduler's one-minute delivery precision.
+- The generated worker uses bounded concurrency and emits Belt `JobEnqueued`, `JobSucceeded`, and `JobFailed` metrics.
+
+## Removing
+
+```sh
+belt d job nightly_cleanup
+belt d jobs
+```
+
+## Development
+
+```sh
+bundle install
+bundle exec rspec
+bundle exec rubocop
+bundle exec rake build
+```
+
+## License
+
+MIT

data/lib/belt/generators/job_generator.rb

@@ -0,0 +1,170 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'yaml'
+require_relative 'jobs_generator'
+require_relative '../jobs'
+
+module Belt
+  module Generators
+    class JobGenerator
+      def self.description
+        'Generate an Active Job class (optionally recurring)'
+      end
+
+      def self.run(args)
+        return print_help if args.include?('--help') || args.include?('-h')
+
+        new(args).generate
+      end
+
+      def self.destroy(args)
+        new(args).destroy
+      end
+
+      def self.print_help
+        puts <<~HELP
+          Generate an Active Job class.
+
+          Usage: belt generate job NAME [options]
+
+          Options:
+            --queue NAME           Active Job queue name (default: default)
+            --schedule SCHEDULE    cron(...), rate(...), "every N minutes", or "every day at 9am"
+            --timezone ZONE        EventBridge schedule timezone (default: UTC)
+            --force                Overwrite an existing job
+
+          Examples:
+            belt g job nightly_cleanup
+            belt g job daily_digest --schedule "every day at 9am" --timezone America/New_York
+            belt d job daily_digest
+        HELP
+      end
+
+      def initialize(args)
+        @options = parse(args)
+        @name = @options.delete(:name)
+        @force = @options.delete(:force)
+      end
+
+      def generate
+        abort_with_usage unless @name
+        JobsGenerator.run([]) unless File.exist?('lambda/jobs/application_job.rb')
+        destination = "lambda/jobs/#{underscored_name}_job.rb"
+        if File.exist?(destination) && !@force
+          puts "  skip    #{destination} (already exists, use --force to overwrite)"
+        else
+          FileUtils.mkdir_p(File.dirname(destination))
+          File.write(destination, job_source)
+          puts "  create  #{destination}"
+        end
+        add_recurring_schedule if @options[:schedule]
+        puts "\n✓ Generated #{class_name}"
+      end
+
+      def destroy
+        abort_with_usage unless @name
+        destination = "lambda/jobs/#{underscored_name}_job.rb"
+        FileUtils.rm_f(destination)
+        remove_recurring_schedule
+        puts "\n✓ Removed #{class_name}"
+      end
+
+      private
+
+      def parse(args)
+        options = { queue: 'default', timezone: 'UTC', force: false }
+        index = 0
+        while index < args.length
+          argument = args[index]
+          case argument
+          when '--queue', '--schedule', '--timezone'
+            index += 1
+            options[argument.delete_prefix('--').to_sym] = args[index]
+          when /\A--(queue|schedule|timezone)=(.*)\z/
+            options[Regexp.last_match(1).to_sym] = Regexp.last_match(2)
+          when '--force'
+            options[:force] = true
+          else
+            raise Belt::Jobs::ConfigurationError, "unknown option #{argument}" if argument.start_with?('-')
+
+            options[:name] ||= argument
+          end
+          index += 1
+        end
+        options
+      end
+
+      def underscored_name
+        @name.to_s.gsub('::', '/').gsub(/([a-z\d])([A-Z])/, '\\1_\\2').tr('-', '_').downcase.sub(/_job\z/, '')
+      end
+
+      def class_name
+        "#{underscored_name.split('/').map do |part|
+          part.split('_').map(&:capitalize).join
+        end.join('::')}Job"
+      end
+
+      def schedule_key
+        underscored_name.tr('/', '_')
+      end
+
+      def job_source
+        modules = class_name.split('::')
+        klass = modules.pop
+        indent = ''
+        source = +"# frozen_string_literal: true\n\n"
+        modules.each do |mod|
+          source << "#{indent}module #{mod}\n"
+          indent += '  '
+        end
+        source << <<~RUBY.gsub(/^/, indent)
+          class #{klass} < ApplicationJob
+            queue_as :#{@options[:queue]}
+
+            def perform(*args)
+              # Do work here. Jobs are at-least-once; keep this method idempotent.
+            end
+          end
+        RUBY
+        modules.reverse_each do
+          indent = indent[0...-2]
+          source << "#{indent}end\n"
+        end
+        source
+      end
+
+      def add_recurring_schedule
+        path = 'config/recurring.yml'
+        config = File.exist?(path) ? (YAML.safe_load_file(path, aliases: true) || {}) : {}
+        config[schedule_key] = {
+          'class' => class_name,
+          'args' => [],
+          'schedule' => Belt::Jobs::Schedule.compile(@options[:schedule]),
+          'timezone' => @options[:timezone],
+          'queue' => @options[:queue],
+          'enabled' => true
+        }
+        Belt::Jobs::RecurringConfig.new(config).validate!
+        FileUtils.mkdir_p(File.dirname(path))
+        File.write(path, YAML.dump(config))
+        puts "  update  #{path}"
+      end
+
+      def remove_recurring_schedule
+        path = 'config/recurring.yml'
+        return unless File.exist?(path)
+
+        config = YAML.safe_load_file(path, aliases: true) || {}
+        return unless config.delete(schedule_key)
+
+        File.write(path, YAML.dump(config))
+        puts "  update  #{path}"
+      end
+
+      def abort_with_usage
+        raise Belt::Jobs::ConfigurationError, 'Usage: belt generate job NAME [options]'
+      end
+    end
+  end
+end

data/lib/belt/generators/jobs_generator.rb

@@ -0,0 +1,224 @@
+# frozen_string_literal: true
+
+require 'erb'
+require 'fileutils'
+
+module Belt
+  module Generators
+    class JobsGenerator
+      TEMPLATE_DIR = File.expand_path('../jobs/templates', __dir__)
+      APP_MODULE = 'infrastructure/modules/app/main.tf'
+      ENVIRONMENT = 'lambda/config/environment.rb'
+      JOBS_MODULE_MARKER = /\n?# belt-jobs:module:begin\n.*?# belt-jobs:module:end\n?/m
+      ENV_REFS_MARKER = /\s*# belt-jobs:env-refs:begin\n.*?# belt-jobs:env-refs:end/m
+      SHARED_IAM_MARKER = /\s*# belt-jobs:shared-iam:begin\n.*?# belt-jobs:shared-iam:end/m
+      BOOT_MARKER = /\n# belt-jobs:boot:begin\n.*?# belt-jobs:boot:end\n?/m
+
+      def self.description
+        'Install Active Job on SQS + Lambda with delayed and recurring schedules'
+      end
+
+      def self.run(args)
+        return print_help if args.include?('--help') || args.include?('-h')
+
+        new(args).generate
+      end
+
+      def self.destroy(args)
+        new(args).destroy
+      end
+
+      def self.print_help
+        puts <<~HELP
+          Install Active Job infrastructure for a Belt application.
+
+          Usage: belt generate jobs [options]
+
+          Options:
+            --force               Overwrite generated files
+            -h, --help            Show this help
+
+          Creates:
+            infrastructure/modules/jobs/   SQS, DLQs, Scheduler, IAM, alarms
+            config/lambda/jobs.yml         Worker Lambda config and SQS trigger
+            config/recurring.yml           Recurring jobs map
+            lambda/jobs.rb                 SQS Lambda entry point
+            lambda/jobs/application_job.rb Active Job base class
+
+          The generator also wires the jobs module, Lambda environment references,
+          producer IAM, shared job packaging, and app boot into the standard Belt app module.
+
+          Examples:
+            belt g jobs
+            belt g job nightly_cleanup
+            belt d jobs
+        HELP
+      end
+
+      def initialize(args)
+        @force = args.include?('--force')
+      end
+
+      def generate
+        generate_files
+        wire_app_module
+        wire_environment
+        puts <<~SUCCESS
+
+          ✓ Jobs installed!
+
+          Generate a job:
+            belt g job nightly_cleanup
+            belt g job daily_digest --schedule "every day at 9am" --timezone America/New_York
+
+          Then deploy normally:
+            belt deploy <env>
+        SUCCESS
+      end
+
+      def destroy
+        FileUtils.rm_rf('infrastructure/modules/jobs')
+        FileUtils.rm_rf('lambda/jobs')
+        %w[config/lambda/jobs.yml config/recurring.yml lambda/jobs.rb].each do |path|
+          FileUtils.rm_f(path)
+        end
+        unwire_app_module
+        unwire_environment
+        puts "\n✓ Jobs removed!"
+      end
+
+      private
+
+      def generate_files
+        {
+          'terraform/main.tf.erb' => 'infrastructure/modules/jobs/main.tf',
+          'terraform/variables.tf.erb' => 'infrastructure/modules/jobs/variables.tf',
+          'terraform/outputs.tf.erb' => 'infrastructure/modules/jobs/outputs.tf',
+          'config/jobs.yml.erb' => 'config/lambda/jobs.yml',
+          'config/recurring.yml.erb' => 'config/recurring.yml',
+          'lambda/jobs.rb.erb' => 'lambda/jobs.rb',
+          'lambda/application_job.rb.erb' => 'lambda/jobs/application_job.rb'
+        }.each { |template, destination| write_template(template, destination) }
+      end
+
+      def wire_app_module
+        return warn_missing(APP_MODULE) unless File.exist?(APP_MODULE)
+
+        content = File.read(APP_MODULE)
+        original = content.dup
+        unless content.include?('# belt-jobs:module:begin')
+          block = <<~HCL
+            # belt-jobs:module:begin
+            module "jobs" {
+              source      = "../jobs"
+              app_name    = var.app_name
+              environment = var.environment
+              aws_region  = var.aws_region
+            }
+            # belt-jobs:module:end
+
+          HCL
+          content.sub!('resource "conveyor_belt" "main" {', "#{block}resource \"conveyor_belt\" \"main\" {")
+        end
+
+        unless content.include?('# belt-jobs:env-refs:begin')
+          content.sub!(/^(\s*)lambda_env_refs\s*=\s*var\.lambda_env_refs\s*$/) do
+            indent = Regexp.last_match(1)
+            <<~HCL.chomp
+              #{indent}# belt-jobs:env-refs:begin
+              #{indent}lambda_env_refs = merge(var.lambda_env_refs, {
+              #{indent}  belt_jobs_queue_url          = module.jobs.queue_url
+              #{indent}  belt_jobs_queue_arn          = module.jobs.queue_arn
+              #{indent}  belt_jobs_scheduler_role_arn = module.jobs.scheduler_role_arn
+              #{indent}  belt_jobs_scheduler_dlq_arn  = module.jobs.scheduler_dlq_arn
+              #{indent}  belt_jobs_schedule_group     = module.jobs.schedule_group_name
+              #{indent}})
+              #{indent}# belt-jobs:env-refs:end
+            HCL
+          end
+        end
+
+        unless content.include?('shared_iam_policy_arns')
+          content.sub!(/\n}\s*\z/, <<~HCL)
+
+              # belt-jobs:shared-iam:begin
+              shared_iam_policy_arns = [module.jobs.producer_policy_arn]
+              # belt-jobs:shared-iam:end
+            }
+          HCL
+        end
+
+        content.sub!(/lambda_shared_dirs\s*=\s*\[([^\]]*)\]/) do |match|
+          next match if match.include?('"jobs"')
+
+          "#{match.sub(/\]\z/, ', "jobs"]')} # belt-jobs:shared-dir"
+        end
+        write_update(APP_MODULE, original, content)
+      end
+
+      def wire_environment
+        return warn_missing(ENVIRONMENT) unless File.exist?(ENVIRONMENT)
+
+        content = File.read(ENVIRONMENT)
+        return if content.include?('jobs_dir =') || content.include?('# belt-jobs:boot:begin')
+
+        original = content.dup
+        content << <<~RUBY
+
+          # belt-jobs:boot:begin
+          jobs_dir = File.join(__dir__, '..', 'jobs')
+          all_jobs = Dir[File.join(jobs_dir, '**', '*.rb')].sort
+          application_job = all_jobs.find { |file| File.basename(file) == 'application_job.rb' }
+          require application_job if application_job
+          (all_jobs - [application_job].compact).each { |file| require file }
+          # belt-jobs:boot:end
+        RUBY
+        write_update(ENVIRONMENT, original, content)
+      end
+
+      def unwire_app_module
+        return unless File.exist?(APP_MODULE)
+
+        content = File.read(APP_MODULE)
+        original = content.dup
+        content.gsub!(JOBS_MODULE_MARKER, '')
+        content.gsub!(ENV_REFS_MARKER, "\n  lambda_env_refs = var.lambda_env_refs")
+        content.gsub!(SHARED_IAM_MARKER, '')
+        content.gsub!(/,\s*"jobs"\]\s*# belt-jobs:shared-dir/, ']')
+        write_update(APP_MODULE, original, content)
+      end
+
+      def unwire_environment
+        return unless File.exist?(ENVIRONMENT)
+
+        content = File.read(ENVIRONMENT)
+        original = content.dup
+        content.gsub!(BOOT_MARKER, "\n")
+        write_update(ENVIRONMENT, original, content)
+      end
+
+      def write_template(template, destination)
+        if File.exist?(destination) && !@force
+          puts "  skip    #{destination} (already exists)"
+          return
+        end
+
+        source = File.join(TEMPLATE_DIR, template)
+        FileUtils.mkdir_p(File.dirname(destination))
+        File.write(destination, ERB.new(File.read(source), trim_mode: '-').result(binding))
+        puts "  create  #{destination}"
+      end
+
+      def write_update(path, original, content)
+        return if original == content
+
+        File.write(path, content)
+        puts "  update  #{path}"
+      end
+
+      def warn_missing(path)
+        puts "  warn    #{path} not found; wire the jobs module manually"
+      end
+    end
+  end
+end