geneva_drive 0.4.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 277866fbccabdb2fc26e37af632234df751620057a2b42d94607d8ad5fa180e7
-  data.tar.gz: 054417cf89b79e8534e6e3268bb8577df7f5d21da6b0ccc37b8dce1862596377
+  metadata.gz: a4a73216fcf8355550c3b5e26e76eee14468eb301e8eb315a979bc8f34bdb670
+  data.tar.gz: a9b9cc0d8998f6c479806158dd12725774431e7dba8486c5781ed68f238175b5
 SHA512:
-  metadata.gz: '013668a74a368b8ed7b75b75d0536c5ffd3d7f4ed8a6f110ff740f186ede6dbe79eb6f3e480295c895b280dd64f804ca4e701427c05543aa03e5b3637ee1c702'
-  data.tar.gz: 7bc7f954847dd03f4b2501ae84bc4395a7c51d22681a289e58f0d49e3ae0d5dc00f66bd14f437abfb99de114b7d72aac22afafb7bc7aeebcc60e56e9b83c7d7b
+  metadata.gz: b77f44d451d256d081af6d7b30d62bcaa46bfd41b0606cdcd1d07529e18db7ff5e2d1753d52f6bd24f600f92b7e14587146072a53e46e1052728c760cf6ded78
+  data.tar.gz: 9bdd9f474cd8370ebcc77b4126617833a249b6e1237ed8021463b66d082db63e17bc024fa8718ec3aee1d031731c72b3a9c90903d2ce40ced2f01eb0b1c421ef

data/CHANGELOG.md

@@ -2,6 +2,44 @@
 
 ## [Unreleased]
 
+## [0.6.0]
+
+- Add resumable steps for cursor-based iteration over large collections. `resumable_step :name do |iter| ... end` passes an `IterableStep` (API-compatible with Rails 8.1's `ActiveJob::Continuation::Step`) whose cursor is checkpointed to the database, so the step can be interrupted — by `max_iterations:`, `max_runtime:`, queue shutdown, external pause/cancel, or explicit `suspend!`/`skip_to!` — and continued in a later job execution from where it left off. Interruption completes the current step execution with outcome `continued` and creates a successor execution linked via `continues_from_id`, keeping a full audit chain instead of introducing a suspended state. Resumable steps run through the same `Executor` as regular steps: exception policies (step- and class-level, `max_reattempts`, `terminal_action:`, `report:`), tagged/injected loggers, `Rails.error` context, and merged job options all apply, with `reattempt!` and `resume!` continuing from the last checkpoint (use `reattempt!(rewind: true)` to start over). `HousekeepingJob` recovery is cursor-aware. The feature needs two new columns (`cursor`, `continues_from_id`) added by the installer migration; until it runs, everything else degrades gracefully (regular steps, pause/resume and housekeeping are unaffected) and executing an actual `resumable_step` raises a configuration error pointing at the missing migration. Cursors are limited to 128 KB of serialized JSON by default — the cursor is a position marker, not a payload — configurable via `GenevaDrive.max_cursor_size` (`nil` disables the check). Test helpers: `speedrun_current_step` follows the execution chain, `run_iterations(workflow, count:)` exercises partial progress, plus `assert_cursor` and `assert_step_has_successor`.
+- Validate `job_options` uniformly across `set_step_job_options`, per-step `job_options:`, and per-instance `workflow.step_job_options=`. Only the keys Active Job's `set` actually consumes (`:queue`, `:priority`, `:wait`, `:wait_until`) are accepted, and values are type-checked. Typos like `piority:` now raise `StepConfigurationError` at configuration time instead of being silently ignored.
+- Add partial index on `geneva_drive_step_executions(started_at) WHERE state = 'in_progress'` and order `HousekeepingJob` recovery batches by the filter column (`started_at` for stuck-in-progress, `scheduled_for` for stuck-scheduled). Fixes `HousekeepingJob` timing out on large in-progress sets where most rows are fresh — the planner previously walked the whole `state = 'in_progress'` set looking for old rows and tripped the caller's `statement_timeout`. Fixes AppSignal incident b9a7d1d1.
+- Add a normalized `geneva_drive.paused_ratio` gauge emitted by `HousekeepingJob`. For each workflow class (STI `type`) it reports a float in `0.0..1.0`, tagged `workflow: <ClassName>`, equal to that class's `paused` count divided by its total population (all states). Unlike absolute paused counts, this is bounded and easy to alert on (50 paused of 50 is a fire; 50 of 500,000 is noise). Derived from the existing single grouped query — no extra database load.
+- Add optional `metadata` JSON column to workflows (mirrors the existing step executions pattern). Exposes `step_job_options` / `step_job_options=` for per-instance job option overrides (queue, priority, etc.) that merge with class-level `set_step_job_options` and persist across step boundaries. Dedupe is unaffected since metadata is not part of the uniqueness constraint.
+- Add GenevaDrive workflow and step metadata to `Rails.error` execution context when steps execute, so Rails error reports include workflow id/class, step execution id/name, and hero identifiers.
+- Add per-step Active Job option overrides with `step :name, job_options: { queue: :critical, priority: 0 }`. Step options take precedence over workflow-level `set_step_job_options` and per-instance options, and are preserved across reattempts and resume re-enqueueing.
+- Add `report:` option to exception policies and class-level `on_exception` for controlling when exceptions are reported to `Rails.error.report`. Accepts `:always` (default — preserves existing behavior), `:never` (suppress reporting for expected exceptions like rate limits), or `:terminal_only` (suppress during reattempts, report only when `terminal_action` fires). The executor now defers error reporting until after policy resolution.
+
+## [0.5.0]
+
+- Add `ongoing?` predicate and model-level uniqueness validation for ongoing workflows. Mirrors the database unique index on `(type, hero_type, hero_id)` so that `Workflow.create` (without bang) returns validation errors instead of raising a database constraint violation.
+- Allow `hero` to be NULL on workflows. The model declares `belongs_to :hero, optional: true` but the migration enforced `NOT NULL` on `hero_type` and `hero_id`. New installs get nullable columns; existing installs get an upgrade migration (Postgres/MySQL via `change_column_null`, SQLite via atomic table swap).
+- Detect dominant UUID default function from application tables when generating migrations. When the host app uses a custom UUID default (e.g. `uuidv7()` instead of `gen_random_uuid()`), GenevaDrive tables now inherit the same default by quorum from existing tables.
+- Add class-level `on_exception` for declaring exception policies that apply to all steps in a workflow. Supports blanket policies, exception-class-specific policies, and imperative block handlers. Step-level `on_exception:` overrides class-level policies.
+- Add `GenevaDrive::ExceptionPolicy` value object for reusable exception handling configuration. Can be passed directly to `on_exception:` at step level or class level.
+- Add `terminal_action:` option to `on_exception: :reattempt!` to control what happens when `max_reattempts` is exceeded. Accepts `:pause!` (default), `:cancel!`, or `:skip!` (to skip the failed step and continue the workflow).
+- Add optional `metadata` JSON column to step executions for structured data (reattempt reasons, exception info). The column is optional — writing is a silent no-op if the migration has not been applied yet.
+- Double-write exception info (class, message, backtrace) into step execution metadata alongside the dedicated error columns, preparing for future column removal.
+- Add injectable logger support to `Executor.execute!`. Callers (background jobs, controllers) can pass a `logger:` parameter to inject a pre-tagged logger as the base for all workflow logging during step execution. The injected logger will have workflow and step-specific tags added on top. `PerformStepJob` now passes its own logger to the executor.
+- Add `GenevaDrive.enqueue_after_commit` configuration setting to control job enqueueing behavior. When `true` (default in production), jobs are enqueued via `after_all_transactions_commit` to ensure records are visible to workers. When `false` (default in test), jobs are enqueued inline to avoid issues with transactional tests that never commit.
+- Fix MySQL compatibility in `HousekeepingJob`: wrap LIMIT subqueries in an extra SELECT (MySQL doesn't support LIMIT in IN subqueries), and interpolate LIMIT values directly (MySQL doesn't handle bind parameters for LIMIT clauses).
+- Fix MySQL foreign key compatibility in generator: add foreign keys separately using `add_foreign_key` and skip for MySQL due to unsigned/signed bigint type mismatch.
+- Fix Rails 8 deprecation: use `false` instead of `:never` for `enqueue_after_transaction_commit` setting.
+- Fix double-deferral bug where `PerformStepJob` could silently fail to enqueue with SolidQueue and other adapters that opt into `enqueue_after_transaction_commit`. When `perform_later` was called from inside an `after_all_transactions_commit` callback, ActiveJob could see the transaction as still "open" and defer the queue INSERT into a second callback that never fires — leaving the step execution with a `job_id` but no corresponding job in the queue backend. Fix: set `enqueue_after_transaction_commit = :never` on `PerformStepJob` (GenevaDrive already handles its own transaction-awareness) and add a poll-retry on step execution lookup as defense-in-depth against replication lag.
+- Reduce default `stuck_scheduled_threshold` from 1 hour to 15 minutes for faster recovery when jobs are lost.
+- Change `pause!` to preserve scheduled step executions instead of canceling them. Previously, calling `pause!` would cancel the scheduled execution with outcome "workflow_paused". Now, the scheduled execution remains in "scheduled" state, making it visible in the timeline as "overdue" if time passes while paused. On `resume!`, the same execution is re-enqueued (or a new one created only if the executor canceled it while paused).
+- Improve `HousekeepingJob` to process all eligible records by looping through batches instead of stopping after the first batch. Also uses efficient SQL DELETEs with INNER JOIN for step executions cleanup and fixes cutoff times at the start of each operation for deterministic behavior.
+- Add `GenevaDrive.with_inline_enqueue` block method for bulk workflow creation. Temporarily disables deferred job enqueueing so that bulk enqueueing libraries (e.g., BulkEnqueue) can capture and batch job inserts. Only recommended for use with co-committing, database-backed ActiveJob adapters (SolidQueue, GoodJob, Gouda) on the same database.
+- Extract `CombinedExceptionPolicy` to encapsulate composable exception policy resolution. Multiple policies can be combined in an array — resolution walks them in order and enforces a global reattempt cap (minimum `max_reattempts` across all constituents). Both declarative and imperative (block) policies support `matching:` for targeting specific exception classes.
+- Add workflow state gauges via Measurometer. The housekeeping job now reports workflow count gauges (`geneva_drive.<state>`) grouped by state and workflow class.
+- Fix metadata accessor persisting Hashes via text-type `#to_s` instead of JSON serialization. Instances loaded before the lazy attribute type registration would silently corrupt metadata on save.
+- Fix step code logger to include step execution tags. Previously, calling `logger` inside step code returned the workflow-tagged logger without execution_id and step_name tags.
+- Clarify executor log messages — replace opaque "dropping through" phrasing with specific, actionable messages.
+- Allow `GenevaDrive::Workflow.cancel!` to be called outside steps as well - previously it could only be called from within a step.
+
 ## [0.4.0]
 
 - Preserve original scheduled time when resuming a paused workflow. When a workflow with a future-scheduled step is paused and then resumed before that time, the step is rescheduled for the original time (not run immediately). If the original time has passed, the step runs immediately.