lepus-saga-active_record 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 72048dd94435e22305e296501e08f2d19234da3081fbf3c4c2fd75afdecd79f9
-  data.tar.gz: df6d77f4f98b6596449de33c4fba15a479d74e5c285f8cc406776727e7d72328
+  metadata.gz: 31354b38e13fe1a57459dc0c215b05ae8c80f7437b4545269fc1c9a53dd801e1
+  data.tar.gz: 61413e70d7aebca66b160eb7e986fe94ed0d764276ecfe3c638a1a25a723e03e
 SHA512:
-  metadata.gz: '09d4fbe4123daf7310d7a387f63db9f0e9e4b4055eff5035a3bb8c7285a547d53fecc7850dab249d780fa72aa7d19fb3d8e61382b77e1f01a00c5f9b6bfc404d'
-  data.tar.gz: 9ace56afa8214880ae7d97b39347e6cefdd5906095befb3bbd01064b5e856fcf41555f985b8792ab7f65322ea1f4d4f218d9df637281e7e7c15950676838fc34
+  metadata.gz: f5f25f07e133616a20952014f5c3541b01525fe689c99f35a4423f831096a5cecd6ee5d1526504f623bc7f9191f1b230614d65fec703d59c279cb8eff66abee2
+  data.tar.gz: 6f6e72dd03fedf4b922c9187441fa2e6a2de0f22ec6d4bfc5c79e4f2a00abde2b354d0ae08a29372c02a33a0b58b75b4a0278072a8ea49483a73f62fb68711eb

data/CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.1 - Unreleased
+* Fixed: executions list in time order when their ids are not time-ordered (e.g. random UUIDv4): `list_executions` sorts and pages by (sort value, id) with `after`, `executions_for_resource` and `recent_first` order by started_at, then id
+* Added: the `duration_ms` bigint column, the `sort`/`direction`/`after` options of `list_executions` (keyset paging in at most two queries that start the `(column, id)` index at the cursor, executions without a duration last), the `finished_from`/`finished_to` filters and `executed_steps` in one query
+* Changed: the `from`/`to` filters and the series' starts use started_at; `lepus-saga ~> 0.3.1` is required
+* Changed: dashboard indexes folded into the install migration (pre-release): `lepus:saga:install` indexes `saga_executions (started_at, id)`, `(duration_ms, id)`, `(finished_at)`, `(saga_name, started_at)`, `(status, started_at)`, `(partition, started_at)` and the resource index on `(resource_type, resource_id, started_at)`; the created_at indexes and the `lepus:saga:dashboard_indexes` generator are gone
+
 ## 0.3.0 - Unreleased
 * Added: brand mark and README logo
 * Added: `Lepus::Saga::Stores::ActiveRecord`, selected with `Lepus::Saga.configure { |c| c.store = :active_record }`

data/README.md

@@ -22,7 +22,7 @@ bin/rails generate lepus:saga:install
 bin/rails db:migrate
 ```
 
-The generator writes one migration that creates both tables. Ids are UUIDv7 strings generated in Ruby, the context columns are `json` and `failure_reason` is `text` (a step's error message can outgrow `VARCHAR(255)`, and MySQL/MariaDB strict mode would reject the write that fails the saga).
+The generator writes one migration that creates both tables and their indexes. Ids are UUIDv7 strings generated in Ruby, the context columns are `json` and `failure_reason` is `text` (a step's error message can outgrow `VARCHAR(255)`, and MySQL/MariaDB strict mode would reject the write that fails the saga). `saga_executions.duration_ms` is a `bigint` the engine writes when an execution ends. Besides the resource, progress and attempt indexes the engine uses, it indexes what the console sorts and filters on: `saga_executions (started_at, id)`, `(duration_ms, id)`, `(finished_at)`, `(saga_name, started_at)`, `(status, started_at)` and `(partition, started_at)`, and `saga_step_attempts (started_at)` and `(error_class, started_at)`.
 
 ```ruby
 # config/initializers/lepus_saga.rb
@@ -68,11 +68,14 @@ Lepus::Saga::ActiveRecord::Execution.where.not(id: pinned_ids).prune(finished_be
 | `with_locked_execution(id)` | `transaction { Execution.lock.find_by(id:) }`, one query |
 | `increment_resume_count` | `update_counters(id, resume_count: 1, touch: true)`, then reads the stored value back |
 | `create_step_attempt` | `create!`; `ActiveRecord::RecordNotUnique` from `idx_saga_step_attempts_dedup` becomes `Lepus::Saga::DuplicateAttemptError` |
-| `stalled_executions(before:)` | `in_progress.where(last_progress_at: ...before).order(:last_progress_at)` |
+| `stalled_executions(before:)` | `in_progress.where(last_progress_at: ...before).order(:last_progress_at, :id)` |
+| `executions_for_resource` | `for_resource(type, id).recent_first`: `ORDER BY started_at DESC, id DESC` on the resource index |
 | attempt queries | `where(saga_execution_id:, step_name:, direction:, epoch:)` plus `count`/`exists?` |
-| `list_executions` | Filters on the executions table, `order(id: :desc)` with `id < before_id` for keyset paging; the resource search is `LOWER(column) LIKE ? ESCAPE '!'`; the error filters are `id IN (SELECT saga_execution_id FROM saga_step_attempts WHERE status = 'failed' ...)` |
+| `list_executions` | Filters on the executions table, ordered by `(started_at, id)`, `(duration_ms, id)` or `(finished_at, id)` in the requested direction. Paging is keyset: rows after `after` compare against its stored value in a subquery (`col <= (SELECT col ... WHERE id = ?) AND (col < (SELECT ...) OR id < ?)` descending, `>=` and `>` ascending); the bare bound lets the `(column, id)` index start at the cursor instead of reading every row before it. For a nullable sort it runs at most two queries, the rows with a value and then, when the page is not full, the rows without one by id, so both use the `(column, id)` index and nulls come last either way; the resource search is `LOWER(column) LIKE ? ESCAPE '!'`; the error filters are `id IN (SELECT saga_execution_id FROM saga_step_attempts WHERE status = 'failed' ...)` |
 | `count_executions` | `group(...).count`; grouping by partition also groups by saga name to resolve executions without one |
-| `execution_series` | Plucks `created_at`, and `finished_at` and `status` of terminal executions, newest first and at most `max_rows` + 1; buckets them in Ruby |
+| `finished_executions` | Base's: `count_executions` grouped by saga and status over `finished_from`/`finished_to`, then one `list_executions(sort: :finished_at)` per saga, on the `finished_at` index |
+| `executed_steps` | One `SELECT DISTINCT saga_execution_id, step_name FROM saga_step_attempts WHERE saga_execution_id IN (...) AND direction = 'execute'` for every execution `/api/recent` lists |
+| `execution_series` | Plucks `started_at`, and `finished_at` and `status` of terminal executions, newest first and at most `max_rows` + 1; buckets them in Ruby |
 | `step_latency` | Joins attempts to executions and plucks the newest `max_rows` + 1 durations; percentiles in Ruby |
 | `failure_summary` | `GROUP BY error_class, saga_name, step_name` with `COUNT(*)` and `COUNT(DISTINCT saga_execution_id)`, ordered by `COUNT(*)` and `MAX(finished_at)`, then one query per group for its latest failure |
 | `saga_names` | `distinct.order(:saga_name).pluck(:saga_name)` |
@@ -89,16 +92,23 @@ The `belongs_to :execution` of a step attempt is optional, because the foreign k
 
 ## Existing tables
 
-Tables created by hand or before the generator (PostgreSQL `uuid` ids with `gen_random_uuid()` defaults, `jsonb` context, a `lock_version` column) work unchanged: the models set UUIDv7 ids before insert, read `jsonb` through the `json` attribute type and ignore `lock_version`.
+Tables created by hand or before the generator (PostgreSQL `uuid` ids with `gen_random_uuid()` defaults, `jsonb` context, a `lock_version` column) work unchanged: the models set UUIDv7 ids before insert, read `jsonb` through the `json` attribute type and ignore `lock_version`. Rows written with ids that are not time-ordered (e.g. random UUIDv4) list in time order, since the store sorts by started_at (or duration) and only breaks ties by id.
 
-Tables created before the install migration indexed the query columns lack the indexes the query methods filter on. Add them with:
+Such tables, and tables created by the 0.3.0 install generator, lack the `duration_ms` column and some of the indexes the install migration creates. Add them in a migration of your own; `if_not_exists: true` skips the indexes a table already has:
 
-```bash
-bin/rails generate lepus:saga:dashboard_indexes
-bin/rails db:migrate
+```ruby
+add_column :saga_executions, :duration_ms, :bigint, if_not_exists: true
+add_index :saga_executions, [:started_at, :id], if_not_exists: true
+add_index :saga_executions, [:duration_ms, :id], if_not_exists: true
+add_index :saga_executions, :finished_at, if_not_exists: true
+add_index :saga_executions, [:saga_name, :started_at], if_not_exists: true
+add_index :saga_executions, [:status, :started_at], if_not_exists: true
+add_index :saga_executions, [:partition, :started_at], if_not_exists: true
+add_index :saga_step_attempts, :started_at, if_not_exists: true
+add_index :saga_step_attempts, [:error_class, :started_at], if_not_exists: true
 ```
 
-The migration adds `saga_executions (saga_name, created_at)`, `(status, created_at)` and `(finished_at)`, and `saga_step_attempts (started_at)` and `(error_class, started_at)`, skipping any that already exist.
+and backfill the duration of finished executions, for example with `Lepus::Saga::ActiveRecord::Execution.where.not(finished_at: nil).find_each { |execution| execution.update_columns(duration_ms: ((execution.finished_at - execution.started_at) * 1000).round) }`. The `created_at` indexes of 0.3.0 (`[:saga_name, :created_at]`, `[:status, :created_at]`, `[:partition, :created_at]`) are no longer used and can be dropped.
 
 ## Architecture
 
@@ -112,7 +122,6 @@ StepAttemptBehaviour        <───   Lepus::Saga::ActiveRecord::StepAttempt
 "a lepus saga engine"       ───>   (contract and full engine on every database)
 "a lepus saga query store"  ───>
                                    lib/generators/lepus/saga/install
-                                   lib/generators/lepus/saga/dashboard_indexes
 ```
 
 This gem is also the reference for writing another adapter (Sequel, ROM, a key-value store):

data/lib/generators/lepus/saga/install/templates/create_lepus_saga_tables.rb.erb

@@ -14,19 +14,22 @@ class CreateLepusSagaTables < ActiveRecord::Migration<%= migration_version %>
       t.string   :partition
       t.datetime :started_at,       null: false
       t.datetime :finished_at
+      t.bigint   :duration_ms
       t.datetime :last_progress_at, null: false
       t.integer  :resume_count,     null: false, default: 0
       t.timestamps
     end
-    add_index :saga_executions, [:resource_type, :resource_id, :created_at],
+    add_index :saga_executions, [:resource_type, :resource_id, :started_at],
               name: "idx_saga_executions_on_resource"
     add_index :saga_executions, [:status, :updated_at]
     add_index :saga_executions, [:status, :last_progress_at],
               name: "index_saga_executions_on_status_and_progress"
-    add_index :saga_executions, [:partition, :created_at]
-    add_index :saga_executions, [:saga_name, :created_at]
-    add_index :saga_executions, [:status, :created_at]
+    add_index :saga_executions, [:started_at, :id]
+    add_index :saga_executions, [:duration_ms, :id]
     add_index :saga_executions, :finished_at
+    add_index :saga_executions, [:partition, :started_at]
+    add_index :saga_executions, [:saga_name, :started_at]
+    add_index :saga_executions, [:status, :started_at]
 
     create_table :saga_step_attempts, id: :string, limit: 36 do |t|
       t.references :saga_execution, type: :string, limit: 36, null: false, foreign_key: true

data/lib/lepus/saga/active_record/execution.rb

@@ -30,7 +30,7 @@ module Lepus
         scope :for_resource, ->(type, id) { where(resource_type: type, resource_id: id) }
         scope :in_progress, -> { where(status: IN_PROGRESS_STATUSES) }
         scope :completed, -> { where(status: "completed") }
-        scope :recent_first, -> { order(created_at: :desc) }
+        scope :recent_first, -> { order(started_at: :desc, id: :desc) }
         # Deliberately not `updated_at`: a re-publish, a lock or an attempt
         # opened and abandoned all touch the row while the cursor stands
         # still, so a wedged execution would keep resetting its own stall clock.

data/lib/lepus/saga/active_record/version.rb

@@ -3,7 +3,7 @@
 module Lepus
   module Saga
     module ActiveRecord
-      VERSION = "0.3.0"
+      VERSION = "0.3.1"
     end
   end
 end

data/lib/lepus/saga/stores/active_record.rb

@@ -61,7 +61,7 @@ module Lepus
         end
 
         def stalled_executions(before:)
-          execution_class.in_progress.where(last_progress_at: ...before).order(:last_progress_at)
+          execution_class.in_progress.where(last_progress_at: ...before).order(:last_progress_at, :id)
         end
 
         def executions_for_resource(resource_type, resource_id)
@@ -83,6 +83,15 @@ module Lepus
           attempts_of(execution).chronological
         end
 
+        def executed_steps(executions)
+          steps = executions.to_h { |execution| [execution.id, []] }
+          return steps if steps.empty?
+
+          step_attempt_class.where(saga_execution_id: steps.keys, direction: "execute").distinct
+            .pluck(:saga_execution_id, :step_name).each { |id, step| steps[id] << step }
+          steps
+        end
+
         def count_step_attempts(execution, step_name:, direction:, epoch:)
           attempts_of(execution).where(step_name: step_name, direction: direction, epoch: epoch).count
         end
@@ -104,10 +113,29 @@ module Lepus
 
         def queries? = true
 
-        def list_executions(before_id: nil, limit: 50, **filters)
+        # Keyset paging in up to two index-friendly queries: the executions
+        # with a value in (value, id) order, then those without one by id.
+        def list_executions(sort: :started_at, direction: :desc, after: nil, limit: 50, **filters)
+          column, descending, nullable = list_order(sort, direction, filters)
           scope = matching_executions(**filters)
-          scope = scope.where(id: ...before_id) if before_id
-          scope.order(id: :desc).limit(limit).to_a
+          order = descending ? :desc : :asc
+          if after
+            cursor = execution_class.where(id: after).pluck(:id, column).first
+            return [] unless cursor
+          end
+
+          in_nulls = cursor && cursor.last.nil?
+          rows = []
+          unless in_nulls
+            valued = nullable ? scope.where.not(column => nil) : scope
+            valued = valued.where(beyond(column, after, descending)) if cursor
+            rows = valued.order(column => order, :id => order).limit(limit).to_a
+          end
+          return rows unless nullable && rows.size < limit
+
+          unvalued = scope.where(column => nil)
+          unvalued = unvalued.where(execution_class.arel_table[:id].public_send(descending ? :lt : :gt, after)) if in_nulls
+          rows + unvalued.order(id: order).limit(limit - rows.size).to_a
         end
 
         def count_executions(group_by:, **filters)
@@ -122,7 +150,7 @@ module Lepus
           seconds = series_bucket(from, to, bucket)
           scope = matching_executions(saga_name: saga_name)
           started, started_truncated = take_rows(
-            scope.where(created_at: from...to).order(created_at: :desc).limit(max_rows + 1).pluck(:created_at),
+            scope.where(started_at: from...to).order(started_at: :desc).limit(max_rows + 1).pluck(:started_at),
             max_rows
           )
           finished, finished_truncated = take_rows(
@@ -176,8 +204,22 @@ module Lepus
           step_attempt_class.where(saga_execution_id: execution.id)
         end
 
+        # Rows strictly after the execution `after` in (column, id) order,
+        # comparing against its stored value in a subquery so no precision is
+        # lost on the way through Ruby. The bare bound on the column lets the
+        # (column, id) index start at the cursor instead of filtering every
+        # row before it.
+        def beyond(column, after, descending)
+          table = execution_class.arel_table
+          value = execution_class.where(id: after).select(column).arel
+          strict, bound = descending ? %i[lt lteq] : %i[gt gteq]
+          table[column].public_send(bound, value)
+            .and(table[column].public_send(strict, value).or(table[:id].public_send(strict, after)))
+        end
+
         def matching_executions(saga_name: nil, status: nil, current_step: nil, partition: nil, resource: nil, from: nil, to: nil,
-          progress_from: nil, progress_to: nil, error_class: nil, error_step: nil, error_from: nil, error_to: nil)
+          progress_from: nil, progress_to: nil, finished_from: nil, finished_to: nil, error_class: nil, error_step: nil, error_from: nil,
+          error_to: nil)
           sagas = filter_values(saga_name)
           statuses = filter_values(status)
           steps = filter_values(current_step)
@@ -185,10 +227,12 @@ module Lepus
           scope = scope.where(saga_name: sagas) if sagas
           scope = scope.where(status: statuses) if statuses
           scope = scope.where(current_step: steps) if steps
-          scope = scope.where(created_at: from..) if from
-          scope = scope.where(created_at: ...to) if to
+          scope = scope.where(started_at: from..) if from
+          scope = scope.where(started_at: ...to) if to
           scope = scope.where(last_progress_at: progress_from..) if progress_from
           scope = scope.where(last_progress_at: ...progress_to) if progress_to
+          scope = scope.where(finished_at: finished_from..) if finished_from
+          scope = scope.where(finished_at: ...finished_to) if finished_to
           failing = failed_attempts(filter_values(error_class), filter_values(error_step), error_from, error_to)
           scope = scope.where(id: failing.select(:saga_execution_id)) if failing
           scope = scope.and(in_partition(partition.to_s)) if partition.present?

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: lepus-saga-active_record
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Marcos G. Zimmermann
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.0
+        version: 0.3.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.0
+        version: 0.3.1
 - !ruby/object:Gem::Dependency
   name: activerecord
   requirement: !ruby/object:Gem::Requirement
@@ -199,8 +199,6 @@ files:
 - docs/brand/mark-small-on-light.svg
 - docs/brand/mark-small.svg
 - docs/brand/mark.svg
-- lib/generators/lepus/saga/dashboard_indexes/dashboard_indexes_generator.rb
-- lib/generators/lepus/saga/dashboard_indexes/templates/add_lepus_saga_dashboard_indexes.rb.erb
 - lib/generators/lepus/saga/install/install_generator.rb
 - lib/generators/lepus/saga/install/templates/create_lepus_saga_tables.rb.erb
 - lib/lepus/saga/active_record.rb

data/lib/generators/lepus/saga/dashboard_indexes/dashboard_indexes_generator.rb

@@ -1,26 +0,0 @@
-# frozen_string_literal: true
-
-require "rails/generators"
-require "rails/generators/active_record"
-
-module Lepus
-  module Saga
-    class DashboardIndexesGenerator < Rails::Generators::Base
-      include ::ActiveRecord::Generators::Migration
-
-      source_root File.expand_path("templates", __dir__)
-
-      desc "Adds the indexes the store's query methods use to saga tables created before they were in lepus:saga:install"
-
-      def create_migration_file
-        migration_template "add_lepus_saga_dashboard_indexes.rb.erb", File.join(db_migrate_path, "add_lepus_saga_dashboard_indexes.rb")
-      end
-
-      private
-
-      def migration_version
-        "[#{::ActiveRecord::VERSION::MAJOR}.#{::ActiveRecord::VERSION::MINOR}]"
-      end
-    end
-  end
-end

data/lib/generators/lepus/saga/dashboard_indexes/templates/add_lepus_saga_dashboard_indexes.rb.erb

@@ -1,17 +0,0 @@
-class AddLepusSagaDashboardIndexes < ActiveRecord::Migration<%= migration_version %>
-  def up
-    add_index :saga_executions, [:saga_name, :created_at], if_not_exists: true
-    add_index :saga_executions, [:status, :created_at], if_not_exists: true
-    add_index :saga_executions, :finished_at, if_not_exists: true
-    add_index :saga_step_attempts, :started_at, if_not_exists: true
-    add_index :saga_step_attempts, [:error_class, :started_at], if_not_exists: true
-  end
-
-  def down
-    remove_index :saga_step_attempts, [:error_class, :started_at], if_exists: true
-    remove_index :saga_step_attempts, :started_at, if_exists: true
-    remove_index :saga_executions, :finished_at, if_exists: true
-    remove_index :saga_executions, [:status, :created_at], if_exists: true
-    remove_index :saga_executions, [:saga_name, :created_at], if_exists: true
-  end
-end