lepus-saga-sequel 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 688935ea658850abc331020a8bef9b9a11a940c99e19a65c054fb593871406b9
-  data.tar.gz: 69feb77e326f97a3a344c29b79f6ebb80f254906239a7d3ff99a70bbd38df562
+  metadata.gz: 1ab2a2679b384739bc0e66d34a24999eea651f6e4d2f8d1cef2de9cc1c5caed1
+  data.tar.gz: da5a456ca77c48694aa8ba157836850a80cf4cf996225b42ed1b1e4c6217aa72
 SHA512:
-  metadata.gz: 36b993b968e9781e5eb1cedc84bb52fa2329bfa5b48f6e81b9f941cf8a6b3b46890108eaa68d758bd70451101c69c82969f03cdf52e51622d02292acc5eeacf1
-  data.tar.gz: df9f85d83bdc581a1db3da45616a19e2c8b2f4fedcd636bff5322aae401f8505f062811fb71b58b84e6498f842754b0100af1720a0d312adaaa804dd15dced45
+  metadata.gz: 1ce0d02bddd34b58f0e27e3a2064e05cc755b035aa4bafe3d0fcadcfd4469d236e554b52d0b59874af70857c191fd0de9d269af652012b5c5a87dc0ada412d07
+  data.tar.gz: 21e9d75ea12a719c97c25354cbca1eb864838fa2eff443d1210d574b2014e03a4abdd6e0a3ce1e30cdb3be6cd1ccb38b10048db0e3b3b9eeb9be8cbe31c62087

data/CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.1 - Unreleased
+* Fixed: executions list in time order when their ids are not time-ordered (e.g. random UUIDv4): `list_executions` sorts and pages by (sort value, id) with `after`, `executions_for_resource` and `recent_first` order by started_at, then id
+* Added: the `duration_ms` bigint column, the `sort`/`direction`/`after` options of `list_executions` (keyset paging in at most two queries that start the `(column, id)` index at the cursor, executions without a duration last), the `finished_from`/`finished_to` filters and `executed_steps` in one query
+* Changed: the `from`/`to` filters and the series' starts use started_at; `lepus-saga ~> 0.3.1` is required
+* Changed: dashboard indexes folded into the install migration (pre-release): migration 001 indexes `saga_executions (started_at, id)`, `(duration_ms, id)`, `(finished_at)`, `(saga_name, started_at)`, `(status, started_at)`, `(partition, started_at)` and the resource index on `(resource_type, resource_id, started_at)`; the created_at indexes and migration 002 are gone, so a database that ran 002 needs its tables recreated
+
 ## 0.3.0 - Unreleased
 * Added: brand mark and README logo
 * Added: `Lepus::Saga::Stores::Sequel`, selected with `Lepus::Saga.configure { |c| c.store = :sequel }`, and `Lepus::Saga::Sequel.db=` to bind the models to a database other than `Sequel::Model.db`

data/README.md

@@ -27,7 +27,7 @@ Lepus::Saga::Sequel.migrate(DB, target: 0)   # drop them
 
 The version is tracked in its own `lepus_saga_schema_info` table, so it never collides with your app's `schema_info` or `schema_migrations`. To keep everything in one migration history instead, copy the files from `lib/lepus/saga/sequel/migrations` into your own migrations directory.
 
-Ids are UUIDv7 strings generated in Ruby, the context columns are `json` and `failure_reason` is `text` (a step's error message can outgrow `VARCHAR(255)`, and MySQL/MariaDB strict mode would reject the write that fails the saga). The table, column and index names are the ones the lepus-saga-active_record generator creates, so an app can move between the two gems on the same tables. Column types can differ slightly: Sequel creates unsized strings as `text` on PostgreSQL and `duration_ms` as a double-precision float on MySQL/MariaDB.
+Ids are UUIDv7 strings generated in Ruby, the context columns are `json` and `failure_reason` is `text` (a step's error message can outgrow `VARCHAR(255)`, and MySQL/MariaDB strict mode would reject the write that fails the saga). The table, column and index names are the ones the lepus-saga-active_record generator creates, so an app can move between the two gems on the same tables. Column types can differ slightly: Sequel creates unsized strings as `text` on PostgreSQL and the step attempts' `duration_ms` as a double-precision float on MySQL/MariaDB. `saga_executions.duration_ms` is a `bigint` the engine writes when an execution ends. Besides the resource, progress and attempt indexes the engine uses, the migration indexes what the console sorts and filters on: `saga_executions (started_at, id)`, `(duration_ms, id)`, `(finished_at)`, `(saga_name, started_at)`, `(status, started_at)` and `(partition, started_at)`, and `saga_step_attempts (started_at)` and `(error_class, started_at)`.
 
 ## Configuration
 
@@ -106,11 +106,14 @@ Lepus::Saga::Sequel::Execution.exclude(id: pinned_ids).prune(finished_before: Ti
 | `save_execution`, `save_step_attempt` | `save`, which writes every column, so a context changed in place is persisted |
 | `increment_resume_count` | `UPDATE ... SET resume_count = resume_count + 1, updated_at = ?`, then reads the stored value back |
 | `create_step_attempt` | `create`; `Sequel::UniqueConstraintViolation` from `idx_saga_step_attempts_dedup` becomes `Lepus::Saga::DuplicateAttemptError` |
-| `stalled_executions(before:)` | `in_progress.where(last_progress_at < before).order(:last_progress_at).all` |
+| `stalled_executions(before:)` | `in_progress.where(last_progress_at < before).order(:last_progress_at, :id).all` |
+| `executions_for_resource` | `for_resource(type, id).recent_first`: `ORDER BY started_at DESC, id DESC` on the resource index |
 | attempt queries | `where(saga_execution_id:, step_name:, direction:, epoch:)` plus `count`/`empty?` |
-| `list_executions` | Filters on the executions table, `order(Sequel.desc(:id))` with `id < before_id` for keyset paging; the resource search is `ILIKE` with the LIKE wildcards escaped; the error filters are `id IN (SELECT saga_execution_id FROM saga_step_attempts WHERE status = 'failed' ...)` |
+| `list_executions` | Filters on the executions table, ordered by `(started_at, id)`, `(duration_ms, id)` or `(finished_at, id)` in the requested direction. Paging is keyset: rows after `after` compare against its stored value in a subquery (`col <= (SELECT col ... WHERE id = ?) AND (col < (SELECT ...) OR id < ?)` descending, `>=` and `>` ascending); the bare bound lets the `(column, id)` index start at the cursor instead of reading every row before it. For a nullable sort it runs at most two queries, the rows with a value and then, when the page is not full, the rows without one by id, so both use the `(column, id)` index and nulls come last either way; the resource search is `ILIKE` with the LIKE wildcards escaped; the error filters are `id IN (SELECT saga_execution_id FROM saga_step_attempts WHERE status = 'failed' ...)` |
 | `count_executions` | `group_and_count(...)`; grouping by partition also groups by saga name to resolve executions without one |
-| `execution_series` | Selects `created_at`, and `finished_at` and `status` of terminal executions, newest first and at most `max_rows` + 1; buckets them in Ruby |
+| `finished_executions` | Base's: `count_executions` grouped by saga and status over `finished_from`/`finished_to`, then one `list_executions(sort: :finished_at)` per saga, on the `finished_at` index |
+| `executed_steps` | One `SELECT DISTINCT saga_execution_id, step_name FROM saga_step_attempts WHERE saga_execution_id IN (...) AND direction = 'execute'` for every execution `/api/recent` lists |
+| `execution_series` | Selects `started_at`, and `finished_at` and `status` of terminal executions, newest first and at most `max_rows` + 1; buckets them in Ruby |
 | `step_latency` | Joins attempts to executions and selects the newest `max_rows` + 1 durations; percentiles in Ruby |
 | `failure_summary` | `GROUP BY error_class, saga_name, step_name` with `COUNT(*)` and `COUNT(DISTINCT saga_execution_id)`, ordered by `COUNT(*)` and `MAX(finished_at)`, then one query per group for its latest failure |
 | `saga_names` | `distinct.order(:saga_name).select_map(:saga_name)` |
@@ -129,11 +132,9 @@ The `many_to_one :execution` of a step attempt is not validated, because the for
 
 ## Existing tables
 
-The first migration uses `create_table?`, so it adopts tables that already exist (created by hand or by the lepus-saga-active_record generator). Note that rolling it back drops them.
+The migration uses `create_table?`, so it adopts tables that already exist (created by hand or by the lepus-saga-active_record generator). Note that rolling it back drops them.
 
-Tables created before the indexes the query methods filter on existed lack them. The second migration adds `saga_executions (saga_name, created_at)`, `(status, created_at)` and `(finished_at)`, and `saga_step_attempts (started_at)` and `(error_class, started_at)`, skipping any that already exist.
-
-Tables created by hand (PostgreSQL `uuid` ids, `jsonb` context) work too: the models set UUIDv7 ids before insert and read the JSON columns whether the driver returns a String or a parsed value (the `pg_json` extension).
+Tables created by hand (PostgreSQL `uuid` ids, `jsonb` context) work too: the models set UUIDv7 ids before insert and read the JSON columns whether the driver returns a String or a parsed value (the `pg_json` extension). Rows written with ids that are not time-ordered (e.g. random UUIDv4) list in time order, since the store sorts by started_at (or duration) and only breaks ties by id. Adopted tables keep their own columns and indexes: add `saga_executions.duration_ms` (`Bignum`) and the indexes listed above in a migration of your own, and backfill the duration of finished executions.
 
 ## Architecture
 

data/lib/lepus/saga/sequel/execution.rb

@@ -15,7 +15,7 @@ module Lepus
         dataset_module do
           where :in_progress, status: IN_PROGRESS_STATUSES
           where :completed, status: "completed"
-          order :recent_first, ::Sequel.desc(:created_at)
+          order :recent_first, ::Sequel.desc(:started_at), ::Sequel.desc(:id)
 
           def for_resource(type, id)
             where(resource_type: type, resource_id: id.to_s)

data/lib/lepus/saga/sequel/migrations/001_create_lepus_saga_tables.rb

@@ -17,18 +17,21 @@ Sequel.migration do
       String :partition
       DateTime :started_at, null: false
       DateTime :finished_at
+      Bignum :duration_ms
       DateTime :last_progress_at, null: false
       Integer :resume_count, null: false, default: 0
       DateTime :created_at, null: false
       DateTime :updated_at, null: false
 
-      index %i[resource_type resource_id created_at], name: :idx_saga_executions_on_resource
+      index %i[resource_type resource_id started_at], name: :idx_saga_executions_on_resource
       index %i[status updated_at], name: :index_saga_executions_on_status_and_updated_at
       index %i[status last_progress_at], name: :index_saga_executions_on_status_and_progress
-      index %i[partition created_at], name: :index_saga_executions_on_partition_and_created_at
-      index %i[saga_name created_at], name: :index_saga_executions_on_saga_name_and_created_at
-      index %i[status created_at], name: :index_saga_executions_on_status_and_created_at
+      index %i[started_at id], name: :index_saga_executions_on_started_at_and_id
+      index %i[duration_ms id], name: :index_saga_executions_on_duration_ms_and_id
       index :finished_at, name: :index_saga_executions_on_finished_at
+      index %i[partition started_at], name: :index_saga_executions_on_partition_and_started_at
+      index %i[saga_name started_at], name: :index_saga_executions_on_saga_name_and_started_at
+      index %i[status started_at], name: :index_saga_executions_on_status_and_started_at
     end
 
     create_table?(:saga_step_attempts) do

data/lib/lepus/saga/sequel/version.rb

@@ -3,7 +3,7 @@
 module Lepus
   module Saga
     module Sequel
-      VERSION = "0.3.0"
+      VERSION = "0.3.1"
     end
   end
 end

data/lib/lepus/saga/stores/sequel.rb

@@ -61,7 +61,7 @@ module Lepus
         end
 
         def stalled_executions(before:)
-          execution_class.in_progress.where(::Sequel[:last_progress_at] < before).order(:last_progress_at).all
+          execution_class.in_progress.where(::Sequel[:last_progress_at] < before).order(:last_progress_at, :id).all
         end
 
         def executions_for_resource(resource_type, resource_id)
@@ -83,6 +83,15 @@ module Lepus
           attempts_of(execution).chronological.all
         end
 
+        def executed_steps(executions)
+          steps = executions.to_h { |execution| [execution.id, []] }
+          return steps if steps.empty?
+
+          step_attempt_class.where(saga_execution_id: steps.keys, direction: "execute").distinct
+            .select_map(%i[saga_execution_id step_name]).each { |id, step| steps[id] << step }
+          steps
+        end
+
         def count_step_attempts(execution, step_name:, direction:, epoch:)
           attempts_of(execution).where(step_name: step_name, direction: direction, epoch: epoch).count
         end
@@ -107,10 +116,29 @@ module Lepus
 
         def queries? = true
 
-        def list_executions(before_id: nil, limit: 50, **filters)
+        # Keyset paging in up to two index-friendly queries: the executions
+        # with a value in (value, id) order, then those without one by id.
+        def list_executions(sort: :started_at, direction: :desc, after: nil, limit: 50, **filters)
+          column, descending, nullable = list_order(sort, direction, filters)
           scope = matching_executions(**filters)
-          scope = scope.where(::Sequel[:id] < before_id) if before_id
-          scope.order(::Sequel.desc(:id)).limit(limit).all
+          order = ->(name) { descending ? ::Sequel.desc(name) : ::Sequel.asc(name) }
+          if after
+            cursor = execution_class.where(id: after).get([:id, column])
+            return [] unless cursor
+          end
+
+          in_nulls = cursor && cursor.last.nil?
+          rows = []
+          unless in_nulls
+            valued = nullable ? scope.exclude(column => nil) : scope
+            valued = valued.where(beyond(column, after, descending)) if cursor
+            rows = valued.order(order.call(column), order.call(:id)).limit(limit).all
+          end
+          return rows unless nullable && rows.size < limit
+
+          unvalued = scope.where(column => nil)
+          unvalued = unvalued.where(descending ? ::Sequel[:id] < after : ::Sequel[:id] > after) if in_nulls
+          rows + unvalued.order(order.call(:id)).limit(limit - rows.size).all
         end
 
         def count_executions(group_by:, **filters)
@@ -124,8 +152,8 @@ module Lepus
           seconds = series_bucket(from, to, bucket)
           scope = matching_executions(saga_name: saga_name)
           started, started_truncated = take_rows(
-            in_window(scope, :created_at, from, to)
-              .order(::Sequel.desc(:created_at)).limit(max_rows + 1).select_map(:created_at),
+            in_window(scope, :started_at, from, to)
+              .order(::Sequel.desc(:started_at)).limit(max_rows + 1).select_map(:started_at),
             max_rows
           )
           finished, finished_truncated = take_rows(
@@ -211,8 +239,24 @@ module Lepus
           scope
         end
 
+        # Rows strictly after the execution `after` in (column, id) order,
+        # comparing against its stored value in a subquery so no precision is
+        # lost on the way through Ruby. The bare bound on the column lets the
+        # (column, id) index start at the cursor instead of filtering every
+        # row before it.
+        def beyond(column, after, descending)
+          value = execution_class.where(id: after).select(column)
+          strict, bound = descending ? %i[< <=] : %i[> >=]
+          compare = ->(operator, left, right) { ::Sequel::SQL::BooleanExpression.new(operator, left, right) }
+          ::Sequel.&(
+            compare.call(bound, ::Sequel[column], value),
+            ::Sequel.|(compare.call(strict, ::Sequel[column], value), compare.call(strict, ::Sequel[:id], after))
+          )
+        end
+
         def matching_executions(saga_name: nil, status: nil, current_step: nil, partition: nil, resource: nil, from: nil, to: nil,
-          progress_from: nil, progress_to: nil, error_class: nil, error_step: nil, error_from: nil, error_to: nil)
+          progress_from: nil, progress_to: nil, finished_from: nil, finished_to: nil, error_class: nil, error_step: nil, error_from: nil,
+          error_to: nil)
           sagas = filter_values(saga_name)
           statuses = filter_values(status)
           steps = filter_values(current_step)
@@ -220,8 +264,9 @@ module Lepus
           scope = scope.where(saga_name: sagas) if sagas
           scope = scope.where(status: statuses) if statuses
           scope = scope.where(current_step: steps) if steps
-          scope = in_window(scope, :created_at, from, to)
+          scope = in_window(scope, :started_at, from, to)
           scope = in_window(scope, :last_progress_at, progress_from, progress_to)
+          scope = in_window(scope, :finished_at, finished_from, finished_to)
           failing = failed_attempts(filter_values(error_class), filter_values(error_step), error_from, error_to)
           scope = scope.where(id: failing.select(:saga_execution_id)) if failing
           scope = scope.where(in_partition(partition.to_s)) unless blank?(partition)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: lepus-saga-sequel
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Marcos G. Zimmermann
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.0
+        version: 0.3.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.0
+        version: 0.3.1
 - !ruby/object:Gem::Dependency
   name: sequel
   requirement: !ruby/object:Gem::Requirement
@@ -174,7 +174,6 @@ files:
 - lib/lepus/saga/sequel.rb
 - lib/lepus/saga/sequel/execution.rb
 - lib/lepus/saga/sequel/migrations/001_create_lepus_saga_tables.rb
-- lib/lepus/saga/sequel/migrations/002_add_lepus_saga_dashboard_indexes.rb
 - lib/lepus/saga/sequel/record.rb
 - lib/lepus/saga/sequel/step_attempt.rb
 - lib/lepus/saga/sequel/version.rb

data/lib/lepus/saga/sequel/migrations/002_add_lepus_saga_dashboard_indexes.rb

@@ -1,31 +0,0 @@
-# frozen_string_literal: true
-
-# The indexes the query methods filter on, for tables created before 001
-# declared them. A no-op on tables created by 001.
-Sequel.migration do
-  dashboard_indexes = {
-    saga_executions: {
-      index_saga_executions_on_saga_name_and_created_at: %i[saga_name created_at],
-      index_saga_executions_on_status_and_created_at: %i[status created_at],
-      index_saga_executions_on_finished_at: %i[finished_at]
-    },
-    saga_step_attempts: {
-      index_saga_step_attempts_on_started_at: %i[started_at],
-      index_saga_step_attempts_on_error_class_and_started_at: %i[error_class started_at]
-    }
-  }
-
-  up do
-    dashboard_indexes.each do |table, named|
-      present = indexes(table).keys
-      named.each { |name, columns| add_index(table, columns, name: name) unless present.include?(name) }
-    end
-  end
-
-  down do
-    dashboard_indexes.each do |table, named|
-      present = indexes(table).keys
-      named.each { |name, columns| drop_index(table, columns, name: name) if present.include?(name) }
-    end
-  end
-end