lepus-saga 0.3.0 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7886960ade3be84a12c25f22f85047e113bdd2ca19055b6f065624c15d1bb3af
-  data.tar.gz: 4cd2d1fd471c40db739ccb41dcef0bf52196a1a240f95c53404fced9bb264566
+  metadata.gz: 78b15c3001956fd3fc7b24559d5445a0992398c9e9a648a374071f7017376e55
+  data.tar.gz: 868a8b17b8284a4cd09e3bde7ddd5d61be7cfea9e525eac2fe2df4d8c8d36d3b
 SHA512:
-  metadata.gz: c1e827eb0eb4924c0951a5dd0c6aa08233fa141c7393e275ea0d19f1fbd5d038f2a675f7747762246814ec6dfca1f3ea6949fda0cc4d16ecc85c4ceda3891138
-  data.tar.gz: 3f9759919aa086d1afda979ee040010b549b9dd353944e4f5eb03a3373c6c2b9de4beaffb7a995cebf5ad30706aa19f43a880483bf3198ca65e274fb5a808d79
+  metadata.gz: 74b9f0adadfeb24255abeab7fea95d26b4168442291690bd2e7f3a9c118cd243096a3d1a32958083cc8ba36e07f7dfc689477b2a08677f6266d627b032553bd1
+  data.tar.gz: 6bcf0f25574c99f05ddf173269c9c16ee638389a59a01e44275c4c4581edaa2128d1290841c6d65dbc569c0ea0a6eb05638b3f6fea700eb89a20f2fe6e8a36e0

data/CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.2 - Unreleased
+* Fixed: the console's execution page no longer shows a stray "null" under the summary when the execution has no failure reason
+
+## 0.3.1 - Unreleased
+* Fixed: executions list in time order when their ids are not time-ordered (e.g. random UUIDv4): `list_executions` sorts by started_at and pages by `after` (the last row's id) instead of by id, `executions_for_resource` orders by started_at, and the stalled list and failure samples break ties by id only after time
+* Added: `duration_ms` on executions (`ExecutionBehaviour::ATTRIBUTES`), written by the engine when an execution ends (completed, compensated, failed, compensation_failed, abandoned) and cleared when a retry reopens it; `finish_attributes` and `reopen_attributes` on `ExecutionBehaviour`
+* Added: `list_executions(sort:, direction:)`: `:started_at` (default), `:duration` or `:finished_at`, `:desc` (default) or `:asc`, ties by id, executions without a value last in either direction; an unknown `after` returns an empty page. The console's Started and Took headers sort the executions list (buttons with `aria-sort`, a select on phones, kept in the URL)
+* Changed: the executions list pages along the active sort ("Show older" follows it, `/api/executions` takes `sort`, `dir`, `after` and `limit` and returns `next`), changing the sort starts the list over and each refresh reloads the rows shown in that order. `before_id` and `next_before` are gone
+* Changed: the `from`/`to` filters and the series' starts use started_at instead of created_at
+* Added: `finished_executions(from:, to:, listed_from:, saga_name:, per_saga:, limit:)`, `executed_steps(executions)` and the `finished_from`/`finished_to` filters in the query contract, `GET /api/recent?since=&replay_from=` and the server's `now` on `/api/sagas` and `/api/moving`
+* Added: the map replays executions that finished between refreshes: tokens seen moving run on to how they ended, executions that started and finished in between replay their path in a lighter shade (at most 8 a line, a "+N just finished" pill for the rest), a pause collapses into "+N finished while paused", reduced motion shows per-line summaries, and the map key gains "Just finished"
+* Added: `bin/web-demo` starts quick webhook deliveries (more with `HEAVY=1`) that finish between refreshes
+
 ## 0.3.0 - Unreleased
 * Added: brand mark and README logo
 * Added: saga engine: `Lepus::Saga::Definition`, `Executor`, `ExecutorConsumer`, `StepProducer`, `QueueDepth`, `Instrumentation`, `LogSubscriber` and `Testing.drain`

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    lepus-saga (0.3.0)
+    lepus-saga (0.3.2)
       lepus (>= 0.2.1)
       rack (>= 2.2)
 
@@ -189,7 +189,7 @@ CHECKSUMS
   json (3.1.0) sha256=3ce7ccf8fde58960177a3a756c31f292a4c1e1dedee884547096a5b0604e3bd8
   language_server-protocol (3.17.0.6) sha256=5ef2c0c138f8267e1bc631d3328347d354f96724b0af22f2c79516120443b7f0
   lepus (0.2.2) sha256=7de6bcfb13e92b03af09d4a018c765aee6381624d8ace650705d5a87bea07fb7
-  lepus-saga (0.3.0)
+  lepus-saga (0.3.2)
   lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87
   logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
   method_source (1.1.0) sha256=181301c9c45b731b4769bc81e8860e72f9161ad7d66dd99103c9ab84f560f5c5

data/README.md

@@ -151,8 +151,8 @@ Adapters usually offer richer queries on their own records (`Lepus::Saga::Active
 
 `Lepus::Saga::Web` is a Rack app for the engineer on call. It has three screens:
 
-- **Map**: every saga drawn as a line of its steps. Tokens are the executions moving right now, placed along the step by how long they have spent in it against the step's median, up to three lanes per step and a "+N running" pill for the rest (static when the viewer prefers reduced motion). Each step shows its stalled executions and its failures in the selected time range (1 h, 24 h or 7 d, kept in the URL). The headline reads as the key: how many executions are moving, compensating and stalled, and how many failed. Below the map, a full-width throughput chart (with a table view), **Needs attention** (failed attempts grouped by error, saga and step, each linking to exactly the executions that failed that way in the range) and **Waiting the longest** (the stalled executions with the oldest progress).
-- **Executions**: a paged list filtered by status, saga, step, error, time range or stalled.
+- **Map**: every saga drawn as a line of its steps. Tokens are the executions moving right now, placed along the step by how long they have spent in it against the step's median, up to three lanes per step and a "+N running" pill for the rest (static when the viewer prefers reduced motion). Each step shows its stalled executions and its failures in the selected time range (1 h, 24 h or 7 d, kept in the URL). The headline counts how many executions are moving, compensating and stalled, and how many failed; the key under it names the marks (running, compensating, stalled, failed, just finished). Between refreshes the map also shows what finished: a token seen moving runs on from where it is to its end (completed ones reach the end terminal and fade, failed ones stop at the failing step and flash red, compensated ones turn back along the compensation rail), and executions that started and finished between two refreshes replay their whole path in about a second, in a lighter shade, staggered over the refresh in the order they started. At most 8 replay per line; a "+N just finished" pill at the end of the line counts the rest. The first load shows only what is running. After a pause or a hidden tab only the last refresh interval is animated and the rest becomes "+N finished while paused" (or "while away" for a hidden tab). With reduced motion nothing moves: each line ends with a short summary instead ("+3 completed · 1 failed"). Below the map, a full-width throughput chart (with a table view), **Needs attention** (failed attempts grouped by error, saga and step, each linking to exactly the executions that failed that way in the range) and **Waiting the longest** (the stalled executions with the oldest progress).
+- **Executions**: a list filtered by status, saga, step, error, time range or stalled, sorted by Started (the default, latest first) or Took, either way; the sort is kept in the URL and on phones it is a select. "Show older" (or "Show more" in another order) pages along the active sort, and each refresh reloads the rows shown so far in that order.
 - **Execution**: a replay of the execution along its line, its attempt history with errors and backtraces, its context and other runs of the same resource. Retry and compensate are one click plus a confirmation.
 
 It reads through the store's optional query methods, so the store must report `queries?` (`Stores::Memory`, `Stores::ActiveRecord` and `Stores::Sequel` do).
@@ -204,11 +204,11 @@ end
 
 `authorize` runs on every request: a falsy answer for `:read` returns 403 for the whole console, and for `:retry` or `:compensate` it refuses that action and the page hides its button. Retry and compensate are POSTs that need a JSON body, an `X-Lepus-Saga` header and a same-origin `Origin` (or `Referer`), so a cookie-authenticated session cannot be driven from another site.
 
-The page refreshes itself by polling the JSON API. The Live button in the top bar picks the interval: Paused, 2, 5, 15 or 60 seconds (`default_poll_interval` until the viewer picks one). Polling also pauses while the tab is hidden, and the map's tokens freeze while paused. The theme follows the system, or Light or Dark when picked; the interval, the theme and the time range are kept in the browser.
+The page refreshes itself by polling the JSON API. Responses that feed the map carry the server's `now`; the map asks `/api/recent?since=` with the previous one (and `replay_from` a little earlier, to catch late writes, which are drawn but not counted), so what finished in between is never measured against the browser's clock. The Live button in the top bar picks the interval: Paused, 2, 5, 15 or 60 seconds (`default_poll_interval` until the viewer picks one). Polling also pauses while the tab is hidden, and the map's tokens freeze while paused. The theme follows the system, or Light or Dark when picked; the interval, the theme and the time range are kept in the browser.
 
 Content Security Policy: the console loads only its own files (no inline scripts or styles, no `eval`, no CDN) and sends `default-src 'self'` with its page. A host-wide policy works as long as it allows `'self'` for scripts, styles, fonts and fetches, and `data:` images. Static files are served with ETags; the page references them under a digest path, cached for a year.
 
-To see the console without a database or RabbitMQ, run `bin/web-demo` from a checkout: it seeds an in-memory store with a week of executions generated by the real engine, keeps starting new ones, and serves the console on http://127.0.0.1:9292 (`PORT`, `HOST` and `DAYS` change it). `HEAVY=1` adds the load of a bad day: thousands of failed attempts, hundreds of executions crowding one step and hundreds of stalled ones. It is not part of the gem.
+To see the console without a database or RabbitMQ, run `bin/web-demo` from a checkout: it seeds an in-memory store with a week of executions generated by the real engine, keeps starting new ones (among them quick webhook deliveries that start and finish between two refreshes, so the map has replays to show), and serves the console on http://127.0.0.1:9292 (`PORT`, `HOST` and `DAYS` change it). `HEAVY=1` adds the load of a bad day: thousands of failed attempts, hundreds of executions crowding one step and hundreds of stalled ones. It is not part of the gem.
 
 The font is Overpass, under the SIL Open Font License (`web/assets/fonts/overpass/OFL.txt`).
 
@@ -254,7 +254,7 @@ ExecutorConsumer <────────────────────
 
 - `Definition`, `Executor`, `ExecutorConsumer`, `StepProducer`, `QueueDepth`, `Instrumentation`, `LogSubscriber` and `Testing` only talk to `Lepus::Saga.store`.
 - `Lepus::Saga::Stores::Base` is the persistence contract. Every method is documented in [`lib/lepus/saga/stores/base.rb`](lib/lepus/saga/stores/base.rb).
-- `Lepus::Saga::ExecutionBehaviour` and `Lepus::Saga::StepAttemptBehaviour` hold what records mean: statuses and predicates (`running?`, `terminal?`, `compensate_direction?`), `record_progress`, `definition_class`, `partition`, `current_step_scope`, `duration` and `finish_attributes`. Adapters include them into their own record classes.
+- `Lepus::Saga::ExecutionBehaviour` and `Lepus::Saga::StepAttemptBehaviour` hold what records mean: statuses and predicates (`running?`, `terminal?`, `compensate_direction?`), `record_progress`, `definition_class`, `partition`, `current_step_scope`, `duration`, `finish_attributes` and, on executions, `reopen_attributes`. The engine stores an execution's `duration_ms` (finished_at minus started_at, in milliseconds) whenever it ends, and clears it with finished_at when a retry reopens it. Adapters include them into their own record classes.
 
 ### The store contract
 
@@ -267,7 +267,7 @@ ExecutorConsumer <────────────────────
 | `with_locked_execution(id) { \|execution_or_nil\| }` | Same, from an id; Base implements it with `find_execution` + `with_lock`, adapters may do it in one query |
 | `increment_resume_count(execution)` | One atomic `resume_count = resume_count + 1` that touches updated_at; sets and returns the stored value |
 | `stalled_executions(before:)` | Running or compensating executions with `last_progress_at < before`, oldest first, as an Enumerable the caller may stop early |
-| `executions_for_resource(type, id)` | A resource's executions, newest first |
+| `executions_for_resource(type, id)` | A resource's executions, latest started_at first, ties by id |
 | `create_step_attempt(execution, attributes)` | Inserts a `running` attempt; a second one with the same execution, step, direction, epoch and attempt raises `Lepus::Saga::DuplicateAttemptError` |
 | `save_step_attempt(attempt)` | Persists the attempt, returns it |
 | `step_attempts(execution)` | Attempts ordered by started_at, created_at |
@@ -283,9 +283,11 @@ The query methods are optional. The engine never calls them; read-only tools suc
 | Method | Semantics |
 | --- | --- |
 | `queries?` | Whether the store implements the methods below |
-| `list_executions(saga_name:, status:, current_step:, partition:, resource:, from:, to:, progress_from:, progress_to:, error_class:, error_step:, error_from:, error_to:, before_id:, limit: 50)` | Executions newest first by id, `limit` of them, the page after `before_id` when given. Every filter is optional: `saga_name`, `status` and `current_step` take a value or a list, `partition` matches the column or (when it is nil) the definition's partition, `resource` is a case-insensitive substring of resource_type or resource_id, `from`/`to` bound created_at and `progress_from`/`progress_to` bound last_progress_at (inclusive/exclusive), `error_class` and `error_step` (a value or a list) keep executions with a failed attempt of that error class at that step, started between `error_from` and `error_to` when given |
+| `list_executions(sort: :started_at, direction: :desc, after: nil, limit: 50, **filters)` | Executions in `sort` order (`:started_at`, `:duration` for duration_ms, or `:finished_at`), `direction` `:desc` or `:asc`, ties broken by id in the same direction, executions without a value (in progress) last in either direction. Ids need not follow time: rows with ids that are not time-ordered (e.g. random UUIDv4) still list in time order. `after` is the id of the last execution of the previous page; the page holds the executions strictly after it by (sort value, id), and an `after` naming no execution returns `[]`. Every filter is optional: `saga_name`, `status` and `current_step` take a value or a list, `partition` matches the column or (when it is nil) the definition's partition, `resource` is a case-insensitive substring of resource_type or resource_id, `from`/`to` bound started_at, `progress_from`/`progress_to` bound last_progress_at and `finished_from`/`finished_to` bound finished_at (inclusive/exclusive), `error_class` and `error_step` (a value or a list) keep executions with a failed attempt of that error class at that step, started between `error_from` and `error_to` when given |
 | `count_executions(group_by:, **filters)` | A Hash of counts keyed by `:saga_name`, `:status`, `:partition` or `:current_step`, or by arrays of values for an array such as `[:saga_name, :status]`; partition groups resolve nil to the definition's partition |
-| `execution_series(from:, to:, bucket:, saga_name: nil, max_rows:)` | `{rows: [[bucket_start, {"started" => n, "completed" => n, "compensated" => n, "failed" => n, "compensation_failed" => n}], ...], truncated:}`: starts by created_at, terminal statuses by finished_at, every bucket of the window aligned to multiples of `bucket` seconds, at most `max_rows` timestamps of each kind (the newest) |
+| `finished_executions(from:, to:, listed_from: nil, saga_name: nil, per_saga: nil, limit: 100)` | `{executions: [...], counts: {[saga_name, status] => n}}`: executions that finished in the window, latest finish first (ties by id), at most `per_saga` of each saga and `limit` in all, with counts of every one. `listed_from` (before `from`) also lists, without counting, the ones that finished from then on. Base builds it on `list_executions` and `count_executions` |
+| `executed_steps(executions)` | `{id => [step_name, ...]}`: the steps each execution ran forward, every id present. Base calls `step_attempts` per execution; stores override it with one query |
+| `execution_series(from:, to:, bucket:, saga_name: nil, max_rows:)` | `{rows: [[bucket_start, {"started" => n, "completed" => n, "compensated" => n, "failed" => n, "compensation_failed" => n}], ...], truncated:}`: starts by started_at, terminal statuses by finished_at, every bucket of the window aligned to multiples of `bucket` seconds, at most `max_rows` timestamps of each kind (the newest) |
 | `step_latency(from:, to:, saga_name: nil, max_rows:)` | `{rows: [{saga_name:, step_name:, direction:, count:, p50:, p95:, p99:, max:}, ...], truncated:}`: nearest-rank percentiles in milliseconds of succeeded and failed attempts started in the window, from the newest `max_rows` |
 | `failure_summary(from:, to:, saga_name: nil, limit: 20)` | `[{error_class:, saga_name:, step_name:, count:, execution_count:, last_seen_at:, sample_message:, sample_execution_id:}, ...]`: failed attempts started in the window grouped by error class, saga and step, most frequent first, `execution_count` being the distinct executions and the sample the latest failure |
 | `saga_names` | The distinct saga names in storage, sorted |

data/lib/lepus/saga/execution_behaviour.rb

@@ -8,7 +8,7 @@ module Lepus
       ATTRIBUTES = %i[
         id saga_name resource_type resource_id status current_step direction
         epoch context correlation_id failure_reason partition started_at
-        finished_at last_progress_at resume_count created_at updated_at
+        finished_at duration_ms last_progress_at resume_count created_at updated_at
       ].freeze
 
       STATUSES = %w[running completed compensating compensated failed compensation_failed].freeze
@@ -53,6 +53,15 @@ module Lepus
       def duration
         (finished_at || Time.now.utc) - started_at
       end
+
+      # The attributes that end this execution at `now`: finished_at and the
+      # whole run's duration_ms, which the console sorts by.
+      def finish_attributes(now = Time.now.utc)
+        {finished_at: now, duration_ms: ((now - started_at) * 1000).round}
+      end
+
+      # The attributes that reopen a finished execution.
+      def reopen_attributes = {finished_at: nil, duration_ms: nil}
     end
   end
 end

data/lib/lepus/saga/executor.rb

@@ -30,7 +30,7 @@ module Lepus
               current_step: target.name, failure_reason: reason)
           else
             store.update_execution(execution,
-              status: "failed", failure_reason: reason, finished_at: Time.now.utc)
+              status: "failed", failure_reason: reason, **execution.finish_attributes)
           end
           execution
         end
@@ -182,7 +182,7 @@ module Lepus
         finished = with_fresh_lock do
           now = Time.now.utc
           store.finish_step_attempt(attempt, "succeeded", context_after: new_context)
-          store.update_execution(execution, context: new_context, status: "completed", finished_at: now,
+          store.update_execution(execution, context: new_context, status: "completed", **execution.finish_attributes(now),
             last_progress_at: now, resume_count: 0)
         end
         return :ack unless finished
@@ -231,7 +231,7 @@ module Lepus
       def apply_failure(reason)
         if execution.compensate_direction?
           store.update_execution(execution, status: "compensation_failed", failure_reason: reason,
-            finished_at: Time.now.utc)
+            **execution.finish_attributes)
           :compensation_failed
         else
           self.class.begin_compensation(execution, reason: reason)
@@ -270,7 +270,7 @@ module Lepus
 
       def complete_as(status)
         execution.status = status
-        execution.finished_at = Time.now.utc
+        execution.finish_attributes.each { |name, value| execution.public_send(:"#{name}=", value) }
         "saga.#{status}"
       end
 

data/lib/lepus/saga/rspec/engine_examples.rb

@@ -152,6 +152,7 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
 
       expect(execution.status).to eq("completed")
       expect(execution.finished_at).not_to be_nil
+      expect(execution.duration_ms).to be_within(1).of((execution.finished_at - execution.started_at) * 1000)
       expect(execution.context).to include("first_output" => "ok", "second_output" => "ok")
       expect(succeeded_steps(execution)).to eq(%w[first second third])
     end
@@ -174,6 +175,7 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
       execution = start_and_drain("complete_on_first" => true)
 
       expect(execution.status).to eq("completed")
+      expect(execution.duration_ms).to be >= 0
       expect(execution.context["completed_reason"]).to eq("early exit")
       expect(attempts(execution).map(&:step_name)).to eq(%w[first])
     end
@@ -197,6 +199,7 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
       execution = start_and_drain("second_transient_failures" => 99)
 
       expect(execution.status).to eq("compensated")
+      expect(execution.duration_ms).to be_within(1).of((execution.finished_at - execution.started_at) * 1000)
       expect(execution.failure_reason).to eq("transient boom")
       expect(attempt_statuses(execution, "second")).to eq(%w[failed failed])
       expect(sole(attempts(execution, step_name: "first", direction: "compensate")).status).to eq("succeeded")
@@ -215,6 +218,7 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
       execution = start_and_drain("fail_first" => true)
 
       expect(execution.status).to eq("failed")
+      expect(execution.duration_ms).to be >= 0
       expect(execution.failure_reason).to eq("first rejected")
       expect(attempts(execution, direction: "compensate")).to be_empty
     end
@@ -231,6 +235,7 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
       execution = start_and_drain("fail_second" => true, "fail_undo_first" => true)
 
       expect(execution).to have_attributes(status: "compensation_failed", current_step: "first", direction: "compensate")
+      expect(execution.duration_ms).to be >= 0
     end
 
     it "acks stale-epoch messages without running anything" do
@@ -353,7 +358,8 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
       execution = start_and_drain("fail_first" => true)
 
       expect(Lepus::Saga.retry(execution)).to equal(execution)
-      expect(execution).to have_attributes(status: "running", epoch: 1, failure_reason: nil, finished_at: nil)
+      expect(execution).to have_attributes(status: "running", epoch: 1, failure_reason: nil, finished_at: nil, duration_ms: nil)
+      expect(reload(execution)).to have_attributes(finished_at: nil, duration_ms: nil)
     end
 
     it "refuses to retry completed executions" do
@@ -533,6 +539,7 @@ RSpec.shared_examples "a lepus saga engine" do |concurrency: true|
       expect(Lepus::Saga.resume_stuck).to eq(0)
       expect(reload(execution).status).to eq("failed")
       expect(reload(execution).finished_at).not_to be_nil
+      expect(reload(execution).duration_ms).to be >= 0
     end
 
     it "gives an abandoned execution a fresh budget on manual retry" do

data/lib/lepus/saga/rspec/query_examples.rb

@@ -34,12 +34,37 @@ RSpec.shared_examples "a lepus saga query store" do
     end)
   end
 
-  def query_execution(saga_name: checkout, status: "running", created_at: hour, finished_at: nil, **attributes)
+  def query_execution(saga_name: checkout, status: "running", created_at: hour, finished_at: nil, duration_ms: nil, **attributes)
     execution = store.create_execution({
       saga_name: saga_name, resource_type: "Order", resource_id: "order-1", current_step: "reserve",
       partition: nil, started_at: created_at, last_progress_at: created_at
     }.merge(attributes))
-    store.update_execution(execution, status: status, created_at: created_at, finished_at: finished_at)
+    store.update_execution(execution, status: status, created_at: created_at, finished_at: finished_at, duration_ms: duration_ms)
+  end
+
+  # Executions created in the order given, so their ids (time-ordered by
+  # creation) do not follow their started_at.
+  def started_at_minutes(*offsets, **attributes)
+    offsets.map { |minute| query_execution(created_at: hour + minutes(minute), **attributes) }
+  end
+
+  def finished(duration_ms, created_at: hour, **attributes)
+    query_execution(status: "completed", created_at: created_at, finished_at: created_at + duration_ms / 1000.0, duration_ms: duration_ms, **attributes)
+  end
+
+  def ids(executions) = executions.map(&:id)
+
+  def all_pages(size, **options)
+    pages = []
+    after = nil
+    loop do
+      page = store.list_executions(**options, after: after, limit: size)
+      break if page.empty?
+
+      pages << ids(page)
+      after = page.last.id
+    end
+    pages
   end
 
   def query_attempt(execution, step_name: "charge", attempt: 1, direction: "execute", status: "succeeded",
@@ -57,23 +82,82 @@ RSpec.shared_examples "a lepus saga query store" do
   end
 
   describe "#list_executions" do
-    it "returns execution records, newest id first" do
-      ids = Array.new(3) { query_execution.id }
+    it "returns execution records, latest start first, whatever the ids" do
+      executions = started_at_minutes(3, 0, 4, 1, 2)
+      by_start = ids(executions.sort_by(&:started_at).reverse)
+
+      expect(store.list_executions).to all(be_a(Lepus::Saga::ExecutionBehaviour))
+      expect(ids(store.list_executions)).to eq(by_start)
+      expect(ids(store.list_executions(sort: :started_at, direction: :asc))).to eq(by_start.reverse)
+      expect(ids(store.list_executions(sort: "started_at", direction: "desc", limit: 2))).to eq(by_start.first(2))
+    end
 
-      executions = store.list_executions
+    it "breaks ties by id in the direction of the sort" do
+      tied = ids(started_at_minutes(1, 1, 1)).sort
+      later, earlier = started_at_minutes(2, 0)
 
-      expect(executions).to all(be_a(Lepus::Saga::ExecutionBehaviour))
-      expect(executions.map(&:id)).to eq(ids.sort.reverse)
+      expect(ids(store.list_executions)).to eq([later.id, *tied.reverse, earlier.id])
+      expect(ids(store.list_executions(direction: :asc))).to eq([earlier.id, *tied, later.id])
     end
 
-    it "pages with limit and before_id" do
-      ids = Array.new(5) { query_execution.id }.sort.reverse
+    it "pages after the last execution shown, through ties, without gaps or repeats" do
+      executions = started_at_minutes(2, 1, 1, 3, 1, 1, 0)
 
-      first_page = store.list_executions(limit: 2)
-      second_page = store.list_executions(limit: 2, before_id: first_page.last.id)
-      last_page = store.list_executions(limit: 2, before_id: second_page.last.id)
+      [:desc, :asc].each do |direction|
+        expected = ids(store.list_executions(direction: direction, limit: 100))
+        pages = all_pages(2, direction: direction)
 
-      expect([first_page, second_page, last_page].map { |page| page.map(&:id) }).to eq(ids.each_slice(2).to_a)
+        expect(pages.flatten).to eq(expected)
+        expect(pages.map(&:size)).to eq([2, 2, 2, 1])
+        expect(expected).to match_array(ids(executions))
+      end
+    end
+
+    it "sorts by duration with executions that have none last in either direction" do
+      slow = finished(9_000)
+      running = query_execution
+      fast = finished(100)
+      compensating = query_execution(status: "compensating")
+      medium = finished(2_500)
+      unfinished = [running, compensating].map(&:id).sort
+
+      expect(ids(store.list_executions(sort: :duration))).to eq([slow.id, medium.id, fast.id, *unfinished.reverse])
+      expect(ids(store.list_executions(sort: :duration, direction: :asc))).to eq([fast.id, medium.id, slow.id, *unfinished])
+      expect(store.list_executions(sort: :duration, limit: 1).first.duration_ms).to eq(9_000)
+    end
+
+    it "pages by duration through ties and across executions without one" do
+      [400, nil, 200, 400, nil, 100, 400, nil].each_with_index do |duration, index|
+        duration ? finished(duration, created_at: hour + minutes(index)) : query_execution(created_at: hour + minutes(index))
+      end
+
+      page_sizes = [1, 2, 3, 5]
+      {desc: [400, 400, 400, 200, 100, nil, nil, nil], asc: [100, 200, 400, 400, 400, nil, nil, nil]}.each do |direction, durations|
+        listed = store.list_executions(sort: :duration, direction: direction, limit: 100)
+        page_sizes.each do |size|
+          expect(all_pages(size, sort: :duration, direction: direction).flatten).to eq(ids(listed))
+        end
+        expect(listed.map(&:duration_ms)).to eq(durations)
+      end
+    end
+
+    it "pages after an execution the filters leave out by its sort value" do
+      first, hidden, last = started_at_minutes(3, 2, 1)
+      hidden = store.update_execution(hidden, status: "failed")
+
+      expect(ids(store.list_executions(status: "running", after: hidden.id))).to eq([last.id])
+      expect(ids(store.list_executions(status: "running", after: first.id))).to eq([last.id])
+    end
+
+    it "returns an empty page after an id that names no execution" do
+      query_execution
+
+      expect(store.list_executions(after: "00000000-0000-4000-8000-000000000000")).to eq([])
+    end
+
+    it "rejects unknown sorts and directions" do
+      expect { store.list_executions(sort: :resource_id) }.to raise_error(ArgumentError)
+      expect { store.list_executions(direction: :sideways) }.to raise_error(ArgumentError)
     end
 
     it "filters by saga name and status, single or listed" do
@@ -128,7 +212,7 @@ RSpec.shared_examples "a lepus saga query store" do
       expect(store.list_executions(resource: "a_b").map(&:id)).to contain_exactly(underscore.id)
     end
 
-    it "bounds created_at by from (inclusive) and to (exclusive)" do
+    it "bounds started_at by from (inclusive) and to (exclusive)" do
       query_execution(created_at: hour - minutes(1))
       at_from = query_execution(created_at: hour)
       inside = query_execution(created_at: hour + minutes(30))
@@ -140,6 +224,94 @@ RSpec.shared_examples "a lepus saga query store" do
     end
   end
 
+  describe "finished filter" do
+    it "bounds finished_at by finished_from (inclusive) and finished_to (exclusive)" do
+      query_execution(status: "completed", finished_at: hour - 1)
+      at_from = query_execution(status: "failed", finished_at: hour)
+      inside = query_execution(status: "completed", finished_at: hour + minutes(30))
+      query_execution(status: "completed", finished_at: hour + hours(1))
+      query_execution
+
+      expect(ids(store.list_executions(finished_from: hour, finished_to: hour + hours(1)))).to contain_exactly(at_from.id, inside.id)
+      expect(store.count_executions(group_by: :status, finished_from: hour)).to eq("failed" => 1, "completed" => 2)
+      expect(store.list_executions(finished_to: hour).size).to eq(1)
+    end
+  end
+
+  describe "#finished_executions" do
+    it "returns the executions that finished in the window, latest finish first, with counts by saga and status" do
+      query_execution(status: "completed", finished_at: hour - 1)
+      at_from = query_execution(status: "completed", finished_at: hour)
+      failed = query_execution(status: "failed", finished_at: hour + minutes(2))
+      compensated = query_execution(saga_name: invoice, status: "compensated", finished_at: hour + minutes(1))
+      query_execution(status: "completed", finished_at: hour + minutes(5))
+      query_execution
+
+      result = store.finished_executions(from: hour, to: hour + minutes(5))
+
+      expect(result[:executions]).to all(be_a(Lepus::Saga::ExecutionBehaviour))
+      expect(ids(result[:executions])).to eq([failed.id, compensated.id, at_from.id])
+      expect(result[:counts]).to eq([checkout, "completed"] => 1, [checkout, "failed"] => 1, [invoice, "compensated"] => 1)
+    end
+
+    it "keeps the latest per_saga of each saga and at most limit, counting every one" do
+      checkouts = [4, 1, 3, 2].map { |minute| query_execution(status: "completed", finished_at: hour + minutes(minute)) }
+      invoice_failed = query_execution(saga_name: invoice, status: "failed", finished_at: hour)
+      window = {from: hour, to: hour + hours(1)}
+
+      result = store.finished_executions(**window, per_saga: 2)
+
+      expect(ids(result[:executions])).to eq([checkouts[0].id, checkouts[2].id, invoice_failed.id])
+      expect(result[:counts]).to eq([checkout, "completed"] => 4, [invoice, "failed"] => 1)
+      expect(ids(store.finished_executions(**window, limit: 2)[:executions])).to eq([checkouts[0].id, checkouts[2].id])
+      expect(ids(store.finished_executions(**window, saga_name: [invoice])[:executions])).to eq([invoice_failed.id])
+    end
+
+    it "breaks finish ties by id, latest first" do
+      tied = Array.new(3) { query_execution(status: "completed", finished_at: hour) }
+
+      expect(ids(store.finished_executions(from: hour, to: hour + 1)[:executions])).to eq(ids(tied).sort.reverse)
+    end
+
+    it "is empty when nothing finished in the window" do
+      query_execution
+
+      expect(store.finished_executions(from: hour, to: hour + hours(1))).to eq(executions: [], counts: {})
+    end
+
+    it "lists, without counting, the executions that finished from listed_from" do
+      before = query_execution(saga_name: invoice, status: "failed", finished_at: hour - minutes(1))
+      query_execution(status: "completed", finished_at: hour - minutes(5))
+      inside = query_execution(status: "completed", finished_at: hour + minutes(1))
+
+      result = store.finished_executions(from: hour, to: hour + hours(1), listed_from: hour - minutes(2), per_saga: 5)
+
+      expect(ids(result[:executions])).to eq([inside.id, before.id])
+      expect(result[:counts]).to eq([checkout, "completed"] => 1)
+    end
+  end
+
+  describe "#executed_steps" do
+    it "names the steps each execution ran forward, keyed by id, every id present" do
+      failed = query_execution(status: "failed")
+      query_attempt(failed, step_name: "reserve")
+      query_attempt(failed, step_name: "charge", status: "failed")
+      query_attempt(failed, step_name: "charge", attempt: 2, status: "failed")
+      compensated = query_execution(status: "compensated")
+      query_attempt(compensated, step_name: "reserve")
+      query_attempt(compensated, step_name: "reserve", direction: "compensate")
+      untouched = query_execution
+
+      steps = store.executed_steps([failed, compensated, untouched])
+
+      expect(steps.keys).to contain_exactly(failed.id, compensated.id, untouched.id)
+      expect(steps[failed.id]).to contain_exactly("reserve", "charge")
+      expect(steps[compensated.id]).to eq(["reserve"])
+      expect(steps[untouched.id]).to eq([])
+      expect(store.executed_steps([])).to eq({})
+    end
+  end
+
   describe "progress and error filters" do
     it "bounds last_progress_at by progress_from (inclusive) and progress_to (exclusive)" do
       query_execution(last_progress_at: hour - minutes(1))
@@ -228,7 +400,7 @@ RSpec.shared_examples "a lepus saga query store" do
   end
 
   describe "#execution_series" do
-    it "counts starts by created_at and terminal statuses by finished_at in aligned buckets" do
+    it "counts starts by started_at and terminal statuses by finished_at in aligned buckets" do
       query_execution(created_at: hour + minutes(5))
       query_execution(created_at: hour + minutes(10), status: "completed", finished_at: hour + minutes(20))
       query_execution(created_at: hour - hours(1), status: "failed", finished_at: hour + minutes(70))

data/lib/lepus/saga/rspec/store_examples.rb

@@ -55,7 +55,7 @@ RSpec.shared_examples "a lepus saga store" do |concurrency: true|
 
       expect(execution).to have_attributes(
         status: "running", direction: "execute", epoch: 0, resume_count: 0,
-        failure_reason: nil, finished_at: nil
+        failure_reason: nil, finished_at: nil, duration_ms: nil
       )
       expect(execution.created_at).to be_within(1).of(Time.now.utc)
       expect(execution.updated_at).to be_within(1).of(Time.now.utc)
@@ -117,11 +117,12 @@ RSpec.shared_examples "a lepus saga store" do |concurrency: true|
       execution.failure_reason = "boom " * 100
       execution.context = {"after" => true}
       execution.finished_at = Time.now.utc
+      execution.duration_ms = 86_400_000 * 400
 
       expect(store.save_execution(execution)).to eq(execution)
       expect(store.find_execution(execution.id)).to have_attributes(
         status: "compensating", direction: "compensate", current_step: "second", epoch: 2,
-        failure_reason: "boom " * 100, context: {"after" => true}
+        failure_reason: "boom " * 100, context: {"after" => true}, duration_ms: 86_400_000 * 400
       )
     end
 
@@ -367,14 +368,14 @@ RSpec.shared_examples "a lepus saga store" do |concurrency: true|
   end
 
   describe "#executions_for_resource" do
-    it "returns the resource's executions, newest first" do
-      older = create_execution(resource_id: "r-1")
-      sleep 0.01
-      newer = create_execution(resource_id: "r-1")
+    it "returns the resource's executions, latest start first, whatever the ids" do
+      latest = create_execution(resource_id: "r-1", started_at: ago(minutes(1)))
+      earliest = create_execution(resource_id: "r-1", started_at: ago(hours(2)))
+      middle = create_execution(resource_id: "r-1", started_at: ago(hours(1)))
       create_execution(resource_id: "r-2")
       create_execution(resource_type: "Other", resource_id: "r-1")
 
-      expect(store.executions_for_resource("Resource", "r-1").map(&:id)).to eq([newer.id, older.id])
+      expect(store.executions_for_resource("Resource", "r-1").map(&:id)).to eq([latest.id, middle.id, earliest.id])
     end
   end