pgbus 0.20.0 → 0.20.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dd8fbf7676d82bc5e1648aba550d325d6d5e1ee30b2f87b483a123ac493506ec
-  data.tar.gz: 47e9a9949d84ea0489c3d5aba6b47bb343c3895d8d8ce15255ec699ad203f752
+  metadata.gz: c2c853eab253c7e6bbf570e0cd92ada59c071752fb3d6d8a0a5f796790aedfa3
+  data.tar.gz: 8922bdb2140718d03a2bf1cf3600c714fc2b181c566f1213091f83b28385b74c
 SHA512:
-  metadata.gz: 0dd9781845545f3203aac84f297c865604cc2b0560f6fdb2f630d99642e0b50f8a3a2a673d2d9dc91b5353b09b9b83ca073f7464a78e44cc3762b81a257ddd00
-  data.tar.gz: 791c9590b9544a5cbbf89386da81ce4b5daaceccb0df08079d95960e7b434646c63335217fcdb9960bc59124320549b6f78a54ae57e30ed692bd0eaccd50cf62
+  metadata.gz: d9d38d6bb97efc078db1afa031b51cfe2512aac285d664d09e23ce8a4a3b1a3eda075134a8b7e8d234834c31c543a6095c3cdb3bbef77cec3f3f53744e7019eb
+  data.tar.gz: 0e15d3375c20815ea3f0fbc34381e824c308f7210b67189fae56a1f11abeb3381d9b71b2b418e46c4d24bf5f5b80df3360adfccf5c7b77a02a6533d963b30699

data/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ### Added
 
+- **feat(workflow): `on_timeout: :notify` — a soft wait deadline that reports a late signal and keeps waiting.** A fifth `on_timeout:` value for a `wait_for:` step with `wait_timeout:`. When the sweep (or `expire_wait`) finds the wait past `timeout_at`, it records the `wait_timed_out` event and instruments `pgbus.workflow.wait_timed_out` with `action: "notify"`, as the other values do, and then leaves the run and the step as they are: same attempt, still `waiting`, no step outcome, no StepJob. Only `timeout_at` moves, to the next deadline on the `wait_timeout:` grid (one `wait_timeout:` on; a sweep that missed several deadlines reports them once and moves to the first one ahead of now), so a late signal still wakes that attempt and its body runs with `received_signal` set and `timed_out?` false. A paused run's wait still does not fire while paused. Replaces the `on_timeout: :run` + `reattempt!` workaround, whose `reattempted` outcomes counted as failures in `pgbus_workflow_steps_total`. New counter `pgbus_workflow_wait_overdue_total{workflow_class, step}` (Prometheus, StatsD, AppSignal), one per late notice (a sweep that missed several deadlines sends one). The dashboard says "Waiting for signal `approved` since 3h ago (late since 2h ago). The next late notice goes out in 1h" once a deadline passed (and "A late notice goes out …" before), and the timeline marks every `wait_timed_out` with a dashed line, listed as text below the chart; the workflow rows (and so the `pgbus_workflows` / `pgbus_workflow` MCP tools) gain `on_timeout` and `late_since` (the first missed deadline). The bundled AppSignal dashboard gets a "Late workflow waits" graph. Refs #565.
 - **feat(mcp): workflow write tools, opt-in — resume, cancel, skip step, resume at, run now, signal (issue #550).** `config.mcp_write_tools = true` (default `false`, validated as a boolean, read when `Pgbus::MCP::Server.build` runs) adds `pgbus_workflow_resume`, `pgbus_workflow_cancel`, `pgbus_workflow_skip_step`, `pgbus_workflow_resume_at` (`step`), `pgbus_workflow_run_now` and `pgbus_workflow_signal` (`name`, a JSON object `payload` capped at `workflow_max_payload_size`, an optional `idempotency_key`; name and key at most 255 characters) to the MCP server; off, they are never constructed and `tools/list` does not show them, and the read tools are the same either way. Each takes `workflow_id`, a required `reason` (1–500 characters after trimming) and an optional `expected_status` (a run that has moved on by the time the call starts answers `ok: false` and nothing is done; the check runs just before the action, whose transition is guarded again under the run's row lock); a verb the run's `available_actions` does not allow answers `ok: false` with the allowed list; otherwise the call goes through the dashboard's `Web::DataSource` action, is recorded on the run's audit trail with its reason as actor `mcp` (`mcp:<client name>` when the transport knows the client; the name is stripped of control characters and capped at 100), and answers `{ ok:, reason:, workflow: }` with the `pgbus_workflow` detail as the run is after the call (refusals included), payloads always redacted, plus `available_actions` for a disallowed verb and `code`/`status` when the guarded transition refused it; an unexpected failure answers a fixed reason rather than the exception's text (the DataSource logged it). The tools live on their own base class, `Pgbus::MCP::WriteTool` (not `BaseTool`), annotated `readOnlyHint: false, destructiveHint: true` (`idempotentHint: true` on cancel only: a retried skip or resume at can act again once the run parked or paused again). No bulk tools: `pgbus_workflows_attention` groups the stuck runs and `pgbus_workflows` lists a group's runs, one call per run. The server's instructions name them when they are on, `pgbus mcp` logs it at start, and an unauthenticated `Pgbus::MCP.rack_app` with them on warns that anyone who reaches it can change workflow runs. `Workflow#signal!`, `Workflow.signal!` (by subject), `Signals.deliver` and `DataSource#signal_workflow` take an optional `reason:`, recorded on the `signal_received` event. Refs #550.
 - **feat(dashboard): workflow insights — per-class pipeline, step performance, funnel, per-revision numbers, MCP stats (issue #549).** `/pgbus/workflows/definitions` (linked from the Workflows page as "Insights by class") has one row per workflow class — every registered `Pgbus::Workflow` subclass and every class name still in `pgbus_workflows` — with its ongoing, paused and stuck runs, the runs that finished or were cancelled in the window and their p50/p95 end-to-end duration, a "definition not loaded" badge for a class that no longer loads and a count of ongoing runs on removed steps. `/pgbus/workflows/definitions/:workflow_class` (`OrderWorkflow` or `Billing::OrderWorkflow`) shows the class's steps in order as a pipeline: each step with the runs sitting at it now (scheduled, running, waiting, paused), the p50/p95/p99 run time and pickup latency (scheduled → started) of its attempts that finished in the window and their failure (an error was recorded), reattempt and skip rates, with a "few samples" note under 20 attempts; then runs started, finished and cancelled per bucket and the end-to-end duration distribution (ApexCharts, each with a caption and a data table), a funnel (of the runs started in the window, how many reached each step and where each one is now) and the same numbers for the last five app revisions, newest first, marking a step whose failure rate rose more than five points over the revision before. Window 1 h, 24 h (default), 7 d or 30 d, capped at `workflow_retention`. Every count of runs a list filter can express links to the Workflows list filtered to exactly those runs (percentiles, rates, the funnel's "reached" and the per-revision rows are text), which gains `ended_from`/`ended_to` filters (when a run finished or was cancelled; also on `pgbus_workflows`); need-attention counts open `/pgbus/workflows/attention?workflow_class=…`, the attention groups narrowed to the class (also `pgbus_workflows_attention(workflow_class:)`), and each removed step a run still sits on links to those runs. Throughput buckets follow the dashboard zone's wall clock: a day bucket starts at local midnight on both sides of a DST change (at its first hour where a DST change skips midnight). Runs and attempts dated after now are not counted. Read from the step and run rows on the existing indexes (`Web::WorkflowInsightsQueries`, one query per block): on ~1 M step rows over 20 classes a class's page reads in 26 ms and the list in 86 ms over 24 h (median; budgets 250 / 500 ms, `rake bench:workflow_insights`), about 300 ms over 30 d (reported, no budget); `WorkflowEntry.stuck_reason_counts` takes `workflow_class:` so a class's page counts only its own stuck runs. A class name from the URL that no class or run has is never constantized. `Web::DataSource` gains `workflow_definitions(window:)` and `workflow_definition(workflow_class, window:)`; the MCP server gains the read-only `pgbus_workflow_definitions` and `pgbus_workflow_stats`. The workflows guide gains a Monitoring section with example Prometheus alerts. Refs #549.
 - **feat(dashboard): the Workflows pages — list and filters, needs attention, step timeline, audit trail, operator actions, MCP read tools (issue #542).** `/pgbus/workflows` lists every run newest first (ongoing by default, each state, or all) with its class, subject, status badge, current step, start and last activity, and says in words what it is doing and what happens next ("Waiting for batch `OrderWorkflow#notify_vendors` (7 of 12 jobs done)", "Paused on `capture`: Timeout::Error — resume or cancel", "Overdue: `welcome` was due 2h ago; the sweep sends it again"); filters on class, current step, error class, subject type and id, workflow id and a created-at range combine and survive paging. `/pgbus/workflows/:workflow_id` shows the arguments, an ApexCharts range-bar timeline (one row per attempt split into waited, queued and ran, coloured by outcome, with waits and pauses as their own segments, zoomable), every attempt with the same durations as text, its worker, app revision, error, backtrace and cause chain, the signals, the audit trail ("Resumed by dashboard:alice: SMTP fixed"), whether it started on an older definition, rerun links both ways, and the actions `available_actions` allows: Resume (labelled "retry `capture`" or "continue with `receipt`"), Resume at a chosen step, Run now, Skip step, Send signal (a JSON object, sent once per form render), Rerun and Cancel, each with a confirm and an optional reason, each reporting the real outcome ("Not done: the run is cancelled now."). `/pgbus/workflows/attention` groups every stuck run by class, step, reason and error class with a count, first/last seen and the newest message, with bulk Resume/Cancel (500 runs per press, each through its own guarded transition: "Resumed 212; 2 had already moved on"). The home page gains a Workflows card ("N ongoing (M paused) · K need attention") and the nav a Workflows link. New settings: `config.web_actor` (`->(request) { name }`, recorded as `"dashboard:<name>"`, or `"dashboard"`) and `config.workflow_subject_url` (`->(subject_type, subject_id) { url }`, linked only when it is an http(s) URL or a same-site path). `Web::DataSource` gains `workflows`, `workflows_count`, `workflow_detail`, `workflow_attention_groups`, the home counts and the actions; `Pgbus::WorkflowEntry.attention_groups` / `attention_ids` share the stuck-reason `CASE` with the gauge. The MCP server gains the read-only `pgbus_workflows`, `pgbus_workflow` and `pgbus_workflows_attention`, with arguments, outputs, signal payloads and error text behind `include_payloads`. Refs #542.
@@ -18,6 +19,8 @@
 
 ### Fixed
 
+- **An `:until_executed` job keeps its uniqueness key while its retry waits (issue #564).** A job that rescheduled itself with `retry_job(wait:)` (or `retry_on … wait:`) enqueued the retry through the key it held (#333), but the key row kept pointing at the running attempt, and the executor deleted the row by key when that attempt archived. For the whole wait nothing held the key: a fresh `perform_later` with the same key ran beside the retry, and the retry's own success then deleted the newcomer's row. Now the row is handed to the retry the way a first enqueue takes one: before the retry is sent or parked it is unbound onto the retry's queue and re-stamped (`UniquenessKey.unbind!`), and once sent it is bound to the retry (`UniquenessKey.bind!(…, acquired_at:)`). A retry parked behind its `limits_concurrency` slot (with `to: 1, on_conflict: :block` the attempt holds that slot, so its retry always parks) keeps the unbound row until it runs; a retry dropped by its limit, or whose send failed before any message existed, hands the row back to the attempt. A finishing message (success, dead-letter, or a failed `:while_executing` attempt) releases only the row it holds (`UniquenessKey.release_held_by!`): the row bound to it, or an unbound row acquired no later than the message was sent, so a retry that finishes before its bind still releases it; a row handed to a retry, or that a successor took after an operator released it, stays. The dashboard's discards (a job, a failed job, a dead-lettered copy, an event, the bulk discards and queue purges) release the same way, through `UniquenessKey.release_all_held_by!` for the bulk ones, judged by when the job's message was sent: a failed job's queued message gives its own `enqueued_at`, and the dead-letter header gains `source_msg_id` and `source_enqueued_at` so a DLQ copy names its original and when it was sent. Without a send time only a lock bound to the message is freed (the reaper takes the rest). The Locks page's Release still deletes by key. A retry enqueued outside the executor, or by a job running inline inside another job, moves nothing; one enqueued by an attempt that no longer holds the key moves nothing and logs a warning. The dispatcher's reaper now counts a key carried by a job parked in `pgbus_blocked_executions` (legacy double-encoded rows included) as live: a parked `:until_executed` job's unbound row was reaped once it was older than twice the visibility timeout. A parked job's `created_at` now comes from the database clock, like its lock's, so discarding it from the dashboard frees its lock even when the app's clock runs behind the database's. A job's lock now records its queue the way the queue's table spells it (`Uniqueness.lock_queue_name`: `orders-handler` → `orders_handler`), at every place a job lock is written or released (adapter, executor, recurring scheduler), so a release that knows only the table (the dashboard) or a worker that spells the queue the table's way still finds it; a multi-queue worker also hands the executor the configured queue name instead of one cut from the table name. For such a queue, a job in flight during the upgrade holds its lock under the old spelling: its first retry after the upgrade does not take the lock (logged), so it and that job's later retries run without one until the job finishes, and the stale lock is reclaimed by the reaper once its message is gone. `config.queue_prefix` is now stored the way the queue tables spell it (`"my-app"` reads back as `"my_app"`): table names are unchanged, and every place that strips the prefix off a table name (the dashboard, the worker, the lock queue above) strips it whole. A prefix with a leading, trailing or doubled underscore (`"myapp_"`) is kept as written and now warned about at boot: `queue_name` collapses the separator after it only for a queue name that needs normalizing (`myapp__orders`, but `myapp_orders_handler` for `orders-handler`), so the reaper cannot resolve such a queue's lock back to its table: a lock orphaned there (a worker crash before its release) is never reclaimed and has to be released by key. A job's own release and the dashboard still match. `Pgbus::ActiveJob::Delivery.current` gains `enqueued_at` and `job_id`. Refs #564.
+- **Presence statements run under the ActiveRecord adapter's lock (issue #566).** `Pgbus::Streams::Presence` ran every statement on `connection.raw_connection`, and `raw_connection` holds the adapter's lock only long enough to hand the `PG::Connection` out. On a pinned pool, as in transactional tests, every thread shares one adapter and that lock is all that serialises them, so the streamer's seat sweep (`expire_seats!` on the dispatcher thread) could be in flight on the same libpq connection a request thread was about to use. Every presence statement (`join`, `leave`, `touch`, `set_state`, `members`, `count`, `sweep!`, `expire_seats!` and the seat-schema probe) now goes through `Presence.exec_params`, which holds `adapter.lock` for the whole statement. The adapter lock is reentrant, so a request already inside a transaction re-enters it; unpinned, it is a `NullLock` and nothing changes. The seat-schema probe also no longer runs while holding the probe's own mutex: a sweep that probed under it while a pinned request inside `ActiveRecord::Base.transaction` (which holds the adapter's lock for its whole block) asked `seats?` deadlocked both threads. Two threads may now probe at once: a true answer one of them stored is never taken back, and a probe that a `reset_schema_cache!` overtook stores nothing. `Presence.connection` still returns the raw `PG::Connection`.
 - **A blank signal `idempotency_key` is no key (issue #550).** `wf.signal!(…, idempotency_key: "")` (or whitespace) stored `""` as a real key, so every later signal sent with a blank key dedup'd against the first: it answered `duplicate_delivery?` and was never stored, leaving a step waiting for it. `Signals.deliver` now treats a blank key as none, for app code, the dashboard and the MCP tool alike. Refs #550.
 - **`Pgbus.batch` inside a database transaction counts the jobs that enqueue after commit, and waits for them.** A job with Active Job's `enqueue_after_transaction_commit` added inside an open transaction is enqueued only when the transaction commits, after the block has returned and its thread tag is gone. The batch counted nothing, finished empty when the block returned and sent its `on_complete` (and `on_success`) callback before any member job existed; the jobs then ran outside the batch. Even a rolled-back transaction sent the callback. Now `b.add`, `batch.add` and `perform_later` record the batch on the job instance (`enqueue_batch_id`, transient: never serialized, cleared once the enqueue is done), the adapter tags and counts each job from it at the real enqueue, and the batch starts processing (or an open batch re-checks for finish) through `ActiveRecord.after_all_transactions_commit`, after every deferred enqueue has counted itself in. On a rollback nothing is sent, no callback fires and the batch row is rolled back with the transaction. A deferred first attempt discarded at enqueue (uniqueness or concurrency `on_conflict: :discard`) is uncounted as before, and a `retry_on` re-enqueue still re-joins without a second count. Outside a transaction nothing changes, including the queries a batch runs. Known gaps: a bare `ActiveJob.perform_all_later` inside the block does not record the batch (use `b.add`); a `batch.add` inside a transaction raises `AlreadyFinished` when the transaction commits, not at the `add` line, if the batch finished in the meantime; a batch row on a separate `connects_to` database is not part of the caller's transaction. Refs #553.
 - **AppSignal: the job's transaction is current while the job runs, and carries its error (issue #552).** The AppSignal subscriber opened the `background_job` transaction from a finish-only notification callback, after the job had run, and completed it in the same call. `pgbus.job_failed` fired after that, so its `Appsignal.set_error` found no current transaction and reported nothing: a job whose class no longer exists (`ActiveJob::UnknownJobClassError`) or whose arguments cannot be deserialized retried and dead-lettered with nothing in AppSignal but the counters, an error raised inside `perform` arrived only as an action-less sample from the Rails error reporter, and job code calling `Appsignal.add_tags`, `set_error` or `instrument` attached to nothing. Event handlers had the same shape. The subscriber now listens to `pgbus.executor.execute` and `pgbus.event_processed` on both sides (`start` and `finish`): the `<JobClass>#perform` / `<Handler>#handle` transaction opens before the job or handler runs, is current for the whole execution (deserialize, perform, archive), records the exception the execution raised and completes at the end, so an error raised inside `perform` lands on it once, with its duration and queue start. A failure raised before that span opens (a message body that is not JSON, an event that cannot be built, a dead-letter move that failed) is reported from `pgbus.job_failed` / `pgbus.event_failed` on a transaction of its own with the same action and tags; an error the span already recorded is not reported twice. A transaction that is already current is reused and left to its owner to complete, as AppSignal's own ActiveJob hook does, and a span inside another of its kind restores the outer one when it finishes. Like that hook, the span records any exception that ends the execution, so a job interrupted between `perform` and the archive (an `Async::Stop` on shutdown under `execution_mode: :async`, an `Interrupt`) now shows up as an error sample too. The counters and the `pgbus_job_duration_ms` / `pgbus_event_duration_ms` distributions are recorded as before, once per execution. Refs #552.

data/README.md

@@ -887,7 +887,7 @@ wf.events.map(&:event)                              # ["started", "step_schedule
 - **Idempotent by contract**: `output(charge_id: …)` is durable once the step completes and later steps read it with `output_of(:charge)`; `step_idempotency_key` is the same on every attempt, for the external API; `start!(order, idempotency_key: "checkout-#{order.id}")` returns the existing run.
 - **One ongoing run per subject** and class: a second `start!` raises `Pgbus::Workflow::AlreadyRunning`. Subject-less runs are not limited.
 - **Fan-out**: `fan_out :notify_vendors do |b| b.add(*jobs) end` opens one batch for the jobs and parks the step until every member finished; the batch's `on_complete` resumes the run, and a dead-lettered member applies `on_batch_failure:` (`:pause!` by default) with `Pgbus::Workflow::FanOutFailed` recorded. A reattempt, or `resume!` of a failed fan-out step, re-runs the whole body on a fresh batch, so members must be idempotent; a run resumed while its batch is still running keeps waiting on that batch.
-- **Signals**: `step :capture, wait_for: :payment_confirmed do … received_signal.payload["amount"] … end` parks until `wf.signal!(:payment_confirmed, payload: {…}, idempotency_key: event.id)`, or by subject, `OrderWorkflow.signal!(order, :payment_confirmed, …, missing: :ignore)`. Early signals are buffered, a redelivery with the same key is a no-op (`duplicate_delivery?`), and matchers (`Pgbus::Workflow::SignalMatcher.new(:shipped) { |payload| … }`) narrow what a step takes. `wait_timeout: 2.days, on_timeout: :pause!` (or `:cancel!`, `:skip!`, `:run`) puts a deadline on the wait, fired by the dispatcher's sweep.
+- **Signals**: `step :capture, wait_for: :payment_confirmed do … received_signal.payload["amount"] … end` parks until `wf.signal!(:payment_confirmed, payload: {…}, idempotency_key: event.id)`, or by subject, `OrderWorkflow.signal!(order, :payment_confirmed, …, missing: :ignore)`. Early signals are buffered, a redelivery with the same key is a no-op (`duplicate_delivery?`), and matchers (`Pgbus::Workflow::SignalMatcher.new(:shipped) { |payload| … }`) narrow what a step takes. `wait_timeout: 2.days, on_timeout: :pause!` (or `:cancel!`, `:skip!`, `:run`) puts a deadline on the wait, fired by the dispatcher's sweep; `on_timeout: :notify` makes it a soft deadline that reports the late signal (a `wait_timed_out` event and `pgbus_workflow_wait_overdue_total`) and keeps waiting for it, its deadline moving one `wait_timeout:` on each time (a sweep that missed several reports them once).
 - **Housekeeping**: a worker that dies mid-step is noticed when its message comes back (the visibility heartbeat keeps a live one invisible); a hung step stalls after its `timeout:` or `workflow_stall_threshold`; the dispatcher re-sends lost steps and deletes finished runs after `workflow_retention` (30 days).
 - **Metrics** for Prometheus/StatsD and AppSignal: per-step duration histograms and counters, and gauges of runs by status and by stuck reason.
 - **Dashboard**: `/pgbus/workflows` lists runs with what each one does next, filters by state, class, step, error, subject, id and creation date, and shows a run's timeline, attempts, signals and audit trail with its actions (Resume, Resume at, Run now, Skip step, Send signal, Rerun, Cancel); `/pgbus/workflows/attention` groups the stuck runs for bulk Resume/Cancel. `config.web_actor = ->(request) { … }` names the operator on the audit trail and `config.workflow_subject_url = ->(type, id) { … }` links the subject. The MCP server reads the same through `pgbus_workflows`, `pgbus_workflow` and `pgbus_workflows_attention`, and with `config.mcp_write_tools = true` acts on one run per call through `pgbus_workflow_resume`, `_cancel`, `_skip_step`, `_resume_at`, `_run_now` and `_signal` ([write tools](#write-tools-opt-in)).
@@ -1073,6 +1073,7 @@ c.metrics_backend = MyOpenTelemetryBackend.new
 | `pgbus_workflows_finished_total` / `pgbus_workflows_cancelled_total` | counter | `workflow_class` |
 | `pgbus_workflows_paused_total` | counter | `workflow_class`, `error_class` (absent for a manual pause) |
 | `pgbus_workflow_stalls_total` | counter | `workflow_class`, `step`, `detection` (`redelivery`/`sweep`) |
+| `pgbus_workflow_wait_overdue_total` | counter | `workflow_class`, `step` — one per late notice of an `on_timeout: :notify` wait (a sweep that missed several deadlines sends one) |
 | `pgbus_workflows` | gauge | `workflow_class`, `status` |
 | `pgbus_workflows_stuck` | gauge | `workflow_class`, `reason` (`paused_error`/`stalled`/`paused_manual`/`overdue`/`waiting_overdue`) |
 
@@ -1102,7 +1103,7 @@ ActiveSupport::Notifications.subscribe(/^pgbus\./) do |name, start, finish, _id,
 end
 ```
 
-Metrics-relevant events emitted (the subset the metrics subscriber maps; see `lib/pgbus/instrumentation.rb` for the full `pgbus.*` catalog): `pgbus.executor.execute`, `pgbus.job_completed`, `pgbus.job_failed`, `pgbus.job_dead_lettered`, `pgbus.event_processed`, `pgbus.event_failed`, `pgbus.event_dead_lettered`, `pgbus.event_skipped`, `pgbus.job_visibility_extended`, `pgbus.client.send_message`, `pgbus.client.send_batch`, `pgbus.client.read_batch`, `pgbus.client.pool`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`, `pgbus.consumer.recycle`, `pgbus.workflow.step`, `pgbus.workflow_finished`, `pgbus.workflow_paused`, `pgbus.workflow_cancelled`, `pgbus.workflow.stall_detected`, `pgbus.workflow.gauge`. Payload keys are documented in `lib/pgbus/instrumentation.rb`.
+Metrics-relevant events emitted (the subset the metrics subscriber maps; see `lib/pgbus/instrumentation.rb` for the full `pgbus.*` catalog): `pgbus.executor.execute`, `pgbus.job_completed`, `pgbus.job_failed`, `pgbus.job_dead_lettered`, `pgbus.event_processed`, `pgbus.event_failed`, `pgbus.event_dead_lettered`, `pgbus.event_skipped`, `pgbus.job_visibility_extended`, `pgbus.client.send_message`, `pgbus.client.send_batch`, `pgbus.client.read_batch`, `pgbus.client.pool`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`, `pgbus.consumer.recycle`, `pgbus.workflow.step`, `pgbus.workflow_finished`, `pgbus.workflow_paused`, `pgbus.workflow_cancelled`, `pgbus.workflow.stall_detected`, `pgbus.workflow.wait_timed_out`, `pgbus.workflow.gauge`. Payload keys are documented in `lib/pgbus/instrumentation.rb`.
 
 ### Connection pool metrics
 

data/app/frontend/pgbus/modules/charts.js

@@ -233,9 +233,27 @@ export async function renderTimeline(data, i18n) {
     yaxis: { labels: { style: { colors: colors.text } } },
     legend: { position: "top", labels: { colors: colors.text } },
     tooltip: { theme: colors.mode, custom: timelineTooltip },
+    annotations: { xaxis: waitTimeoutMarks(data, i18n, colors) },
   });
 }
 
+// A dashed line where a wait passed its deadline (issue #565): the waiting
+// colour, labelled in the page's text colour on the card, so it reads in
+// both themes. The page lists the same moments as text.
+function waitTimeoutMarks(data, i18n, colors) {
+  return (data.wait_timeouts || []).map((mark) => ({
+    x: mark.at,
+    borderColor: colors.timeline.waiting,
+    strokeDashArray: 4,
+    label: {
+      text: i18n.wait_timed_out,
+      orientation: "horizontal",
+      borderColor: colors.timeline.waiting,
+      style: { color: colors.text, background: colors.card },
+    },
+  }));
+}
+
 // The workflow insights (issue #549): runs started, finished and cancelled
 // per bucket (cancelled in the neutral grey the dashboard uses for it, not
 // the failure red), and the end-to-end duration histogram of the finished

data/app/frontend/pgbus/modules/charts.min.js

@@ -1,2 +1,2 @@
-function x(t){return{mode:t?"dark":"light",text:t?"#d1d5db":"#4b5563",grid:t?"#374151":"#e5e7eb",axis:t?"#4b5563":"#d1d5db",card:t?"#1f2937":"#ffffff",throughput:t?"#818cf8":"#6366f1",latencyAvg:t?"#818cf8":"#6366f1",latencyP95:t?"#fbbf24":"#f59e0b",status:{success:"#10b981",failed:"#ef4444",dead_lettered:"#f97316"},other:t?"#9ca3af":"#6b7280",timeline:{waited:t?"#4b5563":"#d1d5db",queued:t?"#64748b":"#94a3b8",waiting:t?"#fbbf24":"#f59e0b",paused:t?"#fb923c":"#ea580c",ran_ok:"#10b981",ran_failed:"#ef4444",ran_open:t?"#818cf8":"#6366f1",ran_cancelled:t?"#9ca3af":"#6b7280"}}}function w(){return window.matchMedia("(prefers-reduced-motion: reduce)").matches}function l(t,o,r){return{chart:{id:t,type:o,height:280,toolbar:{show:!1},background:"transparent",foreColor:r.text,animations:{enabled:!w()}},theme:{mode:r.mode},tooltip:{theme:r.mode},grid:{borderColor:r.grid},legend:{labels:{colors:r.text}},dataLabels:{enabled:!1}}}function p(t){return{type:"datetime",labels:{style:{colors:t.text}},axisBorder:{color:t.axis},axisTicks:{color:t.axis},tooltip:{enabled:!1}}}function f(t,o){return{labels:{style:{colors:t.text},formatter:o},axisBorder:{show:!0,color:t.axis}}}function c(t,o){return t.map((r)=>({x:new Date(r.time).getTime(),y:r[o]}))}var m=null;function b(){return m||=import("apexcharts").then(()=>globalThis.ApexCharts).catch((t)=>{throw m=null,t}),m}var u={};function i(t,o,r){if(u[o])u[o].destroy();delete u[o];let e=document.getElementById(o);if(!e)return;u[o]=new t(e,r),u[o].render()}async function v(t,o){let r=await b(),e=x(document.documentElement.classList.contains("dark"));i(r,"throughput-chart",{...l("throughput-chart","area",e),series:[{name:o.seriesName,data:c(t.throughput||[],"count")}],stroke:{curve:"smooth",width:2},fill:{type:"gradient",gradient:{shadeIntensity:1,opacityFrom:0.4,opacityTo:0.05,stops:[0,100]}},colors:[e.throughput],xaxis:p(e),yaxis:f(e,(a)=>Math.round(a))});let s=Object.keys(t.status_counts||{});i(r,"status-chart",{...l("status-chart","donut",e),series:s.map((a)=>t.status_counts[a]),labels:s.map((a)=>o.statuses&&o.statuses[a]||a),colors:s.map((a)=>e.status[a]||e.other),stroke:{colors:[e.card]},legend:{position:"bottom",labels:{colors:e.text}},plotOptions:{pie:{donut:{size:"60%",labels:{show:!0,name:{color:e.text},value:{color:e.text},total:{show:!0,color:e.text}}}}}}),i(r,"latency-chart",{...l("latency-chart","line",e),series:[{name:o.latencyAvg,data:c(t.latency_trend||[],"avg_ms")},{name:o.latencyP95,data:c(t.latency_trend||[],"p95_ms")}],stroke:{curve:"smooth",width:[2,2],dashArray:[0,5]},colors:[e.latencyAvg,e.latencyP95],xaxis:p(e),yaxis:f(e,(a)=>o.milliseconds.replace("%n",Math.round(a))),legend:{position:"top",labels:{colors:e.text}}})}var g=["waited","queued","waiting","ran_ok","ran_failed","ran_open","ran_cancelled","paused"];function _(t,o){let r=Object.fromEntries(g.map((s)=>[s,[]])),e=(s,a,n)=>({x:s,y:[a.from,a.to],duration:a.duration,status:n&&o.statuses?o.statuses[n.status]||n.status:null,error:n?n.error_class:null});return(t.rows||[]).forEach((s)=>{s.segments.forEach((a)=>{if(r[a.series])r[a.series].push(e(s.label,a,s))})}),(t.paused||[]).forEach((s)=>r.paused.push(e(o.paused_row,s,null))),g.filter((s)=>r[s].length>0).map((s)=>({name:o.series&&o.series[s]||s,key:s,data:r[s]}))}function d(t){let o=document.createElement("div");return o.textContent=t==null?"":String(t),o.innerHTML}function E({seriesIndex:t,dataPointIndex:o,w:r}){let e=r.config.series[t],s=e.data[o],a=[`${d(e.name)}: ${d(s.duration)}`];if(s.status)a.push(d(s.status));if(s.error)a.push(d(s.error));return`<div class="apexcharts-tooltip-title">${d(s.x)}</div><div style="padding: 4px 10px">${a.join("<br>")}</div>`}async function A(t,o){let r=await b(),e=x(document.documentElement.classList.contains("dark")),s=_(t,o),a=(t.rows||[]).length+((t.paused||[]).length>0?1:0),n=l("workflow-timeline-chart","rangeBar",e);i(r,"workflow-timeline-chart",{...n,chart:{...n.chart,height:Math.max(160,a*36+90),zoom:{enabled:!0,type:"x"},toolbar:{show:!0,tools:{download:!1,selection:!1,zoom:!0,zoomin:!0,zoomout:!0,pan:!0,reset:!0}}},series:s,colors:s.map((y)=>e.timeline[y.key]||e.other),plotOptions:{bar:{horizontal:!0,rangeBarGroupRows:!0,barHeight:"60%"}},xaxis:p(e),yaxis:{labels:{style:{colors:e.text}}},legend:{position:"top",labels:{colors:e.text}},tooltip:{theme:e.mode,custom:E}})}async function C(t,o){let r=await b(),e=x(document.documentElement.classList.contains("dark")),s=t.throughput||[];i(r,"workflow-throughput-chart",{...l("workflow-throughput-chart","line",e),series:[{name:o.started,data:c(s,"started")},{name:o.finished,data:c(s,"finished")},{name:o.cancelled,data:c(s,"cancelled")}],stroke:{curve:"smooth",width:[2,2,2],dashArray:[0,0,5]},colors:[e.throughput,e.status.success,e.other],xaxis:p(e),yaxis:f(e,(n)=>Math.round(n)),legend:{position:"top",labels:{colors:e.text}}});let a=t.durations||[];i(r,"workflow-duration-chart",{...l("workflow-duration-chart","bar",e),series:[{name:o.runs,data:a.map((n)=>n.count)}],colors:[e.throughput],plotOptions:{bar:{columnWidth:"70%",borderRadius:2}},xaxis:{categories:a.map((n)=>n.label),labels:{style:{colors:e.text}},axisBorder:{color:e.axis},axisTicks:{color:e.axis}},yaxis:f(e,(n)=>Math.round(n))})}var h=null;function T(t,o,r=v){if(h)h.disconnect();h=new MutationObserver(()=>{let e=typeof t==="function"?t():t;if(e)r(e,o)}),h.observe(document.documentElement,{attributes:!0,attributeFilter:["class"]})}export{T as observeThemeChanges,v as renderCharts,A as renderTimeline,C as renderWorkflowInsights};
+function x(t){return{mode:t?"dark":"light",text:t?"#d1d5db":"#4b5563",grid:t?"#374151":"#e5e7eb",axis:t?"#4b5563":"#d1d5db",card:t?"#1f2937":"#ffffff",throughput:t?"#818cf8":"#6366f1",latencyAvg:t?"#818cf8":"#6366f1",latencyP95:t?"#fbbf24":"#f59e0b",status:{success:"#10b981",failed:"#ef4444",dead_lettered:"#f97316"},other:t?"#9ca3af":"#6b7280",timeline:{waited:t?"#4b5563":"#d1d5db",queued:t?"#64748b":"#94a3b8",waiting:t?"#fbbf24":"#f59e0b",paused:t?"#fb923c":"#ea580c",ran_ok:"#10b981",ran_failed:"#ef4444",ran_open:t?"#818cf8":"#6366f1",ran_cancelled:t?"#9ca3af":"#6b7280"}}}function w(){return window.matchMedia("(prefers-reduced-motion: reduce)").matches}function l(t,o,s){return{chart:{id:t,type:o,height:280,toolbar:{show:!1},background:"transparent",foreColor:s.text,animations:{enabled:!w()}},theme:{mode:s.mode},tooltip:{theme:s.mode},grid:{borderColor:s.grid},legend:{labels:{colors:s.text}},dataLabels:{enabled:!1}}}function p(t){return{type:"datetime",labels:{style:{colors:t.text}},axisBorder:{color:t.axis},axisTicks:{color:t.axis},tooltip:{enabled:!1}}}function m(t,o){return{labels:{style:{colors:t.text},formatter:o},axisBorder:{show:!0,color:t.axis}}}function i(t,o){return t.map((s)=>({x:new Date(s.time).getTime(),y:s[o]}))}var f=null;function b(){return f||=import("apexcharts").then(()=>globalThis.ApexCharts).catch((t)=>{throw f=null,t}),f}var u={};function c(t,o,s){if(u[o])u[o].destroy();delete u[o];let e=document.getElementById(o);if(!e)return;u[o]=new t(e,s),u[o].render()}async function _(t,o){let s=await b(),e=x(document.documentElement.classList.contains("dark"));c(s,"throughput-chart",{...l("throughput-chart","area",e),series:[{name:o.seriesName,data:i(t.throughput||[],"count")}],stroke:{curve:"smooth",width:2},fill:{type:"gradient",gradient:{shadeIntensity:1,opacityFrom:0.4,opacityTo:0.05,stops:[0,100]}},colors:[e.throughput],xaxis:p(e),yaxis:m(e,(r)=>Math.round(r))});let a=Object.keys(t.status_counts||{});c(s,"status-chart",{...l("status-chart","donut",e),series:a.map((r)=>t.status_counts[r]),labels:a.map((r)=>o.statuses&&o.statuses[r]||r),colors:a.map((r)=>e.status[r]||e.other),stroke:{colors:[e.card]},legend:{position:"bottom",labels:{colors:e.text}},plotOptions:{pie:{donut:{size:"60%",labels:{show:!0,name:{color:e.text},value:{color:e.text},total:{show:!0,color:e.text}}}}}}),c(s,"latency-chart",{...l("latency-chart","line",e),series:[{name:o.latencyAvg,data:i(t.latency_trend||[],"avg_ms")},{name:o.latencyP95,data:i(t.latency_trend||[],"p95_ms")}],stroke:{curve:"smooth",width:[2,2],dashArray:[0,5]},colors:[e.latencyAvg,e.latencyP95],xaxis:p(e),yaxis:m(e,(r)=>o.milliseconds.replace("%n",Math.round(r))),legend:{position:"top",labels:{colors:e.text}}})}var g=["waited","queued","waiting","ran_ok","ran_failed","ran_open","ran_cancelled","paused"];function v(t,o){let s=Object.fromEntries(g.map((a)=>[a,[]])),e=(a,r,n)=>({x:a,y:[r.from,r.to],duration:r.duration,status:n&&o.statuses?o.statuses[n.status]||n.status:null,error:n?n.error_class:null});return(t.rows||[]).forEach((a)=>{a.segments.forEach((r)=>{if(s[r.series])s[r.series].push(e(a.label,r,a))})}),(t.paused||[]).forEach((a)=>s.paused.push(e(o.paused_row,a,null))),g.filter((a)=>s[a].length>0).map((a)=>({name:o.series&&o.series[a]||a,key:a,data:s[a]}))}function d(t){let o=document.createElement("div");return o.textContent=t==null?"":String(t),o.innerHTML}function E({seriesIndex:t,dataPointIndex:o,w:s}){let e=s.config.series[t],a=e.data[o],r=[`${d(e.name)}: ${d(a.duration)}`];if(a.status)r.push(d(a.status));if(a.error)r.push(d(a.error));return`<div class="apexcharts-tooltip-title">${d(a.x)}</div><div style="padding: 4px 10px">${r.join("<br>")}</div>`}async function C(t,o){let s=await b(),e=x(document.documentElement.classList.contains("dark")),a=v(t,o),r=(t.rows||[]).length+((t.paused||[]).length>0?1:0),n=l("workflow-timeline-chart","rangeBar",e);c(s,"workflow-timeline-chart",{...n,chart:{...n.chart,height:Math.max(160,r*36+90),zoom:{enabled:!0,type:"x"},toolbar:{show:!0,tools:{download:!1,selection:!1,zoom:!0,zoomin:!0,zoomout:!0,pan:!0,reset:!0}}},series:a,colors:a.map((y)=>e.timeline[y.key]||e.other),plotOptions:{bar:{horizontal:!0,rangeBarGroupRows:!0,barHeight:"60%"}},xaxis:p(e),yaxis:{labels:{style:{colors:e.text}}},legend:{position:"top",labels:{colors:e.text}},tooltip:{theme:e.mode,custom:E},annotations:{xaxis:A(t,o,e)}})}function A(t,o,s){return(t.wait_timeouts||[]).map((e)=>({x:e.at,borderColor:s.timeline.waiting,strokeDashArray:4,label:{text:o.wait_timed_out,orientation:"horizontal",borderColor:s.timeline.waiting,style:{color:s.text,background:s.card}}}))}async function k(t,o){let s=await b(),e=x(document.documentElement.classList.contains("dark")),a=t.throughput||[];c(s,"workflow-throughput-chart",{...l("workflow-throughput-chart","line",e),series:[{name:o.started,data:i(a,"started")},{name:o.finished,data:i(a,"finished")},{name:o.cancelled,data:i(a,"cancelled")}],stroke:{curve:"smooth",width:[2,2,2],dashArray:[0,0,5]},colors:[e.throughput,e.status.success,e.other],xaxis:p(e),yaxis:m(e,(n)=>Math.round(n)),legend:{position:"top",labels:{colors:e.text}}});let r=t.durations||[];c(s,"workflow-duration-chart",{...l("workflow-duration-chart","bar",e),series:[{name:o.runs,data:r.map((n)=>n.count)}],colors:[e.throughput],plotOptions:{bar:{columnWidth:"70%",borderRadius:2}},xaxis:{categories:r.map((n)=>n.label),labels:{style:{colors:e.text}},axisBorder:{color:e.axis},axisTicks:{color:e.axis}},yaxis:m(e,(n)=>Math.round(n))})}var h=null;function T(t,o,s=_){if(h)h.disconnect();h=new MutationObserver(()=>{let e=typeof t==="function"?t():t;if(e)s(e,o)}),h.observe(document.documentElement,{attributes:!0,attributeFilter:["class"]})}export{T as observeThemeChanges,_ as renderCharts,C as renderTimeline,k as renderWorkflowInsights};
 //# sourceMappingURL=charts.map

data/app/frontend/pgbus/modules/charts.min.js.map

@@ -1 +1 @@
-{"version":3,"sources":["src/charts.js"],"sourcesContent":["// Charts module for the Pgbus Insights page, the workflow timeline and the\n// workflow insights.\n// Loaded only on those pages via an inline <script type=\"module\">, which\n// passes it the series the page embeds. ApexCharts is fetched here, on first\n// draw, so no other dashboard page downloads it (issue #517). The page renders each chart's caption,\n// data table and empty state; this module only draws into the containers that\n// exist, and redraws them when <html class=\"dark\"> toggles.\n\n// Colours for both themes in one place (issue #492). card is the chart card's\n// own background (white / gray-800): the donut's slice separators use it so\n// they read as gaps, not white lines, in dark mode.\nfunction palette(isDark) {\n  return {\n    mode: isDark ? \"dark\" : \"light\",\n    text: isDark ? \"#d1d5db\" : \"#4b5563\",\n    grid: isDark ? \"#374151\" : \"#e5e7eb\",\n    axis: isDark ? \"#4b5563\" : \"#d1d5db\",\n    card: isDark ? \"#1f2937\" : \"#ffffff\",\n    throughput: isDark ? \"#818cf8\" : \"#6366f1\",\n    latencyAvg: isDark ? \"#818cf8\" : \"#6366f1\",\n    latencyP95: isDark ? \"#fbbf24\" : \"#f59e0b\",\n    status: {\n      success: \"#10b981\",\n      failed: \"#ef4444\",\n      dead_lettered: \"#f97316\",\n    },\n    other: isDark ? \"#9ca3af\" : \"#6b7280\",\n    // The workflow timeline's segments (issue #542): the waits in neutral\n    // tones, ran coloured by how the attempt ended.\n    timeline: {\n      waited: isDark ? \"#4b5563\" : \"#d1d5db\",\n      queued: isDark ? \"#64748b\" : \"#94a3b8\",\n      waiting: isDark ? \"#fbbf24\" : \"#f59e0b\",\n      paused: isDark ? \"#fb923c\" : \"#ea580c\",\n      ran_ok: \"#10b981\",\n      ran_failed: \"#ef4444\",\n      ran_open: isDark ? \"#818cf8\" : \"#6366f1\",\n      ran_cancelled: isDark ? \"#9ca3af\" : \"#6b7280\",\n    },\n  };\n}\n\nfunction reducedMotion() {\n  return window.matchMedia(\"(prefers-reduced-motion: reduce)\").matches;\n}\n\n// Options every chart shares: theme, text, axes, grid, tooltip, animation.\nfunction baseOptions(id, type, colors) {\n  return {\n    chart: {\n      id,\n      type,\n      height: 280,\n      toolbar: { show: false },\n      background: \"transparent\",\n      foreColor: colors.text,\n      animations: { enabled: !reducedMotion() },\n    },\n    theme: { mode: colors.mode },\n    tooltip: { theme: colors.mode },\n    grid: { borderColor: colors.grid },\n    legend: { labels: { colors: colors.text } },\n    dataLabels: { enabled: false },\n  };\n}\n\nfunction timeAxis(colors) {\n  return {\n    type: \"datetime\",\n    labels: { style: { colors: colors.text } },\n    axisBorder: { color: colors.axis },\n    axisTicks: { color: colors.axis },\n    tooltip: { enabled: false },\n  };\n}\n\nfunction valueAxis(colors, formatter) {\n  return {\n    labels: { style: { colors: colors.text }, formatter },\n    axisBorder: { show: true, color: colors.axis },\n  };\n}\n\nfunction points(series, key) {\n  return series.map((p) => ({ x: new Date(p.time).getTime(), y: p[key] }));\n}\n\n// vendor/apexcharts.js is a UMD bundle: run as a module it defines the global.\n// Memoized, so the theme-toggle redraws reuse the one download. Every caller\n// awaits the same promise, and its callbacks run in call order, so a redraw\n// can't overtake the first render. A failed import clears the memo, so the\n// next draw (a theme toggle) retries instead of awaiting the same rejection.\nlet apexCharts = null;\nfunction loadApexCharts() {\n  apexCharts ||= import(\"apexcharts\")\n    .then(() => globalThis.ApexCharts)\n    .catch((error) => {\n      apexCharts = null;\n      throw error;\n    });\n  return apexCharts;\n}\n\nconst charts = {};\n\nfunction draw(ApexCharts, id, options) {\n  if (charts[id]) charts[id].destroy();\n  delete charts[id];\n\n  const el = document.getElementById(id);\n  if (!el) return;\n\n  charts[id] = new ApexCharts(el, options);\n  charts[id].render();\n}\n\nexport async function renderCharts(data, i18n) {\n  const ApexCharts = await loadApexCharts();\n  const colors = palette(document.documentElement.classList.contains(\"dark\"));\n\n  draw(ApexCharts, \"throughput-chart\", {\n    ...baseOptions(\"throughput-chart\", \"area\", colors),\n    series: [{ name: i18n.seriesName, data: points(data.throughput || [], \"count\") }],\n    stroke: { curve: \"smooth\", width: 2 },\n    fill: { type: \"gradient\", gradient: { shadeIntensity: 1, opacityFrom: 0.4, opacityTo: 0.05, stops: [0, 100] } },\n    colors: [colors.throughput],\n    xaxis: timeAxis(colors),\n    yaxis: valueAxis(colors, (v) => Math.round(v)),\n  });\n\n  const statuses = Object.keys(data.status_counts || {});\n  draw(ApexCharts, \"status-chart\", {\n    ...baseOptions(\"status-chart\", \"donut\", colors),\n    series: statuses.map((s) => data.status_counts[s]),\n    labels: statuses.map((s) => (i18n.statuses && i18n.statuses[s]) || s),\n    colors: statuses.map((s) => colors.status[s] || colors.other),\n    stroke: { colors: [colors.card] },\n    legend: { position: \"bottom\", labels: { colors: colors.text } },\n    plotOptions: {\n      pie: {\n        donut: {\n          size: \"60%\",\n          labels: {\n            show: true,\n            name: { color: colors.text },\n            value: { color: colors.text },\n            total: { show: true, color: colors.text },\n          },\n        },\n      },\n    },\n  });\n\n  draw(ApexCharts, \"latency-chart\", {\n    ...baseOptions(\"latency-chart\", \"line\", colors),\n    series: [\n      { name: i18n.latencyAvg, data: points(data.latency_trend || [], \"avg_ms\") },\n      { name: i18n.latencyP95, data: points(data.latency_trend || [], \"p95_ms\") },\n    ],\n    stroke: { curve: \"smooth\", width: [2, 2], dashArray: [0, 5] },\n    colors: [colors.latencyAvg, colors.latencyP95],\n    xaxis: timeAxis(colors),\n    yaxis: valueAxis(colors, (v) => i18n.milliseconds.replace(\"%n\", Math.round(v))),\n    legend: { position: \"top\", labels: { colors: colors.text } },\n  });\n}\n\nconst TIMELINE_SERIES = [\"waited\", \"queued\", \"waiting\", \"ran_ok\", \"ran_failed\", \"ran_open\", \"ran_cancelled\", \"paused\"];\n\n// One range bar per segment, grouped by series so each kind keeps its colour.\n// Rows are the attempts (oldest first) plus a \"Paused\" row for the pauses.\nfunction timelineSeries(data, i18n) {\n  const points = Object.fromEntries(TIMELINE_SERIES.map((name) => [name, []]));\n  const point = (label, segment, row) => ({\n    x: label,\n    y: [segment.from, segment.to],\n    duration: segment.duration,\n    status: row && i18n.statuses ? i18n.statuses[row.status] || row.status : null,\n    error: row ? row.error_class : null,\n  });\n\n  (data.rows || []).forEach((row) => {\n    row.segments.forEach((segment) => {\n      if (points[segment.series]) points[segment.series].push(point(row.label, segment, row));\n    });\n  });\n  (data.paused || []).forEach((segment) => points.paused.push(point(i18n.paused_row, segment, null)));\n\n  return TIMELINE_SERIES.filter((name) => points[name].length > 0).map((name) => ({\n    name: (i18n.series && i18n.series[name]) || name,\n    key: name,\n    data: points[name],\n  }));\n}\n\n// The tooltip: the segment's kind and duration, then the attempt's status and\n// error class. Text only (textContent-safe): every value is escaped.\nfunction escapeText(value) {\n  const div = document.createElement(\"div\");\n  div.textContent = value == null ? \"\" : String(value);\n  return div.innerHTML;\n}\n\nfunction timelineTooltip({ seriesIndex, dataPointIndex, w }) {\n  const series = w.config.series[seriesIndex];\n  const p = series.data[dataPointIndex];\n  const lines = [`${escapeText(series.name)}: ${escapeText(p.duration)}`];\n  if (p.status) lines.push(escapeText(p.status));\n  if (p.error) lines.push(escapeText(p.error));\n  return `<div class=\"apexcharts-tooltip-title\">${escapeText(p.x)}</div><div style=\"padding: 4px 10px\">${lines.join(\"<br>\")}</div>`;\n}\n\nexport async function renderTimeline(data, i18n) {\n  const ApexCharts = await loadApexCharts();\n  const colors = palette(document.documentElement.classList.contains(\"dark\"));\n  const series = timelineSeries(data, i18n);\n  const rows = (data.rows || []).length + ((data.paused || []).length > 0 ? 1 : 0);\n  const base = baseOptions(\"workflow-timeline-chart\", \"rangeBar\", colors);\n\n  draw(ApexCharts, \"workflow-timeline-chart\", {\n    ...base,\n    chart: {\n      ...base.chart,\n      height: Math.max(160, rows * 36 + 90),\n      // Waits can span days: zoom and pan along time, reset to see it all.\n      zoom: { enabled: true, type: \"x\" },\n      toolbar: { show: true, tools: { download: false, selection: false, zoom: true, zoomin: true, zoomout: true, pan: true, reset: true } },\n    },\n    series,\n    colors: series.map((s) => colors.timeline[s.key] || colors.other),\n    plotOptions: { bar: { horizontal: true, rangeBarGroupRows: true, barHeight: \"60%\" } },\n    xaxis: timeAxis(colors),\n    yaxis: { labels: { style: { colors: colors.text } } },\n    legend: { position: \"top\", labels: { colors: colors.text } },\n    tooltip: { theme: colors.mode, custom: timelineTooltip },\n  });\n}\n\n// The workflow insights (issue #549): runs started, finished and cancelled\n// per bucket (cancelled in the neutral grey the dashboard uses for it, not\n// the failure red), and the end-to-end duration histogram of the finished\n// runs. The bucket times arrive as the dashboard zone's wall clock written\n// as UTC, which the UTC time axis prints as is.\nexport async function renderWorkflowInsights(data, i18n) {\n  const ApexCharts = await loadApexCharts();\n  const colors = palette(document.documentElement.classList.contains(\"dark\"));\n  const throughput = data.throughput || [];\n\n  draw(ApexCharts, \"workflow-throughput-chart\", {\n    ...baseOptions(\"workflow-throughput-chart\", \"line\", colors),\n    series: [\n      { name: i18n.started, data: points(throughput, \"started\") },\n      { name: i18n.finished, data: points(throughput, \"finished\") },\n      { name: i18n.cancelled, data: points(throughput, \"cancelled\") },\n    ],\n    stroke: { curve: \"smooth\", width: [2, 2, 2], dashArray: [0, 0, 5] },\n    colors: [colors.throughput, colors.status.success, colors.other],\n    xaxis: timeAxis(colors),\n    yaxis: valueAxis(colors, (v) => Math.round(v)),\n    legend: { position: \"top\", labels: { colors: colors.text } },\n  });\n\n  const durations = data.durations || [];\n  draw(ApexCharts, \"workflow-duration-chart\", {\n    ...baseOptions(\"workflow-duration-chart\", \"bar\", colors),\n    series: [{ name: i18n.runs, data: durations.map((bin) => bin.count) }],\n    colors: [colors.throughput],\n    plotOptions: { bar: { columnWidth: \"70%\", borderRadius: 2 } },\n    xaxis: {\n      categories: durations.map((bin) => bin.label),\n      labels: { style: { colors: colors.text } },\n      axisBorder: { color: colors.axis },\n      axisTicks: { color: colors.axis },\n    },\n    yaxis: valueAxis(colors, (v) => Math.round(v)),\n  });\n}\n\nlet themeObserver = null;\n\n// The page toggles dark mode with a class on <html>; redraw in the new palette.\n// render is the page's draw function (renderCharts on Insights, renderTimeline\n// on a workflow, renderWorkflowInsights on a workflow class).\nexport function observeThemeChanges(getDataFn, i18n, render = renderCharts) {\n  if (themeObserver) themeObserver.disconnect();\n\n  themeObserver = new MutationObserver(() => {\n    const data = typeof getDataFn === \"function\" ? getDataFn() : getDataFn;\n    if (data) render(data, i18n);\n  });\n  themeObserver.observe(document.documentElement, { attributes: true, attributeFilter: [\"class\"] });\n}\n"],"mappings":"AAWA,SAAS,CAAO,CAAC,EAAQ,CACvB,MAAO,CACL,KAAM,EAAS,OAAS,QACxB,KAAM,EAAS,UAAY,UAC3B,KAAM,EAAS,UAAY,UAC3B,KAAM,EAAS,UAAY,UAC3B,KAAM,EAAS,UAAY,UAC3B,WAAY,EAAS,UAAY,UACjC,WAAY,EAAS,UAAY,UACjC,WAAY,EAAS,UAAY,UACjC,OAAQ,CACN,QAAS,UACT,OAAQ,UACR,cAAe,SACjB,EACA,MAAO,EAAS,UAAY,UAG5B,SAAU,CACR,OAAQ,EAAS,UAAY,UAC7B,OAAQ,EAAS,UAAY,UAC7B,QAAS,EAAS,UAAY,UAC9B,OAAQ,EAAS,UAAY,UAC7B,OAAQ,UACR,WAAY,UACZ,SAAU,EAAS,UAAY,UAC/B,cAAe,EAAS,UAAY,SACtC,CACF,EAGF,SAAS,CAAa,EAAG,CACvB,OAAO,OAAO,WAAW,kCAAkC,EAAE,QAI/D,SAAS,CAAW,CAAC,EAAI,EAAM,EAAQ,CACrC,MAAO,CACL,MAAO,CACL,KACA,OACA,OAAQ,IACR,QAAS,CAAE,KAAM,EAAM,EACvB,WAAY,cACZ,UAAW,EAAO,KAClB,WAAY,CAAE,QAAS,CAAC,EAAc,CAAE,CAC1C,EACA,MAAO,CAAE,KAAM,EAAO,IAAK,EAC3B,QAAS,CAAE,MAAO,EAAO,IAAK,EAC9B,KAAM,CAAE,YAAa,EAAO,IAAK,EACjC,OAAQ,CAAE,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,EAC1C,WAAY,CAAE,QAAS,EAAM,CAC/B,EAGF,SAAS,CAAQ,CAAC,EAAQ,CACxB,MAAO,CACL,KAAM,WACN,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,CAAE,EACzC,WAAY,CAAE,MAAO,EAAO,IAAK,EACjC,UAAW,CAAE,MAAO,EAAO,IAAK,EAChC,QAAS,CAAE,QAAS,EAAM,CAC5B,EAGF,SAAS,CAAS,CAAC,EAAQ,EAAW,CACpC,MAAO,CACL,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,EAAG,WAAU,EACpD,WAAY,CAAE,KAAM,GAAM,MAAO,EAAO,IAAK,CAC/C,EAGF,SAAS,CAAM,CAAC,EAAQ,EAAK,CAC3B,OAAO,EAAO,IAAI,CAAC,KAAO,CAAE,EAAG,IAAI,KAAK,EAAE,IAAI,EAAE,QAAQ,EAAG,EAAG,EAAE,EAAK,EAAE,EAQzE,IAAI,EAAa,KACjB,SAAS,CAAc,EAAG,CAOxB,OANA,IAAsB,qBACnB,KAAK,IAAM,WAAW,UAAU,EAChC,MAAM,CAAC,IAAU,CAEhB,MADA,EAAa,KACP,EACP,EACI,EAGT,IAAM,EAAS,CAAC,EAEhB,SAAS,CAAI,CAAC,EAAY,EAAI,EAAS,CACrC,GAAI,EAAO,GAAK,EAAO,GAAI,QAAQ,EACnC,OAAO,EAAO,GAEd,IAAM,EAAK,SAAS,eAAe,CAAE,EACrC,GAAI,CAAC,EAAI,OAET,EAAO,GAAM,IAAI,EAAW,EAAI,CAAO,EACvC,EAAO,GAAI,OAAO,EAGpB,eAAsB,CAAY,CAAC,EAAM,EAAM,CAC7C,IAAM,EAAa,MAAM,EAAe,EAClC,EAAS,EAAQ,SAAS,gBAAgB,UAAU,SAAS,MAAM,CAAC,EAE1E,EAAK,EAAY,mBAAoB,IAChC,EAAY,mBAAoB,OAAQ,CAAM,EACjD,OAAQ,CAAC,CAAE,KAAM,EAAK,WAAY,KAAM,EAAO,EAAK,YAAc,CAAC,EAAG,OAAO,CAAE,CAAC,EAChF,OAAQ,CAAE,MAAO,SAAU,MAAO,CAAE,EACpC,KAAM,CAAE,KAAM,WAAY,SAAU,CAAE,eAAgB,EAAG,YAAa,IAAK,UAAW,KAAM,MAAO,CAAC,EAAG,GAAG,CAAE,CAAE,EAC9G,OAAQ,CAAC,EAAO,UAAU,EAC1B,MAAO,EAAS,CAAM,EACtB,MAAO,EAAU,EAAQ,CAAC,IAAM,KAAK,MAAM,CAAC,CAAC,CAC/C,CAAC,EAED,IAAM,EAAW,OAAO,KAAK,EAAK,eAAiB,CAAC,CAAC,EACrD,EAAK,EAAY,eAAgB,IAC5B,EAAY,eAAgB,QAAS,CAAM,EAC9C,OAAQ,EAAS,IAAI,CAAC,IAAM,EAAK,cAAc,EAAE,EACjD,OAAQ,EAAS,IAAI,CAAC,IAAO,EAAK,UAAY,EAAK,SAAS,IAAO,CAAC,EACpE,OAAQ,EAAS,IAAI,CAAC,IAAM,EAAO,OAAO,IAAM,EAAO,KAAK,EAC5D,OAAQ,CAAE,OAAQ,CAAC,EAAO,IAAI,CAAE,EAChC,OAAQ,CAAE,SAAU,SAAU,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,EAC9D,YAAa,CACX,IAAK,CACH,MAAO,CACL,KAAM,MACN,OAAQ,CACN,KAAM,GACN,KAAM,CAAE,MAAO,EAAO,IAAK,EAC3B,MAAO,CAAE,MAAO,EAAO,IAAK,EAC5B,MAAO,CAAE,KAAM,GAAM,MAAO,EAAO,IAAK,CAC1C,CACF,CACF,CACF,CACF,CAAC,EAED,EAAK,EAAY,gBAAiB,IAC7B,EAAY,gBAAiB,OAAQ,CAAM,EAC9C,OAAQ,CACN,CAAE,KAAM,EAAK,WAAY,KAAM,EAAO,EAAK,eAAiB,CAAC,EAAG,QAAQ,CAAE,EAC1E,CAAE,KAAM,EAAK,WAAY,KAAM,EAAO,EAAK,eAAiB,CAAC,EAAG,QAAQ,CAAE,CAC5E,EACA,OAAQ,CAAE,MAAO,SAAU,MAAO,CAAC,EAAG,CAAC,EAAG,UAAW,CAAC,EAAG,CAAC,CAAE,EAC5D,OAAQ,CAAC,EAAO,WAAY,EAAO,UAAU,EAC7C,MAAO,EAAS,CAAM,EACtB,MAAO,EAAU,EAAQ,CAAC,IAAM,EAAK,aAAa,QAAQ,KAAM,KAAK,MAAM,CAAC,CAAC,CAAC,EAC9E,OAAQ,CAAE,SAAU,MAAO,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,CAC7D,CAAC,EAGH,IAAM,EAAkB,CAAC,SAAU,SAAU,UAAW,SAAU,aAAc,WAAY,gBAAiB,QAAQ,EAIrH,SAAS,CAAc,CAAC,EAAM,EAAM,CAClC,IAAM,EAAS,OAAO,YAAY,EAAgB,IAAI,CAAC,IAAS,CAAC,EAAM,CAAC,CAAC,CAAC,CAAC,EACrE,EAAQ,CAAC,EAAO,EAAS,KAAS,CACtC,EAAG,EACH,EAAG,CAAC,EAAQ,KAAM,EAAQ,EAAE,EAC5B,SAAU,EAAQ,SAClB,OAAQ,GAAO,EAAK,SAAW,EAAK,SAAS,EAAI,SAAW,EAAI,OAAS,KACzE,MAAO,EAAM,EAAI,YAAc,IACjC,GASA,OAPC,EAAK,MAAQ,CAAC,GAAG,QAAQ,CAAC,IAAQ,CACjC,EAAI,SAAS,QAAQ,CAAC,IAAY,CAChC,GAAI,EAAO,EAAQ,QAAS,EAAO,EAAQ,QAAQ,KAAK,EAAM,EAAI,MAAO,EAAS,CAAG,CAAC,EACvF,EACF,GACA,EAAK,QAAU,CAAC,GAAG,QAAQ,CAAC,IAAY,EAAO,OAAO,KAAK,EAAM,EAAK,WAAY,EAAS,IAAI,CAAC,CAAC,EAE3F,EAAgB,OAAO,CAAC,IAAS,EAAO,GAAM,OAAS,CAAC,EAAE,IAAI,CAAC,KAAU,CAC9E,KAAO,EAAK,QAAU,EAAK,OAAO,IAAU,EAC5C,IAAK,EACL,KAAM,EAAO,EACf,EAAE,EAKJ,SAAS,CAAU,CAAC,EAAO,CACzB,IAAM,EAAM,SAAS,cAAc,KAAK,EAExC,OADA,EAAI,YAAc,GAAS,KAAO,GAAK,OAAO,CAAK,EAC5C,EAAI,UAGb,SAAS,CAAe,EAAG,cAAa,iBAAgB,KAAK,CAC3D,IAAM,EAAS,EAAE,OAAO,OAAO,GACzB,EAAI,EAAO,KAAK,GAChB,EAAQ,CAAC,GAAG,EAAW,EAAO,IAAI,MAAM,EAAW,EAAE,QAAQ,GAAG,EACtE,GAAI,EAAE,OAAQ,EAAM,KAAK,EAAW,EAAE,MAAM,CAAC,EAC7C,GAAI,EAAE,MAAO,EAAM,KAAK,EAAW,EAAE,KAAK,CAAC,EAC3C,MAAO,yCAAyC,EAAW,EAAE,CAAC,yCAAyC,EAAM,KAAK,MAAM,UAG1H,eAAsB,CAAc,CAAC,EAAM,EAAM,CAC/C,IAAM,EAAa,MAAM,EAAe,EAClC,EAAS,EAAQ,SAAS,gBAAgB,UAAU,SAAS,MAAM,CAAC,EACpE,EAAS,EAAe,EAAM,CAAI,EAClC,GAAQ,EAAK,MAAQ,CAAC,GAAG,SAAW,EAAK,QAAU,CAAC,GAAG,OAAS,EAAI,EAAI,GACxE,EAAO,EAAY,0BAA2B,WAAY,CAAM,EAEtE,EAAK,EAAY,0BAA2B,IACvC,EACH,MAAO,IACF,EAAK,MACR,OAAQ,KAAK,IAAI,IAAK,EAAO,GAAK,EAAE,EAEpC,KAAM,CAAE,QAAS,GAAM,KAAM,GAAI,EACjC,QAAS,CAAE,KAAM,GAAM,MAAO,CAAE,SAAU,GAAO,UAAW,GAAO,KAAM,GAAM,OAAQ,GAAM,QAAS,GAAM,IAAK,GAAM,MAAO,EAAK,CAAE,CACvI,EACA,SACA,OAAQ,EAAO,IAAI,CAAC,IAAM,EAAO,SAAS,EAAE,MAAQ,EAAO,KAAK,EAChE,YAAa,CAAE,IAAK,CAAE,WAAY,GAAM,kBAAmB,GAAM,UAAW,KAAM,CAAE,EACpF,MAAO,EAAS,CAAM,EACtB,MAAO,CAAE,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,CAAE,CAAE,EACpD,OAAQ,CAAE,SAAU,MAAO,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,EAC3D,QAAS,CAAE,MAAO,EAAO,KAAM,OAAQ,CAAgB,CACzD,CAAC,EAQH,eAAsB,CAAsB,CAAC,EAAM,EAAM,CACvD,IAAM,EAAa,MAAM,EAAe,EAClC,EAAS,EAAQ,SAAS,gBAAgB,UAAU,SAAS,MAAM,CAAC,EACpE,EAAa,EAAK,YAAc,CAAC,EAEvC,EAAK,EAAY,4BAA6B,IACzC,EAAY,4BAA6B,OAAQ,CAAM,EAC1D,OAAQ,CACN,CAAE,KAAM,EAAK,QAAS,KAAM,EAAO,EAAY,SAAS,CAAE,EAC1D,CAAE,KAAM,EAAK,SAAU,KAAM,EAAO,EAAY,UAAU,CAAE,EAC5D,CAAE,KAAM,EAAK,UAAW,KAAM,EAAO,EAAY,WAAW,CAAE,CAChE,EACA,OAAQ,CAAE,MAAO,SAAU,MAAO,CAAC,EAAG,EAAG,CAAC,EAAG,UAAW,CAAC,EAAG,EAAG,CAAC,CAAE,EAClE,OAAQ,CAAC,EAAO,WAAY,EAAO,OAAO,QAAS,EAAO,KAAK,EAC/D,MAAO,EAAS,CAAM,EACtB,MAAO,EAAU,EAAQ,CAAC,IAAM,KAAK,MAAM,CAAC,CAAC,EAC7C,OAAQ,CAAE,SAAU,MAAO,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,CAC7D,CAAC,EAED,IAAM,EAAY,EAAK,WAAa,CAAC,EACrC,EAAK,EAAY,0BAA2B,IACvC,EAAY,0BAA2B,MAAO,CAAM,EACvD,OAAQ,CAAC,CAAE,KAAM,EAAK,KAAM,KAAM,EAAU,IAAI,CAAC,IAAQ,EAAI,KAAK,CAAE,CAAC,EACrE,OAAQ,CAAC,EAAO,UAAU,EAC1B,YAAa,CAAE,IAAK,CAAE,YAAa,MAAO,aAAc,CAAE,CAAE,EAC5D,MAAO,CACL,WAAY,EAAU,IAAI,CAAC,IAAQ,EAAI,KAAK,EAC5C,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,CAAE,EACzC,WAAY,CAAE,MAAO,EAAO,IAAK,EACjC,UAAW,CAAE,MAAO,EAAO,IAAK,CAClC,EACA,MAAO,EAAU,EAAQ,CAAC,IAAM,KAAK,MAAM,CAAC,CAAC,CAC/C,CAAC,EAGH,IAAI,EAAgB,KAKb,SAAS,CAAmB,CAAC,EAAW,EAAM,EAAS,EAAc,CAC1E,GAAI,EAAe,EAAc,WAAW,EAE5C,EAAgB,IAAI,iBAAiB,IAAM,CACzC,IAAM,EAAO,OAAO,IAAc,WAAa,EAAU,EAAI,EAC7D,GAAI,EAAM,EAAO,EAAM,CAAI,EAC5B,EACD,EAAc,QAAQ,SAAS,gBAAiB,CAAE,WAAY,GAAM,gBAAiB,CAAC,OAAO,CAAE,CAAC","names":[]}
+{"version":3,"sources":["src/charts.js"],"sourcesContent":["// Charts module for the Pgbus Insights page, the workflow timeline and the\n// workflow insights.\n// Loaded only on those pages via an inline <script type=\"module\">, which\n// passes it the series the page embeds. ApexCharts is fetched here, on first\n// draw, so no other dashboard page downloads it (issue #517). The page renders each chart's caption,\n// data table and empty state; this module only draws into the containers that\n// exist, and redraws them when <html class=\"dark\"> toggles.\n\n// Colours for both themes in one place (issue #492). card is the chart card's\n// own background (white / gray-800): the donut's slice separators use it so\n// they read as gaps, not white lines, in dark mode.\nfunction palette(isDark) {\n  return {\n    mode: isDark ? \"dark\" : \"light\",\n    text: isDark ? \"#d1d5db\" : \"#4b5563\",\n    grid: isDark ? \"#374151\" : \"#e5e7eb\",\n    axis: isDark ? \"#4b5563\" : \"#d1d5db\",\n    card: isDark ? \"#1f2937\" : \"#ffffff\",\n    throughput: isDark ? \"#818cf8\" : \"#6366f1\",\n    latencyAvg: isDark ? \"#818cf8\" : \"#6366f1\",\n    latencyP95: isDark ? \"#fbbf24\" : \"#f59e0b\",\n    status: {\n      success: \"#10b981\",\n      failed: \"#ef4444\",\n      dead_lettered: \"#f97316\",\n    },\n    other: isDark ? \"#9ca3af\" : \"#6b7280\",\n    // The workflow timeline's segments (issue #542): the waits in neutral\n    // tones, ran coloured by how the attempt ended.\n    timeline: {\n      waited: isDark ? \"#4b5563\" : \"#d1d5db\",\n      queued: isDark ? \"#64748b\" : \"#94a3b8\",\n      waiting: isDark ? \"#fbbf24\" : \"#f59e0b\",\n      paused: isDark ? \"#fb923c\" : \"#ea580c\",\n      ran_ok: \"#10b981\",\n      ran_failed: \"#ef4444\",\n      ran_open: isDark ? \"#818cf8\" : \"#6366f1\",\n      ran_cancelled: isDark ? \"#9ca3af\" : \"#6b7280\",\n    },\n  };\n}\n\nfunction reducedMotion() {\n  return window.matchMedia(\"(prefers-reduced-motion: reduce)\").matches;\n}\n\n// Options every chart shares: theme, text, axes, grid, tooltip, animation.\nfunction baseOptions(id, type, colors) {\n  return {\n    chart: {\n      id,\n      type,\n      height: 280,\n      toolbar: { show: false },\n      background: \"transparent\",\n      foreColor: colors.text,\n      animations: { enabled: !reducedMotion() },\n    },\n    theme: { mode: colors.mode },\n    tooltip: { theme: colors.mode },\n    grid: { borderColor: colors.grid },\n    legend: { labels: { colors: colors.text } },\n    dataLabels: { enabled: false },\n  };\n}\n\nfunction timeAxis(colors) {\n  return {\n    type: \"datetime\",\n    labels: { style: { colors: colors.text } },\n    axisBorder: { color: colors.axis },\n    axisTicks: { color: colors.axis },\n    tooltip: { enabled: false },\n  };\n}\n\nfunction valueAxis(colors, formatter) {\n  return {\n    labels: { style: { colors: colors.text }, formatter },\n    axisBorder: { show: true, color: colors.axis },\n  };\n}\n\nfunction points(series, key) {\n  return series.map((p) => ({ x: new Date(p.time).getTime(), y: p[key] }));\n}\n\n// vendor/apexcharts.js is a UMD bundle: run as a module it defines the global.\n// Memoized, so the theme-toggle redraws reuse the one download. Every caller\n// awaits the same promise, and its callbacks run in call order, so a redraw\n// can't overtake the first render. A failed import clears the memo, so the\n// next draw (a theme toggle) retries instead of awaiting the same rejection.\nlet apexCharts = null;\nfunction loadApexCharts() {\n  apexCharts ||= import(\"apexcharts\")\n    .then(() => globalThis.ApexCharts)\n    .catch((error) => {\n      apexCharts = null;\n      throw error;\n    });\n  return apexCharts;\n}\n\nconst charts = {};\n\nfunction draw(ApexCharts, id, options) {\n  if (charts[id]) charts[id].destroy();\n  delete charts[id];\n\n  const el = document.getElementById(id);\n  if (!el) return;\n\n  charts[id] = new ApexCharts(el, options);\n  charts[id].render();\n}\n\nexport async function renderCharts(data, i18n) {\n  const ApexCharts = await loadApexCharts();\n  const colors = palette(document.documentElement.classList.contains(\"dark\"));\n\n  draw(ApexCharts, \"throughput-chart\", {\n    ...baseOptions(\"throughput-chart\", \"area\", colors),\n    series: [{ name: i18n.seriesName, data: points(data.throughput || [], \"count\") }],\n    stroke: { curve: \"smooth\", width: 2 },\n    fill: { type: \"gradient\", gradient: { shadeIntensity: 1, opacityFrom: 0.4, opacityTo: 0.05, stops: [0, 100] } },\n    colors: [colors.throughput],\n    xaxis: timeAxis(colors),\n    yaxis: valueAxis(colors, (v) => Math.round(v)),\n  });\n\n  const statuses = Object.keys(data.status_counts || {});\n  draw(ApexCharts, \"status-chart\", {\n    ...baseOptions(\"status-chart\", \"donut\", colors),\n    series: statuses.map((s) => data.status_counts[s]),\n    labels: statuses.map((s) => (i18n.statuses && i18n.statuses[s]) || s),\n    colors: statuses.map((s) => colors.status[s] || colors.other),\n    stroke: { colors: [colors.card] },\n    legend: { position: \"bottom\", labels: { colors: colors.text } },\n    plotOptions: {\n      pie: {\n        donut: {\n          size: \"60%\",\n          labels: {\n            show: true,\n            name: { color: colors.text },\n            value: { color: colors.text },\n            total: { show: true, color: colors.text },\n          },\n        },\n      },\n    },\n  });\n\n  draw(ApexCharts, \"latency-chart\", {\n    ...baseOptions(\"latency-chart\", \"line\", colors),\n    series: [\n      { name: i18n.latencyAvg, data: points(data.latency_trend || [], \"avg_ms\") },\n      { name: i18n.latencyP95, data: points(data.latency_trend || [], \"p95_ms\") },\n    ],\n    stroke: { curve: \"smooth\", width: [2, 2], dashArray: [0, 5] },\n    colors: [colors.latencyAvg, colors.latencyP95],\n    xaxis: timeAxis(colors),\n    yaxis: valueAxis(colors, (v) => i18n.milliseconds.replace(\"%n\", Math.round(v))),\n    legend: { position: \"top\", labels: { colors: colors.text } },\n  });\n}\n\nconst TIMELINE_SERIES = [\"waited\", \"queued\", \"waiting\", \"ran_ok\", \"ran_failed\", \"ran_open\", \"ran_cancelled\", \"paused\"];\n\n// One range bar per segment, grouped by series so each kind keeps its colour.\n// Rows are the attempts (oldest first) plus a \"Paused\" row for the pauses.\nfunction timelineSeries(data, i18n) {\n  const points = Object.fromEntries(TIMELINE_SERIES.map((name) => [name, []]));\n  const point = (label, segment, row) => ({\n    x: label,\n    y: [segment.from, segment.to],\n    duration: segment.duration,\n    status: row && i18n.statuses ? i18n.statuses[row.status] || row.status : null,\n    error: row ? row.error_class : null,\n  });\n\n  (data.rows || []).forEach((row) => {\n    row.segments.forEach((segment) => {\n      if (points[segment.series]) points[segment.series].push(point(row.label, segment, row));\n    });\n  });\n  (data.paused || []).forEach((segment) => points.paused.push(point(i18n.paused_row, segment, null)));\n\n  return TIMELINE_SERIES.filter((name) => points[name].length > 0).map((name) => ({\n    name: (i18n.series && i18n.series[name]) || name,\n    key: name,\n    data: points[name],\n  }));\n}\n\n// The tooltip: the segment's kind and duration, then the attempt's status and\n// error class. Text only (textContent-safe): every value is escaped.\nfunction escapeText(value) {\n  const div = document.createElement(\"div\");\n  div.textContent = value == null ? \"\" : String(value);\n  return div.innerHTML;\n}\n\nfunction timelineTooltip({ seriesIndex, dataPointIndex, w }) {\n  const series = w.config.series[seriesIndex];\n  const p = series.data[dataPointIndex];\n  const lines = [`${escapeText(series.name)}: ${escapeText(p.duration)}`];\n  if (p.status) lines.push(escapeText(p.status));\n  if (p.error) lines.push(escapeText(p.error));\n  return `<div class=\"apexcharts-tooltip-title\">${escapeText(p.x)}</div><div style=\"padding: 4px 10px\">${lines.join(\"<br>\")}</div>`;\n}\n\nexport async function renderTimeline(data, i18n) {\n  const ApexCharts = await loadApexCharts();\n  const colors = palette(document.documentElement.classList.contains(\"dark\"));\n  const series = timelineSeries(data, i18n);\n  const rows = (data.rows || []).length + ((data.paused || []).length > 0 ? 1 : 0);\n  const base = baseOptions(\"workflow-timeline-chart\", \"rangeBar\", colors);\n\n  draw(ApexCharts, \"workflow-timeline-chart\", {\n    ...base,\n    chart: {\n      ...base.chart,\n      height: Math.max(160, rows * 36 + 90),\n      // Waits can span days: zoom and pan along time, reset to see it all.\n      zoom: { enabled: true, type: \"x\" },\n      toolbar: { show: true, tools: { download: false, selection: false, zoom: true, zoomin: true, zoomout: true, pan: true, reset: true } },\n    },\n    series,\n    colors: series.map((s) => colors.timeline[s.key] || colors.other),\n    plotOptions: { bar: { horizontal: true, rangeBarGroupRows: true, barHeight: \"60%\" } },\n    xaxis: timeAxis(colors),\n    yaxis: { labels: { style: { colors: colors.text } } },\n    legend: { position: \"top\", labels: { colors: colors.text } },\n    tooltip: { theme: colors.mode, custom: timelineTooltip },\n    annotations: { xaxis: waitTimeoutMarks(data, i18n, colors) },\n  });\n}\n\n// A dashed line where a wait passed its deadline (issue #565): the waiting\n// colour, labelled in the page's text colour on the card, so it reads in\n// both themes. The page lists the same moments as text.\nfunction waitTimeoutMarks(data, i18n, colors) {\n  return (data.wait_timeouts || []).map((mark) => ({\n    x: mark.at,\n    borderColor: colors.timeline.waiting,\n    strokeDashArray: 4,\n    label: {\n      text: i18n.wait_timed_out,\n      orientation: \"horizontal\",\n      borderColor: colors.timeline.waiting,\n      style: { color: colors.text, background: colors.card },\n    },\n  }));\n}\n\n// The workflow insights (issue #549): runs started, finished and cancelled\n// per bucket (cancelled in the neutral grey the dashboard uses for it, not\n// the failure red), and the end-to-end duration histogram of the finished\n// runs. The bucket times arrive as the dashboard zone's wall clock written\n// as UTC, which the UTC time axis prints as is.\nexport async function renderWorkflowInsights(data, i18n) {\n  const ApexCharts = await loadApexCharts();\n  const colors = palette(document.documentElement.classList.contains(\"dark\"));\n  const throughput = data.throughput || [];\n\n  draw(ApexCharts, \"workflow-throughput-chart\", {\n    ...baseOptions(\"workflow-throughput-chart\", \"line\", colors),\n    series: [\n      { name: i18n.started, data: points(throughput, \"started\") },\n      { name: i18n.finished, data: points(throughput, \"finished\") },\n      { name: i18n.cancelled, data: points(throughput, \"cancelled\") },\n    ],\n    stroke: { curve: \"smooth\", width: [2, 2, 2], dashArray: [0, 0, 5] },\n    colors: [colors.throughput, colors.status.success, colors.other],\n    xaxis: timeAxis(colors),\n    yaxis: valueAxis(colors, (v) => Math.round(v)),\n    legend: { position: \"top\", labels: { colors: colors.text } },\n  });\n\n  const durations = data.durations || [];\n  draw(ApexCharts, \"workflow-duration-chart\", {\n    ...baseOptions(\"workflow-duration-chart\", \"bar\", colors),\n    series: [{ name: i18n.runs, data: durations.map((bin) => bin.count) }],\n    colors: [colors.throughput],\n    plotOptions: { bar: { columnWidth: \"70%\", borderRadius: 2 } },\n    xaxis: {\n      categories: durations.map((bin) => bin.label),\n      labels: { style: { colors: colors.text } },\n      axisBorder: { color: colors.axis },\n      axisTicks: { color: colors.axis },\n    },\n    yaxis: valueAxis(colors, (v) => Math.round(v)),\n  });\n}\n\nlet themeObserver = null;\n\n// The page toggles dark mode with a class on <html>; redraw in the new palette.\n// render is the page's draw function (renderCharts on Insights, renderTimeline\n// on a workflow, renderWorkflowInsights on a workflow class).\nexport function observeThemeChanges(getDataFn, i18n, render = renderCharts) {\n  if (themeObserver) themeObserver.disconnect();\n\n  themeObserver = new MutationObserver(() => {\n    const data = typeof getDataFn === \"function\" ? getDataFn() : getDataFn;\n    if (data) render(data, i18n);\n  });\n  themeObserver.observe(document.documentElement, { attributes: true, attributeFilter: [\"class\"] });\n}\n"],"mappings":"AAWA,SAAS,CAAO,CAAC,EAAQ,CACvB,MAAO,CACL,KAAM,EAAS,OAAS,QACxB,KAAM,EAAS,UAAY,UAC3B,KAAM,EAAS,UAAY,UAC3B,KAAM,EAAS,UAAY,UAC3B,KAAM,EAAS,UAAY,UAC3B,WAAY,EAAS,UAAY,UACjC,WAAY,EAAS,UAAY,UACjC,WAAY,EAAS,UAAY,UACjC,OAAQ,CACN,QAAS,UACT,OAAQ,UACR,cAAe,SACjB,EACA,MAAO,EAAS,UAAY,UAG5B,SAAU,CACR,OAAQ,EAAS,UAAY,UAC7B,OAAQ,EAAS,UAAY,UAC7B,QAAS,EAAS,UAAY,UAC9B,OAAQ,EAAS,UAAY,UAC7B,OAAQ,UACR,WAAY,UACZ,SAAU,EAAS,UAAY,UAC/B,cAAe,EAAS,UAAY,SACtC,CACF,EAGF,SAAS,CAAa,EAAG,CACvB,OAAO,OAAO,WAAW,kCAAkC,EAAE,QAI/D,SAAS,CAAW,CAAC,EAAI,EAAM,EAAQ,CACrC,MAAO,CACL,MAAO,CACL,KACA,OACA,OAAQ,IACR,QAAS,CAAE,KAAM,EAAM,EACvB,WAAY,cACZ,UAAW,EAAO,KAClB,WAAY,CAAE,QAAS,CAAC,EAAc,CAAE,CAC1C,EACA,MAAO,CAAE,KAAM,EAAO,IAAK,EAC3B,QAAS,CAAE,MAAO,EAAO,IAAK,EAC9B,KAAM,CAAE,YAAa,EAAO,IAAK,EACjC,OAAQ,CAAE,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,EAC1C,WAAY,CAAE,QAAS,EAAM,CAC/B,EAGF,SAAS,CAAQ,CAAC,EAAQ,CACxB,MAAO,CACL,KAAM,WACN,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,CAAE,EACzC,WAAY,CAAE,MAAO,EAAO,IAAK,EACjC,UAAW,CAAE,MAAO,EAAO,IAAK,EAChC,QAAS,CAAE,QAAS,EAAM,CAC5B,EAGF,SAAS,CAAS,CAAC,EAAQ,EAAW,CACpC,MAAO,CACL,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,EAAG,WAAU,EACpD,WAAY,CAAE,KAAM,GAAM,MAAO,EAAO,IAAK,CAC/C,EAGF,SAAS,CAAM,CAAC,EAAQ,EAAK,CAC3B,OAAO,EAAO,IAAI,CAAC,KAAO,CAAE,EAAG,IAAI,KAAK,EAAE,IAAI,EAAE,QAAQ,EAAG,EAAG,EAAE,EAAK,EAAE,EAQzE,IAAI,EAAa,KACjB,SAAS,CAAc,EAAG,CAOxB,OANA,IAAsB,qBACnB,KAAK,IAAM,WAAW,UAAU,EAChC,MAAM,CAAC,IAAU,CAEhB,MADA,EAAa,KACP,EACP,EACI,EAGT,IAAM,EAAS,CAAC,EAEhB,SAAS,CAAI,CAAC,EAAY,EAAI,EAAS,CACrC,GAAI,EAAO,GAAK,EAAO,GAAI,QAAQ,EACnC,OAAO,EAAO,GAEd,IAAM,EAAK,SAAS,eAAe,CAAE,EACrC,GAAI,CAAC,EAAI,OAET,EAAO,GAAM,IAAI,EAAW,EAAI,CAAO,EACvC,EAAO,GAAI,OAAO,EAGpB,eAAsB,CAAY,CAAC,EAAM,EAAM,CAC7C,IAAM,EAAa,MAAM,EAAe,EAClC,EAAS,EAAQ,SAAS,gBAAgB,UAAU,SAAS,MAAM,CAAC,EAE1E,EAAK,EAAY,mBAAoB,IAChC,EAAY,mBAAoB,OAAQ,CAAM,EACjD,OAAQ,CAAC,CAAE,KAAM,EAAK,WAAY,KAAM,EAAO,EAAK,YAAc,CAAC,EAAG,OAAO,CAAE,CAAC,EAChF,OAAQ,CAAE,MAAO,SAAU,MAAO,CAAE,EACpC,KAAM,CAAE,KAAM,WAAY,SAAU,CAAE,eAAgB,EAAG,YAAa,IAAK,UAAW,KAAM,MAAO,CAAC,EAAG,GAAG,CAAE,CAAE,EAC9G,OAAQ,CAAC,EAAO,UAAU,EAC1B,MAAO,EAAS,CAAM,EACtB,MAAO,EAAU,EAAQ,CAAC,IAAM,KAAK,MAAM,CAAC,CAAC,CAC/C,CAAC,EAED,IAAM,EAAW,OAAO,KAAK,EAAK,eAAiB,CAAC,CAAC,EACrD,EAAK,EAAY,eAAgB,IAC5B,EAAY,eAAgB,QAAS,CAAM,EAC9C,OAAQ,EAAS,IAAI,CAAC,IAAM,EAAK,cAAc,EAAE,EACjD,OAAQ,EAAS,IAAI,CAAC,IAAO,EAAK,UAAY,EAAK,SAAS,IAAO,CAAC,EACpE,OAAQ,EAAS,IAAI,CAAC,IAAM,EAAO,OAAO,IAAM,EAAO,KAAK,EAC5D,OAAQ,CAAE,OAAQ,CAAC,EAAO,IAAI,CAAE,EAChC,OAAQ,CAAE,SAAU,SAAU,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,EAC9D,YAAa,CACX,IAAK,CACH,MAAO,CACL,KAAM,MACN,OAAQ,CACN,KAAM,GACN,KAAM,CAAE,MAAO,EAAO,IAAK,EAC3B,MAAO,CAAE,MAAO,EAAO,IAAK,EAC5B,MAAO,CAAE,KAAM,GAAM,MAAO,EAAO,IAAK,CAC1C,CACF,CACF,CACF,CACF,CAAC,EAED,EAAK,EAAY,gBAAiB,IAC7B,EAAY,gBAAiB,OAAQ,CAAM,EAC9C,OAAQ,CACN,CAAE,KAAM,EAAK,WAAY,KAAM,EAAO,EAAK,eAAiB,CAAC,EAAG,QAAQ,CAAE,EAC1E,CAAE,KAAM,EAAK,WAAY,KAAM,EAAO,EAAK,eAAiB,CAAC,EAAG,QAAQ,CAAE,CAC5E,EACA,OAAQ,CAAE,MAAO,SAAU,MAAO,CAAC,EAAG,CAAC,EAAG,UAAW,CAAC,EAAG,CAAC,CAAE,EAC5D,OAAQ,CAAC,EAAO,WAAY,EAAO,UAAU,EAC7C,MAAO,EAAS,CAAM,EACtB,MAAO,EAAU,EAAQ,CAAC,IAAM,EAAK,aAAa,QAAQ,KAAM,KAAK,MAAM,CAAC,CAAC,CAAC,EAC9E,OAAQ,CAAE,SAAU,MAAO,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,CAC7D,CAAC,EAGH,IAAM,EAAkB,CAAC,SAAU,SAAU,UAAW,SAAU,aAAc,WAAY,gBAAiB,QAAQ,EAIrH,SAAS,CAAc,CAAC,EAAM,EAAM,CAClC,IAAM,EAAS,OAAO,YAAY,EAAgB,IAAI,CAAC,IAAS,CAAC,EAAM,CAAC,CAAC,CAAC,CAAC,EACrE,EAAQ,CAAC,EAAO,EAAS,KAAS,CACtC,EAAG,EACH,EAAG,CAAC,EAAQ,KAAM,EAAQ,EAAE,EAC5B,SAAU,EAAQ,SAClB,OAAQ,GAAO,EAAK,SAAW,EAAK,SAAS,EAAI,SAAW,EAAI,OAAS,KACzE,MAAO,EAAM,EAAI,YAAc,IACjC,GASA,OAPC,EAAK,MAAQ,CAAC,GAAG,QAAQ,CAAC,IAAQ,CACjC,EAAI,SAAS,QAAQ,CAAC,IAAY,CAChC,GAAI,EAAO,EAAQ,QAAS,EAAO,EAAQ,QAAQ,KAAK,EAAM,EAAI,MAAO,EAAS,CAAG,CAAC,EACvF,EACF,GACA,EAAK,QAAU,CAAC,GAAG,QAAQ,CAAC,IAAY,EAAO,OAAO,KAAK,EAAM,EAAK,WAAY,EAAS,IAAI,CAAC,CAAC,EAE3F,EAAgB,OAAO,CAAC,IAAS,EAAO,GAAM,OAAS,CAAC,EAAE,IAAI,CAAC,KAAU,CAC9E,KAAO,EAAK,QAAU,EAAK,OAAO,IAAU,EAC5C,IAAK,EACL,KAAM,EAAO,EACf,EAAE,EAKJ,SAAS,CAAU,CAAC,EAAO,CACzB,IAAM,EAAM,SAAS,cAAc,KAAK,EAExC,OADA,EAAI,YAAc,GAAS,KAAO,GAAK,OAAO,CAAK,EAC5C,EAAI,UAGb,SAAS,CAAe,EAAG,cAAa,iBAAgB,KAAK,CAC3D,IAAM,EAAS,EAAE,OAAO,OAAO,GACzB,EAAI,EAAO,KAAK,GAChB,EAAQ,CAAC,GAAG,EAAW,EAAO,IAAI,MAAM,EAAW,EAAE,QAAQ,GAAG,EACtE,GAAI,EAAE,OAAQ,EAAM,KAAK,EAAW,EAAE,MAAM,CAAC,EAC7C,GAAI,EAAE,MAAO,EAAM,KAAK,EAAW,EAAE,KAAK,CAAC,EAC3C,MAAO,yCAAyC,EAAW,EAAE,CAAC,yCAAyC,EAAM,KAAK,MAAM,UAG1H,eAAsB,CAAc,CAAC,EAAM,EAAM,CAC/C,IAAM,EAAa,MAAM,EAAe,EAClC,EAAS,EAAQ,SAAS,gBAAgB,UAAU,SAAS,MAAM,CAAC,EACpE,EAAS,EAAe,EAAM,CAAI,EAClC,GAAQ,EAAK,MAAQ,CAAC,GAAG,SAAW,EAAK,QAAU,CAAC,GAAG,OAAS,EAAI,EAAI,GACxE,EAAO,EAAY,0BAA2B,WAAY,CAAM,EAEtE,EAAK,EAAY,0BAA2B,IACvC,EACH,MAAO,IACF,EAAK,MACR,OAAQ,KAAK,IAAI,IAAK,EAAO,GAAK,EAAE,EAEpC,KAAM,CAAE,QAAS,GAAM,KAAM,GAAI,EACjC,QAAS,CAAE,KAAM,GAAM,MAAO,CAAE,SAAU,GAAO,UAAW,GAAO,KAAM,GAAM,OAAQ,GAAM,QAAS,GAAM,IAAK,GAAM,MAAO,EAAK,CAAE,CACvI,EACA,SACA,OAAQ,EAAO,IAAI,CAAC,IAAM,EAAO,SAAS,EAAE,MAAQ,EAAO,KAAK,EAChE,YAAa,CAAE,IAAK,CAAE,WAAY,GAAM,kBAAmB,GAAM,UAAW,KAAM,CAAE,EACpF,MAAO,EAAS,CAAM,EACtB,MAAO,CAAE,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,CAAE,CAAE,EACpD,OAAQ,CAAE,SAAU,MAAO,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,EAC3D,QAAS,CAAE,MAAO,EAAO,KAAM,OAAQ,CAAgB,EACvD,YAAa,CAAE,MAAO,EAAiB,EAAM,EAAM,CAAM,CAAE,CAC7D,CAAC,EAMH,SAAS,CAAgB,CAAC,EAAM,EAAM,EAAQ,CAC5C,OAAQ,EAAK,eAAiB,CAAC,GAAG,IAAI,CAAC,KAAU,CAC/C,EAAG,EAAK,GACR,YAAa,EAAO,SAAS,QAC7B,gBAAiB,EACjB,MAAO,CACL,KAAM,EAAK,eACX,YAAa,aACb,YAAa,EAAO,SAAS,QAC7B,MAAO,CAAE,MAAO,EAAO,KAAM,WAAY,EAAO,IAAK,CACvD,CACF,EAAE,EAQJ,eAAsB,CAAsB,CAAC,EAAM,EAAM,CACvD,IAAM,EAAa,MAAM,EAAe,EAClC,EAAS,EAAQ,SAAS,gBAAgB,UAAU,SAAS,MAAM,CAAC,EACpE,EAAa,EAAK,YAAc,CAAC,EAEvC,EAAK,EAAY,4BAA6B,IACzC,EAAY,4BAA6B,OAAQ,CAAM,EAC1D,OAAQ,CACN,CAAE,KAAM,EAAK,QAAS,KAAM,EAAO,EAAY,SAAS,CAAE,EAC1D,CAAE,KAAM,EAAK,SAAU,KAAM,EAAO,EAAY,UAAU,CAAE,EAC5D,CAAE,KAAM,EAAK,UAAW,KAAM,EAAO,EAAY,WAAW,CAAE,CAChE,EACA,OAAQ,CAAE,MAAO,SAAU,MAAO,CAAC,EAAG,EAAG,CAAC,EAAG,UAAW,CAAC,EAAG,EAAG,CAAC,CAAE,EAClE,OAAQ,CAAC,EAAO,WAAY,EAAO,OAAO,QAAS,EAAO,KAAK,EAC/D,MAAO,EAAS,CAAM,EACtB,MAAO,EAAU,EAAQ,CAAC,IAAM,KAAK,MAAM,CAAC,CAAC,EAC7C,OAAQ,CAAE,SAAU,MAAO,OAAQ,CAAE,OAAQ,EAAO,IAAK,CAAE,CAC7D,CAAC,EAED,IAAM,EAAY,EAAK,WAAa,CAAC,EACrC,EAAK,EAAY,0BAA2B,IACvC,EAAY,0BAA2B,MAAO,CAAM,EACvD,OAAQ,CAAC,CAAE,KAAM,EAAK,KAAM,KAAM,EAAU,IAAI,CAAC,IAAQ,EAAI,KAAK,CAAE,CAAC,EACrE,OAAQ,CAAC,EAAO,UAAU,EAC1B,YAAa,CAAE,IAAK,CAAE,YAAa,MAAO,aAAc,CAAE,CAAE,EAC5D,MAAO,CACL,WAAY,EAAU,IAAI,CAAC,IAAQ,EAAI,KAAK,EAC5C,OAAQ,CAAE,MAAO,CAAE,OAAQ,EAAO,IAAK,CAAE,EACzC,WAAY,CAAE,MAAO,EAAO,IAAK,EACjC,UAAW,CAAE,MAAO,EAAO,IAAK,CAClC,EACA,MAAO,EAAU,EAAQ,CAAC,IAAM,KAAK,MAAM,CAAC,CAAC,CAC/C,CAAC,EAGH,IAAI,EAAgB,KAKb,SAAS,CAAmB,CAAC,EAAW,EAAM,EAAS,EAAc,CAC1E,GAAI,EAAe,EAAc,WAAW,EAE5C,EAAgB,IAAI,iBAAiB,IAAM,CACzC,IAAM,EAAO,OAAO,IAAc,WAAa,EAAU,EAAI,EAC7D,GAAI,EAAM,EAAO,EAAM,CAAI,EAC5B,EACD,EAAc,QAAQ,SAAS,gBAAiB,CAAE,WAAY,GAAM,gBAAiB,CAAC,OAAO,CAAE,CAAC","names":[]}

data/app/helpers/pgbus/workflows_helper.rb

@@ -73,7 +73,7 @@ module Pgbus
         case key
         when :step, :signal then [key, tag.code(value, class: "font-mono")]
         when :batch then [key, pgbus_workflow_batch_link(value, row[:batch_id])]
-        when :ago then [key, pgbus_time(value)]
+        when :ago, :late then [key, pgbus_time(value)]
         when :time then [key, pgbus_time(value, clock: true)]
         else [key, value]
         end
@@ -158,7 +158,7 @@ module Pgbus
     def pgbus_workflow_chart_i18n
       series = %w[waited queued waiting paused ran_ok ran_failed ran_open ran_cancelled]
       { series: series.index_with { |name| t("pgbus.workflows.timeline.series.#{name}") },
-        paused_row: t("pgbus.workflows.timeline.paused_row"),
+        paused_row: t("pgbus.workflows.timeline.paused_row"), wait_timed_out: t("pgbus.workflows.events.wait_timed_out"),
         statuses: Workflow::States::STEP.index_with { |status| t("pgbus.workflows.step_statuses.#{status}") } }
     end
   end

data/app/models/pgbus/blocked_execution.rb

@@ -62,5 +62,17 @@ module Pgbus
         .limit(limit)
         .pluck(:concurrency_key)
     end
+
+    # The uniqueness keys, out of +lock_keys+, that a parked job carries. Its
+    # :until_executed row stays unbound until it runs and it sits in no queue,
+    # so the uniqueness reaper asks here before calling the row an orphan. A
+    # row parked before the double-encoding fix is read through its string,
+    # since the reaper can run before the promote sweep repairs it.
+    def self.parked_uniqueness_keys(lock_keys)
+      key = Uniqueness::METADATA_KEY
+      key_path = "CASE WHEN jsonb_typeof(payload) = 'string' " \
+                 "THEN (payload #>> '{}')::jsonb ->> '#{key}' ELSE payload ->> '#{key}' END"
+      where("#{key_path} IN (?)", lock_keys).distinct.pluck(Arel.sql(key_path)).to_set
+    end
   end
 end

data/app/models/pgbus/uniqueness_key.rb

@@ -40,10 +40,11 @@ module Pgbus
     # Does not touch created_at — the reaper's age floor is from acquire time.
     # Restricted to this enqueue's unbound row (msg_id=0, matching created_at
     # when acquire! stamped one) so a completed job's bind cannot retarget a
-    # successor that re-acquired the key.
-    def self.bind!(lock_key, queue_name:, msg_id:)
+    # successor that re-acquired the key. A retry passes the stamp unbind!
+    # returned as +acquired_at+.
+    def self.bind!(lock_key, queue_name:, msg_id:, acquired_at: nil)
       stamps = Thread.current[:pgbus_uniqueness_created_at]
-      created_at = stamps&.delete(lock_key)
+      created_at = acquired_at || stamps&.delete(lock_key)
       sql = "UPDATE #{table_name} SET queue_name = $2, msg_id = $3 " \
             "WHERE lock_key = $1 AND msg_id = 0"
       binds = [lock_key, queue_name, msg_id]
@@ -86,6 +87,90 @@ module Pgbus
       )
     end
 
+    # The row a message holds: bound to that message (queue + msg_id, see
+    # release_if_bound!), or still unbound — a parked job's row, or one whose
+    # bind failed — and acquired no later than the message was sent ($4). A
+    # lock is always taken before its message exists, so an unbound row
+    # acquired after the send is a successor's. created_at is a timestamp
+    # without time zone written in the session's zone, and the cast compares
+    # it in that same zone; PGMQ's enqueued_at string carries its offset.
+    # Unbound means msg_id <= 0 only: "pending" (the placeholder queue) is a
+    # name a real queue can have too, and a lock bound to one of its messages
+    # is bound.
+    HELD_BY_CONDITION = "lock_key = $1 AND ((queue_name = $2 AND msg_id = $3 AND msg_id > 0) OR " \
+                        "(msg_id <= 0 AND created_at <= $4::timestamptz))"
+    private_constant :HELD_BY_CONDITION
+
+    # Release the lock a finishing message holds (success or DLQ). Unlike the
+    # key-only release!, it leaves a row that has moved on: handed to the
+    # message's retry (unbind!), or taken by a successor after this
+    # message's own row went. +enqueued_at+ is when that message was sent;
+    # without it only the row bound to the message goes (an unbound row could
+    # be a retry's it handed over), and the reaper takes an orphan.
+    def self.release_held_by!(lock_key, queue_name:, msg_id:, enqueued_at:)
+      Thread.current[:pgbus_uniqueness_created_at]&.delete(lock_key)
+      connection.exec_delete(
+        "DELETE FROM #{table_name} WHERE #{HELD_BY_CONDITION}",
+        "UniquenessKey Release Held By", held_by_binds(lock_key, queue_name, msg_id, sent_at_literal(enqueued_at))
+      )
+    end
+
+    # Hand the running attempt's lock to its retry, which goes out through the
+    # key the attempt still holds (issue #333) and never acquires one. Called
+    # before the retry exists, as a first enqueue acquires before its send:
+    # the row is left unbound on the retry's queue and stamped now, after the
+    # attempt was sent, so the attempt's release no longer counts it as its
+    # own while the retry's (sent later) does, even if the retry finishes
+    # before bind! points the row at it. Returns the new stamp for that
+    # bind!, or nil when the attempt no longer holds the key. Without the
+    # attempt's enqueued_at only a row bound to it is taken.
+    def self.unbind!(lock_key, from_queue_name:, from_msg_id:, from_enqueued_at:, queue_name:)
+      binds = held_by_binds(lock_key, from_queue_name, from_msg_id, sent_at_literal(from_enqueued_at))
+      result = connection.exec_query(
+        "UPDATE #{table_name} SET queue_name = $5, msg_id = 0, created_at = CURRENT_TIMESTAMP " \
+        "WHERE #{HELD_BY_CONDITION} RETURNING created_at",
+        "UniquenessKey Unbind", binds + [queue_name.to_s]
+      )
+      result.rows.first&.first
+    end
+
+    # release_held_by! for many messages at once — the dashboard's bulk
+    # discards and queue purges — as [lock_key, queue_name, msg_id,
+    # enqueued_at] tuples, one statement per thousand.
+    def self.release_all_held_by!(holders)
+      holders.each_slice(1000).sum do |slice|
+        values = slice.map do |lock_key, queue_name, msg_id, enqueued_at|
+          "(#{connection.quote(lock_key.to_s)}, #{connection.quote(queue_name.to_s)}, #{msg_id.to_i}, " \
+            "#{connection.quote(sent_at_literal(enqueued_at))}::timestamptz)"
+        end
+        connection.exec_delete(
+          "DELETE FROM #{table_name} AS u USING (VALUES #{values.join(", ")}) AS m(lock_key, queue_name, msg_id, sent_at) " \
+          "WHERE u.lock_key = m.lock_key AND ((u.queue_name = m.queue_name AND u.msg_id = m.msg_id AND u.msg_id > 0) OR " \
+          "(u.msg_id <= 0 AND u.created_at <= m.sent_at))",
+          "UniquenessKey Release All Held By"
+        )
+      end
+    end
+
+    def self.held_by_binds(lock_key, queue_name, msg_id, sent_at)
+      [lock_key, queue_name.to_s, msg_id.to_i, sent_at]
+    end
+    private_class_method :held_by_binds
+
+    # A message's enqueued_at as a timestamptz literal that carries its own
+    # offset. PGMQ hands over a String that already does; a Time (the
+    # dashboard reads it through ActiveRecord) or epoch seconds would
+    # otherwise be bound as zone-less digits and read in the session's zone.
+    # An unknown send time is -infinity: no unbound row counts as the
+    # message's own.
+    def self.sent_at_literal(enqueued_at)
+      sent_at = enqueued_at.is_a?(Numeric) ? Time.at(enqueued_at) : enqueued_at
+      return "-infinity" if sent_at.nil?
+
+      sent_at.respond_to?(:iso8601) ? sent_at.iso8601(6) : sent_at.to_s
+    end
+    private_class_method :sent_at_literal
+
     # Release a lock only while it is still UNBOUND (msg_id = 0) AND was
     # acquired no later than +acquired_before+. Used when discarding a parked
     # job: a parked job never got a msg_id, so its lock was never bound, and

data/app/models/pgbus/workflow_step.rb

@@ -55,6 +55,13 @@ module Pgbus
       where(id: id, status: "waiting").update_all(status: "in_progress")
     end
 
+    # A waiting row's next deadline (on_timeout: :notify keeps the attempt
+    # waiting). Written without the model callback, so no StepJob goes out.
+    # Returns the number of rows moved.
+    def self.move_deadline!(id, timeout_at:)
+      where(id: id, status: "waiting").update_all(timeout_at: timeout_at)
+    end
+
     # waiting rows of waiting runs past their timeout_at (the sweep applies
     # their on_timeout).
     def self.wait_expired(now: Time.current)

data/app/views/pgbus/workflows/_timeline.html.erb

@@ -1,6 +1,7 @@
 <%# The timeline (issue #542): one range-bar row per attempt, split into
     waited, queued and ran (ran coloured by how the attempt ended), with the
-    waits and pauses as their own segments. charts.js renderTimeline draws
+    waits and pauses as their own segments and a dashed line per missed wait
+    deadline. charts.js renderTimeline draws
     it from the payload embedded once below; the attempts table under it is
     the full text alternative. %>
 <% timeline = @workflow[:timeline] %>
@@ -24,6 +25,15 @@
         <% end %>
       </ul>
     <% end %>
+    <%# The missed wait deadlines as text: the chart's dashed lines (issue #565). %>
+    <% if timeline[:wait_timeouts].any? %>
+      <ul class="mt-2 space-y-1 text-sm text-gray-700 dark:text-gray-300" data-testid="workflow-wait-timeouts">
+        <% timeline[:wait_timeouts].each do |mark| %>
+          <li><%= t("pgbus.workflows.timeline.wait_timeout_html", step: tag.code(mark[:step], class: "font-mono"),
+                    time: pgbus_absolute_time(Time.at(mark[:at] / 1000.0)), action: tag.code(mark[:action], class: "font-mono")) %></li>
+        <% end %>
+      </ul>
+    <% end %>
     <%# json_escape keeps a value from closing the tag; a data block needs no nonce. %>
     <%= tag.script(json_escape(timeline.to_json).html_safe, type: "application/json", id: "pgbus-workflow-timeline-data") %> <%# herb:disable erb-no-unsafe-raw %>
     <script type="module" nonce="<%= content_security_policy_nonce %>">

data/config/locales/da.yml

@@ -1349,6 +1349,8 @@ da:
         unknown_html: 'Status: %{status}.'
         waiting_batch_html: Venter på batch %{batch} (%{done} af %{total} jobs færdige).
         waiting_signal_html: Venter på signalet %{signal} siden %{ago}.
+        waiting_signal_late_html: Venter på signalet %{signal} siden %{ago} (forsinket siden %{late}). Næste forsinkelsesbesked sendes %{time}.
+        waiting_signal_notify_html: Venter på signalet %{signal} siden %{ago}. En forsinkelsesbesked sendes %{time}, og ventetiden fortsætter.
         waiting_signal_timeout_html: Venter på signalet %{signal} siden %{ago}. Ventetiden udløber %{time}.
         waiting_step_html: Venter på %{step}.
       not_found:
@@ -1451,7 +1453,7 @@ da:
         ready: Klar
         waiting: Venter
       timeline:
-        caption: 'Hvert forsøg, ældste øverst: ventede (dets ventetid), i kø (forfalden, endnu ikke hentet), kørte (farvet efter udfald) og venter (på et signal eller sin batch), med pauserne i en egen række. Forsøgstabellen nedenfor viser de samme varigheder.'
+        caption: 'Hvert forsøg, ældste øverst: ventede (dets ventetid), i kø (forfalden, endnu ikke hentet), kørte (farvet efter udfald) og venter (på et signal eller sin batch), med pauserne i en egen række. Forsøgstabellen nedenfor viser de samme varigheder. En stiplet linje markerer hver gang en ventetid passerede sin frist.'
         empty: Ingen forsøg endnu.
         paused_period_html: Sat på pause %{from} i %{duration}
         paused_row: På pause
@@ -1465,3 +1467,4 @@ da:
           waited: Ventede
           waiting: Venter
         title: Tidslinje
+        wait_timeout_html: 'Ventetiden for %{step} passerede sin frist %{time} (on_timeout: %{action}).'

data/config/locales/de.yml

@@ -1349,6 +1349,8 @@ de:
         unknown_html: 'Status: %{status}.'
         waiting_batch_html: Wartet auf Stapel %{batch} (%{done} von %{total} Jobs erledigt).
         waiting_signal_html: Wartet auf Signal %{signal} seit %{ago}.
+        waiting_signal_late_html: Wartet auf Signal %{signal} seit %{ago} (verspätet seit %{late}). Die nächste Verspätungsmeldung folgt %{time}.
+        waiting_signal_notify_html: Wartet auf Signal %{signal} seit %{ago}. Eine Verspätungsmeldung folgt %{time}, und das Warten geht weiter.
         waiting_signal_timeout_html: Wartet auf Signal %{signal} seit %{ago}. Das Warten läuft ab %{time}.
         waiting_step_html: Wartet auf %{step}.
       not_found:
@@ -1451,7 +1453,7 @@ de:
         ready: Bereit
         waiting: Wartend
       timeline:
-        caption: 'Jeder Versuch, der älteste oben: gewartet (seine Wartezeit), eingereiht (fällig, noch nicht abgeholt), gelaufen (nach Ausgang eingefärbt) und wartend (auf ein Signal oder seinen Stapel), die Pausen in einer eigenen Zeile. Die Versuchstabelle darunter listet dieselben Dauern.'
+        caption: 'Jeder Versuch, der älteste oben: gewartet (seine Wartezeit), eingereiht (fällig, noch nicht abgeholt), gelaufen (nach Ausgang eingefärbt) und wartend (auf ein Signal oder seinen Stapel), die Pausen in einer eigenen Zeile. Die Versuchstabelle darunter listet dieselben Dauern. Eine gestrichelte Linie markiert jedes Mal, wenn ein Warten seine Frist überschritt.'
         empty: Noch keine Versuche.
         paused_period_html: Pausiert ab %{from} für %{duration}
         paused_row: Pausiert
@@ -1465,3 +1467,4 @@ de:
           waited: Gewartet
           waiting: Wartend
         title: Zeitachse
+        wait_timeout_html: 'Das Warten von %{step} überschritt um %{time} seine Frist (on_timeout: %{action}).'

data/config/locales/en.yml

@@ -1349,6 +1349,8 @@ en:
         unknown_html: 'Status: %{status}.'
         waiting_batch_html: Waiting for batch %{batch} (%{done} of %{total} jobs done).
         waiting_signal_html: Waiting for signal %{signal} since %{ago}.
+        waiting_signal_late_html: Waiting for signal %{signal} since %{ago} (late since %{late}). The next late notice goes out %{time}.
+        waiting_signal_notify_html: Waiting for signal %{signal} since %{ago}. A late notice goes out %{time}, and the wait goes on.
         waiting_signal_timeout_html: Waiting for signal %{signal} since %{ago}. The wait times out %{time}.
         waiting_step_html: Waiting on %{step}.
       not_found:
@@ -1451,7 +1453,7 @@ en:
         ready: Ready
         waiting: Waiting
       timeline:
-        caption: 'Each attempt, oldest at the top: waited (its wait), queued (due, not yet picked up), ran (coloured by how it ended) and waiting (for a signal or its batch), with the pauses on their own row. The attempts table below lists the same durations.'
+        caption: 'Each attempt, oldest at the top: waited (its wait), queued (due, not yet picked up), ran (coloured by how it ended) and waiting (for a signal or its batch), with the pauses on their own row. The attempts table below lists the same durations. A dashed line marks each time a wait passed its deadline.'
         empty: No attempts yet.
         paused_period_html: Paused at %{from} for %{duration}
         paused_row: Paused
@@ -1465,3 +1467,4 @@ en:
           waited: Waited
           waiting: Waiting
         title: Timeline
+        wait_timeout_html: "%{step}'s wait passed its deadline at %{time} (on_timeout: %{action})."

data/config/locales/es.yml

@@ -1349,6 +1349,8 @@ es:
         unknown_html: 'Estado: %{status}.'
         waiting_batch_html: Esperando el lote %{batch} (%{done} de %{total} trabajos terminados).
         waiting_signal_html: Esperando la señal %{signal} desde %{ago}.
+        waiting_signal_late_html: Esperando la señal %{signal} desde %{ago} (con retraso desde %{late}). El próximo aviso de retraso sale %{time}.
+        waiting_signal_notify_html: Esperando la señal %{signal} desde %{ago}. Un aviso de retraso sale %{time} y la espera continúa.
         waiting_signal_timeout_html: Esperando la señal %{signal} desde %{ago}. La espera caduca %{time}.
         waiting_step_html: Esperando a %{step}.
       not_found:
@@ -1451,7 +1453,7 @@ es:
         ready: Listos
         waiting: En espera
       timeline:
-        caption: 'Cada intento, el más antiguo arriba: esperó (su espera), en cola (vencido, aún sin tomar), se ejecutó (coloreado según cómo terminó) y en espera (de una señal o de su lote), con las pausas en su propia fila. La tabla de intentos de abajo muestra las mismas duraciones.'
+        caption: 'Cada intento, el más antiguo arriba: esperó (su espera), en cola (vencido, aún sin tomar), se ejecutó (coloreado según cómo terminó) y en espera (de una señal o de su lote), con las pausas en su propia fila. La tabla de intentos de abajo muestra las mismas duraciones. Una línea discontinua marca cada vez que una espera superó su plazo.'
         empty: Aún no hay intentos.
         paused_period_html: En pausa desde %{from} durante %{duration}
         paused_row: En pausa
@@ -1465,3 +1467,4 @@ es:
           waited: Esperó
           waiting: En espera
         title: Cronología
+        wait_timeout_html: 'La espera de %{step} superó su plazo en %{time} (on_timeout: %{action}).'

data/config/locales/fi.yml

@@ -1349,6 +1349,8 @@ fi:
         unknown_html: 'Tila: %{status}.'
         waiting_batch_html: Odottaa erää %{batch} (%{done} / %{total} tehtävää valmis).
         waiting_signal_html: Odottaa signaalia %{signal}, alkoi %{ago}.
+        waiting_signal_late_html: Odottaa signaalia %{signal}, alkoi %{ago} (myöhässä alkaen %{late}). Seuraava myöhästymisilmoitus lähtee %{time}.
+        waiting_signal_notify_html: Odottaa signaalia %{signal}, alkoi %{ago}. Myöhästymisilmoitus lähtee %{time}, ja odotus jatkuu.
         waiting_signal_timeout_html: Odottaa signaalia %{signal}, alkoi %{ago}. Odotus aikakatkaistaan %{time}.
         waiting_step_html: Odottaa vaihetta %{step}.
       not_found:
@@ -1451,7 +1453,7 @@ fi:
         ready: Valmiina
         waiting: Odottaa
       timeline:
-        caption: 'Jokainen yritys, vanhin ylimpänä: odotettu (sen odotus), jonossa (erääntynyt, ei vielä otettu), ajettu (väri kertoo lopputuloksen) ja odottaa (signaalia tai erää), keskeytykset omalla rivillään. Alla oleva yritystaulukko listaa samat kestot.'
+        caption: 'Jokainen yritys, vanhin ylimpänä: odotettu (sen odotus), jonossa (erääntynyt, ei vielä otettu), ajettu (väri kertoo lopputuloksen) ja odottaa (signaalia tai erää), keskeytykset omalla rivillään. Alla oleva yritystaulukko listaa samat kestot. Katkoviiva merkitsee jokaista kertaa, kun odotus ylitti määräaikansa.'
         empty: Ei vielä yrityksiä.
         paused_period_html: Keskeytetty %{from} ajaksi %{duration}
         paused_row: Keskeytetty
@@ -1465,3 +1467,4 @@ fi:
           waited: Odotettu
           waiting: Odottaa
         title: Aikajana
+        wait_timeout_html: 'Vaiheen %{step} odotus ylitti määräaikansa %{time} (on_timeout: %{action}).'

data/config/locales/fr.yml

@@ -1349,6 +1349,8 @@ fr:
         unknown_html: 'Statut : %{status}.'
         waiting_batch_html: En attente du lot %{batch} (%{done} travaux sur %{total} terminés).
         waiting_signal_html: En attente du signal %{signal} depuis %{ago}.
+        waiting_signal_late_html: En attente du signal %{signal} depuis %{ago} (en retard depuis %{late}). Le prochain avis de retard part %{time}.
+        waiting_signal_notify_html: En attente du signal %{signal} depuis %{ago}. Un avis de retard part %{time} et l'attente continue.
         waiting_signal_timeout_html: En attente du signal %{signal} depuis %{ago}. L'attente expire %{time}.
         waiting_step_html: En attente de %{step}.
       not_found:
@@ -1451,7 +1453,7 @@ fr:
         ready: Prêts
         waiting: En attente
       timeline:
-        caption: 'Chaque tentative, la plus ancienne en haut : attente (son délai), en file (due, pas encore prise), exécution (colorée selon son issue) et en attente (d''un signal ou de son lot), avec les pauses sur leur propre ligne. Le tableau des tentatives ci-dessous donne les mêmes durées.'
+        caption: 'Chaque tentative, la plus ancienne en haut : attente (son délai), en file (due, pas encore prise), exécution (colorée selon son issue) et en attente (d''un signal ou de son lot), avec les pauses sur leur propre ligne. Le tableau des tentatives ci-dessous donne les mêmes durées. Une ligne pointillée marque chaque fois qu''une attente a dépassé son échéance.'
         empty: Aucune tentative pour l'instant.
         paused_period_html: En pause à %{from} pendant %{duration}
         paused_row: En pause
@@ -1465,3 +1467,4 @@ fr:
           waited: Attendu
           waiting: En attente
         title: Chronologie
+        wait_timeout_html: 'L''attente de %{step} a dépassé son échéance à %{time} (on_timeout: %{action}).'