dispatch_policy 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6eb153991642d0669fffd7cd3c8c2133c837978faf934bf6b14bf44ae8628907
-  data.tar.gz: 6ce8ff07f09fbd7763cb191b6305f316106b21a5f11b20a25ecdac3e3e2e1f2c
+  metadata.gz: 24ab8c2fe85abc57507f84edc955c8263f59a96505522ecd9ceb6ce60e14bcba
+  data.tar.gz: 152dc560f5b1169d5ef6f4a27065ae629da426ceefb0725fa6bc7a8d13c62a3f
 SHA512:
-  metadata.gz: ea97e0959378bedd5a024888a8beab4880b30025216102b420b03deb5dff0c0b2ffd07beec2eb8e37d713ea3ae763378c8e50b143850563c01bf17bfb1ff47d9
-  data.tar.gz: 5ca3f5205c79ab6c8cbe76cd60820602b5ed69bb57c80109dd0ac240988a7a021a3473c3d0c67ae14181913516bae3cdf46e00ec47cf7310a22c8bf49da087ae
+  metadata.gz: 88f5adb73f3e7bb1893eab32e098a5dda1d7b147c0cd000ee35eb6771b2d2c21f0fe1541c16725b537a1e3a1121166b2681331e7fcea79de26e3d3926137bea0
+  data.tar.gz: c2b17c50ccd765c95bb6d316a68f3e7dece5134ae9ca26fcb918356e0baf611ebbfba19f695e41b1b7402f0021f0b9d3d08111c7938448efb9fb5accda0a4131

data/CHANGELOG.md

@@ -1,43 +1,113 @@
 # Changelog
 
-## 0.2.0
+## 0.3.0
 
 ### Added
-- `round_robin_by` supports `weight: :time` to balance per-tick quanta
-  by recent perform compute time instead of by request count (#3).
-- GitHub Actions CI matrix covering Ruby 3.4 and Rails 7.2 / 8.1 (#5).
-- Integration tests for gate combinations and throttle bucket
-  boundaries (#12).
-- Resilience tests covering failure paths and dedupe state transitions
-  (#13).
-- `bin/release` wrapper around `rake release` (#2).
+- TX-atomic admission: the DELETE on `staged_jobs`, the pre-INSERT
+  in `inflight_jobs` and the adapter handoff (`good_job` /
+  `solid_queue`) all run inside the same transaction, so any failure
+  rolls everything back with no loss window between admission COMMIT
+  and adapter enqueue.
+- `:adaptive_concurrency` gate that auto-tunes per-partition
+  `current_max` via AIMD against an EWMA of `queue_lag` (time from
+  admission to perform start), with a safety valve that floors
+  `remaining` at `initial_max` when `in_flight == 0` so idle
+  partitions can recover after a shrink.
+- In-tick fairness layer: claimed partitions are reordered by
+  `decayed_admits` (EWMA, default `half_life = 60s`) and capped by
+  `fair_share = ceil(tick_cap / N)`. Composes with
+  `:adaptive_concurrency` — fairness writes
+  `partitions.decayed_admits`, adaptive writes
+  `dispatch_policy_adaptive_concurrency_stats.current_max`, no
+  shared locks.
+- `shard_by` to split a policy's partitions across parallel tick
+  loops; the shard is pinned on first write so partitions don't jump
+  between tick workers.
+- Policy-level `partition_by`: a single canonical scope shared by
+  the staged job's `partition_key` and the concurrency gate's
+  `inflight_partition_key`, so no gate suffers scope dilution.
+- Gates are no longer required — a policy with `partition_by` and
+  no gates is valid and still benefits from in-tick fairness.
+- `dispatch_policy_inflight_jobs` is populated for every admitted
+  job (not only concurrency-gated ones), with a heartbeat thread
+  refreshing `heartbeat_at` during perform.
+- Bulk handoff via `ActiveJob.perform_all_later` and bulk-flush of
+  deny-path partition state at the end of a tick (single
+  `UPDATE…FROM(VALUES…)` instead of N per-partition statements).
+- Per-tick metrics layer (`dispatch_policy_tick_samples`) feeding
+  the admin UI: throughput, P50/P95 round-trip ages, capacity
+  headroom, pending trend, fail %, and operator hints.
+- Admin UI improvements: cursor-based pagination of `/partitions`,
+  sort + only-pending filter, auto-refresh control (off / 2s / 5s /
+  10s) via Turbo Drive, per-partition and per-policy Drain action,
+  redesigned dummy demo page with cards + storm controls.
+- `config.enabled` master switch for cutovers.
+- `TickLoop` `busy_pause` to throttle busy iterations.
+- `bin/release` wrapper around `rake release`.
+- Manual benchmark suite, plus a real-adapter end-to-end bench
+  covering `good_job` and `solid_queue`.
 
 ### Changed
-- Admin partition breakdown caps its aggregations to keep the page
-  responsive on policies with many partitions (#9).
-- Admin pending list no longer loads the `arguments` jsonb column
-  (#6).
+- **Breaking:** `partition_by` is policy-level only. Per-gate
+  `partition_by:` was removed; if omitted, `Policy#validate!` raises
+  `InvalidPolicy: partition_by required`. For different per-gate
+  scopes, use separate policies.
+- `partitions.context` is refreshed on every `perform_later` via
+  UPSERT, so changes in the host DB take effect on the next enqueue
+  without redeploys. Gates read this ctx, not the historical
+  `staged_jobs.context`.
+- Tick-claim ordering kept at `last_checked_at NULLS FIRST, id`
+  (anti-stagnation): each partition with pending is processed every
+  ⌈N/B⌉ ticks. Fairness reorder happens after the claim, in memory.
+- Non-PG adapters now warn at boot (`warn_unsupported_adapter`)
+  instead of hard-failing — a custom PG-backed adapter still works.
+- `config.database_role` lets the admission TX target a specific
+  Rails multi-DB role (e.g. `solid_queue` on a separate DB).
 
 ### Fixed
-- Admission is reverted when the underlying adapter silently declines
-  to enqueue, so the staged row doesn't stay marked as admitted (#14).
-- `consumed_ms_by_partition` window is padded to survive
-  minute-boundary races in the time-weighted round-robin fetch (#11).
-- ThrottleBucket row locks are taken in a deterministic key order to
-  remove a deadlock window when multiple ticks contend on the same
-  set of partitions (#8).
+- `BulkEnqueue.perform_all_later` checks `Bypass.active?` and
+  delegates to `super` when active, breaking an infinite re-staging
+  loop on the deserialize + `perform_all_later` path under Bypass.
+- `JobExtension.ensure_arguments_materialized!` is called before
+  reading `job.arguments` in both single and bulk paths — previously
+  the public `arguments` getter returned `[]` for deserialised jobs
+  until `perform_now` triggered private materialization, so the
+  context proc fell back to its defaults.
+- `:adaptive_concurrency` updates `current_max` in a single SQL
+  statement that uses the post-update `ewma_latency_ms` value in
+  its CASE expression, removing read-modify-write races between
+  concurrent workers.
+- Adaptive's feedback signal is measured in `InflightTracker.track`
+  before `block.call` so perform duration doesn't pollute the
+  `queue_lag` signal.
+- Heartbeat thread refreshes `inflight_jobs.heartbeat_at` during
+  perform so long-running jobs aren't reaped as stale.
+- Deny-only ticks persist `next_eligible_at`.
+- Tick samples query no longer depends on `date_bin` (works on
+  Postgres 13).
+- Admin UI preserves scroll position on auto-refresh, and skips
+  auto-refresh while a Turbo visit is in flight.
+- P95/P50 round-trip ages were inverted in the metrics view.
+- Railtie no longer auto-merges the gem's `db/migrate` into the
+  host's paths.
+- "Pending is growing" hint silenced when the backlog has drained.
 
 ### Removed
-- Stale custom `InstallGenerator` — the engine's migration generator
-  is the supported install path (#7).
+- Per-gate `partition_by:` declarations (see Changed).
+- Denormalised `partitions.in_flight_count` counter — `inflight_jobs`
+  is the source of truth.
+- `unclaim!` / `preinserted_inflight_ids` — TX rollback covers the
+  failure case.
 
 ## 0.1.0
 
 Initial release.
 
-- Rails engine + ActiveJob integration (`DispatchPolicy::Dispatchable`).
-- Gates: `:throttle`, `:concurrency`, `:global_cap`, `:fair_interleave`, `:adaptive_concurrency`.
-- Staged jobs with dedupe, round-robin fairness, per-partition counters, and throttle buckets.
-- Admin UI (Chart.js + Turbo) with watched partitions, sparklines, and EWMA queue-lag charts.
-- PostgreSQL required (uses `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`, and `jsonb`).
-- Experimental — being trialed on pulso.run.
+- Rails engine + ActiveJob integration intercepting `perform_later`
+  via `JobExtension`.
+- Gates: `:throttle`, `:concurrency`.
+- Staged jobs admitted by a periodic tick loop, per-partition
+  counters, and token-bucket throttle state.
+- Admin UI showing partitions, pending counts, and recent ticks.
+- PostgreSQL required (uses `FOR UPDATE SKIP LOCKED`, `ON CONFLICT`,
+  and `jsonb`).

data/MIT-LICENSE

@@ -1,21 +1,20 @@
-MIT License
-
 Copyright (c) 2026 José Galisteo
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.