pgbus 0.9.7 → 0.9.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +367 -0
- data/README.md +454 -25
- data/Rakefile +10 -1
- data/app/helpers/pgbus/application_helper.rb +37 -0
- data/app/views/pgbus/processes/_processes_table.html.erb +4 -1
- data/config/locales/da.yml +4 -0
- data/config/locales/de.yml +4 -0
- data/config/locales/en.yml +4 -0
- data/config/locales/es.yml +4 -0
- data/config/locales/fi.yml +4 -0
- data/config/locales/fr.yml +4 -0
- data/config/locales/it.yml +4 -0
- data/config/locales/ja.yml +4 -0
- data/config/locales/nb.yml +4 -0
- data/config/locales/nl.yml +4 -0
- data/config/locales/pt.yml +4 -0
- data/config/locales/sv.yml +4 -0
- data/lib/generators/pgbus/{add_job_locks_generator.rb → add_uniqueness_keys_generator.rb} +5 -5
- data/lib/pgbus/active_job/adapter.rb +1 -1
- data/lib/pgbus/active_job/executor.rb +25 -4
- data/lib/pgbus/cli/dlq.rb +164 -0
- data/lib/pgbus/cli.rb +18 -1
- data/lib/pgbus/client/connection_health.rb +194 -0
- data/lib/pgbus/client.rb +592 -73
- data/lib/pgbus/config_loader.rb +50 -4
- data/lib/pgbus/configuration.rb +294 -79
- data/lib/pgbus/dedup_cache.rb +8 -0
- data/lib/pgbus/doctor.rb +275 -0
- data/lib/pgbus/engine.rb +15 -0
- data/lib/pgbus/execution_pools/async_pool.rb +9 -2
- data/lib/pgbus/execution_pools/thread_pool.rb +7 -0
- data/lib/pgbus/generators/config_converter.rb +7 -5
- data/lib/pgbus/generators/migration_detector.rb +20 -16
- data/lib/pgbus/instrumentation.rb +3 -1
- data/lib/pgbus/integrations/appsignal/probe.rb +23 -1
- data/lib/pgbus/integrations/appsignal/subscriber.rb +17 -2
- data/lib/pgbus/metrics/backend.rb +38 -0
- data/lib/pgbus/metrics/backends/prometheus.rb +123 -0
- data/lib/pgbus/metrics/backends/statsd.rb +64 -0
- data/lib/pgbus/metrics/prometheus_exporter.rb +34 -0
- data/lib/pgbus/metrics/subscriber.rb +214 -0
- data/lib/pgbus/metrics.rb +43 -0
- data/lib/pgbus/pgmq_schema/pgmq_v1.11.1.sql +2126 -0
- data/lib/pgbus/pgmq_schema.rb +7 -2
- data/lib/pgbus/process/consumer.rb +354 -18
- data/lib/pgbus/process/consumer_priority.rb +34 -0
- data/lib/pgbus/process/dispatcher.rb +265 -41
- data/lib/pgbus/process/heartbeat.rb +18 -5
- data/lib/pgbus/process/memory_usage.rb +48 -0
- data/lib/pgbus/process/notify_listener.rb +26 -7
- data/lib/pgbus/process/notify_probe.rb +96 -0
- data/lib/pgbus/process/primary_validator.rb +53 -0
- data/lib/pgbus/process/signal_handler.rb +6 -0
- data/lib/pgbus/process/supervisor.rb +423 -50
- data/lib/pgbus/process/worker.rb +288 -35
- data/lib/pgbus/recurring/schedule.rb +1 -2
- data/lib/pgbus/recurring/scheduler.rb +15 -1
- data/lib/pgbus/serializer.rb +4 -4
- data/lib/pgbus/streams/broadcastable_override.rb +0 -8
- data/lib/pgbus/streams/signed_name.rb +2 -2
- data/lib/pgbus/streams/turbo_broadcastable.rb +7 -5
- data/lib/pgbus/table_maintenance.rb +13 -2
- data/lib/pgbus/uniqueness.rb +11 -12
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/data_source.rb +36 -4
- data/lib/pgbus/web/health_app.rb +102 -0
- data/lib/pgbus/web/health_server.rb +144 -0
- data/lib/pgbus/web/payload_filter.rb +3 -3
- data/lib/pgbus/web/streamer/instance.rb +58 -2
- data/lib/pgbus/web/streamer/listener.rb +69 -21
- data/lib/pgbus.rb +77 -0
- data/lib/tasks/pgbus_doctor.rake +12 -0
- metadata +19 -4
- data/app/models/pgbus/job_lock.rb +0 -98
- data/lib/generators/pgbus/templates/add_job_locks.rb.erb +0 -21
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9848fb03a422ec1c28eda5f7c021ae41ecbdba5a9d4b8246635e14ea10d32213
|
|
4
|
+
data.tar.gz: 766159bb02748854335f90fac50b5fa4bbae82307bcbeacea366d3428a5b6c75
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6da2678bd4e68b0d18fd264af33f9bb463b1944b33db8ec4ecfc0dfdf20167275b7b8067fa9c5b082e3ff9294185c6ea8a9c71538c63704b03165e6d85690e07
|
|
7
|
+
data.tar.gz: c39c733fe2c91107adedd58d1ae27b94123681e87cb7ec3ab15d60338f7b99b49fe05152f07ece13ab1a94efca4a68bdfce8a234bdbffe87ebd51fc153185135
|
data/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,71 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Breaking Changes
|
|
4
|
+
|
|
5
|
+
- **The error hierarchy is unified under `Pgbus::Error` for the 1.0 API freeze.** Several call sites raised bare stdlib errors that no `rescue Pgbus::Error` could catch, freezing an inconsistent contract into 1.0. Now: **`Configuration#validate!` and every config setter raise `Pgbus::ConfigurationError`** (was `ArgumentError`) — this is the one that can break existing code: a `rescue ArgumentError` around `Pgbus.configure` or a boot-time config read will no longer catch config failures; switch to `rescue Pgbus::Error` (or `rescue Pgbus::ConfigurationError`). The ActiveJob adapter's `perform_all_later` msg_id-count mismatch now raises **`Pgbus::EnqueueError`** (new, was `RuntimeError`); `ExecutionPools::AsyncPool` shutdown/at-capacity raises now use **`Pgbus::ExecutionPoolError`** (new, was `RuntimeError`); `Serializer#locate_global_id` GlobalID rejections raise **`Pgbus::SerializationError`** (was `ArgumentError`). Four error classes that bypassed the base — `PgmqSchema::VersionNotFoundError`, `Streams::SignedName::InvalidSignedName`/`MissingSecret`, and `Generators::ConfigConverter::Error` — are re-parented under `Pgbus::Error`. Three classes that reject a malformed *argument shape* — `Configuration::CapsuleDSL::ParseError`, `Streams::Cursor::InvalidCursor`, `Streams::StreamNameTooLong` — **deliberately stay `ArgumentError` subclasses**; the policy ("argument-shape errors are `ArgumentError`, operational errors are `Pgbus::Error`") is documented in `lib/pgbus.rb` and pinned by `spec/pgbus/error_hierarchy_spec.rb`. Messages are unchanged. Refs #282.
|
|
6
|
+
|
|
3
7
|
### Added
|
|
4
8
|
|
|
9
|
+
- **`Pgbus.publish` / `Pgbus.publish_later` top-level shortcuts and a `config.drain_timeout` key (1.0 API freeze).** The event bus now has the same top-level ergonomics as streams: `Pgbus.publish(routing_key, payload, headers:, delay:)` and `Pgbus.publish_later(routing_key, payload, delay:, headers:)` delegate to `Pgbus::EventBus::Publisher` (long form still works), so tutorials no longer fossilize the fully-qualified call. `config.drain_timeout` (validated positive number, default 30) replaces the hardcoded `Worker::DRAIN_TIMEOUT = 30` constant — the graceful-shutdown drain deadline is now operator-tunable, so jobs that legitimately run longer than 30s aren't silently abandoned on every deploy/recycle. `pgbus doctor` gains a seventh check that warns (never fails) when `allowed_global_id_models` is `nil` (allow-all) in a Rails production env — the default is kept for upgrade continuity but is security-relevant. Refs #283.
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- **1.0 API freeze: config renames as deprecated aliases, and removal of provably-dead surface.** Renaming now is cheap; renaming after 1.0 is a major-version break, so the semver-audit renames land as **deprecated aliases** (old name works, warns once, removed in 2.0): `skip_recurring` → `recurring_enabled` (positive polarity, the only negative toggle among the `*_enabled` switches — the alias inverts the boolean); `dashboard_filter_parameters`/`dashboard_filter_sensitive` → `web_filter_parameters`/`web_filter_sensitive` (unify on the incumbent `web_` prefix); and `recurring_tasks_file` (singular), which now warns once when set alongside the plural `recurring_tasks_files` instead of being silently ignored. `ConfigLoader` accepts both spellings automatically (it dispatches on `respond_to?`). **Removed** (each verified zero-caller in the audit): the `lock_ttl:` keyword on `ensures_uniqueness` (validated and stored in metadata but read by nothing — passing it now raises `ArgumentError` naming the removal and the upgrade guide); the `pgbus:add_job_locks` generator and the `Pgbus::JobLock` model (new installs use `pgbus:add_uniqueness_keys`; `pgbus:migrate_job_locks` still retires the legacy `pgbus_job_locks` table); the internal `with_pgbus_durable` streams helper (use `with_pgbus_broadcast_opts(durable:)`); and the streamer's `reconnect_via_reset` fallback (`connection_factory` is now required and always injected, so reconnect always rebuilds a fresh connection). `log_format=` no longer clobbers a custom logger's formatter, and `streams_presence_patterns`/`streams_presence_member`, `group_mode`, and `streams_falcon_streaming_body` are annotated **experimental** (exempt from the 1.0 stability promise). The `workers=` Array form is documented as permanent (not "legacy") — it is the only way to express N identical anonymous capsules. Refs #283.
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
|
|
17
|
+
### Security
|
|
18
|
+
|
|
19
|
+
## [0.9.8.1] - 2026-07-04
|
|
20
|
+
|
|
21
|
+
### Changed
|
|
22
|
+
|
|
23
|
+
- **CI now proves the `railties >= 7.1, < 9.0` support matrix at both endpoints, not just the top.** Every CI leg previously resolved a single `Gemfile.lock` pinned to the latest 8.x, so Rails 7.1 — the lower bound the gemspec promises — was never exercised; shipping 1.0 with an untested support claim is a semver-integrity risk. Added `gemfiles/rails_7_1.gemfile` (+ a checked-in `gemfiles/rails_7_1.gemfile.lock`) that pins the Rails components to `~> 7.1.0` via a plain `BUNDLE_GEMFILE` override (no appraisal gem), and the `test` job now runs the unit suite (`spec/pgbus/ spec/generators/`) against both the main 8.x Gemfile (Ruby 3.3/3.4/4.0) and the 7.1 endpoint (Ruby 3.3 floor + 4.0 ceiling). The 7.1 legs pass green — the only fix needed was test-side: `spec/pgbus/web/data_source_spec.rb` mocked the ActiveRecord connection but let real model side-paths (`QueueState.paused`, `RecurringTask.count`) hit it, and AR 7.1's eager schema load raised RSpec's `MockExpectationError` (an `Exception`, not a `StandardError`, so the methods' rescue missed it); AR 8.1 resolves schema lazily and never surfaced the gap. The main Gemfile now reads an optional `RAILS_VERSION` env override (defaulting to the full supported range) so the endpoint gemfile can pin the lower bound without duplicating gem declarations. A `lint_floor` CI job also runs rubocop + `rake build` on Ruby 3.3 (the gemspec's `required_ruby_version` floor). Refs #284.
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- **Metrics: `pgbus.consumer.recycle` now reaches both metrics subscribers, and `pgbus.client.pool` reaches `Metrics::Subscriber`.** Since v0.9.7 (#200, #205) two instrumentation events had no metrics consumer: `pgbus.client.pool` (emitted per worker heartbeat) was subscribed by neither `Metrics::Subscriber` nor the AppSignal subscriber — Prometheus/StatsD users got zero pool visibility, and even AppSignal only saw pool gauges via the separate minutely `Probe#track_pool` path; `pgbus.consumer.recycle` was subscribed by neither, so consumer recycling (#205) was invisible everywhere while `pgbus.worker.recycle` alone incremented `pgbus_worker_recycled`. `Metrics::Subscriber` now subscribes both: `pgbus.client.pool` forwards `size`/`available` as `pgbus_pool_size` / `pgbus_pool_available` gauges tagged `hostname` (matching the AppSignal probe's naming), and `pgbus.consumer.recycle` increments the *existing* `pgbus_worker_recycled` counter (no new metric) tagged `kind: "consumer"` alongside `reason`; `pgbus.worker.recycle` now also carries `kind: "worker"` so the tag is always present. The AppSignal subscriber gains the same `pgbus.consumer.recycle` handling but deliberately does **not** subscribe `pgbus.client.pool` — the minutely probe already covers AppSignal, and subscribing both would double-report and skew the gauges (documented as a comment at the subscription site). Also corrected the drifted `lib/pgbus/metrics.rb` docstring, which claimed a stale event count, to point at `Pgbus::Instrumentation` as the single source of truth for the emitted `pgbus.*` event catalog. Refs #275.
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
|
|
31
|
+
- **CI: nightly dependency-watch Action for the PGMQ SQL and pgmq-ruby upstreams.** Pgbus hand-tracks two upstreams with nothing watching them: the vendored PGMQ SQL (`lib/pgbus/pgmq_schema/pgmq_v*.sql`) and the `pgmq-ruby` gem (`~> 0.7.0` in `pgbus.gemspec`). New `.github/workflows/dependency-watch.yml` runs nightly (`0 5 * * *`, plus `workflow_dispatch`) and does two plain bash + `gh api`/`curl` + `jq` checks — no Ruby application boot: (1) derives the highest vendored PGMQ version from the `pgmq_v*.sql` filenames (never hardcoded) and compares it against `pgmq/pgmq`'s `releases/latest` tag; (2) resolves the locked `pgmq-ruby` version via `bundle lock` (Gemfile.lock is gitignored for this gem, so a fresh checkout has none) and compares it against the RubyGems API. Both comparisons use `sort -V` so a newer patch release (e.g. 1.11.0 → 1.11.1) is caught correctly. On drift it opens a GitHub issue titled `Upstream release: <dep> <version>`, deduplicated by an exact-title search so repeat runs are a no-op, and best-effort posts to Slack via the optional `SLACK_WEBHOOK_URL` repo secret (logged and skipped, never failing the job, when unset). Notification only — it never auto-vendors SQL or auto-bumps the gem. Refs #287.
|
|
32
|
+
|
|
33
|
+
- **Process model: Consumer brought to full operational parity with Worker — stall detection, circuit breaker, StatBuffer, and pool heartbeat.** #205 gave `Pgbus::Process::Consumer` recycling and NOTIFY wake-up, but four safety-net mechanisms Worker relies on were never carried over, leaving a consumer fork invisible to every operational net. All four now mirror Worker directly (no shared module — the third process type will trigger the extraction). **Stall detection:** the consumer stamps a wall-clock loop beacon (`@loop_tick_at`) at the top of every loop iteration and pokes an OS-level liveness pipe when the supervisor forked one — exactly like Worker — and its heartbeat now persists `loop_tick_at` via a `loop_tick_supplier`. `Supervisor#check_stalled_workers` and `#db_loop_tick_ages` were widened from worker-only to `%w[worker consumer]`, and `fork_consumer` now wires the same liveness pipe (`liveness_reader`/`last_pipe_tick_at`/`pipe_seen`) `fork_worker` does, so a wedged consumer is SIGKILLed and restarted through both the DB-row and the DB-independent pipe channel. `Web::DataSource#stall_threshold_for` gained a `"consumer"` branch — previously it returned `nil`, so `derive_process_status` reported a wedged consumer as `:healthy` forever. The consumer's empty-read wait can legitimately reach `NOTIFY_FALLBACK_POLL_SECONDS` (15s) when a live listener drives wake-up, so the threshold widens by `base + [polling_interval, NOTIFY_FALLBACK_POLL_SECONDS].max` when notify wake-up is enabled and `base + polling_interval` otherwise (mirroring how dispatcher/scheduler widen by their sleep interval). **Circuit breaker:** the consumer now builds a `Pgbus::CircuitBreaker` and consults it in the fetch path — a tripped queue is skipped instead of hammered forever — records success/failure per handled message, and rescues `Pgbus::ConnectionCircuitOpenError` before the generic rescue to idle the poll with no `ErrorReporter` call during a database outage (so a consumer pool no longer floods the error tracker). **StatBuffer:** when `config.stats_enabled`, the consumer builds a `Pgbus::StatBuffer` (using `stats_flush_size`/`stats_flush_interval`), records a `success`/`failed`/`dead_lettered` stat per handled message into the same `pgbus_job_stats` table workers use, and flushes at drain entry, on recycle, and on stop. **Pool heartbeat:** the consumer's heartbeat `on_beat` now emits `pgbus.client.pool` once per beat, so consumer processes surface connection-pool utilization like workers. Worker, Dispatcher, and Scheduler behavior is unchanged. Refs #274.
|
|
34
|
+
|
|
35
|
+
- **PGMQ: vendored schema v1.11.1, proving the multi-version upgrade path.** Pgbus vendored exactly one PGMQ schema (`lib/pgbus/pgmq_schema/pgmq_v1.11.0.sql`), so `PgmqSchema.latest_version`, the `pgbus_pgmq_schema_versions` tracking table, `rails g pgbus:upgrade_pgmq`, and `rake pgbus:pgmq:status`/`:versions` had never been exercised against a real delta. Vendoring upstream PGMQ v1.11.1 (2026-04-19) alongside v1.11.0 (keeping both) gives version discovery a genuine second target: dropping `pgmq_v1.11.1.sql` into the schema dir makes `available_versions` list both and `latest_version` return `"1.11.1"` with no code change (the loader globs `pgmq_v*.sql` and sorts by `Gem::Version`). The v1.11.1 delta is purely additive/cosmetic — the upstream release relocated the `pg_monitor` GRANT block to just after `pgmq.meta` and reworded one `pg_dump` comment; every function body is byte-identical to what pgbus already shipped (which was already a customized superset including `read_grouped_head` plus pgbus topic-routing and notify-insert extensions), so no function pgbus calls through `Pgbus::Client` (`create`, `send`, `read`, `archive`, `delete`, `purge_queue`, `metrics`, …) changed. No new `Client` API was added for `read_grouped_head` (the locked pgmq-ruby 0.7.0 client doesn't expose it and pgbus has no consumer). Proving the upgrade path end-to-end (install v1.11.0 → run the generated `upgrade_pgmq` migration's SQL → v1.11.1 with existing queues and messages surviving) surfaced and fixed a latent bug in `PgmqSchema.drop_pgmq_functions_sql`: its composite-type drop loop skipped every standalone `CREATE TYPE … AS (…)` (they get a `pg_class` shell with `relkind = 'c'`, which the old `pg_class.reltype` guard treated as a table-backed row-type), so `message_record`/`queue_record`/`metrics_result` were never dropped and the subsequent `install_sql`'s bare `CREATE TYPE` raised "type already exists" — meaning any upgrade migration failed. The guard now only skips true relation row-types (`relkind <> 'c'`), so the drop covers the composite type shells and the upgrade re-creates them cleanly. A DB-backed integration spec (guarded by `PGBUS_DATABASE_URL`, skipped cleanly without it) proves the full round-trip. Refs #286.
|
|
36
|
+
|
|
37
|
+
- **Docs: backfilled seven streams features shipped since v0.9.7 (#165–#171) that were name-dropped or missing from the docs site and README.** The [Real-time streams](https://pgbus.zoolutions.llc/docs/streams) page gains a "Broadcast options" table (`exclude:`, `event:`, `coalesce:`, `durable:`, `visible_to:`) plus focused subsections: `exclude:`/actor-echo suppression with the full `X-Pgbus-Connection` header round-trip and `pgbus:connected` frame (#165); `broadcast_render`'s three renderable resolutions — `String` → `#call` → `#render_in` → `#to_s` — and content-less actions (#166); typed SSE event names — server `event:` kwarg → SSE frame → `pgbus:<event>`/`pgbus:event` client dispatch, with the `listen-events` reconnect caveat (#170); `coalesce:` publish-side debounce semantics — trailing-edge-with-max-wait, safe only for idempotent replace/update, superseded frames never hit PGMQ (#171); connection-driven presence — auto-join/leave/touch lifecycle, the context-extractor contract, alongside the existing manual presence API (#169); `stream_key` single-String idempotency and `stream_key!` (#167); and the `msg_id`/revision client-side reconciliation recipe — track the highest applied msgId per target, skip stale morphs (#168). `ConfigReference::streams_presence_patterns`/`streams_presence_member` moved from `INTERNAL_ONLY` to the "Streams (SSE)" `GROUPS` entry in `docs/app/models/config_reference.rb` since they're user-facing opt-in configuration, unlike the genuinely internal streams tuning keys around them — the drift spec (`docs/spec/config_reference_spec.rb`) keeps the move honest. README's "Real-time broadcasts" section gets the same parity pass. No runtime code changed. Refs #280.
|
|
38
|
+
|
|
39
|
+
- **Docs: backfill operations & DX documentation shipped since 0.9.7.** `pgbus doctor`/`rake pgbus:doctor`, `pgbus dlq` (list/show/retry/retry-all/purge), the health endpoints (`Pgbus::Web::HealthApp`, `/livez`/`/readyz`, standalone `health_port`/`health_bind`), eager configuration validation (`eager_validation`), `Client#pool_stats`, the supervisor boot diagnostics banner, per-worker throughput rates on the dashboard's Processes panel, and libpq-native read timeouts (dedicated-connection `statement_timeout`/`tcp_user_timeout` vs. shared-AR `database.yml` guidance) were implemented in prior releases but undocumented on both the docs site and the README. Backfilled into `docs/app/views/docs/pages/cli_generators.rb`, `observability.rb`, `running_workers.rb`, `configuration.rb`, and `dashboard.rb`, plus matching README sections — the client-level `Client::ConnectionHealth` circuit breaker gets its own subsection distinguishing it from the per-queue `Pgbus::CircuitBreaker`, since conflating the two is the likeliest source of operator confusion. Refs #279.
|
|
40
|
+
|
|
41
|
+
- **Docs: "Upgrading pgbus" guide covering the 0.9.7 → 0.9.8 → 1.0.0 path.** No version-to-version upgrade narrative existed — the Migrate group covered switching from other gems only, and the `pgbus:update` story lived as a callout inside the installation page. New `docs/app/views/docs/pages/upgrading_pgbus.rb` (Migrate group) documents the standard upgrade procedure every hop follows (`bundle update pgbus` → `rails generate pgbus:update --dry-run`/apply → `rails db:migrate` → `rake pgbus:pgmq:status` + `rails generate pgbus:upgrade_pgmq` if the vendored PGMQ schema moved → deploy → `bundle exec pgbus doctor`), with a rolling-deploy note on heartbeat/metadata compatibility across mixed versions. A 0.9.x → 0.9.8 section documents the two behavior changes (queue-name validation now rejects dashes; eager configuration validation at boot, with the `eager_validation = false` escape hatch) plus the PGMQ 1.11.0 → 1.11.1 vendored schema bump as a worked example, and links the new opt-in features (`metrics_backend`, `health_port`, `pgbus dlq`, `pgbus doctor`). A forward-looking 0.9.8 → 1.0.0 section stubs the semver commitment and the planned renames/removals tracked by #282 (unified `Pgbus::Error` hierarchy) and #283 (config renames + dead-surface removal) — explicitly marked as a stub those issues must update as they land. The README's "Upgrading from an older pgbus?" callout now links the page. Refs #278.
|
|
42
|
+
|
|
43
|
+
- **Docs: a documentation site at [pgbus.zoolutions.llc](https://pgbus.zoolutions.llc).** A nested, stateless [docs-kit](https://github.com/mhenrixon/docs-kit) Rails app under `docs/` (the same monorepo pattern as docs-kit's own dogfood site and phlex-reactive) turns the single 1,786-line README into a browsable site: 25 pages across six groups (Getting started, Guide, Operations, Testing, Migrate, Reference), five hand-built theme-aware SVG flow diagrams (architecture, message lifecycle, event fanout, outbox, streams), full-text search, per-page Markdown twins, `/llms.txt`/`/llms-full.txt`, and a live read-only MCP endpoint (`POST /mcp`). Recurring tasks — previously implemented but undocumented — now have a dedicated page written from the source. The Configuration reference is generated from a data constant guarded by a drift spec that checks it against the real `Pgbus::Configuration` accessors in both directions, so a renamed or added option fails the build. The gem package is unchanged: `spec.files` globs only `app/ config/ exe/ lib/` + root files, so nothing under `docs/` ships in the gem. The three `docs/switch_from_*.md` migration guides now point to their site pages. Deployed via Kamal + the shared docs-kit reusable workflow; the docs site has its own CI job (`.github/workflows/docs-ci.yml`) scoped to `docs/**`. Refs #266.
|
|
44
|
+
|
|
45
|
+
- **Observability: per-worker throughput rates persisted in heartbeat metadata.** Each worker already tracked live throughput in a `RateCounter` (incremented on `:dequeued`/`:processed`/`:failed` and snapshotted every heartbeat via the worker's `on_beat` hook), but those rates never left the process: `Worker#stats` was in-process only and `Heartbeat#beat` persisted just `last_heartbeat_at` plus `loop_tick_at`. The dashboard's Processes page could show only static boot-time metadata (queues/threads/pid) — operators had no cluster-wide view of per-worker throughput. `Pgbus::Process::Heartbeat#initialize` now accepts a `metadata_supplier:` (a callable returning a Hash, default `nil`); `#beat` calls it **after** `@on_beat` (so the rate counter's `snapshot!` has already refreshed the rates) and merges the result into the persisted `metadata` alongside `loop_tick_at`, all inside the existing `rescue StandardError` so a raising supplier logs a warning and the timer survives. `Worker#start_heartbeat` wires a supplier returning `{ "rates" => {processed, failed, dequeued}, "jobs_processed", "jobs_failed", "in_flight" }`. Heartbeats with no supplier (supervisor, dispatcher, consumer, scheduler, outbox poller, streamer) are untouched — `beat_metadata` returns `nil` for them and leaves the column exactly as before. No SQL changes: `Web::DataSource#format_process` already passes `metadata` through, keeping the no-raw-SQL-outside-`Web::DataSource` rule intact. The Processes table renders the rates human-readable via a new `pgbus_worker_rates` helper (e.g. `12.4/s processed · 0.2/s failed`, zero rates omitted), with i18n keys added under `pgbus.processes.processes_table.rates` across all 12 locales; the raw `rates`/counter keys are hidden from the generic metadata dump. Refs #221.
|
|
46
|
+
|
|
47
|
+
- **Observability: HTTP liveness and readiness endpoints for orchestrators.** Pgbus had no HTTP health surface: the `OK`/`DEGRADED`/`STALLED` verdict in `Pgbus::MCP::HealthAnalyzer` was reachable only through the MCP JSON-RPC tool or the auth-gated dashboard, and the supervisor process (which forks and watches workers) had no HTTP surface at all — so a Kubernetes kubelet could not distinguish "supervisor alive" from "workers silently wedged". New `Pgbus::Web::HealthApp` (`lib/pgbus/web/health_app.rb`) is a plain Rack app exposing `GET /livez` (→ `200 text/plain "ok"`, unconditionally, **no** database access — the serving process is up) and `GET /readyz` (builds a `Web::DataSource`, runs `HealthAnalyzer#verdict`: `OK`/`DEGRADED` → `200`, `STALLED` → `503`, body = the verdict JSON; any `StandardError` such as an unreachable DB → `503 {"status":"ERROR"}`, logged via `Pgbus.logger` and never swallowed). Unknown paths return `404`; non-`GET` returns `405`. `DEGRADED` deliberately stays ready — only the silent-wedge `STALLED` signal fails readiness. Mount it in a web pod: `mount Pgbus::Web::HealthApp.new => "/pgbus/health"`. For worker pods that run the supervisor (no Puma to mount into), set `config.health_port` (Integer, default `nil` = disabled) and `config.health_bind` (default `"127.0.0.1"`) and `Supervisor#run` serves both paths standalone via `Pgbus::Web::HealthServer` (`lib/pgbus/web/health_server.rb`) — a single accept-loop thread over a `TCPServer` that parses only the HTTP request line, synthesizes a minimal Rack env, and dispatches to the same `HealthApp#call`, so routing and the verdict→status mapping live in exactly one place. The analyzer is required lazily (it has no `mcp`-gem dependency), so the standalone server works without the optional `mcp` gem installed. Refs #218.
|
|
48
|
+
|
|
49
|
+
- **Observability: generic metrics adapter with Prometheus and StatsD backends.** `lib/pgbus/instrumentation.rb` documents 16 `pgbus.*` `ActiveSupport::Notifications` events, but the only shipped consumer was the AppSignal integration (`lib/pgbus/integrations/appsignal/subscriber.rb` subscribes 13 of them) — teams on Prometheus, Datadog, or plain StatsD got no metrics without hand-writing subscribers. New `Pgbus::Metrics` namespace (Zeitwerk-managed under `lib/pgbus/metrics/`, unlike the explicitly-required `integrations/`) adds a vendor-neutral second consumer: `Metrics::Backend` (interface — `increment(name, value = 1, tags = {})`, `gauge(name, value, tags = {})`, `histogram(name, value, tags = {})` — plus a `Null` no-op subclass); `Metrics::Subscriber` (mirrors the AppSignal subscriber's 13 subscriptions and identical `pgbus_`-prefixed metric names/tags — `pgbus_queue_job_count` with `status`, `pgbus_job_duration_ms`, `pgbus_messages_sent`/`pgbus_messages_read`, `pgbus_event_count`/`pgbus_event_duration_ms`, `pgbus_outbox_published`, `pgbus_recurring_enqueued`, `pgbus_worker_recycled`, `pgbus_stream_broadcast_count` — forwarding to the configured backend, every handler wrapped in a `safely` rescue+log so a raising backend never propagates into the producer thread); `Metrics::Backends::Prometheus` (dependency-free, Mutex-guarded in-process registry, histograms as sum+count summaries, text-exposition rendering with label escaping); `Metrics::PrometheusExporter` (a self-contained Rack app rendering exposition format v0.0.4, mountable like `StreamApp`, `503` when the configured backend is not Prometheus); and `Metrics::Backends::Statsd` (`UDPSocket` datagrams — `|c`/`|g`/`|ms` with DogStatsD `|#tag:value` tags, no gem dependency). Config keys in `lib/pgbus/configuration.rb`: `metrics_backend` (`nil` default | `:prometheus` | `:statsd` | a `Backend` instance — validated eagerly in `validate!`), `statsd_host` (`"127.0.0.1"`), `statsd_port` (`8125`); `ConfigLoader` auto-recognizes them. The subscriber installs from a new `pgbus.metrics` engine initializer that mirrors the AppSignal one — `next if metrics_backend.nil?` so the default installs no subscription (zero overhead). AppSignal is untouched; both integrations can run simultaneously. README documents configuration, the emitted metrics, and mounting the exporter. Refs #219.
|
|
50
|
+
|
|
51
|
+
- **DX: `pgbus dlq` — dead-letter management from the CLI.** Dead-letter inspect/drain operations existed only behind the mounted web dashboard (`Web::DataSource#dlq_messages`/`#dlq_message_detail`/`#retry_dlq_message`/`#discard_dlq_message`/`#retry_all_dlq`/`#discard_all_dlq`), so headless deployments and incident runbooks could not inspect or drain a DLQ without ad-hoc SQL — which project rules forbid. New `Pgbus::CLI::DLQ` (`lib/pgbus/cli/dlq.rb`), dispatched from `CLI.start`, adds five subcommands that route through `Web::DataSource` so retry semantics stay identical to the dashboard (origin-queue re-enqueue by stripping `DEAD_LETTER_SUFFIX`, transactional produce+delete, lock release on discard) with zero raw SQL and no direct PGMQ calls: `pgbus dlq list [--page N] [--per-page N]` (table of msg_id, DLQ queue, origin queue, read_ct, enqueued_at plus a total-count footer — never payloads), `pgbus dlq show MSG_ID` (message metadata with the payload passed through `Web::PayloadFilter.filter_json` so the dashboard's sensitive-data filtering applies identically), `pgbus dlq retry MSG_ID` (resolve the DLQ queue via `dlq_message_detail`, then `retry_dlq_message`), `pgbus dlq retry-all` (prints the re-enqueued count), and `pgbus dlq purge MSG_ID` / `pgbus dlq purge --all --yes` (`--all` without `--yes` makes no changes and exits 1). An unknown msg_id and an unknown subcommand both exit 1. Refs #216.
|
|
52
|
+
|
|
53
|
+
- **DX: boot diagnostics banner at supervisor start.** `Supervisor#run` logged only `"[Pgbus] Supervisor starting pid=…"` — the settings that decide whether a deployment actually works (which database it connected to, LISTEN/NOTIFY vs polling, the resolved pool size, which roles and capsules boot) were never stated, so a misconfigured deployment that sat idle forced operators to attach a console and inspect `Pgbus.configuration` by hand. `Supervisor#run` now emits a one-block banner (a run of `"[Pgbus] boot:"`-prefixed `info` lines that render cleanly under both the `:text` and `:json` log formatters) right after `start_heartbeat` and before `bootstrap_queues`: `Pgbus::VERSION`; the connection target reduced to `host/dbname` (never the password) across all three `connection_options` forms — `database_url` string, `connection_params` hash, and AR-derived hash; `config.resolved_pool_size`; `config.pgmq_schema_mode` and the installed PGMQ version (best-effort via `Client#pgmq_schema_version`, `unknown` on any error); `config.listen_notify` and `config.worker_notify_wakeup?`; the roles that will actually boot honoring `config.roles`; and one line per worker capsule (name or `anonymous`, queues, threads, execution mode via `config.execution_mode_for`) and per event consumer (topics, threads). Every DB-dependent field is wrapped so a transient failure degrades that field to `unknown` — the banner can never abort boot. Refs #213.
|
|
54
|
+
|
|
55
|
+
- **DX: configuration is now validated eagerly at boot.** `Configuration#validate!` had no call site in `lib/` — `Pgbus.configure` just yielded and `ConfigLoader.apply` just assigned keys, so an invalid value like `visibility_timeout = 0` sat dormant until a worker code path consumed it, failing at runtime far from the misconfiguration (the opposite of the fail-loud-at-boot behavior `roles=` and the duration setters already had). `Pgbus.configure` now runs `configuration.validate!` after the yield, and `ConfigLoader.apply` runs `config.validate!` after its key loop — and since both `ConfigLoader.load` branches route through `apply` and the engine's `pgbus.configure` initializer loads `config/pgbus.yml` through it, a bad YAML value now aborts Rails boot with an `ArgumentError` naming the offending key. `validate!` stays DB-free (pure value checks), so eager validation adds no boot-time DB dependency; defaults are valid, so partial and sequential `configure` blocks pass. A new `attr_accessor :eager_validation` (default `true`) is the escape hatch for exotic setups that intentionally hold a transiently-invalid config: set `eager_validation = false` before or inside a `configure` block (or as a key in `pgbus.yml`) to suppress the automatic call — both call sites check the flag after applying settings, and explicit `validate!` still works. **Backward-incompatible in one direction:** an invalid-but-previously-unread config now raises at boot instead of silently later; that is the intended change, and `eager_validation = false` restores the old behavior. Refs #215.
|
|
56
|
+
|
|
57
|
+
- **DX: `pgbus doctor` — a single preflight command for deploy/CI gating.** Nothing answered "is this pgbus environment healthy enough to run?" in one place: config validity was never checked at boot (`Configuration#validate!` was never called), DB reachability only surfaced when the supervisor forked children that crash-looped, PGMQ version lived in `rake pgbus:pgmq:status`, and the process-liveness verdict was reachable only through the MCP stdio server (which needs the optional `mcp` gem). New `Pgbus::Doctor#run` runs six checks — each a `{ name:, status: :ok|:warn|:fail, detail: }` — and `#success?`: (1) `Configuration#validate!`, (2) DB connectivity via a new public `Client#ping` (`SELECT 1` through `with_raw_connection`), (3) PGMQ schema presence and installed-vs-vendored version via a new `Client#pgmq_schema_version` (the same tracking-table read `pgbus:pgmq:status` uses, kept inside the Client so no raw SQL escapes it), (4) queue existence — configured queues (via `Client#configured_queues`, prefixed through `config.queue_name`) diffed against `Client#list_queues`, (5) LISTEN/NOTIFY liveness per configured queue via a new `Client#notify_enabled?`, and (6) process liveness via `Pgbus::MCP::HealthAnalyzer` (required directly — it only needs a `Web::DataSource`, not the `mcp` gem). Doctor never touches PGMQ/PostgreSQL directly and never raises: a broken environment turns every probe into a `:fail` result rather than a crash. `#report` prints one line per check plus a resolved-config summary (queue_prefix, default_queue, pgmq_schema_mode, resolved_pool_size, roles, capsules) with passwords redacted from `database_url`/`connection_params`. Wired into `pgbus doctor` (exit 1 unless `success?`, so it gates a deploy or CI run) and `rake pgbus:doctor`. Refs #212.
|
|
58
|
+
|
|
59
|
+
- **Process: exponential cooldown on the worker queue evict/restore cycle.** When every queue a worker owned had been evicted — the queue table permanently deleted — `restore_evicted_queues` reinstated the initial queue set immediately, both from `evict_missing_queues` and again at the top of every `fetch_messages`. The worker then oscillated forever: fetch → undefined-table error → evict → restore → fetch → error…, emitting an error+warn pair every loop tick (default 0.1s), spamming logs and hammering the database with doomed reads. Restore is now gated by an exponential cooldown: the first restore after a full eviction waits `RESTORE_COOLDOWN_BASE` (30s), and each consecutive failed restore doubles the wait up to `RESTORE_COOLDOWN_MAX` (300s), tracked by a monotonic `@last_evicted_at` window and a `@restore_streak` counter. While the cooldown is pending, `fetch_messages` leaves `@queues` empty (the caller idles on the existing empty-`active_queues` path) and logs exactly one deferral `warn` per window ("all queues evicted; next restore attempt in Ns") instead of an error pair per tick. A fetch that completes without an undefined-queue error resets the streak to 0, so a recreated queue is reinstated promptly after the next eviction. Wildcard workers are untouched — both restore call sites remain guarded by `!@wildcard`. Refs #209.
|
|
60
|
+
|
|
61
|
+
- **Process model: Consumer brought to parity with Worker — recycling and NOTIFY wake-up.** `Pgbus::Process::Consumer` had neither mechanism: its loop never checked `max_jobs_per_worker` / `max_memory_mb` / `max_worker_lifetime`, so a long-lived consumer fork grew memory unboundedly (the exact solid_queue problem recycling exists to fix), and its empty-read path always slept `polling_interval` even when a matching event had just arrived. Consumer now mirrors Worker: a per-iteration `check_recycle` trips once when any configured limit is exceeded (`@jobs_processed`, an `AtomicFixnum` incremented in an `ensure` on every handled message — success, DLQ-routed, and rescued failure alike, mirroring `Worker#process_message` so `max_jobs` recycling still trips on a poison/all-failing queue; a monotonic `@started_at_monotonic`; and the shared, TTL-cached `Pgbus::Process::MemoryUsage`), logs the reason, emits a `pgbus.consumer.recycle` instrumentation event, and sets `@shutting_down` so the loop drains the pool and exits cleanly — the clean exit earns an immediate supervisor restart, replacing the fork before its memory grows. For wake-up, the two `interruptible_sleep(polling_interval)` calls are replaced with `@wake_signal.wait(timeout: wake_timeout)`, and (when `config.worker_notify_wakeup?`) a `Pgbus::Process::NotifyListener` over the physical names of the consumer's subscribed queues fires `WakeSignal#notify!` the moment a row is inserted, collapsing idle latency from a full poll interval to near-instant. A live listener raises the empty-read wait to a `NOTIFY_FALLBACK_POLL_SECONDS` (15s) safety-net ceiling; a listener that fails to start or whose thread dies logs and falls back to polling, with `ensure_notify_listener` self-healing it on a 5s→300s exponential backoff. With no recycle config and notify wakeup off, behavior is unchanged. All PGMQ access stays behind `Pgbus::Client`. Refs #205.
|
|
62
|
+
|
|
63
|
+
- **Connections: connection-pool utilization surfaced, and pool-exhaustion errors made actionable.** The PGMQ connection pool was invisible: pgmq-ruby exposes `size`/`available` counters, but pgbus never read them, so the first sign of an undersized or leaking pool was an opaque `PGMQ::Errors::ConnectionError: Connection pool timeout` after `pool_timeout` seconds — with no pool state and no gauge to alert on *before* saturation. New `Pgbus::Client#pool_stats` returns `{ size:, available:, pool_timeout: }` (pgmq-ruby's `stats` merged with the configured timeout) and works on both the dedicated-pool path and the shared-`Proc` path (`size` is 1 there); it is purely observational and rescues internally to `{}` so reading the pool can never break job processing. The worker heartbeat now emits a `pgbus.client.pool` instrumentation event once per beat (heartbeat-only, never on a per-job hot path) carrying that payload, and the AppSignal minutely probe reports `pgbus_pool_size` / `pgbus_pool_available` gauges tagged by hostname (the pool is per-process, unlike the cluster-wide queue/summary gauges). Finally, a bare `Connection pool timeout` error is re-raised as the *same* `PGMQ::Errors::ConnectionError` class (so `with_stale_connection_retry` semantics are unchanged and pool timeouts remain non-retryable) but with the live pool state and an actionable hint appended — "raise `Pgbus.configuration.pool_size` or reduce worker threads" — so the first signal of saturation is diagnosable. Refs #200.
|
|
64
|
+
|
|
65
|
+
- **Connections: client-level circuit breaker for database-down conditions.** Resilience was previously per-queue only: `Pgbus::CircuitBreaker` records failures from job *execution* and persists its pause state in the database — useless when the database itself is down (its `check_paused` rescues and returns `false`, so nothing trips). Meanwhile every worker's `fetch_messages` rescued each `PGMQ::Errors::ConnectionError`, reported it to `ErrorReporter`, and re-polled every `polling_interval`, so a whole fleet hammered a dead database and flooded the error tracker for the entire outage. A new in-memory, process-local latch — `Pgbus::Client::ConnectionHealth`, owned by `Pgbus::Client` — trips after `OPEN_THRESHOLD` (5) consecutive `PGMQ::Errors::ConnectionError` across any operation. Once open, the read paths (`read_message`, `read_batch`, `read_multi`, `read_batch_prioritized`, `read_grouped*`, `read_with_poll`) fail fast with a new `Pgbus::ConnectionCircuitOpenError` *without checking out a pool connection*; a single half-open probe is admitted after a monotonic backoff window (1s base, doubling per re-open, capped at 60s), whose success closes and resets the breaker and whose failure re-opens it with a doubled window. Enqueues (`send_message`/`send_batch`) are deliberately never short-circuited — callers must see enqueue failures. `Worker#fetch_messages` rescues `ConnectionCircuitOpenError` before its generic rescue and idles the poll with no `ErrorReporter` call, so an outage produces two log lines total (a `warn` on open, an `info` on close) instead of one per worker per poll. The latch is `Mutex`-guarded (single shared state, atomic transitions); a half-open probe that fails with *any* error — including a `Pgbus::ReadTimeoutError` from a hung socket, which the read paths raise from inside the guard — re-opens the breaker rather than wedging it half-open, and a straggler read admitted while closed cannot re-close a breaker another thread just tripped. Refs #197.
|
|
66
|
+
|
|
67
|
+
- **Connections: LISTEN/NOTIFY delivery self-probe at listener start.** A worker or streamer pointed at a transaction-mode pooler (PgBouncer drops `LISTEN` at transaction boundaries) or a read-only replica (`pg_notify()` raises) previously "connected successfully" but never received a wake-up, silently degrading to slow fallback polling / stalled SSE with no explanation. Each listener now runs a one-shot self-probe right after building its dedicated `PG::Connection`: `LISTEN` a unique per-process channel, `SELECT pg_notify()` to itself, and wait up to 2s for delivery. On failure it logs a single actionable `error` naming the direct-connection overrides to set (`worker_notify_database_url` / `worker_notify_host` / `worker_notify_port` for workers; `streams_database_url` / `streams_host` / `streams_port` for the streamer), best-effort `UNLISTEN`s the probe channel, and returns — the worker/streamer still start and degrade gracefully rather than crash. The probe runs on the initial connection only; reconnects skip it to stay cheap. New `Pgbus::Process::NotifyProbe`. Refs #191 (epic #190).
|
|
68
|
+
|
|
5
69
|
- **Streams: optional publish-side coalescing for high-frequency broadcasts.** A chatty reactive component (live cursor, typing indicator, progress bar) can fan out many small broadcasts per second. Pass `coalesce:` (a window in milliseconds, or `true` for the 50ms default) with `target:` to `broadcast`/`broadcast_render` to batch per `(stream, target)` and publish only the *latest* frame within the window — superseded frames never hit the bus (no PGMQ insert, no NOTIFY, no fan-out). Last-write-wins, so it is opt-in and only safe for idempotent `replace`/`update` of a stable target (exactly the high-frequency case). The new `Pgbus::Streams::Coalescer` is a process-wide, thread-safe, trailing-edge-with-max-wait debounce: the first submit per window schedules the flush (bounding latency to one window) and later submits only overwrite the buffered payload; the flush re-enters the normal broadcast path, so a coalesced frame is just a deferred ordinary broadcast that still composes with `visible_to`/`exclude`/`event`/`durable`. Refs #171.
|
|
6
70
|
|
|
7
71
|
- **Streams: typed SSE event names on broadcasts.** A broadcast can set the SSE `event:` field — `Pgbus.stream(name).broadcast(html, event: "presence")` (and `broadcast_render(..., event:)`) — while keeping the payload a Turbo Stream, so clients route on the typed name instead of sniffing the HTML. The default (`nil` or `"turbo-stream"`) is omitted from the JSONB payload to avoid redundancy, but is still set on the SSE frame's `event:` line (the connection adapter falls back to `turbo-stream`), so default consumers still get the standard `message`/turbo-stream path. The event flows through the JSONB payload → `StreamEnvelope.event` → the SSE frame's `event:` line (both the Puma and Falcon connection adapters). On the client, `<pgbus-stream-source>` dispatches a typed broadcast as a generic `pgbus:event` (`{ event, data, msgId }`) and a named `pgbus:<event>` (`{ data, msgId }`) for `addEventListener` ergonomics; declare typed event names via the element's `listen-events` attribute so they survive the EventSource reconnect path. Refs #170.
|
|
@@ -16,6 +80,12 @@
|
|
|
16
80
|
|
|
17
81
|
- **Streams: `Pgbus.stream_key` is idempotent for an already-built key, plus `Pgbus.stream_key!`.** A single `String` argument is now treated as a pre-built pgbus stream key and returned unchanged (after the queue-name budget check) instead of tripping the colon-separator guard. This lets a consumer hold one `stream_key` value and pass it to both `turbo_stream_from` and the broadcaster without `stream_key("chat:lobby")` raising `ArgumentError`. The guard still fires for the genuinely ambiguous multi-fragment join (`stream_key("a:b", :c)`), and `Symbol`/record fragments with colons are still rejected (a colon there never came from `stream_key`). `Pgbus.stream_key!(key)` accepts a pre-built key explicitly (String required, budget enforced). Refs #167.
|
|
18
82
|
|
|
83
|
+
- **Observability: StatBuffer flushes on drain entry and its flush thresholds are now configurable.** `Pgbus::StatBuffer` accumulated up to `DEFAULT_FLUSH_SIZE = 100` entries / `DEFAULT_FLUSH_INTERVAL = 5`s of job stats in memory, flushed only by the worker's periodic `flush_if_due` tick and the drain-loop `stop`. When the supervisor watchdog `SIGKILL`s a stalled worker — which cannot be trapped — everything buffered since the last flush was silently lost, and the drain phase's up-to-30s wait widened that window. `Worker#graceful_shutdown` and `Worker#check_recycle` now call `@stat_buffer&.flush` immediately after `transition_to(:draining)`, so entering drain on either a graceful `TERM` or a recycle threshold (`max_jobs`/`max_memory`/`max_lifetime`) persists the buffer before the watchdog can intervene. Both run on the main loop thread (signals are dispatched via `process_signals`, never in trap context), so the DB write is safe; `flush` is thread-safe and no-ops on an empty buffer and when stats are disabled (`@stat_buffer` is nil). New config keys `stats_flush_size` (default `100`, positive integer) and `stats_flush_interval` (default `5`, positive number) on `Configuration` are validated in `validate!`, auto-recognized by `ConfigLoader`, and passed into `StatBuffer.new` by the worker — tune them down for a tighter loss window on high-throughput deployments or up to reduce insert frequency. The residual loss window (≤ `stats_flush_interval` seconds / `stats_flush_size` entries lost only on an un-trappable `SIGKILL`, affecting insights accuracy only — never a job payload) is documented in the README stats section. Refs #220.
|
|
84
|
+
|
|
85
|
+
### Changed
|
|
86
|
+
|
|
87
|
+
- **DX: worker and event-consumer config keys are normalized once, at assignment.** Worker/consumer config hashes arrived with symbol keys (the Ruby DSL and `capsule`) or string keys (YAML through `ConfigLoader`), so every read site did a dual `w[:x] || w["x"]` lookup — and any new option read without the string fallback silently ignored a YAML-provided value, a recurring bug class. `Configuration#workers=` now maps each `Array`-form entry through a private `normalize_entry` (`transform_keys(&:to_sym)`; entries are flat hashes, no deep recursion) and raises `ArgumentError` for a non-Hash entry; `event_consumers` becomes an `attr_reader` plus a matching `event_consumers=` setter (nil passes through). The string/DSL path already yielded symbols, so it is unaffected. With both setters normalizing, the string fallbacks were deleted from every downstream site — `supervisor` (`fork_worker`, `fork_consumer`), `client` (`collect_configured_queues`), `configuration` (`execution_mode_for`, `validate!`, `capsule_name`, `validate_no_queue_overlap!`, `sum_thread_counts`), `doctor` (`capsule_summary`), and `cli` (`apply_capsule_filter`) — which now use symbol access only. YAML `workers:`/`event_consumers:` sections still boot with correct queues/threads/modes because `ConfigLoader` routes them through the same setters; the CLI `--queues`/`--capsule` overrides route through `workers=` too, so they get normalization for free. Out of scope: the config-converter generator (operates on raw YAML before `Configuration`) and the DB `metadata` JSON read in `consumer_priority` (Postgres always yields string keys there, so no symbol fallback exists to remove). Refs #214.
|
|
88
|
+
|
|
19
89
|
### Breaking Changes
|
|
20
90
|
|
|
21
91
|
- **Queue names must be alphanumeric and underscores only.** Queue names containing dashes (e.g., `my-app-queue`) will now raise `ArgumentError`. Rename to underscored form (e.g., `my_app_queue`) before upgrading. This restriction prevents SQL injection via PGMQ queue identifiers, which are interpolated into table names and cannot be parameterized.
|
|
@@ -31,8 +101,305 @@
|
|
|
31
101
|
|
|
32
102
|
### Fixed
|
|
33
103
|
|
|
104
|
+
- **The documented uniqueness onboarding path now creates the table the runtime actually reads.** The README's uniqueness Setup section told a new user to run `rails generate pgbus:add_job_locks`, which creates `pgbus_job_locks` — a table nothing in the runtime reads. `ensures_uniqueness` (`Pgbus::UniquenessKey` in `app/models/pgbus/uniqueness_key.rb`, `lib/pgbus/uniqueness.rb`) and the dispatcher's reaper (`cleanup_job_locks`/`reap_orphaned_uniqueness_keys`) work exclusively against `pgbus_uniqueness_keys`; the modern `add_uniqueness_keys.rb.erb` migration template existed but was reachable only through `pgbus:update`'s `MigrationDetector`, so a fresh install following the README silently never locked anything. New standalone generator `pgbus:add_uniqueness_keys` (mirrors `pgbus:add_queue_states`, including `--database=`) exposes that template directly; `MigrationDetector::GENERATOR_MAP[:uniqueness_keys]` now invokes it instead of `pgbus:migrate_job_locks`. `pgbus:add_job_locks` is not removed — installs mid-migration still legitimately need it alongside `pgbus:migrate_job_locks` — but it now prints a deprecation warning pointing new installs at `pgbus:add_uniqueness_keys`. README and the docs-site `concurrency-uniqueness`/`cli-generators` pages retargeted to the new generator. Refs #277.
|
|
105
|
+
|
|
106
|
+
- **The Turbo broadcast path now honors `streams_durable_patterns`.** `Pgbus::Streams::TurboBroadcastable#broadcast_stream_to` — the patch every `Turbo::Broadcastable` call (`broadcasts_to`, `broadcast_replace_to`, …) and phlex-reactive's `broadcast_*_to` funnel through — resolved durability from the thread-local override or `streams_default_broadcast_mode` only, then always passed an explicit `durable:` to `Pgbus.stream`. That short-circuited `Pgbus.stream`'s own resolver (`durable.nil? ? stream_durable?(name) : durable`), so `streams_durable_patterns` was dead configuration for the entire broadcast funnel: a matching pattern never promoted a broadcast to durable, and large component renders (> 8000 bytes) that were supposed to route through PGMQ instead went ephemeral through `ActionCable` and died with `PG::ProgramLimitExceeded: payload string too long`. `broadcast_stream_to` now resolves `override.nil? ? Pgbus.configuration.stream_durable?(name) : override`, letting the pattern resolver (which already falls back to `streams_default_broadcast_mode`) participate whenever no explicit thread-local override is present. Apps without `streams_durable_patterns` are unaffected; the thread-local override (set via `broadcastable_override`'s `durable:` kwarg) still wins over a matching pattern. Refs #267.
|
|
107
|
+
- **Reads are now bounded by libpq-native timeouts instead of Ruby `Timeout`.** `Pgbus::Client#with_read_timeout` wrapped every read in `Timeout.timeout(config.read_timeout, Pgbus::ReadTimeoutError)`, which interrupts via `Thread#raise`. That can fire at an arbitrary point — including mid-libpq call — and leave a pooled `PG::Connection` in a state that re-hangs or returns wrong results on reuse; on the shared-AR path it does so on a connection ActiveRecord also queries. On the dedicated-connection path (`database_url` / `connection_params`), `Client#initialize` now bakes two libpq-native bounds into the connection options: (1) a server-side `statement_timeout` (`options=-c statement_timeout=<read_timeout>ms`) so Postgres cleanly cancels an overrunning query — surfaced by pgmq-ruby as `PGMQ::Errors::ConnectionError` ("canceling statement due to statement timeout") and re-raised as `Pgbus::ReadTimeoutError`; and (2) client-side `tcp_user_timeout` + `keepalives` (sized at `read_timeout + 5s`, and only when the linked libpq is ≥ 12 — older libpq rejects the `tcp_user_timeout` conninfo keyword and would fail the connection) so a dead/hung socket makes libpq raise `PG::ConnectionBad` synchronously — no `Thread#raise`, no buffer corruption — which the circuit breaker and stale-connection retry already handle. A caller-supplied `:options` connection param (e.g. `-c search_path=…`) is preserved — the `statement_timeout` flag is appended, not overwritten. On Linux with `read_timeout` set and libpq ≥ 12, these fully cover both a slow query and a hung socket, so Ruby `Timeout` is never wired in (detected at init via `Socket.const_defined?(:TCP_USER_TIMEOUT)` and `PG.library_version`). Ruby `Timeout` remains only as a narrow last resort on a *dedicated* connection where libpq can't bound the socket — non-Linux hosts (`tcp_user_timeout`/`keepalives` are no-ops there) or libpq < 12. The shared-AR `Proc` path gets neither a baked-in bound nor the Ruby `Timeout` (pgbus doesn't own that socket); instead `Client#initialize` logs a one-time hint to configure the same libpq timeouts in `database.yml` (`variables: { statement_timeout: <ms> }` plus `tcp_user_timeout`/`keepalives`), which ActiveRecord passes straight through. A new custom RuboCop cop `Pgbus/NoRubyTimeout` bans `Timeout.timeout` project-wide (no autocorrect) to prevent regressions. Note: on the dedicated path writes gain the same connection-wide `statement_timeout` (acceptable — an enqueue that can't complete within `read_timeout` is already failing). No per-call hot-path change (`client_bench` unchanged; the added cost is a one-time transform in `initialize`). Refs #198.
|
|
108
|
+
- **Stream archive pruning no longer silently no-ops.** `Dispatcher#prune_stream_archives` read `config.archive_compaction_batch_size`, an accessor that was culled from `Configuration` into the `Pgbus::Process::Dispatcher::ARCHIVE_COMPACTION_BATCH_SIZE` constant in the 0.x config-cleanup refactor (#82). Because `prune_stream_archives` was introduced *after* that refactor and copied the pre-cull pattern, every invocation raised `NoMethodError`, which the method's outer `rescue StandardError` swallowed — so per-stream archive tables were never pruned and grew unbounded regardless of `streams_retention`/`streams_default_retention`. It now uses the constant, matching `#compact_archives`. Refs #204.
|
|
109
|
+
- **Long dispatcher maintenance passes are now interruptible during shutdown.** `graceful_shutdown`/`immediate_shutdown` only set `@shutting_down`, which the run loop checked between maintenance passes but never inside them. With many queues or large bloated tables, a `TERM` arriving mid-pass blocked graceful shutdown until the entire pass finished — overrunning deploy windows until the supervisor escalated to `KILL`. The per-queue loops in `compact_archives`, `prune_stream_archives`, and `sweep_orphan_streams` now break before the next queue when a shutdown begins, and `TableMaintenance.run_maintenance` accepts a `stop_check:` callable (the dispatcher passes `-> { @shutting_down }`) checked before each table. An interrupted task simply re-runs next interval; retry-on-failure semantics are unchanged. Refs #204.
|
|
34
110
|
- **EventBus handler invocations now run inside `Rails.application.executor` (or `.reloader` in dev).** `Pgbus::EventBus::Handler#process` previously ran outside any Rails executor wrapper, so `claim_idempotency?`'s `ProcessedEvent.insert` (and any `handle` body that touches AR) leased a connection that was never returned to the pool. With a small AR pool (Rails dev default with `RAILS_MAX_THREADS=3` is 6 connections), a handful of consumed events leaked the entire pool. The next request that triggers `clear_reloadable_connections!` (any reloadable change in dev) hung in `with_exclusively_acquired_all_connections`, surfacing as a confusing `Rack::Timeout` with a trace ending in `MonitorMixin#wait_for_cond`. Mirrors the executor-wrap pattern already used by `Pgbus::ActiveJob::Executor#execute_job`. No-op when Rails isn't loaded.
|
|
35
111
|
- **Defensive retry on stale pooled pgmq connections in the enqueue path.** `Pgbus::Client#send_message`, `#send_batch`, and `#publish_to_topic` now retry once when `@pgmq.produce*` raises `PGMQ::Errors::ConnectionError` with a message indicating the pooled `PG::Connection` was killed beneath pgmq-ruby — typically by PgBouncer hitting `server_idle_timeout` / `client_idle_timeout`, an admin disconnect, or a TCP RST. Observed in production as `PQsocket() can't get socket descriptor` on the first produce following an idle window. pgmq-ruby's `auto_reconnect` recovers on the *next* pool checkout, so a single retry is sufficient; non-stale errors (pool timeout, misconfiguration, unreachable database) still propagate unchanged. Upstream pgmq-ruby fix for the underlying misclassification is in-flight at mensfeld/pgmq-ruby#94.
|
|
112
|
+
- **Stale-connection retry now backs off and attempts twice.** The single immediate retry above recovers one dead pooled socket, but during a PgBouncer restart or a brief failover window the immediate second attempt lands in the same unavailable window and the error propagated — failing an enqueue the caller may never retry. `Pgbus::Client#with_stale_connection_retry` now retries a matched stale error up to `STALE_RETRY_ATTEMPTS` (2) times with a short backoff between attempts (`STALE_RETRY_DELAYS = [0.1, 0.5]`), giving a transient window time to clear. Safety is unchanged: the pattern list still matches only pre-flight/idle-socket errors (no SQL was sent), so retrying non-idempotent ops stays safe. The backoff runs error-path only — never on the success path — and the sleep is issued outside `synchronized`, so `@pgmq_mutex` is never held during a delay on the shared-connection path. No happy-path throughput or allocation change (`client_bench` identical before/after). Refs #196.
|
|
113
|
+
- **Stalled dispatcher and scheduler are no longer reported healthy on the processes page.** `Web::DataSource#derive_process_status` evaluated the loop beacon only for workers, returning `:healthy` for every other kind, and neither the dispatcher (`{ pid }`) nor the scheduler (`{ pid, tasks }`) heartbeat carried a beacon. Because heartbeats run on an independent `Concurrent::TimerTask` thread, a dispatcher wedged inside `run_maintenance` (e.g. a stuck vacuum in `run_table_maintenance`) or a scheduler stuck in `tick` kept heartbeating and showed healthy forever while doing no work. Both processes now stamp a wall-clock loop beacon (`@loop_tick_at`, a `Concurrent::AtomicReference`) at the top of each main-loop iteration and pass `loop_tick_supplier:` to their heartbeat, mirroring `Worker#stamp_loop_tick`; the heartbeat persists it as `metadata.loop_tick_at` on every beat. `derive_process_status` now evaluates the beacon for `"dispatcher"` and `"scheduler"` too, with kind-aware thresholds that account for each loop's sleep — workers keep `config.stall_threshold` (90s), dispatcher uses `stall_threshold + config.dispatch_interval`, scheduler uses `stall_threshold + config.recurring_schedule_interval` — so a beacon older than the threshold renders the `:stalled` badge. A missing beacon (an older process during a rolling deploy) stays `:healthy`, and `stall_threshold = nil` disables the check as before. Worker stall detection is unchanged. Refs #222.
|
|
114
|
+
- **Config hardening: three silent-failure holes closed.** (1) `Configuration#validate!` now validates `statsd_host`/`statsd_port`/`health_bind` unconditionally alongside the other keys added in the same window — a typo'd host or an out-of-range port previously passed boot and only failed later when the UDP socket opened or the health server bound. (2) `ConfigLoader.sectioned?` no longer misclassifies a legitimate flat `pgbus.yml` whose every top-level setting happens to be Hash-valued (`connection_params:`, `streams_retention:`, `recurring_tasks:`, …) as "sectioned" — it now also checks whether any top-level key is a real `Configuration` setter (env section names like `production` never are), and `ConfigLoader.load` logs a warning naming the current env and the available sections whenever a genuinely sectioned file has no matching section, instead of silently applying nothing. (3) `Pgbus::Generators::ConfigConverter::DEPRECATED_SETTINGS` no longer lists `pool_size` — it is a live `Configuration#pool_size` accessor operators set to override auto-tuning, not a removed setting, so `rails generate pgbus:update` no longer silently drops a deliberate override when converting legacy YAML. Refs #276.
|
|
115
|
+
|
|
116
|
+
## [0.9.7] - 2026-06-29
|
|
117
|
+
|
|
118
|
+
### Security
|
|
119
|
+
|
|
120
|
+
- **Dashboard: filter sensitive data in payload display.** New `Pgbus::Web::PayloadFilter` redacts sensitive keys (password, token, secret, api_key, etc.) as `[FILTERED]` in rendered job payloads; configurable via `dashboard_filter_parameters` and `dashboard_filter_sensitive`. Refs #186.
|
|
121
|
+
|
|
122
|
+
## [0.9.6] - 2026-06-28
|
|
123
|
+
|
|
124
|
+
### Added
|
|
125
|
+
|
|
126
|
+
- **AppSignal: magic dashboard definitions for auto-detected metrics.** Ships `dashboard.json` (main overview) plus health, throughput, and streams dashboards in `public_config` format so AppSignal auto-creates pgbus dashboards. Refs #185.
|
|
127
|
+
- **Metrics: `pgbus_queue_latency` gauge (oldest message age in ms) and `hostname` tag on all probe gauges.** Hostname tagging matches the Sidekiq/Puma convention. Refs #185.
|
|
128
|
+
|
|
129
|
+
### Changed
|
|
130
|
+
|
|
131
|
+
- **Dashboard: insights default time range is now 1h (with a new 4h option) instead of 30d,** better suited to post-deploy monitoring. Refs #184.
|
|
132
|
+
- **Dashboard: DLQ page gains pagination** with "Showing X–Y of N", prev/next controls, and a total-count badge. Refs #184.
|
|
133
|
+
- **Dashboard: UX polish** — Failed/DLQ stats card labels each number, truncated error messages show full text on hover, and queue detail pages get back-to-queues navigation. Refs #184.
|
|
134
|
+
|
|
135
|
+
## [0.9.5] - 2026-06-28
|
|
136
|
+
|
|
137
|
+
### Added
|
|
138
|
+
|
|
139
|
+
- **MCP: read-only MCP diagnostic server.** `pgbus mcp` (stdio) exposes 12 read-only tools over `Web::DataSource` (headline `pgbus_health` returns OK/DEGRADED/STALLED); payloads redacted by default, optional `PGBUS_MCP_TOKEN` gate. Refs #182.
|
|
140
|
+
- **Worker: liveness probe to detect and heal stalled claim loops.** Workers stamp a `loop_tick_at` beacon; a supervisor watchdog SIGKILLs and re-forks workers whose loop stalls past `stall_threshold` (default 90s), and client reads are wrapped in `read_timeout` (default 30s) raising `Pgbus::ReadTimeoutError`. Adds a three-state process status (healthy/stalled/stale) to the dashboard. Refs #181, #179.
|
|
141
|
+
|
|
142
|
+
## [0.9.4] - 2026-06-24
|
|
143
|
+
|
|
144
|
+
### Added
|
|
145
|
+
|
|
146
|
+
- **Streams: forward `exclude:`/`visible_to:`/`event:` through Turbo broadcast helpers.** Turbo `broadcast_*_to` helpers now thread all four pgbus opts (durable, exclude, visible_to, event) to `Pgbus.stream.broadcast`, completing the actor-echo suppression loop for gems that broadcast via standard Turbo helpers. Refs #178.
|
|
147
|
+
|
|
148
|
+
## [0.9.3] - 2026-06-22
|
|
149
|
+
|
|
150
|
+
### Added
|
|
151
|
+
|
|
152
|
+
- **Worker: NOTIFY-gated worker wakeups.** Each worker fork owns a dedicated PG connection (`NotifyListener`) that LISTENs on its queues' INSERT channels and wakes the worker immediately on insert instead of waiting for the next polling tick. On by default when `listen_notify` is enabled; configurable via `worker_notify_wakeup`, `worker_notify_host`, `worker_notify_port`, `worker_notify_database_url`. Refs #174.
|
|
153
|
+
|
|
154
|
+
### Fixed
|
|
155
|
+
|
|
156
|
+
- **Worker: prevent silent poll stall.** When `evict_missing_queues` removed all queues from a worker's list, it left an empty list that silently skipped every read forever; the list now recovers from the initial configured set. Added diagnostic logging when circuit-breaker filtering removes all active queues (previously silent), and mutex-guarded `NotifyListener` shared state with hardened reconnect. Refs #174.
|
|
157
|
+
|
|
158
|
+
## [0.9.2] - 2026-06-22
|
|
159
|
+
|
|
160
|
+
### Added
|
|
161
|
+
|
|
162
|
+
- **Streams: reactive primitives for phlex-reactive.** Idempotent `stream_key` for pre-built keys (+ `stream_key!`), actor-echo suppression via `exclude:` on broadcasts, `Stream#broadcast_render` (render + broadcast in one call), client-exposed `msg_id` revision for optimistic UI (`pgbus:message`), optional connection-driven presence (`streams_presence_patterns`), typed SSE `event:` names, and publish-side coalescing for high-frequency broadcasts (`coalesce:`). Refs #165, #166, #167, #168, #169, #170, #171, #173.
|
|
163
|
+
|
|
164
|
+
### Security
|
|
165
|
+
|
|
166
|
+
- **Streams: sanitize typed SSE event names.** `Envelope.message` strips CR/LF from `event` and the dispatcher drops non-string/blank/CRLF values, preventing forged `id:`/`data:` SSE field injection via a crafted event name. Refs #173.
|
|
167
|
+
|
|
168
|
+
### Fixed
|
|
169
|
+
|
|
170
|
+
- **Async: replace blocking sleep with IO pipe wake in reactor.** The reactor no longer polls with `Kernel#sleep`, which starved the Async scheduler and silently stalled all queue processing ~20-30s after boot under load; `post`/`shutdown`/`restore_capacity` now write a wake byte for sub-millisecond fiber-aware wake. Refs #174.
|
|
171
|
+
- **Dashboard: add `turbo_frame: _top` to batch links.** Batch links now break out of the enclosing Turbo frame. Refs #172.
|
|
172
|
+
|
|
173
|
+
## [0.9.0] - 2026-06-19
|
|
174
|
+
|
|
175
|
+
### Added
|
|
176
|
+
|
|
177
|
+
- **PGMQ v1.11 features via pgmq-ruby upgrade.** `Client` gains grouped reads (`read_grouped`, `read_grouped_rr`, `read_grouped_head`), FIFO index management (`create_fifo_index`, `create_fifo_indexes_all`), `wait_for_notify`, notify-insert throttle controls, and `convert_archive_partitioned` (pg_partman); new `pgbus:queues` / `pgbus:archives` rake tasks and group-mode config. Refs #161.
|
|
178
|
+
- **Dashboard Batches tab.** New `BatchesController` (index + show) with progress bars, status badges, job counts, and callback info; `Web::DataSource` batch methods, a `pgbus_batch_status_badge` helper, a nav link between Events and DLQ, and i18n for all 12 locales. Refs #162.
|
|
179
|
+
|
|
180
|
+
### Changed
|
|
181
|
+
|
|
182
|
+
- **Pin released pgmq-ruby v0.7.0** (`~> 0.7.0`), dropping the git-master pin. `Client#tune_autovacuum` now delegates to pgmq-ruby's `tune_autovacuum`, applying the same queue/archive vacuum and fillfactor settings in one pooled checkout. Refs #163.
|
|
183
|
+
|
|
184
|
+
## [0.8.4] - 2026-05-21
|
|
185
|
+
|
|
186
|
+
### Added
|
|
187
|
+
|
|
188
|
+
- **Streams: `streams_host` / `streams_port` / `streams_database_url` connection overrides.** Point only the Streamer's dedicated LISTEN/NOTIFY connection at a direct Postgres port while the rest of pgbus stays on a transaction-pool pooler (e.g. PgBouncer, PlanetScale Postgres), which otherwise silently drops live SSE delivery; `streams_database_url` wins over the host/port surgicals. Refs #160.
|
|
189
|
+
|
|
190
|
+
## [0.8.3] - 2026-05-20
|
|
191
|
+
|
|
192
|
+
### Fixed
|
|
193
|
+
|
|
194
|
+
- **Streams: release ActiveRecord connections after each dispatcher iteration.** The `StreamEventDispatcher` fiber kept its AR connection leased while parked on `@queue.pop`, so the next code reload's `clear_reloadable_connections!` blocked for the full rack-timeout (10s); it now calls `clear_active_connections!` in an ensure block per loop iteration. Refs #158.
|
|
195
|
+
|
|
196
|
+
## [0.8.2] - 2026-05-14
|
|
197
|
+
|
|
198
|
+
### Added
|
|
199
|
+
|
|
200
|
+
- **Streams: per-Turbo-helper durable opt-in.** All Turbo broadcast helpers now accept a `durable:` kwarg (e.g. `broadcasts_to :account, durable: true`, `broadcast_replace_to(record, durable: true)`) to control durable mode at the call site; omitting it falls back to `streams_durable_patterns` → `streams_default_broadcast_mode`. Refs #154, #153.
|
|
201
|
+
- **Streams: in-memory counters and auto-enabled stats for ephemeral broadcasts.** Adds always-on `StreamCounter` (broadcasts, active/total connections per stream, via `Concurrent::AtomicFixnum`) surfaced as "Live Stream Counters" in the Insights dashboard, and ephemeral broadcasts now always write `pgbus_stream_stats` regardless of `streams_stats_enabled`. Refs #156.
|
|
202
|
+
|
|
203
|
+
### Fixed
|
|
204
|
+
|
|
205
|
+
- **Streams: ephemeral broadcasts now reach browsers.** Ephemeral envelopes (negative `msg_id`) bypassed the durable-replay cursor check so every fresh SSE connection (since_id=0) rejected them; they now skip the cursor filter and do not advance `last_msg_id_sent`. Also fixes duplicate `DisconnectMessage`s over-decrementing `active_connections` by only decrementing when `Registry#unregister` actually removes a connection. Refs #157.
|
|
206
|
+
- **Client: wrap `notify_stream` in `with_stale_connection_retry`.** Stream wake-up notifications now survive a stale database connection. Refs #155.
|
|
207
|
+
|
|
208
|
+
## [0.8.1] - 2026-05-08
|
|
209
|
+
|
|
210
|
+
### Added
|
|
211
|
+
|
|
212
|
+
- **Streams: per-broadcast and per-stream durable opt-in.** `stream.broadcast(html, durable: true/false)` overrides the mode for a single call, and `config.streams_durable_patterns` (Array of exact strings or Regexps) resolves the mode for `Pgbus.stream(name)` before falling back to `streams_default_broadcast_mode`; `Pgbus.stream`'s `durable:` default changed from `true` to `nil` so the resolver runs. Refs #152.
|
|
213
|
+
|
|
214
|
+
## [0.8.0] - 2026-05-08
|
|
215
|
+
|
|
216
|
+
### Fixed
|
|
217
|
+
|
|
218
|
+
- **Streams: Turbo broadcasts default to ephemeral mode (PG NOTIFY only), preventing orphan PGMQ queues from accumulating unread messages.** Adds `durable: true/false` on `Stream#broadcast`, an hourly dispatcher sweep of empty stream queues, batched dashboard queue metrics, and `streams_default_broadcast_mode`/`streams_orphan_threshold`/`streams_orphan_sweep_interval` config. Refs #151, #149.
|
|
219
|
+
- **Streams: inline SSE `<script>` tag now carries a CSP nonce when available, fixing broken real-time updates under `script-src` nonce policies.** Falls back gracefully when the nonce is unavailable. Refs #150.
|
|
220
|
+
|
|
221
|
+
## [0.7.9] - 2026-05-06
|
|
222
|
+
|
|
223
|
+
### Added
|
|
224
|
+
|
|
225
|
+
- **Integrations: native AppSignal integration.** Passive `pgbus.*` subscriber (ActiveJob runs + event-bus handler invocations as background-job transactions, counters, distributions), a minutely probe reporting queue depth, oldest-message age, DLQ size, failed-events, dead tuples, MVCC horizon age, active processes, and stream stats, plus three importable dashboards (throughput/latency, health, streams). Gated on `defined?(::Appsignal)` via an Engine initializer; opt out with `config.appsignal_enabled` / `config.appsignal_probe_enabled`. Refs #148.
|
|
226
|
+
- **Instrumentation: new `AS::Notifications` events** — `pgbus.job_dead_lettered`, `pgbus.event_failed`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`. Existing executor/handler instrument calls enriched with `job_id`, `provider_job_id`, `enqueued_at`, `arguments`, `routing_key`, `exception_object`; usable by Datadog/New Relic/OTel subscribers too. Refs #148.
|
|
227
|
+
|
|
228
|
+
## [0.7.8] - 2026-04-29
|
|
229
|
+
|
|
230
|
+
### Fixed
|
|
231
|
+
|
|
232
|
+
- **Event bus: `Handler#process` now runs inside the Rails executor** so ActiveRecord connections are checked back into the pool after each handler run, preventing connection leaks under sustained event processing. Refs #147.
|
|
233
|
+
|
|
234
|
+
## [0.7.7] - 2026-04-25
|
|
235
|
+
|
|
236
|
+
### Fixed
|
|
237
|
+
|
|
238
|
+
- **Executor: jobs now pick up code changes in development.** The executor wraps jobs in `Rails.application.reloader.wrap` in dev/test (and `executor.wrap` in production) instead of always using `executor.wrap`, so workers no longer run stale class definitions. Supports `enable_reloading` (Rails 7.1+) and `cache_classes` (older). Refs #146, closes #145.
|
|
239
|
+
|
|
240
|
+
## [0.7.6] - 2026-04-23
|
|
241
|
+
|
|
242
|
+
### Fixed
|
|
243
|
+
|
|
244
|
+
- **Client: wrap all PGMQ calls in stale-connection retry.** Added SSL-layer EOF variants ("PQconsumeInput() SSL error: unexpected eof while reading", "SSL SYSCALL error") to `STALE_CONNECTION_PATTERNS`, and extended `with_stale_connection_retry` to every `@pgmq.*` call site — reads (consumer path), ack/nack (delete/archive/set_vt), admin/dashboard ops (metrics/list/purge/drop), `ensure_queue`, and stream setup — so transient Postgres/SSL drops no longer crash worker loops or leak job redeliveries. Refs #143, #144.
|
|
245
|
+
|
|
246
|
+
## [0.7.5] - 2026-04-19
|
|
247
|
+
|
|
248
|
+
### Added
|
|
249
|
+
|
|
250
|
+
- **Maintenance: proactive table maintenance to reduce PGMQ bloat.** `fillfactor=70` on queue tables, dispatcher-driven targeted `VACUUM` via `pg_stat_user_tables` above a 10% dead-tuple threshold, and `REINDEX TABLE CONCURRENTLY`; existing installs upgrade via `rails generate pgbus:tune_fillfactor`. Refs #141.
|
|
251
|
+
- **Recurring: support multiple recurring task files.** Refs #142.
|
|
252
|
+
|
|
253
|
+
### Fixed
|
|
254
|
+
|
|
255
|
+
- **Client: retry enqueue once when a pooled pgmq connection was killed.** `send_message`, `send_batch`, and `publish_to_topic` now rescue `PGMQ::Errors::ConnectionError` and retry once when the pooled `PG::Connection` was closed server-side (e.g. behind PgBouncer); other variants propagate unchanged. Refs #140.
|
|
256
|
+
|
|
257
|
+
## [0.7.4] - 2026-04-14
|
|
258
|
+
|
|
259
|
+
### Added
|
|
260
|
+
|
|
261
|
+
- **Dashboard: pending event management.** Discard, edit-and-retry, reroute, mark-handled, and bulk-discard stuck/misrouted events in handler queues from the Events page. Refs #138.
|
|
262
|
+
|
|
263
|
+
### Fixed
|
|
264
|
+
|
|
265
|
+
- **Event bus: embed `routing_key` in the Publisher message body** so non-wildcard pattern handlers are actually invoked; previously they were silently skipped. Refs #137, closes #136.
|
|
266
|
+
|
|
267
|
+
## [0.7.3] - 2026-04-12
|
|
268
|
+
|
|
269
|
+
### Added
|
|
270
|
+
|
|
271
|
+
- **Testing: EventBus test primitives.** `Pgbus::Testing` with `:fake`/`:inline`/`:disabled` modes, a thread-safe `EventStore`, shared assertions (`assert_pgbus_published`, `perform_published_events`), RSpec `have_published_event` matchers (`.with_payload`/`.with_headers`/`.exactly`), and Minitest auto-setup/teardown; Zeitwerk-ignored so it never leaks into production. Refs #132, #131.
|
|
272
|
+
|
|
273
|
+
### Fixed
|
|
274
|
+
|
|
275
|
+
- **Streams: `streams_test_mode` prevents rack.hijack DB connection leaks in tests.** StreamApp returns a stub SSE response without hijacking or spawning polling threads; entering EventBus `fake!`/`inline!` toggles it automatically and restores it on scoped-block exit. Refs #134, #133.
|
|
276
|
+
|
|
277
|
+
## [0.7.2] - 2026-04-11
|
|
278
|
+
|
|
279
|
+
### Added
|
|
280
|
+
|
|
281
|
+
- **Observability: executor lifecycle logging and zombie message detection.** The ActiveJob executor now emits lifecycle log lines and the worker detects zombie messages, with a new configuration option and failed-event recording. Refs #130.
|
|
282
|
+
|
|
283
|
+
## [0.7.1] - 2026-04-11
|
|
284
|
+
|
|
285
|
+
### Fixed
|
|
286
|
+
|
|
287
|
+
- **Executor: fiber interrupts no longer silently leak uniqueness locks.** Under `execution_mode: :async` with `:fiber` isolation, `Async::Stop`/`Async::Cancel` (which inherit from `Exception`, not `StandardError`) propagated past the `rescue StandardError` in `Executor#execute` and `AsyncPool#perform`, skipping `release_lock` and stranding the key in `pgbus_uniqueness_keys` until VT expiry. Both rescues now catch `Exception` (still re-raising `SystemExit`/`Interrupt`/`SignalException`/`NoMemoryError`/`SystemStackError`), producing a visible `:failed` result, a `pgbus_failed_events` row, and a `pgbus.job_failed` notification; debug-level phase markers added for diagnosis. Refs #126, #129.
|
|
288
|
+
- **Bootstrap: `enable_notify_insert` is now idempotent to prevent fork deadlocks.** Multiple forked children racing `bootstrap_queues` triggered concurrent DROP+CREATE of the pgmq notify trigger under `AccessExclusiveLock`, deadlocking on the same queue table. A new `notify_trigger_current?` check skips the destructive path when the trigger already exists at the correct interval, and the supervisor now bootstraps queues in the parent before forking. Refs #125, #128.
|
|
289
|
+
|
|
290
|
+
## [0.7.0] - 2026-04-11
|
|
291
|
+
|
|
292
|
+
### Added
|
|
293
|
+
|
|
294
|
+
- **Streams: short-name helpers and tighter queue-name budget.** New `Pgbus::Streams::Key` (`stream_key`, `short_id`, `normalize`, hashing AR models to a 64-bit SHA-256 suffix), `Pgbus::Streams::Streamable` AR mixin, and top-level `Pgbus.stream_key(*parts)`; `MAX_QUEUE_NAME_LENGTH` dropped to 47 to match pgmq-ruby's runtime ceiling, and over-long names now raise `Pgbus::Streams::StreamNameTooLong` at stream construction before PGMQ is touched. Refs #120.
|
|
295
|
+
- **Error reporters: route caught exceptions to APM services.** New `Pgbus::ErrorReporter` and `config.error_reporters` array wired into all critical rescue sites (executor, worker, dispatcher, circuit breaker, supervisor, outbox, failed-event recording) so exceptions can reach Appsignal/Sentry instead of only being logged. Refs #121.
|
|
296
|
+
- **JSON log formatter and structured logging.** New `Pgbus::LogFormatter` with Text and JSON formatters; set `config.log_format = :json` for structured lines with separate timestamp/pid/tid/severity/component fields plus thread-local `ctx`. Refs #121.
|
|
297
|
+
|
|
298
|
+
### Fixed
|
|
299
|
+
|
|
300
|
+
- **Worker: cap `read_multi` total so multi-queue fetch cannot overflow the pool.** pgmq-ruby treats `qty:` as per-queue, so multi-queue capsules could fetch `queue_count * qty` messages and overflow the fixed-size execution pool, crash-looping the worker fork; `limit:` is now threaded through `Client#read_multi` and capped at pool capacity. Refs #124, #123.
|
|
301
|
+
|
|
302
|
+
## [0.6.9] - 2026-04-11
|
|
303
|
+
|
|
304
|
+
### Fixed
|
|
305
|
+
|
|
306
|
+
- **Generators: `--database` routing now honors `migrations_paths`.** All 14 migration generators previously hardcoded `db/pgbus_migrate/`, ignoring the path configured in `database.yml`; they now delegate to Rails' `db_migrate_path` via a shared `MigrationPath` module. Refs #119.
|
|
307
|
+
- **Schema: autovacuum tuning is reapplied after `db:schema:load`.** Ruby-format `schema.rb` drops `ALTER TABLE ... SET (reloptions)`, so fresh setups lost queue-table tuning; a new idempotent `pgbus:tune_autovacuum` task is hooked into `db:schema:load` (and `db:schema:load:pgbus`) to restore it. Refs #119.
|
|
308
|
+
|
|
309
|
+
## [0.6.8] - 2026-04-11
|
|
310
|
+
|
|
311
|
+
### Added
|
|
312
|
+
|
|
313
|
+
- **Dashboard: queue health observability and worker pool utilization.** Surfaces PostgreSQL vacuum stats, dead tuple counts, per-table bloat, MVCC horizon age, and worker pool utilization in the dashboard, queue detail views, and Prometheus endpoint (new `pgbus_table_dead_tuples`, `pgbus_table_bloat_ratio`, `pgbus_table_last_vacuum_age_seconds`, `pgbus_oldest_transaction_age_seconds`, `pgbus_worker_pool_capacity`, `pgbus_worker_pool_busy`, `pgbus_worker_pool_utilization` gauges). Refs #117.
|
|
314
|
+
- **Schema: autovacuum tuning for high-churn PGMQ and pgbus tables.** New `Pgbus::AutovacuumTuning` module and `pgbus:tune_autovacuum` generator apply aggressive vacuum settings to queue, archive, `pgbus_semaphores`, `pgbus_uniqueness_keys`, and `pgbus_processed_events` tables; applied on queue creation and auto-detected by `pgbus:update`. Refs #117.
|
|
315
|
+
|
|
316
|
+
### Fixed
|
|
317
|
+
|
|
318
|
+
- **i18n: fill in missing Swedish dashboard translations.** Refs #118.
|
|
319
|
+
|
|
320
|
+
## [0.6.7] - 2026-04-11
|
|
321
|
+
|
|
322
|
+
### Fixed
|
|
323
|
+
|
|
324
|
+
- **Streams: `pgbus_stream_from` now works from Rack middleware contexts.** `TurboStreamOverride` includes `Pgbus::StreamsHelper` so the helper is reachable when host tools (e.g. hotwire-livereload) call it via `ActionController::Base.helpers` outside engine views. Refs #116.
|
|
325
|
+
- **Streams: `stream_source_element.js` auto-loads for host apps.** The JS ships in `app/assets/javascripts/pgbus/` (Propshaft/Sprockets) with engine auto-precompile and importmap auto-pin, and `pgbus_stream_from` emits a module import tag so the `<pgbus-stream-source>` element activates and opens its SSE connection. Refs #116.
|
|
326
|
+
- **Streams: script + element output preserves `html_safe`.** Concatenation no longer collapses to a plain String, so the tags render instead of being HTML-escaped as visible text. Refs #116.
|
|
327
|
+
- **Streams: asset-pipeline JS imports from `@hotwired/turbo-rails`.** Matches the importmap pin name, fixing the browser `TypeError` from the unmapped `@hotwired/turbo` specifier. Refs #116.
|
|
328
|
+
- **Streams: SSE reconnect bugs fixed.** Reattached elements reconnect (`closed` reset in `connectedCallback`), a gracefully-ended fetch stream falls through to `EventSource`, and the initial `EventSource` URL carries a `?since=` watermark since the constructor cannot send `Last-Event-ID`. Refs #116.
|
|
329
|
+
|
|
330
|
+
## [0.6.6] - 2026-04-10
|
|
331
|
+
|
|
332
|
+
### Fixed
|
|
333
|
+
|
|
334
|
+
- **Streams: `turbo_stream_from` now uses the pgbus SSE transport.** `TurboStreamOverride` prepends onto `Turbo::StreamsHelper` and delegates `turbo_stream_from` to `pgbus_stream_from` when `streams_enabled`, so third-party gems (e.g. hotwire-livereload) subscribe over PGMQ/SSE instead of ActionCable — matching the publish side and delivering broadcasts. Falls through to turbo-rails when streams are disabled. Refs #115.
|
|
335
|
+
|
|
336
|
+
## [0.6.5] - 2026-04-10
|
|
337
|
+
|
|
338
|
+
### Added
|
|
339
|
+
|
|
340
|
+
- **Process: async/fiber execution mode for workers.** New `ExecutionPools` abstraction (`ThreadPool` + `AsyncPool`) runs jobs as bounded fibers on a single async reactor thread instead of a thread pool, cutting PostgreSQL connection usage for I/O-bound workloads; configurable via `config.execution_mode` (with per-worker override) and the CLI `--execution-mode` flag. Refs #112.
|
|
341
|
+
- **Streams: Falcon-native streaming body support.** New `streams_falcon_streaming_body` flag makes `StreamApp` return a `Protocol::HTTP::Body::Writable` instead of hijacking the socket, giving Falcon proper fiber-driven SSE streaming (takes priority over `rack.hijack?`). Refs #113.
|
|
342
|
+
- **Config: auto-tune pool size for async workers.** Pool-size auto-tuning now contributes 3 connections per async worker (reactor + polling + headroom) instead of one per fiber, and honors a global `config.execution_mode = :async` fallback. Refs #114.
|
|
343
|
+
|
|
344
|
+
## [0.6.4] - 2026-04-10
|
|
345
|
+
|
|
346
|
+
### Added
|
|
347
|
+
|
|
348
|
+
- **Streams: `config.streams_path` decouples the SSE endpoint URL.** Point `<pgbus-stream-source>` elements at a public route when the engine is mounted behind an auth constraint; resolution order is `config.streams_path` > engine route helper > fallback. Refs #111.
|
|
349
|
+
|
|
350
|
+
## [0.6.3] - 2026-04-10
|
|
351
|
+
|
|
352
|
+
### Added
|
|
353
|
+
|
|
354
|
+
- **Retry: exponential backoff for VT-based retries.** Failed jobs now back off exponentially via the visibility timeout instead of retrying at a fixed interval. Refs #110.
|
|
355
|
+
- **Web: Prometheus metrics export endpoint.** New endpoint exposes pgbus metrics in Prometheus format for scraping. Refs #109.
|
|
356
|
+
|
|
357
|
+
### Changed
|
|
358
|
+
|
|
359
|
+
- **Streamer: renamed `Web::Streamer::Dispatcher` to `StreamEventDispatcher`.** Disambiguates from `Process::Dispatcher`; log tags shift from `[Pgbus::Streamer::Dispatcher]` to `[Pgbus::Streamer::StreamEventDispatcher]` — update pinned log filters. Refs #98, #107.
|
|
360
|
+
|
|
361
|
+
### Fixed
|
|
362
|
+
|
|
363
|
+
- **Streams: normalize queue names with hyphens/dots instead of rejecting them.** Refs #108.
|
|
364
|
+
- **Streams: case-insensitive match for a missing PGMQ stream queue.** Refs #106.
|
|
365
|
+
- **Stats: JobStat memoization survives transient connection errors.** A single PG blip during boot no longer permanently disables job stat and latency recording for the process lifetime — only successful probes are memoized. Refs #98, #105.
|
|
366
|
+
|
|
367
|
+
## [0.6.2] - 2026-04-09
|
|
368
|
+
|
|
369
|
+
### Fixed
|
|
370
|
+
|
|
371
|
+
- **Streams: handle a missing PGMQ queue table on the first watermark read.** `Client::ReadAfter#stream_current_msg_id`, `#stream_oldest_msg_id`, and `#read_after` now rescue `PG::UndefinedTable` for the stream's own queue/archive table — treating "no queue" as "no messages" (watermark 0, oldest nil, empty replay) — so first-page renders of ephemeral per-subject streams no longer raise before the first broadcast lazily creates the table. Refs #101, #103.
|
|
372
|
+
|
|
373
|
+
## [0.6.1] - 2026-04-09
|
|
374
|
+
|
|
375
|
+
### Added
|
|
376
|
+
|
|
377
|
+
- **Generators: `pgbus:update` auto-detects and installs missing migrations.** Inspects the live DB schema against known pgbus tables/columns/indexes and invokes the matching sub-generators (e.g. `add_presence`, `add_stream_stats`) in one pass; auto-passes `--database=<name>` when a separate database is configured. Refs #100.
|
|
378
|
+
|
|
379
|
+
### Changed
|
|
380
|
+
|
|
381
|
+
- **Web: dashboard pagination and failed-archive pushed to SQL.** Moves paging and the failed-events archive off in-memory handling and into SQL queries. Refs #95.
|
|
382
|
+
|
|
383
|
+
### Fixed
|
|
384
|
+
|
|
385
|
+
- **Streams: rename `StreamStat` scopes to avoid turbo-rails collision.** `broadcasts`/`connects`/`disconnects` renamed to `broadcast_events`/`connect_events`/`disconnect_events`; the `broadcasts` scope collided with `Turbo::Broadcastable`'s class macro and crashed eager-load in any app combining streams with turbo-rails. `.summary` keys unchanged. Refs #94.
|
|
386
|
+
- **Generators: `pgbus:update` no longer leaks env-specific settings across all environments.** A setting present in only some environments (e.g. `polling_interval: 0.01` under `test:`) was emitted as an unconditional initializer line, applying test tuning to dev and prod; now classified constant only when present in every env and agreeing. Refs #96.
|
|
387
|
+
|
|
388
|
+
## [0.6.0] - 2026-04-09
|
|
389
|
+
|
|
390
|
+
### Added
|
|
391
|
+
|
|
392
|
+
- **Streams: SSE-based turbo-rails replacement.** New `Pgbus::Web::Streamer` subsystem — a threaded coordinator inside each Puma worker delivering Turbo Streams over SSE, fixing page-born-stale (rails/rails#52420) and lost-messages-on-reconnect (hotwired/turbo#1261). Adds 11 `streams_*` configuration options, `streams_helper`, `TurboBroadcastable`, watermark cache middleware, and a Puma plugin. Refs #73, #75, #76, #77.
|
|
393
|
+
- **Streams: transactional broadcasts.** Broadcasts enqueued inside a DB transaction only fire on commit. Refs #81.
|
|
394
|
+
- **Streams: Falcon server support.** `stream_app` runs under Falcon in addition to Puma. Refs #83.
|
|
395
|
+
- **Streams: `broadcasts_with_replay` via `replay:` helper option.** Late subscribers receive missed messages on connect. Refs #84.
|
|
396
|
+
- **Streams: server-side audience filtering.** New `Pgbus::Streams::Filters` scopes broadcasts per connection at the dispatcher. Refs #85.
|
|
397
|
+
- **Streams: presence tracking.** New `Pgbus::Streams::Presence` plus `add_presence` generator and migration. Refs #87.
|
|
398
|
+
- **Streams: opt-in stream stats + Insights integration.** New `StreamStat` model surfaced in the Insights dashboard. Refs #91.
|
|
399
|
+
|
|
400
|
+
### Changed
|
|
401
|
+
|
|
402
|
+
- **Streams: wake coalescing + Stream caching.** Dispatcher coalesces wake-ups and caches Stream lookups; adds a `streams_bench` benchmark harness. Refs #88.
|
|
36
403
|
|
|
37
404
|
## [0.5.1] - 2026-04-08
|
|
38
405
|
|