pg_reports 0.9.1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e2063c9302c5b737dc5272ef382242464d7f11d1dbc7d926bd5db0b55a842b54
-  data.tar.gz: 61ca606a24d3e6fed362acb58270a2c03526803c18c7c89e2e31cd13cc35f3ad
+  metadata.gz: cbcba1cda9a70325b903c293f961716b5658bd865ec9a34dcb3be6f6761df177
+  data.tar.gz: d1289877fd1f6732e5017ee565537c7956fca3fd828188dfb5b5a76faaafddc5
 SHA512:
-  metadata.gz: bf7867350c74791dfb0a36ec2ad013e7f42ef5186f8a29209472599e36ca147b76117039f2009bac86d456370e41bca9d49e45a4b6c51ae34d0829bf1ae0d300
-  data.tar.gz: 75910cc42fc73da32e88f2161b10054ffe1775e8cbfce8085c488d419355eacc3b8a5ed42af1b9149198c64f4a252a6ee0178aeb99ccc2b63fba4a7847c40466
+  metadata.gz: 719d9ec8f59ae5307fcf8bd4a63d83231b1c12ef68ea6f0c1d37a5ce0225b1fc7289b6ee27515e0f7175e4377912894c3d6149a0964a323b8efb95b8e4ad8171
+  data.tar.gz: caf8cb61ea270241f69dc9eda2cd812d00979dda507f313d4472abb549dd8ef3cd214f834ee37ede93fa1970a446af8adac35844b8d89dfd02dd6982be93a537

data/CHANGELOG.md

@@ -7,6 +7,78 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.0.0] - 2026-11-07
+
+### Upgrading
+
+- **Rails 6.1 is now the minimum.** Older versions were declared but never tested.
+- **The dashboard and `/metrics` answer only local requests without credentials, and none in production.** Set `dashboard_auth`, or `config.dashboard_auth = :host_managed` if routing already protects them. A Prometheus on another host, including Docker, needs `grafana_metrics_token`.
+- **The standalone server no longer loads `./pg_reports.rb` or `config/pg_reports.rb` implicitly.** Pass `--config`.
+- **Reports stop after 30 seconds, or after 3 seconds waiting for a lock.** Raise `report_statement_timeout_ms` if a report needs longer.
+- **`pg_reports_issues` will report problems it has been missing.** Review alerts built on it.
+- **CSV exports may start a cell with an apostrophe.** Anything that parses them should expect it.
+
+### Known issues
+
+- `bloated_indexes` overstates bloat: it puts a freshly built index at 65-75%. It has no thresholds until the estimate improves, so read its numbers as a rough ordering.
+
+### Security
+
+- **CSV exports could run formulas from the database.** A table name, query or source path such as `=HYPERLINK(...)` went into report and Query Monitor exports unchanged, and a spreadsheet evaluated it when the file was opened. Such cells now start with an apostrophe, which makes them text. Numbers are unchanged.
+- **The rate limit could be switched off, or lock a client out for good.** It read and wrote its counter separately, so concurrent requests lost counts; every refused request restarted the window, so a client that kept retrying never recovered; on a `NullStore` it did nothing, and when the cache failed it let everything through. It now uses the cache's atomic increment in fixed windows, keeps counting per process when the cache cannot, and answers `429` with `Retry-After`. It also covers `reset_statistics`, `enable_pg_stat_statements`, `send_to_telegram`, `start_query_monitoring` and `load_query_history`.
+- **Captured SQL held secrets in the clear.** The Query Monitor kept string literals, so passwords, tokens and personal data reached the panel, its exports and `log/pg_reports.log`, which was world-readable and grew without limit. Outside development, `query_monitor_redact_literals` now replaces them with `'?'`, including a literal cut off by a length limit. The log is created with mode `0600`, an existing one is tightened, and it rotates at `query_monitor_log_max_bytes` (10 MB), keeping one older file.
+- **Standalone server hardening.** It answered any `Host` header, so a web page could reach the unauthenticated dashboard on `127.0.0.1` through DNS rebinding; without `dashboard_auth` it now answers only to loopback names. Error pages no longer show routes and paths. `./pg_reports.rb` and `config/pg_reports.rb` are no longer loaded implicitly, because starting the tool inside a cloned repository ran that repository's Ruby: **pass `--config` explicitly, this is a breaking change.** SQL parameters are filtered from the log, and a failed connection no longer prints the password from `DATABASE_URL`.
+- **Generate Migration accepted Ruby from the client and from index names.** The server wrote whatever `code` the request carried into `db/migrate/`, over any existing file of that name, and `rails db:migrate` then ran it. The code itself interpolated index and table names as bare symbols, so a crafted index name became Ruby. The client now sends only the schema and index name, and the server builds the migration from `pg_indexes`, on the primary database only, with every identifier written as an escaped string.
+- **The dashboard and `/metrics` no longer open themselves to the network.** Without `dashboard_auth`, anyone who could reach the port had the dashboard, and without `grafana_metrics_token`, the metrics. Without credentials they now answer only requests from the same machine, and none in production. **This is a breaking change:** a dashboard protected at the routing level must set `config.dashboard_auth = :host_managed`, and a Prometheus on another host, including Docker, needs `grafana_metrics_token`. The standalone server warns at startup when it is bound beyond loopback without `dashboard_auth`.
+- **Raw queries could write, and could leave settings on a pooled connection.** The SELECT-only check reads the query text, so `SELECT * INTO new_table`, `nextval()`, `lo_create()` and write-capable functions passed it. Worse, `SELECT set_config('statement_timeout', '1', false)` survived the transaction and went back to the pool, possibly the host application's. Execute Query, EXPLAIN ANALYZE and SQL Console now run in a `READ ONLY` transaction that always ends in `ROLLBACK`, and calls to administrative functions (`pg_read_file`, `pg_terminate_backend`, `pg_reload_conf`, `dblink` and others) are refused. This narrows the risk without removing it: connect with a dedicated role that is not a superuser.
+- **State changes through report URLs.** The `:category/:report` routes dispatched to any public method of the report modules, checking only `respond_to?`. A plain GET, which CSRF protection does not cover, could therefore reset `pg_stat_statements`, try to create the extension, or call `Module#freeze`, and the 422 it returned hid that the action had happened. Only reports in the registry can run now.
+- **SQL injection through EXPLAIN ANALYZE and Execute Query parameters.** Values were substituted into the SQL text one after another, so a value containing `$1` was rewritten by the next substitution and left its quotes, and the result went out over the simple protocol, which runs every statement it is given. With raw query execution enabled, this ran arbitrary SQL past the SELECT-only check, which only ever saw the query before substitution. Values are now sent as bind parameters, and SQL Console, Execute Query and EXPLAIN ANALYZE now use the extended protocol, which refuses a second statement. These statements no longer emit `sql.active_record`, so they do not appear in the Rails log.
+- **Stored XSS from database content in the dashboard.** Anyone able to name a table or an index, or to run a query with a comment, could execute script in the session of whoever viewed the report. A path in a SQL comment closed the `href` of a source link, which ran as soon as the report rendered; a table name containing `"` escaped the JSON serialized into the problem indicator and the Save button; a quote in an index name escaped the Generate Migration handler; and the Query Monitor built its IDE menu from the same unescaped path. Inline handlers now carry a row index instead of the row, the IDE menu is built from DOM nodes, and source paths are percent-encoded in every IDE URL.
+
+### Added
+
+- **Most Frequent Queries report (`frequent_queries`).** Ranks queries by execution rate in calls per second, from highest down to `frequent_query_threshold_per_second` (default `1`). `heavy_queries` ranks by the cumulative call count, which favours whatever has been running longest; this report divides by the time each query has been tracked, so a query's place does not depend on when the statistics were last reset. The window comes from the per-entry `stats_since` on PostgreSQL 17+ and from `pg_stat_statements_info.stats_reset` on 14–16, and is shown as `tracked_since`, because the figure is an average over that window rather than the current load. Requires PostgreSQL 14+ and pg_stat_statements 1.9+. Tagged `NEW` in the dashboard.
+- **SQL Query Monitor in standalone mode.** It used to be hidden there, because subscribing to `ActiveSupport::Notifications` would only see the dashboard's own SQL. It now samples `pg_stat_activity` every 0.5s on the database selected when the session starts and feeds the same table, so it sees every client of that database. A backend only remembers its latest statement, so this is a sample rather than a log: fast queries can slip between samples, and the panel says so. pg_reports' own connections are excluded by `application_name`, and without `pg_read_all_stats` the panel reports how many sessions it cannot read.
+- **A switch between the Query Monitor's two capture methods in development.** The panel offers **App** (`sql.active_record` events) and **Database** (sampling `pg_stat_activity` for every client of the selected database, which needs no extension). The source belongs to the session, and `query_monitor_sources` decides what is on offer: both in development, `:notifications` elsewhere, and the server rejects anything else. Standalone mode only ever samples. Dashboard SQL now carries a `/* pg_reports */` comment, the only thing that identifies it in `pg_stat_activity` when it shares the host application's connections.
+- **`query_monitor_enabled`, `query_monitor_cache_store` and `query_monitor_max_duration`.** The monitor can be removed entirely, given a cache of its own when `Rails.cache` is not shared, and limited in time.
+
+### Changed
+
+- **Booting the host application no longer queries the database.** The PostgreSQL version check ran in every process, consoles and rake tasks included, even with the Query Monitor off. It now runs once per process, when the dashboard home page is first opened, and any warning goes to the server's log.
+- **The Query Monitor is off by default in production.** Every process subscribes to SQL events while it is on, so a deploy now adds nothing to the application's queries unless you opt in with `config.query_monitor_enabled = true`. Until then the panel is hidden and `/query_monitor/*` answers 403. Standalone mode is not affected.
+- **The Query Monitor is a compact table.** Each query is one line (time, duration, source, SQL) instead of a card, and a click shows the full statement. Rows are added rather than redrawn every 2 seconds, so an expanded query no longer collapses on the next poll. The panel shows the time left instead of a session id.
+- **Query Monitor sessions stop by themselves** after `query_monitor_max_duration` (15 minutes by default), instead of the previous 24-hour cache lifetime, so a forgotten session does not keep recording user data in production.
+- **`/query_monitor/feed` no longer reports a stopped session as a failure.** It now answers with `success: true, enabled: false` rather than `success: false`, because a session ending is an ordinary state and the page keeps its rows.
+
+### Performance
+
+- **Opening a report page no longer runs the report.** The result was computed and thrown away, and Run Report then ran it again.
+- **The database list no longer costs a walk of every database's files on each request.** That included the 5-second live metrics poll, where it took 137 ms on a cluster with 20,000 tables instead of about 4 ms. Polling and JSON endpoints now read only the names, and the pages that show sizes cache them for 60 seconds per target.
+- **SQL Console and Execute Query cannot return more than 100 rows**, whatever `LIMIT` the query carries, and a trailing `--` comment no longer drops the limit or breaks the count. One query could send 26 MB and add 290 MB to the process.
+
+### Fixed
+
+- **Report filters ignored their defaults outside the dashboard form.** `/metrics`, downloads, Telegram and direct calls ran `long_running_queries` and `low_cache_hit_queries` with no minimum, so they returned rows the dashboard hid. Both now use the declared defaults of 60 seconds and 100 calls. **`/metrics` for these reports may drop to fewer rows.**
+- `recently_modified` showed three empty columns because it asked for the source column names. It now shows inserts, updates and deletes.
+- `Executor#execute` substituted parameters one after another, so a value containing `:other` was rewritten by the next substitution and `:limit` also matched `:limit_x`. Values are now substituted in one pass. No call in the gem passed such a value.
+- **A report could hang for as long as a lock was held.** Report SQL ran with no time limit: while a migration held `ACCESS EXCLUSIVE` on a table, `table_sizes` waited indefinitely, holding a connection, possibly one from the host application's pool. Reports, `/metrics` and live metrics now stop after `report_statement_timeout_ms` (30 s), or after 3 s waiting for a lock, and say which it was. The limits end with the query and never reach a transaction of the host application. **A report that needs more than 30 s now fails until you raise the limit.**
+- **Problem highlighting and `pg_reports_issues` were blind in about half the reports that define thresholds.** Thirteen reports judged a field their SQL does not return, and the three cache reports compared percentages with fractions, so `cache_hit_ratio` of 60 passed a limit of 0.95. Those reports never flagged a row, and Grafana showed them as healthy. Every threshold now reads a field the report returns, on the scale it uses, and a spec keeps it that way. `index_correlation` judges the absolute value, because a strong negative correlation is ordered too. `bloated_indexes` has no threshold for now: its estimate puts freshly built indexes at 65-75% bloat. `connection_stats`, `locks` and `idle_connections` gained `idle_in_transaction`, `waiting_seconds` and `idle_in_transaction_seconds`. `active_connections` no longer has a threshold, because a list of connections has no per-row limit: use `pool_usage`. **After upgrading, `pg_reports_issues` will show problems it has been missing, so review alerts built on it.**
+- `bloated_indexes` failed entirely when any index name needed quoting, such as `MyIdx`, or lived outside the `search_path`.
+- `cache_stats` declared an `index_hit_ratio` column that its SQL never returned, and always showed it empty. It now shows `blks_hit` and `blks_read`.
+- **The database selector failed for a role without `CONNECT` on every database.** `pg_database_size` raised for the first one it could not read, and the dashboard lost the whole list. Such a database is now listed without a size.
+- **Standalone startup.** An unreachable database went unnoticed on Rails 7.2+, which connects lazily, and the failure surfaced on the first request; it is now reported at startup. `PGHOST` set to a Unix socket directory or an IPv6 address produced a broken URL. Without `rackup`, as on Rails before 7.1, startup failed with a `LoadError` instead of finding a web server.
+- **Generated migrations.** They could not be rolled back, always declared `Migration[7.0]`, and targeted `public` whatever the index's schema; they now restore the index on `down`, use the host's Rails version and qualify the schema. The button followed `Rails.env` and the Create action followed `allow_raw_query_execution`; both now follow `allow_migration_creation`.
+- **SQL Console, Execute Query and EXPLAIN ANALYZE ignored the selected database and target.** They always ran on the application's primary connection, whatever the selector showed, so a query could run against a different database or cluster than the one on screen, and EXPLAIN ANALYZE really executes it. They now use the same connection as reports, and the statement timeout is set on that connection too.
+- **EXPLAIN ANALYZE and Execute Query on captured queries with `LIMIT $1` or `interval $1`.** The first failed with a syntax error because a second `LIMIT` was appended, and the second is the form `pg_stat_statements` reduces `interval '1 day'` to, which PostgreSQL only accepts for a literal. It is now run as `$1::interval`, likewise for `date`, `time`, `timestamp` and `timestamptz`.
+- **The Query Monitor did not work with more than one server process.** Pressing Start enabled it in one worker only: on a three-worker Puma, 10 of 15 status requests answered "disabled", the first such feed response made the page stop polling and show monitoring as off, and only 3 of 12 queries were captured. The shared flag was written but never read (`sync_from_cache` assigned the attribute to itself since the fix for recursion with database-backed caches), the buffer was per-process, and a worker only subscribed if it happened to serve the Start request. Every process now subscribes at boot and follows the session in the cache, and captured queries go to a bounded ring in that cache using atomic sequence numbers, so any worker can answer, and any worker can stop a session for all of them. Verified on a three-worker cluster against both `FileStore` and Redis. A cache that cannot be shared is reported on the panel.
+- **Query Monitor dropped queries under concurrency.** The reentrancy guard was an instance variable shared by all threads, so a query arriving while another thread was being recorded was silently lost. The guard is now per-thread, and a failure while capturing can no longer raise into the host application's query.
+- **Query Monitor added latency to the host application.** Each query walked the backtrace twice; it now does so once, and cache and log writes happen on a background thread.
+- **Query Monitor downloads failed after stopping,** with "No queries to download" while the button stayed visible, because stopping cleared the buffer. The last session's results now stay available for an hour.
+- **`query_monitor_backtrace_filter` had no effect.** The option was documented but never consulted.
+- **The Query Monitor failed on cache stores that deserialize what they read, such as RailsBrotliCache.** Its counter is stored raw, and reading it back raised `marshal data too short`: the feed was silently empty and the status request failed, so the page showed monitoring as off. The counter is now only ever touched through `increment`, a store that cannot support the monitor is refused at Start, and read or write failures appear on the panel rather than only in the log.
+- **The Query Monitor ignored source locations already in the SQL,** and could report a Ruby internal frame such as `<internal:numeric>`. Marginalia and Rails QueryLogs annotations now provide the source, and are removed from the displayed statement.
+- **Query Monitor file logging only happened on stop,** and only in the worker that received the stop. Each process now appends as it goes.
+
 ## [0.9.1] - 2026-09-16
 
 ### Added

data/README.md

@@ -2,7 +2,7 @@
 
 [![Gem Version](https://img.shields.io/gem/v/pg_reports.svg)](https://rubygems.org/gems/pg_reports)
 [![Ruby](https://img.shields.io/badge/Ruby-2.7%2B-red.svg)](https://www.ruby-lang.org/)
-[![Rails](https://img.shields.io/badge/Rails-5.0%2B-red.svg)](https://rubyonrails.org/)
+[![Rails](https://img.shields.io/badge/Rails-6.1%2B-red.svg)](https://rubyonrails.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 A comprehensive PostgreSQL monitoring and analysis library for Rails applications. Get insights into query performance, index usage, table statistics, connection health, and more — across **every database on the cluster**, switchable from the dashboard with no extra configuration. Includes a beautiful web dashboard, a Grafana / Prometheus exporter, and Telegram delivery.
@@ -29,7 +29,7 @@ A comprehensive PostgreSQL monitoring and analysis library for Rails application
 - 📌 **Comparison Mode** - Save records to compare before/after optimization
 - 📊 **EXPLAIN ANALYZE** - Advanced query plan analyzer with problem detection and recommendations
 - 🖥️ **SQL Console** - Free-form SQL editor in a modal, run SELECT queries and view results directly from the dashboard
-- 🔍 **SQL Query Monitoring** - Real-time monitoring of all executed SQL queries with source location tracking (not available in standalone mode)
+- 🔍 **SQL Query Monitoring** - Real-time monitoring of executed SQL queries across all your app's processes, with source location tracking
 - 🔌 **Connection Pool Analytics** - Monitor pool usage, wait times, saturation warnings, and connection churn
 - 🤖 **AI Prompt Export** - Copy a ready-to-paste prompt for Claude Code, Cursor, or Codex with problem context and report data
 - 🗑️ **Migration Generator** - Generate Rails migrations to drop unused indexes
@@ -76,7 +76,7 @@ DATABASE_URL=postgres://user:pass@localhost/myapp bundle exec pg_reports server
 ./bin/pg_reports server --allow-raw-query-execution   # opt into the Run SQL panel
 ```
 
-Settings come from `PG_REPORTS_*` env vars, an auto-detected `./pg_reports.rb` config file (full `PgReports.configure` access), or CLI flags — in that order of precedence. Adds no runtime dependencies to the gem. **[Standalone guide → docs/standalone.md](docs/standalone.md)**
+Settings come from `PG_REPORTS_*` env vars, a config file passed with `--config` (full `PgReports.configure` access), or CLI flags — in that order of precedence. Adds no runtime dependencies to the gem. **[Standalone guide → docs/standalone.md](docs/standalone.md)**
 
 ## Usage
 
@@ -118,7 +118,8 @@ PgReports.configure do |config|
   config.unused_index_threshold_scans = 50
   config.bloat_threshold_percent      = 20
 
-  # Strongly recommended in production
+  # Required outside localhost, and always in production: without it the
+  # dashboard answers 403. Use :host_managed if your routes already protect it.
   config.dashboard_auth = -> {
     authenticate_or_request_with_http_basic do |user, pass|
       user == ENV["PG_REPORTS_USER"] && pass == ENV["PG_REPORTS_PASSWORD"]
@@ -200,17 +201,20 @@ Requires `config.allow_raw_query_execution = true`.
 <details>
 <summary><strong>SQL Query Monitor — real-time query capture</strong></summary>
 
-Live capture of all SQL executed by your Rails app. Click **▶ Start Monitoring**, run any operation, watch the queries appear with:
+Live capture of the SQL executed by your Rails app, across all of its processes. It is off by default in production: set `query_monitor_enabled = true` to use it there. Click **Start Monitoring**, run any operation, and watch the queries appear in a compact table, one line each. Click a row for the full SQL.
 
-- SQL with syntax highlighting
 - Duration (color-coded: 🟢 <10ms, 🟡 <100ms, 🔴 >100ms)
 - Source location with click-to-IDE
 - Timestamp
+- Stops by itself after 15 minutes
 
-Built on `ActiveSupport::Notifications` (`sql.active_record`). Filters internal queries (SCHEMA / CACHE / pg_reports' own). Logged to `log/pg_reports.log` (JSON Lines). Configurable buffer size and backtrace filter:
+Built on `ActiveSupport::Notifications` (`sql.active_record`), with the session and results kept in `Rails.cache`, so **it needs a cache shared by your processes** (Redis, Memcached, SolidCache) to work with several workers; the panel warns you if there isn't one. Filters internal queries (SCHEMA / CACHE / pg_reports' own). Logged to `log/pg_reports.log` (JSON Lines).
 
 ```ruby
 PgReports.configure do |config|
+  config.query_monitor_enabled = true # off by default in production
+  config.query_monitor_cache_store = ActiveSupport::Cache::RedisCacheStore.new(url: ENV["REDIS_URL"])
+  config.query_monitor_max_duration = 5.minutes
   config.query_monitor_log_file = Rails.root.join("log", "custom_monitor.log")
   config.query_monitor_max_queries = 200
   config.query_monitor_backtrace_filter = ->(loc) { !loc.path.match?(%r{/(gems|ruby|railties)/}) }
@@ -219,7 +223,7 @@ end
 
 Use cases: debugging N+1, identifying slow queries during feature development, tracking down unexpected queries, teaching ActiveRecord behavior.
 
-Not available in [standalone mode](docs/standalone.md) — there is no host application process to subscribe to, so the panel and its API are both disabled.
+In [standalone mode](docs/standalone.md#sql-query-monitor) there is no application to subscribe to, so the monitor samples `pg_stat_activity` instead: it sees every client of the selected database, but fast queries can slip between samples.
 
 </details>