rails-ai-context 5.35.2 → 5.36.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +197 -1
- data/CONTRIBUTING.md +1 -0
- data/docs/COMPATIBILITY.md +14 -1
- data/docs/SECURITY.md +12 -10
- data/docs/TOOLS.md +12 -11
- data/docs/TROUBLESHOOTING.md +1 -1
- data/lib/rails_ai_context/database_family/mysql_runtime.rb +17 -2
- data/lib/rails_ai_context/database_family/postgresql_plan.rb +6 -2
- data/lib/rails_ai_context/database_family/postgresql_runtime.rb +41 -6
- data/lib/rails_ai_context/database_family/sqlite_runtime.rb +9 -3
- data/lib/rails_ai_context/database_family.rb +27 -5
- data/lib/rails_ai_context/introspector.rb +12 -0
- data/lib/rails_ai_context/introspectors/eager_load.rb +12 -2
- data/lib/rails_ai_context/introspectors/job_introspector.rb +57 -14
- data/lib/rails_ai_context/introspectors/model_introspector.rb +13 -11
- data/lib/rails_ai_context/introspectors/schema_introspector.rb +12 -1
- data/lib/rails_ai_context/payload.rb +2 -3
- data/lib/rails_ai_context/secret_columns.rb +59 -37
- data/lib/rails_ai_context/sql_gate.rb +118 -25
- data/lib/rails_ai_context/tools/get_conventions.rb +10 -5
- data/lib/rails_ai_context/tools/get_schema.rb +35 -11
- data/lib/rails_ai_context/tools/query.rb +12 -2
- data/lib/rails_ai_context/tools/runtime_info.rb +15 -10
- data/lib/rails_ai_context/version.rb +1 -1
- metadata +1 -23
- data/.rspec +0 -3
- data/.rubocop.yml +0 -16
- data/.ruby-version +0 -1
- data/AGENTS.md +0 -15
- data/CLAUDE.md +0 -3
- data/CODE_OF_CONDUCT.md +0 -44
- data/CONTEXT.md +0 -242
- data/Rakefile +0 -80
- data/docs/RELEASING.md +0 -72
- data/docs/_config.yml +0 -15
- data/docs/_includes/head-custom.html +0 -16
- data/docs/adr/0001-own-listener-registration.md +0 -9
- data/docs/adr/0002-static-tier-declared-not-detected.md +0 -5
- data/docs/adr/0003-shared-tool-cache-semantics.md +0 -31
- data/docs/adr/0004-install-e2e-stays-example-rich.md +0 -28
- data/docs/adr/0008-query-runs-only-where-it-can-guard.md +0 -38
- data/docs/adr/0009-dbhub-shapes-in-ruby.md +0 -26
- data/docs/adr/0010-one-config-tool-with-sections.md +0 -32
- data/docs/agents/domain.md +0 -38
- data/docs/agents/issue-tracker.md +0 -45
- data/docs/agents/triage-labels.md +0 -15
- data/server.json +0 -21
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7ae3eb5f18027a0abe6cf4a8872f9ecce9fb792109ab243b9b7edbba3558cb6d
|
|
4
|
+
data.tar.gz: c6c7b740f4e18e494a371215c99c05bad848ed43ebf1f84f24020ad4348612ac
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fe22c7af7f32d7aa16abdc3b6534653614f3e38fc25a1406c5e29ab9dbb0cf609878fd633210cb3006352007b156d5eaba887db3d1e9ad53cec0ae57357622c0
|
|
7
|
+
data.tar.gz: 6b0aff79f555f43322eab5c3c81ebcc8a7477a8674891d1f3d2988a47cd23044db5383aacbbd6db3ec15cdeb882dc37b5af4e9cb98c73f1150ba8c73d55cf283
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,202 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [5.36.1] - 2026-10-15
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **On PostgreSQL the refusal of a name on the read-only VOLATILE list says the
|
|
13
|
+
list trusts the server's own function or a C function.** 5.36.0 answered "an
|
|
14
|
+
extension's C function", narrower than the rule it applies: any C function of
|
|
15
|
+
that name is trusted, since only a superuser can create one.
|
|
16
|
+
- **The docs say which reads of a `rails_query` call run with no time limit.**
|
|
17
|
+
`docs/TOOLS.md` and `docs/SECURITY.md` said every statement a call sends is
|
|
18
|
+
bounded, and the reads of a setting's own value for the settings note, new in
|
|
19
|
+
5.36.0, are not. They wait on no lock. `docs/SECURITY.md` also said `PRAGMA
|
|
20
|
+
query_only` is refused by name: a `PRAGMA` statement is no read, and the
|
|
21
|
+
`pragma_query_only` function is the one refused by name.
|
|
22
|
+
- **`docs/TOOLS.md` says when `rails_get_mailers` names where a mailer's actions
|
|
23
|
+
come from.** A mailer with no action of its own names a gem base it reaches
|
|
24
|
+
without passing `ApplicationMailer`, such as `Devise::Mailer`. It names none
|
|
25
|
+
past `ApplicationMailer` and never Rails' own base, on both tiers, as 5.36.0
|
|
26
|
+
does. The page said any gem base was named.
|
|
27
|
+
|
|
28
|
+
## [5.36.0] - 2026-10-14
|
|
29
|
+
|
|
30
|
+
### Security
|
|
31
|
+
|
|
32
|
+
- **On PostgreSQL a name on the read-only VOLATILE list vouches for the server's
|
|
33
|
+
own function and an extension's C function alone.** The list trusted a name in
|
|
34
|
+
any schema, so a VOLATILE function the app defined under one of those names
|
|
35
|
+
ran. While the database holds a VOLATILE function of a listed name in another
|
|
36
|
+
language outside `pg_catalog`, every call of the name is now refused, and the
|
|
37
|
+
refusal says so. 5.35.2 and earlier versions had the gap.
|
|
38
|
+
- **On PostgreSQL 18 a set operation over a secret column is refused, as it is
|
|
39
|
+
on older servers.** PostgreSQL 18 plans a set operation over the tables
|
|
40
|
+
themselves, and the plan check that refuses a secret column passed through a
|
|
41
|
+
set operation did not read that shape. A set operation now counts every column
|
|
42
|
+
its inputs hand up as a use of that column. 5.35.2 and earlier versions had
|
|
43
|
+
the gap on PostgreSQL 18.
|
|
44
|
+
|
|
45
|
+
### Changed
|
|
46
|
+
|
|
47
|
+
- **On SQLite a call of `changes()`, `total_changes()` or
|
|
48
|
+
`last_insert_rowid()` is refused also where the app has a table of the same
|
|
49
|
+
name.** The counter answers for the connection that runs the statement,
|
|
50
|
+
which is `rails_query`'s own. 5.35.1 and 5.35.2 ran such a call when a table
|
|
51
|
+
carried the counter's name. Read the counter on the app's own connection, in
|
|
52
|
+
`bin/rails runner` or the console.
|
|
53
|
+
- **The packaged gem holds only what a user of the gem runs or reads.** That is
|
|
54
|
+
`lib/`, `exe/`, `app/`, `config/`, the README, the changelog, the license,
|
|
55
|
+
`SECURITY.md`, the user docs in `docs/`, and the files those docs link to
|
|
56
|
+
(`CONTRIBUTING.md` and three ADRs), so every link works in the unpacked gem.
|
|
57
|
+
5.35.2 also packed the files that serve the repository alone: `AGENTS.md`,
|
|
58
|
+
`CLAUDE.md`, `CONTEXT.md`, `CODE_OF_CONDUCT.md`, `docs/RELEASING.md`, the
|
|
59
|
+
other ADRs, `docs/agents/`, the docs site config, `.rspec`, `.rubocop.yml`,
|
|
60
|
+
`.ruby-version`, `Rakefile` and `server.json`. A tool that indexes an app's
|
|
61
|
+
bundle could read the two assistant files as instructions about the app.
|
|
62
|
+
|
|
63
|
+
### Fixed
|
|
64
|
+
|
|
65
|
+
- **On PostgreSQL a read with `TABLESAMPLE` runs.** The name of the sampling
|
|
66
|
+
method (`BERNOULLI`, `SYSTEM`) was read as a call of a VOLATILE function, and
|
|
67
|
+
the statement was refused. The statement never calls it, so it is no longer
|
|
68
|
+
read as a call. A VOLATILE function called anywhere in the statement, one
|
|
69
|
+
named like a sampling method included, is still refused. 5.35.2 refused the
|
|
70
|
+
read.
|
|
71
|
+
- **On PostgreSQL an `EXISTS` inside an `EXISTS` over a table with a secret
|
|
72
|
+
column runs.** The plan puts a Result node over the inner scan, and the plan
|
|
73
|
+
check counted every column of that scan as passed through, so the statement
|
|
74
|
+
was refused with a sentence saying it passed the secret column on. A Result
|
|
75
|
+
node is now read like a join: a column counts only where the Result names
|
|
76
|
+
it. A plain read of a few columns under a condition that holds no column
|
|
77
|
+
(`WHERE EXISTS (...)`) runs for the same reason. 5.35.2 refused them.
|
|
78
|
+
- **On PostgreSQL `EXPLAIN (GENERIC_PLAN)` answers the plan.** The plan check
|
|
79
|
+
planned the statement under it without the option, so a statement with `$1` in
|
|
80
|
+
it answered an SQL error. It is now planned with the option, read as the
|
|
81
|
+
server reads it: its last value counts, quoted or not. A server older than
|
|
82
|
+
PostgreSQL 16 has no such option and still answers its error. 5.35.2 answered
|
|
83
|
+
the error.
|
|
84
|
+
- **On PostgreSQL the `pg_sequences` view can be read.** It reads each
|
|
85
|
+
sequence's value through `pg_sequence_last_value`, a VOLATILE function that
|
|
86
|
+
writes nothing, and the statement was refused. The function now joins the
|
|
87
|
+
list of read-only VOLATILE functions. `nextval`, `setval`, `currval` and
|
|
88
|
+
`lastval` are still refused. 5.35.2 refused the view.
|
|
89
|
+
- **A query leaves a transaction the app opened with a raw `BEGIN` open, with
|
|
90
|
+
its writes.** Active Record does not count such a transaction, so on
|
|
91
|
+
PostgreSQL the query sent its own `BEGIN` and then `ROLLBACK`, which ended
|
|
92
|
+
the app's transaction and threw its writes away; the answer also missed the
|
|
93
|
+
rows the transaction had written. PostgreSQL now reads the server's own
|
|
94
|
+
transaction status and guards inside a savepoint there, under the same
|
|
95
|
+
read-only transaction and statement timeout. On MySQL and MariaDB the answer
|
|
96
|
+
is the refusal an open transaction gets ("MySQL cannot make a transaction
|
|
97
|
+
that is already open read-only"), where 5.35.2 answered the server's error
|
|
98
|
+
1568, and no `BEGIN`, which would commit the app's transaction, is sent. On
|
|
99
|
+
a SQLite file the statement runs on the app's own connection and reads the
|
|
100
|
+
transaction's rows, where 5.35.2 answered from the file without them.
|
|
101
|
+
5.35.2 and every earlier version ended the transaction on PostgreSQL.
|
|
102
|
+
- **On PostgreSQL a statement another session cancels is answered as the SQL
|
|
103
|
+
error it is.** A `pg_cancel_backend` from another session raises the same
|
|
104
|
+
error class as the statement timeout, and 5.35.2 answered "Query exceeded <N>
|
|
105
|
+
second timeout". The server's message tells the two apart, so a cancel now
|
|
106
|
+
answers `SQL error: PG::QueryCanceled: ERROR: canceling statement due to user
|
|
107
|
+
request`. A real timeout keeps its sentence. A server that writes its messages
|
|
108
|
+
in another language still answers a cancel as the timeout.
|
|
109
|
+
- **A query that reads a setting `rails_query` sets for each statement says so,
|
|
110
|
+
and gives the session's own value.** `SHOW statement_timeout` answered the
|
|
111
|
+
bound the tool set for the statement (`2s`) with nothing to say it was not the
|
|
112
|
+
session's (`50s`), and so did `transaction_read_only` on PostgreSQL, and the
|
|
113
|
+
statement limit (`max_execution_time` or `max_statement_time`) and
|
|
114
|
+
`lock_wait_timeout` on MySQL and MariaDB, by `@@` (the name backquoted or
|
|
115
|
+
not), by `SHOW VARIABLES` or by MariaDB's
|
|
116
|
+
`information_schema.SESSION_VARIABLES`. The answer now ends with a note, for
|
|
117
|
+
example "statement_timeout here is the value rails_query sets for each
|
|
118
|
+
statement it runs. The session's own: statement_timeout 50s." A name used as
|
|
119
|
+
an alias or a column, one in a string or a comment, and a global value on
|
|
120
|
+
MySQL and MariaDB get no note. 5.35.2 and earlier gave no note.
|
|
121
|
+
- **On PostgreSQL 13 and 14 a temporary table runs, with its secret column
|
|
122
|
+
filtered.** Before PostgreSQL 15 a plan names the session's temporary schema
|
|
123
|
+
by its number (`pg_temp_3`), where the column reads name it `pg_temp`, so
|
|
124
|
+
5.35.2 refused every statement over a temporary table there as one whose
|
|
125
|
+
columns could not be read. A whole row of a temporary table that holds a
|
|
126
|
+
secret column stays refused.
|
|
127
|
+
- **On SQLite a column whose own name starts with `pragma_` runs.** A
|
|
128
|
+
`pragma_*` name is now refused where SQLite reads it as the pragma function:
|
|
129
|
+
called, or where a table stands (after `FROM`, `JOIN` or `IN`, after a comma
|
|
130
|
+
of a `FROM` list, inside a parenthesis that opens one, or after a schema and a
|
|
131
|
+
dot there), bare, quoted or as a string. Anywhere else it is a column's or an
|
|
132
|
+
alias's name, and the statement runs. 5.35.1 and 5.35.2 refused such a name
|
|
133
|
+
wherever it stood, so `SELECT id FROM t WHERE pragma_flag = 1` was refused.
|
|
134
|
+
- **On SQLite a secret column name covers only the columns SQLite reads it as.**
|
|
135
|
+
SQLite folds ASCII letters alone, so `"é"` and `"É"` are two columns, and the
|
|
136
|
+
secret column rule now compares names that way there. With `"É"` in
|
|
137
|
+
`query_redacted_columns`, `SELECT "é" FROM t` runs and `SELECT *` filters `É`
|
|
138
|
+
alone; every earlier version refused `é` too and filtered both. A name is
|
|
139
|
+
still read with every letter folded for what it means, so a column that folds
|
|
140
|
+
to a built-in name or one the heuristic flags stays secret, and so does one a
|
|
141
|
+
configured name folds to when no column spells that name as SQLite folds it.
|
|
142
|
+
PostgreSQL, MySQL and MariaDB fold every letter as before.
|
|
143
|
+
- **On SQLite a statement that names a view over a pragma function or a
|
|
144
|
+
counter that tells of the connection is refused, by the view and the
|
|
145
|
+
function.** The view's definition runs on `rails_query`'s own connection, so
|
|
146
|
+
`CREATE VIEW v AS SELECT name FROM pragma_database_list` answered without
|
|
147
|
+
the app's attached databases, and a view calling `changes()` answered for the
|
|
148
|
+
wrong connection, both as a success. 5.35.1 and 5.35.2 refused the function
|
|
149
|
+
written in the statement alone. A view over a pragma function that describes
|
|
150
|
+
the schema or the database file, such as `pragma_table_info`, runs.
|
|
151
|
+
- **Every preset names each database's adapter and lists the databases.** On
|
|
152
|
+
`preset: standard`, which runs no `multi_database` introspector, the static
|
|
153
|
+
tier called the primary's adapter `unknown` in `rails_onboard` and in the
|
|
154
|
+
generated files, named no adapter beside a second database, and left the `-
|
|
155
|
+
Databases:` line out of CLAUDE.md and `.claude/rules/rails-context.md`. The
|
|
156
|
+
context now carries the app databases list on every preset, and where the
|
|
157
|
+
`multi_database` introspector failed, so these answer what `preset: full`
|
|
158
|
+
answers. 5.35.2 and earlier versions had the gap.
|
|
159
|
+
- **A model whose class body raises is named with what it raised.** Ruby
|
|
160
|
+
registers a class before its body runs, so a model file that raised partway
|
|
161
|
+
down its class was listed as `[VERIFIED]` with only what ran before the
|
|
162
|
+
raise, and with no word that loading it failed. It is now named with the
|
|
163
|
+
error, as a file that raises before its class opens has been since 5.35.2.
|
|
164
|
+
5.35.2 and earlier versions listed it.
|
|
165
|
+
- **A booted model keeps its STI hierarchy while nothing has queried its
|
|
166
|
+
database.** A model whose table is in a second database, and so not in
|
|
167
|
+
`db/schema.rb`, lost its STI parent, children and type column until a first
|
|
168
|
+
query connected that database. The table is now read as not known to lack the
|
|
169
|
+
type column, as the static tier reads it, and a model with no STI parent or
|
|
170
|
+
child still answers no STI. An abstract base, such as the class a second
|
|
171
|
+
database connects through, is no STI parent. 5.35.2 and earlier versions had
|
|
172
|
+
the gap.
|
|
173
|
+
- **The static mailer list adds the actions a mailer inherits from the app's own
|
|
174
|
+
mailers.** A `ShopMailer < BaseMailer` listed only the actions in its own
|
|
175
|
+
file, where Rails delivers the parent's public methods too and the booted tier
|
|
176
|
+
lists them. Each name now takes the visibility the nearest class gives it, so
|
|
177
|
+
a method the child makes private, under `private` or with `private :name`,
|
|
178
|
+
stays out. The methods of `ApplicationMailer` itself stay out too, as the
|
|
179
|
+
booted tier leaves them out, and on both tiers a mailer with no action of its
|
|
180
|
+
own names where they come from only for a parent past the app's own mailers,
|
|
181
|
+
such as `Devise::Mailer`, and none past `ApplicationMailer`. 5.35.2 and
|
|
182
|
+
earlier versions had the gap.
|
|
183
|
+
- **`rails_get_schema` says when two databases share `db/schema.rb`.** Rails
|
|
184
|
+
dumps both the first database and the one named `primary` to that file, so it
|
|
185
|
+
holds whichever was dumped last. A table of the other database was called
|
|
186
|
+
missing from the connected one, with advice to run `rails db:migrate`, which
|
|
187
|
+
changes nothing. The answer now says "_main and primary both dump to
|
|
188
|
+
db/schema.rb, so it may describe primary. Not in main: events._", and asked
|
|
189
|
+
for that table it says the same, with no word of a migration. 5.35.2 and
|
|
190
|
+
earlier versions had the gap.
|
|
191
|
+
- **The conventions section says how many entries each of its lists of five
|
|
192
|
+
leaves out.** `rails_get_config(section: "conventions")` shows the first five
|
|
193
|
+
authorization checks, denials, flash notices, flash alerts and not-found
|
|
194
|
+
patterns, and now ends a longer list with `- _...and N more._`. Every earlier
|
|
195
|
+
version cut each list at five and said nothing about the rest.
|
|
196
|
+
- **`rails_runtime_info` shows Sidekiq's queue counts as the app's only when
|
|
197
|
+
Sidekiq is the Active Job adapter.** Under another adapter the jobs section
|
|
198
|
+
says live queue stats are only read from Sidekiq and names that adapter. When
|
|
199
|
+
the app also has Sidekiq workers of its own, Sidekiq's counts follow under a
|
|
200
|
+
`### Sidekiq` heading that says they are those workers' counts. Since 4.3.1
|
|
201
|
+
any app with Sidekiq loaded got Sidekiq's counts under the adapter's name,
|
|
202
|
+
where they read as that adapter's queue.
|
|
203
|
+
|
|
8
204
|
## [5.35.2] - 2026-10-14
|
|
9
205
|
|
|
10
206
|
### Security
|
|
@@ -8440,7 +8636,7 @@ Every introspector that scans `.rb` files has been migrated from regex-based pat
|
|
|
8440
8636
|
|
|
8441
8637
|
### Added - 8 new introspectors closing RAILS_NERVOUS_SYSTEM.md gaps
|
|
8442
8638
|
|
|
8443
|
-
An audit against
|
|
8639
|
+
An audit against `RAILS_NERVOUS_SYSTEM.md` identified 9 framework sections where introspection was missing or partial. This release ships the 8 introspectors needed to close them (the 9th - §11 Query interface - was already covered by `:conventions` via `load_async` / `.async_*` scanning). All are wired into `PRESETS[:full]`. The gem now exposes **39 introspectors** (up from 31).
|
|
8444
8640
|
|
|
8445
8641
|
- **`InitializerIntrospector` (`:initializers`, §2).** Enumerates `Rails.application.initializers` - every initializer's name, owner, declared `before:` / `after:` ordering edges, and the `source_location` of its block. Also summarizes each `config/initializers/*.rb` file (initializer count + the top `config.*` setters it touches) so AI can jump straight to user-owned boot code.
|
|
8446
8642
|
- **`AutoloadIntrospector` (`:autoload`, §3).** Zeitwerk presence, both `Rails.autoloaders.main` and `.once` with their collapsed dirs + ignored paths + root dirs, raw `autoload_paths` / `autoload_once_paths` / `eager_load_paths`, the resolved `eager_load` boolean, plus custom inflection rules extracted from `config/initializers/*.rb` (`inflect.acronym` / `.plural` / `.singular` / `.irregular` / `.uncountable` / `.human`). Paths are root-relative.
|
data/CONTRIBUTING.md
CHANGED
|
@@ -109,6 +109,7 @@ RAC_TEST_MYSQL_URL=trilogy://root:secret@127.0.0.1:3306/rac \
|
|
|
109
109
|
The specs create and drop tables and views whose names start with `rac_`, and leave the `postgres_fdw` extension installed on PostgreSQL, so point them at a scratch database.
|
|
110
110
|
CI's `database_servers` job runs them on PostgreSQL 16 in every leg. Rails 8.0 runs two legs: MySQL 8.0 with MariaDB 10.6, and MySQL 8.4 with MariaDB 11.4. Rails 7.0 runs MySQL 8.0 with MariaDB 10.6, and Rails 8.1 runs MySQL 8.4 with MariaDB 11.4.
|
|
111
111
|
Rails 7.0 has no Trilogy adapter, so its leg runs PostgreSQL and mysql2 only.
|
|
112
|
+
The `postgresql_versions` job runs the PostgreSQL specs on PostgreSQL 13, 14, 15, 17 and 18 on Rails 8.0, since the plan checks read the plan the server prints.
|
|
112
113
|
The job builds mysql2 against MariaDB's client library (`libmariadb-dev`, through `BUNDLE_BUILD__MYSQL2`), which signs in to MySQL and to MariaDB. A MySQL client from 9.0 on cannot sign in to MariaDB.
|
|
113
114
|
|
|
114
115
|
### Static fixture
|
data/docs/COMPATIBILITY.md
CHANGED
|
@@ -317,6 +317,16 @@ Proof sources:
|
|
|
317
317
|
on the v5.35.1 code and on the fixed code, with every changed answer traced
|
|
318
318
|
to its fix; and for each fix an example that fails on the v5.35.1 code, on a
|
|
319
319
|
real server where the fix is a server's.
|
|
320
|
+
In the v5.36.0 release QA: for each fix an example that fails on the v5.35.2
|
|
321
|
+
code, on a real server where the fix is a server's; `rails_query` through
|
|
322
|
+
its tool call in CI on PostgreSQL 16, MySQL 8.0 and 8.4 and MariaDB 10.6 and
|
|
323
|
+
11.4 on Rails 7.0 (mysql2), 8.0 and 8.1 (mysql2 and Trilogy), and on
|
|
324
|
+
PostgreSQL 13, 14, 15, 17 and 18 on Rails 8.0; and the unit suite and the
|
|
325
|
+
E2E harness on Linux, macOS and Windows.
|
|
326
|
+
In the v5.36.1 release QA: every issue repro of 5.36.0 run again on the
|
|
327
|
+
published code on PostgreSQL 13 to 18, MySQL 8.0 and 8.4 and MariaDB 10.6
|
|
328
|
+
and 11.4, the booted and static mailer lists compared on one app, and the
|
|
329
|
+
unit suite and the E2E harness on Linux, macOS and Windows.
|
|
320
330
|
2. Non-crash coverage for every built-in tool including `get_view` in
|
|
321
331
|
`spec/e2e/in_gemfile_install_spec.rb`'s full-tool sweep; output correctness
|
|
322
332
|
(ivar cross-check, render-form detection, partial interfaces) verified
|
|
@@ -363,7 +373,10 @@ row estimates and index usage say they are not available there. CI's
|
|
|
363
373
|
8.4 and MariaDB 10.6 and 11.4 on Rails 8.0, through mysql2 and Trilogy. It also
|
|
364
374
|
runs PostgreSQL 16 with MySQL 8.0 and MariaDB 10.6 on Rails 7.0, which has no
|
|
365
375
|
Trilogy adapter, and with MySQL 8.4 and MariaDB 11.4 on Rails 8.1; see
|
|
366
|
-
[CONTRIBUTING.md](../CONTRIBUTING.md#real-database-servers).
|
|
376
|
+
[CONTRIBUTING.md](../CONTRIBUTING.md#real-database-servers). The
|
|
377
|
+
`postgresql_versions` job runs the PostgreSQL examples on PostgreSQL 13, 14,
|
|
378
|
+
15, 17 and 18 as well, since the plan checks read the plan the server prints and
|
|
379
|
+
its shape changes between major versions.
|
|
367
380
|
|
|
368
381
|
Postgres itself - live query safety (read-only transactions, the
|
|
369
382
|
`BLOCKED_FUNCTIONS` guard against `pg_read_file`/`dblink`/`COPY ... PROGRAM`,
|
data/docs/SECURITY.md
CHANGED
|
@@ -62,7 +62,7 @@ Before any query reaches the database, the SQL gate (`RailsAiContext::SqlGate`)
|
|
|
62
62
|
- **Blocks file and network functions**, each on the database family that has it. PostgreSQL: `pg_read_file`, `pg_read_binary_file`, `pg_ls_dir`, `pg_stat_file`, `lo_import`, `lo_export`, `dblink*` and their relatives. MySQL and MariaDB: `LOAD DATA` and `LOAD_FILE`. SQLite: `load_extension`. They are checked before and after comment stripping
|
|
63
63
|
- **Blocks functions that return a whole table as one value**: PostgreSQL's `table_to_xml`, `query_to_xml`, `cursor_to_xml`, `schema_to_xml`, `database_to_xml` and their `_to_xmlschema` and `_to_xml_and_xmlschema` relatives. Such a value carries every column of the table, its secret columns included, under a name no column check reads. The name with `(` after it is refused on the raw text: "Blocked: function <name> returns a whole table as one value, secret columns included. SELECT the columns you need from the table itself."
|
|
64
64
|
- **Blocks relations and settings that hold sampled values, other sessions' statements or stored credentials**: no check on a column's name applies to what they hold, so they are not read at all. "Relations and settings that are never read" below lists them by database family, and each name is refused on the family that has it
|
|
65
|
-
- **On SQLite, blocks functions that tell of the connection**: `rails_query` runs a statement on a connection of its own and sets some of that connection's values itself, so such an answer would not be the app's. A `pragma_*` function runs only when it describes the schema or the database file: `pragma_table_info`, `pragma_table_xinfo`, `pragma_table_list`, `pragma_index_list`, `pragma_index_info`, `pragma_index_xinfo`, `pragma_foreign_key_list`, `pragma_foreign_key_check`, `pragma_integrity_check`, `pragma_quick_check`, `pragma_user_version`, `pragma_application_id`, `pragma_schema_version`, `pragma_page_size`, `pragma_page_count`, `pragma_freelist_count`, `pragma_encoding`, `pragma_auto_vacuum`, `pragma_journal_mode`, `pragma_compile_options` and `pragma_pragma_list`. Every other `pragma_*` name is refused
|
|
65
|
+
- **On SQLite, blocks functions that tell of the connection**: `rails_query` runs a statement on a connection of its own and sets some of that connection's values itself, so such an answer would not be the app's. A `pragma_*` function runs only when it describes the schema or the database file: `pragma_table_info`, `pragma_table_xinfo`, `pragma_table_list`, `pragma_index_list`, `pragma_index_info`, `pragma_index_xinfo`, `pragma_foreign_key_list`, `pragma_foreign_key_check`, `pragma_integrity_check`, `pragma_quick_check`, `pragma_user_version`, `pragma_application_id`, `pragma_schema_version`, `pragma_page_size`, `pragma_page_count`, `pragma_freelist_count`, `pragma_encoding`, `pragma_auto_vacuum`, `pragma_journal_mode`, `pragma_compile_options` and `pragma_pragma_list`. Every other `pragma_*` name is refused where SQLite reads it as the function: called, or where a table stands (after `FROM`, `JOIN`, `IN`, a comma of a `FROM` list, a parenthesis that opens one, or a schema's dot there), quoted or in a string literal too. A call of `changes()`, `total_changes()` or `last_insert_rowid()` is refused wherever it stands, inside a string literal too, whatever tables the app has. The answer is "Blocked: `<name>` tells of the connection that runs it. rails_query runs a statement on a connection of its own and sets some of that connection's values itself, so the answer would not be the app's. A pragma function that describes the schema or the database file runs, for example `pragma_table_info`." A view the statement names, and every view its definition names in turn, is read the same way, since SQLite runs its definition on the same connection. Such a view is refused with "Blocked: `<name>`, which the view `<view>` reads, tells of the connection that runs it." and the rest of the sentence above. A view over a pragma function that describes the schema or the database file runs. An app table whose own name starts with `pragma_` runs, since SQLite reads the table under that name, and so does a column whose own name starts with `pragma_`
|
|
66
66
|
- **On SQLite, blocks the tables that hold another table's rows under other names**: the page modules, a shadow table of a virtual table, and a virtual table that holds a secret column. See Layer 4
|
|
67
67
|
- **Blocks a function called by a quoted name**: a quoted name can spell any function, past every check in this list that reads a function's name. A name in double quotes, backticks or brackets with `(` after it is refused on every family, a `U&"..."` name with `UESCAPE` included. The answer is "Blocked: a quoted name is followed by a parenthesis, the way a function is called. rails_query reads a function's name to decide whether it runs, so write the name without quotes." A quoted column inside a call (`count("id")`) and a quoted table before `USING (...)` run
|
|
68
68
|
- **Blocks session-effecting / administrative functions**: a read-only transaction does not stop these, and their effect lingers on the pooled connection, so a call is matched on the raw text: the name with `(` after it is refused, inside a string literal too. A bare name inside a literal is data. PostgreSQL: `pg_terminate_backend`, `pg_cancel_backend`, `pg_reload_conf`, `pg_stat_reset*`, `pg_switch_wal`, `pg_create_restore_point`, replication-slot and -origin functions, `pg_logical_emit_message`, `pg_advisory_lock*`, `set_config`, `pg_notify`, `txid_current` and their relatives; MySQL and MariaDB: `GET_LOCK`, `RELEASE_LOCK`, `RELEASE_ALL_LOCKS`, `IS_FREE_LOCK`, `IS_USED_LOCK`, `MASTER_POS_WAIT`, `SOURCE_POS_WAIT`. Each list is held to its own family. This name list is the first layer; on PostgreSQL the plan's VOLATILE-function check (below) closes the rest of the family
|
|
@@ -84,14 +84,14 @@ The textual layer cannot see a whole-row serialisation (`SELECT row_to_json(u.*)
|
|
|
84
84
|
- calls a function that returns a whole table as one value, or reads a setting that can hold a password (`current_setting` with a refused name, or with a name the plan does not show as one constant);
|
|
85
85
|
- holds a **whole-row reference** (`alias.*`) to a relation that has a secret column - this closes `row_to_json(u)`, `to_json(u)`, `u::text`, `array_agg(u)`, `json_agg(u)`, `hstore(u)`, `concat(u.*)`, `string_agg(u::text, ...)`, a bare `SELECT u`, and the same through a view, subquery or CTE. An alias is read as the plan prints it, so two aliases that differ only by case are two relations. The planner numbers an alias a statement uses twice (`p_1` for the `p` inside a subquery), and that name stands for its own relation alone. The number is taken off only for the members of an Append or a Merge Append, which is how the partitions of one table are named. The refusal names the relation whose row the statement reads;
|
|
86
86
|
- **uses a secret column** anywhere but as a plain pass-through output - a `ROW(u.*)` the planner expanded into named columns, or a `WHERE` / `ORDER BY` oracle, an ordinal `ORDER BY 3` over a wildcard included. The statement a `postgres_fdw` foreign scan sends to the remote server (the plan's `Remote SQL`) is read the same way, so a condition, a sort or a group on a secret column of a foreign table is refused where the remote server would run it. A column that carries another name on the remote server (the `column_name` option) is told by one more catalog read, made only for a plan that holds a foreign scan. A stated limit: only the remote statement of `postgres_fdw` is read, and another foreign data wrapper prints keys of its own. A join that `postgres_fdw` runs on the remote server has one scan for all its tables, which names them with their aliases in its `Relations` line: they are read from there, so a whole row of one is refused and a column of one is traced and filtered like a local table's;
|
|
87
|
-
- **passes a secret column** through an intermediate the plan's top node does not name - a `MATERIALIZED` CTE, a subquery scan, or a set operation (`INTERSECT` / `EXCEPT`) - so its value would reach a result column that the output-provenance redaction (Layer 4) cannot pin. These are refused, since no redaction could reach them. A column that a scan hands to an aggregate or a
|
|
88
|
-
- names any **VOLATILE function** (`pg_proc.provolatile = 'v'`) outside a short read-only allowlist. `pg_proc` marks a function VOLATILE when its answer may change within one statement, which is as true of a reader of live server state as of a writer, so VOLATILE alone does not mean "changes state". The allowlist holds only built-in C functions that write nothing: value generators (`random`, `clock_timestamp`, `gen_random_uuid`, `timeofday`, `uuid_generate_v4`, and `pg_sleep*`, which the statement timeout bounds); the size readers (`pg_total_relation_size`, `pg_relation_size`, `pg_table_size`, `pg_indexes_size`, `pg_database_size`, `pg_tablespace_size`); the lock-table snapshot readers (`pg_lock_status` - what the `pg_locks` view reads - `pg_blocking_pids`, `pg_safe_snapshot_blocking_pids`); and the recovery / WAL-position readers (`pg_is_in_recovery`, `pg_is_wal_replay_paused`, `pg_get_wal_replay_pause_state`, `pg_current_wal_lsn`, `pg_current_wal_insert_lsn`, `pg_current_wal_flush_lsn`, `pg_last_wal_receive_lsn`, `pg_last_wal_replay_lsn`, `pg_last_xact_replay_timestamp`). The functions the statement's text names are checked beside those the plan prints, since a plan prints no row of a `VALUES` list and no `LIMIT` expression. Anything else VOLATILE is not run, and the refusal says plainly that the function is VOLATILE and outside the read-only list. This closes the admin-function family without a name list that has to grow with every PostgreSQL release.
|
|
87
|
+
- **passes a secret column** through an intermediate the plan's top node does not name - a `MATERIALIZED` CTE, a subquery scan, or a set operation (`INTERSECT` / `EXCEPT`) - so its value would reach a result column that the output-provenance redaction (Layer 4) cannot pin. These are refused, since no redaction could reach them. A column that a scan hands to an aggregate, a join or a Result node is counted only when that node names it, so `count(*)` over a table with a secret column runs, and so does an `EXISTS` inside an `EXISTS`, whose plan puts a Result over the inner scan;
|
|
88
|
+
- names any **VOLATILE function** (`pg_proc.provolatile = 'v'`) outside a short read-only allowlist. `pg_proc` marks a function VOLATILE when its answer may change within one statement, which is as true of a reader of live server state as of a writer, so VOLATILE alone does not mean "changes state". The allowlist holds only built-in functions and extension C functions that write nothing: value generators (`random`, `clock_timestamp`, `gen_random_uuid`, `timeofday`, `uuid_generate_v4`, and `pg_sleep*`, which the statement timeout bounds); the size readers (`pg_total_relation_size`, `pg_relation_size`, `pg_table_size`, `pg_indexes_size`, `pg_database_size`, `pg_tablespace_size`); the lock-table snapshot readers (`pg_lock_status` - what the `pg_locks` view reads - `pg_blocking_pids`, `pg_safe_snapshot_blocking_pids`); and the recovery / WAL-position readers (`pg_is_in_recovery`, `pg_is_wal_replay_paused`, `pg_get_wal_replay_pause_state`, `pg_current_wal_lsn`, `pg_current_wal_insert_lsn`, `pg_current_wal_flush_lsn`, `pg_last_wal_receive_lsn`, `pg_last_wal_replay_lsn`, `pg_last_xact_replay_timestamp`); and `pg_sequence_last_value`, which reads a sequence's stored value the way `SELECT last_value FROM a_sequence` does and is what the `pg_sequences` view reads. A name on the list vouches for the server's own function or a C function alone, such as an extension's `uuid_generate_v4`: while the database holds a VOLATILE function of that name in another language outside `pg_catalog`, every call of the name is refused. The functions the statement's text names are checked beside those the plan prints, since a plan prints no row of a `VALUES` list and no `LIMIT` expression. The name after `TABLESAMPLE` is a sampling method, whose handler the statement never calls, so it is not read as a call; the method's arguments and its `REPEATABLE` seed are, as the plan prints them. Anything else VOLATILE is not run, and the refusal says plainly that the function is VOLATILE and outside the read-only list, or, for a name on the list, that the database holds another VOLATILE function of that name. This closes the admin-function family without a name list that has to grow with every PostgreSQL release.
|
|
89
89
|
|
|
90
|
-
Every statement except `SHOW` is planned, and an `EXPLAIN` is planned as the statement it explains. A statement whose plan cannot be read does not run: when planning times out the answer is the timeout, when planning fails it is the SQL error, and when the server returns no plan the statement is refused.
|
|
90
|
+
Every statement except `SHOW` is planned, and an `EXPLAIN` is planned as the statement it explains. `EXPLAIN (GENERIC_PLAN)` is planned with that option too, read as the server reads it (its last value, quoted or not), so a statement with `$1` in it is planned as PostgreSQL 16 and later plan it. A statement whose plan cannot be read does not run: when planning times out the answer is the timeout, when planning fails it is the SQL error, and when the server returns no plan the statement is refused.
|
|
91
91
|
|
|
92
92
|
A whole-row read of a relation with no secret column (`row_to_json(posts.*)`) is allowed. Before these checks scan a plan expression, its string constants are masked (PostgreSQL's deparser always doubles quotes and never emits an `E''` string, so a constant is matched exactly), so a secret word inside a `'%key%'::text` filter or a literal argument is not mistaken for a column or a function.
|
|
93
93
|
|
|
94
|
-
A subquery or a join over a table that holds a secret column runs when it reads none of them. A scan under an aggregate or a
|
|
94
|
+
A subquery or a join over a table that holds a secret column runs when it reads none of them. A scan under an aggregate, a join or a Result node hands up every column of its table, and the plan check counts a secret column only when that node names it. So `SELECT (SELECT count(*) FROM users) AS n`, `count(id)`, `max(id)` and `EXISTS (SELECT 1 FROM users)` run, while `max(api_token)`, `count(api_token)`, `string_agg(api_token, ',')`, `row_to_json(users)`, a whole row and `count(users)` are refused.
|
|
95
95
|
|
|
96
96
|
#### Relations and settings that are never read
|
|
97
97
|
|
|
@@ -148,7 +148,7 @@ What bounds each statement a `rails_query` call sends:
|
|
|
148
148
|
| The query, and `explain: true` | `statement_timeout`, a wait on a lock included | the session's statement limit, and `lock_wait_timeout` for a wait on a lock | the child process's deadline |
|
|
149
149
|
| The reads beside it: the schema read behind the secret column rule, and on PostgreSQL the plan, its `pg_proc` lookup and the column trace | `statement_timeout`, a wait on a lock included | the session's statement limit, and `lock_wait_timeout` for a wait on a lock | no statement limit; a wait on another writer ends at the connection's busy timeout |
|
|
150
150
|
|
|
151
|
-
The query has `query_timeout` to itself. The reads beside it share a second `query_timeout`, and each is charged the time it ran, so the statements of a call take about twice `query_timeout` at most. Building the app's context, which a cold call does first, is charged to neither, and so are the gate's checks on the statement's text, which the statement length limit bounds. The schema read is a few statements whatever the number of tables: four at most on MySQL and MariaDB, five on PostgreSQL and six on SQLite. On a SQLite file database where a query process can run, two more reads ask what the app's own connection alone holds, so eight reads come before the statement there. On PostgreSQL a plan that holds a foreign scan adds one catalog read. A statement or a view definition that names another schema or database adds the reads of that one.
|
|
151
|
+
The query has `query_timeout` to itself. The reads beside it share a second `query_timeout`, and each is charged the time it ran, so the statements of a call take about twice `query_timeout` at most. On PostgreSQL, MySQL and MariaDB the reads with no bound are of a setting's own value for the settings note, which wait on no lock. Building the app's context, which a cold call does first, is charged to neither, and so are the gate's checks on the statement's text, which the statement length limit bounds. The schema read is a few statements whatever the number of tables: four at most on MySQL and MariaDB, five on PostgreSQL and six on SQLite. On a SQLite file database where a query process can run, two more reads ask what the app's own connection alone holds, so eight reads come before the statement there. On PostgreSQL a plan that holds a foreign scan adds one catalog read. A statement or a view definition that names another schema or database adds the reads of that one.
|
|
152
152
|
|
|
153
153
|
A read that runs out of that time ends the call with an error that says which side of the statement it was on:
|
|
154
154
|
|
|
@@ -166,13 +166,15 @@ Stated limits of the time bound:
|
|
|
166
166
|
|
|
167
167
|
A database leased by name connects with the `connect_timeout` of its `config/database.yml` entry, or `query_timeout` rounded up to whole seconds when the entry sets none, so a server that never answers the connection ends the call. On MySQL and MariaDB such a connection also gets a `read_timeout` of twice `query_timeout` plus 5 seconds unless its entry sets one, so a server that goes silent after the handshake ends the call. On PostgreSQL `connect_timeout` is the only client bound. Taking the lease down does not replace the outcome of the call: its connections are thrown away and its pool removed, and an error there is not raised. The primary uses the app's own pool and its settings.
|
|
168
168
|
|
|
169
|
-
The MySQL session variable bounds a statement that starts with `WITH` or `EXPLAIN ANALYZE` too; the `MAX_EXECUTION_TIME` hint binds only the outermost `SELECT`, so it is not used. `query_timeout` is a positive number, and PostgreSQL, MySQL and MariaDB bound a value under a millisecond at one millisecond. A timeout is read from the error's class or the database's native code alone, so an SQL error whose message names a column called `timeout` is reported as an SQL error. A wait that PostgreSQL's own `lock_timeout` ended is answered as the timeout. A connection the server dropped is no timeout: on Rails 7.1 and later the answer is an error that starts "The <name> database stopped answering during the query:".
|
|
169
|
+
The MySQL session variable bounds a statement that starts with `WITH` or `EXPLAIN ANALYZE` too; the `MAX_EXECUTION_TIME` hint binds only the outermost `SELECT`, so it is not used. `query_timeout` is a positive number, and PostgreSQL, MySQL and MariaDB bound a value under a millisecond at one millisecond. A timeout is read from the error's class or the database's native code alone, so an SQL error whose message names a column called `timeout` is reported as an SQL error. On PostgreSQL a statement another session cancels (`pg_cancel_backend`) raises the timeout's class too, and the server's message alone tells them apart: a cancel, whose message ends "due to user request", is answered as the SQL error it is. A server that writes its messages in another language answers such a cancel as the timeout. A wait that PostgreSQL's own `lock_timeout` ended is answered as the timeout. A connection the server dropped is no timeout: on Rails 7.1 and later the answer is an error that starts "The <name> database stopped answering during the query:".
|
|
170
170
|
|
|
171
171
|
On MySQL and MariaDB the session that holds the read-only transaction and the time limit is the only one the statement is sent on. Rails replaces a connection it lost before the transaction begins, and the new session has neither guard. So the session's id is read inside the transaction, with one `SELECT CONNECTION_ID()`, and compared with the one the limits were set on. When they differ the answer is "The <name> database stopped answering during the query: the connection was replaced before the statement was sent, and the new session has no read-only transaction and no time limit, so the statement was not sent".
|
|
172
172
|
|
|
173
173
|
On PostgreSQL and MySQL the query executes inside a transaction, then rolls back (even if it could write, it can't). MySQL's read-only transaction still lets `FOR SHARE`, `LOCK IN SHARE MODE` and `INTO` through, so the Layer 1 checks stay on every family.
|
|
174
174
|
|
|
175
|
-
On a connection that is already inside a transaction (a tool called from a transactional test, a mounted server inside a request's transaction) the guard leaves that transaction as it found it. PostgreSQL guards inside a savepoint, so the outer transaction stays writable. MySQL and MariaDB cannot make an open transaction read-only, so the statement is not sent and the answer is "Query failed: MySQL cannot make a transaction that is already open read-only, so the statement was not sent. Run it outside the transaction." SQLite runs the statement on the app's own connection, so it reads the rows of the open transaction. It runs under `PRAGMA query_only = ON`, and the pragma is put back as it was found.
|
|
175
|
+
On a connection that is already inside a transaction (a tool called from a transactional test, a mounted server inside a request's transaction, a transaction the app opened with a raw `BEGIN` that Active Record does not count) the guard leaves that transaction open as it found it, with its writes. PostgreSQL guards inside a savepoint, chosen by the server's own transaction status, so the outer transaction stays writable. MySQL and MariaDB cannot make an open transaction read-only: the server refuses `SET TRANSACTION READ ONLY` inside one, and a `BEGIN` there would commit it, so no `BEGIN` is sent, the statement is not sent, and the answer is "Query failed: MySQL cannot make a transaction that is already open read-only, so the statement was not sent. Run it outside the transaction." SQLite runs the statement on the app's own connection, so it reads the rows of the open transaction. It runs under `PRAGMA query_only = ON`, and the pragma is put back as it was found.
|
|
176
|
+
|
|
177
|
+
A statement that reads a setting the guard sets for it answers the guard's value: `statement_timeout` and `transaction_read_only` on PostgreSQL, the statement limit (`max_execution_time` or `max_statement_time`) and `lock_wait_timeout` on MySQL and MariaDB. When the statement reads one of them (`SHOW name`, `@@name`, `@@SESSION.name`, `@@LOCAL.name` or `current_setting('name')`, the name backquoted or not), or a `SHOW VARIABLES` row or MariaDB's `information_schema.SESSION_VARIABLES` names one, the answer ends with a note that says so and gives the session's own value, for example "statement_timeout here is the value rails_query sets for each statement it runs. The session's own: statement_timeout 50s." A name used as an alias or a column, one in a string or a comment, and a global value on MySQL and MariaDB get no note. On SQLite a `PRAGMA` statement is no read, and the `pragma_query_only` function is refused by name.
|
|
176
178
|
|
|
177
179
|
An adapter with no family (SQL Server, Oracle, CockroachDB and any other) is refused: `rails_query` answers with the adapter's name and sends no statement, because it cannot give that database a read-only transaction and a timeout it has verified. The family is read from the database's configuration, so the refusal comes before any connection is opened or verified. CockroachDB's adapter is built on PostgreSQL's, but the adapter's name alone chooses the family, so it is refused too.
|
|
178
180
|
|
|
@@ -256,7 +258,7 @@ minus anything in `config.query_allowed_columns`. An `encrypts` column is the ex
|
|
|
256
258
|
|
|
257
259
|
**Default redacted patterns:** `password_digest`, `encrypted_password`, `password_hash`, `reset_password_token`, `confirmation_token`, `unlock_token`, `otp_secret`, `session_data`, `secret_key`, `api_key`, `api_secret`, `access_token`, `refresh_token`, `jti`. Those are the defaults of `config.query_redacted_columns`. The fixed built-in list adds `password_reset_token`, `remember_token`, `secret`, `private_key`, `authentication_string`, `rolpassword` and `setconfig`. The last three are where MySQL and PostgreSQL keep password hashes and stored settings.
|
|
258
260
|
|
|
259
|
-
The pre-execution refusal matches on two things, as whole identifiers. Names are folded the way the databases compare them
|
|
261
|
+
The pre-execution refusal matches on two things, as whole identifiers. Names are folded the way the databases compare them. On SQLite that is ASCII letters alone, so `"é"` and `"É"` are two columns there and a refusal of one leaves the other. On PostgreSQL, MySQL and MariaDB every letter is folded to lowercase, with MySQL's reading of a dotted capital I as `i`, since a collation or a database encoding can fold more than ASCII there, and a false refusal is the safe side. What a name means is read with every letter folded on every family: a column whose name folds to a built-in name or to one the heuristic flags is a secret column, and so is one a configured name folds to when no column of the database spells that name as the database folds it.
|
|
260
262
|
|
|
261
263
|
- **the fixed names above**, on every family whether or not they are real columns;
|
|
262
264
|
- **every real column of your schema that the rule flags** - read from the connection the query runs on, so a database named with `database` is checked against its own columns. This catches an app-specific column the heuristic flags (`api_token`, `auth_token`) through any alias or expression. Rails' own bookkeeping tables (`ar_internal_metadata`, `schema_migrations`) are left out: their columns hold no secrets, and `ar_internal_metadata.key` ends in "key", so counting it would refuse every query whose text held the word "key".
|
|
@@ -271,7 +273,7 @@ The rule follows what a statement reaches. It starts from these columns, by data
|
|
|
271
273
|
|
|
272
274
|
A schema or database is told by a name before a dot, and the server says which of those names are schemas. Each name is read backward from its dot, bare or in backticks or double quotes, so nothing written before the name changes how it is read, a quote character inside a string included. A name in brackets is not read: SQLite reads every schema whatever a statement names, and on MySQL and MariaDB Layer 1 refuses the bracket. A view the statement names can name a further schema or database in its definition, and a view named there a further one. The rule follows those views three reads deep. A view no statement names is not followed, so a statement that names no other schema or database and no such view costs no extra read. On PostgreSQL the plan reads through a plain view, so only a materialized view's definition is followed. A statement that names a view that leads deeper is not run: "The view `<view>` leads through more databases than rails_query follows, so the secret columns it could read cannot be told and no statement that names it runs." On PostgreSQL the sentence says "schemas". A schema whose columns cannot be read stops the statement too: "The columns of `<name>` could not be read, so the secret columns there cannot be told and no query that reads it runs." On PostgreSQL a relation the plan names and the rule could not read is refused by the plan check of Layer 1b. So a statement that names a secret column of a table in another schema or database is refused by name, as one in the connection's own is.
|
|
273
275
|
|
|
274
|
-
A refusal names a view, a table or a relation as the database spells it. The rule itself compares names case folded, with a run of spaces as one space. So two names that differ by a run of spaces alone are one name to the rule, and a refusal names the first of them it read.
|
|
276
|
+
A refusal names a view, a table or a relation as the database spells it. The rule itself compares names case folded as above, with a run of spaces as one space. So two names that differ by a run of spaces alone are one name to the rule, and a refusal names the first of them it read.
|
|
275
277
|
|
|
276
278
|
The schema read fails closed. When the columns of the database cannot be read, no query runs on it and the answer is "The columns of the <name> database could not be read, so its secret columns cannot be told and no query runs on it." Only the primary falls back to the cached context's schema, and it is refused the same way when that has no tables either.
|
|
277
279
|
|
data/docs/TOOLS.md
CHANGED
|
@@ -147,10 +147,7 @@ dump was not read is named with "(tables not read)" and left out of the table
|
|
|
147
147
|
count, and the summary says so: "5 tables in 2 databases, the warehouse
|
|
148
148
|
database not read". It ends with the app's custom rake tasks (Rakefile,
|
|
149
149
|
`lib/tasks`, `rakelib`), each with its arguments, description and file: the
|
|
150
|
-
first 15 at `standard`, every one at `full`.
|
|
151
|
-
`preset: standard` on the static tier the stack line reads the primary's
|
|
152
|
-
adapter as `unknown`, and a database's heading in the generated files can name
|
|
153
|
-
no adapter.
|
|
150
|
+
first 15 at `standard`, every one at `full`.
|
|
154
151
|
Then the app's own generators under `lib/generators` (the `bin/rails generate`
|
|
155
152
|
command and its USAGE line), the `lib/templates` files that replace a built-in
|
|
156
153
|
generator's template, and the Railties under `lib/` with their initializers.
|
|
@@ -170,7 +167,10 @@ generator's template, and the Railties under `lib/` with their initializers.
|
|
|
170
167
|
Database schema with column types, indexes, defaults, encrypted hints. Booted,
|
|
171
168
|
a table `db/schema.rb` declares and the connected database does not have is
|
|
172
169
|
named as a migration that has not run, and the listing header says when the
|
|
173
|
-
two table counts disagree.
|
|
170
|
+
two table counts disagree. Where two databases both dump to `db/schema.rb` (the
|
|
171
|
+
first one and the one named `primary`), the file may hold the other one's
|
|
172
|
+
tables, so such a table is named as not in the connected database, with no
|
|
173
|
+
word of a migration.
|
|
174
174
|
|
|
175
175
|
The other databases are listed after the primary's tables: every database any
|
|
176
176
|
environment in `config/database.yml` configures, so a Rails 8 app's queue,
|
|
@@ -184,10 +184,9 @@ or `schema_format` says.
|
|
|
184
184
|
A database only another environment configures carries a mark, as in
|
|
185
185
|
`cache (production only)`, and a first database that is not named `primary`
|
|
186
186
|
is listed under its own name where another database is the one named
|
|
187
|
-
`primary`. Both are the same under every `preset
|
|
188
|
-
`multi_database` introspector off, as on `preset: standard`,
|
|
189
|
-
|
|
190
|
-
from the app databases list.
|
|
187
|
+
`primary`. Both are the same under every `preset`, and so is each database's
|
|
188
|
+
adapter: with the `multi_database` introspector off, as on `preset: standard`,
|
|
189
|
+
the context still carries the app databases list.
|
|
191
190
|
|
|
192
191
|
Column hints follow the leftmost-prefix rule: `[indexed]` means some index leads
|
|
193
192
|
with the column, so a lookup on it alone can use one; `[unique]` means a unique
|
|
@@ -655,7 +654,7 @@ ViewComponent/Phlex components: props, slots, previews, sidecar assets, usage ex
|
|
|
655
654
|
|
|
656
655
|
### `rails_get_mailers`
|
|
657
656
|
|
|
658
|
-
ActionMailer mailers: every mailer class with its delivery actions and delivery method. A mailer that declares no action of its own - one taking them from a gem base or a mixin, or one called through class methods - is listed with where its actions come from
|
|
657
|
+
ActionMailer mailers: every mailer class with its delivery actions and delivery method. The actions include the public methods a mailer inherits from the app's own mailers below `ApplicationMailer`, on both tiers. A mailer that declares no action of its own - one taking them from a gem base or a mixin, or one called through class methods - is listed with where its actions come from when that is a gem base it reaches without passing `ApplicationMailer`, such as `Devise::Mailer`, and never Rails' own base. A base other mailers inherit from is not one: it leaves the listing, and a line above names the ones left out. Asked for by name, a base answers with what every mailer below it inherits - its layout, helpers, defaults, callbacks and mixins as the app wrote them, the methods it defines - and which mailers inherit it.
|
|
659
658
|
|
|
660
659
|
Each mailer also shows its own `default`, `layout`, `helper` and action and
|
|
661
660
|
deliver callbacks as written, the template formats of each action under
|
|
@@ -747,7 +746,7 @@ Safe read-only SQL with layered security: the SQL gate, a PostgreSQL plan check
|
|
|
747
746
|
|
|
748
747
|
It runs on PostgreSQL, MySQL (with MariaDB) and SQLite, each with its own read-only transaction and statement timeout. On any other adapter it answers with that adapter's name and runs nothing. A statement that runs past `query_timeout` is answered as a timeout and any other database error as an SQL error. Both are error responses: `isError` over MCP and a non-zero exit on the CLI. So is a database no connection reaches, which the answer names with what to do about it. So are a locked SQLite database and the refusal in production.
|
|
749
748
|
|
|
750
|
-
On PostgreSQL, MySQL and MariaDB every statement a call sends has a time limit. The query has `query_timeout` to itself and the reads beside it share a second one, so the statements of a call take about twice `query_timeout` at most. On SQLite the query has `query_timeout`, and the reads beside it run on the app's own connection with no statement limit. A SQLite statement that names something the app's connection alone holds, such as a temp table or an attached database, runs on that connection, and its answer says it ran without a time limit. The [security model](SECURITY.md#layer-2---database-level-read-only) says what bounds each statement and lists the stated limits.
|
|
749
|
+
On PostgreSQL, MySQL and MariaDB every statement a call sends has a time limit, apart from the reads of a setting's own value for the settings note, which wait on no lock. The query has `query_timeout` to itself and the reads beside it share a second one, so the statements of a call take about twice `query_timeout` at most. On SQLite the query has `query_timeout`, and the reads beside it run on the app's own connection with no statement limit. A SQLite statement that names something the app's connection alone holds, such as a temp table or an attached database, runs on that connection, and its answer says it ran without a time limit. The [security model](SECURITY.md#layer-2---database-level-read-only) says what bounds each statement and lists the stated limits.
|
|
751
750
|
|
|
752
751
|
On MySQL, MariaDB and SQLite a statement that names a view whose definition names a secret column is refused. So is a `*` inside a subquery when the statement names a table that holds a secret column, and on SQLite `IN` before a table's name (`x IN t`). On PostgreSQL a materialized view whose definition names a secret column is refused. On SQLite a virtual table that holds a secret column, a shadow table and the page modules are refused, and only the pragma functions that describe the schema or the database file run.
|
|
753
752
|
|
|
@@ -834,6 +833,8 @@ Table sizes come from PostgreSQL, MySQL (base tables only) and SQLite, and index
|
|
|
834
833
|
|
|
835
834
|
The pool and the cache are read inside the MCP server process, and the answer says so: the pool's counts are that process's own connections, and a `MemoryStore`'s entries are its own, since each app server process holds its own. A store the processes share (Redis, Memcached, Solid Cache, a file store) is the app's. Another environment that configures a shared store is named, as where the app's cache lives there. Solid Cache's stats are read from its database, so when that is a SQLite file that is not there, the cache section says the database does not exist yet and creates nothing.
|
|
836
835
|
|
|
836
|
+
The jobs section names the Active Job adapter the running environment uses. Queue counts are read from Sidekiq alone: they are the app's when Sidekiq is that adapter, and under another adapter they are shown under their own heading only when the app has Sidekiq workers of its own, as those workers' counts.
|
|
837
|
+
|
|
837
838
|
| Parameter | Type | Default | Description |
|
|
838
839
|
|:----------|:-----|:--------|:------------|
|
|
839
840
|
| `detail` | enum | `standard` | `summary`, `standard`, `full` |
|
data/docs/TROUBLESHOOTING.md
CHANGED
|
@@ -311,7 +311,7 @@ A view over one of these is refused with "its definition names `<table>`, which
|
|
|
311
311
|
|
|
312
312
|
### "Blocked: `pragma_foreign_keys` tells of the connection that runs it"
|
|
313
313
|
|
|
314
|
-
On SQLite `rails_query` runs a statement on a connection of its own, so a value that belongs to a connection would not be the app's. The pragma functions that describe the schema or the database file run: `pragma_table_info`, `pragma_index_list`, `pragma_foreign_key_list`, `pragma_integrity_check`, `pragma_page_count` and the others [SECURITY.md](SECURITY.md#layer-1---the-sql-gate-statement-checks) lists. Every other `pragma_*` function is refused, and so are `changes()`, `total_changes()` and `last_insert_rowid()`. Read those on the app's own connection, in `bin/rails runner` or the console. A
|
|
314
|
+
On SQLite `rails_query` runs a statement on a connection of its own, so a value that belongs to a connection would not be the app's. The pragma functions that describe the schema or the database file run: `pragma_table_info`, `pragma_index_list`, `pragma_foreign_key_list`, `pragma_integrity_check`, `pragma_page_count` and the others [SECURITY.md](SECURITY.md#layer-1---the-sql-gate-statement-checks) lists. Every other `pragma_*` function is refused, and so are `changes()`, `total_changes()` and `last_insert_rowid()`. Read those on the app's own connection, in `bin/rails runner` or the console. A column whose own name starts with `pragma_` runs. Its name is refused only where SQLite would read the function: called, or where a table stands.
|
|
315
315
|
|
|
316
316
|
### "Blocked: the statement is ... bytes long"
|
|
317
317
|
|
|
@@ -7,6 +7,9 @@ module RailsAiContext
|
|
|
7
7
|
module MysqlRuntime
|
|
8
8
|
module_function
|
|
9
9
|
|
|
10
|
+
# A transaction is open that Active Record does not count, which a BEGIN would commit.
|
|
11
|
+
TRANSACTION_ALREADY_OPEN = 1568
|
|
12
|
+
|
|
10
13
|
IN_TRANSACTION = "MySQL cannot make a transaction that is already open read-only, so the statement " \
|
|
11
14
|
"was not sent. Run it outside the transaction."
|
|
12
15
|
|
|
@@ -41,7 +44,13 @@ module RailsAiContext
|
|
|
41
44
|
# before the first statement in the block. Unscoped, it applies to the next transaction only.
|
|
42
45
|
def in_read_only_transaction(conn, sql, session)
|
|
43
46
|
result = nil
|
|
44
|
-
|
|
47
|
+
begin
|
|
48
|
+
conn.execute("SET TRANSACTION READ ONLY")
|
|
49
|
+
rescue ActiveRecord::StatementInvalid => e
|
|
50
|
+
raise Unsupported, IN_TRANSACTION if DatabaseFamily.native_code(e) == TRANSACTION_ALREADY_OPEN
|
|
51
|
+
|
|
52
|
+
raise
|
|
53
|
+
end
|
|
45
54
|
conn.transaction do
|
|
46
55
|
# Rails replaces a connection it lost before BEGIN, and a new session has neither guard.
|
|
47
56
|
raise ActiveRecord::ConnectionNotEstablished, REPLACED unless conn.select_value("SELECT CONNECTION_ID()").to_i == session.to_i
|
|
@@ -52,6 +61,12 @@ module RailsAiContext
|
|
|
52
61
|
result
|
|
53
62
|
end
|
|
54
63
|
|
|
64
|
+
# The settings bounded sets for each statement => the read of the session's own value.
|
|
65
|
+
def bound_settings(conn)
|
|
66
|
+
limit = conn.mariadb? ? "max_statement_time" : "max_execution_time"
|
|
67
|
+
{ limit => "SELECT @@SESSION.#{limit}", "lock_wait_timeout" => "SELECT @@SESSION.lock_wait_timeout" }
|
|
68
|
+
end
|
|
69
|
+
|
|
55
70
|
def explain(conn, sql, timeout)
|
|
56
71
|
# MySQL 8.3+ defaults to TREE output, which has no table/type/key columns. MariaDB has no FORMAT=TRADITIONAL.
|
|
57
72
|
prefix = conn.mariadb? ? "EXPLAIN" : "EXPLAIN FORMAT=TRADITIONAL"
|
|
@@ -90,7 +105,7 @@ module RailsAiContext
|
|
|
90
105
|
[]
|
|
91
106
|
end
|
|
92
107
|
|
|
93
|
-
def plan(_conn, _sql, _timeout)
|
|
108
|
+
def plan(_conn, _sql, _timeout, **)
|
|
94
109
|
nil
|
|
95
110
|
end
|
|
96
111
|
|
|
@@ -10,11 +10,15 @@ module RailsAiContext
|
|
|
10
10
|
NAME = /(?:([A-Za-z_]\w*)|"((?:""|[^"])+)")/
|
|
11
11
|
BASE_VAR = /\A#{NAME}\.#{NAME}\z/
|
|
12
12
|
# Nodes whose every read of a child's column is an expression of their own that names it.
|
|
13
|
-
READS_BY_NAME = [ "Aggregate", "Hash Join", "Merge Join", "Nested Loop" ].freeze
|
|
13
|
+
READS_BY_NAME = [ "Aggregate", "Hash Join", "Merge Join", "Nested Loop", "Result" ].freeze
|
|
14
14
|
|
|
15
15
|
# Catalogs and TOAST tables: the never-read list and the catalog credentials hold what is secret there.
|
|
16
16
|
SYSTEM_SCHEMA = /\A(?:pg_catalog|information_schema|pg_toast(?:_temp_\d+)?)\z/
|
|
17
17
|
|
|
18
|
+
# Before PostgreSQL 15 a plan names the session's temporary schema by its number, where the catalog reads say pg_temp.
|
|
19
|
+
# Another session's temporary table can be planned but never read, so a statement over one fails when it runs.
|
|
20
|
+
OWN_TEMP_SCHEMA = /\Apg_temp_\d+\z/
|
|
21
|
+
|
|
18
22
|
APPENDS = [ "Append", "Merge Append" ].freeze
|
|
19
23
|
REMOTE_NAME = /"(?:""|[^"])+"|[^\s()."]+/
|
|
20
24
|
REMOTE_RELATION = /\((?:(#{REMOTE_NAME})\.)?(#{REMOTE_NAME})(?: (#{REMOTE_NAME}))?\)/
|
|
@@ -95,7 +99,7 @@ module RailsAiContext
|
|
|
95
99
|
# no relation of its own: postgres_fdw names its tables there, each with its alias, as `(schema.table alias)`.
|
|
96
100
|
def scanned(node)
|
|
97
101
|
rel = node["Relation Name"]
|
|
98
|
-
return [ [ node["Schema"], rel.to_s, (node["Alias"] || rel).to_s ] ] if rel
|
|
102
|
+
return [ [ node["Schema"]&.sub(OWN_TEMP_SCHEMA, "pg_temp"), rel.to_s, (node["Alias"] || rel).to_s ] ] if rel
|
|
99
103
|
|
|
100
104
|
node["Relations"].to_s.scan(REMOTE_RELATION).map do |parts|
|
|
101
105
|
schema, name, as = parts.map { |part| part&.delete_prefix('"')&.delete_suffix('"')&.gsub('""', '"') }
|
|
@@ -9,6 +9,12 @@ module RailsAiContext
|
|
|
9
9
|
module_function
|
|
10
10
|
|
|
11
11
|
def read_only(conn, sql, timeout)
|
|
12
|
+
# An EXPLAIN answers text, and one of a generic plan holds $1, which Rails 7.0's select_all sends as an empty bind.
|
|
13
|
+
if sql.match?(/\A\s*EXPLAIN\b/i)
|
|
14
|
+
result = bounded(conn, timeout) { conn.execute(sql) }
|
|
15
|
+
return Rows.new(result.fields, result.values, false)
|
|
16
|
+
end
|
|
17
|
+
|
|
12
18
|
result = bounded(conn, timeout) { conn.select_all(sql) }
|
|
13
19
|
Rows.new(result.columns, result.rows, false)
|
|
14
20
|
end
|
|
@@ -18,16 +24,41 @@ module RailsAiContext
|
|
|
18
24
|
def bounded(conn, timeout)
|
|
19
25
|
result = nil
|
|
20
26
|
DatabaseFamily.bounding do
|
|
21
|
-
conn
|
|
27
|
+
rolling_back(conn) do
|
|
22
28
|
conn.execute("SET TRANSACTION READ ONLY")
|
|
23
29
|
conn.execute("SET LOCAL statement_timeout = '#{[ (timeout * 1000).to_i, 1 ].max}'")
|
|
24
30
|
result = yield
|
|
25
|
-
raise ActiveRecord::Rollback
|
|
26
31
|
end
|
|
27
32
|
end
|
|
28
33
|
result
|
|
29
34
|
end
|
|
30
35
|
|
|
36
|
+
# PG::PQTRANS_IDLE, which needs the pg gem loaded.
|
|
37
|
+
TRANSACTION_IDLE = 0
|
|
38
|
+
|
|
39
|
+
# The server's own transaction status decides: Active Record does not count a transaction the app opened with a
|
|
40
|
+
# raw BEGIN, and its own BEGIN and ROLLBACK there would end the app's.
|
|
41
|
+
def rolling_back(conn)
|
|
42
|
+
if conn.raw_connection.transaction_status == TRANSACTION_IDLE
|
|
43
|
+
conn.transaction(requires_new: true) do
|
|
44
|
+
yield
|
|
45
|
+
raise ActiveRecord::Rollback
|
|
46
|
+
end
|
|
47
|
+
else
|
|
48
|
+
conn.execute("SAVEPOINT rac_bounded")
|
|
49
|
+
begin
|
|
50
|
+
yield
|
|
51
|
+
ensure
|
|
52
|
+
DatabaseFamily.restore(conn, "ROLLBACK TO SAVEPOINT rac_bounded; RELEASE SAVEPOINT rac_bounded")
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# The settings bounded sets for each statement => the read of the session's own value.
|
|
58
|
+
def bound_settings(_conn)
|
|
59
|
+
{ "statement_timeout" => "SHOW statement_timeout", "transaction_read_only" => "SHOW transaction_read_only" }
|
|
60
|
+
end
|
|
61
|
+
|
|
31
62
|
def explain(conn, sql, timeout)
|
|
32
63
|
raw = read_only(conn, "EXPLAIN (FORMAT JSON, ANALYZE) #{sql}", timeout).rows.map { |r| r.first.to_s }.join("\n")
|
|
33
64
|
scans = []
|
|
@@ -44,8 +75,8 @@ module RailsAiContext
|
|
|
44
75
|
|
|
45
76
|
# Planning only: EXPLAIN without ANALYZE runs nothing, not even a VOLATILE function.
|
|
46
77
|
# A timeout and an SQL error are raised, so the caller never runs what it could not plan.
|
|
47
|
-
def plan(conn, sql, timeout)
|
|
48
|
-
json = read_only(conn, "EXPLAIN (VERBOSE, FORMAT JSON) #{sql}", timeout).rows.first&.first
|
|
78
|
+
def plan(conn, sql, timeout, generic: false)
|
|
79
|
+
json = read_only(conn, "EXPLAIN (VERBOSE, FORMAT JSON#{", GENERIC_PLAN" if generic}) #{sql}", timeout).rows.first&.first
|
|
49
80
|
data = JSON.parse(json.to_s)
|
|
50
81
|
data.is_a?(Array) && data.first.is_a?(Hash) ? data.first["Plan"] : nil
|
|
51
82
|
rescue JSON::ParserError
|
|
@@ -82,8 +113,12 @@ module RailsAiContext
|
|
|
82
113
|
[ i, names[col], relation ] if names[col]
|
|
83
114
|
end
|
|
84
115
|
rescue StandardError => e
|
|
85
|
-
# The driver raises its own class here. Half a trace is no answer, so a timeout is raised as the call's.
|
|
86
|
-
|
|
116
|
+
# The driver raises its own class here. Half a trace is no answer, so a timeout is raised as the call's, and a cancel as itself.
|
|
117
|
+
if defined?(PG::QueryCanceled) && e.is_a?(PG::QueryCanceled)
|
|
118
|
+
raise if DatabaseFamily.canceled?(e)
|
|
119
|
+
|
|
120
|
+
raise TimedOut, e.message
|
|
121
|
+
end
|
|
87
122
|
raise if e.is_a?(ActiveRecord::StatementInvalid) && DatabaseFamily.timed_out?(e)
|
|
88
123
|
|
|
89
124
|
[]
|