ros-apartment 4.0.0.alpha7 → 4.0.0.alpha9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +37 -3
- data/lib/apartment/CLAUDE.md +4 -3
- data/lib/apartment/adapters/abstract_adapter.rb +14 -6
- data/lib/apartment/adapters/postgresql_database_adapter.rb +2 -0
- data/lib/apartment/adapters/postgresql_schema_adapter.rb +2 -0
- data/lib/apartment/adapters/postgresql_transaction_state.rb +34 -0
- data/lib/apartment/config.rb +69 -2
- data/lib/apartment/errors.rb +18 -9
- data/lib/apartment/migrator.rb +5 -1
- data/lib/apartment/patches/connection_handling.rb +20 -1
- data/lib/apartment/patches/postgresql_sequence_name.rb +62 -0
- data/lib/apartment/pool_manager.rb +11 -0
- data/lib/apartment/pool_observer.rb +40 -2
- data/lib/apartment/pool_reaper.rb +5 -0
- data/lib/apartment/transaction_taint.rb +123 -0
- data/lib/apartment/version.rb +1 -1
- data/lib/apartment.rb +111 -15
- data/lib/generators/apartment/install/templates/apartment.rb +14 -3
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5b4b1a4532066ffd3ebfdfa8c43b0f0cdd4e304f31f9905af8a91550e71c44a6
|
|
4
|
+
data.tar.gz: 75868c2063131bc20022d35142d87aff2c15f7c8b65b7fa3d8cbee43be4f11d1
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1f3c98442bf8ea826918440071a3a702615494d8761ef20f990135a10697650e05b72926f28a6cf1053b2aa969b67aee1b8c43249efe73ff58e8802c916f92a3
|
|
7
|
+
data.tar.gz: f23fd1d357e9b5b598516960a0d480f22512b0c8b4481827bb137cc074589a73d4684e4879dc1b2355a12a108fe1b25eaf54eff283aa6c89557c823d91a3e13a
|
data/README.md
CHANGED
|
@@ -118,15 +118,19 @@ All options are set in `config/initializers/apartment.rb` inside an `Apartment.c
|
|
|
118
118
|
|
|
119
119
|
### Pool Settings
|
|
120
120
|
|
|
121
|
-
`tenant_pool_size`: max connections per tenant pool. Default `nil` — each tenant pool inherits the app's base pool size (`DB_POOL_SIZE` / `pool:` in `database.yml`). Set it to
|
|
121
|
+
`tenant_pool_size`: max connections per tenant pool. Default `nil` — each tenant pool inherits the app's base pool size (`DB_POOL_SIZE` / `pool:` in `database.yml`). **Set it to at least the peak number of threads/fibers in one process that can touch the _same_ tenant at once** (e.g. a Sidekiq role's concurrency for a same-tenant job fan-out); a smaller pool makes those threads block on connection checkout. It is a lazy ceiling — connections are created on demand and idle pools are reaped — so sizing for peak does not hold that many connections open at steady state.
|
|
122
122
|
|
|
123
123
|
`pool_idle_timeout`: seconds an idle tenant pool must exceed before it is eligible for reaping (default: 300).
|
|
124
124
|
|
|
125
125
|
`reaper_interval`: seconds between background reap passes. Default `nil` — derives from `pool_idle_timeout`. Set it lower to reap more often without shrinking the idle window.
|
|
126
126
|
|
|
127
|
-
`
|
|
127
|
+
`max_tenant_pools`: ceiling on the number of live tenant pools per process; `nil` for unlimited (default: `nil`). Enforced synchronously at pool-creation time (see `pool_overflow_policy`) and trimmed continuously by the background reaper.
|
|
128
128
|
|
|
129
|
-
`
|
|
129
|
+
`max_tenant_connections`: ceiling on total *tenant-pool* connections per process (default: `nil`); requires `tenant_pool_size`. The admission controller derives the pool budget as `floor(max_tenant_connections / tenant_pool_size)`. Per-process tenant-pool connections ≈ `effective_pool_budget × tenant_pool_size`, where `effective_pool_budget = min(max_tenant_pools, floor(max_tenant_connections / tenant_pool_size))`; the default pool and any separate pinned pool are additional. A tenant using multiple roles (e.g. `writing` + `reading`) holds one pool per role (`tenant:role` keys), counting once per role against the budget. For a hard external-pooler budget, pair this with `pool_overflow_policy: :raise` (the ceiling is soft under the default `:evict_idle`).
|
|
130
|
+
|
|
131
|
+
`max_total_connections`: **deprecated** (removed in v5) — it capped tenant-pool *count*, not connections. It now aliases `max_tenant_pools`; rename it. For a true connection ceiling, set `max_tenant_connections`.
|
|
132
|
+
|
|
133
|
+
`pool_overflow_policy`: behavior when a new pool would breach the pool budget (`max_tenant_pools` / `max_tenant_connections`) and every existing pool is pinned or in use (no idle pool to evict). `:evict_idle` (default) — allow the new pool, emit a `cap_unmet` notification (soft cap, prioritizes availability). `:raise` — raise `Apartment::PoolCapacityReached` (hard cap, sheds load). When an idle pool *is* available it is always evicted inline regardless of policy. See `docs/designs/pool-admission-control.md`.
|
|
130
134
|
|
|
131
135
|
`reap_in_test`: keep the background reaper running under `Rails.env.test?` (default `false` — the Railtie stops it in test, where fixture transactions make mid-example eviction a liability). Set `true` if a deployed process can run under test-env semantics and must keep reaping — that's cleaner than guarding `RAILS_ENV` at boot to avoid silently leaking connections. It applies to *every* `Rails.env.test?` process, including CI, so enable it only when a real deployment needs it.
|
|
132
136
|
|
|
@@ -334,6 +338,36 @@ Platform notes: parallel migrations use threads. On macOS, libpq has known fork-
|
|
|
334
338
|
|
|
335
339
|
## Known Limitations
|
|
336
340
|
|
|
341
|
+
### Connection poolers in transaction mode (PgBouncer, RDS Proxy)
|
|
342
|
+
|
|
343
|
+
> [!WARNING]
|
|
344
|
+
> With the PostgreSQL `:schema` strategy, **PgBouncer in `transaction` pooling mode can
|
|
345
|
+
> silently serve one tenant another tenant's data.** No error is raised — queries return the
|
|
346
|
+
> wrong tenant's rows. It is safe only on **PostgreSQL 18+** with
|
|
347
|
+
> `track_extra_parameters = IntervalStyle,search_path`; on PostgreSQL 17 and older,
|
|
348
|
+
> transaction mode cannot be made safe and you must use `session` mode.
|
|
349
|
+
|
|
350
|
+
Tenant isolation under `:schema` rests on `search_path`, which is session-scoped state.
|
|
351
|
+
Transaction-mode pooling hands a different backend to each transaction, so a `search_path`
|
|
352
|
+
set by one client is not guaranteed to be the one in effect for the next query. This affects
|
|
353
|
+
v3 and v4 alike; v4 reduces the number of `SET search_path` statements to one per connection
|
|
354
|
+
but does not eliminate them, and one is enough.
|
|
355
|
+
|
|
356
|
+
**Which pooler to use:** PgBouncer on **PostgreSQL 18+** with
|
|
357
|
+
`track_extra_parameters = IntervalStyle,search_path` is the only setup that safely multiplexes
|
|
358
|
+
a schema-per-tenant app — and it needs no changes to Apartment or your application. **RDS Proxy
|
|
359
|
+
is safe but reduces no connections**: a Rails app pins on it *with or without Apartment* (three
|
|
360
|
+
unconditional `SET`s in `configure_connection`, plus the extended query protocol even at
|
|
361
|
+
`prepared_statements: false` — [rails/rails#40207](https://github.com/rails/rails/issues/40207)),
|
|
362
|
+
and AWS offers no session-pinning filters for PostgreSQL. Keep RDS Proxy if you use it for
|
|
363
|
+
failover or IAM; don't adopt it to cut connections.
|
|
364
|
+
|
|
365
|
+
Database-per-tenant (`:database_name`) is not affected.
|
|
366
|
+
|
|
367
|
+
**Read [Connection Poolers](docs/connection-poolers.md) before putting a pooler in front of a
|
|
368
|
+
schema-per-tenant app**, and verify your configuration rather than assuming it: the failure
|
|
369
|
+
is silent, so a working app proves nothing.
|
|
370
|
+
|
|
337
371
|
### `connects_to` with Separate Databases
|
|
338
372
|
|
|
339
373
|
If a model (or its abstract base class) uses `connects_to` to point at a completely different database (not just different roles on the same DB), Apartment's `connection_pool` patch will attempt to create a tenant pool for it.
|
data/lib/apartment/CLAUDE.md
CHANGED
|
@@ -20,7 +20,8 @@ lib/apartment/
|
|
|
20
20
|
│ └── mysql_config.rb # MysqlConfig: placeholder
|
|
21
21
|
├── elevators/ # Rack middleware for tenant detection (see CLAUDE.md); v4 uses constructor keyword args, no class-level state; Generic, Subdomain, FirstSubdomain, Domain, Host, HostHash, Header
|
|
22
22
|
├── patches/ # ActiveRecord patches for tenant-aware connections
|
|
23
|
-
│
|
|
23
|
+
│ ├── connection_handling.rb # Prepends on AR::Base — tenant-aware connection_pool
|
|
24
|
+
│ └── postgresql_sequence_name.rb # Prepends on the PG adapter — schema-agnostic Model.sequence_name memoization
|
|
24
25
|
├── tasks/ # Rake task utilities; v4.rake for apartment:create/drop/migrate/seed/rollback
|
|
25
26
|
├── config.rb # Configuration with validate!/freeze!
|
|
26
27
|
├── current.rb # Fiber-safe tenant context (CurrentAttributes)
|
|
@@ -59,7 +60,7 @@ lib/apartment/
|
|
|
59
60
|
|
|
60
61
|
### pool_manager.rb — Pool Cache
|
|
61
62
|
|
|
62
|
-
`Concurrent::Map` storing connection pools by tenant key. Monotonic clock timestamps for idle/LRU tracking. `stats_for` returns `{ seconds_idle: N }`. `clear` disconnects all pools before clearing. When `max_total_connections` is
|
|
63
|
+
`Concurrent::Map` storing connection pools by tenant key. Monotonic clock timestamps for idle/LRU tracking. `stats_for` returns `{ seconds_idle: N }`. `clear` disconnects all pools before clearing. When a pool budget is configured (`max_tenant_pools` and/or `max_tenant_connections`, resolved by `Config#effective_pool_budget`; `max_total_connections` is the deprecated alias of `max_tenant_pools`), `Apartment.configure` wires an `admission_controller` (the reaper) so cold creates route through a serialized, capacity-bounded path; otherwise the lock-free `compute_if_absent` fast path is used. See `docs/designs/pool-connection-budget.md` and `docs/designs/pool-admission-control.md`.
|
|
63
64
|
|
|
64
65
|
### pool_reaper.rb — Pool Eviction + Admission
|
|
65
66
|
|
|
@@ -97,7 +98,7 @@ All inherit from `AbstractAdapter`. Override `resolve_connection_config`, `creat
|
|
|
97
98
|
|
|
98
99
|
### pool_observer.rb — Observability (opt-in)
|
|
99
100
|
|
|
100
|
-
Sink-agnostic subscriber for the pool events (`create`/`evict`/`cap_unmet`/`skip_evict`/`reaper_stopped`) + an optional `Concurrent::TimerTask` gauge sampler
|
|
101
|
+
Sink-agnostic subscriber for the pool events (`create`/`evict`/`cap_unmet`/`skip_evict`/`reaper_stopped`) + an optional `Concurrent::TimerTask` gauge sampler. Gauges: `tenant_pools_live`, optional adopter `backend_count`, and per-pass checkout-pressure gauges from each tenant pool's AR `ConnectionPool#stat` — `pools_waiting` (blocked on checkout), `pools_saturated` (`busy >= size`), `max_checkout_waiting` (worst pool's `waiting`, tenant in payload not a dimension). Per-pool `#stat` is rescued so one tearing-down pool can't abort the pass. Normalizes each to a `Sample` via the shared `emit_gauge` helper and forwards to a caller `sink`; ships no transport. Error-isolated — never raises into instrumentation. See `docs/observability.md`.
|
|
101
102
|
|
|
102
103
|
### railtie.rb — v4 Rails Integration
|
|
103
104
|
|
|
@@ -68,15 +68,13 @@ module Apartment
|
|
|
68
68
|
end
|
|
69
69
|
|
|
70
70
|
# Drop a tenant.
|
|
71
|
-
def drop(tenant)
|
|
71
|
+
def drop(tenant)
|
|
72
72
|
drop_tenant(tenant)
|
|
73
73
|
removed_pools = Apartment.pool_manager&.remove_tenant(tenant) || []
|
|
74
74
|
removed_pools.each do |pool_key, pool|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
warn "[Apartment] Pool disconnect failed for '#{pool_key}': #{e.class}: #{e.message}"
|
|
79
|
-
end
|
|
75
|
+
# remove_tenant already took these out of the manager, so deregister_shard's
|
|
76
|
+
# own removal returns nil and cannot disconnect them — we close them here.
|
|
77
|
+
Apartment.disconnect_removed_pool(pool, pool_key)
|
|
80
78
|
begin
|
|
81
79
|
deregister_shard_from_ar_handler(pool_key)
|
|
82
80
|
rescue StandardError => e
|
|
@@ -118,6 +116,16 @@ module Apartment
|
|
|
118
116
|
false
|
|
119
117
|
end
|
|
120
118
|
|
|
119
|
+
# Whether +conn+ sits in an aborted-transaction state that every subsequent
|
|
120
|
+
# statement will fail against until the transaction ends. PostgreSQL is the
|
|
121
|
+
# only supported engine with such a state (PQTRANS_INERROR); MySQL fails the
|
|
122
|
+
# statement and leaves the transaction usable, and its raw connection has no
|
|
123
|
+
# transaction_status at all. Base is conservative: never reclassify.
|
|
124
|
+
# See docs/designs/transaction-taint-detection.md (Evidence E).
|
|
125
|
+
def aborted_transaction?(_conn)
|
|
126
|
+
false
|
|
127
|
+
end
|
|
128
|
+
|
|
121
129
|
# Request-path fail-safe contract. The elevator wraps the
|
|
122
130
|
# tenant switch; on one of these error classes it asks
|
|
123
131
|
# #tenant_container_gone? whether the tenant's storage actually vanished (a
|
|
@@ -10,6 +10,8 @@ module Apartment
|
|
|
10
10
|
# to the environmentified tenant name. Lifecycle operations (create/drop)
|
|
11
11
|
# execute DDL against the default connection.
|
|
12
12
|
class PostgresqlDatabaseAdapter < AbstractAdapter
|
|
13
|
+
include PostgresqlTransactionState
|
|
14
|
+
|
|
13
15
|
def resolve_connection_config(tenant, base_config: nil)
|
|
14
16
|
config = base_config || send(:base_config)
|
|
15
17
|
config.merge('database' => environmentify(tenant))
|
|
@@ -12,6 +12,8 @@ module Apartment
|
|
|
12
12
|
# Apartment.config.postgres_config. Lifecycle operations (create/drop)
|
|
13
13
|
# execute DDL against the default connection.
|
|
14
14
|
class PostgresqlSchemaAdapter < AbstractAdapter
|
|
15
|
+
include PostgresqlTransactionState
|
|
16
|
+
|
|
15
17
|
def shared_pinned_connection?
|
|
16
18
|
!Apartment.config.force_separate_pinned_pool
|
|
17
19
|
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Apartment
|
|
4
|
+
module Adapters
|
|
5
|
+
# PostgreSQL's aborted-transaction state, shared by both PG adapters
|
|
6
|
+
# (schema-per-tenant and database-per-tenant). Their implementations are
|
|
7
|
+
# identical, so this is the DRY seam rather than two copies.
|
|
8
|
+
#
|
|
9
|
+
# A failed statement inside a transaction moves the connection to
|
|
10
|
+
# PQTRANS_INERROR, where every subsequent statement raises
|
|
11
|
+
# PG::InFailedSqlTransaction until the transaction ends. Apartment heals this
|
|
12
|
+
# at pool checkin; see docs/designs/transaction-taint-detection.md.
|
|
13
|
+
module PostgresqlTransactionState
|
|
14
|
+
def aborted_transaction?(conn)
|
|
15
|
+
# NOT conn.raw_connection. That is the escape hatch for code about to *use*
|
|
16
|
+
# the driver, so it materializes lazy transactions, marks the connection
|
|
17
|
+
# dirty, and CONNECTS one that was never connected — at every checkin, on
|
|
18
|
+
# every pooled connection, which defeats Rails' lazy connect and burns a
|
|
19
|
+
# backend per tenant pool. We only read a status flag, so we take the handle
|
|
20
|
+
# directly. `connected?` keeps us off the never-connected path.
|
|
21
|
+
return false unless conn.connected?
|
|
22
|
+
|
|
23
|
+
raw = conn.instance_variable_get(:@raw_connection)
|
|
24
|
+
return false unless raw.respond_to?(:transaction_status)
|
|
25
|
+
|
|
26
|
+
raw.transaction_status == ::PG::PQTRANS_INERROR
|
|
27
|
+
rescue StandardError
|
|
28
|
+
# A connection too broken to report its own status is not ours to classify;
|
|
29
|
+
# AR's own verify!/reconnect path owns that case.
|
|
30
|
+
false
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
data/lib/apartment/config.rb
CHANGED
|
@@ -17,7 +17,8 @@ module Apartment
|
|
|
17
17
|
attr_accessor :tenants_provider, :default_tenant,
|
|
18
18
|
:default_tenant_switch_allowed,
|
|
19
19
|
:tenant_pool_size, :pool_idle_timeout, :reaper_interval,
|
|
20
|
-
:max_total_connections, :
|
|
20
|
+
:max_total_connections, :max_tenant_pools, :max_tenant_connections,
|
|
21
|
+
:pool_overflow_policy,
|
|
21
22
|
:seed_after_create, :seed_data_file,
|
|
22
23
|
:schema_load_strategy, :schema_file,
|
|
23
24
|
:parallel_migration_threads,
|
|
@@ -26,7 +27,8 @@ module Apartment
|
|
|
26
27
|
:active_record_log, :sql_query_tags,
|
|
27
28
|
:shard_key_prefix,
|
|
28
29
|
:migration_role, :app_role, :schema_cache_per_tenant, :check_pending_migrations,
|
|
29
|
-
:force_separate_pinned_pool, :test_fixture_cleanup, :reap_in_test
|
|
30
|
+
:force_separate_pinned_pool, :test_fixture_cleanup, :reap_in_test,
|
|
31
|
+
:heal_tainted_connections
|
|
30
32
|
|
|
31
33
|
def initialize # rubocop:disable Metrics/AbcSize
|
|
32
34
|
@tenant_strategy = nil
|
|
@@ -38,6 +40,8 @@ module Apartment
|
|
|
38
40
|
@pool_idle_timeout = 300
|
|
39
41
|
@reaper_interval = nil
|
|
40
42
|
@max_total_connections = nil
|
|
43
|
+
@max_tenant_pools = nil
|
|
44
|
+
@max_tenant_connections = nil
|
|
41
45
|
@pool_overflow_policy = :evict_idle
|
|
42
46
|
@seed_after_create = false
|
|
43
47
|
@seed_data_file = nil
|
|
@@ -60,6 +64,13 @@ module Apartment
|
|
|
60
64
|
@check_pending_migrations = true
|
|
61
65
|
@force_separate_pinned_pool = false
|
|
62
66
|
@reap_in_test = false
|
|
67
|
+
# Reset a tenant connection left in an aborted transaction when it is checked
|
|
68
|
+
# back into its pool. On by default: without it the poisoned connection is
|
|
69
|
+
# served to the next caller, and under pool-per-tenant that connection is the
|
|
70
|
+
# tenant's ONLY connection, so the tenant is dead on that worker until the
|
|
71
|
+
# process restarts. ActiveRecord's active? cannot detect the state.
|
|
72
|
+
# PostgreSQL-only in effect. See docs/designs/transaction-taint-detection.md.
|
|
73
|
+
@heal_tainted_connections = true
|
|
63
74
|
@test_fixture_cleanup = true
|
|
64
75
|
end
|
|
65
76
|
|
|
@@ -124,6 +135,17 @@ module Apartment
|
|
|
124
135
|
# Reap on the idle-timeout cadence unless an explicit interval decouples
|
|
125
136
|
# the two (reap more often without shrinking the idle window).
|
|
126
137
|
@reaper_interval = @pool_idle_timeout if @reaper_interval.nil?
|
|
138
|
+
|
|
139
|
+
# max_total_connections is deprecated: the name said "connections" but it
|
|
140
|
+
# always capped tenant-pool COUNT. Alias it to its true meaning without
|
|
141
|
+
# changing behavior. Only fill when max_tenant_pools was not set explicitly,
|
|
142
|
+
# so validate!'s both-set guard can still catch a genuine double-spec.
|
|
143
|
+
return unless @max_total_connections
|
|
144
|
+
|
|
145
|
+
warn '[Apartment] DEPRECATION: config.max_total_connections is deprecated and will be ' \
|
|
146
|
+
'removed in v5. It caps tenant-pool COUNT, not connections; rename it to ' \
|
|
147
|
+
'max_tenant_pools. For a true connection ceiling, set max_tenant_connections.'
|
|
148
|
+
@max_tenant_pools = @max_total_connections if @max_tenant_pools.nil?
|
|
127
149
|
end
|
|
128
150
|
|
|
129
151
|
# Validate configuration completeness and consistency.
|
|
@@ -171,6 +193,35 @@ module Apartment
|
|
|
171
193
|
"max_total_connections must be a positive integer or nil, got: #{@max_total_connections.inspect}")
|
|
172
194
|
end
|
|
173
195
|
|
|
196
|
+
if @max_tenant_pools && (!@max_tenant_pools.is_a?(Integer) || @max_tenant_pools < 1)
|
|
197
|
+
raise(ConfigurationError,
|
|
198
|
+
"max_tenant_pools must be a positive integer or nil, got: #{@max_tenant_pools.inspect}")
|
|
199
|
+
end
|
|
200
|
+
|
|
201
|
+
if @max_total_connections && @max_tenant_pools && @max_total_connections != @max_tenant_pools
|
|
202
|
+
raise(ConfigurationError,
|
|
203
|
+
'max_total_connections and max_tenant_pools are the same setting under two names; ' \
|
|
204
|
+
"set only one. Got max_total_connections=#{@max_total_connections}, " \
|
|
205
|
+
"max_tenant_pools=#{@max_tenant_pools}")
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
if @max_tenant_connections && (!@max_tenant_connections.is_a?(Integer) || @max_tenant_connections < 1)
|
|
209
|
+
raise(ConfigurationError,
|
|
210
|
+
"max_tenant_connections must be a positive integer or nil, got: #{@max_tenant_connections.inspect}")
|
|
211
|
+
end
|
|
212
|
+
|
|
213
|
+
if @max_tenant_connections && @tenant_pool_size.nil?
|
|
214
|
+
raise(ConfigurationError,
|
|
215
|
+
'max_tenant_connections requires tenant_pool_size to be set (the pool budget is ' \
|
|
216
|
+
'derived as max_tenant_connections / tenant_pool_size)')
|
|
217
|
+
end
|
|
218
|
+
|
|
219
|
+
if @max_tenant_connections && @tenant_pool_size && @max_tenant_connections < @tenant_pool_size
|
|
220
|
+
raise(ConfigurationError,
|
|
221
|
+
"max_tenant_connections (#{@max_tenant_connections}) must be >= tenant_pool_size " \
|
|
222
|
+
"(#{@tenant_pool_size}); it cannot fit a single tenant pool")
|
|
223
|
+
end
|
|
224
|
+
|
|
174
225
|
unless %i[evict_idle raise].include?(@pool_overflow_policy)
|
|
175
226
|
raise(ConfigurationError,
|
|
176
227
|
'pool_overflow_policy must be :evict_idle or :raise, ' \
|
|
@@ -215,6 +266,12 @@ module Apartment
|
|
|
215
266
|
"reap_in_test must be true or false, got: #{@reap_in_test.inspect}")
|
|
216
267
|
end
|
|
217
268
|
|
|
269
|
+
unless [true, false].include?(@heal_tainted_connections)
|
|
270
|
+
raise(ConfigurationError,
|
|
271
|
+
'heal_tainted_connections must be true or false, ' \
|
|
272
|
+
"got: #{@heal_tainted_connections.inspect}")
|
|
273
|
+
end
|
|
274
|
+
|
|
218
275
|
unless @tenant_validator.nil? || @tenant_validator == false || @tenant_validator.respond_to?(:call)
|
|
219
276
|
raise(ConfigurationError,
|
|
220
277
|
'tenant_validator must be nil, false, or a callable, ' \
|
|
@@ -238,5 +295,15 @@ module Apartment
|
|
|
238
295
|
def rails_env_name
|
|
239
296
|
(Rails.env if defined?(Rails.env)) || ENV['RAILS_ENV'] || ENV['RACK_ENV'] || 'default_env'
|
|
240
297
|
end
|
|
298
|
+
|
|
299
|
+
# The pool-count bound the admission controller enforces: the stricter of the
|
|
300
|
+
# explicit pool cap (max_tenant_pools) and the pool budget derived from the
|
|
301
|
+
# connection ceiling (max_tenant_connections / tenant_pool_size, floored).
|
|
302
|
+
# Returns nil (uncapped) when neither knob is set. Relies on a global
|
|
303
|
+
# tenant_pool_size, so tenant-pool connections <= budget * tenant_pool_size.
|
|
304
|
+
def effective_pool_budget
|
|
305
|
+
derived = @max_tenant_connections && @tenant_pool_size ? @max_tenant_connections / @tenant_pool_size : nil
|
|
306
|
+
[@max_tenant_pools, derived].compact.min
|
|
307
|
+
end
|
|
241
308
|
end
|
|
242
309
|
end
|
data/lib/apartment/errors.rb
CHANGED
|
@@ -33,22 +33,31 @@ module Apartment
|
|
|
33
33
|
# Raised when the tenant connection pool is exhausted.
|
|
34
34
|
class PoolExhausted < ApartmentError; end
|
|
35
35
|
|
|
36
|
-
# Raised when admitting a new tenant pool would exceed
|
|
37
|
-
# and no idle pool can be evicted to make
|
|
38
|
-
# policy. Distinct from PoolExhausted (a single
|
|
39
|
-
# the process-wide pool-
|
|
36
|
+
# Raised when admitting a new tenant pool would exceed the effective pool
|
|
37
|
+
# budget (Config#effective_pool_budget) and no idle pool can be evicted to make
|
|
38
|
+
# room, under the :raise overflow policy. Distinct from PoolExhausted (a single
|
|
39
|
+
# pool's connections) — this is the process-wide pool-COUNT ceiling. The message
|
|
40
|
+
# names the knobs that set it, never the deprecated max_total_connections: that
|
|
41
|
+
# name is what made the budget look like a connection count in the first place.
|
|
42
|
+
# See docs/designs/pool-connection-budget.md and pool-admission-control.md.
|
|
40
43
|
class PoolCapacityReached < ApartmentError
|
|
44
|
+
# +max_total+ is the effective POOL budget, not a connection count. The
|
|
45
|
+
# reader keeps its name for compatibility with anything already rescuing this.
|
|
41
46
|
attr_reader :max_total, :current
|
|
42
47
|
|
|
48
|
+
alias pool_budget max_total
|
|
49
|
+
|
|
43
50
|
def initialize(max_total: nil, current: nil)
|
|
44
51
|
@max_total = max_total
|
|
45
52
|
@current = current
|
|
46
53
|
super(
|
|
47
|
-
"Tenant pool capacity reached: #{current.inspect} pools open, " \
|
|
48
|
-
"
|
|
49
|
-
'be evicted to admit another (all pinned or in use). Raise ' \
|
|
50
|
-
'
|
|
51
|
-
'
|
|
54
|
+
"Tenant pool capacity reached: #{current.inspect} tenant pools open, " \
|
|
55
|
+
"the effective pool budget is #{max_total.inspect}, and no idle pool " \
|
|
56
|
+
'could be evicted to admit another (all pinned or in use). Raise ' \
|
|
57
|
+
'max_tenant_pools, or raise max_tenant_connections (which derives the ' \
|
|
58
|
+
'pool budget as max_tenant_connections / tenant_pool_size), reduce ' \
|
|
59
|
+
'concurrent tenants, or set pool_overflow_policy to :evict_idle to ' \
|
|
60
|
+
'allow soft overflow.'
|
|
52
61
|
)
|
|
53
62
|
end
|
|
54
63
|
end
|
data/lib/apartment/migrator.rb
CHANGED
|
@@ -222,7 +222,11 @@ module Apartment
|
|
|
222
222
|
role = Apartment.config.migration_role
|
|
223
223
|
return unless role && Apartment.pool_manager
|
|
224
224
|
|
|
225
|
-
Apartment.pool_manager.evict_by_role(role).each do |pool_key,
|
|
225
|
+
Apartment.pool_manager.evict_by_role(role).each do |pool_key, pool|
|
|
226
|
+
# evict_by_role already removed it from the manager, so deregister_shard has
|
|
227
|
+
# nothing left to disconnect; close it here or a pool AR no longer registers
|
|
228
|
+
# leaks its backend. See docs/designs/out-of-band-tenant-ddl.md.
|
|
229
|
+
Apartment.disconnect_removed_pool(pool, pool_key)
|
|
226
230
|
Apartment.deregister_shard(pool_key)
|
|
227
231
|
end
|
|
228
232
|
rescue StandardError => e
|
|
@@ -86,15 +86,34 @@ module Apartment
|
|
|
86
86
|
# to the reaper and to max_total accounting (a connection leak that also
|
|
87
87
|
# undercounts the cap). Deregister it before re-raising so AR and the
|
|
88
88
|
# manager stay consistent. The next request re-establishes cleanly.
|
|
89
|
+
#
|
|
90
|
+
# deregister_ar_shard, NOT deregister_shard: we are running inside
|
|
91
|
+
# PoolManager's create block, and the full form removes from PoolManager's
|
|
92
|
+
# Concurrent::Map — whose MRI backend guards compute_if_absent and delete
|
|
93
|
+
# with the same non-reentrant mutex, so that would raise ThreadError
|
|
94
|
+
# ("deadlock; recursive locking") and skip this deregistration entirely,
|
|
95
|
+
# orphaning the very pool this rescue exists to reclaim. There is nothing
|
|
96
|
+
# to remove from the manager here regardless: the pool is not stored until
|
|
97
|
+
# this block returns.
|
|
98
|
+
#
|
|
99
|
+
# Reached with +send+ because the AR-only half is private: it is a
|
|
100
|
+
# half-operation, and leaving one publicly reachable is the exact footgun
|
|
101
|
+
# this seam exists to close. This is the one place it is correct.
|
|
89
102
|
begin
|
|
90
103
|
raise(Apartment::PendingMigrationError, tenant) if check_pending_migrations?(pool)
|
|
91
104
|
|
|
92
105
|
load_tenant_schema_cache(tenant, pool) if cfg.schema_cache_per_tenant
|
|
93
106
|
rescue StandardError
|
|
94
|
-
Apartment.
|
|
107
|
+
Apartment.send(:deregister_ar_shard, pool_key)
|
|
95
108
|
raise
|
|
96
109
|
end
|
|
97
110
|
|
|
111
|
+
# After the post-establish checks (a pool that fails them is discarded, so
|
|
112
|
+
# extending it would be wasted), and before the pool is handed out (so the
|
|
113
|
+
# very first checkin is already covered).
|
|
114
|
+
# See docs/designs/transaction-taint-detection.md.
|
|
115
|
+
Apartment::TransactionTaint.install(pool, tenant: tenant, pool_key: pool_key)
|
|
116
|
+
|
|
98
117
|
pool
|
|
99
118
|
end
|
|
100
119
|
rescue Apartment::ApartmentError
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Apartment
|
|
4
|
+
module Patches
|
|
5
|
+
# Keeps ActiveRecord's class-level Model.sequence_name memoization
|
|
6
|
+
# schema-agnostic under pool-per-tenant.
|
|
7
|
+
#
|
|
8
|
+
# Rails resolves default_sequence_name via pg_get_serial_sequence with an
|
|
9
|
+
# unqualified table name, so PostgreSQL answers through the current
|
|
10
|
+
# connection's search_path and returns a schema-QUALIFIED name — the schema
|
|
11
|
+
# of whichever tenant's pool happened to resolve it first. ActiveRecord then
|
|
12
|
+
# memoizes that value once per model class, process-wide. Consumers that
|
|
13
|
+
# render it into SQL (activerecord-import prefetches primary keys as a
|
|
14
|
+
# literal nextval(Model.sequence_name)) would draw ids from the
|
|
15
|
+
# first-resolver tenant's sequence in every tenant: wrong-tenant ids, silent
|
|
16
|
+
# sequence drift, and eventual PG::UniqueViolation.
|
|
17
|
+
#
|
|
18
|
+
# Stripping the connection's own current_schema prefix makes the memoized
|
|
19
|
+
# value schema-agnostic: nextval('widgets_id_seq') re-resolves through each
|
|
20
|
+
# pool's search_path, per tenant — the same invariant v4 relies on for table
|
|
21
|
+
# names (see docs/designs/v4-connection-model-rationale.md). A prefix naming
|
|
22
|
+
# any OTHER schema is preserved on purpose: persistent-schema tables and
|
|
23
|
+
# pinned models (whose table names are default_tenant-qualified and may
|
|
24
|
+
# execute on a shared tenant connection) are only correct BECAUSE they stay
|
|
25
|
+
# qualified.
|
|
26
|
+
#
|
|
27
|
+
# v3 shipped this guarantee as PostgreSqlAdapterPatch; it was lost in the
|
|
28
|
+
# Phase 2.5 v3 deletion (PR #356). Reset-on-switch is not an alternative:
|
|
29
|
+
# v4 serves tenants concurrently in one process, so only a schema-agnostic
|
|
30
|
+
# memoized value can be correct.
|
|
31
|
+
module PostgresqlSequenceName
|
|
32
|
+
def default_sequence_name(table_name, primary_key = 'id')
|
|
33
|
+
res = super
|
|
34
|
+
return res if res.nil?
|
|
35
|
+
# A schema-qualified table name is a PINNED model: Apartment qualifies
|
|
36
|
+
# those to the default tenant, and they resolve on whatever connection is
|
|
37
|
+
# current (the schema strategy shares the tenant's connection). Its
|
|
38
|
+
# sequence must stay qualified — stripping here is the mirror-image bug,
|
|
39
|
+
# because an unqualified name would later re-resolve against a *tenant's*
|
|
40
|
+
# search_path and draw ids from that tenant's stale copy of the pinned
|
|
41
|
+
# table's sequence. That order is the common one, not a corner case:
|
|
42
|
+
# pinned models are typically first touched at boot, on the default pool.
|
|
43
|
+
# v3 guarded the same case by force-adding the default_tenant prefix.
|
|
44
|
+
return res if table_name.to_s.include?('.')
|
|
45
|
+
|
|
46
|
+
# Routed table: drop whatever schema PostgreSQL qualified the sequence
|
|
47
|
+
# with — not just the connection's own. The two differ when the tenant
|
|
48
|
+
# schema is missing the table and search_path falls through to a schema
|
|
49
|
+
# that has it: the resolved sequence then names the FALLBACK schema, and
|
|
50
|
+
# preserving that would memoize it process-wide, so every tenant —
|
|
51
|
+
# including correctly-migrated ones — would draw from the fallback's
|
|
52
|
+
# sequence forever. Unqualified is the only value that lets the sequence
|
|
53
|
+
# re-resolve per pool exactly like the table name does.
|
|
54
|
+
#
|
|
55
|
+
# Parsed with AR's own splitter (rather than by hand) because it is what
|
|
56
|
+
# produced `res`, so it round-trips quoting and dotted identifiers.
|
|
57
|
+
ActiveRecord::ConnectionAdapters::PostgreSQL::Utils
|
|
58
|
+
.extract_schema_qualified_name(res).identifier
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
@@ -40,12 +40,22 @@ module Apartment
|
|
|
40
40
|
|
|
41
41
|
# Delete pool first, then timestamp. This ordering prevents a concurrent
|
|
42
42
|
# #get from orphaning a timestamp (get checks @pools, skips touch if absent).
|
|
43
|
+
#
|
|
44
|
+
# @api private
|
|
45
|
+
# Forgetting a pool here is only HALF of discarding it — the pool stays
|
|
46
|
+
# registered in AR's ConnectionHandler, leaking the registration and a live
|
|
47
|
+
# backend if the tenant is never re-accessed. Callers outside the gem want
|
|
48
|
+
# Apartment.deregister_shard (one pool) or Apartment.reset_tenant_pools! (all).
|
|
49
|
+
# These mutators also bypass PoolReaper's in-use guard: they will drop a pool
|
|
50
|
+
# with a checked-out connection or an open transaction.
|
|
51
|
+
# See docs/designs/out-of-band-tenant-ddl.md.
|
|
43
52
|
def remove(tenant_key)
|
|
44
53
|
pool = @pools.delete(tenant_key)
|
|
45
54
|
@timestamps.delete(tenant_key)
|
|
46
55
|
pool
|
|
47
56
|
end
|
|
48
57
|
|
|
58
|
+
# @api private — see #remove.
|
|
49
59
|
def remove_tenant(tenant)
|
|
50
60
|
prefix = "#{tenant}:"
|
|
51
61
|
removed = []
|
|
@@ -58,6 +68,7 @@ module Apartment
|
|
|
58
68
|
removed
|
|
59
69
|
end
|
|
60
70
|
|
|
71
|
+
# @api private — see #remove.
|
|
61
72
|
def evict_by_role(role)
|
|
62
73
|
suffix = ":#{role}"
|
|
63
74
|
removed = []
|
|
@@ -66,14 +66,15 @@ module Apartment
|
|
|
66
66
|
# when supplied. Safe to call from start_sampler! or an external scheduler.
|
|
67
67
|
def sample!
|
|
68
68
|
total = Apartment.pool_manager&.stats&.fetch(:total_pools, 0) || 0
|
|
69
|
-
|
|
69
|
+
emit_gauge(:tenant_pools_live, total)
|
|
70
|
+
emit_checkout_pressure!
|
|
70
71
|
|
|
71
72
|
return unless @backend_count
|
|
72
73
|
|
|
73
74
|
backends = @backend_count.call
|
|
74
75
|
return if backends.nil?
|
|
75
76
|
|
|
76
|
-
|
|
77
|
+
emit_gauge(:backend_connections, backends)
|
|
77
78
|
rescue StandardError => e
|
|
78
79
|
warn_failure('sample!', e)
|
|
79
80
|
end
|
|
@@ -104,6 +105,43 @@ module Apartment
|
|
|
104
105
|
@sampler = nil
|
|
105
106
|
end
|
|
106
107
|
|
|
108
|
+
# Per-tenant-pool checkout pressure, aggregated to low cardinality. waiting>0
|
|
109
|
+
# means threads are blocked acquiring a connection (the same-tenant fan-out
|
|
110
|
+
# starvation signal). The worst tenant is carried in the Sample payload, NOT
|
|
111
|
+
# as a metric dimension, to avoid unbounded time-series churn. Per-pool #stat
|
|
112
|
+
# is rescued so a pool tearing down mid-sample can't abort the whole pass.
|
|
113
|
+
def emit_checkout_pressure!
|
|
114
|
+
manager = Apartment.pool_manager
|
|
115
|
+
return unless manager
|
|
116
|
+
|
|
117
|
+
stats = collect_pool_stats(manager)
|
|
118
|
+
worst = stats.max_by { |s| s[:pending] } || { tenant: nil, pending: 0 }
|
|
119
|
+
payload = worst[:pending].positive? ? { tenant: worst[:tenant] } : {}
|
|
120
|
+
|
|
121
|
+
emit_gauge(:pools_waiting, stats.count { |s| s[:pending].positive? })
|
|
122
|
+
emit_gauge(:pools_saturated, stats.count { |s| s[:saturated] })
|
|
123
|
+
emit_gauge(:max_checkout_waiting, worst[:pending], payload: payload)
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
def emit_gauge(name, value, payload: {})
|
|
127
|
+
emit(Sample.new(name: name, kind: :gauge, value: value, dimensions: {}, payload: payload))
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
# Read each tenant pool's checkout stats into [{ tenant:, pending:, saturated: }],
|
|
131
|
+
# skipping any pool whose #stat raises (mid-teardown) so one bad pool can't
|
|
132
|
+
# abort the whole pass.
|
|
133
|
+
def collect_pool_stats(manager)
|
|
134
|
+
stats = []
|
|
135
|
+
manager.each_pair do |tenant_key, pool|
|
|
136
|
+
stat = pool.stat
|
|
137
|
+
stats << { tenant: tenant_key, pending: stat[:waiting].to_i,
|
|
138
|
+
saturated: stat[:busy].to_i >= stat[:size].to_i }
|
|
139
|
+
rescue StandardError
|
|
140
|
+
next
|
|
141
|
+
end
|
|
142
|
+
stats
|
|
143
|
+
end
|
|
144
|
+
|
|
107
145
|
def record_event(event, payload)
|
|
108
146
|
# Copy the notification payload so a sink that mutates Sample#payload
|
|
109
147
|
# can't corrupt it for other subscribers of the same event.
|
|
@@ -142,6 +142,11 @@ module Apartment
|
|
|
142
142
|
# on_evict with a nil pool.
|
|
143
143
|
return nil unless (pool = @pool_manager.remove(tenant))
|
|
144
144
|
|
|
145
|
+
# We removed the pool ourselves, so deregister_shard's own removal returns nil
|
|
146
|
+
# and it has nothing left to disconnect. AR disconnects only a pool it still
|
|
147
|
+
# finds registered, so a pool that has drifted out of AR's handler would leak
|
|
148
|
+
# its backend unless we close it here. See docs/designs/out-of-band-tenant-ddl.md.
|
|
149
|
+
Apartment.disconnect_removed_pool(pool, tenant)
|
|
145
150
|
deregister_from_ar_handler(tenant)
|
|
146
151
|
Instrumentation.instrument(:evict, tenant: tenant, reason: reason)
|
|
147
152
|
@on_evict&.call(tenant, pool)
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Apartment
|
|
4
|
+
# Heals a tenant connection left in an aborted transaction (PostgreSQL's
|
|
5
|
+
# PQTRANS_INERROR) at the moment it is checked back into its pool.
|
|
6
|
+
#
|
|
7
|
+
# WHY CHECKIN. ActiveRecord's +active?+ probes with an empty query, which does
|
|
8
|
+
# NOT error in an aborted transaction, so +verify!+ pronounces a poisoned
|
|
9
|
+
# connection healthy and +checkin+ does not reset it. The pool then serves it to
|
|
10
|
+
# the next caller. Under pool-per-tenant that connection is the ONLY connection
|
|
11
|
+
# for its tenant, so the tenant is dead on that worker until the process
|
|
12
|
+
# restarts, while every other tenant looks fine and nothing in the logs explains
|
|
13
|
+
# it. That amplification is what makes this ours to fix: the defect is generic
|
|
14
|
+
# Rails/PG, the consequence is specific to pool-per-tenant.
|
|
15
|
+
#
|
|
16
|
+
# Checkin is also the one seam that serves both populations correctly. A
|
|
17
|
+
# FIXTURE-PINNED connection must KEEP its transaction (teardown_fixtures owns the
|
|
18
|
+
# rollback) and never reaches ConnectionPool#checkin's body; a production
|
|
19
|
+
# connection must be reset before reuse. We still test +pinned+ explicitly,
|
|
20
|
+
# because this override runs BEFORE the early return in AR's own +checkin+.
|
|
21
|
+
#
|
|
22
|
+
# NEVER issue a raw ROLLBACK here. It destroys the ENCLOSING transaction while
|
|
23
|
+
# ActiveRecord still believes its stack is intact, raises nothing, and lets
|
|
24
|
+
# subsequent writes autocommit — turning a loud failure into silent, permanent
|
|
25
|
+
# database pollution. +reset!+ is ActiveRecord's own primitive: it rolls back,
|
|
26
|
+
# DISCARDs session state, and resets AR's transaction bookkeeping together. It
|
|
27
|
+
# also re-applies the pool's schema_search_path, so a healed tenant connection
|
|
28
|
+
# still points at its own schema (verified — the alternative would be a
|
|
29
|
+
# cross-tenant leak, which is strictly worse than the bug being fixed).
|
|
30
|
+
#
|
|
31
|
+
# Design: docs/designs/transaction-taint-detection.md
|
|
32
|
+
module TransactionTaint
|
|
33
|
+
# Extended onto Apartment-owned tenant pools only. The primary pool and every
|
|
34
|
+
# app-owned pool are left alone: the same Rails defect reaches them, but they
|
|
35
|
+
# are not ours, and the blast radius there is one connection of N rather than a
|
|
36
|
+
# dead tenant. That gap is reported upstream, not patched here.
|
|
37
|
+
module PoolHeal
|
|
38
|
+
attr_accessor :apartment_tenant, :apartment_pool_key, :apartment_taint_warned
|
|
39
|
+
|
|
40
|
+
def checkin(conn)
|
|
41
|
+
TransactionTaint.heal(conn, self)
|
|
42
|
+
super
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
class << self
|
|
47
|
+
# Extend +pool+ with the heal. Called once, at tenant-pool creation.
|
|
48
|
+
def install(pool, tenant:, pool_key:)
|
|
49
|
+
return pool unless Apartment.config&.heal_tainted_connections
|
|
50
|
+
|
|
51
|
+
pool.extend(PoolHeal)
|
|
52
|
+
pool.apartment_tenant = tenant.to_s
|
|
53
|
+
pool.apartment_pool_key = pool_key
|
|
54
|
+
pool
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Best-effort, and it must never raise: an exception here would abort
|
|
58
|
+
# ConnectionPool#checkin and leak the connection out of the pool permanently,
|
|
59
|
+
# which is worse than the taint we came to fix.
|
|
60
|
+
def heal(conn, pool)
|
|
61
|
+
return if conn.pinned
|
|
62
|
+
return unless Apartment.adapter&.aborted_transaction?(conn)
|
|
63
|
+
|
|
64
|
+
open_transactions = conn.open_transactions
|
|
65
|
+
conn.reset!
|
|
66
|
+
warn_once(pool)
|
|
67
|
+
instrument(pool, open_transactions: open_transactions, healed: true)
|
|
68
|
+
rescue StandardError => e
|
|
69
|
+
discard(conn, pool, e)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
private
|
|
73
|
+
|
|
74
|
+
# reset! failed, and +super+ is about to put this connection back in the pool's
|
|
75
|
+
# available list. Checking a STILL-POISONED connection back in would rebuild the
|
|
76
|
+
# exact outage we came to fix — the next lease gets it and the tenant stays dead.
|
|
77
|
+
# So drop the handle instead: disconnect! sends no SQL and cannot fail to nil out
|
|
78
|
+
# @raw_connection, so the pool keeps a dead shell that Rails transparently
|
|
79
|
+
# reconnects, fresh, on the next checkout. Checking in a disconnected connection
|
|
80
|
+
# is normal Rails (it is what flush/reap leave behind).
|
|
81
|
+
def discard(conn, pool, error)
|
|
82
|
+
conn.disconnect!
|
|
83
|
+
warn('[Apartment] could not reset the tainted connection for tenant ' \
|
|
84
|
+
"'#{pool.apartment_tenant}' (#{error.class}: #{error.message}); the connection " \
|
|
85
|
+
'was dropped and will be reopened on next use.')
|
|
86
|
+
instrument(pool, open_transactions: nil, healed: false, error: error.class.name)
|
|
87
|
+
rescue StandardError
|
|
88
|
+
nil # Nothing further is safe to attempt inside checkin.
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def instrument(pool, open_transactions:, healed:, error: nil)
|
|
92
|
+
Instrumentation.instrument(
|
|
93
|
+
'transaction_taint',
|
|
94
|
+
tenant: pool.apartment_tenant,
|
|
95
|
+
pool_key: pool.apartment_pool_key,
|
|
96
|
+
open_transactions: open_transactions,
|
|
97
|
+
healed: healed,
|
|
98
|
+
error: error
|
|
99
|
+
)
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
# Once per pool per process. Healing means each taint is a distinct incident
|
|
103
|
+
# rather than a cascade, but an app that poisons on every request would still
|
|
104
|
+
# warn on every request — flooding the log during the exact incident where the
|
|
105
|
+
# signal matters. The notification fires every time; it is the countable
|
|
106
|
+
# channel, and the one to alert on.
|
|
107
|
+
def warn_once(pool)
|
|
108
|
+
return if pool.apartment_taint_warned
|
|
109
|
+
|
|
110
|
+
pool.apartment_taint_warned = true
|
|
111
|
+
warn(
|
|
112
|
+
"[Apartment] the connection for tenant '#{pool.apartment_tenant}' was checked in " \
|
|
113
|
+
'while in an aborted transaction (PostgreSQL PQTRANS_INERROR) and has been reset. ' \
|
|
114
|
+
'Something ran a statement that failed inside a transaction Rails did not unwind. ' \
|
|
115
|
+
'Contain it by wrapping the failing call in ' \
|
|
116
|
+
'ActiveRecord::Base.transaction(requires_new: true); subscribe to ' \
|
|
117
|
+
'transaction_taint.apartment to find the call site. Do NOT add a raw ROLLBACK loop — ' \
|
|
118
|
+
'see docs/testing.md.'
|
|
119
|
+
)
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
end
|
data/lib/apartment/version.rb
CHANGED
data/lib/apartment.rb
CHANGED
|
@@ -213,23 +213,81 @@ module Apartment # rubocop:disable Metrics/ModuleLength
|
|
|
213
213
|
ActiveRecord::QueryLogs.tags = ActiveRecord::QueryLogs.tags + [:tenant]
|
|
214
214
|
end
|
|
215
215
|
|
|
216
|
-
#
|
|
216
|
+
# Discard one tenant pool, whole: deregister its shard from AR's ConnectionHandler
|
|
217
|
+
# and forget it in PoolManager, disconnecting it either way.
|
|
217
218
|
# Safe to call when AR is not loaded or config is not set (no-op).
|
|
218
219
|
# Used by PoolReaper eviction, AbstractAdapter#drop, and teardown.
|
|
220
|
+
#
|
|
221
|
+
# Both registries are updated because either one alone is wrong: deregistering
|
|
222
|
+
# from AR while the manager still holds the pool wedges the tenant permanently
|
|
223
|
+
# (the manager keeps handing back a pool AR has forgotten), and forgetting it in
|
|
224
|
+
# the manager alone leaks the AR registration and a live backend when the tenant
|
|
225
|
+
# is never re-accessed. Every internal caller already removes from the manager
|
|
226
|
+
# first, so the removal here is a no-op for them.
|
|
227
|
+
#
|
|
228
|
+
# The pool is disconnected here rather than left to AR, which disconnects only a
|
|
229
|
+
# pool it actually finds registered (ConnectionHandler#disconnect_pool_from_pool_manager
|
|
230
|
+
# guards `pool_config.disconnect!` behind `if pool_config`). A manager-held pool
|
|
231
|
+
# with no matching AR registration is reachable — the integration suite swaps the
|
|
232
|
+
# ConnectionHandler per example — and since we have just removed it from the
|
|
233
|
+
# manager, no later `PoolManager#clear` will disconnect it either. Mirrors
|
|
234
|
+
# AbstractAdapter#drop, which already removes, disconnects, then deregisters.
|
|
235
|
+
# (Callers that remove the pool from the manager THEMSELVES must disconnect it
|
|
236
|
+
# themselves too — the removal here returns nil for them, so there is nothing left
|
|
237
|
+
# for us to disconnect. PoolReaper#evict_tenant and Migrator#evict_migration_pools
|
|
238
|
+
# do exactly that.)
|
|
239
|
+
# See docs/designs/out-of-band-tenant-ddl.md.
|
|
240
|
+
#
|
|
241
|
+
# NOT SAFE inside a PoolManager create block — use +deregister_ar_shard+ there.
|
|
242
|
+
# PoolManager's @pools is a Concurrent::Map whose MRI backend guards
|
|
243
|
+
# +compute_if_absent+ and +delete+ with the SAME non-reentrant mutex, so removing
|
|
244
|
+
# a pool from inside the create block raises ThreadError ("deadlock; recursive
|
|
245
|
+
# locking"). The manager removal below is exactly that call.
|
|
246
|
+
#
|
|
247
|
+
# ORDER: AR first, manager removal in an +ensure+. The manager removal is an
|
|
248
|
+
# in-memory delete that cannot meaningfully fail, so it is the step that may run
|
|
249
|
+
# unconditionally; AR's removal does IO and is the one that can raise. Running the
|
|
250
|
+
# fallible step first, with the infallible one guaranteed after, means the call
|
|
251
|
+
# always ends with BOTH registries clear.
|
|
252
|
+
#
|
|
253
|
+
# Manager-first would be a race: between the manager delete and AR's removal, a
|
|
254
|
+
# concurrent tenant switch misses the manager, calls establish_connection — which
|
|
255
|
+
# RETURNS THE STILL-REGISTERED OLD POOL (ConnectionHandler#establish_connection
|
|
256
|
+
# reuses a pool whose db_config is equal) — and stores that doomed pool back in
|
|
257
|
+
# the manager. AR then unregisters and disconnects it, leaving the manager holding
|
|
258
|
+
# a dead pool: the permanent wedge this method exists to prevent, reintroduced.
|
|
259
|
+
# In this order the same interleaving costs at most one failed request, and the
|
|
260
|
+
# ensure clears the manager so the next switch rebuilds cleanly.
|
|
261
|
+
#
|
|
262
|
+
# Deliberately un-rescued at this level: both steps rescue their own failures, and
|
|
263
|
+
# swallowing everything here is what once hid a ThreadError, silently orphaning the
|
|
264
|
+
# pool the caller asked us to discard. Misuse should be loud.
|
|
219
265
|
def deregister_shard(pool_key)
|
|
220
266
|
return unless @config && defined?(ActiveRecord::Base)
|
|
221
267
|
|
|
222
|
-
|
|
223
|
-
|
|
268
|
+
begin
|
|
269
|
+
deregister_ar_shard(pool_key)
|
|
270
|
+
ensure
|
|
271
|
+
disconnect_removed_pool(@pool_manager&.remove(pool_key), pool_key)
|
|
272
|
+
end
|
|
273
|
+
end
|
|
224
274
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
275
|
+
# @api private
|
|
276
|
+
# Disconnect a pool that has been removed from PoolManager. Idempotent with AR's
|
|
277
|
+
# own disconnect on the happy path (ConnectionPool#disconnect! empties @connections,
|
|
278
|
+
# so a second call has nothing left to close); load-bearing only when AR has no
|
|
279
|
+
# matching registration to disconnect, in which case nothing else will close it.
|
|
280
|
+
#
|
|
281
|
+
# Public because the callers that remove a pool from the manager THEMSELVES — and
|
|
282
|
+
# therefore get nil back from deregister_shard's own removal — must disconnect what
|
|
283
|
+
# they removed: PoolReaper#evict_tenant, Migrator#evict_migration_pools,
|
|
284
|
+
# AbstractAdapter#drop. Rescued so one broken pool cannot abort the caller.
|
|
285
|
+
def disconnect_removed_pool(pool, pool_key)
|
|
286
|
+
return unless pool.respond_to?(:disconnect!)
|
|
287
|
+
|
|
288
|
+
pool.disconnect!
|
|
231
289
|
rescue StandardError => e
|
|
232
|
-
warn "[Apartment] Failed to
|
|
290
|
+
warn "[Apartment] Failed to disconnect pool for #{pool_key}: #{e.class}: #{e.message}"
|
|
233
291
|
end
|
|
234
292
|
|
|
235
293
|
# Deregister all tenant pools from AR's ConnectionHandler and clear the
|
|
@@ -255,6 +313,32 @@ module Apartment # rubocop:disable Metrics/ModuleLength
|
|
|
255
313
|
|
|
256
314
|
private
|
|
257
315
|
|
|
316
|
+
# The ActiveRecord half of a discard: deregister the shard (which disconnects the
|
|
317
|
+
# pool AR holds), leaving PoolManager untouched. The ONLY form that is safe inside
|
|
318
|
+
# a PoolManager create block — see the re-entrancy note on +deregister_shard+ —
|
|
319
|
+
# and correct there anyway: compute_if_absent does not store the pool until the
|
|
320
|
+
# block returns, so there is no manager entry to remove.
|
|
321
|
+
#
|
|
322
|
+
# PRIVATE ON PURPOSE. This is a half-operation, and a reachable half-operation is
|
|
323
|
+
# the bug this whole seam exists to eliminate: called on its own, it leaves AR
|
|
324
|
+
# without the pool while PoolManager keeps handing it out, wedging the tenant for
|
|
325
|
+
# the life of the process. The one legitimate caller reaches it with +send+.
|
|
326
|
+
def deregister_ar_shard(pool_key)
|
|
327
|
+
return unless @config && defined?(ActiveRecord::Base)
|
|
328
|
+
|
|
329
|
+
_, separator, role_str = pool_key.to_s.rpartition(':')
|
|
330
|
+
role = separator.empty? || role_str.empty? ? ActiveRecord.writing_role : role_str.to_sym
|
|
331
|
+
|
|
332
|
+
shard_key = :"#{@config.shard_key_prefix}_#{pool_key}"
|
|
333
|
+
ActiveRecord::Base.connection_handler.remove_connection_pool(
|
|
334
|
+
'ActiveRecord::Base',
|
|
335
|
+
role: role,
|
|
336
|
+
shard: shard_key
|
|
337
|
+
)
|
|
338
|
+
rescue StandardError => e
|
|
339
|
+
warn "[Apartment] Failed to deregister AR pool for #{pool_key}: #{e.class}: #{e.message}"
|
|
340
|
+
end
|
|
341
|
+
|
|
258
342
|
# Double-checked locking: the common path (already built) skips the mutex;
|
|
259
343
|
# concurrent first callers serialize so exactly one validator is built.
|
|
260
344
|
# TenantValidator.new subscribes to ActiveSupport::Notifications, so a
|
|
@@ -265,21 +349,23 @@ module Apartment # rubocop:disable Metrics/ModuleLength
|
|
|
265
349
|
end
|
|
266
350
|
|
|
267
351
|
# Build the pool manager + reaper for a freshly-validated config and start
|
|
268
|
-
# the background reaper. When
|
|
269
|
-
#
|
|
270
|
-
#
|
|
352
|
+
# the background reaper. When a pool budget is configured (max_tenant_pools
|
|
353
|
+
# and/or max_tenant_connections, via Config#effective_pool_budget), wire the
|
|
354
|
+
# reaper as the pool manager's admission controller so cold creates are
|
|
355
|
+
# bounded synchronously; otherwise the manager keeps its lock-free create path.
|
|
271
356
|
def setup_pools!(new_config)
|
|
357
|
+
budget = new_config.effective_pool_budget
|
|
272
358
|
@pool_manager = PoolManager.new
|
|
273
359
|
@pool_reaper = PoolReaper.new(
|
|
274
360
|
pool_manager: @pool_manager,
|
|
275
361
|
interval: new_config.reaper_interval,
|
|
276
362
|
idle_timeout: new_config.pool_idle_timeout,
|
|
277
|
-
max_total:
|
|
363
|
+
max_total: budget,
|
|
278
364
|
default_tenant: new_config.default_tenant,
|
|
279
365
|
shard_key_prefix: new_config.shard_key_prefix,
|
|
280
366
|
overflow_policy: new_config.pool_overflow_policy
|
|
281
367
|
)
|
|
282
|
-
@pool_manager.admission_controller = @pool_reaper if
|
|
368
|
+
@pool_manager.admission_controller = @pool_reaper if budget
|
|
283
369
|
@pool_reaper.start
|
|
284
370
|
end
|
|
285
371
|
|
|
@@ -370,6 +456,16 @@ module Apartment # rubocop:disable Metrics/ModuleLength
|
|
|
370
456
|
end
|
|
371
457
|
end
|
|
372
458
|
|
|
459
|
+
# Prepend the sequence-name patch whenever the PostgreSQL adapter loads
|
|
460
|
+
# (immediately, if it already has). Registered at gem load rather than in
|
|
461
|
+
# activate! because ActiveRecord memoizes Model.sequence_name at first touch,
|
|
462
|
+
# which can happen during boot before Apartment.activate! runs. No-op for apps
|
|
463
|
+
# that never load the PostgreSQL adapter, so MySQL/SQLite consumers never pull
|
|
464
|
+
# in pg. See the patch file for the full rationale.
|
|
465
|
+
ActiveSupport.on_load(:active_record_postgresqladapter) do
|
|
466
|
+
prepend(Apartment::Patches::PostgresqlSequenceName)
|
|
467
|
+
end
|
|
468
|
+
|
|
373
469
|
# Load Railtie when Rails is present (standard gem convention).
|
|
374
470
|
# Railtie is Zeitwerk-ignored — this explicit require is the only load path.
|
|
375
471
|
require_relative 'apartment/railtie' if defined?(Rails::Railtie)
|
|
@@ -37,9 +37,20 @@ Apartment.configure do |config|
|
|
|
37
37
|
|
|
38
38
|
# == Connection Pool =====================================================
|
|
39
39
|
|
|
40
|
-
#
|
|
41
|
-
#
|
|
42
|
-
#
|
|
40
|
+
# tenant_pool_size MUST be >= the peak number of threads/fibers in one process
|
|
41
|
+
# that can touch the SAME tenant at once (e.g. a Sidekiq role's concurrency for a
|
|
42
|
+
# same-tenant job fan-out); otherwise those threads block on connection checkout.
|
|
43
|
+
# It is a lazy ceiling — connections are created on demand and idle pools reaped.
|
|
44
|
+
# config.tenant_pool_size = 5
|
|
45
|
+
# config.pool_idle_timeout = 300
|
|
46
|
+
#
|
|
47
|
+
# Bound how many tenant pools exist per process:
|
|
48
|
+
# config.max_tenant_pools = nil
|
|
49
|
+
# ...or bound total tenant-pool CONNECTIONS (requires tenant_pool_size); the
|
|
50
|
+
# admission controller derives the pool budget as floor(value / tenant_pool_size):
|
|
51
|
+
# config.max_tenant_connections = nil
|
|
52
|
+
#
|
|
53
|
+
# config.max_total_connections = nil # DEPRECATED: alias of max_tenant_pools; removed in v5
|
|
43
54
|
|
|
44
55
|
# == Elevator (Request Tenant Detection) =================================
|
|
45
56
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ros-apartment
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 4.0.0.
|
|
4
|
+
version: 4.0.0.alpha9
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ryan Brunner
|
|
@@ -168,6 +168,7 @@ files:
|
|
|
168
168
|
- lib/apartment/adapters/mysql2_adapter.rb
|
|
169
169
|
- lib/apartment/adapters/postgresql_database_adapter.rb
|
|
170
170
|
- lib/apartment/adapters/postgresql_schema_adapter.rb
|
|
171
|
+
- lib/apartment/adapters/postgresql_transaction_state.rb
|
|
171
172
|
- lib/apartment/adapters/sqlite3_adapter.rb
|
|
172
173
|
- lib/apartment/adapters/trilogy_adapter.rb
|
|
173
174
|
- lib/apartment/cli.rb
|
|
@@ -194,6 +195,7 @@ files:
|
|
|
194
195
|
- lib/apartment/migrator.rb
|
|
195
196
|
- lib/apartment/patches/connection_handling.rb
|
|
196
197
|
- lib/apartment/patches/live_tenant_propagation.rb
|
|
198
|
+
- lib/apartment/patches/postgresql_sequence_name.rb
|
|
197
199
|
- lib/apartment/pool_manager.rb
|
|
198
200
|
- lib/apartment/pool_observer.rb
|
|
199
201
|
- lib/apartment/pool_reaper.rb
|
|
@@ -206,6 +208,7 @@ files:
|
|
|
206
208
|
- lib/apartment/tenant_name_validator.rb
|
|
207
209
|
- lib/apartment/tenant_validator.rb
|
|
208
210
|
- lib/apartment/test_fixtures.rb
|
|
211
|
+
- lib/apartment/transaction_taint.rb
|
|
209
212
|
- lib/apartment/version.rb
|
|
210
213
|
- lib/generators/apartment/install/USAGE
|
|
211
214
|
- lib/generators/apartment/install/install_generator.rb
|