partition_gardener 0.3.2 → 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/README.md +18 -5
- data/SECURITY.md +8 -8
- data/docs/THIRD_PARTY_LICENSE_MANIFEST.tsv +12 -12
- data/docs/application_contract.md +6 -4
- data/docs/decision_flow.md +1 -1
- data/docs/partition_engines.md +6 -6
- data/docs/partition_landscape.md +95 -6
- data/lib/partition_gardener/date_range_maintenance.rb +15 -1
- data/lib/partition_gardener/strategy/date_range.rb +4 -4
- data/lib/partition_gardener/version.rb +1 -1
- metadata +4 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2ad4860950aedb375b6703ededf1602006203f1defa7406465f6f87b5d38083b
|
|
4
|
+
data.tar.gz: 0af3c3dadfe7c123c5193e9aa1b78b3d8fb5dc1a294a3819c790ad20b18a4d53
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 68c44b5b9851c2f167801c26f33b9050f5ba684a0355820ea8738f7309ceebaf35ed3ab2ef3ee8a87830a145c9593b8a994f06ae7b3625901fb7b92963cec2d7
|
|
7
|
+
data.tar.gz: 8026d9241cf10f037655ded5999cae214dafe6658b6d3c66a5d336561e87d20b078a63a82321bed1d0b74d0b9b513579196dbe3d3d31235a61c10aedb6d864ab
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.3.3 (2026-08-13)
|
|
6
|
+
|
|
7
|
+
- Roll the sliding-window current partition before attaching the prior month's archive so a month rollover detaches the old current span and attaches that month as its own child
|
|
8
|
+
- Skip archive attach when a monthly bucket starts at or after the attached current partition lower bound
|
|
9
|
+
- Expand routing documentation: routing layers, recovery ladder, parent timestamp example, window sizing, global index tables, time-embedded ids, two-tier composite hints, and post-maintenance cache invalidation ([partition_landscape.md](docs/partition_landscape.md))
|
|
10
|
+
- Documentation audit: remove duplicate portable routing section from partition engines guide; consolidate sharded-app guidance; tighten cross-links between decision flow, application contract, and landscape docs
|
|
11
|
+
|
|
5
12
|
## 0.3.2 (2026-07-13)
|
|
6
13
|
|
|
7
14
|
- Route `HotSwitchConcern` DDL through `connection.execute` instead of `ActiveRecord::Migration#execute` so migration safety tools such as Strong Migrations do not intercept hot-switch SQL.
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Partition Gardener
|
|
2
2
|
|
|
3
|
-
[](https://badge.fury.io/rb/partition_gardener) [](https://github.com/amkisko/partition_gardener.rb/actions/workflows/test.yml) [](https://codecov.io/
|
|
3
|
+
[](https://badge.fury.io/rb/partition_gardener) [](https://github.com/amkisko/partition_gardener.rb/actions/workflows/test.yml) [](https://app.codecov.io/github/amkisko/partition_gardener.rb)
|
|
4
4
|
|
|
5
5
|
PostgreSQL partition lifecycle: archive, current, and future zones, heat-driven splits inside the active window, cursor-based rebalance, mandatory default drain, and hot-switch migration helpers. Rails-integrated by default; works standalone with `pg` and a JSON registry.
|
|
6
6
|
|
|
@@ -18,8 +18,8 @@ Complements migration gems (e.g. [pg_party](https://github.com/rkrage/pg_party))
|
|
|
18
18
|
### Choosing and configuring
|
|
19
19
|
|
|
20
20
|
- [docs/decision_flow.md](docs/decision_flow.md) — when to partition, layout, and method choices
|
|
21
|
-
- [docs/partition_landscape.md](docs/partition_landscape.md) — templates, Rails sharding, pruning, UI, aggregate snapshots, materialized views
|
|
22
|
-
- [docs/partition_engines.md](docs/partition_engines.md) — portable patterns and other database engines
|
|
21
|
+
- [docs/partition_landscape.md](docs/partition_landscape.md) — templates, Rails sharding, pruning, routing layers and hints, UI, aggregate snapshots, materialized views
|
|
22
|
+
- [docs/partition_engines.md](docs/partition_engines.md) — portable maintenance patterns and other database engines
|
|
23
23
|
- [docs/configuration.md](docs/configuration.md) — global config, registry, per-table options, JSON import
|
|
24
24
|
- [docs/tooling_split.md](docs/tooling_split.md) — pg_party vs pg_partman vs Gardener
|
|
25
25
|
- [docs/related_postgres_tooling.md](docs/related_postgres_tooling.md) — PgHero, Dexter, pgsync, pgslice vs Gardener
|
|
@@ -37,7 +37,7 @@ Complements migration gems (e.g. [pg_party](https://github.com/rkrage/pg_party))
|
|
|
37
37
|
### Application and migration
|
|
38
38
|
|
|
39
39
|
- [docs/cutover.md](docs/cutover.md) — hot-switch playbook and template upgrades
|
|
40
|
-
- [docs/application_contract.md](docs/application_contract.md) — writes, moves, bulk load, replicas
|
|
40
|
+
- [docs/application_contract.md](docs/application_contract.md) — writes, moves, routing when the key is missing, bulk load, replicas
|
|
41
41
|
- [docs/naming.md](docs/naming.md) — child partition naming catalog
|
|
42
42
|
- [docs/host_testing.md](docs/host_testing.md) — CI and staging for host apps
|
|
43
43
|
|
|
@@ -138,7 +138,7 @@ See [docs/decision_flow.md](docs/decision_flow.md) for when to pick other templa
|
|
|
138
138
|
|
|
139
139
|
`Templates.premake_monthly` (layout `:premake_monthly`) — cron-style premake bridge; migrate to sliding window.
|
|
140
140
|
|
|
141
|
-
See [docs/partition_landscape.md](docs/partition_landscape.md) for the template catalog, Rails sharding, composite keys, partition pruning, UI and product surfaces, aggregate snapshots, and materialized view limits. Operations: [operations.md](docs/operations.md), [cutover.md](docs/cutover.md), [monitoring.md](docs/monitoring.md).
|
|
141
|
+
See [docs/partition_landscape.md](docs/partition_landscape.md) for the template catalog, Rails sharding, composite keys, partition pruning, routing hints when the key is missing, UI and product surfaces, aggregate snapshots, and materialized view limits. Operations: [operations.md](docs/operations.md), [cutover.md](docs/cutover.md), [monitoring.md](docs/monitoring.md).
|
|
142
142
|
|
|
143
143
|
## Hot-switch migrations
|
|
144
144
|
|
|
@@ -191,6 +191,19 @@ bundle exec rspec --exclude-pattern "spec/integration/**/*_spec.rb"
|
|
|
191
191
|
INTEGRATION=1 DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/partition_gardener_test bundle exec rake spec:integration
|
|
192
192
|
```
|
|
193
193
|
|
|
194
|
+
## Links
|
|
195
|
+
|
|
196
|
+
- [GitHub](https://github.com/amkisko/partition_gardener.rb)
|
|
197
|
+
- [GitLab](https://gitlab.com/amkisko/partition_gardener.rb)
|
|
198
|
+
- [RubyGems](https://rubygems.org/gems/partition_gardener)
|
|
199
|
+
- [Versions Atom](https://rubygems.org/gems/partition_gardener/versions.atom) (feed id `11544106111602`)
|
|
200
|
+
- [libraries.io](https://libraries.io/rubygems/partition_gardener)
|
|
201
|
+
- [Deps.dev](https://deps.dev/rubygems/partition_gardener)
|
|
202
|
+
- [SonarCloud](https://sonarcloud.io/project/overview?id=amkisko_partition_gardener.rb)
|
|
203
|
+
- [Snyk](https://snyk.io/test/github/amkisko/partition_gardener.rb)
|
|
204
|
+
- [Codecov](https://app.codecov.io/github/amkisko/partition_gardener.rb)
|
|
205
|
+
- [OpenSSF Scorecard](https://scorecard.dev/viewer/?uri=github.com/amkisko/partition_gardener.rb)
|
|
206
|
+
|
|
194
207
|
## License
|
|
195
208
|
|
|
196
209
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/SECURITY.md
CHANGED
|
@@ -1,27 +1,27 @@
|
|
|
1
|
-
#
|
|
1
|
+
# SECURITY
|
|
2
2
|
|
|
3
|
-
## Reporting a
|
|
3
|
+
## Reporting a Vulnerability
|
|
4
4
|
|
|
5
|
-
Do not open a public
|
|
5
|
+
Do not open a public issue for security vulnerabilities.
|
|
6
6
|
|
|
7
7
|
Email security details to: security@kiskolabs.com
|
|
8
8
|
|
|
9
9
|
Include: description, steps to reproduce, potential impact, and suggested fix (if available).
|
|
10
10
|
|
|
11
|
-
### Response
|
|
11
|
+
### Response Timeline
|
|
12
12
|
|
|
13
13
|
- We will acknowledge receipt of your report
|
|
14
14
|
- We will provide an initial assessment
|
|
15
15
|
- We will keep you informed of our progress and resolution timeline
|
|
16
16
|
|
|
17
|
-
### Disclosure
|
|
17
|
+
### Disclosure Policy
|
|
18
18
|
|
|
19
19
|
- We will work with you to understand and resolve the issue
|
|
20
20
|
- We will credit you for the discovery (unless you prefer to remain anonymous)
|
|
21
21
|
- We will publish a security advisory after the vulnerability is patched
|
|
22
22
|
- We will coordinate public disclosure with you
|
|
23
23
|
|
|
24
|
-
## Automation
|
|
24
|
+
## Automation Security
|
|
25
25
|
|
|
26
|
-
- Context
|
|
27
|
-
- Supply
|
|
26
|
+
- Context Isolation: It is strictly forbidden to include production credentials, API keys, or Personally Identifiable Information (PII) in prompts sent to third-party LLMs or automation services.
|
|
27
|
+
- Supply Chain: All automated dependencies must be verified.
|
|
@@ -1,34 +1,34 @@
|
|
|
1
1
|
bundle gem version licenses
|
|
2
|
-
root activemodel 8.1.3 MIT
|
|
3
|
-
root activerecord 8.1.3 MIT
|
|
4
|
-
root activesupport 8.1.3 MIT
|
|
2
|
+
root activemodel 8.1.3.1 MIT
|
|
3
|
+
root activerecord 8.1.3.1 MIT
|
|
4
|
+
root activesupport 8.1.3.1 MIT
|
|
5
5
|
root appraisal 2.5.0 MIT
|
|
6
6
|
root ast 2.4.3 MIT
|
|
7
7
|
root base64 0.3.0 Ruby|BSD-2-Clause
|
|
8
8
|
root bigdecimal 4.1.2 Ruby|BSD-2-Clause
|
|
9
|
-
root bundler 4.0.
|
|
10
|
-
root concurrent-ruby 1.3.
|
|
9
|
+
root bundler 4.0.16 MIT
|
|
10
|
+
root concurrent-ruby 1.3.8 MIT
|
|
11
11
|
root connection_pool 3.0.2 MIT
|
|
12
12
|
root diff-lcs 1.6.2 MIT|Artistic-1.0-Perl|GPL-2.0-or-later
|
|
13
13
|
root drb 2.2.3 Ruby|BSD-2-Clause
|
|
14
14
|
root i18n 1.15.2 MIT
|
|
15
|
-
root json 2.
|
|
15
|
+
root json 2.21.2 Ruby
|
|
16
16
|
root language_server-protocol 3.17.0.6 MIT
|
|
17
17
|
root lint_roller 1.1.0 MIT
|
|
18
18
|
root logger 1.7.0 Ruby|BSD-2-Clause
|
|
19
19
|
root minitest 6.0.6 MIT
|
|
20
20
|
root parallel 2.1.0 MIT
|
|
21
|
-
root parser 3.3.
|
|
22
|
-
root partition_gardener 0.3.
|
|
21
|
+
root parser 3.3.12.0 MIT
|
|
22
|
+
root partition_gardener 0.3.3 MIT
|
|
23
23
|
root pg 1.6.3 BSD-2-Clause
|
|
24
|
-
root polyrun 2.2.
|
|
24
|
+
root polyrun 2.2.2 MIT
|
|
25
25
|
root prism 1.9.0 MIT
|
|
26
26
|
root prosopite 2.2.0 Apache-2.0
|
|
27
27
|
root racc 1.8.1 Ruby|BSD-2-Clause
|
|
28
28
|
root rack 3.2.6 MIT
|
|
29
29
|
root rainbow 3.1.1 MIT
|
|
30
30
|
root rake 13.4.2 MIT
|
|
31
|
-
root rbs
|
|
31
|
+
root rbs 4.1.2 BSD-2-Clause|Ruby
|
|
32
32
|
root regexp_parser 2.12.0 MIT
|
|
33
33
|
root rspec 3.13.2 MIT
|
|
34
34
|
root rspec-core 3.13.6 MIT
|
|
@@ -36,7 +36,7 @@ root rspec-expectations 3.13.5 MIT
|
|
|
36
36
|
root rspec-mocks 3.13.8 MIT
|
|
37
37
|
root rspec-support 3.13.7 MIT
|
|
38
38
|
root rspec_junit_formatter 0.6.0 MIT
|
|
39
|
-
root rubocop 1.
|
|
39
|
+
root rubocop 1.88.2 MIT
|
|
40
40
|
root rubocop-ast 1.50.0 MIT
|
|
41
41
|
root rubocop-capybara 2.23.0 MIT
|
|
42
42
|
root rubocop-factory_bot 2.28.0 MIT
|
|
@@ -47,7 +47,7 @@ root rubocop-rspec_rails 2.32.0 MIT
|
|
|
47
47
|
root rubocop-thread_safety 0.7.3 MIT
|
|
48
48
|
root ruby-progressbar 1.13.0 MIT
|
|
49
49
|
root securerandom 0.4.1 Ruby|BSD-2-Clause
|
|
50
|
-
root standard 1.
|
|
50
|
+
root standard 1.56.0
|
|
51
51
|
root standard-custom 1.0.2 MIT
|
|
52
52
|
root standard-performance 1.9.0 MIT
|
|
53
53
|
root standard-rails 1.6.0 MIT
|
|
@@ -7,6 +7,8 @@ How host applications should behave around partitioned tables: queries, writes,
|
|
|
7
7
|
- Hot-path reads and writes include the partition key in plain predicates (no wrappers on the key column).
|
|
8
8
|
- `conflict_key` columns match the parent unique index; updates and deletes use `query_constraints` when the logical id is not globally unique ([partition_landscape.md](partition_landscape.md#rails-application-contract)).
|
|
9
9
|
- Inserts supply a routable partition key value so rows land in named children, not only in `default`.
|
|
10
|
+
- When only a logical id or parent reference is available, follow the recovery ladder in [partition_landscape.md](partition_landscape.md#routing-hints-when-the-key-is-not-in-hand).
|
|
11
|
+
- After maintenance with high `rows_moved`, refresh `id → partition_key` mappings when the partition key can change.
|
|
10
12
|
|
|
11
13
|
## During maintenance (row moves)
|
|
12
14
|
|
|
@@ -54,7 +56,7 @@ Gardener does not add or remove FKs during maintenance.
|
|
|
54
56
|
|
|
55
57
|
## Sharded applications
|
|
56
58
|
|
|
57
|
-
|
|
59
|
+
Shard plus partition filters on hot paths; see [Rails horizontal sharding](partition_landscape.md#rails-horizontal-sharding). Per-shard maintenance: [operations.md](operations.md#sharded-registries).
|
|
58
60
|
|
|
59
61
|
```ruby
|
|
60
62
|
ApplicationRecord.connected_to(shard: :tenant_a) do
|
|
@@ -62,12 +64,12 @@ ApplicationRecord.connected_to(shard: :tenant_a) do
|
|
|
62
64
|
end
|
|
63
65
|
```
|
|
64
66
|
|
|
65
|
-
|
|
67
|
+
Registry JSON may differ per shard only if layouts differ (unusual).
|
|
66
68
|
|
|
67
69
|
## Admin and operator surfaces
|
|
68
70
|
|
|
69
71
|
- Default filters: current month or selected tenant, not all history.
|
|
70
|
-
- Global id search is an advanced, slow path; require date or tenant hint.
|
|
72
|
+
- Global id search is an advanced, slow path; require date or tenant hint, or resolve through a mapping table before scanning children.
|
|
71
73
|
- Export flows chunk by bucket; show progress per period.
|
|
72
74
|
- Totals read from snapshot tables with `computed_at`, not live `SUM` across all children or a stale materialized view over the full fact table ([partition_landscape.md](partition_landscape.md#materialized-views)).
|
|
73
75
|
|
|
@@ -77,6 +79,6 @@ See [host_testing.md](host_testing.md) for CI registry fixtures and integration
|
|
|
77
79
|
|
|
78
80
|
## Related
|
|
79
81
|
|
|
80
|
-
- [partition_landscape.md](partition_landscape.md) — pruning, UI, snapshots
|
|
82
|
+
- [partition_landscape.md](partition_landscape.md) — pruning, routing layers and hints, UI, snapshots
|
|
81
83
|
- [cutover.md](cutover.md) — backfill and switch
|
|
82
84
|
- [naming.md](naming.md) — child table names
|
data/docs/decision_flow.md
CHANGED
|
@@ -87,7 +87,7 @@ Operational split:
|
|
|
87
87
|
|
|
88
88
|
Both cadences use the same Gardener primitives (`plan`, `audit`, `apply`, `run!`). The difference is when `apply` runs and how heavily the application depends on snapshots on hot paths.
|
|
89
89
|
|
|
90
|
-
Partition Gardener does not replace application query contracts.
|
|
90
|
+
Partition Gardener does not replace application query contracts. See [application_contract.md](application_contract.md) and [partition_landscape.md](partition_landscape.md#routing-layers). The gem does not enforce routing; migrations, reviews, and `EXPLAIN` do.
|
|
91
91
|
|
|
92
92
|
## Reliability invariants
|
|
93
93
|
|
data/docs/partition_engines.md
CHANGED
|
@@ -4,7 +4,7 @@ Partition Gardener implements runtime maintenance for PostgreSQL native declarat
|
|
|
4
4
|
|
|
5
5
|
The problems Gardener solves are not PostgreSQL-only. Large tables split by time or category need the same operational invariants on every engine that supports partition-shaped lifecycle: bounded catalog size, headroom for future inserts, retention without bulk delete, and a single maintainer that reconciles layout against policy. This page maps those portable patterns to other database engines, states where Gardener's current implementation applies as-is, and where teams should borrow the design without expecting this gem to run unchanged.
|
|
6
6
|
|
|
7
|
-
Use this page with [partition_landscape.md](partition_landscape.md) (templates
|
|
7
|
+
Use this page with [partition_landscape.md](partition_landscape.md) (templates, pruning, routing, PostgreSQL scope), [decision_flow.md](decision_flow.md) (when to partition), [application_contract.md](application_contract.md) (host app queries and writes), and [tooling_split.md](tooling_split.md) (creation vs runtime maintenance).
|
|
8
8
|
|
|
9
9
|
## Portable patterns Gardener encodes
|
|
10
10
|
|
|
@@ -78,7 +78,7 @@ Patterns that transfer from Gardener without semantic change:
|
|
|
78
78
|
- Retention by `DROP TABLE` on detached or attached partition children
|
|
79
79
|
- Composite primary key including partition key for uniqueness and pruning
|
|
80
80
|
- List-then-range composite trees for geo or branch plus time
|
|
81
|
-
- Application contract: filters on partition key, UI scoped to period or tenant ([partition_landscape.md](partition_landscape.md#rails-application-contract))
|
|
81
|
+
- Application contract: filters on partition key, UI scoped to period or tenant ([partition_landscape.md](partition_landscape.md#rails-application-contract), [application_contract.md](application_contract.md))
|
|
82
82
|
|
|
83
83
|
Gardener-specific gaps on YugabyteDB:
|
|
84
84
|
|
|
@@ -400,15 +400,15 @@ CockroachDB, ClickHouse, and warehouses need different products (TTL-first or wa
|
|
|
400
400
|
|
|
401
401
|
## Operator checklist when the engine is not PostgreSQL
|
|
402
402
|
|
|
403
|
-
Keep from Gardener regardless of engine:
|
|
403
|
+
Keep from Gardener regardless of engine (application-side detail: [application_contract.md](application_contract.md), routing: [partition_landscape.md](partition_landscape.md#routing-layers)):
|
|
404
404
|
|
|
405
405
|
- Partition key in all unique constraints and hot-path queries
|
|
406
|
-
- One maintainer per table; no overlapping premake crons
|
|
406
|
+
- One maintainer per table; no overlapping premake crons ([tooling_split.md](tooling_split.md))
|
|
407
407
|
- Retention by removing whole partitions or TTL, not unbounded DELETE
|
|
408
408
|
- Plan or dry-run before destructive DDL
|
|
409
|
-
- Snapshot totals for cross-period aggregates
|
|
409
|
+
- Snapshot totals for cross-period aggregates ([partition_landscape.md](partition_landscape.md#aggregates-totals-and-snapshots))
|
|
410
410
|
- UI and APIs default to partition-scoped windows ([partition_landscape.md](partition_landscape.md#ui-and-product-surfaces))
|
|
411
|
-
- After large data movement, refresh statistics and
|
|
411
|
+
- After large data movement, refresh statistics, dependent rollups, and routing mappings when keys can change
|
|
412
412
|
|
|
413
413
|
Replace Gardener-specific steps:
|
|
414
414
|
|
data/docs/partition_landscape.md
CHANGED
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Partition Gardener targets PostgreSQL native declarative partitioning with gardener-owned maintenance plans. This page maps industry patterns to what the gem implements, documents experimental layouts, and lists what belongs outside the gem.
|
|
4
4
|
|
|
5
|
+
Canonical home for routing (layers, recovery ladder, hints), pruning, UI scoping, and aggregate snapshots. Host query and write obligations live in [application_contract.md](application_contract.md). Cross-engine portability lives in [partition_engines.md](partition_engines.md).
|
|
6
|
+
|
|
5
7
|
## Implemented templates
|
|
6
8
|
|
|
7
9
|
sliding_window_monthly — layout sliding_window, bucket month. Default time-series on a date or timestamp column.
|
|
@@ -71,6 +73,8 @@ The default railtie uses `ActiveRecord::Base.connection`. Sharded models should
|
|
|
71
73
|
|
|
72
74
|
Choose Rails sharding when the bottleneck is total row volume or tenant isolation across servers. Choose Gardener templates when the bottleneck is partition catalog size, retention, or hot-month splits inside one database.
|
|
73
75
|
|
|
76
|
+
On hot paths carry both signals: shard routing first (`connected_to(shard:)`), then partition key filters inside that shard. Losing either one pushes the system toward broad scans. Gardener maintenance runs per shard connection ([operations.md](operations.md#sharded-registries)); it does not change the query contract ([application_contract.md](application_contract.md#sharded-applications)).
|
|
77
|
+
|
|
74
78
|
## Composite keys and partition pruning
|
|
75
79
|
|
|
76
80
|
Partitioning only speeds up queries when PostgreSQL can skip child tables. That is partition pruning: the planner uses partition bounds and your `WHERE` clause to exclude partitions that cannot hold matching rows. Without pruning, one large scan becomes many smaller scans and latency often gets worse.
|
|
@@ -119,13 +123,98 @@ Inserts must supply routable partition key values (`Event.create!(occurred_on: .
|
|
|
119
123
|
|
|
120
124
|
If you register `partition_key_column: "created_at::date"`, keep application filters on that expression or use a generated column the planner can reason about.
|
|
121
125
|
|
|
122
|
-
###
|
|
126
|
+
### Routing layers
|
|
123
127
|
|
|
124
|
-
|
|
128
|
+
Partition performance depends on choosing the right slice before PostgreSQL plans the query. Think in layers from outside in:
|
|
129
|
+
|
|
130
|
+
1. Cluster or shard — Rails `connected_to(shard:)`, tenant resolver middleware, or a distributed coordinator (Citus). Picks which database server or node set holds the row.
|
|
131
|
+
2. Partition dimension — plain `WHERE` on `partition_key_column` so declarative pruning limits which child tables participate.
|
|
132
|
+
3. Missing-key recovery — when the call site only has a logical id or parent reference, derive a routable key or bounded window before querying the fact table (below).
|
|
133
|
+
4. Inside one child — btree indexes, heat splits, and Gardener tail layout; pruning already selected the child.
|
|
134
|
+
|
|
135
|
+
Gardener maintains layer 4 and the catalog for layer 2. Layers 1 and 3 are application-owned; the gem does not inject routing middleware or enforce predicates.
|
|
136
|
+
|
|
137
|
+
### Routing hints when the key is not in hand
|
|
138
|
+
|
|
139
|
+
Hot paths should carry the real partition key. Many lookups start with only a logical id, a parent foreign key, or a URL token. The planner still needs a plain predicate on the partition dimension. Use an orientation source to supply that predicate without scanning every child.
|
|
140
|
+
|
|
141
|
+
Recovery ladder (prefer earlier steps):
|
|
142
|
+
|
|
143
|
+
1. Exact key on the row or in the request (denormalized column, generated column, cursor token).
|
|
144
|
+
2. Carry the key beside the logical id (API, jobs, admin links).
|
|
145
|
+
3. Orient from a related record (parent timestamp or business period).
|
|
146
|
+
4. Fast-access column or session context already on the hot path.
|
|
147
|
+
5. Durable mapping table or cache (`id → partition_key`).
|
|
148
|
+
6. Bounded scan of recent buckets (last resort; cap parallelism and bucket count).
|
|
149
|
+
|
|
150
|
+
#### Exact key (preferred)
|
|
151
|
+
|
|
152
|
+
- Denormalize the parent business date (or tenant/branch) onto the child at insert time; filter and join on the child column.
|
|
153
|
+
- Store the partition key next to the logical id in API cursors, job args, outbox payloads, and admin deep links. Opaque tokens may encode `(id, partition_key)` so clients never send id-only.
|
|
154
|
+
- Use a generated column when the key is a stable expression of an existing column the planner can see (`created_at::date` registered and filtered the same way).
|
|
155
|
+
|
|
156
|
+
#### Orientation / range from a related record
|
|
157
|
+
|
|
158
|
+
Parent `created_at`, `occurred_on`, or business date often bounds where child rows live. Load the parent (or a thin projection), then query the child with a half-open window around that timestamp plus the child id.
|
|
159
|
+
|
|
160
|
+
```ruby
|
|
161
|
+
order = Order.select(:id, :ordered_on).find(order_id)
|
|
162
|
+
LineItem.where(
|
|
163
|
+
order_id: order.id,
|
|
164
|
+
occurred_on: order.ordered_on...order.ordered_on + 1.day
|
|
165
|
+
)
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Window sizing:
|
|
169
|
+
|
|
170
|
+
- Same calendar day when children are daily and backdating is rare.
|
|
171
|
+
- Same month when children are monthly and the parent period matches the child's bucket.
|
|
172
|
+
- Parent open period (subscription term, fiscal quarter) when product rules tie children to that span.
|
|
173
|
+
- Widen only after a miss inside the first window; log and alert repeated widens.
|
|
125
174
|
|
|
126
|
-
|
|
175
|
+
Prefer the parent's partition-aligned column over wall-clock "now" when the child follows the parent's period. Timezone boundaries and backdated parent updates can shift the correct bucket; test with production-shaped dates.
|
|
127
176
|
|
|
128
|
-
|
|
177
|
+
#### Cached or fast-access columns
|
|
178
|
+
|
|
179
|
+
Columns already on the hot path for other reasons (workspace id, branch, status period, cached `order_date` on line items) are valid routing inputs when they match `partition_key_column`.
|
|
180
|
+
|
|
181
|
+
- Keep them written at insert/update with the same rules as the true key; do not derive them only in Ruby for reads while the column is null in the row.
|
|
182
|
+
- Session or request context (selected month, current workspace) is a routing hint for list scopes; still persist the key on each row for writes and point lookups.
|
|
183
|
+
|
|
184
|
+
#### Global index and lookup caches
|
|
185
|
+
|
|
186
|
+
A narrow mapping table (`logical_id`, `partition_key`, optional `shard_key`) or Redis entry supports global id search and admin deep links without scanning every child.
|
|
187
|
+
|
|
188
|
+
- Treat cache or index miss as a bounded scan of recent buckets or a rejected request, not as an unbounded parent query.
|
|
189
|
+
- Invalidate or refresh when maintenance moves rows across children if the cached key can change (rare for immutable event dates; common if the key is mutable).
|
|
190
|
+
- After a Gardener run with high `rows_moved`, reconcile mapping rows for affected buckets.
|
|
191
|
+
|
|
192
|
+
#### Time-embedded identifiers
|
|
193
|
+
|
|
194
|
+
ULID, KSUID, and Snowflake-style ids embed creation time. They suggest a default orientation window without a parent lookup, but they are not a substitute for `partition_key_column` on the row unless the partition bound is derived from the same clock and registered consistently. See [UUIDv7 and encoded-time keys](#uuidv7-and-encoded-time-keys).
|
|
195
|
+
|
|
196
|
+
#### Two-tier composite layouts
|
|
197
|
+
|
|
198
|
+
`composite_list_range` and similar templates partition first by LIST (tenant, region) then RANGE (month). Partial hints still prune one dimension:
|
|
199
|
+
|
|
200
|
+
- Tenant id alone limits LIST branches; add a month range inside that tenant for RANGE children.
|
|
201
|
+
- Month alone without tenant may scan every LIST branch; always carry the list key on hot paths for multi-tenant trees.
|
|
202
|
+
|
|
203
|
+
#### Verify routing
|
|
204
|
+
|
|
205
|
+
Run `EXPLAIN (ANALYZE, BUFFERS)` on production-shaped lookups. Pruning-capable plans show `Append` with only relevant children or `Subplans Removed: N`. Sequential scans on every child mean the contract is broken.
|
|
206
|
+
|
|
207
|
+
Do not:
|
|
208
|
+
|
|
209
|
+
- Call `find(id)` on a time-partitioned table and hope runtime pruning saves you when `id` is not globally unique per child.
|
|
210
|
+
- Join only on `parent_id` / `id` without also constraining the child's partition key (or a denormalized copy of it).
|
|
211
|
+
- Invent the key from "current month" for historical rows; orientation must come from the row's period or its parent.
|
|
212
|
+
|
|
213
|
+
Gardener does not invent these hints. Migrations, models, and API contracts own them; reviews and `EXPLAIN` prove they prune.
|
|
214
|
+
|
|
215
|
+
### Gardener `conflict_key`
|
|
216
|
+
|
|
217
|
+
Registry `conflict_key` must match a parent unique index and should include the partition key, for example `%w[id occurred_on]`. Gardener uses it for idempotent keyset moves during default drain and tail rebalance. The same composite shape that satisfies PostgreSQL uniqueness is what lets maintenance target one partition without cross-child ambiguity.
|
|
129
218
|
|
|
130
219
|
### UI and product surfaces
|
|
131
220
|
|
|
@@ -247,7 +336,7 @@ After Gardener moves rows between children, refresh or recompute any materialize
|
|
|
247
336
|
|
|
248
337
|
### Common pitfalls (what teams report)
|
|
249
338
|
|
|
250
|
-
- Admin search by `id` only on a time-partitioned table scans every month until the key encodes time or lookups include `occurred_on`
|
|
339
|
+
- Admin search by `id` only on a time-partitioned table scans every month until the key encodes time or lookups include `occurred_on` (see [Routing hints when the key is not in hand](#routing-hints-when-the-key-is-not-in-hand))
|
|
251
340
|
- Reporting queries over wide date ranges intentionally touch many children; that is correct behavior, not a pruning bug
|
|
252
341
|
- Partition count in the thousands increases planner overhead even when pruning works; Gardener sliding window keeps the active catalog bounded
|
|
253
342
|
- Relying on `default` without monitoring; Gardener drains default last, but application inserts that never match bounds still accumulate there
|
|
@@ -279,7 +368,7 @@ Hypertables use chunk policies and compression, not declarative RANGE children.
|
|
|
279
368
|
|
|
280
369
|
### Citus distributed tables
|
|
281
370
|
|
|
282
|
-
Citus distributes tables across worker nodes inside PostgreSQL with its own shard metadata and rebalance tooling. That is neither Rails `connects_to` sharding nor native declarative partitioning on a single instance. Gardener does not drive Citus shard placement.
|
|
371
|
+
Citus distributes tables across worker nodes inside PostgreSQL with its own shard metadata and rebalance tooling. That is neither Rails `connects_to` sharding nor native declarative partitioning on a single instance. Citus sits in [routing layer](#routing-layers) 1 (distribution column picks the worker); Gardener-style declarative children can still exist per worker for retention. Gardener does not drive Citus shard placement.
|
|
283
372
|
|
|
284
373
|
### Other database engines
|
|
285
374
|
|
|
@@ -11,11 +11,11 @@ module PartitionGardener
|
|
|
11
11
|
def run!
|
|
12
12
|
report_audit_warnings
|
|
13
13
|
ensure_default_partition
|
|
14
|
+
rebalance_tail!
|
|
14
15
|
unless MaintenanceBackend.hybrid?(@config)
|
|
15
16
|
finalize_archive_partitions
|
|
16
17
|
apply_archive_retention!
|
|
17
18
|
end
|
|
18
|
-
rebalance_tail!
|
|
19
19
|
drain_default_partition
|
|
20
20
|
end
|
|
21
21
|
|
|
@@ -115,6 +115,8 @@ module PartitionGardener
|
|
|
115
115
|
end
|
|
116
116
|
|
|
117
117
|
def finalize_archive_from_source!(identifier, source_partition_name)
|
|
118
|
+
return if archive_attach_overlaps_current?(identifier)
|
|
119
|
+
|
|
118
120
|
partition_name = archive_partition_name(identifier)
|
|
119
121
|
where_condition = strategy.bucket_where_condition(identifier)
|
|
120
122
|
|
|
@@ -214,6 +216,18 @@ module PartitionGardener
|
|
|
214
216
|
end
|
|
215
217
|
end
|
|
216
218
|
|
|
219
|
+
def archive_attach_overlaps_current?(identifier)
|
|
220
|
+
return false unless strategy.is_a?(Strategy::DateRange)
|
|
221
|
+
|
|
222
|
+
current_lower_bound = Connection.current_partition_lower_bound(
|
|
223
|
+
table_name,
|
|
224
|
+
current_partition_name(table_name)
|
|
225
|
+
)
|
|
226
|
+
return false unless current_lower_bound
|
|
227
|
+
|
|
228
|
+
strategy.beginning_of_bucket(identifier) >= current_lower_bound
|
|
229
|
+
end
|
|
230
|
+
|
|
217
231
|
def ensure_archive_partition_attached!(identifier, partition_name:, source_partition_name:)
|
|
218
232
|
for_values_clause = archive_for_values_clause(identifier)
|
|
219
233
|
where_condition = strategy.bucket_where_condition(identifier)
|
|
@@ -136,6 +136,10 @@ module PartitionGardener
|
|
|
136
136
|
"#{partition_key_column} >= #{connection.quote(start_range)}::date AND #{partition_key_column} < #{connection.quote(end_range)}::date"
|
|
137
137
|
end
|
|
138
138
|
|
|
139
|
+
def beginning_of_bucket(bucket)
|
|
140
|
+
DateBucket.beginning_of_bucket(bucket.to_date, date_bucket)
|
|
141
|
+
end
|
|
142
|
+
|
|
139
143
|
def archive_bucket?(bucket)
|
|
140
144
|
beginning_of_bucket(bucket) < active_window[:start]
|
|
141
145
|
end
|
|
@@ -202,10 +206,6 @@ module PartitionGardener
|
|
|
202
206
|
@config.fetch(:split_row_threshold, FUTURE_MONTH_PARTITION_ROW_THRESHOLD)
|
|
203
207
|
end
|
|
204
208
|
|
|
205
|
-
def beginning_of_bucket(bucket)
|
|
206
|
-
DateBucket.beginning_of_bucket(bucket.to_date, date_bucket)
|
|
207
|
-
end
|
|
208
|
-
|
|
209
209
|
def end_of_bucket(bucket)
|
|
210
210
|
DateBucket.end_of_bucket(bucket.to_date, date_bucket)
|
|
211
211
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: partition_gardener
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.3.
|
|
4
|
+
version: 0.3.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrei Makarov
|
|
@@ -253,14 +253,14 @@ dependencies:
|
|
|
253
253
|
requirements:
|
|
254
254
|
- - "~>"
|
|
255
255
|
- !ruby/object:Gem::Version
|
|
256
|
-
version: '
|
|
256
|
+
version: '4'
|
|
257
257
|
type: :development
|
|
258
258
|
prerelease: false
|
|
259
259
|
version_requirements: !ruby/object:Gem::Requirement
|
|
260
260
|
requirements:
|
|
261
261
|
- - "~>"
|
|
262
262
|
- !ruby/object:Gem::Version
|
|
263
|
-
version: '
|
|
263
|
+
version: '4'
|
|
264
264
|
description: Runtime maintenance for PostgreSQL declarative partitions — archive /
|
|
265
265
|
current / future zones, heat-driven splits, mandatory default drain, hot-switch
|
|
266
266
|
migrations, and strategy templates.
|
|
@@ -370,7 +370,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
370
370
|
requirements:
|
|
371
371
|
- - ">="
|
|
372
372
|
- !ruby/object:Gem::Version
|
|
373
|
-
version: 3.
|
|
373
|
+
version: '3.4'
|
|
374
374
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
375
375
|
requirements:
|
|
376
376
|
- - ">="
|