partition_gardener 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +36 -0
- data/LICENSE.md +21 -0
- data/README.md +203 -0
- data/SECURITY.md +27 -0
- data/docs/THIRD_PARTY_LICENSE_MANIFEST.tsv +60 -0
- data/docs/application_contract.md +82 -0
- data/docs/audit_reference.md +125 -0
- data/docs/background_job.md +161 -0
- data/docs/cli.md +71 -0
- data/docs/configuration.md +261 -0
- data/docs/cutover.md +180 -0
- data/docs/decision_flow.md +154 -0
- data/docs/host_testing.md +91 -0
- data/docs/monitoring.md +110 -0
- data/docs/naming.md +71 -0
- data/docs/operations.md +131 -0
- data/docs/partition_landscape.md +323 -0
- data/docs/pg_party_recipe.md +44 -0
- data/docs/related_postgres_tooling.md +195 -0
- data/docs/retention.md +65 -0
- data/docs/schemas/partition_garden.schema.json +114 -0
- data/docs/schemas/plan_report.schema.json +82 -0
- data/docs/tooling_split.md +79 -0
- data/exe/partition_gardener +6 -0
- data/lib/partition_gardener/active_record_run_record_store.rb +1 -0
- data/lib/partition_gardener/advisory_lock.rb +48 -0
- data/lib/partition_gardener/archive_retention.rb +88 -0
- data/lib/partition_gardener/audit.rb +93 -0
- data/lib/partition_gardener/blank.rb +13 -0
- data/lib/partition_gardener/cli.rb +190 -0
- data/lib/partition_gardener/config_document.rb +240 -0
- data/lib/partition_gardener/configuration.rb +102 -0
- data/lib/partition_gardener/connection.rb +260 -0
- data/lib/partition_gardener/date_bucket.rb +117 -0
- data/lib/partition_gardener/date_calendar.rb +65 -0
- data/lib/partition_gardener/date_range_maintenance.rb +297 -0
- data/lib/partition_gardener/default_partition.rb +54 -0
- data/lib/partition_gardener/executor.rb +324 -0
- data/lib/partition_gardener/gap_detection.rb +59 -0
- data/lib/partition_gardener/hash_routing.rb +70 -0
- data/lib/partition_gardener/layout/calendar_year.rb +28 -0
- data/lib/partition_gardener/layout/integer_window.rb +28 -0
- data/lib/partition_gardener/layout/sliding_window.rb +28 -0
- data/lib/partition_gardener/layout/three_area.rb +27 -0
- data/lib/partition_gardener/layout/zone_segments.rb +62 -0
- data/lib/partition_gardener/lock_not_acquired.rb +3 -0
- data/lib/partition_gardener/maintenance_backend.rb +73 -0
- data/lib/partition_gardener/memory_run_record_store.rb +19 -0
- data/lib/partition_gardener/migration/hot_switch_concern.rb +445 -0
- data/lib/partition_gardener/missing_conflict_index.rb +3 -0
- data/lib/partition_gardener/naming.rb +29 -0
- data/lib/partition_gardener/orphaned_rebalance_staging.rb +3 -0
- data/lib/partition_gardener/pg_connection.rb +94 -0
- data/lib/partition_gardener/plan.rb +49 -0
- data/lib/partition_gardener/plan_applier.rb +289 -0
- data/lib/partition_gardener/plan_diff.rb +51 -0
- data/lib/partition_gardener/plan_report.rb +95 -0
- data/lib/partition_gardener/planner.rb +21 -0
- data/lib/partition_gardener/predicate.rb +85 -0
- data/lib/partition_gardener/premake_monthly_maintenance.rb +44 -0
- data/lib/partition_gardener/rails.rb +12 -0
- data/lib/partition_gardener/registry.rb +84 -0
- data/lib/partition_gardener/run_failed.rb +10 -0
- data/lib/partition_gardener/run_metrics.rb +65 -0
- data/lib/partition_gardener/run_record.rb +76 -0
- data/lib/partition_gardener/sql_run_record_store.rb +106 -0
- data/lib/partition_gardener/stdlib_extensions.rb +15 -0
- data/lib/partition_gardener/strategy/composite.rb +27 -0
- data/lib/partition_gardener/strategy/cursor_columns.rb +18 -0
- data/lib/partition_gardener/strategy/date_range.rb +303 -0
- data/lib/partition_gardener/strategy/hash_branches.rb +161 -0
- data/lib/partition_gardener/strategy/integer_range.rb +261 -0
- data/lib/partition_gardener/strategy/list_split.rb +125 -0
- data/lib/partition_gardener/strategy/requires_default_partition.rb +19 -0
- data/lib/partition_gardener/strategy.rb +26 -0
- data/lib/partition_gardener/templates.rb +373 -0
- data/lib/partition_gardener/unmoved_rows_remaining.rb +15 -0
- data/lib/partition_gardener/version.rb +3 -0
- data/lib/partition_gardener.rb +215 -0
- data/sig/partition_gardener.rbs +19 -0
- metadata +367 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 9dc13f2d2bdae3ecc6cf8e9d66a27f836c1cebbb06bdfd84452e4698e5cb8c63
|
|
4
|
+
data.tar.gz: e60370a277cdcd06b9d2376f4d73169ac29792ed9e7f150a6177d1a1e5703c92
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 70d5dabf55e8e6b207768233e2f4f154784531ba35cac9f9731cae76141f81252810acd9bb9fded3c9547c1b60c207be7ad0cf5baca47a950869fa0dd6463ea3
|
|
7
|
+
data.tar.gz: 23e10104588d5878c59bb25c3d3116d2cf9ea25793c13dff785a0c90b57f545889ec87bc6b56c5a480a1defdf3b367c0de3ec698ef0b85a1953c9c654c55f64e
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# CHANGELOG
|
|
2
|
+
|
|
3
|
+
## 0.3.0 (2026-07-08)
|
|
4
|
+
|
|
5
|
+
- Remove `suggest_template`, `recommend`, and register-time layout reliability warnings
|
|
6
|
+
- Add structured branch predicates for list-split layouts so operators declare column, operator, and value instead of raw SQL fragments
|
|
7
|
+
- Add shorthand list-branch filters from partition key column plus branch value at registration time
|
|
8
|
+
- Document materialized view limits, manual per-slice patterns, and rollup table alternatives in the partition landscape guide
|
|
9
|
+
- Clarify Gardener maintenance scope applies to declarative partitioned tables, not materialized views
|
|
10
|
+
|
|
11
|
+
## 0.2.0 (2026-07-06)
|
|
12
|
+
|
|
13
|
+
- Add configuration, background job, and CLI guides under `docs/`
|
|
14
|
+
- Add `partition_gardener` CLI with `plan`, `audit`, and `apply` for dry-run planning, layout audit, and maintenance
|
|
15
|
+
- Add dry-run plan reports with gap detection and JSON schema export
|
|
16
|
+
- Add hot-switch migration helpers for zero-downtime cutover to partitioned tables
|
|
17
|
+
- Add layout recommendations for date partition keys
|
|
18
|
+
- Add per-table advisory locks during maintenance runs
|
|
19
|
+
- Add `continue_on_error` so one failing table does not block the rest
|
|
20
|
+
- Add run summaries with per-table duration, plan signature, and rows moved
|
|
21
|
+
- Add sliding-window retention policies with optional concurrent detach
|
|
22
|
+
- Add incremental rebalance and resume checkpoints after partial rebalance failure
|
|
23
|
+
- Add per-table statement timeouts during maintenance
|
|
24
|
+
- Add conflict-index verification before row moves
|
|
25
|
+
- Add durable run records when ActiveRecord is available
|
|
26
|
+
- Fix default advisory lock to transaction-scoped try-lock and skip locked tables
|
|
27
|
+
- Fix hash partition layouts to omit unsupported default partitions
|
|
28
|
+
- Fix hot-switch delta sync when there are no updatable columns
|
|
29
|
+
- Fix batch row moves to avoid `ON CONFLICT` stalls during rebalance
|
|
30
|
+
|
|
31
|
+
## 0.1.0 (2026-07-05)
|
|
32
|
+
|
|
33
|
+
- Initial release of PostgreSQL partition lifecycle maintenance
|
|
34
|
+
- Add templates for sliding-window monthly, calendar year, integer window, list split, composite, and hash layouts
|
|
35
|
+
- Add injectable notifier, connection, statement timeout, and today resolver
|
|
36
|
+
- Add Rails railtie for ActiveRecord connection defaults
|
data/LICENSE.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kisko Labs
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# Partition Gardener
|
|
2
|
+
|
|
3
|
+
[](https://badge.fury.io/rb/partition_gardener) [](https://github.com/amkisko/partition_gardener.rb/actions/workflows/test.yml) [](https://codecov.io/gh/amkisko/partition_gardener.rb)
|
|
4
|
+
|
|
5
|
+
PostgreSQL partition lifecycle: archive, current, and future zones, heat-driven splits inside the active window, cursor-based rebalance, mandatory default drain, and hot-switch migration helpers. Rails-integrated by default; works standalone with `pg` and a JSON registry.
|
|
6
|
+
|
|
7
|
+
## Requirements
|
|
8
|
+
|
|
9
|
+
- Ruby >= 3.2
|
|
10
|
+
- PostgreSQL with declarative partitioning
|
|
11
|
+
- `pg` gem (runtime dependency)
|
|
12
|
+
- Rails >= 7.1 optional — railtie loads when Rails is present and sets Active Record connection plus `Time.zone.today`
|
|
13
|
+
|
|
14
|
+
Complements migration gems (e.g. [pg_party](https://github.com/rkrage/pg_party)) — use those for creation DDL; use this gem for runtime maintenance (scheduled or indicator-driven) and cutover migrations. See [docs/related_postgres_tooling.md](docs/related_postgres_tooling.md) for how Gardener pairs with [PgHero](https://github.com/ankane/pghero), [Dexter](https://github.com/ankane/dexter), [pgsync](https://github.com/ankane/pgsync), and [pgslice](https://github.com/ankane/pgslice).
|
|
15
|
+
|
|
16
|
+
## Documentation
|
|
17
|
+
|
|
18
|
+
### Choosing and configuring
|
|
19
|
+
|
|
20
|
+
- [docs/decision_flow.md](docs/decision_flow.md) — when to partition, layout, and method choices
|
|
21
|
+
- [docs/partition_landscape.md](docs/partition_landscape.md) — templates, Rails sharding, pruning, UI, aggregate snapshots, materialized views
|
|
22
|
+
- [docs/configuration.md](docs/configuration.md) — global config, registry, per-table options, JSON import
|
|
23
|
+
- [docs/tooling_split.md](docs/tooling_split.md) — pg_party vs pg_partman vs Gardener
|
|
24
|
+
- [docs/related_postgres_tooling.md](docs/related_postgres_tooling.md) — PgHero, Dexter, pgsync, pgslice vs Gardener
|
|
25
|
+
- [docs/pg_party_recipe.md](docs/pg_party_recipe.md) — creation DDL with pg_party
|
|
26
|
+
|
|
27
|
+
### Operations
|
|
28
|
+
|
|
29
|
+
- [docs/operations.md](docs/operations.md) — runbook, incidents, RunSummary
|
|
30
|
+
- [docs/audit_reference.md](docs/audit_reference.md) — audit and plan warning catalog
|
|
31
|
+
- [docs/monitoring.md](docs/monitoring.md) — metrics, alerts, SLOs
|
|
32
|
+
- [docs/retention.md](docs/retention.md) — detach, drop, compliance, legal hold
|
|
33
|
+
- [docs/background_job.md](docs/background_job.md) — host job pattern, concurrency, scheduling
|
|
34
|
+
- [docs/cli.md](docs/cli.md) — plan, audit, and apply commands
|
|
35
|
+
|
|
36
|
+
### Application and migration
|
|
37
|
+
|
|
38
|
+
- [docs/cutover.md](docs/cutover.md) — hot-switch playbook and template upgrades
|
|
39
|
+
- [docs/application_contract.md](docs/application_contract.md) — writes, moves, bulk load, replicas
|
|
40
|
+
- [docs/naming.md](docs/naming.md) — child partition naming catalog
|
|
41
|
+
- [docs/host_testing.md](docs/host_testing.md) — CI and staging for host apps
|
|
42
|
+
|
|
43
|
+
### Schemas
|
|
44
|
+
|
|
45
|
+
- [docs/schemas/](docs/schemas/) — JSON schemas for registry and plan reports
|
|
46
|
+
|
|
47
|
+
## Quick start
|
|
48
|
+
|
|
49
|
+
```ruby
|
|
50
|
+
# Gemfile
|
|
51
|
+
gem "partition_gardener"
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
# config/initializers/partition_gardener.rb — see docs/configuration.md for all options
|
|
56
|
+
PartitionGardener.configure do |config|
|
|
57
|
+
config.notifier = ->(message, context: {}) { Rails.logger.info(message) }
|
|
58
|
+
config.today_resolver = -> { Time.zone.today }
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
PartitionGardener::Registry.register_template(
|
|
62
|
+
:sliding_window_monthly,
|
|
63
|
+
table_name: "events",
|
|
64
|
+
partition_key_column: "occurred_on",
|
|
65
|
+
conflict_key: %w[id occurred_on],
|
|
66
|
+
active_months: 12
|
|
67
|
+
)
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
# app/jobs/partition_maintenance_job.rb — see docs/background_job.md
|
|
72
|
+
class PartitionMaintenanceJob < ApplicationJob
|
|
73
|
+
def perform
|
|
74
|
+
PartitionGardener.run!(job_class_name: self.class.name)
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# Dry-run plan from the app directory
|
|
81
|
+
bundle exec partition_gardener --rails plan events --pretty
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### Standalone (no Rails)
|
|
85
|
+
|
|
86
|
+
Set `DATABASE_URL` and point the CLI at a JSON registry:
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
export DATABASE_URL=postgres://postgres@127.0.0.1:5432/mydb
|
|
90
|
+
bundle exec partition_gardener --registry config/partition_garden.json plan events --pretty
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Or configure in Ruby:
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
PartitionGardener.configure do |config|
|
|
97
|
+
config.connection_resolver = -> { PartitionGardener::PgConnection.connect(ENV.fetch("DATABASE_URL")) }
|
|
98
|
+
config.today_resolver = -> { Date.today }
|
|
99
|
+
end
|
|
100
|
+
PartitionGardener::ConfigDocument.load_registry_file!("config/partition_garden.json")
|
|
101
|
+
PartitionGardener.run!
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
## Default layout
|
|
105
|
+
|
|
106
|
+
Default layout: monthly sliding window (`register_template :sliding_window_monthly` in the Rails example above). Three-area layout with planner-enforced non-overlapping ranges, default drained last, and keyset moves on `(partition_key, conflict_key)`.
|
|
107
|
+
|
|
108
|
+
See [docs/decision_flow.md](docs/decision_flow.md) for when to pick other templates.
|
|
109
|
+
|
|
110
|
+
## Templates
|
|
111
|
+
|
|
112
|
+
`Templates.sliding_window_monthly` (layout `:sliding_window`, bucket `:month`) — `RANGE (date)` monthly time-series.
|
|
113
|
+
|
|
114
|
+
`Templates.sliding_window_daily` (bucket `:day`) — daily buckets for short retention telemetry.
|
|
115
|
+
|
|
116
|
+
`Templates.sliding_window_weekly` (bucket `:week`) — ISO-week buckets.
|
|
117
|
+
|
|
118
|
+
`Templates.sliding_window_quarterly` (bucket `:quarter`) — calendar-quarter buckets.
|
|
119
|
+
|
|
120
|
+
`Templates.calendar_year` (layout `:calendar_year`) — `RANGE (date)` yearly buckets.
|
|
121
|
+
|
|
122
|
+
`Templates.rolling_current_monthly` (layout `:rolling_current`) — monthly sliding window without heat splits.
|
|
123
|
+
|
|
124
|
+
`Templates.integer_window` (layout `:integer_window`) — `RANGE (bigint)` id bands.
|
|
125
|
+
|
|
126
|
+
`Templates.list_split` (layout `:list_split`) — fixed `LIST` branches.
|
|
127
|
+
|
|
128
|
+
`Templates.composite_list_hash` (layout `:composite`) — LIST parent plus HASH sub-trees.
|
|
129
|
+
|
|
130
|
+
`Templates.composite_list_range` / `Templates.list_range` — LIST parent plus RANGE sliding-window sub-trees.
|
|
131
|
+
|
|
132
|
+
`Templates.composite_range_hash` — RANGE parent plus HASH child tables.
|
|
133
|
+
|
|
134
|
+
`Templates.composite_range_list` — RANGE parent plus LIST child tables.
|
|
135
|
+
|
|
136
|
+
`Templates.hash_branches` (layout `:hash_branches`) — `HASH` remainders.
|
|
137
|
+
|
|
138
|
+
`Templates.premake_monthly` (layout `:premake_monthly`) — cron-style premake bridge; migrate to sliding window.
|
|
139
|
+
|
|
140
|
+
See [docs/partition_landscape.md](docs/partition_landscape.md) for the template catalog, Rails sharding, composite keys, partition pruning, UI and product surfaces, aggregate snapshots, and materialized view limits. Operations: [operations.md](docs/operations.md), [cutover.md](docs/cutover.md), [monitoring.md](docs/monitoring.md).
|
|
141
|
+
|
|
142
|
+
## Hot-switch migrations
|
|
143
|
+
|
|
144
|
+
`PartitionGardener::Migration::HotSwitchConcern` (alias `HotSwitchPartitionedTable`) consolidates cutover migration helpers.
|
|
145
|
+
|
|
146
|
+
```ruby
|
|
147
|
+
class PartitionEventsHotSwitch < ActiveRecord::Migration[8.0]
|
|
148
|
+
include PartitionGardener::Migration::HotSwitchConcern
|
|
149
|
+
|
|
150
|
+
HOT_SWITCH_CONFIG = {
|
|
151
|
+
current_table: "events",
|
|
152
|
+
partitioned_table: "p_events",
|
|
153
|
+
partition_key_column: "occurred_on",
|
|
154
|
+
conflict_key: %w[id occurred_on],
|
|
155
|
+
partition_config: PartitionGardener::Registry.hot_switch_partition_config("events")
|
|
156
|
+
}.freeze
|
|
157
|
+
|
|
158
|
+
def up
|
|
159
|
+
ensure_future_partitions_exist(months_ahead: 1)
|
|
160
|
+
hot_switch_tables
|
|
161
|
+
end
|
|
162
|
+
end
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
`partition_config` may also be an inline hash for tables not yet registered.
|
|
166
|
+
|
|
167
|
+
## Runtime guarantees
|
|
168
|
+
|
|
169
|
+
- Per-table advisory lock during maintenance (`hashtext` namespace plus table name)
|
|
170
|
+
- `continue_on_error: true` by default — one failing table does not block others (`PartitionGardener::RunFailed` aggregates errors)
|
|
171
|
+
- Batch moves use composite keyset cursors and delete source rows by batch keys
|
|
172
|
+
- `run!` returns `RunSummary` with per-table duration, `plan_signature`, and `rows_moved`
|
|
173
|
+
- `maintenance_backend: :pg_partman` skips gardener for partman-owned tables
|
|
174
|
+
|
|
175
|
+
## Tests
|
|
176
|
+
|
|
177
|
+
App-agnostic PostgreSQL integration specs live under `spec/integration/`. CI and local full-suite runs use [polyrun](https://github.com/amkisko/polyrun) to shard specs across parallel workers (each worker uses its own PostgreSQL database when `DATABASE_URL` is set).
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
# Full suite (unit + integration; requires PostgreSQL)
|
|
181
|
+
make test
|
|
182
|
+
|
|
183
|
+
# Same as make test
|
|
184
|
+
INTEGRATION=1 DATABASE_URL=postgres://postgres@127.0.0.1:5432/partition_gardener_test ./bin/polyrun parallel-rspec --workers 5
|
|
185
|
+
|
|
186
|
+
# Unit specs only
|
|
187
|
+
bundle exec rspec --exclude-pattern "spec/integration/**/*_spec.rb"
|
|
188
|
+
|
|
189
|
+
# Integration only (requires PostgreSQL)
|
|
190
|
+
INTEGRATION=1 DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/partition_gardener_test bundle exec rake spec:integration
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
## License
|
|
194
|
+
|
|
195
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
|
196
|
+
|
|
197
|
+
## Sponsors
|
|
198
|
+
|
|
199
|
+
Sponsored by [Kisko Labs](https://www.kiskolabs.com).
|
|
200
|
+
|
|
201
|
+
<a href="https://www.kiskolabs.com">
|
|
202
|
+
<img src="kisko.svg" width="200" alt="Sponsored by Kisko Labs" />
|
|
203
|
+
</a>
|
data/SECURITY.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Security
|
|
2
|
+
|
|
3
|
+
## Reporting a vulnerability
|
|
4
|
+
|
|
5
|
+
Do not open a public GitHub issue for security vulnerabilities.
|
|
6
|
+
|
|
7
|
+
Email security details to: security@kiskolabs.com
|
|
8
|
+
|
|
9
|
+
Include: description, steps to reproduce, potential impact, and suggested fix (if available).
|
|
10
|
+
|
|
11
|
+
### Response timeline
|
|
12
|
+
|
|
13
|
+
- We will acknowledge receipt of your report
|
|
14
|
+
- We will provide an initial assessment
|
|
15
|
+
- We will keep you informed of our progress and resolution timeline
|
|
16
|
+
|
|
17
|
+
### Disclosure policy
|
|
18
|
+
|
|
19
|
+
- We will work with you to understand and resolve the issue
|
|
20
|
+
- We will credit you for the discovery (unless you prefer to remain anonymous)
|
|
21
|
+
- We will publish a security advisory after the vulnerability is patched
|
|
22
|
+
- We will coordinate public disclosure with you
|
|
23
|
+
|
|
24
|
+
## Automation security
|
|
25
|
+
|
|
26
|
+
- Context isolation. It is strictly forbidden to include production credentials, API keys, or personally identifiable information in prompts sent to third-party LLMs or automation services.
|
|
27
|
+
- Supply chain. All automated dependencies must be verified.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
bundle gem version licenses
|
|
2
|
+
root activemodel 8.1.3 MIT
|
|
3
|
+
root activerecord 8.1.3 MIT
|
|
4
|
+
root activesupport 8.1.3 MIT
|
|
5
|
+
root appraisal 2.5.0 MIT
|
|
6
|
+
root ast 2.4.3 MIT
|
|
7
|
+
root base64 0.3.0 Ruby|BSD-2-Clause
|
|
8
|
+
root bigdecimal 4.1.2 Ruby|BSD-2-Clause
|
|
9
|
+
root bundler 4.0.15 MIT
|
|
10
|
+
root concurrent-ruby 1.3.7 MIT
|
|
11
|
+
root connection_pool 3.0.2 MIT
|
|
12
|
+
root diff-lcs 1.6.2 MIT|Artistic-1.0-Perl|GPL-2.0-or-later
|
|
13
|
+
root drb 2.2.3 Ruby|BSD-2-Clause
|
|
14
|
+
root i18n 1.15.2 MIT
|
|
15
|
+
root json 2.20.0 Ruby
|
|
16
|
+
root language_server-protocol 3.17.0.6 MIT
|
|
17
|
+
root lint_roller 1.1.0 MIT
|
|
18
|
+
root logger 1.7.0 Ruby|BSD-2-Clause
|
|
19
|
+
root minitest 6.0.6 MIT
|
|
20
|
+
root parallel 2.1.0 MIT
|
|
21
|
+
root parser 3.3.11.1 MIT
|
|
22
|
+
root partition_gardener 0.3.0 MIT
|
|
23
|
+
root pg 1.6.3 BSD-2-Clause
|
|
24
|
+
root polyrun 1.5.0 MIT
|
|
25
|
+
root prism 1.9.0 MIT
|
|
26
|
+
root racc 1.8.1 Ruby|BSD-2-Clause
|
|
27
|
+
root rack 3.2.6 MIT
|
|
28
|
+
root rainbow 3.1.1 MIT
|
|
29
|
+
root rake 13.4.2 MIT
|
|
30
|
+
root rbs 3.10.4 BSD-2-Clause|Ruby
|
|
31
|
+
root regexp_parser 2.12.0 MIT
|
|
32
|
+
root rspec 3.13.2 MIT
|
|
33
|
+
root rspec-core 3.13.6 MIT
|
|
34
|
+
root rspec-expectations 3.13.5 MIT
|
|
35
|
+
root rspec-mocks 3.13.8 MIT
|
|
36
|
+
root rspec-support 3.13.7 MIT
|
|
37
|
+
root rspec_junit_formatter 0.6.0 MIT
|
|
38
|
+
root rubocop 1.87.0 MIT
|
|
39
|
+
root rubocop-ast 1.50.0 MIT
|
|
40
|
+
root rubocop-capybara 2.23.0 MIT
|
|
41
|
+
root rubocop-factory_bot 2.28.0 MIT
|
|
42
|
+
root rubocop-performance 1.26.1 MIT
|
|
43
|
+
root rubocop-rails 2.34.3 MIT
|
|
44
|
+
root rubocop-rspec 3.10.2 MIT
|
|
45
|
+
root rubocop-rspec_rails 2.32.0 MIT
|
|
46
|
+
root rubocop-thread_safety 0.7.3 MIT
|
|
47
|
+
root ruby-progressbar 1.13.0 MIT
|
|
48
|
+
root securerandom 0.4.1 Ruby|BSD-2-Clause
|
|
49
|
+
root standard 1.55.0
|
|
50
|
+
root standard-custom 1.0.2 MIT
|
|
51
|
+
root standard-performance 1.9.0 MIT
|
|
52
|
+
root standard-rails 1.6.0 MIT
|
|
53
|
+
root standard-rspec 0.5.0 MIT
|
|
54
|
+
root thor 1.5.0 MIT
|
|
55
|
+
root timeout 0.6.1 Ruby|BSD-2-Clause
|
|
56
|
+
root tsort 0.2.0 Ruby|BSD-2-Clause
|
|
57
|
+
root tzinfo 2.0.6 MIT
|
|
58
|
+
root unicode-display_width 3.2.0 MIT
|
|
59
|
+
root unicode-emoji 4.2.0 MIT
|
|
60
|
+
root uri 1.1.1 Ruby|BSD-2-Clause
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Application contract
|
|
2
|
+
|
|
3
|
+
How host applications should behave around partitioned tables: queries, writes, maintenance side effects, bulk load, and replicas. For UI scoping and aggregate snapshots see [partition_landscape.md](partition_landscape.md). For operator runbooks see [operations.md](operations.md).
|
|
4
|
+
|
|
5
|
+
## Query and write basics
|
|
6
|
+
|
|
7
|
+
- Hot-path reads and writes include the partition key in plain predicates (no wrappers on the key column).
|
|
8
|
+
- `conflict_key` columns match the parent unique index; updates and deletes use `query_constraints` when the logical id is not globally unique ([partition_landscape.md](partition_landscape.md#rails-application-contract)).
|
|
9
|
+
- Inserts supply a routable partition key value so rows land in named children, not only in `default`.
|
|
10
|
+
|
|
11
|
+
## During maintenance (row moves)
|
|
12
|
+
|
|
13
|
+
Maintenance runs move rows in batches keyed by `(partition_key, conflict_key)`:
|
|
14
|
+
|
|
15
|
+
- Moves are delete-from-source, insert-into-target within transactions; logical row identity is preserved when `conflict_key` is stable.
|
|
16
|
+
- Row-level triggers and callbacks fire per batch; prefer idempotent side effects or defer heavy work to jobs keyed by `conflict_key`.
|
|
17
|
+
- Counter caches and Rails `counter_cache` on associations may drift during moves; use snapshots or reconcile after maintenance ([partition_landscape.md](partition_landscape.md#aggregates-totals-and-snapshots)).
|
|
18
|
+
- Outbox and change-data-capture streams may emit move pairs; consumers should treat `(id, partition_key)` as identity.
|
|
19
|
+
|
|
20
|
+
After a run with high `rows_moved`, schedule snapshot recompute for affected buckets and watch replica lag.
|
|
21
|
+
|
|
22
|
+
## Insert routing failures
|
|
23
|
+
|
|
24
|
+
Row in `default` — key outside attached bounds or horizon lag. Application response: fix key; gardener drains default on next run.
|
|
25
|
+
|
|
26
|
+
PostgreSQL routing error — no child accepts key. Application response: `apply` to extend horizon; never silence without ops review.
|
|
27
|
+
|
|
28
|
+
Duplicate key across children — overlapping manual DDL. Application response: stop writes; `plan` / `apply`; remove overlapping attach.
|
|
29
|
+
|
|
30
|
+
Applications should not catch routing errors and retry without the partition key.
|
|
31
|
+
|
|
32
|
+
## Bulk import and backfill
|
|
33
|
+
|
|
34
|
+
- Chunk `COPY` or `insert_all` by bucket window within partition bounds (month, week, day).
|
|
35
|
+
- Set partition key explicitly on every row; do not rely on defaults that omit the key.
|
|
36
|
+
- Run imports during low traffic or against a shadow table during cutover ([cutover.md](cutover.md)).
|
|
37
|
+
- Disable or throttle per-row callbacks on mass load; run bucket snapshot recompute after import completes.
|
|
38
|
+
|
|
39
|
+
## Read replicas
|
|
40
|
+
|
|
41
|
+
- Row moves generate write load on the primary; replicas may lag during large `rows_moved` nights.
|
|
42
|
+
- Do not run reporting aggregates on replicas that must be strictly current during maintenance windows; use snapshots or primary with bounded windows.
|
|
43
|
+
- Logical replication of partitioned parents replicates to the parent; child layout is visible on subscribers. Coordinate major layout changes with replication monitoring.
|
|
44
|
+
|
|
45
|
+
## Foreign keys and references
|
|
46
|
+
|
|
47
|
+
PostgreSQL limits foreign keys referencing partitioned tables and FKs across partition boundaries. Prefer:
|
|
48
|
+
|
|
49
|
+
- Application-level integrity for cross-table references into partitioned facts.
|
|
50
|
+
- Composite references that include the partition key when FKs are required.
|
|
51
|
+
- Document any FK from partitioned child to dimension table in migration reviews.
|
|
52
|
+
|
|
53
|
+
Gardener does not add or remove FKs during maintenance.
|
|
54
|
+
|
|
55
|
+
## Sharded applications
|
|
56
|
+
|
|
57
|
+
Compose Rails shard routing with per-shard partition filters:
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
ApplicationRecord.connected_to(shard: :tenant_a) do
|
|
61
|
+
Event.in_window(month_range).where(workspace_id: workspace.id)
|
|
62
|
+
end
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Maintenance runs per shard; registry JSON may differ per shard only if layouts differ (unusual).
|
|
66
|
+
|
|
67
|
+
## Admin and operator surfaces
|
|
68
|
+
|
|
69
|
+
- Default filters: current month or selected tenant, not all history.
|
|
70
|
+
- Global id search is an advanced, slow path; require date or tenant hint.
|
|
71
|
+
- Export flows chunk by bucket; show progress per period.
|
|
72
|
+
- Totals read from snapshot tables with `computed_at`, not live `SUM` across all children or a stale materialized view over the full fact table ([partition_landscape.md](partition_landscape.md#materialized-views)).
|
|
73
|
+
|
|
74
|
+
## Testing expectations
|
|
75
|
+
|
|
76
|
+
See [host_testing.md](host_testing.md) for CI registry fixtures and integration smoke tests.
|
|
77
|
+
|
|
78
|
+
## Related
|
|
79
|
+
|
|
80
|
+
- [partition_landscape.md](partition_landscape.md) — pruning, UI, snapshots
|
|
81
|
+
- [cutover.md](cutover.md) — backfill and switch
|
|
82
|
+
- [naming.md](naming.md) — child table names
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
# Audit and plan reference
|
|
2
|
+
|
|
3
|
+
Human-readable catalog for `PartitionGardener.audit`, `PartitionGardener.plan`, CLI `audit` / `plan`, and `RunSummary` output. JSON schemas: [plan_report.schema.json](schemas/plan_report.schema.json), [partition_garden.schema.json](schemas/partition_garden.schema.json).
|
|
4
|
+
|
|
5
|
+
## Audit result (`schema_version` 1.0)
|
|
6
|
+
|
|
7
|
+
`table_name` (string) — parent table.
|
|
8
|
+
|
|
9
|
+
`partitioned` (boolean) — declarative parent exists.
|
|
10
|
+
|
|
11
|
+
`default_row_count` (integer) — rows in `{table}_default`.
|
|
12
|
+
|
|
13
|
+
`attached_child_count` (integer) — attached children including default.
|
|
14
|
+
|
|
15
|
+
`horizon_days` (integer or null) — days from `today` to latest finite upper bound among children.
|
|
16
|
+
|
|
17
|
+
`gaps` (array) — structural range gaps in tail layout.
|
|
18
|
+
|
|
19
|
+
`warnings` (array of string) — human-readable issues.
|
|
20
|
+
|
|
21
|
+
Horizon warning threshold: 30 days (`Audit::HORIZON_WARNING_DAYS`). High child count warning: more than 200 attached children.
|
|
22
|
+
|
|
23
|
+
## Audit warning catalog
|
|
24
|
+
|
|
25
|
+
`{table} is not a partitioned table` — parent missing or not partitioned. Fix migration; check `connection_resolver`.
|
|
26
|
+
|
|
27
|
+
`default partition {name} has {n} rows` — rows routed to default. Fix inserts; run `apply` to drain.
|
|
28
|
+
|
|
29
|
+
`default partition {name} is missing` — no default child. Add default partition for layouts that require it.
|
|
30
|
+
|
|
31
|
+
`partition horizon is {d} days ahead (below 30)` — future bound too close. Run `apply`; check premake / `active_*`.
|
|
32
|
+
|
|
33
|
+
`attached child count is {n} (high catalog pressure)` — too many children. Retention; reduce `active_months`; remove orphans.
|
|
34
|
+
|
|
35
|
+
`partition gap: uncovered range between {a} and {b} ({from}..{to})` — hole between adjacent tail segments. `plan` then `apply`; remove overlapping manual DDL.
|
|
36
|
+
|
|
37
|
+
`partition gap: no attached tail partition extends to MAXVALUE` — missing future/open tail. `apply` to attach `_future`.
|
|
38
|
+
|
|
39
|
+
Gap objects include `range_start`, `range_end`, and `message`.
|
|
40
|
+
|
|
41
|
+
## Plan report
|
|
42
|
+
|
|
43
|
+
`schema_version` — `"1.0"`.
|
|
44
|
+
|
|
45
|
+
`table_name` — parent.
|
|
46
|
+
|
|
47
|
+
`layout` — registry layout symbol (e.g. `sliding_window`).
|
|
48
|
+
|
|
49
|
+
`changed` — `true` when operations would mutate catalog or move rows.
|
|
50
|
+
|
|
51
|
+
`plan_signature` — stable hash of target segments; stored in run records for resume.
|
|
52
|
+
|
|
53
|
+
`target_segments` — desired segment list (`name`, `range_start`, `range_end`, `kind`).
|
|
54
|
+
|
|
55
|
+
`attached_segments` — current catalog segments.
|
|
56
|
+
|
|
57
|
+
`operations` — planned steps: `keep`, `create`, `reshape`, `drop`.
|
|
58
|
+
|
|
59
|
+
`gaps` — same structure as audit gaps on attached tail.
|
|
60
|
+
|
|
61
|
+
`hot_buckets` — buckets exceeding heat threshold inside active window.
|
|
62
|
+
|
|
63
|
+
Use `changed: false` to skip apply when only verifying drift.
|
|
64
|
+
|
|
65
|
+
### Operation actions
|
|
66
|
+
|
|
67
|
+
`keep` — segment already matches target.
|
|
68
|
+
|
|
69
|
+
`create` — new child attach.
|
|
70
|
+
|
|
71
|
+
`reshape` — bound or name change; may move rows.
|
|
72
|
+
|
|
73
|
+
`drop` — detach or drop child (retention or layout).
|
|
74
|
+
|
|
75
|
+
## RunSummary (apply / run!)
|
|
76
|
+
|
|
77
|
+
`schema_version` — `"1.0"`.
|
|
78
|
+
|
|
79
|
+
`tables` — array of per-table metrics objects.
|
|
80
|
+
|
|
81
|
+
`errors` — string messages; run raises `RunFailed` if non-empty (with default `continue_on_error`).
|
|
82
|
+
|
|
83
|
+
Per-table metrics (`RunMetrics#to_h`):
|
|
84
|
+
|
|
85
|
+
`table_name` — parent.
|
|
86
|
+
|
|
87
|
+
`duration_ms` — elapsed milliseconds.
|
|
88
|
+
|
|
89
|
+
`plan_signature` — signature applied (null if skipped early).
|
|
90
|
+
|
|
91
|
+
`rows_moved` — rows relocated by maintenance.
|
|
92
|
+
|
|
93
|
+
`skipped` — boolean.
|
|
94
|
+
|
|
95
|
+
`skip_reason` — `lock_not_acquired`, etc.
|
|
96
|
+
|
|
97
|
+
## Run record table
|
|
98
|
+
|
|
99
|
+
When `run_record_enabled`, PostgreSQL table `partition_gardener_run_records` stores:
|
|
100
|
+
|
|
101
|
+
`table_name` — parent.
|
|
102
|
+
|
|
103
|
+
`phase` — rebalance phase checkpoint.
|
|
104
|
+
|
|
105
|
+
`plan_signature` — resume when matches current plan.
|
|
106
|
+
|
|
107
|
+
`staging_row_count` — progress through staging partition.
|
|
108
|
+
|
|
109
|
+
`updated_at` — last checkpoint time.
|
|
110
|
+
|
|
111
|
+
Disable globally or per table if you do not want checkpoint storage ([configuration.md](configuration.md)).
|
|
112
|
+
|
|
113
|
+
## CLI mapping
|
|
114
|
+
|
|
115
|
+
`audit TABLE` — `PartitionGardener.audit(TABLE)`.
|
|
116
|
+
|
|
117
|
+
`plan TABLE` — `PartitionGardener.plan(table_name: TABLE)`.
|
|
118
|
+
|
|
119
|
+
`apply --confirm TABLE` — `PartitionGardener.run!(table_name: TABLE)`.
|
|
120
|
+
|
|
121
|
+
## Related
|
|
122
|
+
|
|
123
|
+
- [operations.md](operations.md) — remediation workflows
|
|
124
|
+
- [monitoring.md](monitoring.md) — metrics from these fields
|
|
125
|
+
- [naming.md](naming.md) — segment names in output
|