partition_gardener 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +36 -0
  3. data/LICENSE.md +21 -0
  4. data/README.md +203 -0
  5. data/SECURITY.md +27 -0
  6. data/docs/THIRD_PARTY_LICENSE_MANIFEST.tsv +60 -0
  7. data/docs/application_contract.md +82 -0
  8. data/docs/audit_reference.md +125 -0
  9. data/docs/background_job.md +161 -0
  10. data/docs/cli.md +71 -0
  11. data/docs/configuration.md +261 -0
  12. data/docs/cutover.md +180 -0
  13. data/docs/decision_flow.md +154 -0
  14. data/docs/host_testing.md +91 -0
  15. data/docs/monitoring.md +110 -0
  16. data/docs/naming.md +71 -0
  17. data/docs/operations.md +131 -0
  18. data/docs/partition_landscape.md +323 -0
  19. data/docs/pg_party_recipe.md +44 -0
  20. data/docs/related_postgres_tooling.md +195 -0
  21. data/docs/retention.md +65 -0
  22. data/docs/schemas/partition_garden.schema.json +114 -0
  23. data/docs/schemas/plan_report.schema.json +82 -0
  24. data/docs/tooling_split.md +79 -0
  25. data/exe/partition_gardener +6 -0
  26. data/lib/partition_gardener/active_record_run_record_store.rb +1 -0
  27. data/lib/partition_gardener/advisory_lock.rb +48 -0
  28. data/lib/partition_gardener/archive_retention.rb +88 -0
  29. data/lib/partition_gardener/audit.rb +93 -0
  30. data/lib/partition_gardener/blank.rb +13 -0
  31. data/lib/partition_gardener/cli.rb +190 -0
  32. data/lib/partition_gardener/config_document.rb +240 -0
  33. data/lib/partition_gardener/configuration.rb +102 -0
  34. data/lib/partition_gardener/connection.rb +260 -0
  35. data/lib/partition_gardener/date_bucket.rb +117 -0
  36. data/lib/partition_gardener/date_calendar.rb +65 -0
  37. data/lib/partition_gardener/date_range_maintenance.rb +297 -0
  38. data/lib/partition_gardener/default_partition.rb +54 -0
  39. data/lib/partition_gardener/executor.rb +324 -0
  40. data/lib/partition_gardener/gap_detection.rb +59 -0
  41. data/lib/partition_gardener/hash_routing.rb +70 -0
  42. data/lib/partition_gardener/layout/calendar_year.rb +28 -0
  43. data/lib/partition_gardener/layout/integer_window.rb +28 -0
  44. data/lib/partition_gardener/layout/sliding_window.rb +28 -0
  45. data/lib/partition_gardener/layout/three_area.rb +27 -0
  46. data/lib/partition_gardener/layout/zone_segments.rb +62 -0
  47. data/lib/partition_gardener/lock_not_acquired.rb +3 -0
  48. data/lib/partition_gardener/maintenance_backend.rb +73 -0
  49. data/lib/partition_gardener/memory_run_record_store.rb +19 -0
  50. data/lib/partition_gardener/migration/hot_switch_concern.rb +445 -0
  51. data/lib/partition_gardener/missing_conflict_index.rb +3 -0
  52. data/lib/partition_gardener/naming.rb +29 -0
  53. data/lib/partition_gardener/orphaned_rebalance_staging.rb +3 -0
  54. data/lib/partition_gardener/pg_connection.rb +94 -0
  55. data/lib/partition_gardener/plan.rb +49 -0
  56. data/lib/partition_gardener/plan_applier.rb +289 -0
  57. data/lib/partition_gardener/plan_diff.rb +51 -0
  58. data/lib/partition_gardener/plan_report.rb +95 -0
  59. data/lib/partition_gardener/planner.rb +21 -0
  60. data/lib/partition_gardener/predicate.rb +85 -0
  61. data/lib/partition_gardener/premake_monthly_maintenance.rb +44 -0
  62. data/lib/partition_gardener/rails.rb +12 -0
  63. data/lib/partition_gardener/registry.rb +84 -0
  64. data/lib/partition_gardener/run_failed.rb +10 -0
  65. data/lib/partition_gardener/run_metrics.rb +65 -0
  66. data/lib/partition_gardener/run_record.rb +76 -0
  67. data/lib/partition_gardener/sql_run_record_store.rb +106 -0
  68. data/lib/partition_gardener/stdlib_extensions.rb +15 -0
  69. data/lib/partition_gardener/strategy/composite.rb +27 -0
  70. data/lib/partition_gardener/strategy/cursor_columns.rb +18 -0
  71. data/lib/partition_gardener/strategy/date_range.rb +303 -0
  72. data/lib/partition_gardener/strategy/hash_branches.rb +161 -0
  73. data/lib/partition_gardener/strategy/integer_range.rb +261 -0
  74. data/lib/partition_gardener/strategy/list_split.rb +125 -0
  75. data/lib/partition_gardener/strategy/requires_default_partition.rb +19 -0
  76. data/lib/partition_gardener/strategy.rb +26 -0
  77. data/lib/partition_gardener/templates.rb +373 -0
  78. data/lib/partition_gardener/unmoved_rows_remaining.rb +15 -0
  79. data/lib/partition_gardener/version.rb +3 -0
  80. data/lib/partition_gardener.rb +215 -0
  81. data/sig/partition_gardener.rbs +19 -0
  82. metadata +367 -0
@@ -0,0 +1,161 @@
1
+ # Background job integration
2
+
3
+ Partition Gardener does not ship an Active Job class, queue adapter, or cron schedule. The gem exposes `PartitionGardener.run!`; the host application owns scheduling, queue choice, and concurrency policy.
4
+
5
+ ## Minimal job
6
+
7
+ ```ruby
8
+ class PartitionMaintenanceJob < ApplicationJob
9
+ queue_as :cleaners
10
+
11
+ def perform
12
+ PartitionGardener.run!(job_class_name: self.class.name)
13
+ end
14
+ end
15
+ ```
16
+
17
+ Pass `job_class_name:` so notifier context and run metadata identify the host job (for example in Sentry or structured logs).
18
+
19
+ ## Recommended host-app practices
20
+
21
+ ### One maintainer at a time per table
22
+
23
+ Gardener acquires a per-table PostgreSQL advisory lock during `run!`. A second worker skips that table with `lock_not_acquired` rather than blocking forever.
24
+
25
+ Still limit full maintenance runs to one concurrent job for the whole registry when possible. Two jobs iterating all tables waste work and inflate log noise even when locks serialize per table.
26
+
27
+ Example with Good Job:
28
+
29
+ ```ruby
30
+ class PartitionMaintenanceJob < ApplicationJob
31
+ queue_as :cleaners
32
+
33
+ include GoodJob::ActiveJobExtensions::Concurrency
34
+
35
+ good_job_control_concurrency_with(
36
+ key: -> { self.class.name },
37
+ total_limit: 1
38
+ )
39
+
40
+ def perform
41
+ PartitionGardener.run!(job_class_name: self.class.name)
42
+ end
43
+ end
44
+ ```
45
+
46
+ Sidekiq, Solid Queue, and other adapters have similar uniqueness or concurrency controls — use whatever matches the host app.
47
+
48
+ ### Schedule
49
+
50
+ Sliding-window maintenance is designed for periodic runs, not per-insert work. How often to call `run!` depends on service load and move tolerance.
51
+
52
+ #### Scheduled maintenance (default)
53
+
54
+ Run nightly or off-peak when row moves on a fixed cadence are acceptable and audit usually stays clean between runs.
55
+
56
+ Typical cadence:
57
+
58
+ - Production: once per day after the business-date rollover the app uses (`today_resolver`)
59
+ - Staging: same schedule to catch config drift before production
60
+ - After cutover: run manually or on the next schedule slot; verify default partition row count reaches zero
61
+
62
+ #### Indicator-driven maintenance
63
+
64
+ Some services should not run maintenance nightly. Defer `run!` until audit indicators clearly show need: non-zero default, horizon below threshold, heat split warranted, or gap warnings. Schedule read-only `audit` on a fixed cadence; enqueue `apply` / `run!` only when warnings persist or breach operator thresholds ([monitoring.md](monitoring.md), [decision_flow.md](decision_flow.md#maintenance-cadence)).
65
+
66
+ Services on this cadence should serve hot-path totals and rollups from bucket-keyed snapshot tables, not live aggregates across all partitions ([partition_landscape.md](partition_landscape.md#aggregates-totals-and-snapshots)). Snapshot freshness and drift checks become part of the maintenance story alongside layout audit.
67
+
68
+ Example gate job:
69
+
70
+ ```ruby
71
+ class PartitionAuditJob < ApplicationJob
72
+ queue_as :cleaners
73
+
74
+ def perform
75
+ PartitionGardener::Registry.each do |config|
76
+ result = PartitionGardener.audit(config[:table_name])
77
+ next unless needs_maintenance?(result)
78
+
79
+ PartitionMaintenanceJob.perform_later(table_name: config[:table_name])
80
+ end
81
+ end
82
+
83
+ private
84
+
85
+ def needs_maintenance?(audit_result)
86
+ audit_result.warnings.any? { |warning| threshold_breached?(warning) }
87
+ end
88
+ end
89
+ ```
90
+
91
+ Tune `needs_maintenance?` to your SLOs; do not run `apply` on a single transient spike without a second audit or sustained breach.
92
+
93
+ Cron, Good Job cron, or an external scheduler (Kubernetes CronJob, systemd timer) are all valid. The gem does not prescribe one.
94
+
95
+ ### Statement timeouts
96
+
97
+ Wrap maintenance in the host database timeout helper via global config:
98
+
99
+ ```ruby
100
+ PartitionGardener.configure do |config|
101
+ config.statement_timeout_wrapper = ->(timeout, &block) {
102
+ DatabaseTimeout.statement_timeout(timeout, &block)
103
+ }
104
+ end
105
+ ```
106
+
107
+ Per-table overrides: `statement_timeout: 10.minutes` on the registry entry.
108
+
109
+ ### Error handling
110
+
111
+ Default `continue_on_error: true` finishes other tables when one fails, then raises `PartitionGardener::RunFailed` with all errors. The job should retry or alert on `RunFailed`.
112
+
113
+ To fail fast on the first table error:
114
+
115
+ ```ruby
116
+ PartitionGardener.run!(continue_on_error: false, job_class_name: self.class.name)
117
+ ```
118
+
119
+ Notifier hook — wire to your reporting pipeline:
120
+
121
+ ```ruby
122
+ PartitionGardener.configure do |config|
123
+ config.notifier = ->(message_or_error, context: {}) {
124
+ ActionReporter.notify(message_or_error, context: context)
125
+ }
126
+ end
127
+ ```
128
+
129
+ ### Single-table runs
130
+
131
+ For targeted recovery after an incident:
132
+
133
+ ```ruby
134
+ PartitionGardener.run!(table_name: "audits", job_class_name: self.class.name)
135
+ ```
136
+
137
+ The CLI `apply` command accepts one table by default; pass `--all` for the full registry ([cli.md](cli.md)).
138
+
139
+ ## What the job should not do
140
+
141
+ - Do not reimplement premake, default drain, or tail rebalance in the job — call `run!` only.
142
+ - Do not register the same parent with both partman and gardener unless using `hybrid_layout_only` intentionally ([tooling_split.md](tooling_split.md)).
143
+ - Do not run maintenance inside request cycles.
144
+
145
+ ## Alternatives to a job
146
+
147
+ `bundle exec partition_gardener --rails apply TABLE` — manual ops, CI smoke, one-off recovery.
148
+
149
+ `PartitionGardener.run!(dry_run: true)` — pre-deploy plan review.
150
+
151
+ `PartitionGardener.audit("audits")` — read-only layout audit without writes ([audit_reference.md](audit_reference.md)).
152
+
153
+ ## Rollup and snapshot jobs
154
+
155
+ Dashboard totals and cross-period aggregates should refresh in separate background jobs, bucketed like partitions ([partition_landscape.md](partition_landscape.md#aggregates-totals-and-snapshots)). Materialized views are not a substitute for bucket-keyed snapshot tables on hot paths ([materialized views](partition_landscape.md#materialized-views)). Schedule rollup recompute after maintenance when `rows_moved` is high, or on a fixed cadence with drift checks ([monitoring.md](monitoring.md)).
156
+
157
+ ## Related
158
+
159
+ - [configuration.md](configuration.md) — global and per-table options
160
+ - [cli.md](cli.md) — plan, audit, apply without enqueueing a job
161
+ - [operations.md](operations.md) — incident runbook
data/docs/cli.md ADDED
@@ -0,0 +1,71 @@
1
+ # CLI
2
+
3
+ The `partition_gardener` executable plans, audits, and applies maintenance without enqueueing a background job.
4
+
5
+ ```bash
6
+ bundle exec partition_gardener --rails plan audits
7
+ bundle exec partition_gardener --registry config/partition_garden.json audit --all
8
+ bundle exec partition_gardener --rails apply --confirm audits
9
+ ```
10
+
11
+ ## Commands
12
+
13
+ `plan` — read-only. Outputs the target layout diff as JSON ([plan_report.schema.json](schemas/plan_report.schema.json)).
14
+
15
+ `audit` — read-only. Outputs layout audit warnings as JSON (default rows, gaps, horizon).
16
+
17
+ `apply` — writes data. Runs `PartitionGardener.run!` and prints `RunSummary` JSON. Requires `--confirm`.
18
+
19
+ ## Options
20
+
21
+ `--rails` — load registry from the host Rails environment (`config/environment.rb`).
22
+
23
+ `--registry PATH` — load registry from a JSON file ([configuration.md](configuration.md#json-registry-files)).
24
+
25
+ `--table NAME` — table name (default: first positional argument after the command).
26
+
27
+ `--all` — all registered tables (plan and audit only; apply runs full registry when omitted with `--all`).
28
+
29
+ `--pretty` — pretty-print JSON.
30
+
31
+ `--confirm` — required for `apply` (mutates the database).
32
+
33
+ Registry context is required. Use `--rails`, `--registry`, or pre-register tables in Ruby before invoking the CLI.
34
+
35
+ When using `--registry` without `--rails`, set `DATABASE_URL` so the CLI can connect through `PartitionGardener::PgConnection`.
36
+
37
+ ## Examples
38
+
39
+ Dry-run plan for one table:
40
+
41
+ ```bash
42
+ cd /path/to/rails/app
43
+ bundle exec partition_gardener --rails plan user_workdays --pretty
44
+ ```
45
+
46
+ Audit every table in a JSON registry:
47
+
48
+ ```bash
49
+ bundle exec partition_gardener --registry config/partition_garden.json audit --all --pretty
50
+ ```
51
+
52
+ Apply maintenance to one table:
53
+
54
+ ```bash
55
+ bundle exec partition_gardener --rails apply --confirm audits
56
+ ```
57
+
58
+ Apply all registered gardener-owned tables:
59
+
60
+ ```bash
61
+ bundle exec partition_gardener --rails apply --all
62
+ ```
63
+
64
+ Omitting a table name without `--all` also runs the full registry.
65
+
66
+ ## Related
67
+
68
+ - [configuration.md](configuration.md) — registry format and global config
69
+ - [background_job.md](background_job.md) — when to use a job instead of CLI apply
70
+ - [audit_reference.md](audit_reference.md) — audit and plan output reference
71
+ - [operations.md](operations.md) — when to plan vs apply
@@ -0,0 +1,261 @@
1
+ # Configuration
2
+
3
+ Partition Gardener has two configuration layers:
4
+
5
+ 1. Global — `PartitionGardener.configure` in a Rails initializer (or before CLI use)
6
+ 2. Per-table — entries in `PartitionGardener::Registry` built from `Templates.*` or JSON
7
+
8
+ Pick layout and maintainer first ([decision_flow.md](decision_flow.md), [tooling_split.md](tooling_split.md)). Use this document as the option reference.
9
+
10
+ ## Global configuration
11
+
12
+ ```ruby
13
+ # config/initializers/partition_gardener.rb
14
+ PartitionGardener.configure do |config|
15
+ config.notifier = ->(message_or_error, context: {}) {
16
+ Rails.logger.info(message_or_error)
17
+ }
18
+ config.statement_timeout_wrapper = ->(timeout, &block) {
19
+ DatabaseTimeout.statement_timeout(timeout, &block) # host-app helper
20
+ }
21
+ config.today_resolver = -> { Time.zone.today }
22
+ config.continue_on_error = true
23
+ config.advisory_lock_mode = :transaction
24
+ config.run_record_store = PartitionGardener::SqlRunRecordStore.new
25
+ end
26
+ ```
27
+
28
+ When the gem loads inside Rails, the railtie sets `connection_resolver` and `today_resolver` automatically.
29
+
30
+ `notifier` (default: no-op proc) — receives errors and info messages with a `context:` hash (`table_name`, `job`, `action`, …).
31
+
32
+ `connection_resolver` (default: `ActiveRecord::Base.connection` when Rails is loaded, else `PartitionGardener::PgConnection` from `DATABASE_URL`) — database connection for maintenance SQL.
33
+
34
+ `statement_timeout_wrapper` (default: passthrough) — wraps each table's maintenance in a host timeout (recommended in production).
35
+
36
+ `today_resolver` (default: `Date.today`) — reference date for archive, current, and future window math.
37
+
38
+ `schema_name` (default: `"public"`) — schema searched for parent and child partitions.
39
+
40
+ `continue_on_error` (default: `true`) — when one table fails, continue others; raise `RunFailed` at end with all errors.
41
+
42
+ `advisory_lock_mode` (default: `:transaction` with Active Record, `:session` with standalone `PgConnection`) — `:transaction` uses `pg_try_advisory_xact_lock`; `:session` uses `pg_try_advisory_lock`.
43
+
44
+ `analyze_after_rebalance` (default: `false`) — run `ANALYZE` on parent after tail rebalance (global default).
45
+
46
+ `incremental_rebalance` (default: `true`) — skip unchanged tail partitions when resuming (global default).
47
+
48
+ `run_record_enabled` (default: `true`) — persist phase checkpoints for resume (global default).
49
+
50
+ `run_record_store` (default: `SqlRunRecordStore` when a database connection is available, else in-memory) — where rebalance checkpoints are stored.
51
+
52
+ `strict_maintenance_backend_validation` (default: `false`) — when `true`, raise `MaintenanceBackend::ValidationError` on register if `maintenance_backend` disagrees with `partman.parent_config`; when `false`, notify only.
53
+
54
+ `retention_detach_concurrently` (default: `false`) — detach dropped archive partitions with `CONCURRENTLY` (global default).
55
+
56
+ ### Advisory locks
57
+
58
+ Each table run acquires `pg_advisory_lock(hashtext('partition_gardener'), hashtext(table_name))` (or the transaction-scoped variant).
59
+
60
+ `:transaction` (default with Active Record) — lock is held for one table maintenance transaction. If another worker already holds it, the table is skipped (`lock_not_acquired` in run metrics) and maintenance continues for other tables. Prefer `:session` when a single table run can take many minutes, because `:transaction` keeps one database transaction open for the whole run.
61
+
62
+ `:session` — lock spans the whole table run across multiple transactions; released in an `ensure` block. Default when the connection resolver returns `PgConnection`.
63
+
64
+ Do not run two full `run!` jobs against the same table concurrently. Host apps should also limit concurrent maintenance jobs (see [background_job.md](background_job.md)).
65
+
66
+ ### Run records
67
+
68
+ When a database connection is configured (Active Record or `DATABASE_URL`), `SqlRunRecordStore` creates `partition_gardener_run_records` on first use. Run records let incremental rebalance resume after partial failure. Disable with `config.run_record_enabled = false` or per-table `run_record_enabled: false`.
69
+
70
+ ## Per-table registration
71
+
72
+ Register one config hash per partitioned parent (composite layouts expand to child configs internally).
73
+
74
+ ```ruby
75
+ PartitionGardener::Registry.register_template(
76
+ :sliding_window_monthly,
77
+ table_name: "audits",
78
+ partition_key_column: "created_at::date",
79
+ conflict_key: %w[id created_at],
80
+ active_months: 12,
81
+ move_batch_size: 10_000,
82
+ statement_timeout: 10.minutes,
83
+ retention_months: 24
84
+ )
85
+
86
+ # equivalent two-step form:
87
+ # Registry.register(Templates.sliding_window_monthly(...))
88
+
89
+ # or bulk:
90
+ PartitionGardener::Registry.register_all([config_a, config_b])
91
+ ```
92
+
93
+ `Registry.register_template` calls the matching `Templates` builder, then `Registry.register`. `Registry.register` normalizes the config, validates `maintenance_backend` against `partman.part_config`, and replaces any prior entry with the same `table_name`.
94
+
95
+ ### Template builders
96
+
97
+ `Templates.sliding_window_monthly` — layout `:sliding_window`, bucket `:month`; key option `active_months` (default 12).
98
+
99
+ `Templates.sliding_window_daily` — bucket `:day`; key option `active_days` (default 90).
100
+
101
+ `Templates.sliding_window_weekly` — bucket `:week`; key option `active_weeks` (default 52).
102
+
103
+ `Templates.sliding_window_quarterly` — bucket `:quarter`; key option `active_quarters` (default 8).
104
+
105
+ `Templates.rolling_current_monthly` — layout `:rolling_current`; monthly sliding window without heat splits.
106
+
107
+ `Templates.calendar_year` — layout `:calendar_year`; key option `active_years` (default 2).
108
+
109
+ `Templates.premake_monthly` — layout `:premake_monthly`; key option `premake_months` (default 3).
110
+
111
+ `Templates.integer_window` — layout `:integer_window`; key options `active_id_lo`, `active_id_width`, `current_band_size`, `archive_band_size`.
112
+
113
+ `Templates.hash_branches` — layout `:hash_branches`; key option `hash_modulus`.
114
+
115
+ `Templates.list_split` — layout `:list_split`; key option `branches`. Each branch needs `name` and `value`. Provide a structured `predicate` (`column`, `operator`, `value`), or omit it when `partition_key_column` (or `discriminator_column` on composite parents) matches the equality column and `value` is the partition key.
116
+
117
+ Supported predicate operators: `eq`, `ne`, `is_null`, `is_not_null`. Column names must be simple identifiers; values are quoted through the database connection.
118
+
119
+ ```ruby
120
+ Templates.list_split(
121
+ table_name: "repository_packages",
122
+ partition_key_column: "branch",
123
+ conflict_key: %w[id],
124
+ branches: [
125
+ {name: "cached", value: "cached"},
126
+ {
127
+ name: "workspace",
128
+ value: "workspace",
129
+ predicate: {column: "branch", operator: "eq", value: "workspace"}
130
+ }
131
+ ]
132
+ )
133
+ ```
134
+
135
+ Legacy `where_condition` SQL fragments are still accepted but should be migrated to predicates.
136
+
137
+ `Templates.composite_list_hash` — layout `:composite`, `parent_mode: :list`; LIST parent plus HASH branches.
138
+
139
+ `Templates.composite_list_range` / `Templates.list_range` — LIST parent plus RANGE sliding-window branches.
140
+
141
+ `Templates.composite_range_hash` — RANGE parent plus HASH child tables.
142
+
143
+ `Templates.composite_range_list` — RANGE parent plus LIST child tables.
144
+
145
+ See [partition_landscape.md](partition_landscape.md) for the template catalog and out-of-scope patterns.
146
+
147
+ All templates accept shared options passed through to the registry hash:
148
+
149
+ `partition_key_column` (required) — column or SQL expression used for routing and moves.
150
+
151
+ `conflict_key` (required) — unique key columns for idempotent batch moves (must match a parent unique index). Include the partition key so the index is valid on partitioned parents and maintenance can target one child; see [partition_landscape.md](partition_landscape.md).
152
+
153
+ `move_batch_size` (default: `10_000`, constant `MOVE_BATCH_SIZE`) — rows per move batch.
154
+
155
+ `split_row_threshold` (default: `100_000`) — heat split threshold inside the active window.
156
+
157
+ `statement_timeout` (default: `300` seconds global default) — per-table override for `run!`.
158
+
159
+ `retention_months` (default: none) — drop or detach archive partitions older than N months.
160
+
161
+ `retention_apply` (default: `false`) — when `true`, detach or drop expired archive children; when `false`, notifier logs would-drop partitions only. See [retention.md](retention.md).
162
+
163
+ `retention_keep_table` (default: `false`) — detach but do not drop expired partitions.
164
+
165
+ `retention_detach_concurrently` (default: global default) — per-table override for concurrent detach.
166
+
167
+ `maintenance_backend` (default: `:gardener`) — `:gardener`, `:pg_partman`, or `:hybrid_layout_only`.
168
+
169
+ `incremental_rebalance` (default: global default) — per-table override.
170
+
171
+ `run_record_enabled` (default: global default) — per-table override.
172
+
173
+ `analyze_after_rebalance` (default: global default) — per-table override.
174
+
175
+ Advanced Ruby-only keys (not in JSON import): `partition_name_format`, `partition_definition`, `extract_partition_identifier` procs. Prefer templates so these defaults stay consistent.
176
+
177
+ ### Maintenance backend
178
+
179
+ `:gardener` (default) — gardener runs premake, tail rebalance, default drain, and retention; partman runs nothing.
180
+
181
+ `:pg_partman` — gardener does nothing (table skipped in `run!`); partman runs premake and retention.
182
+
183
+ `:hybrid_layout_only` — gardener runs tail rebalance, default drain, and gap repair only; partman runs premake and simple retention.
184
+
185
+ On register, Gardener warns when partman and gardener both claim the same parent. Set `strict_maintenance_backend_validation: true` in `PartitionGardener.configure` to raise `MaintenanceBackend::ValidationError` instead of notifying. See [tooling_split.md](tooling_split.md).
186
+
187
+ ## JSON registry files
188
+
189
+ Portable table fields are defined in [schemas/partition_garden.schema.json](schemas/partition_garden.schema.json). The schema `$id` is the raw GitHub URL (`https://raw.githubusercontent.com/amkisko/partition_gardener.rb/main/docs/schemas/partition_garden.schema.json`); pin a release tag instead of `main` when you need a fixed version for editor validation. The gem validates against the bundled copy under `docs/schemas/`, not by fetching that URL.
190
+
191
+ `statement_timeout` is stored in seconds in JSON. Ruby registration accepts integer seconds; when Active Support is loaded, duration helpers such as `10.minutes` are also accepted.
192
+
193
+ ```json
194
+ {
195
+ "tables": [
196
+ {
197
+ "table_name": "audits",
198
+ "layout": "sliding_window",
199
+ "partition_key_column": "created_at::date",
200
+ "conflict_key": ["id", "created_at"],
201
+ "active_months": 12,
202
+ "move_batch_size": 10000,
203
+ "statement_timeout": 600,
204
+ "retention_months": 24,
205
+ "maintenance_backend": "gardener"
206
+ }
207
+ ]
208
+ }
209
+ ```
210
+
211
+ Load from Ruby:
212
+
213
+ ```ruby
214
+ PartitionGardener::ConfigDocument.load_registry_file!("config/partition_garden.json")
215
+ ```
216
+
217
+ Load from CLI:
218
+
219
+ ```bash
220
+ bundle exec partition_gardener --registry config/partition_garden.json audit --all
221
+ ```
222
+
223
+ Export from registered Ruby configs:
224
+
225
+ ```ruby
226
+ PartitionGardener::ConfigDocument.export(config_hash)
227
+ PartitionGardener::ConfigDocument.export_all(PartitionGardener::Registry.tables)
228
+ ```
229
+
230
+ JSON import supports layouts: `sliding_window` (with optional `bucket`: day, week, month, quarter), `rolling_current`, `calendar_year`, `premake_monthly`, `integer_window`, `hash_branches`. Composite and `list_split` layouts must be registered in Ruby today.
231
+
232
+ `load_registry_file!` validates each entry against [schemas/partition_garden.schema.json](schemas/partition_garden.schema.json) (required keys, allowed keys, importable layouts) before registration.
233
+
234
+ ### Registry trust boundary
235
+
236
+ Registry entries are operator-controlled configuration, not end-user input.
237
+
238
+ `partition_key_column` is embedded in generated SQL as a trusted expression. List-branch filters should use structured `predicate` hashes (rendered with quoted identifiers and values at registration). Legacy `where_condition` strings are still accepted but bypass that validation. Keep registry files and Ruby registration limited to trusted operators. Do not load registry JSON from untrusted upload paths without a separate review step.
239
+
240
+ ## Programmatic API
241
+
242
+ `PartitionGardener.run!(table_name: "audits")` — run maintenance for one table.
243
+
244
+ `PartitionGardener.run!(job_class_name: "PartitionMaintenanceJob")` — run all registered gardener-owned tables.
245
+
246
+ `PartitionGardener.run!(dry_run: true)` — build plan reports without applying.
247
+
248
+ `PartitionGardener.plan(table_name: "audits")` — plan report for one table.
249
+
250
+ `PartitionGardener.audit("audits")` — read-only layout audit (default rows, gaps, horizon).
251
+
252
+ `run!` skips tables that are not partitioned, use `maintenance_backend: :pg_partman`, or fail to acquire the advisory lock. Returns `RunSummary`; raises `RunFailed` when `continue_on_error` is false or after aggregating errors at the end.
253
+
254
+ ## Related
255
+
256
+ - [background_job.md](background_job.md) — schedule `run!` from a host job
257
+ - [cli.md](cli.md) — plan, audit, apply from the shell
258
+ - [decision_flow.md](decision_flow.md) — when and how to partition
259
+ - [operations.md](operations.md) — runbook and RunSummary
260
+ - [audit_reference.md](audit_reference.md) — audit and plan fields
261
+ - [retention.md](retention.md) — retention options in production
data/docs/cutover.md ADDED
@@ -0,0 +1,180 @@
1
+ # Cutover and migration playbook
2
+
3
+ Move a live non-partitioned table to declarative partitioning with minimal downtime. Creation DDL uses pg_party or SQL ([pg_party_recipe.md](pg_party_recipe.md)); runtime layout uses Partition Gardener after switch.
4
+
5
+ ## Prerequisites
6
+
7
+ - Composite unique index including the partition key, matching `conflict_key` ([partition_landscape.md](partition_landscape.md)).
8
+ - Registry entry chosen (typically `sliding_window_monthly`; see [decision_flow.md](decision_flow.md)).
9
+ - Maintenance job scheduled or CLI access for the first `run!` after cutover.
10
+ - Staging environment with production-shaped data and the same registry.
11
+
12
+ ## Phase 1: Creation (no traffic switch)
13
+
14
+ 1. Create shadow parent `p_events` (or your naming convention) with `PARTITION BY RANGE` on the partition key.
15
+ 2. Create `p_events_default` and minimal premake (current month and next month).
16
+ 3. Add indexes on the parent (unique on `conflict_key` columns).
17
+ 4. Do not rename production `events` yet.
18
+
19
+ See [pg_party_recipe.md](pg_party_recipe.md) for a minimal migration outline.
20
+
21
+ ## Phase 2: Backfill
22
+
23
+ 1. Copy historical rows into `p_events` in batches keyed by partition window (month or bucket).
24
+ 2. Throttle batch size to protect production I/O; use the same `conflict_key` for idempotent upserts.
25
+ 3. Compare counts per month between source and shadow (`compare_table_counts` in hot-switch concern).
26
+ 4. Run application read-only queries against shadow in staging to validate pruning and scopes.
27
+
28
+ Backfill window for hot-switch delta sync defaults to roughly three months before and after `today` at switch time; extend backfill if older rows must exist before switch.
29
+
30
+ For cross-database copies (production → staging, or a second cluster during migration), [pgsync](https://github.com/ankane/pgsync) can move filtered subsets with `where` clauses and `--in-batches` on append-only history. Gardener does not run pgsync; schedule it outside `run!` windows so loads do not fight nightly layout work.
31
+
32
+ ## Phase 3: Hot switch
33
+
34
+ Use `PartitionGardener::Migration::HotSwitchConcern` in a migration:
35
+
36
+ ```ruby
37
+ class PartitionEventsHotSwitch < ActiveRecord::Migration[8.0]
38
+ include PartitionGardener::Migration::HotSwitchConcern
39
+
40
+ HOT_SWITCH_CONFIG = {
41
+ current_table: "events",
42
+ partitioned_table: "p_events",
43
+ partition_key_column: "occurred_on",
44
+ conflict_key: %w[id occurred_on],
45
+ swap_lock_timeout: "5s", # optional; nil disables; default is 5s when omitted
46
+ partition_config: PartitionGardener::Registry.hot_switch_partition_config("events")
47
+ }.freeze
48
+
49
+ def up
50
+ add_write_block_trigger("events") # optional: quiesce writes during final sync
51
+ wait_for_active_transactions("events")
52
+ sync_delta_data
53
+ analyze_shadow_partitions!
54
+ ensure_future_partitions_exist(months_ahead: 1)
55
+ hot_switch_tables
56
+ sync_delta_data(swapped: true) # optional: catch rows on events_old if write-block was skipped
57
+ end
58
+ end
59
+ ```
60
+
61
+ Switch sequence (inside `hot_switch_tables`):
62
+
63
+ 1. `SET LOCAL lock_timeout` when `swap_lock_timeout` is set (default `5s`).
64
+ 2. Rename live `events` → `events_old`.
65
+ 3. Rename `p_events` → `events`.
66
+ 4. Rename child partitions to match new parent prefix.
67
+ 5. Repoint serial/identity sequences to the live table (`ALTER SEQUENCE ... OWNED BY`).
68
+ 6. Remove write-block trigger from `events_old`.
69
+
70
+ `months_ahead: 1` at switch is enough; gardener extends the sliding window nightly.
71
+
72
+ `analyze_shadow_partitions!` runs `ANALYZE` on each shadow child and parent before swap. `sync_delta_data` accepts `source_table`, `target_table`, `swapped: true` (post-swap catch-up from `events_old`), `sleep_seconds` between batches, and `batch_size`.
73
+
74
+ ## Cutover comparison: pgslice
75
+
76
+ [pgslice](https://github.com/ankane/pgslice) solves the same cutover problem with a CLI: `prep` → `add_partitions` → `fill` → `analyze` → `swap`, then optional `fill --swapped` for rows that landed on the retired table after the first fill. Gardener's `HotSwitchConcern` covers the swap phase in a Rails migration, plus live-delta sync and optional write quiesce.
77
+
78
+ ### Step comparison
79
+
80
+ Shadow parent — pgslice uses `visits_intermediate` (`LIKE` + `PARTITION BY RANGE`); Gardener uses `p_events` (pg_party or SQL creation).
81
+
82
+ Premake children — pgslice `add_partitions --intermediate`; Gardener `ensure_future_partitions_exist`.
83
+
84
+ Copy rows — pgslice `fill` in id batches within partition bounds; Gardener Phase 2 backfill plus `sync_delta_data` (UPSERT, partition-key window).
85
+
86
+ Statistics — pgslice `analyze` on each child and parent; Gardener `analyze_shadow_partitions!` before swap and `analyze_after_rebalance` on first `run!`.
87
+
88
+ Atomic rename — pgslice renames `visits` → `visits_retired`, intermediate → `visits`; Gardener renames `events` → `events_old`, `p_events` → `events`, and children.
89
+
90
+ Catch-up after rename — pgslice `fill --swapped` from retired to live; Gardener `sync_delta_data(swapped: true)` or write-block trigger before swap.
91
+
92
+ Rollback drill — pgslice `unswap`; Gardener `hot_unswitch_tables`.
93
+
94
+ The rename swap uses the same low-downtime pattern: metadata (table name) flips in one transaction; the app keeps querying `events` without code changes.
95
+
96
+ ### Gardener extras for live cutover
97
+
98
+ Delta sync before switch — `sync_delta_data` upserts only rows in a sliding date window that are missing or stale on the shadow table (`updated_at` comparison). pgslice's first `fill` is id-batch oriented and assumes a numeric primary key; catch-up after rename is a separate `fill --swapped` pass.
99
+
100
+ Write quiesce — optional `add_write_block_trigger` plus `wait_for_active_transactions` narrow the race between last sync and rename.
101
+
102
+ Registry handoff — after swap, `Registry` plus nightly `run!` own runtime maintenance. pgslice stops at swap and expects a cron `add_partitions` you must later disable ([tooling_split.md](tooling_split.md)).
103
+
104
+ ### Hot-switch config reference
105
+
106
+ `swap_lock_timeout` — default `"5s"`. Sets `SET LOCAL lock_timeout` in swap/unswitch transaction; `nil` disables.
107
+
108
+ `sync_batch_size` — default `PartitionGardener::MOVE_BATCH_SIZE`. Rows per `sync_delta_data` batch.
109
+
110
+ `sync_stale_column` — default `"updated_at"`. Stale-row detection for upserts.
111
+
112
+ Sequence repointing uses `pg_get_serial_sequence` per column; custom sequence names are supported.
113
+
114
+ ### Using pgslice for creation, Gardener for switch
115
+
116
+ Valid hybrid when you do not use pg_party:
117
+
118
+ 1. `pgslice prep` / `add_partitions` / `fill` / `analyze` on the shadow tree.
119
+ 2. Register the table in Gardener before switch.
120
+ 3. Replace `pgslice swap` with a migration that calls `analyze_shadow_partitions!`, `sync_delta_data`, `ensure_future_partitions_exist`, and `hot_switch_tables`.
121
+ 4. Disable pgslice `add_partitions` cron; run Gardener nightly.
122
+
123
+ Do not run pgslice premake and Gardener `run!` on the same parent without `maintenance_backend` coordination.
124
+
125
+ ## Phase 4: After cutover
126
+
127
+ 1. Register `events` in `PartitionGardener::Registry` if not already loaded.
128
+ 2. `bundle exec partition_gardener --rails audit events` — target default row count trending to zero.
129
+ 3. `bundle exec partition_gardener --rails plan events` — review first layout diff.
130
+ 4. `bundle exec partition_gardener --rails apply --confirm events` or wait for nightly job.
131
+ 5. Recompute bucket snapshots for dashboard totals if you use rollup tables ([partition_landscape.md](partition_landscape.md#aggregates-totals-and-snapshots)).
132
+ 6. Drop `events_old` only after retention policy, legal review, and verified row counts.
133
+
134
+ ## Rollback
135
+
136
+ Before dropping `events_old`:
137
+
138
+ 1. Stop maintenance job for `events` or set maintenance window.
139
+ 2. Call `hot_unswitch_tables` in a migration, or reverse renames manually: `events` → `p_events`, `events_old` → `events`.
140
+ 3. Re-point application if any code assumed partitioned shape.
141
+ 4. Document registry removal to avoid gardener acting on the wrong parent.
142
+
143
+ Rollback after `events_old` is dropped requires restore from backup; treat drop as irreversible.
144
+
145
+ ## Template upgrades
146
+
147
+ ### premake_monthly → sliding_window_monthly
148
+
149
+ 1. Register new template; `plan` to preview tail and default drain.
150
+ 2. Run `apply` during low traffic; expect `rows_moved` from default and tail reshape.
151
+ 3. Remove legacy cron premake jobs.
152
+
153
+ ### Changing bucket (monthly → weekly)
154
+
155
+ Treat as a new partitioning scheme: new parent or full reload into a new tree. Gardener does not rewrite archive naming in place. Plan a new shadow table, backfill, and hot switch.
156
+
157
+ ### Composite trees
158
+
159
+ Native PostgreSQL sub-partition DDL is your migration responsibility. Register each gardener-maintained node separately ([partition_landscape.md](partition_landscape.md#composite-trees)). Creation migration attaches the full tree; gardener plans each registered parent.
160
+
161
+ ## Cutover checklist
162
+
163
+ - [ ] `conflict_key` matches parent unique index
164
+ - [ ] Shadow backfill complete; counts match per bucket
165
+ - [ ] `ensure_future_partitions_exist(months_ahead: 1)` succeeds
166
+ - [ ] Hot switch migration applied
167
+ - [ ] Sequences owned by live table, not `events_old` (`hot_switch_tables` repoints serial columns; verify after switch)
168
+ - [ ] Registry loaded in all app processes
169
+ - [ ] First `audit`: default rows acceptable or draining
170
+ - [ ] First `apply` or nightly job succeeded
171
+ - [ ] Application scopes and UI use partition key on hot paths
172
+ - [ ] Snapshot rollups invalidated or recomputed for affected buckets
173
+ - [ ] `events_old` drop scheduled with backup verification
174
+
175
+ ## Related
176
+
177
+ - [pg_party_recipe.md](pg_party_recipe.md) — creation DDL sketch
178
+ - [operations.md](operations.md) — post-cutover incidents
179
+ - [host_testing.md](host_testing.md) — CI and staging for host apps
180
+ - [tooling_split.md](tooling_split.md) — do not double-maintain with partman