partition_gardener 0.3.5 → 0.3.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +21 -0
  3. data/README.md +2 -2
  4. data/docs/THIRD_PARTY_LICENSE_MANIFEST.tsv +18 -19
  5. data/docs/application_contract.md +29 -3
  6. data/docs/audit_reference.md +1 -1
  7. data/docs/changelogs/20260709120900_partition-engines-guide.md +1 -1
  8. data/docs/changelogs/20260813142300_rebalance-before-archive-attach.md +19 -0
  9. data/docs/changelogs/20260813144516_relock-appraisal-advisory-json.md +21 -0
  10. data/docs/changelogs/20260902114500_align-child-columns-before-attach.md +19 -0
  11. data/docs/changelogs/20260902124500_cut-patch-release-0-3-4.md +24 -0
  12. data/docs/changelogs/20260909143500_occupied-window-current-origin.md +19 -0
  13. data/docs/changelogs/20260909154900_tail-slot-rfc.md +21 -0
  14. data/docs/changelogs/20260909162554_high-end-future-occupancy.md +21 -0
  15. data/docs/changelogs/20260909164000_shipped-contract-rfcs.md +21 -0
  16. data/docs/changelogs/20260909165500_cut-patch-release-0-3-5.md +24 -0
  17. data/docs/changelogs/20260916144600_public-identifiers-contract.md +23 -0
  18. data/docs/changelogs/20261005135700_checkpoint-table-name.md +23 -0
  19. data/docs/changelogs/20261005141621_cut-patch-release-0-3-6.md +23 -0
  20. data/docs/changelogs/20261021220000_cut-patch-release-0-3-7.md +24 -0
  21. data/docs/configuration.md +3 -1
  22. data/docs/dependencies/20260813144516_json-cve-2026-71847.md +27 -0
  23. data/docs/issues/20260909143500_occupied-window-current-origin.md +25 -0
  24. data/docs/issues/20260909164000_shipped-contract-rfcs.md +25 -0
  25. data/docs/issues/20260916113300_opaque-composite-public-ids.md +55 -0
  26. data/docs/issues/20260917123253_ci-drop-automatic-dependency-audit.md +28 -0
  27. data/docs/operations.md +1 -1
  28. data/docs/partition_landscape.md +6 -4
  29. data/docs/reviews/20261021220000_engineering-audit-0-3-7.md +30 -0
  30. data/docs/reviews/20261028075216_engineering-audit-release-preparation.md +35 -0
  31. data/docs/reviews/20261029150855_engineering-audit-release-preparation.md +43 -0
  32. data/docs/schemas/partition_garden.schema.json +16 -0
  33. data/lib/partition_gardener/archive_retention.rb +7 -3
  34. data/lib/partition_gardener/config_document.rb +4 -0
  35. data/lib/partition_gardener/configuration.rb +9 -1
  36. data/lib/partition_gardener/date_calendar.rb +2 -5
  37. data/lib/partition_gardener/gap_detection.rb +1 -1
  38. data/lib/partition_gardener/migration/hot_switch_concern.rb +24 -15
  39. data/lib/partition_gardener/plan_applier.rb +17 -3
  40. data/lib/partition_gardener/sql_run_record_store.rb +40 -5
  41. data/lib/partition_gardener/version.rb +1 -1
  42. data/sig/partition_gardener.rbs +2 -2
  43. metadata +22 -15
@@ -0,0 +1,55 @@
1
+ # Opaque composite public identifiers
2
+
3
+ ## Participants
4
+
5
+ Andrei Makarov (@amkisko)
6
+
7
+ ## Decisions
8
+
9
+ Treat database uniqueness, public identifier, routing hint, and authorization as four separate contracts. PostgreSQL uniqueness on a partitioned parent is composite because the partition key must sit in every unique index. That does not by itself decide what a person sees in a URL or API.
10
+
11
+ When a single sequence or UUIDv7 keeps id unique across children, keep that logical id as the application identifier. Use self.primary_key = :id so to_param and JSON do not become Rails underscore composites such as 42_100. Still attach a routing hint on point lookups so SELECT can prune. Do not invent that hint from current month.
12
+
13
+ When id is unique only inside a child, the public identifier for point lookups must carry every uniqueness column. Prefer one opaque path segment over Rails extract_value underscore form or a separate query parameter the caller can edit independently.
14
+
15
+ If that segment is encoded, use unpadded base64url from RFC 4648 section 5, not standard base64. Standard base64 uses plus, slash, and padding equals, which are not URL-safe in a path. Encoding is packaging. Anyone can decode and re-encode another date or tenant. Tamper resistance needs an HMAC over a versioned payload, or a server-side mapping table. Encoding is not authorization. Lookups still go through an ownership set.
16
+
17
+ After decode, query with both columns and fail closed. Do not fall back to id-only scan. A bad or unsigned token is 404, not a bounded retry across children.
18
+
19
+ Do not hide browse context. List screens still show period, account, or branch in product language. Tenant and list keys for hot paths come from session or parent context, not from a tunable URL field. Point-lookup tokens may be opaque. Pagination cursors already belong in the same family as opaque tokens.
20
+
21
+ Do not ship an encoder in this gem until a second in-repo caller needs it. Host applications own secrets, token version, and decode. Gardener docs should name the contract.
22
+
23
+ Rails already covers the packaging the first pass sketched. Prefer those APIs over a homemade base64 string.
24
+
25
+ signed_id (Rails 7+) is an HMAC token over record.id, SHA256, JSON, url_safe: true. The docs say the payload is encoded not encrypted. find_signed fails closed on a bad signature. That is the closest out-of-box match to one opaque URL-safe path segment. It only carries both uniqueness columns when primary_key is the composite array. Scalar self.primary_key = :id plus query_constraints still signs and finds by id alone, so SELECT can scan every child. query_constraints never feeds SELECT.
26
+
27
+ to_param plus params.extract_value is the Rails CPK URL form. Default delimiter is underscore (4_2). Callers can edit either half. Dates and other values that contain underscore split wrong unless param_delimiter is changed. Use this for private admin paths, not for a public identifier that must not be tunable.
28
+
29
+ generates_token_for can embed extra JSON (password salt, or occurred_on) and re-check it after fetch. Lookup still yields only the primary key into find_by or find. Extra JSON is not a WHERE predicate and is plaintext in the token. CPK find_by_token_for was fixed in 2024 (Rails 7.2/8.0). find_signed CPK wrap landed later (Rails 8.1 / 2026-04). Gardener's development floor is ActiveRecord 7.1, so hosts on 7.1 cannot treat signed CPK find as solved.
30
+
31
+ has_secure_token stores a random Base58 string (URL-safe by alphabet). It is a mapping column, not an encoding of composite values. PostgreSQL still forbids a unique index on token alone on a partitioned parent unless the partition key is in that index. Lookup by token without the partition key then cannot prune, or the unique index cannot exist. Put the mapping on an unpartitioned table, or keep a globally unique logical id on the fact row.
32
+
33
+ GlobalID locates with Model.find(model_id). Composite keys become gid://app/Event/123/2026-03-15 (slash, CGI.escape, max 20 segments). Unsigned. to_param is unpadded base64url of that URI, still decodable. SignedGlobalID HMAC-wraps the same URI. Locator find prunes only when primary_key is the composite.
34
+
35
+ ActiveSupport::MessageVerifier is the primitive under signed_id, token_for, and SignedGlobalID. Default MessageVerifier is not URL-safe (plus, slash, padding). signed_id and GlobalID::Verifier opt into url_safe. MessageEncryptor exists; do not encrypt identifiers. CurrentAttributes can hold session tenant for list filters; that is not a public id.
36
+
37
+ ## Effects
38
+
39
+ Added Public identifiers to docs/application_contract.md. Pointers from docs/partition_landscape.md Rails application contract, routing hints, and UI surfaces. README application_contract blurb names public identifiers. CHANGELOG Unreleased and usr/docs/changelogs/20260916144600_public-identifiers-contract.md record the doc contract.
40
+
41
+ ## Next
42
+
43
+ Host docs only. No encoder in the gem. No RFC unless a later change makes public identifiers a user-facing gem contract.
44
+
45
+ ## Source
46
+
47
+ Triggered by a review question: should composite ids for users be a single base64 string of composite values so callers cannot easily tune them and so the value stays URL-safe.
48
+
49
+ Later pass accepted the proposal as host documentation. Vendor product names stay out of the host contract; they remain a Source citation only.
50
+
51
+ Existing repo text: docs/partition_landscape.md Opaque tokens may encode (id, partition_key); Rails query_constraints; UI and product surfaces require visible period or tenant for browse. docs/application_contract.md query_constraints when logical id is not globally unique.
52
+
53
+ Materialized in docs/application_contract.md Public identifiers, docs/partition_landscape.md pointers, README application_contract blurb.
54
+
55
+ External: GitLab Database table partitioning, Rails Composite Primary Keys guide to_param 4_2 and extract_value, ActiveRecord::SignedId (url_safe MessageVerifier, find_by primary_key => [id] on main), ActiveRecord::TokenFor payload_for [id, block.as_json], has_secure_token SecureRandom.base58, URI::GID COMPOSITE_MODEL_ID_DELIMITER slash, GlobalID to_param unpadded base64url, Rails PRs 52798 and 57245, RFC 4648 section 5 base64url, GraphQL Relay global object identification, OWASP Insecure Direct Object Reference Prevention Cheat Sheet.
@@ -0,0 +1,28 @@
1
+ CI quality failed on a trailing blank line in spec/support/junit_formatter.rb. Test CI also ran bundler-audit or osv-scanner, which fail whenever an advisory lands even when a separate dependency audit already exists.
2
+
3
+ ## Participants
4
+
5
+ Andrei Makarov (@amkisko)
6
+
7
+ ## Decisions
8
+
9
+ - Drop bundler-audit and security jobs from test.yml so tests no longer wait on advisory freshness.
10
+ - Keep dependency-audit.yml as workflow_dispatch only. Do not run it on push, pull request, or schedule.
11
+ - Disable osv-scanner in Trunk Check.
12
+ - Disable markdownlint MD013 in .markdownlint.yaml. Keep AGENTS.md in Trunk so other markdownlint rules still run.
13
+ - Keep lockfile and spec fixes that were already blocking quality or tests.
14
+
15
+ ## Effects
16
+
17
+ - Test CI runs lint and specs without an advisory gate.
18
+ - Trunk Check no longer treats dependency CVEs as a required check.
19
+ - Manual dependency-audit remains available on workflow_dispatch.
20
+
21
+ ## Next
22
+
23
+ - Run dependency audit on demand when a release or a known advisory needs it.
24
+ - Do not reintroduce advisory scanners into test.yml.
25
+
26
+ ## Source
27
+
28
+ - GitHub Actions test and Trunk Check failures on 2026-09-17
data/docs/operations.md CHANGED
@@ -83,7 +83,7 @@ table is not a partitioned table — registry typo or wrong connection. Action:
83
83
 
84
84
  ## Failed or partial runs
85
85
 
86
- Gardener persists checkpoints in `partition_gardener_run_records` when `run_record_enabled` is true (default).
86
+ Gardener persists checkpoints in `partition_gardener_checkpoints` when `run_record_enabled` is true (default). Override the table with `run_record_table_name`.
87
87
 
88
88
  1. Read `errors` and notifier context (`table_name`, `action`).
89
89
  2. `audit TABLE` — confirm default count and gaps unchanged or worse.
@@ -2,7 +2,7 @@
2
2
 
3
3
  Partition Gardener targets PostgreSQL native declarative partitioning with gardener-owned maintenance plans. This page maps industry patterns to what the gem implements, documents experimental layouts, and lists what belongs outside the gem.
4
4
 
5
- Canonical home for routing (layers, recovery ladder, hints), pruning, UI scoping, and aggregate snapshots. Host query and write obligations live in [application_contract.md](application_contract.md). Cross-engine portability lives in [partition_engines.md](partition_engines.md).
5
+ Canonical home for routing (layers, recovery ladder, hints), pruning, UI scoping, and aggregate snapshots. Host query, write, and public identifier obligations live in [application_contract.md](application_contract.md). Cross-engine portability lives in [partition_engines.md](partition_engines.md).
6
6
 
7
7
  ## Implemented templates
8
8
 
@@ -113,6 +113,8 @@ end
113
113
 
114
114
  `query_constraints` (Rails 7.1+) adds listed columns to `UPDATE` and `DELETE` `WHERE` clauses so writes can prune. It does not add the partition key to `SELECT` lookups: `Event.find(id)` may still scan all partitions unless `id` is globally unique and the planner can prove a single child.
115
115
 
116
+ Public URLs and API ids are a separate contract from `query_constraints`. See [application_contract.md](application_contract.md#public-identifiers).
117
+
116
118
  For reads, encode the contract in scopes used by hot paths:
117
119
 
118
120
  ```ruby
@@ -150,7 +152,7 @@ Recovery ladder (prefer earlier steps):
150
152
  #### Exact key (preferred)
151
153
 
152
154
  - Denormalize the parent business date (or tenant/branch) onto the child at insert time; filter and join on the child column.
153
- - Store the partition key next to the logical id in API cursors, job args, outbox payloads, and admin deep links. Opaque tokens may encode `(id, partition_key)` so clients never send id-only.
155
+ - Store the partition key next to the logical id in API cursors, job args, outbox payloads, and admin deep links. Opaque tokens may encode `(id, partition_key)` so clients never send id-only ([application_contract.md](application_contract.md#public-identifiers)).
154
156
  - Use a generated column when the key is a stable expression of an existing column the planner can see (`created_at::date` registered and filtered the same way).
155
157
 
156
158
  #### Orientation / range from a related record
@@ -218,7 +220,7 @@ Registry `conflict_key` must match a parent unique index and should include the
218
220
 
219
221
  ### UI and product surfaces
220
222
 
221
- Partition boundaries are an application contract, not only a database detail. Screens and APIs should make the same separation key visible to people so routine work stays inside one prune-friendly slice. User-facing copy should describe periods, accounts, or branches in product language, not partitions, shards, or child tables.
223
+ Partition boundaries are an application contract, not only a database detail. Screens and APIs should make the same separation key visible to people so routine work stays inside one prune-friendly slice. User-facing copy should describe periods, accounts, or branches in product language, not partitions, shards, or child tables. Point-lookup URLs may use an opaque token; list and filter screens still show the period or tenant ([application_contract.md](application_contract.md#public-identifiers)).
222
224
 
223
225
  Time-partitioned data (monthly sliding window):
224
226
 
@@ -410,7 +412,7 @@ Operations and migration docs in this repository:
410
412
 
411
413
  - [operations.md](operations.md) — runbook
412
414
  - [cutover.md](cutover.md) — hot-switch playbook
413
- - [application_contract.md](application_contract.md) — host app behavior
415
+ - [application_contract.md](application_contract.md) — host app behavior, public identifiers
414
416
  - [monitoring.md](monitoring.md) — metrics and alerts
415
417
  - [retention.md](retention.md) — archive detach and drop
416
418
  - [audit_reference.md](audit_reference.md) — audit and plan catalog
@@ -0,0 +1,30 @@
1
+ # Engineering audit and 0.3.7 fixes
2
+
3
+ ## Participants
4
+
5
+ Andrei Makarov (@amkisko)
6
+
7
+ ## Decisions
8
+
9
+ - Run a multi-model engineering audit on `partition_gardener` (correctness, tests, hot-path, contracts).
10
+ - Fact-check findings in source before shipping.
11
+ - Ship high-confidence correctness and contract fixes as patch `0.3.7`.
12
+ - Defer structural refactors (PlanApplier phase machine collapse, HotSwitchConcern split, strategy mixin extraction, default statement_timeout wrapper) to later work.
13
+
14
+ ## Effects
15
+
16
+ - Confirmed and fixed: resume skips detach while phase is `detach`; hot-switch child rename skip on `p_` prefix; serial sequence default left on `_old`; unbounded `sync_delta_data` when stale column missing; `CREATE IF NOT EXISTS … PARTITION OF` no-op on detached name; lexicographic integer gap sort; leap-day `add_years`; retention notify says drop when table kept; JSON registry omits integer window keys.
17
+ - Unit specs: 151 examples, 0 failures (`bundle exec rspec` on non-integration paths from `spec/spec_paths.txt`).
18
+ - Integration resume examples extended for row-count preservation and started-but-not-detached phase; not run in this session (needs `INTEGRATION=1` and Postgres).
19
+
20
+ ## Next
21
+
22
+ - Run `INTEGRATION=1` resume and hot-switch specs before cutting the gem.
23
+ - Run `ruby usr/bin/release.rb` when the working tree is clean and ready to push to RubyGems.
24
+ - Follow-up: session advisory lock default for long copies; statement_timeout wrapper; RFC 0002 status; dead `SCHEMA_PATH` wiring or removal.
25
+
26
+ ## Source
27
+
28
+ - Audit agents: correctness, tests/coverage, hot-path/resource, org/contracts.
29
+ - Branch: `patch/engineering-audit-0-3-7`
30
+ - Related changelog: `docs/changelogs/20261021220000_cut-patch-release-0-3-7.md`
@@ -0,0 +1,35 @@
1
+ # Engineering audit and release preparation
2
+
3
+ ## Decisions
4
+
5
+ - Continue on main while preserving the pre-existing uncommitted changes. Do not commit, tag, push, or publish.
6
+ - Retain release version 0.3.7. Reconcile root and appraisal PATH metadata without updating third-party dependency versions.
7
+ - Review correctness, executable coverage, workflow triggers and permissions, package boundaries, resource cost, and trace/identity boundaries. Re-audit the changed paths after corrections.
8
+
9
+ ## Effects
10
+
11
+ - Correct the resume integration fixture's initial expectation: the source rows initially belong to the attached current partition, and the hot monthly partition exists after rebalance.
12
+ - Restore the required lint gate by combining adjacent boundary stubs. Verify all integration examples against an isolated PostgreSQL server, including the existing rebalance and hot-switch changes.
13
+ - Synchronize the own-gem PATH version in the root and appraisal lockfiles to 0.3.7, enable pull-request tests, and restrict workflow permissions to contents: read.
14
+ - Fix the release confirmation guard in usr/bin/release.rb so EOF refuses publication. An explicit Enter still selects the displayed default. The process-boundary regression test executes the release script with command execution replaced at the operating-system boundary; no publication command runs against a real service. The shared regression was observed first in Favorites: 2 examples, 1 failure before correction and 2 examples, 0 failures after correction. The release specs pass in every repository.
15
+ - Final serial JSON report: 188 examples, 0 failures with INTEGRATION=1 and PostgreSQL; no pending integration examples.
16
+ - Required project checks passed: DATABASE_URL=postgres://postgres@127.0.0.1:55432/partition_gardener_test make test. Environment: Ruby 4.0.2, BUNDLE_PATH=vendor/bundle; the installed native extensions require that Ruby build.
17
+ - Build the gem, verify it with Gem::Package#verify, and check that the changelog heading matches the gem version. Final package measurements are recorded in the session output; this gem includes the review note itself.
18
+ - Cold process measurement on macOS arm64 with /usr/bin/time -l and bundle exec ruby -Ilib: 0.35 CPU seconds, 0.7 elapsed seconds, 31817728 peak RSS bytes. Favorites loads Rails before the engine. Local smoke ceilings: 1 MiB compressed gem, 128 MiB RSS, 2 CPU seconds. All measured values fit those ceilings. CPU seconds are the energy proxy; electricity use was not measured.
19
+ - Parse every workflow YAML, verify lockfile own-gem versions, and run git diff --check. All passed. All six assigned repositories remain on main.
20
+
21
+ ## Next
22
+
23
+ - No confirmed correction remains in the reviewed changes. Hosted GitHub Actions and the complete supported Ruby/framework matrix were not executed locally.
24
+ - Scope limits: production traffic, production database size and query plans, external service retention, and co-located resource interference were not measured. The local byte/RSS/CPU results do not establish those properties.
25
+ - Trace review follows local file/log and protocol emissions. No live customer identities or credentials were used. Data retention and deletion in embedding applications remain host contracts.
26
+ - Learned-system training/evaluation and cloud deployment modes do not apply to these library trees. Native memory-boundary mode applies to Polyrun's extension; browser presentation applies to Favorites; HTTP/identity and message companions apply only to repositories that expose those boundaries. Libraries have no independent production health endpoint to assess.
27
+
28
+ ## Source
29
+
30
+ - AGENTS.md
31
+ - .agents/skills/engineering-audit/SKILL.md
32
+ - .agents/skills/changelog-update/SKILL.md
33
+ - CHANGELOG.md
34
+ - Makefile
35
+ - spec/README.md
@@ -0,0 +1,43 @@
1
+ # Engineering audit and release preparation
2
+
3
+ ## Participants
4
+
5
+ Andrei Makarov (@amkisko)
6
+
7
+ ## Decisions
8
+
9
+ - Continue on main while preserving uncommitted work. Do not commit, tag, push, or publish.
10
+ - Keep release version 0.3.7. Set the CHANGELOG heading date to 2026-10-29. RubyGems still lists 0.3.6 as the latest published version.
11
+ - Re-check correctness, executable coverage, workflow triggers and permissions, package boundaries, resource cost, and database-boundary behavior after the prior release-prep pass.
12
+
13
+ ## Effects
14
+
15
+ - Enable `pull_request` on the test workflow, keep token permissions at `contents: read`, and cancel superseded runs.
16
+ - Set the `CHANGELOG.md` heading for 0.3.7 to 2026-10-29 and align the cut note under `docs/changelogs/`.
17
+ - Final serial JSON report: 188 examples, 0 failures with `INTEGRATION=1` and PostgreSQL on port 5432; no pending examples. Parallel polyrun with five workers: exit 0.
18
+ - Required project checks passed: rubocop (116 files, no offenses), rbs validate, `INTEGRATION=1` polyrun parallel-rspec. Environment: Ruby 4.0.2, `BUNDLE_PATH=vendor/bundle`.
19
+ - `release_spec`: 2 examples, 0 failures (EOF refuses gem push; Enter accepts the displayed default with system execution replaced).
20
+ - Build the gem, verify it with `Gem::Package#verify`, and confirm the changelog heading matches version 0.3.7. Compressed package: 117760 bytes; source payload: 495104 bytes across 104 packaged paths. These are compressed versus uncompressed sizes, not interchangeable quantities.
21
+ - Cold process measurement on macOS arm64 with `/usr/bin/time -p` and `bundle exec ruby -Ilib`: 0.17 CPU seconds, 0.26 elapsed seconds. Peak RSS via `/usr/bin/time -l` was unavailable in this session (`sysctl kern.clockrate` refused). Prior release-prep peak RSS on the same require path was 31817728 bytes. Local smoke ceilings: 1 MiB compressed gem, 128 MiB RSS, 2 CPU seconds. Measured gem size and CPU fit those ceilings. CPU seconds are the energy proxy; electricity use was not measured.
22
+ - Parse every workflow YAML, verify lockfile own-gem versions at 0.3.7, and run `git diff --check`. All passed.
23
+
24
+ ## Next
25
+
26
+ - Commit, tag 0.3.7, push, and publish remain operator steps; this session did not run them. When the tree is clean, `make release` runs gem push, tag, and GitHub release.
27
+ - Hosted GitHub Actions and the complete supported Ruby/framework matrix were not executed locally beyond the default Gemfile path used by `make test`.
28
+ - Re-measure peak RSS with `/usr/bin/time -l` outside the restricted session if a fresh number is required before publish.
29
+ - Scope limits: production traffic, production database size and query plans, external service retention, and co-located resource interference were not measured. The local byte/CPU results do not establish those properties.
30
+ - Trace review follows local file/log and protocol emissions. No live customer identities or credentials were used. Data retention and deletion in embedding applications remain host contracts.
31
+ - Learned-system training/evaluation and cloud deployment modes do not apply to this library tree. Database-boundary mode applies to PostgreSQL maintenance DDL and checkpoint resume. The library has no independent production health endpoint to assess.
32
+
33
+ ## Source
34
+
35
+ - AGENTS.md
36
+ - .agents/skills/engineering-audit/SKILL.md
37
+ - .agents/skills/changelog-update/SKILL.md
38
+ - CHANGELOG.md
39
+ - Makefile
40
+ - spec/README.md
41
+ - docs/reviews/20261028075216_engineering-audit-release-preparation.md
42
+ - docs/reviews/20261021220000_engineering-audit-0-3-7.md
43
+ - docs/changelogs/20261021220000_cut-patch-release-0-3-7.md
@@ -60,6 +60,22 @@
60
60
  "minimum": 1,
61
61
  "maximum": 10
62
62
  },
63
+ "active_id_lo": {
64
+ "type": "integer",
65
+ "minimum": 0
66
+ },
67
+ "active_id_width": {
68
+ "type": "integer",
69
+ "minimum": 1
70
+ },
71
+ "current_band_size": {
72
+ "type": "integer",
73
+ "minimum": 1
74
+ },
75
+ "archive_band_size": {
76
+ "type": "integer",
77
+ "minimum": 1
78
+ },
63
79
  "premake_months": {
64
80
  "type": "integer",
65
81
  "minimum": 0,
@@ -35,15 +35,18 @@ module PartitionGardener
35
35
  next if bucket >= DateCalendar.beginning_of_month(cutoff)
36
36
 
37
37
  if apply_retention
38
+ keep_table = @config.fetch(:retention_keep_table, false)
38
39
  @executor.detach_partition(table_name, partition.name, concurrently: detach_concurrently?)
39
- @executor.drop_table(partition.name) unless @config.fetch(:retention_keep_table, false)
40
+ @executor.drop_table(partition.name) unless keep_table
40
41
 
42
+ action = keep_table ? "Detached" : "Dropped"
41
43
  PartitionGardener.configuration.notify(
42
- "[PartitionGardener] Dropped archive partition #{partition.name} (retention #{retention_months} months)",
44
+ "[PartitionGardener] #{action} archive partition #{partition.name} (retention #{retention_months} months)",
43
45
  context: {
44
46
  table_name: table_name,
45
47
  partition_name: partition.name,
46
48
  retention_months: retention_months,
49
+ retention_keep_table: keep_table,
47
50
  job: @job_class_name
48
51
  }
49
52
  )
@@ -51,8 +54,9 @@ module PartitionGardener
51
54
  next
52
55
  end
53
56
 
57
+ preview_action = @config.fetch(:retention_keep_table, false) ? "detach" : "drop"
54
58
  PartitionGardener.configuration.notify(
55
- "[PartitionGardener] Would drop archive partition #{partition.name} (bucket #{bucket}, retention #{retention_months} months)",
59
+ "[PartitionGardener] Would #{preview_action} archive partition #{partition.name} (bucket #{bucket}, retention #{retention_months} months)",
56
60
  context: {
57
61
  table_name: table_name,
58
62
  partition_name: partition.name,
@@ -14,6 +14,10 @@ module PartitionGardener
14
14
  active_weeks
15
15
  active_quarters
16
16
  active_years
17
+ active_id_lo
18
+ active_id_width
19
+ current_band_size
20
+ archive_band_size
17
21
  premake_months
18
22
  split_row_threshold
19
23
  move_batch_size
@@ -15,7 +15,7 @@ module PartitionGardener
15
15
  :strict_maintenance_backend_validation,
16
16
  :current_run_metrics
17
17
 
18
- attr_writer :run_record_store
18
+ attr_writer :run_record_store, :run_record_table_name
19
19
 
20
20
  def initialize
21
21
  @notifier = ->(_message_or_error, context: {}) {}
@@ -28,6 +28,7 @@ module PartitionGardener
28
28
  @analyze_after_rebalance = false
29
29
  @incremental_rebalance = true
30
30
  @run_record_enabled = true
31
+ @run_record_table_name = SqlRunRecordStore::TABLE_NAME
31
32
  @run_record_store = nil
32
33
  @retention_detach_concurrently = false
33
34
  @align_child_columns = true
@@ -39,6 +40,13 @@ module PartitionGardener
39
40
  @run_record_store ||= default_run_record_store
40
41
  end
41
42
 
43
+ def run_record_table_name
44
+ name = @run_record_table_name.to_s.strip
45
+ raise ArgumentError, "run_record_table_name is blank" if name.empty?
46
+
47
+ name
48
+ end
49
+
42
50
  def connection
43
51
  resolved = @connection_resolver.call
44
52
  raise ArgumentError, "PartitionGardener connection is not configured" unless resolved
@@ -52,14 +52,11 @@ module PartitionGardener
52
52
  end
53
53
 
54
54
  def add_years(date, count)
55
- value = to_date(date)
56
- Date.new(value.year + count, value.month, value.day)
55
+ to_date(date) >> (count * 12)
57
56
  end
58
57
 
59
58
  def add_months(date, count)
60
- base = beginning_of_month(date)
61
- month_index = (base.year * 12) + (base.month - 1) + count
62
- Date.new(month_index / 12, (month_index % 12) + 1, 1)
59
+ beginning_of_month(date) >> count
63
60
  end
64
61
  end
65
62
  end
@@ -20,7 +20,7 @@ module PartitionGardener
20
20
  segments = strategy.attached_tail_segments
21
21
  return [] if segments.empty?
22
22
 
23
- sorted = segments.sort_by { |segment| [segment.range_start.to_s, segment.name] }
23
+ sorted = segments.sort_by { |segment| [segment.range_start, segment.name] }
24
24
  gaps = []
25
25
 
26
26
  sorted.each_cons(2) do |left, right|
@@ -175,11 +175,13 @@ module PartitionGardener
175
175
  conflict_match = conflict_key.map { |column| "d.#{connection.quote_column_name(column)} = s.#{connection.quote_column_name(column)}" }.join(" AND ")
176
176
  update_columns = insert_columns - conflict_key
177
177
  update_clause = update_columns.map { |column| "#{connection.quote_column_name(column)} = EXCLUDED.#{connection.quote_column_name(column)}" }.join(", ")
178
- stale_check = if partitioned_columns.include?(stale_column)
179
- "d.#{connection.quote_column_name(stale_column)} >= s.#{connection.quote_column_name(stale_column)}"
180
- else
181
- "FALSE"
178
+ unless current_columns.include?(stale_column) && partitioned_columns.include?(stale_column)
179
+ raise ArgumentError,
180
+ "sync_stale_column #{stale_column.inspect} is missing from source or target; " \
181
+ "refusing unbounded delta sync"
182
182
  end
183
+
184
+ stale_check = "d.#{connection.quote_column_name(stale_column)} >= s.#{connection.quote_column_name(stale_column)}"
183
185
  window_predicate = [
184
186
  "#{current_partition_key_expression} >= #{start_date_sql}",
185
187
  "AND #{current_partition_key_expression} < #{end_date_sql}",
@@ -209,16 +211,11 @@ module PartitionGardener
209
211
  <<~SQL.chomp
210
212
  ON CONFLICT (#{conflict_key_str}) DO NOTHING
211
213
  SQL
212
- elsif partitioned_columns.include?(stale_column)
213
- <<~SQL.chomp
214
- ON CONFLICT (#{conflict_key_str}) DO UPDATE SET
215
- #{update_clause}
216
- WHERE #{quoted_table(target_table)}.#{connection.quote_column_name(stale_column)} < EXCLUDED.#{connection.quote_column_name(stale_column)}
217
- SQL
218
214
  else
219
215
  <<~SQL.chomp
220
216
  ON CONFLICT (#{conflict_key_str}) DO UPDATE SET
221
217
  #{update_clause}
218
+ WHERE #{quoted_table(target_table)}.#{connection.quote_column_name(stale_column)} < EXCLUDED.#{connection.quote_column_name(stale_column)}
222
219
  SQL
223
220
  end
224
221
 
@@ -406,17 +403,29 @@ module PartitionGardener
406
403
  def rename_partition_children!(parent_table, from_prefix, to_prefix)
407
404
  fetch_partitions(parent_table).each do |partition|
408
405
  partition_name = partition.to_s
409
- next if partition_name.include?(to_prefix)
406
+ bare_name = partition_name.delete('"').split(".").last
407
+ next unless bare_name.include?(from_prefix)
410
408
 
411
- new_partition_name = partition_name.gsub(from_prefix, to_prefix)
412
- run_sql "ALTER TABLE #{partition} RENAME TO #{new_partition_name}"
413
- say "Renamed #{partition} to #{new_partition_name}"
409
+ new_bare_name = bare_name.sub(from_prefix, to_prefix)
410
+ next if new_bare_name == bare_name
411
+
412
+ run_sql "ALTER TABLE #{partition} RENAME TO #{connection.quote_column_name(new_bare_name)}"
413
+ say "Renamed #{partition} to #{new_bare_name}"
414
414
  end
415
415
  end
416
416
 
417
417
  def repoint_serial_sequences!(table_name, sequence_pairs)
418
418
  sequence_pairs.each do |column_name, sequence_name|
419
- run_sql "ALTER SEQUENCE #{sequence_name} OWNED BY #{quoted_table(table_name)}.#{connection.quote_column_name(column_name)}"
419
+ quoted_column = connection.quote_column_name(column_name)
420
+ live_table = quoted_table(table_name)
421
+ run_sql <<~SQL
422
+ ALTER TABLE #{live_table}
423
+ ALTER COLUMN #{quoted_column}
424
+ SET DEFAULT nextval(#{connection.quote(sequence_name)}::regclass)
425
+ SQL
426
+ run_sql "ALTER SEQUENCE #{sequence_name} OWNED BY #{live_table}.#{quoted_column}"
427
+ max_id = run_sql("SELECT COALESCE(MAX(#{quoted_column}), 1) AS max_id FROM #{live_table}").first["max_id"].to_i
428
+ run_sql "SELECT setval(#{connection.quote(sequence_name)}, #{[max_id, 1].max})"
420
429
  say "Repointed #{sequence_name} to #{table_name}.#{column_name}"
421
430
  end
422
431
  end
@@ -139,8 +139,10 @@ module PartitionGardener
139
139
  end
140
140
 
141
141
  def apply_resuming!(plan, operations, staging_name, record)
142
- unless record.phase_at_least?("detach")
143
- prepare_staging!(staging_name)
142
+ # RunRecord.start persists phase "detach" before work. Treat detach as done only once
143
+ # the record has advanced to "segments" or later.
144
+ unless record.phase_at_least?("segments")
145
+ prepare_staging!(staging_name) unless Connection.partition_exists?(staging_name)
144
146
  detach_managed_tail_partitions!(staging_name, skip_names: keep_names_from(operations))
145
147
  drain_default_tail_rows_into_staging!(staging_name)
146
148
  record = record.advance!("detach", staging_row_count: staging_row_count(staging_name))
@@ -185,11 +187,21 @@ module PartitionGardener
185
187
  segments.each do |segment|
186
188
  next if Connection.partition_attached?(table_name, segment.name)
187
189
 
190
+ if Connection.partition_exists?(segment.name)
191
+ raise StandardError,
192
+ "partition #{segment.name} exists but is not attached to #{table_name}; " \
193
+ "CREATE TABLE IF NOT EXISTS … PARTITION OF would no-op and leave rows stranded"
194
+ end
195
+
188
196
  @executor.create_partition(
189
197
  table_name,
190
198
  segment.name,
191
199
  segment.for_values_clause(strategy)
192
200
  )
201
+
202
+ unless Connection.partition_attached?(table_name, segment.name)
203
+ raise StandardError, "failed to attach planned partition #{segment.name} to #{table_name}"
204
+ end
193
205
  end
194
206
  end
195
207
 
@@ -205,8 +217,10 @@ module PartitionGardener
205
217
  operations.select { |operation| operation.action == :drop }.each do |operation|
206
218
  partition_name = operation.attached_segment.name
207
219
  next if Connection.partition_attached?(@config[:table_name], partition_name)
220
+ next unless Connection.partition_exists?(partition_name)
221
+ next unless Connection.count_rows_in_partition_table(partition_name).zero?
208
222
 
209
- @executor.drop_table(partition_name) if Connection.partition_exists?(partition_name)
223
+ @executor.drop_table(partition_name)
210
224
  end
211
225
  end
212
226
 
@@ -1,6 +1,7 @@
1
1
  module PartitionGardener
2
2
  class SqlRunRecordStore
3
- TABLE_NAME = "partition_gardener_run_records"
3
+ TABLE_NAME = "partition_gardener_checkpoints"
4
+ LEGACY_TABLE_NAMES = %w[partition_gardener_runs partition_gardener_run_records].freeze
4
5
 
5
6
  def initialize
6
7
  @schema_mutex = Mutex.new
@@ -40,6 +41,7 @@ module PartitionGardener
40
41
  @schema_mutex.synchronize do
41
42
  return if @schema_ready
42
43
 
44
+ migrate_legacy_table!
43
45
  connection.execute(create_table_sql)
44
46
  @schema_ready = true
45
47
  end
@@ -55,17 +57,50 @@ module PartitionGardener
55
57
  connection.quote_table_name(name)
56
58
  end
57
59
 
60
+ def storage_table_name
61
+ PartitionGardener.configuration.run_record_table_name
62
+ end
63
+
64
+ def migrate_legacy_table!
65
+ return unless storage_table_name == TABLE_NAME
66
+ return if catalog_relation_exists?(storage_table_name)
67
+
68
+ LEGACY_TABLE_NAMES.each do |legacy_name|
69
+ next if legacy_name == storage_table_name
70
+ next unless catalog_relation_exists?(legacy_name)
71
+
72
+ connection.execute(
73
+ "ALTER TABLE #{quoted_table(legacy_name)} RENAME TO #{quoted_table(storage_table_name)}"
74
+ )
75
+ break
76
+ end
77
+ end
78
+
79
+ def catalog_relation_exists?(name)
80
+ sql = <<~SQL
81
+ SELECT 1 AS present
82
+ FROM pg_catalog.pg_class AS classes
83
+ INNER JOIN pg_catalog.pg_namespace AS namespaces
84
+ ON namespaces.oid = classes.relnamespace
85
+ WHERE namespaces.nspname = #{connection.quote(PartitionGardener.configuration.schema_name)}
86
+ AND classes.relname = #{connection.quote(name)}
87
+ AND classes.relkind IN ('r', 'p')
88
+ SQL
89
+
90
+ Blank.present?(connection.execute(sql).first)
91
+ end
92
+
58
93
  def load_sql(table_name)
59
94
  <<~SQL
60
95
  SELECT table_name, phase, plan_signature, staging_row_count
61
- FROM #{quoted_table(TABLE_NAME)}
96
+ FROM #{quoted_table(storage_table_name)}
62
97
  WHERE table_name = #{connection.quote(table_name)}
63
98
  SQL
64
99
  end
65
100
 
66
101
  def save_sql(table_name, attributes)
67
102
  <<~SQL
68
- INSERT INTO #{quoted_table(TABLE_NAME)} (
103
+ INSERT INTO #{quoted_table(storage_table_name)} (
69
104
  table_name, phase, plan_signature, staging_row_count, updated_at
70
105
  ) VALUES (
71
106
  #{connection.quote(table_name)},
@@ -84,14 +119,14 @@ module PartitionGardener
84
119
 
85
120
  def clear_sql(table_name)
86
121
  <<~SQL
87
- DELETE FROM #{quoted_table(TABLE_NAME)}
122
+ DELETE FROM #{quoted_table(storage_table_name)}
88
123
  WHERE table_name = #{connection.quote(table_name)}
89
124
  SQL
90
125
  end
91
126
 
92
127
  def create_table_sql
93
128
  <<~SQL
94
- CREATE TABLE IF NOT EXISTS #{quoted_table(TABLE_NAME)} (
129
+ CREATE TABLE IF NOT EXISTS #{quoted_table(storage_table_name)} (
95
130
  table_name text PRIMARY KEY,
96
131
  phase text NOT NULL,
97
132
  plan_signature text NOT NULL,
@@ -1,3 +1,3 @@
1
1
  module PartitionGardener
2
- VERSION = "0.3.5"
2
+ VERSION = "0.3.7"
3
3
  end
@@ -3,10 +3,10 @@ module PartitionGardener
3
3
 
4
4
  MOVE_BATCH_SIZE: Integer
5
5
  FUTURE_MONTH_PARTITION_ROW_THRESHOLD: Integer
6
- DEFAULT_STATEMENT_TIMEOUT: ActiveSupport::Duration
6
+ DEFAULT_STATEMENT_TIMEOUT: Integer
7
7
 
8
8
  def self.run!: (
9
- ?statement_timeout: ActiveSupport::Duration,
9
+ ?statement_timeout: Integer | Numeric,
10
10
  ?job_class_name: String,
11
11
  ?continue_on_error: bool,
12
12
  ?dry_run: bool,