partition_gardener 0.3.5 → 0.3.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -0
- data/README.md +2 -2
- data/docs/THIRD_PARTY_LICENSE_MANIFEST.tsv +18 -19
- data/docs/application_contract.md +29 -3
- data/docs/audit_reference.md +1 -1
- data/docs/changelogs/20260709120900_partition-engines-guide.md +1 -1
- data/docs/changelogs/20260813142300_rebalance-before-archive-attach.md +19 -0
- data/docs/changelogs/20260813144516_relock-appraisal-advisory-json.md +21 -0
- data/docs/changelogs/20260902114500_align-child-columns-before-attach.md +19 -0
- data/docs/changelogs/20260902124500_cut-patch-release-0-3-4.md +24 -0
- data/docs/changelogs/20260909143500_occupied-window-current-origin.md +19 -0
- data/docs/changelogs/20260909154900_tail-slot-rfc.md +21 -0
- data/docs/changelogs/20260909162554_high-end-future-occupancy.md +21 -0
- data/docs/changelogs/20260909164000_shipped-contract-rfcs.md +21 -0
- data/docs/changelogs/20260909165500_cut-patch-release-0-3-5.md +24 -0
- data/docs/changelogs/20260916144600_public-identifiers-contract.md +23 -0
- data/docs/changelogs/20261005135700_checkpoint-table-name.md +23 -0
- data/docs/changelogs/20261005141621_cut-patch-release-0-3-6.md +23 -0
- data/docs/changelogs/20261021220000_cut-patch-release-0-3-7.md +24 -0
- data/docs/configuration.md +3 -1
- data/docs/dependencies/20260813144516_json-cve-2026-71847.md +27 -0
- data/docs/issues/20260909143500_occupied-window-current-origin.md +25 -0
- data/docs/issues/20260909164000_shipped-contract-rfcs.md +25 -0
- data/docs/issues/20260916113300_opaque-composite-public-ids.md +55 -0
- data/docs/issues/20260917123253_ci-drop-automatic-dependency-audit.md +28 -0
- data/docs/operations.md +1 -1
- data/docs/partition_landscape.md +6 -4
- data/docs/reviews/20261021220000_engineering-audit-0-3-7.md +30 -0
- data/docs/reviews/20261028075216_engineering-audit-release-preparation.md +35 -0
- data/docs/reviews/20261029150855_engineering-audit-release-preparation.md +43 -0
- data/docs/schemas/partition_garden.schema.json +16 -0
- data/lib/partition_gardener/archive_retention.rb +7 -3
- data/lib/partition_gardener/config_document.rb +4 -0
- data/lib/partition_gardener/configuration.rb +9 -1
- data/lib/partition_gardener/date_calendar.rb +2 -5
- data/lib/partition_gardener/gap_detection.rb +1 -1
- data/lib/partition_gardener/migration/hot_switch_concern.rb +24 -15
- data/lib/partition_gardener/plan_applier.rb +17 -3
- data/lib/partition_gardener/sql_run_record_store.rb +40 -5
- data/lib/partition_gardener/version.rb +1 -1
- data/sig/partition_gardener.rbs +2 -2
- metadata +22 -15
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Opaque composite public identifiers
|
|
2
|
+
|
|
3
|
+
## Participants
|
|
4
|
+
|
|
5
|
+
Andrei Makarov (@amkisko)
|
|
6
|
+
|
|
7
|
+
## Decisions
|
|
8
|
+
|
|
9
|
+
Treat database uniqueness, public identifier, routing hint, and authorization as four separate contracts. PostgreSQL uniqueness on a partitioned parent is composite because the partition key must sit in every unique index. That does not by itself decide what a person sees in a URL or API.
|
|
10
|
+
|
|
11
|
+
When a single sequence or UUIDv7 keeps id unique across children, keep that logical id as the application identifier. Use self.primary_key = :id so to_param and JSON do not become Rails underscore composites such as 42_100. Still attach a routing hint on point lookups so SELECT can prune. Do not invent that hint from current month.
|
|
12
|
+
|
|
13
|
+
When id is unique only inside a child, the public identifier for point lookups must carry every uniqueness column. Prefer one opaque path segment over Rails extract_value underscore form or a separate query parameter the caller can edit independently.
|
|
14
|
+
|
|
15
|
+
If that segment is encoded, use unpadded base64url from RFC 4648 section 5, not standard base64. Standard base64 uses plus, slash, and padding equals, which are not URL-safe in a path. Encoding is packaging. Anyone can decode and re-encode another date or tenant. Tamper resistance needs an HMAC over a versioned payload, or a server-side mapping table. Encoding is not authorization. Lookups still go through an ownership set.
|
|
16
|
+
|
|
17
|
+
After decode, query with both columns and fail closed. Do not fall back to id-only scan. A bad or unsigned token is 404, not a bounded retry across children.
|
|
18
|
+
|
|
19
|
+
Do not hide browse context. List screens still show period, account, or branch in product language. Tenant and list keys for hot paths come from session or parent context, not from a tunable URL field. Point-lookup tokens may be opaque. Pagination cursors already belong in the same family as opaque tokens.
|
|
20
|
+
|
|
21
|
+
Do not ship an encoder in this gem until a second in-repo caller needs it. Host applications own secrets, token version, and decode. Gardener docs should name the contract.
|
|
22
|
+
|
|
23
|
+
Rails already covers the packaging the first pass sketched. Prefer those APIs over a homemade base64 string.
|
|
24
|
+
|
|
25
|
+
signed_id (Rails 7+) is an HMAC token over record.id, SHA256, JSON, url_safe: true. The docs say the payload is encoded not encrypted. find_signed fails closed on a bad signature. That is the closest out-of-box match to one opaque URL-safe path segment. It only carries both uniqueness columns when primary_key is the composite array. Scalar self.primary_key = :id plus query_constraints still signs and finds by id alone, so SELECT can scan every child. query_constraints never feeds SELECT.
|
|
26
|
+
|
|
27
|
+
to_param plus params.extract_value is the Rails CPK URL form. Default delimiter is underscore (4_2). Callers can edit either half. Dates and other values that contain underscore split wrong unless param_delimiter is changed. Use this for private admin paths, not for a public identifier that must not be tunable.
|
|
28
|
+
|
|
29
|
+
generates_token_for can embed extra JSON (password salt, or occurred_on) and re-check it after fetch. Lookup still yields only the primary key into find_by or find. Extra JSON is not a WHERE predicate and is plaintext in the token. CPK find_by_token_for was fixed in 2024 (Rails 7.2/8.0). find_signed CPK wrap landed later (Rails 8.1 / 2026-04). Gardener's development floor is ActiveRecord 7.1, so hosts on 7.1 cannot treat signed CPK find as solved.
|
|
30
|
+
|
|
31
|
+
has_secure_token stores a random Base58 string (URL-safe by alphabet). It is a mapping column, not an encoding of composite values. PostgreSQL still forbids a unique index on token alone on a partitioned parent unless the partition key is in that index. Lookup by token without the partition key then cannot prune, or the unique index cannot exist. Put the mapping on an unpartitioned table, or keep a globally unique logical id on the fact row.
|
|
32
|
+
|
|
33
|
+
GlobalID locates with Model.find(model_id). Composite keys become gid://app/Event/123/2026-03-15 (slash, CGI.escape, max 20 segments). Unsigned. to_param is unpadded base64url of that URI, still decodable. SignedGlobalID HMAC-wraps the same URI. Locator find prunes only when primary_key is the composite.
|
|
34
|
+
|
|
35
|
+
ActiveSupport::MessageVerifier is the primitive under signed_id, token_for, and SignedGlobalID. Default MessageVerifier is not URL-safe (plus, slash, padding). signed_id and GlobalID::Verifier opt into url_safe. MessageEncryptor exists; do not encrypt identifiers. CurrentAttributes can hold session tenant for list filters; that is not a public id.
|
|
36
|
+
|
|
37
|
+
## Effects
|
|
38
|
+
|
|
39
|
+
Added Public identifiers to docs/application_contract.md. Pointers from docs/partition_landscape.md Rails application contract, routing hints, and UI surfaces. README application_contract blurb names public identifiers. CHANGELOG Unreleased and usr/docs/changelogs/20260916144600_public-identifiers-contract.md record the doc contract.
|
|
40
|
+
|
|
41
|
+
## Next
|
|
42
|
+
|
|
43
|
+
Host docs only. No encoder in the gem. No RFC unless a later change makes public identifiers a user-facing gem contract.
|
|
44
|
+
|
|
45
|
+
## Source
|
|
46
|
+
|
|
47
|
+
Triggered by a review question: should composite ids for users be a single base64 string of composite values so callers cannot easily tune them and so the value stays URL-safe.
|
|
48
|
+
|
|
49
|
+
Later pass accepted the proposal as host documentation. Vendor product names stay out of the host contract; they remain a Source citation only.
|
|
50
|
+
|
|
51
|
+
Existing repo text: docs/partition_landscape.md Opaque tokens may encode (id, partition_key); Rails query_constraints; UI and product surfaces require visible period or tenant for browse. docs/application_contract.md query_constraints when logical id is not globally unique.
|
|
52
|
+
|
|
53
|
+
Materialized in docs/application_contract.md Public identifiers, docs/partition_landscape.md pointers, README application_contract blurb.
|
|
54
|
+
|
|
55
|
+
External: GitLab Database table partitioning, Rails Composite Primary Keys guide to_param 4_2 and extract_value, ActiveRecord::SignedId (url_safe MessageVerifier, find_by primary_key => [id] on main), ActiveRecord::TokenFor payload_for [id, block.as_json], has_secure_token SecureRandom.base58, URI::GID COMPOSITE_MODEL_ID_DELIMITER slash, GlobalID to_param unpadded base64url, Rails PRs 52798 and 57245, RFC 4648 section 5 base64url, GraphQL Relay global object identification, OWASP Insecure Direct Object Reference Prevention Cheat Sheet.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
CI quality failed on a trailing blank line in spec/support/junit_formatter.rb. Test CI also ran bundler-audit or osv-scanner, which fail whenever an advisory lands even when a separate dependency audit already exists.
|
|
2
|
+
|
|
3
|
+
## Participants
|
|
4
|
+
|
|
5
|
+
Andrei Makarov (@amkisko)
|
|
6
|
+
|
|
7
|
+
## Decisions
|
|
8
|
+
|
|
9
|
+
- Drop bundler-audit and security jobs from test.yml so tests no longer wait on advisory freshness.
|
|
10
|
+
- Keep dependency-audit.yml as workflow_dispatch only. Do not run it on push, pull request, or schedule.
|
|
11
|
+
- Disable osv-scanner in Trunk Check.
|
|
12
|
+
- Disable markdownlint MD013 in .markdownlint.yaml. Keep AGENTS.md in Trunk so other markdownlint rules still run.
|
|
13
|
+
- Keep lockfile and spec fixes that were already blocking quality or tests.
|
|
14
|
+
|
|
15
|
+
## Effects
|
|
16
|
+
|
|
17
|
+
- Test CI runs lint and specs without an advisory gate.
|
|
18
|
+
- Trunk Check no longer treats dependency CVEs as a required check.
|
|
19
|
+
- Manual dependency-audit remains available on workflow_dispatch.
|
|
20
|
+
|
|
21
|
+
## Next
|
|
22
|
+
|
|
23
|
+
- Run dependency audit on demand when a release or a known advisory needs it.
|
|
24
|
+
- Do not reintroduce advisory scanners into test.yml.
|
|
25
|
+
|
|
26
|
+
## Source
|
|
27
|
+
|
|
28
|
+
- GitHub Actions test and Trunk Check failures on 2026-09-17
|
data/docs/operations.md
CHANGED
|
@@ -83,7 +83,7 @@ table is not a partitioned table — registry typo or wrong connection. Action:
|
|
|
83
83
|
|
|
84
84
|
## Failed or partial runs
|
|
85
85
|
|
|
86
|
-
Gardener persists checkpoints in `
|
|
86
|
+
Gardener persists checkpoints in `partition_gardener_checkpoints` when `run_record_enabled` is true (default). Override the table with `run_record_table_name`.
|
|
87
87
|
|
|
88
88
|
1. Read `errors` and notifier context (`table_name`, `action`).
|
|
89
89
|
2. `audit TABLE` — confirm default count and gaps unchanged or worse.
|
data/docs/partition_landscape.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Partition Gardener targets PostgreSQL native declarative partitioning with gardener-owned maintenance plans. This page maps industry patterns to what the gem implements, documents experimental layouts, and lists what belongs outside the gem.
|
|
4
4
|
|
|
5
|
-
Canonical home for routing (layers, recovery ladder, hints), pruning, UI scoping, and aggregate snapshots. Host query and
|
|
5
|
+
Canonical home for routing (layers, recovery ladder, hints), pruning, UI scoping, and aggregate snapshots. Host query, write, and public identifier obligations live in [application_contract.md](application_contract.md). Cross-engine portability lives in [partition_engines.md](partition_engines.md).
|
|
6
6
|
|
|
7
7
|
## Implemented templates
|
|
8
8
|
|
|
@@ -113,6 +113,8 @@ end
|
|
|
113
113
|
|
|
114
114
|
`query_constraints` (Rails 7.1+) adds listed columns to `UPDATE` and `DELETE` `WHERE` clauses so writes can prune. It does not add the partition key to `SELECT` lookups: `Event.find(id)` may still scan all partitions unless `id` is globally unique and the planner can prove a single child.
|
|
115
115
|
|
|
116
|
+
Public URLs and API ids are a separate contract from `query_constraints`. See [application_contract.md](application_contract.md#public-identifiers).
|
|
117
|
+
|
|
116
118
|
For reads, encode the contract in scopes used by hot paths:
|
|
117
119
|
|
|
118
120
|
```ruby
|
|
@@ -150,7 +152,7 @@ Recovery ladder (prefer earlier steps):
|
|
|
150
152
|
#### Exact key (preferred)
|
|
151
153
|
|
|
152
154
|
- Denormalize the parent business date (or tenant/branch) onto the child at insert time; filter and join on the child column.
|
|
153
|
-
- Store the partition key next to the logical id in API cursors, job args, outbox payloads, and admin deep links. Opaque tokens may encode `(id, partition_key)` so clients never send id-only.
|
|
155
|
+
- Store the partition key next to the logical id in API cursors, job args, outbox payloads, and admin deep links. Opaque tokens may encode `(id, partition_key)` so clients never send id-only ([application_contract.md](application_contract.md#public-identifiers)).
|
|
154
156
|
- Use a generated column when the key is a stable expression of an existing column the planner can see (`created_at::date` registered and filtered the same way).
|
|
155
157
|
|
|
156
158
|
#### Orientation / range from a related record
|
|
@@ -218,7 +220,7 @@ Registry `conflict_key` must match a parent unique index and should include the
|
|
|
218
220
|
|
|
219
221
|
### UI and product surfaces
|
|
220
222
|
|
|
221
|
-
Partition boundaries are an application contract, not only a database detail. Screens and APIs should make the same separation key visible to people so routine work stays inside one prune-friendly slice. User-facing copy should describe periods, accounts, or branches in product language, not partitions, shards, or child tables.
|
|
223
|
+
Partition boundaries are an application contract, not only a database detail. Screens and APIs should make the same separation key visible to people so routine work stays inside one prune-friendly slice. User-facing copy should describe periods, accounts, or branches in product language, not partitions, shards, or child tables. Point-lookup URLs may use an opaque token; list and filter screens still show the period or tenant ([application_contract.md](application_contract.md#public-identifiers)).
|
|
222
224
|
|
|
223
225
|
Time-partitioned data (monthly sliding window):
|
|
224
226
|
|
|
@@ -410,7 +412,7 @@ Operations and migration docs in this repository:
|
|
|
410
412
|
|
|
411
413
|
- [operations.md](operations.md) — runbook
|
|
412
414
|
- [cutover.md](cutover.md) — hot-switch playbook
|
|
413
|
-
- [application_contract.md](application_contract.md) — host app behavior
|
|
415
|
+
- [application_contract.md](application_contract.md) — host app behavior, public identifiers
|
|
414
416
|
- [monitoring.md](monitoring.md) — metrics and alerts
|
|
415
417
|
- [retention.md](retention.md) — archive detach and drop
|
|
416
418
|
- [audit_reference.md](audit_reference.md) — audit and plan catalog
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Engineering audit and 0.3.7 fixes
|
|
2
|
+
|
|
3
|
+
## Participants
|
|
4
|
+
|
|
5
|
+
Andrei Makarov (@amkisko)
|
|
6
|
+
|
|
7
|
+
## Decisions
|
|
8
|
+
|
|
9
|
+
- Run a multi-model engineering audit on `partition_gardener` (correctness, tests, hot-path, contracts).
|
|
10
|
+
- Fact-check findings in source before shipping.
|
|
11
|
+
- Ship high-confidence correctness and contract fixes as patch `0.3.7`.
|
|
12
|
+
- Defer structural refactors (PlanApplier phase machine collapse, HotSwitchConcern split, strategy mixin extraction, default statement_timeout wrapper) to later work.
|
|
13
|
+
|
|
14
|
+
## Effects
|
|
15
|
+
|
|
16
|
+
- Confirmed and fixed: resume skips detach while phase is `detach`; hot-switch child rename skip on `p_` prefix; serial sequence default left on `_old`; unbounded `sync_delta_data` when stale column missing; `CREATE IF NOT EXISTS … PARTITION OF` no-op on detached name; lexicographic integer gap sort; leap-day `add_years`; retention notify says drop when table kept; JSON registry omits integer window keys.
|
|
17
|
+
- Unit specs: 151 examples, 0 failures (`bundle exec rspec` on non-integration paths from `spec/spec_paths.txt`).
|
|
18
|
+
- Integration resume examples extended for row-count preservation and started-but-not-detached phase; not run in this session (needs `INTEGRATION=1` and Postgres).
|
|
19
|
+
|
|
20
|
+
## Next
|
|
21
|
+
|
|
22
|
+
- Run `INTEGRATION=1` resume and hot-switch specs before cutting the gem.
|
|
23
|
+
- Run `ruby usr/bin/release.rb` when the working tree is clean and ready to push to RubyGems.
|
|
24
|
+
- Follow-up: session advisory lock default for long copies; statement_timeout wrapper; RFC 0002 status; dead `SCHEMA_PATH` wiring or removal.
|
|
25
|
+
|
|
26
|
+
## Source
|
|
27
|
+
|
|
28
|
+
- Audit agents: correctness, tests/coverage, hot-path/resource, org/contracts.
|
|
29
|
+
- Branch: `patch/engineering-audit-0-3-7`
|
|
30
|
+
- Related changelog: `docs/changelogs/20261021220000_cut-patch-release-0-3-7.md`
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Engineering audit and release preparation
|
|
2
|
+
|
|
3
|
+
## Decisions
|
|
4
|
+
|
|
5
|
+
- Continue on main while preserving the pre-existing uncommitted changes. Do not commit, tag, push, or publish.
|
|
6
|
+
- Retain release version 0.3.7. Reconcile root and appraisal PATH metadata without updating third-party dependency versions.
|
|
7
|
+
- Review correctness, executable coverage, workflow triggers and permissions, package boundaries, resource cost, and trace/identity boundaries. Re-audit the changed paths after corrections.
|
|
8
|
+
|
|
9
|
+
## Effects
|
|
10
|
+
|
|
11
|
+
- Correct the resume integration fixture's initial expectation: the source rows initially belong to the attached current partition, and the hot monthly partition exists after rebalance.
|
|
12
|
+
- Restore the required lint gate by combining adjacent boundary stubs. Verify all integration examples against an isolated PostgreSQL server, including the existing rebalance and hot-switch changes.
|
|
13
|
+
- Synchronize the own-gem PATH version in the root and appraisal lockfiles to 0.3.7, enable pull-request tests, and restrict workflow permissions to contents: read.
|
|
14
|
+
- Fix the release confirmation guard in usr/bin/release.rb so EOF refuses publication. An explicit Enter still selects the displayed default. The process-boundary regression test executes the release script with command execution replaced at the operating-system boundary; no publication command runs against a real service. The shared regression was observed first in Favorites: 2 examples, 1 failure before correction and 2 examples, 0 failures after correction. The release specs pass in every repository.
|
|
15
|
+
- Final serial JSON report: 188 examples, 0 failures with INTEGRATION=1 and PostgreSQL; no pending integration examples.
|
|
16
|
+
- Required project checks passed: DATABASE_URL=postgres://postgres@127.0.0.1:55432/partition_gardener_test make test. Environment: Ruby 4.0.2, BUNDLE_PATH=vendor/bundle; the installed native extensions require that Ruby build.
|
|
17
|
+
- Build the gem, verify it with Gem::Package#verify, and check that the changelog heading matches the gem version. Final package measurements are recorded in the session output; this gem includes the review note itself.
|
|
18
|
+
- Cold process measurement on macOS arm64 with /usr/bin/time -l and bundle exec ruby -Ilib: 0.35 CPU seconds, 0.7 elapsed seconds, 31817728 peak RSS bytes. Favorites loads Rails before the engine. Local smoke ceilings: 1 MiB compressed gem, 128 MiB RSS, 2 CPU seconds. All measured values fit those ceilings. CPU seconds are the energy proxy; electricity use was not measured.
|
|
19
|
+
- Parse every workflow YAML, verify lockfile own-gem versions, and run git diff --check. All passed. All six assigned repositories remain on main.
|
|
20
|
+
|
|
21
|
+
## Next
|
|
22
|
+
|
|
23
|
+
- No confirmed correction remains in the reviewed changes. Hosted GitHub Actions and the complete supported Ruby/framework matrix were not executed locally.
|
|
24
|
+
- Scope limits: production traffic, production database size and query plans, external service retention, and co-located resource interference were not measured. The local byte/RSS/CPU results do not establish those properties.
|
|
25
|
+
- Trace review follows local file/log and protocol emissions. No live customer identities or credentials were used. Data retention and deletion in embedding applications remain host contracts.
|
|
26
|
+
- Learned-system training/evaluation and cloud deployment modes do not apply to these library trees. Native memory-boundary mode applies to Polyrun's extension; browser presentation applies to Favorites; HTTP/identity and message companions apply only to repositories that expose those boundaries. Libraries have no independent production health endpoint to assess.
|
|
27
|
+
|
|
28
|
+
## Source
|
|
29
|
+
|
|
30
|
+
- AGENTS.md
|
|
31
|
+
- .agents/skills/engineering-audit/SKILL.md
|
|
32
|
+
- .agents/skills/changelog-update/SKILL.md
|
|
33
|
+
- CHANGELOG.md
|
|
34
|
+
- Makefile
|
|
35
|
+
- spec/README.md
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Engineering audit and release preparation
|
|
2
|
+
|
|
3
|
+
## Participants
|
|
4
|
+
|
|
5
|
+
Andrei Makarov (@amkisko)
|
|
6
|
+
|
|
7
|
+
## Decisions
|
|
8
|
+
|
|
9
|
+
- Continue on main while preserving uncommitted work. Do not commit, tag, push, or publish.
|
|
10
|
+
- Keep release version 0.3.7. Set the CHANGELOG heading date to 2026-10-29. RubyGems still lists 0.3.6 as the latest published version.
|
|
11
|
+
- Re-check correctness, executable coverage, workflow triggers and permissions, package boundaries, resource cost, and database-boundary behavior after the prior release-prep pass.
|
|
12
|
+
|
|
13
|
+
## Effects
|
|
14
|
+
|
|
15
|
+
- Enable `pull_request` on the test workflow, keep token permissions at `contents: read`, and cancel superseded runs.
|
|
16
|
+
- Set the `CHANGELOG.md` heading for 0.3.7 to 2026-10-29 and align the cut note under `docs/changelogs/`.
|
|
17
|
+
- Final serial JSON report: 188 examples, 0 failures with `INTEGRATION=1` and PostgreSQL on port 5432; no pending examples. Parallel polyrun with five workers: exit 0.
|
|
18
|
+
- Required project checks passed: rubocop (116 files, no offenses), rbs validate, `INTEGRATION=1` polyrun parallel-rspec. Environment: Ruby 4.0.2, `BUNDLE_PATH=vendor/bundle`.
|
|
19
|
+
- `release_spec`: 2 examples, 0 failures (EOF refuses gem push; Enter accepts the displayed default with system execution replaced).
|
|
20
|
+
- Build the gem, verify it with `Gem::Package#verify`, and confirm the changelog heading matches version 0.3.7. Compressed package: 117760 bytes; source payload: 495104 bytes across 104 packaged paths. These are compressed versus uncompressed sizes, not interchangeable quantities.
|
|
21
|
+
- Cold process measurement on macOS arm64 with `/usr/bin/time -p` and `bundle exec ruby -Ilib`: 0.17 CPU seconds, 0.26 elapsed seconds. Peak RSS via `/usr/bin/time -l` was unavailable in this session (`sysctl kern.clockrate` refused). Prior release-prep peak RSS on the same require path was 31817728 bytes. Local smoke ceilings: 1 MiB compressed gem, 128 MiB RSS, 2 CPU seconds. Measured gem size and CPU fit those ceilings. CPU seconds are the energy proxy; electricity use was not measured.
|
|
22
|
+
- Parse every workflow YAML, verify lockfile own-gem versions at 0.3.7, and run `git diff --check`. All passed.
|
|
23
|
+
|
|
24
|
+
## Next
|
|
25
|
+
|
|
26
|
+
- Commit, tag 0.3.7, push, and publish remain operator steps; this session did not run them. When the tree is clean, `make release` runs gem push, tag, and GitHub release.
|
|
27
|
+
- Hosted GitHub Actions and the complete supported Ruby/framework matrix were not executed locally beyond the default Gemfile path used by `make test`.
|
|
28
|
+
- Re-measure peak RSS with `/usr/bin/time -l` outside the restricted session if a fresh number is required before publish.
|
|
29
|
+
- Scope limits: production traffic, production database size and query plans, external service retention, and co-located resource interference were not measured. The local byte/CPU results do not establish those properties.
|
|
30
|
+
- Trace review follows local file/log and protocol emissions. No live customer identities or credentials were used. Data retention and deletion in embedding applications remain host contracts.
|
|
31
|
+
- Learned-system training/evaluation and cloud deployment modes do not apply to this library tree. Database-boundary mode applies to PostgreSQL maintenance DDL and checkpoint resume. The library has no independent production health endpoint to assess.
|
|
32
|
+
|
|
33
|
+
## Source
|
|
34
|
+
|
|
35
|
+
- AGENTS.md
|
|
36
|
+
- .agents/skills/engineering-audit/SKILL.md
|
|
37
|
+
- .agents/skills/changelog-update/SKILL.md
|
|
38
|
+
- CHANGELOG.md
|
|
39
|
+
- Makefile
|
|
40
|
+
- spec/README.md
|
|
41
|
+
- docs/reviews/20261028075216_engineering-audit-release-preparation.md
|
|
42
|
+
- docs/reviews/20261021220000_engineering-audit-0-3-7.md
|
|
43
|
+
- docs/changelogs/20261021220000_cut-patch-release-0-3-7.md
|
|
@@ -60,6 +60,22 @@
|
|
|
60
60
|
"minimum": 1,
|
|
61
61
|
"maximum": 10
|
|
62
62
|
},
|
|
63
|
+
"active_id_lo": {
|
|
64
|
+
"type": "integer",
|
|
65
|
+
"minimum": 0
|
|
66
|
+
},
|
|
67
|
+
"active_id_width": {
|
|
68
|
+
"type": "integer",
|
|
69
|
+
"minimum": 1
|
|
70
|
+
},
|
|
71
|
+
"current_band_size": {
|
|
72
|
+
"type": "integer",
|
|
73
|
+
"minimum": 1
|
|
74
|
+
},
|
|
75
|
+
"archive_band_size": {
|
|
76
|
+
"type": "integer",
|
|
77
|
+
"minimum": 1
|
|
78
|
+
},
|
|
63
79
|
"premake_months": {
|
|
64
80
|
"type": "integer",
|
|
65
81
|
"minimum": 0,
|
|
@@ -35,15 +35,18 @@ module PartitionGardener
|
|
|
35
35
|
next if bucket >= DateCalendar.beginning_of_month(cutoff)
|
|
36
36
|
|
|
37
37
|
if apply_retention
|
|
38
|
+
keep_table = @config.fetch(:retention_keep_table, false)
|
|
38
39
|
@executor.detach_partition(table_name, partition.name, concurrently: detach_concurrently?)
|
|
39
|
-
@executor.drop_table(partition.name) unless
|
|
40
|
+
@executor.drop_table(partition.name) unless keep_table
|
|
40
41
|
|
|
42
|
+
action = keep_table ? "Detached" : "Dropped"
|
|
41
43
|
PartitionGardener.configuration.notify(
|
|
42
|
-
"[PartitionGardener]
|
|
44
|
+
"[PartitionGardener] #{action} archive partition #{partition.name} (retention #{retention_months} months)",
|
|
43
45
|
context: {
|
|
44
46
|
table_name: table_name,
|
|
45
47
|
partition_name: partition.name,
|
|
46
48
|
retention_months: retention_months,
|
|
49
|
+
retention_keep_table: keep_table,
|
|
47
50
|
job: @job_class_name
|
|
48
51
|
}
|
|
49
52
|
)
|
|
@@ -51,8 +54,9 @@ module PartitionGardener
|
|
|
51
54
|
next
|
|
52
55
|
end
|
|
53
56
|
|
|
57
|
+
preview_action = @config.fetch(:retention_keep_table, false) ? "detach" : "drop"
|
|
54
58
|
PartitionGardener.configuration.notify(
|
|
55
|
-
"[PartitionGardener] Would
|
|
59
|
+
"[PartitionGardener] Would #{preview_action} archive partition #{partition.name} (bucket #{bucket}, retention #{retention_months} months)",
|
|
56
60
|
context: {
|
|
57
61
|
table_name: table_name,
|
|
58
62
|
partition_name: partition.name,
|
|
@@ -15,7 +15,7 @@ module PartitionGardener
|
|
|
15
15
|
:strict_maintenance_backend_validation,
|
|
16
16
|
:current_run_metrics
|
|
17
17
|
|
|
18
|
-
attr_writer :run_record_store
|
|
18
|
+
attr_writer :run_record_store, :run_record_table_name
|
|
19
19
|
|
|
20
20
|
def initialize
|
|
21
21
|
@notifier = ->(_message_or_error, context: {}) {}
|
|
@@ -28,6 +28,7 @@ module PartitionGardener
|
|
|
28
28
|
@analyze_after_rebalance = false
|
|
29
29
|
@incremental_rebalance = true
|
|
30
30
|
@run_record_enabled = true
|
|
31
|
+
@run_record_table_name = SqlRunRecordStore::TABLE_NAME
|
|
31
32
|
@run_record_store = nil
|
|
32
33
|
@retention_detach_concurrently = false
|
|
33
34
|
@align_child_columns = true
|
|
@@ -39,6 +40,13 @@ module PartitionGardener
|
|
|
39
40
|
@run_record_store ||= default_run_record_store
|
|
40
41
|
end
|
|
41
42
|
|
|
43
|
+
def run_record_table_name
|
|
44
|
+
name = @run_record_table_name.to_s.strip
|
|
45
|
+
raise ArgumentError, "run_record_table_name is blank" if name.empty?
|
|
46
|
+
|
|
47
|
+
name
|
|
48
|
+
end
|
|
49
|
+
|
|
42
50
|
def connection
|
|
43
51
|
resolved = @connection_resolver.call
|
|
44
52
|
raise ArgumentError, "PartitionGardener connection is not configured" unless resolved
|
|
@@ -52,14 +52,11 @@ module PartitionGardener
|
|
|
52
52
|
end
|
|
53
53
|
|
|
54
54
|
def add_years(date, count)
|
|
55
|
-
|
|
56
|
-
Date.new(value.year + count, value.month, value.day)
|
|
55
|
+
to_date(date) >> (count * 12)
|
|
57
56
|
end
|
|
58
57
|
|
|
59
58
|
def add_months(date, count)
|
|
60
|
-
|
|
61
|
-
month_index = (base.year * 12) + (base.month - 1) + count
|
|
62
|
-
Date.new(month_index / 12, (month_index % 12) + 1, 1)
|
|
59
|
+
beginning_of_month(date) >> count
|
|
63
60
|
end
|
|
64
61
|
end
|
|
65
62
|
end
|
|
@@ -20,7 +20,7 @@ module PartitionGardener
|
|
|
20
20
|
segments = strategy.attached_tail_segments
|
|
21
21
|
return [] if segments.empty?
|
|
22
22
|
|
|
23
|
-
sorted = segments.sort_by { |segment| [segment.range_start
|
|
23
|
+
sorted = segments.sort_by { |segment| [segment.range_start, segment.name] }
|
|
24
24
|
gaps = []
|
|
25
25
|
|
|
26
26
|
sorted.each_cons(2) do |left, right|
|
|
@@ -175,11 +175,13 @@ module PartitionGardener
|
|
|
175
175
|
conflict_match = conflict_key.map { |column| "d.#{connection.quote_column_name(column)} = s.#{connection.quote_column_name(column)}" }.join(" AND ")
|
|
176
176
|
update_columns = insert_columns - conflict_key
|
|
177
177
|
update_clause = update_columns.map { |column| "#{connection.quote_column_name(column)} = EXCLUDED.#{connection.quote_column_name(column)}" }.join(", ")
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
178
|
+
unless current_columns.include?(stale_column) && partitioned_columns.include?(stale_column)
|
|
179
|
+
raise ArgumentError,
|
|
180
|
+
"sync_stale_column #{stale_column.inspect} is missing from source or target; " \
|
|
181
|
+
"refusing unbounded delta sync"
|
|
182
182
|
end
|
|
183
|
+
|
|
184
|
+
stale_check = "d.#{connection.quote_column_name(stale_column)} >= s.#{connection.quote_column_name(stale_column)}"
|
|
183
185
|
window_predicate = [
|
|
184
186
|
"#{current_partition_key_expression} >= #{start_date_sql}",
|
|
185
187
|
"AND #{current_partition_key_expression} < #{end_date_sql}",
|
|
@@ -209,16 +211,11 @@ module PartitionGardener
|
|
|
209
211
|
<<~SQL.chomp
|
|
210
212
|
ON CONFLICT (#{conflict_key_str}) DO NOTHING
|
|
211
213
|
SQL
|
|
212
|
-
elsif partitioned_columns.include?(stale_column)
|
|
213
|
-
<<~SQL.chomp
|
|
214
|
-
ON CONFLICT (#{conflict_key_str}) DO UPDATE SET
|
|
215
|
-
#{update_clause}
|
|
216
|
-
WHERE #{quoted_table(target_table)}.#{connection.quote_column_name(stale_column)} < EXCLUDED.#{connection.quote_column_name(stale_column)}
|
|
217
|
-
SQL
|
|
218
214
|
else
|
|
219
215
|
<<~SQL.chomp
|
|
220
216
|
ON CONFLICT (#{conflict_key_str}) DO UPDATE SET
|
|
221
217
|
#{update_clause}
|
|
218
|
+
WHERE #{quoted_table(target_table)}.#{connection.quote_column_name(stale_column)} < EXCLUDED.#{connection.quote_column_name(stale_column)}
|
|
222
219
|
SQL
|
|
223
220
|
end
|
|
224
221
|
|
|
@@ -406,17 +403,29 @@ module PartitionGardener
|
|
|
406
403
|
def rename_partition_children!(parent_table, from_prefix, to_prefix)
|
|
407
404
|
fetch_partitions(parent_table).each do |partition|
|
|
408
405
|
partition_name = partition.to_s
|
|
409
|
-
|
|
406
|
+
bare_name = partition_name.delete('"').split(".").last
|
|
407
|
+
next unless bare_name.include?(from_prefix)
|
|
410
408
|
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
409
|
+
new_bare_name = bare_name.sub(from_prefix, to_prefix)
|
|
410
|
+
next if new_bare_name == bare_name
|
|
411
|
+
|
|
412
|
+
run_sql "ALTER TABLE #{partition} RENAME TO #{connection.quote_column_name(new_bare_name)}"
|
|
413
|
+
say "Renamed #{partition} to #{new_bare_name}"
|
|
414
414
|
end
|
|
415
415
|
end
|
|
416
416
|
|
|
417
417
|
def repoint_serial_sequences!(table_name, sequence_pairs)
|
|
418
418
|
sequence_pairs.each do |column_name, sequence_name|
|
|
419
|
-
|
|
419
|
+
quoted_column = connection.quote_column_name(column_name)
|
|
420
|
+
live_table = quoted_table(table_name)
|
|
421
|
+
run_sql <<~SQL
|
|
422
|
+
ALTER TABLE #{live_table}
|
|
423
|
+
ALTER COLUMN #{quoted_column}
|
|
424
|
+
SET DEFAULT nextval(#{connection.quote(sequence_name)}::regclass)
|
|
425
|
+
SQL
|
|
426
|
+
run_sql "ALTER SEQUENCE #{sequence_name} OWNED BY #{live_table}.#{quoted_column}"
|
|
427
|
+
max_id = run_sql("SELECT COALESCE(MAX(#{quoted_column}), 1) AS max_id FROM #{live_table}").first["max_id"].to_i
|
|
428
|
+
run_sql "SELECT setval(#{connection.quote(sequence_name)}, #{[max_id, 1].max})"
|
|
420
429
|
say "Repointed #{sequence_name} to #{table_name}.#{column_name}"
|
|
421
430
|
end
|
|
422
431
|
end
|
|
@@ -139,8 +139,10 @@ module PartitionGardener
|
|
|
139
139
|
end
|
|
140
140
|
|
|
141
141
|
def apply_resuming!(plan, operations, staging_name, record)
|
|
142
|
-
|
|
143
|
-
|
|
142
|
+
# RunRecord.start persists phase "detach" before work. Treat detach as done only once
|
|
143
|
+
# the record has advanced to "segments" or later.
|
|
144
|
+
unless record.phase_at_least?("segments")
|
|
145
|
+
prepare_staging!(staging_name) unless Connection.partition_exists?(staging_name)
|
|
144
146
|
detach_managed_tail_partitions!(staging_name, skip_names: keep_names_from(operations))
|
|
145
147
|
drain_default_tail_rows_into_staging!(staging_name)
|
|
146
148
|
record = record.advance!("detach", staging_row_count: staging_row_count(staging_name))
|
|
@@ -185,11 +187,21 @@ module PartitionGardener
|
|
|
185
187
|
segments.each do |segment|
|
|
186
188
|
next if Connection.partition_attached?(table_name, segment.name)
|
|
187
189
|
|
|
190
|
+
if Connection.partition_exists?(segment.name)
|
|
191
|
+
raise StandardError,
|
|
192
|
+
"partition #{segment.name} exists but is not attached to #{table_name}; " \
|
|
193
|
+
"CREATE TABLE IF NOT EXISTS … PARTITION OF would no-op and leave rows stranded"
|
|
194
|
+
end
|
|
195
|
+
|
|
188
196
|
@executor.create_partition(
|
|
189
197
|
table_name,
|
|
190
198
|
segment.name,
|
|
191
199
|
segment.for_values_clause(strategy)
|
|
192
200
|
)
|
|
201
|
+
|
|
202
|
+
unless Connection.partition_attached?(table_name, segment.name)
|
|
203
|
+
raise StandardError, "failed to attach planned partition #{segment.name} to #{table_name}"
|
|
204
|
+
end
|
|
193
205
|
end
|
|
194
206
|
end
|
|
195
207
|
|
|
@@ -205,8 +217,10 @@ module PartitionGardener
|
|
|
205
217
|
operations.select { |operation| operation.action == :drop }.each do |operation|
|
|
206
218
|
partition_name = operation.attached_segment.name
|
|
207
219
|
next if Connection.partition_attached?(@config[:table_name], partition_name)
|
|
220
|
+
next unless Connection.partition_exists?(partition_name)
|
|
221
|
+
next unless Connection.count_rows_in_partition_table(partition_name).zero?
|
|
208
222
|
|
|
209
|
-
@executor.drop_table(partition_name)
|
|
223
|
+
@executor.drop_table(partition_name)
|
|
210
224
|
end
|
|
211
225
|
end
|
|
212
226
|
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
module PartitionGardener
|
|
2
2
|
class SqlRunRecordStore
|
|
3
|
-
TABLE_NAME = "
|
|
3
|
+
TABLE_NAME = "partition_gardener_checkpoints"
|
|
4
|
+
LEGACY_TABLE_NAMES = %w[partition_gardener_runs partition_gardener_run_records].freeze
|
|
4
5
|
|
|
5
6
|
def initialize
|
|
6
7
|
@schema_mutex = Mutex.new
|
|
@@ -40,6 +41,7 @@ module PartitionGardener
|
|
|
40
41
|
@schema_mutex.synchronize do
|
|
41
42
|
return if @schema_ready
|
|
42
43
|
|
|
44
|
+
migrate_legacy_table!
|
|
43
45
|
connection.execute(create_table_sql)
|
|
44
46
|
@schema_ready = true
|
|
45
47
|
end
|
|
@@ -55,17 +57,50 @@ module PartitionGardener
|
|
|
55
57
|
connection.quote_table_name(name)
|
|
56
58
|
end
|
|
57
59
|
|
|
60
|
+
def storage_table_name
|
|
61
|
+
PartitionGardener.configuration.run_record_table_name
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def migrate_legacy_table!
|
|
65
|
+
return unless storage_table_name == TABLE_NAME
|
|
66
|
+
return if catalog_relation_exists?(storage_table_name)
|
|
67
|
+
|
|
68
|
+
LEGACY_TABLE_NAMES.each do |legacy_name|
|
|
69
|
+
next if legacy_name == storage_table_name
|
|
70
|
+
next unless catalog_relation_exists?(legacy_name)
|
|
71
|
+
|
|
72
|
+
connection.execute(
|
|
73
|
+
"ALTER TABLE #{quoted_table(legacy_name)} RENAME TO #{quoted_table(storage_table_name)}"
|
|
74
|
+
)
|
|
75
|
+
break
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def catalog_relation_exists?(name)
|
|
80
|
+
sql = <<~SQL
|
|
81
|
+
SELECT 1 AS present
|
|
82
|
+
FROM pg_catalog.pg_class AS classes
|
|
83
|
+
INNER JOIN pg_catalog.pg_namespace AS namespaces
|
|
84
|
+
ON namespaces.oid = classes.relnamespace
|
|
85
|
+
WHERE namespaces.nspname = #{connection.quote(PartitionGardener.configuration.schema_name)}
|
|
86
|
+
AND classes.relname = #{connection.quote(name)}
|
|
87
|
+
AND classes.relkind IN ('r', 'p')
|
|
88
|
+
SQL
|
|
89
|
+
|
|
90
|
+
Blank.present?(connection.execute(sql).first)
|
|
91
|
+
end
|
|
92
|
+
|
|
58
93
|
def load_sql(table_name)
|
|
59
94
|
<<~SQL
|
|
60
95
|
SELECT table_name, phase, plan_signature, staging_row_count
|
|
61
|
-
FROM #{quoted_table(
|
|
96
|
+
FROM #{quoted_table(storage_table_name)}
|
|
62
97
|
WHERE table_name = #{connection.quote(table_name)}
|
|
63
98
|
SQL
|
|
64
99
|
end
|
|
65
100
|
|
|
66
101
|
def save_sql(table_name, attributes)
|
|
67
102
|
<<~SQL
|
|
68
|
-
INSERT INTO #{quoted_table(
|
|
103
|
+
INSERT INTO #{quoted_table(storage_table_name)} (
|
|
69
104
|
table_name, phase, plan_signature, staging_row_count, updated_at
|
|
70
105
|
) VALUES (
|
|
71
106
|
#{connection.quote(table_name)},
|
|
@@ -84,14 +119,14 @@ module PartitionGardener
|
|
|
84
119
|
|
|
85
120
|
def clear_sql(table_name)
|
|
86
121
|
<<~SQL
|
|
87
|
-
DELETE FROM #{quoted_table(
|
|
122
|
+
DELETE FROM #{quoted_table(storage_table_name)}
|
|
88
123
|
WHERE table_name = #{connection.quote(table_name)}
|
|
89
124
|
SQL
|
|
90
125
|
end
|
|
91
126
|
|
|
92
127
|
def create_table_sql
|
|
93
128
|
<<~SQL
|
|
94
|
-
CREATE TABLE IF NOT EXISTS #{quoted_table(
|
|
129
|
+
CREATE TABLE IF NOT EXISTS #{quoted_table(storage_table_name)} (
|
|
95
130
|
table_name text PRIMARY KEY,
|
|
96
131
|
phase text NOT NULL,
|
|
97
132
|
plan_signature text NOT NULL,
|
data/sig/partition_gardener.rbs
CHANGED
|
@@ -3,10 +3,10 @@ module PartitionGardener
|
|
|
3
3
|
|
|
4
4
|
MOVE_BATCH_SIZE: Integer
|
|
5
5
|
FUTURE_MONTH_PARTITION_ROW_THRESHOLD: Integer
|
|
6
|
-
DEFAULT_STATEMENT_TIMEOUT:
|
|
6
|
+
DEFAULT_STATEMENT_TIMEOUT: Integer
|
|
7
7
|
|
|
8
8
|
def self.run!: (
|
|
9
|
-
?statement_timeout:
|
|
9
|
+
?statement_timeout: Integer | Numeric,
|
|
10
10
|
?job_class_name: String,
|
|
11
11
|
?continue_on_error: bool,
|
|
12
12
|
?dry_run: bool,
|