db-purger 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4025ea48c6e9d97505720a8450017e70f2fd527408b518cc8837ddf512ce043d
-  data.tar.gz: 3cb576faf87f8a5d2735fd8edc973e04063469a7edae92148dac7d7a80688be1
+  metadata.gz: 863444350ef64942e0669feacdf4a119514c16369a31eeb99e31a5f5435028ef
+  data.tar.gz: 66b063187fdaa33232a6fffe76f9496264e8bd74e2a835013139b4e19f3e045e
 SHA512:
-  metadata.gz: 3685ddf0b11aff29c97d618026abe46a59c1d21a8310f6d8c038f913b2892e30b77468c1d5d8946978eaf14e4003278815967417e7ec5ec98cc0257b00dbce84
-  data.tar.gz: b6dacd558e0b1a7b4b22c9ed332df7071da6300e43e9eb0476cbcf32899c998c39197aad7005e78e38f86ed821d728bf61e8f9d62fe945695987abe614443d0e
+  metadata.gz: dfd50964ce367548606fe758df1d1cfb48d6891d9ac09993079b851a78658807fa4a5ff8633709a49f42b9d67c9367260721cd2d0f1454dbd7da197f3363f54b
+  data.tar.gz: ccc2bee89229fad63fd2dc7d20682b07b008b29040cb66a92aa1cd4c48bd8f09c5f6b91834ead524be8f3394d54c8af08ce8df5d61bc5067a6229b14187c7ede

data/ARCHITECTURE.md

@@ -0,0 +1,160 @@
+# Architecture
+
+db-purger is small (~900 lines) and splits cleanly into three layers: **describe** a purge (plan + DSL),
+**check** it (validator), and **run** it (purgers + instrumentation).
+
+```
+            plan file / PlanBuilder.build { ... }
+                          │
+                          ▼
+  ┌──────────────┐   ┌─────────┐   ┌─────────┐
+  │ PlanBuilder  │──▶│  Plan   │──▶│  Table  │──┐ each Table owns a nested Plan
+  │ (DSL)        │   │         │   │         │◀─┘ (recursive tree)
+  └──────────────┘   └─────────┘   └─────────┘
+                          │
+        ┌─────────────────┼──────────────────┐
+        ▼                 ▼                  ▼
+  PlanValidator      Executor ─────▶ PurgeTable / PurgeTableScanner
+  (schema check)     (entry point)          │ (PurgeTableHelper)
+                                            ▼
+                               ActiveSupport::Notifications
+                                  (*.db_purger events)
+                                            │
+                                            ▼
+                               MetricSubscriber ─▶ Metrics
+```
+
+## Components
+
+| File | Responsibility |
+|---|---|
+| `lib/db-purger.rb` | Autoloads everything; holds the global `DBPurger.config`. |
+| `config.rb` | Global options: `explain?`, `explain_file`, `datetime_format`. |
+| `table.rb` | Value object for one table in the plan: name, match field, options, and a lazily created nested `Plan`. |
+| `plan.rb` | A node in the plan tree: one optional `base_table` plus lists of parent, child, search and ignored tables. `#purge!` is the run entry point. |
+| `plan_builder.rb` | The DSL. `instance_eval`s a plan file or block against a `Plan`; nested blocks get a new builder bound to that table's nested plan. |
+| `plan_validator.rb` | `ActiveModel::Validations` over plan vs. schema: missing tables, unknown tables, unknown columns. |
+| `executor.rb` | Convenience façade: loads a plan file, applies config options, `verify!`, `purge!`. |
+| `purge_table.rb` | Purges one table by `field = value(s)` in primary-key batches. Recurses into nested tables. |
+| `purge_table_scanner.rb` | Purges a `purge_table_search` table: full `find_in_batches` scan filtered through the user's `search_proc`. |
+| `purge_table_helper.rb` | Shared behaviour for both purgers: nested-table recursion, delete vs. soft delete vs. explain, transactions. |
+| `metrics.rb` / `metric_subscriber.rb` | Aggregate timing and row counts per table from the notification events. |
+| `dynamic_plan_builder.rb` | Generates plan-file source from `has_many` associations; a bootstrap tool, not used at purge time. |
+
+## The plan tree
+
+`PlanBuilder` always attaches tables to the *current* plan's `base_table.nested_plan` once a base table
+exists. So a plan file is really:
+
+```
+Plan (root)
+└── base_table: companies(:id)
+    └── nested Plan
+        ├── parent_tables: [company_tags(:company_id)]
+        ├── child_tables:  [employments(:company_id) ─▶ nested Plan ..., websites(:id, fk: website_id) ...]
+        ├── search_tables: [users(:id)]
+        └── ignore_tables
+```
+
+`Plan#tables` flattens this tree for validation; `Table#foreign_keys` collects the `foreign_key:` columns of
+a table's direct children so the purger can `SELECT` them alongside the primary key.
+
+## Purge algorithm
+
+`Plan#purge!` resets metrics and starts a `PurgeTable` on the base table with the purge value. Each
+`PurgeTable#purge!` does:
+
+```
+each_batch = loop:
+    batch = SELECT pk, <child foreign_keys> FROM t
+            WHERE field IN (values) [AND conditions] [AND pk > last_pk]
+            ORDER BY pk LIMIT batch_size
+    break if batch empty
+
+purge_children(batch) =
+    for each child_table without foreign_key:      # rows pointing at us
+      PurgeTable(child, child.field, batch.pks).purge!    (recursive)
+
+delete_rows(batch) =
+    if any child has foreign_key:                  # rows we point at
+      TRANSACTION
+        delete batch by pk
+        for each fk child: PurgeTable(child, child.field, batch.<fk values>).purge!
+    else
+      delete batch by pk
+
+if table has a primary key:
+  if no parent_tables:
+    each_batch: purge_children(batch); delete_rows(batch)
+  else:                                            # two passes
+    each_batch: purge_children(batch)
+    for each parent_table:                         # siblings sharing the key
+      PurgeTable(parent, parent.field, original purge value).purge!
+    each_batch: delete_rows(batch)
+else:
+  raise if nested child/parent tables              # no ids to propagate
+  single DELETE WHERE field = value [AND conditions]
+
+for each search_table:
+  PurgeTableScanner(search_table).purge!
+```
+
+Key properties:
+
+- **Depth-first, children first.** A row is only deleted after everything referencing it, so FK
+  constraints hold without `ON DELETE CASCADE`.
+- **Two passes when there are parent tables.** A parent table can reference this table (e.g.
+  `company_tags.company_id → companies.id`) *and* be referenced by one of its children, so it is purged
+  between the child pass and the delete pass. Tables without parent tables keep the single pass.
+- **Keyset pagination** (`pk > last_pk`) rather than `OFFSET`, so batches stay cheap on large tables and still
+  advance in explain mode where nothing is actually deleted.
+- **Bounded memory.** Only one batch of ids per level of the tree is held at a time.
+- **Idempotent re-runs.** Every step re-derives its rows from the database; there is no saved cursor, so a
+  crashed purge is resumed by running it again.
+- **Parent vs. child** is about *which value* is propagated: children get the enclosing batch's primary keys,
+  parents get the original purge value.
+
+`PurgeTableScanner` follows the same shape but sources batches from `find_in_batches` over the entire table
+(plus `conditions`) and narrows each batch with `search_proc` before recursing and deleting.
+
+## Delete strategies
+
+`PurgeTableHelper#delete_records_with_instrumentation` picks one of three actions for every scope:
+
+1. **explain** (`DBPurger.config.explain?`) — rewrite `scope.to_sql` into `DELETE`/`UPDATE ... SET`, write it
+   to `explain_file`, return `scope.count`.
+2. **soft delete** (`mark_deleted_field`) — `scope.update_all(field => value)`.
+3. **hard delete** — `scope.delete_all`.
+
+All three go through ActiveRecord's `*_all` methods: no model callbacks or validations run.
+
+## Instrumentation
+
+Every unit of work is wrapped in `ActiveSupport::Notifications.instrument` under the `db_purger` namespace
+(`purge`, `next_batch`, `delete_records`, `search_filter`). The purgers never talk to `Metrics` directly;
+`MetricSubscriber` (an `ActiveSupport::Subscriber`) translates events into `Metrics` counters. This keeps the
+purge code free of reporting concerns and lets callers attach their own subscribers (StatsD, logs, progress
+bars) without changes to the library.
+
+`PurgeTableScanner` uses the lower-level `instrumenter.start/finish` pair for `next_batch` because the batch
+fetch happens inside `find_in_batches` rather than in a block the scanner controls.
+
+## Dependencies and boundaries
+
+- **ActiveRecord** is the only database interface. The library relies on `where`, `select`, `order`,
+  `limit`, `find_in_batches`, `delete_all`, `update_all`, `transaction`, `to_sql` and
+  `connection.quote*`, so it is adapter-agnostic (specs use SQLite).
+- **dynamic-active-model** supplies the `database` object. The purge code only calls `database.models` and
+  matches on `model.table_name`, so any object with that shape works. `DynamicPlanBuilder` additionally
+  relies on `reflect_on_all_associations`.
+- **Config scoping.** `DBPurger.config` returns the config set by `DBPurger.with_config` for the current
+  thread, falling back to a process-wide default. `Executor#purge!` wraps the run in `with_config`, so executors
+  with different explain settings can't interfere. `MetricSubscriber.metrics` is still process-wide.
+
+## Testing
+
+- `spec/db-purger/*` — unit specs for the builder, validator, executor and purger.
+- `spec/integrations/*` — end-to-end purges over the schema in `spec/support/db/schema.rb`, asserting row-count
+  deltas per table and, for explain mode, the exact SQL in `spec/fixtures/delete_plan.sql`.
+- `spec/support/test_db.rb` builds a fresh SQLite database and dynamic models (`TestDB::*`) for each run.
+- `spec/fixtures/*.plan.rb` — plan files used for loading and validation cases.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Douglas Youch
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,271 @@
+# db-purger
+
+[![CI](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml)
+[![Coverage](https://raw.githubusercontent.com/dougyouch/db-purger/badges/coverage.svg)](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml)
+[![Gem Version](https://img.shields.io/gem/v/db-purger)](https://rubygems.org/gems/db-purger)
+
+Purge every row tied to a single top-level record — a company, an account, a tenant — across all of the
+tables that reference it, in batches, from a declarative Ruby plan.
+
+```ruby
+executor = DBPurger::Executor.new(database, 'config/company.plan.rb')
+executor.verify!          # fail fast if the plan doesn't cover the schema
+executor.purge!(42)       # delete company 42 and everything that hangs off it
+```
+
+## Why
+
+Deleting a tenant from a relational database is rarely one `DELETE`. Rows are spread across dozens of
+tables, some keyed directly on the tenant id, some several joins away, some polymorphic, some that must be
+soft-deleted rather than removed. `ON DELETE CASCADE` is often absent, and a single giant delete will lock
+tables and blow out replication.
+
+db-purger lets you describe those relationships once, in a plan file, and then:
+
+- deletes in **primary-key batches** (default 10,000) so no single statement gets too large
+- deletes rows **only after the rows that reference them**, so foreign-key constraints hold without `ON DELETE CASCADE`
+- **validates** the plan against the live schema, so a newly added table can't be silently forgotten
+- supports **soft deletes** (`UPDATE ... SET deleted_at = ...`) per table
+- has an **explain mode** that prints the SQL it would run instead of running it
+- emits **ActiveSupport::Notifications** events, with a built-in subscriber that collects timing and row counts
+
+## Installation
+
+```ruby
+# Gemfile
+gem 'db-purger'
+```
+
+Requires Ruby >= 3.2 and ActiveRecord >= 7.0. Models are supplied by
+[dynamic-active-model](https://github.com/dougyouch/dynamic-active-model), which builds ActiveRecord classes
+directly from the database schema.
+
+## Quick start
+
+### 1. Load the database
+
+db-purger works against a `DynamicActiveModel::Database`. Anything that responds to `#models` (returning
+ActiveRecord classes) will do.
+
+```ruby
+require 'active_record'
+require 'dynamic-active-model'
+require 'db-purger'
+
+module PurgeDB; end
+
+database = DynamicActiveModel::Explorer.explore(
+  PurgeDB,
+  { adapter: 'mysql2', host: 'localhost', database: 'app', username: 'app' },
+  %w[schema_migrations ar_internal_metadata]   # tables to skip entirely
+)
+```
+
+### 2. Write a plan
+
+A plan is Ruby, evaluated with the plan DSL. Given this schema:
+
+```
+companies            (id, website_id, ...)
+employments          (id, company_id, user_id, ...)
+employment_notes     (id, employment_id, ...)
+company_tags         (company_id, tag_id)            -- no primary key
+events               (id, model_type, model_id, ...) -- polymorphic
+websites             (id, content_id, ...)
+contents             (id, ...)
+tags, jobs           -- shared lookup tables, never purged
+```
+
+a plan to purge one company looks like:
+
+```ruby
+# config/company.plan.rb
+base_table(:companies, :id)
+
+# Tables keyed directly on the purge value (company_id = 42)
+parent_table(:company_tags, :company_id)
+
+# Tables keyed on the base table's primary key, purged batch-by-batch
+child_table(:employments, :company_id) do
+  child_table(:employment_notes, :employment_id)
+  child_table(:events, :model_id, conditions: { model_type: 'PurgeDB::Employment' })
+end
+
+child_table(:events, :model_id, conditions: { model_type: 'PurgeDB::Company' })
+
+# The company row points at the website (companies.website_id -> websites.id)
+child_table(:websites, :id, foreign_key: :website_id) do
+  child_table(:contents, :id, foreign_key: :content_id)
+end
+
+# Shared tables that are intentionally left alone
+ignore_table :tags
+ignore_table :jobs
+ignore_table(/\Atmp_/)           # regexps are allowed
+```
+
+### 3. Verify and purge
+
+```ruby
+executor = DBPurger::Executor.new(database, 'config/company.plan.rb')
+executor.verify!            # raises 'purge plan failed verification', errors printed to $stderr
+deleted = executor.purge!(42)
+```
+
+`purge!` returns the number of base-table rows deleted.
+
+## The plan DSL
+
+| Method | Meaning |
+|---|---|
+| `base_table(table, field, opts = {}, &block)` | The root of the purge. Rows where `field = purge_value` are purged. Declare it **first** — every subsequent top-level call nests under it. |
+| `child_table(table, field, opts = {}, &block)` | Rows whose `field` matches the **primary key** of the enclosing table's current batch. Purged before that batch is deleted. |
+| `child_table(table, :id, foreign_key: :col, &block)` | Inverted relationship: the *enclosing* table holds `col` pointing at this table's `id`. Deleted in the same transaction, right after the enclosing batch. |
+| `parent_table(table, field, opts = {}, &block)` | Rows whose `field` matches the original **purge value**. Purged after the enclosing table's child tables but before the enclosing table's own rows, so it may both reference the base (`company_tags.company_id → companies.id`) and be referenced by a child table. Use for sibling tables that share the same key (e.g. `company_id`). |
+| `purge_table_search(table, field, opts = {}) { \|batch\| ... }` | Scans the whole table in batches; the block receives each batch and returns the records to purge. For orphans that can't be reached by a key. |
+| `ignore_table(name_or_regexp)` | Exclude a table from validation. |
+
+Blocks nest arbitrarily deep. Because `purge_table_search` uses its block as the filter, nest tables under it
+with `.nested_plan`:
+
+```ruby
+purge_table_search(:users, :id) do |users|
+  users = users.index_by(&:id)
+  PurgeDB::Employment.where(user_id: users.keys).pluck(:user_id).each { |id| users.delete(id) }
+  users.values                                   # users with no remaining employments
+end.nested_plan do
+  child_table(:events, :model_id, conditions: { model_type: 'PurgeDB::User' })
+end
+```
+
+### Table options
+
+| Option | Default | Description |
+|---|---|---|
+| `batch_size:` | `10_000` | Rows fetched and deleted per batch. |
+| `conditions:` | none | Extra `where` applied to every query for this table (hash or SQL string). |
+| `foreign_key:` | none | See `child_table` above. |
+| `mark_deleted_field:` | none | Soft delete: `UPDATE table SET field = value` instead of `DELETE`. |
+| `mark_deleted_value:` | `1` | Value written to `mark_deleted_field`. `Time` values are formatted with `datetime_format` in explain output. |
+
+Tables **without a primary key** (e.g. join tables) are purged with a single unbatched `DELETE ... WHERE
+field = value`; nested tables are not supported under them.
+
+### Building a plan in code
+
+```ruby
+plan = DBPurger::PlanBuilder.build do
+  base_table(:companies, :id)
+  child_table(:employments, :company_id)
+end
+
+DBPurger::Executor.new(database, plan).purge!(42)
+# or, without the executor:
+plan.purge!(database, 42)
+```
+
+### Generating a starting plan
+
+`DynamicPlanBuilder` walks the `has_many` associations dynamic-active-model discovered and emits a plan file,
+listing every unreachable table as `ignore_table`. Treat the output as a first draft: it only knows about
+conventional `<singular_table>_id` foreign keys and cannot infer polymorphic, soft-delete, or search rules.
+
+```ruby
+puts DBPurger::DynamicPlanBuilder.new(database).build(:companies, :id)
+```
+
+## Validation
+
+`Executor#verify!` (or `DBPurger::PlanValidator.new(database, plan).valid?`) checks that:
+
+- every table in the database is either in the plan or ignored (`missing_tables`)
+- every table in the plan exists in the database (`unknown_tables`)
+- every field, `foreign_key` and `mark_deleted_field` named in the plan is a real column
+- the plan has a `base_table`, and every `batch_size` is positive
+- tables without a primary key have no nested child or parent tables (there would be no ids to propagate)
+
+Run it in CI against your schema so a new table can't ship without a purge decision.
+
+## Explain mode (dry run)
+
+```ruby
+File.open('purge.sql', 'w') do |io|
+  executor = DBPurger::Executor.new(database, 'config/company.plan.rb', explain: true, explain_file: io)
+  executor.purge!(42)
+end
+```
+
+Nothing is deleted; each `DELETE`/`UPDATE` is written to `explain_file` (default `$stdout`). Lookups still run
+against the database, so the output reflects real row ids.
+
+| Executor option | Default |
+|---|---|
+| `explain:` | `false` |
+| `explain_file:` | `$stdout` |
+| `datetime_format:` | `'%Y-%m-%d %H:%M:%S'` |
+
+Each executor keeps its own settings and applies them only for the duration of its `purge!` (per thread), so
+creating another executor can't turn a dry run into a live one. `explain:` must be `true`, `false` or `nil`;
+anything else (such as the string `'true'`) raises `ArgumentError` rather than running for real.
+
+## Metrics and instrumentation
+
+Attach the built-in subscriber once at boot:
+
+```ruby
+DBPurger::MetricSubscriber.auto_attach
+
+executor.purge!(42)
+DBPurger::MetricSubscriber.metrics.as_json
+# => { took: 12.4, started_at: ..., finished_at: ...,
+#      purge_stats:  { employments: { duration:, num_purges:, num_records: } },
+#      delete_stats: { employments: { duration:, num_delete_queries:, num_deleted:, num_expected_to_delete: } },
+#      lookup_stats: { ... }, filter_stats: { ... } }
+```
+
+Metrics are reset at the start of each `Plan#purge!`. To feed your own telemetry, subscribe to the raw events
+(all in the `db_purger` namespace):
+
+| Event | Payload |
+|---|---|
+| `purge.db_purger` | `table_name`, `purge_field`, `deleted` |
+| `next_batch.db_purger` | `table_name`, `start_id`, `num_records` |
+| `delete_records.db_purger` | `table_name`, `num_records`, `records_deleted`, `deleted` |
+| `search_filter.db_purger` | `table_name`, `num_records`, `num_records_selected` |
+
+## Caveats
+
+- **Only the base table is the entry point.** Top-level `parent_table`/`child_table` calls made before
+  `base_table` are ignored by `Plan#purge!`.
+- **Not one big transaction.** Each batch is its own set of statements (foreign-key children share a
+  transaction with their parent batch). An interrupted purge is safe to re-run with the same value.
+- **Soft-deleted rows still match.** A `mark_deleted_field` table is not filtered on that field; add
+  `conditions:` if re-runs should skip already-marked rows.
+- Always run explain mode against a copy of production before the first real purge with a new plan.
+
+## Development
+
+```sh
+bundle install
+bundle exec rspec        # specs run against a throwaway SQLite database
+bundle exec rubocop
+script/console
+```
+
+CI (`.github/workflows/ci.yml`) runs RuboCop and the specs on Ruby 4.0 for every push and pull request.
+The HTML coverage report is attached to each run as the `coverage` artifact, and pushes to `master` refresh
+the coverage badge on the `badges` branch.
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for how the pieces fit together.
+
+## Releasing
+
+1. Bump `s.version` in `db-purger.gemspec` and merge to `master`.
+2. Tag and push: `git tag v0.5.0 && git push origin v0.5.0`
+
+`.github/workflows/release.yml` re-runs CI, checks the tag matches the gemspec version, publishes to RubyGems
+via trusted publishing (no API key), and creates a GitHub release with the `.gem` attached.
+
+## License
+
+MIT — see [LICENSE.txt](LICENSE.txt).

data/lib/db-purger/config.rb

@@ -5,16 +5,30 @@ module DBPurger
   class Config
     DEFAULT_DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
 
-    attr_writer :explain,
-                :explain_file,
+    attr_writer :explain_file,
                 :datetime_format
 
+    def initialize(options = {})
+      self.explain = options[:explain]
+      @explain_file = options[:explain_file]
+      @datetime_format = options[:datetime_format]
+    end
+
+    # Fail closed: a value like 'true' must not silently mean "run for real"
+    def explain=(value)
+      unless [true, false, nil].include?(value)
+        raise(ArgumentError, "explain must be true, false or nil, got #{value.inspect}")
+      end
+
+      @explain = value
+    end
+
     def explain?
       @explain == true
     end
 
     def explain_file
-      (@explain_file || $stdout)
+      @explain_file || $stdout
     end
 
     def datetime_format

data/lib/db-purger/dynamic_plan_builder.rb

@@ -14,7 +14,6 @@ module DBPurger
       @tables = []
     end
 
-    # rubocop:disable Metrics/AbcSize
     def build(base_table_name, field)
       write_table('base', base_table_name.to_s, field, [], nil)
       line_break
@@ -32,7 +31,6 @@ module DBPurger
       ignore_missing_tables
       @output
     end
-    # rubocop:enable Metrics/AbcSize
 
     private
 
@@ -41,7 +39,7 @@ module DBPurger
     end
 
     def write(str)
-      @output << (INDENT * @indent_depth) + str + "\n"
+      @output << "#{INDENT * @indent_depth}#{str}\n"
     end
 
     def line_break
@@ -49,7 +47,7 @@ module DBPurger
     end
 
     def add_parent_tables(base_table_name, field)
-      @database.models.each do |model|
+      sorted_models.each do |model|
         next if model.table_name == base_table_name.to_s
         next unless column?(model, field)
 
@@ -72,7 +70,13 @@ module DBPurger
     end
 
     def find_child_models(model, field)
-      model_has_many_associations(model).map(&:klass).select { |m| column?(m, field) }
+      model_has_many_associations(model).map(&:klass).select { |m| column?(m, field) }.sort_by(&:table_name)
+    end
+
+    # database.models order depends on how the adapter lists tables, which varies by platform;
+    # sort so the generated plan is deterministic
+    def sorted_models
+      @sorted_models ||= @database.models.sort_by(&:table_name)
     end
 
     def model_has_many_associations(model)
@@ -82,7 +86,7 @@ module DBPurger
     end
 
     def foreign_key_name(model)
-      model.table_name.singularize + '_id'
+      "#{model.table_name.singularize}_id"
     end
 
     def column?(model, field)
@@ -101,7 +105,7 @@ module DBPurger
     end
 
     def ignore_missing_tables
-      missing_tables = @database.models.map(&:table_name) - @tables
+      missing_tables = sorted_models.map(&:table_name) - @tables
       return if missing_tables.empty?
 
       line_break

data/lib/db-purger/executor.rb

@@ -8,14 +8,14 @@ module DBPurger
     def initialize(database, plan, options = {})
       @database = database
       @plan = plan.is_a?(Plan) ? plan : load_plan(plan)
-      setup_config(options)
+      @config = Config.new(options)
       @error_io = $stderr
     end
 
     def purge!(purge_value)
       raise('purge_value is nil') if purge_value.nil?
 
-      @plan.purge!(@database, purge_value)
+      ::DBPurger.with_config(@config) { @plan.purge!(@database, purge_value) }
     end
 
     def verify!
@@ -31,12 +31,6 @@ module DBPurger
       @plan_validator ||= PlanValidator.new(@database, @plan)
     end
 
-    def setup_config(options)
-      ::DBPurger.config.explain = options[:explain]
-      ::DBPurger.config.explain_file = options[:explain_file]
-      ::DBPurger.config.datetime_format = options[:datetime_format]
-    end
-
     def load_plan(file)
       PlanBuilder
         .new(Plan.new)

data/lib/db-purger/metric_subscriber.rb

@@ -23,7 +23,7 @@ module DBPurger
       self.class.metrics.update_purge_stats(
         event.payload[:table_name],
         event.duration,
-        event.payload[:deleted]
+        event.payload[:deleted] || 0
       )
     end
 
@@ -31,7 +31,7 @@ module DBPurger
       self.class.metrics.update_delete_records_stats(
         event.payload[:table_name],
         event.duration,
-        event.payload[:records_deleted],
+        event.payload[:records_deleted] || 0,
         event.payload[:num_records]
       )
     end
@@ -40,7 +40,7 @@ module DBPurger
       self.class.metrics.update_lookup_stats(
         event.payload[:table_name],
         event.duration,
-        event.payload[:num_records]
+        event.payload[:num_records] || 0
       )
     end
 
@@ -48,8 +48,8 @@ module DBPurger
       self.class.metrics.update_search_filter_stats(
         event.payload[:table_name],
         event.duration,
-        event.payload[:num_records],
-        event.payload[:num_records_selected]
+        event.payload[:num_records] || 0,
+        event.payload[:num_records_selected] || 0
       )
     end
   end

data/lib/db-purger/plan.rb

@@ -18,6 +18,8 @@ module DBPurger
     end
 
     def purge!(database, purge_value)
+      raise('plan has no base_table') unless @base_table
+
       MetricSubscriber.reset!
       num_deleted = PurgeTable.new(database, @base_table, @base_table.field, purge_value).purge!
       MetricSubscriber.finished!

data/lib/db-purger/plan_validator.rb

@@ -7,6 +7,7 @@ module DBPurger
   class PlanValidator
     include ActiveModel::Validations
 
+    validate :validate_base_table
     validate :validate_no_missing_tables
     validate :validate_no_unknown_tables
     validate :validate_tables
@@ -28,6 +29,10 @@ module DBPurger
 
     private
 
+    def validate_base_table
+      errors.add(:base_table, 'is required') unless @plan.base_table
+    end
+
     def validate_no_missing_tables
       errors.add(:missing_tables, missing_tables.sort.join(',')) unless missing_tables.empty?
     end
@@ -51,6 +56,26 @@ module DBPurger
           errors.add(:table, "#{table.name}.#{field} is missing in the database")
         end
       end
+
+      validate_mark_deleted_field(table, model)
+      validate_batch_size(table)
+      validate_nested_tables_have_primary_key(table, model)
+    end
+
+    def validate_mark_deleted_field(table, model)
+      return if table.mark_deleted_field.nil? || model.column_names.include?(table.mark_deleted_field.to_s)
+
+      errors.add(:table, "#{table.name}.#{table.mark_deleted_field} (mark_deleted_field) is missing in the database")
+    end
+
+    def validate_batch_size(table)
+      errors.add(:table, "#{table.name} batch_size must be positive") unless table.batch_size.to_i.positive?
+    end
+
+    def validate_nested_tables_have_primary_key(table, model)
+      return if model.primary_key || !table.nested_key_tables?
+
+      errors.add(:table, "#{table.name} has no primary key and cannot have nested child or parent tables")
     end
 
     def find_model_for_table(table)

data/lib/db-purger/purge_table.rb

@@ -21,9 +21,11 @@ module DBPurger
       ActiveSupport::Notifications.instrument('purge.db_purger',
                                               table_name: @table.name,
                                               purge_field: @purge_field) do |payload|
+        payload[:deleted] = @num_deleted
         if model.primary_key
           purge_in_batches!
         else
+          ensure_no_nested_key_tables!
           purge_all!
         end
         purge_search_tables
@@ -34,6 +36,13 @@ module DBPurger
 
     private
 
+    # without a primary key there are no batch ids to propagate, so nested tables would be silently skipped
+    def ensure_no_nested_key_tables!
+      return unless @table.nested_key_tables?
+
+      raise("#{@table.name} has no primary key and cannot have nested child or parent tables")
+    end
+
     def purge_all!
       scope = model.where(@purge_field => @purge_value)
       scope = scope.where(@table.conditions) if @table.conditions
@@ -41,13 +50,27 @@ module DBPurger
     end
 
     def purge_in_batches!
+      unless @table.parent_tables?
+        each_batch do |batch|
+          purge_nested_tables(batch) if @table.nested_tables?
+          delete_records(batch)
+        end
+        return
+      end
+
+      # Parent tables may reference this table's rows and be referenced by its child tables,
+      # so purge them after the children but before this table's rows.
+      each_batch { |batch| purge_nested_tables(batch) }
+      purge_parent_tables
+      each_batch { |batch| delete_records(batch) }
+    end
+
+    def each_batch
       start_id = nil
       until (batch = next_batch(start_id)).empty?
-        start_id = batch.last.send(model.primary_key)
-        purge_nested_tables(batch) if @table.nested_tables?
-        delete_records(batch)
+        start_id = batch.last[model.primary_key]
+        yield batch
       end
-      purge_parent_tables
     end
 
     def next_batch(start_id)

data/lib/db-purger/purge_table_helper.rb

@@ -38,8 +38,9 @@ module DBPurger
       end
     end
 
+    # record[] reads the column; send would call a same-named method instead (e.g. a column called "reload")
     def batch_values(batch, field)
-      batch.map { |record| record.send(field) }.compact
+      batch.map { |record| record[field] }.compact
     end
 
     def foreign_tables?
@@ -72,7 +73,7 @@ module DBPurger
         else
           scope.to_sql.sub(/SELECT .*?FROM/, 'DELETE FROM')
         end
-      ::DBPurger.config.explain_file.puts(sql + ';')
+      ::DBPurger.config.explain_file.puts("#{sql};")
       scope.count
     end
 

data/lib/db-purger/table.rb

@@ -32,6 +32,15 @@ module DBPurger
       @nested_plan != nil
     end
 
+    def parent_tables?
+      nested_tables? && !@nested_plan.parent_tables.empty?
+    end
+
+    # nested tables that depend on this table's rows (search tables scan independently)
+    def nested_key_tables?
+      nested_tables? && !(@nested_plan.child_tables.empty? && @nested_plan.parent_tables.empty?)
+    end
+
     def tables
       @nested_plan ? @nested_plan.tables : []
     end
@@ -45,7 +54,7 @@ module DBPurger
     end
 
     def mark_deleted_value
-      @mark_deleted_value || 1
+      @mark_deleted_value.nil? ? 1 : @mark_deleted_value
     end
   end
 end

data/lib/db-purger.rb

@@ -15,7 +15,22 @@ module DBPurger
   autoload :PlanValidator, 'db-purger/plan_validator'
   autoload :Table, 'db-purger/table'
 
+  # The config in effect for the current thread: the one set by with_config, else the global default
   def self.config
-    @config ||= Config.new
+    Thread.current[:db_purger_config] || default_config
+  end
+
+  def self.default_config
+    @default_config ||= Config.new
+  end
+
+  # Runs the block with config in effect for this thread only, so concurrent or
+  # later executors can't change explain mode out from under a running purge
+  def self.with_config(config)
+    previous = Thread.current[:db_purger_config]
+    Thread.current[:db_purger_config] = config
+    yield
+  ensure
+    Thread.current[:db_purger_config] = previous
   end
 end

metadata

@@ -1,49 +1,55 @@
 --- !ruby/object:Gem::Specification
 name: db-purger
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.0
 platform: ruby
 authors:
 - Doug Youch
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-03-16 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: dynamic-active-model
+  name: activerecord
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '7.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '7.0'
 - !ruby/object:Gem::Dependency
-  name: inheritance-helper
+  name: dynamic-active-model
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.7'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
-description: Purge database tables by top level id in batches
+        version: '0.7'
+description: DB Purger deletes (or soft-deletes) every row related to a single top-level
+  record (e.g. a company or account) using a declarative Ruby purge plan. Tables are
+  purged in primary-key batches, child tables before their parents, with plan validation
+  against the live schema, an explain (dry-run) mode that emits SQL, and ActiveSupport::Notifications
+  instrumentation for metrics.
 email: dougyouch@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- ARCHITECTURE.md
+- LICENSE.txt
+- README.md
 - lib/db-purger.rb
 - lib/db-purger/config.rb
 - lib/db-purger/dynamic_plan_builder.rb
@@ -60,8 +66,10 @@ files:
 homepage: https://github.com/dougyouch/db-purger
 licenses:
 - MIT
-metadata: {}
-post_install_message:
+metadata:
+  source_code_uri: https://github.com/dougyouch/db-purger
+  bug_tracker_uri: https://github.com/dougyouch/db-purger/issues
+  rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths:
 - lib
@@ -69,15 +77,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '3.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.3
-signing_key:
+rubygems_version: 4.0.20
 specification_version: 4
-summary: DB Purger by top level id
+summary: Purge all data tied to a top-level id across related tables, in batches
 test_files: []