db-purger 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 863444350ef64942e0669feacdf4a119514c16369a31eeb99e31a5f5435028ef
-  data.tar.gz: 66b063187fdaa33232a6fffe76f9496264e8bd74e2a835013139b4e19f3e045e
+  metadata.gz: 45ab4b4aa9e6e8c3778061c5deb8a8eb37fecf153bb1bae9cf298f2c83433f22
+  data.tar.gz: bbb9aec2aaa9dcd8e7c0609186795bf17e8fce64ffa951ce6dd1877981b7dbcc
 SHA512:
-  metadata.gz: dfd50964ce367548606fe758df1d1cfb48d6891d9ac09993079b851a78658807fa4a5ff8633709a49f42b9d67c9367260721cd2d0f1454dbd7da197f3363f54b
-  data.tar.gz: ccc2bee89229fad63fd2dc7d20682b07b008b29040cb66a92aa1cd4c48bd8f09c5f6b91834ead524be8f3394d54c8af08ce8df5d61bc5067a6229b14187c7ede
+  metadata.gz: 41d99640be3f800277142f9ef9a9ec0f0945785282551c757f92ba92de36dfad331b735533713bb98385023378727ca5365cf995cff84580dfe725cbbf10240d
+  data.tar.gz: bfe3512b7f289cec1b4ac7f466918c4aaa6a5c12f9deb61e60e825153dd3e1badbbcbb1eb578e80dd453504f9b7b08f70a996e4d9d10b7d1e7a654245d12ec00

data/ARCHITECTURE.md

@@ -31,15 +31,18 @@ db-purger is small (~900 lines) and splits cleanly into three layers: **describe
 | `lib/db-purger.rb` | Autoloads everything; holds the global `DBPurger.config`. |
 | `config.rb` | Global options: `explain?`, `explain_file`, `datetime_format`. |
 | `table.rb` | Value object for one table in the plan: name, match field, options, and a lazily created nested `Plan`. |
-| `plan.rb` | A node in the plan tree: one optional `base_table` plus lists of parent, child, search and ignored tables. `#purge!` is the run entry point. |
+| `plan.rb` | A node in the plan tree: one optional `base_table` plus lists of parent, child, nullify, search and ignored tables. `#purge!` is the run entry point; `#root_tables` are the tables it starts from. |
 | `plan_builder.rb` | The DSL. `instance_eval`s a plan file or block against a `Plan`; nested blocks get a new builder bound to that table's nested plan. |
 | `plan_validator.rb` | `ActiveModel::Validations` over plan vs. schema: missing tables, unknown tables, unknown columns. |
 | `executor.rb` | Convenience façade: loads a plan file, applies config options, `verify!`, `purge!`. |
 | `purge_table.rb` | Purges one table by `field = value(s)` in primary-key batches. Recurses into nested tables. |
 | `purge_table_scanner.rb` | Purges a `purge_table_search` table: full `find_in_batches` scan filtered through the user's `search_proc`. |
+| `nullify_table.rb` | Sets a nullable column to `NULL` on rows pointing at a batch of ids about to be deleted (`ON DELETE SET NULL` at purge time). |
 | `purge_table_helper.rb` | Shared behaviour for both purgers: nested-table recursion, delete vs. soft delete vs. explain, transactions. |
 | `metrics.rb` / `metric_subscriber.rb` | Aggregate timing and row counts per table from the notification events. |
-| `dynamic_plan_builder.rb` | Generates plan-file source from `has_many` associations; a bootstrap tool, not used at purge time. |
+| `dynamic_plan_builder.rb` | Generates plan-file source (`build` for a base table, `build_for` for several roots); a bootstrap tool, not used at purge time. |
+| `association_graph.rb` | For the generator: which tables reference a model, and by which column, from its `has_many`/`has_one`/HABTM reflections. |
+| `plan_writer.rb` | For the generator: renders plan DSL text and records which tables it wrote. |
 
 ## The plan tree
 
@@ -52,16 +55,30 @@ Plan (root)
     └── nested Plan
         ├── parent_tables: [company_tags(:company_id)]
         ├── child_tables:  [employments(:company_id) ─▶ nested Plan ..., websites(:id, fk: website_id) ...]
+        ├── nullify_tables: [companies(:acquired_by_company_id)]
         ├── search_tables: [users(:id)]
         └── ignore_tables
 ```
 
+Without a `base_table`, top-level `parent_table`s stay in the root plan's `parent_tables` and each one is a
+root:
+
+```
+Plan (root)
+├── parent_tables: [calls(:oid) ─▶ nested Plan ..., emails(:oid) ─▶ nested Plan ..., sms_messages(:oid) ...]
+├── search_tables
+└── ignore_tables
+```
+
 `Plan#tables` flattens this tree for validation; `Table#foreign_keys` collects the `foreign_key:` columns of
 a table's direct children so the purger can `SELECT` them alongside the primary key.
 
 ## Purge algorithm
 
-`Plan#purge!` resets metrics and starts a `PurgeTable` on the base table with the purge value. Each
+`Plan#purge!` resets metrics and starts a `PurgeTable` with the purge value on each root table
+(`root_tables` = the `base_table`, if any, followed by top-level `parent_tables`, in declaration order), then runs
+any top-level search tables. With a `base_table`, `root_tables` is just `[base_table]`, which is the original
+single-root algorithm. Each
 `PurgeTable#purge!` does:
 
 ```
@@ -72,6 +89,8 @@ each_batch = loop:
     break if batch empty
 
 purge_children(batch) =
+    for each nullify_table:                        # optional rows pointing at us
+      UPDATE nullify SET field = NULL WHERE field IN (batch.pks) [AND conditions]
     for each child_table without foreign_key:      # rows pointing at us
       PurgeTable(child, child.field, batch.pks).purge!    (recursive)
 
@@ -103,6 +122,9 @@ Key properties:
 
 - **Depth-first, children first.** A row is only deleted after everything referencing it, so FK
   constraints hold without `ON DELETE CASCADE`.
+- **Unlink before delete.** A `nullify_table` is updated for each batch before that batch's children and rows
+  are deleted, so a self-referential foreign key (or a reference from a row that must survive) never blocks the
+  delete, whatever the chain depth or id order across batches.
 - **Two passes when there are parent tables.** A parent table can reference this table (e.g.
   `company_tags.company_id → companies.id`) *and* be referenced by one of its children, so it is purged
   between the child pass and the delete pass. Tables without parent tables keep the single pass.
@@ -131,7 +153,7 @@ All three go through ActiveRecord's `*_all` methods: no model callbacks or valid
 ## Instrumentation
 
 Every unit of work is wrapped in `ActiveSupport::Notifications.instrument` under the `db_purger` namespace
-(`purge`, `next_batch`, `delete_records`, `search_filter`). The purgers never talk to `Metrics` directly;
+(`purge`, `next_batch`, `delete_records`, `nullify_records`, `search_filter`). The purgers never talk to `Metrics` directly;
 `MetricSubscriber` (an `ActiveSupport::Subscriber`) translates events into `Metrics` counters. This keeps the
 purge code free of reporting concerns and lets callers attach their own subscribers (StatsD, logs, progress
 bars) without changes to the library.
@@ -157,4 +179,8 @@ fetch happens inside `find_in_batches` rather than in a block the scanner contro
 - `spec/integrations/*` — end-to-end purges over the schema in `spec/support/db/schema.rb`, asserting row-count
   deltas per table and, for explain mode, the exact SQL in `spec/fixtures/delete_plan.sql`.
 - `spec/support/test_db.rb` builds a fresh SQLite database and dynamic models (`TestDB::*`) for each run.
+- `spec/integrations/multi_root_plan_spec.rb` purges one org from a separate outreach schema (`spec/support/outreach_db.rb`,
+  real FOREIGN KEY constraints) seeded with two orgs by `OutreachSeeder`, and asserts the exact surviving rows of
+  every table for the generated plan, a hand-written small-batch plan and the equivalent `base_table` plan.
+- `spec/support/throwaway_db.rb` builds standalone SQLite databases from raw SQL for schema-specific specs.
 - `spec/fixtures/*.plan.rb` — plan files used for loading and validation cases.

data/README.md

@@ -2,6 +2,7 @@
 
 [![CI](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml)
 [![Coverage](https://raw.githubusercontent.com/dougyouch/db-purger/badges/coverage.svg)](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml)
+[![Branch coverage](https://raw.githubusercontent.com/dougyouch/db-purger/badges/branch-coverage.svg)](https://github.com/dougyouch/db-purger/actions/workflows/ci.yml)
 [![Gem Version](https://img.shields.io/gem/v/db-purger)](https://rubygems.org/gems/db-purger)
 
 Purge every row tied to a single top-level record — a company, an account, a tenant — across all of the
@@ -112,16 +113,46 @@ executor.verify!            # raises 'purge plan failed verification', errors pr
 deleted = executor.purge!(42)
 ```
 
-`purge!` returns the number of base-table rows deleted.
+`purge!` returns the number of root-table rows deleted (the base table, plus any top-level `parent_table`s).
+
+### Plans with several top-level tables
+
+`base_table` is shorthand for "one root table, with everything after it nested underneath". When several tables
+are equally top-level (an outreach product's `emails`, `sms_messages` and `calls`, all keyed by `oid`), leave
+`base_table` out and declare each root as a top-level `parent_table`:
+
+```ruby
+# config/outreach.plan.rb
+parent_table(:calls, :oid) do              # calls.email_id -> emails.id, so calls go first
+  child_table(:call_notes, :call_id)
+  child_table(:call_recordings, :call_id)
+  child_table(:call_tags, :call_id)
+end
+
+parent_table(:emails, :oid) do
+  child_table(:email_attachments, :email_id)
+end
+
+parent_table(:sms_messages, :oid) do
+  child_table(:sms_deliveries, :sms_message_id)
+end
+
+ignore_table :users
+```
+
+Each root is purged by `oid = purge_value`, children first, **in declaration order**: when one root's rows
+reference another's, declare the referencing root first. Without a `base_table`, top-level `child_table`s are
+an error (there is no enclosing batch to take ids from). Existing `base_table` plans run exactly as before.
 
 ## The plan DSL
 
 | Method | Meaning |
 |---|---|
-| `base_table(table, field, opts = {}, &block)` | The root of the purge. Rows where `field = purge_value` are purged. Declare it **first** — every subsequent top-level call nests under it. |
+| `base_table(table, field, opts = {}, &block)` | Optional single root. Rows where `field = purge_value` are purged. Declare it **first** — every subsequent top-level call nests under it. |
 | `child_table(table, field, opts = {}, &block)` | Rows whose `field` matches the **primary key** of the enclosing table's current batch. Purged before that batch is deleted. |
 | `child_table(table, :id, foreign_key: :col, &block)` | Inverted relationship: the *enclosing* table holds `col` pointing at this table's `id`. Deleted in the same transaction, right after the enclosing batch. |
-| `parent_table(table, field, opts = {}, &block)` | Rows whose `field` matches the original **purge value**. Purged after the enclosing table's child tables but before the enclosing table's own rows, so it may both reference the base (`company_tags.company_id → companies.id`) and be referenced by a child table. Use for sibling tables that share the same key (e.g. `company_id`). |
+| `parent_table(table, field, opts = {}, &block)` | Rows whose `field` matches the original **purge value**. At the top level of a plan without a `base_table`, each one is a root, purged in declaration order. Purged after the enclosing table's child tables but before the enclosing table's own rows, so it may both reference the base (`company_tags.company_id → companies.id`) and be referenced by a child table. Use for sibling tables that share the same key (e.g. `company_id`). |
+| `nullify_table(table, field, conditions: nil)` | Rows whose `field` matches the **primary key** of the enclosing table's current batch get `field = NULL` instead of being deleted, before anything in that batch is deleted (`ON DELETE SET NULL` at purge time). Use for optional references that must not take the referencing row down with them: a self-referential `parent_id`/"copied from" column, or a link from another tenant's row. Only `conditions:` is supported. |
 | `purge_table_search(table, field, opts = {}) { \|batch\| ... }` | Scans the whole table in batches; the block receives each batch and returns the records to purge. For orphans that can't be reached by a key. |
 | `ignore_table(name_or_regexp)` | Exclude a table from validation. |
 
@@ -166,14 +197,25 @@ plan.purge!(database, 42)
 
 ### Generating a starting plan
 
-`DynamicPlanBuilder` walks the `has_many` associations dynamic-active-model discovered and emits a plan file,
-listing every unreachable table as `ignore_table`. Treat the output as a first draft: it only knows about
-conventional `<singular_table>_id` foreign keys and cannot infer polymorphic, soft-delete, or search rules.
+`DynamicPlanBuilder` walks the `has_many`, `has_one` and `has_and_belongs_to_many` associations
+dynamic-active-model discovered, using each association's real foreign key, and emits a plan file listing every
+unreachable table as `ignore_table`.
 
 ```ruby
-puts DBPurger::DynamicPlanBuilder.new(database).build(:companies, :id)
+builder = DBPurger::DynamicPlanBuilder.new(database)
+puts builder.build(:companies, :id)   # single base_table plan
+puts builder.build_for(:oid)          # one top-level parent_table per table holding oid
 ```
 
+`build_for` makes **every** table holding the field a root, so rows with a null foreign key to another root
+(an `email_recipients` row without an email) are still purged, and orders the roots so a root referencing
+another root's rows comes first. HABTM join tables are emitted as leaves, never walking into the shared table on
+the other side, and nothing is nested under a table without a primary key. A foreign-key cycle is written as a
+comment instead of recursing.
+
+Treat the output as a first draft: it cannot infer polymorphic (`as:`), soft-delete, `belongs_to`-owned
+(`foreign_key:`) or search rules.
+
 ## Validation
 
 `Executor#verify!` (or `DBPurger::PlanValidator.new(database, plan).valid?`) checks that:
@@ -181,8 +223,11 @@ puts DBPurger::DynamicPlanBuilder.new(database).build(:companies, :id)
 - every table in the database is either in the plan or ignored (`missing_tables`)
 - every table in the plan exists in the database (`unknown_tables`)
 - every field, `foreign_key` and `mark_deleted_field` named in the plan is a real column
-- the plan has a `base_table`, and every `batch_size` is positive
-- tables without a primary key have no nested child or parent tables (there would be no ids to propagate)
+- the plan has a `base_table` or at least one top-level `parent_table`, no top-level `child_table` is left
+  unreachable, and every `batch_size` is positive
+- tables without a primary key have no nested child, parent or nullify tables (there would be no ids to propagate)
+- every `nullify_table` field is a nullable column, and no `nullify_table` is left at the top level without a
+  `base_table`
 
 Run it in CI against your schema so a new table can't ship without a purge decision.
 
@@ -220,6 +265,7 @@ DBPurger::MetricSubscriber.metrics.as_json
 # => { took: 12.4, started_at: ..., finished_at: ...,
 #      purge_stats:  { employments: { duration:, num_purges:, num_records: } },
 #      delete_stats: { employments: { duration:, num_delete_queries:, num_deleted:, num_expected_to_delete: } },
+#      nullify_stats: { cadences: { duration:, num_nullify_queries:, num_nullified: } },
 #      lookup_stats: { ... }, filter_stats: { ... } }
 ```
 
@@ -231,12 +277,13 @@ Metrics are reset at the start of each `Plan#purge!`. To feed your own telemetry
 | `purge.db_purger` | `table_name`, `purge_field`, `deleted` |
 | `next_batch.db_purger` | `table_name`, `start_id`, `num_records` |
 | `delete_records.db_purger` | `table_name`, `num_records`, `records_deleted`, `deleted` |
+| `nullify_records.db_purger` | `table_name`, `nullify_field`, `num_records`, `records_nullified` |
 | `search_filter.db_purger` | `table_name`, `num_records`, `num_records_selected` |
 
 ## Caveats
 
-- **Only the base table is the entry point.** Top-level `parent_table`/`child_table` calls made before
-  `base_table` are ignored by `Plan#purge!`.
+- **Declare `base_table` first.** A `child_table` declared before it is never reached (the validator reports
+  it); a `parent_table` declared before it becomes a separate root, purged after the base table.
 - **Not one big transaction.** Each batch is its own set of statements (foreign-key children share a
   transaction with their parent batch). An interrupted purge is safe to re-run with the same value.
 - **Soft-deleted rows still match.** A `mark_deleted_field` table is not filtered on that field; add
@@ -254,14 +301,14 @@ script/console
 
 CI (`.github/workflows/ci.yml`) runs RuboCop and the specs on Ruby 4.0 for every push and pull request.
 The HTML coverage report is attached to each run as the `coverage` artifact, and pushes to `master` refresh
-the coverage badge on the `badges` branch.
+the line and branch coverage badges on the `badges` branch.
 
 See [ARCHITECTURE.md](ARCHITECTURE.md) for how the pieces fit together.
 
 ## Releasing
 
 1. Bump `s.version` in `db-purger.gemspec` and merge to `master`.
-2. Tag and push: `git tag v0.5.0 && git push origin v0.5.0`
+2. Tag and push: `git tag v0.7.0 && git push origin v0.7.0`
 
 `.github/workflows/release.yml` re-runs CI, checks the tag matches the gemspec version, publishes to RubyGems
 via trusted publishing (no API key), and creates a GitHub release with the `.gem` attached.

data/lib/db-purger/association_graph.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+module DBPurger
+  # DBPurger::AssociationGraph answers "which tables reference this model, and by which column" from the
+  # has_many, has_one and has_and_belongs_to_many associations dynamic-active-model discovered
+  class AssociationGraph
+    # a table holding foreign_key that points at the parent model's primary key
+    Edge = Struct.new(:model, :foreign_key)
+
+    def initialize(database)
+      @database = database
+      @edges = {}
+    end
+
+    # database.models order depends on how the adapter lists tables, which varies by platform;
+    # sort so generated plans are deterministic
+    def models
+      @models ||= @database.models.sort_by(&:table_name)
+    end
+
+    def model_for(table_name)
+      models.detect { |model| model.table_name == table_name.to_s }
+    end
+
+    # one edge per (table, foreign key); a habtm join table also reached by a has_many appears once
+    def edges(model)
+      @edges[model] ||= model.reflect_on_all_associations
+                             .filter_map { |reflection| edge_for(reflection) }
+                             .uniq { |edge| edge_key(edge) }
+                             .sort_by { |edge| edge_key(edge) }
+    end
+
+    # every model reachable from model through edges, excluding model itself unless there is a cycle
+    def reachable(model, seen = Set.new)
+      edges(model).each do |edge|
+        next if seen.include?(edge.model)
+
+        seen << edge.model
+        reachable(edge.model, seen)
+      end
+      seen
+    end
+
+    def column?(model, field)
+      model.column_names.include?(field.to_s)
+    end
+
+    private
+
+    def edge_for(reflection)
+      return if skip_reflection?(reflection)
+
+      case reflection
+      when ActiveRecord::Reflection::HasAndBelongsToManyReflection
+        join_table_edge(reflection)
+      when ActiveRecord::Reflection::HasManyReflection, ActiveRecord::Reflection::HasOneReflection
+        Edge.new(reflection.klass, reflection.foreign_key.to_s)
+      end
+    end
+
+    # through associations are reached via their own direct associations; polymorphic (as:) ones need a
+    # type condition the generator cannot infer
+    def skip_reflection?(reflection)
+      reflection.options[:through] || reflection.options[:as]
+    end
+
+    def join_table_edge(reflection)
+      join_model = model_for(reflection.join_table)
+      Edge.new(join_model, reflection.foreign_key.to_s) if join_model
+    end
+
+    def edge_key(edge)
+      [edge.model.table_name, edge.foreign_key]
+    end
+  end
+end

data/lib/db-purger/dynamic_plan_builder.rb

@@ -3,115 +3,124 @@
 module DBPurger
   # DBPurger::DynamicPlanBuilder generates a purge plan based on the database relations
   class DynamicPlanBuilder
-    INDENT = '  '
-
-    attr_reader :output
-
     def initialize(database)
-      @database = database
-      @output = ''.dup
-      @indent_depth = 0
-      @tables = []
+      @graph = AssociationGraph.new(database)
+      @writer = PlanWriter.new
+    end
+
+    def output
+      @writer.output
     end
 
+    # plan rooted at a single base table
     def build(base_table_name, field)
-      write_table('base', base_table_name.to_s, field, [], nil)
-      line_break
-      model = find_model_for_table(base_table_name)
-      foreign_key = foreign_key_name(model)
+      model = @graph.model_for(base_table_name)
+      @writer.table('base', model.table_name, field)
+      @writer.line_break
       if model.primary_key == field.to_s
-        add_parent_tables(base_table_name, foreign_key)
+        add_referencing_parent_tables(model)
       else
-        add_parent_tables(base_table_name, field)
-        unless (child_models = find_child_models(model, foreign_key)).empty?
-          line_break unless field == :id
-          add_child_tables(child_models, foreign_key, 0)
-        end
+        add_sibling_parent_tables(model, field)
+        add_base_child_tables(model)
       end
-      ignore_missing_tables
-      @output
+      finish
+    end
+
+    # plan with one top-level parent_table per table holding field (e.g. :oid), each purged by field
+    # directly so rows with a null foreign key are not missed; the tables referencing each root are nested
+    # under it, and roots are ordered so a root referencing another root's rows is purged first
+    def build_for(field)
+      @root_field = field.to_s
+      ordered_root_models.each_with_index do |model, idx|
+        @writer.line_break if idx.positive?
+        write_table('parent', model, field, [])
+      end
+      finish
     end
 
     private
 
-    def find_model_for_table(base_table_name)
-      @database.models.detect { |m| m.table_name == base_table_name.to_s }
+    def finish
+      @writer.ignore_tables(@graph.models.map(&:table_name) - @writer.table_names)
+      output
     end
 
-    def write(str)
-      @output << "#{INDENT * @indent_depth}#{str}\n"
+    # base_table(:companies, :id): tables holding companies.id are keyed directly on the purge value
+    def add_referencing_parent_tables(model)
+      @graph.edges(model).each { |edge| write_edge('parent', edge, [model]) }
     end
 
-    def line_break
-      @output << "\n"
+    # base_table(:employments, :company_id): other tables holding company_id share the purge value
+    def add_sibling_parent_tables(model, field)
+      @graph.models.each do |sibling|
+        next if sibling == model || !@graph.column?(sibling, field)
+
+        write_table('parent', sibling, field, [model])
+      end
     end
 
-    def add_parent_tables(base_table_name, field)
-      sorted_models.each do |model|
-        next if model.table_name == base_table_name.to_s
-        next unless column?(model, field)
+    def add_base_child_tables(model)
+      edges = nestable_edges(model)
+      return if edges.empty?
 
-        foreign_key = foreign_key_name(model)
-        write_table('parent', model.table_name, field, find_child_models(model, foreign_key), foreign_key)
-      end
+      @writer.line_break
+      edges.each { |edge| write_edge('child', edge, [model]) }
     end
 
-    def add_child_tables(child_models, field, change_indent_by = 1)
-      @indent_depth += change_indent_by
-      child_models.each do |model|
-        add_child_table(model, field)
+    def write_edge(table_type, edge, ancestors)
+      if ancestors.include?(edge.model)
+        @writer.comment("#{table_type}_table(#{edge.model.table_name.to_sym.inspect}, " \
+                        "#{edge.foreign_key.to_sym.inspect}) skipped: cycle back to #{edge.model.table_name}")
+      else
+        write_table(table_type, edge.model, edge.foreign_key, ancestors)
       end
-      @indent_depth -= change_indent_by
     end
 
-    def add_child_table(model, field)
-      foreign_key = foreign_key_name(model)
-      write_table('child', model.table_name, field, find_child_models(model, foreign_key), foreign_key)
+    def write_table(table_type, model, field, ancestors)
+      edges = nestable_edges(model)
+      if edges.empty?
+        @writer.table(table_type, model.table_name, field)
+        warn_unnestable(model)
+      else
+        @writer.table_block(table_type, model.table_name, field) do
+          edges.each { |edge| write_edge('child', edge, ancestors + [model]) }
+        end
+      end
     end
 
-    def find_child_models(model, field)
-      model_has_many_associations(model).map(&:klass).select { |m| column?(m, field) }.sort_by(&:table_name)
-    end
+    # purging nested tables needs this table's primary keys to propagate
+    def nestable_edges(model)
+      return [] unless model.primary_key
 
-    # database.models order depends on how the adapter lists tables, which varies by platform;
-    # sort so the generated plan is deterministic
-    def sorted_models
-      @sorted_models ||= @database.models.sort_by(&:table_name)
+      @graph.edges(model).reject { |edge| root_model?(edge.model) }
     end
 
-    def model_has_many_associations(model)
-      model.reflect_on_all_associations.select do |assoc|
-        assoc.is_a?(ActiveRecord::Reflection::HasManyReflection)
-      end
+    def root_model?(model)
+      @root_field && @graph.column?(model, @root_field)
     end
 
-    def foreign_key_name(model)
-      "#{model.table_name.singularize}_id"
-    end
+    def warn_unnestable(model)
+      return if model.primary_key || (edges = @graph.edges(model)).empty?
 
-    def column?(model, field)
-      model.columns.detect { |c| c.name == field.to_s } != nil
+      @writer.comment("#{model.table_name} has no primary key; cannot nest " \
+                      "#{edges.map { |edge| edge.model.table_name }.join(', ')}")
     end
 
-    def write_table(table_type, table_name, field, child_models, foreign_key)
-      @tables << table_name
-      if child_models.empty?
-        write("#{table_type}_table(#{table_name.to_sym.inspect}, #{field.to_sym.inspect})")
-      else
-        write("#{table_type}_table(#{table_name.to_sym.inspect}, #{field.to_sym.inspect}) do")
-        add_child_tables(child_models, foreign_key)
-        write('end')
+    # repeatedly take the first root (by name) whose referencing roots are already written; on a cycle,
+    # fall back to name order so no root is dropped
+    def ordered_root_models
+      remaining = @graph.models.select { |model| root_model?(model) }
+      ordered = []
+      until remaining.empty?
+        model = remaining.detect { |root| (purged_first(root) & remaining).empty? } || remaining.first
+        ordered << remaining.delete(model)
       end
+      ordered
     end
 
-    def ignore_missing_tables
-      missing_tables = sorted_models.map(&:table_name) - @tables
-      return if missing_tables.empty?
-
-      line_break
-      missing_tables.each do |table_name|
-        write("ignore_table #{table_name.to_sym.inspect}")
-      end
+    # roots holding rows that reference root's rows (directly or through nested tables)
+    def purged_first(root)
+      @graph.reachable(root).select { |model| model != root && root_model?(model) }
     end
   end
 end

data/lib/db-purger/metric_subscriber.rb

@@ -36,6 +36,14 @@ module DBPurger
       )
     end
 
+    def nullify_records(event)
+      self.class.metrics.update_nullify_records_stats(
+        event.payload[:table_name],
+        event.duration,
+        event.payload[:records_nullified] || 0
+      )
+    end
+
     def next_batch(event)
       self.class.metrics.update_lookup_stats(
         event.payload[:table_name],

data/lib/db-purger/metrics.rb

@@ -7,6 +7,7 @@ module DBPurger
                 :finished_at,
                 :purge_stats,
                 :delete_stats,
+                :nullify_stats,
                 :lookup_stats,
                 :filter_stats
 
@@ -18,6 +19,7 @@ module DBPurger
       @started_at = Time.now
       @purge_stats = {}
       @delete_stats = {}
+      @nullify_stats = {}
       @lookup_stats = {}
       @filter_stats = {}
       @finished_at = nil
@@ -48,6 +50,14 @@ module DBPurger
       stats
     end
 
+    def update_nullify_records_stats(table_name, duration, num_nullified)
+      stats = (@nullify_stats[table_name] ||= Hash.new(0))
+      stats[:duration] += duration
+      stats[:num_nullify_queries] += 1
+      stats[:num_nullified] += num_nullified
+      stats
+    end
+
     def update_lookup_stats(table_name, duration, records_found)
       stats = (@lookup_stats[table_name] ||= Hash.new(0))
       stats[:duration] += duration
@@ -72,6 +82,7 @@ module DBPurger
         finished_at: @finished_at,
         purge_stats: @purge_stats,
         delete_stats: @delete_stats,
+        nullify_stats: @nullify_stats,
         lookup_stats: @lookup_stats,
         filter_stats: @filter_stats
       }

data/lib/db-purger/nullify_table.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module DBPurger
+  # DBPurger::NullifyTable clears a nullable column that points at a batch of rows about to be deleted,
+  # the purge-time equivalent of ON DELETE SET NULL. It updates the referencing rows instead of deleting them.
+  class NullifyTable
+    include PurgeTableHelper
+
+    def initialize(database, table, purge_values)
+      @database = database
+      @table = table
+      @purge_values = purge_values
+    end
+
+    def nullify!
+      ActiveSupport::Notifications.instrument('nullify_records.db_purger',
+                                              table_name: @table.name,
+                                              nullify_field: @table.field,
+                                              num_records: @purge_values.size) do |payload|
+        payload[:records_nullified] = ::DBPurger.config.explain? ? explain_nullify : nullify_records
+      end
+    end
+
+    private
+
+    def nullify_records
+      scope.update_all(@table.field => nil)
+    end
+
+    def scope
+      scope = model.where(@table.field => @purge_values)
+      scope = scope.where(@table.conditions) if @table.conditions
+      scope
+    end
+
+    def explain_nullify
+      ::DBPurger.config.explain_file.puts("#{explain_update_sql(scope, field_quoted, 'NULL')};")
+      scope.count
+    end
+
+    def field_quoted
+      model.connection.quote_column_name(@table.field)
+    end
+  end
+end

data/lib/db-purger/plan.rb

@@ -8,39 +8,48 @@ module DBPurger
     attr_reader :parent_tables,
                 :child_tables,
                 :ignore_tables,
-                :search_tables
+                :search_tables,
+                :nullify_tables
 
     def initialize
       @parent_tables = []
       @child_tables = []
       @ignore_tables = []
       @search_tables = []
+      @nullify_tables = []
     end
 
     def purge!(database, purge_value)
-      raise('plan has no base_table') unless @base_table
+      raise('plan has no base_table or top-level parent_table') if root_tables.empty?
+      raise('top-level child_tables require a base_table') unless @base_table || @child_tables.empty?
+      raise('top-level nullify_tables require a base_table') unless @base_table || @nullify_tables.empty?
 
       MetricSubscriber.reset!
-      num_deleted = PurgeTable.new(database, @base_table, @base_table.field, purge_value).purge!
+      num_deleted = purge_root_tables(database, purge_value)
+      purge_search_tables(database)
       MetricSubscriber.finished!
       num_deleted
     end
 
+    # tables that receive the purge value directly: the base_table (if any) and top-level parent_tables
+    def root_tables
+      (@base_table ? [@base_table] : []) + @parent_tables
+    end
+
     def tables
       all_tables = @base_table ? [@base_table] + @base_table.tables : []
       all_tables += @parent_tables + @parent_tables.map(&:tables) +
                     @child_tables + @child_tables.map(&:tables) +
-                    @search_tables + @search_tables.map(&:tables)
+                    @search_tables + @search_tables.map(&:tables) +
+                    @nullify_tables
       all_tables.flatten!
       all_tables.compact!
       all_tables
     end
 
+    # the tables of a nested plan (nested plans never have a base_table)
     def foreign_tables
-      (@base_table ? [@base_table] : []) +
-        @parent_tables +
-        @child_tables +
-        @search_tables
+      @parent_tables + @child_tables + @search_tables
     end
 
     def table_names
@@ -51,7 +60,8 @@ module DBPurger
       @base_table.nil? &&
         @parent_tables.empty? &&
         @child_tables.empty? &&
-        @search_tables.empty?
+        @search_tables.empty? &&
+        @nullify_tables.empty?
     end
 
     def ignore_table?(table_name)
@@ -63,5 +73,20 @@ module DBPurger
         end
       end
     end
+
+    private
+
+    def purge_root_tables(database, purge_value)
+      root_tables.sum do |table|
+        PurgeTable.new(database, table, table.field, purge_value).purge!
+      end
+    end
+
+    # with a base_table these live in its nested plan and are purged by it
+    def purge_search_tables(database)
+      @search_tables.each do |table|
+        PurgeTableScanner.new(database, table).purge!
+      end
+    end
   end
 end

data/lib/db-purger/plan_builder.rb

@@ -3,6 +3,8 @@
 module DBPurger
   # DBPurger::PlanBuilder is used to build the relationships between tables in a convenient way
   class PlanBuilder
+    NULLIFY_TABLE_OPTIONS = %i[conditions].freeze
+
     def initialize(plan)
       @plan = plan
     end
@@ -31,6 +33,22 @@ module DBPurger
       table
     end
 
+    # rows whose field matches the enclosing batch's primary keys get field set to NULL instead of being deleted
+    def nullify_table(table_name, field, options = {})
+      unsupported_options = options.keys - NULLIFY_TABLE_OPTIONS
+      unless unsupported_options.empty?
+        raise(ArgumentError, "nullify_table does not support #{unsupported_options.map(&:inspect).join(', ')}")
+      end
+
+      table = create_table(table_name, field, options)
+      if @plan.base_table
+        @plan.base_table.nested_plan.nullify_tables << table
+      else
+        @plan.nullify_tables << table
+      end
+      table
+    end
+
     def ignore_table(table_name)
       @plan.ignore_tables << table_name
     end

data/lib/db-purger/plan_validator.rb

@@ -11,6 +11,7 @@ module DBPurger
     validate :validate_no_missing_tables
     validate :validate_no_unknown_tables
     validate :validate_tables
+    validate :validate_nullify_tables
 
     def initialize(database, plan)
       @database = database
@@ -29,8 +30,16 @@ module DBPurger
 
     private
 
+    # a plan is rooted either by a base_table or by one or more top-level parent_tables
     def validate_base_table
-      errors.add(:base_table, 'is required') unless @plan.base_table
+      if @plan.root_tables.empty?
+        errors.add(:base_table, 'or a top-level parent_table is required')
+      elsif !@plan.child_tables.empty?
+        # without a base_table there are no ids to propagate; declared before one, they are never reached
+        errors.add(:base_table, 'must be declared before top-level child_tables')
+      elsif !@plan.nullify_tables.empty?
+        errors.add(:base_table, 'must be declared before top-level nullify_tables')
+      end
     end
 
     def validate_no_missing_tables
@@ -45,6 +54,27 @@ module DBPurger
       @plan.tables.each { |table| validate_table_definition(table) }
     end
 
+    # a NOT NULL column can't be unlinked, so the purge would fail at runtime instead of here
+    def validate_nullify_tables
+      nullify_tables.each do |table|
+        next unless (model = find_model_for_table(table)) # reported by validate_tables
+        next unless not_nullable_column?(model, table.field)
+
+        errors.add(:table, "#{table.name}.#{table.field} (nullify_table) is not nullable")
+      end
+    end
+
+    # a missing column is reported by validate_tables
+    def not_nullable_column?(model, field)
+      column = model.columns_hash[field.to_s]
+      column && !column.null
+    end
+
+    def nullify_tables
+      @plan.nullify_tables +
+        @plan.tables.select(&:nested_tables?).flat_map { |table| table.nested_plan.nullify_tables }
+    end
+
     def validate_table_definition(table)
       unless (model = find_model_for_table(table))
         errors.add(:table, "#{table.name} has no model")

data/lib/db-purger/plan_writer.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+module DBPurger
+  # DBPurger::PlanWriter renders plan DSL source and tracks which tables it has written
+  class PlanWriter
+    INDENT = '  '
+
+    attr_reader :output,
+                :table_names
+
+    def initialize
+      @output = ''.dup
+      @indent_depth = 0
+      @table_names = []
+    end
+
+    def table(table_type, table_name, field)
+      @table_names << table_name
+      write(table_call(table_type, table_name, field))
+    end
+
+    def table_block(table_type, table_name, field)
+      @table_names << table_name
+      write("#{table_call(table_type, table_name, field)} do")
+      @indent_depth += 1
+      yield
+      @indent_depth -= 1
+      write('end')
+    end
+
+    def comment(str)
+      write("# #{str}")
+    end
+
+    def ignore_tables(table_names)
+      return if table_names.empty?
+
+      line_break
+      table_names.each { |table_name| write("ignore_table #{table_name.to_sym.inspect}") }
+    end
+
+    def line_break
+      @output << "\n"
+    end
+
+    private
+
+    def write(str)
+      @output << "#{INDENT * @indent_depth}#{str}\n"
+    end
+
+    def table_call(table_type, table_name, field)
+      "#{table_type}_table(#{table_name.to_sym.inspect}, #{field.to_sym.inspect})"
+    end
+  end
+end

data/lib/db-purger/purge_table.rb

@@ -13,10 +13,6 @@ module DBPurger
       @num_deleted = 0
     end
 
-    def model
-      @model ||= @database.models.detect { |m| m.table_name == @table.name.to_s }
-    end
-
     def purge!
       ActiveSupport::Notifications.instrument('purge.db_purger',
                                               table_name: @table.name,

data/lib/db-purger/purge_table_helper.rb

@@ -10,9 +10,19 @@ module DBPurger
     private
 
     def purge_nested_tables(batch)
+      nullify_tables(batch) unless @table.nested_plan.nullify_tables.empty?
       purge_child_tables(batch) unless @table.nested_plan.child_tables.empty?
     end
 
+    # unlink rows that point at this batch before anything in it is deleted
+    def nullify_tables(batch)
+      ids = batch_values(batch, model.primary_key)
+
+      @table.nested_plan.nullify_tables.each do |table|
+        NullifyTable.new(@database, table, ids).nullify!
+      end
+    end
+
     def purge_child_tables(batch)
       ids = batch_values(batch, model.primary_key)
 
@@ -69,7 +79,7 @@ module DBPurger
     def explain(scope)
       sql =
         if @table.mark_deleted_field
-          explain_update_sql(scope)
+          explain_update_sql(scope, mark_deleted_field_quoted, mark_deleted_value_quoted)
         else
           scope.to_sql.sub(/SELECT .*?FROM/, 'DELETE FROM')
         end
@@ -77,10 +87,10 @@ module DBPurger
       scope.count
     end
 
-    def explain_update_sql(scope)
+    def explain_update_sql(scope, field_quoted, value_quoted)
       sql = scope.to_sql.dup
       sql.sub!(/SELECT .*?FROM/, 'UPDATE')
-      sql.sub!('WHERE', "SET #{mark_deleted_field_quoted} = #{mark_deleted_value_quoted} WHERE")
+      sql.sub!('WHERE', "SET #{field_quoted} = #{value_quoted} WHERE")
       sql
     end
 

data/lib/db-purger/purge_table_scanner.rb

@@ -11,10 +11,6 @@ module DBPurger
       @num_deleted = 0
     end
 
-    def model
-      @model ||= @database.models.detect { |m| m.table_name == @table.name.to_s }
-    end
-
     def purge!
       ActiveSupport::Notifications.instrument('purge.db_purger',
                                               table_name: @table.name) do |payload|

data/lib/db-purger/table.rb

@@ -38,7 +38,8 @@ module DBPurger
 
     # nested tables that depend on this table's rows (search tables scan independently)
     def nested_key_tables?
-      nested_tables? && !(@nested_plan.child_tables.empty? && @nested_plan.parent_tables.empty?)
+      nested_tables? &&
+        !(@nested_plan.child_tables.empty? && @nested_plan.parent_tables.empty? && @nested_plan.nullify_tables.empty?)
     end
 
     def tables

data/lib/db-purger.rb

@@ -2,17 +2,20 @@
 
 # DBPurger is a tool to delete data from tables based on a initial purge value
 module DBPurger
+  autoload :AssociationGraph, 'db-purger/association_graph'
   autoload :Config, 'db-purger/config'
   autoload :DynamicPlanBuilder, 'db-purger/dynamic_plan_builder'
   autoload :Executor, 'db-purger/executor'
   autoload :Metrics, 'db-purger/metrics'
   autoload :MetricSubscriber, 'db-purger/metric_subscriber'
+  autoload :NullifyTable, 'db-purger/nullify_table'
   autoload :PurgeTable, 'db-purger/purge_table'
   autoload :PurgeTableHelper, 'db-purger/purge_table_helper'
   autoload :PurgeTableScanner, 'db-purger/purge_table_scanner'
   autoload :Plan, 'db-purger/plan'
   autoload :PlanBuilder, 'db-purger/plan_builder'
   autoload :PlanValidator, 'db-purger/plan_validator'
+  autoload :PlanWriter, 'db-purger/plan_writer'
   autoload :Table, 'db-purger/table'
 
   # The config in effect for the current thread: the one set by with_config, else the global default

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: db-purger
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.7.0
 platform: ruby
 authors:
 - Doug Youch
@@ -29,14 +29,20 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.7'
+        version: '0.9'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.9.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.7'
+        version: '0.9'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.9.1
 description: DB Purger deletes (or soft-deletes) every row related to a single top-level
   record (e.g. a company or account) using a declarative Ruby purge plan. Tables are
   purged in primary-key batches, child tables before their parents, with plan validation
@@ -51,14 +57,17 @@ files:
 - LICENSE.txt
 - README.md
 - lib/db-purger.rb
+- lib/db-purger/association_graph.rb
 - lib/db-purger/config.rb
 - lib/db-purger/dynamic_plan_builder.rb
 - lib/db-purger/executor.rb
 - lib/db-purger/metric_subscriber.rb
 - lib/db-purger/metrics.rb
+- lib/db-purger/nullify_table.rb
 - lib/db-purger/plan.rb
 - lib/db-purger/plan_builder.rb
 - lib/db-purger/plan_validator.rb
+- lib/db-purger/plan_writer.rb
 - lib/db-purger/purge_table.rb
 - lib/db-purger/purge_table_helper.rb
 - lib/db-purger/purge_table_scanner.rb