exwiw 0.5.0 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 224bdc1d3b0f94e08463ad9e42a6e67d0592d902388b5873f5840226dbdbd3fe
-  data.tar.gz: de9ddd4a625565e0bcd28ff3f74df8da06092c443ad1f170d41c5858a24c4802
+  metadata.gz: 567683d65df5d9f147ab9415a67baf48a80e21ad32e1ef7635c624dfc3d28c47
+  data.tar.gz: 1513b577f6f2368df60edc45a54c96495ece4f1ee9b453e92adb8991f182fcdf
 SHA512:
-  metadata.gz: '08f564c07c09561a4b9b825bb7f6ca43a076df5b8262f165addad471639084fa5b5074330215edd36951ce0c427f510533f39d03e0632c1306ba4e9054391b33'
-  data.tar.gz: 2c161f236a676a15774fb097a7a2c4d66f95f38be8c465dfc29788b4c45165aa3b13dee2b1da771e317672e63f089d2496e10cf4fe2ebf16f97885e0e1c49c76
+  metadata.gz: a9680642eb34f99ed3f0c2924154a5171edf541286dfed14befe15b8c029271419a13d3295694847b1143898b0200bf6eb1d6448e6665dbad7bef026e3c3fbbb
+  data.tar.gz: 30e6ef9f988965b85f899fdb0646b6e4e2befd68f95a247a9edfeddb8d3a6088f611f07d5376c7d3b9f4f58f147072e03294a140312dec1622994fb6da175720

data/CHANGELOG.md

@@ -2,6 +2,21 @@
 
 ## [Unreleased]
 
+## [0.5.2] - 2026-06-18
+
+### Fixed
+
+- **PostgreSQL: an extension the restore target cannot create no longer aborts the restore, and `pglogical` is never emitted.** `dump_schema` prepends `CREATE EXTENSION IF NOT EXISTS` for every extension installed on the source, wrapped in a `DO` block that previously swallowed only `feature_not_supported` (the extension's binaries are unavailable on the target). A source on managed Postgres/AlloyDB carrying `pglogical` (logical replication) emitted `CREATE EXTENSION ... SCHEMA "pglogical"`, which on a target lacking that schema fails with `invalid_schema_name` — an error the handler did not catch, so the whole restore aborted. The handler now also catches `invalid_schema_name`, and instead of silently discarding the skip it re-raises it as a `WARNING` (carrying SQLSTATE and the original message) so the skip is visible in the restore logs rather than vanishing. `insufficient_privilege` is intentionally **not** caught: a restore role lacking `CREATE` privilege is a misconfiguration that must fail loudly. Separately, `pglogical` is now excluded from the prepended extensions entirely (alongside `plpgsql` and the `google_*`/`rds_*`/`aiven_*` managed-platform extensions) — it is a replication mechanism of the source, not part of the copied data.
+- **Table processing order no longer aborts on `belongs_to` cycles.** Two distinct problems made `export`/`explain` fail with "Circular belongs_to dependency detected" on schemas that have no resolvable order: (1) a `belongs_to` whose target table is **not part of the run** — most commonly an embedded MongoDB collection, which is masked through its parent and never dumped on its own — was treated as a dependency that could never be satisfied, so every table that (transitively) referenced one froze and was misreported as a cycle; such out-of-run targets are now ignored when ordering. (2) A **genuine** cycle (e.g. `a belongs_to b` and `b belongs_to a`) now **breaks deterministically with a warning** instead of raising: exwiw emits the cycle member (a table in a strongly-connected component of the unresolved-dependency graph) with the fewest unresolved dependencies, preferring one that still has an already-ordered parent so its extraction stays constrained, and logs which `belongs_to` edge was dropped. The dropped edge is not enforced while ordering, so for the mongodb adapter that table may match a superset of rows (the not-yet-processed parent contributes no `$in` filter); mark one of the `belongs_to` entries forming the cycle with `ignore: true` to break it explicitly instead. Acyclic tables that merely wait on a cycle are never reordered ahead of their parents.
+- **MongoDB: `dump_schema` tolerates collections declared in the schema but absent from the source database.** Listing indexes for a non-existent collection makes the driver raise `NamespaceNotFound` (code 26), which aborted the whole export when the schema covered more collections than the connected database actually had (schema/DB drift, or a sparse development database). The existing collections are now resolved once up front and indexes are emitted only for those; `createCollection` is still emitted for every configured collection, so the target schema is created in full.
+- **MongoDB: a related collection that cannot be scoped no longer falls back to dumping the whole collection.** When a non-target collection's `belongs_to` parents all yield no ids to filter by — because every parent matched nothing or is not dumped on its own (e.g. an embedded collection) — the assembled filter is empty. Previously that empty filter was sent as `find({})`, scanning and dumping the **entire collection across every scope** (a cross-scope data-exposure risk). Such a collection is now constrained to match no rows and a warning is logged instead. A collection with no `belongs_to` at all is still treated as reference/master data and dumped in full.
+
+## [0.5.1] - 2026-06-18
+
+### Added
+
+- **Scope-column extraction mode** (`--scope-column`, SQL adapters only). For schemas where many independent top-level tables share the same scope/tenant column instead of converging on a single `belongs_to` root, exwiw can now filter **every** table by that shared column (`--scope-column=COLUMN` with `--ids` as its values) rather than anchoring on one `--target-table`. A table that carries the column is filtered directly; a table that lacks it but `belongs_to` a table that has it is joined up to the nearest such table and filtered there. A table that `belongs_to` a parent which is itself scoped but carries no scope column of its own (e.g. a *hub* table scoped only because an extractable child references it) is constrained to the parent's in-scope ids via a subquery (`fk IN (SELECT parent.pk FROM <parent's scoped query>)`), so the hub's other children ride along to just the in-scope rows — limited to a single forward hop and a single unambiguous scopable parent. A table that cannot be scoped at all (no column and no path to one) makes the run **abort with a list of the offending tables**, so an unscoped table is never silently dumped in full. Two user-owned table-config keys support this and are preserved across `schema:generate` regeneration: **`scope_exempt: true`** exports a genuine reference/master table in full (rails-managed tables are treated as exempt automatically), and **`scope_column`** overrides the filtered column name for a table that stores the same scope value under a different name. `--scope-column` is mutually exclusive with `--target-table`, `--target-collection`, `--ids-column`, and `--ids-field`, can be set in `exwiw.yml`, and works with `exwiw explain`.
+
 ## [0.5.0] - 2026-06-16
 
 ### Added

data/README.md

@@ -129,6 +129,88 @@ exwiw explain \
 
 The `--output-dir`, `--output-format`, `--insert-only`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
 
+### Scope-column mode (`--scope-column`)
+
+The default `--target-table` extraction assumes the schema converges on a single
+root: every table is reached by walking `belongs_to` toward that one table. Some
+schemas are not shaped that way — many independent top-level tables each carry the
+*same* scope/tenant column (e.g. `tenant_id`, `account_uuid`) and there is no
+single root. Choosing one of them as `--target-table` would leave the others
+unrelated to it, and an unrelated table is dumped in full — a problem if it holds
+personal data.
+
+`--scope-column` handles this shape: instead of one anchor table, **every table is
+filtered by a shared column** whose values are `--ids`.
+
+```bash
+exwiw \
+  --adapter=postgresql \
+  --host=localhost --port=5432 --user=reader \
+  --database=app_production \
+  --schema-dir=exwiw/schema \
+  --scope-column=tenant_id \
+  --ids=42,43 \
+  --output-dir=dump
+```
+
+Each table is resolved as follows:
+
+- **Carries the scope column** → `WHERE scope_column IN (ids)`.
+- **Lacks it but `belongs_to` reaches a table that has it** → exwiw joins up to the
+  nearest such table and applies the scope filter there (the same join machinery
+  the single-target mode uses).
+- **`belongs_to` a parent that is itself scoped but carries no scope column of its
+  own** → exwiw constrains this table to the parent's in-scope ids via a subquery
+  (`fk IN (SELECT parent.pk FROM <parent's scoped query>)`). This covers a *hub*
+  table that has no scope column and is scoped only because an extractable child
+  references it (see referenced-by below): the hub's other `belongs_to` children
+  ride along to just the in-scope rows instead of being dumped in full. Limited to
+  a single forward hop and a single unambiguous scopable parent.
+- **Cannot be scoped at all** (no scope column and no path to one) → exwiw
+  **aborts** and lists the offending tables, so an unscoped table is never silently
+  dumped in full. For each, either add a `belongs_to` path, set `ignore: true` to
+  skip it, or mark it `scope_exempt: true` (below) to export it in full.
+
+`--scope-column` is SQL-only (mysql / postgresql / sqlite) and mutually exclusive
+with `--target-table`, `--target-collection`, `--ids-column`, and `--ids-field`.
+It works with `exwiw explain` too, which is the recommended way to preview the
+queries before exporting.
+
+#### `scope_exempt` (intentional full dump)
+
+A genuine reference/master table (no personal data) that has no scope linkage can
+opt out of the strict check and be exported in full:
+
+```json
+{
+  "name": "countries",
+  "primary_key": "id",
+  "scope_exempt": true,
+  "columns": [{ "name": "id" }, { "name": "code" }]
+}
+```
+
+Rails-managed tables (`schema_migrations`, `ar_internal_metadata`) are treated as
+exempt automatically.
+
+#### Per-table `scope_column` override
+
+scope-column mode assumes a single shared **value** space — the same `--ids` apply
+to every scoped table. If a table stores that same value under a differently named
+column, override the column name for that table:
+
+```json
+{
+  "name": "legacy_orders",
+  "primary_key": "id",
+  "scope_column": "legacy_tenant_id",
+  "columns": [{ "name": "id" }, { "name": "legacy_tenant_id" }]
+}
+```
+
+Both `scope_exempt` and `scope_column` are user-maintained and preserved across
+`schema:generate` regeneration (the generators never emit them).
+
 ### Config file (`exwiw.yml`)
 
 Options you would otherwise repeat on every run can be kept in a YAML config file. Pass it with `--config=PATH`; when `--config` is omitted, exwiw automatically loads `exwiw.yml` (or `exwiw.yaml`) from the current directory if present.
@@ -144,7 +226,7 @@ output_format: insert        # insert | copy
 insert_only: false
 after_insert_hook: hooks/seed.rb
 log_level: info              # debug | info
-# target_table / ids / ids_field / ids_column may also be set here
+# target_table / ids / ids_field / ids_column / scope_column may also be set here
 ```
 
 With the file above, only the connection details need to be supplied on the CLI:

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'json'
+require 'set'
 
 # NOTE: This adapter consumes MongodbCollectionConfig (`fields` instead of
 # `columns`, plus `embedded_in`). Top-level collections are dumped as one
@@ -71,16 +72,7 @@ module Exwiw
               { config.primary_key => { "$in" => coerce_ids(dump_target.ids) } }
             end
           else
-            config.belongs_tos.each_with_object({}) do |relation, acc|
-              # Constrain by the parent field this FK actually references
-              # (`relation.references`, default the parent primary_key). The
-              # values were captured from that field's documents in #execute, so
-              # their BSON type already matches the stored FK — no coercion.
-              values = parent_state_for(relation, config_by_name)
-              next if values.nil? || values.empty?
-
-              acc[relation.foreign_key] = { "$in" => values }
-            end
+            related_collection_filter(config, config_by_name)
           end
 
         Exwiw::MongoQuery::Find.new(
@@ -160,6 +152,14 @@ module Exwiw
 
         collections = ordered_tables.reject(&:embedded?)
 
+        # Index listing targets a specific collection, and MongoDB raises
+        # NamespaceNotFound (code 26) for one that does not exist. The schema may
+        # declare collections absent from this database (schema/DB drift, or a
+        # sparse dev DB), so resolve the set that actually exists up front and emit
+        # indexes only for those. `createCollection` is still emitted for every
+        # config below, so the target schema is created in full regardless.
+        existing_collections = db.database.collection_names.to_set
+
         File.open(output_path, 'w') do |file|
           file.puts("// Auto-generated by exwiw. Apply with: mongosh \"$MONGODB_URI\" #{File.basename(output_path)}")
           file.puts
@@ -172,6 +172,11 @@ module Exwiw
 
           collections.each do |config|
             name = config.name
+            unless existing_collections.include?(name)
+              @logger.debug("  Collection '#{name}' is not present in the source database; emitting no indexes.")
+              next
+            end
+
             indexes = db[name].indexes.to_a.reject { |idx| idx['name'] == '_id_' }
             indexes.each do |idx|
               key = idx['key']
@@ -274,6 +279,39 @@ module Exwiw
         ([config.primary_key] + referenced).uniq
       end
 
+      # Build the scoping filter for a non-target collection from its belongs_to
+      # parents' captured ids. Each belongs_to is constrained by the parent field
+      # the FK references (`relation.references`, default the parent primary_key);
+      # the values were captured from that field in #execute, so their BSON type
+      # already matches the stored FK — no coercion.
+      #
+      # A belongs_to whose parent produced no ids contributes no constraint:
+      # either the parent matched nothing, or it is not dumped here (e.g. an
+      # embedded collection, or one excluded from the run). If that leaves the
+      # filter empty even though the collection HAS belongs_to, the collection
+      # cannot be scoped from the dump target — and falling back to an empty `{}`
+      # filter would scan and dump the ENTIRE collection across every scope. That
+      # is never what a scoped extraction wants, so constrain it to match nothing
+      # and warn instead. (A collection with no belongs_to at all is genuine
+      # reference/master data and is still dumped in full via `{}`.)
+      private def related_collection_filter(config, config_by_name)
+        filter = config.belongs_tos.each_with_object({}) do |relation, acc|
+          values = parent_state_for(relation, config_by_name)
+          next if values.nil? || values.empty?
+
+          acc[relation.foreign_key] = { "$in" => values }
+        end
+
+        return filter unless filter.empty? && config.belongs_tos.any?
+
+        @logger.warn(
+          "  Collection '#{config.name}' has belongs_to but no parent produced ids to scope by " \
+          "(parents matched nothing, or are not dumped on their own such as embedded collections). " \
+          "Constraining it to match no rows to avoid an unscoped full-collection dump."
+        )
+        { config.primary_key => { "$in" => [] } }
+      end
+
       # The captured parent-collection values a child belongs_to should be
       # constrained by: the values of the parent field the FK references
       # (`relation.references`, default the parent primary_key). nil when the

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -65,7 +65,18 @@ module Exwiw
           ext_ddl = extensions.map do |extname, schema|
             stmt = "CREATE EXTENSION IF NOT EXISTS #{connection.quote_ident(extname)}"
             stmt += " SCHEMA #{connection.quote_ident(schema)}" unless schema == "public"
-            "DO $$ BEGIN #{stmt}; EXCEPTION WHEN feature_not_supported THEN NULL; END $$;"
+            # Best-effort prepend: a restore target that genuinely cannot create the
+            # extension should not abort the whole restore. Two such cases are caught:
+            #   feature_not_supported (0A000) -- the extension's binaries are unavailable
+            #   invalid_schema_name   (3F000) -- the extension's required schema is absent
+            # insufficient_privilege (42501) is deliberately NOT caught: a restore role
+            # lacking CREATE privilege is a misconfiguration to fix, not to skip silently.
+            # The skip is re-raised as a WARNING so it surfaces in the restore logs
+            # instead of vanishing.
+            warning = connection.escape_literal("exwiw: skipped CREATE EXTENSION #{extname} (SQLSTATE %): %")
+            "DO $$ BEGIN #{stmt}; " \
+              "EXCEPTION WHEN feature_not_supported OR invalid_schema_name THEN " \
+              "RAISE WARNING #{warning}, SQLSTATE, SQLERRM; END $$;"
           end.join("\n") + "\n\n"
           @logger.debug("  Found #{extensions.size} extension(s) to prepend.")
           stdout = ext_ddl + stdout
@@ -382,11 +393,17 @@ module Exwiw
       end
 
       private def query_extensions
+        # Skip plpgsql (always present) and managed-platform bookkeeping extensions
+        # (google_*/rds_*/aiven_*). pglogical is also skipped: it is a logical-
+        # replication mechanism of the source, not part of the data being copied,
+        # and its dedicated `pglogical` schema is typically absent on the restore
+        # target — so prepending CREATE EXTENSION for it only breaks the restore.
         sql = <<~SQL
           SELECT e.extname, n.nspname
           FROM pg_extension e
           JOIN pg_namespace n ON n.oid = e.extnamespace
           WHERE e.extname != 'plpgsql'
+            AND e.extname != 'pglogical'
             AND e.extname NOT LIKE 'google\\_%' ESCAPE '\\'
             AND e.extname NOT LIKE 'rds\\_%' ESCAPE '\\'
             AND e.extname NOT LIKE 'aiven\\_%' ESCAPE '\\'

data/lib/exwiw/after_insert_hook.rb

@@ -38,6 +38,7 @@ module Exwiw
         'EXWIW_DATABASE_USER'    => cli_options[:database_user].to_s,
         'EXWIW_DATABASE_NAME'    => cli_options[:database_name].to_s,
         'EXWIW_TARGET_TABLE'     => cli_options[:target_table].to_s,
+        'EXWIW_SCOPE_COLUMN'     => cli_options[:scope_column].to_s,
         'EXWIW_IDS'              => Array(cli_options[:ids]).join(','),
         'EXWIW_OUTPUT_FORMAT'    => cli_options[:output_format].to_s,
       }

data/lib/exwiw/cli.rb

@@ -35,6 +35,7 @@ module Exwiw
       ids
       ids_field
       ids_column
+      scope_column
     ].freeze
 
     # Database connection settings are environment-specific (and sometimes
@@ -77,6 +78,7 @@ module Exwiw
       @ids = []
       @ids_field = nil
       @ids_column = nil
+      @scope_column = nil
       @output_format = nil
       @insert_only = nil
       @after_insert_hook_path = nil
@@ -109,6 +111,7 @@ module Exwiw
         table_name: @target_table_name,
         ids: @ids,
         ids_field: @ids_field,
+        scope_column: @scope_column,
       )
 
       logger = build_logger
@@ -161,6 +164,7 @@ module Exwiw
       end
 
       resolve_target_collection_alias!
+      resolve_scope_column!
       resolve_ids_column_alias!
       resolve_uri_option!
 
@@ -228,8 +232,13 @@ module Exwiw
         exit 1
       end
 
-      if !@target_table_name && @ids.any?
-        $stderr.puts "--target-table is required when --ids is specified"
+      if @scope_column && @ids.empty?
+        $stderr.puts "--ids is required when --scope-column is specified"
+        exit 1
+      end
+
+      if !@target_table_name && !@scope_column && @ids.any?
+        $stderr.puts "--target-table or --scope-column is required when --ids is specified"
         exit 1
       end
 
@@ -309,6 +318,7 @@ module Exwiw
       end
       @ids_field ||= config["ids_field"]
       @ids_column ||= config["ids_column"]
+      @scope_column ||= config["scope_column"]
     end
 
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
@@ -376,6 +386,33 @@ module Exwiw
       end
     end
 
+    # `--scope-column` switches to scope-column mode: every table is filtered by a
+    # shared column (`--ids` are its values) instead of anchoring on one
+    # `--target-table`. It is SQL-only and mutually exclusive with the single-target
+    # flags. Runs after resolve_target_collection_alias! (so --target-collection is
+    # already folded into @target_table_name) and before resolve_ids_column_alias!
+    # so the clearer "cannot combine" message wins over the generic ids-column one.
+    private def resolve_scope_column!
+      return if @scope_column.nil?
+
+      sql_adapters = ["mysql", "postgresql", "sqlite"]
+      unless sql_adapters.include?(@database_adapter)
+        $stderr.puts "--scope-column is only supported by the sql adapters"
+        exit 1
+      end
+
+      if @target_table_name
+        $stderr.puts "--scope-column cannot be combined with --target-table/--target-collection"
+        exit 1
+      end
+
+      if @ids_field || @ids_column
+        flag = @ids_column ? "--ids-column" : "--ids-field"
+        $stderr.puts "--scope-column cannot be combined with #{flag}"
+        exit 1
+      end
+    end
+
     # `--uri` supplies a full connection string (e.g. `mongodb+srv://...`) and is
     # mongodb-only — the SQL adapters shell out to their own client binaries with
     # discrete host/port/user flags and have no equivalent. Runs after the
@@ -442,6 +479,7 @@ module Exwiw
         target_table: @target_table_name,
         ids: @ids.dup.freeze,
         ids_field: @ids_field,
+        scope_column: @scope_column,
         output_format: @output_format,
         insert_only: @insert_only,
         log_level: @log_level,
@@ -500,6 +538,7 @@ module Exwiw
         opts.on("--ids=[IDS]", "Comma-separated list of identifiers. Required when --target-table is given.") { |v| @ids = v.split(',') }
         opts.on("--ids-field=[FIELD]", "Field on the target collection that --ids is matched against. Defaults to the primary key. (mongodb adapter only)") { |v| @ids_field = v }
         opts.on("--ids-column=[COLUMN]", "Column on the target table that --ids is matched against. Defaults to the primary key. (sql adapters only)") { |v| @ids_column = v }
+        opts.on("--scope-column=[COLUMN]", "Filter every table by this shared column (--ids are its values) instead of a single --target-table. Tables lacking it are reached via belongs_to. SQL adapters only; mutually exclusive with --target-table.") { |v| @scope_column = v }
         opts.on("--output-format=[FORMAT]", "Output format: insert (default) or copy (PostgreSQL only, export subcommand only)") { |v| @output_format = v }
         opts.on("--insert-only", "Do not generate DELETE SQL files (export subcommand only)") { @insert_only = true }
         opts.on("--after-insert-hook=PATH", "Path to a .rb or .sh post-processing hook executed after all insert/delete files are written (export subcommand only)") do |v|

data/lib/exwiw/determine_table_processing_order.rb

@@ -1,35 +1,58 @@
 # frozen_string_literal: true
 
+require "set"
+
 module Exwiw
   module DetermineTableProcessingOrder
     module_function
 
     # @param tables [Array<Exwiw::TableConfig>] tables
+    # @param logger [Logger, nil] receives a warning when a cycle has to be broken
     # @return [Array<String>] sorted table names
-    def run(tables)
+    def run(tables, logger: nil)
       return tables.map(&:name) if tables.size < 2
 
       ordered_table_names = []
+      ordered = Set.new
 
       table_by_name = tables.each_with_object({}) do |table, acc|
         acc[table.name] = table
       end
 
+      # Only belongs_to relations whose target is also in this run constrain the
+      # order. A belongs_to pointing at a table that is not being processed here
+      # — e.g. an embedded MongoDB collection (masked through its parent, never
+      # dumped on its own) or any table excluded from the run — is not something
+      # we can or need to order against, so it must never block resolution.
+      # Without this, such a dependency would stay unresolved forever and
+      # masquerade as a circular dependency, freezing every table that
+      # (transitively) references it.
+      present_names = table_by_name.keys.to_set
+
       loop do
         break if table_by_name.empty?
 
-        tables_with_no_dependencies = table_by_name.values.select do |table|
-          not_resolved_names = compute_table_dependencies(table) - ordered_table_names - [table.name]
-
-          not_resolved_names.empty?
+        resolvable = table_by_name.values.select do |table|
+          unresolved_dependencies(table, present_names, ordered).empty?
         end
 
-        if tables_with_no_dependencies.empty?
-          raise ArgumentError, build_cycle_error_message(table_by_name, ordered_table_names)
+        if resolvable.empty?
+          # No table has all its (in-run) dependencies satisfied, yet tables
+          # remain: the belongs_to graph has a genuine cycle and no strict
+          # topological order exists. Rather than aborting the whole export, break
+          # the cycle by emitting one cycle member; see pick_cycle_victim for how
+          # the member is chosen. Warn so the dropped constraint is visible.
+          victim = pick_cycle_victim(table_by_name.values, present_names, ordered)
+          warn_cycle_break(logger, victim, unresolved_dependencies(victim, present_names, ordered))
+          resolvable = [victim]
         end
 
-        tables_with_no_dependencies.each do |table|
+        # In the normal (acyclic) path, emit every currently-resolvable table in
+        # insertion order — preserving the historical ordering the snapshot specs
+        # depend on. The cycle-break path emits exactly its single chosen victim.
+        resolvable.each do |table|
           ordered_table_names << table.name
+          ordered << table.name
           table_by_name.delete(table.name)
         end
       end
@@ -37,30 +60,124 @@ module Exwiw
       ordered_table_names
     end
 
+    # The belongs_to target table names of `table`. A polymorphic belongs_to is
+    # expanded into one entry per concrete target by schema generation, so each
+    # entry is a plain table name here.
     def compute_table_dependencies(table)
-      table.belongs_tos.each_with_object([]) do |relation, acc|
-        acc << relation.table_name
+      table.belongs_tos.map(&:table_name)
+    end
+
+    # The dependencies still blocking `table`: belongs_to targets that are part
+    # of this run, not yet ordered, and not the table itself (a self-referential
+    # belongs_to never blocks).
+    private_class_method def unresolved_dependencies(table, present_names, ordered)
+      compute_table_dependencies(table).uniq.select do |dep|
+        present_names.include?(dep) && !ordered.include?(dep) && dep != table.name
       end
     end
 
-    # When no table can be resolved but some remain, the belongs_to graph
-    # contains a cycle (e.g. A belongs_to B and B belongs_to A). A topological
-    # order cannot exist, so report the offending tables instead of looping
-    # forever.
-    private_class_method def cycle_diagnostics(table_by_name, ordered_table_names)
-      table_by_name.values.map do |table|
-        unresolved = (compute_table_dependencies(table) - ordered_table_names - [table.name]).uniq
-        "  #{table.name} -> #{unresolved.join(', ')}"
+    # Choose the next table to emit when the order is stuck in a cycle. Only
+    # genuine cycle members are eligible — a table in a non-trivial
+    # strongly-connected component of the unresolved-dependency subgraph — so an
+    # acyclic table that merely waits on a cycle is never reordered ahead of its
+    # parent. Among the members, prefer one that still has at least one
+    # already-ordered parent, so its extraction stays constrained instead of
+    # collapsing to "match every row" (a cross-scope over-extraction risk for the
+    # mongodb adapter); break remaining ties by fewest unresolved dependencies,
+    # then by name, for determinism.
+    private_class_method def pick_cycle_victim(remaining, present_names, ordered)
+      adjacency = remaining.each_with_object({}) do |table, acc|
+        acc[table.name] = unresolved_dependencies(table, present_names, ordered)
+      end
+      cyclic_names = strongly_connected_members(adjacency)
+
+      candidates = remaining.select { |table| cyclic_names.include?(table.name) }
+      candidates = remaining if candidates.empty? # defensive; a stall implies a cycle
+
+      anchored = candidates.select { |table| ordered_parent?(table, present_names, ordered) }
+      pool = anchored.empty? ? candidates : anchored
+
+      pool.min_by { |table| [unresolved_dependencies(table, present_names, ordered).size, table.name] }
+    end
+
+    # True when `table` has a belongs_to whose target was already ordered, so its
+    # extraction filter will be constrained rather than an unscoped full scan.
+    private_class_method def ordered_parent?(table, present_names, ordered)
+      compute_table_dependencies(table).any? do |dep|
+        dep != table.name && present_names.include?(dep) && ordered.include?(dep)
       end
     end
 
-    private_class_method def build_cycle_error_message(table_by_name, ordered_table_names)
-      "Circular belongs_to dependency detected among tables: " \
-        "#{table_by_name.keys.sort.join(', ')}. " \
-        "A processing order cannot be determined. " \
-        "Remove one of the belongs_to entries forming the cycle.\n" \
-        "Unresolved dependencies:\n" \
-        "#{cycle_diagnostics(table_by_name, ordered_table_names).join("\n")}"
+    # Names belonging to a non-trivial strongly-connected component (size > 1) of
+    # `adjacency` (table name -> unresolved dependency names), i.e. the genuine
+    # cycle participants. Iterative Tarjan; nodes and edges are visited in name
+    # order so the result is deterministic. Self-edges are already excluded from
+    # the adjacency, so a size-1 component is never a cycle.
+    private_class_method def strongly_connected_members(adjacency)
+      index = {}
+      low = {}
+      on_stack = {}
+      stack = []
+      counter = 0
+      members = Set.new
+      neighbors = adjacency.each_with_object({}) { |(name, deps), acc| acc[name] = deps.sort }
+
+      adjacency.keys.sort.each do |start|
+        next if index.key?(start)
+
+        work = [[start, 0]]
+        until work.empty?
+          node, edge_i = work.last
+          if edge_i.zero?
+            index[node] = counter
+            low[node] = counter
+            counter += 1
+            stack.push(node)
+            on_stack[node] = true
+          end
+
+          adj = neighbors[node] || []
+          if edge_i < adj.size
+            work.last[1] += 1
+            w = adj[edge_i]
+            next unless adjacency.key?(w) # ignore edges leaving the remaining set
+
+            if index.key?(w)
+              low[node] = [low[node], index[w]].min if on_stack[w]
+            else
+              work.push([w, 0])
+            end
+          else
+            if low[node] == index[node]
+              component = []
+              loop do
+                w = stack.pop
+                on_stack[w] = false
+                component << w
+                break if w == node
+              end
+              members.merge(component) if component.size > 1
+            end
+            work.pop
+            low[work.last[0]] = [low[work.last[0]], low[node]].min unless work.empty?
+          end
+        end
+      end
+
+      members
+    end
+
+    private_class_method def warn_cycle_break(logger, victim, dropped)
+      return if logger.nil?
+
+      logger.warn(
+        "Circular belongs_to dependency detected. Breaking it by ordering " \
+        "'#{victim.name}' before its parent table(s): #{dropped.join(', ')}. The dropped " \
+        "relationship is not enforced while ordering, so '#{victim.name}' is extracted " \
+        "without that parent constraint (the mongodb adapter may then match a superset of " \
+        "rows; SQL output may not load in foreign-key order). To break the cycle explicitly " \
+        "instead, mark one of the belongs_to entries forming it with `ignore: true`."
+      )
     end
   end
 end

data/lib/exwiw/explain_runner.rb

@@ -26,8 +26,11 @@ module Exwiw
       target = table_by_name[@dump_target.table_name]
       adapter.validate_as_dump_target!(target) if target
 
+      dumpable_configs = configs.select { |c| adapter.dumpable?(c) }
+      QueryAstBuilder.validate_scope!(dumpable_configs, table_by_name, @dump_target, @logger)
+
       @logger.debug("Determining table processing order...")
-      ordered_table_names = DetermineTableProcessingOrder.run(configs.select { |c| adapter.dumpable?(c) })
+      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs, logger: @logger)
 
       total_size = ordered_table_names.size
       ordered_table_names.each_with_index do |table_name, idx|

data/lib/exwiw/query_ast_builder.rb

@@ -2,23 +2,58 @@
 
 module Exwiw
   class QueryAstBuilder
-    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true)
-      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse).run
+    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, allow_forward: true)
+      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, allow_forward: allow_forward).run
+    end
+
+    # Scope-column mode classification for a single table. One of
+    # :exempt / :direct / :via_path / :referenced_by / :via_scoped_parent / :unscopable.
+    def self.scope_category(table_name, table_by_name, dump_target, logger)
+      new(table_name, table_by_name, dump_target, logger).scope_category
+    end
+
+    # Strict pre-flight for scope-column mode: abort if any extractable table
+    # cannot be scoped, so an unscoped (potentially sensitive) table is never
+    # silently dumped in full. No-op outside scope mode. `tables` is the set of
+    # dumpable configs (ignore:true tables are skipped — they are not extracted).
+    def self.validate_scope!(tables, table_by_name, dump_target, logger)
+      return if dump_target.scope_column.nil?
+
+      unscopable =
+        tables.reject(&:ignore).select do |table|
+          scope_category(table.name, table_by_name, dump_target, logger) == :unscopable
+        end
+      return if unscopable.empty?
+
+      names = unscopable.map(&:name).sort.join(", ")
+      raise ArgumentError,
+            "scope-column mode: #{unscopable.size} table(s) cannot be scoped by " \
+            "'#{dump_target.scope_column}': #{names}. For each, add `scope_exempt: true` " \
+            "to export it in full, set `ignore: true` to skip it, or add a belongs_to path " \
+            "to a table that carries the scope column (use a per-table `scope_column` if the " \
+            "column name differs on that table)."
     end
 
     attr_reader :table_name, :table_by_name, :dump_target
 
-    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true)
+    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, allow_forward: true)
       @table_name = table_name
       @table_by_name = table_by_name
       @dump_target = dump_target
       @logger = logger
       @allow_reverse = allow_reverse
+      # @allow_forward gates the "scope via an indirectly-scoped belongs_to
+      # parent" rescue (build_belongs_to_scoped_clause). Disabled while building a
+      # parent/child subquery so a single forward hop never recurses into another
+      # (which could loop on a belongs_to cycle).
+      @allow_forward = allow_forward
     end
 
     def run
       table = table_by_name.fetch(table_name)
 
+      return build_scoped(table) if scope_mode?
+
       where_clauses = build_where_clauses(table, dump_target)
       join_clauses = build_join_clauses(table, table_by_name, dump_target)
 
@@ -130,8 +165,10 @@ module Exwiw
         next if relation.nil? || relation.polymorphic?
 
         # Build the child's own extraction query. allow_reverse:false stops a
-        # chain of FK-less tables from recursing back into each other.
-        child_query = self.class.run(other.name, table_by_name, dump_target, @logger, allow_reverse: false)
+        # chain of FK-less tables from recursing back into each other;
+        # allow_forward:false stops the child from forward-scoping back through
+        # this very table (which would loop).
+        child_query = self.class.run(other.name, table_by_name, dump_target, @logger, allow_reverse: false, allow_forward: false)
 
         # Only an *already constrained* child narrows anything; an unconstrained
         # child would select every fk value (i.e. dump all) and not help.
@@ -169,6 +206,64 @@ module Exwiw
       )
     end
 
+    # Scope-column mode. Builds a `fk IN (SELECT parent.pk FROM <parent
+    # extraction query>)` clause for a table whose belongs_to parent is itself
+    # scopable but carries no scope column of its own — so find_path_to_scoped
+    # cannot terminate on it (via_path fails) and nothing references this table
+    # (referenced_by fails). The classic shape is a hub scoped only via
+    # referenced_by (e.g. CDP `customer_accounts`, scoped by the `customers` that
+    # reference it) with sibling detail tables (`customer_account_details`, ...)
+    # hanging off it. Constraining those siblings to the hub's in-scope ids keeps
+    # them out of a full dump. Returns nil when there is no single, unambiguous
+    # scopable parent, leaving the caller on the unscopable path.
+    private def build_belongs_to_scoped_clause(table)
+      candidates = table.belongs_tos.filter_map do |relation|
+        # A polymorphic belongs_to points at several parent tables through one
+        # column, so it cannot project to a single parent id set; skip it.
+        next if relation.polymorphic?
+
+        parent = table_by_name[relation.table_name]
+        next if parent.nil?
+
+        # Build the parent's own scoped query. allow_reverse stays true so the
+        # parent may be scoped via referenced_by; allow_forward:false bounds this
+        # to a single forward hop so a belongs_to cycle cannot loop.
+        parent_query = self.class.run(parent.name, table_by_name, dump_target, @logger, allow_reverse: true, allow_forward: false)
+
+        # Only a constrained parent narrows anything; an unconstrained parent
+        # would select every pk (i.e. dump all) and not help.
+        next unless parent_query.where_clauses.any? || parent_query.join_clauses.any?
+
+        [relation, parent, parent_query]
+      end
+
+      # Only the unambiguous single-parent case. Multiple scopable parents would
+      # need their subqueries combined (not supported); fall back to unscopable.
+      if candidates.size != 1
+        if candidates.size > 1
+          @logger.debug("  #{table.name} has multiple scopable parents; skipping forward scope (unscopable).")
+        end
+        return nil
+      end
+
+      relation, parent, parent_query = candidates.first
+
+      # Project the parent's extraction query down to just its primary key — the
+      # column this table's foreign key points at.
+      pk_column = TableColumn.from_symbol_keys(name: parent.primary_key)
+      projected = QueryAst::Select.new
+      projected.from(parent_query.from_table_name)
+      projected.select([pk_column])
+      parent_query.join_clauses.each { |j| projected.join(j) }
+      parent_query.where_clauses.each { |w| projected.where(w) }
+
+      QueryAst::WhereClause.new(
+        column_name: relation.foreign_key,
+        operator: :in_subquery,
+        value: QueryAst::SelectSubquery.new(query: projected)
+      )
+    end
+
     private def build_where_clauses(table, dump_target)
       clauses = []
 
@@ -264,5 +359,208 @@ module Exwiw
 
       queue
     end
+
+    # ------------------------------------------------------------------
+    # Scope-column mode (Exwiw::DumpTarget#scope_column).
+    #
+    # The single-target machinery above anchors everything on one named table.
+    # Scope mode instead filters every table by a shared column. The relationship
+    # walk is the same idea — the *terminus* is just "any table carrying the
+    # scope column" rather than "the one named target".
+    # ------------------------------------------------------------------
+
+    private def scope_mode?
+      !dump_target.scope_column.nil?
+    end
+
+    # Classifier used by validate_scope! and mirrored by build_scoped below.
+    def scope_category
+      table = table_by_name.fetch(table_name)
+      return :exempt if scope_exempt?(table)
+      return :direct if directly_scoped?(table)
+      return :via_path if build_join_clauses_scoped(table).any?
+      return :referenced_by if @allow_reverse && build_referenced_by_clause(table)
+      return :via_scoped_parent if @allow_forward && build_belongs_to_scoped_clause(table)
+
+      :unscopable
+    end
+
+    private def build_scoped(table)
+      ast = QueryAst::Select.new
+      ast.from(table.name)
+      if table.rails_managed?
+        ast.select_all!
+      else
+        ast.select(table.columns)
+      end
+
+      # Reference/master (or rails-managed) table: export every row.
+      return ast if scope_exempt?(table)
+
+      # Carries the scope column itself: filter on it directly.
+      if directly_scoped?(table)
+        ast.where(scope_where_clause(table))
+        ast.where(table.filter) if table.filter
+        return ast
+      end
+
+      # Reachable via belongs_to: join up to the scoped ancestor (the scope
+      # filter is applied at the terminal join inside build_join_clauses_scoped).
+      join_clauses = build_join_clauses_scoped(table)
+      unless join_clauses.empty?
+        join_clauses.each { |join_clause| ast.join(join_clause) }
+        ast.where(table.filter) if table.filter
+        return ast
+      end
+
+      if @allow_reverse
+        # Referenced by an extractable (scoped) child: constrain via subquery.
+        reverse_clause = build_referenced_by_clause(table)
+        if reverse_clause
+          ast.where(reverse_clause)
+          return ast
+        end
+      end
+
+      if @allow_forward
+        # Belongs_to a parent that is itself scoped but carries no scope column of
+        # its own (so via_path cannot terminate on it) — e.g. a hub table scoped
+        # only via referenced_by. Constrain this table to that parent's in-scope
+        # ids so its rows ride along instead of being dumped in full.
+        parent_clause = build_belongs_to_scoped_clause(table)
+        if parent_clause
+          ast.where(parent_clause)
+          return ast
+        end
+      end
+
+      # Only the genuine top-level build (no rescue disabled) is allowed to fail
+      # hard. The Runner/ExplainRunner pre-flight (validate_scope!) rejects
+      # unscopable tables before extraction, so a top-level build never
+      # legitimately lands here; if it does, raise rather than emit an unfiltered
+      # (potential full PII) dump.
+      if @allow_reverse && @allow_forward
+        raise ArgumentError, scope_unscopable_message(table)
+      end
+
+      # Unscopable during a reverse/forward subquery build (a rescue is disabled):
+      # return the unconstrained AST so the caller's "constrained only" check
+      # filters this candidate out (it never becomes a real dump query).
+      ast
+    end
+
+    # The shared column this table is filtered on: a per-table `scope_column`
+    # override when present, otherwise the global `--scope-column`.
+    private def resolved_scope_column(table)
+      table.scope_column || dump_target.scope_column
+    end
+
+    private def scope_exempt?(table)
+      table.scope_exempt || table.rails_managed?
+    end
+
+    private def directly_scoped?(table)
+      column = resolved_scope_column(table)
+      table.columns.any? { |c| c.name == column }
+    end
+
+    private def scope_where_clause(table)
+      Exwiw::QueryAst::WhereClause.new(
+        column_name: resolved_scope_column(table),
+        operator: :eq,
+        value: dump_target.ids
+      )
+    end
+
+    # BFS over belongs_tos to the nearest *directly scoped* ancestor. Unlike the
+    # target-mode walk, the returned path INCLUDES that ancestor: the scope column
+    # lives on the ancestor itself (not on a foreign key of the child), so the
+    # ancestor must be joined and then filtered.
+    private def find_path_to_scoped(table)
+      visited = {}
+      queue = [[table.name, [table.name]]]
+
+      until queue.empty?
+        current_table_name, path = queue.shift
+        next if visited[current_table_name]
+        visited[current_table_name] = true
+
+        current_table = table_by_name[current_table_name]
+        next if current_table.nil?
+
+        current_table.belongs_tos.each do |relation|
+          next_table_name = relation.table_name
+          next_table = table_by_name[next_table_name]
+          next if next_table.nil?
+
+          next_path = path + [next_table_name]
+          return next_path if directly_scoped?(next_table)
+
+          queue.push([next_table_name, next_path])
+        end
+      end
+
+      []
+    end
+
+    private def build_join_clauses_scoped(table)
+      path_tables = find_path_to_scoped(table)
+      @logger.debug("  Join path from #{table.name} to a scoped table: #{path_tables}")
+
+      return [] if path_tables.size < 2
+
+      path_tables.each_cons(2).map do |from_table_name, to_table_name|
+        from_table = table_by_name[from_table_name]
+        to_table = table_by_name[to_table_name]
+
+        join_clause = build_scoped_join_clause(from_table, to_table)
+
+        # Only the final hop's to_table is directly scoped (the BFS stops there),
+        # so the scope filter rides on that join's where_clauses, compiled against
+        # join_table_name = the scoped ancestor.
+        if directly_scoped?(to_table)
+          join_clause.where_clauses.push scope_where_clause(to_table)
+        end
+
+        if to_table.filter
+          join_clause.where_clauses.push to_table.filter
+        end
+
+        join_clause
+      end
+    end
+
+    # One belongs_to hop as a JoinClause, with the polymorphic type condition
+    # placed on the source table (base_where_clauses) when the hop is polymorphic
+    # — mirroring the target-mode loop in build_join_clauses.
+    private def build_scoped_join_clause(from_table, to_table)
+      relation = from_table.belongs_to(to_table.name)
+
+      join_clause = QueryAst::JoinClause.new(
+        base_table_name: from_table.name,
+        foreign_key: relation.foreign_key,
+        join_table_name: to_table.name,
+        primary_key: to_table.primary_key,
+        where_clauses: [],
+        base_where_clauses: []
+      )
+
+      if relation.polymorphic?
+        join_clause.base_where_clauses.push QueryAst::WhereClause.new(
+          column_name: relation.foreign_type,
+          operator: :eq,
+          value: [relation.type_value]
+        )
+      end
+
+      join_clause
+    end
+
+    private def scope_unscopable_message(table)
+      "Table '#{table.name}' cannot be scoped in scope-column mode: it has no " \
+        "'#{dump_target.scope_column}' column (nor a per-table scope_column override) and no " \
+        "belongs_to path to a table that does. Add `scope_exempt: true` to export it in full, " \
+        "set `ignore: true` to skip it, or add the missing belongs_to."
+    end
   end
 end

data/lib/exwiw/runner.rb

@@ -38,8 +38,13 @@ module Exwiw
       target = table_by_name[@dump_target.table_name]
       adapter.validate_as_dump_target!(target) if target
 
+      dumpable_configs = configs.select { |c| adapter.dumpable?(c) }
+      # Scope-column mode: abort if any extractable table cannot be scoped (no-op
+      # otherwise). Done before extraction so nothing is dumped if it would leak.
+      QueryAstBuilder.validate_scope!(dumpable_configs, table_by_name, @dump_target, @logger)
+
       @logger.info("Determining table processing order...")
-      ordered_table_names = DetermineTableProcessingOrder.run(configs.select { |c| adapter.dumpable?(c) })
+      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs, logger: @logger)
 
       clean_output_dir!
 

data/lib/exwiw/table_config.rb

@@ -26,6 +26,18 @@ module Exwiw
     attribute :columns, array(TableColumn), default: []
     attribute :bulk_insert_chunk_size, optional(Integer), skip_serializing_if_nil: true
     attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    # Scope-column mode only (see Exwiw::DumpTarget#scope_column). Both are
+    # user-configured and never emitted by the schema generators.
+    #
+    # `scope_exempt: true` exports the whole table without scope filtering — the
+    # explicit, auditable escape hatch for genuine reference/master tables under
+    # the strict "every table must be scopable" rule.
+    #
+    # `scope_column` overrides the physical column this table is filtered on when
+    # it differs from the global `--scope-column` name (same scope value, just a
+    # different column name on this table).
+    attribute :scope_exempt, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    attribute :scope_column, optional(String), skip_serializing_if_nil: true
 
     def self.from(hash)
       config = super
@@ -137,6 +149,9 @@ module Exwiw
         merged_table.filter = filter
         merged_table.bulk_insert_chunk_size = passed_table.bulk_insert_chunk_size
         merged_table.ignore = ignore
+        # User-owned, never regenerated: carry over from the existing config.
+        merged_table.scope_exempt = scope_exempt
+        merged_table.scope_column = scope_column
 
         # Structural facts of each belongs_to come from the freshly generated
         # config, but the user-owned `comment`/`ignore`/`references` carry over

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.5.0"
+  VERSION = "0.5.2"
 end

data/lib/exwiw.rb

@@ -39,7 +39,13 @@ module Exwiw
   # `ids_field` optionally overrides which field `--ids` is matched against on
   # the target table. When nil the table's primary key is used (the historical
   # behavior). Currently only honored by the mongodb adapter.
-  DumpTarget = Struct.new(:table_name, :ids, :ids_field, keyword_init: true)
+  #
+  # `scope_column` switches the extraction to scope-column mode: instead of a
+  # single `table_name` anchor, every table is filtered by a shared column
+  # (`scope_column IN ids`) and tables lacking it are reached by walking
+  # belongs_to up to the nearest table that has it. When set, `table_name` is
+  # nil. SQL adapters only.
+  DumpTarget = Struct.new(:table_name, :ids, :ids_field, :scope_column, keyword_init: true)
   # `uri` is an optional full connection string (currently only honored by the
   # mongodb adapter, e.g. `mongodb+srv://...`). When present it is the source of
   # truth for the connection — host/port/user/password are ignored — so TLS,

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.5.2
 platform: ruby
 authors:
 - Shia