exwiw 1.1.5 → 1.1.7

data/docs/mongodb.md

@@ -71,6 +71,10 @@ Where the SQL adapters emit a `UNION` subquery, MongoDB has no cross-collection
 
 Masking (`replace_with`) and `fields` behavior on a reverse-scoped collection are unchanged. Like the SQL key, `reverse_scope` is user-owned: `exwiw:mongoid:schema:generate` never emits it and regeneration preserves a hand-added value.
 
+## Self-referencing `belongs_to`
+
+Unlike the SQL adapters, which keep the ancestors of a narrowed [tree table](../README.md#self-referencing-belongs_to-tree-tables), the MongoDB adapter extracts each collection with a single `find` and does not walk a self-reference. A narrowed collection with a `belongs_to` to itself logs a warning, since a kept document's parent may be missing from the dump.
+
 ## Embedded documents
 
 MongoDB models often store one-to-many relationships as embedded subdocument arrays (e.g. `users` documents with a `posts: [...]` field). To mask fields inside embedded subdocuments, declare a separate config with `embedded_in`:

data/docs/scope-id-set-join-notes.md

@@ -0,0 +1,12 @@
+# Why scope id sets are joined, not `IN (subquery)`
+
+When a table is narrowed by a set of ids that another query produces (referenced-by, `reverse_scope`, the narrowed-parent cascade, polymorphic arms, tree ancestors), the SQL adapters emit a `JOIN` to a `SELECT DISTINCT` derived table instead of `<col> IN (<subquery>)`:
+
+```sql
+… JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids
+    ON <table>.<col> = ids.exwiw_scope_id
+```
+
+Both forms return the same rows; the `DISTINCT` keeps the join from multiplying them. The plans differ on a large table. MySQL cannot turn `IN (… UNION …)` into a materialized semi-join, so it rewrites it into a correlated `EXISTS` that is evaluated once per outer row: a full scan of the outer table, with the union re-run for every row. The `DISTINCT` derived table cannot be merged into the outer query, so it is materialized once and the outer table is probed by its key. On a large `users` table this is the difference between a full scan and an index lookup. Nested cascades are materialized once per level for the same reason.
+
+PostgreSQL casts both sides to `text` when the key types differ (`uuid` against `varchar`).

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -164,13 +164,14 @@ module Exwiw
             # left as Strings rather than guessed at — the caller passes values
             # matching the field's actual type.
             if dump_target.ids_field
-              { dump_target.ids_field => { "$in" => dump_target.ids } }
+              { dump_target.ids_field => { "$in" => dump_target.default_ids } }
             else
-              { config.primary_key => { "$in" => coerce_ids(dump_target.ids) } }
+              { config.primary_key => { "$in" => coerce_ids(dump_target.default_ids) } }
             end
           else
             related_collection_filter(config, config_by_name, dump_target)
           end
+        warn_missing_self_ancestors(config) unless filter.empty?
 
         Exwiw::MongoQuery::Find.new(
           collection: config.name,
@@ -181,6 +182,17 @@ module Exwiw
         )
       end
 
+      # The SQL adapters add the ancestors of a narrowed self-referencing table;
+      # this adapter extracts each collection with a single find, so it cannot.
+      private def warn_missing_self_ancestors(config)
+        return unless config.belongs_tos.any? { |relation| relation.table_name == config.name && !relation.polymorphic? }
+
+        @logger.warn(
+          "  #{config.name} references itself, but the mongodb adapter does not add the ancestors " \
+          "of the documents it keeps; a kept document's parent may be missing."
+        )
+      end
+
       def execute(query)
         @logger.debug("  Executing Mongo find on '#{query.collection}': filter=#{query.filter.inspect} projection=#{query.projection.inspect}")
 

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -353,24 +353,27 @@ module Exwiw
       end
 
       private def compile_subquery(subquery)
-        # A SelectSubquery wraps a full Select (the referencing table's
-        # extraction query, projected to a foreign key); compile it as-is.
-        return compile_ast(subquery.query) if subquery.is_a?(Exwiw::QueryAst::SelectSubquery)
-
-        # A UnionSubquery wraps several such Selects; UNION their compiled forms
-        # into a single id set. Sibling arms can carry the same nested scope
-        # id-set, and MySQL cannot reference one TEMPORARY table twice in a
-        # statement (ER_CANT_REOPEN_TABLE), so nested scopes compile inline here.
-        if subquery.is_a?(Exwiw::QueryAst::UnionSubquery)
-          return without_scope_materialization do
+        case subquery
+        in Exwiw::QueryAst::RecursiveAncestorSubquery
+          compile_ancestor_closure(subquery)
+        in Exwiw::QueryAst::SelectSubquery
+          # A SelectSubquery wraps a full Select (the referencing table's
+          # extraction query, projected to a foreign key); compile it as-is.
+          compile_ast(subquery.query)
+        in Exwiw::QueryAst::UnionSubquery
+          # A UnionSubquery wraps several such Selects; UNION their compiled forms
+          # into a single id set. Sibling arms can carry the same nested scope
+          # id-set, and MySQL cannot reference one TEMPORARY table twice in a
+          # statement (ER_CANT_REOPEN_TABLE), so nested scopes compile inline here.
+          without_scope_materialization do
             subquery.queries.map { |q| compile_ast(q) }.join(' UNION ')
           end
+        in Exwiw::QueryAst::Subquery
+          inner_values = subquery.where_values.map { |v| escape_value(v) }
+          "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
+            "FROM #{quote_table_name(subquery.table_name)} " \
+            "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
         end
-
-        inner_values = subquery.where_values.map { |v| escape_value(v) }
-        "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
-          "FROM #{quote_table_name(subquery.table_name)} " \
-          "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
       end
 
       # Backslash and control-character escapes, matching mysqldump. Escaping

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -461,25 +461,35 @@ module Exwiw
       private def compile_subquery(subquery, outer_table: nil, outer_column: nil)
         cast_to = subquery_cast_to(subquery, outer_table, outer_column)
 
-        if subquery.is_a?(Exwiw::QueryAst::SelectSubquery)
-          return compile_ast(subquery.query, select_cast_to: cast_to)
+        case subquery
+        in Exwiw::QueryAst::RecursiveAncestorSubquery
+          # The outer key is compared with this table's own primary keys, so it needs no cast.
+          compile_ancestor_closure(subquery)
+        in Exwiw::QueryAst::SelectSubquery
+          compile_ast(subquery.query, select_cast_to: cast_to)
+        in Exwiw::QueryAst::UnionSubquery
+          # A UnionSubquery wraps several projected Selects; UNION their compiled
+          # forms. cast_to is the union-wide decision (see union_cast_to): when any
+          # arm's column type would clash with the outer column or another arm,
+          # every arm's projected column and the outer key are cast to text so the
+          # UNION and the enclosing IN comparison resolve to one type.
+          subquery.queries.map { |q| compile_ast(q, select_cast_to: cast_to) }.join(' UNION ')
+        in Exwiw::QueryAst::Subquery
+          inner_values = subquery.where_values.map { |v| escape_value(v) }
+          select_expr = qualified_name(subquery.table_name, subquery.select_column)
+          select_expr = "#{select_expr}::#{cast_to}" if cast_to
+          "SELECT #{select_expr} " \
+            "FROM #{quote_table_name(subquery.table_name)} " \
+            "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
         end
+      end
 
-        # A UnionSubquery wraps several projected Selects; UNION their compiled
-        # forms. cast_to is the union-wide decision (see union_cast_to): when any
-        # arm's column type would clash with the outer column or another arm,
-        # every arm's projected column and the outer key are cast to text so the
-        # UNION and the enclosing IN comparison resolve to one type.
-        if subquery.is_a?(Exwiw::QueryAst::UnionSubquery)
-          return subquery.queries.map { |q| compile_ast(q, select_cast_to: cast_to) }.join(' UNION ')
-        end
+      # A self-referencing foreign key may be stored as text while the key it
+      # points at is a uuid; reconcile them the same way a JOIN does.
+      private def ancestor_link_condition(table, referenced_column, foreign_key)
+        return super unless types_need_cast?(column_pg_type(table, referenced_column), column_pg_type(table, foreign_key))
 
-        inner_values = subquery.where_values.map { |v| escape_value(v) }
-        select_expr = qualified_name(subquery.table_name, subquery.select_column)
-        select_expr = "#{select_expr}::#{cast_to}" if cast_to
-        "SELECT #{select_expr} " \
-          "FROM #{quote_table_name(subquery.table_name)} " \
-          "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
+        "#{qualified_name(table, referenced_column)}::text = #{ANCESTOR_SET_NAME}.#{quote_identifier(foreign_key)}::text"
       end
 
       private def subquery_select_target(subquery)

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -232,20 +232,23 @@ module Exwiw
       end
 
       private def compile_subquery(subquery)
-        # A SelectSubquery wraps a full Select (the referencing table's
-        # extraction query, projected to a foreign key); compile it as-is.
-        return compile_ast(subquery.query) if subquery.is_a?(Exwiw::QueryAst::SelectSubquery)
-
-        # A UnionSubquery wraps several such Selects; UNION their compiled forms
-        # into a single id set.
-        if subquery.is_a?(Exwiw::QueryAst::UnionSubquery)
-          return subquery.queries.map { |q| compile_ast(q) }.join(' UNION ')
+        case subquery
+        in Exwiw::QueryAst::RecursiveAncestorSubquery
+          compile_ancestor_closure(subquery)
+        in Exwiw::QueryAst::SelectSubquery
+          # A SelectSubquery wraps a full Select (the referencing table's
+          # extraction query, projected to a foreign key); compile it as-is.
+          compile_ast(subquery.query)
+        in Exwiw::QueryAst::UnionSubquery
+          # A UnionSubquery wraps several such Selects; UNION their compiled forms
+          # into a single id set.
+          subquery.queries.map { |q| compile_ast(q) }.join(' UNION ')
+        in Exwiw::QueryAst::Subquery
+          inner_values = subquery.where_values.map { |v| escape_value(v) }
+          "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
+            "FROM #{quote_table_name(subquery.table_name)} " \
+            "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
         end
-
-        inner_values = subquery.where_values.map { |v| escape_value(v) }
-        "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
-          "FROM #{quote_table_name(subquery.table_name)} " \
-          "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
       end
 
       private def escape_value(value)

data/lib/exwiw/adapter.rb

@@ -307,7 +307,8 @@ module Exwiw
         where_clause.is_a?(Exwiw::QueryAst::WhereClause) &&
           where_clause.operator == :in_subquery &&
           (where_clause.value.is_a?(Exwiw::QueryAst::SelectSubquery) ||
-            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery))
+            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery) ||
+            where_clause.value.is_a?(Exwiw::QueryAst::RecursiveAncestorSubquery))
       end
 
       # The bare name of the single column a scope subquery projects, used to
@@ -319,8 +320,36 @@ module Exwiw
           subquery.query.columns.first.name
         when Exwiw::QueryAst::UnionSubquery
           subquery.queries.first.columns.first.name
+        when Exwiw::QueryAst::RecursiveAncestorSubquery
+          subquery.primary_key
         end
       end
+
+      ANCESTOR_SET_NAME = "exwiw_ancestors"
+
+      # The WITH clause sits inside the subquery, so the id set stays valid wherever
+      # it is embedded: a derived-table JOIN, a MySQL temporary table, a UNION arm.
+      private def compile_ancestor_closure(subquery)
+        table = subquery.table_name
+        ancestor_columns = [subquery.primary_key, *subquery.links.map(&:foreign_key)].uniq
+        column_list = ancestor_columns.map { |c| quote_identifier(c) }.join(', ')
+        recursive_select = ancestor_columns.map { |c| qualified_name(table, c) }.join(', ')
+        link_conditions = subquery.links.map do |link|
+          ancestor_link_condition(table, link.references || subquery.primary_key, link.foreign_key)
+        end
+
+        "WITH RECURSIVE #{ANCESTOR_SET_NAME} (#{column_list}) AS (" \
+          "#{compile_ast(subquery.base)} " \
+          "UNION " \
+          "SELECT #{recursive_select} FROM #{quote_table_name(table)} " \
+          "JOIN #{ANCESTOR_SET_NAME} ON #{link_conditions.join(' OR ')}" \
+          ") SELECT #{ANCESTOR_SET_NAME}.#{quote_identifier(subquery.primary_key)} FROM #{ANCESTOR_SET_NAME}"
+      end
+
+      # Overridden where a dialect compares the two columns differently.
+      private def ancestor_link_condition(table, referenced_column, foreign_key)
+        "#{qualified_name(table, referenced_column)} = #{ANCESTOR_SET_NAME}.#{quote_identifier(foreign_key)}"
+      end
     end
 
     # @params [Exwiw::QueryAst] query_ast

data/lib/exwiw/after_insert_hook.rb

@@ -4,25 +4,26 @@ require 'erb'
 
 module Exwiw
   class AfterInsertHook
-    def self.run(path:, cli_options:, output_dir:, next_idx:, output_extension:, logger:)
+    def self.run(path:, cli_options:, dump_target:, output_dir:, next_idx:, output_extension:, logger:)
       ext = File.extname(path)
 
       if ext == '.rb'
         run_ruby(
           path: path,
           cli_options: cli_options,
+          dump_target: dump_target,
           output_dir: output_dir,
           next_idx: next_idx,
           output_extension: output_extension,
           logger: logger,
         )
       else
-        run_shell(path: path, cli_options: cli_options, output_dir: output_dir, logger: logger)
+        run_shell(path: path, cli_options: cli_options, dump_target: dump_target, output_dir: output_dir, logger: logger)
       end
     end
 
-    def self.run_ruby(path:, cli_options:, output_dir:, next_idx:, output_extension:, logger:)
-      ctx = Context.new(cli_options, output_extension: output_extension)
+    def self.run_ruby(path:, cli_options:, dump_target:, output_dir:, next_idx:, output_extension:, logger:)
+      ctx = Context.new(cli_options, dump_target: dump_target, output_extension: output_extension)
       ctx.instance_eval(File.read(path), path)
 
       # One output file per buffer, numbered sequentially from next_idx: first
@@ -53,7 +54,7 @@ module Exwiw
       end
     end
 
-    def self.run_shell(path:, cli_options:, output_dir:, logger:)
+    def self.run_shell(path:, cli_options:, dump_target:, output_dir:, logger:)
       env = {
         'EXWIW_OUTPUT_DIR'       => output_dir,
         'EXWIW_SCHEMA_DIR'       => cli_options[:schema_dir].to_s,
@@ -64,9 +65,12 @@ module Exwiw
         'EXWIW_DATABASE_NAME'    => cli_options[:database_name].to_s,
         'EXWIW_TARGET_TABLE'     => cli_options[:target_table].to_s,
         'EXWIW_SCOPE_COLUMN'     => cli_options[:scope_column].to_s,
-        'EXWIW_IDS'              => Array(cli_options[:ids]).join(','),
+        'EXWIW_IDS'              => dump_target.default_ids.join(','),
         'EXWIW_OUTPUT_FORMAT'    => cli_options[:output_format].to_s,
       }
+      dump_target.ids.each do |space, values|
+        env["EXWIW_IDS_#{space.upcase}"] = Array(values).join(',')
+      end
       logger.info("Running after-insert shell hook: #{path}")
       ok = system(env, path)
       raise "after-insert shell hook failed: #{path}" unless ok
@@ -79,13 +83,19 @@ module Exwiw
 
       attr_reader :cli_options, :collected, :collected_by_collection
 
-      def initialize(cli_options, output_extension: nil)
+      def initialize(cli_options, dump_target:, output_extension: nil)
         @cli_options = cli_options
+        @dump_target = dump_target
         @output_extension = output_extension
         @collected = []
         @collected_by_collection = {}
       end
 
+      # The values the run was scoped by for the given ID space (`default` when omitted).
+      def ids_for(id_space = DEFAULT_ID_SPACE)
+        @dump_target.ids_for(id_space.to_s)
+      end
+
       def insert_sql(template)
         @collected << render(template)
       end

data/lib/exwiw/cli.rb

@@ -111,7 +111,8 @@ module Exwiw
       @database_name = nil
       @target_table_name = nil
       @target_collection_name = nil
-      @ids = []
+      @ids_by_id_space = {}
+      @ids_errors = []
       @ids_field = nil
       @scope_column = nil
       @output_format = nil
@@ -151,7 +152,7 @@ module Exwiw
 
       dump_target = DumpTarget.new(
         table_name: @target_table_name,
-        ids: @ids,
+        ids: @ids_by_id_space,
         ids_field: @ids_field,
         scope_column: @scope_column,
       )
@@ -297,6 +298,9 @@ module Exwiw
       # Fill in any options not given on the CLI from the config file. Done first
       # so a config-provided `adapter` is in place before normalization below.
       # CLI values always win (the merge only fills nil/empty ivars).
+      @ids_errors.each { |message| $stderr.puts message }
+      exit 1 if @ids_errors.any?
+
       apply_config_file!
 
       # Default log level once CLI and config have both had their say.
@@ -316,6 +320,7 @@ module Exwiw
       end
 
       resolve_target_collection_alias!
+      resolve_named_ids!
       resolve_scope_column!
       resolve_ids_field!
       resolve_uri_option!
@@ -391,18 +396,22 @@ module Exwiw
         exit 1
       end
 
-      if @target_table_name && @ids.empty?
+      if @target_table_name && !ids_given?
         $stderr.puts "--ids is required when --target-table is specified"
         exit 1
       end
 
-      if @scope_column && @ids.empty?
+      if @scope_column && !ids_given?
         $stderr.puts "--ids is required when --scope-column is specified"
         exit 1
       end
 
-      if !@target_table_name && !@scope_column && @ids.any?
-        $stderr.puts "--target-table or --scope-column is required when --ids is specified"
+      # For the SQL adapters, --ids without a target is scope-column mode; whether
+      # the schema supports it is checked once it is loaded
+      # (QueryAstBuilder.validate_scope!). MongoDB has no scope-column mode.
+      if @database_adapter == "mongodb" && !@target_table_name && default_ids.any?
+        $stderr.puts "--target-collection (or --target-table) is required when --ids is specified " \
+                     "with the mongodb adapter"
         exit 1
       end
 
@@ -479,11 +488,23 @@ module Exwiw
       @log_level ||= config["log_level"]&.to_sym
       @target_table_name ||= config["target_table"]
       @target_collection_name ||= config["target_collection"]
-      if @ids.empty? && config.key?("ids")
+      # Like the other keys, `ids` is taken from the config only when the CLI
+      # passed no --ids at all; the two are never merged per ID space.
+      if !ids_given? && config.key?("ids")
         raw = config["ids"]
-        # Accept either a YAML list or a "1,2" string; coerce to strings to match
-        # the CLI's `--ids=1,2` -> ["1", "2"] shape.
-        @ids = (raw.is_a?(String) ? raw.split(",") : Array(raw)).map(&:to_s)
+        if raw.is_a?(Hash)
+          raw.each do |space, values|
+            space = space.to_s
+            unless space.match?(ID_SPACE_NAME_PATTERN)
+              $stderr.puts "config 'ids' keys must be ID space names (a lowercase letter followed by " \
+                           "lowercase letters, digits or underscores; got #{space.inspect}) in #{path}"
+              exit 1
+            end
+            @ids_by_id_space[space] = parse_config_ids(values)
+          end
+        else
+          @ids_by_id_space[DEFAULT_ID_SPACE] = parse_config_ids(raw)
+        end
       end
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
@@ -492,6 +513,56 @@ module Exwiw
       @explain_verbosity ||= config["explain_verbosity"]
     end
 
+    # Accept either a YAML list or a "1,2" string; coerce to strings to match
+    # the CLI's `--ids=1,2` -> ["1", "2"] shape.
+    private def parse_config_ids(raw)
+      (raw.is_a?(String) ? raw.split(",") : Array(raw)).map(&:to_s)
+    end
+
+    # `--ids=<id_space>=1,2` gives the values of a named ID space; without the
+    # prefix the values are the default space's. The prefix is recognized only
+    # when it is shaped like an ID space name, so an id containing `=` is passed
+    # as `--ids=default=<ids>`. Each ID space may be given once: a second --ids
+    # for it would otherwise replace the first without notice.
+    private def add_ids_option(value)
+      prefix, rest = value.to_s.split("=", 2)
+      space, list =
+        if rest && prefix.match?(ID_SPACE_NAME_PATTERN)
+          [prefix, rest]
+        else
+          [DEFAULT_ID_SPACE, value.to_s]
+        end
+
+      if @ids_by_id_space.key?(space)
+        @ids_errors << "--ids was given more than once for the ID space '#{space}'; " \
+                       "pass all of its ids in one comma-separated --ids."
+      end
+      @ids_by_id_space[space] = list.split(",")
+    end
+
+    private def ids_given?
+      @ids_by_id_space.each_value.any?(&:any?)
+    end
+
+    private def default_ids
+      @ids_by_id_space.fetch(DEFAULT_ID_SPACE, [])
+    end
+
+    # Named ID spaces exist only for scope-column mode, which the mongodb
+    # adapter does not have. Whether a SQL run is in scope-column mode depends on
+    # the schema, so that is checked once it is loaded (QueryAstBuilder.validate_scope!).
+    private def resolve_named_ids!
+      named = @ids_by_id_space.keys - [DEFAULT_ID_SPACE]
+      return if named.empty?
+
+      if @database_adapter == "mongodb"
+        $stderr.puts "--ids for a named ID space (#{named.map { |space| "'#{space}'" }.join(', ')}) " \
+                     "is not supported by the mongodb adapter, which uses only the default ID space " \
+                     "(--ids without an `<id_space>=` prefix)."
+        exit 1
+      end
+    end
+
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
     # `base` (the config file's directory). Returns nil for a nil value.
     private def expand_dir(value, base)
@@ -542,8 +613,8 @@ module Exwiw
 
     # `--scope-column` is **deprecated**: it selected scope-column mode with a
     # single global column for every table. The preferred way is to declare a
-    # per-table `scope_column:` in the schema config and pass `--target-table`
-    # (the target is then scoped like any other table). The flag still works as
+    # per-table `scope_column:` in the schema config (`--target-table` is then
+    # optional). The flag still works as
     # before — SQL-only and mutually exclusive with `--target-table` — but emits a
     # deprecation warning. Runs after resolve_target_collection_alias! so
     # --target-collection is already folded into @target_table_name.
@@ -567,7 +638,7 @@ module Exwiw
       end
 
       $stderr.puts "warning: --scope-column is deprecated; declare a per-table `scope_column:` " \
-                   "in the schema config and run with --target-table instead."
+                   "in the schema config and drop the flag instead."
     end
 
     # `--uri` supplies a full connection string (e.g. `mongodb+srv://...`) and is
@@ -780,7 +851,7 @@ module Exwiw
         database_adapter: @database_adapter,
         database_name: @database_name,
         target_table: @target_table_name,
-        ids: @ids.dup.freeze,
+        ids: default_ids.dup.freeze,
         ids_field: @ids_field,
         scope_column: @scope_column,
         output_format: @output_format,
@@ -849,11 +920,11 @@ module Exwiw
         opts.on("-a", "--adapter=ADAPTER", "Database adapter: mysql, sqlite, postgresql, mongodb (aliases: mysql2, sqlite3)") { |v| @database_adapter = v }
         opts.on("--uri=URI", "Full MongoDB connection URI (mongodb:// or mongodb+srv://). mongodb adapter only; takes precedence over --host/--port/--user. TLS, replicaSet, authSource and credentials are read from the URI.") { |v| @connection_uri = v }
         opts.on("--database=DATABASE", "Target database name") { |v| @database_name = v }
-        opts.on("--target-table=[TABLE]", "Target table for extraction. If omitted, dump all tables.") { |v| @target_table_name = v }
+        opts.on("--target-table=[TABLE]", "Target table for extraction. If omitted with --ids, run in scope-column mode (SQL adapters; the schema must declare `scope_column:`). If omitted without --ids, dump all tables.") { |v| @target_table_name = v }
         opts.on("--target-collection=[COLLECTION]", "Alias of --target-table for the mongodb adapter.") { |v| @target_collection_name = v }
-        opts.on("--ids=[IDS]", "Comma-separated list of identifiers. Required when --target-table is given.") { |v| @ids = v.split(',') }
+        opts.on("--ids=[IDS]", "Comma-separated list of identifiers. Required when --target-table is given. Prefix with `<id_space>=` (e.g. --ids=org=a,b) to give the values of a named ID space; may be repeated, once per ID space. An id containing `=` is passed as --ids=default=<ids>.") { |v| add_ids_option(v) }
         opts.on("--ids-field=[FIELD]", "Field on the target collection that --ids is matched against. Defaults to the primary key. (mongodb adapter only)") { |v| @ids_field = v }
-        opts.on("--scope-column=[COLUMN]", "DEPRECATED. Filter every table by this shared global column (--ids are its values) instead of a single --target-table. SQL adapters only; mutually exclusive with --target-table. Prefer declaring a per-table `scope_column:` in the schema config and running with --target-table.") { |v| @scope_column = v }
+        opts.on("--scope-column=[COLUMN]", "DEPRECATED. Filter every table by this shared global column (--ids are its values) instead of a single --target-table. SQL adapters only; mutually exclusive with --target-table. Prefer declaring a per-table `scope_column:` in the schema config.") { |v| @scope_column = v }
         opts.on("--output-format=[FORMAT]", "Output format: insert (default) or copy (PostgreSQL only, export subcommand only)") { |v| @output_format = v }
         opts.on("--insert-only", "DEPRECATED: ignored. exwiw no longer generates delete-*.sql files, so every export is insert-only.") do
           $stderr.puts "warning: --insert-only is obsolete and ignored (exwiw no longer generates delete-*.sql files); remove the flag"

data/lib/exwiw/query_ast.rb

@@ -20,12 +20,34 @@ module Exwiw
           reads_table?(where_clause.value.query, table_name)
         when UnionSubquery
           where_clause.value.queries.any? { |q| reads_table?(q, table_name) }
+        when RecursiveAncestorSubquery
+          where_clause.value.table_name == table_name || reads_table?(where_clause.value.base, table_name)
         else
           false
         end
       end
     end
 
+    # The ID spaces whose values `query` filters by, anywhere in it.
+    def self.id_spaces(query)
+      where_clauses = query.where_clauses +
+        query.join_clauses.flat_map { |j| j.where_clauses + j.base_where_clauses }
+
+      where_clauses.flat_map do |where_clause|
+        next [] unless where_clause.is_a?(WhereClause)
+
+        own = where_clause.is_a?(ScopeWhereClause) ? [where_clause.id_space] : []
+        nested =
+          case where_clause.value
+          when SelectSubquery then id_spaces(where_clause.value.query)
+          when UnionSubquery then where_clause.value.queries.flat_map { |q| id_spaces(q) }
+          when RecursiveAncestorSubquery then id_spaces(where_clause.value.base)
+          else []
+          end
+        own + nested
+      end.uniq
+    end
+
     class JoinClause
       # `where_clauses` is compiled against this join's join_table_name (the
       # joined-to table). `base_where_clauses`, on the other hand, is compiled
@@ -65,11 +87,17 @@ module Exwiw
         {
           column_name: column_name,
           operator: operator,
-          value: value.is_a?(Subquery) || value.is_a?(SelectSubquery) || value.is_a?(UnionSubquery) ? value.to_h : value,
+          value: SUBQUERY_VALUE_TYPES.any? { |type| value.is_a?(type) } ? value.to_h : value,
         }
       end
     end
 
+    # A scope-column filter. It compiles like any WhereClause; `id_space` records
+    # which ID space its values come from, so a query mixing spaces can be found.
+    ScopeWhereClause = Class.new(WhereClause) do
+      attr_accessor :id_space
+    end
+
     # Resolves a set of values on `where_column` to the rows' `select_column`
     # via a nested SELECT. Used as the `value` of a WhereClause whose operator
     # is `:in_subquery`, so a non primary-key `ids_field` can filter related
@@ -126,6 +154,36 @@ module Exwiw
       end
     end
 
+    # The primary keys of the rows `base` keeps plus every ancestor reachable
+    # through the table's own self-referencing foreign keys (`links`), so a tree
+    # table never keeps a row whose parent was left out:
+    #
+    #   WITH RECURSIVE exwiw_ancestors (<pk>, <fk>...) AS (
+    #     <base projected to pk and every link's fk>
+    #     UNION
+    #     SELECT <table>.<pk>, <table>.<fk>... FROM <table>
+    #       JOIN exwiw_ancestors ON <table>.<ref> = exwiw_ancestors.<fk> OR ...
+    #   ) SELECT <pk> FROM exwiw_ancestors
+    #
+    # UNION (not UNION ALL) stops the walk on cyclic data.
+    RecursiveAncestorSubquery = Struct.new(:base, :table_name, :primary_key, :links, keyword_init: true) do
+      def to_h
+        {
+          ancestors_of: base.to_h,
+          table_name: table_name,
+          primary_key: primary_key,
+          links: links.map(&:to_h),
+        }
+      end
+    end
+
+    # One self-referencing foreign key of a RecursiveAncestorSubquery.
+    # `references` is the column the foreign key points at (the primary key
+    # unless the belongs_to says otherwise).
+    AncestorLink = Struct.new(:foreign_key, :references, keyword_init: true)
+
+    SUBQUERY_VALUE_TYPES = [Subquery, SelectSubquery, UnionSubquery, RecursiveAncestorSubquery].freeze
+
     module ColumnValue
       Base = Struct.new(:name, :value, keyword_init: true)
       Plain = Class.new(Base)