exwiw 1.1.6 → 1.1.8

data/docs/mongodb.md

@@ -71,6 +71,10 @@ Where the SQL adapters emit a `UNION` subquery, MongoDB has no cross-collection
 
 Masking (`replace_with`) and `fields` behavior on a reverse-scoped collection are unchanged. Like the SQL key, `reverse_scope` is user-owned: `exwiw:mongoid:schema:generate` never emits it and regeneration preserves a hand-added value.
 
+## Self-referencing `belongs_to`
+
+Unlike the SQL adapters, which keep the ancestors of a narrowed [tree table](../README.md#self-referencing-belongs_to-tree-tables), the MongoDB adapter extracts each collection with a single `find` and does not walk a self-reference. A narrowed collection with a `belongs_to` to itself logs a warning, since a kept document's parent may be missing from the dump.
+
 ## Embedded documents
 
 MongoDB models often store one-to-many relationships as embedded subdocument arrays (e.g. `users` documents with a `posts: [...]` field). To mask fields inside embedded subdocuments, declare a separate config with `embedded_in`:

data/docs/scope-id-set-join-notes.md

@@ -0,0 +1,12 @@
+# Why scope id sets are joined, not `IN (subquery)`
+
+When a table is narrowed by a set of ids that another query produces (referenced-by, `reverse_scope`, the narrowed-parent cascade, polymorphic arms, tree ancestors), the SQL adapters emit a `JOIN` to a `SELECT DISTINCT` derived table instead of `<col> IN (<subquery>)`:
+
+```sql
+… JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids
+    ON <table>.<col> = ids.exwiw_scope_id
+```
+
+Both forms return the same rows; the `DISTINCT` keeps the join from multiplying them. The plans differ on a large table. MySQL cannot turn `IN (… UNION …)` into a materialized semi-join, so it rewrites it into a correlated `EXISTS` that is evaluated once per outer row: a full scan of the outer table, with the union re-run for every row. The `DISTINCT` derived table cannot be merged into the outer query, so it is materialized once and the outer table is probed by its key. On a large `users` table this is the difference between a full scan and an index lookup. Nested cascades are materialized once per level for the same reason.
+
+PostgreSQL casts both sides to `text` when the key types differ (`uuid` against `varchar`).

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -171,6 +171,7 @@ module Exwiw
           else
             related_collection_filter(config, config_by_name, dump_target)
           end
+        warn_missing_self_ancestors(config) unless filter.empty?
 
         Exwiw::MongoQuery::Find.new(
           collection: config.name,
@@ -181,6 +182,17 @@ module Exwiw
         )
       end
 
+      # The SQL adapters add the ancestors of a narrowed self-referencing table;
+      # this adapter extracts each collection with a single find, so it cannot.
+      private def warn_missing_self_ancestors(config)
+        return unless config.belongs_tos.any? { |relation| relation.table_name == config.name && !relation.polymorphic? }
+
+        @logger.warn(
+          "  #{config.name} references itself, but the mongodb adapter does not add the ancestors " \
+          "of the documents it keeps; a kept document's parent may be missing."
+        )
+      end
+
       def execute(query)
         @logger.debug("  Executing Mongo find on '#{query.collection}': filter=#{query.filter.inspect} projection=#{query.projection.inspect}")
 

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -103,6 +103,7 @@ module Exwiw
         # MariaDB's mysqldump does not recognise the MySQL-specific
         # --set-gtid-purged flag and exits with code 7.
         gtid_flags = mariadb_mysqldump?(mysqldump_bin) ? [] : ['--set-gtid-purged=OFF']
+        ssl_flags = mysqldump_ssl_flags(mariadb: mariadb_mysqldump?(mysqldump_bin))
 
         cmd = [
           mysqldump_bin,
@@ -115,6 +116,7 @@ module Exwiw
           '--skip-set-charset',
           '--no-tablespaces', # skip tablespace query that requires PROCESS privilege (unavailable on managed MySQL like RDS)
           *gtid_flags,
+          *ssl_flags,
           '--compact',
           @connection_config.database_name,
         ]
@@ -353,24 +355,27 @@ module Exwiw
       end
 
       private def compile_subquery(subquery)
-        # A SelectSubquery wraps a full Select (the referencing table's
-        # extraction query, projected to a foreign key); compile it as-is.
-        return compile_ast(subquery.query) if subquery.is_a?(Exwiw::QueryAst::SelectSubquery)
-
-        # A UnionSubquery wraps several such Selects; UNION their compiled forms
-        # into a single id set. Sibling arms can carry the same nested scope
-        # id-set, and MySQL cannot reference one TEMPORARY table twice in a
-        # statement (ER_CANT_REOPEN_TABLE), so nested scopes compile inline here.
-        if subquery.is_a?(Exwiw::QueryAst::UnionSubquery)
-          return without_scope_materialization do
+        case subquery
+        in Exwiw::QueryAst::RecursiveAncestorSubquery
+          compile_ancestor_closure(subquery)
+        in Exwiw::QueryAst::SelectSubquery
+          # A SelectSubquery wraps a full Select (the referencing table's
+          # extraction query, projected to a foreign key); compile it as-is.
+          compile_ast(subquery.query)
+        in Exwiw::QueryAst::UnionSubquery
+          # A UnionSubquery wraps several such Selects; UNION their compiled forms
+          # into a single id set. Sibling arms can carry the same nested scope
+          # id-set, and MySQL cannot reference one TEMPORARY table twice in a
+          # statement (ER_CANT_REOPEN_TABLE), so nested scopes compile inline here.
+          without_scope_materialization do
             subquery.queries.map { |q| compile_ast(q) }.join(' UNION ')
           end
+        in Exwiw::QueryAst::Subquery
+          inner_values = subquery.where_values.map { |v| escape_value(v) }
+          "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
+            "FROM #{quote_table_name(subquery.table_name)} " \
+            "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
         end
-
-        inner_values = subquery.where_values.map { |v| escape_value(v) }
-        "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
-          "FROM #{quote_table_name(subquery.table_name)} " \
-          "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
       end
 
       # Backslash and control-character escapes, matching mysqldump. Escaping
@@ -434,6 +439,23 @@ module Exwiw
           end
       end
 
+      private def mysqldump_ssl_flags(mariadb:)
+        mode = @connection_config.ssl_mode
+        ca_flags = @connection_config.ssl_ca ? ["--ssl-ca=#{@connection_config.ssl_ca}"] : []
+        return ca_flags + (mode ? ["--ssl-mode=#{mode.upcase}"] : []) unless mariadb
+
+        # MariaDB's client has no --ssl-mode, and from 11.4 verifies the server
+        # certificate by default, so `required` must opt out to stay unverified.
+        mode_flags =
+          case mode
+          when nil, "preferred" then []
+          when "disabled" then ["--skip-ssl"]
+          when "required" then ["--ssl", "--skip-ssl-verify-server-cert"]
+          else ["--ssl", "--ssl-verify-server-cert"]
+          end
+        ca_flags + mode_flags
+      end
+
       private def connection
         @connection ||= MysqlClient.new(@connection_config)
       end

data/lib/exwiw/adapter/mysql_client.rb

@@ -20,6 +20,16 @@ module Exwiw
 
       DRIVERS = [:mysql2, :trilogy].freeze
 
+      SSL_MODES = %w[disabled preferred required verify_ca verify_identity].freeze
+
+      TRILOGY_SSL_MODE_CONSTANTS = {
+        "disabled" => :SSL_DISABLED,
+        "preferred" => :SSL_PREFERRED_NOVERIFY,
+        "required" => :SSL_REQUIRED_NOVERIFY,
+        "verify_ca" => :SSL_VERIFY_CA,
+        "verify_identity" => :SSL_VERIFY_IDENTITY,
+      }.freeze
+
       # Pick the available driver, preferring mysql2 (exwiw's historical default).
       # require returns false when already loaded, so this is safe to call repeatedly.
       #
@@ -188,10 +198,31 @@ module Exwiw
         }.compact
 
         case @driver
-        when :mysql2 then Mysql2::Client.new(options)
-        when :trilogy then Trilogy.new(options)
+        when :mysql2 then Mysql2::Client.new(options.merge(mysql2_ssl_options))
+        when :trilogy then Trilogy.new(options.merge(trilogy_ssl_options))
         end
       end
+
+      private def mysql2_ssl_options
+        { ssl_mode: mysql2_ssl_mode, sslca: @connection_config.ssl_ca }.compact
+      end
+
+      # mysql2 linked against MariaDB Connector/C defines unsupported modes as 0 and
+      # ignores them with a warning, so fall back to the nearest mode it supports.
+      private def mysql2_ssl_mode
+        mode = @connection_config.ssl_mode
+        return mode if mode.nil? || !Mysql2::Client.const_get("SSL_MODE_#{mode.upcase}").zero?
+
+        "verify_identity" if mode == "verify_ca"
+      end
+
+      private def trilogy_ssl_options
+        mode = @connection_config.ssl_mode
+        {
+          ssl_mode: mode && Trilogy.const_get(TRILOGY_SSL_MODE_CONSTANTS.fetch(mode)),
+          ssl_ca: @connection_config.ssl_ca,
+        }.compact
+      end
     end
   end
 end

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -461,25 +461,35 @@ module Exwiw
       private def compile_subquery(subquery, outer_table: nil, outer_column: nil)
         cast_to = subquery_cast_to(subquery, outer_table, outer_column)
 
-        if subquery.is_a?(Exwiw::QueryAst::SelectSubquery)
-          return compile_ast(subquery.query, select_cast_to: cast_to)
+        case subquery
+        in Exwiw::QueryAst::RecursiveAncestorSubquery
+          # The outer key is compared with this table's own primary keys, so it needs no cast.
+          compile_ancestor_closure(subquery)
+        in Exwiw::QueryAst::SelectSubquery
+          compile_ast(subquery.query, select_cast_to: cast_to)
+        in Exwiw::QueryAst::UnionSubquery
+          # A UnionSubquery wraps several projected Selects; UNION their compiled
+          # forms. cast_to is the union-wide decision (see union_cast_to): when any
+          # arm's column type would clash with the outer column or another arm,
+          # every arm's projected column and the outer key are cast to text so the
+          # UNION and the enclosing IN comparison resolve to one type.
+          subquery.queries.map { |q| compile_ast(q, select_cast_to: cast_to) }.join(' UNION ')
+        in Exwiw::QueryAst::Subquery
+          inner_values = subquery.where_values.map { |v| escape_value(v) }
+          select_expr = qualified_name(subquery.table_name, subquery.select_column)
+          select_expr = "#{select_expr}::#{cast_to}" if cast_to
+          "SELECT #{select_expr} " \
+            "FROM #{quote_table_name(subquery.table_name)} " \
+            "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
         end
+      end
 
-        # A UnionSubquery wraps several projected Selects; UNION their compiled
-        # forms. cast_to is the union-wide decision (see union_cast_to): when any
-        # arm's column type would clash with the outer column or another arm,
-        # every arm's projected column and the outer key are cast to text so the
-        # UNION and the enclosing IN comparison resolve to one type.
-        if subquery.is_a?(Exwiw::QueryAst::UnionSubquery)
-          return subquery.queries.map { |q| compile_ast(q, select_cast_to: cast_to) }.join(' UNION ')
-        end
+      # A self-referencing foreign key may be stored as text while the key it
+      # points at is a uuid; reconcile them the same way a JOIN does.
+      private def ancestor_link_condition(table, referenced_column, foreign_key)
+        return super unless types_need_cast?(column_pg_type(table, referenced_column), column_pg_type(table, foreign_key))
 
-        inner_values = subquery.where_values.map { |v| escape_value(v) }
-        select_expr = qualified_name(subquery.table_name, subquery.select_column)
-        select_expr = "#{select_expr}::#{cast_to}" if cast_to
-        "SELECT #{select_expr} " \
-          "FROM #{quote_table_name(subquery.table_name)} " \
-          "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
+        "#{qualified_name(table, referenced_column)}::text = #{ANCESTOR_SET_NAME}.#{quote_identifier(foreign_key)}::text"
       end
 
       private def subquery_select_target(subquery)

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -232,20 +232,23 @@ module Exwiw
       end
 
       private def compile_subquery(subquery)
-        # A SelectSubquery wraps a full Select (the referencing table's
-        # extraction query, projected to a foreign key); compile it as-is.
-        return compile_ast(subquery.query) if subquery.is_a?(Exwiw::QueryAst::SelectSubquery)
-
-        # A UnionSubquery wraps several such Selects; UNION their compiled forms
-        # into a single id set.
-        if subquery.is_a?(Exwiw::QueryAst::UnionSubquery)
-          return subquery.queries.map { |q| compile_ast(q) }.join(' UNION ')
+        case subquery
+        in Exwiw::QueryAst::RecursiveAncestorSubquery
+          compile_ancestor_closure(subquery)
+        in Exwiw::QueryAst::SelectSubquery
+          # A SelectSubquery wraps a full Select (the referencing table's
+          # extraction query, projected to a foreign key); compile it as-is.
+          compile_ast(subquery.query)
+        in Exwiw::QueryAst::UnionSubquery
+          # A UnionSubquery wraps several such Selects; UNION their compiled forms
+          # into a single id set.
+          subquery.queries.map { |q| compile_ast(q) }.join(' UNION ')
+        in Exwiw::QueryAst::Subquery
+          inner_values = subquery.where_values.map { |v| escape_value(v) }
+          "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
+            "FROM #{quote_table_name(subquery.table_name)} " \
+            "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
         end
-
-        inner_values = subquery.where_values.map { |v| escape_value(v) }
-        "SELECT #{qualified_name(subquery.table_name, subquery.select_column)} " \
-          "FROM #{quote_table_name(subquery.table_name)} " \
-          "WHERE #{qualified_name(subquery.table_name, subquery.where_column)} IN (#{inner_values.join(', ')})"
       end
 
       private def escape_value(value)

data/lib/exwiw/adapter.rb

@@ -307,7 +307,8 @@ module Exwiw
         where_clause.is_a?(Exwiw::QueryAst::WhereClause) &&
           where_clause.operator == :in_subquery &&
           (where_clause.value.is_a?(Exwiw::QueryAst::SelectSubquery) ||
-            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery))
+            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery) ||
+            where_clause.value.is_a?(Exwiw::QueryAst::RecursiveAncestorSubquery))
       end
 
       # The bare name of the single column a scope subquery projects, used to
@@ -319,8 +320,36 @@ module Exwiw
           subquery.query.columns.first.name
         when Exwiw::QueryAst::UnionSubquery
           subquery.queries.first.columns.first.name
+        when Exwiw::QueryAst::RecursiveAncestorSubquery
+          subquery.primary_key
         end
       end
+
+      ANCESTOR_SET_NAME = "exwiw_ancestors"
+
+      # The WITH clause sits inside the subquery, so the id set stays valid wherever
+      # it is embedded: a derived-table JOIN, a MySQL temporary table, a UNION arm.
+      private def compile_ancestor_closure(subquery)
+        table = subquery.table_name
+        ancestor_columns = [subquery.primary_key, *subquery.links.map(&:foreign_key)].uniq
+        column_list = ancestor_columns.map { |c| quote_identifier(c) }.join(', ')
+        recursive_select = ancestor_columns.map { |c| qualified_name(table, c) }.join(', ')
+        link_conditions = subquery.links.map do |link|
+          ancestor_link_condition(table, link.references || subquery.primary_key, link.foreign_key)
+        end
+
+        "WITH RECURSIVE #{ANCESTOR_SET_NAME} (#{column_list}) AS (" \
+          "#{compile_ast(subquery.base)} " \
+          "UNION " \
+          "SELECT #{recursive_select} FROM #{quote_table_name(table)} " \
+          "JOIN #{ANCESTOR_SET_NAME} ON #{link_conditions.join(' OR ')}" \
+          ") SELECT #{ANCESTOR_SET_NAME}.#{quote_identifier(subquery.primary_key)} FROM #{ANCESTOR_SET_NAME}"
+      end
+
+      # Overridden where a dialect compares the two columns differently.
+      private def ancestor_link_condition(table, referenced_column, foreign_key)
+        "#{qualified_name(table, referenced_column)} = #{ANCESTOR_SET_NAME}.#{quote_identifier(foreign_key)}"
+      end
     end
 
     # @params [Exwiw::QueryAst] query_ast

data/lib/exwiw/cli.rb

@@ -65,7 +65,7 @@ module Exwiw
     # Database connection settings are environment-specific (and sometimes
     # secret-adjacent), so they must be passed via CLI/env, never the committed
     # config file. `adapter` is the one connection-ish key allowed in config.
-    REJECTED_CONNECTION_KEYS = %w[host port user database uri password].freeze
+    REJECTED_CONNECTION_KEYS = %w[host port user database uri password ssl_mode ssl_ca].freeze
 
     # Keys that only make sense for `export`. They are skipped when merging config
     # for `explain` so a shared config file does not trip validate_explain_only!,
@@ -103,6 +103,8 @@ module Exwiw
       @database_user = nil
       @database_password = ENV["DATABASE_PASSWORD"]
       @connection_uri = nil
+      @ssl_mode = nil
+      @ssl_ca = nil
       @output_dir = nil
       @schema_dir = nil
       @from_db = false
@@ -148,6 +150,8 @@ module Exwiw
         database_name: @database_name,
         uri: @connection_uri,
         mongodb_query_timeout_ms: @mongodb_query_timeout_ms,
+        ssl_mode: @ssl_mode,
+        ssl_ca: @ssl_ca,
       )
 
       dump_target = DumpTarget.new(
@@ -324,6 +328,7 @@ module Exwiw
       resolve_scope_column!
       resolve_ids_field!
       resolve_uri_option!
+      resolve_ssl_options!
       resolve_parallel_workers!
       resolve_mongodb_query_timeout_ms!
 
@@ -654,6 +659,31 @@ module Exwiw
       end
     end
 
+    private def resolve_ssl_options!
+      return if @ssl_mode.nil? && @ssl_ca.nil?
+
+      if @database_adapter != "mysql"
+        $stderr.puts "--ssl-mode / --ssl-ca are only supported by the mysql adapter"
+        exit 1
+      end
+
+      if @ssl_ca&.empty?
+        $stderr.puts "--ssl-ca must be a path to a CA certificate file"
+        exit 1
+      end
+
+      if @ssl_mode.nil?
+        $stderr.puts "--ssl-ca requires --ssl-mode (verify_ca or verify_identity to verify the server)"
+        exit 1
+      end
+
+      @ssl_mode = @ssl_mode.downcase
+      unless Adapter::MysqlClient::SSL_MODES.include?(@ssl_mode)
+        $stderr.puts "Invalid --ssl-mode. Available options are: #{Adapter::MysqlClient::SSL_MODES.join(', ')}"
+        exit 1
+      end
+    end
+
     # `--parallel-workers` opts into the MongoDB fork-parallel dump schedule
     # (docs/mongodb-dump-parallelism-2x-notes.md). It is mongodb-only (the SQL
     # adapters shell out to their own dumpers) and must be a positive integer;
@@ -919,6 +949,8 @@ module Exwiw
         end
         opts.on("-a", "--adapter=ADAPTER", "Database adapter: mysql, sqlite, postgresql, mongodb (aliases: mysql2, sqlite3)") { |v| @database_adapter = v }
         opts.on("--uri=URI", "Full MongoDB connection URI (mongodb:// or mongodb+srv://). mongodb adapter only; takes precedence over --host/--port/--user. TLS, replicaSet, authSource and credentials are read from the URI.") { |v| @connection_uri = v }
+        opts.on("--ssl-mode=MODE", "TLS mode for the connection and mysqldump: #{Adapter::MysqlClient::SSL_MODES.join(', ')}. mysql adapter only; defaults to the client library's behavior.") { |v| @ssl_mode = v }
+        opts.on("--ssl-ca=PATH", "CA certificate file used to verify the server. mysql adapter only.") { |v| @ssl_ca = v }
         opts.on("--database=DATABASE", "Target database name") { |v| @database_name = v }
         opts.on("--target-table=[TABLE]", "Target table for extraction. If omitted with --ids, run in scope-column mode (SQL adapters; the schema must declare `scope_column:`). If omitted without --ids, dump all tables.") { |v| @target_table_name = v }
         opts.on("--target-collection=[COLLECTION]", "Alias of --target-table for the mongodb adapter.") { |v| @target_collection_name = v }

data/lib/exwiw/query_ast.rb

@@ -20,6 +20,8 @@ module Exwiw
           reads_table?(where_clause.value.query, table_name)
         when UnionSubquery
           where_clause.value.queries.any? { |q| reads_table?(q, table_name) }
+        when RecursiveAncestorSubquery
+          where_clause.value.table_name == table_name || reads_table?(where_clause.value.base, table_name)
         else
           false
         end
@@ -39,6 +41,7 @@ module Exwiw
           case where_clause.value
           when SelectSubquery then id_spaces(where_clause.value.query)
           when UnionSubquery then where_clause.value.queries.flat_map { |q| id_spaces(q) }
+          when RecursiveAncestorSubquery then id_spaces(where_clause.value.base)
           else []
           end
         own + nested
@@ -84,7 +87,7 @@ module Exwiw
         {
           column_name: column_name,
           operator: operator,
-          value: value.is_a?(Subquery) || value.is_a?(SelectSubquery) || value.is_a?(UnionSubquery) ? value.to_h : value,
+          value: SUBQUERY_VALUE_TYPES.any? { |type| value.is_a?(type) } ? value.to_h : value,
         }
       end
     end
@@ -151,6 +154,36 @@ module Exwiw
       end
     end
 
+    # The primary keys of the rows `base` keeps plus every ancestor reachable
+    # through the table's own self-referencing foreign keys (`links`), so a tree
+    # table never keeps a row whose parent was left out:
+    #
+    #   WITH RECURSIVE exwiw_ancestors (<pk>, <fk>...) AS (
+    #     <base projected to pk and every link's fk>
+    #     UNION
+    #     SELECT <table>.<pk>, <table>.<fk>... FROM <table>
+    #       JOIN exwiw_ancestors ON <table>.<ref> = exwiw_ancestors.<fk> OR ...
+    #   ) SELECT <pk> FROM exwiw_ancestors
+    #
+    # UNION (not UNION ALL) stops the walk on cyclic data.
+    RecursiveAncestorSubquery = Struct.new(:base, :table_name, :primary_key, :links, keyword_init: true) do
+      def to_h
+        {
+          ancestors_of: base.to_h,
+          table_name: table_name,
+          primary_key: primary_key,
+          links: links.map(&:to_h),
+        }
+      end
+    end
+
+    # One self-referencing foreign key of a RecursiveAncestorSubquery.
+    # `references` is the column the foreign key points at (the primary key
+    # unless the belongs_to says otherwise).
+    AncestorLink = Struct.new(:foreign_key, :references, keyword_init: true)
+
+    SUBQUERY_VALUE_TYPES = [Subquery, SelectSubquery, UnionSubquery, RecursiveAncestorSubquery].freeze
+
     module ColumnValue
       Base = Struct.new(:name, :value, keyword_init: true)
       Plain = Class.new(Base)