exwiw 0.5.1 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2af5a1cc29946424a2b6498f19d7ad77714f108194575ddd6014c5fc2d829416
-  data.tar.gz: 3113e80b88ab11a95344140f819a9247eadc3706c45a9b2a665f946a88a945b7
+  metadata.gz: 567683d65df5d9f147ab9415a67baf48a80e21ad32e1ef7635c624dfc3d28c47
+  data.tar.gz: 1513b577f6f2368df60edc45a54c96495ece4f1ee9b453e92adb8991f182fcdf
 SHA512:
-  metadata.gz: 539f4ab428d75f97714475d607d91ae2870a2bf860ac8ab2f732d5a5709d5e0cf2fd085c0bc3bbdf9b095f969542eb866c1f3c3766cc5e3a8be0294cf3cfcf0f
-  data.tar.gz: 7b41f191e5e6d5ff60a1c927111d06f85b6c461a1dd63d5665b254338eafcde037233bb7f67027306258515e71aa1af3cf38e6aadf20b02ec9f6a91f05c7f2c4
+  metadata.gz: a9680642eb34f99ed3f0c2924154a5171edf541286dfed14befe15b8c029271419a13d3295694847b1143898b0200bf6eb1d6448e6665dbad7bef026e3c3fbbb
+  data.tar.gz: 30e6ef9f988965b85f899fdb0646b6e4e2befd68f95a247a9edfeddb8d3a6088f611f07d5376c7d3b9f4f58f147072e03294a140312dec1622994fb6da175720

data/CHANGELOG.md

@@ -2,6 +2,15 @@
 
 ## [Unreleased]
 
+## [0.5.2] - 2026-06-18
+
+### Fixed
+
+- **PostgreSQL: an extension the restore target cannot create no longer aborts the restore, and `pglogical` is never emitted.** `dump_schema` prepends `CREATE EXTENSION IF NOT EXISTS` for every extension installed on the source, wrapped in a `DO` block that previously swallowed only `feature_not_supported` (the extension's binaries are unavailable on the target). A source on managed Postgres/AlloyDB carrying `pglogical` (logical replication) emitted `CREATE EXTENSION ... SCHEMA "pglogical"`, which on a target lacking that schema fails with `invalid_schema_name` — an error the handler did not catch, so the whole restore aborted. The handler now also catches `invalid_schema_name`, and instead of silently discarding the skip it re-raises it as a `WARNING` (carrying SQLSTATE and the original message) so the skip is visible in the restore logs rather than vanishing. `insufficient_privilege` is intentionally **not** caught: a restore role lacking `CREATE` privilege is a misconfiguration that must fail loudly. Separately, `pglogical` is now excluded from the prepended extensions entirely (alongside `plpgsql` and the `google_*`/`rds_*`/`aiven_*` managed-platform extensions) — it is a replication mechanism of the source, not part of the copied data.
+- **Table processing order no longer aborts on `belongs_to` cycles.** Two distinct problems made `export`/`explain` fail with "Circular belongs_to dependency detected" on schemas that have no resolvable order: (1) a `belongs_to` whose target table is **not part of the run** — most commonly an embedded MongoDB collection, which is masked through its parent and never dumped on its own — was treated as a dependency that could never be satisfied, so every table that (transitively) referenced one froze and was misreported as a cycle; such out-of-run targets are now ignored when ordering. (2) A **genuine** cycle (e.g. `a belongs_to b` and `b belongs_to a`) now **breaks deterministically with a warning** instead of raising: exwiw emits the cycle member (a table in a strongly-connected component of the unresolved-dependency graph) with the fewest unresolved dependencies, preferring one that still has an already-ordered parent so its extraction stays constrained, and logs which `belongs_to` edge was dropped. The dropped edge is not enforced while ordering, so for the mongodb adapter that table may match a superset of rows (the not-yet-processed parent contributes no `$in` filter); mark one of the `belongs_to` entries forming the cycle with `ignore: true` to break it explicitly instead. Acyclic tables that merely wait on a cycle are never reordered ahead of their parents.
+- **MongoDB: `dump_schema` tolerates collections declared in the schema but absent from the source database.** Listing indexes for a non-existent collection makes the driver raise `NamespaceNotFound` (code 26), which aborted the whole export when the schema covered more collections than the connected database actually had (schema/DB drift, or a sparse development database). The existing collections are now resolved once up front and indexes are emitted only for those; `createCollection` is still emitted for every configured collection, so the target schema is created in full.
+- **MongoDB: a related collection that cannot be scoped no longer falls back to dumping the whole collection.** When a non-target collection's `belongs_to` parents all yield no ids to filter by — because every parent matched nothing or is not dumped on its own (e.g. an embedded collection) — the assembled filter is empty. Previously that empty filter was sent as `find({})`, scanning and dumping the **entire collection across every scope** (a cross-scope data-exposure risk). Such a collection is now constrained to match no rows and a warning is logged instead. A collection with no `belongs_to` at all is still treated as reference/master data and dumped in full.
+
 ## [0.5.1] - 2026-06-18
 
 ### Added

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'json'
+require 'set'
 
 # NOTE: This adapter consumes MongodbCollectionConfig (`fields` instead of
 # `columns`, plus `embedded_in`). Top-level collections are dumped as one
@@ -71,16 +72,7 @@ module Exwiw
               { config.primary_key => { "$in" => coerce_ids(dump_target.ids) } }
             end
           else
-            config.belongs_tos.each_with_object({}) do |relation, acc|
-              # Constrain by the parent field this FK actually references
-              # (`relation.references`, default the parent primary_key). The
-              # values were captured from that field's documents in #execute, so
-              # their BSON type already matches the stored FK — no coercion.
-              values = parent_state_for(relation, config_by_name)
-              next if values.nil? || values.empty?
-
-              acc[relation.foreign_key] = { "$in" => values }
-            end
+            related_collection_filter(config, config_by_name)
           end
 
         Exwiw::MongoQuery::Find.new(
@@ -160,6 +152,14 @@ module Exwiw
 
         collections = ordered_tables.reject(&:embedded?)
 
+        # Index listing targets a specific collection, and MongoDB raises
+        # NamespaceNotFound (code 26) for one that does not exist. The schema may
+        # declare collections absent from this database (schema/DB drift, or a
+        # sparse dev DB), so resolve the set that actually exists up front and emit
+        # indexes only for those. `createCollection` is still emitted for every
+        # config below, so the target schema is created in full regardless.
+        existing_collections = db.database.collection_names.to_set
+
         File.open(output_path, 'w') do |file|
           file.puts("// Auto-generated by exwiw. Apply with: mongosh \"$MONGODB_URI\" #{File.basename(output_path)}")
           file.puts
@@ -172,6 +172,11 @@ module Exwiw
 
           collections.each do |config|
             name = config.name
+            unless existing_collections.include?(name)
+              @logger.debug("  Collection '#{name}' is not present in the source database; emitting no indexes.")
+              next
+            end
+
             indexes = db[name].indexes.to_a.reject { |idx| idx['name'] == '_id_' }
             indexes.each do |idx|
               key = idx['key']
@@ -274,6 +279,39 @@ module Exwiw
         ([config.primary_key] + referenced).uniq
       end
 
+      # Build the scoping filter for a non-target collection from its belongs_to
+      # parents' captured ids. Each belongs_to is constrained by the parent field
+      # the FK references (`relation.references`, default the parent primary_key);
+      # the values were captured from that field in #execute, so their BSON type
+      # already matches the stored FK — no coercion.
+      #
+      # A belongs_to whose parent produced no ids contributes no constraint:
+      # either the parent matched nothing, or it is not dumped here (e.g. an
+      # embedded collection, or one excluded from the run). If that leaves the
+      # filter empty even though the collection HAS belongs_to, the collection
+      # cannot be scoped from the dump target — and falling back to an empty `{}`
+      # filter would scan and dump the ENTIRE collection across every scope. That
+      # is never what a scoped extraction wants, so constrain it to match nothing
+      # and warn instead. (A collection with no belongs_to at all is genuine
+      # reference/master data and is still dumped in full via `{}`.)
+      private def related_collection_filter(config, config_by_name)
+        filter = config.belongs_tos.each_with_object({}) do |relation, acc|
+          values = parent_state_for(relation, config_by_name)
+          next if values.nil? || values.empty?
+
+          acc[relation.foreign_key] = { "$in" => values }
+        end
+
+        return filter unless filter.empty? && config.belongs_tos.any?
+
+        @logger.warn(
+          "  Collection '#{config.name}' has belongs_to but no parent produced ids to scope by " \
+          "(parents matched nothing, or are not dumped on their own such as embedded collections). " \
+          "Constraining it to match no rows to avoid an unscoped full-collection dump."
+        )
+        { config.primary_key => { "$in" => [] } }
+      end
+
       # The captured parent-collection values a child belongs_to should be
       # constrained by: the values of the parent field the FK references
       # (`relation.references`, default the parent primary_key). nil when the

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -65,7 +65,18 @@ module Exwiw
           ext_ddl = extensions.map do |extname, schema|
             stmt = "CREATE EXTENSION IF NOT EXISTS #{connection.quote_ident(extname)}"
             stmt += " SCHEMA #{connection.quote_ident(schema)}" unless schema == "public"
-            "DO $$ BEGIN #{stmt}; EXCEPTION WHEN feature_not_supported THEN NULL; END $$;"
+            # Best-effort prepend: a restore target that genuinely cannot create the
+            # extension should not abort the whole restore. Two such cases are caught:
+            #   feature_not_supported (0A000) -- the extension's binaries are unavailable
+            #   invalid_schema_name   (3F000) -- the extension's required schema is absent
+            # insufficient_privilege (42501) is deliberately NOT caught: a restore role
+            # lacking CREATE privilege is a misconfiguration to fix, not to skip silently.
+            # The skip is re-raised as a WARNING so it surfaces in the restore logs
+            # instead of vanishing.
+            warning = connection.escape_literal("exwiw: skipped CREATE EXTENSION #{extname} (SQLSTATE %): %")
+            "DO $$ BEGIN #{stmt}; " \
+              "EXCEPTION WHEN feature_not_supported OR invalid_schema_name THEN " \
+              "RAISE WARNING #{warning}, SQLSTATE, SQLERRM; END $$;"
           end.join("\n") + "\n\n"
           @logger.debug("  Found #{extensions.size} extension(s) to prepend.")
           stdout = ext_ddl + stdout
@@ -382,11 +393,17 @@ module Exwiw
       end
 
       private def query_extensions
+        # Skip plpgsql (always present) and managed-platform bookkeeping extensions
+        # (google_*/rds_*/aiven_*). pglogical is also skipped: it is a logical-
+        # replication mechanism of the source, not part of the data being copied,
+        # and its dedicated `pglogical` schema is typically absent on the restore
+        # target — so prepending CREATE EXTENSION for it only breaks the restore.
         sql = <<~SQL
           SELECT e.extname, n.nspname
           FROM pg_extension e
           JOIN pg_namespace n ON n.oid = e.extnamespace
           WHERE e.extname != 'plpgsql'
+            AND e.extname != 'pglogical'
             AND e.extname NOT LIKE 'google\\_%' ESCAPE '\\'
             AND e.extname NOT LIKE 'rds\\_%' ESCAPE '\\'
             AND e.extname NOT LIKE 'aiven\\_%' ESCAPE '\\'

data/lib/exwiw/determine_table_processing_order.rb

@@ -1,35 +1,58 @@
 # frozen_string_literal: true
 
+require "set"
+
 module Exwiw
   module DetermineTableProcessingOrder
     module_function
 
     # @param tables [Array<Exwiw::TableConfig>] tables
+    # @param logger [Logger, nil] receives a warning when a cycle has to be broken
     # @return [Array<String>] sorted table names
-    def run(tables)
+    def run(tables, logger: nil)
       return tables.map(&:name) if tables.size < 2
 
       ordered_table_names = []
+      ordered = Set.new
 
       table_by_name = tables.each_with_object({}) do |table, acc|
         acc[table.name] = table
       end
 
+      # Only belongs_to relations whose target is also in this run constrain the
+      # order. A belongs_to pointing at a table that is not being processed here
+      # — e.g. an embedded MongoDB collection (masked through its parent, never
+      # dumped on its own) or any table excluded from the run — is not something
+      # we can or need to order against, so it must never block resolution.
+      # Without this, such a dependency would stay unresolved forever and
+      # masquerade as a circular dependency, freezing every table that
+      # (transitively) references it.
+      present_names = table_by_name.keys.to_set
+
       loop do
         break if table_by_name.empty?
 
-        tables_with_no_dependencies = table_by_name.values.select do |table|
-          not_resolved_names = compute_table_dependencies(table) - ordered_table_names - [table.name]
-
-          not_resolved_names.empty?
+        resolvable = table_by_name.values.select do |table|
+          unresolved_dependencies(table, present_names, ordered).empty?
         end
 
-        if tables_with_no_dependencies.empty?
-          raise ArgumentError, build_cycle_error_message(table_by_name, ordered_table_names)
+        if resolvable.empty?
+          # No table has all its (in-run) dependencies satisfied, yet tables
+          # remain: the belongs_to graph has a genuine cycle and no strict
+          # topological order exists. Rather than aborting the whole export, break
+          # the cycle by emitting one cycle member; see pick_cycle_victim for how
+          # the member is chosen. Warn so the dropped constraint is visible.
+          victim = pick_cycle_victim(table_by_name.values, present_names, ordered)
+          warn_cycle_break(logger, victim, unresolved_dependencies(victim, present_names, ordered))
+          resolvable = [victim]
         end
 
-        tables_with_no_dependencies.each do |table|
+        # In the normal (acyclic) path, emit every currently-resolvable table in
+        # insertion order — preserving the historical ordering the snapshot specs
+        # depend on. The cycle-break path emits exactly its single chosen victim.
+        resolvable.each do |table|
           ordered_table_names << table.name
+          ordered << table.name
           table_by_name.delete(table.name)
         end
       end
@@ -37,30 +60,124 @@ module Exwiw
       ordered_table_names
     end
 
+    # The belongs_to target table names of `table`. A polymorphic belongs_to is
+    # expanded into one entry per concrete target by schema generation, so each
+    # entry is a plain table name here.
     def compute_table_dependencies(table)
-      table.belongs_tos.each_with_object([]) do |relation, acc|
-        acc << relation.table_name
+      table.belongs_tos.map(&:table_name)
+    end
+
+    # The dependencies still blocking `table`: belongs_to targets that are part
+    # of this run, not yet ordered, and not the table itself (a self-referential
+    # belongs_to never blocks).
+    private_class_method def unresolved_dependencies(table, present_names, ordered)
+      compute_table_dependencies(table).uniq.select do |dep|
+        present_names.include?(dep) && !ordered.include?(dep) && dep != table.name
       end
     end
 
-    # When no table can be resolved but some remain, the belongs_to graph
-    # contains a cycle (e.g. A belongs_to B and B belongs_to A). A topological
-    # order cannot exist, so report the offending tables instead of looping
-    # forever.
-    private_class_method def cycle_diagnostics(table_by_name, ordered_table_names)
-      table_by_name.values.map do |table|
-        unresolved = (compute_table_dependencies(table) - ordered_table_names - [table.name]).uniq
-        "  #{table.name} -> #{unresolved.join(', ')}"
+    # Choose the next table to emit when the order is stuck in a cycle. Only
+    # genuine cycle members are eligible — a table in a non-trivial
+    # strongly-connected component of the unresolved-dependency subgraph — so an
+    # acyclic table that merely waits on a cycle is never reordered ahead of its
+    # parent. Among the members, prefer one that still has at least one
+    # already-ordered parent, so its extraction stays constrained instead of
+    # collapsing to "match every row" (a cross-scope over-extraction risk for the
+    # mongodb adapter); break remaining ties by fewest unresolved dependencies,
+    # then by name, for determinism.
+    private_class_method def pick_cycle_victim(remaining, present_names, ordered)
+      adjacency = remaining.each_with_object({}) do |table, acc|
+        acc[table.name] = unresolved_dependencies(table, present_names, ordered)
+      end
+      cyclic_names = strongly_connected_members(adjacency)
+
+      candidates = remaining.select { |table| cyclic_names.include?(table.name) }
+      candidates = remaining if candidates.empty? # defensive; a stall implies a cycle
+
+      anchored = candidates.select { |table| ordered_parent?(table, present_names, ordered) }
+      pool = anchored.empty? ? candidates : anchored
+
+      pool.min_by { |table| [unresolved_dependencies(table, present_names, ordered).size, table.name] }
+    end
+
+    # True when `table` has a belongs_to whose target was already ordered, so its
+    # extraction filter will be constrained rather than an unscoped full scan.
+    private_class_method def ordered_parent?(table, present_names, ordered)
+      compute_table_dependencies(table).any? do |dep|
+        dep != table.name && present_names.include?(dep) && ordered.include?(dep)
       end
     end
 
-    private_class_method def build_cycle_error_message(table_by_name, ordered_table_names)
-      "Circular belongs_to dependency detected among tables: " \
-        "#{table_by_name.keys.sort.join(', ')}. " \
-        "A processing order cannot be determined. " \
-        "Remove one of the belongs_to entries forming the cycle.\n" \
-        "Unresolved dependencies:\n" \
-        "#{cycle_diagnostics(table_by_name, ordered_table_names).join("\n")}"
+    # Names belonging to a non-trivial strongly-connected component (size > 1) of
+    # `adjacency` (table name -> unresolved dependency names), i.e. the genuine
+    # cycle participants. Iterative Tarjan; nodes and edges are visited in name
+    # order so the result is deterministic. Self-edges are already excluded from
+    # the adjacency, so a size-1 component is never a cycle.
+    private_class_method def strongly_connected_members(adjacency)
+      index = {}
+      low = {}
+      on_stack = {}
+      stack = []
+      counter = 0
+      members = Set.new
+      neighbors = adjacency.each_with_object({}) { |(name, deps), acc| acc[name] = deps.sort }
+
+      adjacency.keys.sort.each do |start|
+        next if index.key?(start)
+
+        work = [[start, 0]]
+        until work.empty?
+          node, edge_i = work.last
+          if edge_i.zero?
+            index[node] = counter
+            low[node] = counter
+            counter += 1
+            stack.push(node)
+            on_stack[node] = true
+          end
+
+          adj = neighbors[node] || []
+          if edge_i < adj.size
+            work.last[1] += 1
+            w = adj[edge_i]
+            next unless adjacency.key?(w) # ignore edges leaving the remaining set
+
+            if index.key?(w)
+              low[node] = [low[node], index[w]].min if on_stack[w]
+            else
+              work.push([w, 0])
+            end
+          else
+            if low[node] == index[node]
+              component = []
+              loop do
+                w = stack.pop
+                on_stack[w] = false
+                component << w
+                break if w == node
+              end
+              members.merge(component) if component.size > 1
+            end
+            work.pop
+            low[work.last[0]] = [low[work.last[0]], low[node]].min unless work.empty?
+          end
+        end
+      end
+
+      members
+    end
+
+    private_class_method def warn_cycle_break(logger, victim, dropped)
+      return if logger.nil?
+
+      logger.warn(
+        "Circular belongs_to dependency detected. Breaking it by ordering " \
+        "'#{victim.name}' before its parent table(s): #{dropped.join(', ')}. The dropped " \
+        "relationship is not enforced while ordering, so '#{victim.name}' is extracted " \
+        "without that parent constraint (the mongodb adapter may then match a superset of " \
+        "rows; SQL output may not load in foreign-key order). To break the cycle explicitly " \
+        "instead, mark one of the belongs_to entries forming it with `ignore: true`."
+      )
     end
   end
 end

data/lib/exwiw/explain_runner.rb

@@ -30,7 +30,7 @@ module Exwiw
       QueryAstBuilder.validate_scope!(dumpable_configs, table_by_name, @dump_target, @logger)
 
       @logger.debug("Determining table processing order...")
-      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs)
+      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs, logger: @logger)
 
       total_size = ordered_table_names.size
       ordered_table_names.each_with_index do |table_name, idx|

data/lib/exwiw/runner.rb

@@ -44,7 +44,7 @@ module Exwiw
       QueryAstBuilder.validate_scope!(dumpable_configs, table_by_name, @dump_target, @logger)
 
       @logger.info("Determining table processing order...")
-      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs)
+      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs, logger: @logger)
 
       clean_output_dir!
 

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.5.1"
+  VERSION = "0.5.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.5.2
 platform: ruby
 authors:
 - Shia