exwiw 0.8.2 → 0.8.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eb6879b8efb94ef9a9a0bd37df32962257106df9eec4c3fc9de844a12c6360cd
-  data.tar.gz: b43b0bfde090111d7ce23083db7471f15974292ddb5886da682de9bc443febfa
+  metadata.gz: ee150e9fd830a7829be8c0eebbeca19f0e2d1b1dbcce50fe78e63d62e878997c
+  data.tar.gz: 31377340d8737ece209e1c0755eeac49ac40930562503e2c4d04f6362938e9d9
 SHA512:
-  metadata.gz: e8cca59b376a369b9253d1482f3ee64ae8c542b00c1c3eddb192d273efe0e3c969e3aec1770f087977efe6b75fa52f375538e286d31a87d7f87a197739a4eeaf
-  data.tar.gz: 78a3ece09cd6ec9ecb6fd7b0a2f166bc41d51720ea36935ca4cd974bf766008f11688f0e0dc7bbd8a63ce29f4719806e1a4705b0390850a857a192fcfaa56976
+  metadata.gz: 26151e0bf763b6f48e993d2025b4bac804a373fe2e3cfd71bb70d91b86b453826838ec683ebfd3f4f4336bb83b8a12d567d7dbda7f8a4ef48fc7f50bbe98145a
+  data.tar.gz: b29a2da42750dde93cd53eb56a11a592478134b690ccba31b59cd312f3685dad2713e63897d09cb77246a8be37dfe55890f1a8505c1aa8b5fd020b7bc7760a03

data/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## [Unreleased]
 
+## [0.8.4] - 2026-06-24
+
+### Fixed
+
+- **Scope id-sets are materialized and probed by JOIN instead of `<col> IN (subquery)`, removing a correlated full-scan on large tables.** The three id-set scope shapes — the multi-referencer `reverse_scope` `UNION`, the single-referencer reverse/`referenced_by` extraction, and the multi-hop forward (`via_scoped_parent`) cascade — were emitted as `<col> IN (<subquery>)`. On a large global-identity table (e.g. `users`) MySQL cannot turn a `UNION` subquery into a materialized semi-join and falls back to its IN-to-`EXISTS` rewrite: a correlated `DEPENDENT SUBQUERY`/`DEPENDENT UNION` re-evaluated **per outer row**, so the driving table is full-scanned and the union re-run for every row (the plan that ran for minutes and timed out on a production-scale identity table, even for an empty tenant). These clauses are now lifted into a `JOIN` against a materialized derived table — `JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids ON <table>.<col> = ids.exwiw_scope_id` — so the engine evaluates the id set **once** (the `DISTINCT` makes the derived table non-mergeable, hence materialized) and probes the outer table by primary key. The `DISTINCT` also dedups, so the result set is identical to the old `IN` form; the cascade nests the same way (each level materialized once); NULL exclusion, the forward-path cycle guard, single-parent/polymorphic skips, and PostgreSQL's `uuid`/`varchar` `::text` reconciliation are all preserved. All three SQL adapters (mysql / postgresql / sqlite). See the [README](README.md#why-a-join-not-in-subquery).
+
+## [0.8.3] - 2026-06-24
+
+### Fixed
+
+- **Forward scope (`via_scoped_parent`) now cascades across multiple `belongs_to` hops instead of dying after one.** A table with no scope column of its own is scoped by constraining it to its `belongs_to` parent's in-scope ids (`fk IN (SELECT parent.pk FROM <parent's scoped query>)`). Previously the parent was rebuilt with forward scoping turned off, so if the parent was *itself* scoped only through *its* parent (e.g. an identity-family table two or more hops below a `reverse_scope`/`referenced_by` table — `users ← end_users ← end_user_profiles`), the rebuilt parent came back unconstrained and the child was classified `:unscopable` — forcing a `scope_exempt` full dump and re-introducing the bloat the prune removes. The boolean single-hop bound is replaced by a forward-path guard: the rescue keeps forward scoping enabled while rebuilding the parent (appending the current table to the path), so the cascade recurses N levels and produces a correspondingly nested `IN (subquery)`; it terminates only on a genuine `belongs_to` cycle (a table already on the path is not revisited, falling through to `:unscopable`). The single-unambiguous-parent rule and the polymorphic-skip are unchanged, and the reverse arms still cannot loop back through the table being reverse-scoped. SQL adapters only.
+
 ## [0.8.2] - 2026-06-24
 
 ### Added

data/README.md

@@ -179,12 +179,18 @@ Each table is resolved as follows:
   nearest such table and applies the scope filter there (the same join machinery
   the single-target mode uses).
 - **`belongs_to` a parent that is itself scoped but carries no scope column of its
-  own** → exwiw constrains this table to the parent's in-scope ids via a subquery
-  (`fk IN (SELECT parent.pk FROM <parent's scoped query>)`). This covers a *hub*
-  table that has no scope column and is scoped only because an extractable child
-  references it (see referenced-by below): the hub's other `belongs_to` children
-  ride along to just the in-scope rows instead of being dumped in full. Limited to
-  a single forward hop and a single unambiguous scopable parent.
+  own** → exwiw constrains this table to the parent's in-scope ids by joining it to
+  the parent's scoped query, materialized as a derived table
+  (`JOIN (SELECT DISTINCT parent.pk … FROM <parent's scoped query>) … ON fk = …`).
+  This covers a *hub* table that has no scope column and is scoped only because an
+  extractable child references it (see referenced-by below): the hub's other
+  `belongs_to` children ride along to just the in-scope rows instead of being dumped
+  in full. The parent itself may be scoped the same way, so this **cascades across
+  multiple hops** (each a single unambiguous scopable parent) and the derived-table
+  JOINs nest correspondingly; the recursion terminates on a genuine `belongs_to`
+  cycle (a table already on the path is left `:unscopable` rather than looped on).
+  (See [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery) for the
+  materialization rationale.)
 - **Cannot be scoped at all** (no scope column and no path to one) → exwiw
   **aborts** and lists the offending tables, so an unscoped table is never silently
   dumped in full. For each, either declare a `scope_column`, add a `belongs_to`
@@ -565,15 +571,19 @@ The same type filter is applied on the join path — and in the matching `delete
 ActiveStorage is handled automatically — no ActiveStorage-specific configuration is required. The `has_one_attached` / `has_many_attached` macros don't add a column to the owning model; they generate ordinary associations that exwiw already understands:
 
 - **`active_storage_attachments`** is the polymorphic join row (`belongs_to :record, polymorphic: true` + `belongs_to :blob`). `exwiw:schema:generate` expands the polymorphic `record` into one `belongs_to` per model that declared `has_*_attached` (found via the generated `has_* ..., as: :record` reflections), exactly like any other [polymorphic `belongs_to`](#polymorphic-belongs_to). So only the attachments whose owner is among the dumped rows are extracted.
-- **`active_storage_blobs`** has no `belongs_to` of its own (attachments point *at* it), so it has no path to the dump target. exwiw narrows it via **reverse / "referenced_by" extraction**: a parent table referenced by exactly one constrained, non-polymorphic child is constrained to just the referenced ids instead of dumping every row:
+- **`active_storage_blobs`** has no `belongs_to` of its own (attachments point *at* it), so it has no path to the dump target. exwiw narrows it via **reverse / "referenced_by" extraction**: a parent table referenced by exactly one constrained, non-polymorphic child is constrained to just the referenced ids instead of dumping every row. The id set is materialized once and joined back (see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery)):
 
   ```sql
   SELECT active_storage_blobs.* FROM active_storage_blobs
-  WHERE active_storage_blobs.id IN (
-    SELECT active_storage_attachments.blob_id FROM active_storage_attachments
-    WHERE active_storage_attachments.record_id IN (/* owner subquery */)
-      AND active_storage_attachments.record_type = '...'
-  )
+  JOIN (
+    SELECT DISTINCT exwiw_scope_src_0.blob_id AS exwiw_scope_id
+    FROM (
+      SELECT active_storage_attachments.blob_id FROM active_storage_attachments
+      WHERE active_storage_attachments.record_id IN (/* owner subquery */)
+        AND active_storage_attachments.record_type = '...'
+    ) AS exwiw_scope_src_0
+  ) AS exwiw_scope_ids_0
+    ON active_storage_blobs.id = exwiw_scope_ids_0.exwiw_scope_id
   ```
 
   `active_storage_variant_records` also references blobs, but since it has no path of its own to the dump target it doesn't constrain anything and is ignored as a referencer — blobs stays narrowed to the attachment-referenced ids. (A parent referenced by *multiple* constrained children currently falls back to dumping all of its rows.)
@@ -600,18 +610,22 @@ The automatic reverse extraction above narrows a table referenced by **exactly o
 }
 ```
 
-produces (each arm reuses that referencer's own scope, so a per-tenant run keeps only that tenant's ids):
+produces (each arm reuses that referencer's own scope, so a per-tenant run keeps only that tenant's ids; the `UNION` id set is materialized once and joined back — see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery)):
 
 ```sql
 SELECT users.* FROM users
-WHERE users.id IN (
-  SELECT customers.user_id FROM customers WHERE <customers' scope> AND customers.user_id IS NOT NULL
-  UNION
-  SELECT staff.user_id FROM staff WHERE <staff' scope> AND staff.user_id IS NOT NULL
-  UNION
-  SELECT business_entity_customers.kantan_yoyaku_user_id FROM business_entity_customers
-    WHERE <…' scope> AND business_entity_customers.kantan_yoyaku_user_id IS NOT NULL
-)
+JOIN (
+  SELECT DISTINCT exwiw_scope_src_0.user_id AS exwiw_scope_id
+  FROM (
+    SELECT customers.user_id FROM customers WHERE <customers' scope> AND customers.user_id IS NOT NULL
+    UNION
+    SELECT staff.user_id FROM staff WHERE <staff' scope> AND staff.user_id IS NOT NULL
+    UNION
+    SELECT business_entity_customers.kantan_yoyaku_user_id FROM business_entity_customers
+      WHERE <…' scope> AND business_entity_customers.kantan_yoyaku_user_id IS NOT NULL
+  ) AS exwiw_scope_src_0
+) AS exwiw_scope_ids_0
+  ON users.id = exwiw_scope_ids_0.exwiw_scope_id
 ```
 
 Notes:
@@ -619,9 +633,22 @@ Notes:
 - **`column` is explicit**, so a *non-default* foreign key (e.g. `kantan_yoyaku_user_id`, or `organization_admins.id` which itself references `users.id`) is honored, and even a column with no declared `belongs_to` edge can be enumerated.
 - **Only scoped referencers belong in `via`.** Each arm's query must come out constrained; an unconstrained referencer (e.g. a `scope_exempt` table, or one with no path to a scope) would project *every* id and union the whole table back — so such an arm is **skipped with a warning** rather than silently widening the dump. An unknown table is likewise skipped with a warning. If no arm survives, the table stays unscopable and (in [scope-column mode](#scope-column-mode)) the run aborts via `validate_scope!`.
 - **NULLs are excluded** per arm (`IS NOT NULL`).
-- **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`.
+- **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`. The cascade is **multi-hop**, so a table several `belongs_to` hops below the reverse-scoped table (e.g. `end_user_profiles → end_users → users`) also tightens automatically, with no config of its own.
 - Works in both single-target and scope-column mode. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
 
+### Why a JOIN, not `IN (subquery)`
+
+Every scope id-set above — the multi-referencer `reverse_scope` `UNION`, the single-referencer reverse extraction, and the multi-hop forward cascade — is emitted as a `JOIN` to a `SELECT DISTINCT` derived table rather than `<col> IN (<subquery>)`:
+
+```sql
+… JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids
+    ON <table>.<col> = ids.exwiw_scope_id
+```
+
+Both forms select the **same rows** — the `DISTINCT` dedups, so the join never fans out — but the query plans differ sharply on a large table. As `<col> IN (… UNION …)`, MySQL cannot turn a `UNION` subquery into a materialized semi-join and falls back to its IN-to-`EXISTS` rewrite: a **correlated `DEPENDENT SUBQUERY`** re-evaluated for every outer row, i.e. a full scan of the (potentially huge) outer table multiplied by the cost of the union. The derived-table form forces the engine to evaluate the id set **once** (the `DISTINCT` makes the derived table non-mergeable, hence materialized) and then probe the outer table by its primary key. On a global-identity table such as `users` this is the difference between a full table scan and an index lookup; the cascade nests the same way, so each level is materialized once instead of being re-evaluated by the level above.
+
+All three SQL adapters (mysql / postgresql / sqlite) emit this shape. PostgreSQL additionally reconciles a `uuid`/`varchar` type mismatch by casting the join key and the projected id to `text`, exactly as the old `IN` form did.
+
 ### Rails-managed tables (special `type` values)
 
 Some tables are owned by Rails itself rather than the application — they have no ActiveRecord model and Rails reserves the right to evolve their column shape between versions (e.g. `schema_migrations`, `ar_internal_metadata`). exwiw treats them as a distinct category via the `type` field on a table config:

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -229,11 +229,18 @@ module Exwiw
       def compile_ast(query_ast, count_only: false)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if count_only
                  "COUNT(*)"
                elsif query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
                end
@@ -256,14 +263,43 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table:
+      #
+      #   JOIN (SELECT DISTINCT src.<proj> AS exwiw_scope_id
+      #         FROM (<subquery>) AS src) AS ids
+      #     ON <table>.<col> = ids.exwiw_scope_id
+      #
+      # The DISTINCT makes the derived table non-mergeable, so MySQL materializes
+      # the id-set once and probes this table by it (PK/index lookup) — instead
+      # of full-scanning this table and re-evaluating a correlated
+      # `IN (… UNION …)` per row (the DEPENDENT SUBQUERY / IN-to-EXISTS fallback,
+      # which a UNION subquery cannot be turned into a materialized semi-join).
+      # DISTINCT also dedups, so the join never fans out: the row set is identical
+      # to `<col> IN (subquery)`.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -301,9 +301,16 @@ module Exwiw
       def compile_ast(query_ast, select_cast_to: nil)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  cols = query_ast.columns.map { |col| compile_column_name(query_ast, col) }
                  cols = cols.map { |c| "#{c}::#{select_cast_to}" } if select_cast_to
@@ -337,14 +344,45 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table
+      # (see MysqlAdapter#compile_scope_join for the full rationale). The DISTINCT
+      # forces the engine to materialize the id-set once and probe this table by
+      # it, instead of full-scanning and re-evaluating a correlated subquery per
+      # row; it also dedups, so the join is row-for-row identical to
+      # `<col> IN (subquery)`.
+      #
+      # Type reconciliation mirrors the old IN form: when the outer column and
+      # the projected id clash (e.g. uuid vs varchar), #compile_subquery already
+      # casts every arm to text, so the derived `exwiw_scope_id` is text and the
+      # outer key is cast to match.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+
+        inner_sql = compile_subquery(subquery, outer_table: from_table_name, outer_column: where_clause.column_name)
+        cast_to = subquery_cast_to(subquery, from_table_name, where_clause.column_name)
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+        outer_key = "#{outer_key}::#{cast_to}" if cast_to
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{inner_sql}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -198,11 +198,18 @@ module Exwiw
       def compile_ast(query_ast, count_only: false)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if count_only
                  "COUNT(*)"
                elsif query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
                end
@@ -225,14 +232,36 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table
+      # (see MysqlAdapter#compile_scope_join for the full rationale). The DISTINCT
+      # forces the engine to materialize the id-set once and probe this table by
+      # it, instead of full-scanning and re-evaluating a correlated subquery per
+      # row; it also dedups, so the join is row-for-row identical to
+      # `<col> IN (subquery)`.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter.rb

@@ -242,6 +242,43 @@ module Exwiw
       private def null_preserving(ast, column, masked_expr)
         "CASE WHEN #{ast.from_table_name}.#{column.name} IS NOT NULL THEN #{masked_expr} ELSE NULL END"
       end
+
+      # Split an outer query's WHERE clauses into the scope id-set clauses to
+      # lift into a materialized derived-table JOIN (see each adapter's
+      # #compile_scope_join) and the remaining plain clauses (kept in WHERE).
+      # Returns [scope_clauses, plain_clauses]; #partition keeps each clause in
+      # its original order *within* its own group. The two groups are emitted in
+      # different SQL positions (a JOIN vs the WHERE), so their interleaving is
+      # irrelevant — only the order within each group matters, and that is kept.
+      private def partition_scope_clauses(where_clauses)
+        where_clauses.partition { |where_clause| scope_subquery_clause?(where_clause) }
+      end
+
+      # Whether a WHERE clause is a scope id-set probe that should be lifted into
+      # a JOIN against a materialized derived table. Only the SelectSubquery /
+      # UnionSubquery shapes (reverse_scope UNION, forward cascade, single
+      # referenced_by) qualify: they project over potentially huge tables and, as
+      # `<col> IN (subquery)`, can degrade into a correlated DEPENDENT SUBQUERY
+      # re-evaluated per outer row. The flat ids_field `Subquery` is deliberately
+      # left as a plain IN — it is a small, bounded, uncorrelated probe.
+      private def scope_subquery_clause?(where_clause)
+        where_clause.is_a?(Exwiw::QueryAst::WhereClause) &&
+          where_clause.operator == :in_subquery &&
+          (where_clause.value.is_a?(Exwiw::QueryAst::SelectSubquery) ||
+            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery))
+      end
+
+      # The bare name of the single column a scope subquery projects, used to
+      # reference it inside the materialized derived table. For a UNION the
+      # output column name comes from the first arm.
+      private def subquery_projection_name(subquery)
+        case subquery
+        when Exwiw::QueryAst::SelectSubquery
+          subquery.query.columns.first.name
+        when Exwiw::QueryAst::UnionSubquery
+          subquery.queries.first.columns.first.name
+        end
+      end
     end
 
     # @params [Exwiw::QueryAst] query_ast

data/lib/exwiw/query_ast_builder.rb

@@ -2,8 +2,8 @@
 
 module Exwiw
   class QueryAstBuilder
-    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, allow_forward: true)
-      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, allow_forward: allow_forward).run
+    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
+      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path).run
     end
 
     # Scope-column mode classification for a single table. One of
@@ -49,17 +49,20 @@ module Exwiw
 
     attr_reader :table_name, :table_by_name, :dump_target
 
-    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, allow_forward: true)
+    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
       @table_name = table_name
       @table_by_name = table_by_name
       @dump_target = dump_target
       @logger = logger
       @allow_reverse = allow_reverse
-      # @allow_forward gates the "scope via an indirectly-scoped belongs_to
-      # parent" rescue (build_belongs_to_scoped_clause). Disabled while building a
-      # parent/child subquery so a single forward hop never recurses into another
-      # (which could loop on a belongs_to cycle).
-      @allow_forward = allow_forward
+      # @forward_path is the chain of tables currently being forward-resolved by
+      # the "scope via an indirectly-scoped belongs_to parent" rescue
+      # (build_belongs_to_scoped_clause). Each forward hop appends the table it is
+      # descending from, so the rescue recurses N levels (users -> end_users ->
+      # end_user_profiles -> ...) and stops only on a real belongs_to cycle: a
+      # table already on the path is not re-resolved, falling through to
+      # :unscopable instead of looping forever.
+      @forward_path = forward_path
     end
 
     def run
@@ -187,10 +190,11 @@ module Exwiw
         next if relation.nil? || relation.polymorphic?
 
         # Build the child's own extraction query. allow_reverse:false stops a
-        # chain of FK-less tables from recursing back into each other;
-        # allow_forward:false stops the child from forward-scoping back through
-        # this very table (which would loop).
-        child_query = self.class.run(other.name, table_by_name, dump_target, @logger, allow_reverse: false, allow_forward: false)
+        # chain of FK-less tables from recursing back into each other; adding this
+        # table to forward_path stops the child from forward-scoping back through
+        # it (which would loop) while still letting the child forward-scope
+        # through other tables.
+        child_query = self.class.run(other.name, table_by_name, dump_target, @logger, allow_reverse: false, forward_path: @forward_path + [table.name])
 
         # Only an *already constrained* child narrows anything; an unconstrained
         # child would select every fk value (i.e. dump all) and not help.
@@ -248,12 +252,12 @@ module Exwiw
           next
         end
 
-        # Build the referencer's own scoped extraction query. allow_reverse /
-        # allow_forward are disabled to bound recursion exactly as the
-        # single-referencer path does (a referencer scopable only via its own
-        # reverse/forward rescue would loop or, worse, recurse back into this
-        # table).
-        ref_query = self.class.run(referencer.name, table_by_name, dump_target, @logger, allow_reverse: false, allow_forward: false)
+        # Build the referencer's own scoped extraction query. allow_reverse is
+        # disabled and this table is added to forward_path to bound recursion
+        # exactly as the single-referencer path does (a referencer that could only
+        # be scoped by recursing back into this table would loop); the referencer
+        # may still forward-scope through other tables.
+        ref_query = self.class.run(referencer.name, table_by_name, dump_target, @logger, allow_reverse: false, forward_path: @forward_path + [table.name])
 
         unless ref_query.where_clauses.any? || ref_query.join_clauses.any?
           @logger.warn(
@@ -297,6 +301,13 @@ module Exwiw
     # them out of a full dump. Returns nil when there is no single, unambiguous
     # scopable parent, leaving the caller on the unscopable path.
     private def build_belongs_to_scoped_clause(table)
+      # This table plus every ancestor currently being forward-resolved. A
+      # candidate parent already on this path would close a belongs_to cycle, so
+      # it is skipped; threading the grown path into the parent build lets the
+      # cascade recurse N hops (users -> end_users -> end_user_profiles -> ...)
+      # and terminate only when a table reappears.
+      forward_path = @forward_path + [table.name]
+
       candidates = table.belongs_tos.filter_map do |relation|
         # A polymorphic belongs_to points at several parent tables through one
         # column, so it cannot project to a single parent id set; skip it.
@@ -305,10 +316,15 @@ module Exwiw
         parent = table_by_name[relation.table_name]
         next if parent.nil?
 
+        # Cycle guard: descending into a parent already on the forward path would
+        # loop (a -> b -> a). Stop, leaving this table on the :unscopable path.
+        next if forward_path.include?(parent.name)
+
         # Build the parent's own scoped query. allow_reverse stays true so the
-        # parent may be scoped via referenced_by; allow_forward:false bounds this
-        # to a single forward hop so a belongs_to cycle cannot loop.
-        parent_query = self.class.run(parent.name, table_by_name, dump_target, @logger, allow_reverse: true, allow_forward: false)
+        # parent may be scoped via referenced_by, and forward scoping stays
+        # enabled so a parent that is itself scoped via *its* parent resolves
+        # too — this is what makes the cascade multi-hop.
+        parent_query = self.class.run(parent.name, table_by_name, dump_target, @logger, allow_reverse: true, forward_path: forward_path)
 
         # Only a constrained parent narrows anything; an unconstrained parent
         # would select every pk (i.e. dump all) and not help.
@@ -460,11 +476,18 @@ module Exwiw
       return :direct if directly_scoped?(table)
       return :via_path if build_join_clauses_scoped(table).any?
       return :referenced_by if @allow_reverse && build_referenced_by_clause(table)
-      return :via_scoped_parent if @allow_forward && build_belongs_to_scoped_clause(table)
+      return :via_scoped_parent if forward_scope_allowed?(table) && build_belongs_to_scoped_clause(table)
 
       :unscopable
     end
 
+    # True when this table may still attempt the forward "scope via a scoped
+    # belongs_to parent" rescue: it is not already on the forward-resolution
+    # path, so descending into its parent cannot revisit it (a belongs_to cycle).
+    private def forward_scope_allowed?(table)
+      !@forward_path.include?(table.name)
+    end
+
     private def build_scoped(table)
       ast = QueryAst::Select.new
       ast.from(table.name)
@@ -502,11 +525,13 @@ module Exwiw
         end
       end
 
-      if @allow_forward
+      if forward_scope_allowed?(table)
         # Belongs_to a parent that is itself scoped but carries no scope column of
         # its own (so via_path cannot terminate on it) — e.g. a hub table scoped
-        # only via referenced_by. Constrain this table to that parent's in-scope
-        # ids so its rows ride along instead of being dumped in full.
+        # only via referenced_by, or a parent that is itself scoped through *its*
+        # parent. Constrain this table to that parent's in-scope ids so its rows
+        # ride along instead of being dumped in full; the parent build recurses
+        # the cascade further up.
         parent_clause = build_belongs_to_scoped_clause(table)
         if parent_clause
           ast.where(parent_clause)
@@ -514,12 +539,13 @@ module Exwiw
         end
       end
 
-      # Only the genuine top-level build (no rescue disabled) is allowed to fail
-      # hard. The Runner/ExplainRunner pre-flight (validate_scope!) rejects
-      # unscopable tables before extraction, so a top-level build never
-      # legitimately lands here; if it does, raise rather than emit an unfiltered
-      # (potential full PII) dump.
-      if @allow_reverse && @allow_forward
+      # Only the genuine top-level build (allow_reverse on, forward_path empty —
+      # i.e. no rescue subquery in progress) is allowed to fail hard. The
+      # Runner/ExplainRunner pre-flight (validate_scope!) rejects unscopable
+      # tables before extraction, so a top-level build never legitimately lands
+      # here; if it does, raise rather than emit an unfiltered (potential full
+      # PII) dump.
+      if @allow_reverse && @forward_path.empty?
         raise ArgumentError, scope_unscopable_message(table)
       end
 

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.2"
+  VERSION = "0.8.4"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.2
+  version: 0.8.4
 platform: ruby
 authors:
 - Shia