exwiw 0.8.3 → 0.8.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 90d949cb54565ed644b599102cfabc16236fe2d83523594cd7b14f269118a9b2
-  data.tar.gz: 7236bb6ee6b9dda38aec93491f0266e334890e3c271b4f7d5d072d46a1e0f88b
+  metadata.gz: ee150e9fd830a7829be8c0eebbeca19f0e2d1b1dbcce50fe78e63d62e878997c
+  data.tar.gz: 31377340d8737ece209e1c0755eeac49ac40930562503e2c4d04f6362938e9d9
 SHA512:
-  metadata.gz: bcfac0aaf220b55dfa3172f94d2ad0a6e828f53253e367aa9203db7061695fa6a9188e22a92c18d51bd9a55b6a9bf840cf780a3647d4aca2a0cccff1a785a347
-  data.tar.gz: d891fc7101fea30d674b933213715a8a4534c459e191b9bf4fce31b3bda8a9be037bf09627419f43ef914d2a9288aa40e1cd4836a4d8e1e0901d73e320218cd3
+  metadata.gz: 26151e0bf763b6f48e993d2025b4bac804a373fe2e3cfd71bb70d91b86b453826838ec683ebfd3f4f4336bb83b8a12d567d7dbda7f8a4ef48fc7f50bbe98145a
+  data.tar.gz: b29a2da42750dde93cd53eb56a11a592478134b690ccba31b59cd312f3685dad2713e63897d09cb77246a8be37dfe55890f1a8505c1aa8b5fd020b7bc7760a03

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+## [0.8.4] - 2026-06-24
+
+### Fixed
+
+- **Scope id-sets are materialized and probed by JOIN instead of `<col> IN (subquery)`, removing a correlated full-scan on large tables.** The three id-set scope shapes — the multi-referencer `reverse_scope` `UNION`, the single-referencer reverse/`referenced_by` extraction, and the multi-hop forward (`via_scoped_parent`) cascade — were emitted as `<col> IN (<subquery>)`. On a large global-identity table (e.g. `users`) MySQL cannot turn a `UNION` subquery into a materialized semi-join and falls back to its IN-to-`EXISTS` rewrite: a correlated `DEPENDENT SUBQUERY`/`DEPENDENT UNION` re-evaluated **per outer row**, so the driving table is full-scanned and the union re-run for every row (the plan that ran for minutes and timed out on a production-scale identity table, even for an empty tenant). These clauses are now lifted into a `JOIN` against a materialized derived table — `JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids ON <table>.<col> = ids.exwiw_scope_id` — so the engine evaluates the id set **once** (the `DISTINCT` makes the derived table non-mergeable, hence materialized) and probes the outer table by primary key. The `DISTINCT` also dedups, so the result set is identical to the old `IN` form; the cascade nests the same way (each level materialized once); NULL exclusion, the forward-path cycle guard, single-parent/polymorphic skips, and PostgreSQL's `uuid`/`varchar` `::text` reconciliation are all preserved. All three SQL adapters (mysql / postgresql / sqlite). See the [README](README.md#why-a-join-not-in-subquery).
+
 ## [0.8.3] - 2026-06-24
 
 ### Fixed

data/README.md

@@ -179,15 +179,18 @@ Each table is resolved as follows:
   nearest such table and applies the scope filter there (the same join machinery
   the single-target mode uses).
 - **`belongs_to` a parent that is itself scoped but carries no scope column of its
-  own** → exwiw constrains this table to the parent's in-scope ids via a subquery
-  (`fk IN (SELECT parent.pk FROM <parent's scoped query>)`). This covers a *hub*
-  table that has no scope column and is scoped only because an extractable child
-  references it (see referenced-by below): the hub's other `belongs_to` children
-  ride along to just the in-scope rows instead of being dumped in full. The parent
-  itself may be scoped the same way, so this **cascades across multiple hops**
-  (each a single unambiguous scopable parent) and the subquery nests
-  correspondingly; the recursion terminates on a genuine `belongs_to` cycle (a
-  table already on the path is left `:unscopable` rather than looped on).
+  own** → exwiw constrains this table to the parent's in-scope ids by joining it to
+  the parent's scoped query, materialized as a derived table
+  (`JOIN (SELECT DISTINCT parent.pk … FROM <parent's scoped query>) … ON fk = …`).
+  This covers a *hub* table that has no scope column and is scoped only because an
+  extractable child references it (see referenced-by below): the hub's other
+  `belongs_to` children ride along to just the in-scope rows instead of being dumped
+  in full. The parent itself may be scoped the same way, so this **cascades across
+  multiple hops** (each a single unambiguous scopable parent) and the derived-table
+  JOINs nest correspondingly; the recursion terminates on a genuine `belongs_to`
+  cycle (a table already on the path is left `:unscopable` rather than looped on).
+  (See [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery) for the
+  materialization rationale.)
 - **Cannot be scoped at all** (no scope column and no path to one) → exwiw
   **aborts** and lists the offending tables, so an unscoped table is never silently
   dumped in full. For each, either declare a `scope_column`, add a `belongs_to`
@@ -568,15 +571,19 @@ The same type filter is applied on the join path — and in the matching `delete
 ActiveStorage is handled automatically — no ActiveStorage-specific configuration is required. The `has_one_attached` / `has_many_attached` macros don't add a column to the owning model; they generate ordinary associations that exwiw already understands:
 
 - **`active_storage_attachments`** is the polymorphic join row (`belongs_to :record, polymorphic: true` + `belongs_to :blob`). `exwiw:schema:generate` expands the polymorphic `record` into one `belongs_to` per model that declared `has_*_attached` (found via the generated `has_* ..., as: :record` reflections), exactly like any other [polymorphic `belongs_to`](#polymorphic-belongs_to). So only the attachments whose owner is among the dumped rows are extracted.
-- **`active_storage_blobs`** has no `belongs_to` of its own (attachments point *at* it), so it has no path to the dump target. exwiw narrows it via **reverse / "referenced_by" extraction**: a parent table referenced by exactly one constrained, non-polymorphic child is constrained to just the referenced ids instead of dumping every row:
+- **`active_storage_blobs`** has no `belongs_to` of its own (attachments point *at* it), so it has no path to the dump target. exwiw narrows it via **reverse / "referenced_by" extraction**: a parent table referenced by exactly one constrained, non-polymorphic child is constrained to just the referenced ids instead of dumping every row. The id set is materialized once and joined back (see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery)):
 
   ```sql
   SELECT active_storage_blobs.* FROM active_storage_blobs
-  WHERE active_storage_blobs.id IN (
-    SELECT active_storage_attachments.blob_id FROM active_storage_attachments
-    WHERE active_storage_attachments.record_id IN (/* owner subquery */)
-      AND active_storage_attachments.record_type = '...'
-  )
+  JOIN (
+    SELECT DISTINCT exwiw_scope_src_0.blob_id AS exwiw_scope_id
+    FROM (
+      SELECT active_storage_attachments.blob_id FROM active_storage_attachments
+      WHERE active_storage_attachments.record_id IN (/* owner subquery */)
+        AND active_storage_attachments.record_type = '...'
+    ) AS exwiw_scope_src_0
+  ) AS exwiw_scope_ids_0
+    ON active_storage_blobs.id = exwiw_scope_ids_0.exwiw_scope_id
   ```
 
   `active_storage_variant_records` also references blobs, but since it has no path of its own to the dump target it doesn't constrain anything and is ignored as a referencer — blobs stays narrowed to the attachment-referenced ids. (A parent referenced by *multiple* constrained children currently falls back to dumping all of its rows.)
@@ -603,18 +610,22 @@ The automatic reverse extraction above narrows a table referenced by **exactly o
 }
 ```
 
-produces (each arm reuses that referencer's own scope, so a per-tenant run keeps only that tenant's ids):
+produces (each arm reuses that referencer's own scope, so a per-tenant run keeps only that tenant's ids; the `UNION` id set is materialized once and joined back — see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery)):
 
 ```sql
 SELECT users.* FROM users
-WHERE users.id IN (
-  SELECT customers.user_id FROM customers WHERE <customers' scope> AND customers.user_id IS NOT NULL
-  UNION
-  SELECT staff.user_id FROM staff WHERE <staff' scope> AND staff.user_id IS NOT NULL
-  UNION
-  SELECT business_entity_customers.kantan_yoyaku_user_id FROM business_entity_customers
-    WHERE <…' scope> AND business_entity_customers.kantan_yoyaku_user_id IS NOT NULL
-)
+JOIN (
+  SELECT DISTINCT exwiw_scope_src_0.user_id AS exwiw_scope_id
+  FROM (
+    SELECT customers.user_id FROM customers WHERE <customers' scope> AND customers.user_id IS NOT NULL
+    UNION
+    SELECT staff.user_id FROM staff WHERE <staff' scope> AND staff.user_id IS NOT NULL
+    UNION
+    SELECT business_entity_customers.kantan_yoyaku_user_id FROM business_entity_customers
+      WHERE <…' scope> AND business_entity_customers.kantan_yoyaku_user_id IS NOT NULL
+  ) AS exwiw_scope_src_0
+) AS exwiw_scope_ids_0
+  ON users.id = exwiw_scope_ids_0.exwiw_scope_id
 ```
 
 Notes:
@@ -625,6 +636,19 @@ Notes:
 - **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`. The cascade is **multi-hop**, so a table several `belongs_to` hops below the reverse-scoped table (e.g. `end_user_profiles → end_users → users`) also tightens automatically, with no config of its own.
 - Works in both single-target and scope-column mode. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
 
+### Why a JOIN, not `IN (subquery)`
+
+Every scope id-set above — the multi-referencer `reverse_scope` `UNION`, the single-referencer reverse extraction, and the multi-hop forward cascade — is emitted as a `JOIN` to a `SELECT DISTINCT` derived table rather than `<col> IN (<subquery>)`:
+
+```sql
+… JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids
+    ON <table>.<col> = ids.exwiw_scope_id
+```
+
+Both forms select the **same rows** — the `DISTINCT` dedups, so the join never fans out — but the query plans differ sharply on a large table. As `<col> IN (… UNION …)`, MySQL cannot turn a `UNION` subquery into a materialized semi-join and falls back to its IN-to-`EXISTS` rewrite: a **correlated `DEPENDENT SUBQUERY`** re-evaluated for every outer row, i.e. a full scan of the (potentially huge) outer table multiplied by the cost of the union. The derived-table form forces the engine to evaluate the id set **once** (the `DISTINCT` makes the derived table non-mergeable, hence materialized) and then probe the outer table by its primary key. On a global-identity table such as `users` this is the difference between a full table scan and an index lookup; the cascade nests the same way, so each level is materialized once instead of being re-evaluated by the level above.
+
+All three SQL adapters (mysql / postgresql / sqlite) emit this shape. PostgreSQL additionally reconciles a `uuid`/`varchar` type mismatch by casting the join key and the projected id to `text`, exactly as the old `IN` form did.
+
 ### Rails-managed tables (special `type` values)
 
 Some tables are owned by Rails itself rather than the application — they have no ActiveRecord model and Rails reserves the right to evolve their column shape between versions (e.g. `schema_migrations`, `ar_internal_metadata`). exwiw treats them as a distinct category via the `type` field on a table config:

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -229,11 +229,18 @@ module Exwiw
       def compile_ast(query_ast, count_only: false)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if count_only
                  "COUNT(*)"
                elsif query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
                end
@@ -256,14 +263,43 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table:
+      #
+      #   JOIN (SELECT DISTINCT src.<proj> AS exwiw_scope_id
+      #         FROM (<subquery>) AS src) AS ids
+      #     ON <table>.<col> = ids.exwiw_scope_id
+      #
+      # The DISTINCT makes the derived table non-mergeable, so MySQL materializes
+      # the id-set once and probes this table by it (PK/index lookup) — instead
+      # of full-scanning this table and re-evaluating a correlated
+      # `IN (… UNION …)` per row (the DEPENDENT SUBQUERY / IN-to-EXISTS fallback,
+      # which a UNION subquery cannot be turned into a materialized semi-join).
+      # DISTINCT also dedups, so the join never fans out: the row set is identical
+      # to `<col> IN (subquery)`.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -301,9 +301,16 @@ module Exwiw
       def compile_ast(query_ast, select_cast_to: nil)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  cols = query_ast.columns.map { |col| compile_column_name(query_ast, col) }
                  cols = cols.map { |c| "#{c}::#{select_cast_to}" } if select_cast_to
@@ -337,14 +344,45 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table
+      # (see MysqlAdapter#compile_scope_join for the full rationale). The DISTINCT
+      # forces the engine to materialize the id-set once and probe this table by
+      # it, instead of full-scanning and re-evaluating a correlated subquery per
+      # row; it also dedups, so the join is row-for-row identical to
+      # `<col> IN (subquery)`.
+      #
+      # Type reconciliation mirrors the old IN form: when the outer column and
+      # the projected id clash (e.g. uuid vs varchar), #compile_subquery already
+      # casts every arm to text, so the derived `exwiw_scope_id` is text and the
+      # outer key is cast to match.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+
+        inner_sql = compile_subquery(subquery, outer_table: from_table_name, outer_column: where_clause.column_name)
+        cast_to = subquery_cast_to(subquery, from_table_name, where_clause.column_name)
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+        outer_key = "#{outer_key}::#{cast_to}" if cast_to
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{inner_sql}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -198,11 +198,18 @@ module Exwiw
       def compile_ast(query_ast, count_only: false)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if count_only
                  "COUNT(*)"
                elsif query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
                end
@@ -225,14 +232,36 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table
+      # (see MysqlAdapter#compile_scope_join for the full rationale). The DISTINCT
+      # forces the engine to materialize the id-set once and probe this table by
+      # it, instead of full-scanning and re-evaluating a correlated subquery per
+      # row; it also dedups, so the join is row-for-row identical to
+      # `<col> IN (subquery)`.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter.rb

@@ -242,6 +242,43 @@ module Exwiw
       private def null_preserving(ast, column, masked_expr)
         "CASE WHEN #{ast.from_table_name}.#{column.name} IS NOT NULL THEN #{masked_expr} ELSE NULL END"
       end
+
+      # Split an outer query's WHERE clauses into the scope id-set clauses to
+      # lift into a materialized derived-table JOIN (see each adapter's
+      # #compile_scope_join) and the remaining plain clauses (kept in WHERE).
+      # Returns [scope_clauses, plain_clauses]; #partition keeps each clause in
+      # its original order *within* its own group. The two groups are emitted in
+      # different SQL positions (a JOIN vs the WHERE), so their interleaving is
+      # irrelevant — only the order within each group matters, and that is kept.
+      private def partition_scope_clauses(where_clauses)
+        where_clauses.partition { |where_clause| scope_subquery_clause?(where_clause) }
+      end
+
+      # Whether a WHERE clause is a scope id-set probe that should be lifted into
+      # a JOIN against a materialized derived table. Only the SelectSubquery /
+      # UnionSubquery shapes (reverse_scope UNION, forward cascade, single
+      # referenced_by) qualify: they project over potentially huge tables and, as
+      # `<col> IN (subquery)`, can degrade into a correlated DEPENDENT SUBQUERY
+      # re-evaluated per outer row. The flat ids_field `Subquery` is deliberately
+      # left as a plain IN — it is a small, bounded, uncorrelated probe.
+      private def scope_subquery_clause?(where_clause)
+        where_clause.is_a?(Exwiw::QueryAst::WhereClause) &&
+          where_clause.operator == :in_subquery &&
+          (where_clause.value.is_a?(Exwiw::QueryAst::SelectSubquery) ||
+            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery))
+      end
+
+      # The bare name of the single column a scope subquery projects, used to
+      # reference it inside the materialized derived table. For a UNION the
+      # output column name comes from the first arm.
+      private def subquery_projection_name(subquery)
+        case subquery
+        when Exwiw::QueryAst::SelectSubquery
+          subquery.query.columns.first.name
+        when Exwiw::QueryAst::UnionSubquery
+          subquery.queries.first.columns.first.name
+        end
+      end
     end
 
     # @params [Exwiw::QueryAst] query_ast

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.3"
+  VERSION = "0.8.4"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.3
+  version: 0.8.4
 platform: ruby
 authors:
 - Shia