jpie 3.8.1 → 3.8.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d9e7099c27c6aec0de41a095baa563dbc63eb934b44ab8195a9e3a51f5152c4e
-  data.tar.gz: 86ee7fbdc3cf8414d9829bbf4f42ac7e016603f8a884a0b674d8e9b869cf1b8d
+  metadata.gz: 4d78efd83c17a8d1e0b9c7e13966c083de82d95d92463dd5b945a4478d981c30
+  data.tar.gz: 71594c1b5687a4d20268d79b4e83e3760d399f3467029b8f0787329b6cf599fa
 SHA512:
-  metadata.gz: a013761543aba53b56e575f8b2f51d3004d1cae53e53570bf506bd744cb2d38891e43a5188ad11ee338923aacdd5c8bc4cf107e1772f5cb73173aafb590d1859
-  data.tar.gz: 105cbbcb454083ebbfae7db54d9fbec5e76c39d3c1d1823c16b34de071abeae47e4cdf5923aadeb3224e609ee85c07712ed37651caa1122719501ac1c479ae07
+  metadata.gz: e1bdbd01c0c181b59be183a8d72decb7d3e13f254c92cd477285f69a909953d4bfe58f6dff6001ddbc437036888e30b6ab9e921675fd65a70003206f714f37ac
+  data.tar.gz: 6bdadef777e82f021a67b820e77e61a7c2051dbb5cddc037447b0c42c26c3142cc738f8a6aa864ef26b2cfb19a3a0f5e6f6a42afd5ec483b1049bd15d24acf52

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    jpie (3.8.1)
+    jpie (3.8.2)
       actionpack (~> 8.1, >= 8.1.0)
       pg_query (>= 4)
       prosopite (>= 1)

data/PERFORMANCE_BASELINE.md

@@ -5,9 +5,12 @@ problem: controller actions with large `include=` trees fire hundreds to
 thousands of SQL queries and allocate hundreds of MB per request. Production
 examples: `/workstreams` with 25 include paths (~300 queries at page 10, up to
 `page[size]=100`), `/policies` with 17 include paths at `page[size]=1` (kills
-the app on AI-conversation data; one index request allocated ~600MB), and
+the app on AI-conversation data; one index request allocated ~600MB),
 `/evidence_checks` with a 45-id batch filter and files at depth 3 (~1,250
-queries).
+queries), `/incidents` with 18 include paths at page 20 (577 ActiveRecord
+calls, 2,794ms), and `/suggested_risk_scenarios` with only two include paths
+(541 ActiveRecord calls, 2,059ms). The last two fail for opposite reasons; see
+"Two shapes, two causes" below.
 
 The numbers come from the benchmark suite in `spec/benchmarks/`. The evaluation
 behind them ran 32 independent code-path analyses; a second wave verified the
@@ -26,15 +29,19 @@ shapes, wall time, allocated/retained bytes, allocation sites, RSS). Diff that
 file against this baseline after any optimization. `DEBUG_QUERIES=1` also dumps
 raw SQL logs to `tmp/benchmarks/query_logs/`.
 
-The suite seeds a stand-in domain (`Assignment`, `Gadget`, `Edition`, `Remark`,
-and others) that copies the production include-tree topology: wide + deep trees,
-polymorphic hops, shared prefixes, STI-free but otherwise faithful. The
-`assignments_full` tree mirrors the production `/workstreams` request
+The suite seeds a stand-in domain (`Assignment`, `Gadget`, `Fault`, `Edition`,
+`Remark`, and others) that copies the production include-tree topology: wide +
+deep trees, polymorphic hops, shared prefixes, STI-free but otherwise faithful.
+The `assignments_full` tree mirrors the production `/workstreams` request
 path-for-path; `assignments_single_conversation` mirrors the `/policies` killer
 (fat document bodies, a 600-remark conversation, ActiveStorage files, per-remark
 children); `gadgets_evidence_shape` mirrors `/evidence_checks` (a 45-id `id_in`
 batch, two branches converging on ActiveStorage files, polymorphic leaves at
-depth 2-3).
+depth 2-3); `faults_index` mirrors `/incidents` (link joins to depth-3 gadget
+and appliance targets whose resources carry resource-default preloads and
+per-record meta, plus a full discussion thread); `suggestions_polymorphic`
+mirrors `/suggested_risk_scenarios` (two depth-1 paths, one of them
+polymorphic, onto target resources that under-declare their preloads).
 
 ## Baseline (large dataset, sqlite, 2026-08-14)
 
@@ -98,18 +105,55 @@ active — the domain shape alone is not the problem.
    `{files_attachments: {blob: {}}}` in the preload hash and reads blobs
    through the loaded associations. See "Optimizations landed" below.
 
-4. **Per-record meta queries, any depth.** Instance `meta` that touches an
-   association or runs a finder fires per serialized record. The batch pass
-   (`preload_included_resource_associations`) only applies resource-default
-   preloads to depth-1 targets, and only from `records` scope values. Probe: 1
-   profile query per serialized user at depth 1 and depth 2; putting the path in
-   `include=` collapses it to one batched query. Production:
-   `PolicyResource#meta` runs 3 version lookups per policy;
-   `PolicyVersionResource#meta` touches `resource.policy` per version.
-   Baseline signature: 40× draft-edition and 40× published-edition lookups in
-   `assignments_full`.
-
-5. **Virtual sort disables everything.** A virtual (resource-method) sort field
+4. **Resource-default preloads reached depth 1 only — FIXED** (see
+   "Optimizations landed" #3). Two separate passes load data, and they did not
+   cover the same ground:
+   - `scope_with_includes` preloads the include tree at **every** depth. It
+     builds one nested hash from all paths, so a 4-level path costs 4 queries,
+     not 4 per record. Control: `users_deep_include` serializes 400 records in
+     3 queries; `assignments_single_conversation` serializes 2,191 records at
+     depth 4 in 22 queries.
+   - `preload_included_resource_associations` applies each resource's own
+     `records` preloads to include targets. It used to read
+     `include_path.split(".").first`, so only **depth-1** targets got them.
+
+   A resource at depth 2 or deeper therefore serialized with its own
+   associations unloaded. Every `meta` method that branches on `loaded?` took
+   the fallback branch and queried per record. Production:
+   `SelectedControlResource.records` preloads `evidence_checks`,
+   `selected_control_checks`, and `framework_criteria`, but `/incidents`
+   reaches `SelectedControl` at depth 2 through
+   `related_controls.selected_control`, so none of the three apply.
+   Anchor: `include_preloading.rb` (`preload_included_resource_associations`).
+   Baseline signature: 390 of 415 queries in `faults_index` are per-record;
+   40× draft-edition and 40× published-edition lookups in `assignments_full`.
+
+   The depth-1 pass itself is sound, including through a polymorphic hop. Probe:
+   `GET /assignments?include=subject` (polymorphic to `Gadget`) batches all five
+   of `GadgetResource`'s declared preloads into one query each, and the count
+   stays flat as records grow. Regression spec:
+   `polymorphic_resource_preload_spec.rb`. Read the two facts together before
+   blaming jpie for an N+1: at depth 1 a declared preload is applied, so a
+   per-record query there means the resource never declared it.
+
+5. **Resources that never declare the preloads their own reads need.** A
+   `meta` method or attribute that walks an association costs one query per
+   serialized record when the resource's `records` scope does not preload it.
+   jpie cannot infer the need; nothing in the include tree names the
+   association. This is an application fault, not a library limit, and it is
+   the whole cost of a shallow request. Production `/suggested_risk_scenarios`
+   sends two depth-1 paths and still runs 541 ActiveRecord calls:
+   `RiskScenarioTemplateResource#meta` walks `inherent_risk_template.category`
+   with no `records` override (2 queries per template);
+   `SelectedControlResource#baseline` reads `control_template`, which its
+   preload list omits; `ComplianceScopeableResource` exposes `in_scope` and
+   `in_scope_justification`, which probe and then load
+   `resource_framework_scopes`, and no resource preloads them.
+   Baseline signature: `suggestions_polymorphic` runs one `baseline_estimates`
+   and one `estimate_categories` query per serialized repair plan, while every
+   association a resource does declare loads in a single batched query.
+
+6. **Virtual sort disables everything.** A virtual (resource-method) sort field
    materializes the whole filtered table as an Array before pagination, and
    `scope_with_includes` bails on Arrays, so nothing preloads. Probe: 10 users,
    `include=posts.comments`: 3 queries with a column sort, 41 with a virtual
@@ -259,3 +303,159 @@ Cumulative against the original baseline: the /policies shape is down from
 1,728 queries to 22; the /evidence_checks shape from 2,550 to 146; page 100
 from 5,102 to 1,233. The remaining residue is mostly mechanism 4 (per-record
 meta lookups: 120 of the 153 queries in assignments_full).
+
+### 3. Apply resource-default preloads at every depth (mechanism 4)
+
+`preload_included_resource_associations` read only the first segment of each
+include path, so a resource reached at depth 2 or deeper never received the
+preloads its own `records` scope declares. It now nests the paths into one tree
+and walks it, applying each target class's preloads at every hop. Shared
+prefixes collapse, so a prefix is visited once however many paths cross it, and
+a polymorphic hop continues once per class present in the loaded targets.
+
+The walk costs no association queries: every hop reads targets that
+`scope_with_includes` already loaded. Pure batching — included counts are
+identical, scenario for scenario, and the whole suite passes unchanged.
+Regression spec: `deep_resource_preload_spec.rb` pins a depth-2 target at a
+flat query count (36 now, 141 before).
+
+Large dataset, before and after:
+
+| Scenario | Queries before | Queries after | Alloc MB before | Alloc MB after |
+|---|---|---|---|---|
+| faults_index | 415 | 81 (−80%) | 117.0 | 114.4 |
+| assignments_full | 292 | 183 (−37%) | 83.7 | 82.4 |
+| assignments_full + authz | 315 | 206 (−35%) | 86.0 | 84.7 |
+| gadgets_single | 87 | 34 (−61%) | 3.09 | 2.57 |
+| assignments_page_1 | 108 | 55 (−49%) | 40.2 | 39.5 |
+| assignments_page_25 | 582 | 385 (−34%) | 156.3 | 154.1 |
+| assignments_page_50 | 1,027 | 693 (−33%) | 280.8 | 277.2 |
+| assignments_page_100 | 1,830 | 1,293 (−29%) | 526.2 | 520.5 |
+| gadgets_evidence_shape (control) | 150 | 150 | 59.2 | 59.3 |
+| assignments_single_conversation (control) | 22 | 22 | 79.7 | 80.0 |
+| suggestions_polymorphic (control) | 84 | 84 | 9.37 | 9.37 |
+| users_deep_include (control) | 3 | 3 | 9.83 | 9.84 |
+
+The controls carry the argument. `suggestions_polymorphic` does not move,
+because its targets already sat at depth 1 — that request's cost is
+mechanism 5, in the application, and no jpie change touches it.
+`gadgets_evidence_shape` and `users_deep_include` do not move either: neither
+reaches a preload-declaring resource below depth 1.
+
+Allocations barely move because the saving is query count, not object count.
+Wall time on sqlite is too noisy between runs to attribute — the same scenario
+varied by 2x across runs with identical query counts. Against production
+postgres, where per-query latency is real, a 334-query drop on `faults_index`
+is the whole point.
+
+## Two shapes, two causes (2026-08-17)
+
+Two staging requests look alike from the outside: both are index pages, both
+spend about 2 seconds, both fire hundreds of ActiveRecord calls. They fail for
+opposite reasons, and the fix for one does nothing for the other.
+
+| | `/incidents` | `/suggested_risk_scenarios` |
+|---|---|---|
+| Include paths | 18, up to depth 3 | 2, both depth 1 |
+| Where targets sit | depth 2-3 | depth 1 |
+| Did jpie apply the target's preloads? | No — depth-1 pass missed them | Yes |
+| Cause | jpie: mechanism 4 | application: mechanism 5 |
+| Fix belongs in | jpie | kiln resources |
+| Benchmark scenario | `faults_index` | `suggestions_polymorphic` |
+| Status | fixed: 415 → 81 queries | open, needs kiln preloads |
+
+### The /incidents shape: mechanism 4
+
+`faults_index` reproduces the production `/incidents` request: 18 include
+paths, `page[size]=20`, `sort=-created_at`, `filter[archived]=false`. Three
+link branches reach `Gadget` and `Appliance` at depth 2-3. Both resources now
+carry what production carries: a `records` scope that preloads what `meta`
+reads, and `meta` methods that fall back to a query when nothing preloaded.
+
+**This change moved the older scenarios too.** `Gadget` and `Appliance` sit in
+the `assignments_full` tree as well, so that scenario went from 153 queries to
+292 on the same code. No optimization regressed; the benchmark domain became
+more faithful. Compare future runs against the table below, not against the
+2026-08-14 recordings.
+
+| Scenario | Primary | Included | Queries | ms (median) | Alloc MB |
+|---|---|---|---|---|---|
+| faults_index (page 20, 18 paths) | 20 | 4,290 | 415 | 2,430 | 117.0 |
+| assignments_full (re-recorded) | 10 | 2,701 | 292 | 1,548 | 83.7 |
+| assignments_full + authz (re-recorded) | 10 | 2,701 | 315 | 1,088 | 86.0 |
+| gadgets_evidence_shape (re-recorded) | 45 | 1,656 | 150 | 767 | 59.2 |
+| assignments_single_conversation (unchanged) | 1 | 2,191 | 22 | 432 | 79.7 |
+| users_deep_include (control) | 20 | 400 | 3 | 95 | 9.8 |
+
+Adding the suggestion domain changed none of these counts. Every scenario above
+reproduced its query count exactly on the run that first measured
+`suggestions_polymorphic`, which confirms the new tables and resources sit
+outside the other include trees.
+
+390 of the 415 `faults_index` queries are per-record. The include tree itself
+costs about 25. The breakdown, against 45 gadgets and 25 appliances reached at
+depth 2-3:
+
+| Count | Query | Source | After the depth fix |
+|---|---|---|---|
+| 70 | `archivals` by owner | `records` preload, unreachable at depth 2 | 2 |
+| 45 | `criteria.id` pluck | `criterium_ids` attribute | 1 |
+| 45 + 45 | `inspections` order-limit, then exists | inspection history in `meta` | 1 |
+| 45 + 45 | `standard_scopes` rows, then exists | scope verdict in `meta` | 1 |
+| 45 | `gadget_checks` exists | checklist verdict in `meta` | 1 |
+| 25 + 25 | `stream_entries` by stream | append-only read, no association | 50 |
+
+Only the last row was beyond jpie's reach: a raw stream read has no association
+for any preloader to batch. Everything above it collapsed to a handful of
+batched queries once resource-default preloads applied at every depth, taking
+the scenario from 415 queries to 81. The 50 stream reads are now the largest
+single item, and they need a change in the application.
+
+### The /suggested_risk_scenarios shape: mechanism 5
+
+`suggestions_polymorphic` reproduces the production request:
+`filter[account_level]=true`, `sort=-priority`, and two include paths —
+`repair_plan` (plain) and `suggestable` (polymorphic, onto `Gadget` and
+`Appliance`). `SuggestionResource.records` copies the production scope, with
+the plain hop on `includes` and the polymorphic hop on `preload`.
+
+Both targets sit at depth 1, so jpie applies their `records` preloads. The
+split in the results is the whole point:
+
+- `Gadget` declares its preloads. All five load in one batched query each,
+  however many gadgets the page holds.
+- `RepairPlan` declares none, and its `meta` walks
+  `baseline_estimate.estimate_category`. That costs two queries per serialized
+  plan — the exact `Risk::InherentRiskTemplate` and `Risk::Category` bands in
+  the staging trace.
+- `Appliance` reads an append-only stream, which has no association to preload
+  at any depth.
+
+| Dataset | Primary | Included | Queries | ms (median) | Alloc MB |
+|---|---|---|---|---|---|
+| small | 5 | 10 | 22 | 14 | - |
+| medium | 25 | 37 | 46 | 31 | 2.9 |
+| large | 100 | 112 | 84 | 203 | 9.4 |
+
+Queries grow far slower than the page, because the repair-plan pool caps at 12
+and only 25 of the 100 suggestables are appliances. Production behaves the same
+way: 18 suggestions reached 13 distinct templates. The large run splits like
+this:
+
+| Count | Query | Verdict |
+|---|---|---|
+| 25 + 25 | `stream_entries` by stream | unfixable by preloading |
+| 12 + 12 | `baseline_estimates`, then `estimate_categories` | resource never declared it |
+| 2 | `archivals` by owner type | correct: one per polymorphic class |
+| 8 | primary, both target classes, plan, declared preloads | correct: batched |
+
+Only 8 of the 84 queries are structural. Every association a resource declares
+is batched, including through the polymorphic hop, and the 24 `RepairPlan`
+queries are the shape a one-line preload removes.
+
+No change to jpie removes the `RepairPlan` queries. The resource has to ask for
+the preload. In production the three to fix are
+`RiskScenarioTemplateResource` (add `inherent_risk_template: :category`),
+`SelectedControlResource` (add `control_template` to its existing preload
+list), and `ComplianceScopeableResource` (add `resource_framework_scopes`,
+which alone accounts for 82 of the 541 calls).

data/lib/json_api/controllers/concerns/resource_actions/include_preloading.rb

@@ -8,10 +8,14 @@ module JSONAPI
       private
 
       def includes_to_hash(paths)
-        hash = paths.each_with_object({}) do |path, h|
-          path.split(".").reduce(h) { |cur, part| cur[part.to_sym] ||= {} }
+        filter_includable(includes_to_tree(paths), model_class)
+      end
+
+      # Nests dotted include paths into one tree, collapsing shared prefixes.
+      def includes_to_tree(paths)
+        paths.each_with_object({}) do |path, tree|
+          path.split(".").reduce(tree) { |node, part| node[part.to_sym] ||= {} }
         end
-        filter_includable(hash, model_class)
       end
 
       # Preloads associations from the include param so the serializer avoids N+1 queries
@@ -82,18 +86,45 @@ module JSONAPI
         assoc.polymorphic? || (value.present? && hash_contains_polymorphic?(value, assoc.klass))
       end
 
+      # Applies each included resource's own `records` preloads to the records
+      # of that resource's class. Walks the whole include tree: a resource
+      # reached at depth 3 gets its preloads just as one reached at depth 1.
+      #
+      # The tree collapses shared prefixes, so a prefix is visited once however
+      # many paths run through it. Walking costs no association queries — every
+      # hop reads targets that `scope_with_includes` already loaded. The only
+      # queries are the batched resource-default preloads themselves, each of
+      # which replaces one query per record during serialization.
       def preload_included_resource_associations(resources, includes)
         return if includes.empty? || resources.empty?
 
-        includes.each do |include_path|
-          association_name = include_path.split(".").first.to_sym
-          assoc_reflection = model_class.reflect_on_association(association_name)
-          next unless assoc_reflection
+        preload_resource_tree(resources, model_class, includes_to_tree(includes))
+      end
 
-          targets = collect_include_targets(resources, association_name)
+      def preload_resource_tree(records, klass, tree)
+        tree.each do |association_name, subtree|
+          reflection = klass.reflect_on_association(association_name)
+          next unless reflection
+
+          targets = collect_include_targets(records, association_name)
           next if targets.empty?
 
-          apply_preloads_for_targets(targets, assoc_reflection.polymorphic?)
+          apply_preloads_for_targets(targets, reflection.polymorphic?)
+          next if subtree.empty?
+
+          descend_resource_tree(targets, reflection, subtree)
+        end
+      end
+
+      # A polymorphic hop has no single target class, so continue once per
+      # class actually present in the loaded targets.
+      def descend_resource_tree(targets, reflection, subtree)
+        if reflection.polymorphic?
+          targets.group_by(&:class).each do |target_class, records|
+            preload_resource_tree(records, target_class, subtree)
+          end
+        else
+          preload_resource_tree(targets, reflection.klass, subtree)
         end
       end
 

data/lib/json_api/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module JSONAPI
-  VERSION = "3.8.1"
+  VERSION = "3.8.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: jpie
 version: !ruby/object:Gem::Version
-  version: 3.8.1
+  version: 3.8.2
 platform: ruby
 authors:
 - Emil Kampp