jpie 3.7.0 → 3.8.2

data/PERFORMANCE_BASELINE.md

@@ -1,102 +1,461 @@
-# jpie Performance Baseline
-
-This document outlines the performance metrics for key operations within the `jpie` gem, comparing the initial baseline with the results after implementing all optimizations.
-
-## Performance Comparison
-
-| Metric | Original Baseline | After Optimizations | Improvement |
-|--------|-------------------|---------------------|-------------|
-| `resource_loader_find_avg_ms` | 0.0182 | 0.0003 | **60x faster** |
-| `resource_loader_find_for_model_avg_ms` | 0.036 | 0.0003 | **120x faster** |
-| `jsonapi_object_avg_ms` | 0.0003 | 0.0001 | **3x faster** |
-| `serialize_single_user_avg_ms` | 0.2168 | 0.0982 | **2.2x faster** |
-| `serialize_10_users_avg_ms` | 1.4318 | 0.975 | **1.5x faster** |
-| `serialize_single_user_query_count` | 4 | 4 | N/A |
-| `serialize_10_users_query_count` | 40 | 40 | N/A |
-| `serialize_single_user_allocations` | 997 | 977 | **2% reduction** |
-| `serialize_10_users_allocations` | 9934 | 9716 | **2% reduction** |
-| `relationship_definitions_avg_ms` | 0.0036 | 0.0002 | **18x faster** |
-| `resource_instantiation_avg_ms` | 0.0003 | 0.0004 | Similar |
-
-## Implemented Optimizations
-
-### 1. ResourceLoader Caching (Phase 4)
-- Thread-safe caching for `find` and `find_for_model` methods
-- Eliminates repeated `constantize` calls
-- **60-120x improvement** in resource class lookups
-
-### 2. jsonapi_object Memoization (Phase 5)
-- Memoized the static JSON:API object
-- Returns frozen object to prevent mutations
-- **3x improvement** in object generation
-
-### 3. Relationship Definitions Caching (Phase 6)
-- Memoized `relationship_definitions` computation
-- Returns frozen array to prevent mutations
-- **18x improvement** in relationship metadata access
-
-### 4. Optional Count Query (Phase 7)
-- `total_count` only computed when pagination is applied
-- Avoids unnecessary COUNT queries for non-paginated requests
-- Reduces database load
-
-### 5. Eager Loading DSL (Phase 8)
-- New `eager_load` class method for resources
-- Automatically included in preloading without explicit `include` param
-- Helps eliminate N+1 queries at the resource level
-
-### 6. Preload for Serialization Hook (Phase 9)
-- New `preload_for_serialization` class method
-- Called before serialization with all records
-- Enables batch-loading of data needed by `meta` methods
-- Thread-local storage for preloaded data
-
-## Usage Examples
-
-### Eager Loading DSL
-
-```ruby
-class WorkstreamResource < ApplicationResource
-  eager_load :conversation, :owners
-
-  # These associations will be automatically eager-loaded
-  # even without an explicit ?include= param
-end
-```
+# jpie performance baseline: N+1 queries and memory bloat
+
+This document records the measured baseline for the request-level performance
+problem: controller actions with large `include=` trees fire hundreds to
+thousands of SQL queries and allocate hundreds of MB per request. Production
+examples: `/workstreams` with 25 include paths (~300 queries at page 10, up to
+`page[size]=100`), `/policies` with 17 include paths at `page[size]=1` (kills
+the app on AI-conversation data; one index request allocated ~600MB),
+`/evidence_checks` with a 45-id batch filter and files at depth 3 (~1,250
+queries), `/incidents` with 18 include paths at page 20 (577 ActiveRecord
+calls, 2,794ms), and `/suggested_risk_scenarios` with only two include paths
+(541 ActiveRecord calls, 2,059ms). The last two fail for opposite reasons; see
+"Two shapes, two causes" below.
+
+The numbers come from the benchmark suite in `spec/benchmarks/`. The evaluation
+behind them ran 32 independent code-path analyses; a second wave verified the
+top findings empirically with probe specs. Every mechanism below carries a
+measured probe result, not an estimate.
 
-### Preload for Serialization Hook
-
-```ruby
-class WorkstreamResource < ApplicationResource
-  def self.preload_for_serialization(records, context = {})
-    # Batch-load stats for all records
-    stats = Preloaders::StatsPreloader.call(records: records)
-    stats  # Store in preloaded_data for access in meta
-  end
-
-  def meta(...)
-    stats = self.class.preloaded_data[record.id] || record.loading_stats
-    super.merge(loading: stats)
-  end
-end
+## How to run
+
+```bash
+rake benchmark
 ```
 
-## Metric Interpretation
-
-- **`resource_loader_find_avg_ms`**: Average time to find a resource class by type
-- **`resource_loader_find_for_model_avg_ms`**: Average time to find a resource class by model
-- **`jsonapi_object_avg_ms`**: Average time to generate the static `jsonapi` object
-- **`serialize_single_user_avg_ms`**: Average time to serialize a single `User` record
-- **`serialize_10_users_avg_ms`**: Average time to serialize a collection of 10 `User` records
-- **`serialize_single_user_query_count`**: Number of SQL queries for single `User` serialization
-- **`serialize_10_users_query_count`**: Number of SQL queries for 10 `User` records serialization
-- **`serialize_single_user_allocations`**: Object allocations for single `User` serialization
-- **`serialize_10_users_allocations`**: Object allocations for 10 `User` records serialization
-- **`relationship_definitions_avg_ms`**: Average time to compute relationship definitions
-- **`resource_instantiation_avg_ms`**: Average time to instantiate a resource
-
-## Notes
-
-- Query counts remain the same in benchmarks as they test the serializer directly
-- Real-world N+1 reduction comes from the `eager_load` DSL and `preload_for_serialization` hook
-- Allocation reduction is modest but consistent across serialization
+This runs `spec/benchmarks/` with `PERFORMANCE=1`, prints a results table, and
+writes `tmp/request_benchmark_results.json` (per-scenario query counts, query
+shapes, wall time, allocated/retained bytes, allocation sites, RSS). Diff that
+file against this baseline after any optimization. `DEBUG_QUERIES=1` also dumps
+raw SQL logs to `tmp/benchmarks/query_logs/`.
+
+The suite seeds a stand-in domain (`Assignment`, `Gadget`, `Fault`, `Edition`,
+`Remark`, and others) that copies the production include-tree topology: wide +
+deep trees, polymorphic hops, shared prefixes, STI-free but otherwise faithful.
+The `assignments_full` tree mirrors the production `/workstreams` request
+path-for-path; `assignments_single_conversation` mirrors the `/policies` killer
+(fat document bodies, a 600-remark conversation, ActiveStorage files, per-remark
+children); `gadgets_evidence_shape` mirrors `/evidence_checks` (a 45-id `id_in`
+batch, two branches converging on ActiveStorage files, polymorphic leaves at
+depth 2-3); `faults_index` mirrors `/incidents` (link joins to depth-3 gadget
+and appliance targets whose resources carry resource-default preloads and
+per-record meta, plus a full discussion thread); `suggestions_polymorphic`
+mirrors `/suggested_risk_scenarios` (two depth-1 paths, one of them
+polymorphic, onto target resources that under-declare their preloads).
+
+## Baseline (large dataset, sqlite, 2026-08-14)
+
+The seeds sit above the worst production reports on purpose: a 600-remark AI
+conversation, 40 editions with ~64KB document bodies, ~4KB remark bodies,
+fanout 10 on subject inspections plus link-owned inspections. The heaviest
+scenarios overshoot the observed ceilings (600MB allocated, 1,250 queries) so
+the bottleneck curves stay visible past them.
+
+| Scenario | Primary | Included | Queries | ms (median) | Alloc MB | Retained MB |
+|---|---|---|---|---|---|---|
+| assignments_full (page 10, 25 paths) | 10 | 2,701 | 782 | 1,499 | 133.9 | 14.1 |
+| assignments_full + authz hook | 10 | 2,701 | 3,403 | 2,200 | 168.2 | 14.6 |
+| assignments_single_conversation (page 1, 17 paths) | 1 | 2,191 | 1,728 | 1,376 | 147.4 | 24.6 |
+| assignments_single_conversation + authz hook | 1 | 2,191 | 4,029 | 2,079 | 176.8 | 25.0 |
+| gadgets_evidence_shape (45-id batch, 9 paths) | 45 | 1,656 | 2,550 | 1,386 | 132.3 | 8.1 |
+| assignments page 1 / 25 / 50 / 100 (fixed tree) | 1-100 | 1,317-16,382 | 345/1,502/2,702/5,102 | 634-11,148 | 60.0/257.2/466.9/884.5 | 6.9-102.1 |
+| assignments tree depth 1 / 2 / full (fixed page 10) | 10 | 67/1,470/2,701 | 129/1,294/782 | 66/639/1,499 | 8.4/64.5/133.9 | 1.7/13.7/14.1 |
+| users_deep_include (control: clean resources) | 20 | 400 | 3 | 97 | 12.0 | 1.6 |
+
+The reproduction exceeds every production report. Queries: 782 at the
+`/workstreams` page-10 shape, 2,550 at the `/evidence_checks` shape (vs 1,250
+observed), 1,728 for ONE primary record at the `/policies` shape, 5,102 at
+page 100. Memory: 884.5MB allocated at page 100 (vs ~600MB observed), 133.9MB
+at page 10. The authorization-hook variants add a narrowing Pundit-style scope
+on every model; production sits between the plain and authz numbers because
+only some models carry narrowing scopes. The control scenario shows the same
+serializer doing 400 included records in 3 queries when no mechanism below is
+active — the domain shape alone is not the problem.
+
+## Verified mechanisms
+
+### N+1 query drivers
+
+1. **Per-visit authorization re-check — FIXED** (with mechanism 2, see
+   "Optimizations landed"). The include walk runs the
+   `authorization_scope` hook once per parent record, per hop, per include path.
+   When the hook changes the scope's SQL, each visit fires one filter query.
+   Probe: 10 posts, `include=comments,comments.author`: 3 queries without a
+   hook, 73 with a narrowing hook. No memoization: 10 distinct users were
+   re-authorized 50 times. Shared prefixes multiply visits (each path re-visits
+   its whole prefix). Scaling: queries ≈ parents × path-visits.
+   Anchors: `include_filtering.rb` (`filter_loaded_records`, `narrowed?`),
+   `includes_serialization.rb`.
+   Baseline signature: 1,170× single-row authorization plucks in
+   `assignments_full_authz`.
+
+2. **Narrowing resource-default scopes re-check per visit — FIXED** (same
+   change as mechanism 1). A resource whose
+   `records` narrows the model scope (production: `MessageResource.records =
+   super.client_visible`) triggers the same per-visit re-check without any
+   authorization hook.
+   Baseline signature: 570× `SELECT remarks.id WHERE body IS NOT NULL AND id = ?`
+   in `assignments_full`; 3,270× at page 100.
+
+3. **ActiveStorage includes never preload — FIXED.** `filter_includable` kept
+   only include keys with an AR reflection; `has_many_attached :files` defines
+   `files_attachments`, not `files`, so the key dropped silently. Serialization
+   then probed attachments per record (2N+1 queries; 2,340 of the 2,550
+   queries in `gadgets_evidence_shape`). The fix maps the attachment key to
+   `{files_attachments: {blob: {}}}` in the preload hash and reads blobs
+   through the loaded associations. See "Optimizations landed" below.
+
+4. **Resource-default preloads reached depth 1 only — FIXED** (see
+   "Optimizations landed" #3). Two separate passes load data, and they did not
+   cover the same ground:
+   - `scope_with_includes` preloads the include tree at **every** depth. It
+     builds one nested hash from all paths, so a 4-level path costs 4 queries,
+     not 4 per record. Control: `users_deep_include` serializes 400 records in
+     3 queries; `assignments_single_conversation` serializes 2,191 records at
+     depth 4 in 22 queries.
+   - `preload_included_resource_associations` applies each resource's own
+     `records` preloads to include targets. It used to read
+     `include_path.split(".").first`, so only **depth-1** targets got them.
+
+   A resource at depth 2 or deeper therefore serialized with its own
+   associations unloaded. Every `meta` method that branches on `loaded?` took
+   the fallback branch and queried per record. Production:
+   `SelectedControlResource.records` preloads `evidence_checks`,
+   `selected_control_checks`, and `framework_criteria`, but `/incidents`
+   reaches `SelectedControl` at depth 2 through
+   `related_controls.selected_control`, so none of the three apply.
+   Anchor: `include_preloading.rb` (`preload_included_resource_associations`).
+   Baseline signature: 390 of 415 queries in `faults_index` are per-record;
+   40× draft-edition and 40× published-edition lookups in `assignments_full`.
+
+   The depth-1 pass itself is sound, including through a polymorphic hop. Probe:
+   `GET /assignments?include=subject` (polymorphic to `Gadget`) batches all five
+   of `GadgetResource`'s declared preloads into one query each, and the count
+   stays flat as records grow. Regression spec:
+   `polymorphic_resource_preload_spec.rb`. Read the two facts together before
+   blaming jpie for an N+1: at depth 1 a declared preload is applied, so a
+   per-record query there means the resource never declared it.
+
+5. **Resources that never declare the preloads their own reads need.** A
+   `meta` method or attribute that walks an association costs one query per
+   serialized record when the resource's `records` scope does not preload it.
+   jpie cannot infer the need; nothing in the include tree names the
+   association. This is an application fault, not a library limit, and it is
+   the whole cost of a shallow request. Production `/suggested_risk_scenarios`
+   sends two depth-1 paths and still runs 541 ActiveRecord calls:
+   `RiskScenarioTemplateResource#meta` walks `inherent_risk_template.category`
+   with no `records` override (2 queries per template);
+   `SelectedControlResource#baseline` reads `control_template`, which its
+   preload list omits; `ComplianceScopeableResource` exposes `in_scope` and
+   `in_scope_justification`, which probe and then load
+   `resource_framework_scopes`, and no resource preloads them.
+   Baseline signature: `suggestions_polymorphic` runs one `baseline_estimates`
+   and one `estimate_categories` query per serialized repair plan, while every
+   association a resource does declare loads in a single batched query.
+
+6. **Virtual sort disables everything.** A virtual (resource-method) sort field
+   materializes the whole filtered table as an Array before pagination, and
+   `scope_with_includes` bails on Arrays, so nothing preloads. Probe: 10 users,
+   `include=posts.comments`: 3 queries with a column sort, 41 with a virtual
+   sort; the primary SELECT has no LIMIT. Not in the sample production requests,
+   but a cliff any client can trigger.
+
+### Memory drivers
+
+Measured attribution for `assignments_full` (133.9MB allocated): the top
+allocation sites are ActiveSupport inflector string churn (24.9MB, driven by
+uncached type-name inflection and `ResourceLoader` resolution per record and per
+identifier), the include walk itself (`includes_serialization.rb`, 10.9MB), and
+row materialization. At page 100 (884.5MB) the inflector alone allocates
+163MB and the include walk 68MB.
+
+1. **Shared included records serialize once per primary.** The dedupe by
+   `type-id` runs after full serialization, so a page of 50 primaries sharing
+   one included record serializes it 50 times and discards 49. Probe: the
+   shared-record request allocates 90% of what 50 distinct records allocate,
+   with 1 record in `included`. Multiplies every other per-record cost by page
+   size on shared subtrees.
+
+2. **`record.attributes` churn.** Exactly 3 fresh full-column hash
+   materializations per model-backed attribute per serialized record
+   (`model_has_attribute?` twice, value read once). Probe: 420 serialized
+   records produced 1,680 `#attributes` calls; the AR attribute machinery
+   accounts for ~21% of request allocations.
+
+3. **Uncached name resolution.** `ResourceLoader.find_for_model` and type-name
+   inflection run per record, per identifier, and per include-hop visit —
+   thousands of calls per request; the inflector is the single largest
+   allocation site in every heavy scenario.
+
+4. **Full-column AR materialization.** Fat text columns (document bodies,
+   remark bodies) load for every included record and dominate retained-during-
+   request memory; the response then holds the document again, and meta can hold
+   it a third time (production `PolicyVersionResource` serializes `html` in
+   attributes and meta; `PolicyResource#meta` adds a `preview` copy).
+
+Retained-after-request memory stays under ~102MB in every scenario (25MB
+outside the page-100 stress point): the bloat is
+transient garbage plus heap high-water marks, not a leak.
+
+### Checked and cleared
+
+The evaluation also refuted several suspicions, with measurements: pagination
+LIMIT does not break include batching (the count query costs 1); the
+`preload`-vs-`includes` decision is query-neutral; polymorphic and STI hops
+batch correctly at load time; the meta resource instance is cheap (~2MB); the
+attribute-transform dup chain never runs on GET; query tracking retains only
+per-request SQL references and is off in production.
+
+## Optimizations applied
+
+- **Name-resolution caching** (`ResourceLoader` + `TypeConversion`): memory
+  driver 3 above. Allocations drop 15-22% and median wall time drops 40-55%
+  across every scenario, with identical query counts. Page 100: 884.5MB ->
+  694.1MB and 11.1s -> 5.2s. The inflector leaves the top allocation sites;
+  the include walk is now the largest jpie-owned allocator.
+- **Shared include context** (dedupe before serialization): memory driver 1
+  above. A record reached from several primaries serializes once, not once per
+  primary. Allocations drop 4-10% on multi-primary pages (page 100: 884.5MB ->
+  797.4MB) and stay flat on single-primary scenarios, where nothing is shared.
+  Query counts are identical. Pages with heavy cross-primary overlap (many
+  primaries sharing the same reference records) gain far more: the probe case
+  of 50 primaries sharing one record cut its included-serialization work 50x.
+  These figures come from the 884.5MB baseline, before name-resolution caching
+  landed. Re-run the benchmark on top of that change for the combined number.
+
+## Reading the numbers
+
+- Query count is the primary regression metric. It transfers from sqlite to
+  postgres; wall time does not (sqlite has near-zero per-query latency, so the
+  same request is far slower against production postgres).
+- `allocated_mb` (memory_profiler) is the primary memory metric. RSS deltas are
+  indicative only.
+- The scaling scenarios pin the shape: queries and bytes grow linearly with
+  page size at a fixed tree, and with tree size at a fixed page. Cost ≈
+  parents × path-visits × per-visit mechanisms.
+
+## Optimizations landed
+
+### 1. Preload attachment includes (mechanism 3)
+
+`filter_includable` now maps an attachment include key to the association pair
+ActiveStorage defines for it, and `blobs_for` reads blobs through the loaded
+associations. Regression spec: `spec/jsonapi_spec/query_parameters/`
+`attachment_include_preload_spec.rb` pins `GET /users?include=avatar` at 3
+queries for any record count (was 2N+1).
+
+Before/after (large dataset; scenarios without attachment includes are
+query-identical, confirming no behavior change outside the mechanism):
+
+| Scenario | Queries before | Queries after | Alloc MB before | Alloc MB after |
+|---|---|---|---|---|
+| gadgets_evidence_shape | 2,550 | 146 (−94%) | 132.3 | 89.5 |
+| assignments_single_conversation | 1,728 | 330 (−81%) | 147.4 | 119.5 |
+| assignments_single_conversation + authz | 4,029 | 2,631 | 176.8 | 148.9 |
+| assignments_full (no attachment path) | 782 | 782 | 133.8 | 133.8 |
+
+The 330 remaining queries in the `/policies` shape are mostly the narrowing
+resource-scope re-check (mechanism 2) — the next target.
+
+### 2. Vet include-filter verdicts once per request (mechanisms 1 and 2)
+
+A request-scoped `IncludeFilterCache` computes each related class's filter
+scopes once, gives each record id one verdict per (class, scope kind), and is
+warmed from the already-loaded include tree: one vetting query per class
+instead of one per parent record per path visit. Pure memoization — the
+allowed set is identical, id for id; the IDOR regression specs hold
+unchanged. Regression spec: `include_filter_batching_spec.rb`.
+
+#### First recording (2026-08-14, base: the attachment-preload branch)
+
+Measured before the dedupe-included change (#66) landed on main. Kept for the
+record — the query counts still hold, the allocation figures do not.
+
+| Scenario | Queries before | Queries after | Alloc MB before | Alloc MB after |
+|---|---|---|---|---|
+| assignments_full | 782 | 153 (−80%) | 133.8 | 111.8 |
+| assignments_full + authz | 3,403 | 176 (−95%) | 168.2 | 114.0 |
+| assignments_single_conversation | 330 | 22 (−93%) | 119.5 | 102.8 |
+| assignments_single_conversation + authz | 2,631 | 34 (−99%) | 148.9 | 104.7 |
+| assignments_page_100 | 5,102 | 1,233 (−76%) | 884.5 | 741.0 |
+| gadgets_evidence_shape (no narrowed scope — control) | 146 | 146 | 89.5 | 79.2 |
+
+#### Second recording (2026-08-17, base: main at 8bff12d, after #64 and #66)
+
+Re-measured after the rebase onto main. Both columns come from one sequential
+run of `rake benchmark` on the same machine, large dataset.
+
+| Scenario | Queries before | Queries after | Alloc MB before | Alloc MB after | ms(med) before | ms(med) after |
+|---|---|---|---|---|---|---|
+| assignments_full | 782 | 153 (−80%) | 98.8 | 80.4 | 708.6 | 522.3 |
+| assignments_full + authz | 3,403 | 176 (−95%) | 133.6 | 82.8 | 1,208.1 | 473.9 |
+| assignments_single_conversation | 330 | 22 (−93%) | 92.6 | 79.6 | 534.1 | 376.4 |
+| assignments_single_conversation + authz | 2,631 | 34 (−99%) | 122.4 | 81.6 | 997.8 | 402.4 |
+| assignments_page_100 | 5,102 | 1,233 (−76%) | 632.2 | 512.3 | 4,800.1 | 3,128.2 |
+| gadgets_evidence_shape (no narrowed scope — control) | 146 | 146 | 64.9 | 57.9 | 443.6 | 340.2 |
+
+Every query count reproduces the first recording exactly, which confirms #66
+changed no query counts in these scenarios. The allocation figures dropped on
+both sides, because #66 cut allocations independently — so the earlier "before"
+column overstated the saving this change alone delivers.
+
+Cumulative against the original baseline: the /policies shape is down from
+1,728 queries to 22; the /evidence_checks shape from 2,550 to 146; page 100
+from 5,102 to 1,233. The remaining residue is mostly mechanism 4 (per-record
+meta lookups: 120 of the 153 queries in assignments_full).
+
+### 3. Apply resource-default preloads at every depth (mechanism 4)
+
+`preload_included_resource_associations` read only the first segment of each
+include path, so a resource reached at depth 2 or deeper never received the
+preloads its own `records` scope declares. It now nests the paths into one tree
+and walks it, applying each target class's preloads at every hop. Shared
+prefixes collapse, so a prefix is visited once however many paths cross it, and
+a polymorphic hop continues once per class present in the loaded targets.
+
+The walk costs no association queries: every hop reads targets that
+`scope_with_includes` already loaded. Pure batching — included counts are
+identical, scenario for scenario, and the whole suite passes unchanged.
+Regression spec: `deep_resource_preload_spec.rb` pins a depth-2 target at a
+flat query count (36 now, 141 before).
+
+Large dataset, before and after:
+
+| Scenario | Queries before | Queries after | Alloc MB before | Alloc MB after |
+|---|---|---|---|---|
+| faults_index | 415 | 81 (−80%) | 117.0 | 114.4 |
+| assignments_full | 292 | 183 (−37%) | 83.7 | 82.4 |
+| assignments_full + authz | 315 | 206 (−35%) | 86.0 | 84.7 |
+| gadgets_single | 87 | 34 (−61%) | 3.09 | 2.57 |
+| assignments_page_1 | 108 | 55 (−49%) | 40.2 | 39.5 |
+| assignments_page_25 | 582 | 385 (−34%) | 156.3 | 154.1 |
+| assignments_page_50 | 1,027 | 693 (−33%) | 280.8 | 277.2 |
+| assignments_page_100 | 1,830 | 1,293 (−29%) | 526.2 | 520.5 |
+| gadgets_evidence_shape (control) | 150 | 150 | 59.2 | 59.3 |
+| assignments_single_conversation (control) | 22 | 22 | 79.7 | 80.0 |
+| suggestions_polymorphic (control) | 84 | 84 | 9.37 | 9.37 |
+| users_deep_include (control) | 3 | 3 | 9.83 | 9.84 |
+
+The controls carry the argument. `suggestions_polymorphic` does not move,
+because its targets already sat at depth 1 — that request's cost is
+mechanism 5, in the application, and no jpie change touches it.
+`gadgets_evidence_shape` and `users_deep_include` do not move either: neither
+reaches a preload-declaring resource below depth 1.
+
+Allocations barely move because the saving is query count, not object count.
+Wall time on sqlite is too noisy between runs to attribute — the same scenario
+varied by 2x across runs with identical query counts. Against production
+postgres, where per-query latency is real, a 334-query drop on `faults_index`
+is the whole point.
+
+## Two shapes, two causes (2026-08-17)
+
+Two staging requests look alike from the outside: both are index pages, both
+spend about 2 seconds, both fire hundreds of ActiveRecord calls. They fail for
+opposite reasons, and the fix for one does nothing for the other.
+
+| | `/incidents` | `/suggested_risk_scenarios` |
+|---|---|---|
+| Include paths | 18, up to depth 3 | 2, both depth 1 |
+| Where targets sit | depth 2-3 | depth 1 |
+| Did jpie apply the target's preloads? | No — depth-1 pass missed them | Yes |
+| Cause | jpie: mechanism 4 | application: mechanism 5 |
+| Fix belongs in | jpie | kiln resources |
+| Benchmark scenario | `faults_index` | `suggestions_polymorphic` |
+| Status | fixed: 415 → 81 queries | open, needs kiln preloads |
+
+### The /incidents shape: mechanism 4
+
+`faults_index` reproduces the production `/incidents` request: 18 include
+paths, `page[size]=20`, `sort=-created_at`, `filter[archived]=false`. Three
+link branches reach `Gadget` and `Appliance` at depth 2-3. Both resources now
+carry what production carries: a `records` scope that preloads what `meta`
+reads, and `meta` methods that fall back to a query when nothing preloaded.
+
+**This change moved the older scenarios too.** `Gadget` and `Appliance` sit in
+the `assignments_full` tree as well, so that scenario went from 153 queries to
+292 on the same code. No optimization regressed; the benchmark domain became
+more faithful. Compare future runs against the table below, not against the
+2026-08-14 recordings.
+
+| Scenario | Primary | Included | Queries | ms (median) | Alloc MB |
+|---|---|---|---|---|---|
+| faults_index (page 20, 18 paths) | 20 | 4,290 | 415 | 2,430 | 117.0 |
+| assignments_full (re-recorded) | 10 | 2,701 | 292 | 1,548 | 83.7 |
+| assignments_full + authz (re-recorded) | 10 | 2,701 | 315 | 1,088 | 86.0 |
+| gadgets_evidence_shape (re-recorded) | 45 | 1,656 | 150 | 767 | 59.2 |
+| assignments_single_conversation (unchanged) | 1 | 2,191 | 22 | 432 | 79.7 |
+| users_deep_include (control) | 20 | 400 | 3 | 95 | 9.8 |
+
+Adding the suggestion domain changed none of these counts. Every scenario above
+reproduced its query count exactly on the run that first measured
+`suggestions_polymorphic`, which confirms the new tables and resources sit
+outside the other include trees.
+
+390 of the 415 `faults_index` queries are per-record. The include tree itself
+costs about 25. The breakdown, against 45 gadgets and 25 appliances reached at
+depth 2-3:
+
+| Count | Query | Source | After the depth fix |
+|---|---|---|---|
+| 70 | `archivals` by owner | `records` preload, unreachable at depth 2 | 2 |
+| 45 | `criteria.id` pluck | `criterium_ids` attribute | 1 |
+| 45 + 45 | `inspections` order-limit, then exists | inspection history in `meta` | 1 |
+| 45 + 45 | `standard_scopes` rows, then exists | scope verdict in `meta` | 1 |
+| 45 | `gadget_checks` exists | checklist verdict in `meta` | 1 |
+| 25 + 25 | `stream_entries` by stream | append-only read, no association | 50 |
+
+Only the last row was beyond jpie's reach: a raw stream read has no association
+for any preloader to batch. Everything above it collapsed to a handful of
+batched queries once resource-default preloads applied at every depth, taking
+the scenario from 415 queries to 81. The 50 stream reads are now the largest
+single item, and they need a change in the application.
+
+### The /suggested_risk_scenarios shape: mechanism 5
+
+`suggestions_polymorphic` reproduces the production request:
+`filter[account_level]=true`, `sort=-priority`, and two include paths —
+`repair_plan` (plain) and `suggestable` (polymorphic, onto `Gadget` and
+`Appliance`). `SuggestionResource.records` copies the production scope, with
+the plain hop on `includes` and the polymorphic hop on `preload`.
+
+Both targets sit at depth 1, so jpie applies their `records` preloads. The
+split in the results is the whole point:
+
+- `Gadget` declares its preloads. All five load in one batched query each,
+  however many gadgets the page holds.
+- `RepairPlan` declares none, and its `meta` walks
+  `baseline_estimate.estimate_category`. That costs two queries per serialized
+  plan — the exact `Risk::InherentRiskTemplate` and `Risk::Category` bands in
+  the staging trace.
+- `Appliance` reads an append-only stream, which has no association to preload
+  at any depth.
+
+| Dataset | Primary | Included | Queries | ms (median) | Alloc MB |
+|---|---|---|---|---|---|
+| small | 5 | 10 | 22 | 14 | - |
+| medium | 25 | 37 | 46 | 31 | 2.9 |
+| large | 100 | 112 | 84 | 203 | 9.4 |
+
+Queries grow far slower than the page, because the repair-plan pool caps at 12
+and only 25 of the 100 suggestables are appliances. Production behaves the same
+way: 18 suggestions reached 13 distinct templates. The large run splits like
+this:
+
+| Count | Query | Verdict |
+|---|---|---|
+| 25 + 25 | `stream_entries` by stream | unfixable by preloading |
+| 12 + 12 | `baseline_estimates`, then `estimate_categories` | resource never declared it |
+| 2 | `archivals` by owner type | correct: one per polymorphic class |
+| 8 | primary, both target classes, plan, declared preloads | correct: batched |
+
+Only 8 of the 84 queries are structural. Every association a resource declares
+is batched, including through the polymorphic hop, and the 24 `RepairPlan`
+queries are the shape a one-line preload removes.
+
+No change to jpie removes the `RepairPlan` queries. The resource has to ask for
+the preload. In production the three to fix are
+`RiskScenarioTemplateResource` (add `inherent_risk_template: :category`),
+`SelectedControlResource` (add `control_template` to its existing preload
+list), and `ComplianceScopeableResource` (add `resource_framework_scopes`,
+which alone accounts for 82 of the 541 calls).

data/README.md

@@ -532,7 +532,7 @@ end
 
 ### Resource-Level Meta
 
-Resource-level meta appears within each resource object. By default, the gem automatically includes `created_at` and `updated_at` timestamps in ISO8601 format if the model responds to these methods.
+Resource-level meta appears within each resource object. By default, the gem automatically includes `created_at` and `updated_at` timestamps if the model responds to these methods. The gem passes the raw `Time` to the JSON encoder, so Rails renders ISO8601 at the precision `ActiveSupport::JSON::Encoding.time_precision` sets — three sub-second digits by default. A timestamp you also expose as an attribute therefore reads the same in both places.
 
 You can also define custom meta in two ways:
 
@@ -578,8 +578,8 @@ The instance method has access to the model instance via `resource`. Custom meta
       "email": "john@example.com"
     },
     "meta": {
-      "created_at": "2024-01-15T10:30:00Z",
-      "updated_at": "2024-01-15T10:30:00Z",
+      "created_at": "2024-01-15T10:30:00.000Z",
+      "updated_at": "2024-01-15T10:30:00.000Z",
       "name_length": 8,
       "custom_field": "value"
     },

data/Rakefile

@@ -7,6 +7,11 @@ RSpec::Core::RakeTask.new(:spec)
 
 task default: :spec
 
+desc "Run the performance benchmark suite (writes tmp/request_benchmark_results.json)"
+task :benchmark do
+  sh({ "PERFORMANCE" => "1" }, "bundle", "exec", "rspec", "spec/benchmarks", "--format", "progress")
+end
+
 # Override release task to require OTP code
 # Usage: GEM_HOST_OTP_CODE=123456 rake release
 Rake::Task["release"].enhance do

data/lib/json_api/active_storage/serialization.rb

@@ -11,19 +11,36 @@ module JSONAPI
         attachment = record.public_send(attachment_name)
         return nil unless attachment.respond_to?(:attached?)
 
-        attachment.attached? ? serialize_attached(attachment) : empty_attachment_value(attachment)
+        blobs = blobs_for(attachment_name, record)
+        return blobs.map { |blob| serialize_blob_identifier(blob) } if attachment.is_a?(::ActiveStorage::Attached::Many)
+
+        blobs.first ? serialize_blob_identifier(blobs.first) : nil
       end
 
-      def serialize_attached(attachment)
+      # Blobs for an attachment. Reads through the ActiveStorage associations
+      # when the include preloader loaded them (zero queries); otherwise falls
+      # back to the probing reader (attached? + blobs, one query each).
+      def blobs_for(attachment_name, record)
+        attachment = record.public_send(attachment_name)
+        return [] unless attachment.respond_to?(:attached?)
+
         if attachment.is_a?(::ActiveStorage::Attached::Many)
-          return attachment.blobs.map { |blob| serialize_blob_identifier(blob) }
+          many_blobs(attachment, record.association(:"#{attachment_name}_attachments"))
+        else
+          one_blob(attachment, record.association(:"#{attachment_name}_attachment"))
         end
+      end
+
+      def many_blobs(attachment, association)
+        return association.target.filter_map(&:blob) if association.loaded?
 
-        serialize_blob_identifier(attachment.blob)
+        attachment.attached? ? attachment.blobs.to_a : []
       end
 
-      def empty_attachment_value(attachment)
-        attachment.is_a?(::ActiveStorage::Attached::Many) ? [] : nil
+      def one_blob(attachment, association)
+        return [association.target&.blob].compact if association.loaded?
+
+        attachment.attached? ? [attachment.blob].compact : []
       end
 
       def serialize_blob_identifier(blob)