paper_trail_diff 0.13.1 → 0.14.0

data/docs/performance.md

@@ -8,14 +8,103 @@ does not establish that a batch API meets that purpose.
 The [follow-up reconstruction investigation](../benchmark/investigation/README.md)
 records the pre-fix N+1 paths and direct-snapshot experiments. Production now uses
 prepared event successors even without changesets, and directly normalizes plain
-root/direct-child states. Scoped and collection-source through batch paths require
-`allow_expensive: true`; that opt-in can still cause N+1 work. Model callbacks,
+root/direct-child states. Scopes and default scopes made of plain conditions,
+and a `limit` or `offset` with a total integer order, are read from prepared
+history; other scoped batch paths, such as owner-dependent scopes, require
+`allow_expensive: true`, and that opt-in can still cause N+1 work. Model callbacks,
 custom types and callable predicates also remain outside a flat-query guarantee.
 
 The benchmark exercises `analyze_scope` with genuine PaperTrail history. It is
 a local SQLite baseline for finding work inside the gem, not a production
 latency guarantee or a comparison of SQL dialects.
 
+## Limited associations: 2026-10-29
+
+A `limit` or `offset` with a total order on integer or boolean columns, and an
+ordered `has_one`, are now read from prepared history: each record's stored
+values decide whether it meets the scope's conditions, and the matching records
+are ranked as the association's query ranks them. Batch methods no longer need
+`allow_expensive: true` for them. The probe's `limited` scenario reports
+articles with `has_many :limited_comments, -> { order(:id).limit(1) }`:
+
+| `limited` | SQL at 1 / 5 / 20 roots | `reify` calls | Model initializations |
+| --- | --- | --- | --- |
+| before, with `allow_expensive: true` | 24 / 72 / 252 | 8 / 40 / 160 | 30 / 150 / 600 |
+| after, without an opt-in | 13 / 17 / 32 | 0 / 0 / 0 | 10 / 50 / 200 |
+
+One query per root remains: the current end loads each article's limited
+comments on its own, because Active Record's preloader applies a limit or
+offset to all owners' records at once rather than to each owner's.
+
+## Scoped associations: 2026-10-29
+
+An association scope, or a target's default scope, made of plain conditions
+(`where(reviewer_id: nil)`, `where.not(body: "hidden")`) used to fall back to
+PT-AT for every record, so batch methods required `allow_expensive: true` for
+it. It is now read from prepared history, checking each record's stored values
+against the scope. The probe below reports articles with their comments,
+through `has_many :scoped_comments, -> { where(reviewer_id: nil) }`
+(`scoped`) and through a `TrackedComment` default scope (`default_scope`):
+
+| Scenario and code | SQL at 1 / 5 / 20 roots | `reify` calls at 1 / 5 / 20 roots |
+| --- | --- | --- |
+| `scoped`, before, with `allow_expensive: true` | 15 / 27 / 72 | 1 / 5 / 20 |
+| `scoped`, after, without an opt-in | 12 / 12 / 12 | 0 / 0 / 0 |
+| `default_scope`, before, with `allow_expensive: true` | 15 / 27 / 72 | 1 / 5 / 20 |
+| `default_scope`, after, without an opt-in | 12 / 12 / 12 | 0 / 0 / 0 |
+
+Model initializations fell from 17 / 85 / 340 (`scoped`) and 19 / 95 / 380
+(`default_scope`) to 10 / 50 / 200, the same as unscoped `comments`. The
+default-scoped comments are read without `reify` because their scope assigns
+no attributes to a new record; one such as `where(hidden: false)` does, so
+those records are still rebuilt with `reify`, though the query count stays
+flat. `limited` (`order(:id).limit(1)`) and `owner` (a scope that takes the
+article) still raise without the opt-in, and cost what they did.
+
+## Fewer `reify` calls: 2026-10-29
+
+The `reify` calls that remained came from one place: building a root's
+snapshot from its own version (`HistoricalSnapshotStore#snapshot_from_version`).
+That snapshot is built from stored attributes only when every requested
+association can be read the same way, and `has_many :through` and polymorphic
+`belongs_to` could not. So the whole root was rebuilt with `reify`, and its
+associations as model instances. Both are now read from stored attributes,
+resolving each part of a through as the record path does, and keeping only
+join records of the `source_type:` when there is one. A type that names no
+class, a default-scoped or unversioned target, and every scoped association
+still take the record path as before.
+
+Counts from `benchmark/investigation/reconstruction.rb` at 1 / 5 / 20 roots.
+`source_type_through` and `polymorphic_belongs_to` are new probe scenarios: a
+holder's links to an author and to a tag, one moving to another author.
+
+| Scenario | `reify` before | `reify` after | Model instances before | Model instances after |
+| --- | --- | --- | --- | --- |
+| `through_has_many` | 7 / 35 / 140 | 0 / 0 / 0 | 36 / 180 / 720 | 14 / 70 / 280 |
+| `source_type_through` | 7 / 35 / 140 | 0 / 0 / 0 | 45 / 225 / 900 | 22 / 110 / 440 |
+| `polymorphic_belongs_to` | 2 / 10 / 40 | 0 / 0 / 0 | 14 / 70 / 280 | 10 / 50 / 200 |
+
+Query counts are unchanged: 20, 20 and 14 at every size. `belongs_to`,
+`comments` and `checkpoints` made no `reify` calls before and are unchanged.
+Model instances still grow with the roots: they are the snapshots' own records
+and, for the two polymorphic scenarios, a few states the activity timeline
+builds as records.
+
+## Through associations: 2026-10-29
+
+A `has_many :through` whose source is a `has_many` used to fall back to PT-AT
+for every record, so batch methods required `allow_expensive: true` for it. It
+now reads prepared history like any other through. The five-root
+`through_has_many` probe below, an author's comments through articles:
+
+| Code | SQL at 1 / 5 / 20 roots | `reify` calls at 1 / 5 / 20 roots |
+| --- | --- | --- |
+| Before, with `allow_expensive: true` | 56 / 200 / 740 | 15 / 75 / 300 |
+| After, without an opt-in | 20 / 20 / 20 | 7 / 35 / 140 |
+
+The `reify` calls still grew with the roots: each root's own versions were
+still rebuilt that way, until the change above.
+
 ## Direct reconstruction follow-up: 2026-09-07
 
 Implementation: `1b6b1b9`, followed by belongs-to discovery fix `f6a4ae2`.
@@ -45,9 +134,10 @@ Ruby memory was about 15.0, 20.8 and 19.9 MiB respectively. These timings do not
 establish a retained-activity improvement.
 
 A querying initialization callback still produced 10 / 26 / 86 queries, and a
-querying historical predicate produced 8 / 12 / 27. Custom behavior, polymorphic
-or advanced graph reconstruction, and non-leaf activity fallbacks may still
-instantiate historical records. There is no blanket no-N+1 guarantee.
+querying historical predicate produced 8 / 12 / 27. Custom behavior, advanced
+graph reconstruction, and non-leaf activity fallbacks may still instantiate
+historical records, as does a polymorphic `belongs_to` to a default-scoped
+class. There is no blanket no-N+1 guarantee.
 
 Production parity matches PaperTrail for STI, enum, decimal, date, boolean, JSON
 and serialized values, including missing fields supplied by callbacks/default
@@ -57,7 +147,7 @@ scripts are not included in the installed gem), reproduce these checks with:
 ```console
 PTD_PRODUCTION=1 mise exec -- bundle exec ruby benchmark/investigation/scalar_parity.rb
 mise exec -- bundle exec ruby benchmark/investigation/reconstruction.rb comments_without_changes 20
-PTD_PROBE_ALLOW_EXPENSIVE=1 mise exec -- bundle exec ruby benchmark/investigation/reconstruction.rb through_has_many 5
+mise exec -- bundle exec ruby benchmark/investigation/reconstruction.rb through_has_many 5
 ```
 
 ## Reproduce
@@ -300,10 +390,13 @@ Deleted history has no original timed counterpart here. Its five queries at both
   a small root ceiling can load a full 250-candidate version batch, subject to the
   separate version budget. Requested analysis history and retained snapshots
   still consume memory proportional to the data involved.
-- **Selection semantics are unchanged.** Historical filters test the states a
-  record held during the window, not continuous-time membership. Scope selection needs a root
-  version in the window; child-only edits do not select an otherwise quiet root.
-
-Further batching of advanced association fallbacks needs temporal parity tests,
-especially around ownership changes. A graph-wide selector for child-only activity
-also needs its own bounded query design.
+- **Selection reads the window's versions.** Historical filters test the states a
+  record held during the window, not continuous-time membership. Activity
+  selection also reaches roots through their requested associations, with one
+  query per association level, not per root. Those queries scan the window's
+  versions of each associated model and their `version_associations` rows; ids
+  stay in subqueries unless `versions.item_id` and the keys differ in type, in
+  which case they are read in slices of 250.
+
+Batching scoped associations needs tests of membership over time like those
+for `has_many :through`, especially around ownership changes.

data/docs/reference.md

@@ -19,6 +19,7 @@ including ones most applications never need.
 - [Ignoring fields](#ignoring-fields)
 - [Reading results](#reading-results)
 - [JSON columns](#json-columns)
+- [What a version recorded](#what-a-version-recorded): `recorded_changes`
 - [Checking your setup](#checking-your-setup)
 - [Models, schemas, and edge cases](#models-schemas-and-edge-cases)
 - [Instrumentation](#instrumentation)
@@ -60,7 +61,7 @@ Take an article created as `"Draft"`, then updated to `"Published"`, then to
                                              only in the table
 ```
 
-Three things follow from this:
+Four things follow from this:
 
 - **A `create` version stands for "no record yet".** Its `reify` returns `nil`.
   Comparing it with a later state reports the record as added
@@ -70,6 +71,14 @@ Three things follow from this:
   change from the current record.
 - **A change shows up between two points, never at one.** That is why the
   timeline methods return steps, each comparing one state with the next.
+- **A change saved without a version joins the change before it.** Writing
+  with `update_columns` or with versioning off creates no version, but it
+  changes the values the record's next version stores, or its current row. The
+  gem reads the result of a change from there, so the write appears in the step
+  for that record's previous recorded change and is attributed to its author.
+  Ending a range at a version leaves out only the writes after it.
+  [`recorded_changes`](#what-a-version-recorded) reads what a version itself
+  recorded.
 
 ## Choosing a method
 
@@ -142,8 +151,9 @@ the current record instead. The step's `to_boundary.kind` is then `:current`.
 This works the same for a window that ends now and one that ended long ago.
 
 Reading the current record has one consequence: it includes changes PaperTrail
-never recorded, such as an `update_columns` call. Those appear in the last step
-of the window.
+never recorded, such as an `update_columns` call. Like any
+[change saved without a version](#how-papertrail-stores-history), each appears
+in the step for that record's last recorded change.
 
 If the window's last change destroyed the record, the window closes on that
 deletion instead. See [a destroyed record](#a-destroyed-record).
@@ -216,6 +226,11 @@ diff.to_h
   `ReversedEndpointsError`, because the result would look valid with every
   change backwards. A current record can be on either side.
 - Both sides must belong to the same record.
+- Associations at a version are read from the history recorded since then, plus
+  the current rows for records that have not changed since. A comment that later
+  moved to another article is still found at the versions where it belonged.
+  This also holds when both sides are versions, or the record has since been
+  deleted.
 - A current record must be saved, not destroyed, and have no unsaved changes.
   The gem reloads it, so in-memory edits and cached associations are ignored.
 - Options: `associations:`, `ignore:`, and `reload_live_endpoints:` (see
@@ -317,9 +332,10 @@ steps.reject(&:empty?).each do |step|
 end
 ```
 
-- Each step's `diff` is what the event at its `source_boundary` changed. Use
-  `source_boundary` for attribution. `to_boundary` is only the next point in
-  time and may belong to someone else.
+- Each step's `diff` is what the event at its `source_boundary` changed, plus
+  anything [saved without a version](#how-papertrail-stores-history) to the
+  same record before its next version. Use `source_boundary` for attribution.
+  `to_boundary` is only the next point in time and may belong to someone else.
 - Boundaries have a `kind`: `:version`, `:current` (the current record), or
   `:destroyed` (after a deletion). They also have `event`, `whodunnit`,
   `recorded_at`, `item_type`, `item_id`, `transaction_id`, and `record`, and
@@ -347,7 +363,7 @@ attributed = steps.reject(&:empty?)
 by_priya = attributed.select { |step| step.source_boundary.whodunnit == "Priya Shah" }
 
 by_priya.first.source_boundary.whodunnit  # "Priya Shah"
-by_priya.first.diff                     # exactly what that one event changed
+by_priya.first.diff                     # what that one event changed
 
 authors = attributed.group_by { |step| step.source_boundary.whodunnit }
                     .transform_values(&:length)
@@ -429,6 +445,13 @@ the range at a version instead (see the
 [checkpoint recipe](#saving-a-checkpoint)), or use a join model with
 `has_many :through`.
 
+PT-AT records HABTM members under the transaction a version was saved in, and
+does not say which record they belong to. If another record with a HABTM of the
+same name, such as a second article, also saved a version in that transaction
+and members were recorded there, reading the root's membership at that version
+raises `IncompleteAssociationHistoryError` instead of mixing both records'
+members. Save each HABTM owner's versions in a transaction of its own.
+
 ## analyze
 
 `analyze(record)` returns an `Analysis` with the net `diff` and the `timeline`
@@ -519,9 +542,18 @@ It accepts `associations:`, `ignore:`, `snapshots:`, `group:`,
 
 ### Which records are selected
 
-- **A record needs its own version in the window.** Editing only a comment does
-  not select its article. Once an article is selected, its comment activity is
-  included when you request `associations: [:comments]`.
+- **A record is selected when it, or a record on a requested association,
+  changed in the window.** With `activity: true`, which `report` always uses,
+  editing only a comment selects its article when you request
+  `associations: [:comments]`. A comment that moved between articles selects
+  both. Nested paths count at every level, and a `has_many :through` counts its
+  join rows too. Without `activity: true`, an analysis compares a record's own
+  versions and cannot show a change to its associations alone, so only a
+  record's own version selects it.
+- **An analysis can still be empty.** A record is selected by any version on its
+  paths, but its analysis compares only the state you requested. A join row
+  whose role changed, or a version that touched only ignored columns, selects a
+  record with nothing to report.
 - **A relation checks current values.** `Order.where(status: "open")` means open
   now, not open during the window. Use `historical_filter:` to select by past
   values.
@@ -533,14 +565,17 @@ It accepts `associations:`, `ignore:`, `snapshots:`, `group:`,
   ID tie-breaker to any other order. If a join repeats a record, it is analyzed
   once, in order of first appearance.
 - Selection still scans the versions in the window, so keep windows reasonable
-  for large tables.
+  for large tables. Each requested association adds one query per level, however
+  many records it selects. Changes to associations are found from PT-AT's
+  `version_associations`, so history recorded before PT-AT was installed selects
+  nothing.
 
 ### Deleted records
 
 A deleted record cannot be tested against a relation's conditions, because it
 has no row. Records that changed in the window but no longer exist are listed in
-`unreachable`. It is not paginated and does not mean those records matched the
-relation.
+`unreachable`, including those selected by a change to their associations. It
+is not paginated and does not mean those records matched the relation.
 
 `include_deleted: true` also analyzes them from their `destroy` versions and
 returns the results in `deleted`, keyed the same way. A value is `nil` when no
@@ -552,7 +587,10 @@ missing_history = report.deleted.select { |_identity, analysis| analysis.nil? }.
 ```
 
 A deleted record's `diff` and `to_snapshot` end at "no record". Its `timeline`
-lists only edits; its `activity_timeline` ends with the deletion.
+lists only edits; its `activity_timeline` ends with the deletion. That last
+step's `diff.record_presence_change.from` is the record just before it was
+deleted, even when it was also created inside the window, where
+`from_snapshot` is `nil`.
 
 ### Selecting by past values: `historical_filter:`
 
@@ -592,13 +630,75 @@ model instance and may run queries of its own.
 
 ### Costly associations
 
-Most associations are loaded for all records together. A few still need queries
-per record: associations with a scope (including `limit` or owner-dependent
-scopes), targets with a default scope, and `has_many :through` associations
-whose source is not a `belongs_to`. Batch methods raise
-`UnsupportedBatchPathError` for these by default and name the association.
-Remove it from `associations:`, or pass `allow_expensive: true` to accept the
-per-record cost. Single-record methods always allow them.
+Most associations are loaded for all records together. That includes an
+association whose scope, or whose target's default scope, is plain conditions
+on the target's own columns:
+
+```ruby
+has_many :active_cards, -> { where(archived: false) }, class_name: "Card"
+has_many :urgent_cards, -> { where(priority: [1, 2]).order(:id) }, class_name: "Card"
+default_scope { where.not(hidden: true) }
+```
+
+The gem reads such a scope once and checks each record's stored values against
+it at every point, so a record that moves into or out of the scope is added or
+removed at the version that moved it. It reads equality, `IN`, `nil`, their
+`where.not` forms, and `and`/`or` combinations of them on integer, boolean,
+and enum columns. String and text columns are read only on SQLite and
+PostgreSQL, and only without a column collation: MySQL and SQL Server compare
+strings without regard to case or trailing spaces by default, which the gem
+cannot repeat.
+
+A `limit` or `offset`, and the order of a `has_one`, are read the same way
+when the order is total and the gem can repeat it:
+
+```ruby
+has_many :first_comments, -> { order(:position, :id).limit(3) }
+has_one :latest_comment, -> { order(id: :desc) }
+```
+
+At every point the gem ranks the records that meet the scope's conditions by
+the order, then skips the offset and keeps the limit, as the association's own
+query would have. The order must reach the primary key, so that no two records
+tie, and each key before it must be an integer, boolean, integer enum, date,
+or datetime column of the target, ascending or descending. NULL sorts where the
+database puts it by default: last in an ascending order on PostgreSQL, first on
+SQLite, MySQL, and SQL Server.
+
+A datetime key, such as `order(created_at: :desc)`, is read when its column
+declares a precision, as Rails 7 and later give every new `datetime` column,
+when history keeps that many fractional digits, and when Active Record stores
+times in UTC, its default. Active Record truncates a time to its column's
+precision before writing it, so history then holds the value the database
+sorts by. YAML history keeps every digit; JSON history, and a `json` or
+`jsonb` object column, keep `ActiveSupport::JSON::Encoding.time_precision`
+digits, 3 by default, so with them only a `datetime(3)` or coarser column is
+read. A time-of-day column is not read: Active Record keeps the date it was
+given in memory but stores only the time.
+
+A `has_many :through` reads its parts as Active Record's own query joins them:
+it keeps each part's conditions but not its `limit`, `offset`, or `has_one`
+cut, so a through over `has_one :latest_comment` holds the records of every
+comment that meets the part's conditions, not only the latest. Its own `limit`
+or `offset` ranks the rows the join yields, so a record reached through two
+join records fills two places, as in the query. A `has_one :through` whose
+own scope or whose parts have a limit, an offset, or a `has_one` order is not
+read.
+
+A few associations still need queries per record: a `limit` or `offset` whose
+order is not total or sorts by another column type (strings follow a
+collation) or by a datetime history cannot rank exactly, scopes that take the
+owner (`->(article) { ... }`), scopes written as SQL strings or ranges, scopes
+that join or select, and conditions on other column types or other tables. Batch methods raise `UnsupportedBatchPathError`
+for these by default and name the association. Remove it from
+`associations:`, or pass `allow_expensive: true` to accept the per-record
+cost. Single-record methods always allow them.
+
+A polymorphic `belongs_to` is loaded together too, with one query per target
+class. Its classes are known only from the records, though, so batch methods
+cannot reject one with a scope they cannot read in advance: a record that
+points at such a class is read on its own, without an error. A scope on the
+polymorphic `belongs_to` itself is rejected.
 
 This check covers known association cases only. Your model callbacks, custom
 attribute types, and callable filters can still run queries. See the
@@ -642,9 +742,57 @@ comments.changed.first.associations["replies"].added
   the association already reports it. The same applies to the foreign key a
   `has_one` or `has_many` child points back with. Without the association, the
   key is an ordinary attribute.
+- A polymorphic `belongs_to` reads each record's target from its type
+  column. Because the targets' classes differ, a path cannot continue past
+  it: `"notable.comments"` raises `UnsupportedAssociationError`. A record
+  whose type names a model without `has_paper_trail` raises
+  `UnversionedAssociationError` when its history is read; the gem cannot
+  check this earlier, since only the record names the class. A type that
+  names no class, such as one a renamed model left in its versions, raises
+  `IncompleteAssociationHistoryError` naming the record.
 - `has_many :through` uses the versioned join model's history, so join-model
   attributes are available. HABTM reports membership and the target's
   attributes; it has no join model, so it has no join attributes.
+- A `has_many :through` holds, at each point, the records the owner reached
+  through the join records it had then. A record that moves to another join
+  record leaves one owner and joins the other, and so does every record under
+  a join record that moves to another owner.
+- A `has_many :through` with `source_type:` holds only the records reached
+  through join records whose type column names that class, as Active Record
+  reads it. A join record that points at another type is never a member, and
+  its changes add no activity step. A join record whose type changes adds or
+  removes its target in the step of that version.
+- A `has_many :through` that goes through a HABTM, as its `through:`, its
+  `source:`, or further down its chain, raises `UnsupportedAssociationError`
+  up front in every method. Active Record joins a HABTM through a hidden
+  model with no versions, so the through cannot be rebuilt from history.
+  Replace the HABTM with a `has_many :through` a versioned join model to
+  compare it. `association_paths` leaves such a through out.
+- A HABTM can only be rebuilt on the root record. PT-AT records membership
+  when the record that owns it saves a version, in rows that name the
+  transaction and the association but not the owner, so a root checkpoint
+  holds the root's membership and no one else's. A HABTM later in a path, such
+  as `"tags.articles"` or `"author.articles.tags"`, raises
+  `UnsupportedAssociationError` up front in every method, even when the path
+  leads back to the root, as `"comments.article.tags"` does.
+  `association_paths` leaves such a path out. To follow another record's
+  memberships, compare that record's own checkpoints, or replace the HABTM
+  with a `has_many :through` a versioned join model.
+- A HABTM with a scope, such as `-> { where(active: true) }`, or whose target
+  has a default scope, raises `UnsupportedAssociationError` up front in every
+  method. PT-AT records a HABTM's members by loading the association through
+  its scope when the owner saves, so a target outside the scope then is never
+  recorded, and one that enters the scope before the owner's next save is a
+  member history cannot show. Replace it with a `has_many :through` a
+  versioned join model, which can be scoped. `association_paths` leaves it
+  out.
+- A root that uses single-table inheritance gets the associations of the
+  class it had at each version, so you can request one that only a subclass
+  declares. Without an `item_subtype` column, PaperTrail stores only the base
+  class; the gem reads the subclass from the type column the version stored.
+  A version where the record was a class without that association raises
+  `UnknownAssociationError`, as a current record of that class would. A
+  stored type that names no class raises `InvalidEndpointError`.
 - Requesting associations for a historical state without PT-AT raises
   `AssociationTrackingUnavailableError`. Unknown names raise
   `UnknownAssociationError`; unsupported macros raise
@@ -671,7 +819,10 @@ end
 ```
 
 Without `synchronize_version_creation_timestamp: false`, the checkpoint takes the
-record's old `updated_at` as its time and lands in the past.
+record's old `updated_at` as its time and lands in the past. Save one record per
+transaction: HABTM membership cannot be read from a transaction in which
+another record with the same HABTM also saved a version (see the
+[HABTM limitation](#habtm-limitation)).
 
 ### Files from ActiveStorage
 
@@ -837,6 +988,56 @@ PaperTrailDiff.nested_changes(from_value, to_value) # also accepts two values
   has the whole change.
 - Keys such as `:a` and `"a"` in one object raise `AmbiguousNestedKeyError`.
 
+## What a version recorded
+
+`recorded_changes(version)` returns the changes PaperTrail stored with one
+version in its `object_changes` column, as a frozen hash of `ValueChange`
+objects in attribute-name order:
+
+```ruby
+PaperTrailDiff.recorded_changes(version)
+# => { "title" => <from "Draft" to "Published"> }
+```
+
+A step's `diff` compares the record before and after an event instead. The two
+usually agree. They differ when:
+
+- something was [saved without a version](#how-papertrail-stores-history)
+  before the record's next version. The diff includes it; this does not.
+- `has_paper_trail` ignores an attribute or leaves it out of `only:`. PaperTrail
+  stores its value but not its changes, so only the diff reports them.
+- the version is a `create` or a `destroy`. The diff reports
+  `record_presence_change`; this lists each attribute changing from `nil`, or
+  to `nil`.
+- an included association's foreign key changed. The diff reports the
+  association instead; this keeps the column.
+
+Other details:
+
+- It returns `nil` when nothing was recorded: the version table has no
+  `object_changes` column, or this version has no value in it. A version that
+  recorded no changes, such as a [checkpoint](#saving-a-checkpoint), returns
+  `{}`.
+- `ignore:` takes an array of attribute names and defaults to `updated_at`, as
+  elsewhere. It does not take `paths:`, because a version does not say which
+  association path it was reached through. Primary keys are left out.
+- Unlike PaperTrail's `version.changeset`, it never loads the record. That
+  saves a query per version, and a deleted record's values keep their types,
+  such as an enum's label rather than the integer PaperTrail stored.
+
+Steps name their source version by ID. Load the versions in one query rather
+than one per step; a model with its own version class needs that class here:
+
+<!-- executable:reference-recorded-changes -->
+```ruby
+ids = steps.map { |step| step.source_boundary.version_id }
+recorded = PaperTrail::Version.where(id: ids).to_h do |version|
+  [version.id, PaperTrailDiff.recorded_changes(version)]
+end
+
+recorded.fetch(steps.last.source_boundary.version_id)
+```
+
 ## Checking your setup
 
 List the associations a model supports, for example to build a settings
@@ -850,6 +1051,14 @@ PaperTrailDiff.association_paths(Article, max_depth: 3).map(&:to_h)
 Each descriptor has `path`, `kind`, `target_type`, `through`, `polymorphic`,
 and `cycle`. Cycles are marked and not followed.
 
+Only associations a request accepts are listed, so each path can be passed to
+`associations:`. The PaperTrail versions association, a `has_many :through`
+that goes through a HABTM, a scoped HABTM, and a HABTM below the root are left
+out. A
+`has_many :through` Active Record cannot follow, such as one through a
+polymorphic `belongs_to`, raises Active Record's own error from any model
+discovery reaches, as requesting it does.
+
 Check two versions for known problems before relying on association history:
 
 ```ruby
@@ -933,7 +1142,7 @@ All errors inherit from `PaperTrailDiff::Error`.
 | --- | --- |
 | `ConfigurationError` | An option is invalid |
 | `UnsupportedBatchPathError` | A batch requests a [costly association](#costly-associations) without `allow_expensive: true` |
-| `InvalidEndpointError` | A record is unsaved, destroyed, or has unsaved changes |
+| `InvalidEndpointError` | A record is unsaved, destroyed, or has unsaved changes, or `recorded_changes` is given something other than a version |
 | `UnloadedAssociationError` | `reload_live_endpoints: false` and an association is not loaded |
 | `VersionMismatchError` | Two endpoints belong to different records |
 | `ReversedEndpointsError` | Two versions are given newest first |
@@ -945,9 +1154,9 @@ All errors inherit from `PaperTrailDiff::Error`.
 | `FilterLimitExceededError` | `historical_filter:` needs more than `max_filter_versions:` |
 | `AssociationTrackingUnavailableError` | Association history is requested without PT-AT |
 | `UnknownAssociationError` | An association name does not exist |
-| `UnsupportedAssociationError` | An association's macro is not supported |
-| `UnversionedAssociationError` | An association's target model has no `has_paper_trail` |
-| `IncompleteAssociationHistoryError` | PT-AT did not record enough to rebuild an association, such as HABTM without a transaction |
+| `UnsupportedAssociationError` | An association's macro is not supported, a path continues past a polymorphic `belongs_to`, a `has_many :through` goes through a HABTM, or a HABTM is scoped or below the root |
+| `UnversionedAssociationError` | An association's target model has no `has_paper_trail`, including the class a polymorphic `belongs_to` names |
+| `IncompleteAssociationHistoryError` | PT-AT did not record enough to rebuild an association, such as HABTM without a transaction or in a transaction another record with that HABTM also saved a version in, or a polymorphic type names no class |
 | `UnsupportedLiveActivityError` | A HABTM activity range ends at the current record |
 | `AmbiguousNestedKeyError` | `nested_changes` finds keys like `:a` and `"a"` together |
 
@@ -956,14 +1165,18 @@ All errors inherit from `PaperTrailDiff::Error`.
 Results can only be as complete as the history PaperTrail and PT-AT recorded.
 
 - Changes made with versioning off, with `update_columns`, or to `skip:` columns
-  were never recorded. A range ending at the current record shows the first two
-  in its last step, attributed to the change before them.
+  were never recorded. The first two still appear, in the step for the record's
+  previous recorded change and attributed to its author; see
+  [how PaperTrail stores history](#how-papertrail-stores-history).
 - Deleted versions, schema changes, and serializer changes can affect how old
   versions read.
 - Several queries against the current record are not one database snapshot.
   Use a transaction with a suitable isolation level if that matters.
 - HABTM membership history exists only where PT-AT recorded it in a
-  transaction. HABTM changes cannot become their own activity steps.
+  transaction, and only for the root record without a scope: a scoped HABTM,
+  or one below the root, raises `UnsupportedAssociationError`, and a transaction that another record with
+  that HABTM also saved a version in raises `IncompleteAssociationHistoryError`. HABTM changes cannot become their own
+  activity steps.
 - PT-AT has its own
   [limitations](https://github.com/westonganger/paper_trail-association_tracking#limitations),
   especially for some `has_one` histories. The gem follows PT-AT's configured

data/docs/testing.md

@@ -46,19 +46,40 @@ passes is behavior nothing checks. `mutant.yml` lists the classes it mutates:
 - what counts as a change: `Engine`, `CollectionComparator`,
   `CollectionIdentityIndex`, and `NestedComparator`;
 - which columns are compared: `SnapshotNormalizer` and `SnapshotPool`;
-- which history a range or filter reads: `TimelineRange`, `RootVersionPlan`,
-  and `HistoricalFilter`.
+- which versions a range reads: `TimeRange`, `TimelineRange`, `VersionRange`,
+  `TimeVersionRange`, and `RootVersionPlan`;
+- which versions a batch or filter reads: `BatchedRootVersions`,
+  `DeletedAnalysisBatch`, and `HistoricalFilter`;
+- what a version recorded: `RecordedChanges`;
+- which records a scope or report selects: `ScopedRootSelection`,
+  `AssociationActivitySelection`, and `AssociationActivityEdge`;
+- which children and targets a record had at each point:
+  `PreparedHistoryLoader`, `PreparedEdgeLoader`,
+  `ActivityTargetCandidateLoader`, and `PreparedActivityIndex`;
+- where an activity range starts and ends, and which association history its
+  query reads: `ActivityRange` and `VersionAssociationCandidateScope`;
+- what activity a record shows: `ActivityTimelineBuilder` and
+  `ActivityTransactionGrouper`;
+- which association paths a request accepts: `AssociationTraversal`;
+- whether a transaction's HABTM members are one record's:
+  `HabtmTransactionOwners`;
+- which association paths a batch refuses: `BatchSafety`;
+- which records an association scope admits, and how it ranks them:
+  `ScopeCondition` and `ScopeRanking`.
 
 The specs for the classes that read the database use tables of their own
-(`spec/support/filter_database.rb`, `spec/support/normalizer_database.rb`), so
-they run in both the core and the association suite.
+(`spec/support/filter_database.rb`, `spec/support/normalizer_database.rb`,
+`spec/support/recorded_changes_database.rb`), so they run in both the core and
+the association suite. The association classes need PT-AT, so their specs are
+in `spec/association` and define any extra tables in the spec file itself.
 
-Mutant does not run every spec for every mutation. It runs the specs whose
+Mutant reads `spec/unit` and `spec/association`, never `spec/integration`.
+Even there it does not run every spec for every mutation, only those whose
 description names the mutated class (`RSpec.describe PaperTrailDiff::Engine`),
-and only falls back to wider names such as `PaperTrailDiff` when none does. A
-spec named after a behavior, as most integration specs are, never runs. So
-each mutated class has a unit spec of its own that catches every mutation in
-it, and a class added to `mutant.yml` needs one too.
+falling back to wider names such as `PaperTrailDiff` when none does. A spec
+named after a behavior, as most integration specs are, never runs. So each
+mutated class has a unit spec of its own that catches every mutation in it,
+and a class added to `mutant.yml` needs one too.
 
 Mutant counts a mutation that runs past its time limit as caught, so
 `mutant.yml` sets the limit well above the time the whole suite takes. With
@@ -87,8 +108,10 @@ BUNDLE_GEMFILE=gemfiles/mutant.gemfile mise exec -- bundle exec mutant run --sin
 When a mutation survives, either add the spec that would catch it, or, if the
 mutated code means the same thing, simplify the code to that form. Reserve a
 `mutant:disable` comment for a method whose surviving mutations all mean the
-same thing as the code, and say why next to it. Mutant forks a worker per CPU; `spec/support/mutant_hooks.rb`
-gives each worker its own copy of the SQLite fixture database.
+same thing as the code, and say why next to it.
+
+Mutant forks a worker per CPU; `spec/support/mutant_hooks.rb` gives each
+worker its own copy of the SQLite fixture database.
 
 ## Test clock