paper_trail_diff 0.14.0 → 0.15.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8b8f753f8bf15574c43fe572252b467b6066f097d0ad9990b6f1d8a74f41d938
-  data.tar.gz: 536266a31306236cc26e71b3024af058f96a0544d363f0fb971b5b49f9b1b578
+  metadata.gz: 4cc17a3d2f233d6f9bcc44605687ef4cabf1611f957ad41d0471b35236b8ab8d
+  data.tar.gz: e123b779da0c52350cb05b4ed18f8585626d8cbcf45e3c737cfe43777f7bcf14
 SHA512:
-  metadata.gz: 5070f94bc6bb46a58d6a80352aebd2f7c2392e0486c0197fe8977cb257af3b1a7644643fc1587df33bf6b0138df25ac67a3629ccf004fc627de3be883b321db1
-  data.tar.gz: 8771c47edd7d8ec39347b7d76883af7c2ff7137ac7670064672dbc5fccea0c196d5e461624c183c8d51890b8a84472625fba8cc283c72d1375a0331fd3cb1efd
+  metadata.gz: 65f32e4a83fd05ba8cfece667be5842d41f9452ae84f49f495ff8ad0051b87c76ebddfb07c1a99c09227689eaa393286ca0737042bf6b6f7c6dffea0b77039bf
+  data.tar.gz: 217119550346fc0eed08759dd20aa1a2fb53bd71dde161e43ef1243f191159fc7c1225924a6c39059dda8e2df4c4b17f0f1abed694938a2824a89e722e88e0d7

data/CHANGELOG.md

@@ -3,6 +3,349 @@
 All notable changes to this project will be documented in this file. The
 project follows [Semantic Versioning](https://semver.org/).
 
+## [0.15.0] - 2026-10-31
+
+### Breaking changes
+
+- An association whose model keeps its versions, or its PT-AT version
+  associations, in different tables than its owner's raises
+  `UnsupportedAssociationError` in every method, and `association_paths`
+  leaves it out. History read through it was wrong: PT-AT records a child's
+  membership with the child's versions but reads it from the owner's tables,
+  so a child that moved to another owner was never reported as removed, and a
+  root with its own version table raised `ActiveRecord::StatementInvalid`.
+
+- An association keyed on a column other than a primary key
+  (`primary_key: :legacy_id`), or a `has_many :through` with such a part,
+  raises `UnsupportedAssociationError` up front in every method, and
+  `association_paths` leaves it out. PT-AT records which record a child
+  pointed at by that record's ID, and history matched children by ID, so an
+  article's history showed another article's comments, and a comment's past
+  article was the wrong one. Comparisons against the current record were
+  right, so the same range disagreed with itself.
+- An association path history cannot read, such as one with a scope written
+  as SQL text, a scope that takes the owner, or a condition on a string
+  column on MySQL or SQL Server, raises `UnsupportedAssociationError` in every
+  method that reads history, unless you pass the new `allow_approximate: true`.
+  The gem reads such a path with PT-AT, which rebuilds an association from its
+  current members and never checks the scope against them: a record that
+  entered the scope since was reported as a member it was not, so a comment
+  moved into `-> { where("position > 2") }` showed as changed rather than
+  added. Single-record methods did this without warning, and batch methods
+  described it only as costly.
+- `include_deleted: true` keeps only the deleted records that met the
+  relation's conditions when they were destroyed, in both `deleted` and
+  `unreachable`. It returned every deleted record of the model that changed in
+  the window, so a report on one account's articles included other accounts'
+  deleted articles with their contents. The relation's conditions must be
+  plain conditions on the model's own columns; a join or SQL text raises
+  `ConfigurationError`.
+
+### Added
+
+- `diagnose` reports every reason `compare` would refuse the same request as
+  an error, with `compare`'s message, so `ok?` is false whenever `compare`
+  would refuse. It said `ok?` for a path `compare` reads only approximately,
+  such as a scope written as SQL text, raised on an unknown or unsupported
+  association instead of reporting it, and checked only the join model a
+  `has_many :through` goes through first, not every model it reads. It
+  takes `allow_approximate:`, and with `true` reports such a path as an
+  `:approximate_association` warning. The new error codes are
+  `:unknown_association`, `:unsupported_association`, and
+  `:approximate_association`; an unversioned target's message now matches
+  `compare`'s.
+
+- `diagnose` takes the current record as either endpoint, as `compare` does,
+  so `compare(version, record)` can be checked before it runs. Its version
+  checks then run from the version to the latest one. An endpoint that is
+  neither a version nor a saved record raises `InvalidEndpointError` naming
+  that, rather than `VersionMismatchError`.
+
+- Activity steps list the events whose changes they show as `step.sources`,
+  oldest first. A step from `group: :transaction` lists every event in the
+  transaction, each with its own author, record, and version ID, where
+  `source_boundary` names only the first. `to_h(metadata: true)` adds a
+  `sources` list.
+- `recorded_changes: true` on `activity_timeline`, on `analyze`, `analyze_many`,
+  and `analyze_scope` with `activity: true`, and on `report` gives each
+  boundary the changes its version recorded, as `recorded_changes(version)`
+  returns them. The versions are already loaded, so it adds no queries, and it
+  replaces the reference recipe that loaded every step's version again.
+  Without the option, `boundary.recorded_changes` raises `ConfigurationError`
+  rather than returning `nil`. Names in `ignore:` or `ignore: { all: }` are
+  left out, and so are names on the root path, `"$"`, for the root's own
+  versions. Names on an association path are not, because a child's version
+  does not say which path reached it.
+- `change_rows` on an activity step, on an `Analysis` built with
+  `activity: true`, and on a `report` or `analyze_scope` result flattens
+  activity into `ChangeRow` objects: one per edited field, and one per record
+  that appeared or disappeared followed by its fields. Each names its record,
+  association path, and source version, and `to_h` gives a flat hash. It
+  replaces the README recipe for turning a report into rows, which had no rows
+  for whole records, so a record added or removed with only blank fields left
+  no trace.
+
+- `associations:` takes what `includes` takes: a single name
+  (`associations: :comments`) or hashes of names to what to include below them
+  (`associations: [:author, { comments: :replies }]`), as well as the arrays of
+  names and dotted paths it took before.
+
+- `nested_changes` compares two top-level arrays, as a `json` column can hold,
+  and reports one `ArrayChange` at the empty path, `[]`. It returned `{}`,
+  although arrays nested in an object were compared.
+
+### Changed
+
+- The reference describes how PT-AT records HABTM membership and how the gem
+  credits it. A change made between two root versions is credited to the
+  step after the earlier one, so a checkpoint's capture is credited to whoever
+  saved the version before it; a checkpoint just before the change as well
+  credits it to its author. Only the first version an instance saves after
+  changing a HABTM is right; PT-AT never forgets the instance's changes, not
+  even on `reload`. A later save from it records the old membership, and the
+  gem reports members removed and added again that never left.
+
+- The performance guide describes current behavior instead of logging each
+  past change: what the batch methods cost, what still grows with a report,
+  the version indexes, what costs a query per record, historical filters,
+  large reports, and how to measure. Its measurements were taken again. The
+  earlier ones remain in `benchmark/results/` and in the changelog.
+
+- `historical_filter:` reads the history of the relation's records only, and
+  of deleted records, which a relation cannot test. It read every record of
+  the model that changed in the window, so a report on one account's articles
+  raised `BatchLimitExceededError` when another account had more matching
+  articles than `limit:`, and spent `max_filter_versions:` on their versions.
+  Matching deleted records no longer count toward `limit:` unless
+  `include_deleted: true` is given, as without a filter.
+
+- The reference says how a record created without a version, such as one
+  imported with `insert_all`, and history pruned with `has_paper_trail
+  limit:` read.
+
+- `IncompleteTimeRangeError` no longer advises the deprecated
+  `close_on: :current`. The quickstart and performance guide no longer say
+  that `has_many :through` or limited associations need the opt-ins, and the
+  reference says that `association_paths` lists paths that need
+  `allow_approximate: true`.
+
+- The quickstart sets PaperTrail's `yaml_column_permitted_classes`. In a new
+  Rails 8.1 app, versions hold timestamps as `ActiveSupport::TimeWithZone`,
+  and the quickstart's first `reify` raised `Psych::DisallowedClass` without
+  it. Its documentation spec now reads a version in that form.
+
+- The README and architecture guide recommend a second version index,
+  `(item_type, created_at, item_id)`. Batch methods with `since:` or
+  `within:` find changed records by type and time, which the one index they
+  recommended cannot narrow, so each report read every version of the model:
+  about 230 ms instead of 2 ms on PostgreSQL with 5 million versions.
+- The gemspec requires Active Record 7.1 to 8.1. CI tests 7.2 through 8.1,
+  and the suite passes on 7.1. It named only PaperTrail, so Bundler could
+  install the gem with Active Record 6.1 or 7.0, whose relations and Arel
+  nodes the gem reads differently.
+- `close_on:` is deprecated and warns through the new
+  `PaperTrailDiff.deprecator`. It has had no effect since a window began
+  closing on the current record by default; remove it. In a Rails app,
+  register the deprecator with
+  `Rails.application.deprecators[:paper_trail_diff] = PaperTrailDiff.deprecator`.
+- `allow_expensive:` is deprecated and has no effect. It refused exactly the
+  paths `allow_approximate:` refuses, so batch methods needed both or
+  neither; `allow_approximate: true` alone now accepts such a path in every
+  method, and `UnsupportedBatchPathError` is no longer raised. Batch methods
+  refuse such a path before their first query, as before, now with
+  `UnsupportedAssociationError`.
+- `reload_live_endpoints:` is deprecated and warns. It still works until it
+  is removed. Skipping the reload saves a query for the records and one per
+  association level in each call, 2 for 500 articles with their comments,
+  and reads values from objects that may be stale; it also needs every
+  requested association loaded, and reads limited ones again anyway. It was
+  on `compare`, `compare_many`, and `activity_timeline` only. Remove it.
+
+### Fixed
+
+- A HABTM member created in the same transaction as the checkpoint that
+  records it is a member at that checkpoint, as a member attached there
+  already was. It was left out, because it had no state before the
+  transaction, so the checkpoint missed it and a comparison from the
+  checkpoint to now reported it added.
+
+- `association_paths` leaves out an association whose target, or a model its
+  `has_many :through` goes through, has no `has_paper_trail`, such as the
+  `file_attachment` and `file_blob` that `has_one_attached` defines. Every
+  request that reads history refused them with `UnversionedAssociationError`,
+  so a settings screen built from the list offered paths that always failed.
+
+- `diagnose` reports `:tied_version_timestamps`, and `compare` emits
+  `ambiguous_association_boundary.paper_trail_diff`, only when the request can
+  read a path with PT-AT: one history reads only approximately, a composite
+  foreign key, or a polymorphic `belongs_to`. PT-AT places association changes
+  by timestamp alone. For every other path, since 0.14.0, versions that share a
+  timestamp are ordered by ID, so a change between them is reported, yet both
+  warned that it could not be detected. A request refused for a path PT-AT
+  would read now raises before the notification.
+
+- `timeline`, `activity_timeline`, and `analyze` read every version of a
+  model whose versions association has a scope
+  (`has_paper_trail versions: { scope: }`), as the other methods already did.
+  They read only the versions the scope kept, so a change saved in a version
+  it left out was credited to the next one kept: with a bot's edits hidden,
+  an editor's step showed the bot's later edit too, and the same record and
+  window gave different steps in `analyze` and `analyze_many`. A version the
+  scope leaves out is also accepted as an endpoint. To report only some
+  changes, pass `version_scope:`.
+
+- Activity steps leave out the versions a record saved while it was outside
+  the root's tree. A comment edited on another article, then moved to this
+  one, brought that edit into the article's timeline as a step, and a
+  transaction group's `sources` listed it. Saved together with a comment added
+  to the article, the edit took the credit for the addition. The same went for
+  a comment edited after it moved away or left a scope, and for an author
+  renamed after the article changed authors. A version now counts when its
+  record is in the tree before or after its transaction. The versions of a
+  `has_many :through`'s join rows are all still kept.
+
+- `timeline`, `activity_timeline`, `analyze`, and `analyze_many` raise
+  `InvalidEndpointError` naming what they were given when it is not a record
+  of a versioned model, such as `nil`, a model class, a relation, or a
+  version. They raised a bare `NoMethodError`, or said to provide `from` and
+  `to`. `report` names `since` when it is `nil` or a range.
+
+- A scope ordered by a datetime, such as
+  `-> { order(created_at: :desc, id: :desc).limit(3) }`, is read from history
+  under Rails' default time-zone-aware attributes. Rails wraps those
+  attributes' types, which hid the datetime, so every method raised
+  `UnsupportedAssociationError` for the documented recipe.
+
+- A `Date` bound of `within:` or `since:` is a whole day in `Time.zone`, and
+  a string a time in `Time.zone`, or in UTC without one. They were read in
+  the server's own zone, so the same window covered different hours on
+  different servers, and a same-day `Date..Date` window covered one instant
+  and found nothing.
+
+- A `has_many :through` whose join model is not versioned raises
+  `UnversionedAssociationError` naming the join model when its history is
+  read, and `diagnose` reports it as an error, so `ok?` is false. A
+  comparison raised a bare `KeyError` while `diagnose` reported only a
+  warning.
+
+- `reload_live_endpoints: false` reads an association with a `limit` or
+  `offset`, directly or in a part it goes through, or a scope that takes the
+  owner, again for each record. Active Record's `preload` applies the limit across all the records
+  it loads, so with three articles preloaded, two kept one comment each and
+  the third none, and their unchanged comments were reported as removed.
+
+- A root and its associations that share one version table other than
+  PaperTrail's `versions` can be compared. Association lookups joined PT-AT's
+  version associations to the `versions` table whatever table the versions
+  were in, so every method raised `ActiveRecord::StatementInvalid`.
+
+- A `within:` or `since:` window whose edge falls inside a transaction that
+  saved a version of the root record reports the whole transaction in the
+  later window. A window starting between a comment edit and the `touch: true`
+  article version saved with it showed the edit at the article's next change,
+  credited to that change's author, and the window ending there left it out;
+  when the article saved first, its version was in one window and the comment
+  edit in the other. Each root's edges move back to the transaction's start,
+  read in a fixed number of queries per batch.
+
+- `activity_timeline`, and `analyze` with `activity: true`, no longer report a
+  child version saved in the same transaction as the root version that ends
+  the range. A `touch: true` parent version is saved this way after every
+  child save. PT-AT rebuilds the parent's associations at that version as they
+  were before the transaction, so the child's step was always empty and its
+  change was reported nowhere. The range now ends before that transaction, as
+  `compare` and `timeline` already did. End at a later version or at the
+  current record to include the change, with `group: :transaction` to credit it
+  to the child's version.
+- A range that starts at a version now starts before that version's
+  transaction, as one that ends there already ended. A child version saved
+  earlier in the same transaction, such as the update behind a `touch: true`
+  parent version, was read after its change, so `compare`, `analyze`,
+  `timeline`, and `activity_timeline` from that version left the change out.
+  It was also missing from the range ending there, so a history split at that
+  version reported it nowhere. It is now inside the range that starts there.
+- A past `within:` window in `activity_timeline`, and in `analyze`,
+  `analyze_many`, `analyze_scope`, and `report` with `activity: true`, no
+  longer includes association changes made after it. When the root had no
+  version after the window, the window closed on current state, so a comment
+  edited in August appeared in a July window, in its last step, `to_snapshot`,
+  and `diff`; with a later root version the same window was right. It now
+  closes on current state only when nothing on a requested path was recorded
+  after it. The reasoning in 0.8.0, that nothing had been recorded since,
+  checked only the root's own versions. `timeline`, and `analyze` without
+  `activity:`, still run their last step to the root's next version or the
+  current record; the reference now says so.
+- Association scopes read from history now cover `IS NULL` and `IS NOT NULL`
+  on a column of any type, such as a soft-delete scope's
+  `where(deleted_at: nil)`, and `NOT` with SQL's rules for NULL, such as
+  `where.not(position: [1, nil])`. These were read with PT-AT before, so a
+  record restored from a soft delete showed as changed rather than added, and
+  batch methods refused them without `allow_expensive: true`. A condition on a
+  PostgreSQL `citext` column, which compares without regard to case, is no
+  longer read as exact text.
+- A `historical_filter:` hash casts its values, and the recorded ones, to each
+  attribute's type before comparing them, as `where` does. An enum given as a
+  symbol (`{ status: :published }`), and a date or time read from a `json` or
+  `jsonb` version column, which PaperTrail returns as a string, never matched,
+  so the report silently selected nothing.
+- Versions from two version tables saved in the same instant raise
+  `AmbiguousVersionOrderError`, as the reference promises, instead of being
+  ordered by their ids. Each table counts its ids separately, so their order
+  said nothing about which change came first, and an activity timeline built
+  from them could credit a change to the wrong step.
+- `compare_many`, `analyze_many`, `analyze_scope`, and `report` load a
+  `has_one` of every current record in one query again, as before 0.14.0.
+  0.14.0 meant to load only a `has_one` with a declared limit or offset per
+  record, but checked the association's scope, to which Active Record adds
+  `limit(1)` for every `has_one`, so every `has_one` cost one query per
+  record.
+- `report`, `analyze_scope`, and `analyze_many` with `activity: true` no
+  longer read a root's current row once more for activity through a
+  `has_many :through` after its last version. The batch has already loaded
+  it, so the query count no longer grows with the number of roots.
+- A snapshot's collections list their records by type, then ID, as the
+  reference says, wherever the snapshot was read from. Snapshots rebuilt from
+  history or carried through an activity timeline listed them in the order
+  they were found, and a ranked association in its query's order, so `to_h` of
+  the same state differed between methods. Diffs were already ordered.
+- With a `json` or `jsonb` `object` column, an activity step no longer
+  reports an unchanged timestamp as changed. PaperTrail returns such a
+  column's times as strings, and a step after a child's update carried them
+  into the snapshot as they were, so the next step that read the child again
+  reported `created_at` changing from a string to the same time. Activity now
+  casts them as it does every other recorded state. The specs had covered the
+  JSON serializer on a text column only, which PaperTrail casts.
+
+### Development
+
+- A new oracle suite, `rake spec_oracle`, checks rebuilt history against the
+  truth rather than against other gem methods. It makes seeded random edits to
+  an article graph one transaction at a time, records the live state after
+  each, and compares every endpoint, timeline step, activity step, window, and
+  batch result with those states. It also checks that two windows meeting
+  between transactions or inside one report each step once, scoped membership
+  against Active Record, and a tree of one model whose children have
+  encrypted, serialized, enum, custom-type, decimal, and JSON attributes.
+  `rake spec` runs it, so CI runs it on every database. `ORACLE_SEEDS` and
+  `ORACLE_STEPS` widen a run. See `docs/testing.md`.
+- `PTD_PROBE_ALLOW_APPROXIMATE=1` makes the reconstruction probe pass
+  `allow_approximate: true`, so its PT-AT-read scenario runs rather than
+  raising, and neither benchmark passes the deprecated `close_on:` any more.
+- On SQL Server the association specs keep their versions 5 ms apart. Specs
+  that compare the gem with PT-AT failed at random there: PT-AT compares
+  version times through SQL text, which the SQL Server adapter writes as
+  `datetime` values up to 1/300 s off, and after the host clock stepped back
+  the test clock packed versions microseconds apart until it caught up.
+- The reporting benchmark loads the monotonic test clock, so its results no
+  longer depend on the host clock. A clock that stepped backwards, as WSL2's
+  does, could date a root's destroy before its last update, and
+  `deleted_roots` then raised `IncompleteTimeRangeError`. The seeded history
+  and query counts are unchanged. Allocations rise by under 1% in
+  `many_roots` and `retained_activity`, where the gem reads the clock once
+  per root. The investigation scripts that seed through the same workload
+  load the clock too; `scalar_cost.rb` freezes time in front of it, so its
+  parity check still sees one instant.
+
 ## [0.14.0] - 2026-10-29
 
 ### Breaking changes

data/QUICKSTART.md

@@ -37,6 +37,19 @@ table. `--with-changes` adds `object_changes`, which PaperTrail uses for
 for each event. This gem can also compare the full earlier records saved in
 PaperTrail's `object` column.
 
+PaperTrail stores earlier records as YAML, and Rails reads back only the YAML
+classes you allow. Without this, reading the first version raises
+`Psych::DisallowedClass`. Add PaperTrail's list to `config/application.rb`,
+inside `class Application`:
+
+<!-- executable:quickstart-config -->
+```ruby
+config.active_record.yaml_column_permitted_classes = [
+  ActiveRecord::Type::Time::Value, ActiveSupport::TimeWithZone, ActiveSupport::TimeZone,
+  BigDecimal, Date, Symbol, Time
+]
+```
+
 ## 2. Version a model
 
 Create an article model and tell PaperTrail to track its changes:
@@ -233,10 +246,11 @@ The same `within:` option works on `activity_timeline`, `analyze`,
 | Recent activity through now, for the records a relation selects | `report(scope, since:, limit:)` |
 
 Use the batch methods for many records so they can share database queries.
-Some scoped associations and `has_many :through` histories still need queries
-for each record. Batch calls reject those with `UnsupportedBatchPathError`.
-The error names the association. `allow_expensive: true` permits that extra work;
-see the [performance guide](docs/performance.md) before using it in a large report.
+A few scoped associations, such as one whose scope is SQL text, can be read
+only with PT-AT, approximately and one record at a time. Every method rejects
+them with `UnsupportedAssociationError`, naming the association; passing
+`allow_approximate: true` accepts both costs. See
+[costly associations](docs/reference.md#costly-associations).
 
 When associations are included, `timeline` compares their values at the article's
 versions. `activity_timeline` also reports the comments' own version events, even
@@ -388,6 +402,9 @@ window to select that article.
   comments does not automatically include every association on each comment.
 - Historical output cannot contain data that PaperTrail or PT-AT did not record
   or can no longer reconstruct.
+- A report with `since:` or `within:` gets slow as the version table grows
+  unless you add the
+  [two version indexes](docs/architecture.md#database-indexes).
 
 For association support, performance, error handling, ignore rules, and all
 available result methods, see the

data/README.md

@@ -122,13 +122,14 @@ and `item_id`. Its `recorded_at` is the version's `created_at`.
 Use this source information for attribution: the version used to read the
 resulting state, exposed as `to_boundary`, may have been saved by someone else.
 A change saved without a version, such as with `update_columns`, appears in the
-step for that record's previous recorded change. To read only what a version
-recorded, use `PaperTrailDiff.recorded_changes(version)`; see
-[what a version recorded](docs/reference.md#what-a-version-recorded).
+step for that record's previous recorded change. To also read only what each
+version recorded, pass `recorded_changes: true` and read
+`step.source_boundary.recorded_changes`; see
+[what each event recorded](docs/reference.md#what-each-event-recorded-recorded_changes-true).
 
 `group: :transaction` combines changes from one database transaction. Its source
-information comes from the first event in that group. Omit `group:` if you need
-to attribute each version separately.
+information comes from the first event in that group, and `step.sources` lists
+every event in it, each with its own author.
 
 `snapshots: true` also keeps copies of the compared records' data, including
 unchanged fields. These read-only copies are called snapshots. Access them as
@@ -195,8 +196,11 @@ Three selection rules matter:
   `associations: [:comments]`, editing only a comment selects its article, and
   the article's report shows the edit.
 - A relation such as `Article.where(status: "open")` checks today's values.
-  It cannot establish whether deleted articles matched that condition. Deleted
-  records are therefore listed separately.
+  Deleted records have none, so they are listed separately. With
+  `include_deleted: true`, the gem tests each against the relation as it was
+  when destroyed and keeps only those that match, so a report on one account's
+  records holds only that account's deletions. That needs plain conditions on
+  the model's own columns; a join or SQL text raises `ConfigurationError`.
 - `limit: 100` is a safety ceiling, not pagination. Selecting too many records
   raises `BatchLimitExceededError`. With `include_deleted: true`, both existing
   and deleted records count toward it. Use a narrower relation or time window,
@@ -215,9 +219,9 @@ own.
 
 ## 4. Turn a report into rows
 
-For a history table or an export, flatten a report into one row per changed
-field. Each row names the record, the change, and who made it. This example
-requires the association setup in the quickstart:
+For a history table or an export, `change_rows` flattens a report into one row
+per change. Each row names the record, the change, and the version it came
+from. This example requires the association setup in the quickstart:
 
 <!-- executable:recipe-summary -->
 ```ruby
@@ -225,48 +229,35 @@ report = PaperTrailDiff.report(
   Article.all, since: 1.day.ago, limit: 100,
   associations: [:comments], include_deleted: true
 )
-analyses = report.analyses.merge(report.deleted.compact)
-
-rows = analyses.flat_map do |root, analysis|
-  analysis.activity_timeline.flat_map do |step|
-    source = step.source_boundary
-    step.diff.each_entry.filter_map do |entry|
-      next unless entry.attribute # skip entries for whole records and associations
-
-      from, to, change =
-        if entry.change?
-          [entry.value.from, entry.value.to, :changed]
-        elsif entry.state == :after
-          [nil, entry.value, :added]
-        else
-          [entry.value, nil, :removed]
-        end
-      next if from.nil? && to.nil? # a blank field of an added or removed record
-
-      record = entry.record # nil for the article itself
-
-      {
-        root: root,
-        association: entry.association_path.join("."), # "" for the article itself
-        record: record ? [record.type, record.id.to_s] : root,
-        change: change, attribute: entry.attribute, from: from, to: to,
-        actor: source.whodunnit, event: source.event, recorded_at: source.recorded_at
-      }
-    end
-  end
+
+rows = report.change_rows.flat_map do |root, changes|
+  changes.map { |row| row.to_h.merge(root: root) }
 end
 ```
 
-`change` is `:changed` for an edited field. It is `:added` or `:removed` for
-each field of a record that appeared or disappeared: the article was created
-or deleted, or a comment joined or left it. A comment moved from another
-article is also `:added`; its `event` is then `"update"` rather than
-`"create"`. `actor`, `event`, and `recorded_at` describe the step's source
-version. With `group: :transaction`, that is the transaction's first event.
-
-Rows follow the report's record order, with deleted records last, then each
-record's steps in time order. Labels, grouping, and formatting are left to your
-application.
+`report.change_rows` is a hash keyed like `report.analyses`, with deleted
+records last. Each value lists that record's rows in time order. Each row
+covers one of two things:
+
+- **A field**: `attribute` names it, and `change` is `:changed` for an edit.
+  For a record that appeared or disappeared, each of its fields gets an
+  `:added` or `:removed` row, holding its value in `to` or `from`. Blank
+  fields of such a record get no row.
+- **A whole record**: `attribute` is `nil`, and `change` is `:added` or
+  `:removed`. The record was created or deleted, a comment joined or left the
+  article, or a `belongs_to` pointed somewhere new, which gives a `:removed`
+  row for the old record and an `:added` row for the new one. A comment moved
+  from another article is `:added` too; its `event` is then `"update"`.
+
+`record` is the record the row is about, or `nil` for the article itself, and
+`association_path` is how it was reached, such as `["comments"]`. `to_h` adds
+the step's source version: `version_id`, `event`, `whodunnit`, `recorded_at`,
+and `transaction_id`. With `group: :transaction`, that is the transaction's
+first event; `step.sources` lists every event. `row.source` is the
+`ActivityBoundary` itself.
+
+`change_rows` is also on an `Analysis` built with `activity: true` and on each
+activity step. Labels, grouping, and formatting are left to your application.
 
 ## Choose a method
 
@@ -298,15 +289,27 @@ Scopes and default scopes made of plain conditions, such as
 `-> { where(archived: false) }`, are batched too, and so is a `limit` or
 `offset` ordered by integer, boolean, date, or datetime columns down to the
 primary key, such as `-> { order(:position, :id).limit(3) }`, on a direct
-association or a `has_many :through`. Some associations still need per-record
-queries, such as a `limit` ordered by a string or scopes that take
-the owner.
-Batch methods reject those by default with `UnsupportedBatchPathError`. The
-error names the association. Remove it from `associations:`, or pass
-`allow_expensive: true` if you accept that cost.
+association or a `has_many :through`. History cannot read a few scopes, such
+as a `limit` ordered by a string, a scope that takes the owner, or SQL text.
+PT-AT reads those instead, one record at a time and only approximately, so
+every method raises `UnsupportedAssociationError` naming the association.
+Remove it from `associations:`, or pass `allow_approximate: true` to accept
+PT-AT's answer and its cost. See
+[costly associations](docs/reference.md#costly-associations).
 Callbacks and callable historical filters can also issue queries. See the
 [performance guide](docs/performance.md) for measured behavior and limitations.
 
+On a large version table, add these indexes, or every `since:` or `within:`
+report reads all of the model's versions. See
+[database indexes](docs/architecture.md#database-indexes) for why.
+
+```ruby
+add_index :versions, %i[item_type item_id created_at id],
+          name: "index_versions_on_item_and_timeline"
+add_index :versions, %i[item_type created_at item_id],
+          name: "index_versions_on_type_and_time"
+```
+
 - `updated_at` is ignored. Passing `ignore:` replaces the default list.
 - Records passed as current state must be saved and have no unsaved changes.
   They are reloaded from the database.

data/docs/architecture.md

@@ -66,17 +66,19 @@ PT-AT transaction are applied together.
 
 These association shapes cannot be read from the loaded history. The gem asks
 PT-AT to rebuild them at each point instead, which costs queries per record and
-resolves association membership by timestamp only:
+resolves association membership by timestamp only. PT-AT starts from the
+association's current members and swaps in their older versions, without
+checking a scope against them, so its membership is approximate:
 
 - associations with a scope the gem cannot read as plain conditions and an
   exact ranking, including owner-dependent scopes and a `limit` without a
   total integer order;
-- targets with such a default scope;
-- targets that are not versioned;
+- targets with such a default scope, including a polymorphic `belongs_to`
+  target, whose class and scope are known only from each record;
 - composite foreign keys.
 
-Batch methods reject the scoped cases unless you pass
-`allow_expensive: true`; see [costly associations](reference.md#costly-associations).
+Every method that reads history rejects the scoped cases unless you pass
+`allow_approximate: true`; see [costly associations](reference.md#costly-associations).
 
 HABTM membership comes from the snapshots PT-AT records with each transaction,
 so it is only known at root versions saved in a transaction. Those snapshots
@@ -88,18 +90,32 @@ HABTM of the same name saved a version holds both records' members;
 
 ## Database indexes
 
-For large histories, add a composite index that matches how the gem reads
-versions:
+For large histories, add two indexes to each version table, one for each way
+the gem reads versions:
 
 ```ruby
 add_index :versions, %i[item_type item_id created_at id],
           name: "index_versions_on_item_and_timeline"
+add_index :versions, %i[item_type created_at item_id],
+          name: "index_versions_on_type_and_time"
 ```
 
-It helps most with repeated `within:` queries, which filter by `item_type`,
-`item_id`, and `created_at`, then order by time and ID. Keep PT-AT's own index
-on `version_associations` (`foreign_key_name`, `foreign_key_id`,
-`foreign_type`).
+The first serves reading one record's history: it filters by `item_type` and
+`item_id`, then by time, in time and ID order. It covers PaperTrail's own
+`(item_type, item_id)` index, which you can then drop.
+
+The second serves finding which records changed in a window, as `report`,
+`analyze_scope`, and the other batch methods do with `since:` or `within:`.
+They filter by `item_type` and `created_at` alone, which the first index cannot
+narrow, so without the second the database reads every version of the model.
+On PostgreSQL with 5 million article versions, a report on five articles
+changed in the last hour took about 230 ms without it and 2 ms with it.
+Ending the index with `item_id` lets the database answer from the index alone,
+and SQLite, which keeps no table statistics unless you run `ANALYZE`, picks it
+only then.
+
+Keep PT-AT's own index on `version_associations` (`foreign_key_name`,
+`foreign_key_id`, `foreign_type`).
 
 ## Remaining costs
 
@@ -109,3 +125,8 @@ on `version_associations` (`foreign_key_name`, `foreign_key_id`,
 - Repeated events in one very wide collection still copy that collection's
   list of records for each step. The comparison work stays small, but memory
   traffic grows with *events × collection size*.
+- With PT-AT installed, a `within:` or `since:` window looks up each root's
+  versions on either side of both edges, to move an edge that falls inside a
+  transaction (see [time windows](reference.md#time-windows)). These are
+  four indexed lookups per root, sent 200 roots per query, whatever the
+  length of the root's history: about 0.2 ms per root on SQLite.