paper_trail_diff 0.13.1 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (181) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +630 -0
  3. data/QUICKSTART.md +30 -10
  4. data/README.md +91 -13
  5. data/docs/architecture.md +62 -18
  6. data/docs/performance.md +116 -298
  7. data/docs/reference.md +554 -102
  8. data/docs/testing.md +93 -14
  9. data/lib/paper_trail_diff/activity_belongs_to_event_applier.rb +19 -9
  10. data/lib/paper_trail_diff/activity_boundary.rb +83 -26
  11. data/lib/paper_trail_diff/activity_child_candidate_loader.rb +2 -1
  12. data/lib/paper_trail_diff/activity_collection_event_applier.rb +11 -1
  13. data/lib/paper_trail_diff/activity_collection_record_updater.rb +16 -2
  14. data/lib/paper_trail_diff/activity_event.rb +41 -10
  15. data/lib/paper_trail_diff/activity_event_record_resolver.rb +20 -1
  16. data/lib/paper_trail_diff/activity_event_route_finder.rb +2 -1
  17. data/lib/paper_trail_diff/activity_event_snapshot_refresher.rb +1 -1
  18. data/lib/paper_trail_diff/activity_history.rb +34 -9
  19. data/lib/paper_trail_diff/activity_range.rb +32 -5
  20. data/lib/paper_trail_diff/activity_relationship.rb +12 -0
  21. data/lib/paper_trail_diff/activity_root_snapshot_refresher.rb +5 -2
  22. data/lib/paper_trail_diff/activity_snapshot_sequence.rb +76 -37
  23. data/lib/paper_trail_diff/activity_target_candidate_loader.rb +131 -0
  24. data/lib/paper_trail_diff/activity_timeline_builder.rb +281 -120
  25. data/lib/paper_trail_diff/activity_transaction_grouper.rb +2 -1
  26. data/lib/paper_trail_diff/activity_tree_filter.rb +114 -0
  27. data/lib/paper_trail_diff/activity_version_collector.rb +37 -24
  28. data/lib/paper_trail_diff/activity_view.rb +69 -0
  29. data/lib/paper_trail_diff/analysis.rb +13 -1
  30. data/lib/paper_trail_diff/analysis_batch.rb +24 -30
  31. data/lib/paper_trail_diff/approximate_paths.rb +48 -0
  32. data/lib/paper_trail_diff/association_activity_edge.rb +236 -0
  33. data/lib/paper_trail_diff/association_activity_selection.rb +197 -0
  34. data/lib/paper_trail_diff/association_discovery.rb +14 -11
  35. data/lib/paper_trail_diff/association_key.rb +56 -0
  36. data/lib/paper_trail_diff/association_traversal.rb +180 -47
  37. data/lib/paper_trail_diff/batch_boundary_resolver.rb +3 -4
  38. data/lib/paper_trail_diff/batch_preload.rb +53 -0
  39. data/lib/paper_trail_diff/batched_root_analyzer.rb +12 -14
  40. data/lib/paper_trail_diff/batched_root_versions.rb +53 -47
  41. data/lib/paper_trail_diff/change_row.rb +133 -0
  42. data/lib/paper_trail_diff/collection_comparator.rb +5 -32
  43. data/lib/paper_trail_diff/comparison_batch.rb +37 -10
  44. data/lib/paper_trail_diff/configuration.rb +35 -12
  45. data/lib/paper_trail_diff/deleted_analysis_batch.rb +45 -27
  46. data/lib/paper_trail_diff/deleted_root_filter.rb +79 -0
  47. data/lib/paper_trail_diff/deprecation.rb +61 -0
  48. data/lib/paper_trail_diff/diagnostics.rb +105 -102
  49. data/lib/paper_trail_diff/direct_record_state_loader.rb +10 -1
  50. data/lib/paper_trail_diff/endpoint.rb +26 -3
  51. data/lib/paper_trail_diff/errors.rb +5 -2
  52. data/lib/paper_trail_diff/fallback_paths.rb +38 -0
  53. data/lib/paper_trail_diff/habtm_transaction_owners.rb +68 -0
  54. data/lib/paper_trail_diff/historical_association_reifier.rb +34 -6
  55. data/lib/paper_trail_diff/historical_filter.rb +19 -10
  56. data/lib/paper_trail_diff/historical_snapshot_store.rb +63 -76
  57. data/lib/paper_trail_diff/live_endpoint_batch_loader.rb +12 -13
  58. data/lib/paper_trail_diff/live_graph_collector.rb +1 -1
  59. data/lib/paper_trail_diff/nested_comparator.rb +19 -18
  60. data/lib/paper_trail_diff/paper_trail_adapter.rb +136 -125
  61. data/lib/paper_trail_diff/preloaded_endpoint_batch_loader.rb +14 -2
  62. data/lib/paper_trail_diff/prepared_activity_index.rb +110 -36
  63. data/lib/paper_trail_diff/prepared_edge_loader.rb +31 -46
  64. data/lib/paper_trail_diff/prepared_history.rb +70 -53
  65. data/lib/paper_trail_diff/prepared_history_loader.rb +34 -43
  66. data/lib/paper_trail_diff/prepared_record_index.rb +64 -7
  67. data/lib/paper_trail_diff/prepared_snapshot_builder.rb +1 -1
  68. data/lib/paper_trail_diff/prepared_state_association_reader.rb +119 -29
  69. data/lib/paper_trail_diff/recorded_changes.rb +83 -0
  70. data/lib/paper_trail_diff/recorded_class.rb +53 -0
  71. data/lib/paper_trail_diff/root_version_selection.rb +4 -2
  72. data/lib/paper_trail_diff/scope_condition.rb +386 -0
  73. data/lib/paper_trail_diff/scope_ranking.rb +206 -0
  74. data/lib/paper_trail_diff/scoped_analysis.rb +11 -0
  75. data/lib/paper_trail_diff/scoped_analysis_builder.rb +20 -11
  76. data/lib/paper_trail_diff/scoped_root_selection.rb +105 -49
  77. data/lib/paper_trail_diff/snapshot.rb +16 -1
  78. data/lib/paper_trail_diff/snapshot_normalizer.rb +10 -3
  79. data/lib/paper_trail_diff/support.rb +38 -6
  80. data/lib/paper_trail_diff/time_range.rb +37 -18
  81. data/lib/paper_trail_diff/time_version_range.rb +5 -5
  82. data/lib/paper_trail_diff/timeline_builder.rb +7 -7
  83. data/lib/paper_trail_diff/timeline_range.rb +9 -1
  84. data/lib/paper_trail_diff/timeline_snapshot_provider.rb +24 -17
  85. data/lib/paper_trail_diff/transaction_cut.rb +181 -0
  86. data/lib/paper_trail_diff/traversal_preparer.rb +134 -29
  87. data/lib/paper_trail_diff/version.rb +1 -1
  88. data/lib/paper_trail_diff/version_association_candidate_scope.rb +13 -3
  89. data/lib/paper_trail_diff/version_boundary.rb +37 -8
  90. data/lib/paper_trail_diff/version_query.rb +34 -10
  91. data/lib/paper_trail_diff/version_range.rb +8 -15
  92. data/lib/paper_trail_diff/version_scope_filter.rb +20 -13
  93. data/lib/paper_trail_diff/version_sequence_diagnostics.rb +31 -16
  94. data/lib/paper_trail_diff/version_table.rb +60 -0
  95. data/lib/paper_trail_diff.rb +126 -55
  96. data/sig/generated/paper_trail_diff/activity_belongs_to_event_applier.rbs +10 -2
  97. data/sig/generated/paper_trail_diff/activity_boundary.rbs +54 -19
  98. data/sig/generated/paper_trail_diff/activity_collection_event_applier.rbs +5 -0
  99. data/sig/generated/paper_trail_diff/activity_collection_record_updater.rbs +5 -0
  100. data/sig/generated/paper_trail_diff/activity_event.rbs +25 -4
  101. data/sig/generated/paper_trail_diff/activity_event_record_resolver.rbs +7 -0
  102. data/sig/generated/paper_trail_diff/activity_history.rbs +21 -6
  103. data/sig/generated/paper_trail_diff/activity_range.rbs +14 -3
  104. data/sig/generated/paper_trail_diff/activity_relationship.rbs +3 -0
  105. data/sig/generated/paper_trail_diff/activity_root_snapshot_refresher.rbs +2 -0
  106. data/sig/generated/paper_trail_diff/activity_snapshot_sequence.rbs +48 -9
  107. data/sig/generated/paper_trail_diff/activity_target_candidate_loader.rbs +79 -0
  108. data/sig/generated/paper_trail_diff/activity_timeline_builder.rbs +138 -28
  109. data/sig/generated/paper_trail_diff/activity_tree_filter.rbs +73 -0
  110. data/sig/generated/paper_trail_diff/activity_version_collector.rbs +12 -4
  111. data/sig/generated/paper_trail_diff/activity_view.rbs +42 -0
  112. data/sig/generated/paper_trail_diff/analysis.rbs +8 -1
  113. data/sig/generated/paper_trail_diff/analysis_batch.rbs +8 -15
  114. data/sig/generated/paper_trail_diff/approximate_paths.rbs +23 -0
  115. data/sig/generated/paper_trail_diff/association_activity_edge.rbs +148 -0
  116. data/sig/generated/paper_trail_diff/association_activity_selection.rbs +112 -0
  117. data/sig/generated/paper_trail_diff/association_discovery.rbs +8 -5
  118. data/sig/generated/paper_trail_diff/association_key.rbs +29 -0
  119. data/sig/generated/paper_trail_diff/association_traversal.rbs +88 -9
  120. data/sig/generated/paper_trail_diff/batch_preload.rbs +29 -0
  121. data/sig/generated/paper_trail_diff/batched_root_analyzer.rbs +9 -10
  122. data/sig/generated/paper_trail_diff/batched_root_versions.rbs +14 -10
  123. data/sig/generated/paper_trail_diff/change_row.rbs +86 -0
  124. data/sig/generated/paper_trail_diff/collection_comparator.rbs +2 -16
  125. data/sig/generated/paper_trail_diff/comparison_batch.rbs +22 -4
  126. data/sig/generated/paper_trail_diff/configuration.rbs +19 -3
  127. data/sig/generated/paper_trail_diff/deleted_analysis_batch.rbs +20 -8
  128. data/sig/generated/paper_trail_diff/deleted_root_filter.rbs +48 -0
  129. data/sig/generated/paper_trail_diff/deprecation.rbs +35 -0
  130. data/sig/generated/paper_trail_diff/diagnostics.rbs +44 -32
  131. data/sig/generated/paper_trail_diff/direct_record_state_loader.rbs +6 -0
  132. data/sig/generated/paper_trail_diff/endpoint.rbs +11 -2
  133. data/sig/generated/paper_trail_diff/errors.rbs +5 -2
  134. data/sig/generated/paper_trail_diff/fallback_paths.rbs +22 -0
  135. data/sig/generated/paper_trail_diff/habtm_transaction_owners.rbs +40 -0
  136. data/sig/generated/paper_trail_diff/historical_association_reifier.rbs +13 -0
  137. data/sig/generated/paper_trail_diff/historical_filter.rbs +12 -2
  138. data/sig/generated/paper_trail_diff/historical_snapshot_store.rbs +36 -30
  139. data/sig/generated/paper_trail_diff/live_endpoint_batch_loader.rbs +5 -3
  140. data/sig/generated/paper_trail_diff/nested_comparator.rbs +10 -7
  141. data/sig/generated/paper_trail_diff/paper_trail_adapter.rbs +55 -48
  142. data/sig/generated/paper_trail_diff/preloaded_endpoint_batch_loader.rbs +9 -1
  143. data/sig/generated/paper_trail_diff/prepared_activity_index.rbs +52 -10
  144. data/sig/generated/paper_trail_diff/prepared_edge_loader.rbs +4 -12
  145. data/sig/generated/paper_trail_diff/prepared_history.rbs +40 -24
  146. data/sig/generated/paper_trail_diff/prepared_history_loader.rbs +15 -13
  147. data/sig/generated/paper_trail_diff/prepared_record_index.rbs +34 -2
  148. data/sig/generated/paper_trail_diff/prepared_state_association_reader.rbs +56 -11
  149. data/sig/generated/paper_trail_diff/recorded_changes.rbs +47 -0
  150. data/sig/generated/paper_trail_diff/recorded_class.rbs +22 -0
  151. data/sig/generated/paper_trail_diff/root_version_selection.rbs +2 -0
  152. data/sig/generated/paper_trail_diff/scope_condition.rbs +219 -0
  153. data/sig/generated/paper_trail_diff/scope_ranking.rbs +125 -0
  154. data/sig/generated/paper_trail_diff/scoped_analysis.rbs +6 -0
  155. data/sig/generated/paper_trail_diff/scoped_analysis_builder.rbs +8 -3
  156. data/sig/generated/paper_trail_diff/scoped_root_selection.rbs +49 -11
  157. data/sig/generated/paper_trail_diff/snapshot.rbs +8 -0
  158. data/sig/generated/paper_trail_diff/snapshot_normalizer.rbs +6 -0
  159. data/sig/generated/paper_trail_diff/support.rbs +20 -2
  160. data/sig/generated/paper_trail_diff/time_range.rbs +20 -2
  161. data/sig/generated/paper_trail_diff/time_version_range.rbs +4 -4
  162. data/sig/generated/paper_trail_diff/timeline_builder.rbs +2 -2
  163. data/sig/generated/paper_trail_diff/timeline_range.rbs +2 -0
  164. data/sig/generated/paper_trail_diff/timeline_snapshot_provider.rbs +17 -7
  165. data/sig/generated/paper_trail_diff/transaction_cut.rbs +94 -0
  166. data/sig/generated/paper_trail_diff/traversal_preparer.rbs +66 -8
  167. data/sig/generated/paper_trail_diff/version_association_candidate_scope.rbs +7 -0
  168. data/sig/generated/paper_trail_diff/version_boundary.rbs +20 -3
  169. data/sig/generated/paper_trail_diff/version_query.rbs +18 -2
  170. data/sig/generated/paper_trail_diff/version_range.rbs +4 -2
  171. data/sig/generated/paper_trail_diff/version_scope_filter.rbs +7 -4
  172. data/sig/generated/paper_trail_diff/version_sequence_diagnostics.rbs +16 -9
  173. data/sig/generated/paper_trail_diff/version_table.rbs +30 -0
  174. data/sig/generated/paper_trail_diff.rbs +47 -25
  175. metadata +62 -10
  176. data/lib/paper_trail_diff/activity_grouping.rb +0 -38
  177. data/lib/paper_trail_diff/batch_safety.rb +0 -55
  178. data/lib/paper_trail_diff/time_activity_timeline_builder.rb +0 -224
  179. data/sig/generated/paper_trail_diff/activity_grouping.rbs +0 -22
  180. data/sig/generated/paper_trail_diff/batch_safety.rbs +0 -24
  181. data/sig/generated/paper_trail_diff/time_activity_timeline_builder.rbs +0 -92
data/docs/reference.md CHANGED
@@ -19,6 +19,7 @@ including ones most applications never need.
19
19
  - [Ignoring fields](#ignoring-fields)
20
20
  - [Reading results](#reading-results)
21
21
  - [JSON columns](#json-columns)
22
+ - [What a version recorded](#what-a-version-recorded): `recorded_changes`
22
23
  - [Checking your setup](#checking-your-setup)
23
24
  - [Models, schemas, and edge cases](#models-schemas-and-edge-cases)
24
25
  - [Instrumentation](#instrumentation)
@@ -60,7 +61,7 @@ Take an article created as `"Draft"`, then updated to `"Published"`, then to
60
61
  only in the table
61
62
  ```
62
63
 
63
- Three things follow from this:
64
+ Six things follow from this:
64
65
 
65
66
  - **A `create` version stands for "no record yet".** Its `reify` returns `nil`.
66
67
  Comparing it with a later state reports the record as added
@@ -70,6 +71,24 @@ Three things follow from this:
70
71
  change from the current record.
71
72
  - **A change shows up between two points, never at one.** That is why the
72
73
  timeline methods return steps, each comparing one state with the next.
74
+ - **A change saved without a version joins the change before it.** Writing
75
+ with `update_columns` or with versioning off creates no version, but it
76
+ changes the values the record's next version stores, or its current row. The
77
+ gem reads the result of a change from there, so the write appears in the step
78
+ for that record's previous recorded change and is attributed to its author.
79
+ Ending a range at a version leaves out only the writes after it.
80
+ [`recorded_changes`](#what-a-version-recorded) reads what a version itself
81
+ recorded, and activity steps can [include it](#what-each-event-recorded-recorded_changes-true).
82
+ - **A record created without a version looks as if it always existed.** A
83
+ row written with versioning off, or imported with `insert_all`, has no
84
+ `create` version, so history never reports it as added: its earliest
85
+ version's stored values are its state from before history began, and an
86
+ association includes it there. Rows from before PaperTrail was installed
87
+ are the same, which is usually what you want.
88
+ - **Pruned history is gone.** With `has_paper_trail limit:`, PaperTrail
89
+ deletes a record's oldest versions, and the earliest one left becomes its
90
+ start: states before it cannot be read, and an older version passed in is
91
+ no longer there to pass.
73
92
 
74
93
  ## Choosing a method
75
94
 
@@ -119,6 +138,12 @@ PaperTrailDiff.timeline(article, within: 1.week.ago..Time.current)
119
138
  - `to:` defaults to the record itself, meaning its current state. Pass a
120
139
  version instead for a result that never changes when the record is edited
121
140
  again.
141
+ - A version stands for the state before its transaction, since PT-AT rebuilds
142
+ associations as they were before it. Child changes saved in the same
143
+ transaction, such as the update behind a `touch: true` parent version, fall
144
+ after a range that ends there and inside a range that starts there. So a
145
+ history split at that version reports each such change exactly once, in the
146
+ later part.
122
147
  - Versions must belong to the record and be given in time order.
123
148
  - A record with no versions has an empty history, not an error.
124
149
 
@@ -136,20 +161,46 @@ window = Time.zone.parse("2026-08-01")...Time.zone.parse("2026-09-01")
136
161
  steps = PaperTrailDiff.timeline(article, within: window)
137
162
  ```
138
163
 
139
- To show what the window's last change produced, the gem reads the next version
140
- after the window. If there is none, nothing has been recorded since, so it reads
141
- the current record instead. The step's `to_boundary.kind` is then `:current`.
142
- This works the same for a window that ends now and one that ended long ago.
164
+ A `Date` bound is a whole day in `Time.zone`, so `Date.today..Date.today` is
165
+ all of today and `august...september` stops before September 1. A string is a
166
+ time in `Time.zone` unless it names an offset. Without `Time.zone`, both are
167
+ read in UTC, never in the server's own zone.
168
+
169
+ A window edge that falls inside a transaction moves back to where that
170
+ transaction began, so the whole transaction is in the later window, as it is
171
+ when a range [starts or ends at a version](#choosing-a-range). This applies to
172
+ transactions that saved a version of the root record, such as a comment edit
173
+ and the `touch: true` article version saved with it. So back-to-back windows,
174
+ or a `report` with `since:` after an earlier report, show each change exactly
175
+ once, credited to the transaction that made it. Each root moves its own edges,
176
+ so in a batch two roots can start a few moments apart.
177
+
178
+ To show what the window's last change produced, the gem reads the state just
179
+ before the next change after the window:
180
+
181
+ - With `activity_timeline`, or `analyze` with `activity: true`, that is the next
182
+ version of the record or of any record on a requested association path. If
183
+ none was recorded after the window, it reads the current record instead,
184
+ and the last step's `to_boundary.kind` is `:current`. So a window that ended
185
+ long ago ends at its own end, even when only a comment changed since.
186
+ - `timeline`, and `analyze` without `activity:`, compare the record's own
187
+ versions. The last step runs to the record's next version, or to the
188
+ current record when it has none, and an association change made in between
189
+ is part of that step, credited to its author. Use `activity: true` to keep
190
+ a past window's associations as they were at its end.
143
191
 
144
192
  Reading the current record has one consequence: it includes changes PaperTrail
145
- never recorded, such as an `update_columns` call. Those appear in the last step
146
- of the window.
193
+ never recorded, such as an `update_columns` call. Like any
194
+ [change saved without a version](#how-papertrail-stores-history), each appears
195
+ in the step for that record's last recorded change.
147
196
 
148
197
  If the window's last change destroyed the record, the window closes on that
149
198
  deletion instead. See [a destroyed record](#a-destroyed-record).
150
199
 
151
- `close_on: :current` asks for the behavior above explicitly. It is the default
152
- now and is still accepted.
200
+ `close_on: :current` is deprecated and has no effect; passing it warns through
201
+ `PaperTrailDiff.deprecator`. Remove it. In a Rails app, register the deprecator
202
+ with `Rails.application.deprecators[:paper_trail_diff] = PaperTrailDiff.deprecator`
203
+ to handle its warnings as the app's own.
153
204
 
154
205
  `timeline` returns no steps for a window with no versions of the root record.
155
206
  `activity_timeline` still reports chosen associations that changed in such a
@@ -216,10 +267,16 @@ diff.to_h
216
267
  `ReversedEndpointsError`, because the result would look valid with every
217
268
  change backwards. A current record can be on either side.
218
269
  - Both sides must belong to the same record.
270
+ - Associations at a version are read from the history recorded since then, plus
271
+ the current rows for records that have not changed since. A comment that later
272
+ moved to another article is still found at the versions where it belonged.
273
+ This also holds when both sides are versions, or the record has since been
274
+ deleted.
219
275
  - A current record must be saved, not destroyed, and have no unsaved changes.
220
276
  The gem reloads it, so in-memory edits and cached associations are ignored.
221
- - Options: `associations:`, `ignore:`, and `reload_live_endpoints:` (see
222
- [reusing loaded records](#reusing-loaded-records)).
277
+ - Options: `associations:`, `ignore:`, and `allow_approximate:` (see
278
+ [costly associations](#costly-associations)). `reload_live_endpoints:` is
279
+ deprecated (see [reusing loaded records](#reusing-loaded-records)).
223
280
 
224
281
  ## compare_many
225
282
 
@@ -243,31 +300,26 @@ diffs.fetch(["Order", orders.first.id.to_s]) # => PaperTrailDiff::Diff
243
300
  record, so `{ from: :first, to: :first }` raises.
244
301
  - A record with no versions gets an empty `Diff`.
245
302
  - Each record can appear in only one pair, because results are keyed by record.
246
- - Options: `associations:`, `ignore:`, `reload_live_endpoints:`, and
247
- `allow_expensive:` (see [costly associations](#costly-associations)).
303
+ - Options: `associations:`, `ignore:`, and `allow_approximate:` (see
304
+ [costly associations](#costly-associations)).
248
305
 
249
306
  If you look up versions yourself, do it in bulk. Looking up one record at a
250
307
  time brings back the per-record queries this method avoids.
251
308
 
252
309
  ### Reusing loaded records
253
310
 
254
- `compare`, `compare_many`, and `activity_timeline` reload current records by
255
- default. If you already loaded them with every requested association, you can
256
- skip the reload:
257
-
258
- ```ruby
259
- orders = Order.where(id: order_ids).preload(line_items: :product).to_a
260
-
261
- diffs = PaperTrailDiff.compare_many(
262
- orders.map { |order| { from: :first, to: order } },
263
- associations: ["line_items.product"],
264
- reload_live_endpoints: false
265
- )
266
- ```
267
-
268
- The values then come from your objects, so keeping them fresh is up to you. A
269
- requested association that is not loaded raises `UnloadedAssociationError`
270
- instead of quietly running a query per record.
311
+ Every method reloads the current records it compares with, together with the
312
+ requested associations, in a query for the records and one per association
313
+ level. `reload_live_endpoints: false` on `compare`, `compare_many`, and
314
+ `activity_timeline` skips that reload and reads your loaded objects instead.
315
+ It is deprecated and will be removed: it saves those few queries per call (2
316
+ for 500 articles with their comments), and the values then come from objects
317
+ that may be stale. Until it is removed it still works,
318
+ with a warning. A requested association that is not loaded raises
319
+ `UnloadedAssociationError`, and one with a `limit` or `offset`, directly or in
320
+ a part it goes through, or a scope that takes the owner, is read again for
321
+ each record, because Active Record's `preload` applies a limit to all the
322
+ records it loads together.
271
323
 
272
324
  ## timeline
273
325
 
@@ -299,7 +351,7 @@ steps.reject(&:empty?) # hide steps with no reported changes
299
351
  `item_type`, `item_id`, and `recorded_at`. `to_h(metadata: true)` adds author,
300
352
  event, transaction ID, and record to each boundary.
301
353
  - Options: `from:`, `to:`, `within:`, `associations:`, `ignore:`,
302
- `version_scope:`, `close_on:`.
354
+ `version_scope:`, `allow_approximate:`.
303
355
 
304
356
  ## activity_timeline
305
357
 
@@ -317,21 +369,29 @@ steps.reject(&:empty?).each do |step|
317
369
  end
318
370
  ```
319
371
 
320
- - Each step's `diff` is what the event at its `source_boundary` changed. Use
321
- `source_boundary` for attribution. `to_boundary` is only the next point in
322
- time and may belong to someone else.
372
+ - Each step's `diff` is what the event at its `source_boundary` changed, plus
373
+ anything [saved without a version](#how-papertrail-stores-history) to the
374
+ same record before its next version. Use `source_boundary` for attribution.
375
+ `to_boundary` is only the next point in time and may belong to someone else.
323
376
  - Boundaries have a `kind`: `:version`, `:current` (the current record), or
324
377
  `:destroyed` (after a deletion). They also have `event`, `whodunnit`,
325
378
  `recorded_at`, `item_type`, `item_id`, `transaction_id`, and `record`, and
326
379
  the predicates `version?`, `current?`, and `destroyed?`.
380
+ - `step.sources` lists the events whose changes the step shows, oldest first.
381
+ That is just `source_boundary`, unless the step is a
382
+ [transaction group](#one-step-per-transaction-group-transaction).
383
+ - A record's versions are steps only while it is in the tree. A comment edited
384
+ on another article and moved to this one later shows up when it moves, not
385
+ when it was edited there.
327
386
  - `step.to_h` returns `{ from:, to:, diff: }`. `to_h(metadata: true)` adds
328
- author, event, transaction ID, and record. `to_h(snapshots: true)` adds the
329
- retained snapshots.
387
+ author, event, transaction ID, and record, and a `sources` list.
388
+ `to_h(snapshots: true)` adds the retained snapshots.
330
389
  - A child's change becomes visible at the next recorded event, or at the
331
390
  current record. You do not need to save the article after each comment change.
332
391
  - Options: `from:`, `to:`, `within:`, `associations:`, `ignore:`,
333
- `version_scope:`, `close_on:`, `reload_live_endpoints:`, `snapshots:`,
334
- `group:`.
392
+ `version_scope:`, `snapshots:`, `group:`, `recorded_changes:`,
393
+ `allow_approximate:`. `reload_live_endpoints:` is deprecated (see
394
+ [reusing loaded records](#reusing-loaded-records)).
335
395
 
336
396
  ### One person's changes
337
397
 
@@ -347,7 +407,7 @@ attributed = steps.reject(&:empty?)
347
407
  by_priya = attributed.select { |step| step.source_boundary.whodunnit == "Priya Shah" }
348
408
 
349
409
  by_priya.first.source_boundary.whodunnit # "Priya Shah"
350
- by_priya.first.diff # exactly what that one event changed
410
+ by_priya.first.diff # what that one event changed
351
411
 
352
412
  authors = attributed.group_by { |step| step.source_boundary.whodunnit }
353
413
  .transform_values(&:length)
@@ -374,7 +434,9 @@ PaperTrailDiff.activity_timeline(article, associations: [:comments], group: :tra
374
434
  - A combined step compares the state before the transaction with the state after
375
435
  it. A field set and then restored inside the transaction is not reported.
376
436
  - Its `source_boundary` is the first event in the transaction. That event's
377
- author is not necessarily the author of every grouped change.
437
+ author is not necessarily the author of every grouped change. `step.sources`
438
+ lists every event in the transaction, each with its own author, except
439
+ those of records that were outside the tree before and after it.
378
440
  - Versions without a `transaction_id` are never grouped.
379
441
  - Also accepted by `analyze`, `analyze_many`, and `analyze_scope` with
380
442
  `activity: true`, and by `report`.
@@ -382,7 +444,9 @@ PaperTrailDiff.activity_timeline(article, associations: [:comments], group: :tra
382
444
  Without grouping, a step inside a transaction can look empty. A child's new
383
445
  value is only visible at the next recorded event, and inside a transaction that
384
446
  event is often another part of the same save. Nothing is lost; the change
385
- appears one step later.
447
+ appears one step later. When the range ends at the version that closes that
448
+ save, there is no later step, so the change is
449
+ [outside the range](#choosing-a-range).
386
450
 
387
451
  ### Keeping snapshots: `snapshots: true`
388
452
 
@@ -403,6 +467,49 @@ step.from_snapshot.associations["comments"].records.first.attributes["author"]
403
467
  - Also accepted by `analyze`, `analyze_many`, and `analyze_scope` with
404
468
  `activity: true`, and by `report`.
405
469
 
470
+ ### What each event recorded: `recorded_changes: true`
471
+
472
+ A step's `diff` compares the state before and after an event. To also see what
473
+ PaperTrail [stored with each version](#what-a-version-recorded), such as for an
474
+ audit log, pass `recorded_changes: true`:
475
+
476
+ <!-- executable:reference-step-sources -->
477
+ ```ruby
478
+ steps = PaperTrailDiff.activity_timeline(
479
+ article, associations: [:comments], group: :transaction, recorded_changes: true
480
+ )
481
+
482
+ steps.reject(&:empty?).flat_map(&:sources).map do |source|
483
+ {
484
+ version_id: source.version_id,
485
+ record: source.record.to_h, # { type: "Comment", id: 7 }
486
+ event: source.event,
487
+ actor: source.whodunnit,
488
+ recorded_at: source.recorded_at,
489
+ transaction_id: source.transaction_id,
490
+ changes: source.recorded_changes # { "body" => <from "Old" to "New"> }
491
+ }
492
+ end
493
+ ```
494
+
495
+ - Each boundary's `recorded_changes` is what
496
+ `PaperTrailDiff.recorded_changes(version)` returns for its version: `nil`
497
+ when nothing was recorded, and always `nil` for the current record. The
498
+ versions are already loaded, so this adds no queries, but decoding each
499
+ version's changes takes time: about a third more for a `report` of 200
500
+ articles with comments, with history stored as YAML. Pass it only when you
501
+ use them.
502
+ - Without the option, `recorded_changes` raises `ConfigurationError` instead of
503
+ returning `nil`, so a missing option cannot look like a version that recorded
504
+ nothing.
505
+ - `to_h(metadata: true)` includes them on each boundary and source.
506
+ - Names in `ignore:`, or in `ignore: { all: }`, are left out. So are names
507
+ ignored on the root path, `"$"`, for the root's own versions. Names ignored
508
+ on an association path are not left out, because a child's version does not
509
+ say which path reached it.
510
+ - Also accepted by `analyze`, `analyze_many`, and `analyze_scope` with
511
+ `activity: true`, and by `report`.
512
+
406
513
  ### A destroyed record
407
514
 
408
515
  When the range ends at the record's `destroy` version, the activity timeline
@@ -429,6 +536,34 @@ the range at a version instead (see the
429
536
  [checkpoint recipe](#saving-a-checkpoint)), or use a join model with
430
537
  `has_many :through`.
431
538
 
539
+ PT-AT records HABTM members under the transaction a version was saved in, and
540
+ does not say which record they belong to. If another record with a HABTM of the
541
+ same name, such as a second article, also saved a version in that transaction
542
+ and members were recorded there, reading the root's membership at that version
543
+ raises `IncompleteAssociationHistoryError` instead of mixing both records'
544
+ members. Save each HABTM owner's versions in a transaction of its own.
545
+
546
+ PT-AT records membership only when the owner saves a version, so the gem knows
547
+ it only at root versions. A change made between two root versions is reported
548
+ in the first step after the earlier one, and credited to that step's source:
549
+ usually the earlier root version, which someone else may have saved. A member
550
+ created between them is reported in the step that starts with its creation
551
+ instead. A change attached before a checkpoint is therefore credited to
552
+ whoever saved the version before the checkpoint, not to the checkpoint's
553
+ author. To credit a change to the person who made it, save a checkpoint just
554
+ before the change as well as after it: the step between the two checkpoints is
555
+ theirs.
556
+
557
+ PT-AT works out the membership a version records from the instance that saves
558
+ it: the instance's current members, with the changes that instance made undone,
559
+ so that the version holds the membership from before its own event. It never
560
+ forgets those changes, not even on `reload`. Only the first version an instance
561
+ saves after changing a HABTM is right; each later save from it records the
562
+ membership from before that change again. The gem then reports the change late,
563
+ or reports a member that never left as removed and added again. Load the record
564
+ again, with `find`, before saving it a second time, and save checkpoints from a
565
+ freshly loaded record, as the [checkpoint recipe](#saving-a-checkpoint) does.
566
+
432
567
  ## analyze
433
568
 
434
569
  `analyze(record)` returns an `Analysis` with the net `diff` and the `timeline`
@@ -446,10 +581,18 @@ analysis.to_h(snapshots: true, metadata: true)
446
581
  ```
447
582
 
448
583
  - It accepts the same ranges as `timeline`.
449
- - `activity: true` also builds the activity timeline. `snapshots:` and `group:`
450
- apply to the activity timeline and require `activity: true`.
584
+ - When nothing in the range is reported, the analysis is empty: an empty
585
+ `diff` and timelines, and `nil` for `from_snapshot` and `to_snapshot`. Here
586
+ `nil` means no state was read, not that the record did not exist; reading
587
+ one would rebuild the record's whole graph, a cost per record in a batch.
588
+ Use `compare` with a version and the record to see a state that did not
589
+ change.
590
+ - `activity: true` also builds the activity timeline. `snapshots:`, `group:`,
591
+ and `recorded_changes:` apply to the activity timeline and require
592
+ `activity: true`.
451
593
  - Options: `from:`, `to:`, `within:`, `associations:`, `ignore:`, `activity:`,
452
- `version_scope:`, `close_on:`, `snapshots:`, `group:`.
594
+ `version_scope:`, `snapshots:`, `group:`, `recorded_changes:`,
595
+ `allow_approximate:`.
453
596
 
454
597
  ## Reports for many records
455
598
 
@@ -473,10 +616,11 @@ results.fetch(["Order", order.id.to_s]).diff
473
616
 
474
617
  - Without `within:`, each record's whole history through its current state is
475
618
  analyzed.
476
- - A record with nothing to report gets an empty `Analysis`.
619
+ - A record with nothing to report gets an empty `Analysis`, whose snapshots
620
+ are `nil` (see [analyze](#analyze)).
477
621
  - Records must be saved and unique.
478
622
  - Options: `within:`, `associations:`, `ignore:`, `activity:`, `version_scope:`,
479
- `close_on:`, `snapshots:`, `group:`, `allow_expensive:`.
623
+ `snapshots:`, `group:`, `recorded_changes:`, `allow_approximate:`.
480
624
 
481
625
  ### analyze_scope
482
626
 
@@ -514,14 +658,23 @@ report = PaperTrailDiff.report(Article.all, since: 1.day.ago, limit: 100)
514
658
  ```
515
659
 
516
660
  It accepts `associations:`, `ignore:`, `snapshots:`, `group:`,
517
- `include_deleted:`, `historical_filter:`, `max_filter_versions:`, and
518
- `allow_expensive:`.
661
+ `recorded_changes:`, `include_deleted:`, `historical_filter:`, `max_filter_versions:`,
662
+ and `allow_approximate:`.
519
663
 
520
664
  ### Which records are selected
521
665
 
522
- - **A record needs its own version in the window.** Editing only a comment does
523
- not select its article. Once an article is selected, its comment activity is
524
- included when you request `associations: [:comments]`.
666
+ - **A record is selected when it, or a record on a requested association,
667
+ changed in the window.** With `activity: true`, which `report` always uses,
668
+ editing only a comment selects its article when you request
669
+ `associations: [:comments]`. A comment that moved between articles selects
670
+ both. Nested paths count at every level, and a `has_many :through` counts its
671
+ join rows too. Without `activity: true`, an analysis compares a record's own
672
+ versions and cannot show a change to its associations alone, so only a
673
+ record's own version selects it.
674
+ - **An analysis can still be empty.** A record is selected by any version on its
675
+ paths, but its analysis compares only the state you requested. A join row
676
+ whose role changed, or a version that touched only ignored columns, selects a
677
+ record with nothing to report.
525
678
  - **A relation checks current values.** `Order.where(status: "open")` means open
526
679
  now, not open during the window. Use `historical_filter:` to select by past
527
680
  values.
@@ -533,26 +686,43 @@ It accepts `associations:`, `ignore:`, `snapshots:`, `group:`,
533
686
  ID tie-breaker to any other order. If a join repeats a record, it is analyzed
534
687
  once, in order of first appearance.
535
688
  - Selection still scans the versions in the window, so keep windows reasonable
536
- for large tables.
689
+ for large tables. Each requested association adds one query per level, however
690
+ many records it selects. Changes to associations are found from PT-AT's
691
+ `version_associations`, so history recorded before PT-AT was installed selects
692
+ nothing.
537
693
 
538
694
  ### Deleted records
539
695
 
540
- A deleted record cannot be tested against a relation's conditions, because it
541
- has no row. Records that changed in the window but no longer exist are listed in
542
- `unreachable`. It is not paginated and does not mean those records matched the
543
- relation.
544
-
545
- `include_deleted: true` also analyzes them from their `destroy` versions and
546
- returns the results in `deleted`, keyed the same way. A value is `nil` when no
547
- `destroy` version was recorded. With this option, existing and deleted records
548
- together count toward `limit:`.
696
+ A relation's conditions read the table, and a deleted record has no row.
697
+ Records that changed in the window but no longer exist are listed in
698
+ `unreachable`, including those selected by a change to their associations. It
699
+ is not paginated and, without `include_deleted:`, does not mean those records
700
+ matched the relation.
701
+
702
+ `include_deleted: true` tests each of them against the relation's conditions,
703
+ as it stood when it was destroyed, and keeps only those that match. So a
704
+ report on `Article.where(account_id: account.id)` returns that account's
705
+ deletions, not every account's. The relation's conditions must be plain
706
+ conditions on the model's own columns, as a [readable association
707
+ scope's](#costly-associations) must; anything else, such as a join or SQL
708
+ text, raises `ConfigurationError`. Its order, limit, and offset are ignored
709
+ here, since they choose a page of existing rows. A record whose destroy
710
+ PaperTrail did not record has no state to test and stays listed.
711
+
712
+ It then analyzes the kept records from their `destroy` versions and returns the
713
+ results in `deleted`, keyed the same way. A value is `nil` when no `destroy`
714
+ version was recorded. With this option, existing and deleted records together
715
+ count toward `limit:`.
549
716
 
550
717
  ```ruby
551
718
  missing_history = report.deleted.select { |_identity, analysis| analysis.nil? }.keys
552
719
  ```
553
720
 
554
721
  A deleted record's `diff` and `to_snapshot` end at "no record". Its `timeline`
555
- lists only edits; its `activity_timeline` ends with the deletion.
722
+ lists only edits; its `activity_timeline` ends with the deletion. That last
723
+ step's `diff.record_presence_change.from` is the record just before it was
724
+ deleted, even when it was also created inside the window, where
725
+ `from_snapshot` is `nil`.
556
726
 
557
727
  ### Selecting by past values: `historical_filter:`
558
728
 
@@ -569,7 +739,10 @@ report = PaperTrailDiff.report(
569
739
 
570
740
  The filter checks the record as it was when the window started and after each
571
741
  change made inside it. A record matches if any of those states match. With a
572
- hash, every key must be equal (`==`). Values are read from the stored history
742
+ hash, every key must be equal (`==`) once both sides are cast to the
743
+ attribute's type, as `where` casts its values: `{ status: :published }` matches
744
+ an enum, and a date or time matches whether the history stored it as YAML or
745
+ as a string in a `json` column. Values are read from the stored history
573
746
  without calling `reify`, and unknown or unrecorded attributes raise an error
574
747
  instead of matching `nil`. For other checks, pass a callable such as
575
748
  `->(article) { article.title.start_with?("Pub") }`; it receives each state as a
@@ -577,14 +750,18 @@ model instance and may run queries of its own.
577
750
 
578
751
  - It looks only at the record itself, not its associations, and does not check
579
752
  whether a value was held for the whole window.
580
- - The relation still filters current rows. Matching deleted records are listed
753
+ - Only the history of records the relation selects is read, and the relation's
754
+ order, `limit`, and `offset` then page the ones that match. Deleted records,
755
+ which a relation cannot test, are read too, and those that match are listed
581
756
  in `unreachable`.
582
- - Every matching record counts toward `limit:`, including deleted ones and ones
583
- the relation later excludes. Matching stops as soon as the limit is exceeded.
757
+ - Matching records the relation selects count toward `limit:`, and matching
758
+ stops as soon as the limit is exceeded. Matching deleted records count only
759
+ with `include_deleted: true`, as without a filter.
584
760
  - `max_filter_versions:` limits how many versions the filter loads, including
585
- the one version after the window it reads for each record. The default is
586
- 10,000. Going over raises `FilterLimitExceededError` before the extra batch is
587
- checked. It does not limit queries inside a callable.
761
+ the one version after the window it reads for each record, and the versions
762
+ of deleted records. The default is 10,000. Going over raises
763
+ `FilterLimitExceededError` before the extra batch is checked. It does not
764
+ limit queries inside a callable.
588
765
  - `version_scope:` narrows which changes are reported, not which states are
589
766
  tested here.
590
767
  - It is accepted by `analyze_scope` and `report`, which select records from a
@@ -592,13 +769,88 @@ model instance and may run queries of its own.
592
769
 
593
770
  ### Costly associations
594
771
 
595
- Most associations are loaded for all records together. A few still need queries
596
- per record: associations with a scope (including `limit` or owner-dependent
597
- scopes), targets with a default scope, and `has_many :through` associations
598
- whose source is not a `belongs_to`. Batch methods raise
599
- `UnsupportedBatchPathError` for these by default and name the association.
600
- Remove it from `associations:`, or pass `allow_expensive: true` to accept the
601
- per-record cost. Single-record methods always allow them.
772
+ Most associations are loaded for all records together. That includes an
773
+ association whose scope, or whose target's default scope, is plain conditions
774
+ on the target's own columns:
775
+
776
+ ```ruby
777
+ has_many :active_cards, -> { where(archived: false) }, class_name: "Card"
778
+ has_many :urgent_cards, -> { where(priority: [1, 2]).order(:id) }, class_name: "Card"
779
+ default_scope { where.not(hidden: true) }
780
+ ```
781
+
782
+ The gem reads such a scope once and checks each record's stored values against
783
+ it at every point, so a record that moves into or out of the scope is added or
784
+ removed at the version that moved it. It reads equality, `IN`, their
785
+ `where.not` forms, and `and`/`or`/`not` combinations of them on integer,
786
+ boolean, and enum columns, with SQL's rules for NULL. A `nil` test
787
+ (`IS NULL`, `IS NOT NULL`) is read on a column of any type, so a soft-delete
788
+ scope such as `where(deleted_at: nil)` is read too. String and text columns
789
+ are read only on SQLite and PostgreSQL, and only without a column collation:
790
+ MySQL and SQL Server compare strings without regard to case or trailing spaces
791
+ by default, which the gem cannot repeat, and neither is PostgreSQL's
792
+ `citext`.
793
+
794
+ A `limit` or `offset`, and the order of a `has_one`, are read the same way
795
+ when the order is total and the gem can repeat it:
796
+
797
+ ```ruby
798
+ has_many :first_comments, -> { order(:position, :id).limit(3) }
799
+ has_one :latest_comment, -> { order(id: :desc) }
800
+ ```
801
+
802
+ At every point the gem ranks the records that meet the scope's conditions by
803
+ the order, then skips the offset and keeps the limit, as the association's own
804
+ query would have. The order must reach the primary key, so that no two records
805
+ tie, and each key before it must be an integer, boolean, integer enum, date,
806
+ or datetime column of the target, ascending or descending. NULL sorts where the
807
+ database puts it by default: last in an ascending order on PostgreSQL, first on
808
+ SQLite, MySQL, and SQL Server.
809
+
810
+ A datetime key, such as `order(created_at: :desc)`, is read when its column
811
+ declares a precision, as Rails 7 and later give every new `datetime` column,
812
+ when history keeps that many fractional digits, and when Active Record stores
813
+ times in UTC, its default. Active Record truncates a time to its column's
814
+ precision before writing it, so history then holds the value the database
815
+ sorts by. YAML history keeps every digit; JSON history, and a `json` or
816
+ `jsonb` object column, keep `ActiveSupport::JSON::Encoding.time_precision`
817
+ digits, 3 by default, so with them only a `datetime(3)` or coarser column is
818
+ read. A time-of-day column is not read: Active Record keeps the date it was
819
+ given in memory but stores only the time.
820
+
821
+ A `has_many :through` reads its parts as Active Record's own query joins them:
822
+ it keeps each part's conditions but not its `limit`, `offset`, or `has_one`
823
+ cut, so a through over `has_one :latest_comment` holds the records of every
824
+ comment that meets the part's conditions, not only the latest. Its own `limit`
825
+ or `offset` ranks the rows the join yields, so a record reached through two
826
+ join records fills two places, as in the query. A `has_one :through` whose
827
+ own scope or whose parts have a limit, an offset, or a `has_one` order is not
828
+ read.
829
+
830
+ The gem cannot read the rest: a `limit` or `offset` whose order is not total
831
+ or sorts by another column type (strings follow a collation) or by a datetime
832
+ history cannot rank exactly, scopes that take the owner
833
+ (`->(article) { ... }`), scopes written as SQL strings or ranges, scopes that
834
+ join or select, and conditions on other column types or other tables, such as
835
+ a string column on MySQL. For these it asks PT-AT, which costs queries per
836
+ record and is approximate: PT-AT rebuilds an association from its current
837
+ members, swapping in their older versions, and never checks the scope against
838
+ them. A record that entered the scope since is reported as a member it was
839
+ not, so a comment moved into `-> { where("position > 2") }`, or restored from
840
+ a soft delete, shows as changed rather than added.
841
+
842
+ So every method that reads history raises `UnsupportedAssociationError` for
843
+ such a path and names it. Remove it from `associations:`, rewrite its scope as
844
+ plain conditions on the target's own columns, or pass `allow_approximate:
845
+ true` to accept PT-AT's answer, which also accepts its cost of queries per
846
+ record in batch methods. A comparison of two current records reads no history
847
+ and needs no opt-in. `allow_expensive:` is deprecated and has no effect.
848
+
849
+ A polymorphic `belongs_to` is loaded together too, with one query per target
850
+ class. Its classes are known only from the records, though, so batch methods
851
+ cannot reject one with a scope they cannot read in advance: a record that
852
+ points at such a class is read on its own, without an error. A scope on the
853
+ polymorphic `belongs_to` itself is rejected.
602
854
 
603
855
  This check covers known association cases only. Your model callbacks, custom
604
856
  attribute types, and callable filters can still run queries. See the
@@ -613,6 +865,10 @@ paths; the parents in a path are included automatically:
613
865
  diff = PaperTrailDiff.compare(from, to, associations: [:author, "comments.replies"])
614
866
  ```
615
867
 
868
+ `associations:` also takes what `includes` takes: a single name
869
+ (`associations: :comments`) or hashes of names to what to include below them
870
+ (`associations: [:author, { comments: :replies }]`).
871
+
616
872
  Supported: `belongs_to`, `has_one`, `has_many` (including `:through`), and
617
873
  `has_and_belongs_to_many`. Associated models must use `has_paper_trail`, and
618
874
  historical states need PT-AT installed and enabled
@@ -642,9 +898,64 @@ comments.changed.first.associations["replies"].added
642
898
  the association already reports it. The same applies to the foreign key a
643
899
  `has_one` or `has_many` child points back with. Without the association, the
644
900
  key is an ordinary attribute.
901
+ - A polymorphic `belongs_to` reads each record's target from its type
902
+ column. Because the targets' classes differ, a path cannot continue past
903
+ it: `"notable.comments"` raises `UnsupportedAssociationError`. A record
904
+ whose type names a model without `has_paper_trail` raises
905
+ `UnversionedAssociationError` when its history is read; the gem cannot
906
+ check this earlier, since only the record names the class. A type that
907
+ names no class, such as one a renamed model left in its versions, raises
908
+ `IncompleteAssociationHistoryError` naming the record.
645
909
  - `has_many :through` uses the versioned join model's history, so join-model
646
910
  attributes are available. HABTM reports membership and the target's
647
911
  attributes; it has no join model, so it has no join attributes.
912
+ - A `has_many :through` holds, at each point, the records the owner reached
913
+ through the join records it had then. A record that moves to another join
914
+ record leaves one owner and joins the other, and so does every record under
915
+ a join record that moves to another owner.
916
+ - A `has_many :through` with `source_type:` holds only the records reached
917
+ through join records whose type column names that class, as Active Record
918
+ reads it. A join record that points at another type is never a member, and
919
+ its changes add no activity step. A join record whose type changes adds or
920
+ removes its target in the step of that version.
921
+ - A `has_many :through` that goes through a HABTM, as its `through:`, its
922
+ `source:`, or further down its chain, raises `UnsupportedAssociationError`
923
+ up front in every method. Active Record joins a HABTM through a hidden
924
+ model with no versions, so the through cannot be rebuilt from history.
925
+ Replace the HABTM with a `has_many :through` a versioned join model to
926
+ compare it. `association_paths` leaves such a through out.
927
+ - A HABTM can only be rebuilt on the root record. PT-AT records membership
928
+ when the record that owns it saves a version, in rows that name the
929
+ transaction and the association but not the owner, so a root checkpoint
930
+ holds the root's membership and no one else's. A HABTM later in a path, such
931
+ as `"tags.articles"` or `"author.articles.tags"`, raises
932
+ `UnsupportedAssociationError` up front in every method, even when the path
933
+ leads back to the root, as `"comments.article.tags"` does.
934
+ `association_paths` leaves such a path out. To follow another record's
935
+ memberships, compare that record's own checkpoints, or replace the HABTM
936
+ with a `has_many :through` a versioned join model.
937
+ - A HABTM with a scope, such as `-> { where(active: true) }`, or whose target
938
+ has a default scope, raises `UnsupportedAssociationError` up front in every
939
+ method. PT-AT records a HABTM's members by loading the association through
940
+ its scope when the owner saves, so a target outside the scope then is never
941
+ recorded, and one that enters the scope before the owner's next save is a
942
+ member history cannot show. Replace it with a `has_many :through` a
943
+ versioned join model, which can be scoped. `association_paths` leaves it
944
+ out.
945
+ - An association keyed on a column other than a primary key, such as
946
+ `has_many :comments, primary_key: :legacy_id`, raises
947
+ `UnsupportedAssociationError` up front in every method, and so does a
948
+ `has_many :through` with such a part. PT-AT records which record a child
949
+ pointed at by that record's ID, so history read that way would show other
950
+ records' children. `association_paths` leaves it out. Naming the primary key
951
+ itself, `primary_key: :id`, is accepted.
952
+ - A root that uses single-table inheritance gets the associations of the
953
+ class it had at each version, so you can request one that only a subclass
954
+ declares. Without an `item_subtype` column, PaperTrail stores only the base
955
+ class; the gem reads the subclass from the type column the version stored.
956
+ A version where the record was a class without that association raises
957
+ `UnknownAssociationError`, as a current record of that class would. A
958
+ stored type that names no class raises `InvalidEndpointError`.
648
959
  - Requesting associations for a historical state without PT-AT raises
649
960
  `AssociationTrackingUnavailableError`. Unknown names raise
650
961
  `UnknownAssociationError`; unsupported macros raise
@@ -671,13 +982,22 @@ end
671
982
  ```
672
983
 
673
984
  Without `synchronize_version_creation_timestamp: false`, the checkpoint takes the
674
- record's old `updated_at` as its time and lands in the past.
985
+ record's old `updated_at` as its time and lands in the past. Save one record per
986
+ transaction: HABTM membership cannot be read from a transaction in which
987
+ another record with the same HABTM also saved a version (see the
988
+ [HABTM limitation](#habtm-limitation)). A checkpoint records membership as it
989
+ is when saved, including a member attached or created earlier in the same
990
+ transaction. A change it captures is credited to the step before it, so its
991
+ author is whoever saved the version before the checkpoint, unless that is a
992
+ checkpoint the same person saved just before the change. Don't save the
993
+ instance that made the HABTM change again; it records the old membership.
675
994
 
676
995
  ### Files from ActiveStorage
677
996
 
678
997
  `has_one_attached` and `has_many_attached` point to ActiveStorage's own models,
679
998
  which PaperTrail does not version. Comparing them historically raises
680
- `UnversionedAssociationError` rather than claiming nothing changed. Versioning
999
+ `UnversionedAssociationError` rather than claiming nothing changed, and
1000
+ `association_paths` leaves them out. Versioning
681
1001
  ActiveStorage's models does not help either, because `has_one_attached` is a
682
1002
  scoped `has_one`.
683
1003
 
@@ -765,6 +1085,7 @@ for numeric IDs).
765
1085
  | `Step` | one `timeline` step |
766
1086
  | `ActivityStep` | one `activity_timeline` step |
767
1087
  | `ActivityBoundary` | one end of a step |
1088
+ | `ChangeRow` | one row from `change_rows` |
768
1089
  | `Analysis` | result of `analyze` |
769
1090
  | `ScopedAnalysis` | result of `analyze_scope` and `report` |
770
1091
  | `TraversalEntry`, `RecordReference` | entries from `each_change` and `each_entry` |
@@ -814,6 +1135,38 @@ end
814
1135
  Both return an `Enumerator` without a block. They never query the database. A
815
1136
  record reachable through two requested paths appears at both.
816
1137
 
1138
+ ### Change rows
1139
+
1140
+ For a table or an export, `change_rows` flattens activity into one row per
1141
+ change, each attributed to its step's source version:
1142
+
1143
+ ```ruby
1144
+ steps.last.change_rows # one activity step
1145
+ analysis.change_rows # an Analysis built with activity: true
1146
+ report.change_rows # { identity => rows }, deleted records last
1147
+
1148
+ row = report.change_rows.values.first.first
1149
+ row.change # :changed, :added, or :removed
1150
+ row.attribute # "body", or nil for a row about a whole record
1151
+ row.from # nil for :added
1152
+ row.to # nil for :removed
1153
+ row.record # a RecordReference, or nil for the root
1154
+ row.association_path # ["comments"], or [] for the root
1155
+ row.source # the step's source_boundary
1156
+ row.to_h # the above, plus version_id, event, whodunnit, recorded_at,
1157
+ # and transaction_id
1158
+ ```
1159
+
1160
+ - An edited field is one `:changed` row. A record that appeared or disappeared
1161
+ gets an `:added` or `:removed` row with `attribute` `nil`, then a row for each
1162
+ field that has a value and for each record it holds.
1163
+ - A replaced `belongs_to` or `has_one` target gives the old record `:removed`
1164
+ rows, then the new one `:added` rows.
1165
+ - Rows follow the diff's walking order, and an analysis's rows follow its steps
1166
+ in time order.
1167
+ - `Analysis#change_rows` raises `ConfigurationError` without `activity: true`.
1168
+ `report.change_rows` leaves out a deleted record without destroy history.
1169
+
817
1170
  ## JSON columns
818
1171
 
819
1172
  A `json` or `jsonb` column is compared as one value. `nested_changes` reports
@@ -833,10 +1186,63 @@ PaperTrailDiff.nested_changes(from_value, to_value) # also accepts two values
833
1186
  inside arrays are compared whole.
834
1187
  - A missing key is `PaperTrailDiff::NestedComparator::ABSENT`, not `nil`, because
835
1188
  `{"a": null}` and `{}` are different.
836
- - A value that is not JSON on both sides returns `{}`; the attribute diff still
837
- has the whole change.
1189
+ - Two arrays at the top, as a `json` column can hold, return one `ArrayChange`
1190
+ at the empty path: `{ [] => <added [3], removed [2]> }`.
1191
+ - A value that is not JSON on both sides, or an object facing an array, returns
1192
+ `{}`; the attribute diff still has the whole change.
838
1193
  - Keys such as `:a` and `"a"` in one object raise `AmbiguousNestedKeyError`.
839
1194
 
1195
+ ## What a version recorded
1196
+
1197
+ `recorded_changes(version)` returns the changes PaperTrail stored with one
1198
+ version in its `object_changes` column, as a frozen hash of `ValueChange`
1199
+ objects in attribute-name order:
1200
+
1201
+ ```ruby
1202
+ PaperTrailDiff.recorded_changes(version)
1203
+ # => { "title" => <from "Draft" to "Published"> }
1204
+ ```
1205
+
1206
+ A step's `diff` compares the record before and after an event instead. The two
1207
+ usually agree. They differ when:
1208
+
1209
+ - something was [saved without a version](#how-papertrail-stores-history)
1210
+ before the record's next version. The diff includes it; this does not.
1211
+ - `has_paper_trail` ignores an attribute or leaves it out of `only:`. PaperTrail
1212
+ stores its value but not its changes, so only the diff reports them.
1213
+ - the version is a `create` or a `destroy`. The diff reports
1214
+ `record_presence_change`; this lists each attribute changing from `nil`, or
1215
+ to `nil`.
1216
+ - an included association's foreign key changed. The diff reports the
1217
+ association instead; this keeps the column.
1218
+
1219
+ Other details:
1220
+
1221
+ - It returns `nil` when nothing was recorded: the version table has no
1222
+ `object_changes` column, or this version has no value in it. A version that
1223
+ recorded no changes, such as a [checkpoint](#saving-a-checkpoint), returns
1224
+ `{}`.
1225
+ - `ignore:` takes an array of attribute names and defaults to `updated_at`, as
1226
+ elsewhere. It does not take `paths:`, because a version does not say which
1227
+ association path it was reached through. Primary keys are left out.
1228
+ - Unlike PaperTrail's `version.changeset`, it never loads the record. That
1229
+ saves a query per version, and a deleted record's values keep their types,
1230
+ such as an enum's label rather than the integer PaperTrail stored.
1231
+
1232
+ For activity steps, pass `recorded_changes: true` instead of loading the
1233
+ versions again. Each step's sources then carry what their versions recorded,
1234
+ with no extra queries; see
1235
+ [what each event recorded](#what-each-event-recorded-recorded_changes-true):
1236
+
1237
+ <!-- executable:reference-recorded-changes -->
1238
+ ```ruby
1239
+ steps = PaperTrailDiff.activity_timeline(
1240
+ article, associations: [:comments], recorded_changes: true
1241
+ )
1242
+
1243
+ steps.last.source_boundary.recorded_changes
1244
+ ```
1245
+
840
1246
  ## Checking your setup
841
1247
 
842
1248
  List the associations a model supports, for example to build a settings
@@ -850,10 +1256,25 @@ PaperTrailDiff.association_paths(Article, max_depth: 3).map(&:to_h)
850
1256
  Each descriptor has `path`, `kind`, `target_type`, `through`, `polymorphic`,
851
1257
  and `cycle`. Cycles are marked and not followed.
852
1258
 
853
- Check two versions for known problems before relying on association history:
1259
+ Only associations a request accepts are listed, so each path can be passed to
1260
+ `associations:`. The PaperTrail versions association, a `has_many :through`
1261
+ that goes through a HABTM, a scoped HABTM, a HABTM below the root, an
1262
+ association keyed on a column other than a primary key, one whose ends keep
1263
+ their versions in different tables, and one whose target, or a model its
1264
+ `has_many :through` goes through, has no `has_paper_trail` are left out. A path
1265
+ history can read only approximately, such as one with a scope written as SQL
1266
+ text, is listed, but requesting it also takes `allow_approximate: true` (see
1267
+ [costly associations](#costly-associations)). A `has_many :through` Active
1268
+ Record cannot follow, such as one through a polymorphic `belongs_to`, raises
1269
+ Active Record's own error from any model discovery reaches, as requesting it
1270
+ does.
1271
+
1272
+ Check a comparison's endpoints for known problems before relying on
1273
+ association history. They are what `compare` takes: two versions, or a version
1274
+ and the current record.
854
1275
 
855
1276
  ```ruby
856
- report = PaperTrailDiff.diagnose(from_version, to_version, associations: [:tags, "comments.replies"])
1277
+ report = PaperTrailDiff.diagnose(from_version, article, associations: [:tags, "comments.replies"])
857
1278
 
858
1279
  report.ok? # no errors among the checks that ran
859
1280
  report.errors.map(&:code)
@@ -861,7 +1282,12 @@ report.warnings.map(&:code)
861
1282
  ```
862
1283
 
863
1284
  Association checks run only for requested associations. The version-order check
864
- always runs. `ok?` does not prove the history is complete.
1285
+ always runs. Every reason `compare` would refuse the same request is an error,
1286
+ with the message `compare` would raise, so `ok?` is false whenever `compare`
1287
+ would refuse: an unknown or unsupported association, missing association
1288
+ tracking, an unversioned target, or a path read only approximately. Pass
1289
+ `allow_approximate: true`, as you would to `compare`, and such a path is a
1290
+ warning instead. `ok?` does not prove the history is complete.
865
1291
 
866
1292
  ## Models, schemas, and edge cases
867
1293
 
@@ -871,14 +1297,27 @@ always runs. `ok?` does not prove the history is complete.
871
1297
  - **Non-integer primary keys**, such as UUIDs, are supported, and so is a string
872
1298
  `versions.item_id` column.
873
1299
  - **Custom version classes** (`has_paper_trail versions: { class_name: }`) are
874
- supported.
1300
+ supported. For association history, a model and each association you ask
1301
+ for must keep their versions, and their PT-AT version associations, in the
1302
+ same tables. PT-AT reads an association's history from its owner's tables
1303
+ and numbers transactions within one table, so it cannot follow an
1304
+ association into a different version table. Such an association raises
1305
+ `UnsupportedAssociationError` naming both tables, and `association_paths`
1306
+ leaves it out. One custom table shared by the models works like
1307
+ PaperTrail's own.
1308
+ - **A scoped versions association** (`has_paper_trail versions: { scope: }`)
1309
+ does not change what is read: every method reads all of a record's
1310
+ versions, because a history replayed without some of them would credit
1311
+ their changes to the next version kept. To report only some changes, pass
1312
+ [`version_scope:`](#reporting-only-some-changes-version_scope).
875
1313
 
876
1314
  ### Version order
877
1315
 
878
1316
  Versions are ordered by `created_at`, then by ID when timestamps are equal. That
879
- works when IDs increase with each insert, as autoincrement IDs do. UUID version
880
- IDs do not, so two UUID versions with the same timestamp cannot be ordered; the
881
- gem raises `AmbiguousVersionOrderError` naming both instead of guessing.
1317
+ works when IDs increase with each insert, as autoincrement IDs do, in one
1318
+ table. UUID version IDs do not, and models with their own version tables count
1319
+ IDs separately, so two such versions with the same timestamp cannot be ordered;
1320
+ the gem raises `AmbiguousVersionOrderError` naming both instead of guessing.
882
1321
  Timestamps stored to the second, as on a MySQL `datetime` column, make ties
883
1322
  common. `diagnose` reports this as an error.
884
1323
 
@@ -888,11 +1327,18 @@ earlier time. Keep server clocks synchronized.
888
1327
 
889
1328
  Association changes between versions with the same timestamp are placed using
890
1329
  the same order. A few association shapes fall back to PT-AT's own reifier, which
891
- uses timestamps only: scoped associations, unversioned targets, and the others
892
- listed in the [architecture notes](architecture.md). For those, `diagnose` warns
893
- about tied timestamps, and a comparison between two tied versions emits the
894
- `ambiguous_association_boundary.paper_trail_diff` notification. Recording
895
- timestamps below one second avoids the problem.
1330
+ uses timestamps only: a path history reads only approximately, which takes
1331
+ `allow_approximate: true`, a composite foreign key, and a polymorphic
1332
+ `belongs_to` whose target's default scope history cannot read. The
1333
+ [architecture notes](architecture.md) list them. When a request includes such a
1334
+ path, or any polymorphic `belongs_to`, whose targets are known only per record,
1335
+ `diagnose` warns about tied timestamps and a comparison between two tied
1336
+ versions emits the `ambiguous_association_boundary.paper_trail_diff`
1337
+ notification. Recording timestamps below one second avoids the problem. On SQL Server, PT-AT compares
1338
+ those timestamps only to about 3 milliseconds (a `datetime` tick, 1/300 s),
1339
+ because the adapter writes the times in its queries that way, so for these
1340
+ shapes two versions saved closer together than that may be read in the wrong
1341
+ order.
896
1342
 
897
1343
  ### History stored as JSON
898
1344
 
@@ -900,7 +1346,9 @@ PaperTrail's JSON serializer, and a `json` or `jsonb` `object` column, store
900
1346
  times to milliseconds by default (`ActiveSupport::JSON::Encoding.time_precision`).
901
1347
  For those histories the gem compares all times at that precision, so an
902
1348
  unchanged timestamp is not reported as changed. A change smaller than that
903
- precision was never recorded and cannot be reported.
1349
+ precision was never recorded and cannot be reported. A `json` or `jsonb`
1350
+ column also returns times and dates as strings, which PaperTrail leaves as
1351
+ they are; the gem casts them back to the attribute's type.
904
1352
 
905
1353
  ## Instrumentation
906
1354
 
@@ -932,9 +1380,9 @@ All errors inherit from `PaperTrailDiff::Error`.
932
1380
  | Error | Raised when |
933
1381
  | --- | --- |
934
1382
  | `ConfigurationError` | An option is invalid |
935
- | `UnsupportedBatchPathError` | A batch requests a [costly association](#costly-associations) without `allow_expensive: true` |
936
- | `InvalidEndpointError` | A record is unsaved, destroyed, or has unsaved changes |
937
- | `UnloadedAssociationError` | `reload_live_endpoints: false` and an association is not loaded |
1383
+ | `UnsupportedBatchPathError` | No longer raised; kept until `allow_expensive:` is removed |
1384
+ | `InvalidEndpointError` | A record is unsaved, destroyed, or has unsaved changes, or `recorded_changes` is given something other than a version |
1385
+ | `UnloadedAssociationError` | The deprecated `reload_live_endpoints: false`, and an association is not loaded |
938
1386
  | `VersionMismatchError` | Two endpoints belong to different records |
939
1387
  | `ReversedEndpointsError` | Two versions are given newest first |
940
1388
  | `InvalidTimelineRangeError` | A range is invalid, for example `within:` with `from:`, or `:last` |
@@ -945,9 +1393,9 @@ All errors inherit from `PaperTrailDiff::Error`.
945
1393
  | `FilterLimitExceededError` | `historical_filter:` needs more than `max_filter_versions:` |
946
1394
  | `AssociationTrackingUnavailableError` | Association history is requested without PT-AT |
947
1395
  | `UnknownAssociationError` | An association name does not exist |
948
- | `UnsupportedAssociationError` | An association's macro is not supported |
949
- | `UnversionedAssociationError` | An association's target model has no `has_paper_trail` |
950
- | `IncompleteAssociationHistoryError` | PT-AT did not record enough to rebuild an association, such as HABTM without a transaction |
1396
+ | `UnsupportedAssociationError` | An association's macro is not supported, a path continues past a polymorphic `belongs_to`, a `has_many :through` goes through a HABTM, a HABTM is scoped or below the root, an association is keyed on a column other than a primary key, or history cannot read a [costly association](#costly-associations) and `allow_approximate: true` was not passed |
1397
+ | `UnversionedAssociationError` | An association's target model, or a model a `has_many :through` goes through, has no `has_paper_trail`, including the class a polymorphic `belongs_to` names |
1398
+ | `IncompleteAssociationHistoryError` | PT-AT did not record enough to rebuild an association, such as HABTM without a transaction or in a transaction another record with that HABTM also saved a version in, or a polymorphic type names no class |
951
1399
  | `UnsupportedLiveActivityError` | A HABTM activity range ends at the current record |
952
1400
  | `AmbiguousNestedKeyError` | `nested_changes` finds keys like `:a` and `"a"` together |
953
1401
 
@@ -956,14 +1404,18 @@ All errors inherit from `PaperTrailDiff::Error`.
956
1404
  Results can only be as complete as the history PaperTrail and PT-AT recorded.
957
1405
 
958
1406
  - Changes made with versioning off, with `update_columns`, or to `skip:` columns
959
- were never recorded. A range ending at the current record shows the first two
960
- in its last step, attributed to the change before them.
1407
+ were never recorded. The first two still appear, in the step for the record's
1408
+ previous recorded change and attributed to its author; see
1409
+ [how PaperTrail stores history](#how-papertrail-stores-history).
961
1410
  - Deleted versions, schema changes, and serializer changes can affect how old
962
1411
  versions read.
963
1412
  - Several queries against the current record are not one database snapshot.
964
1413
  Use a transaction with a suitable isolation level if that matters.
965
1414
  - HABTM membership history exists only where PT-AT recorded it in a
966
- transaction. HABTM changes cannot become their own activity steps.
1415
+ transaction, and only for the root record without a scope: a scoped HABTM,
1416
+ or one below the root, raises `UnsupportedAssociationError`, and a transaction that another record with
1417
+ that HABTM also saved a version in raises `IncompleteAssociationHistoryError`. HABTM changes cannot become their own
1418
+ activity steps.
967
1419
  - PT-AT has its own
968
1420
  [limitations](https://github.com/westonganger/paper_trail-association_tracking#limitations),
969
1421
  especially for some `has_one` histories. The gem follows PT-AT's configured