orfeas_lyra 0.9.2 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (93) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +2021 -17
  3. data/README.md +22 -9
  4. data/Rakefile +40 -1
  5. data/app/controllers/lyra/application_controller.rb +127 -0
  6. data/app/controllers/lyra/dashboard_controller.rb +33 -31
  7. data/app/controllers/lyra/flow_controller.rb +32 -34
  8. data/app/controllers/lyra/privacy_controller.rb +57 -18
  9. data/app/views/lyra/dashboard/discrepancies.html.erb +5 -5
  10. data/app/views/lyra/dashboard/index.html.erb +1 -0
  11. data/app/views/lyra/flow/crud_mapping.html.erb +1 -1
  12. data/app/views/lyra/privacy/pii_detection.html.erb +1 -1
  13. data/app/views/lyra/privacy/policy.html.erb +7 -0
  14. data/app/views/lyra/privacy/rights.html.erb +78 -0
  15. data/config/privacy_policies.rb +0 -3
  16. data/config/routes.rb +1 -0
  17. data/lib/lyra/access_log.rb +187 -6
  18. data/lib/lyra/advisory_lock.rb +236 -0
  19. data/lib/lyra/aggregate.rb +1 -1
  20. data/lib/lyra/associations/habtm_links.rb +382 -19
  21. data/lib/lyra/associations/maintenance.rb +229 -0
  22. data/lib/lyra/attribution.rb +48 -0
  23. data/lib/lyra/bypass_events.rb +446 -46
  24. data/lib/lyra/command_handler.rb +19 -14
  25. data/lib/lyra/configuration.rb +237 -10
  26. data/lib/lyra/consistency/read_your_writes.rb +4 -2
  27. data/lib/lyra/correlation.rb +121 -0
  28. data/lib/lyra/domain_events.rb +4 -0
  29. data/lib/lyra/dual_view.rb +53 -16
  30. data/lib/lyra/encrypted_attributes.rb +5 -0
  31. data/lib/lyra/engine.rb +67 -0
  32. data/lib/lyra/erasure.rb +1011 -106
  33. data/lib/lyra/event.rb +35 -1
  34. data/lib/lyra/event_data.rb +38 -0
  35. data/lib/lyra/event_flow.rb +31 -15
  36. data/lib/lyra/event_store_adapter.rb +28 -0
  37. data/lib/lyra/event_window.rb +58 -0
  38. data/lib/lyra/genesis.rb +548 -44
  39. data/lib/lyra/interceptors/association_interceptor.rb +689 -34
  40. data/lib/lyra/interceptors/crud_interceptor.rb +578 -91
  41. data/lib/lyra/internal.rb +26 -0
  42. data/lib/lyra/mode_sync.rb +51 -3
  43. data/lib/lyra/mode_transition.rb +257 -30
  44. data/lib/lyra/model_selection.rb +35 -0
  45. data/lib/lyra/own_connection.rb +17 -2
  46. data/lib/lyra/privacy/adapters/pam.rb +25 -2
  47. data/lib/lyra/privacy/gdpr_compliance.rb +314 -48
  48. data/lib/lyra/privacy/interface.rb +27 -7
  49. data/lib/lyra/privacy/subject_links.rb +143 -0
  50. data/lib/lyra/privacy/subject_records.rb +369 -0
  51. data/lib/lyra/projection.rb +7 -1
  52. data/lib/lyra/projections/async_checkpoint.rb +32 -0
  53. data/lib/lyra/projections/async_projection_job.rb +166 -23
  54. data/lib/lyra/projections/cached_joins.rb +33 -18
  55. data/lib/lyra/projections/cached_projection.rb +74 -33
  56. data/lib/lyra/projections/cached_relation.rb +3175 -321
  57. data/lib/lyra/projections/default_scopes.rb +254 -0
  58. data/lib/lyra/projections/deferred_relation.rb +23 -3
  59. data/lib/lyra/projections/event_store_reader.rb +47 -8
  60. data/lib/lyra/projections/foreign_keys.rb +92 -0
  61. data/lib/lyra/projections/lazy_projection.rb +165 -27
  62. data/lib/lyra/projections/model_projection.rb +165 -14
  63. data/lib/lyra/projections/rebuild.rb +34 -6
  64. data/lib/lyra/projections/reference_links.rb +463 -72
  65. data/lib/lyra/projections/typed_values.rb +629 -0
  66. data/lib/lyra/projections/unique_indexes.rb +595 -0
  67. data/lib/lyra/projections/uniqueness.rb +784 -0
  68. data/lib/lyra/projections/value_digest.rb +93 -0
  69. data/lib/lyra/purpose_bound_reads.rb +912 -77
  70. data/lib/lyra/repair.rb +119 -28
  71. data/lib/lyra/retention.rb +114 -8
  72. data/lib/lyra/rights/consent_store.rb +415 -0
  73. data/lib/lyra/rights/erasure_guard.rb +135 -0
  74. data/lib/lyra/rights/pending.rb +74 -0
  75. data/lib/lyra/rights/queries.rb +269 -0
  76. data/lib/lyra/rights/store.rb +512 -0
  77. data/lib/lyra/rights/subject_keys.rb +224 -0
  78. data/lib/lyra/rights.rb +281 -0
  79. data/lib/lyra/saves_without_effect.rb +69 -0
  80. data/lib/lyra/stream_tail.rb +319 -0
  81. data/lib/lyra/strict_data_access.rb +916 -106
  82. data/lib/lyra/temporal.rb +4 -2
  83. data/lib/lyra/touch_registry.rb +404 -0
  84. data/lib/lyra/verification/bypass_workflow.rb +452 -42
  85. data/lib/lyra/verification/rights_workflows.rb +170 -0
  86. data/lib/lyra/version.rb +1 -1
  87. data/lib/lyra/visualization/event_graph.rb +1 -1
  88. data/lib/lyra/visualization/timeline.rb +2 -2
  89. data/lib/lyra.rb +43 -2
  90. data/lib/tasks/lyra_mode.rake +110 -8
  91. data/lib/tasks/lyra_projections.rake +43 -2
  92. metadata +34 -4
  93. data/lib/lyra/associations/event_aware.rb +0 -224
data/CHANGELOG.md CHANGED
@@ -7,10 +7,2017 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [1.0.0] - 2026-11-22
11
+
12
+ ### Breaking
13
+ - **Lyra requires Rails 8.1: `rails >= 8.1, < 8.2`** (`lyra.gemspec`), where it declared
14
+ `rails >= 8.0`. Rails 8.0 was never tested: Lyra is developed and tested on Rails 8.1.4.1 (with
15
+ Ruby 4.0.10 and RailsEventStore 3.2.0), and CI tests only the lockfile. The upper bound is new
16
+ because Lyra patches Active Record internals (relations, associations, persistence), which a
17
+ Rails minor release may change, so a future Rails minor needs a Lyra release that is tested on
18
+ it. The `rails_event_store ~> 3.0` and Ruby `>= 4.0` requirements are unchanged. The blog app
19
+ asks for Ruby `>= 4.0`, Lyra's floor; no locked gem version changed.
20
+
21
+ - **A query uses every declared field it names, whatever it returns.** Purpose-bound reads checked
22
+ record loads and `pluck`; aggregates, filters and `exists?` were not checked, so under a purpose
23
+ that does not allow `name`, `Registration.group(:name).count`, `maximum(:name)` and
24
+ `where(name: "Ann").exists?` answered, and `find_by(name: "Ann")` raised for Ann and returned
25
+ `nil` for anyone else: a membership test. Now the declared fields a query names (its select or
26
+ pluck list, the aggregated column, `group`, `where`, `having`, `order`, the conditions of
27
+ `find_by`, class-level too, and `exists?`, a subquery, and the filter of `update_all`,
28
+ `delete_all`, `touch_all`, `delete_by` and `destroy_by`) are checked before it runs, so such code
29
+ under a purpose now raises `PamDsl::PurposeFieldMismatchError` (or, with no purpose under
30
+ `reads_without_purpose :deny`, `PurposeRequiredError`). `count`, `count(:all)` and `size` name
31
+ nothing and are not checked; `count(:name)` uses `name`. The check is split by what can be known:
32
+ a record load has its query's fields checked without the consent step (PAM's new
33
+ `validate_fields!`), then each record returned as before, consent included, for what it holds
34
+ and what the query named; aggregates, `exists?`, `ids`, `pluck` and bulk writes read everyone at
35
+ once (subject `Model$*`) and are refused under a purpose whose basis is consent. A joined
36
+ model's declared fields are checked against its own policy (subject `Other$*`). `find_each` and
37
+ `in_batches` check and record a query once, not per batch. ES-NoProj gives the same answers:
38
+ its relations keep what their query named, and it no longer disagrees with Monitor on
39
+ `exists?(hash)`. Rails' uniqueness validation and an association's `dependent:` handling when a record
40
+ is destroyed (`:delete_all`, `:nullify`, `:delete`, the restrict options, and the lookup of the
41
+ dependents `:destroy` destroys) run as part of the write and are not checked; `find_or_create_by`
42
+ and an application's own `where(author_id: id).delete_all` are. In ES-NoProj a block given to
43
+ `select` (now returning an Array, as in Active Record), `find` or `many?` sees only checked
44
+ records; it saw the cached records unchecked. So do its association readers (`belongs_to`,
45
+ `has_one`), `records`, `each` and `find_each` without a block, and `inspect`, which handed
46
+ records out unchecked. A subquery or CTE reads other people, so what it names is checked whole
47
+ (subject `Model$*`), even on the queried model; a `table.column` inside an SQL fragment belongs
48
+ to that table's model; a column named without its table that the queried model lacks belongs
49
+ to the joined model that has it; a condition on a `belongs_to` association names its foreign
50
+ key; `update_all`'s SET expressions read the columns they name (`"email = name"`); the
51
+ `returning:` columns of `insert_all` and `upsert_all` are read; and `cache_version` and
52
+ `cache_key_with_version` are aggregates. A refused check is never remembered, and the
53
+ application's block inside `find_each` or `in_batches` is checked each time it queries; only the
54
+ loop's own cursor queries are checked once. The purpose and Lyra's internal-read flag are kept
55
+ in Ruby's fiber storage, tied to their thread, so the fiber an Enumerator runs in
56
+ (`relation.to_enum.next`) has them and another thread does not; code that read
57
+ `Thread.current[:lyra_purpose]` uses `Lyra::PurposeBoundReads.current`. With no purpose and `reads_without_purpose :allow` the hooks return before
58
+ looking at the query. Limits: hand-written SQL, string joins, views, functions, triggers, custom
59
+ Arel nodes and CTEs are not seen, and an SQL fragment is matched by the queried model's declared
60
+ names only.
61
+ - **Lyra's escape hatches are Lyra's.** `Lyra.projection_write`, `Lyra::PurposeBoundReads.internal`,
62
+ `Lyra::Projections::ModelProjection.project` and `Lyra::Projections::Rebuild.project_events`
63
+ switch off the checks that keep every write in the log and every read under its purpose. They now
64
+ take a token only Lyra's own code can name, and raise `Lyra::InternalApiError` without it;
65
+ `ModelProjection.project_create`, `project_update` and `project_destroy` are private. Application
66
+ code that wrote without events (seeding, fixtures, a wipe whose history is not wanted) uses
67
+ `Lyra.unrecorded_write(reason:) { ... }`. It runs the block the same way and records that it ran
68
+ (reason, who, the caller) in the stream `Lyra::UnrecordedWrites`; `config.unrecorded_writes =
69
+ :deny` refuses it. The token keeps code from calling them by accident; it is no barrier against
70
+ code out to defeat it. A refused call leaves the flags of the block it ran in as they were.
71
+ - **`DualView#crud_state` reads the row as its caller does,** so under `reads_without_purpose :deny`
72
+ it needs a purpose that allows the declared fields, as any read does; before, it read as Lyra.
73
+ `compare`, `compare_all` and `find_discrepancies` follow it. Lyra's mode gate, the sampler,
74
+ repair still read as Lyra. The dashboard reads under a purpose, as the application does: the
75
+ request's `?purpose=`, else the new `config.dashboard_purpose`; a read the policy refuses answers
76
+ 403 and says how to name one.
77
+ - **PAM DSL: a purpose that declares no `basis` is `:undetermined`,** where it was `:consent`.
78
+ The register printed such a purpose as Art. 6(1)(a), and nothing checked it, because no consent
79
+ block asked for one. Access under it now raises `PamDsl::UndeterminedBasisError`, and the
80
+ Article 30 register lists it as a gap, as for a purpose the policy generator writes. A policy
81
+ that relied on the default must declare the basis it means (`basis :consent` included).
82
+
83
+ - **The Art. 15 export no longer has `processing_activities`.** `data_export`'s
84
+ `processing_activities` listed, per event, a purpose guessed from keywords in its
85
+ `metadata[:source]` (Lyra's events carry `lyra_interceptor` or `lyra_command_handler`, so unless
86
+ a `metadata_proc` set another source every event read "Application functionality"), whatever
87
+ the policy declared. The key is gone: the
88
+ copy's purposes are `processing_context`, taken from each record's model's policy (basis,
89
+ recipients, transfers, retention and sources per purpose). Code that read
90
+ `data_export[:processing_activities]` reads `processing_context`. The per-subject Art. 30 method
91
+ `GDPRCompliance#processing_activities`, and the GDPR report's key of that name, are unchanged.
92
+
93
+ - **PAM DSL: the consent grammar refuses what the Regulation does not allow, and a `:consent`
94
+ purpose always needs consent.** On a purpose whose basis is `:consent`, `required! false` (Art.
95
+ 6(1)(a)) and `withdrawable! false` (Art. 7(3)) raise `PamDsl::ConsentGrammarError` when the
96
+ purpose or its consent block is declared, in either order, through `define_policy` or the
97
+ methods of `Policy`. `granular!`, recorded and never enforced, is removed: it raises
98
+ `ConsentGrammarError`, and the consent report has no `granular` key. Access under a `:consent`
99
+ purpose now needs an active consent whether or not a consent block is declared for it; with no
100
+ block it passed the legal-basis check unchecked.
101
+
102
+ - **`created_at` and `updated_at` are ordinary columns, and event logs written before 1.0.0 are
103
+ not supported.** Every event Lyra writes now carries the two timestamps like any other column
104
+ (a create's `attributes`, an update's `changes` when they moved), DualView compares them, and
105
+ nothing treats them specially any more. The compatibility code for older logs is gone:
106
+ `Lyra::Event.created_attributes` no longer recovers a create's timestamps from its `changes`
107
+ (Monitor creates written by 0.9.x left them out of `attributes`), and `ModelProjection` no longer
108
+ stamps `created_at`/`updated_at` with the projection time when an event lacks them. An update
109
+ that left `updated_at` alone (`update_columns`, an `update_all` that does not set it) now leaves
110
+ it alone in the projected row too, where before a rebuild stamped the rebuild time and the row
111
+ differed from its events. An application with a log from Lyra 0.9.x should start a new event
112
+ store at 1.0.0 (Genesis imports the existing rows) rather than replay the old one; the per-mode
113
+ event shapes of 0.9.x are not supported.
114
+
115
+ - **`DualView#compare` and `#calculate_differences` return `{}` when the row and its replayed
116
+ state agree,** where they returned `{ no_differences: true }`. An agreement is now an empty hash,
117
+ and every entry of a non-empty one has the same shape, `{ crud:, event_sourced: }`: an attribute
118
+ that differs, or `exists: { crud:, event_sourced: }` (`true`/`false` each) when only one view has
119
+ the record, where it was `{ exists_mismatch: true }`. That entry keeps the result non-empty, so a
120
+ check for `{}` cannot take a missing record for agreement. Code that tested
121
+ `differences[:no_differences]` or compared with `{ no_differences: true }` must test
122
+ `differences.empty?`; code that tested `differences[:exists_mismatch]` must test
123
+ `differences.key?(:exists)`. Attribute differences are keyed by column name as a **string**
124
+ (`differences["email"]`, as Active Record keys `attributes`, and as HABTM link differences already
125
+ were), where they were symbols; the existence entry is the symbol `:exists`, so no column, a
126
+ column named `exists` included, can collide with it. Lyra's own callers
127
+ (`ModeTransition`, `Repair`, `DualView.find_discrepancies`, the recommendations) and the replay
128
+ verifiers of the BPI and Olist examples are updated.
129
+
130
+ ### Added
131
+ - **The switch into ES-NoProj refuses a database foreign key it cannot honour.** ES-NoProj leaves
132
+ the tables of monitored models empty: their records live in the event log. A database foreign
133
+ key from a table no monitored model writes into such a table (an unmonitored `Match` whose
134
+ `registration_id` references `registrations`) therefore made every write there that sets the
135
+ column fail with `ActiveRecord::InvalidForeignKey`. `Lyra::ModeTransition.to!` into ES-NoProj now
136
+ refuses such a constraint up front (`ModeTransition::Refused`, as for a default scope ES-NoProj
137
+ cannot evaluate), naming the referencing table and column, the referenced table, the constraint
138
+ and the loaded models that write the table, and the fix: drop the constraint
139
+ (`remove_foreign_key :table, name: "..."`) and keep the reference in the model (`belongs_to`
140
+ validates it), or monitor the referencing model. `force: true` switches anyway. Foreign keys
141
+ between monitored tables, and from a monitored table into an unmonitored one, sit on a table
142
+ that gets no rows in ES-NoProj and pass. The check reads `connection.foreign_keys` for each
143
+ table of each database the monitored models use, at the switch only
144
+ (`Lyra::Projections::ForeignKeys.blocking(models)`). ES-NoProj set by configuration at boot
145
+ (`config.projection_mode = :disabled`, which no switch gates) now gets a warning
146
+ (`Rails.logger.warn`, else `Kernel#warn`) for each such constraint and each default scope
147
+ ES-NoProj cannot evaluate, for the models enrolled by then (with eager loading, all of them);
148
+ the boot goes on, as for a retention rule that cannot act. `Lyra::ModeTransition.check_es_noproj!`
149
+ runs that check on demand and returns the warnings.
150
+ - **A warning for a retention rule that can never act.** `:soft_delete`, PAM's strategy for a
151
+ rule that names none, sets `deleted_at` or `discarded_at`; on a table with neither,
152
+ `Lyra::Retention` skipped every record past its period and kept it, and said so only in each
153
+ action's `detail`. Lyra now warns (`Rails.logger.warn`, else `Kernel#warn`), naming the model,
154
+ its policy and rule, and the fix (add the column, or `on_expiry :anonymize` or `:hard_delete`):
155
+ at boot for the models enrolled by then, and in each run that skips records for it, once per
156
+ model, not once per record. `Lyra::Retention.check_rules!(models:)` makes the boot check on
157
+ demand and returns the names of the models it warned about. Nothing else changes: the records
158
+ are still skipped, and only models with a rule of their own are acted on (the policy's
159
+ `default` period is not executed; now documented).
160
+ - **`config.record_touches`: keep timestamp-only touches out of the log, in Monitor.** An instance
161
+ `touch`, a `touch_all` and a `belongs_to ... touch: true` cascade each append an event, and an
162
+ application that touches a lot fills the log with writes that change only `updated_at` (on the
163
+ Olist replay through Solidus, cascades touching orders and shipments). `true`, the default,
164
+ records every touch as before. `false` leaves out a touch whose only changes are the model's
165
+ update timestamps (`updated_at`, `updated_on`); a touch that also changes another column
166
+ (`touch(:completed_at)`, a `lock_version` increment) is recorded in full, and so is an explicit
167
+ `update_all(updated_at: ...)` or `update_columns(updated_at: ...)`, which are not touches.
168
+ DualView, and with it the sampler, the gate's check and `lyra:repair`, then takes a difference
169
+ confined to those columns as agreement; a difference in any other column is reported with the
170
+ timestamps beside it. Monitor (and Disabled) only: Hijack and event sourcing take the log as the
171
+ record of every change and rebuild rows from it, where a missing touch would rebuild a stale
172
+ `updated_at`, and Hijack passes into event sourcing without a check. So `record_touches =
173
+ false` raises `ArgumentError` in those modes, and while it is false `mode=`, `hijack_enabled =
174
+ true`, `ModeTransition.apply_label` and an ungated `enable_hijack!` or `enable_event_sourcing!`
175
+ raise `ArgumentError`. A process with the option off registers in `lyra_mode_transitions` (kind
176
+ `touch_skipping`: host, pid, since, Lyra's version, last seen; `Lyra::TouchRegistry`) before it
177
+ leaves a touch out, refreshes the entry at most every 60 seconds from the touch path and
178
+ ModeSync's check, ends it when the option is set back to `true` and at exit, and logs a warning
179
+ once. An entry not refreshed for 180 seconds (the database's `clock_timestamp()`) is no longer
180
+ live. Registry writes never join the caller's transaction: each commits on its own, on a second
181
+ connection when the caller's is in a transaction (checked out for at most one second, with a
182
+ two-second `lock_timeout`); when none can be had, the touch is recorded. Each takes the
183
+ registry's advisory lock and re-reads the application's mode under it, and a switch takes the
184
+ same lock for its last look at the registry and its record (a check for its last look and its
185
+ certificate), so a registration and a switch are serialized: either the switch is refused, or the
186
+ registering process ends its entry, records the touch and is fenced. While this process has the
187
+ option off or any process has a live entry, every way into Hijack or event sourcing is refused
188
+ with `ModeTransition::Refused`, naming the processes and the way up: `ModeTransition.to!` (`force: true` included, and so the gated
189
+ `enable_*!` helpers), `check` (at its start and before it certifies; `rake lyra:mode:check`
190
+ prints it and exits non-zero), ModeSync's adoption (the switch stays unseen, and is adopted once
191
+ the entry has ended or expired) and a boot into such a mode; `rake lyra:mode:status` lists the
192
+ live processes. No certificate issued before an entry's latest refresh or end is taken. A process
193
+ with the option off does not boot into an application whose mode is Hijack or event sourcing,
194
+ cannot set the option false at runtime in one, and one that finds itself in such a mode later is
195
+ fenced: each ModeSync check raises, and so does every touch it would have left out, with the save
196
+ that cascades it. `Lyra::Projections::Rebuild.rebuild` is refused (`Rebuild::Refused`) with the
197
+ option off or a live entry, since it would set every touched row
198
+ back. To escalate: restart every process with the option on, run `lyra:repair` in Monitor (the
199
+ check then sees the stale timestamps; the repair records them, one `Updated` event per record),
200
+ then switch. A row with no stream that is touched with the option off is imported by Genesis as
201
+ it then is. Strict mode is unchanged (a `touch` is allowed, `touch_all` refused as
202
+ `update_all`). An unrecorded touch has no event, so no event names it as its cause.
203
+ `LYRA_RECORD_TOUCHES=false` sets it in the Solidus example. Accepts `"true"`/`"false"` strings.
204
+ On 200 Olist orders replayed through Solidus in Monitor, the log falls from 8,691 events (2,547
205
+ of them touches) to 6,331; the 187 touches left are order completions (`touch(:completed_at)`),
206
+ and the oracle (197/200 orders, as in Disabled), the mails and DualView (1,575 records, no
207
+ discrepancy) are unchanged.
208
+
209
+ ### Fixed
210
+ - **Every `causation_id` Lyra gives names an event in the log.** A write's scope fixed its event
211
+ id before its callbacks ran, and the writes they made recorded it as their cause. A save that
212
+ changes nothing records no event (see Changed), so its callbacks' writes named an event that
213
+ never existed: replaying 200 Olist orders through Solidus, 203 of 793 causation ids in Monitor,
214
+ ES-Sync and ES-NoProj alike, each a `LineItemUpdated` inside an `Order#save` with nothing to
215
+ change, which still saves its line items (has_many autosave). A write is now the cause of the
216
+ writes made after its own event is in the log; when it records none, they take the cause that
217
+ was in scope when the write began. The writes its `before_*` callbacks make (a `belongs_to`
218
+ autosave, a `dependent: :destroy`) come before anyone knows and name the write, as before. A
219
+ save that changes nothing still records nothing in its stream; when a write made in its
220
+ `before_*` callbacks names it as cause, Lyra records a `SaveWithoutEffect` marker with its id in
221
+ an internal stream, `Lyra::SavesWithoutEffect`, in the write's transaction. So it does for every
222
+ write that ends without its event in the log after naming it: an update or destroy that reaches
223
+ no row (Monitor's stale copy), an aborted save, a command refused after the id was taken
224
+ (Hijack, event sourcing), a write whose Monitor append failed, and a write that raised, which a
225
+ caller may rescue inside an enclosing transaction and commit (then in a savepoint, best effort).
226
+ The marker holds the model, the record's key, the operation and the reason (`no change`,
227
+ `no row`, `not saved`, `append failed`), no attribute value, and `replay: false`; replay, the
228
+ projections, ES-Lazy's catch-up, DualView, the dashboard's event window and the subject reports
229
+ pass it over. A write's scope now wraps its whole save or destroy callback chain, a model's own
230
+ `after_save` and `after_destroy` included: it opens around `create_or_update` and `destroy`, and
231
+ around a destroy chain an application runs itself with `run_callbacks(:destroy)` (a soft delete,
232
+ such as paranoia's). As an `around_save`/`around_destroy` callback it had closed before them
233
+ (Active Model prepends `after_*` callbacks), so their writes (Solidus's `Payment#update_order`)
234
+ had no cause and opened a correlation of their own. Bypass writes (`touch`, `update_columns`)
235
+ and jobs created during a write take their cause the same way. An access-log entry, which
236
+ commits on its own connection and outlives a write that rolls back, names no write in scope: its
237
+ cause is the one from outside every write scope (`Lyra::Causation.outside_write_scopes`), and
238
+ its correlation ties it to the writes. `Lyra::Causation.integrity` reports the causation ids
239
+ that name no event in a store; none do once the transactions commit, except a cause the
240
+ application sets itself and an append lost under Monitor's log-and-continue policy. The
241
+ correlation id the outermost write opens is still its event id when it records nothing: a
242
+ correlation id is a group's label, not a reference.
243
+ - **Hijack no longer keeps the event of a row write that failed.** Hijack appends a write's event
244
+ before its row write. On a model with optimistic locking the row write can raise
245
+ `StaleObjectError` after the append, and a caller that rescued it inside an enclosing
246
+ transaction committed an `Updated` (or `Destroyed`) for a write that never happened: the log and
247
+ the row parted, and DualView reported the record. The append and the row write now share a
248
+ savepoint, which takes the event with it when the row write raises. The row lock taken before
249
+ the append is still held to the end of the transaction when the write succeeds. Models without
250
+ optimistic locking take no savepoint: their row write does not raise for a row that moved.
251
+ - **DualView on a model without `created_at` or `updated_at`.** `DualView#crud_state` read both
252
+ whatever the model had, so a model with only `updated_on`, or no timestamps, raised
253
+ `NoMethodError` in DualView, the mode gate's check and `lyra:repair`. Its `timestamps` now hold
254
+ the model's own timestamp columns (`timestamp_attributes_for_create_in_model` and
255
+ `..._for_update_in_model`), none for a model without.
256
+ - **ES-NoProj refuses an Active Record relation's condition it cannot read, instead of skipping
257
+ it.** A named scope called on a relation, a relation passed to `merge`, and a default scope are
258
+ read as hash conditions on the model's own columns. An equality or `IN` whose left side was not
259
+ a column was skipped: `scope :p, -> { where(Arel.sql("lower(name)").eq("pear")) }` called as
260
+ `Item.where(...).p` returned every record the other conditions selected, and a default scope
261
+ made only of such a condition was taken as evaluable and not applied. A column of another
262
+ table was read as the model's column of the same name, a right side that is not a value
263
+ (another column, an SQL literal) was compared as if it were one, and a column named twice
264
+ (`where(qty: 2).where(qty: 5)`) kept only the last condition. Each now raises
265
+ `UnsupportedQuery`, except the column named twice, whose conditions are both applied, as SQL
266
+ does. Orders likewise: an order on another table's column (`"posts.title"`, an Arel attribute
267
+ of another table) sorted by the model's column of that name, and a direction other than `asc`
268
+ or `desc` (`order(qty: "desc nulls last")`) sorted ascending; both are refused.
269
+ - **ES-NoProj's `exists?(false)` and `exists?(nil)` are `false`, as in Active Record.** They
270
+ answered whether the relation had any record.
271
+ - **ES-NoProj's `limit!` limits the relation it is called on**, as Active Record's; it returned a
272
+ limited copy and left the relation whole. The bang query methods it does not implement
273
+ (`offset!`, `reorder!`, `distinct!`, `group!`, `none!`, ...) are refused, as before.
274
+ - **ES-NoProj's `only` keeps the parts named and drops the rest, as Active Record's**
275
+ (`only(:where)` keeps the conditions and drops the order, limit, select and group; `only(:order)`
276
+ orders every record). It returned the relation unchanged, its order and limit kept. The
277
+ conditions and order are taken off as `unscope` takes them off; `only` of a joined relation
278
+ without its joins is refused (`UnsupportedQuery`).
279
+ - **ES-NoProj's pagination is Kaminari 1.2.2's.** `page(n)` is `limit(size).offset((n - 1) *
280
+ size)`, the size the model's `paginates_per` (else Kaminari's `default_per_page`, else 25), or its
281
+ `max_paginates_per` when that is smaller; a page number below 1, `nil` or not a number is page 1.
282
+ `page(n)` without `per` answered with every record, and the maximum was not applied. `per(m)` is
283
+ the page of that size with the same number, at most `max_per_page:`, the relation's
284
+ `max_paginates_per(n)` (kept on the relation; it set the model's) or the model's; `per(nil)`, a
285
+ negative or a non-numeric size keep the page as it is (they gave an empty page). `padding`,
286
+ `next_page`, `prev_page`, `first_page?`, `last_page?`, `out_of_range?`, `total_pages` and
287
+ `current_page` are Kaminari's (the first five were missing), as are, on a page only,
288
+ `entry_name` (`page_entries_info`'s), `without_count` (its `last_page?` and `out_of_range?`
289
+ look at the page; `total_count` raises Kaminari's error) and the deprecated `current_per_page`.
290
+ `total_count` and `total_pages`
291
+ count the records the relation selects before the window, through the same purpose check and
292
+ rights as any count; they counted the page's records, so a paginated list showed one page of
293
+ one. `Model.page(n)`, Kaminari's own, gives the same page: the relation answers what Kaminari's
294
+ relation modules define (extending it with them was refused when read). `page` no longer
295
+ changes the relation it is called on.
296
+ - **ES-NoProj takes a limit or offset after the relation's conditions and order, as SQL does.**
297
+ A `where`, `where.not`, `order`, scope, `merge`, `select` or `unscope` called after `limit` or
298
+ `offset` acted on the window's records (`Model.order(:code).limit(2).where(status: "draft")`
299
+ filtered the first two records); it now applies before the window. A second `limit` or `offset`
300
+ replaces the first (`limit(1).limit(3)` is three records, `limit(2).offset(1)` records 2 and 3);
301
+ they applied to the window. `limit_value` and `offset_value` are the values given, as in Active
302
+ Record (`limit_value` was the page size, `nil` after `limit`; `offset_value` was 0 without an
303
+ offset). `find`, `find_by` and `exists?` with a condition on a window apply the condition first
304
+ and keep the window's offset, as Active Record's `where(...).take` does. `sum`, `average`,
305
+ `minimum` and `maximum` of a window are Active Record's: its one aggregate row, which the window
306
+ keeps (the aggregate over every record the relation selects) or leaves out (an offset, a limit of
307
+ 0: `0` for `sum`, `nil` otherwise); they were computed over the window's records. `or` of two
308
+ windows needs the same window, as in Active Record (an `ArgumentError` otherwise). A grouped
309
+ aggregate of a window, which Active Record limits to some of the groups, is refused. Under a
310
+ right in force, a window's count, pluck, ids, exists? and batch sizes are taken from the records
311
+ the right does not leave out, as SQL takes LIMIT and OFFSET after its WHERE (the window was
312
+ taken first, and then shortened by a record left out of it). `find` with
313
+ several ids (`find(a, b)`, `find([a, b])`) is Active Record's, window included; it raised
314
+ `ArgumentError`.
315
+ - **ES-NoProj's `or` returns the records in the relation's order** (else by primary key), each once;
316
+ it returned this relation's records, then the other's.
317
+ - **ES-NoProj's `reselect` replaces the selected columns, a second `select` adds to the first, and
318
+ `unscope(:select)` and `unscope(:group)` take them off, as in Active Record.** `reselect` did
319
+ nothing and a second `select` replaced the first, so the records came back without the columns
320
+ asked for (`MissingAttributeError`); `unscope(:select)` and `unscope(:group)` were ignored.
321
+ `select` and `reselect` with no column raise Active Record's `ArgumentError`. The relation's
322
+ plan (what `unscope`, `rewhere` and `only` replay) now survives `select`, `group` and
323
+ `create_with`, and a replay keeps them.
324
+ - **ES-NoProj's `unscope(:where)` of a relation filtered by an SQL fragment is refused** (`Model
325
+ .where("qty > ?", 2).unscope(:where)`); it kept the fragment's condition. `where.not` now
326
+ continues the relation's plan, so `unscope`, `rewhere` and `merge` can take it off, as Active
327
+ Record does (they were refused after it). `unscope(where: :owner_id)` takes off a condition
328
+ named by its `belongs_to` association (`where(owner: ann)`); it was kept.
329
+ - **ES-NoProj's `distinct` of selected columns (`select(:status).distinct`, `SELECT DISTINCT`) is
330
+ refused**, in either order and through `merge`: its rows are distinct values, not records, and
331
+ it answered with the records.
332
+ - **ES-NoProj batches as Active Record does.** `find_in_batches`, `find_each` and `in_batches` go
333
+ by primary key (or the `cursor:` columns, in `order: :asc` or `:desc`, from `start:` to `finish:`),
334
+ whatever the relation's order (logged, or `ArgumentError` with `error_on_ignore: true`), keeping its
335
+ limit; they followed the relation's order and took no options. Each `in_batches` batch is the
336
+ relation narrowed to the batch's records, in the relation's order (`load: true`: the batch's
337
+ records). `in_batches` without a block returns Active Record's `BatchEnumerator` (`each`,
338
+ `each_record`, `update_all`, `delete_all`, ...); it raised `LocalJumpError`. The enumerators of
339
+ `find_each` and `find_in_batches` have Active Record's sizes. Counted (those sizes, and
340
+ `in_batches`' batches of keys), the records leave out those a right in force leaves out, before
341
+ the relation's limit, as a count does; a restricted record was counted. Loaded (`find_each`,
342
+ `find_in_batches`, `in_batches(load: true)`, `each_record`, or a batch's records), each is
343
+ checked as a load is, and a restricted record raises, as in Active Record. The batches of a
344
+ relation with no conditions are ranges of keys, as Active Record's are (`use_ranges:`), so a
345
+ record the keys left out is in one, and loading that batch raises as Active Record's does.
346
+ Batches of a relation with an offset are taken as Active Record takes them: it applies the
347
+ offset to each batch's query (see below).
348
+ - **An ES-NoProj bulk write writes what its relation selects when it is called, as Active Record's
349
+ does, and no longer brings back a record destroyed since.** A relation, and each batch
350
+ `in_batches` yields, held the records it selected when it was built, loaded or not, and
351
+ `update_all`, `delete_all`, `touch_all`, `destroy_all`, `update_counters` and `update` wrote
352
+ those. A record deleted in the meantime (by another relation's `delete_all`, a `destroy`, an
353
+ erasure, or during an `in_batches` loop) got an `Updated` event after its `Destroyed` and read
354
+ as live again, so a stale relation could undo a delete or an erasure; `delete_all` appended a
355
+ second `Destroyed`; `destroy_all`, `update` and `update!` raised `Lyra::RecordDestroyedError`
356
+ part-way. A record changed or created since was written or left out by the old selection, and a
357
+ window was not taken again: `order(:code).offset(1).limit(3)`, after one of its records was
358
+ deleted, wrote the two it still held where Active Record writes the next three. A relation now
359
+ keeps how it was built (`Lyra::Projections::Rebuildable`): where it started (every record of
360
+ the model, a foreign-key lookup, an association, a `:through` association with no association
361
+ scope on the way, its default scopes kept as they were built) and the query methods called on it since, with their arguments
362
+ copied when they were called, so that a caller changing an array, hash or string afterwards
363
+ does not change what the write selects, as it does not change Active Record's query. A scope or
364
+ class method called on the relation is the application's code and is not run again: what it
365
+ built is recorded as the query methods it called, with the values it used then (`Time.current`,
366
+ the ids it plucked), as Active Record binds them. A step that changes no selection (`extending`,
367
+ `includes`, `readonly`, ...) is not recorded; one with another block, or an argument that cannot
368
+ be copied, leaves the relation without a recipe. `unscope`, `rewhere`, `except` and `only`,
369
+ which rebuild the relation from its plan, keep the recipe that rebuild records (where the plan
370
+ started, read again), so they too select at write time; they wrote the records the plan held
371
+ when the relation was built, with their old values. A write builds the relation again from the
372
+ event store: the aggregate locks of the records it holds are taken, the lock a single record's
373
+ update or destroy takes, all or none (`Lyra::AdvisoryLock.xact_lock_all`: tried without waiting
374
+ in one statement; when one is held, the others are given back and the next try first waits for
375
+ that one, so two writers over overlapping records wait for each other instead of deadlocking),
376
+ then the selection is read; a record that came to match since is locked in turn and the
377
+ selection read again (up to four times, after which a record not yet locked is left out, as if
378
+ it had matched after the write). A destroyed record is not in the selection. The count returned
379
+ is Active Record's. A relation keyed by primary key (`where(id: ...)`) is read again by those
380
+ keys only. `destroy_all`, `update` and `update!` load the selection as Active Record's do, or,
381
+ once the application has loaded the relation (`to_a`, `each`, ...), work on the records loaded,
382
+ as Active Record's do: there a record destroyed since is left out of `destroy_all` (Active
383
+ Record also runs `destroy` on it, reaching no row, and returns it) and takes `update`'s
384
+ attributes in memory without being written. `destroy_all` delivers its records as a load does,
385
+ so a record under a restriction raises `RestrictedProcessingError` before any is destroyed, as
386
+ in Monitor; it was destroyed. It checks each record under its lock before its destroy, which
387
+ checks again after its `before_destroy` callbacks (one that deletes the record is refused with
388
+ `Lyra::RecordDestroyedError`, and rolled back). Each batch of `in_batches` is the relation
389
+ narrowed to the batch's keys, as in Active Record, so a record of a later batch that no longer
390
+ matches is not written by the batch's bulk write (and the batches are read as the loop reaches
391
+ them: see below). A relation without a recipe (made from given records, such as a
392
+ `dependent: :delete_all` or `:nullify`'s children, or from a scope or argument that could not
393
+ be recorded) writes the records it holds, less those whose streams end in a destroy, read under
394
+ the same locks (`Lyra::StreamTail.destroyed_streams`: one query for all of them where each last
395
+ event is a create or an update). The cost is on writes only: one locking statement (in a
396
+ savepoint) and one read of the selection per bulk write, the read of the records with the given
397
+ keys for a keyed relation; one read of the streams for a relation without a recipe;
398
+ `destroy_all` adds two locks and two stream reads per record. On 2,047 records, a
399
+ `where(id: x).update_all` takes about what it took (165 ms against 156), a `where(code: ...)`
400
+ one about three times (one more read of the model's records at the write). Reads are unchanged
401
+ but for recording each query method's call. With `lock_aggregate_before_append` off nothing is
402
+ locked or checked, as for a single record. ES-Sync, Hijack and Monitor write through SQL and
403
+ were not affected; for ES-Async and ES-Lazy, see below.
404
+ - **ES-Async bulk writes leave out a row whose destroy is waiting for its projection job.** The
405
+ row stays in the table until the job runs, and `update_all`, `delete_all` and `touch_all` wrote
406
+ it and appended an `Updated` after the `Destroyed` (or a second `Destroyed`), bringing the record
407
+ back. The write now reads the keys it matches, takes their aggregate locks (all or none), then
408
+ locks the rows (and Genesis admits rows that predate Lyra), reads the rows' streams in one query
409
+ and leaves out a destroyed record's row from the write, the events and the count, as Active
410
+ Record's reaches no row of it. A row that came to match while the locks were taken gives the row
411
+ locks back and the keys are read again (up to three times). `dependent: :nullify` leaves out a
412
+ destroyed child the same way: no event, and its row is not written. The same
413
+ `bulk_write_lock_limit` applies. On 1,000 rows the write costs what it did (4,013 queries against
414
+ 4,005, 3.05 s). For ES-Lazy, see below.
415
+ - **A record's locks are taken in one order: its aggregate lock, then its row.** ES-Async's and
416
+ ES-Lazy's writes through callbacks take the aggregate (advisory) lock only, Hijack's and
417
+ ES-Sync's the row's only; the writes that skip callbacks (`update_columns`, `touch`, `delete`,
418
+ ES-Async's `update_all` and `delete_all`) and an erasure now take the aggregate lock first, then
419
+ write or lock the row. An erasure took the row first, and with the checks above a bulk write
420
+ took the rows first too, which deadlocked against a write holding the aggregate lock (an ES-Async
421
+ `update_all` against concurrent `update_columns`: 232 deadlocks in 300 writes; none now). The
422
+ order is written down at `Lyra::AdvisoryLock`.
423
+ - **A stale copy's `update_columns`, `touch` and `delete` no longer bring back a destroyed record
424
+ or log a second destroy.** A copy read before another copy, a relation's `delete_all` or an
425
+ erasure destroyed the record recorded an `Updated` after the `Destroyed` (`update_columns`,
426
+ `touch`) or a second `Destroyed` (`delete`), in ES-NoProj, ES-Async and ES-Lazy, and `delete`
427
+ did in Monitor, Hijack and ES-Sync as well. In the first three they now read the record's stream
428
+ under its aggregate lock first; elsewhere `delete` locks the row first. When the record is gone
429
+ they record nothing and return what Active Record returns for a row already gone: `false` from
430
+ `update_columns` and `touch` (the attributes are set in memory), the frozen record from
431
+ `delete`. A row still in the table (in ES-Async, before the destroy's job deletes it) is not
432
+ written either: Active Record's own write runs in a savepoint that is rolled back.
433
+ - **ES-NoProj's class-level counters write the record.** `Model.increment_counter`,
434
+ `decrement_counter`, `update_counters(id, ...)` (with `touch:`) and a record's `increment!` and
435
+ `decrement!` went to Active Record's SQL on the empty table: they returned 0 and recorded no
436
+ event. They now write as a relation's `update_counters` does (locks, the counters' current
437
+ values, an event per record), reading only the records with those keys, and return the number of
438
+ records written. With optimistic locking the lock version moves too, as Active Record's
439
+ `Locking::Optimistic.update_counters` moves it, so a copy read before raises `StaleObjectError`
440
+ on save. A counter cache, which Active Record's associations keep through these, is still left
441
+ as it is: ES-NoProj does not keep counter caches.
442
+ - **ES-NoProj's `update_counters` on a relation adds to the counters' current values, and two of
443
+ them no longer deadlock.** It added to the values the relation had read, so an update made since
444
+ (`update!(qty: 100)`, then `+1` gave the old value plus one, not 101) or a concurrent increment
445
+ was lost (two threads, 25 increments each over seven records, ended 175 short); Active Record's
446
+ SQL (`qty = COALESCE(qty, 0) + 1`) adds to the row as it is. It also wrote one `update_all` per
447
+ set of current values, each taking its records' locks, so two writers whose relations ordered
448
+ the records differently took them in opposite orders and deadlocked (30 of 50 calls). It now
449
+ takes every selected record's lock (all or none), reads the counters from the event store under
450
+ those locks, and writes the groups without locking or reading again.
451
+ - **`config.bulk_write_lock_limit` (default 5,000) caps the records one bulk write locks.** An
452
+ ES-NoProj bulk write (`update_all`, `delete_all`, `touch_all`, `update_counters`, a `dependent:
453
+ :delete_all` or `:nullify`, and a `destroy_all` inside a transaction already open), and an
454
+ ES-Async or ES-Lazy `update_all`, `delete_all` or `touch_all`, holds one advisory lock per record until its
455
+ transaction ends, and PostgreSQL's lock table is shared by every session: past about 12,800
456
+ locks with the default `max_locks_per_transaction` and `max_connections` 100 (about 23,200 with
457
+ 200), every session's writes failed with "out of shared memory". A larger write now raises
458
+ `Lyra::BulkWriteLockLimitError` (its `count` and `limit`) before it locks or writes anything,
459
+ and a parent's destroy whose dependent write is refused rolls back. Write in batches
460
+ (`in_batches(of: n) { |batch| batch.update_all(...) }`), each of which commits and gives its
461
+ locks back on its own, but not inside a transaction already open, which holds them all until it
462
+ ends; `nil` removes the limit.
463
+ - **ES-Lazy writes that skip callbacks see a destroy committed after their catch-up.** A bulk
464
+ write (`update_all`, `delete_all`, `touch_all`), `update_columns`, `touch`, `delete` and a
465
+ `dependent: :nullify` bring the tables up to date and then write; a destroy committed in
466
+ between left its row in the table, and the write wrote it and appended an `Updated` after the
467
+ `Destroyed` (or a second `Destroyed`). They now take the records' aggregate locks after the
468
+ catch-up and read their streams (as in ES-NoProj and ES-Async), leaving such a record out, and
469
+ replay onto its row each record whose stream has an event the tables have not had yet, so an
470
+ update committed in between is the row the write starts from (it was lost under the write, or
471
+ `update_all("tag = tag || ...")` read the old value). The
472
+ write's own reads no longer catch up again: a catch-up there waited for the catch-up lock while
473
+ the write held rows that a catch-up running elsewhere waited for, a deadlock that bulk writes
474
+ and writes that skip callbacks met against catch-ups under load before 1.0.0 too (9 in a run of
475
+ a bulk write, writes through and past callbacks and a catch-up in a loop, against none to 1
476
+ now: a catch-up writing rows in the log's order and a bulk write locking them in the table's can
477
+ still meet on rows alone). A catch-up that a catch-up under way refused no longer clears the
478
+ under-way flag of the outer one.
479
+ - **ES-NoProj's batches are read when the loop reaches them, as Active Record queries each.**
480
+ `in_batches`, `find_in_batches` and `find_each` on a relation that can be built again (see
481
+ above) read each batch then: the records after the last batch's last cursor values, in cursor
482
+ order, from `start:` to `finish:`, `of` of them, so a record inserted, changed or deleted during
483
+ the loop is seen as Active Record sees it (a later batch no longer yields a record deleted or no
484
+ longer matching, and yields one inserted). `use_ranges`, `order: :desc`, `load: true`,
485
+ `each_record`, a custom cursor and the rights (loads raise, counts leave out) are as before. A
486
+ relation with no recipe takes its batches from the records it holds, as before.
487
+ Each batch costs one read: where the relation starts from every record of the model and the
488
+ cursor is the integer primary key, of the model's keys and then of the next records by key (about
489
+ the batch's own cost); otherwise of the whole relation. Locally, `in_batches(of: 1000)` over
490
+ 10,000 records takes 2.8 s against 1.3 s (one read of every record up front), `find_each` 2.3 s
491
+ against 0.9 s.
492
+ - **ES-NoProj's batches of a relation with a limit or an offset are read as the loop reaches
493
+ them, and take the limit and offset as Active Record does.** They read every record up front,
494
+ and an offset was refused. Each batch is read when the loop reaches it, as above; a limit stops
495
+ the loop after its records (a batch reads `of` records, or what is left of the limit when that
496
+ is less), and the offset skips that many records after the last batch's cursor in every batch's
497
+ query, as Active Record applies `OFFSET` to each (`offset(1).find_each(batch_size: 2)` skips a
498
+ record before each batch). An `in_batches` batch is the relation narrowed to the batch's keys
499
+ with its limit and offset, as Active Record's `rewhere(cursor => values)` is, so the offset
500
+ skips records of the batch again, and its writes select so at write time (a batch's keys were
501
+ those of its window); a batch of a relation that starts from every record of the model is
502
+ built again from its own keys (`Rebuildable::Keyed`), not from every record. An
503
+ `in_batches(load: true)` batch holds the records read for it; its query methods, aggregates,
504
+ finders (`find_by`, `find`, `find_or_create_by`, `find_sole_by`, ...) and `update_all`,
505
+ `delete_all` and `touch_all` are that narrowed relation's, read again from the event store each
506
+ time, as Active Record's batch keeps its query and runs it (`CachedRelation#query_relation`).
507
+ Active Record's batch query of a relation with an offset has no `ORDER BY`, so which records
508
+ it skips is the database's plan (an index scan, or the table's physical order after an
509
+ update): `in_batches` of such a relation without an order raises `UnsupportedQuery`, and so do
510
+ the queries of its `load: true` batches (their records are read in cursor order, and given);
511
+ `find_each` and `find_in_batches`, ordered by the cursor, answer. `in_batches(use_ranges:
512
+ true)` with an offset raises it too (each range is read with the `OFFSET` and no `ORDER BY`),
513
+ and with `limit(0)` (Active Record's query has `OFFSET -1`, which the database rejects). The
514
+ enumerators' sizes take the offset too. The batches of a loaded relation (after `load`, `to_a`,
515
+ `each`, ...) are its loaded records, from `start:` to `finish:`, in cursor order, as Active
516
+ Record's are; they were read again. Each `in_batches` batch of it is loaded with its records,
517
+ and its query methods, aggregates, finders and `update_all`, `delete_all` and `touch_all` are
518
+ the relation's, as Active Record's batch is the relation loaded with them (each batch's
519
+ `update_all` writes every record the relation selects); `destroy_all` and `update` work on the
520
+ batch's records. The batches read by key took every record a step brought in: an `or` (whose
521
+ other relation is read whole) gave records of other chunks and from before the cursor, so a
522
+ batch repeated records and, without a limit, `find_each` and `find_in_batches` never ended
523
+ (since the batches were read as the loop reaches them, above). Each chunk now keeps only its
524
+ own records, which a step keeps or leaves out as it would among every record; `distinct` and
525
+ `group` are read whole. Cost, locally, 10,000 records: `limit(5000).find_each(batch_size:
526
+ 1000)` takes 1.36 s against 0.84 s (11 queries against one read of every record up front),
527
+ `in_batches(of: 1000)` 1.34 s against 0.85 s, as the batches without a limit cost (one read
528
+ of the keys and one of the batch's records per batch); `find`, `where`, `to_a`, `count` and
529
+ `first` are unchanged (same queries, time and allocations).
530
+ - **ES-NoProj's `load`, `reload`, `reset` and `loaded?` answer as Active Record's.** They raised
531
+ `UnsupportedQuery`. `load` loads the records as any read delivers them (the purpose check, each
532
+ record checked, a restricted one raising `RestrictedProcessingError`), passes each to its block,
533
+ and returns the relation, `loaded?` true after (before: false on `Model.all`, nil on a relation
534
+ derived from another, as in Active Record); a relation loaded already is not loaded again.
535
+ `reset` reads the relation again from the event store (Rebuildable), as a relation built at that
536
+ moment, and leaves it not loaded; `reload` is `reset` and `load`. A relation that cannot be
537
+ built again (one of given records) raises `UnsupportedQuery` there. A loaded relation's reads
538
+ do not read again, its `destroy_all` and `update` work on its loaded records and its
539
+ `update_all`, `delete_all` and `touch_all` select at write time, as before and as in Active
540
+ Record. `destroy_all` then resets the relation, as Active Record's does: it no longer holds the
541
+ records it destroyed (it held them, still loaded). A bang query method (`where!`, `order!`,
542
+ `limit!`, ...) on a loaded relation raises `ActiveRecord::UnmodifiableRelation`, as in Active
543
+ Record; it changed the loaded records. `DeferredRelation#loaded?`, which said whether an
544
+ association's records had been read, is `records_read?`.
545
+ - **ES-NoProj's `find_or_create_by` and `find_or_create_by!` create as `create_or_find_by` does,
546
+ as Active Record 8.1's.** A record a unique index of the table would conflict with is found
547
+ (or, outside the relation, `RecordNotFound`) instead of created again.
548
+ - **ES-NoProj's `excluding`, `without` and `in_order_of` are query methods, as Active Record's.**
549
+ They were Enumerable's, which returned an Array, so a write on them raised `NoMethodError`.
550
+ `excluding(*records)` (and `without`) is `where.not(primary key: ids)` of the records and of the
551
+ relations given; a record or relation of another model raises Active Record's `ArgumentError`.
552
+ `in_order_of(column, values, filter: true)` is a `where` of the column (`IS NULL` for `nil`; no
553
+ `where` with `filter: false`), which `rewhere`, `unscope(where: ...)`, `unscope(:where)` and
554
+ `except(:where)` take off as in Active Record, and an order by the column's position among the
555
+ values after any order the relation has, as Active Record's `CASE` added by `order!`, which
556
+ `unscope(:order)` and `reorder` take off; the values are cast as Active Record casts them (an
557
+ enum's label to its value, one the column cannot hold left out, `nil` a position of its own),
558
+ and none left is `none`. The column may name the model's table (`"items.code"`); an SQL
559
+ expression raises `UnsupportedQuery`. Both keep the relation's recipe, so `update_all`,
560
+ `delete_all`, `touch_all` and `destroy_all` on them select at write time, with a limit, offset,
561
+ `where` or order chained before or after.
562
+ - **ES-NoProj's `merge` takes the other relation's conditions where both name a column, and its
563
+ further arguments as options, as Active Record (7 and later) does.** `where(status: "active")
564
+ .merge(where(status: "draft"))` is the draft records; it was the records with both, none. A
565
+ condition named by a `belongs_to` association (`where(owner: ann)`) is one on its key. This
566
+ relation's conditions on such a column are taken off as `unscope` takes them off, a `where.not`
567
+ and an earlier merge included; where they cannot be (after an SQL fragment or a join, an
568
+ association's key, a condition on several columns at once, `where([:a, :b] => ...)`, or an `or`
569
+ of conditions on that one column) the merge is refused. An `or` over different columns is kept,
570
+ as Active Record keeps it. `merge(r1, r2)` intersected with `r2` too, and `merge(r, rewhere:
571
+ true)` was refused; both are now `merge(r1)`. `merge(nil)` and `merge(false)` raise
572
+ `ArgumentError` (they returned the relation), as does a value that is not a relation; `merge`
573
+ of an Array is the loaded records both have, and of a Proc the Proc run on the relation. The
574
+ other relation's limit and offset (an event-store relation's window, or an Active Record
575
+ relation's) replace this one's, its group and select columns are added, its distinct taken and,
576
+ if it reorders, its order replaces this one's; they were dropped (an Active Record relation's
577
+ limit was taken inside this one's). A relation joined on one side, or both to the same
578
+ associations, merges as the join's rows (it was refused); a condition on a joined table that the
579
+ other relation names too, and relations joined to different associations, are refused.
580
+ - **ES-NoProj's `relation.update(id, attributes)` and `update!(id, ...)` are Active Record's
581
+ `Model.update(id, ...)`:** the model's record with that id is found and written through the
582
+ model's own read and write paths (its default scope, the purpose check, an event), whether or
583
+ not the relation selects it, as Active Record does. They raised `RecordNotFound` for a record
584
+ outside the relation.
585
+ - **ES-NoProj's `update_counters({})` raises Active Record's `ArgumentError`** ("Empty list of
586
+ attributes to change"); it returned 0. `update_counters(touch: true)` with no counter touches the
587
+ records, as in Active Record; it did nothing.
588
+ - **A scope answers the same under ES-NoProj on the model and on a relation.** A scope called on
589
+ a relation now runs its body on that relation, as Active Record runs it (it was built as an Active
590
+ Record relation and the relation narrowed by it). A scope with an Arel condition
591
+ (`where(arel_table[:qty].eq(2))`) called on the model was refused while the same scope on a
592
+ relation was answered; both are answered, by building that scope as Active Record builds it. A
593
+ scope whose body returns a value (a count, a record) returns it, as in Active Record; on the model
594
+ it returned the relation, on a relation it was refused. One that returns an Active Record
595
+ relation of the model answers with that relation's records (it returned the relation the scope was
596
+ called on). A scope's `where.not` of a list is now answered on a relation as `where.not` is (it
597
+ was refused). As in Active Record, the body runs with no current scope (a body reading
598
+ `Model.where(...)` read the relation the scope was called on), and a bang method in it changes a
599
+ copy of that relation (`limit!(1)` limited the caller's). A `scoping` block's relation is seen
600
+ inside a fiber or an enumerator in it, as Active Record's current scope is (they read every
601
+ record).
602
+ - **`update_all(foreign_key: nil)` on an Active Record relation of a monitored model under
603
+ ES-NoProj** (counted as a dependent nullify) published events for the records the event store
604
+ holds with that foreign key, every other condition of the relation skipped: `where(author_id: 1,
605
+ status: "published")` nullified the author's drafts too in the log. The records are now those the
606
+ relation's where clause selects, all of it, read from the event store (not the table first); a
607
+ condition that cannot be read refuses the write (`UnsupportedQuery`). The events and the write
608
+ are one transaction where events are required, so a write that fails after its events were
609
+ published takes them back (a refused write used to leave its records nullified in the log).
610
+ `author.articles.update_all(author_id: nil)` now nullifies under ES-NoProj what Monitor does.
611
+ - **A bulk write with an offset in Monitor and the other table modes** (`order(:code).offset(5)
612
+ .delete_all`, a page's `update_all`) wrote no row: the write by the snapshot's keys applied the
613
+ offset again. It now writes the rows the relation selects, as Active Record does.
614
+ - **A polymorphic `has_one ... as:` under ES-NoProj matches the owner's type**, as the `has_many`
615
+ path does; it returned a record of another type of owner with the same id.
616
+ - **A `has_one` with a scope of its own is narrowed by it under ES-NoProj.** `has_one
617
+ :selected_rate, -> { where(selected: true) }` and `has_one :latest, -> { order(id: :desc) }`
618
+ answered with whichever record of the owner's the event store found first; the scope is now
619
+ applied to the owner's records and the first in its order is the answer, as Active Record's
620
+ `LIMIT 1`. A scope ES-NoProj cannot evaluate raises `UnsupportedQuery`. The record is checked
621
+ as a loaded row, as a `has_one` without a scope's is; it was returned unchecked.
622
+ - **A `has_one :through` a plain `has_one` reaches its target as Active Record's join does under
623
+ ES-NoProj.** It followed only the intermediate record the `has_one` reads, where the join reaches
624
+ the targets of every intermediate record the owner has (those the `has_one`'s scope selects),
625
+ narrowed by the association's own scope. When either scope orders, the first target in that
626
+ order is the answer; in no order, the one target there is (it could be `nil` before, when the
627
+ intermediate record read had none), and with more than one ES-NoProj now refuses
628
+ (`UnsupportedQuery`) rather than answer with one of them. The intermediate records are read as
629
+ Lyra's own read, without a purpose check, as the join reads them in SQL; the `has_one` reader
630
+ used before was checked, and under a purpose that does not allow the intermediate model's fields
631
+ raised `PamDsl::PurposeFieldMismatchError`.
632
+ - **ES-NoProj reads a `:through` association's intermediate records afresh, as Active Record's
633
+ join reads the tables.** A `has_one :through` a `belongs_to`, and a `has_many :through`, took
634
+ them from the owner's association when it was loaded: after the intermediate record changed
635
+ through another instance (moved to another owner), the answer came from the stale copy. Reading
636
+ the target also left the intermediate association loaded on the owner, which the join does not.
637
+ Every record on the way is now read on an association object of its own (from the event store,
638
+ or with SQL for a model not read from events), and the owner's associations are left as they
639
+ were. As the join, such a read reads the intermediate records each time: a loaded intermediate
640
+ association no longer saves the reads.
641
+ - **ES-NoProj checks only the records a `:through` association returns, as Active Record's join
642
+ does.** The intermediate records reached through a `belongs_to` or a `has_many` were checked as
643
+ loaded rows (a purpose that does not allow the intermediate model's fields raised
644
+ `PamDsl::PurposeFieldMismatchError`), and every target was checked before the association's
645
+ own scope, order and limit: a restricted target the scope leaves out (`scoped_tags`,
646
+ `tags.where(...)`, a limit, the other candidates of a `has_one :through`) raised where Monitor
647
+ answers. The records on the way are now read as Lyra's own read; the records returned are
648
+ checked, and a query on the association (`count`, `exists?`, `pluck`) is checked as a query
649
+ through the intermediate models, as the join's SQL is (a restriction in force for one of them
650
+ refuses it, as in Monitor).
651
+ - **An association scope that unscopes brings back a `:through` association's target that the
652
+ target's default scope leaves out, under ES-NoProj.** `has_one :x, -> { with_discarded },
653
+ through: ...` answered `nil` for a soft-deleted target, and a `has_many :through` left it out,
654
+ because the source association had already applied the target's default scope; and a `has_many
655
+ :through` whose source unscopes (`belongs_to :variant, -> { with_discarded }`) applied the whole
656
+ default scope again afterwards. The targets are now read without the default scope and narrowed
657
+ by it as the join applies it: less what the source's scope or the association's own unscopes.
658
+ The intermediate models' default scopes apply as before, as in the join, except on a step on
659
+ the target's own table (`cat.grandchildren` through a soft-deleted child), which the join
660
+ aliases and applies none to. `unscope(where: :col)` called on such an association drops the
661
+ column from the intermediate models' default scopes too in Active Record (a soft-deleted
662
+ intermediate record's targets come back); ES-NoProj read those records through their default
663
+ scopes, and refuses an unscope of a column one of them conditions (`UnsupportedQuery`).
664
+ - **A `:through` association reaches every candidate of each `has_one` on the way under
665
+ ES-NoProj, in the join's order.** Through another `has_one :through`, a `has_one :through`
666
+ followed only the target that association read (and was refused when that one was ambiguous);
667
+ through a `has_one` source with several records, or a `has_one` step on a model not read from
668
+ events (read with its `LIMIT 1`), it answered with the one record read. The candidates are now
669
+ all read, as the join reaches them. The targets were in the intermediate records' order, so an
670
+ order of the source's scope or the association's own came second (`has_one :latest_tag, through:
671
+ :first_item` answered the first item's latest tag, where the join orders by the tag's label
672
+ first and the item's position after). They are now ordered as the join orders them: the
673
+ target's default scope, the next intermediate model's default scope, the association's own
674
+ scope and the source's (an order on the intermediate model's columns included, as Solidus's
675
+ `Product#prices` has), then the intermediate step's scope; an `unscope(:order)` in a scope on
676
+ the target takes the default scopes' and the steps' orders off, as in the join. In no order at
677
+ all, the targets are in primary key order, as Active Record's `first` takes them. Where the
678
+ join's order cannot be reproduced (an order on another table, or a default scope ordering a
679
+ step past the first while an association scope orders too), a read that depends on the order
680
+ (loading, `first`, `last`, `take`, a `limit` or `offset`, `pluck` and `ids`, a `has_one
681
+ :through`'s answer) is refused (`UnsupportedQuery`) when there is more than one record;
682
+ `count`, `size`, `length`, `exists?` and sums answer, and so does a read after `reorder`. A
683
+ `has_one :through` answers with the first target in that order; in no order, with the one
684
+ target there is, and more than one is refused. `first`, `last` and `take` on a `has_many
685
+ :through` follow the same order; they took the first record by primary key.
686
+ - **A `has_many :through` that reaches a target through two intermediate records has it twice
687
+ under ES-NoProj**, as Active Record's join has (`author.article_authors` through two articles);
688
+ ES-NoProj kept one, and counted one. `-> { distinct }` keeps one, as before.
689
+ - **`update_all` on a `has_many` under ES-NoProj** (`author.articles.update_all(author_id: nil)`,
690
+ a `has_many :through` too) writes the records the association selects, as Active Record's runs
691
+ through the association's scope, recording an event for each, and returns how many; it ran
692
+ against the table and was refused. A record a `has_many :through` reaches twice is written once,
693
+ as SQL updates its row once (also for `delete_all`, `touch_all` and `destroy_all` on such a
694
+ relation).
695
+ - **Inside a transaction that recorded rights, a restriction committed before it began is in
696
+ force.** A write of rights drops its subject from the store's shared snapshot until the next
697
+ read, and a connection with rights of its own in its open transaction never read the snapshot
698
+ again, so a subject restricted just before such a transaction was not restricted inside it
699
+ (`active_for`, `active`, `keys_in_force`). Such a connection now reads the rights of every other
700
+ subject into a snapshot of its own, at most once per refresh interval and checking scope, and its
701
+ own subjects as its transaction left them; the shared snapshot is never written from it (under
702
+ `REPEATABLE READ` it reads the database as its transaction began). Outside such a transaction
703
+ nothing changes, and no query is added. `keys_in_force` inside a transaction that lifted a
704
+ restriction no longer lists its subject, as `active` does not.
705
+ - **ES-NoProj answers a `has_one :through` by following the through association.** The lookup
706
+ used the source's foreign key on the target: Solidus's `InventoryUnit has_one :order, through:
707
+ :shipment` was asked as `Spree::Order.find_by(order_id: <the unit's id>)`, a column only
708
+ `spree_shipments` has. 0.9.x returned `nil` for such an association under ES-NoProj, so
709
+ `carton.orders` was empty and Solidus sent no shipped email. 1.0.0 reads the intermediate record
710
+ and then the source association, each as that association reads under ES-NoProj, and narrows
711
+ the result by the association's own scope (a scope it cannot evaluate raises
712
+ `UnsupportedQuery`). A `has_many :through` whose source is a `has_one :through`
713
+ (`carton.orders`) follows it too. With `source_type:` (a polymorphic source), a `has_one` or
714
+ `has_many :through` follows only the intermediate records whose type column names that class,
715
+ as Active Record's join does; a `has_many :through` returned records of whatever class the
716
+ column named.
717
+ - **ES-NoProj let Rails' uniqueness validation pass every duplicate.** `validates :x, uniqueness:
718
+ true` asks the table whether another row holds the value; in ES-NoProj a monitored model's table
719
+ is empty, so every value passed, and the table's unique index could not catch the duplicate
720
+ either, since no row is written. The testbed saved a second bank-transfer payment with an RF code
721
+ already issued (`BankPayment#remittance_info`) and a second web invoice with a series and number
722
+ already used (`Finance::WebInvoice`). The relation Rails builds for the check is now answered
723
+ from the records rebuilt from events (`Lyra::Projections::Uniqueness`), as Active Record and the
724
+ database answer it from the table: Rails still builds it, so `scope:` (columns and
725
+ associations), `conditions:` (hash conditions, `where.not`, ranges, `or`, named scopes, a lambda
726
+ taking the record), `case_sensitive:`, `allow_nil`/`allow_blank`, the record itself left out on
727
+ update by its primary key (a custom one too), `Model.unscoped` (a soft-delete default scope is
728
+ not applied, as in Rails) and, under STI, the type condition of the class that declared the
729
+ validation all follow Rails' own construction. Values compare through the attribute's type (a
730
+ decimal `"250.00"` is `250`, an enum's label, a datetime in another zone), with SQL's NULL
731
+ semantics; `LOWER(column) = LOWER(value)` and a `citext` column's equality are worked out by the
732
+ database's own `LOWER` (texts it lowered once are remembered). A form that cannot be evaluated exactly (an SQL fragment
733
+ or a subquery in `conditions:`, a join, `LIKE`, a string column with a collation of its own)
734
+ raises `UnsupportedQuery` instead of passing or refusing the value on a guess. The check is
735
+ still exempt from the purpose check. Other modes keep Active Record's relation and the table's
736
+ answer. A uniqueness check is a read and then a write in every mode; where the table modes
737
+ have the unique index to refuse the second of two concurrent writers, ES-NoProj now takes a
738
+ transaction-level advisory lock on the model, attribute and value before the check reads
739
+ (PostgreSQL; under `case_sensitive: false` or on `citext`, on `LOWER(value)` as the database
740
+ computes it), so the second writer's check waits for the first's commit and then finds its
741
+ record. It is taken only inside a transaction (`save`, `create`, `update`) and only by a save
742
+ that claims a value (a new record, a changed attribute or scope column, or a check with
743
+ `conditions:`): a save that keeps its value does not wait for, or deadlock with, other writers.
744
+ It closes the race under READ COMMITTED, not for a REPEATABLE READ transaction whose snapshot
745
+ predates the other's commit. Each check reads every record of the model; a per-process copy
746
+ stamped as the projection cache's entries are reads only the streams written since the last
747
+ check (24 ms instead of 94 ms at 3,000 records in the test environment). A check answers from
748
+ the entries it read itself, so another thread replacing one meanwhile cannot show it that
749
+ thread's uncommitted state; the copies are kept under 32 MB (models with more than 20,000
750
+ streams are read through the projection cache), expire after an hour as the cache's memory
751
+ entries do, and are rebuilt when a model's columns change.
752
+ - **An erasure in another process did not reach warm ES-NoProj reads.** `Lyra::Erasure` rewrites
753
+ the record's events in place and appends an `ErasureApplied`. The eraser drops its own cache
754
+ entries; another process brought its entry up to date by replaying only the events after the
755
+ entry's stamp, the `ErasureApplied`, which changes no attribute, and kept the erased values:
756
+ `find_by`, `where` and the uniqueness check answered with the erased e-mail until the entry
757
+ expired. An entry with an `ErasureApplied` after its stamp is now rebuilt from the rewritten
758
+ stream.
759
+ - **ES-NoProj's lists, `character(n)` columns and serialized attributes compared as the table does
760
+ not.** A hash condition's list of two or more values is an `IN` of the values the type binds:
761
+ Active Record leaves out one it binds as NULL (`"abc"` on an integer, an enum's unknown label,
762
+ `""` on a boolean), and ES-NoProj kept it as NULL, so `where.not(qty: [9, "abc"])` returned no
763
+ row instead of every row but 9. An enum's unknown label alone is bound as `IS NULL`, as Active
764
+ Record binds it; an integer the enum has no label for (`where(status: 5)`) is bound as `= 5`,
765
+ which no row equals, NULL or not, and stays a member of a list (`where.not(status: [5, 7])`
766
+ returned no row). A `character(n)` value compares without its trailing spaces (`'ab'` is `'ab '`
767
+ in PostgreSQL). A serialized attribute compares as the text it is stored as
768
+ (`{"a"=>1,"b"=>2}` is not `{"b"=>2,"a"=>1}` there), and a Hash given for one is its value, not a
769
+ condition on a table named after it (`where(prefs: {...})` was refused). Hash conditions in
770
+ `where`, `where.not`, `find_by`, `exists?` and `count`, and the uniqueness check, all follow.
771
+ - **ES-NoProj compared a condition's value as given, not cast through the column's type.**
772
+ Active Record binds a condition's value through the attribute's type before the database
773
+ compares it; ES-NoProj compared the Ruby values. `find_by(amount: 250.0)` on a decimal column
774
+ compared the event's `"250.0"` with the Float and found nothing, with a BigDecimal too (the
775
+ testbed created a second bank-transfer payment code instead of reusing the first), and
776
+ `where(transaction_type: :card)` matched no row, as did a Symbol on any string column or enum.
777
+ Conditions in `where`, `where.not`, `find_by`, `exists?`, `count` and the joined conditions now
778
+ bind the value as Active Record binds it (the attribute's type serializes it) and read it back as
779
+ a record holds it, in equality, `IN` lists and ranges (each end): a decimal, float or integer
780
+ given as a String, Float or BigDecimal (`3.5` on an integer is 3, `250.004` on a decimal(10, 2)
781
+ is 250.00); a date given as a String or Time; a `datetime`, `timestamptz` or `time` given as a
782
+ String or Date, read in the database's clock (UTC) as Active Record binds it, not in
783
+ `Time.zone`, and a `time` compared as its time of day in UTC; a boolean given as `"t"`, `1`,
784
+ `"0"`; an enum, integer- or string-backed, given as its label, a Symbol or its column's value
785
+ (`where(status: "0")`); a deterministically encrypted attribute as its ciphertext compares
786
+ (`downcase: true` included); an alias for its attribute. A value the type binds as NULL (`"abc"`
787
+ or `""` on an integer column, `""` on a boolean) matches nothing, a value out of an integer
788
+ column's range matches nothing (and leaves a range end open), and NULL is unknown as in SQL:
789
+ `where.not(tag: "x")` and `where.not(amount: 250)` no longer return the rows whose column is
790
+ NULL. An equality on a non-deterministically encrypted attribute matches no row, as on the
791
+ table, and a range, a pattern or an SQL fragment on one is refused (`UnsupportedQuery`). SQL
792
+ fragments (`where("at >= ?", date)`) bind as Active Record quotes: a date or a string on a
793
+ datetime in the database's clock (the testbed's refund list filtered `requested_at` by a date
794
+ in `Time.zone` "Athens"), a number not rounded to the column's scale, and a value the database
795
+ rejects (`"qty >= ?", 2.5` on an integer column) refused. A condition on a name that is not an
796
+ attribute (`where(id: …)` where the primary key is `code`) is refused, as the table rejects
797
+ it; it matched nothing. An order puts `false` before `true` (they compared as equal) and orders
798
+ an enum by the value its column holds, not by its label. A record rebuilt from events also read
799
+ an integer-backed enum wrong: the event holds the label, which was read as the column's
800
+ integer, so every record had the first label (`0`); and a serialized attribute (`serialize
801
+ :prefs, coder: JSON`) failed to load. Both are now turned into the column's value, as a
802
+ projection writing the row does, once when the record's cache entry is built (the entries'
803
+ format is new: `CachedProjection::CACHE_VERSION` 3), with an STI subclass's own enums and
804
+ serialized attributes, and also in `Lyra::Temporal` reads and the rights' subject lookup. A
805
+ record that cannot be built from its events raises; it was left out of the answer. Strings are
806
+ still ordered by code point, not by the database's collation.
807
+ - **Retention did nothing in ES-NoProj.** `Lyra::Retention` (and `lyra:retention:apply`) listed
808
+ the model's ids from its streams but loaded each record from its table, which ES-NoProj leaves
809
+ empty or stale: no record was found, a dry run listed nothing and no rule was applied, and
810
+ `:hard_delete` and `:soft_delete` would have failed reloading the record from its table. In
811
+ ES-NoProj retention now finds the records, their anchor column (`created_at` or
812
+ `config.retention_anchors`) included, in the event store, with no default scope applied (a
813
+ soft-deleted record is still due), and acts on them there: `:hard_delete` destroys the record
814
+ rebuilt from its events (a destroy event) and `:soft_delete` updates it (an update event). The
815
+ other modes are unchanged.
816
+ - **The Art. 15 copy read a column an association owns as a link to another model too.** The
817
+ `<type>_id` convention of `Lyra::Privacy::SubjectLinks` also read `owner_id` of
818
+ `belongs_to :owner, class_name: "Person"` as a link to a model named `Owner`, and a polymorphic
819
+ `owner_id` as one whatever `owner_type` said, so the export for `Owner` 13 included the note
820
+ of `Person` 13. A column an association owns now links only to that association's class (a
821
+ polymorphic one's, only with its type column naming the subject's class), and the convention
822
+ reads only columns no association owns, for a subject type that is a monitored model or names
823
+ no model at all.
824
+ - **ES-NoProj reads apply the model's default scope.** Active Record builds every read of a
825
+ table from the model's `default_scope`; ES-NoProj reads, rebuilt from the event log, left it
826
+ out, so soft-deleted records (`default_scope { where(deleted_at: nil) }`, as paranoia, discard
827
+ and Solidus's `Spree::SoftDeletable` use) were found, counted and returned by associations.
828
+ `find`, `find_by`, `find_by!`, `exists?`, `where`, `all` (and so `count`, `pluck` and the other
829
+ calculations), `first`, `last` and named scopes now read through the default scope, as do
830
+ `belongs_to` and `has_one` readers and `has_many` readers, counts, sizes and collection queries
831
+ (through the target's default scope less what the association's scope unscopes, as Active
832
+ Record's association scope). A default `order` orders the results; on a `CachedRelation` an
833
+ `order` now adds to the existing order and `reorder` replaces it, as in Active Record, and
834
+ `first`/`last` follow the relation's order (`last(n)` returns the last n in that order, no
835
+ longer reversed). `Model.unscoped` read the table, which ES-NoProj leaves empty or stale; in
836
+ application code it now returns every record from the event store, and reads inside
837
+ `Model.unscoped { ... }` (and `reload`) leave the default scope out. Active Record's internals
838
+ and Lyra's own code keep Active Record's relation from `Model.unscoped`, also through a
839
+ model's own `def self.unscoped = super`. Scopes that take the default scope off
840
+ (`with_deleted`, `with_discarded`, `unscope(where: :deleted_at)`) and `rewhere` work after
841
+ `where`, `order` and named scopes, on the model and on an association, as on Active Record
842
+ (the relation replays its conditions without what is taken off); **after a `limit`,
843
+ `where.not` or an SQL fragment they raise `UnsupportedQuery`**, since the records they would
844
+ bring back are no longer in the relation. **A default scope ES-NoProj cannot evaluate (an SQL
845
+ string, a join, an ordering by an expression) now raises `UnsupportedQuery` on every read of
846
+ the model**, rather than answer with the records it leaves out, and
847
+ `Lyra::ModeTransition.to!` refuses the switch into ES-NoProj while a monitored model has one
848
+ (`force: true` overrides). An association's scope is evaluated on each read, so one that
849
+ decides at run time (`-> { Current.admin? ? with_deleted : all }`) is honoured. A find by
850
+ primary key reads one record even under an ordered default scope. Models without a default
851
+ scope read as before.
852
+ - **ES-NoProj relations no longer act on records their query did not select.** A
853
+ `CachedRelation` answers exactly or raises `UnsupportedQuery` before anything is written.
854
+ `merge` and `merge!` returned the relation unchanged, so `Model.unscoped.merge(Model.where(...))
855
+ .delete_all` deleted every record; they now evaluate the other relation (its conditions,
856
+ order and limit, or the records of an event-store relation) and refuse another model's.
857
+ `where!` with anything but hash conditions kept every record and now
858
+ raises; `exists?(["title = ?", x])` ignored its condition and now evaluates it as `where`
859
+ does (or refuses it). Relation methods Lyra did not implement ran on a bare relation of the model, past the
860
+ relation's conditions: `update` and `update_counters` changed every record, `first_or_create!`
861
+ and `create_or_find_by!` created records without the conditions, `create_with` was dropped and
862
+ `to_sql` raised `NoMethodError`. `update`, `update!`, `update_counters` (with `touch:`),
863
+ `first_or_initialize`, `first_or_create(!)`, `create_or_find_by(!)` (a record is found instead
864
+ when a unique index of the table would refuse it, as Active Record finds it on the violation)
865
+ and `create_with` now work on the relation's records and conditions, and `sole` and
866
+ `find_sole_by` raise `RecordNotFound` or `SoleRecordExceeded` as Active Record does; `lock`, `annotate`,
867
+ `optimizer_hints` and `strict_loading` are accepted and do nothing (writes take Lyra's
868
+ aggregate lock); `to_sql` and every other Active Record relation method raise
869
+ `UnsupportedQuery` before running. A model's own class methods called on a relation run with
870
+ it as the current scope, as Active Record runs them: their reads and writes act on its
871
+ records.
872
+ - **PAM DSL prints a retention period given in plain seconds correctly.** With ActiveSupport
873
+ loaded, as in a Lyra application, an Integer number of seconds (`keep_for 86400`,
874
+ `default 7 * 365 * 86400`, or an Integer from configuration) was read as years: 86400 printed
875
+ "0 years" in the Article 30 register, the policy summary and the retention check, and the
876
+ policy comparator printed 365 days of seconds as "365 days" and seven years of seconds as
877
+ "6 years". Plain seconds are now worded the same with and without ActiveSupport, with the
878
+ polyfill's fixed-length units (1 month = 30 days, 1 year = 365 days), rounded down to the
879
+ largest whole unit: 3600 is "1 hour", 86400 "1 day", 30 days "1 month", 365 days "1 year".
880
+ `ActiveSupport::Duration` values (`1.day`, `7.years`) print as before. Retention enforcement
881
+ was not affected: it already read plain seconds as seconds in both modes.
882
+ - **Lyra's GDPR retention check dropped a policy's retention given in plain seconds** and
883
+ applied the configured or default period (7 years) instead; it now keeps a plain number of
884
+ seconds as it keeps an `ActiveSupport::Duration`.
885
+ - **An erased record's row kept personal values inside `json`, `jsonb` and array columns that
886
+ its events had lost,** so it no longer replayed to its events. This happened with or without
887
+ `everywhere`. The erasure rewrote the events recursively but replaced only top-level string
888
+ columns in the row. `Lyra::Erasure` now scrubs every column of the row with the rule its events
889
+ get: the record's own row with the key and value rule, another record's row reached by
890
+ `everywhere` with the value rule only, so a third party's email under an `email` key there is
891
+ kept. `json`, `jsonb` and array columns and serialized attributes are recursed into and written
892
+ back whole in the same `UPDATE`. A `text` column holding JSON is still compared as a string, as
893
+ in its events. A column declared personal that holds a hash or an array is replaced whole, in
894
+ the row and in the events alike (each side of a change too): `nil`, or, when it is NOT NULL,
895
+ its own default as the database stores it (`[]`, `{}`); before, the events kept it, or got its
896
+ elements replaced one by one. Inside a hash, the strings that are identifying for the field's type (an
897
+ email for `:email`, a phone number for `:phone`) count as the person's values, and those under a
898
+ nested personal field; their copies elsewhere in the record, and with `everywhere` in other
899
+ records, are erased too. Labels such as `"home"` are not. An earlier erasure's `"erased:<id>"`
900
+ is no one's value, so a second erasure neither reports nor searches for it. The row is scrubbed by
901
+ the attribute's type, as its events are, so `hstore` columns and attributes cast as json are
902
+ recursed into too. An
903
+ encrypted attribute is compared in clear in its events, as in its row, and a replacement is
904
+ encrypted again; before, its ciphertext stayed in the events and replayed back to the value.
905
+ The erasure moves the row's optimistic-lock column, as any write does, so a copy loaded before
906
+ it fails with `StaleObjectError` instead of saving the erased values back; the move is recorded
907
+ as an ordinary update of that column alone (metadata `source: "lyra_erasure"`), so the row still
908
+ equals its replay. Before, the lock moved in the row only, and the row no longer replayed to its
909
+ events. Virtual attributes are no longer written, which made the erasure fail. An
910
+ event that is not a CRUD event (an application's own, in the record's stream) gets the key rule
911
+ at its top level, so a personal field's hash there is replaced whole. With `everywhere`, a
912
+ record is listed in `result.copies` only if its row no longer holds the value anywhere. A value
913
+ held as a JSON key, or inside a longer string, is not erased, and the record is listed in the
914
+ new `result.left_behind` as `{ record:, columns: }`, never with the value; `lyra:erase` prints
915
+ it. Before, such a record was listed as erased. The erased record's own row is checked the same
916
+ way, with or without `everywhere`, and listed first. A primary or foreign key is never rewritten;
917
+ one that holds an erased value is reported in `left_behind` as `"Model <key withheld>"` (and
918
+ `result.key_withheld`), and no result, reason or task output prints it. With `everywhere`, each
919
+ record found is locked and its stream read again before it is erased, so a copy written during
920
+ the search is not missed; candidates are found without case, and a `citext` copy differing in
921
+ case is reported. The new `result.not_searched` names the encrypted columns of monitored
922
+ models, which a text search of the log cannot read. In ES-Async the lock moves from the version
923
+ the log last recorded.
924
+ - **`everywhere` erased a third party's own record when the person's data named them.** In the
925
+ person's own record a value under a personal field's name is the person's (the key rule), so a
926
+ manager's email nested inside the person's JSON (`{"manager" => {"email" => …}}`), or a friend's
927
+ email at the top of another event in the person's stream (an invitation), was erased there and
928
+ then searched for in other records, the third party's own included. Now only the person's own
929
+ values drive the search: those of the row's personal fields and at the top of a personal field in
930
+ the record's own CRUD events (an attribute, either side of a change, a payload entry), with the
931
+ strings inside a structure declared personal taken by its type as before. The others are still
932
+ erased in the person's record, as its events are, and no longer anywhere else.
933
+ - **`everywhere` missed rows that held a value their events did not.** The search reached rows
934
+ only through events, so a row given the value by raw SQL, written before monitoring, or changed
935
+ by a write Lyra could not intercept kept it, and nothing reported it; and a record reached
936
+ through an event that mentioned the value only inside a longer string kept a whole copy in its
937
+ row. With `everywhere`, the tables of monitored models with a policy are now searched directly,
938
+ one query per table that returns keys only (string and text columns by equality, `json`/`jsonb`
939
+ as text by a `LIKE` on the value as a JSON string, string arrays with `= ANY`, `hstore` values);
940
+ not in ES-NoProj, where the tables are absent or stale. Every record reached has its row scrubbed
941
+ by the value rule, whether or not its events held the value whole. When only the row held it,
942
+ its `ErasureApplied` says `row_only: true` and it is listed in the new
943
+ `result.rows_without_events` (its row already differed from its replay: `lyra:repair` in Monitor
944
+ or Disabled, a rebuild of its table in the other modes; the task says which). A row with no
945
+ stream is erased first and then imported the Genesis way, so its `Imported` event holds the
946
+ replacement, never the value (an `ErasureApplied` alone would have started a stream Genesis never
947
+ imports); if a Genesis run imported it first, that stream is rewritten instead. A key column is
948
+ still never rewritten, and a row holding a value only as a JSON key or in another case in a
949
+ `citext` column is reported in `left_behind`. The log is searched once for all the values (it
950
+ was searched again per value to count holders), and candidate events are read in batches.
951
+ - **`everywhere`'s threshold counted streams, not people's records.** A value was taken as shared
952
+ when more than `max_copies` other streams held it, so one person's many records (and a record's
953
+ reference links) counted against it. Holders are now counted by record, through events (each
954
+ confirmed by an exact match) and rows alike, and the person's linked records, found by the rule
955
+ of the subject-access copy (`Lyra::Privacy::SubjectLinks`: any `belongs_to` to the erased
956
+ record's model naming it, by id or by its `primary_key:` column, a polymorphic one, a
957
+ `<model>_id` column), do not count. `max_copies:` must be a positive Integer (`ArgumentError`
958
+ before anything is written; a String failed half-way before).
959
+ - **`everywhere` rewrote Lyra's own record streams.** An event in a stream that was not a monitored
960
+ record's was rewritten when it held an erased value, so a consent recorded under an email as its
961
+ subject key (`Lyra::Consent$<email>`) lost it, and the consent store, rebuilt from those events,
962
+ lost the consent. Restrictions and objections (`Lyra::Rights`), consent (`Lyra::Consent`), the
963
+ access log and its erasures and the subject-access exports, with their per-subject streams, are
964
+ now never rewritten by `everywhere`, nor counted as holders.
965
+ - **Genesis's lock was per class, not per table.** An import through an STI subclass did not wait
966
+ for one through its base class. The lock is now named after the table's stream owner (the STI
967
+ base class), which the erasure of a row with no stream also takes.
968
+ - **Genesis imported a value the model cannot represent as the model read it.** A row holding a
969
+ value outside a string or integer enum's mapping (`refund_requests.status = '2'`) was imported
970
+ with `status: nil`: the `Imported` event disagreed with its row, and the disagreement surfaced
971
+ later and elsewhere (ES-Lazy's catch-up failed with `PG::NotNullViolation`). Genesis now checks
972
+ every attribute of every row it imports: the value the model reads, written back by the model's
973
+ type, must be the value stored, compared as the column's own type reads both or as the database
974
+ would be sent them. `'1'` and `1`, time zones and timestamp precision, a time of day, a decimal's
975
+ trailing zeros, jsonb key order or a normalized attribute's untrimmed legacy value are no
976
+ difference; an enum value outside the mapping, a lossy attribute type (`attribute :n, :integer`
977
+ over `10.50`), an inet value's host bits (which Rails drops; in `inet[]` arrays too) and a value
978
+ out of the model's integer range are. Whatever the type, a stored value the model writes back
979
+ as NULL is lost, in a nullable column too: a JSON `null` in a json/jsonb column, an empty range,
980
+ a blank or `'null'` JSON text or an empty YAML hash in a serialized column. A serialized,
981
+ encrypted or JSON attribute's text is otherwise one of many encodings of its value and is not
982
+ compared (a value of another class is). Columns the model ignores are imported as stored.
983
+ - By default a model holding such rows is refused: every row is checked before anything is
984
+ written, and `Lyra::Genesis::Refused` lists model, id, attribute, stored value and value read
985
+ (at most `Genesis::LISTED`, 20, with the count; a value the privacy policy declares, or an
986
+ encrypted one, is not shown). Nothing of the model is imported; inside a caller's transaction
987
+ the import is a savepoint, so a caller that rescues the refusal commits nothing of it. A
988
+ refused first use is remembered for `Genesis::REFUSAL_RECHECK` (30) seconds from the scan
989
+ that found it, so repeated uses do not scan the table again; `lyra:genesis` always scans.
990
+ - With `config.genesis_unrepresentable = :skip` (or `unrepresentable: :skip`, or
991
+ `UNREPRESENTABLE=skip` for `lyra:genesis`) those rows are left without a stream, the rest is
992
+ imported, and the rows are listed in the result (`Lyra::Genesis.import`, new, returns a
993
+ `Result`; `import_all` still returns the count) and in the log. A write to a skipped row,
994
+ through callbacks or bypassing them (`update_columns`, `touch`, `update_all`, `touch_all`,
995
+ `delete_all`, `upsert_all`, ...), and an ES-NoProj `find` of it, import it once the model can
996
+ represent it and raise `Refused` (naming the rows) until then, so its stream never begins with
997
+ the write's event. A skipped row is therefore fixed in the database; the next use of it, or
998
+ `lyra:genesis`, imports it. A process knows only the rows it skipped itself. `dependent:
999
+ :nullify` over a skipped child refuses the parent's destroy where events are required; in
1000
+ Monitor that child's event is left out (logged) and the other children still get theirs.
1001
+ - A write that bypasses callbacks now also runs the model's first use, as a write through
1002
+ callbacks does: on a model not yet imported in the process, `update_all` and the like import
1003
+ the rows that predate Lyra before recording their events.
1004
+ - A gated switch into Hijack or event sourcing is refused with such rows as discrepancies,
1005
+ whatever the setting (`Report#unrepresentable` counts them); `lyra:genesis` lists a refused
1006
+ model's rows and exits 1. `Genesis.import_one` (an erasure of a row with no stream) logs such
1007
+ a value and imports the row: the erasure does not fail on it.
1008
+ - `Lyra::Repair` wrote what the model read too (an `Imported` event for a row without events,
1009
+ an `Updated` event for the changed columns). It now applies the same check
1010
+ (`Genesis.unrepresentable`) under the same setting: by default `Lyra::Repair::Unrepresentable`
1011
+ (a `Repair::Refused`, with the listing) is raised before anything is written; with `:skip`
1012
+ (`Repair.run(unrepresentable: :skip)`, `UNREPRESENTABLE=skip` for `lyra:repair`) those
1013
+ records stay out of line, the rest is repaired, and `Repair::Result` (`skipped`,
1014
+ `unrepresentable`) and `lyra:repair` list them; a dry run lists them.
1015
+ - The error of work run on a connection of its own (`Lyra::OwnConnection`) is raised in the
1016
+ caller only, no longer also printed by its thread.
1017
+ - **An erasure could miss an update that committed while it ran.** It read the record's stream
1018
+ before its transaction and took the row lock only when it wrote the row, after rewriting the
1019
+ events, so a Monitor update that committed in between appended an event the erasure never saw,
1020
+ and that event kept the value. `Lyra::Erasure.erase!` now takes the row lock, then the record's
1021
+ advisory lock (the locks a writer holds, in the order writers take them), and only then reads
1022
+ the stream, all in one transaction; everything derived from the stream is computed under the
1023
+ locks, so a refusal there rolls back with nothing changed. A write already in flight finishes
1024
+ first; one that starts later waits and is new data. In ES-NoProj, with no row, only the advisory
1025
+ lock is taken.
1026
+ - **An access-log entry was lost when the reader's transaction rolled back,** though the read had
1027
+ already returned its value. `Lyra::AccessLog` now appends on a connection of its own
1028
+ (`Lyra::OwnConnection`), so the entry survives the rollback; an access that cannot be recorded
1029
+ still does not go ahead. Inside a transaction this costs a second connection per access, so size
1030
+ the pool for it (twice the threads avoids any wait): when no connection comes free within the
1031
+ pool's checkout timeout, or the connection is pinned (Rails' transactional tests), the entry is
1032
+ written inline, in the reader's transaction, and lost if that rolls back.
1033
+ - **PAM DSL works without ActiveSupport again.** Loaded on its own, with its standard-library
1034
+ polyfill, the Article 30 register raised `NoMethodError` (`String#presence` is ActiveSupport),
1035
+ and JSON export (`Reporter#export_json`, `GDPRCompliance#portable_export`) relied on
1036
+ ActiveSupport having loaded `json`. The polyfill now provides `presence` and treats a
1037
+ whitespace-only string as blank, as ActiveSupport does, and both files require `json`. Behaviour
1038
+ under ActiveSupport is unchanged. `rake test:all` now runs the PAM DSL suite twice, with
1039
+ ActiveSupport and with the polyfill (`rake test:both`), so CI covers both.
1040
+ - **A model whose single primary key is not `id`** (`self.primary_key = "code"`, a string or
1041
+ uuid column, no `id` column) is supported in every mode. `Lyra::DualView` looked the row up
1042
+ with `find_by(id:)`, so `DualView#compare`, `compare_all`, `find_discrepancies`,
1043
+ `Lyra::ModeTransition.discrepancy`, the mode gate's check and `lyra:repair` raised on such a
1044
+ model; sync projection and rebuild updated and deleted rows `where(id:)`. Hijack and event
1045
+ sourcing ignored the key the application gave, reserved another (a uuid), and put an `id`
1046
+ attribute the table does not have into the `Created` event. ES-NoProj's and `as_of`'s
1047
+ reconstructed records and the dashboard's record pages also used `id`. All now use the
1048
+ model's primary key. A composite primary key is still not supported, and
1049
+ is now refused clearly: `monitor_with_lyra` and `config.models` raise `ArgumentError` naming
1050
+ the key, and `verify_mapping!` reports it.
1051
+ - **ES-NoProj: `has_many ... dependent: :delete_all` left the children alive.** Rails deletes
1052
+ them through the association's table scope, which matches no row in the events-only store, so
1053
+ no `Destroyed` event was written and the children were still read back after their parent was
1054
+ destroyed. A `dependent: :nullify` found its children by foreign key alone, so it also cleared
1055
+ those outside a scoped association (`has_many :x, -> { where(...) }`), and a polymorphic one
1056
+ (`as:`) wrote no events at all. Both now find the children the association selects in the event
1057
+ store, as Active Record's association scope does: the owner's records, narrowed by the child's
1058
+ default scopes (less what the association's scope unscopes, so a soft-deleted child is left
1059
+ alone) and by the association's scope. Each gets the event `delete_all` or `update_all` writes
1060
+ there (`delete_all`, `dependent_association` or `update_all`, as in Monitor); strict mode
1061
+ refuses `delete_all` as it does in Monitor. **A dependent association whose scope, or whose
1062
+ child's default scope, cannot be evaluated against the event store (an SQL string scope such
1063
+ as `-> { where("status = 'x'") }`) now raises `UnsupportedQuery` and the parent's destroy is
1064
+ refused**, rather than delete or nullify children the association might not select. The same
1065
+ applies to `delete_all`, `delete(record)` and assigning a new set on such an association;
1066
+ `delete(record)` and assignment raised `TypeError` before (a condition on a record's query
1067
+ constraints, `where(["id"] => [[1]])`, which `CachedRelation` now evaluates). The owner's
1068
+ in-memory counter cache is no longer lowered (it went negative): ES-NoProj does not maintain
1069
+ counter caches. `has_one` `:delete` and `:nullify` were not affected. Monitor, Hijack and
1070
+ ES-Sync are unchanged.
1071
+ - **ES-NoProj answers `delete_by` and `destroy_by`,** as `where(...).delete_all` and
1072
+ `where(...).destroy_all`; they raised `UnsupportedQuery`. The partners of an in-memory join are
1073
+ read as Lyra's own read: under a purpose, a join on a model with a policy was refused for the
1074
+ partners' fields though no partner is returned; so are the children a `dependent: :nullify`
1075
+ rewrites, read for their events.
1076
+ - **Monitor appended `Updated` after `Destroyed`** when a stale copy was saved after another
1077
+ process had destroyed the record. Its UPDATE reached no row, or, when nothing had changed, no
1078
+ UPDATE ran; Active Record still ran `after_update`, and the stream became
1079
+ `[Created, Destroyed, Updated]`. Monitor now records no update when the UPDATE reached no row or
1080
+ nothing changed.
1081
+ - **Monitor appended a second `Destroyed`** when two stale copies of a record were both destroyed,
1082
+ when a copy was destroyed after `delete_all` or hand-written SQL had deleted its row, and when
1083
+ the same copy was destroyed twice. Without optimistic locking Active Record does not check what
1084
+ its DELETE reached, and it runs `after_destroy` even when the DELETE reached no row, or when it
1085
+ ran none for a record that is not persisted; the stream became `[Created, Destroyed, Destroyed]`,
1086
+ which no replay can apply. A destroy now records nothing unless it deleted the record's row, so
1087
+ a new record's `destroy` no longer appends an event without an id in any mode either. A row
1088
+ deleted by hand-written SQL keeps its stream open, as any write that bypasses Active Record does,
1089
+ until `lyra:repair` appends the `Destroyed`. Hijack and the event-sourcing modes refuse a stale
1090
+ destroy, as before; a soft delete (paranoia) and optimistic locking are unchanged.
1091
+ - **`belongs_to ... touch: true` cascades published nothing.** Rails touches the parent through
1092
+ `touch_later`, which sets the timestamps and clears their change before `touch` runs at commit,
1093
+ so Lyra saw no change. The parent's row moved and its stream did not; with timestamps compared,
1094
+ DualView flagged every touched parent and the mode gate refused. Lyra now keeps the row's value
1095
+ before the first deferred touch and records the change.
1096
+ - **An application's `metadata_proc` `source` was overwritten** by Lyra's (`lyra_interceptor`,
1097
+ `lyra_command_handler`). Lyra's value is now the default the application's keys override.
1098
+ - **ES-NoProj: `where(...).touch_all` touched every record** (an unscoped `touch_all` through the
1099
+ scope branch, with an event each) and then raised `UnsupportedQuery`. It now touches only the
1100
+ selected records, with one time for all, as Rails's `touch_all` does.
1101
+ - **The GDPR helpers lost a related record's updates and destroy** (Art. 15/17/20 reports):
1102
+ they matched the subject's foreign key in each event alone, and an update carries only its
1103
+ changes. Each record is now followed through the log, and an event is the subject's when the
1104
+ record pointed at them before or after it.
1105
+ - **A record created in Monitor lost `created_at` and `updated_at` when the log became
1106
+ authoritative.** Monitor's event left the two columns out of its `attributes` snapshot (its
1107
+ `changes` carried them), and replay and projection read a create's `attributes`. After a switch
1108
+ to event sourcing an ES-NoProj read raised `MissingAttributeError` on them, and a rebuild reset
1109
+ them to the rebuild time; DualView ignored the two columns, so it reported neither. `attributes`
1110
+ now holds every column, timestamps included, in Monitor events and in the events of bypass writes
1111
+ that create or delete records (`insert_all`, `delete_all` and the like), as Hijack and the
1112
+ event-sourcing modes already did. Logs written before this change are not supported (see
1113
+ Breaking).
1114
+ - **`ModelProjection.project_create` overwrote a string-keyed `created_at` or `updated_at` with
1115
+ the projection time.** It defaulted the two with symbol keys, and stringifying the keys let the
1116
+ default replace the value the event carried. A rebuild therefore reset the timestamps of every
1117
+ record whose event attributes had string keys, as Monitor's do.
1118
+ - **A destroyed record could be written again after events that are not replayed.** The check
1119
+ that refuses an update or destroy of a destroyed record (0.9.3) looked at the stream's last
1120
+ event only. A destroy followed by an event that is not replayed (the additional domain event of
1121
+ an `also:` rule on destroy, an `ErasureApplied` from erasing the destroyed record, a
1122
+ `has_and_belongs_to_many` link made with a stale copy and linked into its stream) let a stale
1123
+ copy update the record in ES-NoProj, ES-Async and ES-Lazy, appending after the destroy. The
1124
+ check now looks at the stream's last replayed event: the last one `Lyra::Event.operation_of`
1125
+ gives an operation.
1126
+ - **Strict data access tells Rails' association writes from the application's.** Any
1127
+ `update_all` setting one `*_id` column to nil was taken for a `dependent: :nullify` by its
1128
+ shape, and let through: `Article.where(...).update_all(author_id: nil)` cleared keys in strict
1129
+ mode. Rails' association code now marks the one relation or record it is about to write, just
1130
+ before the write (`Lyra::Associations::Maintenance`), and only that write is let through:
1131
+ a write the application makes meanwhile (in an `after_find` while the association loads its
1132
+ record, in a destroy callback) is its own, and refused. The marked writes are a `has_many`'s
1133
+ nullify (`dependent: :nullify`, and removing children with `delete`, `delete_all`, `clear` or
1134
+ by assigning a new set or ids), a `has_many :through`'s `dependent: :nullify` of its through
1135
+ records, a `has_one`'s `dependent: :nullify`, and the deletes of rows that link records:
1136
+ `has_and_belongs_to_many` join rows, the same rows through a model the application declared on
1137
+ the join table, and a `has_many :through`'s through records when records are removed from it.
1138
+ A `has_many`'s `dependent: :delete_all` of its children stays refused. The same writes made by
1139
+ the application are bulk writes like any other: refused in strict mode, and, where allowed,
1140
+ admitted by Genesis, checked against destroyed streams and the lock limit, and recorded as
1141
+ `update_all` or `delete_all` (an `update_all(fk: nil)` in ES-NoProj on the records the event
1142
+ store holds that the relation selects, as before). Several of Rails' own writes were refused
1143
+ in strict mode and now run: a polymorphic `dependent: :nullify` (two columns, `has_many` and
1144
+ `has_one`; a `has_many`'s is now recorded as `dependent_association`, with both columns'
1145
+ changes, where it was an `update_all`), a `has_many`'s removal of a child whose key is not named
1146
+ `*_id` (`foreign_key: :parent`), a `has_one`'s `dependent: :nullify` (its `update_columns`; the
1147
+ owner could not be destroyed), removing records from a `has_many :through` over a monitored
1148
+ model (its through records' `delete_all`, recorded as before), Rails' deletes through an
1149
+ application model on a join table, and, in ES-NoProj, a `has_many :through`'s
1150
+ `dependent: :nullify`, now recorded as `dependent_association` there as in the other modes. To
1151
+ mark the relation a `has_many :through` writes, Lyra reproduces Active Record's
1152
+ `HasManyThroughAssociation#delete_records`, checked identical in 8.0.5.1 and 8.1.4.1, and
1153
+ installs it for Rails 8.0 and 8.1 only; on another version Rails' own method runs with the
1154
+ through model marked for the whole of it (a write of that model made meanwhile is taken for
1155
+ Rails'), and a warning says so once.
1156
+ - **Deleting `has_and_belongs_to_many` join rows yourself is refused in strict mode.** The
1157
+ association's own deletes (`delete`, `delete_all`, `clear`, `destroy`, `destroy_all`,
1158
+ assigning records or ids, destroying the owner) are Rails' bookkeeping, allowed as before, and
1159
+ each records its `AssociationUnlinked`. A `delete_all` the application makes on the join model
1160
+ (`Item.const_get(:HABTM_Tags)`) was always allowed; it is now refused in strict mode when either
1161
+ side's model is monitored, as is one on a model the application declared on the join table
1162
+ (not monitored itself), which deleted links with no event at all and is now recorded as an
1163
+ unlink where it runs. Several monitored models' `has_and_belongs_to_many` can share a join
1164
+ table (both sides of one association, a self-referential one, two owners on columns of their
1165
+ own): a delete through the application's model of it, its own or Rails' (a `has_many :through`
1166
+ over it, its `dependent: :delete_all`), records on the stream of every monitored owner the
1167
+ deleted rows name the `AssociationUnlinked` Rails' own unlink from that side records, linked
1168
+ into the associated records' streams that have none of their own; a side whose model is not
1169
+ monitored records nothing. Links made and removed from a side Lyra does not monitor (its
1170
+ `has_and_belongs_to_many` to a monitored model) were not recorded at all: Rails' link and
1171
+ unlink, and the application's `delete_all` where allowed, are now recorded on the monitored
1172
+ records' streams (appended to the first, linked into the others), as a link made from their
1173
+ side is seen there.
1174
+ - **`insert_all` in ES-NoProj decides on primary keys as the event store has them.** The table
1175
+ decided: a record created through callbacks (no row there) was inserted again, a second
1176
+ `Created` on its live stream with its attributes overwritten, where Monitor skips the row
1177
+ (`insert_all!` raises `ActiveRecord::RecordNotUnique`); and a record inserted with `insert_all`
1178
+ and destroyed kept its row, so inserting its key again was skipped with no event, where
1179
+ Monitor creates the record again. The rows' keys are now locked and read from the event store:
1180
+ a record it holds is skipped (`insert_all`) or refused before anything is written
1181
+ (`insert_all!`, and `insert_all` whose `unique_by` is another index, whose `ON CONFLICT` does
1182
+ not cover the key), and the table's row of any other key, left by a destroyed record, is
1183
+ replaced (a row Genesis left without a stream is a record, and conflicts). A key whose record was destroyed is created again, `Created` after
1184
+ `Destroyed` as in Monitor, and the stream reads as the record created last (find, queries,
1185
+ `StreamTail`, the writes that follow). Rows that give no key take the table's sequence, as
1186
+ before.
1187
+ - **An ES-Async or ES-Lazy `upsert_all` no longer brings back a destroyed record whose row is
1188
+ still in the table.** Before the destroy's projection job ran (ES-Async), or before the next
1189
+ catch-up (ES-Lazy, a destroy committed after the one the write began with), `ON CONFLICT`
1190
+ updated the row and appended an `Updated` after the `Destroyed`: the record read as live in the
1191
+ log, the job then deleted its row, and `find` raised. The rows the upsert conflicts with now
1192
+ have their aggregate locks taken (all or none, within `bulk_write_lock_limit`) and their streams
1193
+ read. In ES-Async (see the next entry) the rows are then brought up to date with their streams,
1194
+ and a destroyed record is inserted again, as in Monitor. In ES-Lazy a destroyed one refuses
1195
+ the whole upsert with `Lyra::RecordDestroyedError` before anything is written, as an update of
1196
+ a destroyed record is refused: replaying the destroy onto the row there would be applied again
1197
+ by the next catch-up, which would delete the new row. Once the destroy is in the table the
1198
+ upsert inserts, as in Monitor. In ES-Lazy the locked rows are brought up to date with their
1199
+ streams first, so the events carry the values the stream has. ES-Sync, Hijack and Monitor
1200
+ delete the row with the destroy and are unchanged; ES-NoProj refuses `upsert_all` as before.
1201
+ - **ES-Async: `insert_all`, `insert_all!` and `upsert_all` conflict as Monitor's do while a
1202
+ projection job is pending.** The id of a record whose destroy was still waiting for its job met
1203
+ the dead row: `insert_all` skipped it (no event; the job then deleted the row), where Monitor
1204
+ creates the record again, and `upsert_all` was refused. The id of a record whose create was
1205
+ waiting had no row yet: the insert wrote one and logged a second `Created` on the live stream,
1206
+ where Monitor's `insert_all` skips it, `insert_all!` raises `ActiveRecord::RecordNotUnique` and
1207
+ `upsert_all` updates it. Deleting the dead row first was no answer: the destroy's job would then
1208
+ delete the new one. The records of the keys the rows give (and, for `upsert_all`, the rows its
1209
+ conflict key finds) are now locked (aggregate locks, within `bulk_write_lock_limit`) and their
1210
+ rows brought up to date with their streams, as their jobs would
1211
+ (`AsyncProjectionJob.project_pending`): under the job's own per-stream lock, the events after
1212
+ the stream's checkpoint are applied and the checkpoint moved past them, in the write's
1213
+ transaction. The pending jobs then apply only what is past the checkpoint, the write's own
1214
+ event at most, which is in the row already: a destroy is never applied to the row created
1215
+ again. A destroyed record is created again with the values given and the column defaults (not
1216
+ the dead row's), `Created` after `Destroyed`; a live one conflicts. `upsert_all` reads the rows
1217
+ its conflict key finds again once the locks are held, so a job that committed in between is
1218
+ seen. Rows already up to date cost one query. The write then moves the checkpoints of the
1219
+ records it brought up to date, and of those it created, to its own events (one statement):
1220
+ writing the same keys again finds nothing to project. Rows that give no key take the table's
1221
+ sequence; a conflict on another unique index is looked up in the event store (see the entry on
1222
+ ES-Sync, ES-Async and ES-Lazy below). A projection job no longer
1223
+ waits for its record's row while it holds the stream's lock: it takes the row with its lock
1224
+ (one statement more per job, no measurable time) and, when another transaction holds the row,
1225
+ lets the lock go and waits for the row first, so a transaction that holds the row and then
1226
+ inserts or upserts its key does not deadlock against the job.
1227
+ - **ES-NoProj: `insert_all` and `insert_all!` check the model's other unique indexes against the
1228
+ event store.** Only the primary key was; the other unique indexes were checked against the
1229
+ table, which holds no row of a record created through callbacks: a duplicate (an email taken,
1230
+ no id given) was inserted, a second record with the value, where Monitor's `insert_all` skips it
1231
+ and `insert_all!` raises; and the row a destroyed record left there skipped a value that is free.
1232
+ Each row's values for each unique index (not partial, not an expression; NULL conflicts only
1233
+ under NULLS NOT DISTINCT) are now locked as a uniqueness validation of the value locks them
1234
+ and compared with the records the event store holds that may hold them (found through the
1235
+ values' streams, next entry) and with the table's rows holding them: a conflict is skipped
1236
+ (`insert_all`, on any index or on its `unique_by`) or raises `ActiveRecord::RecordNotUnique`
1237
+ before anything is written (`insert_all!`, and a conflict on an index other than `unique_by`).
1238
+ The table's rows of destroyed records holding the values are deleted, and the row of a live
1239
+ record that holds a value its record gave up since (updated through callbacks) is brought up to
1240
+ the record's values. A row Genesis left without a stream (`genesis_unrepresentable :skip`)
1241
+ conflicts, by its key and by its values, as in Monitor; it was deleted as a destroyed record's
1242
+ row, by its key. The values' locks and the keys' count together against
1243
+ `bulk_write_lock_limit`, so a model with several unique indexes reaches it at fewer rows. A
1244
+ `create` or `update` through callbacks is checked against the same indexes (next entry).
1245
+ - **ES-NoProj: `create`, `update`, `update_columns`, `update_column` and `update_all` enforce the
1246
+ table's unique indexes.** ES-NoProj writes no row, so the index never saw these writes: a
1247
+ duplicate email was saved and recorded (unless a uniqueness validation caught it), where Monitor
1248
+ raises `ActiveRecord::RecordNotUnique` from the `INSERT` or `UPDATE`.
1249
+ - *Which indexes.* Unique, on columns, not partial, other than the primary key, and on no column
1250
+ with a nondeterministic collation (whose equality only the database knows; a uniqueness
1251
+ validation refuses such a column too): the list `insert_all` checks too
1252
+ (`Lyra::Projections::UniqueIndexes`). They are read from the database once per model and
1253
+ process, again after `reset_column_information` and every `config.unique_index_refresh`
1254
+ seconds (60), so an index a migration adds is enforced in a process started before it. An
1255
+ index is checked when the write sets one of its columns.
1256
+ - *When.* Where the table modes write the row: after the `before_*` callbacks and before the
1257
+ event is built, so a conflict raises `ActiveRecord::RecordNotUnique` (with PostgreSQL's
1258
+ message, naming the index) before any `after_*` callback and with nothing appended;
1259
+ `update_columns` after its destroyed-record check; `update_all` before any of its events, two
1260
+ of its records given one value conflicting with each other. An update claims the tuple the
1261
+ row would hold: the columns it does not write as the record's stream has them, read under the
1262
+ record's aggregate lock (a copy read before another writer moved the record to another shop
1263
+ claimed its old pair and saved a duplicate). NULL conflicts with nothing unless the index is
1264
+ `NULLS NOT DISTINCT`; values compare as the column holds them, a `citext` column's by the
1265
+ database's `LOWER`, as `citext` compares (Ruby's `downcase` disagreed on "İ"); a destroyed
1266
+ record holds no value, and a record keeps its own.
1267
+ - *Locks.* Each claimed value is locked with the key a uniqueness validation and `insert_all`
1268
+ take, so of two concurrent writers of one value the second waits and raises, as the index
1269
+ makes the second `INSERT` wait. An update, `update_columns` and `update_all` take the value
1270
+ locks after the records' aggregate locks, as Monitor's `UPDATE` takes the row and then the
1271
+ index entry, so they wait where Monitor waits and do not deadlock where it does not (taking
1272
+ them before deadlocked an `update_all` against a save of the same record, and two
1273
+ transactions crossing two records and one value). An update of a model with a uniqueness
1274
+ validation takes the record's aggregate lock before its validations (once, earlier), so the
1275
+ validation's value lock comes after it as well: a validated save and an `update_all` of the
1276
+ record and value, and two validated transactions crossing two records and one value, no
1277
+ longer deadlock. The lock key holds a keyed digest of the value, not the value, which reached
1278
+ the statement log (`hashtext('lyra_unique/User/email/ann@…')`); the `LOWER` of a `citext`
1279
+ value is asked for with the value bound. The links are caught up without waiting for another
1280
+ thread, the links' lock or a connection (a write that cannot have them checks against every
1281
+ record of the model, once), so a model's first use with every connection of the pool taken
1282
+ by its writers finishes. Without `config.value_digest_key` or `secret_key_base`, every environment but development and
1283
+ test raises `ValueDigest::KeyMissing` at first use. The unique indexes ES-NoProj leaves out
1284
+ (partial, expression, nondeterministic collation) are named in the log once per model.
1285
+ - *Lookup.* The value is looked up in a value stream (`"Model!column$<digest>"`, linked by
1286
+ `ReferenceLinks` for each event that sets an indexed column, in one `INSERT`): its cost is the
1287
+ value's history, not the model's size. The digest is an HMAC-SHA256 under a secret of the
1288
+ installation (`config.value_digest_key`, or derived from `secret_key_base`;
1289
+ `Lyra::Projections::ValueDigest`), so a dictionary does not reverse it; a value stream linked
1290
+ under another key is deleted and linked again on its next use. An erasure moves the links of
1291
+ the events it rewrites, so no digest of an erased value is kept. Each column's links have a
1292
+ checkpoint of their own: the first check of a model links only the columns behind (the
1293
+ foreign keys are not read again), before its value locks, in one read of the events; a column
1294
+ added to an index is linked from the first event. Processes that ran an earlier Lyra, or read
1295
+ the indexes before a migration added one, write without the column's links; once they are
1296
+ gone, `bin/rails lyra:link_values` links every value again (a process of this release that
1297
+ reads the new index links its own earlier writes itself).
1298
+ - *Cost.* On the development VM (PostgreSQL in Docker, 1,000 records, 200 interleaved writes) a
1299
+ create or an update that sets a fresh unique email costs three queries more (the lock, the
1300
+ stream read, the link's `INSERT`: 11 against 8 for a create, 12 against 9 for an update),
1301
+ about 90 more objects allocated (of some 2,500) and 2 to 3 ms (of 15 to 25, on a loaded VM);
1302
+ a model without such an index, and an update that sets none of its columns, are unchanged (the
1303
+ same queries, allocations within noise). On the testbed in ES-NoProj the benchmarked writes
1304
+ (`Registration` create, update and destroy: its unique indexes are all partial, so none is
1305
+ checked) are unchanged (12, 10 and 10.5 queries per write, as before), and a `BankPayment`
1306
+ create (a unique `remittance_info`, validated too) costs 13 queries against 10. ES-NoProj's
1307
+ `insert_all` reads the records the values' streams name instead of every record of the model
1308
+ (`Uniqueness::Snapshot`: 52.8 s at 10,000 records, cold), and `Model.insert_all` (and the
1309
+ other inserts on the model) no longer builds the relation of every record first (Active
1310
+ Record sends them through `Model.all`): a cold two-row `insert_all` at 3,000 records took
1311
+ 10.2 s, now 77 ms. With `config.reference_links = false` every record of the model is read,
1312
+ as a uniqueness validation reads them.
1313
+ - *`create_or_find_by`.* In ES-NoProj it looked for a conflict on an index made of its
1314
+ attributes alone, so a value its block set was created twice; it now creates in a savepoint
1315
+ and, when the index refuses the record, finds it (`RecordNotFound` when the attributes select
1316
+ none), and `find_or_create_by` creates through it, as Active Record 8.1 does.
1317
+ - *Also.* An `update_columns` on a copy read before another writer changed the value it sets
1318
+ back recorded nothing in ES-NoProj and left the record as it was (its old value was the
1319
+ copy's): it now takes the old values from the record's stream, as the table's `UPDATE` writes
1320
+ the row. ES-Sync, ES-Async and ES-Lazy: next entry.
1321
+ - **ES-Sync, ES-Async and ES-Lazy: a duplicate's event was appended before the projection met
1322
+ the unique index.** In these modes the event store is authoritative and the table a projection:
1323
+ a `create`, `save`, `update`, `update_columns` or `update_all` setting a value a unique index
1324
+ already held appended its event, then the projection's `INSERT` or `UPDATE` failed and was only
1325
+ logged (`strict_projections` off). The log held a record the table could not; in ES-Lazy the
1326
+ next read's catch-up then failed on it, and with it the reads of every model. Monitor raises
1327
+ `ActiveRecord::RecordNotUnique` and writes nothing. The writes are now checked before the event
1328
+ is built, as ES-NoProj's are (previous entry; `Lyra::Projections::UniqueIndexes`, the same
1329
+ indexes, claims, value locks and message, naming the mode), so a duplicate raises
1330
+ `ActiveRecord::RecordNotUnique` with no event appended, no projection job enqueued, no row
1331
+ written and nothing logged as a failed projection, and ES-Lazy's reads go on.
1332
+ - *Where the value is looked up.* ES-Sync: in its table, under the value's lock. Its projection
1333
+ writes the row in the writer's transaction, so once the lock is held the table has every
1334
+ committed claim and the transaction's own, compared by the database as its index compares
1335
+ them; one statement (the lock) and one `SELECT` per value claimed, and no links. ES-Async and
1336
+ ES-Lazy, whose tables lag the event store (a create whose job has not run, a write since the
1337
+ last read): in the value streams, as ES-NoProj, which are now linked in these modes too
1338
+ (`ReferenceLinks.values_enabled?`; the foreign-key links stay ES-NoProj's, and a model with
1339
+ no unique index links nothing). The first check of a model in a process links the events
1340
+ stored before (per column, once).
1341
+ - *Writes that skip callbacks.* `update_columns`, `update_column`, `update_all`, `insert_all`,
1342
+ `insert_all!` and `upsert_all` take the values' locks in these modes too, so a write through
1343
+ callbacks checking the table waits for them, as they wait for it (in ES-Sync
1344
+ `update_columns` locks its row first, `FOR NO KEY UPDATE`, and `update_all` its rows, as
1345
+ Monitor's `UPDATE` takes the row and then the index entry). ES-Sync's statement then checks
1346
+ the index itself. In ES-Async and ES-Lazy the values are first looked up in the event store,
1347
+ where a holder whose row is not projected yet was missed by the table: `update_columns` and
1348
+ `update_all` raise; `insert_all` leaves the row out (or raises, for an index other than its
1349
+ `unique_by`), `insert_all!` raises; `upsert_all` raises for a conflict on an index other than
1350
+ its conflict columns (the record it updates excepted), and on its conflict columns updates
1351
+ the holder, whose row ES-Async brings up to date first (it raises where that row is still not
1352
+ in the table). This was the known leftover of the ES-Async insert entry above.
1353
+ - *A holder destroyed or moved away, its row still in the table.* The value is free, as in
1354
+ Monitor, whose row is gone: a `create` (or any of the writes above) saves it. ES-Async brings
1355
+ such a row up to date with its stream in the write's transaction, as its job would, where that
1356
+ waits for nothing (the record's aggregate lock, its row and its stream's projection lock are
1357
+ only tried; a record one is taken for is left to its job), so neither the write's own `UPDATE`
1358
+ or `INSERT` nor its projection job meets the stale value. ES-Lazy's catch-up applies the
1359
+ destroy first (log order).
1360
+ - *Cost.* On the testbed (development VM, PostgreSQL in Docker, 100 writes each after warm-up),
1361
+ a model without a checked unique index and a write that sets none of its columns issue the
1362
+ same statements and locks as before in all three modes: the writes the benchmark campaign
1363
+ measures (`Registration` create, update and destroy, 8/8/13 statements in ES-Sync, 7/8/13 in
1364
+ ES-Async, 7/8/26.8 in ES-Lazy; its unique indexes are all partial, so none is checked; its
1365
+ `insert_all` and `update_all` likewise), and a `BankPayment` update of its amount (8). In
1366
+ ES-Async and ES-Lazy each append of such a model also looks the model up for its value
1367
+ columns, in memory (about 8 objects more per write; times within noise). A write that sets a
1368
+ checked value costs, per value: in ES-Sync the lock and a `SELECT` on the table (a
1369
+ `BankPayment` create, unique `remittance_info`, 11 statements against 9; an update of the
1370
+ code 11 against 9); in ES-Async and ES-Lazy the lock, the value stream's read and the link's
1371
+ `INSERT` (create 11 against 8 in ES-Async, 19 against 16 in ES-Lazy; update of the code 12
1372
+ against 9, 21 against 18), 1 to 3 ms, and a rebuild of the records the stream names when the
1373
+ value has a history. ES-Async's projection jobs are unchanged (10 to 12 statements).
1374
+ - *ES-Sync inside `ReadYourWrites.with_guaranteed_read`.* A write's row was deferred to the
1375
+ block's end, so the block's own reads missed it and so did the check of every write after it
1376
+ (which reads ES-Sync's table): two creates of one email in a block were both saved, the
1377
+ second's row dropped. ES-Sync now writes the row at once inside the block too, as outside it
1378
+ (and as `each_write: true` did); the block has nothing left to project there.
1379
+ - *A projected create that met another record's value inserted nothing, silently.* The
1380
+ projection's `INSERT ... ON CONFLICT DO NOTHING` skipped a row whose value another row held, as
1381
+ it skips a replay of the record's own row. In ES-Async, when the holder's row was held by
1382
+ another transaction (so the write left it to its job) and the new record's job ran before the
1383
+ holder's, the job completed and moved its checkpoint past the create: the log held a record
1384
+ the table never got, and nothing was logged. A create that inserts nothing although the table
1385
+ has no row with its key (looked up on the table's class: a record of an STI subclass whose
1386
+ type changed since keeps its row) now raises (one query, only then), after reading the rows
1387
+ that hold its values (a tuple with a NULL holds nothing, unless its index is `NULLS NOT
1388
+ DISTINCT`) and their records' streams. When each such record gave the value up and only its
1389
+ row lags, an ES-Async job brings those rows up to date itself, where nothing has to be waited
1390
+ for (so a holder whose own job was lost does not block it), and inserts again; a row still
1391
+ held by another transaction raises `Lyra::Projections::ProjectionBehind`, which the job
1392
+ retries at most every five minutes for about a day (`AsyncProjectionJob::BEHIND_ATTEMPTS`,
1393
+ 300; the general retry gave up after about seven minutes of a held row, and the record never
1394
+ reached the table), then reports (log, `projection_error_handler`) and gives up, leaving the
1395
+ record to a rebuild. `Lyra::Projections::DuplicateInLog` when a live record holds the value in
1396
+ the log too (two records of the log hold it), a row with no stream holds it, or no holder is
1397
+ found once the insert was tried again (an expression index, which Lyra does not read): logged
1398
+ once per record and process (debug after; 10,000 remembered) and passed to
1399
+ `projection_error_handler`, the job discarded. Both are `ActiveRecord::RecordNotUnique`s:
1400
+ ES-Sync's write fails with either, a rebuild stops at it. ES-Lazy's catch-up logs it (once)
1401
+ and goes on, since raising there would stop every read for a record the log holds for good.
1402
+ An update whose value another row holds raised already.
1403
+ - *ES-Sync: a database error in the projection was logged inside an aborted transaction.* With
1404
+ `strict_projections` off, the error the projection's `INSERT` or `UPDATE` raised was logged and
1405
+ the save reported success, but PostgreSQL had aborted the transaction, so its `COMMIT` rolled
1406
+ the write and its event back: a value committed after a `REPEATABLE READ` transaction's
1407
+ snapshot, which the check's `SELECT` cannot see, ended in a save reported as done with no
1408
+ event. A database error there now fails the write (`ActiveRecord::StatementInvalid` and its
1409
+ subclasses: here `RecordNotUnique` or `SerializationFailure`), whatever `strict_projections`
1410
+ says; other errors are logged as before.
1411
+ - *Not covered.* Transactions at `REPEATABLE READ` or `SERIALIZABLE`, in every event-sourcing
1412
+ mode (as for a uniqueness validation): the check reads the transaction's snapshot, which
1413
+ misses a value another transaction committed after it began. ES-Sync's write then fails at
1414
+ its projection (above), as Monitor's does; ES-Async and ES-Lazy save both records, and the
1415
+ second's projection fails with `DuplicateInLog` (logged once; ES-Async discards the job,
1416
+ ES-Lazy's catch-up goes on). Partial and expression indexes, and columns with a nondeterministic collation,
1417
+ are left to the projection, as in ES-NoProj (named in the log once per model).
1418
+ - **`insert_all` and `upsert_all` with `returning:` as raw SQL without the primary key recorded
1419
+ nothing.** `insert_all(rows, returning: Arel.sql("code"))` wrote the rows and published no
1420
+ event, in every mode: the keys could not be read back. The primary key is now returned with the
1421
+ caller's SQL under an alias of Lyra's own and taken out of the result the caller gets.
1422
+ - **The result of `insert_all`, `insert_all!` and `upsert_all` with `returning:` is Active
1423
+ Record's own.** Where Lyra added the primary key to `RETURNING` (a symbol, an array, raw SQL,
1424
+ `Arel.sql("*")`) it rebuilt the result without its column types: `column_types` came back empty
1425
+ and `cast_values` cast nothing; `returning: false` lost the affected rows. The column types (by
1426
+ name and by index) and the affected rows are kept.
1427
+ - **A model whose primary key has several columns** (one its table declares, enrolled before
1428
+ Active Record read it; Lyra refuses one it knows of) raised `PG::UndefinedColumn` on
1429
+ `insert_all` and `upsert_all` with `returning:` (the key was asked for as one quoted name) and,
1430
+ without it, wrote the rows with no event. In Monitor and Hijack each key column is now returned
1431
+ and taken out of the caller's result, and each row records its `Created`, the record's key
1432
+ being the array of its columns' values. The event-sourcing modes, whose projections, catch-ups
1433
+ and rebuilds cannot apply such an event (ES-Lazy's catch-up then failed on every read of every
1434
+ monitored model), refuse the three writes with an `ArgumentError` before anything is written,
1435
+ and Genesis refuses the model on its first use there (it failed with `PG::UndefinedColumn`).
1436
+ - **`has_and_belongs_to_many`: the application's `insert_all`, `insert_all!` and `upsert_all`
1437
+ of join rows were neither refused nor recorded.** Through Rails' join model
1438
+ (`Item::HABTM_Tags`) or a model of the application's on the join table (and `insert`, `insert!`,
1439
+ `upsert`, and an owner's `item_tags.insert_all(...)`), they are now refused in strict mode
1440
+ unless inside `Lyra.without_strict_access`, as its `delete_all` of join rows is, when either
1441
+ side is monitored, and where they run each row written (read back with `RETURNING`) is recorded
1442
+ as the `AssociationLinked` Rails' own link from each side records, on every monitored owner's
1443
+ stream the row names; a row `ON CONFLICT` skipped records nothing, and an upsert that moves a
1444
+ row to another pair unlinks the old pair. Rails' own links are inserted one row at a time
1445
+ through the join model's callbacks, as before.
1446
+ - **`has_and_belongs_to_many`: the join table of an owner Lyra does not monitor, whose monitored
1447
+ side has no association back onto it, was not recognised** for a model of the application's on
1448
+ that table: its deletes went unguarded and unrecorded, though Rails' own links and unlinks there
1449
+ are recorded on the monitored side's streams. Lyra now keeps every `has_and_belongs_to_many`
1450
+ declared (those of models loaded before its hook was installed included), and the table is a
1451
+ join table when one whose links it records names it. A
1452
+ `has_many :through` over a model Lyra does not monitor is not one: no link of it is recorded,
1453
+ from either side, as before.
1454
+ - **`has_and_belongs_to_many`: Rails' own unlink of a join row naming another association's
1455
+ owner too** (two owners' columns filled on one row of a shared table, which only the
1456
+ application's code writes) recorded the unlink of the side Rails deleted from only; the other
1457
+ owner's stream kept a link whose row was gone. It is now recorded on every monitored owner's
1458
+ stream the row names, as the application's delete of the row already was. Rows naming no other
1459
+ owner record exactly what they did.
1460
+ - **`config.bulk_write_lock_limit` takes `nil` or a whole number of at least 1.** 0 (or a
1461
+ negative number) refused every write that locks a record, a single `update_columns`, `touch` or
1462
+ `delete` included; any other value raises `ArgumentError` now, and a string is read as its
1463
+ number. `BulkWriteLockLimitError`'s message no longer reads "a update_columns of 1 ... records".
1464
+ - **A bulk write is no longer aborted for the deadlocks of the transactions it waits behind.**
1465
+ `Lyra::AdvisoryLock.xact_lock_all` gave back every lock it had taken whenever one was held, and
1466
+ waited for that one without a limit, in its queue; there it was the one PostgreSQL aborted when
1467
+ the transactions ahead of it deadlocked among themselves, and transactions that came and went with
1468
+ a few of its records kept it out. A bulk write of 8 records, for 15 s, against three transactions
1469
+ that each updated one record and then bulk-wrote two others: in ES-NoProj it went through 2 times
1470
+ in 21 attempts and was aborted 19 times (four runs), in ES-Async 50 times in 66 attempts, 16
1471
+ aborted (three runs). A bulk write that holds no advisory lock of its own yet now keeps the locks
1472
+ it took while it waits for the others, for at most 100 ms in all, and waits holding none for at
1473
+ most half of `deadlock_timeout`; then it gives them back and tries again. No wait of its lasts as
1474
+ long as `deadlock_timeout`, so PostgreSQL never reports it deadlocked, and a transaction that
1475
+ holds one of its records and asks for another it took goes on once it gives way. The same runs:
1476
+ ES-NoProj 7 of 8 attempts went through (the one aborted already held locks, in a later round of
1477
+ the write), ES-Async 42 of 42 (four runs each); deadlocks in all, the three transactions' among
1478
+ themselves included, 47 against 56 in ES-NoProj, and in ES-Async 29 in four runs against 43 in
1479
+ three (the transactions' own: 46 against 37, and 29 against 27). A bulk write of 6 records, beside
1480
+ one of 4 and a transaction like those, went through 62 and 88 times (two runs) instead of 24 to 30
1481
+ in ES-Async, and 60 instead of 12 in ES-NoProj (the one of 4 records: 130 and 135 instead of 152
1482
+ to 160, and 121 as before), with no deadlock before or after; the runs took longer (ES-Async 42 s
1483
+ against 22 to 25, ES-NoProj 415 s against 236). It is no cure for starvation: in ES-NoProj a write
1484
+ still waited 2 to 28 s, and in ES-Async its turns cost the three transactions about a third of
1485
+ their throughput. A transaction that already holds locks of its own (it wrote a record, then
1486
+ bulk-writes others) waits as before, holding none of the new ones: keeping them, or taking all in
1487
+ sorted order and waiting, as Monitor's row locks do, made such a transaction deadlock nearly every
1488
+ time (119 of 120 in that ES-Async run, against none). The lock limit is still checked before any
1489
+ lock is taken, nothing is appended until every lock is held, and a write that finds its records
1490
+ free takes them as before, in one statement in a savepoint. A `lock_timeout` of the caller's own
1491
+ (session or transaction) shorter than the bounded wait stays in force, and its timeout raises
1492
+ `ActiveRecord::LockWaitTimeout`, as before, instead of being taken for the bounded wait's own
1493
+ limit and tried again.
1494
+
1495
+ ### Changed
1496
+ - **The gems' MIT-LICENSE copyright line no longer names the University of the Aegean**, which is
1497
+ the author's affiliation, not a copyright holder.
1498
+ - **Ruby 4.0.9, Rails 8.1.4.1, pg 1.7.0.** Security releases: Ruby 4.0.9 fixes memory
1499
+ corruption when compiling crafted regular expressions, Rails 8.1.4.1 fixes advisories in
1500
+ Active Support and Action View (Active Record unchanged). Lyra's requirements are unchanged
1501
+ (Ruby 4.0+, Rails 8.0+); the project and the example app now run these versions.
1502
+ - **Ruby 4.0.10.** The project and the example app run Ruby 4.0.10 (`.ruby-version`); Lyra's
1503
+ requirement is unchanged (Ruby 4.0+).
1504
+ - **`config.erasure_max_copies`** (default 10) sets `everywhere`'s threshold for the application;
1505
+ `erase!(max_copies:)` and `lyra:erase MAX_COPIES=` still override it per call. Each value left
1506
+ as shared is reported in the new `result.shared` as `{ field:, type:, holders: }` (never the
1507
+ value); `result.shared_values` stays an Integer, their count. **`erase!(dry_run: true)`**
1508
+ (`lyra:erase DRY_RUN=1`) runs the erasure and rolls it back, reporting what it would do, the
1509
+ shared values and their holders included, and writing nothing: once a person's values are
1510
+ erased, a later run cannot find the copies left as shared, so the threshold is chosen with a dry
1511
+ run first, and the task's hint says so. `ErasureApplied` gains `row_only` and `imported`
1512
+ (above). The documentation now gives the operator's `VACUUM` step after an erasure (old row and
1513
+ event versions stay on disk, in WAL and in backups until reclaimed or expired), states that a
1514
+ database error logged by a failed bypass publish can contain the failing row, and describes a
1515
+ narrow wait between an erasure and a model's first use in another transaction.
1516
+ - **Erasure can delete the subject's access log; off by default.** `Lyra::Erasure.erase!` left the
1517
+ record's access stream (`Lyra::DataAccess$<Model>$<id>`: who read the record, when, under which
1518
+ purpose and basis) whole and unreported, and `everywhere` never reached it, since it holds field
1519
+ names, not values. Whether it goes is the controller's decision (it may be evidence to keep, Art.
1520
+ 5(2)). `config.erase_access_log` (default `false`) deletes it once the record is fully erased
1521
+ (this erasure and the earlier ones together cover every personal field);
1522
+ `erase!(access_log: true|false)` (`lyra:erase ACCESS_LOG=1|0`) decides per call, `true` at once. Its events are deleted
1523
+ from every stream and from the events table, in the erasure's transaction. `access_log_subjects:`
1524
+ (`ACCESS_LOG_SUBJECTS=`) names the ids the application passed as the person's subject
1525
+ (`validate_access!(…, subject: "user-7")`), whose logs follow the same rule. `ErasureApplied` and
1526
+ the `Result` record `access_log` (`"erased"` or `"kept"`) and `access_log_entries`, and
1527
+ `ErasureApplied` how many such ids (`access_log_subjects`), never which.
1528
+ `Lyra::AccessLog.erase!(subject, reason:, erased_by: nil)` (`lyra:erase_access_log SUBJECT=
1529
+ REASON= [OPERATOR=]`) deletes one subject's log alone, a record's or a bare id's, and records
1530
+ it as `AccessLogErased` in `Lyra::AccessLogErasures`. A model-wide entry (`Lyra::DataAccess$<Model>$*`, a `pluck`'s) names
1531
+ no subject and is refused (`Lyra::AccessLog::ModelWideSubject`). Retention's `:hard_delete` of a
1532
+ record erased before with its log kept deletes the log too when the setting is on, and records
1533
+ an `ErasureApplied` that names no fields: **code that reads `ErasureApplied` must allow an empty
1534
+ `fields`**. Not covered: the access logs of other records `everywhere` reaches, and an access
1535
+ recorded (on its own connection) while the erasure runs.
1536
+ - **Mode switches, erasures and repairs record what the operating system says beside the operator's
1537
+ label.** The actor (`OPERATOR=`, `Current.user`, `LYRA_OPERATOR`) is a label nothing checks.
1538
+ `Lyra::Attribution` adds the user the process runs as (by uid, not the `USER` variable), the host
1539
+ and the pid: `lyra_mode_transitions` gains `os_user`, `host` and `pid` (added to an existing table
1540
+ on first use), the metadata of `ErasureApplied` and of `AccessLogErased` (an access log erased on
1541
+ its own) carries them beside `erased_by`, and a repair's events record
1542
+ `repaired_by` (the label: `lyra:repair` passes `OPERATOR`, `Lyra::Repair.run` takes `operator:`),
1543
+ `os_user`, `host` and `pid`, where a repair recorded no actor at all. `ErasureApplied`'s data gains
1544
+ `any_fields`, `requested_fields` (what the request named; `nil` for the declared columns) and
1545
+ `beyond_policy` (the columns erased only because of `any_fields`). This authenticates no one: it
1546
+ makes the label comparable with the OS and SSH audit logs, which remain the attribution control.
1547
+ - **PAM DSL: restriction of processing (Art. 18) and the right to object (Art. 21).** PAM had no
1548
+ construct for either: a restriction could only be approximated by narrowing a purpose for
1549
+ everyone or withdrawing a consent, and a purpose on contract or legal obligation could not be
1550
+ stopped for one person. A purpose now declares the Art. 18(2) grounds on which it continues
1551
+ while a subject's processing is restricted (`allowed_when_restricted :consent`, `:legal_claims`,
1552
+ `:protection_of_rights`, or `:important_public_interest, reference: "<the law>"`), and whether
1553
+ it is direct marketing (`direct_marketing!`, Art. 21(2)) or research or statistics
1554
+ (`research_or_statistics! public_interest_task: true|false`, Art. 21(6)); `Purpose#objectable`
1555
+ derives the article that lets a subject object (Art. 21(1) for public task and legitimate
1556
+ interests, research or not), or none. `PamDsl.rights` records, per subject and independent of any policy,
1557
+ restrictions (`restrict!`, then `notify_lift!` and `lift!`, which raises without the Art. 18(3)
1558
+ notice) and objections (`object!`: upheld at once for direct marketing, pending and restricting
1559
+ that purpose for Art. 21(1) and 21(6) until `decide!`, recorded as rejected and raised as
1560
+ `ObjectionNotApplicableError` for a purpose with no right to object; `withdraw_objection!`), as
1561
+ events in `PamDsl.rights_store`, with an `on_rights_event` hook. An upheld objection to direct
1562
+ marketing stops every direct-marketing purpose for that subject; objecting again while an
1563
+ objection is in force returns it; times must be `Time` values that run forwards. `validate_access!` gains
1564
+ Condition 6, checked after every other condition: `RestrictedProcessingError` for a restriction
1565
+ that covers the access when the purpose declares no exception that applies, and
1566
+ `ObjectedProcessingError` for an upheld objection. The Article 30 report prints each purpose's
1567
+ right to object and its exceptions. PAM DSL ships the store protocol and an in-memory store
1568
+ only; the in-memory store refuses to record outside development and test
1569
+ (`RightsStoreNotDurableError`). Lyra's durable store and its enforcement on reads, erasure and
1570
+ retention are not in this change, and writes are not checked under a restriction in 1.0.0.
1571
+ - **Lyra keeps the data subject's restrictions, objections and consent durably, shared by every
1572
+ process.** With PAM DSL loaded, Lyra installs `Lyra::Rights` into it. `PamDsl.rights_store`
1573
+ becomes Lyra's: each request, notice, decision and lift is an event in
1574
+ `Lyra::Rights$<subject key>` (linked to `Lyra::Rights`), appended with RubyEventStore's expected
1575
+ version so that two processes cannot both lift or decide one request (the second gets
1576
+ `PamDsl::RightsConcurrencyError` and nothing is stored), and an append that does not replay is
1577
+ refused before anything is written. The new table `lyra_rights` (one row per restriction or
1578
+ objection: subject key, kind, policy, purpose, fields, ground, state, times) is written in the
1579
+ same transaction and created on first use; `bin/rails lyra:rights:rebuild` rebuilds it.
1580
+ Condition 6 answers from a per-process snapshot of the committed requests in force, refreshed at
1581
+ most every `config.rights_refresh_interval` seconds (default 1, new) and on
1582
+ `Lyra::Rights.refresh!`, so a check costs a clock read when nothing is in force; a transaction
1583
+ that records a right sees it at once and other threads only after it commits, and a rebuild
1584
+ locks the table against transitions. `PamDsl.subject_key_resolver` maps a record to its own key
1585
+ and to the keys of the people it names, by the subject-access copy's selection rule (link
1586
+ columns a record was loaded without are read from its row, or the access fails with
1587
+ `Lyra::Rights::UnresolvedSubjectError`), so a restriction recorded against a person
1588
+ (`"Person$7"`) refuses a read of their invoice under a purpose with no Art. 18(2) exception.
1589
+ Lyra installs its stores when it loads, and at boot only where the application has not
1590
+ installed its own. Consent moves to the same store: PAM DSL's new `PamDsl.consent_store`, when set, is
1591
+ the store of every policy's `consent_policy`, and Lyra's keeps each request, grant and withdrawal
1592
+ as an event in `Lyra::Consent$<subject key>` with the table `lyra_consents` as its projection.
1593
+ Before, consent was held in memory, per process and per policy object: lost on restart, not seen
1594
+ by other processes or by another policy, and gone when a policy was defined again. A person's
1595
+ consent now covers the records that name them; a grant that repeats a granted consent records
1596
+ nothing. Limit: a consent check on a record that names more than one person (a payer and a
1597
+ payee) is refused with `Lyra::Rights::ConsentStore::SharedRecordConsentError`, a
1598
+ `PamDsl::ConsentRequiredError`, whatever each consented to; whose consent such a record needs
1599
+ is future work. Condition 4 and the `:consent` exception of Art. 18(2) read the same record (a
1600
+ consent store's refusal under that exception is now reported with the restriction rather than
1601
+ raised), and `consent_audit` lists the store's records (`recorded_consents`). Without Lyra, PAM
1602
+ DSL keeps consent in memory as before. Neither stream
1603
+ is erased by `Lyra::Erasure` or with the access log: they hold names, grounds and reasons, never
1604
+ values. Model-wide filtering of restricted subjects, `lyra_processable`, and restriction-aware
1605
+ erasure and retention are not in this change.
1606
+ - **Lyra enforces restriction and objection where it checks reads, in erasure and in retention,
1607
+ and reports them.** Under a purpose (or `reads_without_purpose` `:audit`/`:deny`):
1608
+ - A record load that names no declared field (`select(:id, :payer_id)`) is checked too, for a
1609
+ restriction of all of its subjects' data, a pending objection's restriction of the purpose and
1610
+ an upheld objection (PAM DSL's new `validate_rights!`); before, such a load was never checked.
1611
+ A loaded restricted record raises, as a non-consenting one does.
1612
+ - Aggregates, `pluck`, `pick`, `ids`, `exists?` and `cache_version` (subject `Model$*`) leave the
1613
+ restricted subjects out: the record's own key `NOT IN` the restricted records and each link
1614
+ column `IS NULL OR NOT IN` the restricted people, so rows that name no one are kept; ES-NoProj
1615
+ filters its in-memory records by their keys, and fails closed
1616
+ (`Lyra::Rights::UnresolvedSubjectError`) when a record's links cannot be read. A field-scoped
1617
+ restriction applies only to queries naming one of its fields. Audit mode leaves no one out and
1618
+ logs it. With nothing in force for the query, its SQL is unchanged. Before, these queries
1619
+ included the restricted people.
1620
+ - A join, subquery or CTE that reaches a model with such a subject raises
1621
+ `PamDsl::RestrictedProcessingError` rather than include or silently drop them.
1622
+ - `Model.lyra_processable(fields: nil)`, a scope without the records the purpose in scope may not
1623
+ process, for batch jobs (Active Record and ES-NoProj).
1624
+ - The rights and consent are read afresh at the first check of each checking scope: a
1625
+ `Lyra.with_purpose` block, a web request (`Lyra::Rights::Middleware`, installed by the
1626
+ engine), an Active Job (`Lyra::Rights::Job`) and each batch of `in_batches`, `find_each` and
1627
+ `find_in_batches` (`Lyra::Rights.scope`). A right recorded in another process applies from the
1628
+ next scope; before, up to `rights_refresh_interval` later, mid-request included. The first
1629
+ check in a scope makes one indexed query; code that checks nothing (no purpose, `:allow`) makes
1630
+ none.
1631
+ `config.rights_refresh = :interval` (new; the default is `:scope`) keeps scopes but drops the
1632
+ forced read: rights and consent are read again only after `rights_refresh_interval`, as before.
1633
+ - `Lyra::Erasure.erase!` refuses a record that a restriction in force covers (its own, a person's
1634
+ it names, of all the data or of a field being erased) with `PamDsl::RestrictedProcessingError`;
1635
+ `lifting: restriction_id`, after the Art. 18(3) notice, lifts it and erases in one transaction.
1636
+ A dry run (`dry_run: true`) is refused the same way and, with `lifting:`, lifts nothing and
1637
+ lists in `Result#lifting` what it would lift (a real erasure lists what it lifted).
1638
+ `Lyra::Retention` skips restricted records with outcome `:restricted` (`Result#restricted`,
1639
+ printed by `lyra:retention:apply`). Before, both acted on data the subject had asked to be kept.
1640
+ - The subject-access copy (`data_export`) has `restrictions` and `objections`: the subject's own
1641
+ requests, never another person's. The per-subject `processing_activities` gives each declared
1642
+ purpose's right to object and Art. 18(2) exceptions (`rights_by_purpose`); with Lyra's
1643
+ per-model policies it gave none. A dashboard page, `privacy/rights`, lists the open requests
1644
+ with their age against the month of Art. 12(3); the policy page shows each purpose's
1645
+ declarations; `RestrictedProcessingError` and `ObjectedProcessingError` in a dashboard read
1646
+ answer 403.
1647
+ - `Lyra::Verification::DataLifecycleWorkflow` (the data lifecycle with a Restricted place,
1648
+ restrict, notice and lift) and `ObjectionWorkflow`, PetriFlow nets whose tests check that
1649
+ PAM DSL's transitions are exactly theirs and that Deleted is unreachable from Restricted
1650
+ without the lift.
1651
+ - After review: SQL strings are read for the tables they reach (`FROM`/`JOIN`, with aliases), so a
1652
+ string join or subquery reaching a restricted model is refused, and a subquery's words are
1653
+ checked against the policy of the table they belong to (a change to the purpose check too); a
1654
+ `FROM`/`JOIN` Lyra cannot name is refused while a right is in force (fail closed). A polymorphic
1655
+ filter covers STI subclass names. A subject's people are resolved only when a request in force
1656
+ could cover the access, and aggregate rows are not subjects (a grouped `select` or `find_by_sql`
1657
+ no longer raises `UnresolvedSubjectError` because of an unrelated right); ES-NoProj reads a
1658
+ narrowed record's people from the whole record. ES-NoProj's `Model.exists?(id)` and an undeclared
1659
+ purpose follow the rights. Erasure reads a record's people from its events when it has no row
1660
+ (ES-NoProj), refuses one it cannot resolve, lifts with `lifting:` only a restriction of the record
1661
+ erased, and with `everywhere:` leaves another record's restricted copy unmodified
1662
+ (`Result#restricted_copies`, printed by `lyra:erase`, which takes `LIFTING=`). The subject-access copy
1663
+ lists record restrictions under ES-NoProj too. The dashboard's event routes refuse (403) or leave
1664
+ out restricted records (`X-Lyra-Rights-Withheld`). The rights page gives a deadline only to an
1665
+ objection awaiting a decision. The excluded subjects are computed once per snapshot (one pass over
1666
+ the requests in force), the condition compiled once, and a scope whose rights did not change costs
1667
+ one single-row query: a count with 300 restrictions in force went from about 13 ms to about 1 ms.
1668
+ Writes (`update`, `destroy`, bulk writes) are not checked under a restriction in 1.0.0.
1669
+ - **PAM DSL: a type for every Art. 9(1) special category, and for Art. 10 data.** Only `:health`
1670
+ and `:biometric` were special-category types, so a column of any other Art. 9(1) category had
1671
+ to be declared under another type (`:custom`), and a purpose with only an Art. 6 basis could
1672
+ read, say, trade-union membership without PAM raising anything. The new types `:genetic`,
1673
+ `:racial_ethnic_origin`, `:political_opinion`, `:religious_belief` (religious or philosophical
1674
+ beliefs), `:trade_union_membership` and `:sex_life_orientation` are special categories: a purpose
1675
+ that reads one needs an `art9_basis`, else `PamDsl::SpecialCategoryViolationError`. Art. 10 data
1676
+ (criminal convictions and offences) gets the type `:criminal_offence` and a condition of its own,
1677
+ since Art. 10 permits processing only under official authority or when Union or Member State law
1678
+ authorises it, not on an Art. 9(2) ground: a purpose that reads such a field names that
1679
+ authorisation with `art10_authorisation "<the authority or the law>"`, else
1680
+ `PamDsl::CriminalDataViolationError` (strict raises it, audit records it, in `validate_access!`
1681
+ and `validate_fields!`). An Art. 9(2) basis does not satisfy Art. 10, nor an Art. 10
1682
+ authorisation Art. 9. The Article 30 report prints a purpose's Art. 9(2) bases and Art. 10
1683
+ authorisations where it reads such data, the export carries them, and `article_30_gaps` lists
1684
+ a purpose that reads such data without them. The detector, and so the policy generator,
1685
+ proposes the new types from names that say what a column holds (`ethnicity`, `political_party`,
1686
+ `religion`, `trade_union_member`, `sexual_orientation`, `criminal_record`), never from a bare
1687
+ `party`, `union`, `orientation` or `offence`, and the generator marks such fields with the
1688
+ condition a purpose reading them must meet. The dashboard refuses a read that raises the new
1689
+ error as it refuses the others. PAM now has 25 field types (18 before). No field of the Aegean
1690
+ testbed changes type.
1691
+ - **The Art. 15 export is a copy of the data.** `Lyra::Privacy::GDPRCompliance#data_export` keeps
1692
+ its keys, but for `processing_activities` (see Breaking), and adds five. `records` holds each of the subject's records whole: every attribute (not
1693
+ only those a detector flags; encrypted columns in plaintext) and its stored history, `from`/`to`
1694
+ per change. A record is the subject's by what it is: the subject's own row, the rows of monitored
1695
+ models whose `<type>_id` names them (indexed queries), and the records the subject's events touch,
1696
+ so a row that pre-dates monitoring, a destroyed record and an ES-NoProj record are all in it
1697
+ (`state` `:current`, `:destroyed`, `:replayed`); a record whose foreign key moved is the
1698
+ subject's only for the periods it named them (`:reassigned`, `periods`). `accesses` is the access
1699
+ log for those records (`nil` when `record_access_events` is off). `processing_context` takes each
1700
+ record's purposes, bases, recipients, transfers, retention (field periods included) and sources
1701
+ from its own model's policy; `notice` is the policy's `access_notice`; `gaps` lists what the
1702
+ policy does not declare. Who acted is a role label (`"you"`, `"staff"`, `"system"`) unless
1703
+ `disclose_actors: true`. History and the access log are included by default; `history: false`
1704
+ and `accesses: false` are for a request the subject narrowed. A field the policy declares
1705
+ `withheld_from_access: "<basis>"` is named with that basis instead of its value, and one declared
1706
+ `about:` another person goes to `review`; Lyra picks no basis. See
1707
+ [PRIVACY_COMPLIANCE.md](docs/PRIVACY_COMPLIANCE.md#the-copy-of-the-data-from-100).
1708
+ - *Links.* A record refers to the subject through any `belongs_to` whose class is the subject's
1709
+ model (`belongs_to :owner, class_name: "Person"`), a polymorphic `belongs_to` whose type
1710
+ column names it, or the `<type>_id` column; the subject's own model is not linked to itself.
1711
+ - *Moved records.* An event that moves a record between people gives each only their side: the
1712
+ previous subject sees the old values and the link's new value as "moved to another person",
1713
+ the new one only the new values. A record whose links no longer name the subject is
1714
+ `:reassigned` with its state just before the move, whatever comes after. A value last set
1715
+ while the record was another person's is not given: the field is listed under `withheld` with
1716
+ the basis "set while the record was another person's". Accesses to a moved record are the
1717
+ subject's only within their periods.
1718
+ - *The old keys follow the copy's rules* (their content changes): `pii_inventory` and
1719
+ `data_lineage` leave out fields declared `withheld_from_access:` or `about:` another person,
1720
+ give encrypted values in plaintext, and, like `events`, see a moving event with the subject's
1721
+ side only.
1722
+ - *Retention* is what the policy declares: with no rule and no `default`, the period is `nil`
1723
+ and `gaps` lists "Art. 15(1)(d) retention not declared", rather than PAM's silent 7 years.
1724
+ Columns the policy does not declare (no purpose, basis or retention) are listed in `gaps`.
1725
+ - *Accesses* include those recorded under the bare subject id (`access_log_subjects:` on
1726
+ `GDPRCompliance.new` names others, or `[]` for none) and, with the access log off, the
1727
+ subject's earlier exports, with a gap saying the log is off.
1728
+ - **Events record the acting user's class** (`user_type` in the metadata, beside `user_id`, from
1729
+ `Current.user`: the table's owner class for an Active Record user), in the write path, bulk
1730
+ writes and the access log. The copy's role labels compare id and class: "you" only when both
1731
+ are the subject's; "staff or you" when an event records an id equal to the subject's but no
1732
+ class (written before 1.0.0, or by a `metadata_proc` that sets none); "staff" otherwise.
1733
+ - **Each explicit export is recorded,** as `Lyra::Events::SubjectAccessExported` in a stream of
1734
+ its own, `Lyra::SubjectAccessExports$<Type>$<id>`: when, which sections, how many records,
1735
+ `verified_by:` and who exported, never values. Erasing the subject's access log
1736
+ (`AccessLog.erase!`, or `Lyra::Erasure` with `access_log:`) leaves it, so the evidence that a
1737
+ request was answered outlives the log; the copy's `accesses` still lists earlier exports. Only an
1738
+ explicit `data_export` call is recorded: `data_export(record: false)`, `full_report` and the
1739
+ dashboard's subject and GDPR routes are views of the data and record nothing. On by default;
1740
+ `config.record_subject_access_exports = false` turns it off. Nothing is recorded in disabled mode.
1741
+ - **The GDPR reports read each model's policy.** The wrapper passes PAM the policy of each record's
1742
+ model (its `privacy_policy`, else `config.privacy_policy`). `processing_activities` takes the
1743
+ purpose, legal basis, categories, recipients and retention from it, where it printed "Not
1744
+ specified" and guessed the purpose from `metadata[:source]`; `rectification_history` lists an
1745
+ update that changed a declared field as well as one the detector flags.
1746
+ - **The privacy stamp's purposes are those the policy admits the field for,** from either side:
1747
+ the field's `allow_for` and every purpose that lists it in `requires` or `optionally`, as
1748
+ `Policy#allowed?` decides. A policy that lists its fields on the purpose side stamped every field
1749
+ with `"purposes" => []`. `allowed_purposes` on the PAM adapter follows.
1750
+ - **PAM DSL field options and the notice declaration:** `withheld_from_access:`, `about:` and
1751
+ `collected_from` on `field`, and `access_notice` on the policy (see the PAM DSL changelog).
1752
+ - **ES-Lazy reads cost less, with the same consistency.** The check before every read reads the
1753
+ checkpoint and the newest event id in one query instead of two. A reader that finds the log
1754
+ ahead of the tables tries the catch-up lock (`pg_try_advisory_xact_lock`); if another reader holds
1755
+ it, it waits for that catch-up to end, without holding the lock, and reads the checkpoint again.
1756
+ It is done if the checkpoint has reached the newest event it saw and no event the checkpoint
1757
+ lists as missing is visible to it; otherwise it catches up itself, as before. Before, it queued
1758
+ for the lock to repeat a catch-up that, under a write-heavy mix, had mostly been done already. A
1759
+ catch-up that applies nothing no longer rewrites the checkpoint. The three rules hold: only the
1760
+ lock's holder applies events or moves the checkpoint, in the log's order and in its own
1761
+ transaction, and a read sees every event it saw before (a late commit below the checkpoint
1762
+ still within `GAP_RECHECK` on the fast path). In a local profile (30% writes, 70% reads by id,
1763
+ one process) the mean read took about 45% less time at 16 threads and 30% less at 4.
1764
+ - **PAM DSL: a Condition 5 failure raises `PamDsl::SpecialCategoryViolationError`,** where it
1765
+ raised `PamDsl::SensitivityViolationError`. The condition is about special categories (Art. 9,
1766
+ decided by the field's type); sensitivity is a separate Art. 32 risk tier that gates nothing,
1767
+ and the old name mixed the two. `SensitivityViolationError` stays as a deprecated alias of the
1768
+ new class until 2.0, so `rescue PamDsl::SensitivityViolationError` still catches it.
1769
+ - **ModeSync adopts a switch the gate checks only if the gate cleared it.** It adopted the newest
1770
+ `applied` row of `lyra_mode_transitions` and checked nothing, so one `INSERT` moved every process
1771
+ from Monitor to ES-NoProj. A switch now records where it came from (`from_config`), whether the
1772
+ gate was overridden (`forced`) and the certificate of the clean check that cleared it
1773
+ (`certificate_id`). Other processes adopt a switch the gate checks (an escalation from Disabled
1774
+ or Monitor to Hijack or event sourcing, or leaving ES-NoProj, ES-Lazy or ES-Async for a mode
1775
+ that reads the tables) only if the row names that certificate (for the same `from -> to`,
1776
+ recorded before the row, named by no earlier switch) or is forced; whether the gate checks the
1777
+ switch is judged from the row's `from` and from where the previous switch left the application,
1778
+ so a row cannot claim a `from` that avoids the gate. A switch the gate does not check (Monitor to
1779
+ Disabled, ES-Sync to ES-NoProj) is adopted as recorded. No clock is compared, so a switch near
1780
+ the end of its certificate's life or under clock skew is adopted. With the gate off, switches are
1781
+ adopted as recorded. A refusal is logged once; an error while deciding leaves the
1782
+ row to be tried again. Columns Lyra adds to its own tables are added outside the caller's
1783
+ transaction and safely when two processes add one at once. This guards against mistakes and
1784
+ naive writes, not against anyone who can write the table.
1785
+
1786
+ - **The dashboard and `Lyra::EventFlow` read a window of the log, not all of it.** The flow and
1787
+ privacy routes read the newest `config.dashboard_max_events` events (10,000 by default; `?limit=`
1788
+ may only lower it), leaving out the access log's own entries, which would otherwise fill the
1789
+ window as soon as the access log is on, and report `window: { limit:, returned:, truncated:,
1790
+ skipped: }`. A subject's timeline reads that record's own stream (its newest events, up to the
1791
+ limit), not the log; what a user did to other records is the `flow/user_actions` route, which
1792
+ lists at most `?actions=` actions. With `config.record_access_events` on, a dashboard read is
1793
+ recorded in `Lyra::DataAccess$Dashboard` (route, scope, how much came back, never a value). The
1794
+ GDPR report helpers still read the whole log, a batch of 1,000 at a time.
1795
+ - **A save that changes nothing records no event, in every mode.** Monitor, Hijack and event
1796
+ sourcing each appended an `Updated` with no changes for a `save` that changed no attribute.
1797
+ An update carries its changes, and there were none; Hijack and event sourcing still refuse such a
1798
+ save on a destroyed record, as for any write.
1799
+ - **One event shape in every mode**. Monitor, the command handler of Hijack and event
1800
+ sourcing, bypass writes, Genesis and repair now build the payload with `Lyra::EventData`: a
1801
+ create or an import carries the record's `attributes` and no `changes`; an update carries its
1802
+ `changes` as `[old, new]` pairs and no `attributes`; a destroy carries neither. Before, Monitor's
1803
+ events carried the record on an update and a destroy and listed every column in a create's
1804
+ `changes`, and a Hijack or event-sourcing update stored its delta twice (`attributes: { changes: … }`
1805
+ beside `changes`). Every mode now sets `metadata[:source]` (`lyra_interceptor` for Monitor,
1806
+ `lyra_command_handler` for the others). Replay reads a create's `attributes` and an update's
1807
+ `changes`; events in the old per-mode shapes are not supported (see Breaking).
1808
+ The smaller events cost less to serialise and store, and copy less personal data into the log.
1809
+ - **Personal data is detected in an event's fields.** `Lyra::Event.fields_of(event)` gives a create's
1810
+ attributes or an update's new values; the privacy and flow controllers, `EventFlow`, `EventGraph`,
1811
+ `Timeline`, Lyra's GDPR helpers and the PAM DSL's compliance reports use it. They read an event's
1812
+ `attributes` alone, so an update, which now carries none, would have reported no personal data
1813
+ (a Hijack or event-sourcing update already did, because its `attributes` held only `changes`).
1814
+ - **A destroy no longer keeps the record's final state** in Monitor. It is in the events before it;
1815
+ the audit trail view shows no final state for a destroy.
1816
+ - **DualView compares `created_at` and `updated_at`.** They were left out of the comparison; a row
1817
+ whose timestamp differs from its events is now a difference, and `lyra:repair` records it like
1818
+ any other.
1819
+ - **A bare `touch_all` publishes an update per row** whose only change is `updated_at`, as an
1820
+ instance `touch` already did. Bulk bypass events (`update_all`, `upsert_all` and the like) now
1821
+ list `created_at` and `updated_at` among their changes when those moved, so an upsert's event
1822
+ carries `updated_at` beside the columns it set.
1823
+ - **A cheaper destroyed check.** Before every update and destroy in ES-NoProj, ES-Async and
1824
+ ES-Lazy, the check that the record was not destroyed (0.9.3) read and deserialized the stream's
1825
+ last event through Rails Event Store, its data and metadata, to learn its operation. It now
1826
+ decides from the stream's last replayed event (Fixed, above), read with a query over the event
1827
+ store's own tables (`Lyra::StreamTail`) that returns each event's id, type and stored data and
1828
+ deserializes nothing. The usual update costs that one query: the last event is positively a
1829
+ replayed create, import or update when its type ends in `Created`, `Updated` or `Imported` and
1830
+ its data records no destroy's operation and no `replay: false`. Data in a `json` or `jsonb`
1831
+ column comes back as a Hash and is decided from directly. Any other event is read through Rails
1832
+ Event Store by its id, one query each, and an event that is not replayed sends the check to the
1833
+ one before it. The walk back reads one event, then pages of 50, and skips in SQL the events Lyra
1834
+ never replays (`AssociationLinked`, `AssociationUnlinked`, `ErasureApplied`), so a record linked
1835
+ by hundreds of others still costs one query. Domain-event rules and `event_mapping` may no longer
1836
+ use those three names (`ArgumentError`). The type alone cannot decide, since custom event names
1837
+ and domain events can store a destroy under any name. The shortcut is taken only with
1838
+ `RubyEventStore::ActiveRecord::EventRepository`, RES's default mapper and the YAML, JSON or
1839
+ `Lyra::EventSerializer` serializer, and for those it gives the answer a read through Rails Event
1840
+ Store gives. With a mapper that transforms records on load, another serializer or another
1841
+ repository, the stream is read through Rails Event Store. The same happens when the query fails
1842
+ the first time it is tried, for example after a Rails Event Store upgrade that changed its
1843
+ tables. That first try runs during the first check, in a savepoint when a transaction is open,
1844
+ so it cannot abort the write. The lock is unchanged. Locally (no YJIT), the check on a stream
1845
+ ending in an update takes 0.9 ms and 43 allocations instead of 2.3 ms and about 900, and an
1846
+ update or destroy in these three modes allocates about 860 fewer objects. Refusing a write to a
1847
+ destroyed record costs about what it did.
1848
+ - **The bypass-write net (`Lyra::Verification::BypassWorkflow`) models the writes that skip
1849
+ callbacks as 1.0.0 makes them.** A third store, a table that lags the event store (ES-Async,
1850
+ ES-Lazy: `streams_checked` with `table_backed`, as `BypassEvents.checks_streams?` and
1851
+ `locks_rows?`), whose writes read each record's stream under its aggregate lock as ES-NoProj's
1852
+ do; a new ending, `unchanged`, for a write to a record destroyed since it was read (its stream
1853
+ ends in a destroy, or, in ES-Sync, Hijack and Monitor, its row is gone: no write, no event);
1854
+ the `Genesis::Refused` rejection of a row the model could not import, and the
1855
+ `BulkWriteLockLimitError` of a bulk write or `dependent: :nullify` over
1856
+ `bulk_write_lock_limit` or `upsert_all` where the streams are checked (and of `update_all`,
1857
+ `insert_all(!)` and `upsert_all` whose unique values' locks exceed it, ES-Sync included), and that upsert's
1858
+ `RecordDestroyedError` (see Fixed); and `insert_all(!)` as a class of its own (a new row:
1859
+ nothing to admit, written to the table in ES-NoProj too; a key already there is skipped by
1860
+ `insert_all`, unchanged, and fails `insert_all!`). `BypassWorkflow.coverage`
1861
+ counts a dead marking as covered only when it ends in exactly one way: `rejected` alone,
1862
+ `store_changed` with `event_logged`, or `unchanged` alone (and returns how many end
1863
+ unchanged, `unchanged:`); a store change without an event, an event logged for an unchanged
1864
+ write, or a rejection beside a write or an event is reported. `ROUTED` lists ES-NoProj's
1865
+ class-level `update_counters`, `increment_counter` and `decrement_counter`, which write through
1866
+ `update_all`. Rails' association writes (`Lyra::Associations::Maintenance`, see Fixed) enter
1867
+ past the strict guard: `request_nullify` is the nullify Rails marked (an application's
1868
+ `update_all(fk: nil)` is `request_bulk`), a `has_one`'s `dependent: :nullify` the column path
1869
+ (`request_association_column`), removing records from a `has_many :through` over a monitored
1870
+ model the bulk path (`request_through_delete`). `has_and_belongs_to_many` join-row deletes are
1871
+ a class of their own, `link`: Rails' marked deletes are never refused (`request_unlink`), the
1872
+ application's `delete_all` on a join model is guarded (`request_join`), and either writes the
1873
+ table, in every store, then logs `AssociationUnlinked` (`write_unlink`, `publish_unlink`); the
1874
+ application's `insert_all`, `insert_all!` and `upsert_all` of join rows are guarded the same
1875
+ way (`request_join_insert`), write, then log `AssociationLinked` (`write_link`, `publish_link`),
1876
+ or end unchanged (`skip_conflict_link`) or refused (`refuse_conflict_link`). Bulk inserts and
1877
+ upserts are first brought to the rows Monitor's table would have where the streams are checked
1878
+ (`bring_insert_keys_up_to_date`, `bring_upsert_rows_up_to_date`; `insert_as_given` and
1879
+ `upsert_as_given` elsewhere), and a bulk insert's locks can exceed the limit
1880
+ (`refuse_lock_limit_insert`); where it reads keys or unique values from the event store, Genesis
1881
+ refuses it for a model whose import was refused (`refuse_genesis_insert`). An `update_column(s)` or `update_all` whose value a unique index
1882
+ refuses is rejected, writing and logging nothing (`refuse_unique_column`, `refuse_unique_bulk`:
1883
+ the table's `UPDATE` raises, and in ES-NoProj, ES-Async and ES-Lazy the event store is checked,
1884
+ see Fixed), and so is an `upsert_all` whose value of an index other than its conflict columns
1885
+ another record holds (`refuse_unique_upsert`: the table's statement, or in ES-Async and ES-Lazy
1886
+ the event store). The net now has 38 places, 87 transitions and 181 reachable markings (was 25,
1887
+ 36 and 72); it is 1-bounded, has no dead transition, and all 18 dead markings (was 7) are
1888
+ covered.
1889
+
1890
+ ### Removed
1891
+ - **`Lyra::Associations::EventAware`** (`event_aware_belongs_to`, `event_aware_has_many`), an
1892
+ opt-in concern that nothing used, superseded by the association interceptor, read-your-writes
1893
+ and ES-Lazy. Its `has_many` never added pending records (its `PendingRecords` returned none),
1894
+ and the records its `belongs_to` rebuilt from events (`VirtualRecord`) bypassed Lyra's read
1895
+ path: their values were the events' own, not converted to the attribute types; the target's
1896
+ default scopes were not applied, so a record the scoped lookup did not find (soft-deleted, say)
1897
+ came back from its events; and purpose-bound reads did not check them.
1898
+
1899
+ ### Documented
1900
+ - **`allow_for` widens, it does not restrict.** A field may be used for a purpose when its
1901
+ `allow_for` names the purpose or the purpose's `requires`/`optionally` names the field
1902
+ (`Policy#allowed?` takes the union, as before). The PAM DSL README, PRIVACY_COMPLIANCE.md,
1903
+ API_REFERENCE.md and the methods' comments now say so. No behaviour change.
1904
+
1905
+
1906
+ ### Known issues
1907
+ - **ES-Lazy: a transaction that reads and then takes a record's aggregate lock can deadlock against a
1908
+ bulk write.** The catch-up a read begins with writes the rows of the records whose events it
1909
+ applies without their aggregate locks, and inside a transaction those rows stay locked until it
1910
+ ends. When the transaction then takes a record's aggregate lock (a write through callbacks,
1911
+ `update_all`) that a bulk write holds while it waits for one of those rows (to bring it up to
1912
+ date before writing it), the two wait for each other; PostgreSQL detects it and aborts one with
1913
+ `ActiveRecord::Deadlocked`, and nothing of the aborted one is written or recorded. Retry the
1914
+ transaction, or do the reads outside it (docs/TROUBLESHOOTING.md, Concurrency Issues). A
1915
+ transaction that writes several records in an order of its own can deadlock against another, or
1916
+ a bulk write, as plain Active Record's row locks do; take them in one order (by primary key) and
1917
+ retry on `ActiveRecord::Deadlocked` (same section).
1918
+
1919
+ ## [0.9.3] - 2026-11-04
1920
+
1921
+ **Security update.** A thorough review of Lyra's code against the claims made for it found the
1922
+ defects below. None of them touches the gradual migration itself: the modes, the interception
1923
+ of writes, the mapping of operations to events, the checked transitions between modes, and the
1924
+ results that rest on them are unchanged. The defects are in the dashboard, the GDPR helpers, the
1925
+ hand-over from PaperTrail and the operator tasks. Upgrade if the dashboard is open to anyone, or if
1926
+ the Art. 15/20 export, the retention check, `lyra:erase` or the policy generator is used. PAM DSL
1927
+ moves to 0.9.3 for its policy generator, and PetriFlow to 0.9.3 for the way it runs Graphviz
1928
+ (below).
1929
+
1930
+ ### Security
1931
+ - **The dashboard resolves only monitored models.** The routes that take a model
1932
+ (`dashboard/compare`, `discrepancies`, `model_overview`, `audit_trail`, `model_heatmap`,
1933
+ `event_list`, `flow/event_chain`, `flow/visualization`) called `constantize` on the request
1934
+ parameter and returned the record's attributes. Any user the dashboard admitted could therefore
1935
+ read the rows of any Active Record model, monitored or not, including password digests and
1936
+ tokens. They now answer 404 for a model Lyra does not monitor. List sizes, day ranges and graph
1937
+ depth are clamped (`limit` at most 1,000, 500 or 100 by route, `days` at most 90, `depth` at most
1938
+ 3). As before, the dashboard is closed outside development and test unless
1939
+ `config.dashboard_authorization` admits the request.
1940
+ - **The Art. 15/20 export gathers the subject's data, not the subject's edits.**
1941
+ `Lyra::Privacy::GDPRCompliance` selected every event whose metadata named the subject as its
1942
+ *actor*. A staff member's export therefore held the records they had edited, which is other
1943
+ people's data, while a subject's own records written by staff could be missed. It now gathers
1944
+ the events of the subject's own record, of records that refer to the subject by foreign key
1945
+ (`<subject_type>_id`, e.g. `user_id`), and of events that name the subject explicitly. The
1946
+ dashboard's subject timeline does the same; who did what remains the `user_actions` view.
1947
+ - **PetriFlow runs Graphviz without a shell.** `VerificationRunner` (its PNG, PDF and SVG
1948
+ images) and `Visualization::Graphviz#render` built a shell command from the report directory,
1949
+ the workflow's name, the output path and the layout, so a path or name with shell syntax in it
1950
+ would have run as a command. They now pass Graphviz its arguments one by one, and `render`
1951
+ accepts only Graphviz's layout engines (`Graphviz::LAYOUTS`) and plain format names. The
1952
+ values come from the developer, not from a request; a static analysis (Brakeman) flagged both.
1953
+
1954
+ ### Fixed
1955
+ - **PaperTrail is switched off per model, not per request.** Where Lyra takes over a write
1956
+ (Hijack and the event-sourcing modes), it set `PaperTrail.request.enabled = false`. That turned
1957
+ versioning off for every model for the rest of the request or job thread and never turned it
1958
+ back on, so writes to versioned models Lyra does not monitor lost their versions, with no event
1959
+ in their place. Lyra now calls `PaperTrail.request.disable_model` for the model it takes over.
1960
+ - **The retention check reads the policies.** `GDPRCompliance#retention_compliance_check` ignored
1961
+ the policies' `retention` blocks in favour of hard-coded periods for models named Payment,
1962
+ Invoice, Student and Enrollment, and raised `TypeError` when `config.retention_policy` was a
1963
+ single duration such as `7.years`. It now takes each monitored model's period from its policy,
1964
+ over the configured default. `retention_compliance_with_policy` called a method that did not
1965
+ exist.
1966
+ - **Erasures and mode switches name who made them.** `rake lyra:erase` passes `OPERATOR=` as
1967
+ `erased_by`, falling back to `LYRA_OPERATOR`, then the operating-system user and process; before,
1968
+ it recorded no one. Every mode switch records the same in a new `actor` column of
1969
+ `lyra_mode_transitions`, added on first use to an existing table, and logs it. A switch to
1970
+ Disabled turns purpose checks and the access log off, so it is now attributable.
1971
+ - **Erasure is limited to personal fields.** `Lyra::Erasure.erase!` (and `lyra:erase FIELDS=`)
1972
+ erased any column it was named, rewriting history in place under an Art. 17 reason. It now
1973
+ refuses a column that neither the record's policy declares personal nor the events' privacy
1974
+ stamps list, unless called with `any_fields: true` (`ANY_FIELDS=1`).
1975
+ - **Bulk writes lock the rows they write.** `update_all` and `delete_all` (and the writes routed
1976
+ through them: `update_counters`, `touch_all`, `delete_by`, `dependent: :delete_all`) took their
1977
+ snapshot with a plain read, so under READ COMMITTED a row another transaction changed or
1978
+ inserted between the snapshot and the write could be written without an event. They now lock
1979
+ the matched rows (`FOR UPDATE`, through a subquery of keys so DISTINCT and joins work) and write
1980
+ exactly those, inside a transaction in every mode that keeps tables. A failed append rolls back
1981
+ only its savepoint, so in Monitor the write still stands.
1982
+ - **A destroyed record is not written again.** Two writers of one record, a destroy and an
1983
+ update, could both pass: the second found no row to lock, fell back to the advisory lock and
1984
+ appended after the Destroyed event, leaving a stream that brought the record back
1985
+ (`Lyra.state_at`, DualView). An update or destroy of a record whose stream ends in a destroy now
1986
+ raises `Lyra::RecordDestroyedError` (a `RecordNotFound`). To check it, ES-NoProj and ES-Lazy now
1987
+ take the lock on the aggregate for updates and destroys, as the other event-sourcing modes did.
1988
+ - **PAM DSL 0.9.3: the policy generator no longer chooses legal bases.** It wrote one for each
1989
+ purpose it drafted: contract for billing, which tax law often makes a legal obligation;
1990
+ legitimate interests for analytics, which a public authority cannot use for its tasks; consent
1991
+ without a consent block. A drafted purpose now reads `basis :undetermined` with the usual
1992
+ candidate and its caveats as a comment. Access under it is refused
1993
+ (`PamDsl::UndeterminedBasisError`), and the Article 30 register lists it as a gap, until the
1994
+ controller decides.
1995
+
1996
+ ### Documented
1997
+ - **The dashboard is a development tool.** Opening it in production with
1998
+ `config.dashboard_authorization` is at the deployer's risk: whoever the rule admits reads every
1999
+ data subject's records and history, with no declared purpose and no access-log entry, and the
2000
+ rule covers the whole dashboard. What bringing it to production would take is listed in the
2001
+ README (Dashboard) as a future feature.
2002
+
2003
+ ### Dependencies
2004
+ - **The development lockfile is updated against published advisories** (bundler-audit,
2005
+ 2026-11-04): `rack`, `nokogiri`, `loofah`, `rails-html-sanitizer`, `crass` and `sqlite3` (tests
2006
+ only). The gems' own dependency declarations are unchanged: none pins a vulnerable version, and
2007
+ an application resolves its own.
2008
+
2009
+ ### Known limitations (unchanged)
2010
+ - PAM DSL keeps consent records in memory, per process. Persistent consent storage is future
2011
+ work; an application that relies on consent must store it. Documented in the PAM DSL README
2012
+ ("Runtime consent store"), on `ConsentStore`, and in `docs/PRIVACY_COMPLIANCE.md`.
2013
+
10
2014
  ## [0.9.2] - 2026-11-03
11
2015
 
12
2016
  One fix in Lyra; PAM DSL and PetriFlow are unchanged (0.9.1).
13
2017
 
2018
+ ### Released
2019
+ - `orfeas_lyra` on RubyGems, built from tag `v0.9.2` and pushed on 2026-11-03
2020
+
14
2021
  ### Fixed
15
2022
  - **ES-Async read-your-writes no longer races a projection job.** With read-your-writes
16
2023
  (`each_write: true`), the writer projected its own event without the per-stream lock that
@@ -231,8 +2238,8 @@ Everything since 0.8.0, the version the PAM article describes.
231
2238
  replays to the anonymized row. Replacement is nil where allowed, `"erased:<id>"` for a NOT NULL
232
2239
  string, else the column default. Works in every mode, with strict data access on and inside a
233
2240
  purpose. On request only; not policy-driven, and crypto-shredding remains future work.
234
- - **Event store failure policies, named** (`Lyra::EventStoreUnavailableError`, `Lyra.append_events`;
235
- FEATURE_GAP_PLAN F1) — every event Lyra writes now goes through `Lyra.append_events`, which
2241
+ - **Event store failure policies, named** (`Lyra::EventStoreUnavailableError`, `Lyra.append_events`)
2242
+ — every event Lyra writes now goes through `Lyra.append_events`, which
236
2243
  raises `EventStoreUnavailableError` (the store's error as its `cause`, the stream in its message)
237
2244
  when the store fails. The policy is fixed by the mode: **fail-closed** in Hijack and every
238
2245
  event-sourcing mode (the write fails with the error and rolls back), **log-and-continue** in
@@ -248,7 +2255,7 @@ Everything since 0.8.0, the version the PAM article describes.
248
2255
  where the events are authoritative (rebuild the tables instead). `ModeTransition.record_ids` and
249
2256
  `ModeTransition.discrepancy` expose the check it uses.
250
2257
  - **Access log** (`config.record_access_events`, default false; `Lyra::AccessLog`,
251
- `Lyra::Events::DataAccessed`, `Lyra::Events::DataAccessDenied`; FEATURE_GAP_PLAN F6) — when on,
2258
+ `Lyra::Events::DataAccessed`, `Lyra::Events::DataAccessDenied`) — when on,
252
2259
  every access the privacy policy validates (`validate_access!`) is recorded in the subject's
253
2260
  access stream (`Lyra::DataAccess$<Model>$<id>`): `DataAccessed` when it went ahead (outcome
254
2261
  `granted`, or `audited` with the violations audit mode let through), `DataAccessDenied` when
@@ -264,7 +2271,7 @@ Everything since 0.8.0, the version the PAM article describes.
264
2271
  validate access, and records it in its benchmark reports. The Article 30 register stays
265
2272
  PAM's `article_30_report`, generated from the declared policy.
266
2273
  - **Privacy stamps on events** (`config.annotate_privacy`, default false; `Lyra::Privacy.stamp`,
267
- `Lyra::Privacy.stamp_of`; FEATURE_GAP_PLAN F5) — when on, every event that carries an
2274
+ `Lyra::Privacy.stamp_of`) — when on, every event that carries an
268
2275
  attribute the model's privacy policy declares is stamped as it is built with
269
2276
  `metadata[:privacy]`: the policy's name and, per attribute, its type, sensitivity, allowed
270
2277
  purposes, retention period (ISO 8601) and the contexts it has a transformation for — never a
@@ -287,7 +2294,7 @@ Everything since 0.8.0, the version the PAM article describes.
287
2294
  switch reaches. On wherever the Mode Transition Safety gate is (not in the test environment);
288
2295
  the raw `config.mode =` setter is left alone. the migration guide (`docs/MIGRATION_GUIDE.md`, Switching modes) documents the
289
2296
  deploy-time rule.
290
- - **Mode Transition Safety** (`Lyra::ModeTransition`, `rake lyra:mode:check`; FEATURE_GAP_PLAN F3)
2297
+ - **Mode Transition Safety** (`Lyra::ModeTransition`, `rake lyra:mode:check`)
291
2298
  — a mode switch that changes the authoritative store is allowed only when rows and events
292
2299
  agree for every monitored record: escalating from Disabled or Monitor to Hijack or event
293
2300
  sourcing (rows that predate Lyra are imported by Genesis first), and leaving ES-NoProj,
@@ -304,8 +2311,8 @@ Everything since 0.8.0, the version the PAM article describes.
304
2311
  `config.dual_view_discrepancy_handler`) — after a write commits, a sampled share of writes
305
2312
  is compared with its events; a discrepancy is logged and passed to the handler, and the write
306
2313
  is never failed. Skipped in ES-NoProj, ES-Lazy and ES-Async, whose tables lag by design.
307
- - **Domain events** (`monitor_with_lyra domain_events: [...]`, `Lyra::DomainEvents`;
308
- FEATURE_GAP_PLAN F8) — rules name a write's event after what it means instead of the CRUD
2314
+ - **Domain events** (`monitor_with_lyra domain_events: [...]`, `Lyra::DomainEvents`)
2315
+ — rules name a write's event after what it means instead of the CRUD
309
2316
  operation: `{ name: "PaymentCompleted", on: %i[create update], if: ->(payment, changes) { ... } }`.
310
2317
  The first matching rule names the write's own event, which keeps Lyra's envelope (model, id,
311
2318
  operation, attributes, changes), so replay, projections, DualView and `Lyra.state_at` work
@@ -318,7 +2325,7 @@ Everything since 0.8.0, the version the PAM article describes.
318
2325
  callbacks are still recorded as CRUD bypass events. Generated event classes are registered
319
2326
  at boot. A rule whose block raises fails the write where events are the record (Hijack,
320
2327
  event sourcing) and is logged in Monitor. Commands now carry their `record`.
321
- - **Configuration surface** (FEATURE_GAP_PLAN F2): `config.models = %w[Order Payment]` (or a
2328
+ - **Configuration surface**: `config.models = %w[Order Payment]` (or a
322
2329
  hash of name => options) declares the monitored models by name in the initializer; once the
323
2330
  application's code has loaded, each is given `monitor_with_lyra`, with no model file edited,
324
2331
  and a name that does not resolve fails the boot. `config.verify_mapping!` verifies the
@@ -333,8 +2340,7 @@ Everything since 0.8.0, the version the PAM article describes.
333
2340
  destroyed. `Model.as_of(time).find(id)` / `.find_by_id(id)` return read-only records and
334
2341
  `.all` every record that existed then. For a row imported by Genesis, a time before its
335
2342
  import raises `Lyra::Temporal::HistoryNotRecorded` instead of guessing (a time before its own
336
- `created_at` is `nil`). Works in every mode that records events, Monitor included
337
- (FEATURE_GAP_PLAN F7).
2343
+ `created_at` is `nil`). Works in every mode that records events, Monitor included.
338
2344
  - **ES-NoProj joins, evaluated in memory** (`Lyra::Projections::CachedJoins`) — `joins`,
339
2345
  `left_joins`/`left_outer_joins`, `where.missing` and `where.associated` on `belongs_to`,
340
2346
  `has_many` and `has_one` associations, with hash conditions on the joined table
@@ -353,9 +2359,8 @@ Everything since 0.8.0, the version the PAM article describes.
353
2359
  no provider is installed. PAM is an adapter (`Lyra::Privacy::Adapters::Pam`), loaded when
354
2360
  `pam_dsl` is present. `Lyra::Privacy.annotations_for(event)` returns the annotation of every
355
2361
  declared attribute an event touched. The foundation for `config.privacy_policy=` and PII
356
- annotation in event metadata (`FEATURE_GAP_PLAN.md`, F2 and F5). From `thesis-restructure`
357
- (1adadfe); the testbed binds `Registration` and `PaymentTransaction` to its `:epay_data`
358
- policy (d0df542), which adds nothing to the write path.
2362
+ annotation in event metadata. The testbed binds `Registration` and `PaymentTransaction` to its
2363
+ `:epay_data` policy, which adds nothing to the write path.
359
2364
  - **`Lyra::IdGenerator.reserves_safely?(model)`** — true for UUID keys and for integer keys
360
2365
  backed by a PostgreSQL sequence. It uses the cached sequence lookup, so it adds no query per
361
2366
  write once a table's sequence is known.
@@ -366,8 +2371,8 @@ Everything since 0.8.0, the version the PAM article describes.
366
2371
  row that changed; `delete_all` "destroyed" per row; inserts "created" per row; `upsert_all`
367
2372
  finds existing rows by its conflict key (primary key, `unique_by:` columns or index name) and
368
2373
  publishes "updated" for those, "created" for the rest. The caller's `returning:` result is
369
- unchanged. Brought over from the `thesis-restructure` branch (3ba39bb, 975dc6e), keeping
370
- master's instance methods (events built after the write).
2374
+ unchanged, and the instance methods (`update_column(s)`, `touch`, `delete`) still build their
2375
+ events after the write.
371
2376
  - **`Lyra.projection_write { ... }`** — Lyra's own read-model writes: skips strict data access
372
2377
  and publishes no bypass events. `ModelProjection` and `Rebuild` use it; use it for writes that
373
2378
  must not reach the event stream, such as seeding or wiping a table.
@@ -778,8 +2783,7 @@ ignored like any unknown option.
778
2783
  return records a write had already changed. Stamped entries (see Changed) make all three
779
2784
  impossible: a stale entry is never used.
780
2785
  - **An explicit id was replaced** — `CommandHandler#handle_create` generated an id even when the
781
- application set one; it now keeps it, in the event and in the row. From `thesis-restructure`
782
- (526908e), applied to master's own Hijack fix (cd3ea89).
2786
+ application set one; it now keeps it, in the event and in the row.
783
2787
  - **ES-NoProj `insert_all` ran, then raised** — through `CachedRelation`'s scope branch the
784
2788
  insert executed and the caller then got `UnsupportedQuery` ("not a scope"). Inserts and
785
2789
  upserts now go to the table relation and return its result.