orfeas_lyra 0.9.3 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +1923 -17
- data/README.md +15 -8
- data/Rakefile +40 -1
- data/app/controllers/lyra/application_controller.rb +108 -0
- data/app/controllers/lyra/dashboard_controller.rb +17 -15
- data/app/controllers/lyra/flow_controller.rb +30 -33
- data/app/controllers/lyra/privacy_controller.rb +57 -18
- data/app/views/lyra/dashboard/discrepancies.html.erb +5 -5
- data/app/views/lyra/dashboard/index.html.erb +1 -0
- data/app/views/lyra/flow/crud_mapping.html.erb +1 -1
- data/app/views/lyra/privacy/pii_detection.html.erb +1 -1
- data/app/views/lyra/privacy/policy.html.erb +7 -0
- data/app/views/lyra/privacy/rights.html.erb +78 -0
- data/config/privacy_policies.rb +0 -3
- data/config/routes.rb +1 -0
- data/lib/lyra/access_log.rb +187 -6
- data/lib/lyra/advisory_lock.rb +236 -0
- data/lib/lyra/aggregate.rb +1 -1
- data/lib/lyra/associations/habtm_links.rb +382 -19
- data/lib/lyra/associations/maintenance.rb +229 -0
- data/lib/lyra/attribution.rb +48 -0
- data/lib/lyra/bypass_events.rb +426 -45
- data/lib/lyra/command_handler.rb +19 -14
- data/lib/lyra/configuration.rb +237 -10
- data/lib/lyra/consistency/read_your_writes.rb +4 -2
- data/lib/lyra/correlation.rb +121 -0
- data/lib/lyra/domain_events.rb +4 -0
- data/lib/lyra/dual_view.rb +53 -16
- data/lib/lyra/encrypted_attributes.rb +5 -0
- data/lib/lyra/engine.rb +67 -0
- data/lib/lyra/erasure.rb +1002 -116
- data/lib/lyra/event.rb +35 -1
- data/lib/lyra/event_data.rb +38 -0
- data/lib/lyra/event_flow.rb +31 -15
- data/lib/lyra/event_store_adapter.rb +28 -0
- data/lib/lyra/event_window.rb +58 -0
- data/lib/lyra/genesis.rb +548 -44
- data/lib/lyra/interceptors/association_interceptor.rb +689 -34
- data/lib/lyra/interceptors/crud_interceptor.rb +547 -95
- data/lib/lyra/internal.rb +26 -0
- data/lib/lyra/mode_sync.rb +51 -3
- data/lib/lyra/mode_transition.rb +245 -34
- data/lib/lyra/model_selection.rb +35 -0
- data/lib/lyra/own_connection.rb +17 -2
- data/lib/lyra/privacy/adapters/pam.rb +25 -2
- data/lib/lyra/privacy/gdpr_compliance.rb +264 -36
- data/lib/lyra/privacy/interface.rb +27 -7
- data/lib/lyra/privacy/subject_links.rb +143 -0
- data/lib/lyra/privacy/subject_records.rb +369 -0
- data/lib/lyra/projection.rb +7 -1
- data/lib/lyra/projections/async_checkpoint.rb +32 -0
- data/lib/lyra/projections/async_projection_job.rb +166 -23
- data/lib/lyra/projections/cached_joins.rb +33 -18
- data/lib/lyra/projections/cached_projection.rb +74 -33
- data/lib/lyra/projections/cached_relation.rb +3175 -321
- data/lib/lyra/projections/default_scopes.rb +254 -0
- data/lib/lyra/projections/deferred_relation.rb +23 -3
- data/lib/lyra/projections/event_store_reader.rb +47 -8
- data/lib/lyra/projections/foreign_keys.rb +92 -0
- data/lib/lyra/projections/lazy_projection.rb +165 -27
- data/lib/lyra/projections/model_projection.rb +165 -14
- data/lib/lyra/projections/rebuild.rb +34 -6
- data/lib/lyra/projections/reference_links.rb +463 -72
- data/lib/lyra/projections/typed_values.rb +629 -0
- data/lib/lyra/projections/unique_indexes.rb +595 -0
- data/lib/lyra/projections/uniqueness.rb +784 -0
- data/lib/lyra/projections/value_digest.rb +93 -0
- data/lib/lyra/purpose_bound_reads.rb +912 -77
- data/lib/lyra/repair.rb +119 -28
- data/lib/lyra/retention.rb +114 -8
- data/lib/lyra/rights/consent_store.rb +415 -0
- data/lib/lyra/rights/erasure_guard.rb +135 -0
- data/lib/lyra/rights/pending.rb +74 -0
- data/lib/lyra/rights/queries.rb +269 -0
- data/lib/lyra/rights/store.rb +512 -0
- data/lib/lyra/rights/subject_keys.rb +224 -0
- data/lib/lyra/rights.rb +281 -0
- data/lib/lyra/saves_without_effect.rb +69 -0
- data/lib/lyra/stream_tail.rb +319 -0
- data/lib/lyra/strict_data_access.rb +906 -107
- data/lib/lyra/temporal.rb +4 -2
- data/lib/lyra/touch_registry.rb +404 -0
- data/lib/lyra/verification/bypass_workflow.rb +452 -42
- data/lib/lyra/verification/rights_workflows.rb +170 -0
- data/lib/lyra/version.rb +1 -1
- data/lib/lyra/visualization/event_graph.rb +1 -1
- data/lib/lyra/visualization/timeline.rb +2 -2
- data/lib/lyra.rb +43 -2
- data/lib/tasks/lyra_mode.rake +107 -8
- data/lib/tasks/lyra_projections.rake +43 -2
- metadata +34 -4
- data/lib/lyra/associations/event_aware.rb +0 -224
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,1915 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [1.0.0] - 2026-11-22
|
|
11
|
+
|
|
12
|
+
### Breaking
|
|
13
|
+
- **Lyra requires Rails 8.1: `rails >= 8.1, < 8.2`** (`lyra.gemspec`), where it declared
|
|
14
|
+
`rails >= 8.0`. Rails 8.0 was never tested: Lyra is developed and tested on Rails 8.1.4.1 (with
|
|
15
|
+
Ruby 4.0.10 and RailsEventStore 3.2.0), and CI tests only the lockfile. The upper bound is new
|
|
16
|
+
because Lyra patches Active Record internals (relations, associations, persistence), which a
|
|
17
|
+
Rails minor release may change, so a future Rails minor needs a Lyra release that is tested on
|
|
18
|
+
it. The `rails_event_store ~> 3.0` and Ruby `>= 4.0` requirements are unchanged. The blog app
|
|
19
|
+
asks for Ruby `>= 4.0`, Lyra's floor; no locked gem version changed.
|
|
20
|
+
|
|
21
|
+
- **A query uses every declared field it names, whatever it returns.** Purpose-bound reads checked
|
|
22
|
+
record loads and `pluck`; aggregates, filters and `exists?` were not checked, so under a purpose
|
|
23
|
+
that does not allow `name`, `Registration.group(:name).count`, `maximum(:name)` and
|
|
24
|
+
`where(name: "Ann").exists?` answered, and `find_by(name: "Ann")` raised for Ann and returned
|
|
25
|
+
`nil` for anyone else: a membership test. Now the declared fields a query names (its select or
|
|
26
|
+
pluck list, the aggregated column, `group`, `where`, `having`, `order`, the conditions of
|
|
27
|
+
`find_by`, class-level too, and `exists?`, a subquery, and the filter of `update_all`,
|
|
28
|
+
`delete_all`, `touch_all`, `delete_by` and `destroy_by`) are checked before it runs, so such code
|
|
29
|
+
under a purpose now raises `PamDsl::PurposeFieldMismatchError` (or, with no purpose under
|
|
30
|
+
`reads_without_purpose :deny`, `PurposeRequiredError`). `count`, `count(:all)` and `size` name
|
|
31
|
+
nothing and are not checked; `count(:name)` uses `name`. The check is split by what can be known:
|
|
32
|
+
a record load has its query's fields checked without the consent step (PAM's new
|
|
33
|
+
`validate_fields!`), then each record returned as before, consent included, for what it holds
|
|
34
|
+
and what the query named; aggregates, `exists?`, `ids`, `pluck` and bulk writes read everyone at
|
|
35
|
+
once (subject `Model$*`) and are refused under a purpose whose basis is consent. A joined
|
|
36
|
+
model's declared fields are checked against its own policy (subject `Other$*`). `find_each` and
|
|
37
|
+
`in_batches` check and record a query once, not per batch. ES-NoProj gives the same answers:
|
|
38
|
+
its relations keep what their query named, and it no longer disagrees with Monitor on
|
|
39
|
+
`exists?(hash)`. Rails' uniqueness validation and an association's `dependent:` handling when a record
|
|
40
|
+
is destroyed (`:delete_all`, `:nullify`, `:delete`, the restrict options, and the lookup of the
|
|
41
|
+
dependents `:destroy` destroys) run as part of the write and are not checked; `find_or_create_by`
|
|
42
|
+
and an application's own `where(author_id: id).delete_all` are. In ES-NoProj a block given to
|
|
43
|
+
`select` (now returning an Array, as in Active Record), `find` or `many?` sees only checked
|
|
44
|
+
records; it saw the cached records unchecked. So do its association readers (`belongs_to`,
|
|
45
|
+
`has_one`), `records`, `each` and `find_each` without a block, and `inspect`, which handed
|
|
46
|
+
records out unchecked. A subquery or CTE reads other people, so what it names is checked whole
|
|
47
|
+
(subject `Model$*`), even on the queried model; a `table.column` inside an SQL fragment belongs
|
|
48
|
+
to that table's model; a column named without its table that the queried model lacks belongs
|
|
49
|
+
to the joined model that has it; a condition on a `belongs_to` association names its foreign
|
|
50
|
+
key; `update_all`'s SET expressions read the columns they name (`"email = name"`); the
|
|
51
|
+
`returning:` columns of `insert_all` and `upsert_all` are read; and `cache_version` and
|
|
52
|
+
`cache_key_with_version` are aggregates. A refused check is never remembered, and the
|
|
53
|
+
application's block inside `find_each` or `in_batches` is checked each time it queries; only the
|
|
54
|
+
loop's own cursor queries are checked once. The purpose and Lyra's internal-read flag are kept
|
|
55
|
+
in Ruby's fiber storage, tied to their thread, so the fiber an Enumerator runs in
|
|
56
|
+
(`relation.to_enum.next`) has them and another thread does not; code that read
|
|
57
|
+
`Thread.current[:lyra_purpose]` uses `Lyra::PurposeBoundReads.current`. With no purpose and `reads_without_purpose :allow` the hooks return before
|
|
58
|
+
looking at the query. Limits: hand-written SQL, string joins, views, functions, triggers, custom
|
|
59
|
+
Arel nodes and CTEs are not seen, and an SQL fragment is matched by the queried model's declared
|
|
60
|
+
names only.
|
|
61
|
+
- **Lyra's escape hatches are Lyra's.** `Lyra.projection_write`, `Lyra::PurposeBoundReads.internal`,
|
|
62
|
+
`Lyra::Projections::ModelProjection.project` and `Lyra::Projections::Rebuild.project_events`
|
|
63
|
+
switch off the checks that keep every write in the log and every read under its purpose. They now
|
|
64
|
+
take a token only Lyra's own code can name, and raise `Lyra::InternalApiError` without it;
|
|
65
|
+
`ModelProjection.project_create`, `project_update` and `project_destroy` are private. Application
|
|
66
|
+
code that wrote without events (seeding, fixtures, a wipe whose history is not wanted) uses
|
|
67
|
+
`Lyra.unrecorded_write(reason:) { ... }`. It runs the block the same way and records that it ran
|
|
68
|
+
(reason, who, the caller) in the stream `Lyra::UnrecordedWrites`; `config.unrecorded_writes =
|
|
69
|
+
:deny` refuses it. The token keeps code from calling them by accident; it is no barrier against
|
|
70
|
+
code out to defeat it. A refused call leaves the flags of the block it ran in as they were.
|
|
71
|
+
- **`DualView#crud_state` reads the row as its caller does,** so under `reads_without_purpose :deny`
|
|
72
|
+
it needs a purpose that allows the declared fields, as any read does; before, it read as Lyra.
|
|
73
|
+
`compare`, `compare_all` and `find_discrepancies` follow it. Lyra's mode gate, the sampler,
|
|
74
|
+
repair still read as Lyra. The dashboard reads under a purpose, as the application does: the
|
|
75
|
+
request's `?purpose=`, else the new `config.dashboard_purpose`; a read the policy refuses answers
|
|
76
|
+
403 and says how to name one.
|
|
77
|
+
- **PAM DSL: a purpose that declares no `basis` is `:undetermined`,** where it was `:consent`.
|
|
78
|
+
The register printed such a purpose as Art. 6(1)(a), and nothing checked it, because no consent
|
|
79
|
+
block asked for one. Access under it now raises `PamDsl::UndeterminedBasisError`, and the
|
|
80
|
+
Article 30 register lists it as a gap, as for a purpose the policy generator writes. A policy
|
|
81
|
+
that relied on the default must declare the basis it means (`basis :consent` included).
|
|
82
|
+
|
|
83
|
+
- **The Art. 15 export no longer has `processing_activities`.** `data_export`'s
|
|
84
|
+
`processing_activities` listed, per event, a purpose guessed from keywords in its
|
|
85
|
+
`metadata[:source]` (Lyra's events carry `lyra_interceptor` or `lyra_command_handler`, so unless
|
|
86
|
+
a `metadata_proc` set another source every event read "Application functionality"), whatever
|
|
87
|
+
the policy declared. The key is gone: the
|
|
88
|
+
copy's purposes are `processing_context`, taken from each record's model's policy (basis,
|
|
89
|
+
recipients, transfers, retention and sources per purpose). Code that read
|
|
90
|
+
`data_export[:processing_activities]` reads `processing_context`. The per-subject Art. 30 method
|
|
91
|
+
`GDPRCompliance#processing_activities`, and the GDPR report's key of that name, are unchanged.
|
|
92
|
+
|
|
93
|
+
- **PAM DSL: the consent grammar refuses what the Regulation does not allow, and a `:consent`
|
|
94
|
+
purpose always needs consent.** On a purpose whose basis is `:consent`, `required! false` (Art.
|
|
95
|
+
6(1)(a)) and `withdrawable! false` (Art. 7(3)) raise `PamDsl::ConsentGrammarError` when the
|
|
96
|
+
purpose or its consent block is declared, in either order, through `define_policy` or the
|
|
97
|
+
methods of `Policy`. `granular!`, recorded and never enforced, is removed: it raises
|
|
98
|
+
`ConsentGrammarError`, and the consent report has no `granular` key. Access under a `:consent`
|
|
99
|
+
purpose now needs an active consent whether or not a consent block is declared for it; with no
|
|
100
|
+
block it passed the legal-basis check unchecked.
|
|
101
|
+
|
|
102
|
+
- **`created_at` and `updated_at` are ordinary columns, and event logs written before 1.0.0 are
|
|
103
|
+
not supported.** Every event Lyra writes now carries the two timestamps like any other column
|
|
104
|
+
(a create's `attributes`, an update's `changes` when they moved), DualView compares them, and
|
|
105
|
+
nothing treats them specially any more. The compatibility code for older logs is gone:
|
|
106
|
+
`Lyra::Event.created_attributes` no longer recovers a create's timestamps from its `changes`
|
|
107
|
+
(Monitor creates written by 0.9.x left them out of `attributes`), and `ModelProjection` no longer
|
|
108
|
+
stamps `created_at`/`updated_at` with the projection time when an event lacks them. An update
|
|
109
|
+
that left `updated_at` alone (`update_columns`, an `update_all` that does not set it) now leaves
|
|
110
|
+
it alone in the projected row too, where before a rebuild stamped the rebuild time and the row
|
|
111
|
+
differed from its events. An application with a log from Lyra 0.9.x should start a new event
|
|
112
|
+
store at 1.0.0 (Genesis imports the existing rows) rather than replay the old one; the per-mode
|
|
113
|
+
event shapes of 0.9.x are not supported.
|
|
114
|
+
|
|
115
|
+
- **`DualView#compare` and `#calculate_differences` return `{}` when the row and its replayed
|
|
116
|
+
state agree,** where they returned `{ no_differences: true }`. An agreement is now an empty hash,
|
|
117
|
+
and every entry of a non-empty one has the same shape, `{ crud:, event_sourced: }`: an attribute
|
|
118
|
+
that differs, or `exists: { crud:, event_sourced: }` (`true`/`false` each) when only one view has
|
|
119
|
+
the record, where it was `{ exists_mismatch: true }`. That entry keeps the result non-empty, so a
|
|
120
|
+
check for `{}` cannot take a missing record for agreement. Code that tested
|
|
121
|
+
`differences[:no_differences]` or compared with `{ no_differences: true }` must test
|
|
122
|
+
`differences.empty?`; code that tested `differences[:exists_mismatch]` must test
|
|
123
|
+
`differences.key?(:exists)`. Attribute differences are keyed by column name as a **string**
|
|
124
|
+
(`differences["email"]`, as Active Record keys `attributes`, and as HABTM link differences already
|
|
125
|
+
were), where they were symbols; the existence entry is the symbol `:exists`, so no column, a
|
|
126
|
+
column named `exists` included, can collide with it. Lyra's own callers
|
|
127
|
+
(`ModeTransition`, `Repair`, `DualView.find_discrepancies`, the recommendations) and the replay
|
|
128
|
+
verifiers of the BPI and Olist examples are updated.
|
|
129
|
+
|
|
130
|
+
### Added
|
|
131
|
+
- **The switch into ES-NoProj refuses a database foreign key it cannot honour.** ES-NoProj leaves
|
|
132
|
+
the tables of monitored models empty: their records live in the event log. A database foreign
|
|
133
|
+
key from a table no monitored model writes into such a table (an unmonitored `Match` whose
|
|
134
|
+
`registration_id` references `registrations`) therefore made every write there that sets the
|
|
135
|
+
column fail with `ActiveRecord::InvalidForeignKey`. `Lyra::ModeTransition.to!` into ES-NoProj now
|
|
136
|
+
refuses such a constraint up front (`ModeTransition::Refused`, as for a default scope ES-NoProj
|
|
137
|
+
cannot evaluate), naming the referencing table and column, the referenced table, the constraint
|
|
138
|
+
and the loaded models that write the table, and the fix: drop the constraint
|
|
139
|
+
(`remove_foreign_key :table, name: "..."`) and keep the reference in the model (`belongs_to`
|
|
140
|
+
validates it), or monitor the referencing model. `force: true` switches anyway. Foreign keys
|
|
141
|
+
between monitored tables, and from a monitored table into an unmonitored one, sit on a table
|
|
142
|
+
that gets no rows in ES-NoProj and pass. The check reads `connection.foreign_keys` for each
|
|
143
|
+
table of each database the monitored models use, at the switch only
|
|
144
|
+
(`Lyra::Projections::ForeignKeys.blocking(models)`). ES-NoProj set by configuration at boot
|
|
145
|
+
(`config.projection_mode = :disabled`, which no switch gates) now gets a warning
|
|
146
|
+
(`Rails.logger.warn`, else `Kernel#warn`) for each such constraint and each default scope
|
|
147
|
+
ES-NoProj cannot evaluate, for the models enrolled by then (with eager loading, all of them);
|
|
148
|
+
the boot goes on, as for a retention rule that cannot act. `Lyra::ModeTransition.check_es_noproj!`
|
|
149
|
+
runs that check on demand and returns the warnings.
|
|
150
|
+
- **A warning for a retention rule that can never act.** `:soft_delete`, PAM's strategy for a
|
|
151
|
+
rule that names none, sets `deleted_at` or `discarded_at`; on a table with neither,
|
|
152
|
+
`Lyra::Retention` skipped every record past its period and kept it, and said so only in each
|
|
153
|
+
action's `detail`. Lyra now warns (`Rails.logger.warn`, else `Kernel#warn`), naming the model,
|
|
154
|
+
its policy and rule, and the fix (add the column, or `on_expiry :anonymize` or `:hard_delete`):
|
|
155
|
+
at boot for the models enrolled by then, and in each run that skips records for it, once per
|
|
156
|
+
model, not once per record. `Lyra::Retention.check_rules!(models:)` makes the boot check on
|
|
157
|
+
demand and returns the names of the models it warned about. Nothing else changes: the records
|
|
158
|
+
are still skipped, and only models with a rule of their own are acted on (the policy's
|
|
159
|
+
`default` period is not executed; now documented).
|
|
160
|
+
- **`config.record_touches`: keep timestamp-only touches out of the log, in Monitor.** An instance
|
|
161
|
+
`touch`, a `touch_all` and a `belongs_to ... touch: true` cascade each append an event, and an
|
|
162
|
+
application that touches a lot fills the log with writes that change only `updated_at` (on the
|
|
163
|
+
Olist replay through Solidus, cascades touching orders and shipments). `true`, the default,
|
|
164
|
+
records every touch as before. `false` leaves out a touch whose only changes are the model's
|
|
165
|
+
update timestamps (`updated_at`, `updated_on`); a touch that also changes another column
|
|
166
|
+
(`touch(:completed_at)`, a `lock_version` increment) is recorded in full, and so is an explicit
|
|
167
|
+
`update_all(updated_at: ...)` or `update_columns(updated_at: ...)`, which are not touches.
|
|
168
|
+
DualView, and with it the sampler, the gate's check and `lyra:repair`, then takes a difference
|
|
169
|
+
confined to those columns as agreement; a difference in any other column is reported with the
|
|
170
|
+
timestamps beside it. Monitor (and Disabled) only: Hijack and event sourcing take the log as the
|
|
171
|
+
record of every change and rebuild rows from it, where a missing touch would rebuild a stale
|
|
172
|
+
`updated_at`, and Hijack passes into event sourcing without a check. So `record_touches =
|
|
173
|
+
false` raises `ArgumentError` in those modes, and while it is false `mode=`, `hijack_enabled =
|
|
174
|
+
true`, `ModeTransition.apply_label` and an ungated `enable_hijack!` or `enable_event_sourcing!`
|
|
175
|
+
raise `ArgumentError`. A process with the option off registers in `lyra_mode_transitions` (kind
|
|
176
|
+
`touch_skipping`: host, pid, since, Lyra's version, last seen; `Lyra::TouchRegistry`) before it
|
|
177
|
+
leaves a touch out, refreshes the entry at most every 60 seconds from the touch path and
|
|
178
|
+
ModeSync's check, ends it when the option is set back to `true` and at exit, and logs a warning
|
|
179
|
+
once. An entry not refreshed for 180 seconds (the database's `clock_timestamp()`) is no longer
|
|
180
|
+
live. Registry writes never join the caller's transaction: each commits on its own, on a second
|
|
181
|
+
connection when the caller's is in a transaction (checked out for at most one second, with a
|
|
182
|
+
two-second `lock_timeout`); when none can be had, the touch is recorded. Each takes the
|
|
183
|
+
registry's advisory lock and re-reads the application's mode under it, and a switch takes the
|
|
184
|
+
same lock for its last look at the registry and its record (a check for its last look and its
|
|
185
|
+
certificate), so a registration and a switch are serialized: either the switch is refused, or the
|
|
186
|
+
registering process ends its entry, records the touch and is fenced. While this process has the
|
|
187
|
+
option off or any process has a live entry, every way into Hijack or event sourcing is refused
|
|
188
|
+
with `ModeTransition::Refused`, naming the processes and the way up: `ModeTransition.to!` (`force: true` included, and so the gated
|
|
189
|
+
`enable_*!` helpers), `check` (at its start and before it certifies; `rake lyra:mode:check`
|
|
190
|
+
prints it and exits non-zero), ModeSync's adoption (the switch stays unseen, and is adopted once
|
|
191
|
+
the entry has ended or expired) and a boot into such a mode; `rake lyra:mode:status` lists the
|
|
192
|
+
live processes. No certificate issued before an entry's latest refresh or end is taken. A process
|
|
193
|
+
with the option off does not boot into an application whose mode is Hijack or event sourcing,
|
|
194
|
+
cannot set the option false at runtime in one, and one that finds itself in such a mode later is
|
|
195
|
+
fenced: each ModeSync check raises, and so does every touch it would have left out, with the save
|
|
196
|
+
that cascades it. `Lyra::Projections::Rebuild.rebuild` is refused (`Rebuild::Refused`) with the
|
|
197
|
+
option off or a live entry, since it would set every touched row
|
|
198
|
+
back. To escalate: restart every process with the option on, run `lyra:repair` in Monitor (the
|
|
199
|
+
check then sees the stale timestamps; the repair records them, one `Updated` event per record),
|
|
200
|
+
then switch. A row with no stream that is touched with the option off is imported by Genesis as
|
|
201
|
+
it then is. Strict mode is unchanged (a `touch` is allowed, `touch_all` refused as
|
|
202
|
+
`update_all`). An unrecorded touch has no event, so no event names it as its cause.
|
|
203
|
+
`LYRA_RECORD_TOUCHES=false` sets it in the Solidus example. Accepts `"true"`/`"false"` strings.
|
|
204
|
+
On 200 Olist orders replayed through Solidus in Monitor, the log falls from 8,691 events (2,547
|
|
205
|
+
of them touches) to 6,331; the 187 touches left are order completions (`touch(:completed_at)`),
|
|
206
|
+
and the oracle (197/200 orders, as in Disabled), the mails and DualView (1,575 records, no
|
|
207
|
+
discrepancy) are unchanged.
|
|
208
|
+
|
|
209
|
+
### Fixed
|
|
210
|
+
- **Every `causation_id` Lyra gives names an event in the log.** A write's scope fixed its event
|
|
211
|
+
id before its callbacks ran, and the writes they made recorded it as their cause. A save that
|
|
212
|
+
changes nothing records no event (see Changed), so its callbacks' writes named an event that
|
|
213
|
+
never existed: replaying 200 Olist orders through Solidus, 203 of 793 causation ids in Monitor,
|
|
214
|
+
ES-Sync and ES-NoProj alike, each a `LineItemUpdated` inside an `Order#save` with nothing to
|
|
215
|
+
change, which still saves its line items (has_many autosave). A write is now the cause of the
|
|
216
|
+
writes made after its own event is in the log; when it records none, they take the cause that
|
|
217
|
+
was in scope when the write began. The writes its `before_*` callbacks make (a `belongs_to`
|
|
218
|
+
autosave, a `dependent: :destroy`) come before anyone knows and name the write, as before. A
|
|
219
|
+
save that changes nothing still records nothing in its stream; when a write made in its
|
|
220
|
+
`before_*` callbacks names it as cause, Lyra records a `SaveWithoutEffect` marker with its id in
|
|
221
|
+
an internal stream, `Lyra::SavesWithoutEffect`, in the write's transaction. So it does for every
|
|
222
|
+
write that ends without its event in the log after naming it: an update or destroy that reaches
|
|
223
|
+
no row (Monitor's stale copy), an aborted save, a command refused after the id was taken
|
|
224
|
+
(Hijack, event sourcing), a write whose Monitor append failed, and a write that raised, which a
|
|
225
|
+
caller may rescue inside an enclosing transaction and commit (then in a savepoint, best effort).
|
|
226
|
+
The marker holds the model, the record's key, the operation and the reason (`no change`,
|
|
227
|
+
`no row`, `not saved`, `append failed`), no attribute value, and `replay: false`; replay, the
|
|
228
|
+
projections, ES-Lazy's catch-up, DualView, the dashboard's event window and the subject reports
|
|
229
|
+
pass it over. A write's scope now wraps its whole save or destroy callback chain, a model's own
|
|
230
|
+
`after_save` and `after_destroy` included: it opens around `create_or_update` and `destroy`, and
|
|
231
|
+
around a destroy chain an application runs itself with `run_callbacks(:destroy)` (a soft delete,
|
|
232
|
+
such as paranoia's). As an `around_save`/`around_destroy` callback it had closed before them
|
|
233
|
+
(Active Model prepends `after_*` callbacks), so their writes (Solidus's `Payment#update_order`)
|
|
234
|
+
had no cause and opened a correlation of their own. Bypass writes (`touch`, `update_columns`)
|
|
235
|
+
and jobs created during a write take their cause the same way. An access-log entry, which
|
|
236
|
+
commits on its own connection and outlives a write that rolls back, names no write in scope: its
|
|
237
|
+
cause is the one from outside every write scope (`Lyra::Causation.outside_write_scopes`), and
|
|
238
|
+
its correlation ties it to the writes. `Lyra::Causation.integrity` reports the causation ids
|
|
239
|
+
that name no event in a store; none do once the transactions commit, except a cause the
|
|
240
|
+
application sets itself and an append lost under Monitor's log-and-continue policy. The
|
|
241
|
+
correlation id the outermost write opens is still its event id when it records nothing: a
|
|
242
|
+
correlation id is a group's label, not a reference.
|
|
243
|
+
- **Hijack no longer keeps the event of a row write that failed.** Hijack appends a write's event
|
|
244
|
+
before its row write. On a model with optimistic locking the row write can raise
|
|
245
|
+
`StaleObjectError` after the append, and a caller that rescued it inside an enclosing
|
|
246
|
+
transaction committed an `Updated` (or `Destroyed`) for a write that never happened: the log and
|
|
247
|
+
the row parted, and DualView reported the record. The append and the row write now share a
|
|
248
|
+
savepoint, which takes the event with it when the row write raises. The row lock taken before
|
|
249
|
+
the append is still held to the end of the transaction when the write succeeds. Models without
|
|
250
|
+
optimistic locking take no savepoint: their row write does not raise for a row that moved.
|
|
251
|
+
- **DualView on a model without `created_at` or `updated_at`.** `DualView#crud_state` read both
|
|
252
|
+
whatever the model had, so a model with only `updated_on`, or no timestamps, raised
|
|
253
|
+
`NoMethodError` in DualView, the mode gate's check and `lyra:repair`. Its `timestamps` now hold
|
|
254
|
+
the model's own timestamp columns (`timestamp_attributes_for_create_in_model` and
|
|
255
|
+
`..._for_update_in_model`), none for a model without.
|
|
256
|
+
- **ES-NoProj refuses an Active Record relation's condition it cannot read, instead of skipping
|
|
257
|
+
it.** A named scope called on a relation, a relation passed to `merge`, and a default scope are
|
|
258
|
+
read as hash conditions on the model's own columns. An equality or `IN` whose left side was not
|
|
259
|
+
a column was skipped: `scope :p, -> { where(Arel.sql("lower(name)").eq("pear")) }` called as
|
|
260
|
+
`Item.where(...).p` returned every record the other conditions selected, and a default scope
|
|
261
|
+
made only of such a condition was taken as evaluable and not applied. A column of another
|
|
262
|
+
table was read as the model's column of the same name, a right side that is not a value
|
|
263
|
+
(another column, an SQL literal) was compared as if it were one, and a column named twice
|
|
264
|
+
(`where(qty: 2).where(qty: 5)`) kept only the last condition. Each now raises
|
|
265
|
+
`UnsupportedQuery`, except the column named twice, whose conditions are both applied, as SQL
|
|
266
|
+
does. Orders likewise: an order on another table's column (`"posts.title"`, an Arel attribute
|
|
267
|
+
of another table) sorted by the model's column of that name, and a direction other than `asc`
|
|
268
|
+
or `desc` (`order(qty: "desc nulls last")`) sorted ascending; both are refused.
|
|
269
|
+
- **ES-NoProj's `exists?(false)` and `exists?(nil)` are `false`, as in Active Record.** They
|
|
270
|
+
answered whether the relation had any record.
|
|
271
|
+
- **ES-NoProj's `limit!` limits the relation it is called on**, as Active Record's; it returned a
|
|
272
|
+
limited copy and left the relation whole. The bang query methods it does not implement
|
|
273
|
+
(`offset!`, `reorder!`, `distinct!`, `group!`, `none!`, ...) are refused, as before.
|
|
274
|
+
- **ES-NoProj's `only` keeps the parts named and drops the rest, as Active Record's**
|
|
275
|
+
(`only(:where)` keeps the conditions and drops the order, limit, select and group; `only(:order)`
|
|
276
|
+
orders every record). It returned the relation unchanged, its order and limit kept. The
|
|
277
|
+
conditions and order are taken off as `unscope` takes them off; `only` of a joined relation
|
|
278
|
+
without its joins is refused (`UnsupportedQuery`).
|
|
279
|
+
- **ES-NoProj's pagination is Kaminari 1.2.2's.** `page(n)` is `limit(size).offset((n - 1) *
|
|
280
|
+
size)`, the size the model's `paginates_per` (else Kaminari's `default_per_page`, else 25), or its
|
|
281
|
+
`max_paginates_per` when that is smaller; a page number below 1, `nil` or not a number is page 1.
|
|
282
|
+
`page(n)` without `per` answered with every record, and the maximum was not applied. `per(m)` is
|
|
283
|
+
the page of that size with the same number, at most `max_per_page:`, the relation's
|
|
284
|
+
`max_paginates_per(n)` (kept on the relation; it set the model's) or the model's; `per(nil)`, a
|
|
285
|
+
negative or a non-numeric size keep the page as it is (they gave an empty page). `padding`,
|
|
286
|
+
`next_page`, `prev_page`, `first_page?`, `last_page?`, `out_of_range?`, `total_pages` and
|
|
287
|
+
`current_page` are Kaminari's (the first five were missing), as are, on a page only,
|
|
288
|
+
`entry_name` (`page_entries_info`'s), `without_count` (its `last_page?` and `out_of_range?`
|
|
289
|
+
look at the page; `total_count` raises Kaminari's error) and the deprecated `current_per_page`.
|
|
290
|
+
`total_count` and `total_pages`
|
|
291
|
+
count the records the relation selects before the window, through the same purpose check and
|
|
292
|
+
rights as any count; they counted the page's records, so a paginated list showed one page of
|
|
293
|
+
one. `Model.page(n)`, Kaminari's own, gives the same page: the relation answers what Kaminari's
|
|
294
|
+
relation modules define (extending it with them was refused when read). `page` no longer
|
|
295
|
+
changes the relation it is called on.
|
|
296
|
+
- **ES-NoProj takes a limit or offset after the relation's conditions and order, as SQL does.**
|
|
297
|
+
A `where`, `where.not`, `order`, scope, `merge`, `select` or `unscope` called after `limit` or
|
|
298
|
+
`offset` acted on the window's records (`Model.order(:code).limit(2).where(status: "draft")`
|
|
299
|
+
filtered the first two records); it now applies before the window. A second `limit` or `offset`
|
|
300
|
+
replaces the first (`limit(1).limit(3)` is three records, `limit(2).offset(1)` records 2 and 3);
|
|
301
|
+
they applied to the window. `limit_value` and `offset_value` are the values given, as in Active
|
|
302
|
+
Record (`limit_value` was the page size, `nil` after `limit`; `offset_value` was 0 without an
|
|
303
|
+
offset). `find`, `find_by` and `exists?` with a condition on a window apply the condition first
|
|
304
|
+
and keep the window's offset, as Active Record's `where(...).take` does. `sum`, `average`,
|
|
305
|
+
`minimum` and `maximum` of a window are Active Record's: its one aggregate row, which the window
|
|
306
|
+
keeps (the aggregate over every record the relation selects) or leaves out (an offset, a limit of
|
|
307
|
+
0: `0` for `sum`, `nil` otherwise); they were computed over the window's records. `or` of two
|
|
308
|
+
windows needs the same window, as in Active Record (an `ArgumentError` otherwise). A grouped
|
|
309
|
+
aggregate of a window, which Active Record limits to some of the groups, is refused. Under a
|
|
310
|
+
right in force, a window's count, pluck, ids, exists? and batch sizes are taken from the records
|
|
311
|
+
the right does not leave out, as SQL takes LIMIT and OFFSET after its WHERE (the window was
|
|
312
|
+
taken first, and then shortened by a record left out of it). `find` with
|
|
313
|
+
several ids (`find(a, b)`, `find([a, b])`) is Active Record's, window included; it raised
|
|
314
|
+
`ArgumentError`.
|
|
315
|
+
- **ES-NoProj's `or` returns the records in the relation's order** (else by primary key), each once;
|
|
316
|
+
it returned this relation's records, then the other's.
|
|
317
|
+
- **ES-NoProj's `reselect` replaces the selected columns, a second `select` adds to the first, and
|
|
318
|
+
`unscope(:select)` and `unscope(:group)` take them off, as in Active Record.** `reselect` did
|
|
319
|
+
nothing and a second `select` replaced the first, so the records came back without the columns
|
|
320
|
+
asked for (`MissingAttributeError`); `unscope(:select)` and `unscope(:group)` were ignored.
|
|
321
|
+
`select` and `reselect` with no column raise Active Record's `ArgumentError`. The relation's
|
|
322
|
+
plan (what `unscope`, `rewhere` and `only` replay) now survives `select`, `group` and
|
|
323
|
+
`create_with`, and a replay keeps them.
|
|
324
|
+
- **ES-NoProj's `unscope(:where)` of a relation filtered by an SQL fragment is refused** (`Model
|
|
325
|
+
.where("qty > ?", 2).unscope(:where)`); it kept the fragment's condition. `where.not` now
|
|
326
|
+
continues the relation's plan, so `unscope`, `rewhere` and `merge` can take it off, as Active
|
|
327
|
+
Record does (they were refused after it). `unscope(where: :owner_id)` takes off a condition
|
|
328
|
+
named by its `belongs_to` association (`where(owner: ann)`); it was kept.
|
|
329
|
+
- **ES-NoProj's `distinct` of selected columns (`select(:status).distinct`, `SELECT DISTINCT`) is
|
|
330
|
+
refused**, in either order and through `merge`: its rows are distinct values, not records, and
|
|
331
|
+
it answered with the records.
|
|
332
|
+
- **ES-NoProj batches as Active Record does.** `find_in_batches`, `find_each` and `in_batches` go
|
|
333
|
+
by primary key (or the `cursor:` columns, in `order: :asc` or `:desc`, from `start:` to `finish:`),
|
|
334
|
+
whatever the relation's order (logged, or `ArgumentError` with `error_on_ignore: true`), keeping its
|
|
335
|
+
limit; they followed the relation's order and took no options. Each `in_batches` batch is the
|
|
336
|
+
relation narrowed to the batch's records, in the relation's order (`load: true`: the batch's
|
|
337
|
+
records). `in_batches` without a block returns Active Record's `BatchEnumerator` (`each`,
|
|
338
|
+
`each_record`, `update_all`, `delete_all`, ...); it raised `LocalJumpError`. The enumerators of
|
|
339
|
+
`find_each` and `find_in_batches` have Active Record's sizes. Counted (those sizes, and
|
|
340
|
+
`in_batches`' batches of keys), the records leave out those a right in force leaves out, before
|
|
341
|
+
the relation's limit, as a count does; a restricted record was counted. Loaded (`find_each`,
|
|
342
|
+
`find_in_batches`, `in_batches(load: true)`, `each_record`, or a batch's records), each is
|
|
343
|
+
checked as a load is, and a restricted record raises, as in Active Record. The batches of a
|
|
344
|
+
relation with no conditions are ranges of keys, as Active Record's are (`use_ranges:`), so a
|
|
345
|
+
record the keys left out is in one, and loading that batch raises as Active Record's does.
|
|
346
|
+
Batches of a relation with an offset are taken as Active Record takes them: it applies the
|
|
347
|
+
offset to each batch's query (see below).
|
|
348
|
+
- **An ES-NoProj bulk write writes what its relation selects when it is called, as Active Record's
|
|
349
|
+
does, and no longer brings back a record destroyed since.** A relation, and each batch
|
|
350
|
+
`in_batches` yields, held the records it selected when it was built, loaded or not, and
|
|
351
|
+
`update_all`, `delete_all`, `touch_all`, `destroy_all`, `update_counters` and `update` wrote
|
|
352
|
+
those. A record deleted in the meantime (by another relation's `delete_all`, a `destroy`, an
|
|
353
|
+
erasure, or during an `in_batches` loop) got an `Updated` event after its `Destroyed` and read
|
|
354
|
+
as live again, so a stale relation could undo a delete or an erasure; `delete_all` appended a
|
|
355
|
+
second `Destroyed`; `destroy_all`, `update` and `update!` raised `Lyra::RecordDestroyedError`
|
|
356
|
+
part-way. A record changed or created since was written or left out by the old selection, and a
|
|
357
|
+
window was not taken again: `order(:code).offset(1).limit(3)`, after one of its records was
|
|
358
|
+
deleted, wrote the two it still held where Active Record writes the next three. A relation now
|
|
359
|
+
keeps how it was built (`Lyra::Projections::Rebuildable`): where it started (every record of
|
|
360
|
+
the model, a foreign-key lookup, an association, a `:through` association with no association
|
|
361
|
+
scope on the way, its default scopes kept as they were built) and the query methods called on it since, with their arguments
|
|
362
|
+
copied when they were called, so that a caller changing an array, hash or string afterwards
|
|
363
|
+
does not change what the write selects, as it does not change Active Record's query. A scope or
|
|
364
|
+
class method called on the relation is the application's code and is not run again: what it
|
|
365
|
+
built is recorded as the query methods it called, with the values it used then (`Time.current`,
|
|
366
|
+
the ids it plucked), as Active Record binds them. A step that changes no selection (`extending`,
|
|
367
|
+
`includes`, `readonly`, ...) is not recorded; one with another block, or an argument that cannot
|
|
368
|
+
be copied, leaves the relation without a recipe. `unscope`, `rewhere`, `except` and `only`,
|
|
369
|
+
which rebuild the relation from its plan, keep the recipe that rebuild records (where the plan
|
|
370
|
+
started, read again), so they too select at write time; they wrote the records the plan held
|
|
371
|
+
when the relation was built, with their old values. A write builds the relation again from the
|
|
372
|
+
event store: the aggregate locks of the records it holds are taken, the lock a single record's
|
|
373
|
+
update or destroy takes, all or none (`Lyra::AdvisoryLock.xact_lock_all`: tried without waiting
|
|
374
|
+
in one statement; when one is held, the others are given back and the next try first waits for
|
|
375
|
+
that one, so two writers over overlapping records wait for each other instead of deadlocking),
|
|
376
|
+
then the selection is read; a record that came to match since is locked in turn and the
|
|
377
|
+
selection read again (up to four times, after which a record not yet locked is left out, as if
|
|
378
|
+
it had matched after the write). A destroyed record is not in the selection. The count returned
|
|
379
|
+
is Active Record's. A relation keyed by primary key (`where(id: ...)`) is read again by those
|
|
380
|
+
keys only. `destroy_all`, `update` and `update!` load the selection as Active Record's do, or,
|
|
381
|
+
once the application has loaded the relation (`to_a`, `each`, ...), work on the records loaded,
|
|
382
|
+
as Active Record's do: there a record destroyed since is left out of `destroy_all` (Active
|
|
383
|
+
Record also runs `destroy` on it, reaching no row, and returns it) and takes `update`'s
|
|
384
|
+
attributes in memory without being written. `destroy_all` delivers its records as a load does,
|
|
385
|
+
so a record under a restriction raises `RestrictedProcessingError` before any is destroyed, as
|
|
386
|
+
in Monitor; it was destroyed. It checks each record under its lock before its destroy, which
|
|
387
|
+
checks again after its `before_destroy` callbacks (one that deletes the record is refused with
|
|
388
|
+
`Lyra::RecordDestroyedError`, and rolled back). Each batch of `in_batches` is the relation
|
|
389
|
+
narrowed to the batch's keys, as in Active Record, so a record of a later batch that no longer
|
|
390
|
+
matches is not written by the batch's bulk write (and the batches are read as the loop reaches
|
|
391
|
+
them: see below). A relation without a recipe (made from given records, such as a
|
|
392
|
+
`dependent: :delete_all` or `:nullify`'s children, or from a scope or argument that could not
|
|
393
|
+
be recorded) writes the records it holds, less those whose streams end in a destroy, read under
|
|
394
|
+
the same locks (`Lyra::StreamTail.destroyed_streams`: one query for all of them where each last
|
|
395
|
+
event is a create or an update). The cost is on writes only: one locking statement (in a
|
|
396
|
+
savepoint) and one read of the selection per bulk write, the read of the records with the given
|
|
397
|
+
keys for a keyed relation; one read of the streams for a relation without a recipe;
|
|
398
|
+
`destroy_all` adds two locks and two stream reads per record. On 2,047 records, a
|
|
399
|
+
`where(id: x).update_all` takes about what it took (165 ms against 156), a `where(code: ...)`
|
|
400
|
+
one about three times (one more read of the model's records at the write). Reads are unchanged
|
|
401
|
+
but for recording each query method's call. With `lock_aggregate_before_append` off nothing is
|
|
402
|
+
locked or checked, as for a single record. ES-Sync, Hijack and Monitor write through SQL and
|
|
403
|
+
were not affected; for ES-Async and ES-Lazy, see below.
|
|
404
|
+
- **ES-Async bulk writes leave out a row whose destroy is waiting for its projection job.** The
|
|
405
|
+
row stays in the table until the job runs, and `update_all`, `delete_all` and `touch_all` wrote
|
|
406
|
+
it and appended an `Updated` after the `Destroyed` (or a second `Destroyed`), bringing the record
|
|
407
|
+
back. The write now reads the keys it matches, takes their aggregate locks (all or none), then
|
|
408
|
+
locks the rows (and Genesis admits rows that predate Lyra), reads the rows' streams in one query
|
|
409
|
+
and leaves out a destroyed record's row from the write, the events and the count, as Active
|
|
410
|
+
Record's reaches no row of it. A row that came to match while the locks were taken gives the row
|
|
411
|
+
locks back and the keys are read again (up to three times). `dependent: :nullify` leaves out a
|
|
412
|
+
destroyed child the same way: no event, and its row is not written. The same
|
|
413
|
+
`bulk_write_lock_limit` applies. On 1,000 rows the write costs what it did (4,013 queries against
|
|
414
|
+
4,005, 3.05 s). For ES-Lazy, see below.
|
|
415
|
+
- **A record's locks are taken in one order: its aggregate lock, then its row.** ES-Async's and
|
|
416
|
+
ES-Lazy's writes through callbacks take the aggregate (advisory) lock only, Hijack's and
|
|
417
|
+
ES-Sync's the row's only; the writes that skip callbacks (`update_columns`, `touch`, `delete`,
|
|
418
|
+
ES-Async's `update_all` and `delete_all`) and an erasure now take the aggregate lock first, then
|
|
419
|
+
write or lock the row. An erasure took the row first, and with the checks above a bulk write
|
|
420
|
+
took the rows first too, which deadlocked against a write holding the aggregate lock (an ES-Async
|
|
421
|
+
`update_all` against concurrent `update_columns`: 232 deadlocks in 300 writes; none now). The
|
|
422
|
+
order is written down at `Lyra::AdvisoryLock`.
|
|
423
|
+
- **A stale copy's `update_columns`, `touch` and `delete` no longer bring back a destroyed record
|
|
424
|
+
or log a second destroy.** A copy read before another copy, a relation's `delete_all` or an
|
|
425
|
+
erasure destroyed the record recorded an `Updated` after the `Destroyed` (`update_columns`,
|
|
426
|
+
`touch`) or a second `Destroyed` (`delete`), in ES-NoProj, ES-Async and ES-Lazy, and `delete`
|
|
427
|
+
did in Monitor, Hijack and ES-Sync as well. In the first three they now read the record's stream
|
|
428
|
+
under its aggregate lock first; elsewhere `delete` locks the row first. When the record is gone
|
|
429
|
+
they record nothing and return what Active Record returns for a row already gone: `false` from
|
|
430
|
+
`update_columns` and `touch` (the attributes are set in memory), the frozen record from
|
|
431
|
+
`delete`. A row still in the table (in ES-Async, before the destroy's job deletes it) is not
|
|
432
|
+
written either: Active Record's own write runs in a savepoint that is rolled back.
|
|
433
|
+
- **ES-NoProj's class-level counters write the record.** `Model.increment_counter`,
|
|
434
|
+
`decrement_counter`, `update_counters(id, ...)` (with `touch:`) and a record's `increment!` and
|
|
435
|
+
`decrement!` went to Active Record's SQL on the empty table: they returned 0 and recorded no
|
|
436
|
+
event. They now write as a relation's `update_counters` does (locks, the counters' current
|
|
437
|
+
values, an event per record), reading only the records with those keys, and return the number of
|
|
438
|
+
records written. With optimistic locking the lock version moves too, as Active Record's
|
|
439
|
+
`Locking::Optimistic.update_counters` moves it, so a copy read before raises `StaleObjectError`
|
|
440
|
+
on save. A counter cache, which Active Record's associations keep through these, is still left
|
|
441
|
+
as it is: ES-NoProj does not keep counter caches.
|
|
442
|
+
- **ES-NoProj's `update_counters` on a relation adds to the counters' current values, and two of
|
|
443
|
+
them no longer deadlock.** It added to the values the relation had read, so an update made since
|
|
444
|
+
(`update!(qty: 100)`, then `+1` gave the old value plus one, not 101) or a concurrent increment
|
|
445
|
+
was lost (two threads, 25 increments each over seven records, ended 175 short); Active Record's
|
|
446
|
+
SQL (`qty = COALESCE(qty, 0) + 1`) adds to the row as it is. It also wrote one `update_all` per
|
|
447
|
+
set of current values, each taking its records' locks, so two writers whose relations ordered
|
|
448
|
+
the records differently took them in opposite orders and deadlocked (30 of 50 calls). It now
|
|
449
|
+
takes every selected record's lock (all or none), reads the counters from the event store under
|
|
450
|
+
those locks, and writes the groups without locking or reading again.
|
|
451
|
+
- **`config.bulk_write_lock_limit` (default 5,000) caps the records one bulk write locks.** An
|
|
452
|
+
ES-NoProj bulk write (`update_all`, `delete_all`, `touch_all`, `update_counters`, a `dependent:
|
|
453
|
+
:delete_all` or `:nullify`, and a `destroy_all` inside a transaction already open), and an
|
|
454
|
+
ES-Async or ES-Lazy `update_all`, `delete_all` or `touch_all`, holds one advisory lock per record until its
|
|
455
|
+
transaction ends, and PostgreSQL's lock table is shared by every session: past about 12,800
|
|
456
|
+
locks with the default `max_locks_per_transaction` and `max_connections` 100 (about 23,200 with
|
|
457
|
+
200), every session's writes failed with "out of shared memory". A larger write now raises
|
|
458
|
+
`Lyra::BulkWriteLockLimitError` (its `count` and `limit`) before it locks or writes anything,
|
|
459
|
+
and a parent's destroy whose dependent write is refused rolls back. Write in batches
|
|
460
|
+
(`in_batches(of: n) { |batch| batch.update_all(...) }`), each of which commits and gives its
|
|
461
|
+
locks back on its own, but not inside a transaction already open, which holds them all until it
|
|
462
|
+
ends; `nil` removes the limit.
|
|
463
|
+
- **ES-Lazy writes that skip callbacks see a destroy committed after their catch-up.** A bulk
|
|
464
|
+
write (`update_all`, `delete_all`, `touch_all`), `update_columns`, `touch`, `delete` and a
|
|
465
|
+
`dependent: :nullify` bring the tables up to date and then write; a destroy committed in
|
|
466
|
+
between left its row in the table, and the write wrote it and appended an `Updated` after the
|
|
467
|
+
`Destroyed` (or a second `Destroyed`). They now take the records' aggregate locks after the
|
|
468
|
+
catch-up and read their streams (as in ES-NoProj and ES-Async), leaving such a record out, and
|
|
469
|
+
replay onto its row each record whose stream has an event the tables have not had yet, so an
|
|
470
|
+
update committed in between is the row the write starts from (it was lost under the write, or
|
|
471
|
+
`update_all("tag = tag || ...")` read the old value). The
|
|
472
|
+
write's own reads no longer catch up again: a catch-up there waited for the catch-up lock while
|
|
473
|
+
the write held rows that a catch-up running elsewhere waited for, a deadlock that bulk writes
|
|
474
|
+
and writes that skip callbacks met against catch-ups under load before 1.0.0 too (9 in a run of
|
|
475
|
+
a bulk write, writes through and past callbacks and a catch-up in a loop, against none to 1
|
|
476
|
+
now: a catch-up writing rows in the log's order and a bulk write locking them in the table's can
|
|
477
|
+
still meet on rows alone). A catch-up that a catch-up under way refused no longer clears the
|
|
478
|
+
under-way flag of the outer one.
|
|
479
|
+
- **ES-NoProj's batches are read when the loop reaches them, as Active Record queries each.**
|
|
480
|
+
`in_batches`, `find_in_batches` and `find_each` on a relation that can be built again (see
|
|
481
|
+
above) read each batch then: the records after the last batch's last cursor values, in cursor
|
|
482
|
+
order, from `start:` to `finish:`, `of` of them, so a record inserted, changed or deleted during
|
|
483
|
+
the loop is seen as Active Record sees it (a later batch no longer yields a record deleted or no
|
|
484
|
+
longer matching, and yields one inserted). `use_ranges`, `order: :desc`, `load: true`,
|
|
485
|
+
`each_record`, a custom cursor and the rights (loads raise, counts leave out) are as before. A
|
|
486
|
+
relation with no recipe takes its batches from the records it holds, as before.
|
|
487
|
+
Each batch costs one read: where the relation starts from every record of the model and the
|
|
488
|
+
cursor is the integer primary key, of the model's keys and then of the next records by key (about
|
|
489
|
+
the batch's own cost); otherwise of the whole relation. Locally, `in_batches(of: 1000)` over
|
|
490
|
+
10,000 records takes 2.8 s against 1.3 s (one read of every record up front), `find_each` 2.3 s
|
|
491
|
+
against 0.9 s.
|
|
492
|
+
- **ES-NoProj's batches of a relation with a limit or an offset are read as the loop reaches
|
|
493
|
+
them, and take the limit and offset as Active Record does.** They read every record up front,
|
|
494
|
+
and an offset was refused. Each batch is read when the loop reaches it, as above; a limit stops
|
|
495
|
+
the loop after its records (a batch reads `of` records, or what is left of the limit when that
|
|
496
|
+
is less), and the offset skips that many records after the last batch's cursor in every batch's
|
|
497
|
+
query, as Active Record applies `OFFSET` to each (`offset(1).find_each(batch_size: 2)` skips a
|
|
498
|
+
record before each batch). An `in_batches` batch is the relation narrowed to the batch's keys
|
|
499
|
+
with its limit and offset, as Active Record's `rewhere(cursor => values)` is, so the offset
|
|
500
|
+
skips records of the batch again, and its writes select so at write time (a batch's keys were
|
|
501
|
+
those of its window); a batch of a relation that starts from every record of the model is
|
|
502
|
+
built again from its own keys (`Rebuildable::Keyed`), not from every record. An
|
|
503
|
+
`in_batches(load: true)` batch holds the records read for it; its query methods, aggregates,
|
|
504
|
+
finders (`find_by`, `find`, `find_or_create_by`, `find_sole_by`, ...) and `update_all`,
|
|
505
|
+
`delete_all` and `touch_all` are that narrowed relation's, read again from the event store each
|
|
506
|
+
time, as Active Record's batch keeps its query and runs it (`CachedRelation#query_relation`).
|
|
507
|
+
Active Record's batch query of a relation with an offset has no `ORDER BY`, so which records
|
|
508
|
+
it skips is the database's plan (an index scan, or the table's physical order after an
|
|
509
|
+
update): `in_batches` of such a relation without an order raises `UnsupportedQuery`, and so do
|
|
510
|
+
the queries of its `load: true` batches (their records are read in cursor order, and given);
|
|
511
|
+
`find_each` and `find_in_batches`, ordered by the cursor, answer. `in_batches(use_ranges:
|
|
512
|
+
true)` with an offset raises it too (each range is read with the `OFFSET` and no `ORDER BY`),
|
|
513
|
+
and with `limit(0)` (Active Record's query has `OFFSET -1`, which the database rejects). The
|
|
514
|
+
enumerators' sizes take the offset too. The batches of a loaded relation (after `load`, `to_a`,
|
|
515
|
+
`each`, ...) are its loaded records, from `start:` to `finish:`, in cursor order, as Active
|
|
516
|
+
Record's are; they were read again. Each `in_batches` batch of it is loaded with its records,
|
|
517
|
+
and its query methods, aggregates, finders and `update_all`, `delete_all` and `touch_all` are
|
|
518
|
+
the relation's, as Active Record's batch is the relation loaded with them (each batch's
|
|
519
|
+
`update_all` writes every record the relation selects); `destroy_all` and `update` work on the
|
|
520
|
+
batch's records. The batches read by key took every record a step brought in: an `or` (whose
|
|
521
|
+
other relation is read whole) gave records of other chunks and from before the cursor, so a
|
|
522
|
+
batch repeated records and, without a limit, `find_each` and `find_in_batches` never ended
|
|
523
|
+
(since the batches were read as the loop reaches them, above). Each chunk now keeps only its
|
|
524
|
+
own records, which a step keeps or leaves out as it would among every record; `distinct` and
|
|
525
|
+
`group` are read whole. Cost, locally, 10,000 records: `limit(5000).find_each(batch_size:
|
|
526
|
+
1000)` takes 1.36 s against 0.84 s (11 queries against one read of every record up front),
|
|
527
|
+
`in_batches(of: 1000)` 1.34 s against 0.85 s, as the batches without a limit cost (one read
|
|
528
|
+
of the keys and one of the batch's records per batch); `find`, `where`, `to_a`, `count` and
|
|
529
|
+
`first` are unchanged (same queries, time and allocations).
|
|
530
|
+
- **ES-NoProj's `load`, `reload`, `reset` and `loaded?` answer as Active Record's.** They raised
|
|
531
|
+
`UnsupportedQuery`. `load` loads the records as any read delivers them (the purpose check, each
|
|
532
|
+
record checked, a restricted one raising `RestrictedProcessingError`), passes each to its block,
|
|
533
|
+
and returns the relation, `loaded?` true after (before: false on `Model.all`, nil on a relation
|
|
534
|
+
derived from another, as in Active Record); a relation loaded already is not loaded again.
|
|
535
|
+
`reset` reads the relation again from the event store (Rebuildable), as a relation built at that
|
|
536
|
+
moment, and leaves it not loaded; `reload` is `reset` and `load`. A relation that cannot be
|
|
537
|
+
built again (one of given records) raises `UnsupportedQuery` there. A loaded relation's reads
|
|
538
|
+
do not read again, its `destroy_all` and `update` work on its loaded records and its
|
|
539
|
+
`update_all`, `delete_all` and `touch_all` select at write time, as before and as in Active
|
|
540
|
+
Record. `destroy_all` then resets the relation, as Active Record's does: it no longer holds the
|
|
541
|
+
records it destroyed (it held them, still loaded). A bang query method (`where!`, `order!`,
|
|
542
|
+
`limit!`, ...) on a loaded relation raises `ActiveRecord::UnmodifiableRelation`, as in Active
|
|
543
|
+
Record; it changed the loaded records. `DeferredRelation#loaded?`, which said whether an
|
|
544
|
+
association's records had been read, is `records_read?`.
|
|
545
|
+
- **ES-NoProj's `find_or_create_by` and `find_or_create_by!` create as `create_or_find_by` does,
|
|
546
|
+
as Active Record 8.1's.** A record a unique index of the table would conflict with is found
|
|
547
|
+
(or, outside the relation, `RecordNotFound`) instead of created again.
|
|
548
|
+
- **ES-NoProj's `excluding`, `without` and `in_order_of` are query methods, as Active Record's.**
|
|
549
|
+
They were Enumerable's, which returned an Array, so a write on them raised `NoMethodError`.
|
|
550
|
+
`excluding(*records)` (and `without`) is `where.not(primary key: ids)` of the records and of the
|
|
551
|
+
relations given; a record or relation of another model raises Active Record's `ArgumentError`.
|
|
552
|
+
`in_order_of(column, values, filter: true)` is a `where` of the column (`IS NULL` for `nil`; no
|
|
553
|
+
`where` with `filter: false`), which `rewhere`, `unscope(where: ...)`, `unscope(:where)` and
|
|
554
|
+
`except(:where)` take off as in Active Record, and an order by the column's position among the
|
|
555
|
+
values after any order the relation has, as Active Record's `CASE` added by `order!`, which
|
|
556
|
+
`unscope(:order)` and `reorder` take off; the values are cast as Active Record casts them (an
|
|
557
|
+
enum's label to its value, one the column cannot hold left out, `nil` a position of its own),
|
|
558
|
+
and none left is `none`. The column may name the model's table (`"items.code"`); an SQL
|
|
559
|
+
expression raises `UnsupportedQuery`. Both keep the relation's recipe, so `update_all`,
|
|
560
|
+
`delete_all`, `touch_all` and `destroy_all` on them select at write time, with a limit, offset,
|
|
561
|
+
`where` or order chained before or after.
|
|
562
|
+
- **ES-NoProj's `merge` takes the other relation's conditions where both name a column, and its
|
|
563
|
+
further arguments as options, as Active Record (7 and later) does.** `where(status: "active")
|
|
564
|
+
.merge(where(status: "draft"))` is the draft records; it was the records with both, none. A
|
|
565
|
+
condition named by a `belongs_to` association (`where(owner: ann)`) is one on its key. This
|
|
566
|
+
relation's conditions on such a column are taken off as `unscope` takes them off, a `where.not`
|
|
567
|
+
and an earlier merge included; where they cannot be (after an SQL fragment or a join, an
|
|
568
|
+
association's key, a condition on several columns at once, `where([:a, :b] => ...)`, or an `or`
|
|
569
|
+
of conditions on that one column) the merge is refused. An `or` over different columns is kept,
|
|
570
|
+
as Active Record keeps it. `merge(r1, r2)` intersected with `r2` too, and `merge(r, rewhere:
|
|
571
|
+
true)` was refused; both are now `merge(r1)`. `merge(nil)` and `merge(false)` raise
|
|
572
|
+
`ArgumentError` (they returned the relation), as does a value that is not a relation; `merge`
|
|
573
|
+
of an Array is the loaded records both have, and of a Proc the Proc run on the relation. The
|
|
574
|
+
other relation's limit and offset (an event-store relation's window, or an Active Record
|
|
575
|
+
relation's) replace this one's, its group and select columns are added, its distinct taken and,
|
|
576
|
+
if it reorders, its order replaces this one's; they were dropped (an Active Record relation's
|
|
577
|
+
limit was taken inside this one's). A relation joined on one side, or both to the same
|
|
578
|
+
associations, merges as the join's rows (it was refused); a condition on a joined table that the
|
|
579
|
+
other relation names too, and relations joined to different associations, are refused.
|
|
580
|
+
- **ES-NoProj's `relation.update(id, attributes)` and `update!(id, ...)` are Active Record's
|
|
581
|
+
`Model.update(id, ...)`:** the model's record with that id is found and written through the
|
|
582
|
+
model's own read and write paths (its default scope, the purpose check, an event), whether or
|
|
583
|
+
not the relation selects it, as Active Record does. They raised `RecordNotFound` for a record
|
|
584
|
+
outside the relation.
|
|
585
|
+
- **ES-NoProj's `update_counters({})` raises Active Record's `ArgumentError`** ("Empty list of
|
|
586
|
+
attributes to change"); it returned 0. `update_counters(touch: true)` with no counter touches the
|
|
587
|
+
records, as in Active Record; it did nothing.
|
|
588
|
+
- **A scope answers the same under ES-NoProj on the model and on a relation.** A scope called on
|
|
589
|
+
a relation now runs its body on that relation, as Active Record runs it (it was built as an Active
|
|
590
|
+
Record relation and the relation narrowed by it). A scope with an Arel condition
|
|
591
|
+
(`where(arel_table[:qty].eq(2))`) called on the model was refused while the same scope on a
|
|
592
|
+
relation was answered; both are answered, by building that scope as Active Record builds it. A
|
|
593
|
+
scope whose body returns a value (a count, a record) returns it, as in Active Record; on the model
|
|
594
|
+
it returned the relation, on a relation it was refused. One that returns an Active Record
|
|
595
|
+
relation of the model answers with that relation's records (it returned the relation the scope was
|
|
596
|
+
called on). A scope's `where.not` of a list is now answered on a relation as `where.not` is (it
|
|
597
|
+
was refused). As in Active Record, the body runs with no current scope (a body reading
|
|
598
|
+
`Model.where(...)` read the relation the scope was called on), and a bang method in it changes a
|
|
599
|
+
copy of that relation (`limit!(1)` limited the caller's). A `scoping` block's relation is seen
|
|
600
|
+
inside a fiber or an enumerator in it, as Active Record's current scope is (they read every
|
|
601
|
+
record).
|
|
602
|
+
- **`update_all(foreign_key: nil)` on an Active Record relation of a monitored model under
|
|
603
|
+
ES-NoProj** (counted as a dependent nullify) published events for the records the event store
|
|
604
|
+
holds with that foreign key, every other condition of the relation skipped: `where(author_id: 1,
|
|
605
|
+
status: "published")` nullified the author's drafts too in the log. The records are now those the
|
|
606
|
+
relation's where clause selects, all of it, read from the event store (not the table first); a
|
|
607
|
+
condition that cannot be read refuses the write (`UnsupportedQuery`). The events and the write
|
|
608
|
+
are one transaction where events are required, so a write that fails after its events were
|
|
609
|
+
published takes them back (a refused write used to leave its records nullified in the log).
|
|
610
|
+
`author.articles.update_all(author_id: nil)` now nullifies under ES-NoProj what Monitor does.
|
|
611
|
+
- **A bulk write with an offset in Monitor and the other table modes** (`order(:code).offset(5)
|
|
612
|
+
.delete_all`, a page's `update_all`) wrote no row: the write by the snapshot's keys applied the
|
|
613
|
+
offset again. It now writes the rows the relation selects, as Active Record does.
|
|
614
|
+
- **A polymorphic `has_one ... as:` under ES-NoProj matches the owner's type**, as the `has_many`
|
|
615
|
+
path does; it returned a record of another type of owner with the same id.
|
|
616
|
+
- **A `has_one` with a scope of its own is narrowed by it under ES-NoProj.** `has_one
|
|
617
|
+
:selected_rate, -> { where(selected: true) }` and `has_one :latest, -> { order(id: :desc) }`
|
|
618
|
+
answered with whichever record of the owner's the event store found first; the scope is now
|
|
619
|
+
applied to the owner's records and the first in its order is the answer, as Active Record's
|
|
620
|
+
`LIMIT 1`. A scope ES-NoProj cannot evaluate raises `UnsupportedQuery`. The record is checked
|
|
621
|
+
as a loaded row, as a `has_one` without a scope's is; it was returned unchecked.
|
|
622
|
+
- **A `has_one :through` a plain `has_one` reaches its target as Active Record's join does under
|
|
623
|
+
ES-NoProj.** It followed only the intermediate record the `has_one` reads, where the join reaches
|
|
624
|
+
the targets of every intermediate record the owner has (those the `has_one`'s scope selects),
|
|
625
|
+
narrowed by the association's own scope. When either scope orders, the first target in that
|
|
626
|
+
order is the answer; in no order, the one target there is (it could be `nil` before, when the
|
|
627
|
+
intermediate record read had none), and with more than one ES-NoProj now refuses
|
|
628
|
+
(`UnsupportedQuery`) rather than answer with one of them. The intermediate records are read as
|
|
629
|
+
Lyra's own read, without a purpose check, as the join reads them in SQL; the `has_one` reader
|
|
630
|
+
used before was checked, and under a purpose that does not allow the intermediate model's fields
|
|
631
|
+
raised `PamDsl::PurposeFieldMismatchError`.
|
|
632
|
+
- **ES-NoProj reads a `:through` association's intermediate records afresh, as Active Record's
|
|
633
|
+
join reads the tables.** A `has_one :through` a `belongs_to`, and a `has_many :through`, took
|
|
634
|
+
them from the owner's association when it was loaded: after the intermediate record changed
|
|
635
|
+
through another instance (moved to another owner), the answer came from the stale copy. Reading
|
|
636
|
+
the target also left the intermediate association loaded on the owner, which the join does not.
|
|
637
|
+
Every record on the way is now read on an association object of its own (from the event store,
|
|
638
|
+
or with SQL for a model not read from events), and the owner's associations are left as they
|
|
639
|
+
were. As the join, such a read reads the intermediate records each time: a loaded intermediate
|
|
640
|
+
association no longer saves the reads.
|
|
641
|
+
- **ES-NoProj checks only the records a `:through` association returns, as Active Record's join
|
|
642
|
+
does.** The intermediate records reached through a `belongs_to` or a `has_many` were checked as
|
|
643
|
+
loaded rows (a purpose that does not allow the intermediate model's fields raised
|
|
644
|
+
`PamDsl::PurposeFieldMismatchError`), and every target was checked before the association's
|
|
645
|
+
own scope, order and limit: a restricted target the scope leaves out (`scoped_tags`,
|
|
646
|
+
`tags.where(...)`, a limit, the other candidates of a `has_one :through`) raised where Monitor
|
|
647
|
+
answers. The records on the way are now read as Lyra's own read; the records returned are
|
|
648
|
+
checked, and a query on the association (`count`, `exists?`, `pluck`) is checked as a query
|
|
649
|
+
through the intermediate models, as the join's SQL is (a restriction in force for one of them
|
|
650
|
+
refuses it, as in Monitor).
|
|
651
|
+
- **An association scope that unscopes brings back a `:through` association's target that the
|
|
652
|
+
target's default scope leaves out, under ES-NoProj.** `has_one :x, -> { with_discarded },
|
|
653
|
+
through: ...` answered `nil` for a soft-deleted target, and a `has_many :through` left it out,
|
|
654
|
+
because the source association had already applied the target's default scope; and a `has_many
|
|
655
|
+
:through` whose source unscopes (`belongs_to :variant, -> { with_discarded }`) applied the whole
|
|
656
|
+
default scope again afterwards. The targets are now read without the default scope and narrowed
|
|
657
|
+
by it as the join applies it: less what the source's scope or the association's own unscopes.
|
|
658
|
+
The intermediate models' default scopes apply as before, as in the join, except on a step on
|
|
659
|
+
the target's own table (`cat.grandchildren` through a soft-deleted child), which the join
|
|
660
|
+
aliases and applies none to. `unscope(where: :col)` called on such an association drops the
|
|
661
|
+
column from the intermediate models' default scopes too in Active Record (a soft-deleted
|
|
662
|
+
intermediate record's targets come back); ES-NoProj read those records through their default
|
|
663
|
+
scopes, and refuses an unscope of a column one of them conditions (`UnsupportedQuery`).
|
|
664
|
+
- **A `:through` association reaches every candidate of each `has_one` on the way under
|
|
665
|
+
ES-NoProj, in the join's order.** Through another `has_one :through`, a `has_one :through`
|
|
666
|
+
followed only the target that association read (and was refused when that one was ambiguous);
|
|
667
|
+
through a `has_one` source with several records, or a `has_one` step on a model not read from
|
|
668
|
+
events (read with its `LIMIT 1`), it answered with the one record read. The candidates are now
|
|
669
|
+
all read, as the join reaches them. The targets were in the intermediate records' order, so an
|
|
670
|
+
order of the source's scope or the association's own came second (`has_one :latest_tag, through:
|
|
671
|
+
:first_item` answered the first item's latest tag, where the join orders by the tag's label
|
|
672
|
+
first and the item's position after). They are now ordered as the join orders them: the
|
|
673
|
+
target's default scope, the next intermediate model's default scope, the association's own
|
|
674
|
+
scope and the source's (an order on the intermediate model's columns included, as Solidus's
|
|
675
|
+
`Product#prices` has), then the intermediate step's scope; an `unscope(:order)` in a scope on
|
|
676
|
+
the target takes the default scopes' and the steps' orders off, as in the join. In no order at
|
|
677
|
+
all, the targets are in primary key order, as Active Record's `first` takes them. Where the
|
|
678
|
+
join's order cannot be reproduced (an order on another table, or a default scope ordering a
|
|
679
|
+
step past the first while an association scope orders too), a read that depends on the order
|
|
680
|
+
(loading, `first`, `last`, `take`, a `limit` or `offset`, `pluck` and `ids`, a `has_one
|
|
681
|
+
:through`'s answer) is refused (`UnsupportedQuery`) when there is more than one record;
|
|
682
|
+
`count`, `size`, `length`, `exists?` and sums answer, and so does a read after `reorder`. A
|
|
683
|
+
`has_one :through` answers with the first target in that order; in no order, with the one
|
|
684
|
+
target there is, and more than one is refused. `first`, `last` and `take` on a `has_many
|
|
685
|
+
:through` follow the same order; they took the first record by primary key.
|
|
686
|
+
- **A `has_many :through` that reaches a target through two intermediate records has it twice
|
|
687
|
+
under ES-NoProj**, as Active Record's join has (`author.article_authors` through two articles);
|
|
688
|
+
ES-NoProj kept one, and counted one. `-> { distinct }` keeps one, as before.
|
|
689
|
+
- **`update_all` on a `has_many` under ES-NoProj** (`author.articles.update_all(author_id: nil)`,
|
|
690
|
+
a `has_many :through` too) writes the records the association selects, as Active Record's runs
|
|
691
|
+
through the association's scope, recording an event for each, and returns how many; it ran
|
|
692
|
+
against the table and was refused. A record a `has_many :through` reaches twice is written once,
|
|
693
|
+
as SQL updates its row once (also for `delete_all`, `touch_all` and `destroy_all` on such a
|
|
694
|
+
relation).
|
|
695
|
+
- **Inside a transaction that recorded rights, a restriction committed before it began is in
|
|
696
|
+
force.** A write of rights drops its subject from the store's shared snapshot until the next
|
|
697
|
+
read, and a connection with rights of its own in its open transaction never read the snapshot
|
|
698
|
+
again, so a subject restricted just before such a transaction was not restricted inside it
|
|
699
|
+
(`active_for`, `active`, `keys_in_force`). Such a connection now reads the rights of every other
|
|
700
|
+
subject into a snapshot of its own, at most once per refresh interval and checking scope, and its
|
|
701
|
+
own subjects as its transaction left them; the shared snapshot is never written from it (under
|
|
702
|
+
`REPEATABLE READ` it reads the database as its transaction began). Outside such a transaction
|
|
703
|
+
nothing changes, and no query is added. `keys_in_force` inside a transaction that lifted a
|
|
704
|
+
restriction no longer lists its subject, as `active` does not.
|
|
705
|
+
- **ES-NoProj answers a `has_one :through` by following the through association.** The lookup
|
|
706
|
+
used the source's foreign key on the target: Solidus's `InventoryUnit has_one :order, through:
|
|
707
|
+
:shipment` was asked as `Spree::Order.find_by(order_id: <the unit's id>)`, a column only
|
|
708
|
+
`spree_shipments` has. 0.9.x returned `nil` for such an association under ES-NoProj, so
|
|
709
|
+
`carton.orders` was empty and Solidus sent no shipped email. 1.0.0 reads the intermediate record
|
|
710
|
+
and then the source association, each as that association reads under ES-NoProj, and narrows
|
|
711
|
+
the result by the association's own scope (a scope it cannot evaluate raises
|
|
712
|
+
`UnsupportedQuery`). A `has_many :through` whose source is a `has_one :through`
|
|
713
|
+
(`carton.orders`) follows it too. With `source_type:` (a polymorphic source), a `has_one` or
|
|
714
|
+
`has_many :through` follows only the intermediate records whose type column names that class,
|
|
715
|
+
as Active Record's join does; a `has_many :through` returned records of whatever class the
|
|
716
|
+
column named.
|
|
717
|
+
- **ES-NoProj let Rails' uniqueness validation pass every duplicate.** `validates :x, uniqueness:
|
|
718
|
+
true` asks the table whether another row holds the value; in ES-NoProj a monitored model's table
|
|
719
|
+
is empty, so every value passed, and the table's unique index could not catch the duplicate
|
|
720
|
+
either, since no row is written. The testbed saved a second bank-transfer payment with an RF code
|
|
721
|
+
already issued (`BankPayment#remittance_info`) and a second web invoice with a series and number
|
|
722
|
+
already used (`Finance::WebInvoice`). The relation Rails builds for the check is now answered
|
|
723
|
+
from the records rebuilt from events (`Lyra::Projections::Uniqueness`), as Active Record and the
|
|
724
|
+
database answer it from the table: Rails still builds it, so `scope:` (columns and
|
|
725
|
+
associations), `conditions:` (hash conditions, `where.not`, ranges, `or`, named scopes, a lambda
|
|
726
|
+
taking the record), `case_sensitive:`, `allow_nil`/`allow_blank`, the record itself left out on
|
|
727
|
+
update by its primary key (a custom one too), `Model.unscoped` (a soft-delete default scope is
|
|
728
|
+
not applied, as in Rails) and, under STI, the type condition of the class that declared the
|
|
729
|
+
validation all follow Rails' own construction. Values compare through the attribute's type (a
|
|
730
|
+
decimal `"250.00"` is `250`, an enum's label, a datetime in another zone), with SQL's NULL
|
|
731
|
+
semantics; `LOWER(column) = LOWER(value)` and a `citext` column's equality are worked out by the
|
|
732
|
+
database's own `LOWER` (texts it lowered once are remembered). A form that cannot be evaluated exactly (an SQL fragment
|
|
733
|
+
or a subquery in `conditions:`, a join, `LIKE`, a string column with a collation of its own)
|
|
734
|
+
raises `UnsupportedQuery` instead of passing or refusing the value on a guess. The check is
|
|
735
|
+
still exempt from the purpose check. Other modes keep Active Record's relation and the table's
|
|
736
|
+
answer. A uniqueness check is a read and then a write in every mode; where the table modes
|
|
737
|
+
have the unique index to refuse the second of two concurrent writers, ES-NoProj now takes a
|
|
738
|
+
transaction-level advisory lock on the model, attribute and value before the check reads
|
|
739
|
+
(PostgreSQL; under `case_sensitive: false` or on `citext`, on `LOWER(value)` as the database
|
|
740
|
+
computes it), so the second writer's check waits for the first's commit and then finds its
|
|
741
|
+
record. It is taken only inside a transaction (`save`, `create`, `update`) and only by a save
|
|
742
|
+
that claims a value (a new record, a changed attribute or scope column, or a check with
|
|
743
|
+
`conditions:`): a save that keeps its value does not wait for, or deadlock with, other writers.
|
|
744
|
+
It closes the race under READ COMMITTED, not for a REPEATABLE READ transaction whose snapshot
|
|
745
|
+
predates the other's commit. Each check reads every record of the model; a per-process copy
|
|
746
|
+
stamped as the projection cache's entries are reads only the streams written since the last
|
|
747
|
+
check (24 ms instead of 94 ms at 3,000 records in the test environment). A check answers from
|
|
748
|
+
the entries it read itself, so another thread replacing one meanwhile cannot show it that
|
|
749
|
+
thread's uncommitted state; the copies are kept under 32 MB (models with more than 20,000
|
|
750
|
+
streams are read through the projection cache), expire after an hour as the cache's memory
|
|
751
|
+
entries do, and are rebuilt when a model's columns change.
|
|
752
|
+
- **An erasure in another process did not reach warm ES-NoProj reads.** `Lyra::Erasure` rewrites
|
|
753
|
+
the record's events in place and appends an `ErasureApplied`. The eraser drops its own cache
|
|
754
|
+
entries; another process brought its entry up to date by replaying only the events after the
|
|
755
|
+
entry's stamp, the `ErasureApplied`, which changes no attribute, and kept the erased values:
|
|
756
|
+
`find_by`, `where` and the uniqueness check answered with the erased e-mail until the entry
|
|
757
|
+
expired. An entry with an `ErasureApplied` after its stamp is now rebuilt from the rewritten
|
|
758
|
+
stream.
|
|
759
|
+
- **ES-NoProj's lists, `character(n)` columns and serialized attributes compared as the table does
|
|
760
|
+
not.** A hash condition's list of two or more values is an `IN` of the values the type binds:
|
|
761
|
+
Active Record leaves out one it binds as NULL (`"abc"` on an integer, an enum's unknown label,
|
|
762
|
+
`""` on a boolean), and ES-NoProj kept it as NULL, so `where.not(qty: [9, "abc"])` returned no
|
|
763
|
+
row instead of every row but 9. An enum's unknown label alone is bound as `IS NULL`, as Active
|
|
764
|
+
Record binds it; an integer the enum has no label for (`where(status: 5)`) is bound as `= 5`,
|
|
765
|
+
which no row equals, NULL or not, and stays a member of a list (`where.not(status: [5, 7])`
|
|
766
|
+
returned no row). A `character(n)` value compares without its trailing spaces (`'ab'` is `'ab '`
|
|
767
|
+
in PostgreSQL). A serialized attribute compares as the text it is stored as
|
|
768
|
+
(`{"a"=>1,"b"=>2}` is not `{"b"=>2,"a"=>1}` there), and a Hash given for one is its value, not a
|
|
769
|
+
condition on a table named after it (`where(prefs: {...})` was refused). Hash conditions in
|
|
770
|
+
`where`, `where.not`, `find_by`, `exists?` and `count`, and the uniqueness check, all follow.
|
|
771
|
+
- **ES-NoProj compared a condition's value as given, not cast through the column's type.**
|
|
772
|
+
Active Record binds a condition's value through the attribute's type before the database
|
|
773
|
+
compares it; ES-NoProj compared the Ruby values. `find_by(amount: 250.0)` on a decimal column
|
|
774
|
+
compared the event's `"250.0"` with the Float and found nothing, with a BigDecimal too (the
|
|
775
|
+
testbed created a second bank-transfer payment code instead of reusing the first), and
|
|
776
|
+
`where(transaction_type: :card)` matched no row, as did a Symbol on any string column or enum.
|
|
777
|
+
Conditions in `where`, `where.not`, `find_by`, `exists?`, `count` and the joined conditions now
|
|
778
|
+
bind the value as Active Record binds it (the attribute's type serializes it) and read it back as
|
|
779
|
+
a record holds it, in equality, `IN` lists and ranges (each end): a decimal, float or integer
|
|
780
|
+
given as a String, Float or BigDecimal (`3.5` on an integer is 3, `250.004` on a decimal(10, 2)
|
|
781
|
+
is 250.00); a date given as a String or Time; a `datetime`, `timestamptz` or `time` given as a
|
|
782
|
+
String or Date, read in the database's clock (UTC) as Active Record binds it, not in
|
|
783
|
+
`Time.zone`, and a `time` compared as its time of day in UTC; a boolean given as `"t"`, `1`,
|
|
784
|
+
`"0"`; an enum, integer- or string-backed, given as its label, a Symbol or its column's value
|
|
785
|
+
(`where(status: "0")`); a deterministically encrypted attribute as its ciphertext compares
|
|
786
|
+
(`downcase: true` included); an alias for its attribute. A value the type binds as NULL (`"abc"`
|
|
787
|
+
or `""` on an integer column, `""` on a boolean) matches nothing, a value out of an integer
|
|
788
|
+
column's range matches nothing (and leaves a range end open), and NULL is unknown as in SQL:
|
|
789
|
+
`where.not(tag: "x")` and `where.not(amount: 250)` no longer return the rows whose column is
|
|
790
|
+
NULL. An equality on a non-deterministically encrypted attribute matches no row, as on the
|
|
791
|
+
table, and a range, a pattern or an SQL fragment on one is refused (`UnsupportedQuery`). SQL
|
|
792
|
+
fragments (`where("at >= ?", date)`) bind as Active Record quotes: a date or a string on a
|
|
793
|
+
datetime in the database's clock (the testbed's refund list filtered `requested_at` by a date
|
|
794
|
+
in `Time.zone` "Athens"), a number not rounded to the column's scale, and a value the database
|
|
795
|
+
rejects (`"qty >= ?", 2.5` on an integer column) refused. A condition on a name that is not an
|
|
796
|
+
attribute (`where(id: …)` where the primary key is `code`) is refused, as the table rejects
|
|
797
|
+
it; it matched nothing. An order puts `false` before `true` (they compared as equal) and orders
|
|
798
|
+
an enum by the value its column holds, not by its label. A record rebuilt from events also read
|
|
799
|
+
an integer-backed enum wrong: the event holds the label, which was read as the column's
|
|
800
|
+
integer, so every record had the first label (`0`); and a serialized attribute (`serialize
|
|
801
|
+
:prefs, coder: JSON`) failed to load. Both are now turned into the column's value, as a
|
|
802
|
+
projection writing the row does, once when the record's cache entry is built (the entries'
|
|
803
|
+
format is new: `CachedProjection::CACHE_VERSION` 3), with an STI subclass's own enums and
|
|
804
|
+
serialized attributes, and also in `Lyra::Temporal` reads and the rights' subject lookup. A
|
|
805
|
+
record that cannot be built from its events raises; it was left out of the answer. Strings are
|
|
806
|
+
still ordered by code point, not by the database's collation.
|
|
807
|
+
- **Retention did nothing in ES-NoProj.** `Lyra::Retention` (and `lyra:retention:apply`) listed
|
|
808
|
+
the model's ids from its streams but loaded each record from its table, which ES-NoProj leaves
|
|
809
|
+
empty or stale: no record was found, a dry run listed nothing and no rule was applied, and
|
|
810
|
+
`:hard_delete` and `:soft_delete` would have failed reloading the record from its table. In
|
|
811
|
+
ES-NoProj retention now finds the records, their anchor column (`created_at` or
|
|
812
|
+
`config.retention_anchors`) included, in the event store, with no default scope applied (a
|
|
813
|
+
soft-deleted record is still due), and acts on them there: `:hard_delete` destroys the record
|
|
814
|
+
rebuilt from its events (a destroy event) and `:soft_delete` updates it (an update event). The
|
|
815
|
+
other modes are unchanged.
|
|
816
|
+
- **The Art. 15 copy read a column an association owns as a link to another model too.** The
|
|
817
|
+
`<type>_id` convention of `Lyra::Privacy::SubjectLinks` also read `owner_id` of
|
|
818
|
+
`belongs_to :owner, class_name: "Person"` as a link to a model named `Owner`, and a polymorphic
|
|
819
|
+
`owner_id` as one whatever `owner_type` said, so the export for `Owner` 13 included the note
|
|
820
|
+
of `Person` 13. A column an association owns now links only to that association's class (a
|
|
821
|
+
polymorphic one's, only with its type column naming the subject's class), and the convention
|
|
822
|
+
reads only columns no association owns, for a subject type that is a monitored model or names
|
|
823
|
+
no model at all.
|
|
824
|
+
- **ES-NoProj reads apply the model's default scope.** Active Record builds every read of a
|
|
825
|
+
table from the model's `default_scope`; ES-NoProj reads, rebuilt from the event log, left it
|
|
826
|
+
out, so soft-deleted records (`default_scope { where(deleted_at: nil) }`, as paranoia, discard
|
|
827
|
+
and Solidus's `Spree::SoftDeletable` use) were found, counted and returned by associations.
|
|
828
|
+
`find`, `find_by`, `find_by!`, `exists?`, `where`, `all` (and so `count`, `pluck` and the other
|
|
829
|
+
calculations), `first`, `last` and named scopes now read through the default scope, as do
|
|
830
|
+
`belongs_to` and `has_one` readers and `has_many` readers, counts, sizes and collection queries
|
|
831
|
+
(through the target's default scope less what the association's scope unscopes, as Active
|
|
832
|
+
Record's association scope). A default `order` orders the results; on a `CachedRelation` an
|
|
833
|
+
`order` now adds to the existing order and `reorder` replaces it, as in Active Record, and
|
|
834
|
+
`first`/`last` follow the relation's order (`last(n)` returns the last n in that order, no
|
|
835
|
+
longer reversed). `Model.unscoped` read the table, which ES-NoProj leaves empty or stale; in
|
|
836
|
+
application code it now returns every record from the event store, and reads inside
|
|
837
|
+
`Model.unscoped { ... }` (and `reload`) leave the default scope out. Active Record's internals
|
|
838
|
+
and Lyra's own code keep Active Record's relation from `Model.unscoped`, also through a
|
|
839
|
+
model's own `def self.unscoped = super`. Scopes that take the default scope off
|
|
840
|
+
(`with_deleted`, `with_discarded`, `unscope(where: :deleted_at)`) and `rewhere` work after
|
|
841
|
+
`where`, `order` and named scopes, on the model and on an association, as on Active Record
|
|
842
|
+
(the relation replays its conditions without what is taken off); **after a `limit`,
|
|
843
|
+
`where.not` or an SQL fragment they raise `UnsupportedQuery`**, since the records they would
|
|
844
|
+
bring back are no longer in the relation. **A default scope ES-NoProj cannot evaluate (an SQL
|
|
845
|
+
string, a join, an ordering by an expression) now raises `UnsupportedQuery` on every read of
|
|
846
|
+
the model**, rather than answer with the records it leaves out, and
|
|
847
|
+
`Lyra::ModeTransition.to!` refuses the switch into ES-NoProj while a monitored model has one
|
|
848
|
+
(`force: true` overrides). An association's scope is evaluated on each read, so one that
|
|
849
|
+
decides at run time (`-> { Current.admin? ? with_deleted : all }`) is honoured. A find by
|
|
850
|
+
primary key reads one record even under an ordered default scope. Models without a default
|
|
851
|
+
scope read as before.
|
|
852
|
+
- **ES-NoProj relations no longer act on records their query did not select.** A
|
|
853
|
+
`CachedRelation` answers exactly or raises `UnsupportedQuery` before anything is written.
|
|
854
|
+
`merge` and `merge!` returned the relation unchanged, so `Model.unscoped.merge(Model.where(...))
|
|
855
|
+
.delete_all` deleted every record; they now evaluate the other relation (its conditions,
|
|
856
|
+
order and limit, or the records of an event-store relation) and refuse another model's.
|
|
857
|
+
`where!` with anything but hash conditions kept every record and now
|
|
858
|
+
raises; `exists?(["title = ?", x])` ignored its condition and now evaluates it as `where`
|
|
859
|
+
does (or refuses it). Relation methods Lyra did not implement ran on a bare relation of the model, past the
|
|
860
|
+
relation's conditions: `update` and `update_counters` changed every record, `first_or_create!`
|
|
861
|
+
and `create_or_find_by!` created records without the conditions, `create_with` was dropped and
|
|
862
|
+
`to_sql` raised `NoMethodError`. `update`, `update!`, `update_counters` (with `touch:`),
|
|
863
|
+
`first_or_initialize`, `first_or_create(!)`, `create_or_find_by(!)` (a record is found instead
|
|
864
|
+
when a unique index of the table would refuse it, as Active Record finds it on the violation)
|
|
865
|
+
and `create_with` now work on the relation's records and conditions, and `sole` and
|
|
866
|
+
`find_sole_by` raise `RecordNotFound` or `SoleRecordExceeded` as Active Record does; `lock`, `annotate`,
|
|
867
|
+
`optimizer_hints` and `strict_loading` are accepted and do nothing (writes take Lyra's
|
|
868
|
+
aggregate lock); `to_sql` and every other Active Record relation method raise
|
|
869
|
+
`UnsupportedQuery` before running. A model's own class methods called on a relation run with
|
|
870
|
+
it as the current scope, as Active Record runs them: their reads and writes act on its
|
|
871
|
+
records.
|
|
872
|
+
- **PAM DSL prints a retention period given in plain seconds correctly.** With ActiveSupport
|
|
873
|
+
loaded, as in a Lyra application, an Integer number of seconds (`keep_for 86400`,
|
|
874
|
+
`default 7 * 365 * 86400`, or an Integer from configuration) was read as years: 86400 printed
|
|
875
|
+
"0 years" in the Article 30 register, the policy summary and the retention check, and the
|
|
876
|
+
policy comparator printed 365 days of seconds as "365 days" and seven years of seconds as
|
|
877
|
+
"6 years". Plain seconds are now worded the same with and without ActiveSupport, with the
|
|
878
|
+
polyfill's fixed-length units (1 month = 30 days, 1 year = 365 days), rounded down to the
|
|
879
|
+
largest whole unit: 3600 is "1 hour", 86400 "1 day", 30 days "1 month", 365 days "1 year".
|
|
880
|
+
`ActiveSupport::Duration` values (`1.day`, `7.years`) print as before. Retention enforcement
|
|
881
|
+
was not affected: it already read plain seconds as seconds in both modes.
|
|
882
|
+
- **Lyra's GDPR retention check dropped a policy's retention given in plain seconds** and
|
|
883
|
+
applied the configured or default period (7 years) instead; it now keeps a plain number of
|
|
884
|
+
seconds as it keeps an `ActiveSupport::Duration`.
|
|
885
|
+
- **An erased record's row kept personal values inside `json`, `jsonb` and array columns that
|
|
886
|
+
its events had lost,** so it no longer replayed to its events. This happened with or without
|
|
887
|
+
`everywhere`. The erasure rewrote the events recursively but replaced only top-level string
|
|
888
|
+
columns in the row. `Lyra::Erasure` now scrubs every column of the row with the rule its events
|
|
889
|
+
get: the record's own row with the key and value rule, another record's row reached by
|
|
890
|
+
`everywhere` with the value rule only, so a third party's email under an `email` key there is
|
|
891
|
+
kept. `json`, `jsonb` and array columns and serialized attributes are recursed into and written
|
|
892
|
+
back whole in the same `UPDATE`. A `text` column holding JSON is still compared as a string, as
|
|
893
|
+
in its events. A column declared personal that holds a hash or an array is replaced whole, in
|
|
894
|
+
the row and in the events alike (each side of a change too): `nil`, or, when it is NOT NULL,
|
|
895
|
+
its own default as the database stores it (`[]`, `{}`); before, the events kept it, or got its
|
|
896
|
+
elements replaced one by one. Inside a hash, the strings that are identifying for the field's type (an
|
|
897
|
+
email for `:email`, a phone number for `:phone`) count as the person's values, and those under a
|
|
898
|
+
nested personal field; their copies elsewhere in the record, and with `everywhere` in other
|
|
899
|
+
records, are erased too. Labels such as `"home"` are not. An earlier erasure's `"erased:<id>"`
|
|
900
|
+
is no one's value, so a second erasure neither reports nor searches for it. The row is scrubbed by
|
|
901
|
+
the attribute's type, as its events are, so `hstore` columns and attributes cast as json are
|
|
902
|
+
recursed into too. An
|
|
903
|
+
encrypted attribute is compared in clear in its events, as in its row, and a replacement is
|
|
904
|
+
encrypted again; before, its ciphertext stayed in the events and replayed back to the value.
|
|
905
|
+
The erasure moves the row's optimistic-lock column, as any write does, so a copy loaded before
|
|
906
|
+
it fails with `StaleObjectError` instead of saving the erased values back; the move is recorded
|
|
907
|
+
as an ordinary update of that column alone (metadata `source: "lyra_erasure"`), so the row still
|
|
908
|
+
equals its replay. Before, the lock moved in the row only, and the row no longer replayed to its
|
|
909
|
+
events. Virtual attributes are no longer written, which made the erasure fail. An
|
|
910
|
+
event that is not a CRUD event (an application's own, in the record's stream) gets the key rule
|
|
911
|
+
at its top level, so a personal field's hash there is replaced whole. With `everywhere`, a
|
|
912
|
+
record is listed in `result.copies` only if its row no longer holds the value anywhere. A value
|
|
913
|
+
held as a JSON key, or inside a longer string, is not erased, and the record is listed in the
|
|
914
|
+
new `result.left_behind` as `{ record:, columns: }`, never with the value; `lyra:erase` prints
|
|
915
|
+
it. Before, such a record was listed as erased. The erased record's own row is checked the same
|
|
916
|
+
way, with or without `everywhere`, and listed first. A primary or foreign key is never rewritten;
|
|
917
|
+
one that holds an erased value is reported in `left_behind` as `"Model <key withheld>"` (and
|
|
918
|
+
`result.key_withheld`), and no result, reason or task output prints it. With `everywhere`, each
|
|
919
|
+
record found is locked and its stream read again before it is erased, so a copy written during
|
|
920
|
+
the search is not missed; candidates are found without case, and a `citext` copy differing in
|
|
921
|
+
case is reported. The new `result.not_searched` names the encrypted columns of monitored
|
|
922
|
+
models, which a text search of the log cannot read. In ES-Async the lock moves from the version
|
|
923
|
+
the log last recorded.
|
|
924
|
+
- **`everywhere` erased a third party's own record when the person's data named them.** In the
|
|
925
|
+
person's own record a value under a personal field's name is the person's (the key rule), so a
|
|
926
|
+
manager's email nested inside the person's JSON (`{"manager" => {"email" => …}}`), or a friend's
|
|
927
|
+
email at the top of another event in the person's stream (an invitation), was erased there and
|
|
928
|
+
then searched for in other records, the third party's own included. Now only the person's own
|
|
929
|
+
values drive the search: those of the row's personal fields and at the top of a personal field in
|
|
930
|
+
the record's own CRUD events (an attribute, either side of a change, a payload entry), with the
|
|
931
|
+
strings inside a structure declared personal taken by its type as before. The others are still
|
|
932
|
+
erased in the person's record, as its events are, and no longer anywhere else.
|
|
933
|
+
- **`everywhere` missed rows that held a value their events did not.** The search reached rows
|
|
934
|
+
only through events, so a row given the value by raw SQL, written before monitoring, or changed
|
|
935
|
+
by a write Lyra could not intercept kept it, and nothing reported it; and a record reached
|
|
936
|
+
through an event that mentioned the value only inside a longer string kept a whole copy in its
|
|
937
|
+
row. With `everywhere`, the tables of monitored models with a policy are now searched directly,
|
|
938
|
+
one query per table that returns keys only (string and text columns by equality, `json`/`jsonb`
|
|
939
|
+
as text by a `LIKE` on the value as a JSON string, string arrays with `= ANY`, `hstore` values);
|
|
940
|
+
not in ES-NoProj, where the tables are absent or stale. Every record reached has its row scrubbed
|
|
941
|
+
by the value rule, whether or not its events held the value whole. When only the row held it,
|
|
942
|
+
its `ErasureApplied` says `row_only: true` and it is listed in the new
|
|
943
|
+
`result.rows_without_events` (its row already differed from its replay: `lyra:repair` in Monitor
|
|
944
|
+
or Disabled, a rebuild of its table in the other modes; the task says which). A row with no
|
|
945
|
+
stream is erased first and then imported the Genesis way, so its `Imported` event holds the
|
|
946
|
+
replacement, never the value (an `ErasureApplied` alone would have started a stream Genesis never
|
|
947
|
+
imports); if a Genesis run imported it first, that stream is rewritten instead. A key column is
|
|
948
|
+
still never rewritten, and a row holding a value only as a JSON key or in another case in a
|
|
949
|
+
`citext` column is reported in `left_behind`. The log is searched once for all the values (it
|
|
950
|
+
was searched again per value to count holders), and candidate events are read in batches.
|
|
951
|
+
- **`everywhere`'s threshold counted streams, not people's records.** A value was taken as shared
|
|
952
|
+
when more than `max_copies` other streams held it, so one person's many records (and a record's
|
|
953
|
+
reference links) counted against it. Holders are now counted by record, through events (each
|
|
954
|
+
confirmed by an exact match) and rows alike, and the person's linked records, found by the rule
|
|
955
|
+
of the subject-access copy (`Lyra::Privacy::SubjectLinks`: any `belongs_to` to the erased
|
|
956
|
+
record's model naming it, by id or by its `primary_key:` column, a polymorphic one, a
|
|
957
|
+
`<model>_id` column), do not count. `max_copies:` must be a positive Integer (`ArgumentError`
|
|
958
|
+
before anything is written; a String failed half-way before).
|
|
959
|
+
- **`everywhere` rewrote Lyra's own record streams.** An event in a stream that was not a monitored
|
|
960
|
+
record's was rewritten when it held an erased value, so a consent recorded under an email as its
|
|
961
|
+
subject key (`Lyra::Consent$<email>`) lost it, and the consent store, rebuilt from those events,
|
|
962
|
+
lost the consent. Restrictions and objections (`Lyra::Rights`), consent (`Lyra::Consent`), the
|
|
963
|
+
access log and its erasures and the subject-access exports, with their per-subject streams, are
|
|
964
|
+
now never rewritten by `everywhere`, nor counted as holders.
|
|
965
|
+
- **Genesis's lock was per class, not per table.** An import through an STI subclass did not wait
|
|
966
|
+
for one through its base class. The lock is now named after the table's stream owner (the STI
|
|
967
|
+
base class), which the erasure of a row with no stream also takes.
|
|
968
|
+
- **Genesis imported a value the model cannot represent as the model read it.** A row holding a
|
|
969
|
+
value outside a string or integer enum's mapping (`refund_requests.status = '2'`) was imported
|
|
970
|
+
with `status: nil`: the `Imported` event disagreed with its row, and the disagreement surfaced
|
|
971
|
+
later and elsewhere (ES-Lazy's catch-up failed with `PG::NotNullViolation`). Genesis now checks
|
|
972
|
+
every attribute of every row it imports: the value the model reads, written back by the model's
|
|
973
|
+
type, must be the value stored, compared as the column's own type reads both or as the database
|
|
974
|
+
would be sent them. `'1'` and `1`, time zones and timestamp precision, a time of day, a decimal's
|
|
975
|
+
trailing zeros, jsonb key order or a normalized attribute's untrimmed legacy value are no
|
|
976
|
+
difference; an enum value outside the mapping, a lossy attribute type (`attribute :n, :integer`
|
|
977
|
+
over `10.50`), an inet value's host bits (which Rails drops; in `inet[]` arrays too) and a value
|
|
978
|
+
out of the model's integer range are. Whatever the type, a stored value the model writes back
|
|
979
|
+
as NULL is lost, in a nullable column too: a JSON `null` in a json/jsonb column, an empty range,
|
|
980
|
+
a blank or `'null'` JSON text or an empty YAML hash in a serialized column. A serialized,
|
|
981
|
+
encrypted or JSON attribute's text is otherwise one of many encodings of its value and is not
|
|
982
|
+
compared (a value of another class is). Columns the model ignores are imported as stored.
|
|
983
|
+
- By default a model holding such rows is refused: every row is checked before anything is
|
|
984
|
+
written, and `Lyra::Genesis::Refused` lists model, id, attribute, stored value and value read
|
|
985
|
+
(at most `Genesis::LISTED`, 20, with the count; a value the privacy policy declares, or an
|
|
986
|
+
encrypted one, is not shown). Nothing of the model is imported; inside a caller's transaction
|
|
987
|
+
the import is a savepoint, so a caller that rescues the refusal commits nothing of it. A
|
|
988
|
+
refused first use is remembered for `Genesis::REFUSAL_RECHECK` (30) seconds from the scan
|
|
989
|
+
that found it, so repeated uses do not scan the table again; `lyra:genesis` always scans.
|
|
990
|
+
- With `config.genesis_unrepresentable = :skip` (or `unrepresentable: :skip`, or
|
|
991
|
+
`UNREPRESENTABLE=skip` for `lyra:genesis`) those rows are left without a stream, the rest is
|
|
992
|
+
imported, and the rows are listed in the result (`Lyra::Genesis.import`, new, returns a
|
|
993
|
+
`Result`; `import_all` still returns the count) and in the log. A write to a skipped row,
|
|
994
|
+
through callbacks or bypassing them (`update_columns`, `touch`, `update_all`, `touch_all`,
|
|
995
|
+
`delete_all`, `upsert_all`, ...), and an ES-NoProj `find` of it, import it once the model can
|
|
996
|
+
represent it and raise `Refused` (naming the rows) until then, so its stream never begins with
|
|
997
|
+
the write's event. A skipped row is therefore fixed in the database; the next use of it, or
|
|
998
|
+
`lyra:genesis`, imports it. A process knows only the rows it skipped itself. `dependent:
|
|
999
|
+
:nullify` over a skipped child refuses the parent's destroy where events are required; in
|
|
1000
|
+
Monitor that child's event is left out (logged) and the other children still get theirs.
|
|
1001
|
+
- A write that bypasses callbacks now also runs the model's first use, as a write through
|
|
1002
|
+
callbacks does: on a model not yet imported in the process, `update_all` and the like import
|
|
1003
|
+
the rows that predate Lyra before recording their events.
|
|
1004
|
+
- A gated switch into Hijack or event sourcing is refused with such rows as discrepancies,
|
|
1005
|
+
whatever the setting (`Report#unrepresentable` counts them); `lyra:genesis` lists a refused
|
|
1006
|
+
model's rows and exits 1. `Genesis.import_one` (an erasure of a row with no stream) logs such
|
|
1007
|
+
a value and imports the row: the erasure does not fail on it.
|
|
1008
|
+
- `Lyra::Repair` wrote what the model read too (an `Imported` event for a row without events,
|
|
1009
|
+
an `Updated` event for the changed columns). It now applies the same check
|
|
1010
|
+
(`Genesis.unrepresentable`) under the same setting: by default `Lyra::Repair::Unrepresentable`
|
|
1011
|
+
(a `Repair::Refused`, with the listing) is raised before anything is written; with `:skip`
|
|
1012
|
+
(`Repair.run(unrepresentable: :skip)`, `UNREPRESENTABLE=skip` for `lyra:repair`) those
|
|
1013
|
+
records stay out of line, the rest is repaired, and `Repair::Result` (`skipped`,
|
|
1014
|
+
`unrepresentable`) and `lyra:repair` list them; a dry run lists them.
|
|
1015
|
+
- The error of work run on a connection of its own (`Lyra::OwnConnection`) is raised in the
|
|
1016
|
+
caller only, no longer also printed by its thread.
|
|
1017
|
+
- **An erasure could miss an update that committed while it ran.** It read the record's stream
|
|
1018
|
+
before its transaction and took the row lock only when it wrote the row, after rewriting the
|
|
1019
|
+
events, so a Monitor update that committed in between appended an event the erasure never saw,
|
|
1020
|
+
and that event kept the value. `Lyra::Erasure.erase!` now takes the row lock, then the record's
|
|
1021
|
+
advisory lock (the locks a writer holds, in the order writers take them), and only then reads
|
|
1022
|
+
the stream, all in one transaction; everything derived from the stream is computed under the
|
|
1023
|
+
locks, so a refusal there rolls back with nothing changed. A write already in flight finishes
|
|
1024
|
+
first; one that starts later waits and is new data. In ES-NoProj, with no row, only the advisory
|
|
1025
|
+
lock is taken.
|
|
1026
|
+
- **An access-log entry was lost when the reader's transaction rolled back,** though the read had
|
|
1027
|
+
already returned its value. `Lyra::AccessLog` now appends on a connection of its own
|
|
1028
|
+
(`Lyra::OwnConnection`), so the entry survives the rollback; an access that cannot be recorded
|
|
1029
|
+
still does not go ahead. Inside a transaction this costs a second connection per access, so size
|
|
1030
|
+
the pool for it (twice the threads avoids any wait): when no connection comes free within the
|
|
1031
|
+
pool's checkout timeout, or the connection is pinned (Rails' transactional tests), the entry is
|
|
1032
|
+
written inline, in the reader's transaction, and lost if that rolls back.
|
|
1033
|
+
- **PAM DSL works without ActiveSupport again.** Loaded on its own, with its standard-library
|
|
1034
|
+
polyfill, the Article 30 register raised `NoMethodError` (`String#presence` is ActiveSupport),
|
|
1035
|
+
and JSON export (`Reporter#export_json`, `GDPRCompliance#portable_export`) relied on
|
|
1036
|
+
ActiveSupport having loaded `json`. The polyfill now provides `presence` and treats a
|
|
1037
|
+
whitespace-only string as blank, as ActiveSupport does, and both files require `json`. Behaviour
|
|
1038
|
+
under ActiveSupport is unchanged. `rake test:all` now runs the PAM DSL suite twice, with
|
|
1039
|
+
ActiveSupport and with the polyfill (`rake test:both`), so CI covers both.
|
|
1040
|
+
- **A model whose single primary key is not `id`** (`self.primary_key = "code"`, a string or
|
|
1041
|
+
uuid column, no `id` column) is supported in every mode. `Lyra::DualView` looked the row up
|
|
1042
|
+
with `find_by(id:)`, so `DualView#compare`, `compare_all`, `find_discrepancies`,
|
|
1043
|
+
`Lyra::ModeTransition.discrepancy`, the mode gate's check and `lyra:repair` raised on such a
|
|
1044
|
+
model; sync projection and rebuild updated and deleted rows `where(id:)`. Hijack and event
|
|
1045
|
+
sourcing ignored the key the application gave, reserved another (a uuid), and put an `id`
|
|
1046
|
+
attribute the table does not have into the `Created` event. ES-NoProj's and `as_of`'s
|
|
1047
|
+
reconstructed records and the dashboard's record pages also used `id`. All now use the
|
|
1048
|
+
model's primary key. A composite primary key is still not supported, and
|
|
1049
|
+
is now refused clearly: `monitor_with_lyra` and `config.models` raise `ArgumentError` naming
|
|
1050
|
+
the key, and `verify_mapping!` reports it.
|
|
1051
|
+
- **ES-NoProj: `has_many ... dependent: :delete_all` left the children alive.** Rails deletes
|
|
1052
|
+
them through the association's table scope, which matches no row in the events-only store, so
|
|
1053
|
+
no `Destroyed` event was written and the children were still read back after their parent was
|
|
1054
|
+
destroyed. A `dependent: :nullify` found its children by foreign key alone, so it also cleared
|
|
1055
|
+
those outside a scoped association (`has_many :x, -> { where(...) }`), and a polymorphic one
|
|
1056
|
+
(`as:`) wrote no events at all. Both now find the children the association selects in the event
|
|
1057
|
+
store, as Active Record's association scope does: the owner's records, narrowed by the child's
|
|
1058
|
+
default scopes (less what the association's scope unscopes, so a soft-deleted child is left
|
|
1059
|
+
alone) and by the association's scope. Each gets the event `delete_all` or `update_all` writes
|
|
1060
|
+
there (`delete_all`, `dependent_association` or `update_all`, as in Monitor); strict mode
|
|
1061
|
+
refuses `delete_all` as it does in Monitor. **A dependent association whose scope, or whose
|
|
1062
|
+
child's default scope, cannot be evaluated against the event store (an SQL string scope such
|
|
1063
|
+
as `-> { where("status = 'x'") }`) now raises `UnsupportedQuery` and the parent's destroy is
|
|
1064
|
+
refused**, rather than delete or nullify children the association might not select. The same
|
|
1065
|
+
applies to `delete_all`, `delete(record)` and assigning a new set on such an association;
|
|
1066
|
+
`delete(record)` and assignment raised `TypeError` before (a condition on a record's query
|
|
1067
|
+
constraints, `where(["id"] => [[1]])`, which `CachedRelation` now evaluates). The owner's
|
|
1068
|
+
in-memory counter cache is no longer lowered (it went negative): ES-NoProj does not maintain
|
|
1069
|
+
counter caches. `has_one` `:delete` and `:nullify` were not affected. Monitor, Hijack and
|
|
1070
|
+
ES-Sync are unchanged.
|
|
1071
|
+
- **ES-NoProj answers `delete_by` and `destroy_by`,** as `where(...).delete_all` and
|
|
1072
|
+
`where(...).destroy_all`; they raised `UnsupportedQuery`. The partners of an in-memory join are
|
|
1073
|
+
read as Lyra's own read: under a purpose, a join on a model with a policy was refused for the
|
|
1074
|
+
partners' fields though no partner is returned; so are the children a `dependent: :nullify`
|
|
1075
|
+
rewrites, read for their events.
|
|
1076
|
+
- **Monitor appended `Updated` after `Destroyed`** when a stale copy was saved after another
|
|
1077
|
+
process had destroyed the record. Its UPDATE reached no row, or, when nothing had changed, no
|
|
1078
|
+
UPDATE ran; Active Record still ran `after_update`, and the stream became
|
|
1079
|
+
`[Created, Destroyed, Updated]`. Monitor now records no update when the UPDATE reached no row or
|
|
1080
|
+
nothing changed.
|
|
1081
|
+
- **Monitor appended a second `Destroyed`** when two stale copies of a record were both destroyed,
|
|
1082
|
+
when a copy was destroyed after `delete_all` or hand-written SQL had deleted its row, and when
|
|
1083
|
+
the same copy was destroyed twice. Without optimistic locking Active Record does not check what
|
|
1084
|
+
its DELETE reached, and it runs `after_destroy` even when the DELETE reached no row, or when it
|
|
1085
|
+
ran none for a record that is not persisted; the stream became `[Created, Destroyed, Destroyed]`,
|
|
1086
|
+
which no replay can apply. A destroy now records nothing unless it deleted the record's row, so
|
|
1087
|
+
a new record's `destroy` no longer appends an event without an id in any mode either. A row
|
|
1088
|
+
deleted by hand-written SQL keeps its stream open, as any write that bypasses Active Record does,
|
|
1089
|
+
until `lyra:repair` appends the `Destroyed`. Hijack and the event-sourcing modes refuse a stale
|
|
1090
|
+
destroy, as before; a soft delete (paranoia) and optimistic locking are unchanged.
|
|
1091
|
+
- **`belongs_to ... touch: true` cascades published nothing.** Rails touches the parent through
|
|
1092
|
+
`touch_later`, which sets the timestamps and clears their change before `touch` runs at commit,
|
|
1093
|
+
so Lyra saw no change. The parent's row moved and its stream did not; with timestamps compared,
|
|
1094
|
+
DualView flagged every touched parent and the mode gate refused. Lyra now keeps the row's value
|
|
1095
|
+
before the first deferred touch and records the change.
|
|
1096
|
+
- **An application's `metadata_proc` `source` was overwritten** by Lyra's (`lyra_interceptor`,
|
|
1097
|
+
`lyra_command_handler`). Lyra's value is now the default the application's keys override.
|
|
1098
|
+
- **ES-NoProj: `where(...).touch_all` touched every record** (an unscoped `touch_all` through the
|
|
1099
|
+
scope branch, with an event each) and then raised `UnsupportedQuery`. It now touches only the
|
|
1100
|
+
selected records, with one time for all, as Rails's `touch_all` does.
|
|
1101
|
+
- **The GDPR helpers lost a related record's updates and destroy** (Art. 15/17/20 reports):
|
|
1102
|
+
they matched the subject's foreign key in each event alone, and an update carries only its
|
|
1103
|
+
changes. Each record is now followed through the log, and an event is the subject's when the
|
|
1104
|
+
record pointed at them before or after it.
|
|
1105
|
+
- **A record created in Monitor lost `created_at` and `updated_at` when the log became
|
|
1106
|
+
authoritative.** Monitor's event left the two columns out of its `attributes` snapshot (its
|
|
1107
|
+
`changes` carried them), and replay and projection read a create's `attributes`. After a switch
|
|
1108
|
+
to event sourcing an ES-NoProj read raised `MissingAttributeError` on them, and a rebuild reset
|
|
1109
|
+
them to the rebuild time; DualView ignored the two columns, so it reported neither. `attributes`
|
|
1110
|
+
now holds every column, timestamps included, in Monitor events and in the events of bypass writes
|
|
1111
|
+
that create or delete records (`insert_all`, `delete_all` and the like), as Hijack and the
|
|
1112
|
+
event-sourcing modes already did. Logs written before this change are not supported (see
|
|
1113
|
+
Breaking).
|
|
1114
|
+
- **`ModelProjection.project_create` overwrote a string-keyed `created_at` or `updated_at` with
|
|
1115
|
+
the projection time.** It defaulted the two with symbol keys, and stringifying the keys let the
|
|
1116
|
+
default replace the value the event carried. A rebuild therefore reset the timestamps of every
|
|
1117
|
+
record whose event attributes had string keys, as Monitor's do.
|
|
1118
|
+
- **A destroyed record could be written again after events that are not replayed.** The check
|
|
1119
|
+
that refuses an update or destroy of a destroyed record (0.9.3) looked at the stream's last
|
|
1120
|
+
event only. A destroy followed by an event that is not replayed (the additional domain event of
|
|
1121
|
+
an `also:` rule on destroy, an `ErasureApplied` from erasing the destroyed record, a
|
|
1122
|
+
`has_and_belongs_to_many` link made with a stale copy and linked into its stream) let a stale
|
|
1123
|
+
copy update the record in ES-NoProj, ES-Async and ES-Lazy, appending after the destroy. The
|
|
1124
|
+
check now looks at the stream's last replayed event: the last one `Lyra::Event.operation_of`
|
|
1125
|
+
gives an operation.
|
|
1126
|
+
- **Strict data access tells Rails' association writes from the application's.** Any
|
|
1127
|
+
`update_all` setting one `*_id` column to nil was taken for a `dependent: :nullify` by its
|
|
1128
|
+
shape, and let through: `Article.where(...).update_all(author_id: nil)` cleared keys in strict
|
|
1129
|
+
mode. Rails' association code now marks the one relation or record it is about to write, just
|
|
1130
|
+
before the write (`Lyra::Associations::Maintenance`), and only that write is let through:
|
|
1131
|
+
a write the application makes meanwhile (in an `after_find` while the association loads its
|
|
1132
|
+
record, in a destroy callback) is its own, and refused. The marked writes are a `has_many`'s
|
|
1133
|
+
nullify (`dependent: :nullify`, and removing children with `delete`, `delete_all`, `clear` or
|
|
1134
|
+
by assigning a new set or ids), a `has_many :through`'s `dependent: :nullify` of its through
|
|
1135
|
+
records, a `has_one`'s `dependent: :nullify`, and the deletes of rows that link records:
|
|
1136
|
+
`has_and_belongs_to_many` join rows, the same rows through a model the application declared on
|
|
1137
|
+
the join table, and a `has_many :through`'s through records when records are removed from it.
|
|
1138
|
+
A `has_many`'s `dependent: :delete_all` of its children stays refused. The same writes made by
|
|
1139
|
+
the application are bulk writes like any other: refused in strict mode, and, where allowed,
|
|
1140
|
+
admitted by Genesis, checked against destroyed streams and the lock limit, and recorded as
|
|
1141
|
+
`update_all` or `delete_all` (an `update_all(fk: nil)` in ES-NoProj on the records the event
|
|
1142
|
+
store holds that the relation selects, as before). Several of Rails' own writes were refused
|
|
1143
|
+
in strict mode and now run: a polymorphic `dependent: :nullify` (two columns, `has_many` and
|
|
1144
|
+
`has_one`; a `has_many`'s is now recorded as `dependent_association`, with both columns'
|
|
1145
|
+
changes, where it was an `update_all`), a `has_many`'s removal of a child whose key is not named
|
|
1146
|
+
`*_id` (`foreign_key: :parent`), a `has_one`'s `dependent: :nullify` (its `update_columns`; the
|
|
1147
|
+
owner could not be destroyed), removing records from a `has_many :through` over a monitored
|
|
1148
|
+
model (its through records' `delete_all`, recorded as before), Rails' deletes through an
|
|
1149
|
+
application model on a join table, and, in ES-NoProj, a `has_many :through`'s
|
|
1150
|
+
`dependent: :nullify`, now recorded as `dependent_association` there as in the other modes. To
|
|
1151
|
+
mark the relation a `has_many :through` writes, Lyra reproduces Active Record's
|
|
1152
|
+
`HasManyThroughAssociation#delete_records`, checked identical in 8.0.5.1 and 8.1.4.1, and
|
|
1153
|
+
installs it for Rails 8.0 and 8.1 only; on another version Rails' own method runs with the
|
|
1154
|
+
through model marked for the whole of it (a write of that model made meanwhile is taken for
|
|
1155
|
+
Rails'), and a warning says so once.
|
|
1156
|
+
- **Deleting `has_and_belongs_to_many` join rows yourself is refused in strict mode.** The
|
|
1157
|
+
association's own deletes (`delete`, `delete_all`, `clear`, `destroy`, `destroy_all`,
|
|
1158
|
+
assigning records or ids, destroying the owner) are Rails' bookkeeping, allowed as before, and
|
|
1159
|
+
each records its `AssociationUnlinked`. A `delete_all` the application makes on the join model
|
|
1160
|
+
(`Item.const_get(:HABTM_Tags)`) was always allowed; it is now refused in strict mode when either
|
|
1161
|
+
side's model is monitored, as is one on a model the application declared on the join table
|
|
1162
|
+
(not monitored itself), which deleted links with no event at all and is now recorded as an
|
|
1163
|
+
unlink where it runs. Several monitored models' `has_and_belongs_to_many` can share a join
|
|
1164
|
+
table (both sides of one association, a self-referential one, two owners on columns of their
|
|
1165
|
+
own): a delete through the application's model of it, its own or Rails' (a `has_many :through`
|
|
1166
|
+
over it, its `dependent: :delete_all`), records on the stream of every monitored owner the
|
|
1167
|
+
deleted rows name the `AssociationUnlinked` Rails' own unlink from that side records, linked
|
|
1168
|
+
into the associated records' streams that have none of their own; a side whose model is not
|
|
1169
|
+
monitored records nothing. Links made and removed from a side Lyra does not monitor (its
|
|
1170
|
+
`has_and_belongs_to_many` to a monitored model) were not recorded at all: Rails' link and
|
|
1171
|
+
unlink, and the application's `delete_all` where allowed, are now recorded on the monitored
|
|
1172
|
+
records' streams (appended to the first, linked into the others), as a link made from their
|
|
1173
|
+
side is seen there.
|
|
1174
|
+
- **`insert_all` in ES-NoProj decides on primary keys as the event store has them.** The table
|
|
1175
|
+
decided: a record created through callbacks (no row there) was inserted again, a second
|
|
1176
|
+
`Created` on its live stream with its attributes overwritten, where Monitor skips the row
|
|
1177
|
+
(`insert_all!` raises `ActiveRecord::RecordNotUnique`); and a record inserted with `insert_all`
|
|
1178
|
+
and destroyed kept its row, so inserting its key again was skipped with no event, where
|
|
1179
|
+
Monitor creates the record again. The rows' keys are now locked and read from the event store:
|
|
1180
|
+
a record it holds is skipped (`insert_all`) or refused before anything is written
|
|
1181
|
+
(`insert_all!`, and `insert_all` whose `unique_by` is another index, whose `ON CONFLICT` does
|
|
1182
|
+
not cover the key), and the table's row of any other key, left by a destroyed record, is
|
|
1183
|
+
replaced (a row Genesis left without a stream is a record, and conflicts). A key whose record was destroyed is created again, `Created` after
|
|
1184
|
+
`Destroyed` as in Monitor, and the stream reads as the record created last (find, queries,
|
|
1185
|
+
`StreamTail`, the writes that follow). Rows that give no key take the table's sequence, as
|
|
1186
|
+
before.
|
|
1187
|
+
- **An ES-Async or ES-Lazy `upsert_all` no longer brings back a destroyed record whose row is
|
|
1188
|
+
still in the table.** Before the destroy's projection job ran (ES-Async), or before the next
|
|
1189
|
+
catch-up (ES-Lazy, a destroy committed after the one the write began with), `ON CONFLICT`
|
|
1190
|
+
updated the row and appended an `Updated` after the `Destroyed`: the record read as live in the
|
|
1191
|
+
log, the job then deleted its row, and `find` raised. The rows the upsert conflicts with now
|
|
1192
|
+
have their aggregate locks taken (all or none, within `bulk_write_lock_limit`) and their streams
|
|
1193
|
+
read. In ES-Async (see the next entry) the rows are then brought up to date with their streams,
|
|
1194
|
+
and a destroyed record is inserted again, as in Monitor. In ES-Lazy a destroyed one refuses
|
|
1195
|
+
the whole upsert with `Lyra::RecordDestroyedError` before anything is written, as an update of
|
|
1196
|
+
a destroyed record is refused: replaying the destroy onto the row there would be applied again
|
|
1197
|
+
by the next catch-up, which would delete the new row. Once the destroy is in the table the
|
|
1198
|
+
upsert inserts, as in Monitor. In ES-Lazy the locked rows are brought up to date with their
|
|
1199
|
+
streams first, so the events carry the values the stream has. ES-Sync, Hijack and Monitor
|
|
1200
|
+
delete the row with the destroy and are unchanged; ES-NoProj refuses `upsert_all` as before.
|
|
1201
|
+
- **ES-Async: `insert_all`, `insert_all!` and `upsert_all` conflict as Monitor's do while a
|
|
1202
|
+
projection job is pending.** The id of a record whose destroy was still waiting for its job met
|
|
1203
|
+
the dead row: `insert_all` skipped it (no event; the job then deleted the row), where Monitor
|
|
1204
|
+
creates the record again, and `upsert_all` was refused. The id of a record whose create was
|
|
1205
|
+
waiting had no row yet: the insert wrote one and logged a second `Created` on the live stream,
|
|
1206
|
+
where Monitor's `insert_all` skips it, `insert_all!` raises `ActiveRecord::RecordNotUnique` and
|
|
1207
|
+
`upsert_all` updates it. Deleting the dead row first was no answer: the destroy's job would then
|
|
1208
|
+
delete the new one. The records of the keys the rows give (and, for `upsert_all`, the rows its
|
|
1209
|
+
conflict key finds) are now locked (aggregate locks, within `bulk_write_lock_limit`) and their
|
|
1210
|
+
rows brought up to date with their streams, as their jobs would
|
|
1211
|
+
(`AsyncProjectionJob.project_pending`): under the job's own per-stream lock, the events after
|
|
1212
|
+
the stream's checkpoint are applied and the checkpoint moved past them, in the write's
|
|
1213
|
+
transaction. The pending jobs then apply only what is past the checkpoint, the write's own
|
|
1214
|
+
event at most, which is in the row already: a destroy is never applied to the row created
|
|
1215
|
+
again. A destroyed record is created again with the values given and the column defaults (not
|
|
1216
|
+
the dead row's), `Created` after `Destroyed`; a live one conflicts. `upsert_all` reads the rows
|
|
1217
|
+
its conflict key finds again once the locks are held, so a job that committed in between is
|
|
1218
|
+
seen. Rows already up to date cost one query. The write then moves the checkpoints of the
|
|
1219
|
+
records it brought up to date, and of those it created, to its own events (one statement):
|
|
1220
|
+
writing the same keys again finds nothing to project. Rows that give no key take the table's
|
|
1221
|
+
sequence; a conflict on another unique index is looked up in the event store (see the entry on
|
|
1222
|
+
ES-Sync, ES-Async and ES-Lazy below). A projection job no longer
|
|
1223
|
+
waits for its record's row while it holds the stream's lock: it takes the row with its lock
|
|
1224
|
+
(one statement more per job, no measurable time) and, when another transaction holds the row,
|
|
1225
|
+
lets the lock go and waits for the row first, so a transaction that holds the row and then
|
|
1226
|
+
inserts or upserts its key does not deadlock against the job.
|
|
1227
|
+
- **ES-NoProj: `insert_all` and `insert_all!` check the model's other unique indexes against the
|
|
1228
|
+
event store.** Only the primary key was; the other unique indexes were checked against the
|
|
1229
|
+
table, which holds no row of a record created through callbacks: a duplicate (an email taken,
|
|
1230
|
+
no id given) was inserted, a second record with the value, where Monitor's `insert_all` skips it
|
|
1231
|
+
and `insert_all!` raises; and the row a destroyed record left there skipped a value that is free.
|
|
1232
|
+
Each row's values for each unique index (not partial, not an expression; NULL conflicts only
|
|
1233
|
+
under NULLS NOT DISTINCT) are now locked as a uniqueness validation of the value locks them
|
|
1234
|
+
and compared with the records the event store holds that may hold them (found through the
|
|
1235
|
+
values' streams, next entry) and with the table's rows holding them: a conflict is skipped
|
|
1236
|
+
(`insert_all`, on any index or on its `unique_by`) or raises `ActiveRecord::RecordNotUnique`
|
|
1237
|
+
before anything is written (`insert_all!`, and a conflict on an index other than `unique_by`).
|
|
1238
|
+
The table's rows of destroyed records holding the values are deleted, and the row of a live
|
|
1239
|
+
record that holds a value its record gave up since (updated through callbacks) is brought up to
|
|
1240
|
+
the record's values. A row Genesis left without a stream (`genesis_unrepresentable :skip`)
|
|
1241
|
+
conflicts, by its key and by its values, as in Monitor; it was deleted as a destroyed record's
|
|
1242
|
+
row, by its key. The values' locks and the keys' count together against
|
|
1243
|
+
`bulk_write_lock_limit`, so a model with several unique indexes reaches it at fewer rows. A
|
|
1244
|
+
`create` or `update` through callbacks is checked against the same indexes (next entry).
|
|
1245
|
+
- **ES-NoProj: `create`, `update`, `update_columns`, `update_column` and `update_all` enforce the
|
|
1246
|
+
table's unique indexes.** ES-NoProj writes no row, so the index never saw these writes: a
|
|
1247
|
+
duplicate email was saved and recorded (unless a uniqueness validation caught it), where Monitor
|
|
1248
|
+
raises `ActiveRecord::RecordNotUnique` from the `INSERT` or `UPDATE`.
|
|
1249
|
+
- *Which indexes.* Unique, on columns, not partial, other than the primary key, and on no column
|
|
1250
|
+
with a nondeterministic collation (whose equality only the database knows; a uniqueness
|
|
1251
|
+
validation refuses such a column too): the list `insert_all` checks too
|
|
1252
|
+
(`Lyra::Projections::UniqueIndexes`). They are read from the database once per model and
|
|
1253
|
+
process, again after `reset_column_information` and every `config.unique_index_refresh`
|
|
1254
|
+
seconds (60), so an index a migration adds is enforced in a process started before it. An
|
|
1255
|
+
index is checked when the write sets one of its columns.
|
|
1256
|
+
- *When.* Where the table modes write the row: after the `before_*` callbacks and before the
|
|
1257
|
+
event is built, so a conflict raises `ActiveRecord::RecordNotUnique` (with PostgreSQL's
|
|
1258
|
+
message, naming the index) before any `after_*` callback and with nothing appended;
|
|
1259
|
+
`update_columns` after its destroyed-record check; `update_all` before any of its events, two
|
|
1260
|
+
of its records given one value conflicting with each other. An update claims the tuple the
|
|
1261
|
+
row would hold: the columns it does not write as the record's stream has them, read under the
|
|
1262
|
+
record's aggregate lock (a copy read before another writer moved the record to another shop
|
|
1263
|
+
claimed its old pair and saved a duplicate). NULL conflicts with nothing unless the index is
|
|
1264
|
+
`NULLS NOT DISTINCT`; values compare as the column holds them, a `citext` column's by the
|
|
1265
|
+
database's `LOWER`, as `citext` compares (Ruby's `downcase` disagreed on "İ"); a destroyed
|
|
1266
|
+
record holds no value, and a record keeps its own.
|
|
1267
|
+
- *Locks.* Each claimed value is locked with the key a uniqueness validation and `insert_all`
|
|
1268
|
+
take, so of two concurrent writers of one value the second waits and raises, as the index
|
|
1269
|
+
makes the second `INSERT` wait. An update, `update_columns` and `update_all` take the value
|
|
1270
|
+
locks after the records' aggregate locks, as Monitor's `UPDATE` takes the row and then the
|
|
1271
|
+
index entry, so they wait where Monitor waits and do not deadlock where it does not (taking
|
|
1272
|
+
them before deadlocked an `update_all` against a save of the same record, and two
|
|
1273
|
+
transactions crossing two records and one value). An update of a model with a uniqueness
|
|
1274
|
+
validation takes the record's aggregate lock before its validations (once, earlier), so the
|
|
1275
|
+
validation's value lock comes after it as well: a validated save and an `update_all` of the
|
|
1276
|
+
record and value, and two validated transactions crossing two records and one value, no
|
|
1277
|
+
longer deadlock. The lock key holds a keyed digest of the value, not the value, which reached
|
|
1278
|
+
the statement log (`hashtext('lyra_unique/User/email/ann@…')`); the `LOWER` of a `citext`
|
|
1279
|
+
value is asked for with the value bound. The links are caught up without waiting for another
|
|
1280
|
+
thread, the links' lock or a connection (a write that cannot have them checks against every
|
|
1281
|
+
record of the model, once), so a model's first use with every connection of the pool taken
|
|
1282
|
+
by its writers finishes. Without `config.value_digest_key` or `secret_key_base`, every environment but development and
|
|
1283
|
+
test raises `ValueDigest::KeyMissing` at first use. The unique indexes ES-NoProj leaves out
|
|
1284
|
+
(partial, expression, nondeterministic collation) are named in the log once per model.
|
|
1285
|
+
- *Lookup.* The value is looked up in a value stream (`"Model!column$<digest>"`, linked by
|
|
1286
|
+
`ReferenceLinks` for each event that sets an indexed column, in one `INSERT`): its cost is the
|
|
1287
|
+
value's history, not the model's size. The digest is an HMAC-SHA256 under a secret of the
|
|
1288
|
+
installation (`config.value_digest_key`, or derived from `secret_key_base`;
|
|
1289
|
+
`Lyra::Projections::ValueDigest`), so a dictionary does not reverse it; a value stream linked
|
|
1290
|
+
under another key is deleted and linked again on its next use. An erasure moves the links of
|
|
1291
|
+
the events it rewrites, so no digest of an erased value is kept. Each column's links have a
|
|
1292
|
+
checkpoint of their own: the first check of a model links only the columns behind (the
|
|
1293
|
+
foreign keys are not read again), before its value locks, in one read of the events; a column
|
|
1294
|
+
added to an index is linked from the first event. Processes that ran an earlier Lyra, or read
|
|
1295
|
+
the indexes before a migration added one, write without the column's links; once they are
|
|
1296
|
+
gone, `bin/rails lyra:link_values` links every value again (a process of this release that
|
|
1297
|
+
reads the new index links its own earlier writes itself).
|
|
1298
|
+
- *Cost.* On the development VM (PostgreSQL in Docker, 1,000 records, 200 interleaved writes) a
|
|
1299
|
+
create or an update that sets a fresh unique email costs three queries more (the lock, the
|
|
1300
|
+
stream read, the link's `INSERT`: 11 against 8 for a create, 12 against 9 for an update),
|
|
1301
|
+
about 90 more objects allocated (of some 2,500) and 2 to 3 ms (of 15 to 25, on a loaded VM);
|
|
1302
|
+
a model without such an index, and an update that sets none of its columns, are unchanged (the
|
|
1303
|
+
same queries, allocations within noise). On the testbed in ES-NoProj the benchmarked writes
|
|
1304
|
+
(`Registration` create, update and destroy: its unique indexes are all partial, so none is
|
|
1305
|
+
checked) are unchanged (12, 10 and 10.5 queries per write, as before), and a `BankPayment`
|
|
1306
|
+
create (a unique `remittance_info`, validated too) costs 13 queries against 10. ES-NoProj's
|
|
1307
|
+
`insert_all` reads the records the values' streams name instead of every record of the model
|
|
1308
|
+
(`Uniqueness::Snapshot`: 52.8 s at 10,000 records, cold), and `Model.insert_all` (and the
|
|
1309
|
+
other inserts on the model) no longer builds the relation of every record first (Active
|
|
1310
|
+
Record sends them through `Model.all`): a cold two-row `insert_all` at 3,000 records took
|
|
1311
|
+
10.2 s, now 77 ms. With `config.reference_links = false` every record of the model is read,
|
|
1312
|
+
as a uniqueness validation reads them.
|
|
1313
|
+
- *`create_or_find_by`.* In ES-NoProj it looked for a conflict on an index made of its
|
|
1314
|
+
attributes alone, so a value its block set was created twice; it now creates in a savepoint
|
|
1315
|
+
and, when the index refuses the record, finds it (`RecordNotFound` when the attributes select
|
|
1316
|
+
none), and `find_or_create_by` creates through it, as Active Record 8.1 does.
|
|
1317
|
+
- *Also.* An `update_columns` on a copy read before another writer changed the value it sets
|
|
1318
|
+
back recorded nothing in ES-NoProj and left the record as it was (its old value was the
|
|
1319
|
+
copy's): it now takes the old values from the record's stream, as the table's `UPDATE` writes
|
|
1320
|
+
the row. ES-Sync, ES-Async and ES-Lazy: next entry.
|
|
1321
|
+
- **ES-Sync, ES-Async and ES-Lazy: a duplicate's event was appended before the projection met
|
|
1322
|
+
the unique index.** In these modes the event store is authoritative and the table a projection:
|
|
1323
|
+
a `create`, `save`, `update`, `update_columns` or `update_all` setting a value a unique index
|
|
1324
|
+
already held appended its event, then the projection's `INSERT` or `UPDATE` failed and was only
|
|
1325
|
+
logged (`strict_projections` off). The log held a record the table could not; in ES-Lazy the
|
|
1326
|
+
next read's catch-up then failed on it, and with it the reads of every model. Monitor raises
|
|
1327
|
+
`ActiveRecord::RecordNotUnique` and writes nothing. The writes are now checked before the event
|
|
1328
|
+
is built, as ES-NoProj's are (previous entry; `Lyra::Projections::UniqueIndexes`, the same
|
|
1329
|
+
indexes, claims, value locks and message, naming the mode), so a duplicate raises
|
|
1330
|
+
`ActiveRecord::RecordNotUnique` with no event appended, no projection job enqueued, no row
|
|
1331
|
+
written and nothing logged as a failed projection, and ES-Lazy's reads go on.
|
|
1332
|
+
- *Where the value is looked up.* ES-Sync: in its table, under the value's lock. Its projection
|
|
1333
|
+
writes the row in the writer's transaction, so once the lock is held the table has every
|
|
1334
|
+
committed claim and the transaction's own, compared by the database as its index compares
|
|
1335
|
+
them; one statement (the lock) and one `SELECT` per value claimed, and no links. ES-Async and
|
|
1336
|
+
ES-Lazy, whose tables lag the event store (a create whose job has not run, a write since the
|
|
1337
|
+
last read): in the value streams, as ES-NoProj, which are now linked in these modes too
|
|
1338
|
+
(`ReferenceLinks.values_enabled?`; the foreign-key links stay ES-NoProj's, and a model with
|
|
1339
|
+
no unique index links nothing). The first check of a model in a process links the events
|
|
1340
|
+
stored before (per column, once).
|
|
1341
|
+
- *Writes that skip callbacks.* `update_columns`, `update_column`, `update_all`, `insert_all`,
|
|
1342
|
+
`insert_all!` and `upsert_all` take the values' locks in these modes too, so a write through
|
|
1343
|
+
callbacks checking the table waits for them, as they wait for it (in ES-Sync
|
|
1344
|
+
`update_columns` locks its row first, `FOR NO KEY UPDATE`, and `update_all` its rows, as
|
|
1345
|
+
Monitor's `UPDATE` takes the row and then the index entry). ES-Sync's statement then checks
|
|
1346
|
+
the index itself. In ES-Async and ES-Lazy the values are first looked up in the event store,
|
|
1347
|
+
where a holder whose row is not projected yet was missed by the table: `update_columns` and
|
|
1348
|
+
`update_all` raise; `insert_all` leaves the row out (or raises, for an index other than its
|
|
1349
|
+
`unique_by`), `insert_all!` raises; `upsert_all` raises for a conflict on an index other than
|
|
1350
|
+
its conflict columns (the record it updates excepted), and on its conflict columns updates
|
|
1351
|
+
the holder, whose row ES-Async brings up to date first (it raises where that row is still not
|
|
1352
|
+
in the table). This was the known leftover of the ES-Async insert entry above.
|
|
1353
|
+
- *A holder destroyed or moved away, its row still in the table.* The value is free, as in
|
|
1354
|
+
Monitor, whose row is gone: a `create` (or any of the writes above) saves it. ES-Async brings
|
|
1355
|
+
such a row up to date with its stream in the write's transaction, as its job would, where that
|
|
1356
|
+
waits for nothing (the record's aggregate lock, its row and its stream's projection lock are
|
|
1357
|
+
only tried; a record one is taken for is left to its job), so neither the write's own `UPDATE`
|
|
1358
|
+
or `INSERT` nor its projection job meets the stale value. ES-Lazy's catch-up applies the
|
|
1359
|
+
destroy first (log order).
|
|
1360
|
+
- *Cost.* On the testbed (development VM, PostgreSQL in Docker, 100 writes each after warm-up),
|
|
1361
|
+
a model without a checked unique index and a write that sets none of its columns issue the
|
|
1362
|
+
same statements and locks as before in all three modes: the writes the benchmark campaign
|
|
1363
|
+
measures (`Registration` create, update and destroy, 8/8/13 statements in ES-Sync, 7/8/13 in
|
|
1364
|
+
ES-Async, 7/8/26.8 in ES-Lazy; its unique indexes are all partial, so none is checked; its
|
|
1365
|
+
`insert_all` and `update_all` likewise), and a `BankPayment` update of its amount (8). In
|
|
1366
|
+
ES-Async and ES-Lazy each append of such a model also looks the model up for its value
|
|
1367
|
+
columns, in memory (about 8 objects more per write; times within noise). A write that sets a
|
|
1368
|
+
checked value costs, per value: in ES-Sync the lock and a `SELECT` on the table (a
|
|
1369
|
+
`BankPayment` create, unique `remittance_info`, 11 statements against 9; an update of the
|
|
1370
|
+
code 11 against 9); in ES-Async and ES-Lazy the lock, the value stream's read and the link's
|
|
1371
|
+
`INSERT` (create 11 against 8 in ES-Async, 19 against 16 in ES-Lazy; update of the code 12
|
|
1372
|
+
against 9, 21 against 18), 1 to 3 ms, and a rebuild of the records the stream names when the
|
|
1373
|
+
value has a history. ES-Async's projection jobs are unchanged (10 to 12 statements).
|
|
1374
|
+
- *ES-Sync inside `ReadYourWrites.with_guaranteed_read`.* A write's row was deferred to the
|
|
1375
|
+
block's end, so the block's own reads missed it and so did the check of every write after it
|
|
1376
|
+
(which reads ES-Sync's table): two creates of one email in a block were both saved, the
|
|
1377
|
+
second's row dropped. ES-Sync now writes the row at once inside the block too, as outside it
|
|
1378
|
+
(and as `each_write: true` did); the block has nothing left to project there.
|
|
1379
|
+
- *A projected create that met another record's value inserted nothing, silently.* The
|
|
1380
|
+
projection's `INSERT ... ON CONFLICT DO NOTHING` skipped a row whose value another row held, as
|
|
1381
|
+
it skips a replay of the record's own row. In ES-Async, when the holder's row was held by
|
|
1382
|
+
another transaction (so the write left it to its job) and the new record's job ran before the
|
|
1383
|
+
holder's, the job completed and moved its checkpoint past the create: the log held a record
|
|
1384
|
+
the table never got, and nothing was logged. A create that inserts nothing although the table
|
|
1385
|
+
has no row with its key (looked up on the table's class: a record of an STI subclass whose
|
|
1386
|
+
type changed since keeps its row) now raises (one query, only then), after reading the rows
|
|
1387
|
+
that hold its values (a tuple with a NULL holds nothing, unless its index is `NULLS NOT
|
|
1388
|
+
DISTINCT`) and their records' streams. When each such record gave the value up and only its
|
|
1389
|
+
row lags, an ES-Async job brings those rows up to date itself, where nothing has to be waited
|
|
1390
|
+
for (so a holder whose own job was lost does not block it), and inserts again; a row still
|
|
1391
|
+
held by another transaction raises `Lyra::Projections::ProjectionBehind`, which the job
|
|
1392
|
+
retries at most every five minutes for about a day (`AsyncProjectionJob::BEHIND_ATTEMPTS`,
|
|
1393
|
+
300; the general retry gave up after about seven minutes of a held row, and the record never
|
|
1394
|
+
reached the table), then reports (log, `projection_error_handler`) and gives up, leaving the
|
|
1395
|
+
record to a rebuild. `Lyra::Projections::DuplicateInLog` when a live record holds the value in
|
|
1396
|
+
the log too (two records of the log hold it), a row with no stream holds it, or no holder is
|
|
1397
|
+
found once the insert was tried again (an expression index, which Lyra does not read): logged
|
|
1398
|
+
once per record and process (debug after; 10,000 remembered) and passed to
|
|
1399
|
+
`projection_error_handler`, the job discarded. Both are `ActiveRecord::RecordNotUnique`s:
|
|
1400
|
+
ES-Sync's write fails with either, a rebuild stops at it. ES-Lazy's catch-up logs it (once)
|
|
1401
|
+
and goes on, since raising there would stop every read for a record the log holds for good.
|
|
1402
|
+
An update whose value another row holds raised already.
|
|
1403
|
+
- *ES-Sync: a database error in the projection was logged inside an aborted transaction.* With
|
|
1404
|
+
`strict_projections` off, the error the projection's `INSERT` or `UPDATE` raised was logged and
|
|
1405
|
+
the save reported success, but PostgreSQL had aborted the transaction, so its `COMMIT` rolled
|
|
1406
|
+
the write and its event back: a value committed after a `REPEATABLE READ` transaction's
|
|
1407
|
+
snapshot, which the check's `SELECT` cannot see, ended in a save reported as done with no
|
|
1408
|
+
event. A database error there now fails the write (`ActiveRecord::StatementInvalid` and its
|
|
1409
|
+
subclasses: here `RecordNotUnique` or `SerializationFailure`), whatever `strict_projections`
|
|
1410
|
+
says; other errors are logged as before.
|
|
1411
|
+
- *Not covered.* Transactions at `REPEATABLE READ` or `SERIALIZABLE`, in every event-sourcing
|
|
1412
|
+
mode (as for a uniqueness validation): the check reads the transaction's snapshot, which
|
|
1413
|
+
misses a value another transaction committed after it began. ES-Sync's write then fails at
|
|
1414
|
+
its projection (above), as Monitor's does; ES-Async and ES-Lazy save both records, and the
|
|
1415
|
+
second's projection fails with `DuplicateInLog` (logged once; ES-Async discards the job,
|
|
1416
|
+
ES-Lazy's catch-up goes on). Partial and expression indexes, and columns with a nondeterministic collation,
|
|
1417
|
+
are left to the projection, as in ES-NoProj (named in the log once per model).
|
|
1418
|
+
- **`insert_all` and `upsert_all` with `returning:` as raw SQL without the primary key recorded
|
|
1419
|
+
nothing.** `insert_all(rows, returning: Arel.sql("code"))` wrote the rows and published no
|
|
1420
|
+
event, in every mode: the keys could not be read back. The primary key is now returned with the
|
|
1421
|
+
caller's SQL under an alias of Lyra's own and taken out of the result the caller gets.
|
|
1422
|
+
- **The result of `insert_all`, `insert_all!` and `upsert_all` with `returning:` is Active
|
|
1423
|
+
Record's own.** Where Lyra added the primary key to `RETURNING` (a symbol, an array, raw SQL,
|
|
1424
|
+
`Arel.sql("*")`) it rebuilt the result without its column types: `column_types` came back empty
|
|
1425
|
+
and `cast_values` cast nothing; `returning: false` lost the affected rows. The column types (by
|
|
1426
|
+
name and by index) and the affected rows are kept.
|
|
1427
|
+
- **A model whose primary key has several columns** (one its table declares, enrolled before
|
|
1428
|
+
Active Record read it; Lyra refuses one it knows of) raised `PG::UndefinedColumn` on
|
|
1429
|
+
`insert_all` and `upsert_all` with `returning:` (the key was asked for as one quoted name) and,
|
|
1430
|
+
without it, wrote the rows with no event. In Monitor and Hijack each key column is now returned
|
|
1431
|
+
and taken out of the caller's result, and each row records its `Created`, the record's key
|
|
1432
|
+
being the array of its columns' values. The event-sourcing modes, whose projections, catch-ups
|
|
1433
|
+
and rebuilds cannot apply such an event (ES-Lazy's catch-up then failed on every read of every
|
|
1434
|
+
monitored model), refuse the three writes with an `ArgumentError` before anything is written,
|
|
1435
|
+
and Genesis refuses the model on its first use there (it failed with `PG::UndefinedColumn`).
|
|
1436
|
+
- **`has_and_belongs_to_many`: the application's `insert_all`, `insert_all!` and `upsert_all`
|
|
1437
|
+
of join rows were neither refused nor recorded.** Through Rails' join model
|
|
1438
|
+
(`Item::HABTM_Tags`) or a model of the application's on the join table (and `insert`, `insert!`,
|
|
1439
|
+
`upsert`, and an owner's `item_tags.insert_all(...)`), they are now refused in strict mode
|
|
1440
|
+
unless inside `Lyra.without_strict_access`, as its `delete_all` of join rows is, when either
|
|
1441
|
+
side is monitored, and where they run each row written (read back with `RETURNING`) is recorded
|
|
1442
|
+
as the `AssociationLinked` Rails' own link from each side records, on every monitored owner's
|
|
1443
|
+
stream the row names; a row `ON CONFLICT` skipped records nothing, and an upsert that moves a
|
|
1444
|
+
row to another pair unlinks the old pair. Rails' own links are inserted one row at a time
|
|
1445
|
+
through the join model's callbacks, as before.
|
|
1446
|
+
- **`has_and_belongs_to_many`: the join table of an owner Lyra does not monitor, whose monitored
|
|
1447
|
+
side has no association back onto it, was not recognised** for a model of the application's on
|
|
1448
|
+
that table: its deletes went unguarded and unrecorded, though Rails' own links and unlinks there
|
|
1449
|
+
are recorded on the monitored side's streams. Lyra now keeps every `has_and_belongs_to_many`
|
|
1450
|
+
declared (those of models loaded before its hook was installed included), and the table is a
|
|
1451
|
+
join table when one whose links it records names it. A
|
|
1452
|
+
`has_many :through` over a model Lyra does not monitor is not one: no link of it is recorded,
|
|
1453
|
+
from either side, as before.
|
|
1454
|
+
- **`has_and_belongs_to_many`: Rails' own unlink of a join row naming another association's
|
|
1455
|
+
owner too** (two owners' columns filled on one row of a shared table, which only the
|
|
1456
|
+
application's code writes) recorded the unlink of the side Rails deleted from only; the other
|
|
1457
|
+
owner's stream kept a link whose row was gone. It is now recorded on every monitored owner's
|
|
1458
|
+
stream the row names, as the application's delete of the row already was. Rows naming no other
|
|
1459
|
+
owner record exactly what they did.
|
|
1460
|
+
- **`config.bulk_write_lock_limit` takes `nil` or a whole number of at least 1.** 0 (or a
|
|
1461
|
+
negative number) refused every write that locks a record, a single `update_columns`, `touch` or
|
|
1462
|
+
`delete` included; any other value raises `ArgumentError` now, and a string is read as its
|
|
1463
|
+
number. `BulkWriteLockLimitError`'s message no longer reads "a update_columns of 1 ... records".
|
|
1464
|
+
- **A bulk write is no longer aborted for the deadlocks of the transactions it waits behind.**
|
|
1465
|
+
`Lyra::AdvisoryLock.xact_lock_all` gave back every lock it had taken whenever one was held, and
|
|
1466
|
+
waited for that one without a limit, in its queue; there it was the one PostgreSQL aborted when
|
|
1467
|
+
the transactions ahead of it deadlocked among themselves, and transactions that came and went with
|
|
1468
|
+
a few of its records kept it out. A bulk write of 8 records, for 15 s, against three transactions
|
|
1469
|
+
that each updated one record and then bulk-wrote two others: in ES-NoProj it went through 2 times
|
|
1470
|
+
in 21 attempts and was aborted 19 times (four runs), in ES-Async 50 times in 66 attempts, 16
|
|
1471
|
+
aborted (three runs). A bulk write that holds no advisory lock of its own yet now keeps the locks
|
|
1472
|
+
it took while it waits for the others, for at most 100 ms in all, and waits holding none for at
|
|
1473
|
+
most half of `deadlock_timeout`; then it gives them back and tries again. No wait of its lasts as
|
|
1474
|
+
long as `deadlock_timeout`, so PostgreSQL never reports it deadlocked, and a transaction that
|
|
1475
|
+
holds one of its records and asks for another it took goes on once it gives way. The same runs:
|
|
1476
|
+
ES-NoProj 7 of 8 attempts went through (the one aborted already held locks, in a later round of
|
|
1477
|
+
the write), ES-Async 42 of 42 (four runs each); deadlocks in all, the three transactions' among
|
|
1478
|
+
themselves included, 47 against 56 in ES-NoProj, and in ES-Async 29 in four runs against 43 in
|
|
1479
|
+
three (the transactions' own: 46 against 37, and 29 against 27). A bulk write of 6 records, beside
|
|
1480
|
+
one of 4 and a transaction like those, went through 62 and 88 times (two runs) instead of 24 to 30
|
|
1481
|
+
in ES-Async, and 60 instead of 12 in ES-NoProj (the one of 4 records: 130 and 135 instead of 152
|
|
1482
|
+
to 160, and 121 as before), with no deadlock before or after; the runs took longer (ES-Async 42 s
|
|
1483
|
+
against 22 to 25, ES-NoProj 415 s against 236). It is no cure for starvation: in ES-NoProj a write
|
|
1484
|
+
still waited 2 to 28 s, and in ES-Async its turns cost the three transactions about a third of
|
|
1485
|
+
their throughput. A transaction that already holds locks of its own (it wrote a record, then
|
|
1486
|
+
bulk-writes others) waits as before, holding none of the new ones: keeping them, or taking all in
|
|
1487
|
+
sorted order and waiting, as Monitor's row locks do, made such a transaction deadlock nearly every
|
|
1488
|
+
time (119 of 120 in that ES-Async run, against none). The lock limit is still checked before any
|
|
1489
|
+
lock is taken, nothing is appended until every lock is held, and a write that finds its records
|
|
1490
|
+
free takes them as before, in one statement in a savepoint. A `lock_timeout` of the caller's own
|
|
1491
|
+
(session or transaction) shorter than the bounded wait stays in force, and its timeout raises
|
|
1492
|
+
`ActiveRecord::LockWaitTimeout`, as before, instead of being taken for the bounded wait's own
|
|
1493
|
+
limit and tried again.
|
|
1494
|
+
|
|
1495
|
+
### Changed
|
|
1496
|
+
- **The gems' MIT-LICENSE copyright line no longer names the University of the Aegean**, which is
|
|
1497
|
+
the author's affiliation, not a copyright holder.
|
|
1498
|
+
- **Ruby 4.0.9, Rails 8.1.4.1, pg 1.7.0.** Security releases: Ruby 4.0.9 fixes memory
|
|
1499
|
+
corruption when compiling crafted regular expressions, Rails 8.1.4.1 fixes advisories in
|
|
1500
|
+
Active Support and Action View (Active Record unchanged). Lyra's requirements are unchanged
|
|
1501
|
+
(Ruby 4.0+, Rails 8.0+); the project and the example app now run these versions.
|
|
1502
|
+
- **Ruby 4.0.10.** The project and the example app run Ruby 4.0.10 (`.ruby-version`); Lyra's
|
|
1503
|
+
requirement is unchanged (Ruby 4.0+).
|
|
1504
|
+
- **`config.erasure_max_copies`** (default 10) sets `everywhere`'s threshold for the application;
|
|
1505
|
+
`erase!(max_copies:)` and `lyra:erase MAX_COPIES=` still override it per call. Each value left
|
|
1506
|
+
as shared is reported in the new `result.shared` as `{ field:, type:, holders: }` (never the
|
|
1507
|
+
value); `result.shared_values` stays an Integer, their count. **`erase!(dry_run: true)`**
|
|
1508
|
+
(`lyra:erase DRY_RUN=1`) runs the erasure and rolls it back, reporting what it would do, the
|
|
1509
|
+
shared values and their holders included, and writing nothing: once a person's values are
|
|
1510
|
+
erased, a later run cannot find the copies left as shared, so the threshold is chosen with a dry
|
|
1511
|
+
run first, and the task's hint says so. `ErasureApplied` gains `row_only` and `imported`
|
|
1512
|
+
(above). The documentation now gives the operator's `VACUUM` step after an erasure (old row and
|
|
1513
|
+
event versions stay on disk, in WAL and in backups until reclaimed or expired), states that a
|
|
1514
|
+
database error logged by a failed bypass publish can contain the failing row, and describes a
|
|
1515
|
+
narrow wait between an erasure and a model's first use in another transaction.
|
|
1516
|
+
- **Erasure can delete the subject's access log; off by default.** `Lyra::Erasure.erase!` left the
|
|
1517
|
+
record's access stream (`Lyra::DataAccess$<Model>$<id>`: who read the record, when, under which
|
|
1518
|
+
purpose and basis) whole and unreported, and `everywhere` never reached it, since it holds field
|
|
1519
|
+
names, not values. Whether it goes is the controller's decision (it may be evidence to keep, Art.
|
|
1520
|
+
5(2)). `config.erase_access_log` (default `false`) deletes it once the record is fully erased
|
|
1521
|
+
(this erasure and the earlier ones together cover every personal field);
|
|
1522
|
+
`erase!(access_log: true|false)` (`lyra:erase ACCESS_LOG=1|0`) decides per call, `true` at once. Its events are deleted
|
|
1523
|
+
from every stream and from the events table, in the erasure's transaction. `access_log_subjects:`
|
|
1524
|
+
(`ACCESS_LOG_SUBJECTS=`) names the ids the application passed as the person's subject
|
|
1525
|
+
(`validate_access!(…, subject: "user-7")`), whose logs follow the same rule. `ErasureApplied` and
|
|
1526
|
+
the `Result` record `access_log` (`"erased"` or `"kept"`) and `access_log_entries`, and
|
|
1527
|
+
`ErasureApplied` how many such ids (`access_log_subjects`), never which.
|
|
1528
|
+
`Lyra::AccessLog.erase!(subject, reason:, erased_by: nil)` (`lyra:erase_access_log SUBJECT=
|
|
1529
|
+
REASON= [OPERATOR=]`) deletes one subject's log alone, a record's or a bare id's, and records
|
|
1530
|
+
it as `AccessLogErased` in `Lyra::AccessLogErasures`. A model-wide entry (`Lyra::DataAccess$<Model>$*`, a `pluck`'s) names
|
|
1531
|
+
no subject and is refused (`Lyra::AccessLog::ModelWideSubject`). Retention's `:hard_delete` of a
|
|
1532
|
+
record erased before with its log kept deletes the log too when the setting is on, and records
|
|
1533
|
+
an `ErasureApplied` that names no fields: **code that reads `ErasureApplied` must allow an empty
|
|
1534
|
+
`fields`**. Not covered: the access logs of other records `everywhere` reaches, and an access
|
|
1535
|
+
recorded (on its own connection) while the erasure runs.
|
|
1536
|
+
- **Mode switches, erasures and repairs record what the operating system says beside the operator's
|
|
1537
|
+
label.** The actor (`OPERATOR=`, `Current.user`, `LYRA_OPERATOR`) is a label nothing checks.
|
|
1538
|
+
`Lyra::Attribution` adds the user the process runs as (by uid, not the `USER` variable), the host
|
|
1539
|
+
and the pid: `lyra_mode_transitions` gains `os_user`, `host` and `pid` (added to an existing table
|
|
1540
|
+
on first use), the metadata of `ErasureApplied` and of `AccessLogErased` (an access log erased on
|
|
1541
|
+
its own) carries them beside `erased_by`, and a repair's events record
|
|
1542
|
+
`repaired_by` (the label: `lyra:repair` passes `OPERATOR`, `Lyra::Repair.run` takes `operator:`),
|
|
1543
|
+
`os_user`, `host` and `pid`, where a repair recorded no actor at all. `ErasureApplied`'s data gains
|
|
1544
|
+
`any_fields`, `requested_fields` (what the request named; `nil` for the declared columns) and
|
|
1545
|
+
`beyond_policy` (the columns erased only because of `any_fields`). This authenticates no one: it
|
|
1546
|
+
makes the label comparable with the OS and SSH audit logs, which remain the attribution control.
|
|
1547
|
+
- **PAM DSL: restriction of processing (Art. 18) and the right to object (Art. 21).** PAM had no
|
|
1548
|
+
construct for either: a restriction could only be approximated by narrowing a purpose for
|
|
1549
|
+
everyone or withdrawing a consent, and a purpose on contract or legal obligation could not be
|
|
1550
|
+
stopped for one person. A purpose now declares the Art. 18(2) grounds on which it continues
|
|
1551
|
+
while a subject's processing is restricted (`allowed_when_restricted :consent`, `:legal_claims`,
|
|
1552
|
+
`:protection_of_rights`, or `:important_public_interest, reference: "<the law>"`), and whether
|
|
1553
|
+
it is direct marketing (`direct_marketing!`, Art. 21(2)) or research or statistics
|
|
1554
|
+
(`research_or_statistics! public_interest_task: true|false`, Art. 21(6)); `Purpose#objectable`
|
|
1555
|
+
derives the article that lets a subject object (Art. 21(1) for public task and legitimate
|
|
1556
|
+
interests, research or not), or none. `PamDsl.rights` records, per subject and independent of any policy,
|
|
1557
|
+
restrictions (`restrict!`, then `notify_lift!` and `lift!`, which raises without the Art. 18(3)
|
|
1558
|
+
notice) and objections (`object!`: upheld at once for direct marketing, pending and restricting
|
|
1559
|
+
that purpose for Art. 21(1) and 21(6) until `decide!`, recorded as rejected and raised as
|
|
1560
|
+
`ObjectionNotApplicableError` for a purpose with no right to object; `withdraw_objection!`), as
|
|
1561
|
+
events in `PamDsl.rights_store`, with an `on_rights_event` hook. An upheld objection to direct
|
|
1562
|
+
marketing stops every direct-marketing purpose for that subject; objecting again while an
|
|
1563
|
+
objection is in force returns it; times must be `Time` values that run forwards. `validate_access!` gains
|
|
1564
|
+
Condition 6, checked after every other condition: `RestrictedProcessingError` for a restriction
|
|
1565
|
+
that covers the access when the purpose declares no exception that applies, and
|
|
1566
|
+
`ObjectedProcessingError` for an upheld objection. The Article 30 report prints each purpose's
|
|
1567
|
+
right to object and its exceptions. PAM DSL ships the store protocol and an in-memory store
|
|
1568
|
+
only; the in-memory store refuses to record outside development and test
|
|
1569
|
+
(`RightsStoreNotDurableError`). Lyra's durable store and its enforcement on reads, erasure and
|
|
1570
|
+
retention are not in this change, and writes are not checked under a restriction in 1.0.0.
|
|
1571
|
+
- **Lyra keeps the data subject's restrictions, objections and consent durably, shared by every
|
|
1572
|
+
process.** With PAM DSL loaded, Lyra installs `Lyra::Rights` into it. `PamDsl.rights_store`
|
|
1573
|
+
becomes Lyra's: each request, notice, decision and lift is an event in
|
|
1574
|
+
`Lyra::Rights$<subject key>` (linked to `Lyra::Rights`), appended with RubyEventStore's expected
|
|
1575
|
+
version so that two processes cannot both lift or decide one request (the second gets
|
|
1576
|
+
`PamDsl::RightsConcurrencyError` and nothing is stored), and an append that does not replay is
|
|
1577
|
+
refused before anything is written. The new table `lyra_rights` (one row per restriction or
|
|
1578
|
+
objection: subject key, kind, policy, purpose, fields, ground, state, times) is written in the
|
|
1579
|
+
same transaction and created on first use; `bin/rails lyra:rights:rebuild` rebuilds it.
|
|
1580
|
+
Condition 6 answers from a per-process snapshot of the committed requests in force, refreshed at
|
|
1581
|
+
most every `config.rights_refresh_interval` seconds (default 1, new) and on
|
|
1582
|
+
`Lyra::Rights.refresh!`, so a check costs a clock read when nothing is in force; a transaction
|
|
1583
|
+
that records a right sees it at once and other threads only after it commits, and a rebuild
|
|
1584
|
+
locks the table against transitions. `PamDsl.subject_key_resolver` maps a record to its own key
|
|
1585
|
+
and to the keys of the people it names, by the subject-access copy's selection rule (link
|
|
1586
|
+
columns a record was loaded without are read from its row, or the access fails with
|
|
1587
|
+
`Lyra::Rights::UnresolvedSubjectError`), so a restriction recorded against a person
|
|
1588
|
+
(`"Person$7"`) refuses a read of their invoice under a purpose with no Art. 18(2) exception.
|
|
1589
|
+
Lyra installs its stores when it loads, and at boot only where the application has not
|
|
1590
|
+
installed its own. Consent moves to the same store: PAM DSL's new `PamDsl.consent_store`, when set, is
|
|
1591
|
+
the store of every policy's `consent_policy`, and Lyra's keeps each request, grant and withdrawal
|
|
1592
|
+
as an event in `Lyra::Consent$<subject key>` with the table `lyra_consents` as its projection.
|
|
1593
|
+
Before, consent was held in memory, per process and per policy object: lost on restart, not seen
|
|
1594
|
+
by other processes or by another policy, and gone when a policy was defined again. A person's
|
|
1595
|
+
consent now covers the records that name them; a grant that repeats a granted consent records
|
|
1596
|
+
nothing. Limit: a consent check on a record that names more than one person (a payer and a
|
|
1597
|
+
payee) is refused with `Lyra::Rights::ConsentStore::SharedRecordConsentError`, a
|
|
1598
|
+
`PamDsl::ConsentRequiredError`, whatever each consented to; whose consent such a record needs
|
|
1599
|
+
is future work. Condition 4 and the `:consent` exception of Art. 18(2) read the same record (a
|
|
1600
|
+
consent store's refusal under that exception is now reported with the restriction rather than
|
|
1601
|
+
raised), and `consent_audit` lists the store's records (`recorded_consents`). Without Lyra, PAM
|
|
1602
|
+
DSL keeps consent in memory as before. Neither stream
|
|
1603
|
+
is erased by `Lyra::Erasure` or with the access log: they hold names, grounds and reasons, never
|
|
1604
|
+
values. Model-wide filtering of restricted subjects, `lyra_processable`, and restriction-aware
|
|
1605
|
+
erasure and retention are not in this change.
|
|
1606
|
+
- **Lyra enforces restriction and objection where it checks reads, in erasure and in retention,
|
|
1607
|
+
and reports them.** Under a purpose (or `reads_without_purpose` `:audit`/`:deny`):
|
|
1608
|
+
- A record load that names no declared field (`select(:id, :payer_id)`) is checked too, for a
|
|
1609
|
+
restriction of all of its subjects' data, a pending objection's restriction of the purpose and
|
|
1610
|
+
an upheld objection (PAM DSL's new `validate_rights!`); before, such a load was never checked.
|
|
1611
|
+
A loaded restricted record raises, as a non-consenting one does.
|
|
1612
|
+
- Aggregates, `pluck`, `pick`, `ids`, `exists?` and `cache_version` (subject `Model$*`) leave the
|
|
1613
|
+
restricted subjects out: the record's own key `NOT IN` the restricted records and each link
|
|
1614
|
+
column `IS NULL OR NOT IN` the restricted people, so rows that name no one are kept; ES-NoProj
|
|
1615
|
+
filters its in-memory records by their keys, and fails closed
|
|
1616
|
+
(`Lyra::Rights::UnresolvedSubjectError`) when a record's links cannot be read. A field-scoped
|
|
1617
|
+
restriction applies only to queries naming one of its fields. Audit mode leaves no one out and
|
|
1618
|
+
logs it. With nothing in force for the query, its SQL is unchanged. Before, these queries
|
|
1619
|
+
included the restricted people.
|
|
1620
|
+
- A join, subquery or CTE that reaches a model with such a subject raises
|
|
1621
|
+
`PamDsl::RestrictedProcessingError` rather than include or silently drop them.
|
|
1622
|
+
- `Model.lyra_processable(fields: nil)`, a scope without the records the purpose in scope may not
|
|
1623
|
+
process, for batch jobs (Active Record and ES-NoProj).
|
|
1624
|
+
- The rights and consent are read afresh at the first check of each checking scope: a
|
|
1625
|
+
`Lyra.with_purpose` block, a web request (`Lyra::Rights::Middleware`, installed by the
|
|
1626
|
+
engine), an Active Job (`Lyra::Rights::Job`) and each batch of `in_batches`, `find_each` and
|
|
1627
|
+
`find_in_batches` (`Lyra::Rights.scope`). A right recorded in another process applies from the
|
|
1628
|
+
next scope; before, up to `rights_refresh_interval` later, mid-request included. The first
|
|
1629
|
+
check in a scope makes one indexed query; code that checks nothing (no purpose, `:allow`) makes
|
|
1630
|
+
none.
|
|
1631
|
+
`config.rights_refresh = :interval` (new; the default is `:scope`) keeps scopes but drops the
|
|
1632
|
+
forced read: rights and consent are read again only after `rights_refresh_interval`, as before.
|
|
1633
|
+
- `Lyra::Erasure.erase!` refuses a record that a restriction in force covers (its own, a person's
|
|
1634
|
+
it names, of all the data or of a field being erased) with `PamDsl::RestrictedProcessingError`;
|
|
1635
|
+
`lifting: restriction_id`, after the Art. 18(3) notice, lifts it and erases in one transaction.
|
|
1636
|
+
A dry run (`dry_run: true`) is refused the same way and, with `lifting:`, lifts nothing and
|
|
1637
|
+
lists in `Result#lifting` what it would lift (a real erasure lists what it lifted).
|
|
1638
|
+
`Lyra::Retention` skips restricted records with outcome `:restricted` (`Result#restricted`,
|
|
1639
|
+
printed by `lyra:retention:apply`). Before, both acted on data the subject had asked to be kept.
|
|
1640
|
+
- The subject-access copy (`data_export`) has `restrictions` and `objections`: the subject's own
|
|
1641
|
+
requests, never another person's. The per-subject `processing_activities` gives each declared
|
|
1642
|
+
purpose's right to object and Art. 18(2) exceptions (`rights_by_purpose`); with Lyra's
|
|
1643
|
+
per-model policies it gave none. A dashboard page, `privacy/rights`, lists the open requests
|
|
1644
|
+
with their age against the month of Art. 12(3); the policy page shows each purpose's
|
|
1645
|
+
declarations; `RestrictedProcessingError` and `ObjectedProcessingError` in a dashboard read
|
|
1646
|
+
answer 403.
|
|
1647
|
+
- `Lyra::Verification::DataLifecycleWorkflow` (the data lifecycle with a Restricted place,
|
|
1648
|
+
restrict, notice and lift) and `ObjectionWorkflow`, PetriFlow nets whose tests check that
|
|
1649
|
+
PAM DSL's transitions are exactly theirs and that Deleted is unreachable from Restricted
|
|
1650
|
+
without the lift.
|
|
1651
|
+
- After review: SQL strings are read for the tables they reach (`FROM`/`JOIN`, with aliases), so a
|
|
1652
|
+
string join or subquery reaching a restricted model is refused, and a subquery's words are
|
|
1653
|
+
checked against the policy of the table they belong to (a change to the purpose check too); a
|
|
1654
|
+
`FROM`/`JOIN` Lyra cannot name is refused while a right is in force (fail closed). A polymorphic
|
|
1655
|
+
filter covers STI subclass names. A subject's people are resolved only when a request in force
|
|
1656
|
+
could cover the access, and aggregate rows are not subjects (a grouped `select` or `find_by_sql`
|
|
1657
|
+
no longer raises `UnresolvedSubjectError` because of an unrelated right); ES-NoProj reads a
|
|
1658
|
+
narrowed record's people from the whole record. ES-NoProj's `Model.exists?(id)` and an undeclared
|
|
1659
|
+
purpose follow the rights. Erasure reads a record's people from its events when it has no row
|
|
1660
|
+
(ES-NoProj), refuses one it cannot resolve, lifts with `lifting:` only a restriction of the record
|
|
1661
|
+
erased, and with `everywhere:` leaves another record's restricted copy unmodified
|
|
1662
|
+
(`Result#restricted_copies`, printed by `lyra:erase`, which takes `LIFTING=`). The subject-access copy
|
|
1663
|
+
lists record restrictions under ES-NoProj too. The dashboard's event routes refuse (403) or leave
|
|
1664
|
+
out restricted records (`X-Lyra-Rights-Withheld`). The rights page gives a deadline only to an
|
|
1665
|
+
objection awaiting a decision. The excluded subjects are computed once per snapshot (one pass over
|
|
1666
|
+
the requests in force), the condition compiled once, and a scope whose rights did not change costs
|
|
1667
|
+
one single-row query: a count with 300 restrictions in force went from about 13 ms to about 1 ms.
|
|
1668
|
+
Writes (`update`, `destroy`, bulk writes) are not checked under a restriction in 1.0.0.
|
|
1669
|
+
- **PAM DSL: a type for every Art. 9(1) special category, and for Art. 10 data.** Only `:health`
|
|
1670
|
+
and `:biometric` were special-category types, so a column of any other Art. 9(1) category had
|
|
1671
|
+
to be declared under another type (`:custom`), and a purpose with only an Art. 6 basis could
|
|
1672
|
+
read, say, trade-union membership without PAM raising anything. The new types `:genetic`,
|
|
1673
|
+
`:racial_ethnic_origin`, `:political_opinion`, `:religious_belief` (religious or philosophical
|
|
1674
|
+
beliefs), `:trade_union_membership` and `:sex_life_orientation` are special categories: a purpose
|
|
1675
|
+
that reads one needs an `art9_basis`, else `PamDsl::SpecialCategoryViolationError`. Art. 10 data
|
|
1676
|
+
(criminal convictions and offences) gets the type `:criminal_offence` and a condition of its own,
|
|
1677
|
+
since Art. 10 permits processing only under official authority or when Union or Member State law
|
|
1678
|
+
authorises it, not on an Art. 9(2) ground: a purpose that reads such a field names that
|
|
1679
|
+
authorisation with `art10_authorisation "<the authority or the law>"`, else
|
|
1680
|
+
`PamDsl::CriminalDataViolationError` (strict raises it, audit records it, in `validate_access!`
|
|
1681
|
+
and `validate_fields!`). An Art. 9(2) basis does not satisfy Art. 10, nor an Art. 10
|
|
1682
|
+
authorisation Art. 9. The Article 30 report prints a purpose's Art. 9(2) bases and Art. 10
|
|
1683
|
+
authorisations where it reads such data, the export carries them, and `article_30_gaps` lists
|
|
1684
|
+
a purpose that reads such data without them. The detector, and so the policy generator,
|
|
1685
|
+
proposes the new types from names that say what a column holds (`ethnicity`, `political_party`,
|
|
1686
|
+
`religion`, `trade_union_member`, `sexual_orientation`, `criminal_record`), never from a bare
|
|
1687
|
+
`party`, `union`, `orientation` or `offence`, and the generator marks such fields with the
|
|
1688
|
+
condition a purpose reading them must meet. The dashboard refuses a read that raises the new
|
|
1689
|
+
error as it refuses the others. PAM now has 25 field types (18 before). No field of the Aegean
|
|
1690
|
+
testbed changes type.
|
|
1691
|
+
- **The Art. 15 export is a copy of the data.** `Lyra::Privacy::GDPRCompliance#data_export` keeps
|
|
1692
|
+
its keys, but for `processing_activities` (see Breaking), and adds five. `records` holds each of the subject's records whole: every attribute (not
|
|
1693
|
+
only those a detector flags; encrypted columns in plaintext) and its stored history, `from`/`to`
|
|
1694
|
+
per change. A record is the subject's by what it is: the subject's own row, the rows of monitored
|
|
1695
|
+
models whose `<type>_id` names them (indexed queries), and the records the subject's events touch,
|
|
1696
|
+
so a row that pre-dates monitoring, a destroyed record and an ES-NoProj record are all in it
|
|
1697
|
+
(`state` `:current`, `:destroyed`, `:replayed`); a record whose foreign key moved is the
|
|
1698
|
+
subject's only for the periods it named them (`:reassigned`, `periods`). `accesses` is the access
|
|
1699
|
+
log for those records (`nil` when `record_access_events` is off). `processing_context` takes each
|
|
1700
|
+
record's purposes, bases, recipients, transfers, retention (field periods included) and sources
|
|
1701
|
+
from its own model's policy; `notice` is the policy's `access_notice`; `gaps` lists what the
|
|
1702
|
+
policy does not declare. Who acted is a role label (`"you"`, `"staff"`, `"system"`) unless
|
|
1703
|
+
`disclose_actors: true`. History and the access log are included by default; `history: false`
|
|
1704
|
+
and `accesses: false` are for a request the subject narrowed. A field the policy declares
|
|
1705
|
+
`withheld_from_access: "<basis>"` is named with that basis instead of its value, and one declared
|
|
1706
|
+
`about:` another person goes to `review`; Lyra picks no basis. See
|
|
1707
|
+
[PRIVACY_COMPLIANCE.md](docs/PRIVACY_COMPLIANCE.md#the-copy-of-the-data-from-100).
|
|
1708
|
+
- *Links.* A record refers to the subject through any `belongs_to` whose class is the subject's
|
|
1709
|
+
model (`belongs_to :owner, class_name: "Person"`), a polymorphic `belongs_to` whose type
|
|
1710
|
+
column names it, or the `<type>_id` column; the subject's own model is not linked to itself.
|
|
1711
|
+
- *Moved records.* An event that moves a record between people gives each only their side: the
|
|
1712
|
+
previous subject sees the old values and the link's new value as "moved to another person",
|
|
1713
|
+
the new one only the new values. A record whose links no longer name the subject is
|
|
1714
|
+
`:reassigned` with its state just before the move, whatever comes after. A value last set
|
|
1715
|
+
while the record was another person's is not given: the field is listed under `withheld` with
|
|
1716
|
+
the basis "set while the record was another person's". Accesses to a moved record are the
|
|
1717
|
+
subject's only within their periods.
|
|
1718
|
+
- *The old keys follow the copy's rules* (their content changes): `pii_inventory` and
|
|
1719
|
+
`data_lineage` leave out fields declared `withheld_from_access:` or `about:` another person,
|
|
1720
|
+
give encrypted values in plaintext, and, like `events`, see a moving event with the subject's
|
|
1721
|
+
side only.
|
|
1722
|
+
- *Retention* is what the policy declares: with no rule and no `default`, the period is `nil`
|
|
1723
|
+
and `gaps` lists "Art. 15(1)(d) retention not declared", rather than PAM's silent 7 years.
|
|
1724
|
+
Columns the policy does not declare (no purpose, basis or retention) are listed in `gaps`.
|
|
1725
|
+
- *Accesses* include those recorded under the bare subject id (`access_log_subjects:` on
|
|
1726
|
+
`GDPRCompliance.new` names others, or `[]` for none) and, with the access log off, the
|
|
1727
|
+
subject's earlier exports, with a gap saying the log is off.
|
|
1728
|
+
- **Events record the acting user's class** (`user_type` in the metadata, beside `user_id`, from
|
|
1729
|
+
`Current.user`: the table's owner class for an Active Record user), in the write path, bulk
|
|
1730
|
+
writes and the access log. The copy's role labels compare id and class: "you" only when both
|
|
1731
|
+
are the subject's; "staff or you" when an event records an id equal to the subject's but no
|
|
1732
|
+
class (written before 1.0.0, or by a `metadata_proc` that sets none); "staff" otherwise.
|
|
1733
|
+
- **Each explicit export is recorded,** as `Lyra::Events::SubjectAccessExported` in a stream of
|
|
1734
|
+
its own, `Lyra::SubjectAccessExports$<Type>$<id>`: when, which sections, how many records,
|
|
1735
|
+
`verified_by:` and who exported, never values. Erasing the subject's access log
|
|
1736
|
+
(`AccessLog.erase!`, or `Lyra::Erasure` with `access_log:`) leaves it, so the evidence that a
|
|
1737
|
+
request was answered outlives the log; the copy's `accesses` still lists earlier exports. Only an
|
|
1738
|
+
explicit `data_export` call is recorded: `data_export(record: false)`, `full_report` and the
|
|
1739
|
+
dashboard's subject and GDPR routes are views of the data and record nothing. On by default;
|
|
1740
|
+
`config.record_subject_access_exports = false` turns it off. Nothing is recorded in disabled mode.
|
|
1741
|
+
- **The GDPR reports read each model's policy.** The wrapper passes PAM the policy of each record's
|
|
1742
|
+
model (its `privacy_policy`, else `config.privacy_policy`). `processing_activities` takes the
|
|
1743
|
+
purpose, legal basis, categories, recipients and retention from it, where it printed "Not
|
|
1744
|
+
specified" and guessed the purpose from `metadata[:source]`; `rectification_history` lists an
|
|
1745
|
+
update that changed a declared field as well as one the detector flags.
|
|
1746
|
+
- **The privacy stamp's purposes are those the policy admits the field for,** from either side:
|
|
1747
|
+
the field's `allow_for` and every purpose that lists it in `requires` or `optionally`, as
|
|
1748
|
+
`Policy#allowed?` decides. A policy that lists its fields on the purpose side stamped every field
|
|
1749
|
+
with `"purposes" => []`. `allowed_purposes` on the PAM adapter follows.
|
|
1750
|
+
- **PAM DSL field options and the notice declaration:** `withheld_from_access:`, `about:` and
|
|
1751
|
+
`collected_from` on `field`, and `access_notice` on the policy (see the PAM DSL changelog).
|
|
1752
|
+
- **ES-Lazy reads cost less, with the same consistency.** The check before every read reads the
|
|
1753
|
+
checkpoint and the newest event id in one query instead of two. A reader that finds the log
|
|
1754
|
+
ahead of the tables tries the catch-up lock (`pg_try_advisory_xact_lock`); if another reader holds
|
|
1755
|
+
it, it waits for that catch-up to end, without holding the lock, and reads the checkpoint again.
|
|
1756
|
+
It is done if the checkpoint has reached the newest event it saw and no event the checkpoint
|
|
1757
|
+
lists as missing is visible to it; otherwise it catches up itself, as before. Before, it queued
|
|
1758
|
+
for the lock to repeat a catch-up that, under a write-heavy mix, had mostly been done already. A
|
|
1759
|
+
catch-up that applies nothing no longer rewrites the checkpoint. The three rules hold: only the
|
|
1760
|
+
lock's holder applies events or moves the checkpoint, in the log's order and in its own
|
|
1761
|
+
transaction, and a read sees every event it saw before (a late commit below the checkpoint
|
|
1762
|
+
still within `GAP_RECHECK` on the fast path). In a local profile (30% writes, 70% reads by id,
|
|
1763
|
+
one process) the mean read took about 45% less time at 16 threads and 30% less at 4.
|
|
1764
|
+
- **PAM DSL: a Condition 5 failure raises `PamDsl::SpecialCategoryViolationError`,** where it
|
|
1765
|
+
raised `PamDsl::SensitivityViolationError`. The condition is about special categories (Art. 9,
|
|
1766
|
+
decided by the field's type); sensitivity is a separate Art. 32 risk tier that gates nothing,
|
|
1767
|
+
and the old name mixed the two. `SensitivityViolationError` stays as a deprecated alias of the
|
|
1768
|
+
new class until 2.0, so `rescue PamDsl::SensitivityViolationError` still catches it.
|
|
1769
|
+
- **ModeSync adopts a switch the gate checks only if the gate cleared it.** It adopted the newest
|
|
1770
|
+
`applied` row of `lyra_mode_transitions` and checked nothing, so one `INSERT` moved every process
|
|
1771
|
+
from Monitor to ES-NoProj. A switch now records where it came from (`from_config`), whether the
|
|
1772
|
+
gate was overridden (`forced`) and the certificate of the clean check that cleared it
|
|
1773
|
+
(`certificate_id`). Other processes adopt a switch the gate checks (an escalation from Disabled
|
|
1774
|
+
or Monitor to Hijack or event sourcing, or leaving ES-NoProj, ES-Lazy or ES-Async for a mode
|
|
1775
|
+
that reads the tables) only if the row names that certificate (for the same `from -> to`,
|
|
1776
|
+
recorded before the row, named by no earlier switch) or is forced; whether the gate checks the
|
|
1777
|
+
switch is judged from the row's `from` and from where the previous switch left the application,
|
|
1778
|
+
so a row cannot claim a `from` that avoids the gate. A switch the gate does not check (Monitor to
|
|
1779
|
+
Disabled, ES-Sync to ES-NoProj) is adopted as recorded. No clock is compared, so a switch near
|
|
1780
|
+
the end of its certificate's life or under clock skew is adopted. With the gate off, switches are
|
|
1781
|
+
adopted as recorded. A refusal is logged once; an error while deciding leaves the
|
|
1782
|
+
row to be tried again. Columns Lyra adds to its own tables are added outside the caller's
|
|
1783
|
+
transaction and safely when two processes add one at once. This guards against mistakes and
|
|
1784
|
+
naive writes, not against anyone who can write the table.
|
|
1785
|
+
|
|
1786
|
+
- **The dashboard and `Lyra::EventFlow` read a window of the log, not all of it.** The flow and
|
|
1787
|
+
privacy routes read the newest `config.dashboard_max_events` events (10,000 by default; `?limit=`
|
|
1788
|
+
may only lower it), leaving out the access log's own entries, which would otherwise fill the
|
|
1789
|
+
window as soon as the access log is on, and report `window: { limit:, returned:, truncated:,
|
|
1790
|
+
skipped: }`. A subject's timeline reads that record's own stream (its newest events, up to the
|
|
1791
|
+
limit), not the log; what a user did to other records is the `flow/user_actions` route, which
|
|
1792
|
+
lists at most `?actions=` actions. With `config.record_access_events` on, a dashboard read is
|
|
1793
|
+
recorded in `Lyra::DataAccess$Dashboard` (route, scope, how much came back, never a value). The
|
|
1794
|
+
GDPR report helpers still read the whole log, a batch of 1,000 at a time.
|
|
1795
|
+
- **A save that changes nothing records no event, in every mode.** Monitor, Hijack and event
|
|
1796
|
+
sourcing each appended an `Updated` with no changes for a `save` that changed no attribute.
|
|
1797
|
+
An update carries its changes, and there were none; Hijack and event sourcing still refuse such a
|
|
1798
|
+
save on a destroyed record, as for any write.
|
|
1799
|
+
- **One event shape in every mode**. Monitor, the command handler of Hijack and event
|
|
1800
|
+
sourcing, bypass writes, Genesis and repair now build the payload with `Lyra::EventData`: a
|
|
1801
|
+
create or an import carries the record's `attributes` and no `changes`; an update carries its
|
|
1802
|
+
`changes` as `[old, new]` pairs and no `attributes`; a destroy carries neither. Before, Monitor's
|
|
1803
|
+
events carried the record on an update and a destroy and listed every column in a create's
|
|
1804
|
+
`changes`, and a Hijack or event-sourcing update stored its delta twice (`attributes: { changes: … }`
|
|
1805
|
+
beside `changes`). Every mode now sets `metadata[:source]` (`lyra_interceptor` for Monitor,
|
|
1806
|
+
`lyra_command_handler` for the others). Replay reads a create's `attributes` and an update's
|
|
1807
|
+
`changes`; events in the old per-mode shapes are not supported (see Breaking).
|
|
1808
|
+
The smaller events cost less to serialise and store, and copy less personal data into the log.
|
|
1809
|
+
- **Personal data is detected in an event's fields.** `Lyra::Event.fields_of(event)` gives a create's
|
|
1810
|
+
attributes or an update's new values; the privacy and flow controllers, `EventFlow`, `EventGraph`,
|
|
1811
|
+
`Timeline`, Lyra's GDPR helpers and the PAM DSL's compliance reports use it. They read an event's
|
|
1812
|
+
`attributes` alone, so an update, which now carries none, would have reported no personal data
|
|
1813
|
+
(a Hijack or event-sourcing update already did, because its `attributes` held only `changes`).
|
|
1814
|
+
- **A destroy no longer keeps the record's final state** in Monitor. It is in the events before it;
|
|
1815
|
+
the audit trail view shows no final state for a destroy.
|
|
1816
|
+
- **DualView compares `created_at` and `updated_at`.** They were left out of the comparison; a row
|
|
1817
|
+
whose timestamp differs from its events is now a difference, and `lyra:repair` records it like
|
|
1818
|
+
any other.
|
|
1819
|
+
- **A bare `touch_all` publishes an update per row** whose only change is `updated_at`, as an
|
|
1820
|
+
instance `touch` already did. Bulk bypass events (`update_all`, `upsert_all` and the like) now
|
|
1821
|
+
list `created_at` and `updated_at` among their changes when those moved, so an upsert's event
|
|
1822
|
+
carries `updated_at` beside the columns it set.
|
|
1823
|
+
- **A cheaper destroyed check.** Before every update and destroy in ES-NoProj, ES-Async and
|
|
1824
|
+
ES-Lazy, the check that the record was not destroyed (0.9.3) read and deserialized the stream's
|
|
1825
|
+
last event through Rails Event Store, its data and metadata, to learn its operation. It now
|
|
1826
|
+
decides from the stream's last replayed event (Fixed, above), read with a query over the event
|
|
1827
|
+
store's own tables (`Lyra::StreamTail`) that returns each event's id, type and stored data and
|
|
1828
|
+
deserializes nothing. The usual update costs that one query: the last event is positively a
|
|
1829
|
+
replayed create, import or update when its type ends in `Created`, `Updated` or `Imported` and
|
|
1830
|
+
its data records no destroy's operation and no `replay: false`. Data in a `json` or `jsonb`
|
|
1831
|
+
column comes back as a Hash and is decided from directly. Any other event is read through Rails
|
|
1832
|
+
Event Store by its id, one query each, and an event that is not replayed sends the check to the
|
|
1833
|
+
one before it. The walk back reads one event, then pages of 50, and skips in SQL the events Lyra
|
|
1834
|
+
never replays (`AssociationLinked`, `AssociationUnlinked`, `ErasureApplied`), so a record linked
|
|
1835
|
+
by hundreds of others still costs one query. Domain-event rules and `event_mapping` may no longer
|
|
1836
|
+
use those three names (`ArgumentError`). The type alone cannot decide, since custom event names
|
|
1837
|
+
and domain events can store a destroy under any name. The shortcut is taken only with
|
|
1838
|
+
`RubyEventStore::ActiveRecord::EventRepository`, RES's default mapper and the YAML, JSON or
|
|
1839
|
+
`Lyra::EventSerializer` serializer, and for those it gives the answer a read through Rails Event
|
|
1840
|
+
Store gives. With a mapper that transforms records on load, another serializer or another
|
|
1841
|
+
repository, the stream is read through Rails Event Store. The same happens when the query fails
|
|
1842
|
+
the first time it is tried, for example after a Rails Event Store upgrade that changed its
|
|
1843
|
+
tables. That first try runs during the first check, in a savepoint when a transaction is open,
|
|
1844
|
+
so it cannot abort the write. The lock is unchanged. Locally (no YJIT), the check on a stream
|
|
1845
|
+
ending in an update takes 0.9 ms and 43 allocations instead of 2.3 ms and about 900, and an
|
|
1846
|
+
update or destroy in these three modes allocates about 860 fewer objects. Refusing a write to a
|
|
1847
|
+
destroyed record costs about what it did.
|
|
1848
|
+
- **The bypass-write net (`Lyra::Verification::BypassWorkflow`) models the writes that skip
|
|
1849
|
+
callbacks as 1.0.0 makes them.** A third store, a table that lags the event store (ES-Async,
|
|
1850
|
+
ES-Lazy: `streams_checked` with `table_backed`, as `BypassEvents.checks_streams?` and
|
|
1851
|
+
`locks_rows?`), whose writes read each record's stream under its aggregate lock as ES-NoProj's
|
|
1852
|
+
do; a new ending, `unchanged`, for a write to a record destroyed since it was read (its stream
|
|
1853
|
+
ends in a destroy, or, in ES-Sync, Hijack and Monitor, its row is gone: no write, no event);
|
|
1854
|
+
the `Genesis::Refused` rejection of a row the model could not import, and the
|
|
1855
|
+
`BulkWriteLockLimitError` of a bulk write or `dependent: :nullify` over
|
|
1856
|
+
`bulk_write_lock_limit` or `upsert_all` where the streams are checked (and of `update_all`,
|
|
1857
|
+
`insert_all(!)` and `upsert_all` whose unique values' locks exceed it, ES-Sync included), and that upsert's
|
|
1858
|
+
`RecordDestroyedError` (see Fixed); and `insert_all(!)` as a class of its own (a new row:
|
|
1859
|
+
nothing to admit, written to the table in ES-NoProj too; a key already there is skipped by
|
|
1860
|
+
`insert_all`, unchanged, and fails `insert_all!`). `BypassWorkflow.coverage`
|
|
1861
|
+
counts a dead marking as covered only when it ends in exactly one way: `rejected` alone,
|
|
1862
|
+
`store_changed` with `event_logged`, or `unchanged` alone (and returns how many end
|
|
1863
|
+
unchanged, `unchanged:`); a store change without an event, an event logged for an unchanged
|
|
1864
|
+
write, or a rejection beside a write or an event is reported. `ROUTED` lists ES-NoProj's
|
|
1865
|
+
class-level `update_counters`, `increment_counter` and `decrement_counter`, which write through
|
|
1866
|
+
`update_all`. Rails' association writes (`Lyra::Associations::Maintenance`, see Fixed) enter
|
|
1867
|
+
past the strict guard: `request_nullify` is the nullify Rails marked (an application's
|
|
1868
|
+
`update_all(fk: nil)` is `request_bulk`), a `has_one`'s `dependent: :nullify` the column path
|
|
1869
|
+
(`request_association_column`), removing records from a `has_many :through` over a monitored
|
|
1870
|
+
model the bulk path (`request_through_delete`). `has_and_belongs_to_many` join-row deletes are
|
|
1871
|
+
a class of their own, `link`: Rails' marked deletes are never refused (`request_unlink`), the
|
|
1872
|
+
application's `delete_all` on a join model is guarded (`request_join`), and either writes the
|
|
1873
|
+
table, in every store, then logs `AssociationUnlinked` (`write_unlink`, `publish_unlink`); the
|
|
1874
|
+
application's `insert_all`, `insert_all!` and `upsert_all` of join rows are guarded the same
|
|
1875
|
+
way (`request_join_insert`), write, then log `AssociationLinked` (`write_link`, `publish_link`),
|
|
1876
|
+
or end unchanged (`skip_conflict_link`) or refused (`refuse_conflict_link`). Bulk inserts and
|
|
1877
|
+
upserts are first brought to the rows Monitor's table would have where the streams are checked
|
|
1878
|
+
(`bring_insert_keys_up_to_date`, `bring_upsert_rows_up_to_date`; `insert_as_given` and
|
|
1879
|
+
`upsert_as_given` elsewhere), and a bulk insert's locks can exceed the limit
|
|
1880
|
+
(`refuse_lock_limit_insert`); where it reads keys or unique values from the event store, Genesis
|
|
1881
|
+
refuses it for a model whose import was refused (`refuse_genesis_insert`). An `update_column(s)` or `update_all` whose value a unique index
|
|
1882
|
+
refuses is rejected, writing and logging nothing (`refuse_unique_column`, `refuse_unique_bulk`:
|
|
1883
|
+
the table's `UPDATE` raises, and in ES-NoProj, ES-Async and ES-Lazy the event store is checked,
|
|
1884
|
+
see Fixed), and so is an `upsert_all` whose value of an index other than its conflict columns
|
|
1885
|
+
another record holds (`refuse_unique_upsert`: the table's statement, or in ES-Async and ES-Lazy
|
|
1886
|
+
the event store). The net now has 38 places, 87 transitions and 181 reachable markings (was 25,
|
|
1887
|
+
36 and 72); it is 1-bounded, has no dead transition, and all 18 dead markings (was 7) are
|
|
1888
|
+
covered.
|
|
1889
|
+
|
|
1890
|
+
### Removed
|
|
1891
|
+
- **`Lyra::Associations::EventAware`** (`event_aware_belongs_to`, `event_aware_has_many`), an
|
|
1892
|
+
opt-in concern that nothing used, superseded by the association interceptor, read-your-writes
|
|
1893
|
+
and ES-Lazy. Its `has_many` never added pending records (its `PendingRecords` returned none),
|
|
1894
|
+
and the records its `belongs_to` rebuilt from events (`VirtualRecord`) bypassed Lyra's read
|
|
1895
|
+
path: their values were the events' own, not converted to the attribute types; the target's
|
|
1896
|
+
default scopes were not applied, so a record the scoped lookup did not find (soft-deleted, say)
|
|
1897
|
+
came back from its events; and purpose-bound reads did not check them.
|
|
1898
|
+
|
|
1899
|
+
### Documented
|
|
1900
|
+
- **`allow_for` widens, it does not restrict.** A field may be used for a purpose when its
|
|
1901
|
+
`allow_for` names the purpose or the purpose's `requires`/`optionally` names the field
|
|
1902
|
+
(`Policy#allowed?` takes the union, as before). The PAM DSL README, PRIVACY_COMPLIANCE.md,
|
|
1903
|
+
API_REFERENCE.md and the methods' comments now say so. No behaviour change.
|
|
1904
|
+
|
|
1905
|
+
|
|
1906
|
+
### Known issues
|
|
1907
|
+
- **ES-Lazy: a transaction that reads and then takes a record's aggregate lock can deadlock against a
|
|
1908
|
+
bulk write.** The catch-up a read begins with writes the rows of the records whose events it
|
|
1909
|
+
applies without their aggregate locks, and inside a transaction those rows stay locked until it
|
|
1910
|
+
ends. When the transaction then takes a record's aggregate lock (a write through callbacks,
|
|
1911
|
+
`update_all`) that a bulk write holds while it waits for one of those rows (to bring it up to
|
|
1912
|
+
date before writing it), the two wait for each other; PostgreSQL detects it and aborts one with
|
|
1913
|
+
`ActiveRecord::Deadlocked`, and nothing of the aborted one is written or recorded. Retry the
|
|
1914
|
+
transaction, or do the reads outside it (docs/TROUBLESHOOTING.md, Concurrency Issues). A
|
|
1915
|
+
transaction that writes several records in an order of its own can deadlock against another, or
|
|
1916
|
+
a bulk write, as plain Active Record's row locks do; take them in one order (by primary key) and
|
|
1917
|
+
retry on `ActiveRecord::Deadlocked` (same section).
|
|
1918
|
+
|
|
10
1919
|
## [0.9.3] - 2026-11-04
|
|
11
1920
|
|
|
12
1921
|
**Security update.** A thorough review of Lyra's code against the claims made for it found the
|
|
@@ -329,8 +2238,8 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
329
2238
|
replays to the anonymized row. Replacement is nil where allowed, `"erased:<id>"` for a NOT NULL
|
|
330
2239
|
string, else the column default. Works in every mode, with strict data access on and inside a
|
|
331
2240
|
purpose. On request only; not policy-driven, and crypto-shredding remains future work.
|
|
332
|
-
- **Event store failure policies, named** (`Lyra::EventStoreUnavailableError`, `Lyra.append_events
|
|
333
|
-
|
|
2241
|
+
- **Event store failure policies, named** (`Lyra::EventStoreUnavailableError`, `Lyra.append_events`)
|
|
2242
|
+
— every event Lyra writes now goes through `Lyra.append_events`, which
|
|
334
2243
|
raises `EventStoreUnavailableError` (the store's error as its `cause`, the stream in its message)
|
|
335
2244
|
when the store fails. The policy is fixed by the mode: **fail-closed** in Hijack and every
|
|
336
2245
|
event-sourcing mode (the write fails with the error and rolls back), **log-and-continue** in
|
|
@@ -346,7 +2255,7 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
346
2255
|
where the events are authoritative (rebuild the tables instead). `ModeTransition.record_ids` and
|
|
347
2256
|
`ModeTransition.discrepancy` expose the check it uses.
|
|
348
2257
|
- **Access log** (`config.record_access_events`, default false; `Lyra::AccessLog`,
|
|
349
|
-
`Lyra::Events::DataAccessed`, `Lyra::Events::DataAccessDenied
|
|
2258
|
+
`Lyra::Events::DataAccessed`, `Lyra::Events::DataAccessDenied`) — when on,
|
|
350
2259
|
every access the privacy policy validates (`validate_access!`) is recorded in the subject's
|
|
351
2260
|
access stream (`Lyra::DataAccess$<Model>$<id>`): `DataAccessed` when it went ahead (outcome
|
|
352
2261
|
`granted`, or `audited` with the violations audit mode let through), `DataAccessDenied` when
|
|
@@ -362,7 +2271,7 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
362
2271
|
validate access, and records it in its benchmark reports. The Article 30 register stays
|
|
363
2272
|
PAM's `article_30_report`, generated from the declared policy.
|
|
364
2273
|
- **Privacy stamps on events** (`config.annotate_privacy`, default false; `Lyra::Privacy.stamp`,
|
|
365
|
-
`Lyra::Privacy.stamp_of
|
|
2274
|
+
`Lyra::Privacy.stamp_of`) — when on, every event that carries an
|
|
366
2275
|
attribute the model's privacy policy declares is stamped as it is built with
|
|
367
2276
|
`metadata[:privacy]`: the policy's name and, per attribute, its type, sensitivity, allowed
|
|
368
2277
|
purposes, retention period (ISO 8601) and the contexts it has a transformation for — never a
|
|
@@ -385,7 +2294,7 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
385
2294
|
switch reaches. On wherever the Mode Transition Safety gate is (not in the test environment);
|
|
386
2295
|
the raw `config.mode =` setter is left alone. the migration guide (`docs/MIGRATION_GUIDE.md`, Switching modes) documents the
|
|
387
2296
|
deploy-time rule.
|
|
388
|
-
- **Mode Transition Safety** (`Lyra::ModeTransition`, `rake lyra:mode:check
|
|
2297
|
+
- **Mode Transition Safety** (`Lyra::ModeTransition`, `rake lyra:mode:check`)
|
|
389
2298
|
— a mode switch that changes the authoritative store is allowed only when rows and events
|
|
390
2299
|
agree for every monitored record: escalating from Disabled or Monitor to Hijack or event
|
|
391
2300
|
sourcing (rows that predate Lyra are imported by Genesis first), and leaving ES-NoProj,
|
|
@@ -402,8 +2311,8 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
402
2311
|
`config.dual_view_discrepancy_handler`) — after a write commits, a sampled share of writes
|
|
403
2312
|
is compared with its events; a discrepancy is logged and passed to the handler, and the write
|
|
404
2313
|
is never failed. Skipped in ES-NoProj, ES-Lazy and ES-Async, whose tables lag by design.
|
|
405
|
-
- **Domain events** (`monitor_with_lyra domain_events: [...]`, `Lyra::DomainEvents
|
|
406
|
-
|
|
2314
|
+
- **Domain events** (`monitor_with_lyra domain_events: [...]`, `Lyra::DomainEvents`)
|
|
2315
|
+
— rules name a write's event after what it means instead of the CRUD
|
|
407
2316
|
operation: `{ name: "PaymentCompleted", on: %i[create update], if: ->(payment, changes) { ... } }`.
|
|
408
2317
|
The first matching rule names the write's own event, which keeps Lyra's envelope (model, id,
|
|
409
2318
|
operation, attributes, changes), so replay, projections, DualView and `Lyra.state_at` work
|
|
@@ -416,7 +2325,7 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
416
2325
|
callbacks are still recorded as CRUD bypass events. Generated event classes are registered
|
|
417
2326
|
at boot. A rule whose block raises fails the write where events are the record (Hijack,
|
|
418
2327
|
event sourcing) and is logged in Monitor. Commands now carry their `record`.
|
|
419
|
-
- **Configuration surface
|
|
2328
|
+
- **Configuration surface**: `config.models = %w[Order Payment]` (or a
|
|
420
2329
|
hash of name => options) declares the monitored models by name in the initializer; once the
|
|
421
2330
|
application's code has loaded, each is given `monitor_with_lyra`, with no model file edited,
|
|
422
2331
|
and a name that does not resolve fails the boot. `config.verify_mapping!` verifies the
|
|
@@ -431,8 +2340,7 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
431
2340
|
destroyed. `Model.as_of(time).find(id)` / `.find_by_id(id)` return read-only records and
|
|
432
2341
|
`.all` every record that existed then. For a row imported by Genesis, a time before its
|
|
433
2342
|
import raises `Lyra::Temporal::HistoryNotRecorded` instead of guessing (a time before its own
|
|
434
|
-
`created_at` is `nil`). Works in every mode that records events, Monitor included
|
|
435
|
-
(FEATURE_GAP_PLAN F7).
|
|
2343
|
+
`created_at` is `nil`). Works in every mode that records events, Monitor included.
|
|
436
2344
|
- **ES-NoProj joins, evaluated in memory** (`Lyra::Projections::CachedJoins`) — `joins`,
|
|
437
2345
|
`left_joins`/`left_outer_joins`, `where.missing` and `where.associated` on `belongs_to`,
|
|
438
2346
|
`has_many` and `has_one` associations, with hash conditions on the joined table
|
|
@@ -451,9 +2359,8 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
451
2359
|
no provider is installed. PAM is an adapter (`Lyra::Privacy::Adapters::Pam`), loaded when
|
|
452
2360
|
`pam_dsl` is present. `Lyra::Privacy.annotations_for(event)` returns the annotation of every
|
|
453
2361
|
declared attribute an event touched. The foundation for `config.privacy_policy=` and PII
|
|
454
|
-
annotation in event metadata
|
|
455
|
-
|
|
456
|
-
policy (d0df542), which adds nothing to the write path.
|
|
2362
|
+
annotation in event metadata. The testbed binds `Registration` and `PaymentTransaction` to its
|
|
2363
|
+
`:epay_data` policy, which adds nothing to the write path.
|
|
457
2364
|
- **`Lyra::IdGenerator.reserves_safely?(model)`** — true for UUID keys and for integer keys
|
|
458
2365
|
backed by a PostgreSQL sequence. It uses the cached sequence lookup, so it adds no query per
|
|
459
2366
|
write once a table's sequence is known.
|
|
@@ -464,8 +2371,8 @@ Everything since 0.8.0, the version the PAM article describes.
|
|
|
464
2371
|
row that changed; `delete_all` "destroyed" per row; inserts "created" per row; `upsert_all`
|
|
465
2372
|
finds existing rows by its conflict key (primary key, `unique_by:` columns or index name) and
|
|
466
2373
|
publishes "updated" for those, "created" for the rest. The caller's `returning:` result is
|
|
467
|
-
unchanged
|
|
468
|
-
|
|
2374
|
+
unchanged, and the instance methods (`update_column(s)`, `touch`, `delete`) still build their
|
|
2375
|
+
events after the write.
|
|
469
2376
|
- **`Lyra.projection_write { ... }`** — Lyra's own read-model writes: skips strict data access
|
|
470
2377
|
and publishes no bypass events. `ModelProjection` and `Rebuild` use it; use it for writes that
|
|
471
2378
|
must not reach the event stream, such as seeding or wiping a table.
|
|
@@ -876,8 +2783,7 @@ ignored like any unknown option.
|
|
|
876
2783
|
return records a write had already changed. Stamped entries (see Changed) make all three
|
|
877
2784
|
impossible: a stale entry is never used.
|
|
878
2785
|
- **An explicit id was replaced** — `CommandHandler#handle_create` generated an id even when the
|
|
879
|
-
application set one; it now keeps it, in the event and in the row.
|
|
880
|
-
(526908e), applied to master's own Hijack fix (cd3ea89).
|
|
2786
|
+
application set one; it now keeps it, in the event and in the row.
|
|
881
2787
|
- **ES-NoProj `insert_all` ran, then raised** — through `CachedRelation`'s scope branch the
|
|
882
2788
|
insert executed and the caller then got `UnsupportedQuery` ("not a scope"). Inserts and
|
|
883
2789
|
upserts now go to the table relation and return its result.
|