orfeas_lyra 0.9.0 → 0.9.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8e940368c4357d27be315d08d22f791d399356c36884472120c6c2a4f82731fe
-  data.tar.gz: f59258ee1575e968c436cd1f4ab0cd8ff567438d5571f8f1fd438ccfb944446c
+  metadata.gz: 4a75016e10096d61497bb69e58b2570d23ca9a99a8206377fe94216d1db26eb9
+  data.tar.gz: a139ef32e0dd3e83ba2c5ce1f182e6cb83360f2208faea20a1330536ee63edae
 SHA512:
-  metadata.gz: 5e6b4a7a5ce88cbc5c10434ef0b6af0a4f09c6109f66af253fdab4a6771b9e23f3965a59bd0f5e56932358eda75c590d6d8d0b415be961e4c9b3cf42a908ca70
-  data.tar.gz: 15d85548a5cdbc546fa86a2fe263e4be6d137e6e0597f17f13e5e75ceba2763b27c12315b3643e43f7712f134851d006ea20263eb8fcb8d6f03c26f6e7ade141
+  metadata.gz: 9d8e2c114fcae82c8dd8735758da868eb4e2aef90f4c7ad53f1385b6f39d8d182f57e9026d8e90d4a283501ae6a7fcdb97889d989183deee427252794fb1d8b4
+  data.tar.gz: 6817174882333de4336a7907cae0a47c1690712f85ec1f1cb96cd676e4b6ada397cf17c64a1249a5a96623cc5a33ee60c728f5c92d1ffc80bcede9ac4aa76e42

data/CHANGELOG.md

@@ -7,10 +7,81 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.9.0] - 2026-10-14
+## [0.9.2] - 2026-11-03
+
+One fix in Lyra; PAM DSL and PetriFlow are unchanged (0.9.1).
+
+### Fixed
+- **ES-Async read-your-writes no longer races a projection job.** With read-your-writes
+  (`each_write: true`), the writer projected its own event without the per-stream lock that
+  `AsyncProjectionJob` holds. A job that had read the record's stream before that event
+  committed could write the row afterwards, putting it back to the older value until the newer
+  event's job ran: a reader in between saw the record step back (an application's suite, run
+  with real background projection, read a refund as "approved" just after it was "processed").
+  `Lyra::Consistency::ReadYourWrites` now takes the job's lock before projecting, held to the
+  end of the writer's transaction. Final states converged before as well. Plain ES-Async (no
+  read-your-writes) was not affected. `test/concurrent_same_record_test.rb` reproduces the
+  interleaving.
+
+## [0.9.1] - 2026-11-02
+
+Lyra's changes since 0.9.0; PAM DSL and PetriFlow change only in their gem descriptions.
+
+### Released
+- The gems on RubyGems (`orfeas_lyra`, `orfeas_petri_flow`, `orfeas_pam_dsl`), built from tag
+  `v0.9.1` and pushed on 2026-11-02
+
+### Added
+- **Writes are grouped into causal chains by default** (`config.correlate_writes`, on). Before,
+  Lyra carried correlation and causation only where the application called
+  `Lyra::Correlation.with_id`/`Lyra::Causation.with_id`, which no example did: every event was its
+  own chain. Now:
+  - every event-producing write runs in a write scope (`around_save`/`around_destroy`, first in
+    the chain) that fixes its event id before its callbacks run, so a write made inside them
+    records that event as its `causation_id`; a record saving itself inside its own callbacks is
+    caused by the outer save;
+  - the outermost write opens the correlation when none is in scope, its own event id, as the
+    event store does for a root event;
+  - `Lyra::Correlation::Middleware`, installed by the engine, gives the writes of one HTTP request
+    one correlation, the request id; a correlation already in scope is kept;
+  - `Lyra::Correlation::Job`, included into `ActiveJob::Base` by the engine, gives the writes of
+    one job the correlation in scope where the job was created, or the job id without one, and a
+    write in scope there as their cause. Both are taken when the job is created, so a job deferred
+    to after the commit (`enqueue_after_transaction_commit`) keeps them, and travel in its payload
+    (`lyra_correlation_id`, `lyra_causation_id`), so a retry does too. A job created inside a
+    transaction and sent at once (the Rails default) could outlive a rollback of its cause; its
+    cause is kept when it runs only if that event is in the log, so no job names a cause the log
+    lacks (one that runs before the enqueuing transaction commits loses the link).
+  Monitor, Hijack and event-sourcing events alike (`DomainEvents.build` takes the pre-assigned
+  `event_id:`, given once per write). The Olist replay runs each step as one request would
+  (`test/write_correlation_test.rb`).
+
+### Changed
+- **ORFEAS is expanded as the words its letters stand for**: Object-Relational Framework for
+  Event-Aware Systems (README, `docs/ARCHITECTURE.md` and its citation, gem descriptions). It
+  read "Object-Relational to Event-Sourcing Architecture", which has no F.
+
+### Fixed
+- **Writes that skip callbacks name who made them.** Events of `update_columns`, `touch`,
+  `delete`, `update_all`, `delete_all`, `insert_all`, `upsert_all` and `dependent: :nullify`, and
+  `has_and_belongs_to_many` links, carried correlation and causation but neither `user_id` nor
+  `request_id`, so the same request's writes named a user only if they ran callbacks. They now
+  take both from `Current`, as the documentation already said. Events written before this keep no
+  actor; their correlation still ties them to the request's other writes.
+- **`AsyncProjectionJob` warns again when its event is not in the log.** The warning sat in a
+  rescue of `RubyEventStore::EventNotFound`, which `read.event` never raises (it returns nil;
+  `read.event!` raises), so it was never logged. The job still did nothing for a missing event.
+
+## [0.9.0] - 2026-10-25
 
 Everything since 0.8.0, the version the PAM article describes.
 
+### Released
+- The gems on RubyGems (`orfeas_lyra`, `orfeas_petri_flow`, `orfeas_pam_dsl`), built from tag
+  `v0.9.0` (2026-10-19); the entries marked *after the tag* are not in them
+- The replication package: https://doi.org/10.5281/zenodo.17770613
+- The benchmark results: https://doi.org/10.5281/zenodo.23640846
+
 ### Added
 - **`config.models` takes namespace patterns, `:all` and a denylist** (`Lyra::ModelSelection`):
   `config.models = %w[Spree::* Order]`, `config.models = :all`,
@@ -344,6 +415,14 @@ Everything since 0.8.0, the version the PAM article describes.
   YAML serializer) was not affected. Found by replaying the BPI Challenge 2017 log
   (`examples/bpi2017_loan_app`), whose timestamps carry milliseconds.
 ### Changed
+- **Documentation** (*after the tag*) — `docs/PERFORMANCE.md` gives the 0.9.0 measurement (three runs, with
+  controls for PaperTrail and for the lock on the aggregate), the SQL statements per write under the
+  lock, and the published runs (https://doi.org/10.5281/zenodo.23640846); the install instructions
+  take the gems from RubyGems (`"~> 0.9"`); the case studies, benchmark harness and formal-model
+  scripts are referred to as the replication package (https://doi.org/10.5281/zenodo.17770613).
+  Comments no longer cite issue numbers; `lib/` changes in comments only.
+- **The Rakefile's testbed tasks** (*after the tag*) say where the testbed is when it is absent, and `stats:all`
+  skips it.
 - **ES-NoProj reads**: a stale cache entry is brought up to date from the events after its stamp
   instead of a full replay; cache entries have a process-local copy in front of the shared cache;
   an association's records are read when first needed, and a reference-link lookup intersects
@@ -454,7 +533,10 @@ ignored like any unknown option.
   write path always uses `Lyra::CommandHandler`.
 
 ### Fixed
-- **A thread that wrote through Lyra kept a database connection until it ended (mpantel/lyra#37).** Lyra
+- **`trace_conformance_test.rb` on a fresh database** (*after the tag*) — the recorder took the commit of Lyra's own
+  checkpoint table, created on a connection of its own, for part of the write; it now records only
+  the write's connection. Test only.
+- **A thread that wrote through Lyra kept a database connection until it ended.** Lyra
   called `Model.connection` (Rails' `lease_connection`) in its write, lock and read paths: the
   aggregate lock, the id generator, Genesis, ES-Async's job, ES-NoProj's cache and reference links,
   ES-Lazy's catch-up, and its maintenance calls (DualView, Rebuild, mode checks, erasure). Since
@@ -470,7 +552,7 @@ ignored like any unknown option.
   returns whatever its work leased. `test/connection_release_test.rb`: in every mode, a thread that
   writes and reads, bare or in `with_connection`, ends without a connection, and `lib/` has no
   `Model.connection` call left.
-- **Genesis and the reference-link catch-up assumed a second connection (#37).** Inside an open
+- **Genesis and the reference-link catch-up assumed a second connection.** Inside an open
   transaction both ran their work on a thread that checked out a connection of its own and waited
   for it. Under Rails' transactional tests the pool's connection is pinned, every checkout returns
   the caller's own, locked connection, and the wait never ended: the Aegean testbed suite in
@@ -491,7 +573,7 @@ ignored like any unknown option.
   association's links with the join table, `Rebuild` restores join tables from the events, and
   Genesis records the links pre-existing rows had. Under ES-NoProj, HABTM writes failed before
   reaching Lyra (`CachedRelation` lacked `where_values_hash`); they now work and read back
-  (`test/habtm_links_test.rb`, every mode; mpantel/lyra#21).
+  (`test/habtm_links_test.rb`, every mode).
 - **Optimistic locking: a stale update silently won under event sourcing.** Active Record checks
   `lock_version` in the UPDATE or DELETE that writes the row; ES-Sync, ES-Async, ES-NoProj and
   ES-Lazy skip that statement, so a stale `update` or `destroy` overwrote the newer write (a lost
@@ -503,7 +585,7 @@ ignored like any unknown option.
   model in every event-sourcing mode), and raise `ActiveRecord::StaleObjectError` as Active Record
   does (`Lyra::OptimisticLocking`). A record that expects a newer version than the log's passes,
   so streams written before the change keep working (`test/optimistic_locking_test.rb`, every
-  mode, concurrent writers included; mpantel/lyra#20).
+  mode, concurrent writers included).
 - **Encrypted attributes (Active Record `encrypts`) reached the event log in clear** while their
   row held ciphertext: events are built from attributes Active Record has already decrypted.
   `Lyra.append_events`, through which every event passes, now stores those attributes' values (in
@@ -511,7 +593,7 @@ ignored like any unknown option.
   (`Lyra::EncryptedAttributes`); projections, rebuilds and DualView decrypt them, and records
   instantiated from events are decrypted by Active Record. Events written before keep their clear
   values and still read; rows written from them are encrypted. No mode leaks the value, no value is
-  encrypted twice (`test/encrypted_attributes_test.rb`; mpantel/lyra#19).
+  encrypted twice (`test/encrypted_attributes_test.rb`).
 - **ES-Lazy stopped projecting after the event store was emptied or restored without its
   checkpoint table**: the checkpoint then lay past the newest event, and every new event looked
   applied. A checkpoint past the newest event is now recognised, warned about and reset (applying
@@ -589,8 +671,8 @@ ignored like any unknown option.
   are now the same whether one mode or all are generated (`monitor_mode_workflow.rb` defines
   `MonitorModeWorkflow`, `es_sync_mode_workflow.rb` `EsSyncModeWorkflow`;
   `WorkflowGenerator.workflow_file_basename(mode)`, `.workflow_class_name(mode)`).
-- **`already initialized constant Lyra::VERSION` in the monorepo** — Bundler's default gemspec
-  search also loaded the nested `lyra-engine/` checkout's `lyra.gemspec`. The root `Gemfile`
+- **`already initialized constant Lyra::VERSION` in a checkout holding a nested copy of the
+  engine** — Bundler's default gemspec search also loaded the nested copy's `lyra.gemspec`. The root `Gemfile`
   uses `gemspec glob: "lyra.gemspec"`, and the example apps' Gemfiles `glob: "lyra.gemspec"`.
 - **Hijack and event-sourcing events lost who wrote them** — the command handler wrote only
   the causal chain, so `user_id`, `request_id`, the user action and `config.metadata_proc`
@@ -651,11 +733,11 @@ ignored like any unknown option.
   (`CachedRelationDelivery`), as are the class-level `find`, `find_by` and `find_by!`.
 - **The all-modes runner skipped ES-Lazy** — `lyra:test:all_modes` and the PAM-less run now cover
   all seven configurations.
-- **Lyra's rake tasks ran twice, and host apps got the monorepo's maintenance tasks** — Rails
+- **Lyra's rake tasks ran twice, and host apps got the development checkout's maintenance tasks** — Rails
   loads every `lib/tasks/*.rake` under an engine's root, and the engine also loaded its three
   task files explicitly, so each `lyra:mode:*`, `lyra:projections:*` and `lyra:schema:*` task
   ran twice (`lyra:mode:check` ran its full check twice), and `gems:*`, `public:*`, `stats:*` and
-  the other monorepo tasks appeared in every host app. The engine now loads `lyra_*.rake` only,
+  the other maintenance tasks appeared in every host app. The engine now loads `lyra_*.rake` only,
   once.
 - **`lyra:mode:check` could certify a switch having checked nothing** — monitored models register
   as their classes load, which a rake task in development does not do, so the check saw no

data/README.md

@@ -2,7 +2,7 @@
 
 **CRUD to Event Sourcing Transformation Engine**
 
-*Part of the ORFEAS (Object-Relational to Event-Sourcing Architecture) Framework*
+*Part of ORFEAS (Object-Relational Framework for Event-Aware Systems)*
 
 Lyra is a Rails engine that records the writes an ActiveRecord application
 makes as events (RailsEventStore), and can then move the application, one
@@ -52,13 +52,14 @@ seven configurations:
 - PostgreSQL. Lyra depends on the `pg` gem; the advisory locks behind Genesis
   and ES-Lazy, and id reservation in Hijack mode, need PostgreSQL.
 
-The gem names differ from the files to require:
+The gems are on RubyGems, at 0.9.1 (2 November 2026). Their names differ from the
+files to require:
 
 ```ruby
 # Gemfile
-gem "orfeas_lyra", path: "path/to/lyra", require: "lyra"
-gem "orfeas_pam_dsl", path: "path/to/lyra/gems/pam_dsl", require: "pam_dsl"          # optional, privacy features
-gem "orfeas_petri_flow", path: "path/to/lyra/gems/petri_flow", require: "petri_flow"  # optional, verification
+gem "orfeas_lyra", "~> 0.9", require: "lyra"
+gem "orfeas_pam_dsl", "~> 0.9", require: "pam_dsl"        # optional, privacy features
+gem "orfeas_petri_flow", "~> 0.9", require: "petri_flow"  # optional, verification
 ```
 
 Create the event store tables with the RailsEventStore 3 generator
@@ -139,7 +140,7 @@ individual calls.
 - **[Architecture Overview](docs/ARCHITECTURE.md)** - System design and components
 - **[Switching Modes](docs/MIGRATION_GUIDE.md#switching-modes)** - Who holds the mode, and how to switch safely (the deploy-time rule)
 - **[Adopting Lyra](docs/MIGRATION_GUIDE.md#adopting-lyra-in-an-existing-application)** - What "non-intrusive" means, the costs, and a checklist for a new codebase
-- **[Performance](docs/PERFORMANCE.md)** - What each mode costs and why (counted in SQL statements per write; throughput being re-measured)
+- **[Performance](docs/PERFORMANCE.md)** - What each mode costs and why (measured on Lyra 0.9.0, with the SQL statements per write)
 - **[Troubleshooting](docs/TROUBLESHOOTING.md)** - Common problems
 - **[Workflow Generator](docs/WORKFLOW_GENERATOR.md)** - The Petri net workflow models of Lyra's modes and their verification
 - **[Changelog](CHANGELOG.md)**
@@ -189,6 +190,18 @@ DOI: [10.1145/3575879.3576006](https://doi.org/10.1145/3575879.3576006)
 EGOVIS 2022, Lecture Notes in Computer Science 13429, Springer, pp. 18–31.
 DOI: [10.1007/978-3-031-12673-4_2](https://doi.org/10.1007/978-3-031-12673-4_2)
 
+### Code and data behind the papers
+
+Each paper's code is a tag or commit of this repository; its data and the rest of its
+replication material are on Zenodo. The case studies (the Aegean ePay testbed, Solidus
+with the Olist orders, the BPI 2017 loan application) are in the replication packages.
+
+| Paper | Code | Data and replication material |
+|:------|:-----|:------------------------------|
+| *From ORM to OEM* (IEEE Software, 2026) | tag [`paperoem_final`](https://github.com/mpantel/lyra-engine/tree/paperoem_final), commit `b863d2e` (numbered 0.6.0, three commits after the [`v0.6.0`](https://github.com/mpantel/lyra-engine/tree/v0.6.0) release) | The benchmark run, its SQL statement trace, the per-trial values and 95% confidence intervals, and the harness as measured: [10.5281/zenodo.23648730](https://doi.org/10.5281/zenodo.23648730) |
+| *PAM* (Requirements Engineering, 2026) | tag [`v0.8.0`](https://github.com/mpantel/lyra-engine/tree/v0.8.0); `orfeas_pam_dsl` 0.8.0 on RubyGems | The policies, the testbed's models, tests and reports, the Solidus replication and the microbenchmark: [10.5281/zenodo.23649008](https://doi.org/10.5281/zenodo.23649008) |
+| *ORFEAS* (IEEE Transactions on Software Engineering, under review) | tag [`v0.9.0`](https://github.com/mpantel/lyra-engine/tree/v0.9.0); `orfeas_lyra` 0.9.0 on RubyGems | Replication package: [10.5281/zenodo.23646667](https://doi.org/10.5281/zenodo.23646667) (all versions: [10.5281/zenodo.17770613](https://doi.org/10.5281/zenodo.17770613)); benchmark results (CC BY-NC-SA 4.0): [10.5281/zenodo.23640846](https://doi.org/10.5281/zenodo.23640846) |
+
 ### Research Areas
 
 - **Event Sourcing Patterns**: Formal models for CRUD-to-event transformation
@@ -269,7 +282,8 @@ If you use this software in academic research, please cite:
   author={Pantelelis, Michail},
   year={2026},
   note={Part of ORFEAS Framework},
-  url={https://github.com/mpantel/lyra-engine}
+  url={https://github.com/mpantel/lyra-engine},
+  doi={10.5281/zenodo.17770613}
 }
 ```
 

data/Rakefile

@@ -57,6 +57,9 @@ task default: :test
 # Shared constants
 # =============================================================================
 TESTBED_DIR = File.expand_path("examples/aegean_epay_testbed", __dir__)
+# The testbed is in the replication package, not in the lyra-engine repository.
+TESTBED_MISSING = "The Aegean testbed is not in this checkout; it is in the replication package " \
+                  "(https://doi.org/10.5281/zenodo.17770613)."
 
 # =============================================================================
 # Benchmark tasks
@@ -69,6 +72,8 @@ namespace :benchmark do
     # Usage: rake benchmark:modes -- --scales=1000,5000 --modes=disabled,monitor
     args = ARGV.drop_while { |a| a != "--" }.drop(1).join(" ")
 
+    abort TESTBED_MISSING unless File.directory?(TESTBED_DIR)
+
     Dir.chdir(TESTBED_DIR) do
       # Use bundle exec with testbed's Gemfile to ensure correct dependencies
       cmd = "bundle exec ruby perf/run_mode_comparison_benchmark.rb #{args}"
@@ -300,8 +305,10 @@ namespace :stats do
     results[:petri_flow] = analyze_directory("gems/petri_flow/lib", "gems/petri_flow/test", "PetriFlow")
 
     # Aegean E-Pay Testbed
-    puts "Analyzing Aegean E-Pay Testbed..."
-    results[:testbed] = analyze_rails_app("examples/aegean_epay_testbed", "Aegean E-Pay Testbed")
+    if File.directory?(TESTBED_DIR)
+      puts "Analyzing Aegean E-Pay Testbed..."
+      results[:testbed] = analyze_rails_app("examples/aegean_epay_testbed", "Aegean E-Pay Testbed")
+    end
 
     puts ""
 
@@ -337,6 +344,7 @@ namespace :stats do
 
   desc "Generate statistics for Aegean E-Pay Testbed"
   task :testbed do
+    abort TESTBED_MISSING unless File.directory?(TESTBED_DIR)
     result = analyze_rails_app("examples/aegean_epay_testbed", "Aegean E-Pay Testbed")
     print_component_report(result)
   end
@@ -538,7 +546,7 @@ namespace :stats do
     require "json"
     require "fileutils"
 
-    report_dir = "examples/aegean_epay_testbed/reports"
+    report_dir = File.directory?(TESTBED_DIR) ? "examples/aegean_epay_testbed/reports" : "tmp/stats"
     FileUtils.mkdir_p(report_dir)
 
     timestamp = Time.now.strftime("%Y%m%d_%H%M%S")

data/app/controllers/lyra/dashboard_controller.rb

@@ -36,6 +36,8 @@ module Lyra
     def discrepancies
       @model_class = params[:model_class].constantize
       @discrepancies = DualView.find_discrepancies(@model_class)
+      # find_discrepancies returns only the records that differ
+      @records_checked = @model_class.count
 
       respond_to do |format|
         format.html

data/app/views/lyra/dashboard/discrepancies.html.erb

@@ -22,10 +22,11 @@
   .lyra-discrepancies .comparison-col h4 { margin: 0 0 10px 0; font-size: 14px; color: #495057; }
   .lyra-discrepancies .comparison-col .source-label { font-size: 12px; color: #6c757d; margin-bottom: 5px; }
   .lyra-discrepancies .field-list { list-style: none; padding: 0; margin: 0; }
-  .lyra-discrepancies .field-list li { padding: 8px 0; border-bottom: 1px solid #eee; display: flex; justify-content: space-between; }
+  .lyra-discrepancies .field-list li { padding: 8px 0; border-bottom: 1px solid #eee; display: flex; justify-content: space-between; gap: 10px; }
   .lyra-discrepancies .field-list li:last-child { border-bottom: none; }
   .lyra-discrepancies .field-name { font-weight: 500; color: #1a1a2e; }
-  .lyra-discrepancies .field-value { color: #495057; font-family: monospace; font-size: 13px; max-width: 200px; overflow: hidden; text-overflow: ellipsis; }
+  .lyra-discrepancies .field-value { color: #495057; font-family: monospace; font-size: 13px; max-width: 70%; overflow-wrap: anywhere; text-align: right; }
+  .lyra-discrepancies .existence-note { margin: 0 0 15px 0; color: #495057; font-size: 14px; }
   .lyra-discrepancies .field-mismatch { background: #fff3cd; padding: 2px 6px; border-radius: 3px; }
   .lyra-discrepancies .empty-state { text-align: center; padding: 60px 40px; color: #6c757d; background: #d4edda; border: 1px solid #28a745; border-radius: 8px; }
   .lyra-discrepancies .empty-state h3 { color: #155724; margin-bottom: 10px; }
@@ -45,9 +46,14 @@
     Overview of data consistency between the database and event store.
   </p>
   <%
-    total_records = @discrepancies.count
-    mismatches = @discrepancies.count { |d| d[:status] == :mismatch || d[:differences]&.any? }
+    # DualView.find_discrepancies returns only the records that differ, each
+    # a DualView#compare result: differences maps a field to
+    # { crud:, event_sourced: }, or is { exists_mismatch: true } when the row
+    # and the log disagree on whether the record exists.
+    mismatches = @discrepancies.size
+    total_records = [@records_checked.to_i, mismatches].max
     matches = total_records - mismatches
+    show_value = ->(value) { value.nil? ? "(none)" : value.to_s }
   %>
   <div class="summary-cards">
     <div class="summary-card">
@@ -69,7 +75,7 @@
     Per-record comparison showing differences between database values and event-reconstructed state.
   </p>
 
-  <% if @discrepancies.empty? || mismatches == 0 %>
+  <% if mismatches == 0 %>
     <div class="empty-state">
       <h3>All Records Match</h3>
       <p>No discrepancies found between the database and event store.</p>
@@ -77,48 +83,71 @@
     </div>
   <% else %>
     <% @discrepancies.each do |discrepancy| %>
-      <% has_differences = discrepancy[:status] == :mismatch || discrepancy[:differences]&.any? %>
+      <%
+        differences = discrepancy[:differences] || {}
+        crud_view = discrepancy[:crud_view] || {}
+        es_view = discrepancy[:event_sourced_view] || {}
+        record_id = discrepancy.dig(:metadata, :model_id)
+      %>
       <div class="discrepancy-card">
-        <div class="discrepancy-header <%= has_differences ? '' : 'match' %>">
-          <h3><%= @model_class.name %> #<%= discrepancy[:id] || discrepancy[:record_id] %></h3>
-          <span class="status <%= has_differences ? 'mismatch' : 'match' %>">
-            <%= has_differences ? 'Mismatch' : 'Match' %>
-          </span>
+        <div class="discrepancy-header">
+          <h3><%= @model_class.name %> #<%= record_id %></h3>
+          <span class="status mismatch">Mismatch</span>
         </div>
-        <% if has_differences %>
-          <div class="discrepancy-body">
-            <div class="comparison-grid">
-              <div class="comparison-col">
-                <p class="source-label">Database State</p>
-                <h4>Current Values</h4>
-                <ul class="field-list">
-                  <% (discrepancy[:differences] || discrepancy[:db_state] || {}).each do |field, values| %>
+        <div class="discrepancy-body">
+          <% if differences[:exists_mismatch] %>
+            <p class="existence-note">
+              <% if crud_view[:exists] %>
+                The row exists, but the event log does not describe this record
+                (<%= es_view[:destroyed] ? "its events end in a destroy" : "no events" %>):
+                it was written before Lyra was enabled, or by a write that bypassed it.
+              <% else %>
+                The event log describes this record, but its row is gone: it was
+                deleted by a write that bypassed Lyra.
+              <% end %>
+            </p>
+          <% end %>
+          <div class="comparison-grid">
+            <div class="comparison-col">
+              <p class="source-label">Database State</p>
+              <h4>Current Values</h4>
+              <ul class="field-list">
+                <% if differences[:exists_mismatch] %>
+                  <li>
+                    <span class="field-name">row</span>
+                    <span class="field-value field-mismatch"><%= crud_view[:exists] ? "present" : "absent" %></span>
+                  </li>
+                <% else %>
+                  <% differences.each do |field, values| %>
                     <li>
                       <span class="field-name"><%= field %></span>
-                      <span class="field-value field-mismatch">
-                        <%= values.is_a?(Array) ? values[0] : values %>
-                      </span>
+                      <span class="field-value field-mismatch"><%= show_value.(values[:crud]) %></span>
                     </li>
                   <% end %>
-                </ul>
-              </div>
-              <div class="comparison-col">
-                <p class="source-label">Event-Sourced State</p>
-                <h4>Projected Values</h4>
-                <ul class="field-list">
-                  <% (discrepancy[:differences] || discrepancy[:event_state] || {}).each do |field, values| %>
+                <% end %>
+              </ul>
+            </div>
+            <div class="comparison-col">
+              <p class="source-label">Event-Sourced State</p>
+              <h4>Projected Values</h4>
+              <ul class="field-list">
+                <% if differences[:exists_mismatch] %>
+                  <li>
+                    <span class="field-name">events</span>
+                    <span class="field-value field-mismatch"><%= es_view[:events_count].to_i %></span>
+                  </li>
+                <% else %>
+                  <% differences.each do |field, values| %>
                     <li>
                       <span class="field-name"><%= field %></span>
-                      <span class="field-value field-mismatch">
-                        <%= values.is_a?(Array) ? values[1] : values %>
-                      </span>
+                      <span class="field-value field-mismatch"><%= show_value.(values[:event_sourced]) %></span>
                     </li>
                   <% end %>
-                </ul>
-              </div>
+                <% end %>
+              </ul>
             </div>
           </div>
-        <% end %>
+        </div>
       </div>
     <% end %>
   <% end %>

data/lib/lyra/associations/habtm_links.rb

@@ -2,7 +2,7 @@
 
 module Lyra
   module Associations
-    # Links and unlinks of has_and_belongs_to_many associations (#21).
+    # Links and unlinks of has_and_belongs_to_many associations.
     #
     # A HABTM join table has no model of its own: Active Record writes it
     # through an anonymous class (Item::HABTM_Tags), which no interceptor saw,
@@ -173,7 +173,7 @@ module Lyra
             rows: rows,
             timestamp: Time.current
           }
-          metadata = { correlation_id: Lyra::Correlation.current_id, causation_id: Lyra::Causation.current_id }.compact
+          metadata = Lyra::BypassEvents.current_metadata
           metadata[:genesis] = true if genesis
           event_class = operation == LINKED ? Lyra::Events::AssociationLinked : Lyra::Events::AssociationUnlinked
           event_class.new(data: data, metadata: Lyra::Privacy.stamp(owner, data, metadata))

data/lib/lyra/bypass_events.rb

@@ -55,11 +55,7 @@ module Lyra
 
       def publish(model_class, id, operation, attributes:, changes: {}, source:)
         event_class = event_class_for(model_class, operation)
-        metadata = {
-          correlation_id: Lyra::Correlation.current_id,
-          causation_id: Lyra::Causation.current_id,
-          source_key(source) => source
-        }.compact
+        metadata = current_metadata.merge(source_key(source) => source)
 
         data = {
           model_class: model_class.name,
@@ -79,6 +75,19 @@ module Lyra
         end
       end
 
+      # Who made the change and what it belongs with, for a write no record's
+      # callbacks attribute. Read from the application's Current, as for a write
+      # through callbacks; a bulk write has no record to ask, so a model's
+      # lyra_current_user_id hook does not apply here.
+      def current_metadata
+        {
+          user_id: current(:user)&.id,
+          request_id: current(:request_id),
+          correlation_id: Lyra::Correlation.current_id,
+          causation_id: Lyra::Causation.current_id
+        }.compact
+      end
+
       # Attributes of every row the relation matches, keyed by primary key.
       # Taken before a bulk write so the events can carry old values.
       def snapshot(model_class, relation)
@@ -162,6 +171,10 @@ module Lyra
 
       private
 
+      def current(attribute)
+        defined?(::Current) && ::Current.respond_to?(attribute) ? ::Current.public_send(attribute) : nil
+      end
+
       # Column => [old, new] for every non-timestamp column that changed.
       def diff(old_attrs, new_attrs)
         new_attrs.each_with_object({}) do |(column, new_value), acc|

data/lib/lyra/command_handler.rb

@@ -139,9 +139,13 @@ module Lyra
     # domain events (Lyra::DomainEvents).
     def create_events(operation, id, data)
       own = create_event(operation, id, data)
+      record = command.record
       Lyra::DomainEvents.build(
         command.model_class, operation, data: own.data, metadata: own.metadata.to_h,
-        default_class: own.class, record: command.record, changes: changes_for(operation, data)
+        default_class: own.class, record: record, changes: changes_for(operation, data),
+        # The id the record's write scope fixed, so the writes its callbacks
+        # made can name this event as their cause.
+        event_id: record.respond_to?(:lyra_take_own_event_id) ? record.lyra_take_own_event_id : nil
       )
     end
 

data/lib/lyra/configuration.rb

@@ -33,6 +33,12 @@ module Lyra
     # (default) follows mode_transition_gate; true; false. Checked at most
     # every mode_sync_interval seconds.
     attr_accessor :mode_sync, :mode_sync_interval
+    # Group writes into causal chains (Lyra::Correlation): the writes of one
+    # HTTP request share a correlation (the request id); a write made inside
+    # another write's callbacks records that write's event as its cause; the
+    # outermost write opens the correlation when none is in scope. On by
+    # default; false leaves both to the application's own with_id calls.
+    attr_accessor :correlate_writes
     # Stamp each event's metadata with the privacy annotation of the personal
     # data it carries (Lyra::Privacy.stamp). Off by default: it adds work to
     # every write, measured separately.
@@ -110,6 +116,7 @@ module Lyra
       @dual_view_discrepancy_handler = nil
       @mode_sync = nil
       @mode_sync_interval = 5
+      @correlate_writes = true
       @annotate_privacy = false
       @record_access_events = false
       @access_metadata_proc = nil

data/lib/lyra/consistency/read_your_writes.rb

@@ -89,12 +89,34 @@ module Lyra
           (Thread.current[:lyra_pending_writes] || []).each { |write| project(write) }
         end
 
+        # Projects under the lock AsyncProjectionJob holds per stream, taken
+        # in the writer's transaction (or one of its own) and held to its end.
+        # A job that had read the stream before this write committed would
+        # otherwise write the row after this projection, putting it back to
+        # the older value until the newer event's job ran. With the lock, this
+        # projection waits for a job in flight; a job that starts after it
+        # reads a stream that already holds this write's event.
         def project(write)
-          Lyra::Projections::ModelProjection.project(write[:model_class], write[:operation], write[:result])
+          model_class = write[:model_class]
+          model_class.transaction do
+            lock_stream(model_class, write[:result])
+            Lyra::Projections::ModelProjection.project(model_class, write[:operation], write[:result])
+          end
         rescue => e
           Rails.logger.error("Lyra: Failed to ensure projection - #{e.message}")
           raise if Lyra.config.strict_projections
         end
+
+        # The same key AsyncProjectionJob#lock_stream! locks: the record's
+        # stream name, from the model id its event carries.
+        def lock_stream(model_class, result)
+          data = result.events&.first&.data
+          model_id = data && (data[:model_id] || data["model_id"])
+          return unless model_id
+
+          stream = Lyra.stream_name(model_class, model_id)
+          model_class.with_connection { |c| Lyra::AdvisoryLock.xact_lock(c, stream, purpose: "read-your-writes projection") }
+        end
       end
     end
 

data/lib/lyra/correlation.rb

@@ -24,6 +24,95 @@ module Lyra
         Thread.current[:lyra_correlation_id] = previous_id
       end
     end
+
+    # The writes of one HTTP request share a correlation: the request id
+    # (ActionDispatch::RequestId), or a new id without one. A correlation
+    # already in scope is kept. Installed by the engine; off with
+    # config.correlate_writes = false.
+    class Middleware
+      def initialize(app)
+        @app = app
+      end
+
+      def call(env)
+        return @app.call(env) unless Lyra.config.correlate_writes && Correlation.current_id.nil?
+
+        Correlation.with_id(env["action_dispatch.request_id"].presence) { @app.call(env) }
+      end
+    end
+
+    # The writes of one background job share a correlation: the one in scope
+    # where the job was created (perform_later in a request, or in another
+    # write's callbacks), or the job id without one. A write in scope there is
+    # the cause of the job's outermost writes. Both are taken when the job is
+    # created, not when it is serialized: a job deferred to after the commit
+    # (enqueue_after_transaction_commit) is serialized outside the write that
+    # enqueued it. They travel in the job's payload, so a retry keeps them.
+    # Included into ActiveJob::Base by the engine; off with
+    # config.correlate_writes = false.
+    #
+    # A job created inside a transaction and sent at once (not deferred: the
+    # Rails default) can outlive a rollback of the write that caused it. Its
+    # cause is unconfirmed, and kept when the job runs only if that event is
+    # in the log, so no job names a cause the log lacks. A job that runs
+    # before the enqueuing transaction commits loses the link, not its
+    # correlation.
+    module Job
+      extend ActiveSupport::Concern
+
+      included do
+        around_perform :lyra_with_enqueuing_scope
+      end
+
+      attr_reader :lyra_correlation_id, :lyra_causation_id
+
+      def initialize(...)
+        super
+        return unless Lyra.config.correlate_writes
+
+        @lyra_correlation_id = Correlation.current_id
+        @lyra_causation_id = Causation.current_id
+        @lyra_cause_unconfirmed = !!@lyra_causation_id && !self.class.enqueue_after_transaction_commit &&
+                                  ActiveRecord::Base.with_connection(&:transaction_open?)
+      end
+
+      def serialize
+        data = super
+        data["lyra_correlation_id"] = lyra_correlation_id if lyra_correlation_id
+        data["lyra_causation_id"] = lyra_causation_id if lyra_causation_id
+        data["lyra_cause_unconfirmed"] = true if @lyra_cause_unconfirmed
+        data
+      end
+
+      def deserialize(job_data)
+        super
+        @lyra_correlation_id = job_data["lyra_correlation_id"]
+        @lyra_causation_id = job_data["lyra_causation_id"]
+        @lyra_cause_unconfirmed = job_data["lyra_cause_unconfirmed"] == true
+      end
+
+      private
+
+      # A correlation already in scope (a job performed inline, inside the
+      # request that created it) is kept, as by the Middleware.
+      def lyra_with_enqueuing_scope
+        return yield unless Lyra.config.correlate_writes && Correlation.current_id.nil?
+
+        Correlation.with_id(lyra_correlation_id || job_id) do
+          Causation.with_id(lyra_confirmed_cause) { yield }
+        end
+      end
+
+      # The cause, once it is in the log; nil if it is not. Decided on the
+      # first attempt: a retry carries the outcome in its payload.
+      def lyra_confirmed_cause
+        return lyra_causation_id unless @lyra_cause_unconfirmed
+
+        @lyra_cause_unconfirmed = false
+        @lyra_causation_id = nil unless Lyra.config.event_store.read.event(lyra_causation_id)
+        lyra_causation_id
+      end
+    end
   end
 
   # Causation system for tracking event chains

data/lib/lyra/domain_events.rb

@@ -89,16 +89,20 @@ module Lyra
       #
       # @param default_class [Class] the CRUD event class for the operation
       # @param data [Hash] the event envelope; metadata [Hash] its metadata
-      def build(model_class, operation, data:, metadata:, default_class:, record:, changes:)
+      # @param event_id [String, nil] the write's own event id when its write
+      #   scope fixed one (CrudInterceptor#lyra_write_scope); nil lets the
+      #   store assign one
+      def build(model_class, operation, data:, metadata:, default_class:, record:, changes:, event_id: nil)
+        id = event_id ? { event_id: event_id } : {}
         rules = rules_for(model_class)
-        return [default_class.new(data: data, metadata: metadata)] if rules.empty? || record.nil?
+        return [default_class.new(**id, data: data, metadata: metadata)] if rules.empty? || record.nil?
 
         changes = normalize_changes(changes)
         matching = rules.select { |rule| rule.applies?(operation, record, changes) }
         own_rule = matching.find { |rule| !rule.also }
 
         own = event_class_for(own_rule, default_class).new(
-          data: with_payload(data, own_rule, record, changes), metadata: metadata
+          **id, data: with_payload(data, own_rule, record, changes), metadata: metadata
         )
         extras = matching.select(&:also).map do |rule|
           event_class_for(rule, nil).new(

data/lib/lyra/encrypted_attributes.rb

@@ -5,7 +5,7 @@ module Lyra
   #
   # Events are built from a model's attributes and changes, which Active
   # Record has already decrypted, so an encrypted column reached the log in
-  # clear while its row held ciphertext (issue #19). Every event Lyra writes
+  # clear while its row held ciphertext. Every event Lyra writes
   # passes through Lyra.append_events, which seals it here: the values of the
   # model's encrypted attributes, in attributes and in changes ([old, new]),
   # are replaced by their ciphertext, as Active Record writes it to the row.

data/lib/lyra/engine.rb

@@ -84,6 +84,14 @@ module Lyra
         end
       end
 
+      # Lyra::Correlation::Middleware and Lyra::Correlation::Job: the writes of
+      # one request share a correlation, and a job's writes the one in scope
+      # where it was created (config.correlate_writes).
+      initializer "lyra.request_correlation" do |app|
+        app.config.middleware.use Lyra::Correlation::Middleware
+        ActiveSupport.on_load(:active_job) { include Lyra::Correlation::Job }
+      end
+
       # config.verify_mapping!: verify at boot, once the models are
       # instrumented; a failure stops the boot.
       initializer "lyra.verify_mapping", after: "lyra.apply_declared_models" do
@@ -119,7 +127,7 @@ module Lyra
       end
 
       # Rails loads every lib/tasks/*.rake under an engine's root into the
-      # host application. In the monorepo that root also holds its own
+      # host application. In a development checkout that root also holds its own
       # maintenance tasks (gems:, public:, stats:, ...), and the Lyra task
       # files loaded here explicitly as well, so each Lyra task ran twice
       # (lyra:mode:check checked everything twice). Host applications get

data/lib/lyra/interceptors/crud_interceptor.rb

@@ -84,6 +84,15 @@ module Lyra
         after_update :lyra_intercept_update, if: :lyra_events_enabled?
         after_destroy :lyra_intercept_destroy, if: :lyra_events_enabled?
 
+        # Every event-producing write runs in a write scope
+        # (config.correlate_writes): its event id is fixed before any of its
+        # callbacks, so the writes they make record it as their cause, and the
+        # outermost write opens the correlation when none is in scope. First
+        # in each chain, so it wraps the model's own callbacks, whatever order
+        # the model declares them in.
+        around_save :lyra_write_scope, if: :lyra_events_enabled?, prepend: true
+        around_destroy :lyra_write_scope, if: :lyra_events_enabled?, prepend: true
+
         # Hijack and event-sourcing modes build their event in WriteHooks, after
         # every before_* callback and just before the row write (see below).
         # Only PaperTrail is switched off here, first, so that its own
@@ -147,7 +156,9 @@ module Lyra
           user_id: lyra_current_user_id,
           request_id: lyra_current_request_id,
           correlation_id: Lyra::Correlation.current_id,
-          causation_id: Lyra::Causation.current_id,
+          # Inside its own write scope the current cause is this write itself;
+          # its own cause is the one in scope when the write began.
+          causation_id: @lyra_in_write_scope ? @lyra_write_cause_id : Lyra::Causation.current_id,
           action_id: lyra_current_action_id,
           user_action: lyra_current_user_action
         }
@@ -155,6 +166,34 @@ module Lyra
         base_metadata.merge(lyra_custom_metadata(operation))
       end
 
+      # The event id fixed for this write by its write scope, for the write's
+      # own event: given once, so no two events can share it; nil outside a
+      # scope or once taken (the store then assigns one).
+      def lyra_take_own_event_id
+        return nil unless @lyra_in_write_scope && !@lyra_write_event_id_taken
+
+        @lyra_write_event_id_taken = true
+        @lyra_write_event_id
+      end
+
+      # around_save/around_destroy (see the callbacks above). Nested scopes of
+      # the same record (a save inside its own callbacks) restore the outer
+      # one on the way out.
+      def lyra_write_scope
+        return yield unless Lyra.config.correlate_writes
+
+        outer = [@lyra_in_write_scope, @lyra_write_event_id, @lyra_write_cause_id, @lyra_write_event_id_taken]
+        @lyra_write_cause_id = Lyra::Causation.current_id
+        @lyra_write_event_id = SecureRandom.uuid
+        @lyra_write_event_id_taken = false
+        @lyra_in_write_scope = true
+        Lyra::Correlation.with_id(Lyra::Correlation.current_id || @lyra_write_event_id) do
+          Lyra::Causation.with_id(@lyra_write_event_id) { yield }
+        end
+      ensure
+        @lyra_in_write_scope, @lyra_write_event_id, @lyra_write_cause_id, @lyra_write_event_id_taken = outer if outer
+      end
+
       class_methods do
         # Enable Lyra monitoring for this model
         # The model as it was at +time+, rebuilt from its event streams
@@ -808,7 +847,8 @@ module Lyra
         events = Lyra::DomainEvents.build(
           self.class, operation, data: data, metadata: metadata,
           default_class: lyra_event_class_for(operation),
-          record: self, changes: operation == :destroyed ? {} : previous_changes
+          record: self, changes: operation == :destroyed ? {} : previous_changes,
+          event_id: lyra_take_own_event_id
         )
 
         Lyra.append_events(events, stream_name: lyra_stream_name)

data/lib/lyra/projections/async_projection_job.rb

@@ -105,11 +105,12 @@ module Lyra
         model_class.with_connection { |c| Lyra::AdvisoryLock.xact_lock(c, stream, purpose: "ES-Async projection") }
       end
 
+      # read.event returns nil for a missing event (read.event! raises): the
+      # warning sat in a rescue of EventNotFound and was never logged.
       def load_event(event_id)
-        Lyra.config.event_store.read.event(event_id)
-      rescue RubyEventStore::EventNotFound
-        Rails.logger.warn("Lyra::AsyncProjectionJob: Event #{event_id} not found")
-        nil
+        event = Lyra.config.event_store.read.event(event_id)
+        Rails.logger.warn("Lyra::AsyncProjectionJob: Event #{event_id} not found") unless event
+        event
       end
     end
   end

data/lib/lyra/projections/reference_links.rb

@@ -117,7 +117,7 @@ module Lyra
         #
         # One thread per model and process catches up; the others wait for it
         # without taking a connection, so a burst of first lookups needs one
-        # connection beyond the callers' own, not one each (issue #37).
+        # connection beyond the callers' own, not one each.
         def catch_up(model_class, force: false)
           model_class = owner(model_class)
           return 0 if !force && ready?(model_class)

data/lib/lyra/version.rb

@@ -1,3 +1,3 @@
 module Lyra
-  VERSION = "0.9.0"
+  VERSION = "0.9.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: orfeas_lyra
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.9.2
 platform: ruby
 authors:
 - Michail Pantelelis