orfeas_lyra 0.6.0 → 0.9.0

data/README.md

@@ -4,984 +4,104 @@
 
 *Part of the ORFEAS (Object-Relational to Event-Sourcing Architecture) Framework*
 
-## Author
-
-**Michail Pantelelis** (mpantel@aegean.gr)
-PhD Candidate, University of the Aegean
-Department of Information and Communication Systems Engineering
-
----
-
-## Table of Contents
-
-1. [Overview](#overview)
-2. [The Problem](#the-problem)
-3. [The Solution: ORFEAS Framework](#the-solution-orfeas-framework)
-4. [Key Features](#key-features)
-5. [Architecture](#architecture)
-6. [Components](#components)
-7. [Installation](#installation)
-8. [Quick Start](#quick-start)
-9. [Usage Guide](#usage-guide)
-10. [Example Application](#example-application)
-11. [Benefits](#benefits)
-12. [Documentation](#documentation)
-13. [Research Foundation](#research-foundation)
-14. [Development Methodology](#development-methodology)
-15. [Contributing](#contributing)
-16. [Citation](#citation)
-17. [Support](#support)
-
----
-
-## Overview
-
-Lyra is a Rails engine that enables **gradual, non-intrusive transformation** of traditional CRUD (Create, Read, Update, Delete) operations into Event Sourcing architectures, with built-in privacy compliance capabilities. It is the practical implementation of the **ORFEAS (Object-Relational to Event-Sourcing Architecture)** framework, developed as part of a PhD thesis addressing the fundamental challenge of bridging 40+ years of ORM dominance with modern event-driven, GDPR-compliant application architectures.
-
-### What Makes Lyra Unique?
-
-- **Dual-Mode Operation**: Monitor existing applications non-intrusively, then gradually transform to full event sourcing
-- **Formal Mathematical Foundation**: Built on Petri nets (P/T nets for verification, CPNs for advanced modeling) and matrix analysis for rigorous CRUD-to-event mapping
-- **Privacy-First Design**: Integrated Privacy Attribute Matrix (PAM) for GDPR compliance
-- **Zero-Downtime Migration**: Maintain CRUD as a safety net while transitioning to event sourcing
-- **Research-Backed**: Grounded in peer-reviewed research on event sourcing patterns and privacy preservation
-
----
-
-## The Problem
-
-Modern enterprise systems face a critical architectural challenge:
-
-### The Legacy Dilemma
-- **40+ Years of ORM Investment**: Decades of business logic encoded in Object-Relational Mapping systems
-- **CRUD Limitations**: State-based systems lose behavioral history and temporal context
-- **Privacy Requirements**: GDPR mandates comprehensive data lineage and processing transparency
-- **Migration Risk**: Complete rewrites from CRUD to Event Sourcing are expensive and risky
-
-### The Event Sourcing Promise
-- **Complete Audit Trails**: Every state change recorded as an immutable event
-- **Temporal Queries**: Query system state at any point in time
-- **Behavioral Analysis**: Understand *how* and *why* state changes occurred
-- **Microservices Alignment**: Natural fit for distributed, event-driven architectures
-
-### The Gap
-**There is no formal, proven method for gradually migrating CRUD systems to event sourcing while maintaining operational safety and privacy compliance.**
-
----
-
-## The Solution: ORFEAS Framework
-
-The **ORFEAS (Object-Relational to Event-Sourcing Architecture)** framework provides:
-
-1. **Formal Mathematical Models**: Petri nets map CRUD operations to event sequences with formal verification (P/T nets for structural proofs, CPNs optional for data modeling)
-2. **Privacy Compliance**: Privacy Attribute Matrix (PAM) ensures GDPR compliance throughout the transformation
-3. **Gradual Migration Path**: Two operational modes support safe, incremental transition
-4. **Dual-View Analysis**: Compare CRUD and event-sourced views to verify correctness
-5. **Practical Tooling**: Lyra implements ORFEAS as a production-ready Rails engine
-
-### Theoretical Foundation
-
-ORFEAS is built on three pillars:
-
-1. **Petri Nets**: Model CRUD-to-event transformations (P/T nets for verification proofs; CPNs with tokens, guards, and arc expressions for advanced data modeling)
-2. **Matrix Analysis**: Complementary linear algebra approach for causation and lineage tracking
-3. **Privacy Attribute Matrix (PAM)**: DSL for declaring field-level privacy policies and transformations
-
-[Read the complete theoretical foundation →](docs/ORFEAS_FRAMEWORK_OVERVIEW.md)
-
----
-
-## Key Features
-
-### Operational Features
-- ✅ **Non-intrusive Monitoring** - Works with existing Rails applications without code changes
-- ✅ **Dual Operational Modes** - Monitor or hijack CRUD operations
-- ✅ **Automatic Event Mapping** - CRUD operations automatically mapped to domain events
-- ✅ **State Reconstruction** - Rebuild state from event streams
-- ✅ **Dual-View Dashboard** - Compare CRUD state vs event-sourced state
-
-### Technical Features
-- ✅ **Rails Event Store Integration** - Built on proven event sourcing infrastructure
-- ✅ **Pluggable Event Backends** - Support for custom event storage (Kafka, EventStoreDB)
-- ✅ **Aggregate Support** - Custom domain aggregates for complex business logic
-- ✅ **Command/Query Separation** - CQRS-ready architecture
-- ✅ **Event Versioning** - Support for event schema evolution
-
-### Privacy & Compliance
-- ✅ **Privacy Attribute Matrix (PAM)** - Built-in PAM DSL for defining privacy policies
-- ✅ **GDPR Compliance** - Purpose-based access control, consent management, retention policies
-- ✅ **PII Detection & Transformation** - Automatic PII field identification and transformation
-- ✅ **Audit Trails** - Complete lineage tracking for regulatory compliance
-
----
-
-## Architecture
-
-### High-Level Architecture
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    Rails Application                         │
-├─────────────────────────────────────────────────────────────┤
-│                                                               │
-│  ActiveRecord Models (with monitor_with_lyra)                │
-│           │                                                   │
-│           ▼                                                   │
-│  ┌──────────────────┐                                        │
-│  │ CRUD Interceptor │                                        │
-│  └────────┬─────────┘                                        │
-│           │                                                   │
-│      ┌────┴────┐                                             │
-│      │         │                                             │
-│   MONITOR    HIJACK                                          │
-│    MODE       MODE                                           │
-│      │         │                                             │
-│      ▼         ▼                                             │
-│  ┌─────┐   ┌─────────┐                                      │
-│  │ Log │   │ Command │                                      │
-│  │Event│   │ Handler │                                      │
-│  └──┬──┘   └────┬────┘                                      │
-│     │           │                                            │
-│     └─────┬─────┘                                            │
-│           ▼                                                   │
-│  ┌─────────────────┐                                        │
-│  │  Event Store    │                                        │
-│  │ (Rails Event    │                                        │
-│  │   Store)        │                                        │
-│  └─────────────────┘                                        │
-│           │                                                   │
-│           ▼                                                   │
-│  ┌─────────────────┐                                        │
-│  │  Aggregates &   │                                        │
-│  │  Projections    │                                        │
-│  └─────────────────┘                                        │
-│                                                               │
-└─────────────────────────────────────────────────────────────┘
-```
-
-### Operational Modes
-
-#### 1. Monitor Mode (Non-intrusive)
-- Observes CRUD operations and logs them as domain events
-- **No changes** to application behavior
-- Perfect for **analysis and planning** migration
-- Zero risk to production systems
-
-#### 2. Hijack Mode (Transformative)
-- Intercepts CRUD operations and routes through event sourcing
-- Replaces traditional relational backend
-- Maintains CRUD interface for **backward compatibility**
-- Full event sourcing benefits
-
-[See detailed architecture documentation →](docs/ARCHITECTURE.md)
-
----
-
-## Components
-
-### Lyra Engine (Core)
-The main Rails engine providing:
-- CRUD interception and monitoring
-- Event mapping and publishing
-- Dual-view analysis
-- Command/Query handlers
-- State reconstruction
-
-### PetriFlow Gem
-Complete Petri net and Colored Petri Net library:
-- Core Petri net components (places, transitions, arcs)
-- Colored extensions (guards, arc expressions, token types)
-- Matrix analysis (CRUD-event mapping, causation, lineage)
-- Visualization (GraphViz, Mermaid, ASCII)
-- Formal verification (reachability, boundedness, liveness)
-- Export functionality (PNML, CPN Tools, JSON, YAML)
-
-[Read the PetriFlow documentation →](gems/petri_flow/README.md)
-
-### PAM DSL Gem
-Privacy Attribute Matrix Domain-Specific Language:
-- Field-level PII classification with sensitivity levels
-- Purpose-based access control aligned with GDPR
-- Retention policies with field-level granularity
-- Consent management with expiration tracking
-- Data transformation for different contexts (display, logging, API)
-
-[Read the PAM DSL documentation →](gems/pam_dsl/README.md)
-
-### Monorepo Structure
-
-This repository is organized as a monorepo:
-- **`/` (root)** - Lyra Rails engine
-- **`gems/petri_flow/`** - PetriFlow Petri net library
-- **`gems/pam_dsl/`** - PAM DSL privacy policy language
-- **`docs/`** - Theoretical documentation
-- **`docs-site/`** - Jekyll documentation site
-- **`examples/`** - Example applications
-
-[See complete monorepo structure →](docs/MONOREPO.md)
-
----
-
-## Installation
-
-### Prerequisites
-
-- Ruby 3.4.5+ (tested up to Ruby 4.0) and Rails 8.0+
-- PostgreSQL 14+ (recommended for Rails Event Store)
-- Bundler
-
-### Add to Gemfile
-
-```ruby
-gem 'lyra', path: 'path/to/lyra'  # or from git/rubygems when published
-gem 'pam_dsl', '~> 0.1.0'         # Privacy Attribute Matrix DSL
-gem 'petri_flow', '~> 0.1.0'      # Petri net library
-```
-
-### Install Dependencies
-
-```bash
-bundle install
-rails generate rails_event_store_active_record:migration
-rails db:migrate
-```
-
----
-
-## Quick Start
-
-### 1. Configure Lyra
-
-Create `config/initializers/lyra.rb`:
-
-```ruby
-Lyra.configure do |config|
-  # Start in monitor mode (non-intrusive)
-  config.mode = :monitor
-
-  # Configure event store
-  config.event_backend = :rails_event_store
-  config.event_store = RailsEventStore::Client.new
-
-  # Optional: User tracking for audit trails (see User Tracking section below)
-  # config.metadata_proc = ->(record, operation) { { user_id: Current.user&.id } }
-end
-```
-
-### 2. Monitor a Model
-
-Add monitoring to your ActiveRecord models:
-
-```ruby
-class Order < ApplicationRecord
-  # Enable Lyra monitoring
-  monitor_with_lyra
-
-  # Your existing code continues to work normally
-  validates :total, presence: true
-  belongs_to :customer
-end
-```
-
-### 3. Observe Events
-
-All CRUD operations are now logged as events:
-
-```ruby
-order = Order.create!(total: 100, customer: customer)
-# => Publishes OrderCreated event
-
-order.update!(total: 150)
-# => Publishes OrderUpdated event
-
-order.destroy!
-# => Publishes OrderDestroyed event
-```
-
-### 4. Analyze State
-
-Compare CRUD vs event-sourced views:
-
-```ruby
-comparison = Lyra::DualView.new(Order, order.id).compare
-
-puts comparison[:differences]
-# => { no_differences: true }
-
-# View complete audit trail
-audit = Lyra::DualView.new(Order, order.id).audit_trail
-# => [{ timestamp: ..., operation: :created, changes: {...} }, ...]
-```
-
-[Continue to full usage guide →](#usage-guide)
+Lyra is a Rails engine that records the writes an ActiveRecord application
+makes as events (RailsEventStore), and can then move the application, one
+checked step at a time, to treating those events as the source of truth.
 
 ---
 
-## Usage Guide
-
-### Basic Model Monitoring
-
-Add monitoring to any ActiveRecord model:
-
-```ruby
-class Order < ApplicationRecord
-  monitor_with_lyra
-
-  validates :total, presence: true
-  belongs_to :customer
-end
-```
-
-### Privacy Policies with PAM DSL
-
-Define privacy policies for your models:
-
-```ruby
-# config/privacy_policies.rb
-PamDsl.define_policy :order_system do
-  # Define PII fields
-  field :email, type: :email, sensitivity: :internal do
-    allow_for :order_processing, :communication
-    transform :display { |v| "#{v[0]}***@#{v.split('@').last}" }
-  end
-
-  field :credit_card, type: :credit_card, sensitivity: :restricted do
-    allow_for :payment_processing
-    transform :display { |v| "****-****-****-#{v[-4..]}" }
-  end
-
-  # Define processing purposes
-  purpose :order_processing do
-    basis :contract
-    requires :email
-  end
-
-  purpose :marketing do
-    basis :consent
-    requires :email
-  end
-
-  # Configure retention
-  retention do
-    for_model 'Order' do
-      keep_for 7.years
-      on_expiry :anonymize
-    end
-  end
-end
-
-# Apply to model
-class Order < ApplicationRecord
-  monitor_with_lyra privacy_policy: :order_system
-end
-```
-
-[See complete PAM DSL guide →](gems/pam_dsl/docs/PAM_DSL_INTEGRATION.md)
-
-### Schema Validation (Strict Mode)
-
-Lyra can enforce schema consistency to prevent silent breaking changes in production:
-
-```ruby
-# config/initializers/lyra.rb
-Lyra.configure do |config|
-  config.mode = :monitor
-
-  # Enable strict schema validation (recommended for production)
-  config.strict_schema = Rails.env.production?
-
-  # Custom schema storage path (optional, defaults to db/lyra_schemas/)
-  config.schema_path = Rails.root.join('db/lyra_schemas')
-
-  config.monitor_model User
-  config.monitor_model Order
-end
-```
-
-#### Schema Management Rake Tasks
-
-```bash
-# Generate initial schema from monitored models
-rake lyra:schema:create
-
-# Check for schema changes without updating
-rake lyra:schema:verify
-
-# Create new schema version (after migrations)
-rake lyra:schema:update
-
-# Display model → event mappings
-rake lyra:schema:report
-
-# Show schema version history
-rake lyra:schema:history
-
-# Compare two schema versions
-rake lyra:schema:diff[1,2]
-```
-
-#### Schema Validation Workflow
-
-1. **Development**: Run `rake lyra:schema:create` to generate initial schema
-2. **After migrations**: Run `rake lyra:schema:verify` to detect changes
-3. **If changes detected**: Run `rake lyra:schema:update` to create new version
-4. **Production**: With `strict_schema = true`, app fails to start if schema changes
-
-#### Schema Change Severity Levels
-
-| Severity | Changes | Impact |
-|----------|---------|--------|
-| **BREAKING** | model_removed, column_removed, column_type_changed | Requires new schema version |
-| **WARNING** | column_nullable_changed, pii_field_added | Review recommended |
-| **INFO** | model_added, column_added | Documentation update |
-
-### User Tracking (metadata_proc)
-
-Lyra can capture custom metadata with each event, such as the current user, for audit trails and GDPR compliance.
-
-#### Configuration
-
-```ruby
-# config/initializers/lyra.rb
-Lyra.configure do |config|
-  config.mode = :monitor
-  config.event_backend = :rails_event_store
-
-  # Custom metadata proc - called for every event
-  # Signature: ->(record, operation) { Hash }
-  config.metadata_proc = lambda do |record, operation|
-    {
-      user_id: Current.user&.id,
-      user_email: Current.user&.email,
-      ip_address: Current.request&.remote_ip,
-      source: 'my_app'
-    }
-  end
-end
-```
-
-#### Using with Rails CurrentAttributes
-
-Rails 5.2+ provides `CurrentAttributes` for request-scoped state:
-
-```ruby
-# app/models/current.rb
-class Current < ActiveSupport::CurrentAttributes
-  attribute :user, :request_id, :request
-end
-
-# app/controllers/application_controller.rb
-class ApplicationController < ActionController::Base
-  before_action :set_current_attributes
-
-  private
-
-  def set_current_attributes
-    Current.user = current_user
-    Current.request_id = request.request_id
-    Current.request = request
-  end
-end
-```
-
-#### Using with Devise/Warden
-
-For Devise-based authentication:
-
-```ruby
-config.metadata_proc = lambda do |record, operation|
-  user = if defined?(Warden) && Thread.current[:request_env]
-    Warden::Proxy.new(Thread.current[:request_env], Warden::Manager.new(nil)).user
-  end
-
-  { user_id: user&.id, user_email: user&.email }
-end
-```
-
-#### Using with Rails Engines (Solidus, Spree, etc.)
-
-Many Rails Engines provide their own `Current` class:
-
-```ruby
-config.metadata_proc = lambda do |record, operation|
-  user_id = nil
-
-  # Try standard Current
-  user_id ||= Current.user&.id if defined?(Current)
-
-  # Try Spree/Solidus Current
-  user_id ||= Spree::Current.user&.id if defined?(Spree::Current)
-
-  { user_id: user_id }
-end
-```
-
-#### Metadata in Events
-
-The custom metadata is merged with Lyra's built-in metadata:
-
-```ruby
-event = Lyra.event_store.read.last
-event.metadata
-# => {
-#   user_id: 123,
-#   user_email: "admin@example.com",
-#   ip_address: "127.0.0.1",
-#   source: "my_app",
-#   request_id: "abc-123",        # Built-in from Current
-#   correlation_id: "corr-456",   # Built-in from Correlation context
-#   causation_id: "cause-789"     # Built-in from Causation context
-# }
-```
-
-### Advanced Configuration
-
-Use custom event names and aggregates:
-
-```ruby
-class Order < ApplicationRecord
-  monitor_with_lyra(
-    event_prefix: 'Order',
-    aggregate_class: OrderAggregate
-  )
-end
-
-class OrderAggregate < Lyra::Aggregate
-  def total
-    get_state(:total)
-  end
-
-  private
-
-  def apply_order_created(event)
-    @id = event.model_id
-    set_state(:total, event.attributes['total'])
-    set_state(:status, 'pending')
-  end
-
-  def apply_order_updated(event)
-    event.changes.each { |k, (old, new)| set_state(k.to_sym, new) }
-  end
-end
-```
-
-### Event Capture Behavior
-
-Lyra uses ActiveRecord `after_commit` callbacks to capture events. This means **all** operations that go through ActiveRecord are captured, regardless of source:
-
-| Source | Captured |
-|--------|----------|
-| Web requests | ✓ |
-| Rails console (`rails c`) | ✓ |
-| Background jobs (Sidekiq, etc.) | ✓ |
-| Rake tasks | ✓ |
-| Rails runner scripts | ✓ |
-| Seeds (`db:seed`) | ✓ |
-
-**Operations NOT captured** (bypass ActiveRecord callbacks):
-
-| Operation | Captured | Alternative |
-|-----------|----------|-------------|
-| `update_columns` / `update_column` | ✗ | Use `update!` |
-| `update_all` / `delete_all` | ✗ | Iterate with `find_each` |
-| `insert_all` / `upsert_all` | ✗ | Use `create!` |
-| Raw SQL (`connection.execute`) | ✗ | Use ActiveRecord methods |
-| `touch` with no callbacks | ✗ | Use `update!` |
-
-### Strict Data Access Mode
-
-To prevent inconsistencies between CRUD state and event store, enable strict mode to raise errors on callback-bypassing operations:
-
-```ruby
-# config/initializers/lyra.rb
-Lyra.configure do |config|
-  config.strict_data_access = true  # Raises on update_columns, etc.
-  # Or only in specific environments:
-  config.strict_data_access = Rails.env.development? || Rails.env.test?
-end
-```
-
-When enabled, monitored models will raise `Lyra::StrictDataAccessViolation` if you attempt:
-
-| Operation | Alternative | Scope |
-|-----------|-------------|-------|
-| `update_columns` | `update!` | Instance |
-| `update_column` | `update!` | Instance |
-| `delete` | `destroy` | Instance |
-| `update_all` | `find_each { \|r\| r.update!(...) }` | Relation |
-| `delete_all` | `find_each(&:destroy)` | Relation |
-| `insert_all` | `records.each { \|attrs\| create!(attrs) }` | Class |
-| `upsert_all` | `find_or_create_by!(...).update!(...)` | Class |
-
-```ruby
-user.update_columns(name: "New")
-# => raises Lyra::StrictDataAccessViolation:
-#    "update_columns bypasses callbacks and won't be captured by Lyra.
-#     Use update! instead. Disable strict_data_access mode if this is intentional."
-
-Registration.where(status: "pending").update_all(status: "expired")
-# => raises Lyra::StrictDataAccessViolation:
-#    "update_all bypasses callbacks. Use find_each { |r| r.update!(...) } instead."
-```
-
-#### Bypassing Strict Mode
-
-For legitimate bulk operations in migrations, seeds, or admin tasks, use `Lyra.without_strict_access`:
-
-```ruby
-# Temporarily bypass strict mode for bulk operations
-Lyra.without_strict_access do
-  User.where(active: false).delete_all  # No error raised
-end
-```
-
-**Note:** Rails association operations (like `dependent: :nullify`) are automatically allowed since they're internal framework operations.
-
-### Dual View Analysis
-
-Compare CRUD state with event-sourced state:
-
-```ruby
-# Single record comparison
-comparison = Lyra::DualView.new(Order, order_id).compare
-
-puts comparison[:crud_view]
-# => { exists: true, attributes: {...} }
-
-puts comparison[:event_sourced_view]
-# => { exists: true, state: {...}, events_count: 5 }
-
-puts comparison[:differences]
-# => { no_differences: true } or differences hash
-
-# Audit trail
-audit = Lyra::DualView.new(Order, order_id).audit_trail
-# => Array of all operations with timestamps and changes
-```
-
-### Batch Analysis
-
-Find discrepancies across all records:
-
-```ruby
-discrepancies = Lyra::DualView.find_discrepancies(Order)
-# => Returns all orders where CRUD state != Event-sourced state
-
-analysis = Lyra::StateAnalyzer.analyze(Order, order_id)
-puts analysis[:recommendations]
-# => Actionable recommendations based on state comparison
-```
-
-### Switching to Hijack Mode
-
-After analyzing in monitor mode, switch to full event sourcing:
-
-```ruby
-# In config/initializers/lyra.rb
-Lyra.configure do |config|
-  config.enable_hijack!
-end
-```
-
-Now CRUD operations are intercepted and routed through event sourcing:
-
-```ruby
-order = Order.create!(total: 100, customer: customer)
-# => CreateCommand processed
-# => OrderCreated event published
-# => Aggregate updated
-# => Database record created with event-sourced ID
-```
-
-### Dashboard
-
-Lyra provides a web dashboard for monitoring and analyzing event-sourced data.
-
-#### Mounting the Dashboard
-
-Add to your `config/routes.rb`:
-
-```ruby
-Rails.application.routes.draw do
-  mount Lyra::Engine, at: "/lyra"
-  # ... rest of your routes
-end
-```
-
-#### Available Routes
-
-| Route | Description |
-|-------|-------------|
-| `/lyra/dashboard` | Main dashboard with monitored models overview |
-| `/lyra/dashboard/model/:class` | Model-specific overview (e.g., `/lyra/dashboard/model/Order`) |
-| `/lyra/dashboard/compare/:class/:id` | Dual View - Compare CRUD state vs Event-sourced state |
-| `/lyra/dashboard/discrepancies/:class` | List records with state discrepancies |
-
-**Event Flow Routes:**
-
-| Route | Description |
-|-------|-------------|
-| `/lyra/flow/timeline` | Global event timeline |
-| `/lyra/flow/event_chain/:class/:id` | Event chain for a specific record |
-| `/lyra/flow/crud_mapping` | CRUD operation to event type mapping |
-| `/lyra/flow/correlation/:correlation_id` | Events by correlation ID |
-| `/lyra/flow/user_actions/:user_id` | Events by user |
-
-**Visualization Routes:**
-
-| Route | Description |
-|-------|-------------|
-| `/lyra/visualizations/event_graph` | Interactive event graph (HTML + Mermaid) |
-| `/lyra/visualizations/event_graph.json` | Event graph data with filters |
-| `/lyra/visualizations/entity_graph/:class/:id.json` | Entity lifecycle graph |
-| `/lyra/visualizations/heatmap` | Activity heatmap view |
-| `/lyra/visualizations/heatmap.json` | Heatmap data for time period |
-| `/lyra/visualizations/event_list.json` | List of entities with event counts |
-
-**Event Graph Features:**
-
-The event graph provides an interactive visualization of entity lifecycles:
-
-- **Entity Picker**: Filter by model class, select specific entities to view
-- **Changed Fields**: Each event node displays which fields were modified (📝 prefix)
-- **Link Types**:
-  - Solid lines → Same entity lifecycle (chronological order)
-  - Dashed lines → Correlated events (same transaction/request)
-- **Node Details**: Includes operation type, timestamp, and changed field names
-
-Example node display:
-```
-┌─────────────────────┐
-│      UPDATED        │
-│     14:30:45        │
-│ 📝 status, amount   │
-└─────────────────────┘
-```
-
-Timestamp fields (`*_at`) are automatically excluded from the changed fields display.
-
-**Schema Management Routes:**
-
-| Route | Description |
-|-------|-------------|
-| `/lyra/dashboard/schema` | Current schema with pending changes |
-| `/lyra/dashboard/schema/history` | Schema version history |
-| `/lyra/dashboard/schema/:version` | View specific schema version |
-
-**Formal Verification Routes (requires PetriFlow):**
-
-| Route | Description |
-|-------|-------------|
-| `/lyra/verification` | Verification dashboard |
-| `/lyra/verification.json` | Verification results (JSON) |
-
-**Privacy & GDPR Routes:**
-
-| Route | Description |
-|-------|-------------|
-| `/lyra/privacy/pii_detection` | Automatic PII field detection |
-| `/lyra/privacy/gdpr_report/:type/:id` | GDPR Article 15 compliant report |
-| `/lyra/privacy/subject/:type/:id` | View all data for a subject |
-
-#### Securing Dashboard Access
-
-**IMPORTANT**: The Lyra dashboard exposes sensitive data including PII fields, event history, and audit trails. Always restrict access in production.
-
-##### Option 1: Authentication Constraint (Recommended)
-
-```ruby
-# config/routes.rb
-Rails.application.routes.draw do
-  # Restrict to authenticated admin users
-  authenticate :user, ->(u) { u.admin? } do
-    mount Lyra::Engine, at: "/lyra"
-  end
-end
-```
-
-##### Option 2: Basic HTTP Authentication
-
-```ruby
-# config/routes.rb
-Rails.application.routes.draw do
-  mount Lyra::Engine, at: "/lyra", constraints: ->(req) {
-    Rack::Auth::Basic::Request.new(req.env).provided? &&
-    Rack::Auth::Basic::Request.new(req.env).credentials ==
-      [ENV['LYRA_USER'], ENV['LYRA_PASSWORD']]
-  }
-end
-```
-
-##### Option 3: IP Whitelist
-
-```ruby
-# config/routes.rb
-Rails.application.routes.draw do
-  constraints ->(req) { ['127.0.0.1', '::1'].include?(req.remote_ip) } do
-    mount Lyra::Engine, at: "/lyra"
-  end
-end
-```
-
-##### Option 4: Custom Middleware
-
-```ruby
-# lib/lyra_auth_middleware.rb
-class LyraAuthMiddleware
-  def initialize(app)
-    @app = app
-  end
-
-  def call(env)
-    if env['PATH_INFO'].start_with?('/lyra')
-      # Your authentication logic here
-      return [403, {}, ['Forbidden']] unless authorized?(env)
-    end
-    @app.call(env)
-  end
-
-  private
-
-  def authorized?(env)
-    # Implement your authorization logic
-    env['warden']&.user&.admin?
-  end
-end
-
-# config/application.rb
-config.middleware.use LyraAuthMiddleware
-```
-
-##### Option 5: Disable in Production
-
-```ruby
-# config/routes.rb
-Rails.application.routes.draw do
-  unless Rails.env.production?
-    mount Lyra::Engine, at: "/lyra"
-  end
-end
-```
-
-#### Environment-Based Configuration
-
-```ruby
-# config/routes.rb
-Rails.application.routes.draw do
-  case Rails.env
-  when 'development'
-    # Open access in development
-    mount Lyra::Engine, at: "/lyra"
-  when 'staging'
-    # Basic auth in staging
-    mount Lyra::Engine, at: "/lyra", constraints: LyraBasicAuth
-  when 'production'
-    # Full authentication in production
-    authenticate :user, ->(u) { u.admin? } do
-      mount Lyra::Engine, at: "/lyra"
-    end
-  end
-end
-```
-
-### Custom Event Mappers
-
-Create custom event mapping logic:
-
-```ruby
-class OrderEventMapper < Lyra::EventMapper
-  def event_data
-    super.merge(
-      business_context: {
-        total: data[:attributes]['total'],
-        items_count: data[:attributes]['items_count']
-      }
-    )
-  end
-end
-
-Lyra::EventMapper.register_mapper(Order, OrderEventMapper)
-```
-
-### State Reconstruction
-
-Rebuild current state from events:
-
-```ruby
-# Using projection
-state = Lyra::StateProjection.rebuild_state(Order, order_id)
-# => { total: 150, status: "confirmed", ... }
-
-# Using aggregate
-aggregate = OrderAggregate.load(order_id)
-aggregate.total  # => 150
-aggregate.status # => "confirmed"
-```
-
-### Pluggable Backends
-
-Implement custom event storage:
+## Modes
+
+The mode is application-wide. Per model you choose only whether it is
+monitored (`monitor_with_lyra` in the model, or `config.models=` in the
+initializer). Event sourcing takes one of four projection modes, which gives
+seven configurations:
+
+| Configuration | `mode` | `projection_mode` | Authoritative store |
+|---|---|---|---|
+| Disabled | `:disabled` | – | tables |
+| Monitor | `:monitor` (default) | – | tables |
+| Hijack | `:hijack` | – | events |
+| ES-Sync | `:event_sourcing` | `:sync` (default) | events |
+| ES-Async | `:event_sourcing` | `:async` | events |
+| ES-NoProj | `:event_sourcing` | `:disabled` | events |
+| ES-Lazy | `:event_sourcing` | `:lazy` | events |
+
+- **Monitor** appends the event in `after_create`/`after_update`/`after_destroy`,
+  inside the write's transaction (not `after_commit`).
+- **Hijack and event sourcing** build the event in hooks prepended to
+  ActiveRecord's persistence, after the model's `before_*` callbacks.
+- **Callback-bypassing writes** (`update_columns`, `update_column`, `touch`,
+  `delete`, `update_all`, `delete_all`, `insert_all`, `insert_all!`, `upsert_all`,
+  and the updates of `dependent: :nullify`) on a monitored model publish bypass
+  events; with `config.strict_data_access = true` they raise instead (except
+  `touch` and the `dependent: :nullify` updates).
+- Each record has its own stream, named `"Model$id"` (for example `"Order$42"`).
+- **ES-NoProj** writes no rows; it answers a query exactly from the events or
+  raises `Lyra::Projections::UnsupportedQuery`.
+- **Switching modes after boot is gated**: `Lyra::ModeTransition.to!` (and
+  `config.enable_hijack!`, `enable_monitor!`, `enable_event_sourcing!`) checks
+  that rows and events agree before a switch that changes the authoritative
+  store, and the engine refuses to boot into such a switch without a
+  certificate from `rake lyra:mode:check`. See
+  [Switching Modes](docs/MIGRATION_GUIDE.md#switching-modes).
+
+## Requirements and installation
+
+- Ruby >= 4.0, Rails >= 8.0, `rails_event_store` ~> 3.0 (from `lyra.gemspec`).
+- PostgreSQL. Lyra depends on the `pg` gem; the advisory locks behind Genesis
+  and ES-Lazy, and id reservation in Hijack mode, need PostgreSQL.
+
+The gem names differ from the files to require:
 
 ```ruby
-class MyEventStore < Lyra::CustomEventStoreAdapter
-  def publish(event, stream_name:)
-    # Custom implementation (e.g., Kafka, EventStoreDB)
-  end
-
-  def read_stream(stream_name)
-    # Custom implementation
-  end
-end
-
-Lyra.configure do |config|
-  config.event_backend = :custom
-  config.event_store = MyEventStore.new
-end
-```
+# Gemfile
+gem "orfeas_lyra", path: "path/to/lyra", require: "lyra"
+gem "orfeas_pam_dsl", path: "path/to/lyra/gems/pam_dsl", require: "pam_dsl"          # optional, privacy features
+gem "orfeas_petri_flow", path: "path/to/lyra/gems/petri_flow", require: "petri_flow"  # optional, verification
+```
+
+Create the event store tables with the RailsEventStore 3 generator
+(`bin/rails generate ruby_event_store:active_record:migration`), then add an
+initializer. The [Migration Guide](docs/MIGRATION_GUIDE.md) walks through the
+whole path from Disabled to event sourcing.
+
+## Features
+
+Beyond the modes above (see the [API Reference](docs/API_REFERENCE.md) for each):
+
+- **Genesis**: rows that predate Lyra get an `Imported` event on first use, or
+  ahead of time with `rake lyra:genesis`.
+- **Failure policies**: Hijack and event sourcing are fail-closed (the write
+  rolls back); Monitor logs and continues, and `rake lyra:repair` brings the
+  event log back in line with the tables.
+- **DualView**: compare a record's table row with its event-sourced state;
+  optionally sampled after commit (`config.dual_view_sample_rate`, off by default).
+- **Point-in-time state**: `Lyra.state_at(Model, id, time)` and `Model.as_of(time)`.
+- **Privacy** (all opt-in; most need the pam_dsl gem): purpose-bound reads,
+  an access log, privacy stamps in event metadata, erasure of one record's
+  personal data from its row and its events (`rake lyra:erase`), and retention
+  rules (`rake lyra:retention:apply`).
+- **Formal verification** (needs petri_flow): Petri net models of each mode's
+  write path, checked with PetriFlow (`rake lyra:workflows:verify`), and a
+  check of the CRUD-to-event mapping (`Lyra.verify_mapping!`).
+- **Dashboard**: `mount Lyra::Engine, at: "/lyra"`. Outside development and
+  test it answers 403 until `config.dashboard_authorization` (a proc run on the
+  controller, e.g. `->(controller) { controller.current_user&.admin? }`) says
+  who may use it.
 
 ---
 
-## Example Applications
-
-### Aegean E-Pay Testbed (Comprehensive)
-
-A full-featured university payment system testbed is included in `examples/aegean_epay_testbed/`. This real-world example demonstrates:
-
-- **CRUD Operations** - Registrations, payments, refunds with automatic event generation
-- **Dual-View Comparison** - Compare CRUD state vs event-sourced state
-- **Privacy Policy Enforcement** - PAM DSL integration for GDPR compliance
-- **Audit Trail Generation** - Complete history from immutable events
-- **State Machine Workflows** - Refund request lifecycle with events
-- **Performance Benchmarking** - Compare overhead with/without Lyra
-
-#### Running the Testbed
-
-```bash
-cd examples/aegean_epay_testbed
-bundle install
-rails db:create db:migrate db:seed
-
-# Run integration tests
-ruby test_lyra_integration.rb
-
-# Or with performance benchmarks
-ruby test_lyra_integration.rb --performance
-```
-
-[Explore the Aegean E-Pay testbed →](examples/aegean_epay_testbed/README.md)
-
-### Blog App (Getting Started)
-
-A simpler example for learning Lyra basics is in `examples/blog_app/`:
-
-```bash
-cd examples/blog_app
-bundle install
-rails db:create db:migrate db:seed
-rails console
-```
+## Example application
 
-[Explore the blog example →](examples/blog_app/README.md)
+`examples/blog_app/` is a small blog (users, posts, comments) in Monitor, on
+Rails 8.1, RailsEventStore 3 and the repository's PostgreSQL
+(`docker compose up -d db`, then `bin/setup` and `bin/rails test` in the app);
+see its [README](examples/blog_app/README.md). The files
+[`examples/usage_examples.rb`](examples/usage_examples.rb),
+[`examples/privacy_examples.rb`](examples/privacy_examples.rb) and
+[`examples/privacy_policy_usage.rb`](examples/privacy_policy_usage.rb) show
+individual calls.
 
 ---
 
@@ -991,21 +111,21 @@ rails console
 - **Orthogonal Analysis**: Compare static vs. dynamic views of system state
 - **Behavioral Analysis**: Understand system evolution over time
 - **Migration Patterns**: Study CRUD-to-ES transformation strategies
-- **Formal Verification**: Prove correctness using Petri net theory
+- **Formal Verification**: Check the CRUD-to-event mapping with Petri net models (petri_flow)
 - **Privacy Compliance**: Research privacy-preserving event sourcing patterns
 
 ### For Development
-- **Zero Downtime Migration**: Gradually transition to event sourcing
-- **Audit Trail**: Complete history of all state changes
-- **Temporal Queries**: Query state at any point in time
+- **Gradual Migration**: Move to event sourcing one checked mode switch at a time
+- **Audit Trail**: History of every write to a monitored model that goes through ActiveRecord
+- **Temporal Queries**: Query state at a point in time
 - **Debugging**: Replay events to understand issues
-- **CQRS Support**: Natural separation of commands and queries
+- **CQRS Support**: Commands and aggregates alongside the projected tables
 
 ### For Operations
-- **Non-intrusive**: Deploy without code changes
-- **Rollback Safety**: Keep CRUD as safety net during transition
-- **Real-time Monitoring**: Compare CRUD vs event-sourced state
-- **Validation**: Verify event sourcing correctness before full migration
+- **Few code changes**: A model is monitored by one line or by naming it in the initializer; controllers and queries stay as they are (see [Adopting Lyra](docs/MIGRATION_GUIDE.md#adopting-lyra-in-an-existing-application) for what this costs)
+- **Rollback**: Every mode can be switched back to the previous one
+- **Monitoring**: Compare table and event-sourced state (DualView)
+- **Validation**: A switch that makes the events authoritative is checked first
 - **Performance Analysis**: Measure overhead before committing
 
 ---
@@ -1013,12 +133,19 @@ rails console
 ## Documentation
 
 ### Core Documentation
-- **[Getting Started Guide](docs/GETTING_STARTED.md)** - Installation and first steps
+- **[Getting Started](docs/GETTING_STARTED.md)** - Install, monitor one model, and check its events
+- **[Migration Guide](docs/MIGRATION_GUIDE.md)** - Installation and the step-by-step move from CRUD to event sourcing
+- **[API Reference](docs/API_REFERENCE.md)** - Configuration options, methods, rake tasks and errors
 - **[Architecture Overview](docs/ARCHITECTURE.md)** - System design and components
-- **[Monorepo Structure](docs/MONOREPO.md)** - Repository organization
+- **[Switching Modes](docs/MIGRATION_GUIDE.md#switching-modes)** - Who holds the mode, and how to switch safely (the deploy-time rule)
+- **[Adopting Lyra](docs/MIGRATION_GUIDE.md#adopting-lyra-in-an-existing-application)** - What "non-intrusive" means, the costs, and a checklist for a new codebase
+- **[Performance](docs/PERFORMANCE.md)** - What each mode costs and why (counted in SQL statements per write; throughput being re-measured)
+- **[Troubleshooting](docs/TROUBLESHOOTING.md)** - Common problems
+- **[Workflow Generator](docs/WORKFLOW_GENERATOR.md)** - The Petri net workflow models of Lyra's modes and their verification
+- **[Changelog](CHANGELOG.md)**
 
 ### Theoretical Foundation
-- **[ORFEAS Framework Overview](docs/ORFEAS_FRAMEWORK_OVERVIEW.md)** - Complete framework description
+- **[ORFEAS Framework Overview](docs/ARCHITECTURE.md#about-orfeas)** - The framework: naming, vision, components and migration path (the opening of ARCHITECTURE.md)
 - **[Petri Nets Model](gems/petri_flow/docs/THEORETICAL_MODEL_PETRI_NETS.md)** - P/T nets for verification, CPNs for data modeling
 - **[Matrix Analysis Model](gems/petri_flow/docs/THEORETICAL_MODEL_MATRICES.md)** - Linear algebra approach
 
@@ -1042,10 +169,26 @@ ORFEAS and Lyra are grounded in peer-reviewed research:
 
 ### Published Papers
 
+**Pantelelis, M., & Kalloniatis, C. (2026).** *From ORM to OEM: Object-to-Events Mapping for the OLEP Era.*
+IEEE Software.
+DOI: [10.1109/MS.2026.3734188](https://doi.org/10.1109/MS.2026.3734188)
+
+**Pantelelis, M., & Kalloniatis, C. (2026).** *PAM: A Domain-Specific Language for Specifying Privacy Requirements from Regulation to Runtime.*
+Requirements Engineering, Springer.
+DOI: [10.1007/s00766-026-00469-6](https://doi.org/10.1007/s00766-026-00469-6)
+
+**Pantelelis, M., & Kalloniatis, C. (2024).** *Create, Read, Update, Delete: Implications on Security and Privacy Principles regarding GDPR.*
+ARES 2024, 21st International Workshop on Trust, Privacy and Security in the Digital Society (TrustBus), ACM.
+DOI: [10.1145/3664476.3670898](https://doi.org/10.1145/3664476.3670898)
+
 **Pantelelis, M., & Kalloniatis, C. (2022).** *Mapping CRUD to Events: Towards an object to event-sourcing framework.*
 26th Pan-Hellenic Conference on Informatics (PCI 2022).
 DOI: [10.1145/3575879.3576006](https://doi.org/10.1145/3575879.3576006)
 
+**Pantelelis, M., & Kalloniatis, C. (2022).** *Object Relational Mapping vs. Event-Sourcing: Systematic Review.*
+EGOVIS 2022, Lecture Notes in Computer Science 13429, Springer, pp. 18–31.
+DOI: [10.1007/978-3-031-12673-4_2](https://doi.org/10.1007/978-3-031-12673-4_2)
+
 ### Research Areas
 
 - **Event Sourcing Patterns**: Formal models for CRUD-to-event transformation
@@ -1097,12 +240,19 @@ cd lyra
 # Install dependencies
 bundle install
 
-# Run tests
-rake test
+# The engine's tests need PostgreSQL on localhost:5433 (user and password
+# "postgres", database lyra_test; see test/dummy/config/database.yml).
+# docker-compose.yml provides one:
+bundle exec rake docker:start
+docker exec lyra_postgres createdb -U postgres lyra_test
+
+# Run the engine's tests, then pam_dsl's and petri_flow's
+bundle exec rake test:all
 
-# Build gems
-cd gems/petri_flow && rake build
-cd gems/pam_dsl && rake build
+# Build the gems
+bundle exec rake build
+(cd gems/pam_dsl && bundle exec rake build)
+(cd gems/petri_flow && bundle exec rake build)
 ```
 
 ---
@@ -1123,7 +273,50 @@ If you use this software in academic research, please cite:
 }
 ```
 
-### Research Paper Citation
+### Research Paper Citations
+
+Lyra's object-to-events mapping, and how its modes compare in cost:
+
+```bibtex
+@article{pantelelis2026oem,
+  title={From {ORM} to {OEM}: Object-to-Events Mapping for the {OLEP} Era},
+  author={Pantelelis, Michail and Kalloniatis, Christos},
+  journal={IEEE Software},
+  year={2026},
+  doi={10.1109/MS.2026.3734188}
+}
+```
+
+PAM, the privacy DSL (`gems/pam_dsl`), from regulation to runtime:
+
+```bibtex
+@article{pantelelis2026pam,
+  title={{PAM}: A Domain-Specific Language for Specifying Privacy Requirements from Regulation to Runtime},
+  author={Pantelelis, Michail and Kalloniatis, Christos},
+  journal={Requirements Engineering},
+  publisher={Springer},
+  year={2026},
+  doi={10.1007/s00766-026-00469-6}
+}
+```
+
+What each CRUD operation means for GDPR's security and privacy principles, the ground of PAM and Lyra's privacy events:
+
+```bibtex
+@inproceedings{pantelelis2024crud,
+  title={Create, Read, Update, Delete: Implications on Security and Privacy Principles regarding {GDPR}},
+  author={Pantelelis, Michail and Kalloniatis, Christos},
+  booktitle={Proceedings of the 19th International Conference on Availability, Reliability and Security (ARES 2024), 21st International Workshop on Trust, Privacy and Security in the Digital Society (TrustBus)},
+  articleno={178},
+  pages={1--7},
+  publisher={ACM},
+  address={Vienna, Austria},
+  year={2024},
+  doi={10.1145/3664476.3670898}
+}
+```
+
+The CRUD-to-events mapping as first proposed:
 
 ```bibtex
 @inproceedings{pantelelis2022mapping,
@@ -1135,6 +328,22 @@ If you use this software in academic research, please cite:
 }
 ```
 
+The literature review behind it, ORM against Event Sourcing:
+
+```bibtex
+@inproceedings{pantelelis2022egovis,
+  title={Object Relational Mapping vs. Event-Sourcing: Systematic Review},
+  author={Pantelelis, Michail and Kalloniatis, Christos},
+  booktitle={Electronic Government and the Information Systems Perspective (EGOVIS 2022)},
+  series={Lecture Notes in Computer Science},
+  volume={13429},
+  pages={18--31},
+  publisher={Springer},
+  year={2022},
+  doi={10.1007/978-3-031-12673-4_2}
+}
+```
+
 ### References
 
 - [Rails Event Store](https://railseventstore.org/) - Event Store implementation