activerecord-materialized 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d3b0e7bdccf7c3354e443d0c824e1ac372392e66517c3422df35d6ccc393adb0
-  data.tar.gz: a405e79709760fa5f7cadcfedae9ccd132f98081e9278cf127f01fdff0c8bbf7
+  metadata.gz: 20172f4721a9fec7ab702812329c1f217524f2b03b1a9334f9645b8439b584c4
+  data.tar.gz: d2c308b700d98bdfa9c3703e02acd9351ccaa4a8ea9655236878bdfe3288dd72
 SHA512:
-  metadata.gz: e92ca73b2ad1c34dd4ba7952293a15ba12c87b357dd1d33efe2c10dbc26580e2e8eef73cc9b4c107e1c568f751482e506ee01acd17dcb7f7db45a0bf098f2779
-  data.tar.gz: d8086c2df447684c4076577c8fd00982d67d4c0e773eef2680b2585f810ab4d83a3a046170dcdc5027793580b2ad50c6565f9e69247938bfc8b23f3b25c9e6cb
+  metadata.gz: 631f0b027ab071ca180ea19f0643b9f7917a508d734f87f06b26ef5a16c871b5103010b63f8ae020ac20cb7d0394d2e297c0f1d7c0b2e4b207eef37323611044
+  data.tar.gz: 8ca8d97f0c6b2fa76fac92c546429373d5d2e4348ac446e18bb5bfe9820653713bb9433d66a423c33e287c8cbf94ecf20cfec0db3fda44d5459425bd2209294a

data/CHANGELOG.md

@@ -1,5 +1,98 @@
 # Changelog
 
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## Unreleased
+
+### Added
+
+- **Pluggable change sources & CDC ingestion.** Change *detection* is now decoupled from view
+  *maintenance* behind a public ingestion API — `publish_write_change!` and `mark_dirty_for_tables!`
+  (with the lower-level `ingest_change` for a normalized change descriptor) — so a view can be fed
+  from ActiveRecord callbacks (still the default), raw SQL, another service, or a CDC stream. Choose
+  the source globally with `config.default_change_source` or per view with `change_source :none`;
+  each view is fed by exactly one source. A dependency-free Debezium envelope adapter
+  (`ingest_debezium_change`) maps a decoded change event straight into the pipeline, and optional
+  monotonic **source watermarks** (`source_ts:`, exposed via `View.source_watermark`) suppress
+  provably-stale out-of-order changes and surface per-view freshness/lag; Debezium's `source.ts_ms`
+  is forwarded as that watermark automatically. Delivery is at-least-once and order-tolerant —
+  duplicates and reordering converge. Validated end-to-end against real MySQL binlog and Postgres
+  logical-replication decoding. See the [Change sources](docs/change-sources.md) docs.
+  (#80, #105, #106, #113, #117, #118)
+- **Out-of-band write capture via triggers/outbox.** A database-native outbox change source
+  (`WriteOutbox.install_triggers!` + `drain_write_outbox!`, with an `activerecord_materialized:outbox`
+  generator) captures writes that bypass both ActiveRecord callbacks and the ingestion API — raw SQL,
+  bulk backfills, or another service on the shared database. Triggers record the changed `GROUP BY`
+  key columns, scoped per operation so a partition-moving update maintains both the old and new
+  partition, and draining is batched, at-least-once, and per-row isolated. Portable across Postgres,
+  MySQL, and SQLite; opt-in. See the [detecting out-of-band writes](docs/out-of-band-writes.md) guide.
+  (#68)
+- **Self-healing reconciliation & data-drift detection.** A `DataVerifier`
+  (`verify_data` / `verify_data!`, `materialized:audit`) recomputes the source per partition and
+  reports `missing` / `extra` / `mismatched` keys, with `:row_count`, `:checksum`, and `:full` modes
+  plus cheap sampling for large views. Built on it, self-healing reconciliation
+  (`reconcile!` / `reconcile_stale!`, `materialized:reconcile`) verifies views on a schedule and
+  **scoped-repairs** whatever the change source missed — never a full `rebuild!` — bounding staleness
+  in time and composing with `max_staleness`. Reconciliation is safe alongside normal refresh: it
+  drains pending maintenance first, defers rather than double-maintains when a refresh already holds
+  the cycle, and isolates per-view failures. See
+  [Data integrity: drift detection & self-healing](docs/reconciliation.md). (#62)
+- **Distributed / HA maintenance.** New `enqueue_stale_refreshes!` / `enqueue_stale_reconciles!`
+  (and matching rake tasks) fan the periodic bounded-staleness backstop out across the job fleet as
+  one job per stale view, backed by a new `ReconcileJob` that re-checks staleness before working.
+  Writer/replica topology support routes maintenance to the primary and verification reads to a
+  replica (`config.maintenance_role` / `config.verification_role`), reads cache and source in a
+  single snapshot to avoid false drift, and folds a `config.replica_lag` budget into `stale?`.
+  Cross-process cycles are serialized by a metadata-row lock so concurrent servers apply additive
+  deltas exactly once and recompute a partition once. Run the periodic tick from a single owner — the
+  gem ships no scheduler or leader election. See the
+  [distributed deployment](docs/distributed-deployment.md) docs. (#92, #93, #94, #95)
+- **Observability via `ActiveSupport::Notifications`.** The read, refresh, and maintenance lifecycle
+  points emit documented events (`read.active_record_materialized`,
+  `refresh.active_record_materialized`, `maintenance.active_record_materialized`) with stable
+  payloads — cache-hit vs read-through and staleness on reads; duration, mode, partitions recomputed,
+  and outcome on refreshes; and a widen-to-full signal on writes. No telemetry-vendor dependency;
+  read events fire only when a subscriber is attached. See the [Observability](docs/observability.md)
+  docs.
+- **Scoped maintenance for joined-table keys.** `partition_key_for(table) { |change| ... }` maps a
+  write on a joined/leaf dependency table — whose own payload lacks the `GROUP BY` key — to the
+  affected partition key(s), keeping maintenance scoped to those partitions instead of widening to a
+  full recompute. See the [joined-table docs](docs/architecture.md#views-whose-group-key-lives-on-a-joined-table).
+
+### Changed
+
+- **`config.refresh_dispatcher` now auto-resolves to `:active_job` when ActiveJob is loaded**
+  (previously the in-process `:async` thread), so a typical multi-server Rails deployment coordinates
+  refresh across servers by default; an explicit `config.refresh_dispatcher` still wins.
+  **Upgrade note:** ensure a worker drains `config.refresh_queue_name` (default `:materialized_views`)
+  or set `config.refresh_dispatcher = :async` explicitly — otherwise refresh-on-write jobs enqueue
+  with nothing to run them and views serve stale reads. A boot warning fires when the in-process
+  refresher is active despite ActiveJob being available. `config.reconcile_queue_name` defaults to the
+  refresh queue. (#93)
+
+### Fixed
+
+- **Cold-view widen correctness.** A full-partition (widen) recompute can't be applied to a never-built
+  (cold) view; letting one through previously killed populate-on-read until a manual `rebuild!`. Every
+  widen producer now funnels through a single `MaintenanceStore#merge!` chokepoint that drops the
+  cold-view recompute and resets the fresh set, and a lock-free per-view **epoch**
+  (`fresh_set_generation`) closes a populate-vs-widen race that could otherwise serve stale data on a
+  cold view. Reads fall through to the always-correct source and repopulate on the next miss;
+  warm/materialized views are unaffected. (#110, #120)
+- **Partition-moving update under-scoping.** A CDC/ingestion `:update` with a partial `before`-image
+  (no `GROUP BY` key — the common non-FULL image: Postgres default `REPLICA IDENTITY`, MySQL
+  `binlog-row-image=minimal`) previously left the *old* partition stale. The delta builder now
+  recognizes it cannot identify every affected partition from a partial image and widens to a full
+  recompute, honoring the ingestion API's documented always-correct guarantee. Scoped updates with
+  full before/after images (or `key_attributes`) are unchanged. (#110)
+
+### Internal
+
+- Shared per-partition-store base (`PartitionKeyedStore`) factored out of `PartitionState` and
+  `SourceWatermark`, and documentation/tooling cleanup. (#115, #124)
+
 ## 0.1.0 (2026-06-18)
 
 Initial release.