standard_audit 0.7.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1ab5fcec99d13f0796742d817c1dd42e45c0e8518f8967033a67194c3179ae6e
4
- data.tar.gz: b0f6272a600a665615cc533b439ad2b654d54c025bd7bc2b378b5c2eac7e850d
3
+ metadata.gz: a1d506a4bb5ad3811dd4a70b6be817374ec9a88ced8d1ffd9415bfddd28cc50b
4
+ data.tar.gz: 52c8aaa70f86810c6b308c86fa81e10c368ab5ddb011d16f422745e7681c88da
5
5
  SHA512:
6
- metadata.gz: 1ab1f514a13f1b03501cfdf18b3c2afcf8ff214a36992edb3e00f052384826c56bf09a35c4d7c4b8f92a4f28513cd8fe5534d946622112bb27bb6f8bfd8826c7
7
- data.tar.gz: 7f9fc2acdb292b06f72b2323ac0c5c780348da8d1b91e9850ae73a61eba2361fb21b3aa431888de9d0003dc2934c0ba1b2a9a060bd4bd3b2a6a1bfb71eea5680
6
+ metadata.gz: 2db9ee0ef4983ce2bd9a94d195bbbf973016f32a3e9e2a53b45476df6c9cc6d30f7edd2bfb7e9442d6839e05cc491263b91bb644462946510740c187eb861e29
7
+ data.tar.gz: cce577cbc3f1b5829429ab30765c451f4a0fdca18b9f6283f075e1b3487b3fb763ceaeca846e39768c53707361bfc87024da65e0f8e662885a47766f814dd5c1
data/CHANGELOG.md CHANGED
@@ -7,6 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.9.0] - 2026-07-31
11
+
12
+ ### Added
13
+
14
+ - **`StandardAudit::Operation` — the operation-audit DSL, extracted from five independent copies.** Every consumer app had written the same contract by hand: `audits "x.y"` / `audit_none!` class declarations, a private `audit!(action, **attrs)` as the single write path, and a `verify_audit_declared!` guard that raises on declaration↔write drift in local environments and writes silently in production. The surface is unchanged from those copies, so operation files do not move — adopting apps swap one `include` line and delete their concern. (`rarebit-one/rarebit-ops#279`)
15
+ - It is a **module, never a base class**. The five host `ApplicationOperation`s range 61–303 lines and diverge deliberately (one refuses a `Result`/`execute` lifecycle; one app has no shared base at all), so the module contributes the audit contract and nothing else.
16
+ - **Both adoption shapes work with no configuration**: a shared base that includes it once, whose subclasses are the real operations, and standalone leaves that each include it directly. The unifier is that a class declaring nothing *and* having subclasses is treated as a base and excluded — so a shared base auto-excludes, while every leaf is retained. A class that did declare is never excluded that way.
17
+ - `audit_abstract!` for an intermediate class the automatic rule can't see (typically one with no subclasses yet).
18
+ - **`config.audit_catalogue`** — the host's action vocabulary, as a **callable** (`-> { AuditCatalogue::ACTIONS }`); referencing an autoloadable constant eagerly from an initializer breaks Zeitwerk reloading. A plain Array is accepted for a frozen literal. `nil` (the default) skips the membership check, so the DSL is adoptable before an app has a catalogue. **Membership is the only rule applied** — no dot-count, case, prefix, or namespace validation, because an action may legitimately carry a notification-bus namespace verbatim and normalising it would orphan historical rows.
19
+ - **`config.raise_on_audit_write_error`** (default `false`) and **`config.audit_write_error_handler`**. Four of the five apps report-and-swallow a failed audit write; one deliberately does not rescue, because for it an unaudited state change is itself a compliance failure. A swallow-only module would have silently downgraded that posture. `StandardAudit::Operation::DeclarationError` is never governed by either — it always re-raises, ahead of any generic rescue.
20
+ - **`config.verify_audit_declarations`** (callable or boolean, default: local environments only) — gates the dev/test guard.
21
+ - **Meta-spec logic as plain Ruby predicates** on `StandardAudit::Operation::Audit`: `.operations(source:)`, `.undeclared`, `.unknown_actions`, `.orphan_actions(within:)`, `.missing_write_sites`, `.unexpected_write_sites`, `.duplicate_catalogue_entries`, `.declared_actions`. Each takes an explicit `operations:` list, so hosts can write bespoke assertions.
22
+ - **`standard_audit/rspec/operation`** — a thin RSpec shared-example layer over those predicates, with no logic of its own. It carries a **`minimum:` registry floor**, the one example that catches "someone stopped including the module" or "eager loading stopped reaching the operations"; without it every other example passes vacuously against an empty set. Also `expected:`, `source:` scoping, and `orphans_within:` for hosts whose catalogue covers writers outside `app/operations/`.
23
+
24
+ Everything is additive and nothing includes the module by default — existing hosts upgrade with no code change.
25
+
26
+ ## [0.8.0] - 2026-07-30
27
+
28
+ ### Added
29
+
30
+ - **`previous_checksum` — every row now records the row it was appended to.** New nullable column; run `rails generate standard_audit:add_previous_checksum && rails db:migrate` to add it. The install migration includes it for new hosts, along with a composite `(created_at, id)` index for the verification walk.
31
+ - **`rake standard_audit:relink_checksums` / `AuditLog.relink_checksums!`** — records, for every row that has no `previous_checksum`, the parent digest it was *actually* signed against, recovering it by search where a concurrent append forked the log. It **never rewrites a `checksum`**: a parent is written only when it reproduces the digest the row has held since it was written, so it adds no attestation the rows did not already carry. Rows it cannot resolve are counted as `unresolved`, left untouched, and keep failing verification — which is the point. This is the repair path for a log broken by the defect below; `backfill_checksums!` is **not** (it re-signs rows from their current contents, so a green result attests only that a script ran).
32
+ - `AuditLog.chain_tip_checksum` and `AuditLog.chain_parent_column?`.
33
+
34
+ ### Fixed
35
+
36
+ - **The chained checksum was not concurrency-safe: `verify_chain` failed for 3,762 of 5,577 rows (67%) on one production log, continuously, while every row was untampered.** `compute_checksum` reads the chain tip with an unsynchronised query from `before_create`. Two transactions cannot see each other's uncommitted rows, so both sign against the same predecessor and the sequence forks; verification, which assumed a strict line, reported all but one of them as tampering. Any host with concurrent audit writes — two Puma threads, a web request and a job — has the same broken log. (`fundbright/delivery-ops#433`)
37
+
38
+ **The log is now treated as a DAG rather than a strict line, and each row records its parent.** A row is verified against the exact digest it signed, whether or not that is its predecessor in the walk, so a fork is verifiable instead of fatal. The tip read stays unlocked deliberately: forcing a strict line means holding a lock until the *enclosing business transaction* commits — a row is invisible to other writers until then, so releasing earlier reopens the race — which would serialise every audited request across the estate behind one mutex, on the hot path of every request. `previous_checksum` needs no protection of its own; it is an input to the row's own digest, so editing it invalidates the row.
39
+
40
+ What this deliberately stops claiming: a single total order, and detection of an *inserted* row. A concurrent append and an inserted row are indistinguishable, so a design that tolerates the first cannot detect the second. The strict-line reading did not detect insertions either — it reported every concurrent append as tampering, which on the measured log meant 67% red and any real signal lost in it.
41
+
42
+ **Rows written before this version verify unchanged, with no migration.** With no `previous_checksum`, verification falls back to the preceding row in the walk and then searches back through the last `recovery_window` (default 256) digests for the one that reproduces the row's checksum — recovering the true parent of a forked row without re-signing anything, and reporting how many needed it in the new `recovered:` count. Tamper detection is not weakened: a row whose fields were altered reproduces no candidate's digest. `verify_chain(strict: true)` skips the search and reports every fork as a failure. A host that never runs the migration keeps working on this path indefinitely.
43
+
44
+ - **`verify_chain` never noticed a row being removed from the middle of the log.** Failures now carry `reason:` — `:digest_mismatch`, or `:missing_parent` when the row a record was appended to is no longer present. Retention pruning does not trip it: if the walk opens on a row whose parent is already gone, that parent digest is exempt wherever else it appears — a pruned row can have several children, which is what a concurrent append leaves behind. A removal from the *middle* is still reported. Every row is exempt under `scope:`, because the log is global and a scoped row's parent usually belongs to another scope.
45
+
46
+ - **`verify_chain` did not walk the order it documents, and neither did `backfill_checksums!`.** Both claimed a `(created_at, id)` walk and both used `in_batches`, which paginates by **primary key** range and applies the sort only *within* each batch — so the global sequence was a concatenation of id-ranges, each internally time-sorted. Both now walk with a keyset cursor ordered by `(created_at, id)`, which is the documented order and still loads only `batch_size` rows at a time.
47
+
48
+ With the UUIDv7 ids `assign_uuid` generates, id order usually coincides with insertion order, which is why this stayed latent — but it is wrong for any host that assigns ids differently, backfills rows with an explicit `created_at`, or has clock skew between writers. It matters more than a latent ordering bug normally would: the chain is an *ordering* claim, so a verifier that walks a different order than the one it documents cannot be trusted to prove or disprove anything about chain integrity, including the concurrent-append defect above. `backfill_checksums!` had the same defect, and there it silently *writes* the wrong chain rather than misreading one.
49
+
10
50
  ## [0.7.0] - 2026-07-30
11
51
 
12
52
  ### Added
data/README.md CHANGED
@@ -162,6 +162,128 @@ This provides:
162
162
  organisation.scoped_audit_logs # all logs scoped to this organisation
163
163
  ```
164
164
 
165
+ ## Operation Audit Contract
166
+
167
+ `StandardAudit::Operation` makes an operation *declare* what it audits, so a new
168
+ mutating operation cannot silently ship with no audit trail. It is a **module,
169
+ not a base class** — it contributes the audit contract and nothing else (no
170
+ `call`, no `Result`, no `execute`), so it drops into any operation style.
171
+
172
+ ```ruby
173
+ class ApplicationOperation
174
+ include StandardAudit::Operation # ← the whole adoption, for a shared base
175
+ end
176
+
177
+ class Orders::CreateOperation < ApplicationOperation
178
+ audits "order.created"
179
+
180
+ def execute
181
+ order = Order.create!(**attrs)
182
+ audit!("order.created", target: order, metadata: { total: order.total })
183
+ end
184
+ end
185
+
186
+ class Orders::ReindexOperation < ApplicationOperation
187
+ audit_none! # projection of already-audited state
188
+ end
189
+ ```
190
+
191
+ - `audits "x.y"` — the action(s) this operation may write. Repeatable args; the
192
+ declaration is **not inherited**, so every leaf states its own intent.
193
+ - `audit_none!` — mutates state but deliberately records nothing. Leave the
194
+ reason as a comment.
195
+ - `audit_abstract!` — a base or intermediate class, not a real operation.
196
+ - `audit!(action, **attrs)` — private; the single write path. Args forward
197
+ straight to `StandardAudit.record`. Call it inside the operation's own
198
+ transaction so the row commits with the state change.
199
+
200
+ ### Adoption shapes
201
+
202
+ Both work with no configuration:
203
+
204
+ 1. **A shared base includes the module once**, and the real operations are its
205
+ subclasses (registered via `inherited`). The base declares nothing and has
206
+ subclasses, so it is excluded from the checks automatically. A class that
207
+ *did* declare is never excluded this way, so subclassing a real operation
208
+ cannot quietly drop it.
209
+ 2. **Every operation includes the module directly**, with no shared base
210
+ (registered via `included`).
211
+
212
+ Both land in one registry, so a single meta-spec covers a codebase mixing them.
213
+
214
+ ### The catalogue
215
+
216
+ The gem has no knowledge of your action vocabulary. Declare it as a **callable** —
217
+ referencing an autoloadable constant eagerly from an initializer pins the
218
+ first-loaded copy and breaks Zeitwerk reloading:
219
+
220
+ ```ruby
221
+ StandardAudit.configure(baseline: true) do |config|
222
+ config.audit_catalogue = -> { AuditCatalogue::ACTIONS }
223
+ end
224
+ ```
225
+
226
+ `nil` (the default) skips the membership check entirely, so the DSL is adoptable
227
+ before you have a catalogue. **Membership is the only rule** — there is no
228
+ dot-count, case, prefix, or namespace validation, because an action may
229
+ legitimately carry a notification-bus namespace verbatim.
230
+
231
+ ### Error policy
232
+
233
+ `StandardAudit::Operation::DeclarationError` — a declaration↔write mismatch —
234
+ **always propagates**, ahead of any generic rescue. It is raised in local
235
+ environments only (`config.verify_audit_declarations`); in production `audit!`
236
+ just writes, because a developer's mistake must not 500 a user. The meta-spec is
237
+ the real gate.
238
+
239
+ A genuine **write** failure is governed by config:
240
+
241
+ ```ruby
242
+ config.raise_on_audit_write_error = true # default false: report and swallow
243
+ config.audit_write_error_handler = ->(error, action:, operation:) { ... }
244
+ ```
245
+
246
+ Default `false` matches most apps, but set it `true` where an unaudited state
247
+ change is itself a compliance failure — the audit write then aborts the
248
+ operation.
249
+
250
+ ### The meta-spec
251
+
252
+ The analysis is **plain Ruby**, so you can assert on it however you like:
253
+
254
+ ```ruby
255
+ StandardAudit::Operation::Audit.operations(source: "/app/operations/")
256
+ StandardAudit::Operation::Audit.undeclared # => [Class, ...]
257
+ StandardAudit::Operation::Audit.unknown_actions # => { Class => ["x.y"] }
258
+ StandardAudit::Operation::Audit.orphan_actions(within: ...)
259
+ StandardAudit::Operation::Audit.missing_write_sites # declares but never writes
260
+ StandardAudit::Operation::Audit.unexpected_write_sites
261
+ StandardAudit::Operation::Audit.duplicate_catalogue_entries
262
+ ```
263
+
264
+ A thin RSpec layer over exactly those predicates:
265
+
266
+ ```ruby
267
+ require "standard_audit/rspec/operation"
268
+
269
+ RSpec.describe "Operation audit declarations" do
270
+ it_behaves_like "standard_audit operation declarations",
271
+ source: "/app/operations/",
272
+ minimum: 100,
273
+ expected: %w[Orders::CreateOperation],
274
+ orphans_within: -> { AuditCatalogue::OPERATION_ACTIONS }
275
+ end
276
+ ```
277
+
278
+ **Set `minimum:`.** It is the only example that fails when someone stops
279
+ including the module or eager loading stops reaching your operations — every
280
+ other example passes vacuously against an empty set. `orphans_within:` is the
281
+ catalogue slice operations own; pass it when your catalogue also covers writers
282
+ outside `app/operations/`, which would otherwise always look orphaned.
283
+
284
+ The registry can only see loaded classes, so the shared example calls
285
+ `Rails.application.eager_load!` by default (`eager_load: false` to opt out).
286
+
165
287
  ## Configuration Reference
166
288
 
167
289
  Use `configure(baseline: true)` in your initializer. It remembers the block so
@@ -218,6 +340,23 @@ StandardAudit.configure(baseline: true) do |config|
218
340
  config.before_checksum { |log| log.scope = MyApp.derive_scope(log) }
219
341
  config.before_checksum :backfill_scope # an AuditLog instance method
220
342
 
343
+ # -- Operation audit contract (StandardAudit::Operation) --
344
+ # Your action vocabulary, as a CALLABLE so Zeitwerk can reload it. nil (the
345
+ # default) skips the membership check entirely.
346
+ config.audit_catalogue = -> { AuditCatalogue::ACTIONS }
347
+
348
+ # Whether `audit!` verifies declarations before writing. Callable or boolean;
349
+ # defaults to local environments only.
350
+ config.verify_audit_declarations = -> { Rails.env.local? }
351
+
352
+ # Whether a failed audit WRITE aborts the operation. Default false (report and
353
+ # swallow). Set true where an unaudited state change is a compliance failure.
354
+ # DeclarationError always propagates regardless.
355
+ config.raise_on_audit_write_error = false
356
+ config.audit_write_error_handler = ->(error, action:, operation:) {
357
+ ErrorReporting.notify(error, component: "operation_audit", audit_action: action)
358
+ }
359
+
221
360
  # -- Metadata Builder --
222
361
  # Optional proc to transform metadata before storage.
223
362
  config.metadata_builder = ->(metadata) { metadata.slice(:relevant_key) }
@@ -406,9 +545,92 @@ which is **still HTTP 200** — it surfaces the advisory in the readiness JSON
406
545
  without failing the probe or blocking a deploy. The check is duck-typed and has
407
546
  no hard dependency on `standard_health`.
408
547
 
548
+ ## Chain Integrity
549
+
550
+ Every row carries a SHA-256 `checksum` over its own fields **and the digest of
551
+ the row it was appended to**, so editing a row in place invalidates it. Each row
552
+ also records that parent explicitly in `previous_checksum`.
553
+
554
+ ### The log is a DAG, not a strict line
555
+
556
+ A writer reads the current tip and links to it. That read is deliberately
557
+ unlocked, so two concurrent transactions — two Puma threads, a web request and a
558
+ job — can read the same tip and the sequence forks. **That is normal for a
559
+ multi-process writer, and it is not an integrity failure.** Forcing a strict
560
+ line would mean holding a lock until the enclosing business transaction commits
561
+ (a row is invisible to other writers until then, so releasing earlier reopens
562
+ the race), which serialises every audited request behind one mutex.
563
+
564
+ Storing the parent is what keeps a forked log verifiable: every row is checked
565
+ against the exact digest it signed, whether or not it is the walk's predecessor.
566
+ `previous_checksum` needs no protection of its own — it is an input to the row's
567
+ own digest, so editing it invalidates the row.
568
+
569
+ ```ruby
570
+ result = StandardAudit::AuditLog.verify_chain
571
+ # => { valid: true, verified: 5577, recovered: 0, failures: [] }
572
+ ```
573
+
574
+ - `failures` carries `reason: :digest_mismatch` (the row's fields no longer
575
+ produce its digest) or `reason: :missing_parent` (the row it was appended to
576
+ is no longer in the log). Retention pruning does not trip `:missing_parent`:
577
+ if the walk opens on a row whose parent is already gone, that parent digest
578
+ is exempt wherever else it appears — a pruned row can have several children.
579
+ A removal from the *middle* is still reported. One caveat:
580
+ `standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
581
+ `created_at`, so rows whose two timestamps disagree can leave a hole rather
582
+ than a pruned start, and a hole is reported — truthfully, since rows really
583
+ are missing.
584
+ - `recovered` counts rows with no `previous_checksum` whose parent had to be
585
+ found by searching back through recent digests — see below.
586
+ - `verify_chain(scope: org)` skips the missing-parent check, because the log is
587
+ global and a scoped row's parent usually belongs to another scope.
588
+
589
+ What this deliberately does **not** claim: that the log is in a single total
590
+ order, or that no row was inserted. A concurrent append and an inserted row look
591
+ alike, so a design that tolerates the first cannot detect the second. The
592
+ previous strict-line reading did not actually detect insertions either — it
593
+ reported every concurrent append as tampering, which on one production log meant
594
+ 67% of rows red and any real signal lost in the noise.
595
+
596
+ ### Rows written before 0.8.0
597
+
598
+ They have no `previous_checksum`. Verification falls back to the preceding row
599
+ in the walk and, failing that, searches back through the last `recovery_window`
600
+ (default 256) digests for the one that reproduces the row's checksum — which
601
+ recovers the true parent of a row that forked, **without re-signing anything**.
602
+ This does not weaken tamper detection: a row whose fields were altered
603
+ reproduces no candidate's digest. Pass `strict: true` to skip the search and see
604
+ every fork as a failure.
605
+
606
+ To make that permanent, add the column and record what each row was actually
607
+ signed against:
608
+
609
+ ```bash
610
+ rails generate standard_audit:add_previous_checksum
611
+ rails db:migrate
612
+ rake standard_audit:relink_checksums
613
+ ```
614
+
615
+ `relink_checksums` never rewrites a `checksum`. It fills in the previously-empty
616
+ `previous_checksum` only when that value reproduces the digest the row has held
617
+ since it was written, so it adds no attestation the rows did not already carry.
618
+ Rows it cannot resolve are reported as `unresolved` and keep failing
619
+ verification — which is the point.
620
+
621
+ **Do not run `backfill_checksums!` to make a red `verify_chain` go green.** It
622
+ re-signs rows from their current contents, so it attests only that a script ran.
623
+ It is for rows that never had a checksum at all (pre-feature data).
624
+
409
625
  ## Rake Tasks
410
626
 
411
627
  ```bash
628
+ # Verify chain integrity (exits non-zero on failures)
629
+ rake standard_audit:verify
630
+
631
+ # Record the parent digest each existing row was signed against
632
+ rake standard_audit:relink_checksums
633
+
412
634
  # Delete logs older than N days (default: retention_days config or 90)
413
635
  rake standard_audit:cleanup[180]
414
636
 
@@ -171,45 +171,210 @@ module StandardAudit
171
171
  OpenSSL::Digest::SHA256.hexdigest(canonical)
172
172
  end
173
173
 
174
- # Verifies the integrity of the audit log chain. Returns a result hash with
175
- # :valid (boolean), :verified (count), and :failures (array of hashes).
174
+ # The checksum of the most recent row the node a new row links to.
175
+ def self.chain_tip_checksum
176
+ order(created_at: :desc, id: :desc).limit(1).pick(:checksum)
177
+ end
178
+
179
+ # True when the table carries the `previous_checksum` column, i.e. the host
180
+ # has run the `standard_audit:add_previous_checksum` migration. Rows written
181
+ # without it are verified by position and parent recovery instead, so the
182
+ # gem keeps working unmigrated.
183
+ def self.chain_parent_column?
184
+ column_names.include?("previous_checksum")
185
+ end
186
+
187
+ # Verifies the integrity of the audit log. Returns a result hash with
188
+ # :valid (boolean), :verified (count), :recovered (count) and :failures
189
+ # (array of hashes carrying :id, :event_type, :created_at, :expected,
190
+ # :actual and :reason).
176
191
  #
177
192
  # Records are processed in (created_at, id) order. Records without a
178
- # checksum (pre-feature data) reset the chain — the next checksummed
179
- # record starts a new independent chain segment.
180
- def self.verify_chain(scope: nil, batch_size: 1000)
193
+ # checksum (pre-feature data) reset the walk — the next checksummed record
194
+ # starts a new independent segment.
195
+ #
196
+ # The log is a DAG, not a strict line. Concurrent writers link to whichever
197
+ # node was the tip when they read it, so two rows can share a parent and
198
+ # the sequence forks. Every row is still verified against the exact digest
199
+ # it signed, in one of two ways:
200
+ #
201
+ # * `previous_checksum` present (written by 0.8+ on a migrated host): the
202
+ # parent the writer asserts. It is covered by this row's own digest, so
203
+ # it cannot be edited to excuse a tampered row.
204
+ # * `previous_checksum` NULL (pre-0.8 rows, or an unmigrated host): the
205
+ # preceding row in the walk, and — unless `strict:` — a search back
206
+ # through the last `recovery_window` digests (and finally "no parent at
207
+ # all", for a row written against an empty table) for the one that
208
+ # actually reproduces this row's checksum. That recovers the true parent
209
+ # of a forked row without re-signing anything. It does not weaken tamper
210
+ # detection: a row whose fields were altered reproduces no candidate's
211
+ # digest, so it still fails.
212
+ #
213
+ # A row whose parent digest is absent from the log is reported with
214
+ # `reason: :missing_parent` — a row was removed. Two exemptions:
215
+ #
216
+ # * `scope:` — the log is global, so a scoped row's parent usually
217
+ # belongs to another scope and is absent for an innocent reason.
218
+ # * a pruned start. If the walk *opens* on a row whose parent is already
219
+ # gone, the log has had its start removed (retention cleanup). That
220
+ # parent digest is then exempt wherever else it appears, because a
221
+ # pruned row can have several children — which is exactly what a
222
+ # concurrent append leaves behind. Removing a row from the middle is
223
+ # still reported: its digest is not the one the walk opened on.
224
+ #
225
+ # `standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
226
+ # `created_at`. Rows whose two timestamps disagree (a backdated
227
+ # `occurred_at`) can leave a hole rather than a prefix, and a hole is
228
+ # reported — truthfully, since rows really are missing.
229
+ def self.verify_chain(scope: nil, batch_size: 1000, recovery_window: 256, strict: false)
181
230
  relation = scope ? where(scope_gid: scope.to_global_id.to_s) : all
231
+ check_parents = scope.nil?
232
+ declared_parents = chain_parent_column?
182
233
 
183
234
  previous_checksum = nil
184
235
  verified = 0
236
+ recovered = 0
185
237
  failures = []
238
+ window = []
239
+ first_row = true
240
+ pruned_parents = []
241
+
242
+ each_in_chain_order(relation, batch_size: batch_size) do |record|
243
+ if record.checksum.blank?
244
+ previous_checksum = nil
245
+ window.clear
246
+ next
247
+ end
186
248
 
187
- relation.in_batches(of: batch_size) do |batch|
188
- batch.order(created_at: :asc, id: :asc).each do |record|
189
- if record.checksum.blank?
190
- previous_checksum = nil
191
- next
192
- end
249
+ verified += 1
250
+ declared = record.previous_checksum if declared_parents
193
251
 
194
- expected = record.compute_checksum_value(previous_checksum: previous_checksum)
252
+ if declared.present?
253
+ expected = record.compute_checksum_value(previous_checksum: declared)
195
254
 
196
255
  if record.checksum != expected
197
- failures << {
198
- id: record.id,
199
- event_type: record.event_type,
200
- created_at: record.created_at,
201
- expected: expected,
202
- actual: record.checksum
203
- }
256
+ failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
257
+ elsif check_parents && !parent_present?(declared, window, relation)
258
+ if first_row
259
+ # The walk opens on a row whose parent is already gone, so the
260
+ # log has had its start removed — retention pruning, typically.
261
+ # That parent is unknowable, and it can have several children
262
+ # (which is what a concurrent append leaves behind), so the
263
+ # exemption is remembered per digest rather than for one row.
264
+ pruned_parents << declared
265
+ elsif !pruned_parents.include?(declared)
266
+ failures << chain_failure(record, expected: expected, reason: :missing_parent)
267
+ end
204
268
  end
269
+ else
270
+ expected = record.compute_checksum_value(previous_checksum: previous_checksum)
205
271
 
206
- verified += 1
207
- previous_checksum = record.checksum
272
+ if record.checksum == expected
273
+ # Links to the row before it, as a linear chain does.
274
+ elsif !strict && recover_parent(record, window)
275
+ recovered += 1
276
+ else
277
+ failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
278
+ end
208
279
  end
280
+
281
+ previous_checksum = record.checksum
282
+ first_row = false
283
+ window << record.checksum
284
+ window.shift if window.size > recovery_window
285
+ end
286
+
287
+ { valid: failures.empty?, verified: verified, recovered: recovered, failures: failures }
288
+ end
289
+
290
+ # Records, for every row that does not already carry one, the parent digest
291
+ # it was actually signed against — recovering it by search where a
292
+ # concurrent append forked the chain. Returns
293
+ # `{ relinked:, unresolved:, skipped: }`.
294
+ #
295
+ # This NEVER rewrites a `checksum`. It writes only the previously-empty
296
+ # `previous_checksum` column, so it adds no attestation the rows did not
297
+ # already carry: a parent is recorded only when it reproduces the digest
298
+ # the row has held since it was written. Rows whose parent cannot be
299
+ # reproduced are left untouched and counted in :unresolved — they are the
300
+ # rows verification should keep reporting.
301
+ def self.relink_checksums!(batch_size: 1000, recovery_window: 256)
302
+ return { relinked: 0, unresolved: 0, skipped: 0 } unless chain_parent_column?
303
+
304
+ previous_checksum = nil
305
+ relinked = 0
306
+ unresolved = 0
307
+ skipped = 0
308
+ window = []
309
+
310
+ each_in_chain_order(all, batch_size: batch_size) do |record|
311
+ if record.checksum.blank?
312
+ previous_checksum = nil
313
+ window.clear
314
+ next
315
+ end
316
+
317
+ found = resolve_parent(record, previous_checksum, window) if record.previous_checksum.blank?
318
+
319
+ if record.previous_checksum.present?
320
+ skipped += 1
321
+ elsif found.nil?
322
+ unresolved += 1
323
+ elsif found.first.present?
324
+ record.update_columns(previous_checksum: found.first)
325
+ relinked += 1
326
+ else
327
+ # A genuine segment root. NULL already says so; nothing to write.
328
+ skipped += 1
329
+ end
330
+
331
+ previous_checksum = record.checksum
332
+ window << record.checksum
333
+ window.shift if window.size > recovery_window
334
+ end
335
+
336
+ { relinked: relinked, unresolved: unresolved, skipped: skipped }
337
+ end
338
+
339
+ def self.chain_failure(record, expected:, reason:)
340
+ {
341
+ id: record.id,
342
+ event_type: record.event_type,
343
+ created_at: record.created_at,
344
+ expected: expected,
345
+ actual: record.checksum,
346
+ reason: reason
347
+ }
348
+ end
349
+ private_class_method :chain_failure
350
+
351
+ # Searches `window` (most recent first, then "no parent at all") for the
352
+ # digest that reproduces the record's stored checksum. Returns a one-element
353
+ # array holding the parent — which may itself be nil, for a row written
354
+ # against an empty table — or nil when nothing reproduces the digest.
355
+ #
356
+ # SHA-256 preimage resistance is what makes this safe: a row whose fields
357
+ # were altered reproduces no candidate's digest, so it is still reported.
358
+ def self.recover_parent(record, window)
359
+ window.reverse_each do |candidate|
360
+ return [candidate] if record.checksum == record.compute_checksum_value(previous_checksum: candidate)
209
361
  end
210
362
 
211
- { valid: failures.empty?, verified: verified, failures: failures }
363
+ [nil] if record.checksum == record.compute_checksum_value(previous_checksum: nil)
212
364
  end
365
+ private_class_method :recover_parent
366
+
367
+ def self.resolve_parent(record, previous_checksum, window)
368
+ return [previous_checksum] if record.checksum == record.compute_checksum_value(previous_checksum: previous_checksum)
369
+
370
+ recover_parent(record, window)
371
+ end
372
+ private_class_method :resolve_parent
373
+
374
+ def self.parent_present?(digest, window, relation)
375
+ window.include?(digest) || relation.exists?(checksum: digest)
376
+ end
377
+ private_class_method :parent_present?
213
378
 
214
379
  # Backfills checksums for records that don't have them (e.g. pre-existing
215
380
  # records before the checksum feature was added).
@@ -217,27 +382,67 @@ module StandardAudit
217
382
  previous_checksum = nil
218
383
  count = 0
219
384
 
220
- in_batches(of: batch_size) do |batch|
221
- batch.order(created_at: :asc, id: :asc).each do |record|
222
- if record.checksum.present?
223
- previous_checksum = record.checksum
224
- next
225
- end
385
+ each_in_chain_order(all, batch_size: batch_size) do |record|
386
+ if record.checksum.present?
387
+ previous_checksum = record.checksum
388
+ next
389
+ end
226
390
 
227
- new_checksum = compute_checksum_value(
228
- record.attributes.slice(*CHECKSUM_FIELDS),
229
- previous_checksum: previous_checksum
230
- )
231
- record.update_columns(checksum: new_checksum)
391
+ new_checksum = compute_checksum_value(
392
+ record.attributes.slice(*CHECKSUM_FIELDS),
393
+ previous_checksum: previous_checksum
394
+ )
395
+ columns = { checksum: new_checksum }
396
+ columns[:previous_checksum] = previous_checksum if chain_parent_column?
397
+ record.update_columns(columns)
232
398
 
233
- previous_checksum = new_checksum
234
- count += 1
235
- end
399
+ previous_checksum = new_checksum
400
+ count += 1
236
401
  end
237
402
 
238
403
  count
239
404
  end
240
405
 
406
+ # Yields every record of `relation` in true (created_at, id) order, loading
407
+ # at most `batch_size` rows at a time via a keyset cursor.
408
+ #
409
+ # `in_batches` cannot do this, and used to be used here: it paginates by
410
+ # PRIMARY KEY range and applies any ordering only *within* each batch, so
411
+ # the global sequence it yields is a concatenation of id-ranges, each
412
+ # internally time-sorted. With the UUIDv7 ids `assign_uuid` generates that
413
+ # usually coincides with insertion order, which is why it went unnoticed —
414
+ # but it is wrong for any host that assigns ids differently, backfills rows
415
+ # with an explicit created_at, or has clock skew between writers. The chain
416
+ # is an ordering claim, so a verifier that walks a different order than the
417
+ # one it documents cannot be trusted to prove or disprove anything about it.
418
+ #
419
+ # The cursor predicate assumes created_at is NOT NULL, which the install
420
+ # migration's `t.timestamps` guarantees. A NULL created_at would compare as
421
+ # NULL and end the walk early rather than loop forever.
422
+ def self.each_in_chain_order(relation, batch_size: 1000)
423
+ cursor = nil
424
+
425
+ loop do
426
+ page = relation.reorder(created_at: :asc, id: :asc).limit(batch_size)
427
+
428
+ if cursor
429
+ page = page.where(
430
+ "created_at > :created_at OR (created_at = :created_at AND id > :id)",
431
+ created_at: cursor.first, id: cursor.last
432
+ )
433
+ end
434
+
435
+ records = page.to_a
436
+ break if records.empty?
437
+
438
+ records.each { |record| yield record }
439
+ break if records.size < batch_size
440
+
441
+ cursor = [records.last.created_at, records.last.id]
442
+ end
443
+ end
444
+ private_class_method :each_in_chain_order
445
+
241
446
  private
242
447
 
243
448
  def emit_created_event
@@ -252,12 +457,25 @@ module StandardAudit
252
457
  Rails.logger.warn("[StandardAudit] Failed to emit event: #{e.class}: #{e.message}")
253
458
  end
254
459
 
255
- # Fetches the most recent record's checksum and chains the new record to it.
256
- # Note: concurrent inserts can read the same "previous" record, forking
257
- # the chain. Use database-level advisory locks if you need serializable
258
- # chain integrity under concurrent writes.
460
+ # Links the new record to whichever row was the chain tip when this writer
461
+ # read it, and RECORDS WHICH ONE THAT WAS.
462
+ #
463
+ # The tip read is deliberately unlocked. Two concurrent transactions cannot
464
+ # see each other's uncommitted rows, so both may read the same tip and the
465
+ # sequence forks — that is not an error, it is what a multi-process writer
466
+ # does, and it is why 67% of one production log failed verification while
467
+ # the rows themselves were untampered (fundbright/delivery-ops#433). The
468
+ # alternative is a lock held until the *enclosing business transaction*
469
+ # commits (a row is invisible to other writers until then, so releasing
470
+ # earlier reopens the race), which would serialise every audited request in
471
+ # the estate behind one mutex. Recording the parent instead makes the fork
472
+ # verifiable rather than preventing it.
473
+ #
474
+ # `previous_checksum` needs no protection of its own: it is an input to
475
+ # this row's own digest, so editing it invalidates the row.
259
476
  def compute_checksum
260
- previous = self.class.order(created_at: :desc, id: :desc).limit(1).pick(:checksum)
477
+ previous = self.class.chain_tip_checksum
478
+ self.previous_checksum = previous if self.class.chain_parent_column?
261
479
  self.checksum = compute_checksum_value(previous_checksum: previous)
262
480
  end
263
481