standard_audit 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1ab5fcec99d13f0796742d817c1dd42e45c0e8518f8967033a67194c3179ae6e
-  data.tar.gz: b0f6272a600a665615cc533b439ad2b654d54c025bd7bc2b378b5c2eac7e850d
+  metadata.gz: 55028cd60be7b29a81c8bd0eccb5cbccadfba4f7691f0d6576214de4367e53c9
+  data.tar.gz: efcf7f59f891570b8076e870f64d632de6d707ed9a4aa1f4b6e35aa7116199d1
 SHA512:
-  metadata.gz: 1ab1f514a13f1b03501cfdf18b3c2afcf8ff214a36992edb3e00f052384826c56bf09a35c4d7c4b8f92a4f28513cd8fe5534d946622112bb27bb6f8bfd8826c7
-  data.tar.gz: 7f9fc2acdb292b06f72b2323ac0c5c780348da8d1b91e9850ae73a61eba2361fb21b3aa431888de9d0003dc2934c0ba1b2a9a060bd4bd3b2a6a1bfb71eea5680
+  metadata.gz: d1d0c3c3807faf64dc9e21c897da2e3317a6fee1ea202aae6b07a948f416bbb8d35c03f8977a3def8bdf5a2bca9944941e3fe1b3c860d7b2b3aa942a8f048e16
+  data.tar.gz: 16278c66d30fe401947d8748e6b675a9dd5bc49dac1d0183635faf27df8e1b9e5a5f2d933aa8a606c1b66a01894dc030088a3813b7ca8eb2aee68ed7bd811e0d

data/CHANGELOG.md

@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-07-30
+
+### Added
+
+- **`previous_checksum` — every row now records the row it was appended to.** New nullable column; run `rails generate standard_audit:add_previous_checksum && rails db:migrate` to add it. The install migration includes it for new hosts, along with a composite `(created_at, id)` index for the verification walk.
+- **`rake standard_audit:relink_checksums` / `AuditLog.relink_checksums!`** — records, for every row that has no `previous_checksum`, the parent digest it was *actually* signed against, recovering it by search where a concurrent append forked the log. It **never rewrites a `checksum`**: a parent is written only when it reproduces the digest the row has held since it was written, so it adds no attestation the rows did not already carry. Rows it cannot resolve are counted as `unresolved`, left untouched, and keep failing verification — which is the point. This is the repair path for a log broken by the defect below; `backfill_checksums!` is **not** (it re-signs rows from their current contents, so a green result attests only that a script ran).
+- `AuditLog.chain_tip_checksum` and `AuditLog.chain_parent_column?`.
+
+### Fixed
+
+- **The chained checksum was not concurrency-safe: `verify_chain` failed for 3,762 of 5,577 rows (67%) on one production log, continuously, while every row was untampered.** `compute_checksum` reads the chain tip with an unsynchronised query from `before_create`. Two transactions cannot see each other's uncommitted rows, so both sign against the same predecessor and the sequence forks; verification, which assumed a strict line, reported all but one of them as tampering. Any host with concurrent audit writes — two Puma threads, a web request and a job — has the same broken log. (`fundbright/delivery-ops#433`)
+
+  **The log is now treated as a DAG rather than a strict line, and each row records its parent.** A row is verified against the exact digest it signed, whether or not that is its predecessor in the walk, so a fork is verifiable instead of fatal. The tip read stays unlocked deliberately: forcing a strict line means holding a lock until the *enclosing business transaction* commits — a row is invisible to other writers until then, so releasing earlier reopens the race — which would serialise every audited request across the estate behind one mutex, on the hot path of every request. `previous_checksum` needs no protection of its own; it is an input to the row's own digest, so editing it invalidates the row.
+
+  What this deliberately stops claiming: a single total order, and detection of an *inserted* row. A concurrent append and an inserted row are indistinguishable, so a design that tolerates the first cannot detect the second. The strict-line reading did not detect insertions either — it reported every concurrent append as tampering, which on the measured log meant 67% red and any real signal lost in it.
+
+  **Rows written before this version verify unchanged, with no migration.** With no `previous_checksum`, verification falls back to the preceding row in the walk and then searches back through the last `recovery_window` (default 256) digests for the one that reproduces the row's checksum — recovering the true parent of a forked row without re-signing anything, and reporting how many needed it in the new `recovered:` count. Tamper detection is not weakened: a row whose fields were altered reproduces no candidate's digest. `verify_chain(strict: true)` skips the search and reports every fork as a failure. A host that never runs the migration keeps working on this path indefinitely.
+
+- **`verify_chain` never noticed a row being removed from the middle of the log.** Failures now carry `reason:` — `:digest_mismatch`, or `:missing_parent` when the row a record was appended to is no longer present. Retention pruning does not trip it: if the walk opens on a row whose parent is already gone, that parent digest is exempt wherever else it appears — a pruned row can have several children, which is what a concurrent append leaves behind. A removal from the *middle* is still reported. Every row is exempt under `scope:`, because the log is global and a scoped row's parent usually belongs to another scope.
+
+- **`verify_chain` did not walk the order it documents, and neither did `backfill_checksums!`.** Both claimed a `(created_at, id)` walk and both used `in_batches`, which paginates by **primary key** range and applies the sort only *within* each batch — so the global sequence was a concatenation of id-ranges, each internally time-sorted. Both now walk with a keyset cursor ordered by `(created_at, id)`, which is the documented order and still loads only `batch_size` rows at a time.
+
+  With the UUIDv7 ids `assign_uuid` generates, id order usually coincides with insertion order, which is why this stayed latent — but it is wrong for any host that assigns ids differently, backfills rows with an explicit `created_at`, or has clock skew between writers. It matters more than a latent ordering bug normally would: the chain is an *ordering* claim, so a verifier that walks a different order than the one it documents cannot be trusted to prove or disprove anything about chain integrity, including the concurrent-append defect above. `backfill_checksums!` had the same defect, and there it silently *writes* the wrong chain rather than misreading one.
+
 ## [0.7.0] - 2026-07-30
 
 ### Added

data/README.md

@@ -406,9 +406,92 @@ which is **still HTTP 200** — it surfaces the advisory in the readiness JSON
 without failing the probe or blocking a deploy. The check is duck-typed and has
 no hard dependency on `standard_health`.
 
+## Chain Integrity
+
+Every row carries a SHA-256 `checksum` over its own fields **and the digest of
+the row it was appended to**, so editing a row in place invalidates it. Each row
+also records that parent explicitly in `previous_checksum`.
+
+### The log is a DAG, not a strict line
+
+A writer reads the current tip and links to it. That read is deliberately
+unlocked, so two concurrent transactions — two Puma threads, a web request and a
+job — can read the same tip and the sequence forks. **That is normal for a
+multi-process writer, and it is not an integrity failure.** Forcing a strict
+line would mean holding a lock until the enclosing business transaction commits
+(a row is invisible to other writers until then, so releasing earlier reopens
+the race), which serialises every audited request behind one mutex.
+
+Storing the parent is what keeps a forked log verifiable: every row is checked
+against the exact digest it signed, whether or not it is the walk's predecessor.
+`previous_checksum` needs no protection of its own — it is an input to the row's
+own digest, so editing it invalidates the row.
+
+```ruby
+result = StandardAudit::AuditLog.verify_chain
+# => { valid: true, verified: 5577, recovered: 0, failures: [] }
+```
+
+- `failures` carries `reason: :digest_mismatch` (the row's fields no longer
+  produce its digest) or `reason: :missing_parent` (the row it was appended to
+  is no longer in the log). Retention pruning does not trip `:missing_parent`:
+  if the walk opens on a row whose parent is already gone, that parent digest
+  is exempt wherever else it appears — a pruned row can have several children.
+  A removal from the *middle* is still reported. One caveat:
+  `standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
+  `created_at`, so rows whose two timestamps disagree can leave a hole rather
+  than a pruned start, and a hole is reported — truthfully, since rows really
+  are missing.
+- `recovered` counts rows with no `previous_checksum` whose parent had to be
+  found by searching back through recent digests — see below.
+- `verify_chain(scope: org)` skips the missing-parent check, because the log is
+  global and a scoped row's parent usually belongs to another scope.
+
+What this deliberately does **not** claim: that the log is in a single total
+order, or that no row was inserted. A concurrent append and an inserted row look
+alike, so a design that tolerates the first cannot detect the second. The
+previous strict-line reading did not actually detect insertions either — it
+reported every concurrent append as tampering, which on one production log meant
+67% of rows red and any real signal lost in the noise.
+
+### Rows written before 0.8.0
+
+They have no `previous_checksum`. Verification falls back to the preceding row
+in the walk and, failing that, searches back through the last `recovery_window`
+(default 256) digests for the one that reproduces the row's checksum — which
+recovers the true parent of a row that forked, **without re-signing anything**.
+This does not weaken tamper detection: a row whose fields were altered
+reproduces no candidate's digest. Pass `strict: true` to skip the search and see
+every fork as a failure.
+
+To make that permanent, add the column and record what each row was actually
+signed against:
+
+```bash
+rails generate standard_audit:add_previous_checksum
+rails db:migrate
+rake standard_audit:relink_checksums
+```
+
+`relink_checksums` never rewrites a `checksum`. It fills in the previously-empty
+`previous_checksum` only when that value reproduces the digest the row has held
+since it was written, so it adds no attestation the rows did not already carry.
+Rows it cannot resolve are reported as `unresolved` and keep failing
+verification — which is the point.
+
+**Do not run `backfill_checksums!` to make a red `verify_chain` go green.** It
+re-signs rows from their current contents, so it attests only that a script ran.
+It is for rows that never had a checksum at all (pre-feature data).
+
 ## Rake Tasks
 
 ```bash
+# Verify chain integrity (exits non-zero on failures)
+rake standard_audit:verify
+
+# Record the parent digest each existing row was signed against
+rake standard_audit:relink_checksums
+
 # Delete logs older than N days (default: retention_days config or 90)
 rake standard_audit:cleanup[180]
 

data/app/models/standard_audit/audit_log.rb

@@ -171,45 +171,210 @@ module StandardAudit
       OpenSSL::Digest::SHA256.hexdigest(canonical)
     end
 
-    # Verifies the integrity of the audit log chain. Returns a result hash with
-    # :valid (boolean), :verified (count), and :failures (array of hashes).
+    # The checksum of the most recent row — the node a new row links to.
+    def self.chain_tip_checksum
+      order(created_at: :desc, id: :desc).limit(1).pick(:checksum)
+    end
+
+    # True when the table carries the `previous_checksum` column, i.e. the host
+    # has run the `standard_audit:add_previous_checksum` migration. Rows written
+    # without it are verified by position and parent recovery instead, so the
+    # gem keeps working unmigrated.
+    def self.chain_parent_column?
+      column_names.include?("previous_checksum")
+    end
+
+    # Verifies the integrity of the audit log. Returns a result hash with
+    # :valid (boolean), :verified (count), :recovered (count) and :failures
+    # (array of hashes carrying :id, :event_type, :created_at, :expected,
+    # :actual and :reason).
     #
     # Records are processed in (created_at, id) order. Records without a
-    # checksum (pre-feature data) reset the chain — the next checksummed
-    # record starts a new independent chain segment.
-    def self.verify_chain(scope: nil, batch_size: 1000)
+    # checksum (pre-feature data) reset the walk — the next checksummed record
+    # starts a new independent segment.
+    #
+    # The log is a DAG, not a strict line. Concurrent writers link to whichever
+    # node was the tip when they read it, so two rows can share a parent and
+    # the sequence forks. Every row is still verified against the exact digest
+    # it signed, in one of two ways:
+    #
+    #   * `previous_checksum` present (written by 0.8+ on a migrated host): the
+    #     parent the writer asserts. It is covered by this row's own digest, so
+    #     it cannot be edited to excuse a tampered row.
+    #   * `previous_checksum` NULL (pre-0.8 rows, or an unmigrated host): the
+    #     preceding row in the walk, and — unless `strict:` — a search back
+    #     through the last `recovery_window` digests (and finally "no parent at
+    #     all", for a row written against an empty table) for the one that
+    #     actually reproduces this row's checksum. That recovers the true parent
+    #     of a forked row without re-signing anything. It does not weaken tamper
+    #     detection: a row whose fields were altered reproduces no candidate's
+    #     digest, so it still fails.
+    #
+    # A row whose parent digest is absent from the log is reported with
+    # `reason: :missing_parent` — a row was removed. Two exemptions:
+    #
+    #   * `scope:` — the log is global, so a scoped row's parent usually
+    #     belongs to another scope and is absent for an innocent reason.
+    #   * a pruned start. If the walk *opens* on a row whose parent is already
+    #     gone, the log has had its start removed (retention cleanup). That
+    #     parent digest is then exempt wherever else it appears, because a
+    #     pruned row can have several children — which is exactly what a
+    #     concurrent append leaves behind. Removing a row from the middle is
+    #     still reported: its digest is not the one the walk opened on.
+    #
+    # `standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
+    # `created_at`. Rows whose two timestamps disagree (a backdated
+    # `occurred_at`) can leave a hole rather than a prefix, and a hole is
+    # reported — truthfully, since rows really are missing.
+    def self.verify_chain(scope: nil, batch_size: 1000, recovery_window: 256, strict: false)
       relation = scope ? where(scope_gid: scope.to_global_id.to_s) : all
+      check_parents = scope.nil?
+      declared_parents = chain_parent_column?
 
       previous_checksum = nil
       verified = 0
+      recovered = 0
       failures = []
+      window = []
+      first_row = true
+      pruned_parents = []
+
+      each_in_chain_order(relation, batch_size: batch_size) do |record|
+        if record.checksum.blank?
+          previous_checksum = nil
+          window.clear
+          next
+        end
 
-      relation.in_batches(of: batch_size) do |batch|
-        batch.order(created_at: :asc, id: :asc).each do |record|
-          if record.checksum.blank?
-            previous_checksum = nil
-            next
-          end
+        verified += 1
+        declared = record.previous_checksum if declared_parents
 
-          expected = record.compute_checksum_value(previous_checksum: previous_checksum)
+        if declared.present?
+          expected = record.compute_checksum_value(previous_checksum: declared)
 
           if record.checksum != expected
-            failures << {
-              id: record.id,
-              event_type: record.event_type,
-              created_at: record.created_at,
-              expected: expected,
-              actual: record.checksum
-            }
+            failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
+          elsif check_parents && !parent_present?(declared, window, relation)
+            if first_row
+              # The walk opens on a row whose parent is already gone, so the
+              # log has had its start removed — retention pruning, typically.
+              # That parent is unknowable, and it can have several children
+              # (which is what a concurrent append leaves behind), so the
+              # exemption is remembered per digest rather than for one row.
+              pruned_parents << declared
+            elsif !pruned_parents.include?(declared)
+              failures << chain_failure(record, expected: expected, reason: :missing_parent)
+            end
           end
+        else
+          expected = record.compute_checksum_value(previous_checksum: previous_checksum)
 
-          verified += 1
-          previous_checksum = record.checksum
+          if record.checksum == expected
+            # Links to the row before it, as a linear chain does.
+          elsif !strict && recover_parent(record, window)
+            recovered += 1
+          else
+            failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
+          end
         end
+
+        previous_checksum = record.checksum
+        first_row = false
+        window << record.checksum
+        window.shift if window.size > recovery_window
+      end
+
+      { valid: failures.empty?, verified: verified, recovered: recovered, failures: failures }
+    end
+
+    # Records, for every row that does not already carry one, the parent digest
+    # it was actually signed against — recovering it by search where a
+    # concurrent append forked the chain. Returns
+    # `{ relinked:, unresolved:, skipped: }`.
+    #
+    # This NEVER rewrites a `checksum`. It writes only the previously-empty
+    # `previous_checksum` column, so it adds no attestation the rows did not
+    # already carry: a parent is recorded only when it reproduces the digest
+    # the row has held since it was written. Rows whose parent cannot be
+    # reproduced are left untouched and counted in :unresolved — they are the
+    # rows verification should keep reporting.
+    def self.relink_checksums!(batch_size: 1000, recovery_window: 256)
+      return { relinked: 0, unresolved: 0, skipped: 0 } unless chain_parent_column?
+
+      previous_checksum = nil
+      relinked = 0
+      unresolved = 0
+      skipped = 0
+      window = []
+
+      each_in_chain_order(all, batch_size: batch_size) do |record|
+        if record.checksum.blank?
+          previous_checksum = nil
+          window.clear
+          next
+        end
+
+        found = resolve_parent(record, previous_checksum, window) if record.previous_checksum.blank?
+
+        if record.previous_checksum.present?
+          skipped += 1
+        elsif found.nil?
+          unresolved += 1
+        elsif found.first.present?
+          record.update_columns(previous_checksum: found.first)
+          relinked += 1
+        else
+          # A genuine segment root. NULL already says so; nothing to write.
+          skipped += 1
+        end
+
+        previous_checksum = record.checksum
+        window << record.checksum
+        window.shift if window.size > recovery_window
+      end
+
+      { relinked: relinked, unresolved: unresolved, skipped: skipped }
+    end
+
+    def self.chain_failure(record, expected:, reason:)
+      {
+        id: record.id,
+        event_type: record.event_type,
+        created_at: record.created_at,
+        expected: expected,
+        actual: record.checksum,
+        reason: reason
+      }
+    end
+    private_class_method :chain_failure
+
+    # Searches `window` (most recent first, then "no parent at all") for the
+    # digest that reproduces the record's stored checksum. Returns a one-element
+    # array holding the parent — which may itself be nil, for a row written
+    # against an empty table — or nil when nothing reproduces the digest.
+    #
+    # SHA-256 preimage resistance is what makes this safe: a row whose fields
+    # were altered reproduces no candidate's digest, so it is still reported.
+    def self.recover_parent(record, window)
+      window.reverse_each do |candidate|
+        return [candidate] if record.checksum == record.compute_checksum_value(previous_checksum: candidate)
       end
 
-      { valid: failures.empty?, verified: verified, failures: failures }
+      [nil] if record.checksum == record.compute_checksum_value(previous_checksum: nil)
     end
+    private_class_method :recover_parent
+
+    def self.resolve_parent(record, previous_checksum, window)
+      return [previous_checksum] if record.checksum == record.compute_checksum_value(previous_checksum: previous_checksum)
+
+      recover_parent(record, window)
+    end
+    private_class_method :resolve_parent
+
+    def self.parent_present?(digest, window, relation)
+      window.include?(digest) || relation.exists?(checksum: digest)
+    end
+    private_class_method :parent_present?
 
     # Backfills checksums for records that don't have them (e.g. pre-existing
     # records before the checksum feature was added).
@@ -217,27 +382,67 @@ module StandardAudit
       previous_checksum = nil
       count = 0
 
-      in_batches(of: batch_size) do |batch|
-        batch.order(created_at: :asc, id: :asc).each do |record|
-          if record.checksum.present?
-            previous_checksum = record.checksum
-            next
-          end
+      each_in_chain_order(all, batch_size: batch_size) do |record|
+        if record.checksum.present?
+          previous_checksum = record.checksum
+          next
+        end
 
-          new_checksum = compute_checksum_value(
-            record.attributes.slice(*CHECKSUM_FIELDS),
-            previous_checksum: previous_checksum
-          )
-          record.update_columns(checksum: new_checksum)
+        new_checksum = compute_checksum_value(
+          record.attributes.slice(*CHECKSUM_FIELDS),
+          previous_checksum: previous_checksum
+        )
+        columns = { checksum: new_checksum }
+        columns[:previous_checksum] = previous_checksum if chain_parent_column?
+        record.update_columns(columns)
 
-          previous_checksum = new_checksum
-          count += 1
-        end
+        previous_checksum = new_checksum
+        count += 1
       end
 
       count
     end
 
+    # Yields every record of `relation` in true (created_at, id) order, loading
+    # at most `batch_size` rows at a time via a keyset cursor.
+    #
+    # `in_batches` cannot do this, and used to be used here: it paginates by
+    # PRIMARY KEY range and applies any ordering only *within* each batch, so
+    # the global sequence it yields is a concatenation of id-ranges, each
+    # internally time-sorted. With the UUIDv7 ids `assign_uuid` generates that
+    # usually coincides with insertion order, which is why it went unnoticed —
+    # but it is wrong for any host that assigns ids differently, backfills rows
+    # with an explicit created_at, or has clock skew between writers. The chain
+    # is an ordering claim, so a verifier that walks a different order than the
+    # one it documents cannot be trusted to prove or disprove anything about it.
+    #
+    # The cursor predicate assumes created_at is NOT NULL, which the install
+    # migration's `t.timestamps` guarantees. A NULL created_at would compare as
+    # NULL and end the walk early rather than loop forever.
+    def self.each_in_chain_order(relation, batch_size: 1000)
+      cursor = nil
+
+      loop do
+        page = relation.reorder(created_at: :asc, id: :asc).limit(batch_size)
+
+        if cursor
+          page = page.where(
+            "created_at > :created_at OR (created_at = :created_at AND id > :id)",
+            created_at: cursor.first, id: cursor.last
+          )
+        end
+
+        records = page.to_a
+        break if records.empty?
+
+        records.each { |record| yield record }
+        break if records.size < batch_size
+
+        cursor = [records.last.created_at, records.last.id]
+      end
+    end
+    private_class_method :each_in_chain_order
+
     private
 
     def emit_created_event
@@ -252,12 +457,25 @@ module StandardAudit
       Rails.logger.warn("[StandardAudit] Failed to emit event: #{e.class}: #{e.message}")
     end
 
-    # Fetches the most recent record's checksum and chains the new record to it.
-    # Note: concurrent inserts can read the same "previous" record, forking
-    # the chain. Use database-level advisory locks if you need serializable
-    # chain integrity under concurrent writes.
+    # Links the new record to whichever row was the chain tip when this writer
+    # read it, and RECORDS WHICH ONE THAT WAS.
+    #
+    # The tip read is deliberately unlocked. Two concurrent transactions cannot
+    # see each other's uncommitted rows, so both may read the same tip and the
+    # sequence forks — that is not an error, it is what a multi-process writer
+    # does, and it is why 67% of one production log failed verification while
+    # the rows themselves were untampered (fundbright/delivery-ops#433). The
+    # alternative is a lock held until the *enclosing business transaction*
+    # commits (a row is invisible to other writers until then, so releasing
+    # earlier reopens the race), which would serialise every audited request in
+    # the estate behind one mutex. Recording the parent instead makes the fork
+    # verifiable rather than preventing it.
+    #
+    # `previous_checksum` needs no protection of its own: it is an input to
+    # this row's own digest, so editing it invalidates the row.
     def compute_checksum
-      previous = self.class.order(created_at: :desc, id: :desc).limit(1).pick(:checksum)
+      previous = self.class.chain_tip_checksum
+      self.previous_checksum = previous if self.class.chain_parent_column?
       self.checksum = compute_checksum_value(previous_checksum: previous)
     end
 

data/lib/generators/standard_audit/add_previous_checksum/add_previous_checksum_generator.rb

@@ -0,0 +1,17 @@
+module StandardAudit
+  module Generators
+    class AddPreviousChecksumGenerator < Rails::Generators::Base
+      include Rails::Generators::Migration
+      source_root File.expand_path("templates", __dir__)
+
+      def self.next_migration_number(dirname)
+        Time.now.utc.strftime("%Y%m%d%H%M%S")
+      end
+
+      def copy_migration
+        migration_template "add_previous_checksum_to_audit_logs.rb.erb",
+          "db/migrate/add_previous_checksum_to_audit_logs.rb"
+      end
+    end
+  end
+end

data/lib/generators/standard_audit/add_previous_checksum/templates/add_previous_checksum_to_audit_logs.rb.erb

@@ -0,0 +1,23 @@
+class AddPreviousChecksumToAuditLogs < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
+  def change
+    # On PostgreSQL with a large audit_logs table, add
+    # `disable_ddl_transaction!` above and pass `algorithm: :concurrently` to
+    # the add_index calls — StrongMigrations will ask for it.
+    #
+    # Nullable and index-only: no data is rewritten. Existing rows keep a NULL
+    # parent and are verified by position and parent recovery, exactly as they
+    # are today. Run `rake standard_audit:relink_checksums` afterwards to
+    # record the parent each existing row was actually signed against.
+    add_column :audit_logs, :previous_checksum, :string, limit: 64
+    # Not used by the gem itself: verification reads previous_checksum off rows
+    # it has already loaded. It is here for forensics — "what forked off this
+    # row?" — which is the question you ask once verification flags something.
+    add_index :audit_logs, :previous_checksum
+    # verify_chain / backfill_checksums! walk the table with a keyset cursor
+    # ordered by (created_at, id).
+    add_index :audit_logs, [:created_at, :id] unless index_exists?(:audit_logs, [:created_at, :id])
+    # verify_chain looks a declared parent up by digest when it falls outside
+    # the in-memory recovery window — the path that reports a removed row.
+    add_index :audit_logs, :checksum unless index_exists?(:audit_logs, :checksum)
+  end
+end

data/lib/generators/standard_audit/install/templates/create_audit_logs.rb.erb

@@ -17,6 +17,10 @@ class CreateAuditLogs < ActiveRecord::Migration[<%= ActiveRecord::Migration.curr
       t.jsonb :metadata, default: {}
       t.datetime :occurred_at, null: false
       t.string :checksum, limit: 64
+      # The digest of the row this one was chained to. Concurrent writers can
+      # share a parent, so the log is a DAG rather than a strict line; storing
+      # the parent is what keeps every row verifiable when it forks.
+      t.string :previous_checksum, limit: 64
       t.timestamps
     end
 
@@ -27,6 +31,17 @@ class CreateAuditLogs < ActiveRecord::Migration[<%= ActiveRecord::Migration.curr
     add_index :audit_logs, :request_id
     add_index :audit_logs, [:occurred_at, :created_at]
     add_index :audit_logs, :created_at
+    # verify_chain / backfill_checksums! walk the table with a keyset cursor
+    # ordered by (created_at, id); the composite index makes each page an index
+    # range scan rather than a scan plus filter.
+    add_index :audit_logs, [:created_at, :id]
+    # verify_chain looks a declared parent up by digest when it falls outside
+    # the in-memory recovery window — the path that reports a removed row.
+    add_index :audit_logs, :checksum
+    # Not used by the gem itself: verification reads previous_checksum off rows
+    # it has already loaded. It is here for forensics — "what forked off this
+    # row?" — which is the question you ask once verification flags something.
+    add_index :audit_logs, :previous_checksum
     add_index :audit_logs, :session_id
     # GIN index requires PostgreSQL; remove if using another database
     add_index :audit_logs, :metadata, using: :gin

data/lib/standard_audit/version.rb

@@ -1,3 +1,3 @@
 module StandardAudit
-  VERSION = "0.7.0"
+  VERSION = "0.8.0"
 end

data/lib/standard_audit.rb

@@ -153,10 +153,8 @@ module StandardAudit
 
     def flush_batch(buffer)
       now = Time.current
-      previous_checksum = StandardAudit::AuditLog
-        .order(created_at: :desc, id: :desc)
-        .limit(1)
-        .pick(:checksum)
+      records_parent = StandardAudit::AuditLog.chain_parent_column?
+      previous_checksum = StandardAudit::AuditLog.chain_tip_checksum
 
       # Generate sorted UUIDs to ensure batch ordering matches id ordering.
       # UUIDv7 within the same millisecond can have non-monotonic lower bits;
@@ -175,6 +173,7 @@ module StandardAudit
           row.stringify_keys,
           previous_checksum: previous_checksum
         )
+        row[:previous_checksum] = previous_checksum if records_parent
         row[:checksum] = checksum
         previous_checksum = checksum
         row

data/lib/tasks/standard_audit_tasks.rake

@@ -63,16 +63,34 @@ namespace :standard_audit do
     puts "============================="
     puts "Records verified: #{result[:verified]}"
     puts "Chain valid: #{result[:valid]}"
+    puts "Forked links recovered: #{result[:recovered]}"
 
     if result[:failures].any?
-      puts "\nTampered records detected: #{result[:failures].size}"
+      puts "\nUnverifiable records detected: #{result[:failures].size}"
       result[:failures].each do |failure|
-        puts "  #{failure[:id]} (#{failure[:event_type]}) at #{failure[:created_at]}"
+        puts "  #{failure[:id]} (#{failure[:event_type]}) at #{failure[:created_at]} — #{failure[:reason]}"
       end
       abort "Chain verification failed"
     end
   end
 
+  desc "Record the parent digest each existing row was signed against (never re-signs)"
+  task relink_checksums: :environment do
+    unless StandardAudit::AuditLog.chain_parent_column?
+      abort "audit_logs has no previous_checksum column — run `rails generate standard_audit:add_previous_checksum` first"
+    end
+
+    result = StandardAudit::AuditLog.relink_checksums!
+
+    puts "Relinked: #{result[:relinked]}"
+    puts "Left as-is (already linked, or a segment root): #{result[:skipped]}"
+    puts "Unresolved: #{result[:unresolved]}"
+    puts ""
+    puts "No checksum was rewritten. A parent is recorded only when it reproduces"
+    puts "the digest the row has held since it was written; unresolved rows keep"
+    puts "failing verification, which is the point."
+  end
+
   desc "Backfill checksums for records that don't have them"
   task backfill_checksums: :environment do
     count = StandardAudit::AuditLog.backfill_checksums!

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: standard_audit
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Jaryl Sim
@@ -99,6 +99,8 @@ files:
 - config/routes.rb
 - lib/generators/standard_audit/add_checksums/add_checksums_generator.rb
 - lib/generators/standard_audit/add_checksums/templates/add_checksum_to_audit_logs.rb.erb
+- lib/generators/standard_audit/add_previous_checksum/add_previous_checksum_generator.rb
+- lib/generators/standard_audit/add_previous_checksum/templates/add_previous_checksum_to_audit_logs.rb.erb
 - lib/generators/standard_audit/install/install_generator.rb
 - lib/generators/standard_audit/install/templates/create_audit_logs.rb.erb
 - lib/generators/standard_audit/install/templates/initializer.rb.erb