standard_audit 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 84f59bffd4df4a9174ba0a88645a4c7d5818eddf862b5badfb55c3880fb5849d
-  data.tar.gz: 7747fdff93658f34f3ba3a9bfa70ca40fdc581125d182c996319a161f5c30809
+  metadata.gz: 55028cd60be7b29a81c8bd0eccb5cbccadfba4f7691f0d6576214de4367e53c9
+  data.tar.gz: efcf7f59f891570b8076e870f64d632de6d707ed9a4aa1f4b6e35aa7116199d1
 SHA512:
-  metadata.gz: 03f2f84e0418106b8899b9aaff009e95fdbb116f9eb98c37308d2b4ae41df3cfbfe9d9b1615bdd4beb4391d983989cb658e4195451cb56214a75e4cf46ba4375
-  data.tar.gz: 6b2e84e5bcd6e6e933535a6ae9e654deb3502e69b575d613044c7b02de843e2a8886c43a0c82bd8cb6313ff012510d36a52f290ca36dd778634c4b3c70cefb80
+  metadata.gz: d1d0c3c3807faf64dc9e21c897da2e3317a6fee1ea202aae6b07a948f416bbb8d35c03f8977a3def8bdf5a2bca9944941e3fe1b3c860d7b2b3aa942a8f048e16
+  data.tar.gz: 16278c66d30fe401947d8748e6b675a9dd5bc49dac1d0183635faf27df8e1b9e5a5f2d933aa8a606c1b66a01894dc030088a3813b7ca8eb2aee68ed7bd811e0d

data/CHANGELOG.md

@@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-07-30
+
+### Added
+
+- **`previous_checksum` — every row now records the row it was appended to.** New nullable column; run `rails generate standard_audit:add_previous_checksum && rails db:migrate` to add it. The install migration includes it for new hosts, along with a composite `(created_at, id)` index for the verification walk.
+- **`rake standard_audit:relink_checksums` / `AuditLog.relink_checksums!`** — records, for every row that has no `previous_checksum`, the parent digest it was *actually* signed against, recovering it by search where a concurrent append forked the log. It **never rewrites a `checksum`**: a parent is written only when it reproduces the digest the row has held since it was written, so it adds no attestation the rows did not already carry. Rows it cannot resolve are counted as `unresolved`, left untouched, and keep failing verification — which is the point. This is the repair path for a log broken by the defect below; `backfill_checksums!` is **not** (it re-signs rows from their current contents, so a green result attests only that a script ran).
+- `AuditLog.chain_tip_checksum` and `AuditLog.chain_parent_column?`.
+
+### Fixed
+
+- **The chained checksum was not concurrency-safe: `verify_chain` failed for 3,762 of 5,577 rows (67%) on one production log, continuously, while every row was untampered.** `compute_checksum` reads the chain tip with an unsynchronised query from `before_create`. Two transactions cannot see each other's uncommitted rows, so both sign against the same predecessor and the sequence forks; verification, which assumed a strict line, reported all but one of them as tampering. Any host with concurrent audit writes — two Puma threads, a web request and a job — has the same broken log. (`fundbright/delivery-ops#433`)
+
+  **The log is now treated as a DAG rather than a strict line, and each row records its parent.** A row is verified against the exact digest it signed, whether or not that is its predecessor in the walk, so a fork is verifiable instead of fatal. The tip read stays unlocked deliberately: forcing a strict line means holding a lock until the *enclosing business transaction* commits — a row is invisible to other writers until then, so releasing earlier reopens the race — which would serialise every audited request across the estate behind one mutex, on the hot path of every request. `previous_checksum` needs no protection of its own; it is an input to the row's own digest, so editing it invalidates the row.
+
+  What this deliberately stops claiming: a single total order, and detection of an *inserted* row. A concurrent append and an inserted row are indistinguishable, so a design that tolerates the first cannot detect the second. The strict-line reading did not detect insertions either — it reported every concurrent append as tampering, which on the measured log meant 67% red and any real signal lost in it.
+
+  **Rows written before this version verify unchanged, with no migration.** With no `previous_checksum`, verification falls back to the preceding row in the walk and then searches back through the last `recovery_window` (default 256) digests for the one that reproduces the row's checksum — recovering the true parent of a forked row without re-signing anything, and reporting how many needed it in the new `recovered:` count. Tamper detection is not weakened: a row whose fields were altered reproduces no candidate's digest. `verify_chain(strict: true)` skips the search and reports every fork as a failure. A host that never runs the migration keeps working on this path indefinitely.
+
+- **`verify_chain` never noticed a row being removed from the middle of the log.** Failures now carry `reason:` — `:digest_mismatch`, or `:missing_parent` when the row a record was appended to is no longer present. Retention pruning does not trip it: if the walk opens on a row whose parent is already gone, that parent digest is exempt wherever else it appears — a pruned row can have several children, which is what a concurrent append leaves behind. A removal from the *middle* is still reported. Every row is exempt under `scope:`, because the log is global and a scoped row's parent usually belongs to another scope.
+
+- **`verify_chain` did not walk the order it documents, and neither did `backfill_checksums!`.** Both claimed a `(created_at, id)` walk and both used `in_batches`, which paginates by **primary key** range and applies the sort only *within* each batch — so the global sequence was a concatenation of id-ranges, each internally time-sorted. Both now walk with a keyset cursor ordered by `(created_at, id)`, which is the documented order and still loads only `batch_size` rows at a time.
+
+  With the UUIDv7 ids `assign_uuid` generates, id order usually coincides with insertion order, which is why this stayed latent — but it is wrong for any host that assigns ids differently, backfills rows with an explicit `created_at`, or has clock skew between writers. It matters more than a latent ordering bug normally would: the chain is an *ordering* claim, so a verifier that walks a different order than the one it documents cannot be trusted to prove or disprove anything about chain integrity, including the concurrent-append defect above. `backfill_checksums!` had the same defect, and there it silently *writes* the wrong chain rather than misreading one.
+
+## [0.7.0] - 2026-07-30
+
+### Added
+
+- **Batch actor/target/scope preloading.** `AuditLog#actor` / `#target` / `#scope` resolved their GlobalID one row at a time, so every audit list N+1'd. Two consuming apps had already fixed this locally, both by reaching into private ivars (`instance_variable_set(:@preloaded_actor, …)`) because the gem exposed no setter. The gem now owns it:
+  - `AuditLog.preload_references(logs, refs: %i[actor target], only: [...], includes: {...})` resolves a page in one query per distinct stored `*_type`, built on `GlobalID::Locator.locate_many` with `ignore_missing: true`.
+  - `preloaded_actor=` / `preloaded_target=` / `preloaded_scope=` public writers, plus `actor_preloaded?` predicates.
+  - The memo is a Hash consulted with `key?`, so *preloaded-but-deleted* memoizes `nil` and reads back without a query — distinct from *not preloaded*. `actor=` / `target=` / `scope=` populate the memo (they already hold the record); `reload` clears it.
+  - `only:` is matched against the **stored type string by class name**. `GlobalID::Locator`'s own `:only` is evaluated as `gid.model_class <= klass`, which constantizes the historical type string *before* deciding whether it was permitted — so a renamed or deleted class raises `NameError` rather than being denied. Consequence: `only:` does not expand to subclasses or modules; list every concrete class. A non-whitelisted reference memoizes `nil`.
+  - `includes:` is treated as a per-type map when every key is a String or Class (`{ "Order" => [:user] }`), and as a uniform Active Record includes spec otherwise (so `{ account: :identifiers }` still works as you'd expect).
+  - A type string that no longer constantizes — or a blank `*_type` on a row that still carries a `*_gid` — is left *unmemoized*, so the per-row reader behaves exactly as it did before preloading was attempted. Preloading can never turn a resolvable reference into a permanent `nil`.
+- `AuditLog#actor_model_id` / `#target_model_id` / `#scope_model_id` (and `AuditLog.reference_model_id(gid)`) — the model id extracted straight from the gid string, a helper three separate consumer call sites had hand-rolled. Deliberately **not** `GlobalID.parse(gid)&.model_id`: audit rows are historical and append-only, so a gid whose app segment differs from the current `GlobalID.app` is a real row that must still yield its id. Mirrors `URI::GID`'s own decoding (drops `?params`, `CGI.unescape`s each segment, returns an Array for a composite primary key) while ignoring the app name, so it agrees with `GlobalID#model_id` wherever that works and keeps working where it doesn't.
+
+
+- **`config.before_checksum`** — registers a hook that runs on `before_create` **between `assign_uuid` and `compute_checksum`**. Definition order is execution order, so a hook may set a `CHECKSUM_FIELDS` member (back-fill `scope`, derive a column, rewrite `metadata`) and the row still passes `AuditLog.verify_chain`. Hosts previously had to register their own `before_create ..., prepend: true` to beat the gem's checksum callback — fragile ordering knowledge no host should need.
+  ```ruby
+  config.before_checksum { |log| log.scope = MyApp.derive_scope(log) }
+  config.before_checksum :backfill_scope   # an AuditLog instance method
+  ```
+  Hooks accumulate and run in registration order. **Each is rescued individually** — a failing hook logs (and reports to `Rails.error`), is **rolled back to the attributes it started from**, and is skipped; the remaining hooks still run and the audit write never fails. The rollback is what makes "skipped" true: without it, a hook that assigns `scope_gid` and then fails a later lookup would leave a half-applied row for the following callbacks to checksum and persist. If a row is created with an explicit `checksum`, hooks still run (most derived columns are not checksummed) but the supplied digest is dropped and re-derived when a hook changed a `CHECKSUM_FIELDS` member. Hooks do not run on the batched `insert_all!` path, which never instantiates a model.
+- **`StandardAudit.configure(baseline: true)`** — remembers the block so `reset_configuration!` replays it onto the fresh Configuration. **Required companion to the `before_checksum` hooks, not optional.** The config object holds *behaviour*, not just data, so a suite that installs `standard_audit/rspec` (whose per-example `reset_configuration!` restores gem defaults) would silently lose its write-time hooks after the first example — and the specs that would have caught it pass vacuously. The install template now uses `configure(baseline: true)`, and the `lib/standard_audit/rspec.rb` docstring no longer instructs hosts to re-apply configuration by hand. `reset_configuration!(replay_baseline: false)`, `clear_baseline_configuration!` and `baseline_configured?` round it out. A plain `configure` is unchanged and registers nothing.
+
+
+- **`config.sensitive_key_patterns`** (Array of Regexp, always applied, default `[]`) — the supported way to catch a family of keys. Stripe's `client_secret` slipping past the exact-match `:secret` is solved by `/secret/i`.
+
+  **There is deliberately no substring matching mode**, and one should not be added. Against the current default key list, substring matching strips real audit content across the estate: `input_tokens` / `output_tokens` (live LLM cost accounting in luminality and sidekick), `token_digest` (rendered in luminality's staff audit UI), `password_reset_sent_at`, `authorization_endpoint`, even `onepassword`. Audit rows are append-only, so it cannot be undone — the content is simply never written from then on. Patterns are opt-in, per-app, and checkable in advance.
+- **`config.sensitive_key_exceptions`** (Array of exact names or Regexps, default `[]`) — never redacted, even when `sensitive_keys` or a pattern matches. Lets an app adopt `[/token/i]` while keeping `input_tokens` / `output_tokens`.
+- **`rake standard_audit:sensitive_keys:dry_run`** — read-only. Reports, per metadata key, what a candidate rule *would* have stripped from the rows you already have, what it would keep, and any nested matches that survive because `filter_nested_metadata` is off. Turns "is this rule safe for my app?" into a command rather than a guess, which matters precisely because the rows are append-only.
+  ```bash
+  bin/rails standard_audit:sensitive_keys:dry_run                 # current config
+  bin/rails "standard_audit:sensitive_keys:dry_run[secret]"       # candidate pattern
+  NESTED=1 bin/rails "standard_audit:sensitive_keys:dry_run[secret|token]"
+  ```
+  Backed by `StandardAudit::SensitiveKeysDryRun.call(...)`, which extracts keys **in Ruby** rather than with `jsonb_object_keys` so it stays backend-neutral.
+- `StandardAudit::MetadataFilter` — `MetadataFilter.call(metadata, config:)` filters metadata; `MetadataFilter.new.filter?(key)` answers the same question for a single key without writing a row. Matching stays **exact on the key name** (string/symbol insensitive), unchanged from 0.6.0. Anything hash-like is filtered (so `ActionController::Parameters` is redacted, not waved through because it isn't a `Hash`); `nil` passes; anything else raises `MetadataFilter::UnfilterableMetadataError` rather than writing unfiltered content to an append-only row.
+
+### Fixed
+
+- **Nested metadata was never redacted on either write path.** `metadata: { stripe: { client_secret: … } }` was written intact even under exact matching — a larger real leak than the top-level case. Opt in with `config.filter_nested_metadata = true` (default `false`, so 0.6.0 behaviour is preserved); redaction then descends into nested Hashes and Hashes inside Arrays. `RESERVED_METADATA_KEYS` (`_tags`, `_source`) are preserved *and their subtree is never descended into, at every depth* — they are gem-owned, and `event_subscriber.rb` sets them after building metadata.
+- **The two write paths applied different sensitive-key filters.** `StandardAudit.record` and `StandardAudit::Subscriber#extract_metadata` each carried an independent copy of the redaction logic, and they had already diverged: the subscriber copy did not subtract `RESERVED_METADATA_KEYS`, so an app that added `:_tags` or `:_source` to `sensitive_keys` had the reserved key preserved on the `record` path and stripped on the `ActiveSupport::Notifications` path. Both now call the single `StandardAudit::MetadataFilter`, and the divergence is resolved in favour of `record`'s behaviour — reserved keys can never be filtered on either path. Parity is driven from one shared example (`spec/support/shared_examples/metadata_filtering.rb`), so a future divergence fails the suite rather than shipping.
+
+### Notes
+
+- `StandardAudit.batch { ... }` flushes via `insert_all!` and never instantiates a model, so none of the above runs on the batched write path — the memo is a no-op there by construction.
+- Minor behaviour change on a single instance: `log.actor = user; user.destroy!; log.actor` now returns the (destroyed) in-memory record instead of `nil`, because the writer memoizes. Reading the row fresh (or calling `reload`) is unchanged and still returns `nil`. This matches how Active Record association writers behave.
+
 ## [0.6.0] - 2026-06-24
 
 ### Added

data/README.md

@@ -164,8 +164,13 @@ organisation.scoped_audit_logs  # all logs scoped to this organisation
 
 ## Configuration Reference
 
+Use `configure(baseline: true)` in your initializer. It remembers the block so
+`StandardAudit.reset_configuration!` replays it — required if your suite loads
+`standard_audit/rspec`, because the config object holds behaviour (`before_checksum`
+hooks) as well as data, and a per-example reset would otherwise drop it.
+
 ```ruby
-StandardAudit.configure do |config|
+StandardAudit.configure(baseline: true) do |config|
   # -- Subscriptions --
   # Subscribe to ActiveSupport::Notifications patterns.
   # Supports wildcards.
@@ -189,9 +194,30 @@ StandardAudit.configure do |config|
   config.current_session_id_resolver = -> { Current.session_id }
 
   # -- Sensitive Data --
-  # Keys automatically stripped from metadata.
+  # Keys automatically stripped from metadata. Matching is EXACT on the key
+  # name; there is deliberately no substring mode (see below).
   config.sensitive_keys += %i[my_custom_secret]  # added to built-in defaults
 
+  # Regexps matched against every key name, in addition to the exact list.
+  # Solves e.g. Stripe's `client_secret`, which does not equal `:secret`.
+  config.sensitive_key_patterns = [/secret/i]
+
+  # Keys (exact names or Regexps) that are never redacted, so a broad pattern
+  # can keep the real audit keys it would otherwise swallow.
+  config.sensitive_key_exceptions = %i[input_tokens output_tokens]
+
+  # Descend into nested Hashes when redacting. OFF by default — without it,
+  # `metadata: { stripe: { client_secret: ... } }` is written intact.
+  config.filter_nested_metadata = true
+
+  # -- Write-time hooks --
+  # Run between the UUID assignment and the checksum computation, so a hook MAY
+  # set a checksummed column and the row still passes `verify_chain`. No
+  # `prepend: true` needed. Each hook is rescued individually and can never fail
+  # the audit write. Not run on the batched `insert_all!` path.
+  config.before_checksum { |log| log.scope = MyApp.derive_scope(log) }
+  config.before_checksum :backfill_scope   # an AuditLog instance method
+
   # -- Metadata Builder --
   # Optional proc to transform metadata before storage.
   config.metadata_builder = ->(metadata) { metadata.slice(:relevant_key) }
@@ -380,9 +406,92 @@ which is **still HTTP 200** — it surfaces the advisory in the readiness JSON
 without failing the probe or blocking a deploy. The check is duck-typed and has
 no hard dependency on `standard_health`.
 
+## Chain Integrity
+
+Every row carries a SHA-256 `checksum` over its own fields **and the digest of
+the row it was appended to**, so editing a row in place invalidates it. Each row
+also records that parent explicitly in `previous_checksum`.
+
+### The log is a DAG, not a strict line
+
+A writer reads the current tip and links to it. That read is deliberately
+unlocked, so two concurrent transactions — two Puma threads, a web request and a
+job — can read the same tip and the sequence forks. **That is normal for a
+multi-process writer, and it is not an integrity failure.** Forcing a strict
+line would mean holding a lock until the enclosing business transaction commits
+(a row is invisible to other writers until then, so releasing earlier reopens
+the race), which serialises every audited request behind one mutex.
+
+Storing the parent is what keeps a forked log verifiable: every row is checked
+against the exact digest it signed, whether or not it is the walk's predecessor.
+`previous_checksum` needs no protection of its own — it is an input to the row's
+own digest, so editing it invalidates the row.
+
+```ruby
+result = StandardAudit::AuditLog.verify_chain
+# => { valid: true, verified: 5577, recovered: 0, failures: [] }
+```
+
+- `failures` carries `reason: :digest_mismatch` (the row's fields no longer
+  produce its digest) or `reason: :missing_parent` (the row it was appended to
+  is no longer in the log). Retention pruning does not trip `:missing_parent`:
+  if the walk opens on a row whose parent is already gone, that parent digest
+  is exempt wherever else it appears — a pruned row can have several children.
+  A removal from the *middle* is still reported. One caveat:
+  `standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
+  `created_at`, so rows whose two timestamps disagree can leave a hole rather
+  than a pruned start, and a hole is reported — truthfully, since rows really
+  are missing.
+- `recovered` counts rows with no `previous_checksum` whose parent had to be
+  found by searching back through recent digests — see below.
+- `verify_chain(scope: org)` skips the missing-parent check, because the log is
+  global and a scoped row's parent usually belongs to another scope.
+
+What this deliberately does **not** claim: that the log is in a single total
+order, or that no row was inserted. A concurrent append and an inserted row look
+alike, so a design that tolerates the first cannot detect the second. The
+previous strict-line reading did not actually detect insertions either — it
+reported every concurrent append as tampering, which on one production log meant
+67% of rows red and any real signal lost in the noise.
+
+### Rows written before 0.8.0
+
+They have no `previous_checksum`. Verification falls back to the preceding row
+in the walk and, failing that, searches back through the last `recovery_window`
+(default 256) digests for the one that reproduces the row's checksum — which
+recovers the true parent of a row that forked, **without re-signing anything**.
+This does not weaken tamper detection: a row whose fields were altered
+reproduces no candidate's digest. Pass `strict: true` to skip the search and see
+every fork as a failure.
+
+To make that permanent, add the column and record what each row was actually
+signed against:
+
+```bash
+rails generate standard_audit:add_previous_checksum
+rails db:migrate
+rake standard_audit:relink_checksums
+```
+
+`relink_checksums` never rewrites a `checksum`. It fills in the previously-empty
+`previous_checksum` only when that value reproduces the digest the row has held
+since it was written, so it adds no attestation the rows did not already carry.
+Rows it cannot resolve are reported as `unresolved` and keep failing
+verification — which is the point.
+
+**Do not run `backfill_checksums!` to make a red `verify_chain` go green.** It
+re-signs rows from their current contents, so it attests only that a script ran.
+It is for rows that never had a checksum at all (pre-feature data).
+
 ## Rake Tasks
 
 ```bash
+# Verify chain integrity (exits non-zero on failures)
+rake standard_audit:verify
+
+# Record the parent digest each existing row was signed against
+rake standard_audit:relink_checksums
+
 # Delete logs older than N days (default: retention_days config or 90)
 rake standard_audit:cleanup[180]
 
@@ -423,8 +532,30 @@ t.jsonb :metadata, default: {}
 
 ```ruby
 config.sensitive_keys += %i[medical_record_number]  # extend the built-in defaults
+config.sensitive_key_patterns = [/secret/i]         # catch a whole family
+config.sensitive_key_exceptions = %i[input_tokens]  # ...minus the real ones
+config.filter_nested_metadata = true                # redact nested Hashes too
 ```
 
+`sensitive_keys` matches **exactly** on the key name. There is deliberately no
+substring mode: against the default list it would strip real audit content —
+`input_tokens` / `output_tokens`, `token_digest`, `password_reset_sent_at`,
+`authorization_endpoint`, `onepassword`. Audit rows are append-only, so that
+cannot be undone. Use `sensitive_key_patterns` and check any rule against your
+own data first:
+
+```bash
+bin/rails "standard_audit:sensitive_keys:dry_run[secret]"   # NESTED=1 to model nesting
+```
+
+The dry run writes nothing. It reports per key what would be stripped, what
+would be kept, and nested matches that survive because
+`filter_nested_metadata` is off.
+
+Nested metadata is **not** redacted unless `filter_nested_metadata` is enabled
+— `metadata: { stripe: { client_secret: ... } }` passes both write paths under
+exact matching by default.
+
 **Performance**: For high-volume applications, enable async processing and ensure your `audit_logs` table has appropriate indexes (the install generator adds them by default). Consider partitioning by `occurred_at` for very large tables.
 
 **Retention**: Set `retention_days` in your configuration and run `rake standard_audit:cleanup` via a scheduled job (e.g., cron or SolidQueue recurring). Archive before deleting if you need long-term storage.