standard_audit 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +40 -0
- data/README.md +222 -0
- data/app/models/standard_audit/audit_log.rb +259 -41
- data/lib/generators/standard_audit/add_previous_checksum/add_previous_checksum_generator.rb +17 -0
- data/lib/generators/standard_audit/add_previous_checksum/templates/add_previous_checksum_to_audit_logs.rb.erb +23 -0
- data/lib/generators/standard_audit/install/templates/create_audit_logs.rb.erb +15 -0
- data/lib/standard_audit/configuration.rb +47 -1
- data/lib/standard_audit/operation/audit.rb +253 -0
- data/lib/standard_audit/operation.rb +205 -0
- data/lib/standard_audit/rspec/operation.rb +136 -0
- data/lib/standard_audit/version.rb +1 -1
- data/lib/standard_audit.rb +4 -4
- data/lib/tasks/standard_audit_tasks.rake +20 -2
- metadata +6 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a1d506a4bb5ad3811dd4a70b6be817374ec9a88ced8d1ffd9415bfddd28cc50b
|
|
4
|
+
data.tar.gz: 52c8aaa70f86810c6b308c86fa81e10c368ab5ddb011d16f422745e7681c88da
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2db9ee0ef4983ce2bd9a94d195bbbf973016f32a3e9e2a53b45476df6c9cc6d30f7edd2bfb7e9442d6839e05cc491263b91bb644462946510740c187eb861e29
|
|
7
|
+
data.tar.gz: cce577cbc3f1b5829429ab30765c451f4a0fdca18b9f6283f075e1b3487b3fb763ceaeca846e39768c53707361bfc87024da65e0f8e662885a47766f814dd5c1
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.9.0] - 2026-07-31
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- **`StandardAudit::Operation` — the operation-audit DSL, extracted from five independent copies.** Every consumer app had written the same contract by hand: `audits "x.y"` / `audit_none!` class declarations, a private `audit!(action, **attrs)` as the single write path, and a `verify_audit_declared!` guard that raises on declaration↔write drift in local environments and writes silently in production. The surface is unchanged from those copies, so operation files do not move — adopting apps swap one `include` line and delete their concern. (`rarebit-one/rarebit-ops#279`)
|
|
15
|
+
- It is a **module, never a base class**. The five host `ApplicationOperation`s range 61–303 lines and diverge deliberately (one refuses a `Result`/`execute` lifecycle; one app has no shared base at all), so the module contributes the audit contract and nothing else.
|
|
16
|
+
- **Both adoption shapes work with no configuration**: a shared base that includes it once, whose subclasses are the real operations, and standalone leaves that each include it directly. The unifier is that a class declaring nothing *and* having subclasses is treated as a base and excluded — so a shared base auto-excludes, while every leaf is retained. A class that did declare is never excluded that way.
|
|
17
|
+
- `audit_abstract!` for an intermediate class the automatic rule can't see (typically one with no subclasses yet).
|
|
18
|
+
- **`config.audit_catalogue`** — the host's action vocabulary, as a **callable** (`-> { AuditCatalogue::ACTIONS }`); referencing an autoloadable constant eagerly from an initializer breaks Zeitwerk reloading. A plain Array is accepted for a frozen literal. `nil` (the default) skips the membership check, so the DSL is adoptable before an app has a catalogue. **Membership is the only rule applied** — no dot-count, case, prefix, or namespace validation, because an action may legitimately carry a notification-bus namespace verbatim and normalising it would orphan historical rows.
|
|
19
|
+
- **`config.raise_on_audit_write_error`** (default `false`) and **`config.audit_write_error_handler`**. Four of the five apps report-and-swallow a failed audit write; one deliberately does not rescue, because for it an unaudited state change is itself a compliance failure. A swallow-only module would have silently downgraded that posture. `StandardAudit::Operation::DeclarationError` is never governed by either — it always re-raises, ahead of any generic rescue.
|
|
20
|
+
- **`config.verify_audit_declarations`** (callable or boolean, default: local environments only) — gates the dev/test guard.
|
|
21
|
+
- **Meta-spec logic as plain Ruby predicates** on `StandardAudit::Operation::Audit`: `.operations(source:)`, `.undeclared`, `.unknown_actions`, `.orphan_actions(within:)`, `.missing_write_sites`, `.unexpected_write_sites`, `.duplicate_catalogue_entries`, `.declared_actions`. Each takes an explicit `operations:` list, so hosts can write bespoke assertions.
|
|
22
|
+
- **`standard_audit/rspec/operation`** — a thin RSpec shared-example layer over those predicates, with no logic of its own. It carries a **`minimum:` registry floor**, the one example that catches "someone stopped including the module" or "eager loading stopped reaching the operations"; without it every other example passes vacuously against an empty set. Also `expected:`, `source:` scoping, and `orphans_within:` for hosts whose catalogue covers writers outside `app/operations/`.
|
|
23
|
+
|
|
24
|
+
Everything is additive and nothing includes the module by default — existing hosts upgrade with no code change.
|
|
25
|
+
|
|
26
|
+
## [0.8.0] - 2026-07-30
|
|
27
|
+
|
|
28
|
+
### Added
|
|
29
|
+
|
|
30
|
+
- **`previous_checksum` — every row now records the row it was appended to.** New nullable column; run `rails generate standard_audit:add_previous_checksum && rails db:migrate` to add it. The install migration includes it for new hosts, along with a composite `(created_at, id)` index for the verification walk.
|
|
31
|
+
- **`rake standard_audit:relink_checksums` / `AuditLog.relink_checksums!`** — records, for every row that has no `previous_checksum`, the parent digest it was *actually* signed against, recovering it by search where a concurrent append forked the log. It **never rewrites a `checksum`**: a parent is written only when it reproduces the digest the row has held since it was written, so it adds no attestation the rows did not already carry. Rows it cannot resolve are counted as `unresolved`, left untouched, and keep failing verification — which is the point. This is the repair path for a log broken by the defect below; `backfill_checksums!` is **not** (it re-signs rows from their current contents, so a green result attests only that a script ran).
|
|
32
|
+
- `AuditLog.chain_tip_checksum` and `AuditLog.chain_parent_column?`.
|
|
33
|
+
|
|
34
|
+
### Fixed
|
|
35
|
+
|
|
36
|
+
- **The chained checksum was not concurrency-safe: `verify_chain` failed for 3,762 of 5,577 rows (67%) on one production log, continuously, while every row was untampered.** `compute_checksum` reads the chain tip with an unsynchronised query from `before_create`. Two transactions cannot see each other's uncommitted rows, so both sign against the same predecessor and the sequence forks; verification, which assumed a strict line, reported all but one of them as tampering. Any host with concurrent audit writes — two Puma threads, a web request and a job — has the same broken log. (`fundbright/delivery-ops#433`)
|
|
37
|
+
|
|
38
|
+
**The log is now treated as a DAG rather than a strict line, and each row records its parent.** A row is verified against the exact digest it signed, whether or not that is its predecessor in the walk, so a fork is verifiable instead of fatal. The tip read stays unlocked deliberately: forcing a strict line means holding a lock until the *enclosing business transaction* commits — a row is invisible to other writers until then, so releasing earlier reopens the race — which would serialise every audited request across the estate behind one mutex, on the hot path of every request. `previous_checksum` needs no protection of its own; it is an input to the row's own digest, so editing it invalidates the row.
|
|
39
|
+
|
|
40
|
+
What this deliberately stops claiming: a single total order, and detection of an *inserted* row. A concurrent append and an inserted row are indistinguishable, so a design that tolerates the first cannot detect the second. The strict-line reading did not detect insertions either — it reported every concurrent append as tampering, which on the measured log meant 67% red and any real signal lost in it.
|
|
41
|
+
|
|
42
|
+
**Rows written before this version verify unchanged, with no migration.** With no `previous_checksum`, verification falls back to the preceding row in the walk and then searches back through the last `recovery_window` (default 256) digests for the one that reproduces the row's checksum — recovering the true parent of a forked row without re-signing anything, and reporting how many needed it in the new `recovered:` count. Tamper detection is not weakened: a row whose fields were altered reproduces no candidate's digest. `verify_chain(strict: true)` skips the search and reports every fork as a failure. A host that never runs the migration keeps working on this path indefinitely.
|
|
43
|
+
|
|
44
|
+
- **`verify_chain` never noticed a row being removed from the middle of the log.** Failures now carry `reason:` — `:digest_mismatch`, or `:missing_parent` when the row a record was appended to is no longer present. Retention pruning does not trip it: if the walk opens on a row whose parent is already gone, that parent digest is exempt wherever else it appears — a pruned row can have several children, which is what a concurrent append leaves behind. A removal from the *middle* is still reported. Every row is exempt under `scope:`, because the log is global and a scoped row's parent usually belongs to another scope.
|
|
45
|
+
|
|
46
|
+
- **`verify_chain` did not walk the order it documents, and neither did `backfill_checksums!`.** Both claimed a `(created_at, id)` walk and both used `in_batches`, which paginates by **primary key** range and applies the sort only *within* each batch — so the global sequence was a concatenation of id-ranges, each internally time-sorted. Both now walk with a keyset cursor ordered by `(created_at, id)`, which is the documented order and still loads only `batch_size` rows at a time.
|
|
47
|
+
|
|
48
|
+
With the UUIDv7 ids `assign_uuid` generates, id order usually coincides with insertion order, which is why this stayed latent — but it is wrong for any host that assigns ids differently, backfills rows with an explicit `created_at`, or has clock skew between writers. It matters more than a latent ordering bug normally would: the chain is an *ordering* claim, so a verifier that walks a different order than the one it documents cannot be trusted to prove or disprove anything about chain integrity, including the concurrent-append defect above. `backfill_checksums!` had the same defect, and there it silently *writes* the wrong chain rather than misreading one.
|
|
49
|
+
|
|
10
50
|
## [0.7.0] - 2026-07-30
|
|
11
51
|
|
|
12
52
|
### Added
|
data/README.md
CHANGED
|
@@ -162,6 +162,128 @@ This provides:
|
|
|
162
162
|
organisation.scoped_audit_logs # all logs scoped to this organisation
|
|
163
163
|
```
|
|
164
164
|
|
|
165
|
+
## Operation Audit Contract
|
|
166
|
+
|
|
167
|
+
`StandardAudit::Operation` makes an operation *declare* what it audits, so a new
|
|
168
|
+
mutating operation cannot silently ship with no audit trail. It is a **module,
|
|
169
|
+
not a base class** — it contributes the audit contract and nothing else (no
|
|
170
|
+
`call`, no `Result`, no `execute`), so it drops into any operation style.
|
|
171
|
+
|
|
172
|
+
```ruby
|
|
173
|
+
class ApplicationOperation
|
|
174
|
+
include StandardAudit::Operation # ← the whole adoption, for a shared base
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
class Orders::CreateOperation < ApplicationOperation
|
|
178
|
+
audits "order.created"
|
|
179
|
+
|
|
180
|
+
def execute
|
|
181
|
+
order = Order.create!(**attrs)
|
|
182
|
+
audit!("order.created", target: order, metadata: { total: order.total })
|
|
183
|
+
end
|
|
184
|
+
end
|
|
185
|
+
|
|
186
|
+
class Orders::ReindexOperation < ApplicationOperation
|
|
187
|
+
audit_none! # projection of already-audited state
|
|
188
|
+
end
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
- `audits "x.y"` — the action(s) this operation may write. Repeatable args; the
|
|
192
|
+
declaration is **not inherited**, so every leaf states its own intent.
|
|
193
|
+
- `audit_none!` — mutates state but deliberately records nothing. Leave the
|
|
194
|
+
reason as a comment.
|
|
195
|
+
- `audit_abstract!` — a base or intermediate class, not a real operation.
|
|
196
|
+
- `audit!(action, **attrs)` — private; the single write path. Args forward
|
|
197
|
+
straight to `StandardAudit.record`. Call it inside the operation's own
|
|
198
|
+
transaction so the row commits with the state change.
|
|
199
|
+
|
|
200
|
+
### Adoption shapes
|
|
201
|
+
|
|
202
|
+
Both work with no configuration:
|
|
203
|
+
|
|
204
|
+
1. **A shared base includes the module once**, and the real operations are its
|
|
205
|
+
subclasses (registered via `inherited`). The base declares nothing and has
|
|
206
|
+
subclasses, so it is excluded from the checks automatically. A class that
|
|
207
|
+
*did* declare is never excluded this way, so subclassing a real operation
|
|
208
|
+
cannot quietly drop it.
|
|
209
|
+
2. **Every operation includes the module directly**, with no shared base
|
|
210
|
+
(registered via `included`).
|
|
211
|
+
|
|
212
|
+
Both land in one registry, so a single meta-spec covers a codebase mixing them.
|
|
213
|
+
|
|
214
|
+
### The catalogue
|
|
215
|
+
|
|
216
|
+
The gem has no knowledge of your action vocabulary. Declare it as a **callable** —
|
|
217
|
+
referencing an autoloadable constant eagerly from an initializer pins the
|
|
218
|
+
first-loaded copy and breaks Zeitwerk reloading:
|
|
219
|
+
|
|
220
|
+
```ruby
|
|
221
|
+
StandardAudit.configure(baseline: true) do |config|
|
|
222
|
+
config.audit_catalogue = -> { AuditCatalogue::ACTIONS }
|
|
223
|
+
end
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
`nil` (the default) skips the membership check entirely, so the DSL is adoptable
|
|
227
|
+
before you have a catalogue. **Membership is the only rule** — there is no
|
|
228
|
+
dot-count, case, prefix, or namespace validation, because an action may
|
|
229
|
+
legitimately carry a notification-bus namespace verbatim.
|
|
230
|
+
|
|
231
|
+
### Error policy
|
|
232
|
+
|
|
233
|
+
`StandardAudit::Operation::DeclarationError` — a declaration↔write mismatch —
|
|
234
|
+
**always propagates**, ahead of any generic rescue. It is raised in local
|
|
235
|
+
environments only (`config.verify_audit_declarations`); in production `audit!`
|
|
236
|
+
just writes, because a developer's mistake must not 500 a user. The meta-spec is
|
|
237
|
+
the real gate.
|
|
238
|
+
|
|
239
|
+
A genuine **write** failure is governed by config:
|
|
240
|
+
|
|
241
|
+
```ruby
|
|
242
|
+
config.raise_on_audit_write_error = true # default false: report and swallow
|
|
243
|
+
config.audit_write_error_handler = ->(error, action:, operation:) { ... }
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
Default `false` matches most apps, but set it `true` where an unaudited state
|
|
247
|
+
change is itself a compliance failure — the audit write then aborts the
|
|
248
|
+
operation.
|
|
249
|
+
|
|
250
|
+
### The meta-spec
|
|
251
|
+
|
|
252
|
+
The analysis is **plain Ruby**, so you can assert on it however you like:
|
|
253
|
+
|
|
254
|
+
```ruby
|
|
255
|
+
StandardAudit::Operation::Audit.operations(source: "/app/operations/")
|
|
256
|
+
StandardAudit::Operation::Audit.undeclared # => [Class, ...]
|
|
257
|
+
StandardAudit::Operation::Audit.unknown_actions # => { Class => ["x.y"] }
|
|
258
|
+
StandardAudit::Operation::Audit.orphan_actions(within: ...)
|
|
259
|
+
StandardAudit::Operation::Audit.missing_write_sites # declares but never writes
|
|
260
|
+
StandardAudit::Operation::Audit.unexpected_write_sites
|
|
261
|
+
StandardAudit::Operation::Audit.duplicate_catalogue_entries
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
A thin RSpec layer over exactly those predicates:
|
|
265
|
+
|
|
266
|
+
```ruby
|
|
267
|
+
require "standard_audit/rspec/operation"
|
|
268
|
+
|
|
269
|
+
RSpec.describe "Operation audit declarations" do
|
|
270
|
+
it_behaves_like "standard_audit operation declarations",
|
|
271
|
+
source: "/app/operations/",
|
|
272
|
+
minimum: 100,
|
|
273
|
+
expected: %w[Orders::CreateOperation],
|
|
274
|
+
orphans_within: -> { AuditCatalogue::OPERATION_ACTIONS }
|
|
275
|
+
end
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
**Set `minimum:`.** It is the only example that fails when someone stops
|
|
279
|
+
including the module or eager loading stops reaching your operations — every
|
|
280
|
+
other example passes vacuously against an empty set. `orphans_within:` is the
|
|
281
|
+
catalogue slice operations own; pass it when your catalogue also covers writers
|
|
282
|
+
outside `app/operations/`, which would otherwise always look orphaned.
|
|
283
|
+
|
|
284
|
+
The registry can only see loaded classes, so the shared example calls
|
|
285
|
+
`Rails.application.eager_load!` by default (`eager_load: false` to opt out).
|
|
286
|
+
|
|
165
287
|
## Configuration Reference
|
|
166
288
|
|
|
167
289
|
Use `configure(baseline: true)` in your initializer. It remembers the block so
|
|
@@ -218,6 +340,23 @@ StandardAudit.configure(baseline: true) do |config|
|
|
|
218
340
|
config.before_checksum { |log| log.scope = MyApp.derive_scope(log) }
|
|
219
341
|
config.before_checksum :backfill_scope # an AuditLog instance method
|
|
220
342
|
|
|
343
|
+
# -- Operation audit contract (StandardAudit::Operation) --
|
|
344
|
+
# Your action vocabulary, as a CALLABLE so Zeitwerk can reload it. nil (the
|
|
345
|
+
# default) skips the membership check entirely.
|
|
346
|
+
config.audit_catalogue = -> { AuditCatalogue::ACTIONS }
|
|
347
|
+
|
|
348
|
+
# Whether `audit!` verifies declarations before writing. Callable or boolean;
|
|
349
|
+
# defaults to local environments only.
|
|
350
|
+
config.verify_audit_declarations = -> { Rails.env.local? }
|
|
351
|
+
|
|
352
|
+
# Whether a failed audit WRITE aborts the operation. Default false (report and
|
|
353
|
+
# swallow). Set true where an unaudited state change is a compliance failure.
|
|
354
|
+
# DeclarationError always propagates regardless.
|
|
355
|
+
config.raise_on_audit_write_error = false
|
|
356
|
+
config.audit_write_error_handler = ->(error, action:, operation:) {
|
|
357
|
+
ErrorReporting.notify(error, component: "operation_audit", audit_action: action)
|
|
358
|
+
}
|
|
359
|
+
|
|
221
360
|
# -- Metadata Builder --
|
|
222
361
|
# Optional proc to transform metadata before storage.
|
|
223
362
|
config.metadata_builder = ->(metadata) { metadata.slice(:relevant_key) }
|
|
@@ -406,9 +545,92 @@ which is **still HTTP 200** — it surfaces the advisory in the readiness JSON
|
|
|
406
545
|
without failing the probe or blocking a deploy. The check is duck-typed and has
|
|
407
546
|
no hard dependency on `standard_health`.
|
|
408
547
|
|
|
548
|
+
## Chain Integrity
|
|
549
|
+
|
|
550
|
+
Every row carries a SHA-256 `checksum` over its own fields **and the digest of
|
|
551
|
+
the row it was appended to**, so editing a row in place invalidates it. Each row
|
|
552
|
+
also records that parent explicitly in `previous_checksum`.
|
|
553
|
+
|
|
554
|
+
### The log is a DAG, not a strict line
|
|
555
|
+
|
|
556
|
+
A writer reads the current tip and links to it. That read is deliberately
|
|
557
|
+
unlocked, so two concurrent transactions — two Puma threads, a web request and a
|
|
558
|
+
job — can read the same tip and the sequence forks. **That is normal for a
|
|
559
|
+
multi-process writer, and it is not an integrity failure.** Forcing a strict
|
|
560
|
+
line would mean holding a lock until the enclosing business transaction commits
|
|
561
|
+
(a row is invisible to other writers until then, so releasing earlier reopens
|
|
562
|
+
the race), which serialises every audited request behind one mutex.
|
|
563
|
+
|
|
564
|
+
Storing the parent is what keeps a forked log verifiable: every row is checked
|
|
565
|
+
against the exact digest it signed, whether or not it is the walk's predecessor.
|
|
566
|
+
`previous_checksum` needs no protection of its own — it is an input to the row's
|
|
567
|
+
own digest, so editing it invalidates the row.
|
|
568
|
+
|
|
569
|
+
```ruby
|
|
570
|
+
result = StandardAudit::AuditLog.verify_chain
|
|
571
|
+
# => { valid: true, verified: 5577, recovered: 0, failures: [] }
|
|
572
|
+
```
|
|
573
|
+
|
|
574
|
+
- `failures` carries `reason: :digest_mismatch` (the row's fields no longer
|
|
575
|
+
produce its digest) or `reason: :missing_parent` (the row it was appended to
|
|
576
|
+
is no longer in the log). Retention pruning does not trip `:missing_parent`:
|
|
577
|
+
if the walk opens on a row whose parent is already gone, that parent digest
|
|
578
|
+
is exempt wherever else it appears — a pruned row can have several children.
|
|
579
|
+
A removal from the *middle* is still reported. One caveat:
|
|
580
|
+
`standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
|
|
581
|
+
`created_at`, so rows whose two timestamps disagree can leave a hole rather
|
|
582
|
+
than a pruned start, and a hole is reported — truthfully, since rows really
|
|
583
|
+
are missing.
|
|
584
|
+
- `recovered` counts rows with no `previous_checksum` whose parent had to be
|
|
585
|
+
found by searching back through recent digests — see below.
|
|
586
|
+
- `verify_chain(scope: org)` skips the missing-parent check, because the log is
|
|
587
|
+
global and a scoped row's parent usually belongs to another scope.
|
|
588
|
+
|
|
589
|
+
What this deliberately does **not** claim: that the log is in a single total
|
|
590
|
+
order, or that no row was inserted. A concurrent append and an inserted row look
|
|
591
|
+
alike, so a design that tolerates the first cannot detect the second. The
|
|
592
|
+
previous strict-line reading did not actually detect insertions either — it
|
|
593
|
+
reported every concurrent append as tampering, which on one production log meant
|
|
594
|
+
67% of rows red and any real signal lost in the noise.
|
|
595
|
+
|
|
596
|
+
### Rows written before 0.8.0
|
|
597
|
+
|
|
598
|
+
They have no `previous_checksum`. Verification falls back to the preceding row
|
|
599
|
+
in the walk and, failing that, searches back through the last `recovery_window`
|
|
600
|
+
(default 256) digests for the one that reproduces the row's checksum — which
|
|
601
|
+
recovers the true parent of a row that forked, **without re-signing anything**.
|
|
602
|
+
This does not weaken tamper detection: a row whose fields were altered
|
|
603
|
+
reproduces no candidate's digest. Pass `strict: true` to skip the search and see
|
|
604
|
+
every fork as a failure.
|
|
605
|
+
|
|
606
|
+
To make that permanent, add the column and record what each row was actually
|
|
607
|
+
signed against:
|
|
608
|
+
|
|
609
|
+
```bash
|
|
610
|
+
rails generate standard_audit:add_previous_checksum
|
|
611
|
+
rails db:migrate
|
|
612
|
+
rake standard_audit:relink_checksums
|
|
613
|
+
```
|
|
614
|
+
|
|
615
|
+
`relink_checksums` never rewrites a `checksum`. It fills in the previously-empty
|
|
616
|
+
`previous_checksum` only when that value reproduces the digest the row has held
|
|
617
|
+
since it was written, so it adds no attestation the rows did not already carry.
|
|
618
|
+
Rows it cannot resolve are reported as `unresolved` and keep failing
|
|
619
|
+
verification — which is the point.
|
|
620
|
+
|
|
621
|
+
**Do not run `backfill_checksums!` to make a red `verify_chain` go green.** It
|
|
622
|
+
re-signs rows from their current contents, so it attests only that a script ran.
|
|
623
|
+
It is for rows that never had a checksum at all (pre-feature data).
|
|
624
|
+
|
|
409
625
|
## Rake Tasks
|
|
410
626
|
|
|
411
627
|
```bash
|
|
628
|
+
# Verify chain integrity (exits non-zero on failures)
|
|
629
|
+
rake standard_audit:verify
|
|
630
|
+
|
|
631
|
+
# Record the parent digest each existing row was signed against
|
|
632
|
+
rake standard_audit:relink_checksums
|
|
633
|
+
|
|
412
634
|
# Delete logs older than N days (default: retention_days config or 90)
|
|
413
635
|
rake standard_audit:cleanup[180]
|
|
414
636
|
|
|
@@ -171,45 +171,210 @@ module StandardAudit
|
|
|
171
171
|
OpenSSL::Digest::SHA256.hexdigest(canonical)
|
|
172
172
|
end
|
|
173
173
|
|
|
174
|
-
#
|
|
175
|
-
|
|
174
|
+
# The checksum of the most recent row — the node a new row links to.
|
|
175
|
+
def self.chain_tip_checksum
|
|
176
|
+
order(created_at: :desc, id: :desc).limit(1).pick(:checksum)
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
# True when the table carries the `previous_checksum` column, i.e. the host
|
|
180
|
+
# has run the `standard_audit:add_previous_checksum` migration. Rows written
|
|
181
|
+
# without it are verified by position and parent recovery instead, so the
|
|
182
|
+
# gem keeps working unmigrated.
|
|
183
|
+
def self.chain_parent_column?
|
|
184
|
+
column_names.include?("previous_checksum")
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
# Verifies the integrity of the audit log. Returns a result hash with
|
|
188
|
+
# :valid (boolean), :verified (count), :recovered (count) and :failures
|
|
189
|
+
# (array of hashes carrying :id, :event_type, :created_at, :expected,
|
|
190
|
+
# :actual and :reason).
|
|
176
191
|
#
|
|
177
192
|
# Records are processed in (created_at, id) order. Records without a
|
|
178
|
-
# checksum (pre-feature data) reset the
|
|
179
|
-
#
|
|
180
|
-
|
|
193
|
+
# checksum (pre-feature data) reset the walk — the next checksummed record
|
|
194
|
+
# starts a new independent segment.
|
|
195
|
+
#
|
|
196
|
+
# The log is a DAG, not a strict line. Concurrent writers link to whichever
|
|
197
|
+
# node was the tip when they read it, so two rows can share a parent and
|
|
198
|
+
# the sequence forks. Every row is still verified against the exact digest
|
|
199
|
+
# it signed, in one of two ways:
|
|
200
|
+
#
|
|
201
|
+
# * `previous_checksum` present (written by 0.8+ on a migrated host): the
|
|
202
|
+
# parent the writer asserts. It is covered by this row's own digest, so
|
|
203
|
+
# it cannot be edited to excuse a tampered row.
|
|
204
|
+
# * `previous_checksum` NULL (pre-0.8 rows, or an unmigrated host): the
|
|
205
|
+
# preceding row in the walk, and — unless `strict:` — a search back
|
|
206
|
+
# through the last `recovery_window` digests (and finally "no parent at
|
|
207
|
+
# all", for a row written against an empty table) for the one that
|
|
208
|
+
# actually reproduces this row's checksum. That recovers the true parent
|
|
209
|
+
# of a forked row without re-signing anything. It does not weaken tamper
|
|
210
|
+
# detection: a row whose fields were altered reproduces no candidate's
|
|
211
|
+
# digest, so it still fails.
|
|
212
|
+
#
|
|
213
|
+
# A row whose parent digest is absent from the log is reported with
|
|
214
|
+
# `reason: :missing_parent` — a row was removed. Two exemptions:
|
|
215
|
+
#
|
|
216
|
+
# * `scope:` — the log is global, so a scoped row's parent usually
|
|
217
|
+
# belongs to another scope and is absent for an innocent reason.
|
|
218
|
+
# * a pruned start. If the walk *opens* on a row whose parent is already
|
|
219
|
+
# gone, the log has had its start removed (retention cleanup). That
|
|
220
|
+
# parent digest is then exempt wherever else it appears, because a
|
|
221
|
+
# pruned row can have several children — which is exactly what a
|
|
222
|
+
# concurrent append leaves behind. Removing a row from the middle is
|
|
223
|
+
# still reported: its digest is not the one the walk opened on.
|
|
224
|
+
#
|
|
225
|
+
# `standard_audit:cleanup` prunes by `occurred_at` while the walk orders by
|
|
226
|
+
# `created_at`. Rows whose two timestamps disagree (a backdated
|
|
227
|
+
# `occurred_at`) can leave a hole rather than a prefix, and a hole is
|
|
228
|
+
# reported — truthfully, since rows really are missing.
|
|
229
|
+
def self.verify_chain(scope: nil, batch_size: 1000, recovery_window: 256, strict: false)
|
|
181
230
|
relation = scope ? where(scope_gid: scope.to_global_id.to_s) : all
|
|
231
|
+
check_parents = scope.nil?
|
|
232
|
+
declared_parents = chain_parent_column?
|
|
182
233
|
|
|
183
234
|
previous_checksum = nil
|
|
184
235
|
verified = 0
|
|
236
|
+
recovered = 0
|
|
185
237
|
failures = []
|
|
238
|
+
window = []
|
|
239
|
+
first_row = true
|
|
240
|
+
pruned_parents = []
|
|
241
|
+
|
|
242
|
+
each_in_chain_order(relation, batch_size: batch_size) do |record|
|
|
243
|
+
if record.checksum.blank?
|
|
244
|
+
previous_checksum = nil
|
|
245
|
+
window.clear
|
|
246
|
+
next
|
|
247
|
+
end
|
|
186
248
|
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
if record.checksum.blank?
|
|
190
|
-
previous_checksum = nil
|
|
191
|
-
next
|
|
192
|
-
end
|
|
249
|
+
verified += 1
|
|
250
|
+
declared = record.previous_checksum if declared_parents
|
|
193
251
|
|
|
194
|
-
|
|
252
|
+
if declared.present?
|
|
253
|
+
expected = record.compute_checksum_value(previous_checksum: declared)
|
|
195
254
|
|
|
196
255
|
if record.checksum != expected
|
|
197
|
-
failures <<
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
256
|
+
failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
|
|
257
|
+
elsif check_parents && !parent_present?(declared, window, relation)
|
|
258
|
+
if first_row
|
|
259
|
+
# The walk opens on a row whose parent is already gone, so the
|
|
260
|
+
# log has had its start removed — retention pruning, typically.
|
|
261
|
+
# That parent is unknowable, and it can have several children
|
|
262
|
+
# (which is what a concurrent append leaves behind), so the
|
|
263
|
+
# exemption is remembered per digest rather than for one row.
|
|
264
|
+
pruned_parents << declared
|
|
265
|
+
elsif !pruned_parents.include?(declared)
|
|
266
|
+
failures << chain_failure(record, expected: expected, reason: :missing_parent)
|
|
267
|
+
end
|
|
204
268
|
end
|
|
269
|
+
else
|
|
270
|
+
expected = record.compute_checksum_value(previous_checksum: previous_checksum)
|
|
205
271
|
|
|
206
|
-
|
|
207
|
-
|
|
272
|
+
if record.checksum == expected
|
|
273
|
+
# Links to the row before it, as a linear chain does.
|
|
274
|
+
elsif !strict && recover_parent(record, window)
|
|
275
|
+
recovered += 1
|
|
276
|
+
else
|
|
277
|
+
failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
|
|
278
|
+
end
|
|
208
279
|
end
|
|
280
|
+
|
|
281
|
+
previous_checksum = record.checksum
|
|
282
|
+
first_row = false
|
|
283
|
+
window << record.checksum
|
|
284
|
+
window.shift if window.size > recovery_window
|
|
285
|
+
end
|
|
286
|
+
|
|
287
|
+
{ valid: failures.empty?, verified: verified, recovered: recovered, failures: failures }
|
|
288
|
+
end
|
|
289
|
+
|
|
290
|
+
# Records, for every row that does not already carry one, the parent digest
|
|
291
|
+
# it was actually signed against — recovering it by search where a
|
|
292
|
+
# concurrent append forked the chain. Returns
|
|
293
|
+
# `{ relinked:, unresolved:, skipped: }`.
|
|
294
|
+
#
|
|
295
|
+
# This NEVER rewrites a `checksum`. It writes only the previously-empty
|
|
296
|
+
# `previous_checksum` column, so it adds no attestation the rows did not
|
|
297
|
+
# already carry: a parent is recorded only when it reproduces the digest
|
|
298
|
+
# the row has held since it was written. Rows whose parent cannot be
|
|
299
|
+
# reproduced are left untouched and counted in :unresolved — they are the
|
|
300
|
+
# rows verification should keep reporting.
|
|
301
|
+
def self.relink_checksums!(batch_size: 1000, recovery_window: 256)
|
|
302
|
+
return { relinked: 0, unresolved: 0, skipped: 0 } unless chain_parent_column?
|
|
303
|
+
|
|
304
|
+
previous_checksum = nil
|
|
305
|
+
relinked = 0
|
|
306
|
+
unresolved = 0
|
|
307
|
+
skipped = 0
|
|
308
|
+
window = []
|
|
309
|
+
|
|
310
|
+
each_in_chain_order(all, batch_size: batch_size) do |record|
|
|
311
|
+
if record.checksum.blank?
|
|
312
|
+
previous_checksum = nil
|
|
313
|
+
window.clear
|
|
314
|
+
next
|
|
315
|
+
end
|
|
316
|
+
|
|
317
|
+
found = resolve_parent(record, previous_checksum, window) if record.previous_checksum.blank?
|
|
318
|
+
|
|
319
|
+
if record.previous_checksum.present?
|
|
320
|
+
skipped += 1
|
|
321
|
+
elsif found.nil?
|
|
322
|
+
unresolved += 1
|
|
323
|
+
elsif found.first.present?
|
|
324
|
+
record.update_columns(previous_checksum: found.first)
|
|
325
|
+
relinked += 1
|
|
326
|
+
else
|
|
327
|
+
# A genuine segment root. NULL already says so; nothing to write.
|
|
328
|
+
skipped += 1
|
|
329
|
+
end
|
|
330
|
+
|
|
331
|
+
previous_checksum = record.checksum
|
|
332
|
+
window << record.checksum
|
|
333
|
+
window.shift if window.size > recovery_window
|
|
334
|
+
end
|
|
335
|
+
|
|
336
|
+
{ relinked: relinked, unresolved: unresolved, skipped: skipped }
|
|
337
|
+
end
|
|
338
|
+
|
|
339
|
+
def self.chain_failure(record, expected:, reason:)
|
|
340
|
+
{
|
|
341
|
+
id: record.id,
|
|
342
|
+
event_type: record.event_type,
|
|
343
|
+
created_at: record.created_at,
|
|
344
|
+
expected: expected,
|
|
345
|
+
actual: record.checksum,
|
|
346
|
+
reason: reason
|
|
347
|
+
}
|
|
348
|
+
end
|
|
349
|
+
private_class_method :chain_failure
|
|
350
|
+
|
|
351
|
+
# Searches `window` (most recent first, then "no parent at all") for the
|
|
352
|
+
# digest that reproduces the record's stored checksum. Returns a one-element
|
|
353
|
+
# array holding the parent — which may itself be nil, for a row written
|
|
354
|
+
# against an empty table — or nil when nothing reproduces the digest.
|
|
355
|
+
#
|
|
356
|
+
# SHA-256 preimage resistance is what makes this safe: a row whose fields
|
|
357
|
+
# were altered reproduces no candidate's digest, so it is still reported.
|
|
358
|
+
def self.recover_parent(record, window)
|
|
359
|
+
window.reverse_each do |candidate|
|
|
360
|
+
return [candidate] if record.checksum == record.compute_checksum_value(previous_checksum: candidate)
|
|
209
361
|
end
|
|
210
362
|
|
|
211
|
-
|
|
363
|
+
[nil] if record.checksum == record.compute_checksum_value(previous_checksum: nil)
|
|
212
364
|
end
|
|
365
|
+
private_class_method :recover_parent
|
|
366
|
+
|
|
367
|
+
def self.resolve_parent(record, previous_checksum, window)
|
|
368
|
+
return [previous_checksum] if record.checksum == record.compute_checksum_value(previous_checksum: previous_checksum)
|
|
369
|
+
|
|
370
|
+
recover_parent(record, window)
|
|
371
|
+
end
|
|
372
|
+
private_class_method :resolve_parent
|
|
373
|
+
|
|
374
|
+
def self.parent_present?(digest, window, relation)
|
|
375
|
+
window.include?(digest) || relation.exists?(checksum: digest)
|
|
376
|
+
end
|
|
377
|
+
private_class_method :parent_present?
|
|
213
378
|
|
|
214
379
|
# Backfills checksums for records that don't have them (e.g. pre-existing
|
|
215
380
|
# records before the checksum feature was added).
|
|
@@ -217,27 +382,67 @@ module StandardAudit
|
|
|
217
382
|
previous_checksum = nil
|
|
218
383
|
count = 0
|
|
219
384
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
end
|
|
385
|
+
each_in_chain_order(all, batch_size: batch_size) do |record|
|
|
386
|
+
if record.checksum.present?
|
|
387
|
+
previous_checksum = record.checksum
|
|
388
|
+
next
|
|
389
|
+
end
|
|
226
390
|
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
391
|
+
new_checksum = compute_checksum_value(
|
|
392
|
+
record.attributes.slice(*CHECKSUM_FIELDS),
|
|
393
|
+
previous_checksum: previous_checksum
|
|
394
|
+
)
|
|
395
|
+
columns = { checksum: new_checksum }
|
|
396
|
+
columns[:previous_checksum] = previous_checksum if chain_parent_column?
|
|
397
|
+
record.update_columns(columns)
|
|
232
398
|
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
end
|
|
399
|
+
previous_checksum = new_checksum
|
|
400
|
+
count += 1
|
|
236
401
|
end
|
|
237
402
|
|
|
238
403
|
count
|
|
239
404
|
end
|
|
240
405
|
|
|
406
|
+
# Yields every record of `relation` in true (created_at, id) order, loading
|
|
407
|
+
# at most `batch_size` rows at a time via a keyset cursor.
|
|
408
|
+
#
|
|
409
|
+
# `in_batches` cannot do this, and used to be used here: it paginates by
|
|
410
|
+
# PRIMARY KEY range and applies any ordering only *within* each batch, so
|
|
411
|
+
# the global sequence it yields is a concatenation of id-ranges, each
|
|
412
|
+
# internally time-sorted. With the UUIDv7 ids `assign_uuid` generates that
|
|
413
|
+
# usually coincides with insertion order, which is why it went unnoticed —
|
|
414
|
+
# but it is wrong for any host that assigns ids differently, backfills rows
|
|
415
|
+
# with an explicit created_at, or has clock skew between writers. The chain
|
|
416
|
+
# is an ordering claim, so a verifier that walks a different order than the
|
|
417
|
+
# one it documents cannot be trusted to prove or disprove anything about it.
|
|
418
|
+
#
|
|
419
|
+
# The cursor predicate assumes created_at is NOT NULL, which the install
|
|
420
|
+
# migration's `t.timestamps` guarantees. A NULL created_at would compare as
|
|
421
|
+
# NULL and end the walk early rather than loop forever.
|
|
422
|
+
def self.each_in_chain_order(relation, batch_size: 1000)
|
|
423
|
+
cursor = nil
|
|
424
|
+
|
|
425
|
+
loop do
|
|
426
|
+
page = relation.reorder(created_at: :asc, id: :asc).limit(batch_size)
|
|
427
|
+
|
|
428
|
+
if cursor
|
|
429
|
+
page = page.where(
|
|
430
|
+
"created_at > :created_at OR (created_at = :created_at AND id > :id)",
|
|
431
|
+
created_at: cursor.first, id: cursor.last
|
|
432
|
+
)
|
|
433
|
+
end
|
|
434
|
+
|
|
435
|
+
records = page.to_a
|
|
436
|
+
break if records.empty?
|
|
437
|
+
|
|
438
|
+
records.each { |record| yield record }
|
|
439
|
+
break if records.size < batch_size
|
|
440
|
+
|
|
441
|
+
cursor = [records.last.created_at, records.last.id]
|
|
442
|
+
end
|
|
443
|
+
end
|
|
444
|
+
private_class_method :each_in_chain_order
|
|
445
|
+
|
|
241
446
|
private
|
|
242
447
|
|
|
243
448
|
def emit_created_event
|
|
@@ -252,12 +457,25 @@ module StandardAudit
|
|
|
252
457
|
Rails.logger.warn("[StandardAudit] Failed to emit event: #{e.class}: #{e.message}")
|
|
253
458
|
end
|
|
254
459
|
|
|
255
|
-
#
|
|
256
|
-
#
|
|
257
|
-
#
|
|
258
|
-
#
|
|
460
|
+
# Links the new record to whichever row was the chain tip when this writer
|
|
461
|
+
# read it, and RECORDS WHICH ONE THAT WAS.
|
|
462
|
+
#
|
|
463
|
+
# The tip read is deliberately unlocked. Two concurrent transactions cannot
|
|
464
|
+
# see each other's uncommitted rows, so both may read the same tip and the
|
|
465
|
+
# sequence forks — that is not an error, it is what a multi-process writer
|
|
466
|
+
# does, and it is why 67% of one production log failed verification while
|
|
467
|
+
# the rows themselves were untampered (fundbright/delivery-ops#433). The
|
|
468
|
+
# alternative is a lock held until the *enclosing business transaction*
|
|
469
|
+
# commits (a row is invisible to other writers until then, so releasing
|
|
470
|
+
# earlier reopens the race), which would serialise every audited request in
|
|
471
|
+
# the estate behind one mutex. Recording the parent instead makes the fork
|
|
472
|
+
# verifiable rather than preventing it.
|
|
473
|
+
#
|
|
474
|
+
# `previous_checksum` needs no protection of its own: it is an input to
|
|
475
|
+
# this row's own digest, so editing it invalidates the row.
|
|
259
476
|
def compute_checksum
|
|
260
|
-
previous = self.class.
|
|
477
|
+
previous = self.class.chain_tip_checksum
|
|
478
|
+
self.previous_checksum = previous if self.class.chain_parent_column?
|
|
261
479
|
self.checksum = compute_checksum_value(previous_checksum: previous)
|
|
262
480
|
end
|
|
263
481
|
|