standard_audit 0.13.0 → 0.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +173 -0
- data/README.md +121 -2
- data/app/models/standard_audit/audit_log.rb +189 -60
- data/lib/standard_audit/checksum/key_order_search.rb +153 -0
- data/lib/standard_audit/checksum.rb +208 -0
- data/lib/standard_audit/configuration.rb +12 -1
- data/lib/standard_audit/version.rb +1 -1
- data/lib/standard_audit.rb +13 -0
- data/lib/tasks/standard_audit_tasks.rake +18 -1
- metadata +10 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7ca3b859a3eb6993caa5c57a9a43621f67a44871c5ee3f05b2ea5d5ca79860b4
|
|
4
|
+
data.tar.gz: 7a85141df78963fa16c7b223da7a7d48270c368538815f26d8e097a09487c0ff
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b74d1bbbd38c3580fb36f4db24f40f7fd01c9f6f807035e3aee8e0a3979692f0c4dc8175cdb53799291d578fdf0e8ca80add04e6df1b09b819425717088abea1
|
|
7
|
+
data.tar.gz: 32dd1f53798d592fe517966dd4677abf42ce5ab7dd650176f56dfcd873025a43a3a81a048e276da37fdf90954cc68d3746fb9355e9c1e40559d123c87ab605af
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,179 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.13.1] - 2026-09-25
|
|
11
|
+
|
|
12
|
+
Fixes the checksum so it survives PostgreSQL `jsonb`
|
|
13
|
+
([fundbright/delivery-ops#689](https://github.com/fundbright/delivery-ops/issues/689)).
|
|
14
|
+
Through 0.13.0, the row digest hashed `metadata.to_json` in the order the
|
|
15
|
+
Ruby hash was built. `jsonb` stores keys in its own order (shortest first,
|
|
16
|
+
then bytewise). So any row whose metadata had more than one key failed
|
|
17
|
+
`verify_chain` with `digest_mismatch`, unless its keys happened to be
|
|
18
|
+
written in that order. In fundbright production that is 17,448 of 25,910
|
|
19
|
+
rows (67%). It, not concurrency, was the main cause behind
|
|
20
|
+
fundbright/delivery-ops#433; the 0.8.0 recovery search rescues 24 of those
|
|
21
|
+
rows. Every host on `jsonb` is affected: fundbright, sidekick, jumpdrive,
|
|
22
|
+
luminality and nutripod. The gem's suite runs on SQLite, which keeps JSON as
|
|
23
|
+
text in insertion order, so it never saw the bug.
|
|
24
|
+
|
|
25
|
+
### ⚠️ Deploy before 2026-10-01T00:00:00Z
|
|
26
|
+
|
|
27
|
+
The new checksum switches on **by the clock, not by deploy**. Rows whose
|
|
28
|
+
`created_at` is at or after `StandardAudit::CANONICAL_CHECKSUM_CUTOVER`
|
|
29
|
+
(**2026-10-01T00:00:00Z**) are signed with the canonical checksum and
|
|
30
|
+
verified strictly with it. **Every process that writes audit rows must run
|
|
31
|
+
0.13.1 before then.** A web worker or job runner still on an older gem after
|
|
32
|
+
the cutover writes legacy-hashed rows with post-cutover timestamps. Those
|
|
33
|
+
rows fail verification as `:digest_mismatch`, and the only honest fix
|
|
34
|
+
afterwards is to explain them.
|
|
35
|
+
|
|
36
|
+
If your rollout will slip, set `config.canonical_checksum_since` to a later
|
|
37
|
+
time **before 2026-10-01**, in a `configure(baseline: true)` block. Never move
|
|
38
|
+
it once the time has passed: verification recomputes the same decision from
|
|
39
|
+
each row's stored `created_at`, so moving it re-judges rows under the other
|
|
40
|
+
algorithm.
|
|
41
|
+
|
|
42
|
+
### Upgrade steps
|
|
43
|
+
|
|
44
|
+
1. Bump to 0.13.1 and deploy **before 2026-10-01T00:00:00Z** (or move
|
|
45
|
+
`config.canonical_checksum_since`, as described above). **No migration or
|
|
46
|
+
new column is needed.**
|
|
47
|
+
2. Regenerate Sorbet RBIs where the app uses Tapioca. This release adds
|
|
48
|
+
`StandardAudit::Checksum` and new `verify_chain` keywords.
|
|
49
|
+
3. After the cutover, run `verify_chain` (or `rake standard_audit:verify`)
|
|
50
|
+
and read the new counts; see "What `verify_chain` reports" below. Rows
|
|
51
|
+
created after the cutover should all verify. Record the
|
|
52
|
+
`legacy_unverifiable` count and alert if it ever grows.
|
|
53
|
+
4. **Decide the policy for legacy rows that can't be reconstructed.** This
|
|
54
|
+
is a human decision (#689 options a to c). The gem reports these rows; it
|
|
55
|
+
does not make the call.
|
|
56
|
+
|
|
57
|
+
**Requires Rails 8.1** (`activerecord`, `activejob`, `activesupport` `>= 8.1`,
|
|
58
|
+
was `>= 8.0`). Every consumer app runs 8.1; 8.0 was never exercised in CI.
|
|
59
|
+
|
|
60
|
+
### Changed
|
|
61
|
+
|
|
62
|
+
- **Canonical checksum for rows created at or after the cutover.** It is
|
|
63
|
+
the SHA-256 of canonical JSON
|
|
64
|
+
`{"fields": {…}, "previous_checksum": …, "v": 2}`:
|
|
65
|
+
- JSON values (metadata, or any Hash/Array field) are round-tripped exactly
|
|
66
|
+
as the column stores them.
|
|
67
|
+
- Object keys are sorted bytewise at every depth.
|
|
68
|
+
- Integral floats hash as integers, and times as UTC ISO 8601 with
|
|
69
|
+
microseconds.
|
|
70
|
+
- Strings are escaped at the byte level.
|
|
71
|
+
- nil is distinct from `""`.
|
|
72
|
+
|
|
73
|
+
Every input to the digest was audited. `CHECKSUM_FIELDS` is unchanged, and
|
|
74
|
+
`metadata` is the only JSON field in the shipped schema. The parent is
|
|
75
|
+
covered. The canonical form also closes a field-shifting ambiguity in the
|
|
76
|
+
legacy form, where `|` inside a value could move content between adjacent
|
|
77
|
+
fields with the same digest.
|
|
78
|
+
- **Rows created before the cutover keep the legacy digest byte for byte**
|
|
79
|
+
(`StandardAudit::Checksum.legacy_digest`). They are written and verified
|
|
80
|
+
exactly as before.
|
|
81
|
+
- The algorithm is chosen from `created_at` on every write path: `create`,
|
|
82
|
+
the batched `insert_all!` path, and `backfill_checksums!`. Verification and
|
|
83
|
+
`relink_checksums!` make the same choice. `created_at` is fixed before the
|
|
84
|
+
checksum is computed, so the writer and the verifier read the same stored
|
|
85
|
+
value. `created_at` is not itself hashed; see "Alert if it grows" below.
|
|
86
|
+
- `compute_checksum_value` (instance and class) picks the algorithm from
|
|
87
|
+
`created_at`. `version:` forces one (`Checksum::LEGACY` / `CANONICAL`).
|
|
88
|
+
- **`verify_chain` returns `reordered:`, `legacy_unverifiable:` and
|
|
89
|
+
`unverifiable:`**, and takes `key_order_search_limit:` (default 720) and
|
|
90
|
+
`fail_on_legacy_unverifiable:` (default false). Existing keys and reasons
|
|
91
|
+
are unchanged.
|
|
92
|
+
- `rake standard_audit:verify` prints the cutover, the new counts and a tally
|
|
93
|
+
per reason. It takes `FAIL_ON_LEGACY_UNVERIFIABLE=1` and
|
|
94
|
+
`KEY_ORDER_SEARCH_LIMIT=<n>`.
|
|
95
|
+
|
|
96
|
+
### What `verify_chain` reports
|
|
97
|
+
|
|
98
|
+
**Rows created at or after the cutover** are checked with the canonical
|
|
99
|
+
digest only, against the declared parent, else the preceding row, else the
|
|
100
|
+
recovery search. A mismatch is `:digest_mismatch`. These rows are never
|
|
101
|
+
classified as legacy. Chain linkage holds across the cutover: the first
|
|
102
|
+
canonical row's parent is the last legacy row's checksum.
|
|
103
|
+
|
|
104
|
+
**Rows created before the cutover** are first checked with the legacy digest
|
|
105
|
+
exactly as in 0.13.0: stored order, against the declared parent, else the
|
|
106
|
+
preceding row, else the recovery search. A row that still doesn't reproduce
|
|
107
|
+
goes through these steps:
|
|
108
|
+
|
|
109
|
+
1. **Key-order reconstruction.** The insertion order can't be read back,
|
|
110
|
+
because `jsonb`'s order depends only on the key set. It can be searched,
|
|
111
|
+
though: an ordering of the stored keys (at every depth) that reproduces the
|
|
112
|
+
digest the row has held since it was written is a witness, in the same
|
|
113
|
+
sense as the parent search. A row whose values were edited reproduces no
|
|
114
|
+
ordering.
|
|
115
|
+
- Such rows count in `reordered` and are valid.
|
|
116
|
+
- The search is bounded by `key_order_search_limit` orderings per row.
|
|
117
|
+
- Orders learned from earlier rows with the same keys are tried first, so
|
|
118
|
+
most rows cost one hash.
|
|
119
|
+
- Parents tried: the declared parent, or else the preceding row and "no
|
|
120
|
+
parent".
|
|
121
|
+
2. **`:digest_mismatch`** if the search covered every ordering against the
|
|
122
|
+
parent the row declares, or if the row has no JSON object with more than
|
|
123
|
+
one key (key order can't be why it fails).
|
|
124
|
+
3. **`:missing_parent`** if the declared parent is gone.
|
|
125
|
+
4. Otherwise **`:legacy_key_order_unverifiable`**: the row has a multi-key
|
|
126
|
+
object and every other check passed. It is listed in `unverifiable` and
|
|
127
|
+
counted in `legacy_unverifiable`.
|
|
128
|
+
|
|
129
|
+
**`valid` semantics.** `valid` is still `failures.empty?`. A
|
|
130
|
+
`:legacy_key_order_unverifiable` row **does not make the chain invalid on its
|
|
131
|
+
own**, because it means "cannot be proven either way": an edited legacy row
|
|
132
|
+
looks exactly like one whose key order was lost. It is never silent. It is
|
|
133
|
+
reported separately with a count, and `fail_on_legacy_unverifiable: true`
|
|
134
|
+
turns these rows into failures.
|
|
135
|
+
|
|
136
|
+
**Alert if it grows.** After the cutover, no new legacy row can be written,
|
|
137
|
+
so under honest operation `legacy_unverifiable` never grows. Growth means a
|
|
138
|
+
pre-cutover row was edited, or a row's `created_at` was moved back across
|
|
139
|
+
the cutover.
|
|
140
|
+
|
|
141
|
+
How many fundbright rows reconstruction rescues depends on their key counts
|
|
142
|
+
and on whether they carry `previous_checksum`. That hasn't been measured on
|
|
143
|
+
production data yet.
|
|
144
|
+
|
|
145
|
+
### Added
|
|
146
|
+
|
|
147
|
+
- `StandardAudit::CANONICAL_CHECKSUM_CUTOVER` (frozen,
|
|
148
|
+
`Time.utc(2026, 10, 1)`) and `config.canonical_checksum_since`, which
|
|
149
|
+
defaults to it.
|
|
150
|
+
- `StandardAudit::Checksum` (`algorithm_for`, `digest`, `legacy_digest`,
|
|
151
|
+
`canonical_digest`, `canonical_json`) and
|
|
152
|
+
`StandardAudit::Checksum::KeyOrderSearch`.
|
|
153
|
+
- **A PostgreSQL CI leg** (`test (postgres)`: `postgres:16-alpine`, `jsonb`
|
|
154
|
+
metadata) runs the full suite. The SQLite matrix stays. The dummy app uses
|
|
155
|
+
`DATABASE_URL` when it is set. New specs cover:
|
|
156
|
+
- both sides of the cutover, a row just before and just after it, the
|
|
157
|
+
config override, and the batch and backfill paths;
|
|
158
|
+
- linkage across the cutover;
|
|
159
|
+
- the `jsonb` regression. On Postgres the specs also assert that the stored
|
|
160
|
+
order really changed and that the legacy digest fails on it.
|
|
161
|
+
|
|
162
|
+
### Not included: re-sealing legacy rows
|
|
163
|
+
|
|
164
|
+
No tool re-signs legacy rows with the canonical digest. This is deliberate,
|
|
165
|
+
and a follow-up if you want one. Re-sealing replaces an attestation made at
|
|
166
|
+
write time with one made today. It needs an operator to attest to the rows
|
|
167
|
+
first, and it has to keep the chain intact, because each row's successor
|
|
168
|
+
hashed the row's *old* checksum as its parent. A safe version would need to:
|
|
169
|
+
|
|
170
|
+
1. keep the original checksum (for example in a `legacy_checksum` column) so
|
|
171
|
+
the successor link can still be checked;
|
|
172
|
+
2. record who re-sealed which rows and when (a `resealed_at` stamp plus an
|
|
173
|
+
audit event);
|
|
174
|
+
3. re-seal only rows classified `:legacy_key_order_unverifiable`, never a
|
|
175
|
+
`:digest_mismatch`;
|
|
176
|
+
4. have `verify_chain` report re-sealed rows separately, so a re-seal never
|
|
177
|
+
reads as original evidence.
|
|
178
|
+
|
|
179
|
+
That is a schema change plus a policy decision, so it is left for a separate
|
|
180
|
+
release. `backfill_checksums!` is still only for rows that never had a
|
|
181
|
+
checksum; don't use it for this.
|
|
182
|
+
|
|
10
183
|
## [0.13.0] - 2026-09-24
|
|
11
184
|
|
|
12
185
|
The Phase 4 release. It removes what 0.12 deprecated, the empty engine
|
data/README.md
CHANGED
|
@@ -600,6 +600,13 @@ StandardAudit.configure(baseline: true) do |config|
|
|
|
600
600
|
# Defaults from STANDARD_AUDIT_RETENTION_DAYS (see Retention below); set here
|
|
601
601
|
# to override per app. Leave unset for infinite retention.
|
|
602
602
|
config.retention_days = 90
|
|
603
|
+
|
|
604
|
+
# -- Checksum cutover --
|
|
605
|
+
# Rows created at/after this time get the canonical (key-order-independent)
|
|
606
|
+
# checksum. Default StandardAudit::CANONICAL_CHECKSUM_CUTOVER (2026-10-01Z).
|
|
607
|
+
# Only move it if your rollout slips, only to a future time, and never after
|
|
608
|
+
# it has passed. See "Checksum algorithm versions".
|
|
609
|
+
# config.canonical_checksum_since = Time.utc(2026, 10, 15)
|
|
603
610
|
end
|
|
604
611
|
```
|
|
605
612
|
|
|
@@ -790,7 +797,8 @@ reproduce its own digest. Since 0.12.0 its stored `checksum` is left untouched
|
|
|
790
797
|
|
|
791
798
|
```ruby
|
|
792
799
|
StandardAudit::AuditLog.verify_chain
|
|
793
|
-
# => { valid: true, verified: 5576, recovered: 0,
|
|
800
|
+
# => { valid: true, verified: 5576, recovered: 0, reordered: 0, redacted: 1,
|
|
801
|
+
# legacy_unverifiable: 0, unverifiable: [], failures: [] }
|
|
794
802
|
```
|
|
795
803
|
|
|
796
804
|
A redacted row's declared parent is still checked, so deleting the row before
|
|
@@ -886,7 +894,8 @@ own digest, so editing it invalidates the row.
|
|
|
886
894
|
|
|
887
895
|
```ruby
|
|
888
896
|
result = StandardAudit::AuditLog.verify_chain
|
|
889
|
-
# => { valid: true, verified: 5577, recovered: 0,
|
|
897
|
+
# => { valid: true, verified: 5577, recovered: 0, reordered: 0, redacted: 0,
|
|
898
|
+
# legacy_unverifiable: 0, unverifiable: [], failures: [] }
|
|
890
899
|
```
|
|
891
900
|
|
|
892
901
|
`redacted` counts rows anonymized by `anonymize_actor!` — see "Anonymization
|
|
@@ -902,8 +911,13 @@ and the checksum chain".
|
|
|
902
911
|
`created_at`, so rows whose two timestamps disagree can leave a hole rather
|
|
903
912
|
than a pruned start, and a hole is reported — truthfully, since rows really
|
|
904
913
|
are missing.
|
|
914
|
+
Pre-cutover rows whose metadata key order is lost are listed separately in
|
|
915
|
+
`unverifiable` (`reason: :legacy_key_order_unverifiable`) — see "Checksum
|
|
916
|
+
algorithm versions".
|
|
905
917
|
- `recovered` counts rows with no `previous_checksum` whose parent had to be
|
|
906
918
|
found by searching back through recent digests — see below.
|
|
919
|
+
- `reordered`, `legacy_unverifiable` and `unverifiable` concern pre-cutover
|
|
920
|
+
rows only — see "Checksum algorithm versions".
|
|
907
921
|
- `verify_chain(scope: org)` skips the missing-parent check, because the log is
|
|
908
922
|
global and a scoped row's parent usually belongs to another scope.
|
|
909
923
|
|
|
@@ -914,6 +928,105 @@ previous strict-line reading did not actually detect insertions either — it
|
|
|
914
928
|
reported every concurrent append as tampering, which on one production log meant
|
|
915
929
|
67% of rows red and any real signal lost in the noise.
|
|
916
930
|
|
|
931
|
+
### Checksum algorithm versions
|
|
932
|
+
|
|
933
|
+
Up to 0.13.0 the digest hashed `metadata.to_json` in the order the Ruby hash
|
|
934
|
+
was built. PostgreSQL `jsonb` (and MySQL `JSON`) store object keys in their own
|
|
935
|
+
order — shortest first, then bytewise — so a row whose metadata had more than
|
|
936
|
+
one key could only be verified if its keys happened to be written in that
|
|
937
|
+
order. On one production log that was 67% of rows
|
|
938
|
+
([fundbright/delivery-ops#689](https://github.com/fundbright/delivery-ops/issues/689)).
|
|
939
|
+
SQLite keeps JSON as text in insertion order, so the gem's own suite never saw
|
|
940
|
+
it; CI now runs the suite on PostgreSQL too.
|
|
941
|
+
|
|
942
|
+
There are two algorithms, and **which one a row uses is decided by its
|
|
943
|
+
`created_at`** — nothing extra is stored and no migration is needed:
|
|
944
|
+
|
|
945
|
+
| Rows created | Algorithm | Digest |
|
|
946
|
+
|--------------|-----------|--------|
|
|
947
|
+
| before `config.canonical_checksum_since` | legacy | `SHA256("<parent>\|field=value\|…")`, Hash values via `to_json` in Ruby key order. Kept byte-for-byte so these rows verify as they were signed. |
|
|
948
|
+
| at or after it | canonical | `SHA256` of canonical JSON `{"fields": {…}, "previous_checksum": …, "v": 2}`: JSON values round-tripped exactly as the column stores them, object keys sorted bytewise at every depth, integral floats as integers, times as UTC ISO 8601 (µs), nil distinct from `""`, no separator ambiguity. |
|
|
949
|
+
|
|
950
|
+
`config.canonical_checksum_since` defaults to
|
|
951
|
+
`StandardAudit::CANONICAL_CHECKSUM_CUTOVER`, **2026-10-01T00:00:00Z**. The
|
|
952
|
+
switch happens by the clock, not by deploy:
|
|
953
|
+
|
|
954
|
+
> **Deploy 0.13.1 or later before the cutover.** A process still running an
|
|
955
|
+
> older gem after it writes legacy-hashed rows with post-cutover
|
|
956
|
+
> `created_at`s, and those fail strict canonical verification as
|
|
957
|
+
> `:digest_mismatch`. If your rollout will slip, set
|
|
958
|
+
> `config.canonical_checksum_since` to a later time **before** the default
|
|
959
|
+
> passes — and never move it once it has passed, because verification
|
|
960
|
+
> recomputes the same decision from each row's stored `created_at`.
|
|
961
|
+
|
|
962
|
+
```ruby
|
|
963
|
+
# config/initializers/standard_audit.rb — only if the rollout slips
|
|
964
|
+
StandardAudit.configure(baseline: true) do |config|
|
|
965
|
+
config.canonical_checksum_since = Time.utc(2026, 10, 15)
|
|
966
|
+
end
|
|
967
|
+
```
|
|
968
|
+
|
|
969
|
+
Rows written between upgrading and the cutover are still legacy-hashed, so a
|
|
970
|
+
multi-key row written then can still end up unverifiable (below). Deploying
|
|
971
|
+
early does not change that by itself. To switch sooner, set
|
|
972
|
+
`canonical_checksum_since` to a time that is still in the future and after
|
|
973
|
+
every process runs 0.13.1. Never set it to a time that has already passed,
|
|
974
|
+
because rows written since then would be re-judged under the other algorithm.
|
|
975
|
+
|
|
976
|
+
**How `verify_chain` judges each row:**
|
|
977
|
+
|
|
978
|
+
- **At or after the cutover:** the canonical digest only — declared parent,
|
|
979
|
+
else the preceding row, else the recovery search. A mismatch is
|
|
980
|
+
`:digest_mismatch`; these rows are never classified as legacy.
|
|
981
|
+
- **Before the cutover:** the legacy digest, exactly as before (declared
|
|
982
|
+
parent, preceding row, recovery search). A row that still does not
|
|
983
|
+
reproduce is then:
|
|
984
|
+
1. **Reconstructed.** The insertion order the legacy digest hashed cannot be
|
|
985
|
+
read back — jsonb's order depends only on the key set — but it can be
|
|
986
|
+
*searched*. If some ordering of the stored keys (at every depth), hashed
|
|
987
|
+
with the row's parent, reproduces the digest the row has held since it
|
|
988
|
+
was written, that is a witness in the same sense as the parent recovery
|
|
989
|
+
search: a row whose values were edited reproduces no ordering. Such rows
|
|
990
|
+
count in `reordered` and are valid. At most `key_order_search_limit:`
|
|
991
|
+
orderings per row (default 720, six keys in one object); orders learned
|
|
992
|
+
from earlier rows with the same keys are tried first, so most rows cost
|
|
993
|
+
one hash. Parents tried: the declared one, or else the preceding row and
|
|
994
|
+
"no parent" (not the 256-row window).
|
|
995
|
+
2. **`:digest_mismatch`** if that search was exhaustive against the parent
|
|
996
|
+
the row *declares* (no key order explains it), or if the row has no JSON
|
|
997
|
+
object with more than one key (key order cannot be why it fails).
|
|
998
|
+
3. **`:missing_parent`** if its declared parent is gone.
|
|
999
|
+
4. Otherwise **`:legacy_key_order_unverifiable`**.
|
|
1000
|
+
|
|
1001
|
+
**What `valid` means.** `valid` is `failures.empty?`. A
|
|
1002
|
+
`:legacy_key_order_unverifiable` row does **not** make the chain invalid on its
|
|
1003
|
+
own: it means "cannot be proven either way" — an edited legacy row looks
|
|
1004
|
+
exactly like one whose key order was lost — and the policy for such rows is
|
|
1005
|
+
the host's. It is never silent, though: `legacy_unverifiable` counts them and
|
|
1006
|
+
`unverifiable` lists them (same shape as a failure).
|
|
1007
|
+
|
|
1008
|
+
```ruby
|
|
1009
|
+
StandardAudit::AuditLog.verify_chain
|
|
1010
|
+
# => { valid: true, verified: 25910, recovered: 24, reordered: <n>, redacted: 0,
|
|
1011
|
+
# legacy_unverifiable: <m>, unverifiable: [...], failures: [] }
|
|
1012
|
+
|
|
1013
|
+
StandardAudit::AuditLog.verify_chain(fail_on_legacy_unverifiable: true)
|
|
1014
|
+
# => the same rows reported in failures, valid: false while any exist
|
|
1015
|
+
```
|
|
1016
|
+
|
|
1017
|
+
After the cutover no new legacy row can be written, so under honest operation
|
|
1018
|
+
`legacy_unverifiable` never grows. **Alert if it does:** it means a pre-cutover
|
|
1019
|
+
row was edited, or a row's `created_at` was moved back across the cutover
|
|
1020
|
+
(`created_at` is not itself hashed).
|
|
1021
|
+
|
|
1022
|
+
**Cost.** A canonical row costs one JSON round trip and one hash. A legacy row
|
|
1023
|
+
that needs the search costs up to `limit × 2` hashes the first time a key set
|
|
1024
|
+
is seen and usually one after that. Raise the limit (e.g. 5040 for seven keys)
|
|
1025
|
+
if your events carry wider metadata.
|
|
1026
|
+
|
|
1027
|
+
**Re-sealing legacy rows is not provided.** See the 0.13.1 CHANGELOG for why
|
|
1028
|
+
and for what a safe version would need.
|
|
1029
|
+
|
|
917
1030
|
### Rows written before 0.8.0
|
|
918
1031
|
|
|
919
1032
|
They have no `previous_checksum`. Verification falls back to the preceding row
|
|
@@ -962,6 +1075,8 @@ the 0.2 → 0.3 upgrade path, was removed in 0.13.0.)
|
|
|
962
1075
|
```bash
|
|
963
1076
|
# Verify chain integrity (exits non-zero on failures)
|
|
964
1077
|
rake standard_audit:verify
|
|
1078
|
+
# ...treating unreconstructable pre-cutover rows as failures too
|
|
1079
|
+
FAIL_ON_LEGACY_UNVERIFIABLE=1 rake standard_audit:verify
|
|
965
1080
|
|
|
966
1081
|
# Record the parent digest each existing row was signed against
|
|
967
1082
|
rake standard_audit:relink_checksums
|
|
@@ -1009,6 +1124,10 @@ For PostgreSQL, edit the generated migration to use `jsonb` instead of `json`:
|
|
|
1009
1124
|
t.jsonb :metadata, default: {}
|
|
1010
1125
|
```
|
|
1011
1126
|
|
|
1127
|
+
`jsonb` and MySQL `JSON` reorder object keys. Rows created since the
|
|
1128
|
+
canonical-checksum cutover do not depend on key order; earlier rows may — see
|
|
1129
|
+
"Checksum algorithm versions".
|
|
1130
|
+
|
|
1012
1131
|
## Best Practices
|
|
1013
1132
|
|
|
1014
1133
|
**What to audit**: Authentication events, data mutations, permission changes, financial transactions, admin actions, data exports, and API access from external services.
|
|
@@ -193,24 +193,28 @@ module StandardAudit
|
|
|
193
193
|
|
|
194
194
|
# Recomputes the checksum from the record's current field values and the
|
|
195
195
|
# given previous checksum. Useful for verification without saving.
|
|
196
|
-
|
|
196
|
+
# The algorithm (StandardAudit::Checksum::LEGACY or CANONICAL) follows the
|
|
197
|
+
# row's `created_at` against `config.canonical_checksum_since`; pass
|
|
198
|
+
# `version:` to force one.
|
|
199
|
+
def compute_checksum_value(previous_checksum: nil, version: nil)
|
|
197
200
|
self.class.compute_checksum_value(
|
|
198
201
|
attributes.slice(*CHECKSUM_FIELDS),
|
|
199
|
-
previous_checksum: previous_checksum
|
|
202
|
+
previous_checksum: previous_checksum,
|
|
203
|
+
version: version || StandardAudit::Checksum.algorithm_for(created_at)
|
|
200
204
|
)
|
|
201
205
|
end
|
|
202
206
|
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
value = value.utc.strftime("%Y-%m-%dT%H:%M:%S.%6NZ") if value.respond_to?(:strftime) && value.respond_to?(:utc)
|
|
208
|
-
"#{f}=#{value}"
|
|
209
|
-
}.join("|")
|
|
207
|
+
# `attrs` may carry "created_at" / :created_at, which picks the algorithm
|
|
208
|
+
# as for a stored row; without it, the algorithm a write now would use.
|
|
209
|
+
def self.compute_checksum_value(attrs, previous_checksum: nil, version: nil)
|
|
210
|
+
version ||= StandardAudit::Checksum.algorithm_for(attrs["created_at"] || attrs[:created_at])
|
|
210
211
|
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
212
|
+
StandardAudit::Checksum.digest(
|
|
213
|
+
attrs,
|
|
214
|
+
fields: CHECKSUM_FIELDS,
|
|
215
|
+
previous_checksum: previous_checksum,
|
|
216
|
+
version: version
|
|
217
|
+
)
|
|
214
218
|
end
|
|
215
219
|
|
|
216
220
|
# Runs the configured `before_checksum` hooks against a row that will be
|
|
@@ -240,10 +244,65 @@ module StandardAudit
|
|
|
240
244
|
column_names.include?("previous_checksum")
|
|
241
245
|
end
|
|
242
246
|
|
|
243
|
-
# Verifies the integrity of the audit log. Returns a result hash
|
|
244
|
-
#
|
|
245
|
-
#
|
|
246
|
-
#
|
|
247
|
+
# Verifies the integrity of the audit log. Returns a result hash:
|
|
248
|
+
#
|
|
249
|
+
# valid: true when :failures is empty (see "Legacy rows")
|
|
250
|
+
# verified: rows whose digest was checked (every checksummed,
|
|
251
|
+
# non-anonymized row)
|
|
252
|
+
# recovered: rows verified against a searched-for parent
|
|
253
|
+
# reordered: legacy rows verified by reconstructing the
|
|
254
|
+
# metadata key order they were signed with
|
|
255
|
+
# redacted: anonymized rows (not digest-checked)
|
|
256
|
+
# legacy_unverifiable: count of :unverifiable
|
|
257
|
+
# unverifiable: legacy rows whose digest can't be reproduced
|
|
258
|
+
# because the key order they were signed with is
|
|
259
|
+
# lost — same shape as a failure, reason
|
|
260
|
+
# :legacy_key_order_unverifiable
|
|
261
|
+
# failures: hashes with :id, :event_type, :created_at,
|
|
262
|
+
# :expected, :actual and :reason
|
|
263
|
+
#
|
|
264
|
+
# == Checksum algorithms and the cutover
|
|
265
|
+
#
|
|
266
|
+
# A row created at or after `config.canonical_checksum_since` (default
|
|
267
|
+
# StandardAudit::CANONICAL_CHECKSUM_CUTOVER) is signed with the canonical
|
|
268
|
+
# digest, which does not depend on how the database orders JSON object
|
|
269
|
+
# keys, and is verified STRICTLY with it: a mismatch is :digest_mismatch,
|
|
270
|
+
# never a legacy classification. An earlier row is legacy: its digest
|
|
271
|
+
# hashed `metadata.to_json` in Ruby insertion order, which `jsonb`
|
|
272
|
+
# discards (fundbright/delivery-ops#689). See StandardAudit::Checksum.
|
|
273
|
+
# The decision is recomputed from each row's stored `created_at`, the
|
|
274
|
+
# same value the writer used.
|
|
275
|
+
#
|
|
276
|
+
# == Legacy rows
|
|
277
|
+
#
|
|
278
|
+
# A legacy row is checked with the legacy digest, exactly as before —
|
|
279
|
+
# declared parent, else the preceding row, else the recovery search
|
|
280
|
+
# below. One that still does not reproduce is, in this order:
|
|
281
|
+
#
|
|
282
|
+
# 1. searched for the metadata key order it was signed with. A key order
|
|
283
|
+
# that reproduces the stored digest is a witness, like the parent
|
|
284
|
+
# search; the row counts in :reordered and is valid. Bounded by
|
|
285
|
+
# `key_order_search_limit` orderings per row
|
|
286
|
+
# (Checksum::KeyOrderSearch), against the declared parent or else the
|
|
287
|
+
# preceding row and "no parent";
|
|
288
|
+
# 2. reported `:digest_mismatch` when that search was EXHAUSTIVE against
|
|
289
|
+
# the parent the row declares (no key order explains it), or when the
|
|
290
|
+
# row has no JSON object with more than one key (key order cannot be
|
|
291
|
+
# why it fails);
|
|
292
|
+
# 3. reported `:missing_parent` when its declared parent is absent;
|
|
293
|
+
# 4. otherwise listed in :unverifiable with reason
|
|
294
|
+
# `:legacy_key_order_unverifiable`.
|
|
295
|
+
#
|
|
296
|
+
# `:legacy_key_order_unverifiable` means "cannot be proven either way":
|
|
297
|
+
# an edited legacy row looks exactly like one whose key order was lost.
|
|
298
|
+
# Such rows do NOT make `valid` false on their own — the gem cannot tell,
|
|
299
|
+
# and the policy for them belongs to the host — but they are never
|
|
300
|
+
# silent: they are counted in :legacy_unverifiable and listed in
|
|
301
|
+
# :unverifiable. Pass `fail_on_legacy_unverifiable: true` to report them
|
|
302
|
+
# as failures instead. After the cutover no new legacy row can be
|
|
303
|
+
# written, so under honest operation :legacy_unverifiable never grows;
|
|
304
|
+
# alert if it does (an edited pre-cutover row, or a `created_at` moved
|
|
305
|
+
# back across the cutover).
|
|
247
306
|
#
|
|
248
307
|
# A row stamped `anonymized_at` (GDPR erasure via `anonymize_actor!`) is
|
|
249
308
|
# counted in :redacted instead of being digest-checked: its checksummed
|
|
@@ -273,7 +332,12 @@ module StandardAudit
|
|
|
273
332
|
# actually reproduces this row's checksum. That recovers the true parent
|
|
274
333
|
# of a forked row without re-signing anything. It does not weaken tamper
|
|
275
334
|
# detection: a row whose fields were altered reproduces no candidate's
|
|
276
|
-
# digest, so it still fails.
|
|
335
|
+
# digest, so it still fails. (The key-order search is not combined with
|
|
336
|
+
# this window search, so a legacy row that was both forked and
|
|
337
|
+
# reordered stays unverifiable.)
|
|
338
|
+
#
|
|
339
|
+
# The chain links across the cutover like anywhere else: the first
|
|
340
|
+
# canonical row's parent is the last legacy row's stored checksum.
|
|
277
341
|
#
|
|
278
342
|
# A row whose parent digest is absent from the log is reported with
|
|
279
343
|
# `reason: :missing_parent` — a row was removed. Two exemptions:
|
|
@@ -291,20 +355,46 @@ module StandardAudit
|
|
|
291
355
|
# `created_at`. Rows whose two timestamps disagree (a backdated
|
|
292
356
|
# `occurred_at`) can leave a hole rather than a prefix, and a hole is
|
|
293
357
|
# reported — truthfully, since rows really are missing.
|
|
294
|
-
def self.verify_chain(scope: nil, batch_size: 1000, recovery_window: 256, strict: false
|
|
358
|
+
def self.verify_chain(scope: nil, batch_size: 1000, recovery_window: 256, strict: false,
|
|
359
|
+
key_order_search_limit: StandardAudit::Checksum::KeyOrderSearch::DEFAULT_LIMIT,
|
|
360
|
+
fail_on_legacy_unverifiable: false)
|
|
295
361
|
relation = scope ? where(scope_gid: scope.to_global_id.to_s) : all
|
|
296
362
|
check_parents = scope.nil?
|
|
297
363
|
declared_parents = chain_parent_column?
|
|
364
|
+
key_orders = StandardAudit::Checksum::KeyOrderSearch.new(limit: key_order_search_limit)
|
|
298
365
|
|
|
299
366
|
previous_checksum = nil
|
|
300
367
|
verified = 0
|
|
301
368
|
recovered = 0
|
|
369
|
+
reordered = 0
|
|
302
370
|
redacted = 0
|
|
371
|
+
unverifiable = []
|
|
303
372
|
failures = []
|
|
304
373
|
window = []
|
|
305
374
|
first_row = true
|
|
306
375
|
pruned_parents = []
|
|
307
376
|
|
|
377
|
+
# Records a :missing_parent failure when `declared` is absent from the
|
|
378
|
+
# log, unless the walk opened on it (a pruned start). True when reported.
|
|
379
|
+
report_missing_parent = lambda do |record, declared, expected|
|
|
380
|
+
next false unless declared.present? && check_parents && !parent_present?(declared, window, relation)
|
|
381
|
+
|
|
382
|
+
if first_row
|
|
383
|
+
# The walk opens on a row whose parent is already gone, so the log
|
|
384
|
+
# has had its start removed — retention pruning, typically. That
|
|
385
|
+
# parent is unknowable, and it can have several children (which is
|
|
386
|
+
# what a concurrent append leaves behind), so the exemption is
|
|
387
|
+
# remembered per digest rather than for one row.
|
|
388
|
+
pruned_parents << declared
|
|
389
|
+
false
|
|
390
|
+
elsif pruned_parents.include?(declared)
|
|
391
|
+
false
|
|
392
|
+
else
|
|
393
|
+
failures << chain_failure(record, expected: expected, reason: :missing_parent)
|
|
394
|
+
true
|
|
395
|
+
end
|
|
396
|
+
end
|
|
397
|
+
|
|
308
398
|
each_in_chain_order(relation, batch_size: batch_size) do |record|
|
|
309
399
|
if record.checksum.blank?
|
|
310
400
|
previous_checksum = nil
|
|
@@ -312,44 +402,45 @@ module StandardAudit
|
|
|
312
402
|
next
|
|
313
403
|
end
|
|
314
404
|
|
|
315
|
-
declared = record.previous_checksum
|
|
405
|
+
declared = declared_parents ? record.previous_checksum : nil
|
|
316
406
|
|
|
317
407
|
if record.anonymized?
|
|
318
408
|
redacted += 1
|
|
319
409
|
# The digest cannot be checked, but the parent it declares can.
|
|
320
|
-
|
|
321
|
-
if first_row
|
|
322
|
-
pruned_parents << declared
|
|
323
|
-
elsif !pruned_parents.include?(declared)
|
|
324
|
-
failures << chain_failure(record, expected: nil, reason: :missing_parent)
|
|
325
|
-
end
|
|
326
|
-
end
|
|
327
|
-
elsif declared.present?
|
|
328
|
-
verified += 1
|
|
329
|
-
expected = record.compute_checksum_value(previous_checksum: declared)
|
|
330
|
-
|
|
331
|
-
if record.checksum != expected
|
|
332
|
-
failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
|
|
333
|
-
elsif check_parents && !parent_present?(declared, window, relation)
|
|
334
|
-
if first_row
|
|
335
|
-
# The walk opens on a row whose parent is already gone, so the
|
|
336
|
-
# log has had its start removed — retention pruning, typically.
|
|
337
|
-
# That parent is unknowable, and it can have several children
|
|
338
|
-
# (which is what a concurrent append leaves behind), so the
|
|
339
|
-
# exemption is remembered per digest rather than for one row.
|
|
340
|
-
pruned_parents << declared
|
|
341
|
-
elsif !pruned_parents.include?(declared)
|
|
342
|
-
failures << chain_failure(record, expected: expected, reason: :missing_parent)
|
|
343
|
-
end
|
|
344
|
-
end
|
|
410
|
+
report_missing_parent.call(record, declared, nil)
|
|
345
411
|
else
|
|
346
412
|
verified += 1
|
|
347
|
-
|
|
413
|
+
algorithm = StandardAudit::Checksum.algorithm_for(record.created_at)
|
|
414
|
+
parent = declared.presence || previous_checksum
|
|
415
|
+
expected = walk_digester(record, algorithm).call(parent)
|
|
348
416
|
|
|
349
417
|
if record.checksum == expected
|
|
350
|
-
# Links to the row before it
|
|
351
|
-
|
|
418
|
+
# Links to its declared parent, or to the row before it as a
|
|
419
|
+
# linear chain does.
|
|
420
|
+
report_missing_parent.call(record, declared, expected)
|
|
421
|
+
elsif declared.blank? && !strict && recover_parent(record, window, algorithm)
|
|
352
422
|
recovered += 1
|
|
423
|
+
elsif algorithm == StandardAudit::Checksum::LEGACY && legacy_key_order_ambiguous?(record)
|
|
424
|
+
attrs = record.attributes.slice(*CHECKSUM_FIELDS)
|
|
425
|
+
parents = declared.present? || strict ? [parent] : [parent, nil].uniq
|
|
426
|
+
|
|
427
|
+
if key_orders.search(attrs, fields: CHECKSUM_FIELDS, checksum: record.checksum, parents: parents)
|
|
428
|
+
reordered += 1
|
|
429
|
+
# The row verified under a reconstructed order, so the stored-
|
|
430
|
+
# order `expected` means nothing for a missing-parent report.
|
|
431
|
+
report_missing_parent.call(record, declared, nil)
|
|
432
|
+
elsif declared.present? && key_orders.exhaustive?(attrs, fields: CHECKSUM_FIELDS)
|
|
433
|
+
# Every key order was tried against the parent the row itself
|
|
434
|
+
# declares, so key order does not explain this row.
|
|
435
|
+
failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
|
|
436
|
+
elsif report_missing_parent.call(record, declared, nil)
|
|
437
|
+
# Reported as a removed row, which is the stronger finding. No
|
|
438
|
+
# `expected`: no digest was established for this row.
|
|
439
|
+
else
|
|
440
|
+
entry = chain_failure(record, expected: expected, reason: :legacy_key_order_unverifiable)
|
|
441
|
+
unverifiable << entry
|
|
442
|
+
failures << entry if fail_on_legacy_unverifiable
|
|
443
|
+
end
|
|
353
444
|
else
|
|
354
445
|
failures << chain_failure(record, expected: expected, reason: :digest_mismatch)
|
|
355
446
|
end
|
|
@@ -361,7 +452,16 @@ module StandardAudit
|
|
|
361
452
|
window.shift if window.size > recovery_window
|
|
362
453
|
end
|
|
363
454
|
|
|
364
|
-
{
|
|
455
|
+
{
|
|
456
|
+
valid: failures.empty?,
|
|
457
|
+
verified: verified,
|
|
458
|
+
recovered: recovered,
|
|
459
|
+
reordered: reordered,
|
|
460
|
+
redacted: redacted,
|
|
461
|
+
legacy_unverifiable: unverifiable.size,
|
|
462
|
+
unverifiable: unverifiable,
|
|
463
|
+
failures: failures
|
|
464
|
+
}
|
|
365
465
|
end
|
|
366
466
|
|
|
367
467
|
# Records, for every row that does not already carry one, the parent digest
|
|
@@ -426,28 +526,54 @@ module StandardAudit
|
|
|
426
526
|
private_class_method :chain_failure
|
|
427
527
|
|
|
428
528
|
# Searches `window` (most recent first, then "no parent at all") for the
|
|
429
|
-
# digest that reproduces the record's stored checksum
|
|
430
|
-
# array holding the parent — which may
|
|
431
|
-
# against an empty table — or nil when
|
|
529
|
+
# digest that reproduces the record's stored checksum under `algorithm`.
|
|
530
|
+
# Returns a one-element array holding the parent — which may
|
|
531
|
+
# itself be nil, for a row written against an empty table — or nil when
|
|
532
|
+
# nothing reproduces the digest.
|
|
432
533
|
#
|
|
433
534
|
# SHA-256 preimage resistance is what makes this safe: a row whose fields
|
|
434
535
|
# were altered reproduces no candidate's digest, so it is still reported.
|
|
435
|
-
def self.recover_parent(record, window)
|
|
536
|
+
def self.recover_parent(record, window, algorithm)
|
|
436
537
|
window.reverse_each do |candidate|
|
|
437
|
-
return [candidate] if record
|
|
538
|
+
return [candidate] if digest_matches?(record, candidate, algorithm)
|
|
438
539
|
end
|
|
439
540
|
|
|
440
|
-
[nil] if record
|
|
541
|
+
[nil] if digest_matches?(record, nil, algorithm)
|
|
441
542
|
end
|
|
442
543
|
private_class_method :recover_parent
|
|
443
544
|
|
|
444
545
|
def self.resolve_parent(record, previous_checksum, window)
|
|
445
|
-
|
|
546
|
+
algorithm = StandardAudit::Checksum.algorithm_for(record.created_at)
|
|
547
|
+
return [previous_checksum] if digest_matches?(record, previous_checksum, algorithm)
|
|
446
548
|
|
|
447
|
-
recover_parent(record, window)
|
|
549
|
+
recover_parent(record, window, algorithm)
|
|
448
550
|
end
|
|
449
551
|
private_class_method :resolve_parent
|
|
450
552
|
|
|
553
|
+
def self.digest_matches?(record, parent, algorithm)
|
|
554
|
+
record.checksum == walk_digester(record, algorithm).call(parent)
|
|
555
|
+
end
|
|
556
|
+
private_class_method :digest_matches?
|
|
557
|
+
|
|
558
|
+
# The row's `parent -> digest` function for `algorithm`, memoised on the
|
|
559
|
+
# loaded record: the parent searches try hundreds of parents per row, and
|
|
560
|
+
# this makes each try one SHA-256 instead of re-serialising the row. Only
|
|
561
|
+
# for records the walk loaded and never mutates.
|
|
562
|
+
def self.walk_digester(record, algorithm)
|
|
563
|
+
memo = record.instance_variable_get(:@walk_digesters) || record.instance_variable_set(:@walk_digesters, {})
|
|
564
|
+
memo[algorithm] ||= StandardAudit::Checksum.digester(
|
|
565
|
+
record.attributes.slice(*CHECKSUM_FIELDS), fields: CHECKSUM_FIELDS, version: algorithm
|
|
566
|
+
)
|
|
567
|
+
end
|
|
568
|
+
private_class_method :walk_digester
|
|
569
|
+
|
|
570
|
+
# True when the row's hashed JSON could have been reordered by the store:
|
|
571
|
+
# some object in it has more than one key.
|
|
572
|
+
def self.legacy_key_order_ambiguous?(record)
|
|
573
|
+
CHECKSUM_FIELDS.any? { |f| StandardAudit::Checksum.key_order_ambiguous?(record[f]) }
|
|
574
|
+
end
|
|
575
|
+
private_class_method :legacy_key_order_ambiguous?
|
|
576
|
+
|
|
451
577
|
def self.parent_present?(digest, window, relation)
|
|
452
578
|
window.include?(digest) || relation.exists?(checksum: digest)
|
|
453
579
|
end
|
|
@@ -465,10 +591,8 @@ module StandardAudit
|
|
|
465
591
|
next
|
|
466
592
|
end
|
|
467
593
|
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
previous_checksum: previous_checksum
|
|
471
|
-
)
|
|
594
|
+
# The row's own created_at picks the algorithm, as verification will.
|
|
595
|
+
new_checksum = record.compute_checksum_value(previous_checksum: previous_checksum)
|
|
472
596
|
columns = { checksum: new_checksum }
|
|
473
597
|
columns[:previous_checksum] = previous_checksum if chain_parent_column?
|
|
474
598
|
record.update_columns(columns)
|
|
@@ -550,7 +674,12 @@ module StandardAudit
|
|
|
550
674
|
#
|
|
551
675
|
# `previous_checksum` needs no protection of its own: it is an input to
|
|
552
676
|
# this row's own digest, so editing it invalidates the row.
|
|
677
|
+
#
|
|
678
|
+
# The algorithm is chosen from `created_at`, which is fixed here (Active
|
|
679
|
+
# Record keeps a timestamp that is already set) so the decision is made
|
|
680
|
+
# from exactly the value that is stored and that verification re-reads.
|
|
553
681
|
def compute_checksum
|
|
682
|
+
self.created_at ||= Time.current if has_attribute?(:created_at)
|
|
554
683
|
previous = self.class.chain_tip_checksum
|
|
555
684
|
self.previous_checksum = previous if self.class.chain_parent_column?
|
|
556
685
|
self.checksum = compute_checksum_value(previous_checksum: previous)
|
|
@@ -0,0 +1,153 @@
|
|
|
1
|
+
module StandardAudit
|
|
2
|
+
module Checksum
|
|
3
|
+
# Reconstructs the key order a legacy row was signed with.
|
|
4
|
+
#
|
|
5
|
+
# A legacy digest covers `metadata.to_json` in the writer's Ruby
|
|
6
|
+
# insertion order. A `jsonb` column has since discarded that order, and it
|
|
7
|
+
# cannot be derived from the stored row: the storage order is a function
|
|
8
|
+
# of the key set alone, so every insertion order maps to the same stored
|
|
9
|
+
# value. What CAN be done is to try the orders. If some ordering of the
|
|
10
|
+
# stored keys, hashed with the row's own parent, reproduces the digest the
|
|
11
|
+
# row has held since it was written, that is a witness in the same sense
|
|
12
|
+
# as the parent recovery search: SHA-256 preimage resistance means a row
|
|
13
|
+
# whose values were altered reproduces no ordering. Trying N orderings
|
|
14
|
+
# costs log2(N) bits of a 256-bit margin.
|
|
15
|
+
#
|
|
16
|
+
# The search is bounded. The number of orderings is the product of `n!`
|
|
17
|
+
# over every object in the value, so it is only enumerated in full when
|
|
18
|
+
# that product is at most `limit`. Orders that reproduced an earlier row
|
|
19
|
+
# with the same key structure are remembered and tried first, because a
|
|
20
|
+
# given event is almost always built by the same code path in the same
|
|
21
|
+
# order — that is what makes the search cheap across a whole log.
|
|
22
|
+
#
|
|
23
|
+
# One instance lives for one verify_chain walk. It never writes anything.
|
|
24
|
+
class KeyOrderSearch
|
|
25
|
+
DEFAULT_LIMIT = 720 # 6 keys in one object, or e.g. 3 × 3! nested
|
|
26
|
+
REMEMBERED_ORDERS = 8
|
|
27
|
+
|
|
28
|
+
attr_reader :limit
|
|
29
|
+
|
|
30
|
+
def initialize(limit: DEFAULT_LIMIT)
|
|
31
|
+
@limit = limit.to_i
|
|
32
|
+
@learned = Hash.new { |h, k| h[k] = [] }
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# Searches the orderings of every Hash-valued field in `attrs` for one
|
|
36
|
+
# that reproduces `checksum` under the legacy digest with one of `parents`.
|
|
37
|
+
# Returns `{ parent:, attrs: }` on a match, nil otherwise.
|
|
38
|
+
def search(attrs, fields:, checksum:, parents:)
|
|
39
|
+
hashed = fields.select { |f| attrs[f].is_a?(Hash) }
|
|
40
|
+
return nil if hashed.empty? || limit <= 0
|
|
41
|
+
|
|
42
|
+
signature = hashed.map { |f| [f, self.class.signature(attrs[f])] }
|
|
43
|
+
|
|
44
|
+
# Remembered orders first: one try each, however large the object.
|
|
45
|
+
@learned[signature].dup.each do |templates|
|
|
46
|
+
candidate = attrs.merge(hashed.zip(templates).to_h { |f, t| [f, self.class.apply(t, attrs[f])] })
|
|
47
|
+
match = try(candidate, fields, checksum, parents)
|
|
48
|
+
return remember(signature, hashed, candidate, match) if match
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
return nil unless exhaustive?(attrs, fields: fields)
|
|
52
|
+
|
|
53
|
+
per_field = hashed.map { |f| self.class.variants(attrs[f]) }
|
|
54
|
+
combos = per_field.first.product(*per_field.drop(1))
|
|
55
|
+
|
|
56
|
+
combos.each do |values|
|
|
57
|
+
candidate = attrs.merge(hashed.zip(values).to_h)
|
|
58
|
+
match = try(candidate, fields, checksum, parents)
|
|
59
|
+
return remember(signature, hashed, candidate, match) if match
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
nil
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
# True when `search` enumerates EVERY ordering of this row, so a miss
|
|
66
|
+
# means no key order explains the digest — with a known parent, the row
|
|
67
|
+
# does not match its signed content.
|
|
68
|
+
def exhaustive?(attrs, fields:)
|
|
69
|
+
count = fields.select { |f| attrs[f].is_a?(Hash) }.reduce(1) { |acc, f| acc * self.class.variant_count(attrs[f]) }
|
|
70
|
+
count <= limit
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
class << self
|
|
74
|
+
def variant_count(node)
|
|
75
|
+
case node
|
|
76
|
+
when Hash then (1..node.size).reduce(1, :*) * node.each_value.reduce(1) { |acc, v| acc * variant_count(v) }
|
|
77
|
+
when Array then node.reduce(1) { |acc, v| acc * variant_count(v) }
|
|
78
|
+
else 1
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Every ordering of every object inside `node`. The first is the
|
|
83
|
+
# stored order. Callers bound the size with variant_count.
|
|
84
|
+
def variants(node)
|
|
85
|
+
case node
|
|
86
|
+
when Hash
|
|
87
|
+
keys = node.keys
|
|
88
|
+
children = keys.to_h { |k| [k, variants(node[k])] }
|
|
89
|
+
keys.permutation.flat_map do |perm|
|
|
90
|
+
choices = perm.map { |k| children[k] }
|
|
91
|
+
product(choices).map { |values| perm.zip(values).to_h }
|
|
92
|
+
end
|
|
93
|
+
when Array
|
|
94
|
+
product(node.map { |v| variants(v) })
|
|
95
|
+
else
|
|
96
|
+
[node]
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
# The key structure of a value, independent of order.
|
|
101
|
+
def signature(node)
|
|
102
|
+
case node
|
|
103
|
+
when Hash then [:h, node.keys.map(&:to_s).sort.map { |k| [k, signature(node[k])] }]
|
|
104
|
+
when Array then [:a, node.map { |v| signature(v) }]
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# The ordering of `node`, as a template `apply` can replay.
|
|
109
|
+
def template(node)
|
|
110
|
+
case node
|
|
111
|
+
when Hash then [:h, node.map { |k, v| [k, template(v)] }]
|
|
112
|
+
when Array then [:a, node.map { |v| template(v) }]
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def apply(template, node)
|
|
117
|
+
case template&.first
|
|
118
|
+
when :h then template.last.to_h { |k, t| [k, apply(t, node[k])] }
|
|
119
|
+
when :a then node.each_with_index.map { |v, i| apply(template.last[i], v) }
|
|
120
|
+
else node
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
private
|
|
125
|
+
|
|
126
|
+
def product(lists)
|
|
127
|
+
return [[]] if lists.empty?
|
|
128
|
+
|
|
129
|
+
lists.first.product(*lists.drop(1))
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
private
|
|
134
|
+
|
|
135
|
+
def try(candidate, fields, checksum, parents)
|
|
136
|
+
parents.each do |parent|
|
|
137
|
+
digest = Checksum.legacy_digest(candidate, fields: fields, previous_checksum: parent)
|
|
138
|
+
return { parent: parent } if digest == checksum
|
|
139
|
+
end
|
|
140
|
+
nil
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
def remember(signature, hashed, candidate, match)
|
|
144
|
+
templates = hashed.map { |f| self.class.template(candidate[f]) }
|
|
145
|
+
list = @learned[signature]
|
|
146
|
+
list.delete(templates)
|
|
147
|
+
list.unshift(templates)
|
|
148
|
+
list.pop while list.size > REMEMBERED_ORDERS
|
|
149
|
+
match.merge(attrs: candidate)
|
|
150
|
+
end
|
|
151
|
+
end
|
|
152
|
+
end
|
|
153
|
+
end
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
require "json"
|
|
2
|
+
require "openssl"
|
|
3
|
+
|
|
4
|
+
module StandardAudit
|
|
5
|
+
# The row digest. There are two algorithms; which one a row uses is decided
|
|
6
|
+
# by its `created_at` against `config.canonical_checksum_since` (default
|
|
7
|
+
# StandardAudit::CANONICAL_CHECKSUM_CUTOVER), at write time and again at
|
|
8
|
+
# verification, from the same stored timestamp. Nothing extra is stored.
|
|
9
|
+
#
|
|
10
|
+
# == Legacy (rows created before the cutover)
|
|
11
|
+
#
|
|
12
|
+
# `SHA256("<parent>|field=value|field=value|…")`, where a Hash value was
|
|
13
|
+
# serialised with `to_json` in whatever key order the Ruby hash had. That is
|
|
14
|
+
# the bug in fundbright/delivery-ops#689: at write time the hash is the one
|
|
15
|
+
# the caller built (insertion order), but PostgreSQL `jsonb` — and MySQL
|
|
16
|
+
# `JSON` — store object keys in their own order (shortest first, then
|
|
17
|
+
# bytewise), so the value read back serialises differently and the digest
|
|
18
|
+
# cannot be reproduced. SQLite keeps JSON as text, so the gem's own suite
|
|
19
|
+
# never saw it. Kept byte-for-byte so legacy rows are verified exactly as
|
|
20
|
+
# they were signed.
|
|
21
|
+
#
|
|
22
|
+
# == Canonical (rows created at or after the cutover)
|
|
23
|
+
#
|
|
24
|
+
# `SHA256(canonical_json({"fields" => {…}, "previous_checksum" => …, "v" => 2}))`.
|
|
25
|
+
# Every input is reduced to a form the database cannot change:
|
|
26
|
+
#
|
|
27
|
+
# * Hash / Array values (the `metadata` jsonb column, or any other JSON
|
|
28
|
+
# column a host adds to the hashed set) are first round-tripped through
|
|
29
|
+
# `ActiveSupport::JSON.encode` + `JSON.parse` — the same encoding the
|
|
30
|
+
# column type applies on write — so symbol keys, Time values, BigDecimals
|
|
31
|
+
# and the like hash as the JSON the database actually stores. Object keys
|
|
32
|
+
# are then sorted bytewise at every depth; array order is kept (arrays are
|
|
33
|
+
# ordered in every JSON store).
|
|
34
|
+
# * Integral floats hash as integers (`1.0` → `1`, `1e20` →
|
|
35
|
+
# `100000000000000000000`), because a JSON store is free to hand back
|
|
36
|
+
# either spelling of the same number.
|
|
37
|
+
# * Times hash as UTC ISO 8601 with microseconds, as in the legacy digest.
|
|
38
|
+
# * Strings are escaped at the byte level (`"`, `\` and C0 controls only),
|
|
39
|
+
# so the output does not depend on the json gem's escaping options.
|
|
40
|
+
# * nil stays `null`, distinct from `""` — the legacy digest conflated them.
|
|
41
|
+
# * Fields are encoded as a JSON object rather than joined with `|`, so a
|
|
42
|
+
# value containing `|field=` can no longer move content between adjacent
|
|
43
|
+
# fields without changing the digest.
|
|
44
|
+
#
|
|
45
|
+
# Known limit: a host with `ActiveSupport.parse_json_times = true` reads
|
|
46
|
+
# ISO 8601 strings in JSON back as Time objects, which re-encode at
|
|
47
|
+
# millisecond precision in the app's zone. A metadata string that was not
|
|
48
|
+
# already in that exact form would then hash differently on read. The
|
|
49
|
+
# setting is off by default.
|
|
50
|
+
module Checksum
|
|
51
|
+
LEGACY = 1
|
|
52
|
+
CANONICAL = 2
|
|
53
|
+
ALGORITHMS = [LEGACY, CANONICAL].freeze
|
|
54
|
+
|
|
55
|
+
TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%6NZ".freeze
|
|
56
|
+
ESCAPE = /["\\\x00-\x1f]/n
|
|
57
|
+
|
|
58
|
+
module_function
|
|
59
|
+
|
|
60
|
+
# The algorithm for a row created at `created_at`: canonical at or after
|
|
61
|
+
# `config.canonical_checksum_since`, legacy before it. A row not yet
|
|
62
|
+
# timestamped is judged by the current time — what a write now would use.
|
|
63
|
+
def algorithm_for(created_at)
|
|
64
|
+
created_at ||= Time.current
|
|
65
|
+
created_at >= StandardAudit.config.canonical_checksum_since ? CANONICAL : LEGACY
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def digest(attrs, fields:, previous_checksum: nil, version:)
|
|
69
|
+
digester(attrs, fields: fields, version: version).call(previous_checksum)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# A callable `parent -> digest` for one row. The row's own serialisation
|
|
73
|
+
# is built once, so trying many candidate parents (the recovery search)
|
|
74
|
+
# costs one SHA-256 each.
|
|
75
|
+
def digester(attrs, fields:, version:)
|
|
76
|
+
case version
|
|
77
|
+
when LEGACY then legacy_digester(attrs, fields: fields)
|
|
78
|
+
when CANONICAL then canonical_digester(attrs, fields: fields)
|
|
79
|
+
else raise ArgumentError, "unknown checksum algorithm #{version.inspect} (known: #{ALGORITHMS.join(", ")})"
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
# The legacy digest, unchanged since 0.3. Do not "fix" this: it is how
|
|
84
|
+
# every pre-cutover row is signed, and it must keep reproducing them.
|
|
85
|
+
def legacy_digest(attrs, fields:, previous_checksum: nil)
|
|
86
|
+
legacy_digester(attrs, fields: fields).call(previous_checksum)
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def legacy_digester(attrs, fields:)
|
|
90
|
+
canonical = fields.map { |f|
|
|
91
|
+
value = attrs[f]
|
|
92
|
+
value = value.to_json if value.is_a?(Hash)
|
|
93
|
+
value = value.utc.strftime(TIME_FORMAT) if time_like?(value)
|
|
94
|
+
"#{f}=#{value}"
|
|
95
|
+
}.join("|")
|
|
96
|
+
|
|
97
|
+
lambda do |previous_checksum|
|
|
98
|
+
input = previous_checksum.present? ? "#{previous_checksum}|#{canonical}" : canonical
|
|
99
|
+
OpenSSL::Digest::SHA256.hexdigest(input)
|
|
100
|
+
end
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
def canonical_digest(attrs, fields:, previous_checksum: nil)
|
|
104
|
+
canonical_digester(attrs, fields: fields).call(previous_checksum)
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
# SHA-256 of canonical_json(canonical_payload(...)). The payload's keys
|
|
108
|
+
# sort as "fields" < "previous_checksum" < "v", so the row's part is
|
|
109
|
+
# serialised once and only the parent is spliced in per call — the
|
|
110
|
+
# result is byte-identical to serialising the whole payload.
|
|
111
|
+
def canonical_digester(attrs, fields:)
|
|
112
|
+
head = String.new("{\"fields\":", encoding: Encoding::BINARY)
|
|
113
|
+
canonical_json(fields.to_h { |f| [f, canonical_value(attrs[f])] }, head)
|
|
114
|
+
head << ",\"previous_checksum\":"
|
|
115
|
+
tail = ",\"v\":#{CANONICAL}}"
|
|
116
|
+
|
|
117
|
+
lambda do |previous_checksum|
|
|
118
|
+
parent = previous_checksum.presence
|
|
119
|
+
OpenSSL::Digest::SHA256.hexdigest(head + (parent ? json_string(parent.to_s, String.new(encoding: Encoding::BINARY)) : "null") + tail)
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
# The documented canonical payload for a row ("v" => 2 names the format), as a plain Hash.
|
|
124
|
+
def canonical_payload(attrs, fields:, previous_checksum: nil)
|
|
125
|
+
{
|
|
126
|
+
"fields" => fields.to_h { |f| [f, canonical_value(attrs[f])] },
|
|
127
|
+
"previous_checksum" => previous_checksum.presence,
|
|
128
|
+
"v" => CANONICAL
|
|
129
|
+
}
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
# The stored-form value of one field, before canonical encoding.
|
|
133
|
+
def canonical_value(value)
|
|
134
|
+
case value
|
|
135
|
+
when nil, String, Integer, Float, true, false then value
|
|
136
|
+
when Hash, Array then json_round_trip(value)
|
|
137
|
+
else time_like?(value) ? value.utc.strftime(TIME_FORMAT) : value.to_s
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
# What a JSON column hands back for `value`: encoded exactly as
|
|
142
|
+
# ActiveRecord's JSON type encodes on write, then parsed. Key ORDER is
|
|
143
|
+
# the one thing this does not settle — canonical_json does.
|
|
144
|
+
def json_round_trip(value)
|
|
145
|
+
JSON.parse(ActiveSupport::JSON.encode(value), max_nesting: false, create_additions: false)
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
# Deterministic JSON: object keys sorted bytewise at every depth, no
|
|
149
|
+
# whitespace, integral floats as integers, byte-level string escaping.
|
|
150
|
+
# Returns a binary String.
|
|
151
|
+
def canonical_json(value, out = String.new(encoding: Encoding::BINARY))
|
|
152
|
+
case value
|
|
153
|
+
when Hash
|
|
154
|
+
out << "{"
|
|
155
|
+
value.map { |k, v| [k.to_s.b, v] }.sort_by(&:first).each_with_index do |(k, v), i|
|
|
156
|
+
out << "," if i.positive?
|
|
157
|
+
json_string(k, out)
|
|
158
|
+
out << ":"
|
|
159
|
+
canonical_json(v, out)
|
|
160
|
+
end
|
|
161
|
+
out << "}"
|
|
162
|
+
when Array
|
|
163
|
+
out << "["
|
|
164
|
+
value.each_with_index do |v, i|
|
|
165
|
+
out << "," if i.positive?
|
|
166
|
+
canonical_json(v, out)
|
|
167
|
+
end
|
|
168
|
+
out << "]"
|
|
169
|
+
when nil then out << "null"
|
|
170
|
+
when true then out << "true"
|
|
171
|
+
when false then out << "false"
|
|
172
|
+
when Integer then out << value.to_s
|
|
173
|
+
when Float then out << (value.finite? && value == value.floor ? value.to_i.to_s : value.to_s)
|
|
174
|
+
else json_string(value.to_s, out)
|
|
175
|
+
end
|
|
176
|
+
out
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
def json_string(string, out)
|
|
180
|
+
out << '"'
|
|
181
|
+
out << string.b.gsub(ESCAPE) do |c|
|
|
182
|
+
case c
|
|
183
|
+
when '"' then '\\"'
|
|
184
|
+
when "\\" then "\\\\"
|
|
185
|
+
else format("\\u%04x", c.ord)
|
|
186
|
+
end
|
|
187
|
+
end
|
|
188
|
+
out << '"'
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
def time_like?(value)
|
|
192
|
+
value.respond_to?(:strftime) && value.respond_to?(:utc)
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
# True when some JSON object inside `value` has more than one key, i.e.
|
|
196
|
+
# a store that reorders keys could have changed how the legacy digest serialised
|
|
197
|
+
# it. A row with no such object cannot fail the legacy digest because of key order.
|
|
198
|
+
def key_order_ambiguous?(value)
|
|
199
|
+
case value
|
|
200
|
+
when Hash then value.size > 1 || value.each_value.any? { |v| key_order_ambiguous?(v) }
|
|
201
|
+
when Array then value.any? { |v| key_order_ambiguous?(v) }
|
|
202
|
+
else false
|
|
203
|
+
end
|
|
204
|
+
end
|
|
205
|
+
end
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
require "standard_audit/checksum/key_order_search"
|
|
@@ -13,7 +13,8 @@ module StandardAudit
|
|
|
13
13
|
:anonymizable_metadata_keys, :retention_days,
|
|
14
14
|
:audit_catalogue, :verify_audit_declarations,
|
|
15
15
|
:raise_on_audit_write_error, :audit_write_error_handler,
|
|
16
|
-
:audit_error_context_key, :error_reporter
|
|
16
|
+
:audit_error_context_key, :error_reporter,
|
|
17
|
+
:canonical_checksum_since
|
|
17
18
|
|
|
18
19
|
def initialize
|
|
19
20
|
@subscriptions = []
|
|
@@ -128,6 +129,16 @@ module StandardAudit
|
|
|
128
129
|
# computed. See Configuration#before_checksum.
|
|
129
130
|
@before_checksum_hooks = []
|
|
130
131
|
|
|
132
|
+
# Rows whose `created_at` is at or after this time are signed with the
|
|
133
|
+
# canonical checksum (object keys sorted, independent of how jsonb
|
|
134
|
+
# stores them) and verified strictly with it; earlier rows keep the
|
|
135
|
+
# legacy algorithm. Defaults to StandardAudit::CANONICAL_CHECKSUM_CUTOVER.
|
|
136
|
+
# Override only if your rollout of this release slips past that date,
|
|
137
|
+
# set it BEFORE the new time arrives, and never change it afterwards:
|
|
138
|
+
# verification recomputes the same decision from each row's stored
|
|
139
|
+
# created_at, so moving it re-judges rows under the other algorithm.
|
|
140
|
+
@canonical_checksum_since = StandardAudit::CANONICAL_CHECKSUM_CUTOVER
|
|
141
|
+
|
|
131
142
|
@anonymizable_metadata_keys = %i[email name ip_address]
|
|
132
143
|
|
|
133
144
|
# ── StandardAudit::Operation (the operation-audit DSL) ────────────────
|
data/lib/standard_audit.rb
CHANGED
|
@@ -1,6 +1,17 @@
|
|
|
1
1
|
require "standard_audit/version"
|
|
2
|
+
|
|
3
|
+
module StandardAudit
|
|
4
|
+
# From this instant (by each row's `created_at`) new rows get the canonical
|
|
5
|
+
# checksum and are verified strictly with it; earlier rows are legacy. See
|
|
6
|
+
# StandardAudit::Checksum and fundbright/delivery-ops#689. Every host must
|
|
7
|
+
# run 0.13.1+ BEFORE this time: an older gem still writing after it produces
|
|
8
|
+
# legacy-hashed rows that fail strict canonical verification.
|
|
9
|
+
# Overridable per host via `config.canonical_checksum_since`.
|
|
10
|
+
CANONICAL_CHECKSUM_CUTOVER = Time.utc(2026, 10, 1, 0, 0, 0).freeze
|
|
11
|
+
end
|
|
2
12
|
require "standard_audit/engine"
|
|
3
13
|
require "standard_audit/configuration"
|
|
14
|
+
require "standard_audit/checksum"
|
|
4
15
|
require "standard_audit/metadata_filter"
|
|
5
16
|
require "standard_audit/record_reference"
|
|
6
17
|
require "standard_audit/sensitive_keys_dry_run"
|
|
@@ -286,6 +297,8 @@ module StandardAudit
|
|
|
286
297
|
created_at: now,
|
|
287
298
|
updated_at: now
|
|
288
299
|
))
|
|
300
|
+
# created_at is set above, so the algorithm is decided by the same
|
|
301
|
+
# stored timestamp verification will read.
|
|
289
302
|
checksum = StandardAudit::AuditLog.compute_checksum_value(
|
|
290
303
|
row.stringify_keys,
|
|
291
304
|
previous_checksum: previous_checksum
|
|
@@ -85,17 +85,34 @@ namespace :standard_audit do
|
|
|
85
85
|
|
|
86
86
|
desc "Verify audit log chain integrity (tamper detection)"
|
|
87
87
|
task verify: :environment do
|
|
88
|
-
|
|
88
|
+
# FAIL_ON_LEGACY_UNVERIFIABLE=1 — treat pre-cutover rows whose metadata
|
|
89
|
+
# key order cannot be reconstructed as failures (see
|
|
90
|
+
# AuditLog.verify_chain). KEY_ORDER_SEARCH_LIMIT=5040 — orderings tried
|
|
91
|
+
# per legacy row.
|
|
92
|
+
options = {}
|
|
93
|
+
options[:fail_on_legacy_unverifiable] = true if %w[1 true].include?(ENV["FAIL_ON_LEGACY_UNVERIFIABLE"])
|
|
94
|
+
if (limit = ENV["KEY_ORDER_SEARCH_LIMIT"].presence)
|
|
95
|
+
options[:key_order_search_limit] = Integer(limit, 10)
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
result = StandardAudit::AuditLog.verify_chain(**options)
|
|
89
99
|
|
|
90
100
|
puts "Audit Log Chain Verification"
|
|
91
101
|
puts "============================="
|
|
102
|
+
puts "Canonical checksums since: #{StandardAudit.config.canonical_checksum_since.utc.iso8601}"
|
|
92
103
|
puts "Records verified: #{result[:verified]}"
|
|
93
104
|
puts "Chain valid: #{result[:valid]}"
|
|
94
105
|
puts "Forked links recovered: #{result[:recovered]}"
|
|
106
|
+
puts "Legacy rows verified by key-order reconstruction: #{result[:reordered]}" if result[:reordered].to_i.positive?
|
|
95
107
|
puts "Anonymized (redacted) records: #{result[:redacted]}" if result[:redacted].to_i.positive?
|
|
108
|
+
if result[:legacy_unverifiable].to_i.positive?
|
|
109
|
+
puts "Legacy rows unverifiable (metadata key order lost): #{result[:legacy_unverifiable]} " \
|
|
110
|
+
"— cannot be proven either way; this count must not grow after the cutover"
|
|
111
|
+
end
|
|
96
112
|
|
|
97
113
|
if result[:failures].any?
|
|
98
114
|
puts "\nUnverifiable records detected: #{result[:failures].size}"
|
|
115
|
+
result[:failures].map { |failure| failure[:reason] }.tally.each { |reason, n| puts " #{reason}: #{n}" }
|
|
99
116
|
result[:failures].each do |failure|
|
|
100
117
|
puts " #{failure[:id]} (#{failure[:event_type]}) at #{failure[:created_at]} — #{failure[:reason]}"
|
|
101
118
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: standard_audit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.13.
|
|
4
|
+
version: 0.13.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Jaryl Sim
|
|
@@ -15,42 +15,42 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: '8.
|
|
18
|
+
version: '8.1'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: '8.
|
|
25
|
+
version: '8.1'
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: activejob
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
30
|
- - ">="
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
|
-
version: '8.
|
|
32
|
+
version: '8.1'
|
|
33
33
|
type: :runtime
|
|
34
34
|
prerelease: false
|
|
35
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
36
36
|
requirements:
|
|
37
37
|
- - ">="
|
|
38
38
|
- !ruby/object:Gem::Version
|
|
39
|
-
version: '8.
|
|
39
|
+
version: '8.1'
|
|
40
40
|
- !ruby/object:Gem::Dependency
|
|
41
41
|
name: activesupport
|
|
42
42
|
requirement: !ruby/object:Gem::Requirement
|
|
43
43
|
requirements:
|
|
44
44
|
- - ">="
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
|
-
version: '8.
|
|
46
|
+
version: '8.1'
|
|
47
47
|
type: :runtime
|
|
48
48
|
prerelease: false
|
|
49
49
|
version_requirements: !ruby/object:Gem::Requirement
|
|
50
50
|
requirements:
|
|
51
51
|
- - ">="
|
|
52
52
|
- !ruby/object:Gem::Version
|
|
53
|
-
version: '8.
|
|
53
|
+
version: '8.1'
|
|
54
54
|
- !ruby/object:Gem::Dependency
|
|
55
55
|
name: globalid
|
|
56
56
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -108,6 +108,8 @@ files:
|
|
|
108
108
|
- lib/standard_audit/audit_scope.rb
|
|
109
109
|
- lib/standard_audit/auditable.rb
|
|
110
110
|
- lib/standard_audit/checks/retention.rb
|
|
111
|
+
- lib/standard_audit/checksum.rb
|
|
112
|
+
- lib/standard_audit/checksum/key_order_search.rb
|
|
111
113
|
- lib/standard_audit/configuration.rb
|
|
112
114
|
- lib/standard_audit/engine.rb
|
|
113
115
|
- lib/standard_audit/event_subscriber.rb
|
|
@@ -146,7 +148,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
146
148
|
- !ruby/object:Gem::Version
|
|
147
149
|
version: '0'
|
|
148
150
|
requirements: []
|
|
149
|
-
rubygems_version: 4.0.
|
|
151
|
+
rubygems_version: 4.0.10
|
|
150
152
|
specification_version: 4
|
|
151
153
|
summary: Database-backed audit logging for Rails via Rails.event and ActiveSupport::Notifications.
|
|
152
154
|
test_files: []
|