reeve 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +117 -0
- data/README.md +236 -9
- data/lib/generators/reeve/install/templates/create_audit_entries.rb.tt +29 -4
- data/lib/generators/reeve/install/templates/initializer.rb.tt +10 -0
- data/lib/generators/reeve/upgrade/templates/add_ledger_retention_to_reeve_audit_entries.rb.tt +37 -0
- data/lib/generators/reeve/upgrade/upgrade_generator.rb +10 -1
- data/lib/generators/reeve/upgrade/upgrade_notes.rb +15 -4
- data/lib/reeve/audit/chain/anchoring.rb +91 -0
- data/lib/reeve/audit/chain/canonical.rb +17 -1
- data/lib/reeve/audit/chain/checkpoint.rb +228 -0
- data/lib/reeve/audit/chain/checkpoint_check.rb +156 -0
- data/lib/reeve/audit/chain/entrypoints.rb +29 -0
- data/lib/reeve/audit/chain/epochs.rb +10 -4
- data/lib/reeve/audit/chain/head.rb +26 -1
- data/lib/reeve/audit/chain/keyring.rb +6 -0
- data/lib/reeve/audit/chain/report.rb +188 -31
- data/lib/reeve/audit/chain/retention/archive.rb +98 -0
- data/lib/reeve/audit/chain/retention/archive_check.rb +179 -0
- data/lib/reeve/audit/chain/retention/check.rb +94 -0
- data/lib/reeve/audit/chain/retention/connection.rb +65 -0
- data/lib/reeve/audit/chain/retention/digest.rb +29 -0
- data/lib/reeve/audit/chain/retention/plan.rb +100 -0
- data/lib/reeve/audit/chain/retention/preflight.rb +48 -0
- data/lib/reeve/audit/chain/retention/purge.rb +155 -0
- data/lib/reeve/audit/chain/retention/purge_result.rb +134 -0
- data/lib/reeve/audit/chain/retention/records.rb +23 -0
- data/lib/reeve/audit/chain/verification.rb +31 -9
- data/lib/reeve/audit/chain/verifier.rb +64 -19
- data/lib/reeve/audit/chain/walk.rb +9 -3
- data/lib/reeve/audit/chain.rb +12 -3
- data/lib/reeve/audit/chain_record.rb +3 -2
- data/lib/reeve/audit/file_anchor.rb +49 -0
- data/lib/reeve/audit/retention_record.rb +24 -0
- data/lib/reeve/audit.rb +16 -0
- data/lib/reeve/configuration.rb +25 -0
- data/lib/reeve/errors.rb +15 -0
- data/lib/reeve/railtie.rb +2 -2
- data/lib/reeve/tasks/descriptions.rb +22 -0
- data/lib/reeve/tasks/purge_options.rb +54 -0
- data/lib/reeve/tasks.rb +82 -14
- data/lib/reeve/testing/checks/ledger_chain.rb +56 -2
- data/lib/reeve/testing/ledger.rb +19 -3
- data/lib/reeve/version.rb +1 -1
- metadata +21 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5fec548cc5008459901b118b5bd79b3a3c47899554199c7a8a686ca3e9528cc8
|
|
4
|
+
data.tar.gz: 277700a1ad87207e90b693e96cfb2536ffec41459d420bd9d93d9fe08a1ebd68
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 077330b44a698212477ea31db93ade9d0242c8c1a85ac546363a85a296ca5faaadb3865b058720ac74fb9e1a71e2e0afa72c42eb0687b78707cd5588cbad3394
|
|
7
|
+
data.tar.gz: f6b915ea4c1fb5ade7a2d07c71cc199b5446eca9dcfb8beecbdc233f1d7574e754b026de3a6ecd309fa82f9205af99b65a65c0182e8e5ba5fe31a7c57585fab8
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,121 @@ All notable changes are recorded here. This project follows [Semantic
|
|
|
4
4
|
Versioning](https://semver.org), with one rule specific to what it does — see
|
|
5
5
|
[Versioning policy](#versioning-policy).
|
|
6
6
|
|
|
7
|
+
## [0.8.0] - 2026-10-25
|
|
8
|
+
|
|
9
|
+
Two additions to the tamper-evident ledger: **anchoring**, checkpoints of the chain kept
|
|
10
|
+
outside the database, so deleting the newest entries or the whole ledger is caught; and
|
|
11
|
+
**retention**, purging the oldest entries up to a checkpoint without breaking verification.
|
|
12
|
+
No audit-entry contract change: it stays `4`.
|
|
13
|
+
|
|
14
|
+
### Upgrading from 0.7.0
|
|
15
|
+
|
|
16
|
+
1. `bundle update reeve`, then `bin/rails generate reeve:upgrade && bin/rails db:migrate`. It
|
|
17
|
+
adds one table, `reeve_audit_retentions`; nothing in the ledger table changes.
|
|
18
|
+
2. Grant the application's database role `SELECT` on `reeve_audit_retentions`.
|
|
19
|
+
3. If anything reads the text output of `bin/rails reeve:verify_ledger`, switch it to
|
|
20
|
+
`FORMAT=json`: every summary line now ends with its anchoring (`; not anchored` until you
|
|
21
|
+
set one up). Existing JSON keys are unchanged.
|
|
22
|
+
|
|
23
|
+
Anchoring and retention are both opt-in; the README shows how to set each up.
|
|
24
|
+
|
|
25
|
+
### Anchoring
|
|
26
|
+
|
|
27
|
+
The chain's own records live in the same database as the entries, so two kinds of tampering
|
|
28
|
+
went unseen: deleting the newest entries and rewinding the head to match, and deleting the
|
|
29
|
+
whole ledger, which read as "no chain". A new epoch nobody had written under yet could also
|
|
30
|
+
be deleted without trace. Now reeve can make a **checkpoint**, one signed line saying how far
|
|
31
|
+
the chain reached, and hand it to an **anchor**: storage you choose, outside the database.
|
|
32
|
+
Verification checks the ledger against every checkpoint.
|
|
33
|
+
|
|
34
|
+
**No contract change.** The audit-entry contract stays `4`. Nothing changes for a ledger
|
|
35
|
+
write. (Retention, below, adds one table.)
|
|
36
|
+
|
|
37
|
+
#### Added
|
|
38
|
+
|
|
39
|
+
- `bin/rails reeve:checkpoint_ledger` (`Reeve::Tasks.checkpoint_ledger`,
|
|
40
|
+
`Reeve::Audit::Chain.checkpoint!`): makes a checkpoint, publishes it, and prints it. Exit `0`
|
|
41
|
+
made, `1` refused (no key, no chain yet) or the anchor failed; the line is printed either way
|
|
42
|
+
so it can be stored by hand.
|
|
43
|
+
- `config.ledger_anchor`: any object with `publish(line)` and `checkpoints`. reeve ships
|
|
44
|
+
`Reeve::Audit::FileAnchor`, a file it only ever appends to. It is a building block: on the
|
|
45
|
+
database host it protects nothing. Point it at storage the database's users cannot change,
|
|
46
|
+
such as a mount backed by an object-locked bucket, and use one per environment.
|
|
47
|
+
- `reeve:verify_ledger` checks against the anchor's checkpoints when one is configured.
|
|
48
|
+
`CHECKPOINTS=path` reads a file of checkpoint lines instead, `CHECKPOINTS=` (empty) uses
|
|
49
|
+
none, and `FROM=latest` walks only what came after the newest checkpoint. A checkpoint line
|
|
50
|
+
that does not read is exit `2`, naming its position.
|
|
51
|
+
- New breaks, all exit `1`: `checkpoint_not_reached`, `checkpoint_mismatch`,
|
|
52
|
+
`checkpoint_altered`. A checkpoint signed by a key not configured is unverifiable (exit `2`).
|
|
53
|
+
With a checkpoint, an emptied ledger is BROKEN, not NO CHAIN.
|
|
54
|
+
- `reeve:rotate_ledger_key` makes a checkpoint after opening an epoch when an anchor is
|
|
55
|
+
configured, so the new epoch is pinned before anything is written under it.
|
|
56
|
+
- `Reeve::Checks::LedgerChain.new(anchored: true)`: also makes a checkpoint, reads it back
|
|
57
|
+
from the anchor and verifies against it. Run it as a test of its own (the README shows how);
|
|
58
|
+
the suite's own `LedgerChain` never reads the anchor, since checkpoints earlier runs left
|
|
59
|
+
name test databases that are gone.
|
|
60
|
+
|
|
61
|
+
#### Changed
|
|
62
|
+
|
|
63
|
+
- **Every summary line that follows a walk now ends with its anchoring**: `; not anchored`,
|
|
64
|
+
or `; anchored at #48,000 (checkpoint 2026-10-25 14:00 UTC), 211 entries after it`. This
|
|
65
|
+
applies whether or not you configure an anchor. If something reads the text report, read
|
|
66
|
+
`FORMAT=json` instead; its existing keys are unchanged, and it gains `checkpoints`,
|
|
67
|
+
`anchored_at`, `beyond_anchor`, `walked_from` and `breaks[].checkpoint`.
|
|
68
|
+
|
|
69
|
+
#### Limits, stated plainly
|
|
70
|
+
|
|
71
|
+
- Entries written after the latest checkpoint can still be removed without trace. The summary
|
|
72
|
+
line counts the entries still there after it, not ones deleted; how often you checkpoint is
|
|
73
|
+
the size of that window.
|
|
74
|
+
- A checkpoint is only as safe as its store. One an attacker can delete or overwrite protects
|
|
75
|
+
nothing, and reeve cannot make a store append-only.
|
|
76
|
+
- A leaked key can still rewrite the chain from any point, but not a checkpoint made before
|
|
77
|
+
the leak: rewriting an entry it saw changes the digest it names.
|
|
78
|
+
- Restoring a backup older than a checkpoint reads as BROKEN, because the restored ledger lost
|
|
79
|
+
entries that checkpoint saw. A database reset on purpose needs a new anchor file.
|
|
80
|
+
- A checkpoint's time is the producing host's clock, for people. Verification never uses it.
|
|
81
|
+
|
|
82
|
+
### Retention
|
|
83
|
+
|
|
84
|
+
The ledger could only grow, and deleting old rows by hand broke the chain and read as
|
|
85
|
+
tampering. Now `bin/rails reeve:purge_ledger` removes the oldest entries up to a checkpoint
|
|
86
|
+
your anchor holds, writes an archive of them first, and leaves a signed **retention record**, so
|
|
87
|
+
the rest still verifies and the purge reads as a purge.
|
|
88
|
+
|
|
89
|
+
**One migration, no contract change.** `bin/rails generate reeve:upgrade` adds the table
|
|
90
|
+
`reeve_audit_retentions`; the audit-entry contract stays `4`. Grant the application's role
|
|
91
|
+
`SELECT` on it. Nothing is purged until you run the task.
|
|
92
|
+
|
|
93
|
+
**The constitution's append-only rule now has one exception** (1.1.0): application code still
|
|
94
|
+
cannot update or delete an entry; only this operator-run purge can, and only behind a verified,
|
|
95
|
+
anchored checkpoint, on a database role the application does not use, after writing a signed
|
|
96
|
+
record.
|
|
97
|
+
|
|
98
|
+
- `config.ledger_retention_connection`: an abstract ActiveRecord class connected to the ledger's
|
|
99
|
+
database as a role with `DELETE`; the purge refuses another database, or the application's
|
|
100
|
+
own user (SQLite has no roles, so there it warns).
|
|
101
|
+
- `bin/rails reeve:purge_ledger`: `OLDER_THAN=365d` or `THROUGH=sequence`; exactly one of
|
|
102
|
+
`ARCHIVE=path` (a new file each run) and `NO_ARCHIVE=1`; nothing removed without `CONFIRM=1`.
|
|
103
|
+
Exit `0` for a dry run, a purge, or nothing to purge (the first line says which); `1` refused,
|
|
104
|
+
with the reason and the fix. It requires the whole ledger to verify INTACT first. A purge
|
|
105
|
+
stopped midway leaves rows "awaiting purge" (INTACT, not a break); any rerun finishes it first.
|
|
106
|
+
The README shows the role, the grants and the `database.yml` entry.
|
|
107
|
+
- `bin/rails reeve:verify_archive ARCHIVE=path`: verifies an archive on its own, from its start to
|
|
108
|
+
its boundary checkpoint. It needs the application (for the table's column types), not the
|
|
109
|
+
purged rows.
|
|
110
|
+
- Verification starts after the latest purge, names the purged range on every summary line,
|
|
111
|
+
reports checkpoints older than it as purged, and gains `retention_altered` (BROKEN) and the JSON
|
|
112
|
+
keys `retention` and `awaiting_purge`.
|
|
113
|
+
|
|
114
|
+
Keep every ledger key: epoch and retention records stay in the ledger and are checked under
|
|
115
|
+
their own keys, and archives need the keys of the rows they hold.
|
|
116
|
+
|
|
117
|
+
Limits, in plain words: what a purge removed without an archive is gone for good, and the record
|
|
118
|
+
proves only that it existed and verified. An archive is a file you must keep safe, as you do the
|
|
119
|
+
anchor. A row edited inside the purged range while the purge runs is not reported by the ledger,
|
|
120
|
+
because it is being removed; the archive holds what was read.
|
|
121
|
+
|
|
7
122
|
## [0.7.0] - 2026-10-23
|
|
8
123
|
|
|
9
124
|
0.6.0 let a ledger be verified across a key rotation, provided the old key was kept as a
|
|
@@ -690,4 +805,6 @@ is for:
|
|
|
690
805
|
alone — unless what its absence means, or what an existing value covers, differs between
|
|
691
806
|
rows written before and after it, which is major (contracts 3 and 4 are both examples).
|
|
692
807
|
Removing or renaming a column, or changing what a value means, is major and bumps it.
|
|
808
|
+
- A new table beside the ledger that leaves its rows unchanged is minor and leaves the
|
|
809
|
+
contract alone (`reeve_audit_retentions`, 0.8.0).
|
|
693
810
|
- The audit-entry contract version is `4`.
|
data/README.md
CHANGED
|
@@ -7,7 +7,7 @@ A Ruby gem that makes it safe for a Rails application to expose MCP tools to AI
|
|
|
7
7
|
declarative per-record authorization, an append-only audit ledger, and a testing kit that
|
|
8
8
|
proves both hold.
|
|
9
9
|
|
|
10
|
-
> **Status: 0.
|
|
10
|
+
> **Status: 0.8.0.** Read the known limitations in
|
|
11
11
|
> [CHANGELOG.md](CHANGELOG.md) before adopting it — particularly the one about a
|
|
12
12
|
> transaction wrapped around an invocation, if your application wraps requests in one.
|
|
13
13
|
|
|
@@ -200,7 +200,7 @@ bin/rails reeve:verify_ledger
|
|
|
200
200
|
```
|
|
201
201
|
|
|
202
202
|
```text
|
|
203
|
-
reeve ledger: INTACT — sequences 1–48,211 verified (head at start: 48,211); 5,000 earlier entries not covered (5,000 when the chain started)
|
|
203
|
+
reeve ledger: INTACT — sequences 1–48,211 verified (head at start: 48,211); 5,000 earlier entries not covered (5,000 when the chain started); not anchored
|
|
204
204
|
```
|
|
205
205
|
|
|
206
206
|
It exits `0` when the chain is intact, `1` when it is broken, and `2` when it could not
|
|
@@ -218,7 +218,8 @@ puts result # the report above, with every break listed under it
|
|
|
218
218
|
|
|
219
219
|
In CI the compliance suite you already run includes `LedgerChain`, in both frameworks:
|
|
220
220
|
it writes entries of its own and fails on a broken chain, a missing key, or entries that
|
|
221
|
-
were not chained.
|
|
221
|
+
were not chained. "Not anchored" on the summary line is explained under **Anchoring the
|
|
222
|
+
chain** below.
|
|
222
223
|
|
|
223
224
|
**The key.** The generated initializer derives it from `secret_key_base`:
|
|
224
225
|
|
|
@@ -256,6 +257,216 @@ rotation.epoch # => 1
|
|
|
256
257
|
Verification then lists each epoch under its summary line, with the key fingerprint, where
|
|
257
258
|
it begins, and whether it verified. The CHANGELOG has the full procedure.
|
|
258
259
|
|
|
260
|
+
**Anchoring the chain.** The chain's own records live in the same database as the entries, so
|
|
261
|
+
someone who deletes the newest entries and rewinds the head to match, or deletes the whole
|
|
262
|
+
ledger, leaves nothing behind that verification can see. A *checkpoint* is a second witness:
|
|
263
|
+
one signed line saying how far the chain reached (its genesis, its head, its latest key
|
|
264
|
+
epoch), kept where the database's users cannot change it. It holds digests and counts only,
|
|
265
|
+
never entry content or a key.
|
|
266
|
+
|
|
267
|
+
Point an anchor at that storage. The line goes at the bottom of
|
|
268
|
+
`config/initializers/reeve.rb`, after `require "reeve/audit"` (which defines `FileAnchor`),
|
|
269
|
+
outside the `Reeve.configure` block, hence `Reeve.config`:
|
|
270
|
+
|
|
271
|
+
```ruby
|
|
272
|
+
require "reeve/audit"
|
|
273
|
+
|
|
274
|
+
Reeve.config.ledger_anchor =
|
|
275
|
+
Reeve::Audit::FileAnchor.new("/mnt/ledger-anchor/#{Rails.env}.checkpoints")
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
Use one anchor per environment, on storage outside the database host, such as a mount backed
|
|
279
|
+
by an object-locked bucket; a file next to the database protects nothing. The hosts that make
|
|
280
|
+
checkpoints need to write the file (it is created on the first one), and the hosts that verify
|
|
281
|
+
need to read it. Any object with `publish(line)` and `checkpoints` works in its place.
|
|
282
|
+
|
|
283
|
+
Then make a checkpoint on a schedule, from one host (hourly, say), and keep verifying as
|
|
284
|
+
before. Alert on anything but `0` from either task. `reeve:checkpoint_ledger` exits `1` when
|
|
285
|
+
it refused (no key, or no chain until the first guarded call) or the anchor failed, and still
|
|
286
|
+
prints the line so it can be stored by hand. Two runs at once are safe.
|
|
287
|
+
|
|
288
|
+
```bash
|
|
289
|
+
bin/rails reeve:checkpoint_ledger # exit 0 made and published
|
|
290
|
+
bin/rails reeve:verify_ledger # checks the ledger against every checkpoint
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
```text
|
|
294
|
+
reeve ledger: INTACT — sequences 1–48,211 verified (head at start: 48,211); 5,000 earlier entries not covered (5,000 when the chain started); anchored at #48,000 (checkpoint 2026-10-25 14:00 UTC), 211 entries after it
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
Every checkpoint is checked: a ledger that no longer reaches one, holds another entry where it
|
|
298
|
+
saw one, or lost an epoch record it named, is BROKEN (exit `1`), and so is an emptied ledger,
|
|
299
|
+
which without a checkpoint reads only as "no chain". The newest entries deleted with the head
|
|
300
|
+
rewound read like this:
|
|
301
|
+
|
|
302
|
+
```text
|
|
303
|
+
reeve ledger: BROKEN — 1 break in sequences 1–46,211; 5,000 earlier entries not covered (5,000 when the chain started); anchored at #44,000 (checkpoint 2026-10-25 13:00 UTC), 2,211 entries after it
|
|
304
|
+
totals: checkpoint_not_reached 1
|
|
305
|
+
checkpoint #48,000 checkpoint_not_reached the head is at 46,211; entries the checkpoint saw were removed
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
What it does not cover: entries written after the latest checkpoint can still be removed
|
|
309
|
+
without trace. "211 entries after it" counts the entries still there past the checkpoint; it
|
|
310
|
+
cannot see ones that were deleted, so how often you checkpoint is the size of that window.
|
|
311
|
+
`reeve:rotate_ledger_key` makes a checkpoint itself after opening an epoch. Writes never touch
|
|
312
|
+
the anchor.
|
|
313
|
+
|
|
314
|
+
*Restores and resets.* Restoring a backup older than a checkpoint reads as BROKEN, truthfully:
|
|
315
|
+
the restored ledger lost entries that checkpoint saw. Resetting a database on purpose (a
|
|
316
|
+
staging rebuild) is the same, so start a new anchor file for the new ledger: change the path
|
|
317
|
+
(`staging-2026-10-25.checkpoints`, say), keep the old file as the record of the old ledger, and
|
|
318
|
+
make a checkpoint. Until then the run reads "not anchored". `CHECKPOINTS=` (empty) verifies
|
|
319
|
+
without any checkpoint, for a one-off check, never as the fix.
|
|
320
|
+
|
|
321
|
+
*Other sources, large ledgers.* `CHECKPOINTS=path` verifies against a file of checkpoint lines
|
|
322
|
+
instead of the anchor, say an auditor's copy. `FROM=latest` walks only what came after the
|
|
323
|
+
newest checkpoint: it first checks that checkpoint (its signature, the genesis and epoch it
|
|
324
|
+
names, and that its entry still has the digest it saw), then verifies every entry after it.
|
|
325
|
+
Entries before it are not re-verified by that run, and the summary says how many; keep a full
|
|
326
|
+
run on a slower schedule. With no checkpoint to start from it exits `2`. From Ruby:
|
|
327
|
+
|
|
328
|
+
```ruby
|
|
329
|
+
Reeve::Audit::Chain.checkpoint!.checkpoint.to_line # => "{\"epoch\":1,…}"
|
|
330
|
+
Reeve::Audit::Chain.verify(checkpoints: lines).anchored_at # the highest checkpoint that held
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
*In CI.* The `LedgerChain` check the compliance suite runs never reads the anchor, so
|
|
334
|
+
configuring one does not change it: a test database is emptied between runs, and old
|
|
335
|
+
checkpoints name ledgers that are gone. To prove the anchor works, run the anchored variant as
|
|
336
|
+
a test of its own, with the test environment's anchor pointed at a scratch file such as
|
|
337
|
+
`Rails.root.join("tmp/ledger.checkpoints")`. It makes a checkpoint, reads it back through the
|
|
338
|
+
anchor, and verifies against that one alone:
|
|
339
|
+
|
|
340
|
+
```ruby
|
|
341
|
+
# RSpec
|
|
342
|
+
it "anchors the ledger" do
|
|
343
|
+
result = Reeve::Checks::LedgerChain.new(anchored: true).call
|
|
344
|
+
expect(result).to be_passed, result.message
|
|
345
|
+
end
|
|
346
|
+
|
|
347
|
+
# Minitest
|
|
348
|
+
def test_anchors_the_ledger
|
|
349
|
+
result = Reeve::Checks::LedgerChain.new(anchored: true).call
|
|
350
|
+
assert result.passed?, result.message
|
|
351
|
+
end
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
**Purging old entries.** The ledger grows until you purge it. Deleting rows by hand breaks
|
|
355
|
+
the chain and is reported as tampering, so reeve purges for you: the oldest entries, up to a
|
|
356
|
+
checkpoint your anchor holds, after verifying the whole ledger INTACT, leaving a signed
|
|
357
|
+
*retention record* so the rest still verifies and the purge reads as a purge. It also removes
|
|
358
|
+
rows written before the chain began, and nothing else. It stops at a checkpoint, so with
|
|
359
|
+
`OLDER_THAN=365d` a few entries just over a year old can remain until the next run.
|
|
360
|
+
|
|
361
|
+
*Setting it up, once.* Run `bin/rails generate reeve:upgrade && bin/rails db:migrate` (it adds
|
|
362
|
+
`reeve_audit_retentions`), then create a database role only the purge uses; the application's
|
|
363
|
+
role never gains `DELETE`. On PostgreSQL (the migration's header shows the same grants):
|
|
364
|
+
|
|
365
|
+
```sql
|
|
366
|
+
CREATE ROLE reeve_retention LOGIN PASSWORD '…';
|
|
367
|
+
GRANT SELECT ON reeve_audit_entries, reeve_audit_chains, reeve_audit_chain_epochs,
|
|
368
|
+
reeve_audit_retentions TO reeve_retention;
|
|
369
|
+
GRANT DELETE ON reeve_audit_entries TO reeve_retention;
|
|
370
|
+
GRANT INSERT ON reeve_audit_retentions TO reeve_retention;
|
|
371
|
+
GRANT USAGE ON SEQUENCE reeve_audit_retentions_id_seq TO reeve_retention;
|
|
372
|
+
GRANT SELECT ON reeve_audit_retentions TO app_role; -- the application only reads it
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
Add the role to `config/database.yml` as a second database entry pointing at the same
|
|
376
|
+
database, and a class that connects through it:
|
|
377
|
+
|
|
378
|
+
```yaml
|
|
379
|
+
production:
|
|
380
|
+
primary:
|
|
381
|
+
<<: *default
|
|
382
|
+
database: app_production
|
|
383
|
+
ledger_retention:
|
|
384
|
+
<<: *default
|
|
385
|
+
database: app_production # the same database
|
|
386
|
+
username: reeve_retention
|
|
387
|
+
password: <%= ENV["REEVE_RETENTION_PASSWORD"] %>
|
|
388
|
+
database_tasks: false # never migrated through this entry
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
```ruby
|
|
392
|
+
# app/models/ledger_retention_record.rb
|
|
393
|
+
class LedgerRetentionRecord < ActiveRecord::Base
|
|
394
|
+
self.abstract_class = true
|
|
395
|
+
connects_to database: { writing: :ledger_retention }
|
|
396
|
+
end
|
|
397
|
+
|
|
398
|
+
# config/initializers/reeve.rb, after require "reeve/audit"
|
|
399
|
+
Reeve.config.ledger_retention_connection = LedgerRetentionRecord
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
The purge refuses a connection to another database, or one that logs in as the application's
|
|
403
|
+
own user. On SQLite, which has no roles, it can only warn. The task needs the ledger key, as
|
|
404
|
+
every chain task does.
|
|
405
|
+
|
|
406
|
+
*Running it.* Monthly, from one host, a dry run first (no `CONFIRM`), then the purge. The
|
|
407
|
+
archive path must be new each run: an existing archive is never overwritten.
|
|
408
|
+
|
|
409
|
+
```bash
|
|
410
|
+
bin/rails reeve:purge_ledger OLDER_THAN=365d ARCHIVE=/mnt/archive/ledger-2026-10.jsonl # dry run
|
|
411
|
+
bin/rails reeve:purge_ledger OLDER_THAN=365d ARCHIVE=/mnt/archive/ledger-2026-10.jsonl CONFIRM=1
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
```text
|
|
415
|
+
reeve ledger purge: WOULD PURGE — sequences 1–6,000 through checkpoint #6,000 (2025-10-01 14:00 UTC): 6,000 chained entries and 5,000 pre-chain rows; archive to /mnt/archive/ledger-2026-10.jsonl; run again with CONFIRM=1
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
In crontab, escape the date's `%`: `ledger-$(date +\%Y-\%m).jsonl`. `THROUGH=6000` names a
|
|
419
|
+
checkpoint's sequence instead of an age. Exactly one of `ARCHIVE=path` and `NO_ARCHIVE=1` is
|
|
420
|
+
required; with neither the task refuses, with both it raises. Exit `0` covers a dry run, a
|
|
421
|
+
purge, and nothing old enough (the first line says which: `WOULD PURGE`, `PURGED`, `NOTHING TO
|
|
422
|
+
PURGE`); exit `1` is a refusal, which removes nothing and names the fix:
|
|
423
|
+
|
|
424
|
+
| Refusal says | Fix |
|
|
425
|
+
|---|---|
|
|
426
|
+
| no anchor configured | set `config.ledger_anchor` and schedule checkpoints (above) |
|
|
427
|
+
| no retention connection / another database / the application's own database user | `config.ledger_retention_connection` and the role above |
|
|
428
|
+
| the ledger does not verify intact | read `bin/rails reeve:verify_ledger`; nothing is purged while anything is broken or unverifiable |
|
|
429
|
+
| no checkpoint the anchor holds is at that sequence | use one of the two it names, or `OLDER_THAN` |
|
|
430
|
+
| the archive … already exists, or cannot be written | a new path, in a directory the task can write |
|
|
431
|
+
|
|
432
|
+
Any other failure (a lost connection, a full disk) raises, exit `1` from Rake, like any task.
|
|
433
|
+
|
|
434
|
+
*If it stops midway.* The archive is written in full and the retention record committed
|
|
435
|
+
before any row is deleted. If the host dies during the deletes, verification reads INTACT,
|
|
436
|
+
exit `0`, with `; 120 entries awaiting purge` beside the purged range, which is not a break.
|
|
437
|
+
Run the purge again, the same command or any other: it first finishes the interrupted purge
|
|
438
|
+
(`PURGED — … (resumed; 120 rows removed this run)`), whatever boundary the new run names, and
|
|
439
|
+
skips the archive, which already exists. Run it once more to purge further.
|
|
440
|
+
|
|
441
|
+
*The archive.* Every purged row with every column, so the range can be checked without the
|
|
442
|
+
ledger's rows:
|
|
443
|
+
|
|
444
|
+
```bash
|
|
445
|
+
bin/rails reeve:verify_archive ARCHIVE=/mnt/archive/ledger-2026-10.jsonl # reeve archive: INTACT — sequences 1–6,000 verified, …
|
|
446
|
+
```
|
|
447
|
+
|
|
448
|
+
It runs inside the application (it needs the entries table's columns, from any database with
|
|
449
|
+
the schema, not the purged rows), with the keys that signed the archived rows configured as
|
|
450
|
+
`ledger_key` or `retired_ledger_keys`; the anchor is not needed. Each archive covers the range
|
|
451
|
+
its purge removed, named in its first line, so name files by the month they were made. Keep
|
|
452
|
+
archives where they cannot be changed, as you do the anchor. `NO_ARCHIVE=1` deletes without
|
|
453
|
+
one, for data you must not keep; the record and the output then say no archive was written.
|
|
454
|
+
|
|
455
|
+
*Keys.* Keep every key you ever signed with. A purge removes entries, not epoch records or
|
|
456
|
+
retention records, and verification still checks those under their own keys; archives need
|
|
457
|
+
the keys of the rows they hold.
|
|
458
|
+
|
|
459
|
+
After a purge, every summary line names what was purged before its coverage; with
|
|
460
|
+
`FORMAT=json`, `retention` and `awaiting_purge` hold the same:
|
|
461
|
+
|
|
462
|
+
```text
|
|
463
|
+
reeve ledger: INTACT — sequences 6,001–10,000 verified (head at start: 10,000); 1–6,000 purged (retention record 0, 2026-10-25); 0 earlier entries not covered (5,000 when the chain started, 5,000 purged); anchored at #10,000 (checkpoint 2026-10-25 14:00 UTC), 0 entries after it
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
Checkpoints older than the boundary read as purged. Deleting entries a retention record does
|
|
467
|
+
not cover, or editing or deleting a record, is BROKEN (`retention_altered` or `missing`), as
|
|
468
|
+
before.
|
|
469
|
+
|
|
259
470
|
A dedicated key decouples the two: `config.ledger_key = Rails.application.credentials.reeve_ledger_key`.
|
|
260
471
|
Keys must be at least 32 bytes and are only ever printed as 16-character fingerprints.
|
|
261
472
|
Without a key, every guarded call fails closed; under `audit_failure_mode = :warn` it is
|
|
@@ -271,22 +482,32 @@ recorded unchained instead, logged each time, and reported by verification.
|
|
|
271
482
|
- the chain record removed while chained rows remain;
|
|
272
483
|
- an entry whose key id or key epoch was changed, or that names an epoch with no record, and
|
|
273
484
|
an epoch record that was changed, or removed while entries name it (one forged under a key
|
|
274
|
-
you do not hold reads as unverifiable, never intact)
|
|
485
|
+
you do not hold reads as unverifiable, never intact);
|
|
486
|
+
- with checkpoints: any entry the latest checkpoint saw removed, even with the head rewound;
|
|
487
|
+
the whole ledger deleted or replaced; an epoch record a checkpoint named, removed or changed;
|
|
488
|
+
a checkpoint edited, or forged without a key you hold.
|
|
275
489
|
|
|
276
490
|
**Not detected** — stated plainly, because these are the limits of the guarantee:
|
|
277
491
|
|
|
278
492
|
1. **A leaked key.** Anyone holding it can rewrite the chain from any point forward. By
|
|
279
|
-
default that is anyone holding `secret_key_base`.
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
493
|
+
default that is anyone holding `secret_key_base`. Not past a checkpoint made before the
|
|
494
|
+
leak, though: rewriting an entry it saw changes the digest it names.
|
|
495
|
+
2. **The newest rows deleted by someone who also rewinds the chain record's head**, unless a
|
|
496
|
+
checkpoint saw them. Rows after the latest checkpoint stay exposed; the summary line counts
|
|
497
|
+
them.
|
|
498
|
+
3. **The whole ledger deleted, chain record included**, without a checkpoint. Reported as
|
|
499
|
+
*no chain*, never as a pass, but indistinguishable from a ledger that was never chained.
|
|
500
|
+
And checkpoints are only as safe as their store: one an attacker can delete protects nothing.
|
|
284
501
|
4. **Rows written before you upgraded.** They are counted as "not covered" and are exactly
|
|
285
502
|
as trustworthy as they were before. Likewise, entries without a key epoch — chained by
|
|
286
503
|
0.6.0, before or during the upgrade — are checked as 0.6.0 checked them: a rewritten key id
|
|
287
504
|
on one of them reads as unverifiable, not as tampering.
|
|
288
505
|
5. **A key you no longer hold.** Entries it signed cannot be checked, by anyone. Epochs make
|
|
289
506
|
that visible and name the epoch; they do not undo it.
|
|
507
|
+
6. **What a purge removed.** Without an archive, nobody can read it again; the retention record
|
|
508
|
+
proves only that it existed, verified, and was removed on purpose. A row edited inside the
|
|
509
|
+
range while the purge runs is not reported by the ledger, since it is being removed; the
|
|
510
|
+
archive holds what was read.
|
|
290
511
|
|
|
291
512
|
And it prevents nothing: it detects a change after the fact. The `GRANT INSERT, SELECT` in
|
|
292
513
|
the generated migration is what prevents one. The chain record needs `UPDATE` as well,
|
|
@@ -310,6 +531,12 @@ same door the built-in ones use, or verification reports them as unchained:
|
|
|
310
531
|
Reeve::Audit::Chain.new(entry_class: Reeve::Audit::Entry).append!(row)
|
|
311
532
|
```
|
|
312
533
|
|
|
534
|
+
**Upgrading from 0.7.0:** `bin/rails generate reeve:upgrade && bin/rails db:migrate` adds
|
|
535
|
+
`reeve_audit_retentions`; grant the application's role `SELECT` on it. Nothing is purged until
|
|
536
|
+
you run the task, and anchoring is opt-in. Every summary line now ends with its anchoring
|
|
537
|
+
(`; not anchored` until you set one up), so anything that matches the whole text line should
|
|
538
|
+
read `FORMAT=json` instead, whose existing keys are unchanged.
|
|
539
|
+
|
|
313
540
|
**Upgrading from 0.6.0:** `bin/rails generate reeve:upgrade && bin/rails db:migrate` before
|
|
314
541
|
the new code serves traffic, and grant `INSERT, SELECT` on `reeve_audit_chain_epochs`. The next guarded call opens epoch 0;
|
|
315
542
|
earlier entries are listed as "before epoch 0" and checked as before.
|
|
@@ -19,6 +19,15 @@
|
|
|
19
19
|
# -- the key epochs: written once each, never updated
|
|
20
20
|
# GRANT INSERT, SELECT ON reeve_audit_chain_epochs TO app_role;
|
|
21
21
|
# GRANT USAGE ON SEQUENCE reeve_audit_chain_epochs_id_seq TO app_role;
|
|
22
|
+
# -- retention records: the application only reads them
|
|
23
|
+
# GRANT SELECT ON reeve_audit_retentions TO app_role;
|
|
24
|
+
#
|
|
25
|
+
# -- the retention role, used only by `bin/rails reeve:purge_ledger`
|
|
26
|
+
# GRANT SELECT ON reeve_audit_entries, reeve_audit_chains, reeve_audit_chain_epochs,
|
|
27
|
+
# reeve_audit_retentions TO retention_role;
|
|
28
|
+
# GRANT DELETE ON reeve_audit_entries TO retention_role;
|
|
29
|
+
# GRANT INSERT ON reeve_audit_retentions TO retention_role;
|
|
30
|
+
# GRANT USAGE ON SEQUENCE reeve_audit_retentions_id_seq TO retention_role;
|
|
22
31
|
#
|
|
23
32
|
# Every row is chained to the one before it with a keyed digest, and names the key epoch
|
|
24
33
|
# that signed it (audit-entry contract 4),
|
|
@@ -30,10 +39,10 @@
|
|
|
30
39
|
# reeve_audit_chains; that role needs UPDATE there, which is why the chain table is
|
|
31
40
|
# granted separately from the ledger.
|
|
32
41
|
#
|
|
33
|
-
#
|
|
34
|
-
#
|
|
35
|
-
#
|
|
36
|
-
#
|
|
42
|
+
# The table grows until you purge it. Deleting rows by hand breaks the chain by design and
|
|
43
|
+
# is reported as tampering; `bin/rails reeve:purge_ledger` removes the oldest entries up to
|
|
44
|
+
# a checkpoint, archives them, and leaves a signed record in reeve_audit_retentions so the
|
|
45
|
+
# rest still verifies. It runs on the retention role above, never the application's.
|
|
37
46
|
#
|
|
38
47
|
# `invocation_id` is a UUID stored as a string rather than a native `uuid` column,
|
|
39
48
|
# because reeve supports every database ActiveRecord does and only PostgreSQL has the
|
|
@@ -133,5 +142,21 @@ class CreateReeveAuditEntries < ActiveRecord::Migration[7.0]
|
|
|
133
142
|
|
|
134
143
|
add_index :reeve_audit_chain_epochs, :epoch, unique: true
|
|
135
144
|
add_index :reeve_audit_chain_epochs, :key_id, unique: true
|
|
145
|
+
|
|
146
|
+
create_table :reeve_audit_retentions do |t|
|
|
147
|
+
t.integer :purge_number, null: false
|
|
148
|
+
t.bigint :through_sequence, null: false
|
|
149
|
+
t.string :through_digest, null: false, limit: 64
|
|
150
|
+
t.bigint :previous_through, null: false
|
|
151
|
+
t.bigint :removed_chained, null: false
|
|
152
|
+
t.bigint :removed_pre_chain, null: false
|
|
153
|
+
t.boolean :archived, null: false
|
|
154
|
+
t.string :archive_sha256, limit: 64
|
|
155
|
+
t.datetime :purged_at, null: false, precision: 6
|
|
156
|
+
t.string :key_id, null: false, limit: 16
|
|
157
|
+
t.string :retention_digest, null: false, limit: 64
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
add_index :reeve_audit_retentions, :purge_number, unique: true
|
|
136
161
|
end
|
|
137
162
|
end
|
|
@@ -102,3 +102,13 @@ end
|
|
|
102
102
|
|
|
103
103
|
# The ledger itself is opt-in, so the core stays loadable without ActiveRecord.
|
|
104
104
|
require "reeve/audit"
|
|
105
|
+
|
|
106
|
+
# Checkpoints of the chain, kept where the database's users cannot change them,
|
|
107
|
+
# catch what the chain alone cannot: the newest entries removed with the head
|
|
108
|
+
# rewound, or the whole ledger deleted. Run `bin/rails reeve:checkpoint_ledger`
|
|
109
|
+
# on a schedule; `reeve:verify_ledger` then checks against them. One anchor per
|
|
110
|
+
# environment, on storage outside the database host — a local file protects
|
|
111
|
+
# nothing:
|
|
112
|
+
#
|
|
113
|
+
# Reeve.config.ledger_anchor =
|
|
114
|
+
# Reeve::Audit::FileAnchor.new("/mnt/ledger-anchor/#{Rails.env}.checkpoints")
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Ledger retention. Generated by `bin/rails generate reeve:upgrade`. The audit-entry contract
|
|
4
|
+
# stays 4: no column of reeve_audit_entries changes.
|
|
5
|
+
#
|
|
6
|
+
# Adds reeve_audit_retentions, where `bin/rails reeve:purge_ledger` records each purge: the
|
|
7
|
+
# checkpoint it purged up to, how many rows it removed, and the archive it wrote, signed with
|
|
8
|
+
# the ledger key. Verification reads it to tell a purge from a deletion. Nothing is purged
|
|
9
|
+
# until you run the task.
|
|
10
|
+
#
|
|
11
|
+
# The application role only reads the new table. The retention role, which only the purge
|
|
12
|
+
# uses, needs DELETE on the ledger and INSERT here:
|
|
13
|
+
#
|
|
14
|
+
# GRANT SELECT ON reeve_audit_retentions TO app_role;
|
|
15
|
+
# GRANT SELECT ON reeve_audit_entries, reeve_audit_chains, reeve_audit_chain_epochs,
|
|
16
|
+
# reeve_audit_retentions TO retention_role;
|
|
17
|
+
# GRANT DELETE ON reeve_audit_entries TO retention_role;
|
|
18
|
+
# GRANT INSERT ON reeve_audit_retentions TO retention_role;
|
|
19
|
+
class AddLedgerRetentionToReeveAuditEntries < ActiveRecord::Migration[7.0]
|
|
20
|
+
def change
|
|
21
|
+
create_table :reeve_audit_retentions do |t|
|
|
22
|
+
t.integer :purge_number, null: false
|
|
23
|
+
t.bigint :through_sequence, null: false
|
|
24
|
+
t.string :through_digest, null: false, limit: 64
|
|
25
|
+
t.bigint :previous_through, null: false
|
|
26
|
+
t.bigint :removed_chained, null: false
|
|
27
|
+
t.bigint :removed_pre_chain, null: false
|
|
28
|
+
t.boolean :archived, null: false
|
|
29
|
+
t.string :archive_sha256, limit: 64
|
|
30
|
+
t.datetime :purged_at, null: false, precision: 6
|
|
31
|
+
t.string :key_id, null: false, limit: 16
|
|
32
|
+
t.string :retention_digest, null: false, limit: 64
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
add_index :reeve_audit_retentions, :purge_number, unique: true
|
|
36
|
+
end
|
|
37
|
+
end
|
|
@@ -58,6 +58,15 @@ module Reeve
|
|
|
58
58
|
creates: %w[reeve_audit_chain_epochs],
|
|
59
59
|
template: "add_key_epochs_to_reeve_audit_entries.rb.tt",
|
|
60
60
|
destination: "add_key_epochs_to_reeve_audit_entries.rb"
|
|
61
|
+
},
|
|
62
|
+
# Retention (specs/005) adds a table and no column, so the contract stays 4.
|
|
63
|
+
{
|
|
64
|
+
contract: 4,
|
|
65
|
+
note: :retention,
|
|
66
|
+
adds: [],
|
|
67
|
+
creates: %w[reeve_audit_retentions],
|
|
68
|
+
template: "add_ledger_retention_to_reeve_audit_entries.rb.tt",
|
|
69
|
+
destination: "add_ledger_retention_to_reeve_audit_entries.rb"
|
|
61
70
|
}
|
|
62
71
|
].freeze
|
|
63
72
|
|
|
@@ -124,7 +133,7 @@ module Reeve
|
|
|
124
133
|
Existing rows are stamped with the contract they were actually written under,
|
|
125
134
|
not with the current one.
|
|
126
135
|
NEXT
|
|
127
|
-
message + UpgradeNotes.for(@emitted
|
|
136
|
+
message + UpgradeNotes.for(@emitted)
|
|
128
137
|
end
|
|
129
138
|
|
|
130
139
|
# Asked of the table rather than of a version the host recorded: the table is the
|
|
@@ -38,13 +38,24 @@ module Reeve
|
|
|
38
38
|
The application role needs INSERT and SELECT on reeve_audit_chain_epochs.
|
|
39
39
|
EPOCHS
|
|
40
40
|
|
|
41
|
-
|
|
41
|
+
# Retention changes nothing until the host runs a purge, but the purge needs a role the
|
|
42
|
+
# application does not have, and the host should hear that before it needs one.
|
|
43
|
+
RETENTION = <<~RETENTION
|
|
44
|
+
|
|
45
|
+
The ledger can now be purged (reeve_audit_retentions): bin/rails reeve:purge_ledger
|
|
46
|
+
removes the oldest entries up to a checkpoint and records the purge, so the rest still
|
|
47
|
+
verifies. Nothing is purged until you run it. The application role needs SELECT on
|
|
48
|
+
the new table; the purge runs on a retention role of its own (see the migration).
|
|
49
|
+
RETENTION
|
|
50
|
+
|
|
51
|
+
# Keyed by the step's note, or by its contract when it has none.
|
|
52
|
+
BY_STEP = { 3 => CHAIN, 4 => EPOCHS, retention: RETENTION }.freeze
|
|
42
53
|
|
|
43
54
|
module_function
|
|
44
55
|
|
|
45
|
-
# The notes for the
|
|
46
|
-
def for(
|
|
47
|
-
|
|
56
|
+
# The notes for the steps just written, oldest first.
|
|
57
|
+
def for(steps)
|
|
58
|
+
steps.filter_map { |step| BY_STEP[step.fetch(:note, step[:contract])] }.join
|
|
48
59
|
end
|
|
49
60
|
end
|
|
50
61
|
end
|