eventsourcingdb-merkle 3.0.0 → 4.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -79,11 +79,13 @@ eventsourcingdb-merkle merkle-root --backup=backup.json
79
79
  The command outputs the Merkle root, the total number of events, and the ID of the last event it covers:
80
80
 
81
81
  ```
82
- Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
82
+ Merkle Root: esdb:merkle:v3:6a0e4b2f9d1c...
83
83
  Total events: 4
84
84
  Last event ID: 3
85
85
  ```
86
86
 
87
+ The root covers the number of events as well, so that a proof against it can name the position of its event, as described under [Verifying Merkle Proofs](#verifying-merkle-proofs).
88
+
87
89
  #### Calculating the Merkle Root of an Earlier State
88
90
 
89
91
  A root that you published once has to be calculated again later, for instance to generate proofs against it. Since events are never changed, a newer backup, or the database itself, holds the events of an older backup, followed by those written since. Pass the last event ID that the root covered with `--last-event-id`, and the command calculates the root over the events up to it, which is the root of the older backup:
@@ -107,12 +109,12 @@ eventsourcingdb-merkle merkle-root --backup=backup.json --salt-key-file=merkle.k
107
109
 
108
110
  The key file holds 64 hexadecimal digits. `generate-salt-key` never overwrites a file, and it lets nobody write the key, and on Linux and macOS, only the owner of the file read it. Windows knows only whether a file is read-only, so there, who can read the key follows from the permissions of its folder. Keep the key as long as the root, and as safely as a password: a root calculated with it is calculated again with that key only, so without it, no further proof can be generated for the root, and whoever has it can calculate the salt of every leaf. A proof carries the salt of its own leaf, so verifying it needs no key.
109
111
 
110
- #### Calculating a Root of Version 1
112
+ #### Calculating a Root of an Earlier Version
111
113
 
112
- Releases before 3.0 built the tree in another way, which is version 1 of the tree: it hashes leaves and inner nodes alike, and pairs the last node of a level with itself where it has no partner, so that a list of hashes has the same root as the same list with its last hash twice. A root of version 2 starts with `esdb:merkle:v2:`, a root of version 1 has no prefix. To calculate a root of version 1 again, or to prove an event against one, pass `--tree-version=1`. Version 1 knows no salts:
114
+ Releases before 4.0 built the tree in other ways. Release 3.0 built version 2 of the tree, which has the leaves and inner nodes of version 3, but a root that does not cover the number of events, and proofs that follow the sides of their siblings, so that they do not bind the ID of their event. Releases before 3.0 built version 1, which hashes leaves and inner nodes alike, and pairs the last node of a level with itself where it has no partner, so that a list of hashes has the same root as the same list with its last hash twice. A root of version 3 starts with `esdb:merkle:v3:`, a root of version 2 with `esdb:merkle:v2:`, and a root of version 1 has no prefix. Version 1 knows no salts. To calculate a root of an earlier version again, or to prove an event against one, pass `--tree-version=1` or `--tree-version=2`:
113
115
 
114
116
  ```shell
115
- eventsourcingdb-merkle merkle-root --backup=backup.json --tree-version=1
117
+ eventsourcingdb-merkle merkle-root --backup=backup.json --tree-version=2
116
118
  ```
117
119
 
118
120
  ### Validating Individual Events
@@ -181,21 +183,24 @@ The command outputs the proof in a human-readable format:
181
183
  Merkle Proof for Event 42
182
184
 
183
185
  Event Hash: esdb:hash:v2:abc123...
184
- Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
186
+ Merkle Root: esdb:merkle:v3:6a0e4b2f9d1c...
187
+ Last Event ID: 99
185
188
 
186
189
  Sibling Hashes (3):
187
- 1. def456... (left)
188
- 2. ghi789... (right)
189
- 3. jkl012... (left)
190
+ 1. def456...
191
+ 2. ghi789...
192
+ 3. jkl012...
190
193
  ```
191
194
 
195
+ A proof of version 3 names the ID of the last event that its root covers, which fixes the position of its event in the tree. A proof of version 1 or 2 names no last event ID, but the side of each sibling instead, such as `def456... (left)`.
196
+
192
197
  To prove an event against the root of an earlier state, pass its last event ID with `--last-event-id`, as for `merkle-root`. An event after it is not part of that tree, so the command refuses to prove it:
193
198
 
194
199
  ```shell
195
200
  eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42 --last-event-id=1234566
196
201
  ```
197
202
 
198
- To prove an event against a root whose leaves are salted, pass the same key with `--salt-key-file`, and to prove it against a root of version 1, pass `--tree-version=1`, as for `merkle-root`. A proof against a salted root carries the salt of the leaf of its event, right after the hash of the event:
203
+ To prove an event against a root whose leaves are salted, pass the same key with `--salt-key-file`, and to prove it against a root of an earlier version, pass `--tree-version`, as for `merkle-root`. A proof against a salted root carries the salt of the leaf of its event, right after the hash of the event:
199
204
 
200
205
  ```shell
201
206
  eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42 --salt-key-file=merkle.key
@@ -206,7 +211,8 @@ Merkle Proof for Event 42
206
211
 
207
212
  Event Hash: esdb:hash:v2:abc123...
208
213
  Salt: 9c1f0b...
209
- Merkle Root: esdb:merkle:v2:5f0c3e...
214
+ Merkle Root: esdb:merkle:v3:5f0c3e...
215
+ Last Event ID: 99
210
216
  ```
211
217
 
212
218
  #### Outputting JSON
@@ -224,24 +230,16 @@ The output can then be saved to a file and shared with others:
224
230
  "eventId": "42",
225
231
  "eventHash": "esdb:hash:v2:abc123...",
226
232
  "siblingHashes": [
227
- {
228
- "hash": "def456...",
229
- "position": "left"
230
- },
231
- {
232
- "hash": "ghi789...",
233
- "position": "right"
234
- },
235
- {
236
- "hash": "jkl012...",
237
- "position": "left"
238
- }
233
+ "def456...",
234
+ "ghi789...",
235
+ "jkl012..."
239
236
  ],
240
- "merkleRoot": "esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab"
237
+ "merkleRoot": "esdb:merkle:v3:6a0e4b2f9d1c...",
238
+ "lastEventId": "99"
241
239
  }
242
240
  ```
243
241
 
244
- A proof against a salted root holds its salt in `salt`, right after `eventHash`.
242
+ A proof against a salted root holds its salt in `salt`, right after `eventHash`. A proof of version 1 or 2 holds no `lastEventId`, and every sibling in it is an object with its `hash` and its `position`, `left` or `right`.
245
243
 
246
244
  ### Verifying Merkle Proofs
247
245
 
@@ -256,10 +254,10 @@ eventsourcingdb-merkle verify-proof proof.json
256
254
  Or with a JSON string:
257
255
 
258
256
  ```shell
259
- eventsourcingdb-merkle verify-proof '{"eventId":"42","eventHash":"...","siblingHashes":[],"merkleRoot":"..."}'
257
+ eventsourcingdb-merkle verify-proof '{"eventId":"42","eventHash":"...","siblingHashes":["..."],"merkleRoot":"esdb:merkle:v3:...","lastEventId":"99"}'
260
258
  ```
261
259
 
262
- The command verifies proofs of either version of the tree, which it tells apart by their root, and takes the salt of a salted proof from the proof itself, so it needs neither the backup nor the key.
260
+ The command verifies proofs of every version of the tree, which it tells apart by their root, and takes the salt of a salted proof from the proof itself, so it needs neither the backup nor the key.
263
261
 
264
262
  If the proof is valid, the command confirms this and displays the event details:
265
263
 
@@ -267,7 +265,8 @@ If the proof is valid, the command confirms this and displays the event details:
267
265
  ✓ Merkle proof is valid
268
266
  Event ID: 42
269
267
  Event Hash: esdb:hash:v2:abc123...
270
- Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
268
+ Merkle Root: esdb:merkle:v3:6a0e4b2f9d1c...
269
+ Last Event ID: 99
271
270
  ```
272
271
 
273
272
  If the proof is invalid, the command indicates this with a non-zero exit code:
@@ -276,9 +275,12 @@ If the proof is invalid, the command indicates this with a non-zero exit code:
276
275
  ✗ Merkle proof is invalid
277
276
  Event ID: 42
278
277
  Event Hash: esdb:hash:v2:abc123...
279
- Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
278
+ Merkle Root: esdb:merkle:v3:6a0e4b2f9d1c...
279
+ Last Event ID: 99
280
280
  ```
281
281
 
282
+ A valid proof shows that the event is part of the tree with the root it names, so compare that root with the one that was published. In version 3, it also shows that the event is the one with its ID, in a tree that ends at the last event ID: its path leads to the root from the leaf at that position only, and the root covers the number of events, so neither ID can be changed without the proof becoming invalid. A proof of version 1 or 2 does not bind the ID of its event, so whoever hands it on could change it, and its ID is only as trustworthy as whoever gave you the proof.
283
+
282
284
  ## Understanding the Technical Details
283
285
 
284
286
  The tool implements Merkle trees using SHA-256 hashing, following the EventSourcingDB specification for event hash calculation, which knows two versions. An event keeps the version of the hash it was written with, so the tool verifies every hash in the version it names:
@@ -288,11 +290,14 @@ The tool implements Merkle trees using SHA-256 hashing, following the EventSourc
288
290
 
289
291
  In both versions, the data is hashed as it appears in the backup or in the event given with `--event`, rather than parsed and serialized again, since that would change escape sequences such as `\u003c` for `<` and the notation of numbers. Only the whitespace between its tokens is left out, which EventSourcingDB never writes, so that an event formatted afterwards, as by a viewer or by `jq` without `-c`, still verifies.
290
292
 
291
- The Merkle tree of version 2 follows [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), as the tree of [eventsourcingdb-auditing](https://github.com/thenativeweb/eventsourcingdb-auditing) does:
293
+ The Merkle trees of versions 2 and 3 follow [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), as the tree of [eventsourcingdb-auditing](https://github.com/thenativeweb/eventsourcingdb-auditing) does:
292
294
 
293
295
  - The data of a leaf is the hash of its event as it is stored, prefix included, as text in UTF-8. Where the leaves are salted, its salt comes first: the HMAC-SHA256 of that text under the salt key. Since nothing marks where the salt ends, the tree takes only hashes in the form of version 1 or 2, which give the data of a leaf other lengths with a salt than without one, and refuses an event with any other hash, as `verify-proof` refuses a proof of one.
294
296
  - The hash of a leaf is the SHA-256 of the byte `0x00` followed by its data, and the hash of an inner node is the SHA-256 of the byte `0x01` followed by the hashes of its two children, so that neither can pass for the other.
295
- - The tree splits its leaves at the largest power of two smaller than their number, and never duplicates a node. Its root is the hash of its top node in hexadecimal, after the prefix `esdb:merkle:v2:`.
297
+ - The tree splits its leaves at the largest power of two smaller than their number, and never duplicates a node.
298
+ - The root of version 2 is the hash of the top node in hexadecimal, after the prefix `esdb:merkle:v2:`. The root of version 3 is the SHA-256 of the byte `0x02`, the number of leaves as 8 bytes, big-endian, and the hash of the top node, in hexadecimal, after the prefix `esdb:merkle:v3:`.
299
+
300
+ A proof of version 3 is verified as [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), section 2.1.3.2, verifies an inclusion proof, with the event ID as the index of its leaf and one more than the last event ID as the size of the tree, from which the side of every sibling follows. EventSourcingDB numbers its events from 0 without gaps, so the leaf at an index is the event with that ID, and the tree of version 3 refuses a source whose IDs do not run like that. Certificate Transparency, for which RFC 9162 was written, signs the size of a tree together with its root. The root of version 3 hashes it in instead, since without it, a proof could bring a size of its own, under which its path would lead to the root from the leaf at another index.
296
301
 
297
302
  The tree of version 1 takes the stored hashes as its leaves, and hashes the concatenation of the hashes of its two children, as text, for an inner node. Where a level has an odd number of nodes, it pairs the last one with itself.
298
303
 
@@ -22,7 +22,7 @@ export default class GetProof extends BaseCommand {
22
22
  description: 'ID of the last event the root covers, to prove against the root of an earlier state',
23
23
  }),
24
24
  json: Flags.boolean({ description: 'Output proof as JSON' }),
25
- ...getTreeFlags('Version of the Merkle tree: 1 to prove against roots that releases before 3.0 calculated'),
25
+ ...getTreeFlags('Version of the Merkle tree: 1 to prove against roots that releases before 3.0 calculated, 2 to prove against roots that release 3.0 calculated'),
26
26
  };
27
27
  static args = {};
28
28
  async run() {
@@ -84,10 +84,17 @@ export default class GetProof extends BaseCommand {
84
84
  this.log(`Salt: ${proof.salt}`);
85
85
  }
86
86
  this.log(`Merkle Root: ${proof.merkleRoot}`);
87
+ if ('lastEventId' in proof) {
88
+ this.log(`Last Event ID: ${proof.lastEventId}`);
89
+ }
87
90
  this.log('');
88
91
  this.log(`Sibling Hashes (${proof.siblingHashes.length}):`);
92
+ // The siblings of a proof of version 3 have no sides, since they follow
93
+ // from the position of the event and the size of the tree.
89
94
  for (const [index, sibling] of proof.siblingHashes.entries()) {
90
- this.log(` ${index + 1}. ${sibling.hash} (${sibling.position})`);
95
+ this.log(typeof sibling === 'string'
96
+ ? ` ${index + 1}. ${sibling}`
97
+ : ` ${index + 1}. ${sibling.hash} (${sibling.position})`);
91
98
  }
92
99
  }
93
100
  catch (error) {
@@ -19,7 +19,7 @@ export default class MerkleRoot extends BaseCommand {
19
19
  'last-event-id': Flags.string({
20
20
  description: 'ID of the last event to cover, to calculate the root of an earlier state again',
21
21
  }),
22
- ...getTreeFlags('Version of the Merkle tree: 1 for roots that releases before 3.0 calculated'),
22
+ ...getTreeFlags('Version of the Merkle tree: 1 for roots that releases before 3.0 calculated, 2 for roots that release 3.0 calculated'),
23
23
  };
24
24
  static args = {};
25
25
  async run() {
@@ -4,6 +4,7 @@ import { findProofStructureProblem, verifyMerkleProof } from '../lib/merkleTree.
4
4
  export default class VerifyProof extends BaseCommand {
5
5
  static description = 'Verify a Merkle proof';
6
6
  static examples = [
7
+ '<%= config.bin %> <%= command.id %> \'{"eventId":"42","eventHash":"...","siblingHashes":["..."],"merkleRoot":"esdb:merkle:v3:...","lastEventId":"99"}\'',
7
8
  '<%= config.bin %> <%= command.id %> \'{"eventId":"42","eventHash":"...","siblingHashes":[],"merkleRoot":"..."}\'',
8
9
  '<%= config.bin %> <%= command.id %> proof.json',
9
10
  ];
@@ -35,18 +36,18 @@ export default class VerifyProof extends BaseCommand {
35
36
  return;
36
37
  }
37
38
  const isValid = verifyMerkleProof(proof);
38
- if (!isValid) {
39
- this.log(`✗ Merkle proof is invalid`);
40
- this.log(`Event ID: ${proof.eventId}`);
41
- this.log(`Event Hash: ${proof.eventHash}`);
42
- this.log(`Merkle Root: ${proof.merkleRoot}`);
43
- process.exitCode = 1;
44
- return;
45
- }
46
- this.log(`✓ Merkle proof is valid`);
39
+ this.log(isValid ? '✓ Merkle proof is valid' : '✗ Merkle proof is invalid');
47
40
  this.log(`Event ID: ${proof.eventId}`);
48
41
  this.log(`Event Hash: ${proof.eventHash}`);
49
42
  this.log(`Merkle Root: ${proof.merkleRoot}`);
43
+ // The root of version 3 covers the size of its tree, so the ID of the last
44
+ // event tells which state of the store it is the root of.
45
+ if ('lastEventId' in proof) {
46
+ this.log(`Last Event ID: ${proof.lastEventId}`);
47
+ }
48
+ if (!isValid) {
49
+ process.exitCode = 1;
50
+ }
50
51
  }
51
52
  catch (error) {
52
53
  this.log(`✗ Failed to verify Merkle proof: ${error instanceof Error ? error.message : String(error)}`);
@@ -2,11 +2,13 @@ import { isKnownStoredHash } from './hash.js';
2
2
  import { isObject } from './isObject.js';
3
3
  import { buildVersion1Root, generateVersion1Proof, verifyVersion1Proof, } from './merkleTreeVersion1.js';
4
4
  import { buildTree, calculateRootFromAuditPath, hashLeaf, } from './merkleTreeVersion2.js';
5
+ import { buildVersion3Tree, calculateVersion3Root } from './merkleTreeVersion3.js';
5
6
  import { deriveSalt } from './saltKey.js';
6
7
  // A root that names the version of its tree starts with this prefix. A root
7
8
  // without it is of version 1, which predates the prefix.
8
9
  const merkleRootPrefix = 'esdb:merkle:';
9
10
  const merkleRootVersion2Prefix = `${merkleRootPrefix}v2:`;
11
+ const merkleRootVersion3Prefix = `${merkleRootPrefix}v3:`;
10
12
  // A salt and the hash of a node are SHA-256 digests, which a proof carries in
11
13
  // hexadecimal. Decoding takes text that is not exactly that only as far as it
12
14
  // can, so a proof that holds more than the digest would still verify.
@@ -17,18 +19,23 @@ export class UnsupportedMerkleTreeVersionError extends Error {
17
19
  this.name = 'UnsupportedMerkleTreeVersionError';
18
20
  }
19
21
  }
20
- // The data of a leaf of version 2 is the stored hash of its event, as text, which
21
- // covers both versions of the hash alike, preceded by its salt where the leaves
22
- // are salted.
22
+ // The data of a leaf of versions 2 and 3 is the stored hash of its event, as
23
+ // text, which covers both versions of the hash alike, preceded by its salt where
24
+ // the leaves are salted.
23
25
  const getLeafHash = (storedHash, salt) => hashLeaf(Buffer.concat([salt ?? Buffer.alloc(0), Buffer.from(storedHash, 'utf-8')]));
24
26
  const getSalt = (saltKey, storedHash) => saltKey === undefined ? undefined : deriveSalt(saltKey, storedHash);
25
27
  const getLeafHashes = (hashes, saltKey) => hashes.map(hash => getLeafHash(hash, getSalt(saltKey, hash)));
26
28
  const formatVersion2Root = (root) => `${merkleRootVersion2Prefix}${root.toString('hex')}`;
29
+ const formatVersion3Root = (root) => `${merkleRootVersion3Prefix}${root.toString('hex')}`;
27
30
  export const calculateMerkleRoot = (hashes, options) => {
28
31
  if (options.treeVersion === 1) {
29
32
  return buildVersion1Root(hashes);
30
33
  }
31
- return formatVersion2Root(buildTree(getLeafHashes(hashes, options.saltKey)).root);
34
+ const leafHashes = getLeafHashes(hashes, options.saltKey);
35
+ if (options.treeVersion === 2) {
36
+ return formatVersion2Root(buildTree(leafHashes).root);
37
+ }
38
+ return formatVersion3Root(buildVersion3Tree(leafHashes).root);
32
39
  };
33
40
  export const generateMerkleProof = (hashes, index, options) => {
34
41
  const eventHash = hashes[index];
@@ -39,15 +46,30 @@ export const generateMerkleProof = (hashes, index, options) => {
39
46
  return generateVersion1Proof(hashes, eventHash);
40
47
  }
41
48
  const salt = getSalt(options.saltKey, eventHash);
42
- const { root, auditPath } = buildTree(getLeafHashes(hashes, options.saltKey), index);
43
- return {
49
+ const leaf = {
44
50
  eventHash,
45
51
  ...(salt === undefined ? {} : { salt: salt.toString('hex') }),
46
- siblingHashes: auditPath.map(({ hash, position }) => ({
47
- hash: hash.toString('hex'),
48
- position,
49
- })),
50
- merkleRoot: formatVersion2Root(root),
52
+ };
53
+ const leafHashes = getLeafHashes(hashes, options.saltKey);
54
+ if (options.treeVersion === 2) {
55
+ const { root, auditPath } = buildTree(leafHashes, index);
56
+ return {
57
+ ...leaf,
58
+ siblingHashes: auditPath.map(({ hash, position }) => ({
59
+ hash: hash.toString('hex'),
60
+ position,
61
+ })),
62
+ merkleRoot: formatVersion2Root(root),
63
+ };
64
+ }
65
+ // The leaf at an index of the tree of version 3 is the event with that ID, so
66
+ // the last event of the tree has the ID one below the number of its leaves.
67
+ const { root, auditPath } = buildVersion3Tree(leafHashes, index);
68
+ return {
69
+ ...leaf,
70
+ siblingHashes: auditPath.map(hash => hash.toString('hex')),
71
+ merkleRoot: formatVersion3Root(root),
72
+ lastEventId: String(hashes.length - 1),
51
73
  };
52
74
  };
53
75
  // getMerkleTreeVersion tells the version of the tree that a root names, the way
@@ -60,21 +82,37 @@ export const getMerkleTreeVersion = (merkleRoot) => {
60
82
  if (merkleRoot.startsWith(merkleRootVersion2Prefix)) {
61
83
  return 2;
62
84
  }
85
+ if (merkleRoot.startsWith(merkleRootVersion3Prefix)) {
86
+ return 3;
87
+ }
63
88
  const [version = ''] = merkleRoot.slice(merkleRootPrefix.length).split(':');
64
89
  throw new UnsupportedMerkleTreeVersionError(version);
65
90
  };
66
- const isSibling = (sibling) => isObject(sibling) &&
67
- typeof sibling.hash === 'string' &&
68
- digestPattern.test(sibling.hash) &&
91
+ const isDigest = (value) => typeof value === 'string' && digestPattern.test(value);
92
+ const isText = (value) => typeof value === 'string';
93
+ // A sibling of version 1 or 2 is an object with its hash and its side. The hash
94
+ // of a sibling of version 2 is a digest, but that of version 1 has no fixed
95
+ // form, since the tree of version 1 takes any stored hash as a leaf, so the
96
+ // first sibling of a proof may be the stored hash of a neighbor.
97
+ const isSiblingWithSide = (isHash) => (sibling) => isObject(sibling) &&
98
+ isHash(sibling.hash) &&
69
99
  (sibling.position === 'left' || sibling.position === 'right');
100
+ // An ID in a proof of version 3 is written as EventSourcingDB writes IDs, since
101
+ // get-proof writes it rather than anyone typing it, so that the same proof has
102
+ // one notation only.
103
+ const eventIdPattern = /^(0|[1-9][0-9]*)$/u;
104
+ const isEventId = (value) => typeof value === 'string' && eventIdPattern.test(value);
70
105
  // findProofStructureProblem describes what keeps a proof from being one that
71
- // can be verified, or returns undefined if nothing does. A salt is part of the
72
- // structure, since whether a proof may have one depends on the version of its
73
- // tree alone. So is the form of the event hash in version 2: the data of a leaf
74
- // is its salt followed by the stored hash, with nothing between them that tells
75
- // where the salt ends, so without that form, a proof whose salt took the first
76
- // bytes of the hash would pass for a proof of the rest of it. With it, the data
77
- // of a leaf has other lengths with a salt than without one.
106
+ // can be verified, or returns undefined if nothing does. Which fields a proof may
107
+ // or has to have depends on the version of its tree alone, so they are part of
108
+ // the structure, as is the form of every field. The form of the event hash in
109
+ // versions 2 and 3 matters for another reason: the data of a leaf is its salt
110
+ // followed by the stored hash, with nothing between them that tells where the
111
+ // salt ends, so without that form, a proof whose salt took the first bytes of
112
+ // the hash would pass for a proof of the rest of it. With it, the data of a leaf
113
+ // has other lengths with a salt than without one. Whether the fields fit one
114
+ // another, such as an event ID beyond the last one, is left to the verification,
115
+ // as RFC 9162 leaves it.
78
116
  export const findProofStructureProblem = (proof) => {
79
117
  if (!isObject(proof) ||
80
118
  typeof proof.eventHash !== 'string' ||
@@ -84,36 +122,71 @@ export const findProofStructureProblem = (proof) => {
84
122
  !Array.isArray(proof.siblingHashes)) {
85
123
  return 'Must contain eventHash, merkleRoot, and siblingHashes';
86
124
  }
125
+ const treeVersion = getMerkleTreeVersion(proof.merkleRoot);
87
126
  const hasSalt = proof.salt !== undefined;
88
- if (getMerkleTreeVersion(proof.merkleRoot) === 1) {
89
- return hasSalt ? 'A proof of Merkle tree version 1 has no salt' : undefined;
127
+ const hasLastEventId = proof.lastEventId !== undefined;
128
+ if (treeVersion === 1) {
129
+ if (hasSalt) {
130
+ return 'A proof of Merkle tree version 1 has no salt';
131
+ }
132
+ if (hasLastEventId) {
133
+ return 'A proof of Merkle tree version 1 has no last event ID';
134
+ }
135
+ if (!proof.siblingHashes.every(isSiblingWithSide(isText))) {
136
+ return 'Every sibling has to have a hash, and the position left or right';
137
+ }
138
+ return undefined;
139
+ }
140
+ if (treeVersion === 2 && hasLastEventId) {
141
+ return 'A proof of Merkle tree version 2 has no last event ID';
142
+ }
143
+ if (treeVersion === 3) {
144
+ if (proof.eventId === undefined || !hasLastEventId) {
145
+ return 'A proof of Merkle tree version 3 has to contain eventId and lastEventId';
146
+ }
147
+ if (!isEventId(proof.eventId)) {
148
+ return 'The event ID has to be digits alone, without a sign or leading zeros';
149
+ }
90
150
  }
91
151
  if (!isKnownStoredHash(proof.eventHash)) {
92
152
  return 'The event hash has to be a hash of version 1 or 2';
93
153
  }
94
- if (hasSalt && (typeof proof.salt !== 'string' || !digestPattern.test(proof.salt))) {
154
+ if (hasSalt && !isDigest(proof.salt)) {
95
155
  return 'The salt has to be 64 hexadecimal digits';
96
156
  }
97
- if (!proof.siblingHashes.every(isSibling)) {
98
- return 'Every sibling has to have a hash of 64 hexadecimal digits, and the position left or right';
157
+ if (treeVersion === 2) {
158
+ return proof.siblingHashes.every(isSiblingWithSide(isDigest))
159
+ ? undefined
160
+ : 'Every sibling has to have a hash of 64 hexadecimal digits, and the position left or right';
161
+ }
162
+ if (!proof.siblingHashes.every(isDigest)) {
163
+ return 'Every sibling has to be a hash of 64 hexadecimal digits';
164
+ }
165
+ if (!isEventId(proof.lastEventId)) {
166
+ return 'The last event ID has to be digits alone, without a sign or leading zeros';
99
167
  }
100
168
  return undefined;
101
169
  };
102
170
  // verifyMerkleProof tells whether a proof leads from the hash of its event to
103
- // its root, in the version of the tree that the root names. A proof in which
104
- // findProofStructureProblem finds a problem proves nothing.
171
+ // its root, in the version of the tree that the root names, and in version 3,
172
+ // whether it does so from the leaf of its event in a tree that ends at its last
173
+ // event. A proof in which findProofStructureProblem finds a problem proves
174
+ // nothing, and one that passes it has the fields of its version.
105
175
  export const verifyMerkleProof = (proof) => {
106
176
  if (findProofStructureProblem(proof) !== undefined) {
107
177
  return false;
108
178
  }
109
- if (getMerkleTreeVersion(proof.merkleRoot) === 1) {
179
+ const treeVersion = getMerkleTreeVersion(proof.merkleRoot);
180
+ if (treeVersion === 1) {
110
181
  return verifyVersion1Proof(proof);
111
182
  }
112
- const auditPath = proof.siblingHashes.map(({ hash, position }) => ({
113
- hash: Buffer.from(hash, 'hex'),
114
- position,
115
- }));
116
183
  const salt = proof.salt === undefined ? undefined : Buffer.from(proof.salt, 'hex');
117
- const root = calculateRootFromAuditPath(getLeafHash(proof.eventHash, salt), auditPath);
118
- return formatVersion2Root(root) === proof.merkleRoot;
184
+ const leafHash = getLeafHash(proof.eventHash, salt);
185
+ if (treeVersion === 2) {
186
+ const auditPath = proof.siblingHashes.map(({ hash, position }) => ({ hash: Buffer.from(hash, 'hex'), position }));
187
+ return formatVersion2Root(calculateRootFromAuditPath(leafHash, auditPath)) === proof.merkleRoot;
188
+ }
189
+ const { eventId, lastEventId, siblingHashes } = proof;
190
+ const root = calculateVersion3Root(leafHash, BigInt(eventId), BigInt(lastEventId) + 1n, siblingHashes.map(hash => Buffer.from(hash, 'hex')));
191
+ return root !== undefined && formatVersion3Root(root) === proof.merkleRoot;
119
192
  };
@@ -0,0 +1,65 @@
1
+ import { createHash } from 'node:crypto';
2
+ import { buildTree, hashNode } from './merkleTreeVersion2.js';
3
+ // The tree of version 3 is the tree of version 2, but its root covers the size
4
+ // of the tree as well. A proof of version 3 locates its leaf by the index and
5
+ // the size of the tree, as RFC 9162 does, and the index of an event is its ID.
6
+ // RFC 9162 binds the index only where the size is known, though: a path that
7
+ // leads to the root from one leaf of a tree of one size can lead to it from a
8
+ // leaf at another index of a tree of another size. Certificate Transparency
9
+ // signs the size along with the root, but a root that the tool calculates is
10
+ // published as it is, so the root hashes the size in, behind a byte of its own,
11
+ // as RFC 6962 puts one in front of a leaf and of an inner node.
12
+ const rootPrefix = Buffer.from([0x02]);
13
+ // RFC 6962 encodes the size of a tree in 8 bytes, as it does in a tree head, so
14
+ // a tree has at most 2^64 - 1 leaves.
15
+ const treeSizeLength = 8;
16
+ const maximumTreeSize = 2n ** 64n - 1n;
17
+ const hashRoot = (treeSize, root) => {
18
+ const size = Buffer.alloc(treeSizeLength);
19
+ size.writeBigUInt64BE(treeSize);
20
+ return createHash('sha256').update(rootPrefix).update(size).update(root).digest();
21
+ };
22
+ export const buildVersion3Tree = (leafHashes, index) => {
23
+ const { root, auditPath } = buildTree(leafHashes, index);
24
+ return {
25
+ root: hashRoot(BigInt(leafHashes.length), root),
26
+ auditPath: auditPath.map(({ hash }) => hash),
27
+ };
28
+ };
29
+ // calculateVersion3Root calculates the root from the audit path of the leaf at
30
+ // the given index of a tree of the given size, as section 2.1.3.2 of RFC 9162
31
+ // verifies an inclusion proof, or returns undefined where the path does not fit
32
+ // that leaf in that tree. The side of each sibling follows from the index of the
33
+ // node on the path and the last index of its level: a node with an odd index is
34
+ // a right child, and so is the last node of a level, which has no partner on its
35
+ // right and is moved up unchanged until it is one. Both indexes halve from one
36
+ // level to the next, and the path fits where it ends at the level of the root.
37
+ export const calculateVersion3Root = (leafHash, leafIndex, treeSize, auditPath) => {
38
+ if (leafIndex >= treeSize || treeSize > maximumTreeSize) {
39
+ return undefined;
40
+ }
41
+ let index = leafIndex;
42
+ let lastIndex = treeSize - 1n;
43
+ let hash = leafHash;
44
+ for (const sibling of auditPath) {
45
+ if (lastIndex === 0n) {
46
+ return undefined;
47
+ }
48
+ if (index % 2n === 1n || index === lastIndex) {
49
+ hash = hashNode(sibling, hash);
50
+ while (index % 2n === 0n && index !== 0n) {
51
+ index /= 2n;
52
+ lastIndex /= 2n;
53
+ }
54
+ }
55
+ else {
56
+ hash = hashNode(hash, sibling);
57
+ }
58
+ index /= 2n;
59
+ lastIndex /= 2n;
60
+ }
61
+ if (lastIndex !== 0n) {
62
+ return undefined;
63
+ }
64
+ return hashRoot(treeSize, hash);
65
+ };
package/dist/lib/tree.js CHANGED
@@ -2,18 +2,18 @@ import { Flags } from '@oclif/core';
2
2
  import { isKnownStoredHash } from './hash.js';
3
3
  import { readSaltKey } from './saltKey.js';
4
4
  // The flags that name the tree a command calculates: its version, and the file
5
- // with the key that salts its leaves. Version 2 is the default, and version 1
6
- // stays for the roots that releases before 3.0 calculated. Only the commands
7
- // that calculate a tree take them, since a proof names its version and carries
8
- // its salt.
5
+ // with the key that salts its leaves. Version 3 is the default, and versions 1
6
+ // and 2 stay for the roots that the releases before 4.0 calculated. Only the
7
+ // commands that calculate a tree take them, since a proof names its version and
8
+ // carries its salt.
9
9
  export const getTreeFlags = (treeVersionDescription) => ({
10
10
  'salt-key-file': Flags.string({
11
11
  description: 'File with the salt key to salt the leaves of the tree with',
12
12
  }),
13
13
  'tree-version': Flags.string({
14
14
  description: treeVersionDescription,
15
- options: ['1', '2'],
16
- default: '2',
15
+ options: ['1', '2', '3'],
16
+ default: '3',
17
17
  }),
18
18
  });
19
19
  // getTree returns the tree that the flags name, or why they name none: the tree
@@ -27,7 +27,7 @@ export const getTree = (flags) => {
27
27
  }
28
28
  return { treeVersion: 1 };
29
29
  }
30
- return { treeVersion: 2, saltKeyFile };
30
+ return { treeVersion: flags['tree-version'] === '2' ? 2 : 3, saltKeyFile };
31
31
  };
32
32
  // readTreeOptions reads the salt key of a tree, if it has one. A command reads it
33
33
  // before its source, so that a key file that cannot be read stops it before it
@@ -36,18 +36,27 @@ export const readTreeOptions = async (tree) => {
36
36
  if (tree.treeVersion === 1 || tree.saltKeyFile === undefined) {
37
37
  return { treeVersion: tree.treeVersion };
38
38
  }
39
- return { treeVersion: 2, saltKey: await readSaltKey(tree.saltKeyFile) };
39
+ return { treeVersion: tree.treeVersion, saltKey: await readSaltKey(tree.saltKeyFile) };
40
40
  };
41
41
  // getTreeHashes returns the stored hashes of the events, as the leaves of the
42
- // tree. The tree of version 2 takes only hashes of a form it knows, since only
43
- // they keep the data of a leaf unambiguous, which is why verify-proof refuses
44
- // any other: a root over such a leaf would hold a leaf that no proof could
45
- // prove. The tree of version 1 takes any hash, as it always did.
42
+ // tree, or throws where the tree could not prove each of them. The trees of
43
+ // versions 2 and 3 take only hashes of a form they know, since only they keep
44
+ // the data of a leaf unambiguous, which is why verify-proof refuses any other.
45
+ // The tree of version 3 also takes the events only where their IDs are their
46
+ // positions, as EventSourcingDB numbers them, in the notation in which it
47
+ // writes them, since a proof of version 3 names the position of its leaf by the
48
+ // ID of its event. Otherwise, the root would cover leaves that no proof could
49
+ // prove. The tree of version 1 takes any events, as it always did, and so does
50
+ // the tree of version 2 as far as their IDs are concerned, as release 3.0 did.
46
51
  export const getTreeHashes = (events, treeOptions) => {
47
- if (treeOptions.treeVersion === 2) {
48
- const event = events.find(({ hash }) => !isKnownStoredHash(hash));
49
- if (event !== undefined) {
50
- throw new Error(`event ${event.id} has no hash of version 1 or 2`);
52
+ if (treeOptions.treeVersion !== 1) {
53
+ for (const [position, event] of events.entries()) {
54
+ if (treeOptions.treeVersion === 3 && event.id !== String(position)) {
55
+ throw new Error(`event ${event.id} is at position ${position} of the source, but the IDs of the events have to run from 0 without gaps`);
56
+ }
57
+ if (!isKnownStoredHash(event.hash)) {
58
+ throw new Error(`event ${event.id} has no hash of version 1 or 2`);
59
+ }
51
60
  }
52
61
  }
53
62
  return events.map(({ hash }) => hash);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "eventsourcingdb-merkle",
3
- "version": "3.0.0",
3
+ "version": "4.0.1",
4
4
  "description": "Merkle tree utilities for EventSourcingDB backups and databases.",
5
5
  "author": {
6
6
  "name": "the native web GmbH",