eventsourcingdb-merkle 2.1.0 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -79,7 +79,7 @@ eventsourcingdb-merkle merkle-root --backup=backup.json
79
79
  The command outputs the Merkle root, the total number of events, and the ID of the last event it covers:
80
80
 
81
81
  ```
82
- Merkle Root: d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
82
+ Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
83
83
  Total events: 4
84
84
  Last event ID: 3
85
85
  ```
@@ -94,6 +94,27 @@ eventsourcingdb-merkle merkle-root --backup=backup-2025.json --last-event-id=3
94
94
 
95
95
  So you do not have to keep the older backup to prove against its root. If the backup or the database holds no event with that ID, the command refuses rather than calculating a root over other events.
96
96
 
97
+ #### Salting the Leaves
98
+
99
+ A proof names the hashes next to the path from its event to the root, and the first of them belongs to a neighboring event. That hash does not reveal the event, but whoever holds the proof and has a guess about the neighbor, such as its type, its subject, its time, and its data, can calculate its hash and so check the guess. Where your events can be guessed like that, salt the leaves of the tree: a random value in front of every leaf, which only a proof of that leaf discloses, makes the other hashes in a proof tell nothing.
100
+
101
+ The tool derives the salt of every leaf from a key, so that no salt has to be kept. Generate the key once, and pass it with `--salt-key-file` whenever you calculate the root, or a proof against it:
102
+
103
+ ```shell
104
+ eventsourcingdb-merkle generate-salt-key --output=merkle.key
105
+ eventsourcingdb-merkle merkle-root --backup=backup.json --salt-key-file=merkle.key
106
+ ```
107
+
108
+ The key file holds 64 hexadecimal digits. `generate-salt-key` never overwrites a file, and it lets nobody write the key, and on Linux and macOS, only the owner of the file read it. Windows knows only whether a file is read-only, so there, who can read the key follows from the permissions of its folder. Keep the key as long as the root, and as safely as a password: a root calculated with it is calculated again with that key only, so without it, no further proof can be generated for the root, and whoever has it can calculate the salt of every leaf. A proof carries the salt of its own leaf, so verifying it needs no key.
109
+
110
+ #### Calculating a Root of Version 1
111
+
112
+ Releases before 3.0 built the tree in another way, which is version 1 of the tree: it hashes leaves and inner nodes alike, and pairs the last node of a level with itself where it has no partner, so that a list of hashes has the same root as the same list with its last hash twice. A root of version 2 starts with `esdb:merkle:v2:`, a root of version 1 has no prefix. To calculate a root of version 1 again, or to prove an event against one, pass `--tree-version=1`. Version 1 knows no salts:
113
+
114
+ ```shell
115
+ eventsourcingdb-merkle merkle-root --backup=backup.json --tree-version=1
116
+ ```
117
+
97
118
  ### Validating Individual Events
98
119
 
99
120
  Sometimes you want to verify the integrity of a single event without processing the entire backup. The `validate-event-hash` command provides two ways to do this.
@@ -159,8 +180,8 @@ The command outputs the proof in a human-readable format:
159
180
  ```
160
181
  Merkle Proof for Event 42
161
182
 
162
- Event Hash: abc123...
163
- Merkle Root: d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
183
+ Event Hash: esdb:hash:v2:abc123...
184
+ Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
164
185
 
165
186
  Sibling Hashes (3):
166
187
  1. def456... (left)
@@ -174,6 +195,20 @@ To prove an event against the root of an earlier state, pass its last event ID w
174
195
  eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42 --last-event-id=1234566
175
196
  ```
176
197
 
198
+ To prove an event against a root whose leaves are salted, pass the same key with `--salt-key-file`, and to prove it against a root of version 1, pass `--tree-version=1`, as for `merkle-root`. A proof against a salted root carries the salt of the leaf of its event, right after the hash of the event:
199
+
200
+ ```shell
201
+ eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42 --salt-key-file=merkle.key
202
+ ```
203
+
204
+ ```
205
+ Merkle Proof for Event 42
206
+
207
+ Event Hash: esdb:hash:v2:abc123...
208
+ Salt: 9c1f0b...
209
+ Merkle Root: esdb:merkle:v2:5f0c3e...
210
+ ```
211
+
177
212
  #### Outputting JSON
178
213
 
179
214
  To get the proof as JSON for programmatic use, add the `--json` flag:
@@ -187,7 +222,7 @@ The output can then be saved to a file and shared with others:
187
222
  ```json
188
223
  {
189
224
  "eventId": "42",
190
- "eventHash": "abc123...",
225
+ "eventHash": "esdb:hash:v2:abc123...",
191
226
  "siblingHashes": [
192
227
  {
193
228
  "hash": "def456...",
@@ -202,10 +237,12 @@ The output can then be saved to a file and shared with others:
202
237
  "position": "left"
203
238
  }
204
239
  ],
205
- "merkleRoot": "d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab"
240
+ "merkleRoot": "esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab"
206
241
  }
207
242
  ```
208
243
 
244
+ A proof against a salted root holds its salt in `salt`, right after `eventHash`.
245
+
209
246
  ### Verifying Merkle Proofs
210
247
 
211
248
  Once you have a Merkle proof, you can verify it without needing the original backup file. This is particularly useful when you receive a proof from someone else and want to verify that the event is indeed part of the backup with the claimed Merkle root.
@@ -222,13 +259,15 @@ Or with a JSON string:
222
259
  eventsourcingdb-merkle verify-proof '{"eventId":"42","eventHash":"...","siblingHashes":[],"merkleRoot":"..."}'
223
260
  ```
224
261
 
262
+ The command verifies proofs of either version of the tree, which it tells apart by their root, and takes the salt of a salted proof from the proof itself, so it needs neither the backup nor the key.
263
+
225
264
  If the proof is valid, the command confirms this and displays the event details:
226
265
 
227
266
  ```
228
267
  ✓ Merkle proof is valid
229
268
  Event ID: 42
230
- Event Hash: abc123...
231
- Merkle Root: d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
269
+ Event Hash: esdb:hash:v2:abc123...
270
+ Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
232
271
  ```
233
272
 
234
273
  If the proof is invalid, the command indicates this with a non-zero exit code:
@@ -236,8 +275,8 @@ If the proof is invalid, the command indicates this with a non-zero exit code:
236
275
  ```
237
276
  ✗ Merkle proof is invalid
238
277
  Event ID: 42
239
- Event Hash: abc123...
240
- Merkle Root: d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
278
+ Event Hash: esdb:hash:v2:abc123...
279
+ Merkle Root: esdb:merkle:v2:d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
241
280
  ```
242
281
 
243
282
  ## Understanding the Technical Details
@@ -249,7 +288,13 @@ The tool implements Merkle trees using SHA-256 hashing, following the EventSourc
249
288
 
250
289
  In both versions, the data is hashed as it appears in the backup or in the event given with `--event`, rather than parsed and serialized again, since that would change escape sequences such as `\u003c` for `<` and the notation of numbers. Only the whitespace between its tokens is left out, which EventSourcingDB never writes, so that an event formatted afterwards, as by a viewer or by `jq` without `-c`, still verifies.
251
290
 
252
- The Merkle tree is constructed by treating event hashes, as they are stored, prefix included, as leaf nodes. Internal nodes are created by hashing the concatenation of their child hashes. If the number of nodes at any level is odd, the last node is duplicated to create a complete binary tree. This process continues until a single root hash remains.
291
+ The Merkle tree of version 2 follows [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962), as the tree of [eventsourcingdb-auditing](https://github.com/thenativeweb/eventsourcingdb-auditing) does:
292
+
293
+ - The data of a leaf is the hash of its event as it is stored, prefix included, as text in UTF-8. Where the leaves are salted, its salt comes first: the HMAC-SHA256 of that text under the salt key. Since nothing marks where the salt ends, the tree takes only hashes in the form of version 1 or 2, which give the data of a leaf other lengths with a salt than without one, and refuses an event with any other hash, as `verify-proof` refuses a proof of one.
294
+ - The hash of a leaf is the SHA-256 of the byte `0x00` followed by its data, and the hash of an inner node is the SHA-256 of the byte `0x01` followed by the hashes of its two children, so that neither can pass for the other.
295
+ - The tree splits its leaves at the largest power of two smaller than their number, and never duplicates a node. Its root is the hash of its top node in hexadecimal, after the prefix `esdb:merkle:v2:`.
296
+
297
+ The tree of version 1 takes the stored hashes as its leaves, and hashes the concatenation of the hashes of its two children, as text, for an inner node. Where a level has an odd number of nodes, it pairs the last one with itself.
253
298
 
254
299
  Backup files are expected to be in one of the two NDJSON formats that EventSourcingDB writes, and the tool reads every backup that a restore of EventSourcingDB takes. It checks the structure of a backup, the hashes, and the chain, but neither the values of the events nor the schemas, which a restore checks with the rules of the server:
255
300
 
@@ -0,0 +1,27 @@
1
+ import { Flags } from '@oclif/core';
2
+ import BaseCommand from '../lib/BaseCommand.js';
3
+ import { describeError } from '../lib/describeError.js';
4
+ import { writeSaltKey } from '../lib/saltKey.js';
5
+ export default class GenerateSaltKey extends BaseCommand {
6
+ static description = 'Generate a key to salt the leaves of Merkle trees with';
7
+ static examples = ['<%= config.bin %> <%= command.id %> --output=merkle.key'];
8
+ static flags = {
9
+ output: Flags.string({
10
+ description: 'File to write the salt key to; an existing file is never overwritten',
11
+ required: true,
12
+ }),
13
+ };
14
+ static args = {};
15
+ async run() {
16
+ const { flags } = await this.parse(GenerateSaltKey);
17
+ try {
18
+ await writeSaltKey(flags.output);
19
+ this.log(`✓ Salt key written to ${flags.output}`);
20
+ this.log('Keep it safe: without it, no further proof can be generated for a root calculated with it.');
21
+ }
22
+ catch (error) {
23
+ this.log(`✗ Failed to generate salt key: ${describeError(error)}`);
24
+ process.exitCode = 1;
25
+ }
26
+ }
27
+ }
@@ -5,12 +5,14 @@ import { describeError } from '../lib/describeError.js';
5
5
  import { parseEventId } from '../lib/eventId.js';
6
6
  import { generateMerkleProof } from '../lib/merkleTree.js';
7
7
  import { getSource, readEvents, sourceFlags } from '../lib/source.js';
8
+ import { getTree, getTreeFlags, getTreeHashes, readTreeOptions } from '../lib/tree.js';
8
9
  export default class GetProof extends BaseCommand {
9
10
  static description = 'Generate a Merkle proof for a specific event of an EventSourcingDB backup file or a running EventSourcingDB';
10
11
  static examples = [
11
12
  '<%= config.bin %> <%= command.id %> --backup=backup.json --event-id=42',
12
13
  '<%= config.bin %> <%= command.id %> --backup=/path/to/backup.json --event-id=0 --json',
13
14
  '<%= config.bin %> <%= command.id %> --backup=backup.json --event-id=42 --last-event-id=1234566',
15
+ '<%= config.bin %> <%= command.id %> --backup=backup.json --event-id=42 --salt-key-file=merkle.key',
14
16
  '<%= config.bin %> <%= command.id %> --esdb-url=http://localhost:3000 --esdb-api-token="$ESDB_API_TOKEN" --event-id=42',
15
17
  ];
16
18
  static flags = {
@@ -20,6 +22,7 @@ export default class GetProof extends BaseCommand {
20
22
  description: 'ID of the last event the root covers, to prove against the root of an earlier state',
21
23
  }),
22
24
  json: Flags.boolean({ description: 'Output proof as JSON' }),
25
+ ...getTreeFlags('Version of the Merkle tree: 1 to prove against roots that releases before 3.0 calculated'),
23
26
  };
24
27
  static args = {};
25
28
  async run() {
@@ -39,7 +42,13 @@ export default class GetProof extends BaseCommand {
39
42
  this.refuseCall(`--last-event-id has to be an event ID, but is '${lastEventId}'`);
40
43
  return;
41
44
  }
45
+ const tree = getTree(flags);
46
+ if ('refusal' in tree) {
47
+ this.refuseCall(tree.refusal);
48
+ return;
49
+ }
42
50
  try {
51
+ const treeOptions = await readTreeOptions(tree);
43
52
  const events = await readEvents(source, { lastEventId });
44
53
  // An event after the last one is not part of the tree, so it has no
45
54
  // proof against its root, although the source holds it.
@@ -58,8 +67,8 @@ export default class GetProof extends BaseCommand {
58
67
  process.exitCode = 1;
59
68
  return;
60
69
  }
61
- const hashes = events.map(e => e.hash);
62
- const proofData = generateMerkleProof(hashes, event.hash);
70
+ const hashes = getTreeHashes(events, treeOptions);
71
+ const proofData = generateMerkleProof(hashes, events.indexOf(event), treeOptions);
63
72
  const proof = {
64
73
  eventId: event.id,
65
74
  ...proofData,
@@ -71,6 +80,9 @@ export default class GetProof extends BaseCommand {
71
80
  this.log(`Merkle Proof for Event ${proof.eventId}`);
72
81
  this.log('');
73
82
  this.log(`Event Hash: ${proof.eventHash}`);
83
+ if (proof.salt !== undefined) {
84
+ this.log(`Salt: ${proof.salt}`);
85
+ }
74
86
  this.log(`Merkle Root: ${proof.merkleRoot}`);
75
87
  this.log('');
76
88
  this.log(`Sibling Hashes (${proof.siblingHashes.length}):`);
@@ -2,13 +2,16 @@ import { Flags } from '@oclif/core';
2
2
  import BaseCommand from '../lib/BaseCommand.js';
3
3
  import { describeError } from '../lib/describeError.js';
4
4
  import { parseEventId } from '../lib/eventId.js';
5
- import { buildMerkleRoot } from '../lib/merkleTree.js';
5
+ import { calculateMerkleRoot } from '../lib/merkleTree.js';
6
6
  import { getSource, readEvents, sourceFlags } from '../lib/source.js';
7
+ import { getTree, getTreeFlags, getTreeHashes, readTreeOptions } from '../lib/tree.js';
7
8
  export default class MerkleRoot extends BaseCommand {
8
9
  static description = 'Calculate the Merkle root for an EventSourcingDB backup file or a running EventSourcingDB';
9
10
  static examples = [
10
11
  '<%= config.bin %> <%= command.id %> --backup=backup.json',
11
12
  '<%= config.bin %> <%= command.id %> --backup=/path/to/backup.json --last-event-id=41',
13
+ '<%= config.bin %> <%= command.id %> --backup=backup.json --salt-key-file=merkle.key',
14
+ '<%= config.bin %> <%= command.id %> --backup=backup.json --tree-version=1',
12
15
  '<%= config.bin %> <%= command.id %> --esdb-url=http://localhost:3000 --esdb-api-token="$ESDB_API_TOKEN"',
13
16
  ];
14
17
  static flags = {
@@ -16,6 +19,7 @@ export default class MerkleRoot extends BaseCommand {
16
19
  'last-event-id': Flags.string({
17
20
  description: 'ID of the last event to cover, to calculate the root of an earlier state again',
18
21
  }),
22
+ ...getTreeFlags('Version of the Merkle tree: 1 for roots that releases before 3.0 calculated'),
19
23
  };
20
24
  static args = {};
21
25
  async run() {
@@ -30,10 +34,16 @@ export default class MerkleRoot extends BaseCommand {
30
34
  this.refuseCall(`--last-event-id has to be an event ID, but is '${lastEventId}'`);
31
35
  return;
32
36
  }
37
+ const tree = getTree(flags);
38
+ if ('refusal' in tree) {
39
+ this.refuseCall(tree.refusal);
40
+ return;
41
+ }
33
42
  try {
43
+ const treeOptions = await readTreeOptions(tree);
34
44
  const events = await readEvents(source, { lastEventId });
35
- const hashes = events.map(event => event.hash);
36
- const merkleRoot = buildMerkleRoot(hashes);
45
+ const hashes = getTreeHashes(events, treeOptions);
46
+ const merkleRoot = calculateMerkleRoot(hashes, treeOptions);
37
47
  // The ID of the last event is what calculates this root again later,
38
48
  // from a source that has grown since.
39
49
  this.log(`Merkle Root: ${merkleRoot}`);
@@ -1,6 +1,6 @@
1
1
  import { Args } from '@oclif/core';
2
2
  import BaseCommand from '../lib/BaseCommand.js';
3
- import { verifyMerkleProof } from '../lib/merkleTree.js';
3
+ import { findProofStructureProblem, verifyMerkleProof } from '../lib/merkleTree.js';
4
4
  export default class VerifyProof extends BaseCommand {
5
5
  static description = 'Verify a Merkle proof';
6
6
  static examples = [
@@ -27,10 +27,10 @@ export default class VerifyProof extends BaseCommand {
27
27
  const fileContent = readFileSync(args.proof, 'utf-8');
28
28
  proof = JSON.parse(fileContent);
29
29
  }
30
- // Validate proof structure
31
- if (!proof.eventHash || !proof.merkleRoot || !proof.siblingHashes) {
30
+ const structureProblem = findProofStructureProblem(proof);
31
+ if (structureProblem !== undefined) {
32
32
  this.log('✗ Invalid proof structure');
33
- this.log('Must contain eventHash, merkleRoot, and siblingHashes');
33
+ this.log(structureProblem);
34
34
  process.exitCode = 1;
35
35
  return;
36
36
  }
package/dist/lib/hash.js CHANGED
@@ -50,6 +50,10 @@ export const assertKnownHashVersion = (storedHash) => {
50
50
  const [version = ''] = storedHash.slice(hashPrefix.length).split(':');
51
51
  throw new UnsupportedHashVersionError(version);
52
52
  };
53
+ // A stored hash of version 1 is a SHA-256 in hexadecimal, and one of version 2
54
+ // the same after its prefix, as EventSourcingDB writes them.
55
+ const knownStoredHashPattern = new RegExp(`^(${hashVersion2Prefix})?[0-9a-f]{64}$`, 'u');
56
+ export const isKnownStoredHash = (text) => knownStoredHashPattern.test(text);
53
57
  // calculateEventHash computes the hash of an event in the version that its
54
58
  // stored hash names, since an event keeps the version of the hash it was
55
59
  // written with. An event without a stored hash names no version, which is what
@@ -1,81 +1,119 @@
1
- import { hashPair } from './hash.js';
2
- export const buildMerkleRoot = (hashes) => {
3
- if (hashes.length === 0) {
4
- throw new Error('Cannot build Merkle root from empty list');
1
+ import { isKnownStoredHash } from './hash.js';
2
+ import { isObject } from './isObject.js';
3
+ import { buildVersion1Root, generateVersion1Proof, verifyVersion1Proof, } from './merkleTreeVersion1.js';
4
+ import { buildTree, calculateRootFromAuditPath, hashLeaf, } from './merkleTreeVersion2.js';
5
+ import { deriveSalt } from './saltKey.js';
6
+ // A root that names the version of its tree starts with this prefix. A root
7
+ // without it is of version 1, which predates the prefix.
8
+ const merkleRootPrefix = 'esdb:merkle:';
9
+ const merkleRootVersion2Prefix = `${merkleRootPrefix}v2:`;
10
+ // A salt and the hash of a node are SHA-256 digests, which a proof carries in
11
+ // hexadecimal. Decoding takes text that is not exactly that only as far as it
12
+ // can, so a proof that holds more than the digest would still verify.
13
+ const digestPattern = /^[0-9a-f]{64}$/iu;
14
+ export class UnsupportedMerkleTreeVersionError extends Error {
15
+ constructor(version) {
16
+ super(`Unsupported Merkle tree version: ${version}`);
17
+ this.name = 'UnsupportedMerkleTreeVersionError';
5
18
  }
6
- let currentLevel = [...hashes];
7
- while (currentLevel.length > 1) {
8
- const nextLevel = [];
9
- for (let i = 0; i < currentLevel.length; i += 2) {
10
- const left = currentLevel[i];
11
- const right = i + 1 < currentLevel.length ? currentLevel[i + 1] : currentLevel[i];
12
- if (left === undefined || right === undefined) {
13
- throw new Error('Unexpected undefined hash while building Merkle tree');
14
- }
15
- nextLevel.push(hashPair(left, right));
16
- }
17
- currentLevel = nextLevel;
19
+ }
20
+ // The data of a leaf of version 2 is the stored hash of its event, as text, which
21
+ // covers both versions of the hash alike, preceded by its salt where the leaves
22
+ // are salted.
23
+ const getLeafHash = (storedHash, salt) => hashLeaf(Buffer.concat([salt ?? Buffer.alloc(0), Buffer.from(storedHash, 'utf-8')]));
24
+ const getSalt = (saltKey, storedHash) => saltKey === undefined ? undefined : deriveSalt(saltKey, storedHash);
25
+ const getLeafHashes = (hashes, saltKey) => hashes.map(hash => getLeafHash(hash, getSalt(saltKey, hash)));
26
+ const formatVersion2Root = (root) => `${merkleRootVersion2Prefix}${root.toString('hex')}`;
27
+ export const calculateMerkleRoot = (hashes, options) => {
28
+ if (options.treeVersion === 1) {
29
+ return buildVersion1Root(hashes);
18
30
  }
19
- const [root] = currentLevel;
20
- if (root === undefined) {
21
- throw new Error('Failed to compute Merkle root');
22
- }
23
- return root;
31
+ return formatVersion2Root(buildTree(getLeafHashes(hashes, options.saltKey)).root);
24
32
  };
25
- export const generateMerkleProof = (hashes, targetHash) => {
26
- const targetIndex = hashes.indexOf(targetHash);
27
- if (targetIndex === -1) {
33
+ export const generateMerkleProof = (hashes, index, options) => {
34
+ const eventHash = hashes[index];
35
+ if (eventHash === undefined) {
28
36
  throw new Error('Target hash not found in the list');
29
37
  }
30
- const siblingHashes = [];
31
- let currentLevel = [...hashes];
32
- let currentIndex = targetIndex;
33
- while (currentLevel.length > 1) {
34
- const nextLevel = [];
35
- let nextIndex = 0;
36
- for (let i = 0; i < currentLevel.length; i += 2) {
37
- const left = currentLevel[i];
38
- const right = i + 1 < currentLevel.length ? currentLevel[i + 1] : currentLevel[i];
39
- if (left === undefined || right === undefined) {
40
- throw new Error('Unexpected undefined hash while generating Merkle proof');
41
- }
42
- if (i === currentIndex || i + 1 === currentIndex) {
43
- if (currentIndex % 2 === 0) {
44
- // Target is on the left
45
- if (i + 1 < currentLevel.length) {
46
- siblingHashes.push({ hash: right, position: 'right' });
47
- }
48
- }
49
- else {
50
- // Target is on the right
51
- siblingHashes.push({ hash: left, position: 'left' });
52
- }
53
- nextIndex = Math.floor(currentIndex / 2);
54
- }
55
- nextLevel.push(hashPair(left, right));
56
- }
57
- currentLevel = nextLevel;
58
- currentIndex = nextIndex;
59
- }
60
- const [merkleRoot] = currentLevel;
61
- if (merkleRoot === undefined) {
62
- throw new Error('Failed to compute Merkle root');
38
+ if (options.treeVersion === 1) {
39
+ return generateVersion1Proof(hashes, eventHash);
63
40
  }
41
+ const salt = getSalt(options.saltKey, eventHash);
42
+ const { root, auditPath } = buildTree(getLeafHashes(hashes, options.saltKey), index);
64
43
  return {
65
- eventHash: targetHash,
66
- siblingHashes,
67
- merkleRoot,
44
+ eventHash,
45
+ ...(salt === undefined ? {} : { salt: salt.toString('hex') }),
46
+ siblingHashes: auditPath.map(({ hash, position }) => ({
47
+ hash: hash.toString('hex'),
48
+ position,
49
+ })),
50
+ merkleRoot: formatVersion2Root(root),
68
51
  };
69
52
  };
53
+ // getMerkleTreeVersion tells the version of the tree that a root names, the way
54
+ // the version of a hash is told: a root without the prefix is of version 1, and
55
+ // every other root with it of a version that the tool does not know.
56
+ export const getMerkleTreeVersion = (merkleRoot) => {
57
+ if (!merkleRoot.startsWith(merkleRootPrefix)) {
58
+ return 1;
59
+ }
60
+ if (merkleRoot.startsWith(merkleRootVersion2Prefix)) {
61
+ return 2;
62
+ }
63
+ const [version = ''] = merkleRoot.slice(merkleRootPrefix.length).split(':');
64
+ throw new UnsupportedMerkleTreeVersionError(version);
65
+ };
66
+ const isSibling = (sibling) => isObject(sibling) &&
67
+ typeof sibling.hash === 'string' &&
68
+ digestPattern.test(sibling.hash) &&
69
+ (sibling.position === 'left' || sibling.position === 'right');
70
+ // findProofStructureProblem describes what keeps a proof from being one that
71
+ // can be verified, or returns undefined if nothing does. A salt is part of the
72
+ // structure, since whether a proof may have one depends on the version of its
73
+ // tree alone. So is the form of the event hash in version 2: the data of a leaf
74
+ // is its salt followed by the stored hash, with nothing between them that tells
75
+ // where the salt ends, so without that form, a proof whose salt took the first
76
+ // bytes of the hash would pass for a proof of the rest of it. With it, the data
77
+ // of a leaf has other lengths with a salt than without one.
78
+ export const findProofStructureProblem = (proof) => {
79
+ if (!isObject(proof) ||
80
+ typeof proof.eventHash !== 'string' ||
81
+ proof.eventHash === '' ||
82
+ typeof proof.merkleRoot !== 'string' ||
83
+ proof.merkleRoot === '' ||
84
+ !Array.isArray(proof.siblingHashes)) {
85
+ return 'Must contain eventHash, merkleRoot, and siblingHashes';
86
+ }
87
+ const hasSalt = proof.salt !== undefined;
88
+ if (getMerkleTreeVersion(proof.merkleRoot) === 1) {
89
+ return hasSalt ? 'A proof of Merkle tree version 1 has no salt' : undefined;
90
+ }
91
+ if (!isKnownStoredHash(proof.eventHash)) {
92
+ return 'The event hash has to be a hash of version 1 or 2';
93
+ }
94
+ if (hasSalt && (typeof proof.salt !== 'string' || !digestPattern.test(proof.salt))) {
95
+ return 'The salt has to be 64 hexadecimal digits';
96
+ }
97
+ if (!proof.siblingHashes.every(isSibling)) {
98
+ return 'Every sibling has to have a hash of 64 hexadecimal digits, and the position left or right';
99
+ }
100
+ return undefined;
101
+ };
102
+ // verifyMerkleProof tells whether a proof leads from the hash of its event to
103
+ // its root, in the version of the tree that the root names. A proof in which
104
+ // findProofStructureProblem finds a problem proves nothing.
70
105
  export const verifyMerkleProof = (proof) => {
71
- let currentHash = proof.eventHash;
72
- for (const { hash, position } of proof.siblingHashes) {
73
- if (position === 'left') {
74
- currentHash = hashPair(hash, currentHash);
75
- }
76
- else {
77
- currentHash = hashPair(currentHash, hash);
78
- }
106
+ if (findProofStructureProblem(proof) !== undefined) {
107
+ return false;
108
+ }
109
+ if (getMerkleTreeVersion(proof.merkleRoot) === 1) {
110
+ return verifyVersion1Proof(proof);
79
111
  }
80
- return currentHash === proof.merkleRoot;
112
+ const auditPath = proof.siblingHashes.map(({ hash, position }) => ({
113
+ hash: Buffer.from(hash, 'hex'),
114
+ position,
115
+ }));
116
+ const salt = proof.salt === undefined ? undefined : Buffer.from(proof.salt, 'hex');
117
+ const root = calculateRootFromAuditPath(getLeafHash(proof.eventHash, salt), auditPath);
118
+ return formatVersion2Root(root) === proof.merkleRoot;
81
119
  };
@@ -0,0 +1,86 @@
1
+ import { hashPair } from './hash.js';
2
+ // The Merkle tree of version 1, which releases before 3.0 calculated. It hashes
3
+ // leaves and inner nodes alike, and pairs a node that is left without a partner
4
+ // with itself, so that a list of hashes has the same root as the same list with
5
+ // its last hash twice. It stays only to calculate the roots that were published
6
+ // with it again, and to verify their proofs.
7
+ export const buildVersion1Root = (hashes) => {
8
+ if (hashes.length === 0) {
9
+ throw new Error('Cannot build Merkle root from empty list');
10
+ }
11
+ let currentLevel = [...hashes];
12
+ while (currentLevel.length > 1) {
13
+ const nextLevel = [];
14
+ for (let i = 0; i < currentLevel.length; i += 2) {
15
+ const left = currentLevel[i];
16
+ const right = i + 1 < currentLevel.length ? currentLevel[i + 1] : currentLevel[i];
17
+ if (left === undefined || right === undefined) {
18
+ throw new Error('Unexpected undefined hash while building Merkle tree');
19
+ }
20
+ nextLevel.push(hashPair(left, right));
21
+ }
22
+ currentLevel = nextLevel;
23
+ }
24
+ const [root] = currentLevel;
25
+ if (root === undefined) {
26
+ throw new Error('Failed to compute Merkle root');
27
+ }
28
+ return root;
29
+ };
30
+ export const generateVersion1Proof = (hashes, targetHash) => {
31
+ const targetIndex = hashes.indexOf(targetHash);
32
+ if (targetIndex === -1) {
33
+ throw new Error('Target hash not found in the list');
34
+ }
35
+ const siblingHashes = [];
36
+ let currentLevel = [...hashes];
37
+ let currentIndex = targetIndex;
38
+ while (currentLevel.length > 1) {
39
+ const nextLevel = [];
40
+ let nextIndex = 0;
41
+ for (let i = 0; i < currentLevel.length; i += 2) {
42
+ const left = currentLevel[i];
43
+ const right = i + 1 < currentLevel.length ? currentLevel[i + 1] : currentLevel[i];
44
+ if (left === undefined || right === undefined) {
45
+ throw new Error('Unexpected undefined hash while generating Merkle proof');
46
+ }
47
+ if (i === currentIndex || i + 1 === currentIndex) {
48
+ if (currentIndex % 2 === 0) {
49
+ // Target is on the left
50
+ if (i + 1 < currentLevel.length) {
51
+ siblingHashes.push({ hash: right, position: 'right' });
52
+ }
53
+ }
54
+ else {
55
+ // Target is on the right
56
+ siblingHashes.push({ hash: left, position: 'left' });
57
+ }
58
+ nextIndex = Math.floor(currentIndex / 2);
59
+ }
60
+ nextLevel.push(hashPair(left, right));
61
+ }
62
+ currentLevel = nextLevel;
63
+ currentIndex = nextIndex;
64
+ }
65
+ const [merkleRoot] = currentLevel;
66
+ if (merkleRoot === undefined) {
67
+ throw new Error('Failed to compute Merkle root');
68
+ }
69
+ return {
70
+ eventHash: targetHash,
71
+ siblingHashes,
72
+ merkleRoot,
73
+ };
74
+ };
75
+ export const verifyVersion1Proof = (proof) => {
76
+ let currentHash = proof.eventHash;
77
+ for (const { hash, position } of proof.siblingHashes) {
78
+ if (position === 'left') {
79
+ currentHash = hashPair(hash, currentHash);
80
+ }
81
+ else {
82
+ currentHash = hashPair(currentHash, hash);
83
+ }
84
+ }
85
+ return currentHash === proof.merkleRoot;
86
+ };
@@ -0,0 +1,51 @@
1
+ import { createHash } from 'node:crypto';
2
+ // RFC 6962 puts a byte of its own in front of what a leaf and an inner node
3
+ // hash, so that neither can pass for the other: without them, the hash of an
4
+ // inner node would also be the hash of a leaf whose data are the hashes of its
5
+ // children.
6
+ const leafPrefix = Buffer.from([0x00]);
7
+ const nodePrefix = Buffer.from([0x01]);
8
+ export const hashLeaf = (data) => createHash('sha256').update(leafPrefix).update(data).digest();
9
+ export const hashNode = (left, right) => createHash('sha256').update(nodePrefix).update(left).update(right).digest();
10
+ // buildTree calculates the root of the tree over the hashes of its leaves, as
11
+ // RFC 6962 defines it, and the audit path of the leaf at the given index. RFC
12
+ // 6962 defines the tree by splitting the leaves at the largest power of two
13
+ // smaller than their number. Pairing the nodes level by level and moving a node
14
+ // that is left without a partner up unchanged builds the same tree, without
15
+ // copying the leaves for every subtree. The node is moved rather than paired
16
+ // with itself, since that would give a list of leaves the same root as the same
17
+ // list with its last leaf twice.
18
+ export const buildTree = (leafHashes, index) => {
19
+ if (leafHashes.length === 0) {
20
+ throw new Error('Cannot build Merkle root from empty list');
21
+ }
22
+ const auditPath = [];
23
+ let level = leafHashes;
24
+ let currentIndex = index;
25
+ while (level.length > 1) {
26
+ if (currentIndex !== undefined) {
27
+ const isLeftChild = currentIndex % 2 === 0;
28
+ const sibling = level[isLeftChild ? currentIndex + 1 : currentIndex - 1];
29
+ if (sibling !== undefined) {
30
+ auditPath.push({ hash: sibling, position: isLeftChild ? 'right' : 'left' });
31
+ }
32
+ currentIndex = Math.floor(currentIndex / 2);
33
+ }
34
+ const nextLevel = [];
35
+ for (let position = 0; position < level.length; position += 2) {
36
+ const left = level[position];
37
+ const right = level[position + 1];
38
+ if (left === undefined) {
39
+ throw new Error('Unexpected undefined hash while building Merkle tree');
40
+ }
41
+ nextLevel.push(right === undefined ? left : hashNode(left, right));
42
+ }
43
+ level = nextLevel;
44
+ }
45
+ const [root] = level;
46
+ if (root === undefined) {
47
+ throw new Error('Failed to compute Merkle root');
48
+ }
49
+ return { root, auditPath };
50
+ };
51
+ export const calculateRootFromAuditPath = (leafHash, auditPath) => auditPath.reduce((hash, sibling) => sibling.position === 'left' ? hashNode(sibling.hash, hash) : hashNode(hash, sibling.hash), leafHash);
@@ -0,0 +1,65 @@
1
+ import { createHmac, randomBytes } from 'node:crypto';
2
+ import { access, open, readFile, unlink } from 'node:fs/promises';
3
+ // A salt key is 32 random bytes: RFC 2104 discourages keys for HMAC that are
4
+ // shorter than the output of the hash, and longer ones add nothing. It is kept
5
+ // as text of 64 hexadecimal digits, so that it can be stored wherever a password
6
+ // can, and whitespace around them is left out, so that a final newline, which
7
+ // one editor adds and another does not, does not make the same key another one.
8
+ const saltKeyLength = 32;
9
+ const saltKeyPattern = /^[0-9a-f]{64}$/iu;
10
+ export const readSaltKey = async (filePath) => {
11
+ try {
12
+ await access(filePath);
13
+ }
14
+ catch {
15
+ throw new Error(`File not found: ${filePath}`);
16
+ }
17
+ const digits = (await readFile(filePath, 'utf-8')).trim();
18
+ // The message does not show what the file holds, since that may be a key
19
+ // that was only written down wrong.
20
+ if (!saltKeyPattern.test(digits)) {
21
+ throw new Error(`the salt key in ${filePath} has to be 64 hexadecimal digits`);
22
+ }
23
+ return Buffer.from(digits, 'hex');
24
+ };
25
+ // writeSaltKey never overwrites a file, since a root whose leaves were salted
26
+ // with a key gets no further proof once the key is gone. Nobody may write the
27
+ // file, so that it is not replaced by accident either, as by a command that
28
+ // redirects its output into it, and on Linux and macOS, only its owner may read
29
+ // it. Windows knows only whether a file is read-only, so there, who may read it
30
+ // follows from the permissions of its folder.
31
+ export const writeSaltKey = async (filePath) => {
32
+ const digits = randomBytes(saltKeyLength).toString('hex');
33
+ let file;
34
+ try {
35
+ file = await open(filePath, 'wx', 0o400);
36
+ }
37
+ catch (error) {
38
+ if (error.code === 'EEXIST') {
39
+ throw new Error(`${filePath} already exists`);
40
+ }
41
+ throw error;
42
+ }
43
+ // A root may be calculated and published with the key right after it was
44
+ // written, so the content of the file reaches the disk before the key counts as
45
+ // written. Its entry in the folder is not flushed on its own, so whether a key
46
+ // written right before a power failure is found afterwards is left to the file
47
+ // system. A file that could not be written completely is removed again, so that
48
+ // it neither passes for a key nor keeps the next attempt from writing one. It
49
+ // is closed first, which Windows needs to remove it, but a failure to close it
50
+ // keeps it neither from being removed nor from reporting why writing it failed.
51
+ try {
52
+ await file.writeFile(`${digits}\n`);
53
+ await file.sync();
54
+ }
55
+ catch (error) {
56
+ await file.close().catch(() => undefined);
57
+ await unlink(filePath);
58
+ throw error;
59
+ }
60
+ await file.close();
61
+ };
62
+ // deriveSalt derives the salt of a leaf from the key and the stored hash of its
63
+ // event, so that no salt has to be kept: whoever holds the key gets the same
64
+ // salts again, and whoever does not cannot tell them from random ones.
65
+ export const deriveSalt = (saltKey, storedHash) => createHmac('sha256', saltKey).update(storedHash, 'utf-8').digest();
@@ -0,0 +1,54 @@
1
+ import { Flags } from '@oclif/core';
2
+ import { isKnownStoredHash } from './hash.js';
3
+ import { readSaltKey } from './saltKey.js';
4
+ // The flags that name the tree a command calculates: its version, and the file
5
+ // with the key that salts its leaves. Version 2 is the default, and version 1
6
+ // stays for the roots that releases before 3.0 calculated. Only the commands
7
+ // that calculate a tree take them, since a proof names its version and carries
8
+ // its salt.
9
+ export const getTreeFlags = (treeVersionDescription) => ({
10
+ 'salt-key-file': Flags.string({
11
+ description: 'File with the salt key to salt the leaves of the tree with',
12
+ }),
13
+ 'tree-version': Flags.string({
14
+ description: treeVersionDescription,
15
+ options: ['1', '2'],
16
+ default: '2',
17
+ }),
18
+ });
19
+ // getTree returns the tree that the flags name, or why they name none: the tree
20
+ // of version 1 has no salts, so a salt key file for it would be ignored, which
21
+ // would leave a root unsalted that was meant to be salted.
22
+ export const getTree = (flags) => {
23
+ const saltKeyFile = flags['salt-key-file'];
24
+ if (flags['tree-version'] === '1') {
25
+ if (saltKeyFile !== undefined) {
26
+ return { refusal: '--salt-key-file and --tree-version=1 cannot be used together' };
27
+ }
28
+ return { treeVersion: 1 };
29
+ }
30
+ return { treeVersion: 2, saltKeyFile };
31
+ };
32
+ // readTreeOptions reads the salt key of a tree, if it has one. A command reads it
33
+ // before its source, so that a key file that cannot be read stops it before it
34
+ // reads a large backup or database for nothing.
35
+ export const readTreeOptions = async (tree) => {
36
+ if (tree.treeVersion === 1 || tree.saltKeyFile === undefined) {
37
+ return { treeVersion: tree.treeVersion };
38
+ }
39
+ return { treeVersion: 2, saltKey: await readSaltKey(tree.saltKeyFile) };
40
+ };
41
+ // getTreeHashes returns the stored hashes of the events, as the leaves of the
42
+ // tree. The tree of version 2 takes only hashes of a form it knows, since only
43
+ // they keep the data of a leaf unambiguous, which is why verify-proof refuses
44
+ // any other: a root over such a leaf would hold a leaf that no proof could
45
+ // prove. The tree of version 1 takes any hash, as it always did.
46
+ export const getTreeHashes = (events, treeOptions) => {
47
+ if (treeOptions.treeVersion === 2) {
48
+ const event = events.find(({ hash }) => !isKnownStoredHash(hash));
49
+ if (event !== undefined) {
50
+ throw new Error(`event ${event.id} has no hash of version 1 or 2`);
51
+ }
52
+ }
53
+ return events.map(({ hash }) => hash);
54
+ };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "eventsourcingdb-merkle",
3
- "version": "2.1.0",
3
+ "version": "3.0.0",
4
4
  "description": "Merkle tree utilities for EventSourcingDB backups and databases.",
5
5
  "author": {
6
6
  "name": "the native web GmbH",