eventsourcingdb-merkle 1.2.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -22,10 +22,10 @@ npx eventsourcingdb-merkle [COMMAND]
22
22
 
23
23
  ### Validating Event Chains
24
24
 
25
- To ensure that a backup file has not been tampered with or corrupted, call the `validate-chain` command and provide the path to your backup file. The command verifies that each event's hash is correctly calculated, that the predecessor hash of each event matches the hash of the previous event, and that the first event has a null predecessor hash as expected:
25
+ To ensure that a backup file has not been tampered with or corrupted, call the `validate-chain` command and provide the path to your backup file with `--backup`. The command verifies that each event's hash is correctly calculated, that the predecessor hash of each event matches the hash of the previous event, and that the first event has a null predecessor hash as expected:
26
26
 
27
27
  ```shell
28
- eventsourcingdb-merkle validate-chain backup.json
28
+ eventsourcingdb-merkle validate-chain --backup=backup.json
29
29
  ```
30
30
 
31
31
  If the chain is valid, you will see a confirmation message with the total number of events:
@@ -57,29 +57,40 @@ Such an event cannot be verified, which is not the same as being tampered with,
57
57
 
58
58
  ### Calculating the Merkle Root
59
59
 
60
- To calculate the Merkle root for all events in a backup file, call the `merkle-root` command and provide the path to your backup file. The Merkle root is a single hash that represents the entire event chain and can be used to verify the integrity of the backup:
60
+ To calculate the Merkle root for all events in a backup file, call the `merkle-root` command and provide the path to your backup file with `--backup`. The Merkle root is a single hash that represents the entire event chain and can be used to verify the integrity of the backup:
61
61
 
62
62
  ```shell
63
- eventsourcingdb-merkle merkle-root backup.json
63
+ eventsourcingdb-merkle merkle-root --backup=backup.json
64
64
  ```
65
65
 
66
- The command outputs the Merkle root and the total number of events:
66
+ The command outputs the Merkle root, the total number of events, and the ID of the last event it covers:
67
67
 
68
68
  ```
69
69
  Merkle Root: d1f6713af7ba3896ff238bb51f3986b2bad53b09bda7f23370e0a8b6eb97eeab
70
70
  Total events: 4
71
+ Last event ID: 3
71
72
  ```
72
73
 
74
+ #### Calculating the Merkle Root of an Earlier State
75
+
76
+ A root that you published once has to be calculated again later, for instance to generate proofs against it. Since events are never changed, a newer backup holds the events of an older one, followed by those written since. Pass the last event ID that the root covered with `--last-event-id`, and the command calculates the root over the events up to it, which is the root of the older backup:
77
+
78
+ ```shell
79
+ eventsourcingdb-merkle merkle-root --backup=backup-2025.json --last-event-id=3
80
+ ```
81
+
82
+ So you do not have to keep the older backup to prove against its root. If the backup holds no event with that ID, the command refuses rather than calculating a root over other events.
83
+
73
84
  ### Validating Individual Events
74
85
 
75
86
  Sometimes you want to verify the integrity of a single event without processing the entire backup. The `validate-event-hash` command provides two ways to do this.
76
87
 
77
88
  #### Validating an Event from a Backup File
78
89
 
79
- To validate an event by its ID from a backup file, call the `validate-event-hash` command with the `--file` and `--event-id` flags:
90
+ To validate an event by its ID from a backup file, call the `validate-event-hash` command with the `--backup` and `--event-id` flags:
80
91
 
81
92
  ```shell
82
- eventsourcingdb-merkle validate-event-hash --file=backup.json --event-id=42
93
+ eventsourcingdb-merkle validate-event-hash --backup=backup.json --event-id=42
83
94
  ```
84
95
 
85
96
  If the hash is valid, you will see a confirmation message:
@@ -107,18 +118,19 @@ You can also validate an event provided as a JSON string. This is useful when yo
107
118
  eventsourcingdb-merkle validate-event-hash --event='{"specversion":"1.0","id":"0",...,"hash":"esdb:hash:v2:..."}'
108
119
  ```
109
120
 
110
- If you provide a CloudEvent without a stored hash, the command simply calculates and displays the hash of version 1, since nothing tells which version is meant:
121
+ If you provide a CloudEvent without a stored hash, the command simply calculates and displays the hash in every version, since nothing tells which version is meant:
111
122
 
112
123
  ```
113
- Calculated hash: abc123...
124
+ Calculated hash (version 1): abc123...
125
+ Calculated hash (version 2): esdb:hash:v2:def456...
114
126
  ```
115
127
 
116
128
  ### Generating Merkle Proofs
117
129
 
118
- A Merkle proof allows you to prove that a specific event is part of a backup without revealing the entire backup. To generate a proof for an event, call the `get-proof` command with the path to your backup file and the event ID:
130
+ A Merkle proof allows you to prove that a specific event is part of a backup without revealing the entire backup. To generate a proof for an event, call the `get-proof` command with the path to your backup file in `--backup` and the event ID in `--event-id`:
119
131
 
120
132
  ```shell
121
- eventsourcingdb-merkle get-proof backup.json 42
133
+ eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42
122
134
  ```
123
135
 
124
136
  The command outputs the proof in a human-readable format:
@@ -135,12 +147,18 @@ Sibling Hashes (3):
135
147
  3. jkl012... (left)
136
148
  ```
137
149
 
150
+ To prove an event against the root of an earlier state, pass its last event ID with `--last-event-id`, as for `merkle-root`. An event after it is not part of that tree, so the command refuses to prove it:
151
+
152
+ ```shell
153
+ eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42 --last-event-id=1234566
154
+ ```
155
+
138
156
  #### Outputting JSON
139
157
 
140
158
  To get the proof as JSON for programmatic use, add the `--json` flag:
141
159
 
142
160
  ```shell
143
- eventsourcingdb-merkle get-proof backup.json 42 --json
161
+ eventsourcingdb-merkle get-proof --backup=backup.json --event-id=42 --json
144
162
  ```
145
163
 
146
164
  The output can then be saved to a file and shared with others:
@@ -212,7 +230,12 @@ In both versions, the data is hashed as it appears in the backup or in the event
212
230
 
213
231
  The Merkle tree is constructed by treating event hashes, as they are stored, prefix included, as leaf nodes. Internal nodes are created by hashing the concatenation of their child hashes. If the number of nodes at any level is odd, the last node is duplicated to create a complete binary tree. This process continues until a single root hash remains.
214
232
 
215
- Backup files are expected to be in the NDJSON format that EventSourcingDB writes on `/api/v1/backup`, where each line contains a JSON object with a `type` and a `payload` field. A line of the type `event` holds a CloudEvents-compliant event and its hash. A line of the type `schema` holds the schema of an event type, which the tool skips, since neither the hashes nor the chain cover it. A backup that holds a line of the type `error` was cut short, and the tool refuses it, as it refuses a line of any other type or one that is not valid JSON, naming the line.
233
+ Backup files are expected to be in one of the two NDJSON formats that EventSourcingDB writes, and the tool reads every backup that a restore of EventSourcingDB takes. It checks the structure of a backup, the hashes, and the chain, but neither the values of the events nor the schemas, which a restore checks with the rules of the server:
234
+
235
+ - A backup of the v1 API, which EventSourcingDB writes on `/api/v1/backup`, holds a JSON object with a `type` and a `payload` field on each line. A line of the type `event` holds a CloudEvents-compliant event and its hash. A line of the type `schema` holds the schema of an event type, which the tool skips, since neither the hashes nor the chain cover it.
236
+ - A backup of the v2 API, which EventSourcingDB writes on `/api/v2/management/create-backup` from release 1.3 on, holds a JSON object with a `type` and a `data` field on each line. It begins with a line of the type `backup`, which names the version of its format, and ends with a line of the type `end`. An incremental backup follows on the last event of the backup before it. To verify one, append it to the full backup and the increments before it, in their order, as a restore takes them, for example with `cat full.ndjson increment-1.ndjson increment-2.ndjson`: across them, the chain is the ordinary chain of the events.
237
+
238
+ A backup that holds a line of the type `error` was cut short, and the tool refuses it, as it refuses a line of any type that its format does not know, a line that is not valid JSON, and a backup of the v2 API whose structure a restore would refuse, naming the line.
216
239
 
217
240
  ## Running Quality Assurance
218
241
 
@@ -1,27 +1,58 @@
1
- import { Args, Flags } from '@oclif/core';
1
+ import { Flags } from '@oclif/core';
2
2
  import BaseCommand from '../lib/BaseCommand.js';
3
3
  import { findEventById, readBackupFile } from '../lib/backupReader.js';
4
+ import { parseEventId } from '../lib/eventId.js';
5
+ import { takeEntriesUpTo } from '../lib/lastEventId.js';
4
6
  import { generateMerkleProof } from '../lib/merkleTree.js';
5
7
  export default class GetProof extends BaseCommand {
6
8
  static description = 'Generate a Merkle proof for a specific event';
7
9
  static examples = [
8
- '<%= config.bin %> <%= command.id %> backup.json 42',
9
- '<%= config.bin %> <%= command.id %> /path/to/backup.json 0 --json',
10
+ '<%= config.bin %> <%= command.id %> --backup=backup.json --event-id=42',
11
+ '<%= config.bin %> <%= command.id %> --backup=/path/to/backup.json --event-id=0 --json',
12
+ '<%= config.bin %> <%= command.id %> --backup=backup.json --event-id=42 --last-event-id=1234566',
10
13
  ];
11
- static args = {
12
- file: Args.string({ description: 'Path to the backup file', required: true }),
13
- 'event-id': Args.string({ description: 'Event ID to generate proof for', required: true }),
14
- };
15
14
  static flags = {
15
+ backup: Flags.string({ description: 'Path to the backup file' }),
16
+ 'event-id': Flags.string({ description: 'Event ID to generate proof for' }),
17
+ 'last-event-id': Flags.string({
18
+ description: 'ID of the last event the root covers, to prove against the root of an earlier state',
19
+ }),
16
20
  json: Flags.boolean({ description: 'Output proof as JSON' }),
17
21
  };
22
+ static args = {};
18
23
  async run() {
19
- const { args, flags } = await this.parse(GetProof);
24
+ const { flags } = await this.parse(GetProof);
25
+ if (!flags.backup) {
26
+ this.refuseCall('--backup must be provided');
27
+ return;
28
+ }
29
+ const eventId = flags['event-id'];
30
+ if (!eventId) {
31
+ this.refuseCall('--event-id is required when using --backup');
32
+ return;
33
+ }
34
+ const lastEventId = flags['last-event-id'];
35
+ if (lastEventId !== undefined && parseEventId(lastEventId) === undefined) {
36
+ this.refuseCall(`--last-event-id has to be an event ID, but is '${lastEventId}'`);
37
+ return;
38
+ }
20
39
  try {
21
- const entries = await readBackupFile(args.file);
22
- const entry = findEventById(entries, args['event-id']);
40
+ const allEntries = await readBackupFile(flags.backup);
41
+ const entries = lastEventId === undefined ? allEntries : takeEntriesUpTo(allEntries, lastEventId);
42
+ // An event after the last one is not part of the tree, so it has no
43
+ // proof against its root, although the backup holds it.
44
+ const eventNumber = parseEventId(eventId);
45
+ const lastEventNumber = lastEventId === undefined ? undefined : parseEventId(lastEventId);
46
+ if (eventNumber !== undefined &&
47
+ lastEventNumber !== undefined &&
48
+ eventNumber > lastEventNumber) {
49
+ this.log(`✗ Event ${eventId} lies beyond the last event ID ${lastEventId}`);
50
+ process.exitCode = 1;
51
+ return;
52
+ }
53
+ const entry = findEventById(entries, eventId);
23
54
  if (!entry) {
24
- this.log(`✗ Event with ID ${args['event-id']} not found`);
55
+ this.log(`✗ Event with ID ${eventId} not found`);
25
56
  process.exitCode = 1;
26
57
  return;
27
58
  }
@@ -1,24 +1,43 @@
1
- import { Args } from '@oclif/core';
1
+ import { Flags } from '@oclif/core';
2
2
  import BaseCommand from '../lib/BaseCommand.js';
3
3
  import { readBackupFile } from '../lib/backupReader.js';
4
+ import { parseEventId } from '../lib/eventId.js';
5
+ import { takeEntriesUpTo } from '../lib/lastEventId.js';
4
6
  import { buildMerkleRoot } from '../lib/merkleTree.js';
5
7
  export default class MerkleRoot extends BaseCommand {
6
8
  static description = 'Calculate the Merkle root for an EventSourcingDB backup file';
7
9
  static examples = [
8
- '<%= config.bin %> <%= command.id %> backup.json',
9
- '<%= config.bin %> <%= command.id %> /path/to/backup.json',
10
+ '<%= config.bin %> <%= command.id %> --backup=backup.json',
11
+ '<%= config.bin %> <%= command.id %> --backup=/path/to/backup.json --last-event-id=41',
10
12
  ];
11
- static args = {
12
- file: Args.string({ description: 'Path to the backup file', required: true }),
13
+ static flags = {
14
+ backup: Flags.string({ description: 'Path to the backup file' }),
15
+ 'last-event-id': Flags.string({
16
+ description: 'ID of the last event to cover, to calculate the root of an earlier state again',
17
+ }),
13
18
  };
19
+ static args = {};
14
20
  async run() {
15
- const { args } = await this.parse(MerkleRoot);
21
+ const { flags } = await this.parse(MerkleRoot);
22
+ if (!flags.backup) {
23
+ this.refuseCall('--backup must be provided');
24
+ return;
25
+ }
26
+ const lastEventId = flags['last-event-id'];
27
+ if (lastEventId !== undefined && parseEventId(lastEventId) === undefined) {
28
+ this.refuseCall(`--last-event-id has to be an event ID, but is '${lastEventId}'`);
29
+ return;
30
+ }
16
31
  try {
17
- const entries = await readBackupFile(args.file);
32
+ const allEntries = await readBackupFile(flags.backup);
33
+ const entries = lastEventId === undefined ? allEntries : takeEntriesUpTo(allEntries, lastEventId);
18
34
  const hashes = entries.map(entry => entry.payload.hash);
19
35
  const merkleRoot = buildMerkleRoot(hashes);
36
+ // The ID of the last event is what calculates this root again later,
37
+ // from a backup that has grown since.
20
38
  this.log(`Merkle Root: ${merkleRoot}`);
21
39
  this.log(`Total events: ${entries.length}`);
40
+ this.log(`Last event ID: ${entries.at(-1)?.payload.event.id}`);
22
41
  }
23
42
  catch (error) {
24
43
  this.log(`✗ Failed to calculate Merkle root: ${error instanceof Error ? error.message : String(error)}`);
@@ -1,20 +1,25 @@
1
- import { Args } from '@oclif/core';
1
+ import { Flags } from '@oclif/core';
2
2
  import BaseCommand from '../lib/BaseCommand.js';
3
3
  import { readBackupFile } from '../lib/backupReader.js';
4
4
  import { validateChain } from '../lib/chainValidator.js';
5
5
  export default class ValidateChain extends BaseCommand {
6
6
  static description = 'Validate the hash chain in an EventSourcingDB backup file';
7
7
  static examples = [
8
- '<%= config.bin %> <%= command.id %> backup.json',
9
- '<%= config.bin %> <%= command.id %> /path/to/backup.json',
8
+ '<%= config.bin %> <%= command.id %> --backup=backup.json',
9
+ '<%= config.bin %> <%= command.id %> --backup=/path/to/backup.json',
10
10
  ];
11
- static args = {
12
- file: Args.string({ description: 'Path to the backup file', required: true }),
11
+ static flags = {
12
+ backup: Flags.string({ description: 'Path to the backup file' }),
13
13
  };
14
+ static args = {};
14
15
  async run() {
15
- const { args } = await this.parse(ValidateChain);
16
+ const { flags } = await this.parse(ValidateChain);
17
+ if (!flags.backup) {
18
+ this.refuseCall('--backup must be provided');
19
+ return;
20
+ }
16
21
  try {
17
- const entries = await readBackupFile(args.file);
22
+ const entries = await readBackupFile(flags.backup);
18
23
  const result = validateChain(entries);
19
24
  if (!result.isValid) {
20
25
  this.log(`✗ Chain validation failed`);
@@ -2,41 +2,37 @@ import { Flags } from '@oclif/core';
2
2
  import BaseCommand from '../lib/BaseCommand.js';
3
3
  import { findEventById, readBackupFile } from '../lib/backupReader.js';
4
4
  import { readEventText } from '../lib/eventText.js';
5
- import { calculateEventHash, UnsupportedHashVersionError } from '../lib/hash.js';
5
+ import { calculateEventHash, calculateEventHashInEveryVersion, UnsupportedHashVersionError, } from '../lib/hash.js';
6
6
  export default class ValidateEventHash extends BaseCommand {
7
7
  static description = 'Validate the hash of a specific event (with or without backup file)';
8
8
  static examples = [
9
- '<%= config.bin %> <%= command.id %> --file=backup.json --event-id=42',
9
+ '<%= config.bin %> <%= command.id %> --backup=backup.json --event-id=42',
10
10
  // biome-ignore lint/security/noSecrets: This is an example command, not a secret
11
11
  '<%= config.bin %> <%= command.id %> --event=\'{"specversion":"1.0",...}\'',
12
12
  ];
13
13
  static flags = {
14
- file: Flags.string({ description: 'Path to the backup file', exclusive: ['event'] }),
14
+ backup: Flags.string({ description: 'Path to the backup file', exclusive: ['event'] }),
15
15
  'event-id': Flags.string({
16
- description: 'Event ID to validate (requires --file)',
16
+ description: 'Event ID to validate (requires --backup)',
17
17
  exclusive: ['event'],
18
18
  }),
19
19
  event: Flags.string({
20
20
  description: 'JSON string of the event to validate',
21
- exclusive: ['file', 'event-id'],
21
+ exclusive: ['backup', 'event-id'],
22
22
  }),
23
23
  };
24
24
  static args = {};
25
25
  async run() {
26
26
  const { flags } = await this.parse(ValidateEventHash);
27
- if (!flags.file && !flags.event) {
28
- this.log('✗ Either --file and --event-id, or --event must be provided');
29
- this.log('See more help with --help');
30
- process.exitCode = 2;
27
+ if (!flags.backup && !flags.event) {
28
+ this.refuseCall('Either --backup with --event-id, or --event must be provided');
31
29
  return;
32
30
  }
33
- if (flags.file && !flags['event-id']) {
34
- this.log('✗ --event-id is required when using --file');
35
- this.log('See more help with --help');
36
- process.exitCode = 2;
31
+ if (flags.backup && !flags['event-id']) {
32
+ this.refuseCall('--event-id is required when using --backup');
37
33
  return;
38
34
  }
39
- const filePath = flags.file;
35
+ const filePath = flags.backup;
40
36
  const eventId = flags['event-id'];
41
37
  try {
42
38
  let event;
@@ -45,15 +41,15 @@ export default class ValidateEventHash extends BaseCommand {
45
41
  if (flags.event) {
46
42
  const eventFromText = readEventText(flags.event);
47
43
  if (eventFromText === undefined) {
48
- this.log('✗ --event is neither an event nor a line of a backup or of read-events');
49
- this.log('See more help with --help');
50
- process.exitCode = 2;
44
+ this.refuseCall('--event is neither an event nor a line of a backup or of read-events');
51
45
  return;
52
46
  }
47
+ // Without a stored hash, nothing tells which version of the hash is
48
+ // meant, so it is calculated in every version.
53
49
  if (eventFromText.storedHash === undefined) {
54
- // No stored hash to compare against
55
- const calculatedHash = calculateEventHash(eventFromText.event, eventFromText.originalDataString);
56
- this.log(`Calculated hash: ${calculatedHash}`);
50
+ for (const { version, hash } of calculateEventHashInEveryVersion(eventFromText.event, eventFromText.originalDataString)) {
51
+ this.log(`Calculated hash (version ${version}): ${hash}`);
52
+ }
57
53
  return;
58
54
  }
59
55
  event = eventFromText.event;
@@ -1,16 +1,23 @@
1
1
  import { Command } from '@oclif/core';
2
2
  export default class BaseCommand extends Command {
3
+ // refuseCall reports a call whose arguments or flags do not fit together, as
4
+ // oclif does for those it refuses itself, with the exit code 2.
5
+ refuseCall(message) {
6
+ this.log(`✗ ${message}`);
7
+ this.log('See more help with --help');
8
+ process.exitCode = 2;
9
+ }
3
10
  catch(error) {
4
- const errorName = error.constructor.name;
5
- // Check if it's a RequiredArgsError or NonExistentFlagsError
6
- if (errorName === 'RequiredArgsError' || errorName === 'NonExistentFlagsError') {
11
+ // oclif ends with the exit code 2 where it refuses the arguments and flags
12
+ // of a call, such as an argument that is missing or unexpected, a flag that
13
+ // does not exist, or two flags that cannot be used together. Such a call
14
+ // gets a message of its own rather than a stack trace.
15
+ if (error.oclif?.exit === 2) {
7
16
  // Remove the "See more help with --help" line from oclif
8
17
  const lines = error.message.split('\n');
9
18
  const filteredLines = lines.filter(line => !line.includes('See more help'));
10
19
  const message = filteredLines.join('\n').trim();
11
- this.log(`✗ ${message}`);
12
- this.log('See more help with --help');
13
- process.exitCode = 2;
20
+ this.refuseCall(message);
14
21
  return Promise.resolve();
15
22
  }
16
23
  // For other errors, use default oclif error handling
@@ -1,7 +1,12 @@
1
1
  import { createReadStream } from 'node:fs';
2
2
  import { access } from 'node:fs/promises';
3
3
  import { createInterface } from 'node:readline';
4
- import { getCompactValue } from './rawJson.js';
4
+ import { parseEventId } from './eventId.js';
5
+ import { isObject } from './isObject.js';
6
+ import { getCompactValue, getRawValue } from './rawJson.js';
7
+ // The version of the format in which the v2 API writes a backup. The backups
8
+ // of the v1 API carry no version of their own.
9
+ const backupFormatVersion = 2;
5
10
  export const readBackupFile = async (filePath) => {
6
11
  // Check if file exists and is readable
7
12
  try {
@@ -16,63 +21,215 @@ export const readBackupFile = async (filePath) => {
16
21
  input: fileStream,
17
22
  crlfDelay: Number.POSITIVE_INFINITY,
18
23
  });
24
+ let formatReader;
19
25
  let lineNumber = 0;
20
- for await (const line of rl) {
26
+ for await (const text of rl) {
21
27
  lineNumber++;
22
- if (line.trim() === '') {
28
+ if (text.trim() === '') {
23
29
  continue;
24
30
  }
25
- const entry = readLine(line, lineNumber);
31
+ const line = parseLine(text, lineNumber);
32
+ // A server sends heartbeats while it reads a large store, and they carry
33
+ // nothing to verify.
34
+ if (line.type === 'heartbeat') {
35
+ continue;
36
+ }
37
+ // A backup that the v1 API wrote begins with a line of its own rather than
38
+ // with a backup line, and holds payloads rather than data, which is how
39
+ // the restore of EventSourcingDB tells the two formats apart as well.
40
+ formatReader ??=
41
+ line.type !== 'backup' && line.payload !== undefined ? createV1Reader() : createV2Reader();
42
+ const entry = formatReader.readLine(line);
26
43
  if (entry !== undefined) {
27
44
  entries.push(entry);
28
45
  }
29
46
  }
47
+ formatReader?.finish();
30
48
  return entries;
31
49
  };
32
- // readLine reads a line of a backup, and returns the event it holds, or nothing
33
- // for a line that holds a schema, which neither the hashes nor the chain cover.
34
- // Every other line ends the reading: an error line means that the backup was
35
- // cut short, and a line of a type the format does not know means that the file
36
- // is no backup the tool can read. Skipping either would let the events before
37
- // it pass as a complete, valid chain.
38
- const readLine = (line, lineNumber) => {
50
+ const parseLine = (text, lineNumber) => {
39
51
  let parsed;
40
52
  try {
41
- parsed = JSON.parse(line);
53
+ parsed = JSON.parse(text);
42
54
  }
43
55
  catch (error) {
44
56
  throw new Error(`line ${lineNumber} is not valid JSON: ${error instanceof Error ? error.message : String(error)}`);
45
57
  }
46
- const { type, payload } = (typeof parsed === 'object' && parsed !== null ? parsed : {});
58
+ const { type, payload, data } = isObject(parsed) ? parsed : {};
47
59
  if (typeof type !== 'string') {
48
60
  throw new Error(`line ${lineNumber} has no type`);
49
61
  }
50
- switch (type) {
51
- case 'schema':
52
- return undefined;
53
- case 'error':
54
- throw new Error(`the backup is incomplete: line ${lineNumber} reports an error: ${typeof payload === 'string' ? payload : JSON.stringify(payload)}`);
55
- case 'event':
56
- return readEventLine(line, payload, lineNumber);
57
- default:
58
- throw new Error(`line ${lineNumber} has an unknown type: ${type}`);
59
- }
62
+ return { text, number: lineNumber, type, payload, data };
63
+ };
64
+ // A backup of the v1 API holds a line for every schema, which neither the hashes
65
+ // nor the chain cover, and a line for every event. Every other line ends the
66
+ // reading: an error line means that the backup was cut short, and a line of a
67
+ // type the format does not know means that the file is no backup the tool can
68
+ // read. Skipping either would let the events before it pass as a complete,
69
+ // valid chain.
70
+ const createV1Reader = () => {
71
+ return {
72
+ readLine: line => {
73
+ switch (line.type) {
74
+ case 'schema':
75
+ return undefined;
76
+ case 'error':
77
+ throw new Error(`the backup is incomplete: line ${line.number} reports an error: ${typeof line.payload === 'string' ? line.payload : JSON.stringify(line.payload)}`);
78
+ case 'event':
79
+ return readV1EventLine(line);
80
+ default:
81
+ throw new Error(`line ${line.number} has an unknown type: ${line.type}`);
82
+ }
83
+ },
84
+ finish: () => {
85
+ // The format has no line that ends it, so a backup that was cut short
86
+ // without an error line cannot be told from a complete one.
87
+ },
88
+ };
60
89
  };
61
- const readEventLine = (line, payload, lineNumber) => {
62
- const { event, hash } = (typeof payload === 'object' && payload !== null ? payload : {});
63
- if (typeof event !== 'object' || event === null) {
64
- throw new Error(`line ${lineNumber} has an event line without an event`);
90
+ const readV1EventLine = (line) => {
91
+ const { event, hash } = isObject(line.payload) ? line.payload : {};
92
+ if (!isObject(event)) {
93
+ throw new Error(`line ${line.number} has an event line without an event`);
65
94
  }
66
95
  if (typeof hash !== 'string') {
67
- throw new Error(`line ${lineNumber} has an event line without a hash`);
96
+ throw new Error(`line ${line.number} has an event line without a hash`);
68
97
  }
69
- const entry = {
98
+ return {
70
99
  type: 'event',
71
- payload: { event: event, hash },
100
+ payload: {
101
+ event: event,
102
+ hash,
103
+ originalDataString: getCompactValue(line.text, ['payload', 'event', 'data']),
104
+ },
105
+ };
106
+ };
107
+ // A backup of the v2 API is a sequence of parts in one file: a full backup,
108
+ // followed by any number of increments, each beginning with a backup line and
109
+ // ending with an end line, as the restore of EventSourcingDB takes them. An
110
+ // increment follows on the event that the part before it ended at, so across
111
+ // the parts the chain is the ordinary chain of the events. The reader checks the
112
+ // structure as the restore does, since a backup that the restore refuses holds
113
+ // no chain that could be restored, but leaves the values of the events and the
114
+ // schemas to the restore, which checks them with the rules of the server.
115
+ const createV2Reader = () => {
116
+ let partIndex = -1;
117
+ let isPartOpen = false;
118
+ let hasPartEvents = false;
119
+ let endedAt;
120
+ const readBackupLine = (line) => {
121
+ if (isPartOpen) {
122
+ throw new Error(`line ${line.number} begins a part before the part before it ended`);
123
+ }
124
+ const { lastBackedUpEvent } = isObject(line.data) ? line.data : {};
125
+ partIndex++;
126
+ // The restore reads the version as the text of a number, so it takes one
127
+ // in quotes as well, and counts null as a version that is left out.
128
+ const rawFormatVersion = getRawValue(line.text, ['data', 'formatVersion']);
129
+ const formatVersion = rawFormatVersion === undefined || rawFormatVersion === 'null'
130
+ ? undefined
131
+ : rawFormatVersion.startsWith('"')
132
+ ? JSON.parse(rawFormatVersion)
133
+ : rawFormatVersion;
134
+ if (formatVersion !== String(backupFormatVersion)) {
135
+ throw new Error(`line ${line.number} has an unsupported backup format version: ${formatVersion ?? 'none'}`);
136
+ }
137
+ // The first part is a full backup, and every later one follows on the event
138
+ // the part before it ended at, or on its anchor, where it held no event.
139
+ if (partIndex === 0 && lastBackedUpEvent !== null && lastBackedUpEvent !== undefined) {
140
+ throw new Error(`line ${line.number} begins an increment, but a backup has to begin with a full backup`);
141
+ }
142
+ if (partIndex > 0 && !isSameEvent(lastBackedUpEvent, endedAt)) {
143
+ throw new Error(`line ${line.number} begins an increment that does not follow on the part before it`);
144
+ }
145
+ isPartOpen = true;
146
+ hasPartEvents = false;
147
+ return undefined;
148
+ };
149
+ const readEventLine = (line) => {
150
+ if (!isObject(line.data)) {
151
+ throw new Error(`line ${line.number} has an event line without an event`);
152
+ }
153
+ const { hash } = line.data;
154
+ if (typeof hash !== 'string') {
155
+ throw new Error(`line ${line.number} has an event line without a hash`);
156
+ }
157
+ const event = line.data;
158
+ hasPartEvents = true;
159
+ endedAt = { id: event.id, hash };
160
+ return {
161
+ type: 'event',
162
+ payload: {
163
+ event,
164
+ hash,
165
+ originalDataString: getCompactValue(line.text, ['data', 'data']),
166
+ },
167
+ };
168
+ };
169
+ return {
170
+ readLine: line => {
171
+ if (!isPartOpen && line.type !== 'backup') {
172
+ throw new Error(`line ${line.number} has to begin a part with a backup line, but has the type ${line.type}`);
173
+ }
174
+ switch (line.type) {
175
+ case 'backup':
176
+ return readBackupLine(line);
177
+ case 'eventSchema':
178
+ if (hasPartEvents) {
179
+ throw new Error(`line ${line.number} holds a schema after the events of its part`);
180
+ }
181
+ return undefined;
182
+ case 'event':
183
+ return readEventLine(line);
184
+ case 'lastBackedUpEvent':
185
+ // The event a part ended at follows from its events or its anchor
186
+ // already, so the line that names it adds nothing to check.
187
+ return undefined;
188
+ case 'error': {
189
+ const { description } = isObject(line.data) ? line.data : {};
190
+ throw new Error(`the backup is incomplete: line ${line.number} reports an error: ${typeof description === 'string' ? description : JSON.stringify(line.data)}`);
191
+ }
192
+ case 'end':
193
+ isPartOpen = false;
194
+ return undefined;
195
+ default:
196
+ throw new Error(`line ${line.number} has an unknown type: ${line.type}`);
197
+ }
198
+ },
199
+ finish: () => {
200
+ if (isPartOpen) {
201
+ throw new Error(`the backup is incomplete: part ${partIndex} lacks its end line`);
202
+ }
203
+ },
72
204
  };
73
- entry.payload.originalDataString = getCompactValue(line, ['payload', 'event', 'data']);
74
- return entry;
75
205
  };
206
+ // isSameEvent tells whether the anchor that an increment names, as its backup
207
+ // line holds it, is the event that the part before it ended at, by its ID as a
208
+ // number, since creating a backup takes the ID in any notation that
209
+ // EventSourcingDB reads, and by its hash. An anchor of null names no event, and
210
+ // follows on a part that ended at none, such as a full backup of an empty store.
211
+ // Any other anchor that does not name an event by its ID and hash follows on
212
+ // nothing.
213
+ const isSameEvent = (anchor, endedAt) => {
214
+ if (anchor === null || anchor === undefined) {
215
+ return endedAt === undefined;
216
+ }
217
+ if (endedAt === undefined || !isObject(anchor)) {
218
+ return false;
219
+ }
220
+ const { id, hash } = anchor;
221
+ if (typeof id !== 'string' || typeof hash !== 'string') {
222
+ return false;
223
+ }
224
+ const anchorId = parseEventId(id);
225
+ return anchorId !== undefined && anchorId === parseEventId(endedAt.id) && hash === endedAt.hash;
226
+ };
227
+ // findEventById compares IDs as numbers, as EventSourcingDB reads them, so that
228
+ // an ID in another notation of the same number finds the event.
76
229
  export const findEventById = (entries, eventId) => {
77
- return entries.find(entry => entry.payload.event.id === eventId);
230
+ const wanted = parseEventId(eventId);
231
+ if (wanted === undefined) {
232
+ return undefined;
233
+ }
234
+ return entries.find(entry => parseEventId(entry.payload.event.id) === wanted);
78
235
  };
@@ -0,0 +1,8 @@
1
+ // An event ID is an integer, which EventSourcingDB reads with an optional sign.
2
+ const eventIdPattern = /^[+-]?\d+$/u;
3
+ // parseEventId returns the number that an event ID names, so that IDs in other
4
+ // notations of the same number compare as equal, or undefined for text that
5
+ // names no number.
6
+ export const parseEventId = (id) => {
7
+ return eventIdPattern.test(id) ? BigInt(id) : undefined;
8
+ };
@@ -1,18 +1,18 @@
1
+ import { isObject } from './isObject.js';
1
2
  import { getCompactValue } from './rawJson.js';
2
- const isObject = (value) => {
3
- return typeof value === 'object' && value !== null && !Array.isArray(value);
4
- };
5
3
  const isEvent = (value) => {
6
4
  return isObject(value) && typeof value.specversion === 'string';
7
5
  };
8
6
  // readEventText reads an event from text in every form one can copy it from:
9
- // a line of a backup, which holds the event and its hash side by side, a line
10
- // of read-events, which holds the event with its hash, or the event alone,
11
- // with its hash or without. The data is taken from the text
12
- // rather than from what JSON.parse makes of it, since the hash covers it as
13
- // EventSourcingDB wrote it. Text in any other form gives undefined, a line
14
- // whose payload is neither of the two included, since a line carries no spec
15
- // version of its own and so is never taken for an event alone.
7
+ // a line of a backup of the v1 API, which holds the event and its hash side by
8
+ // side, a line of read-events of the v1 API, which holds the event with its
9
+ // hash, a line of a backup or of read-events of the v2 API, which holds the
10
+ // event with its hash as data, or the event alone, with its hash or without.
11
+ // The data is taken from the text rather than from what JSON.parse makes of
12
+ // it, since the hash covers it as EventSourcingDB wrote it. Text in any other
13
+ // form gives undefined, a line whose event or hash is missing included, since
14
+ // a line carries no spec version of its own and so is never taken for an event
15
+ // alone.
16
16
  export const readEventText = (text) => {
17
17
  const parsed = JSON.parse(text);
18
18
  if (isObject(parsed) && parsed.type === 'event' && isObject(parsed.payload)) {
@@ -32,6 +32,16 @@ export const readEventText = (text) => {
32
32
  };
33
33
  }
34
34
  }
35
+ if (isObject(parsed) && parsed.type === 'event' && isEvent(parsed.data)) {
36
+ const { data } = parsed;
37
+ if (typeof data.hash === 'string') {
38
+ return {
39
+ event: data,
40
+ storedHash: data.hash,
41
+ originalDataString: getCompactValue(text, ['data', 'data']),
42
+ };
43
+ }
44
+ }
35
45
  if (isEvent(parsed)) {
36
46
  return {
37
47
  event: parsed,
package/dist/lib/hash.js CHANGED
@@ -41,13 +41,14 @@ const calculateEventHashOfVersion2 = (event, dataString) => {
41
41
  };
42
42
  // calculateEventHash computes the hash of an event in the version that its
43
43
  // stored hash names, since an event keeps the version of the hash it was
44
- // written with, or in version 1 without a stored hash to go by. A stored hash
44
+ // written with. An event without a stored hash names no version, which is what
45
+ // calculateEventHashInEveryVersion is for. A stored hash
45
46
  // whose prefix names a version that the tool does not know cannot be verified,
46
47
  // which is not the same as being tampered with, so it is refused rather than
47
48
  // computed in another version that could only fail to match.
48
49
  export const calculateEventHash = (event, originalDataString, storedHash) => {
49
50
  const dataString = originalDataString ?? JSON.stringify(event.data);
50
- if (storedHash === undefined || !storedHash.startsWith(hashPrefix)) {
51
+ if (!storedHash.startsWith(hashPrefix)) {
51
52
  return calculateEventHashOfVersion1(event, dataString);
52
53
  }
53
54
  if (storedHash.startsWith(hashVersion2Prefix)) {
@@ -56,6 +57,16 @@ export const calculateEventHash = (event, originalDataString, storedHash) => {
56
57
  const [version = ''] = storedHash.slice(hashPrefix.length).split(':');
57
58
  throw new UnsupportedHashVersionError(version);
58
59
  };
60
+ // calculateEventHashInEveryVersion computes the hash of an event in every
61
+ // version the tool knows, for an event without a stored hash that would tell
62
+ // which version is meant.
63
+ export const calculateEventHashInEveryVersion = (event, originalDataString) => {
64
+ const dataString = originalDataString ?? JSON.stringify(event.data);
65
+ return [
66
+ { version: 1, hash: calculateEventHashOfVersion1(event, dataString) },
67
+ { version: 2, hash: calculateEventHashOfVersion2(event, dataString) },
68
+ ];
69
+ };
59
70
  export const hashPair = (left, right) => {
60
71
  return crypto.createHash('sha256').update(`${left}${right}`).digest('hex');
61
72
  };
@@ -0,0 +1,3 @@
1
+ export const isObject = (value) => {
2
+ return typeof value === 'object' && value !== null && !Array.isArray(value);
3
+ };
@@ -0,0 +1,16 @@
1
+ import { parseEventId } from './eventId.js';
2
+ // takeEntriesUpTo returns the events of a backup up to the one with the given
3
+ // ID, which it includes, so that the Merkle root of an earlier state can be
4
+ // calculated again from a backup that has grown since. An ID that the backup
5
+ // does not hold is refused rather than taken as the end of the backup, since
6
+ // the root would then cover other events than the ones it was meant to.
7
+ export const takeEntriesUpTo = (entries, lastEventId) => {
8
+ const last = parseEventId(lastEventId);
9
+ const index = last === undefined
10
+ ? -1
11
+ : entries.findIndex(entry => parseEventId(entry.payload.event.id) === last);
12
+ if (index === -1) {
13
+ throw new Error(`the backup holds no event with ID ${lastEventId}`);
14
+ }
15
+ return entries.slice(0, index + 1);
16
+ };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "eventsourcingdb-merkle",
3
- "version": "1.2.0",
3
+ "version": "2.0.0",
4
4
  "description": "Merkle tree utilities for EventSourcingDB backups.",
5
5
  "author": {
6
6
  "name": "the native web GmbH",