@agoric/swing-store 0.9.2-dev-fb3af6c.0 → 0.9.2-dev-d6f50e3.0

package/docs/data-export.md

@@ -188,13 +188,13 @@ Once the new SwingStore is fully populated with the previously-exported data, th
 
 Some of the data maintained by SwingStore is not strictly necessary for kernel execution, at least under normal circumstances. For example, once a vat worker performs a heap snapshot, we no longer need the transcript entries from before the snapshot was taken, since vat replay will start from the snapshot point. We split each vat's transcript into "spans", delimited by heap snapshot events, and the "current span" is the most recent one (still growing), whereas the "historical spans" are all closed and immutable. Likewise, we only really need the most recent heap snapshot for each vat: older snapshots might be interesting for experiments that replay old transcripts with different versions of the XS engine, but no normal kernel will ever need them.
 
-Most validators would prefer to prune this data, to reduce their storage needs. But we can imagine some [extreme upgrade scenarios](https://github.com/Agoric/agoric-sdk/issues/1691) that would require access to these historical transcript spans. Our compromise is to record *validation data* for these historical spans in the export data, but omit the spans themselves from the export artifacts. Validators can delete the old spans at will, and if we ever need them in the future, we can add code that will fetch copies from an archive service, validate them against the export data hashes, and re-insert the relevant entries into the SwingStore.
+Most blockchain validator nodes would prefer to prune this data, to reduce their storage needs. But we can imagine some [extreme upgrade scenarios](https://github.com/Agoric/agoric-sdk/issues/1691) that would require access to these historical transcript spans. Our compromise is to record *validation data* for these historical spans in the export data, but omit the spans themselves from the export artifacts. Validators can delete the old spans at will, and if we ever need them in the future, we can add code that will fetch copies from an archive service, validate them against the export data hashes, and re-insert the relevant entries into the SwingStore.
 
 Likewise, each time a heap snapshot is recorded, we cease to need any previous snapshot. And again, as a hedge against even more drastic recovery scenarios, we strike a compromise between minimizing retained data and the ability to validate old snapshots, by retaining only their hashes.
 
-As a result, for each active vat, the first-stage Export Data contains a record for every old transcript span, plus one for the current span. It also contains a record for every old heap snapshot, plus one for the most recent heap snapshot, plus a `.current` record that points to the most recent snapshot. However the exported artifacts may or may not include blobs for the old transcript spans, or for the old heap snapshots.
+As a result, for each active vat, the first-stage Export Data contains a record for every old heap snapshot, plus one for the most recent heap snapshot, plus a `.current` record that points to the most recent snapshot. It also contains a record for every old transcript span, plus one for the current span. However the exported artifacts may or may not include blobs for the old heap snapshots, or for the old transcript spans.
 
-The `openSwingStore()` function has an option named `keepTranscripts` (which defaults to `true`), which causes the transcriptStore to retain the old transcript items. A second option named `keepSnapshots` (which defaults to `false`) causes the snapStore to retain the old heap snapshots. Opening the swingStore with a `false` option does not necessarily delete the old items immediately, but they'll probably get deleted the next time the kernel triggers a heap snapshot or transcript-span rollover. Validators who care about minimizing their disk usage will want to set both to `false`. In the future, we will arrange the SwingStore SQLite tables to provide easy `sqlite3` CLI commands that will delete the old data, so validators can also periodically use the CLI command to prune it.
+The `openSwingStore()` function has an option named `keepSnapshots` (which defaults to `false`), which causes the snapStore to retain the old heap snapshots. A second option named `keepTranscripts` (which defaults to `true`) causes the transcriptStore to retain the old transcript items. Opening the swingStore with a `false` option does not necessarily delete the old items immediately, but they may get deleted the next time the kernel triggers a heap snapshot or transcript-span rollover. Hosts who care about minimizing their disk usage will want to set both to `false`. In the future, we will arrange the SwingStore SQLite tables to provide easy `sqlite3` CLI commands that will delete the old data, for use in periodic pruning.
 
 When exporting, the `makeSwingStoreExporter()` function takes an `artifactMode` option (in an options bag). This serves to both limit, and provide some minimal guarantees about, the set of artifacts that will be provided in the export. The defined values of `artifactMode` each build upon the previous one:
 
@@ -218,9 +218,9 @@ While `importSwingStore()`'s options bag accepts the same options as `openSwingS
 So, to avoid pruning current-incarnation historical transcript spans when exporting from one swingstore to another, you must set (or avoid overriding) the following options along the way:
 
 * the original swingstore must not be opened with `{ keepTranscripts: false }`, otherwise the old spans will be pruned immediately
-* the export must use `makeSwingStoreExporter(dirpath, { artifactMode: 'replay'})`, otherwise the export will omit the old spans
-* the import must use `importSwingStore(exporter, dirPath, { artifactMode: 'replay'})`, otherwise the import will ignore the old spans
-  * subsequent `openSwingStore` calls must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans as new ones are created (during `rolloverSpan`).
+* the export must use `makeSwingStoreExporter(dirpath, { artifactMode: 'replay' })`, otherwise the export will omit the old spans
+* the import must use `importSwingStore(exporter, dirPath, { artifactMode: 'replay' })`, otherwise the import will ignore the old spans
+  * subsequent `openSwingStore` calls must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans they are replaced during `rolloverSpan`.
 
 ## Implementation Details
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agoric/swing-store",
-  "version": "0.9.2-dev-fb3af6c.0+fb3af6c",
+  "version": "0.9.2-dev-d6f50e3.0+d6f50e3",
   "description": "Persistent storage for SwingSet",
   "type": "module",
   "main": "./src/index.js",
@@ -21,7 +21,7 @@
     "lint:eslint": "eslint ."
   },
   "dependencies": {
-    "@agoric/internal": "0.3.3-dev-fb3af6c.0+fb3af6c",
+    "@agoric/internal": "0.3.3-dev-d6f50e3.0+d6f50e3",
     "@endo/base64": "^1.0.7",
     "@endo/bundle-source": "^3.4.0",
     "@endo/check-bundle": "^1.0.9",
@@ -51,5 +51,5 @@
   "typeCoverage": {
     "atLeast": 76.31
   },
-  "gitHead": "fb3af6c419a81e243809ebee78c5de0763b19412"
+  "gitHead": "d6f50e32f6f19bb14cfe4530812609dff00d21b8"
 }

package/src/transcriptStore.js

@@ -80,15 +80,21 @@ export function makeTranscriptStore(
     )
   `);
 
-  // Transcripts are broken up into "spans", delimited by heap snapshots.  If we
-  // take heap snapshots after deliveries 100 and 200, and have not yet
-  // performed delivery 201, we'll have two non-current (i.e., isCurrent=null)
-  // spans (one with startPos=0, endPos=100, the second with startPos=100,
-  // endPos=200), and a single empty isCurrent==1 span with startPos=200 and
-  // endPos=200.  After we perform delivery 201, the single isCurrent=1 span
-  // will will still have startPos=200 but will now have endPos=201.  For every
-  // vatID, there will be exactly one isCurrent=1 span, and zero or more
-  // non-current (historical) spans.
+  // Transcripts are broken up into "spans", delimited by heap snapshots.
+  // The items of each transcript consist of deliveries and pseudo-deliveries
+  // such as initialize-worker and load-snapshot.
+  // For every vatID, there will be exactly one current span (with isCurrent=1),
+  // and zero or more non-current (historical) spans (with isCurrent=null).
+  // If we take a heap snapshot after the first hundred items and again after
+  // the second hundred (i.e., after zero-indexed items 99 and 199),
+  // and have not yet extended the transcript after the second snapshot, we'll
+  // have two historical spans (one with startPos=0 and endPos=100, the second
+  // with startPos=100 and endPos=200) and a single empty current span with
+  // startPos=200 and endPos=200.  But this situation is transient, and will
+  // generally be followed by a load-snapshot pseudo-delivery before the next
+  // commit (at which point the single current span will still have startPos=200
+  // but will have endPos=201).  After we perform the next delivery, the single
+  // current span will still have startPos=200 but will now have endPos=202.
   //
   // The transcriptItems associated with historical spans may or may not exist,
   // depending on pruning.  However, the items associated with the current span
@@ -246,7 +252,7 @@ export function makeTranscriptStore(
 
   const sqlDeleteOldItems = db.prepare(`
     DELETE FROM transcriptItems
-    WHERE vatID = ? AND position < ?
+    WHERE vatID = ? AND position >= ? AND position < ?
   `);
 
   function doSpanRollover(vatID, isNewIncarnation) {
@@ -276,12 +282,13 @@ export function makeTranscriptStore(
     noteExport(spanMetadataKey(newRec), JSON.stringify(newRec));
 
     if (!keepTranscripts) {
-      // TODO: for #9174 (delete historical transcript spans), we need
-      // this DB statement to only delete the items of the old span
-      // (startPos..endPos), not all previous items, otherwise the
-      // first rollover after switching to keepTranscripts=false will
-      // do a huge DB commit and probably explode
-      sqlDeleteOldItems.run(vatID, endPos);
+      // Delete items of the previously-current span.
+      // There may still be items associated with even older spans, but we leave
+      // those, to avoid excessive DB churn (for details, see #9387 and #9174).
+      // Recovery of space claimed by such ancient items is expected to use an
+      // external mechanism such as restoration from an operational snapshot
+      // that doesn't include them.
+      sqlDeleteOldItems.run(vatID, startPos, endPos);
     }
     return incarnationToUse;
   }

package/test/state.test.js

@@ -155,82 +155,106 @@ test('persistent kvStore maxKeySize write', async t => {
   await hostStorage.close();
 });
 
-async function testTranscriptStore(t, dbDir) {
-  const exportLog = makeExportLog();
-  const { kernelStorage, hostStorage } = initSwingStore(dbDir, {
-    exportCallback: exportLog.callback,
-    keepTranscripts: true, // XXX need to vary
-  });
-  const { transcriptStore } = kernelStorage;
-  const { commit, close } = hostStorage;
+const testTranscriptStore = test.macro({
+  title(prefix = '', { ephemeral, keepTranscripts }) {
+    const type = ephemeral ? 'in-memory' : 'persistent';
+    const detail = keepTranscripts ? 'with retention' : 'without retention';
+    return `${prefix.replace(/.$/, '$& ')}${type} transcriptStore ${detail}`;
+  },
+  async exec(t, { ephemeral, keepTranscripts }) {
+    let dbDir = null;
+    if (!ephemeral) {
+      const [tmpPath, cleanup] = await tmpDir('testdb');
+      t.teardown(cleanup);
+      t.is(isSwingStore(tmpPath), false);
+      dbDir = tmpPath;
+    }
 
-  transcriptStore.initTranscript('st1');
-  transcriptStore.initTranscript('st2');
-  transcriptStore.addItem('st1', 'first');
-  transcriptStore.addItem('st1', 'second');
-  transcriptStore.rolloverSpan('st1');
-  transcriptStore.addItem('st1', 'third');
-  transcriptStore.addItem('st2', 'oneth');
-  transcriptStore.addItem('st1', 'fourth');
-  transcriptStore.addItem('st2', 'twoth');
-  transcriptStore.addItem('st2', 'threeth');
-  transcriptStore.addItem('st2', 'fourst');
-  const reader1 = transcriptStore.readSpan('st1', 0);
-  t.deepEqual(Array.from(reader1), ['first', 'second']);
-  const reader2 = transcriptStore.readSpan('st2', 0);
-  t.deepEqual(Array.from(reader2), ['oneth', 'twoth', 'threeth', 'fourst']);
-
-  t.throws(() => transcriptStore.readSpan('st2', 3), {
-    message: 'no transcript span for "st2" at 3',
-  });
+    const exportLog = makeExportLog();
+    const { kernelStorage, hostStorage } = initSwingStore(dbDir, {
+      exportCallback: exportLog.callback,
+      keepTranscripts,
+    });
+    const { transcriptStore } = kernelStorage;
+    const { commit, close } = hostStorage;
+
+    transcriptStore.initTranscript('st1');
+    transcriptStore.initTranscript('st2');
+    transcriptStore.addItem('st1', 'zeroth');
+    transcriptStore.rolloverSpan('st1');
+    transcriptStore.addItem('st1', 'first');
+    transcriptStore.addItem('st1', 'second');
+    transcriptStore.rolloverSpan('st1');
+    transcriptStore.addItem('st1', 'third');
+    transcriptStore.addItem('st2', 'oneth');
+    transcriptStore.addItem('st1', 'fourth');
+    transcriptStore.addItem('st2', 'twoth');
+    transcriptStore.addItem('st2', 'threeth');
+    transcriptStore.addItem('st2', 'fourst');
+    const reader1a = transcriptStore.readSpan('st1', 0);
+    const reader1b = transcriptStore.readSpan('st1', 1);
+    if (keepTranscripts) {
+      t.deepEqual(Array.from(reader1a), ['zeroth']);
+      t.deepEqual(Array.from(reader1b), ['first', 'second']);
+    } else {
+      const fna = async () => Array.from(reader1a);
+      const fnb = async () => Array.from(reader1b);
+      await t.throwsAsync(fna, undefined, 'pruned spans must not be readable');
+      await t.throwsAsync(fnb, undefined, 'pruned spans must not be readable');
+    }
+    const reader2 = transcriptStore.readSpan('st2', 0);
+    t.deepEqual(Array.from(reader2), ['oneth', 'twoth', 'threeth', 'fourst']);
 
-  const reader1alt = transcriptStore.readSpan('st1');
-  t.deepEqual(Array.from(reader1alt), ['third', 'fourth']);
-  const reader1alt2 = transcriptStore.readSpan('st1', 2);
-  t.deepEqual(Array.from(reader1alt2), ['third', 'fourth']);
+    t.throws(() => transcriptStore.readSpan('st2', 3), {
+      message: 'no transcript span for "st2" at 3',
+    });
 
-  transcriptStore.initTranscript('empty');
-  const readerEmpty = transcriptStore.readSpan('empty');
-  t.deepEqual(Array.from(readerEmpty), []);
+    const reader1alt = transcriptStore.readSpan('st1');
+    t.deepEqual(Array.from(reader1alt), ['third', 'fourth']);
+    const reader1alt2 = transcriptStore.readSpan('st1', 3);
+    t.deepEqual(Array.from(reader1alt2), ['third', 'fourth']);
 
-  t.throws(() => transcriptStore.readSpan('nonexistent'), {
-    message: 'no current transcript for "nonexistent"',
-  });
+    transcriptStore.initTranscript('empty');
+    const readerEmpty = transcriptStore.readSpan('empty');
+    t.deepEqual(Array.from(readerEmpty), []);
 
-  await commit();
-  t.deepEqual(exportLog.getLog(), [
-    [
-      [
-        'transcript.empty.current',
-        '{"vatID":"empty","startPos":0,"endPos":0,"hash":"43e6be43a3a34d60c0ebeb8498b5849b094fc20fc68483a7aeb3624fa10f79f6","isCurrent":1,"incarnation":0}',
-      ],
-      [
-        'transcript.st1.0',
-        '{"vatID":"st1","startPos":0,"endPos":2,"hash":"d385c43882cfb5611d255e362a9a98626ba4e55dfc308fc346c144c696ae734e","isCurrent":0,"incarnation":0}',
-      ],
-      [
-        'transcript.st1.current',
-        '{"vatID":"st1","startPos":2,"endPos":4,"hash":"789342fab468506c624c713c46953992f53a7eaae390d634790d791636b96cab","isCurrent":1,"incarnation":0}',
-      ],
+    t.throws(() => transcriptStore.readSpan('nonexistent'), {
+      message: 'no current transcript for "nonexistent"',
+    });
+
+    await commit();
+    t.deepEqual(exportLog.getLog(), [
       [
-        'transcript.st2.current',
-        '{"vatID":"st2","startPos":0,"endPos":4,"hash":"45de7ae9d2be34148f9cf3000052e5d1374932d663442fe9f39a342d221cebf1","isCurrent":1,"incarnation":0}',
+        [
+          'transcript.empty.current',
+          '{"vatID":"empty","startPos":0,"endPos":0,"hash":"43e6be43a3a34d60c0ebeb8498b5849b094fc20fc68483a7aeb3624fa10f79f6","isCurrent":1,"incarnation":0}',
+        ],
+        [
+          'transcript.st1.0',
+          '{"vatID":"st1","startPos":0,"endPos":1,"hash":"92d0cf6ecd39b60b4e32dc65d4c6f343495928cb041f25b19e2825b17f4daa9a","isCurrent":0,"incarnation":0}',
+        ],
+        [
+          'transcript.st1.1',
+          '{"vatID":"st1","startPos":1,"endPos":3,"hash":"d385c43882cfb5611d255e362a9a98626ba4e55dfc308fc346c144c696ae734e","isCurrent":0,"incarnation":0}',
+        ],
+        [
+          'transcript.st1.current',
+          '{"vatID":"st1","startPos":3,"endPos":5,"hash":"789342fab468506c624c713c46953992f53a7eaae390d634790d791636b96cab","isCurrent":1,"incarnation":0}',
+        ],
+        [
+          'transcript.st2.current',
+          '{"vatID":"st2","startPos":0,"endPos":4,"hash":"45de7ae9d2be34148f9cf3000052e5d1374932d663442fe9f39a342d221cebf1","isCurrent":1,"incarnation":0}',
+        ],
       ],
-    ],
-  ]);
-  await close();
-}
-
-test('in-memory transcriptStore read/write', async t => {
-  await testTranscriptStore(t, null);
+    ]);
+    await close();
+  },
 });
 
-test('persistent transcriptStore read/write', async t => {
-  const [dbDir, cleanup] = await tmpDir('testdb');
-  t.teardown(cleanup);
-  t.is(isSwingStore(dbDir), false);
-  await testTranscriptStore(t, dbDir);
-});
+test(testTranscriptStore, { ephemeral: true, keepTranscripts: true });
+test(testTranscriptStore, { ephemeral: true, keepTranscripts: false });
+test(testTranscriptStore, { ephemeral: false, keepTranscripts: true });
+test(testTranscriptStore, { ephemeral: false, keepTranscripts: false });
 
 test('transcriptStore abort', async t => {
   const [dbDir, cleanup] = await tmpDir('testdb');