@agoric/swing-store 0.9.2-u11wf.0 → 0.9.2-u13.0

package/CHANGELOG.md

@@ -3,6 +3,30 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+### [0.9.2-u13.0](https://github.com/Agoric/agoric-sdk/compare/@agoric/swing-store@0.9.2-u12.0...@agoric/swing-store@0.9.2-u13.0) (2023-12-07)
+
+
+### Features
+
+* add exporter.getHostKV() API ([16435d2](https://github.com/Agoric/agoric-sdk/commit/16435d20e9ede86916a54c7bae54ecfc59e4c950)), closes [#8523](https://github.com/Agoric/agoric-sdk/issues/8523)
+
+
+
+### [0.9.2-u12.0](https://github.com/Agoric/agoric-sdk/compare/@agoric/swing-store@0.9.2-u11wf.0...@agoric/swing-store@0.9.2-u12.0) (2023-11-10)
+
+
+### Features
+
+* **swing-store:** faster import of swing-store ([35aef87](https://github.com/Agoric/agoric-sdk/commit/35aef87ec0f10b7f0cdce462ac0509296e8bd752))
+* **swing-store:** prevent SwingSet usage of imported swing-store ([03f642d](https://github.com/Agoric/agoric-sdk/commit/03f642d39f90ef9465a439723c3a69beef73bd61))
+
+
+### Bug Fixes
+
+* **swing-store:** ensure crank savepoint is wrapped in transaction ([8d738c6](https://github.com/Agoric/agoric-sdk/commit/8d738c65ed37b9159e94fbcf291ed7fe8478ee5a))
+
+
+
 ### [0.9.2-u11wf.0](https://github.com/Agoric/agoric-sdk/compare/@agoric/swing-store@0.9.2-u11.0...@agoric/swing-store@0.9.2-u11wf.0) (2023-09-23)
 
 **Note:** Version bump only for package @agoric/swing-store

package/docs/data-export.md

@@ -125,21 +125,9 @@ Then, on the few occasions when the application needs to build a full state-sync
 
 ## Import
 
-On other end of the export process is an importer. This is a new host application, which wants to start from the contents of the export, rather than initializing a brand new (empty) kernel state.
+On the other end of the export process is an importer. This is used to restore kernel state, so that a new host application can simply continue mostly as if it had been previously executing. The expectation is that the import and the execution are 2 independent events, and the execution doesn't need to be aware it was imported.
 
-When starting a brand new instance, host applications would normally call `openSwingStore(dirPath)` to create a new (empty) SwingStore, then call SwingSet's `initializeSwingset(config, .., kernelStorage)` to let the kernel initialize the DB with a config-dependent starting state:
-
-```js
-// this is done only the first time an instance is created:
-
-import { openSwingStore } from '@agoric/swing-store';
-import { initializeSwingset } from '@agoric/swingset-vat';
-const dirPath = './swing-store';
-const { hostStorage, kernelStorage } = openSwingStore(dirPath);
-await initializeSwingset(config, argv, kernelStorage);
-```
-
-Once the initial state is created, each time the application is launched, it will build a controller around the existing state:
+For reference, after the initial state is created, each time the application is launched, it builds a controller around the existing state:
 
 ```js
 import { openSwingStore } from '@agoric/swing-store';
@@ -150,7 +138,7 @@ const controller = await makeSwingsetController(kernelStorage);
 // ... now do things like controller.run(), etc
 ```
 
-When cloning an existing kernel, the initialization step is replaced with `importSwingStore`. The host application should feed the importer with the export data and artifacts, by passing an object that has the same API as the SwingStore's exporter:
+When cloning an existing kernel, the host application first imports and commits the restored state using `importSwingStore`. The host application should feed the importer with the export data and artifacts, by passing an object that has the same API as the SwingStore's exporter:
 
 ```js
 import { importSwingStore } from '@agoric/swing-store';
@@ -161,11 +149,13 @@ const exporter = {
   getArtifact(name) { // return blob of artifact data },
 };
 const { hostStorage } = importSwingStore(exporter, dirPath);
-hostStorage.commit();
-// now the swingstore is fully populated
+// Update any hostStorage as needed
+await hostStorage.commit();
+await hostStorage.close();
+// now the populated swingstore can be re-opened using `openSwingStore``
 ```
 
-Once the new SwingStore is fully populated with the previously-exported data, the host application can use `makeSwingsetController()` to build a kernel that will start from the exported state.
+Once the new SwingStore is fully populated with the previously-exported data, the host application can update any host specific state before committing and closing the SwingStore. `importSwingStore` returns only the host facet of the SwingStore instance, as it is not suitable for immediate execution.
 
 ## Optional / Historical Data
 
@@ -196,14 +186,14 @@ Also note that when a vat is terminated, we delete all information about it, inc
 
 When importing, the `importSwingStore()` function's options bag takes a property named `artifactMode`, with the same meanings as for export. Importing with the `operational` mode will ignore any artifacts other than those needed for current operations, and will fail unless all such artifacts were available. Importing with `replay` will ignore spans from old incarnations, but will fail unless all spans from current incarnations are present. Importing with `archival` will fail unless all spans from all incarnations are present. There is no `debug` option during import.
 
-`importSwingStore()` returns a swingstore, which means its options bag also contains the same options as `openSwingStore()`, including the `keepTranscripts` option. This defaults to `true`, but if it were overridden to `false`, then the new swingstore will delete transcript spans as soon as they are no longer needed for operational purposes (e.g. when `transcriptStore.rolloverSpan()` is called).
+While `importSwingStore()`'s options bag accepts the same options as `openSwingStore()`, since it returns only the host facet of a SwingStore, some of these options might not be meaningful, such as `keepTranscripts`.
 
 So, to avoid pruning current-incarnation historical transcript spans when exporting from one swingstore to another, you must set (or avoid overriding) the following options along the way:
 
 * the original swingstore must not be opened with `{ keepTranscripts: false }`, otherwise the old spans will be pruned immediately
 * the export must use `makeSwingStoreExporter(dirpath, { artifactMode: 'replay'})`, otherwise the export will omit the old spans
 * the import must use `importSwingStore(exporter, dirPath, { artifactMode: 'replay'})`, otherwise the import will ignore the old spans
-  * the `importSwingStore` call (and all subsequent `openSwingStore` calls) must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans as new ones are created (during `rolloverSpan`).
+  * subsequent `openSwingStore` calls must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans as new ones are created (during `rolloverSpan`).
 
 ## Implementation Details
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agoric/swing-store",
-  "version": "0.9.2-u11wf.0",
+  "version": "0.9.2-u13.0",
   "description": "Persistent storage for SwingSet",
   "type": "module",
   "main": "./src/index.js",
@@ -22,7 +22,7 @@
   },
   "dependencies": {
     "@agoric/assert": "^0.6.1-u11wf.0",
-    "@agoric/internal": "^0.3.3-u11wf.0",
+    "@agoric/internal": "^0.4.0-u13.0",
     "@endo/base64": "0.2.31",
     "@endo/bundle-source": "2.5.2-upstream-rollup",
     "@endo/check-bundle": "0.2.18",
@@ -45,5 +45,5 @@
     ],
     "timeout": "2m"
   },
-  "gitHead": "faf9ba6ab8b2b69bf25f435f262f0b5bd2f2bc97"
+  "gitHead": "5a6cdeb0c18ae9700d706445acf402f8d1e873c3"
 }

package/src/exporter.js

@@ -36,6 +36,11 @@ import { validateArtifactMode } from './internal.js';
  * the concurrent activity of other swingStore instances, the data representing
  * the commit point will stay consistent and available.
  *
+ * @property {(key: string) => string | undefined} getHostKV
+ *
+ * Retrieve a value from the "host" portion of the kvStore, just like
+ * hostStorage.hostKVStore.get() would do.
+ *
  * @property {() => AnyIterableIterator<KVPair>} getExportData
  *
  * Get a full copy of the first-stage export data (key-value pairs) from the
@@ -112,6 +117,33 @@ export function makeSwingStoreExporter(dirPath, options = {}) {
     assertComplete(internal, artifactMode);
   }
 
+  const sqlKVGet = db.prepare(`
+    SELECT value
+    FROM kvStore
+    WHERE key = ?
+  `);
+  sqlKVGet.pluck(true);
+
+  /**
+   * Obtain the value stored for a given host key. This is for the
+   * benefit of clients who need to briefly query the DB to ensure
+   * they are exporting the right thing, and need to avoid modifying
+   * anything (or creating a read-write DB lock) in the process.
+   *
+   * @param {string} key  The key whose value is sought.
+   *
+   * @returns {string | undefined} the (string) value for the given key, or
+   *    undefined if there is no such value.
+   *
+   * @throws if key is not a string, or the key is not in the host
+   * section
+   */
+  function getHostKV(key) {
+    typeof key === 'string' || Fail`key must be a string`;
+    getKeyType(key) === 'host' || Fail`getHostKV requires host keys`;
+    return sqlKVGet.get(key);
+  }
+
   const sqlGetAllKVData = db.prepare(`
     SELECT key, value
     FROM kvStore
@@ -173,6 +205,7 @@ export function makeSwingStoreExporter(dirPath, options = {}) {
   }
 
   return harden({
+    getHostKV,
     getExportData,
     getArtifactNames,
     getArtifact,

package/src/importer.js

@@ -11,14 +11,16 @@ import { assertComplete } from './assertComplete.js';
  */
 
 /**
- * Function used to create a new swingStore from an object implementing the
+ * Function used to populate a swingStore from an object implementing the
  * exporter API. The exporter API may be provided by a swingStore instance, or
- * implemented by a host to restore data that was previously exported.
+ * implemented by a host to restore data that was previously exported. The
+ * returned swingStore is not suitable for execution, and thus only contains
+ * the host facet for committing the populated swingStore.
  *
  * @param {import('./exporter').SwingStoreExporter} exporter
  * @param {string | null} [dirPath]
  * @param {ImportSwingStoreOptions} [options]
- * @returns {Promise<import('./swingStore').SwingStore>}
+ * @returns {Promise<Pick<import('./swingStore').SwingStore, 'hostStorage' | 'debug'>>}
  */
 export async function importSwingStore(exporter, dirPath = null, options = {}) {
   if (dirPath && typeof dirPath !== 'string') {
@@ -27,8 +29,14 @@ export async function importSwingStore(exporter, dirPath = null, options = {}) {
   const { artifactMode = 'operational', ...makeSwingStoreOptions } = options;
   validateArtifactMode(artifactMode);
 
-  const store = makeSwingStore(dirPath, true, makeSwingStoreOptions);
-  const { kernelStorage, internal } = store;
+  const { hostStorage, kernelStorage, internal, debug } = makeSwingStore(
+    dirPath,
+    true,
+    {
+      unsafeFastMode: true,
+      ...makeSwingStoreOptions,
+    },
+  );
 
   // For every exportData entry, we add a DB record. 'kv' entries are
   // the "kvStore shadow table", and are not associated with any
@@ -121,5 +129,5 @@ export async function importSwingStore(exporter, dirPath = null, options = {}) {
   assertComplete(internal, checkMode);
 
   await exporter.close();
-  return store;
+  return { hostStorage, debug };
 }

package/src/swingStore.js

@@ -203,9 +203,29 @@ export function makeSwingStore(dirPath, forceReset, options = {}) {
   // mode that defers merge work for a later attempt rather than block any
   // potential readers or writers. See https://sqlite.org/wal.html for details.
 
+  // However we also allow opening the DB with journaling off, which is unsafe
+  // and doesn't support rollback, but avoids any overhead for large
+  // transactions like for during an import.
+
+  function setUnsafeFastMode(enabled) {
+    const journalMode = enabled ? 'off' : 'wal';
+    const synchronousMode = enabled ? 'normal' : 'full';
+    !db.inTransaction || Fail`must not be in a transaction`;
+
+    db.unsafeMode(!!enabled);
+    // The WAL mode is persistent so it's not possible to switch to a different
+    // mode for an existing DB.
+    const actualMode = db.pragma(`journal_mode=${journalMode}`, {
+      simple: true,
+    });
+    actualMode === journalMode ||
+      filePath === ':memory:' ||
+      Fail`Couldn't set swing-store DB to ${journalMode} mode (is ${actualMode})`;
+    db.pragma(`synchronous=${synchronousMode}`);
+  }
+
   // PRAGMAs have to happen outside a transaction
-  db.exec(`PRAGMA journal_mode=WAL`);
-  db.exec(`PRAGMA synchronous=FULL`);
+  setUnsafeFastMode(options.unsafeFastMode);
 
   // We use IMMEDIATE because the kernel is supposed to be the sole writer of
   // the DB, and if some other process is holding a write lock, we want to find
@@ -370,6 +390,10 @@ export function makeSwingStore(dirPath, forceReset, options = {}) {
     inCrank || Fail`establishCrankSavepoint outside of crank`;
     const savepointOrdinal = savepoints.length;
     savepoints.push(savepoint);
+    // We must be in a transaction when creating the savepoint or releasing it
+    // later will cause an autocommit.
+    // See https://github.com/Agoric/agoric-sdk/issues/8423
+    ensureTxn();
     const sql = db.prepare(`SAVEPOINT t${savepointOrdinal}`);
     sql.run();
   }
@@ -477,7 +501,11 @@ export function makeSwingStore(dirPath, forceReset, options = {}) {
   }
 
   /** @type {import('./internal.js').SwingStoreInternal} */
-  const internal = harden({ snapStore, transcriptStore, bundleStore });
+  const internal = harden({
+    snapStore,
+    transcriptStore,
+    bundleStore,
+  });
 
   async function repairMetadata(exporter) {
     return doRepairMetadata(internal, exporter);

package/test/test-bundles.js

@@ -107,6 +107,9 @@ test('b0 import', async t => {
   const idA = makeB0ID(b0A);
   const nameA = `bundle.${idA}`;
   const exporter = {
+    getHostKV(_key) {
+      return undefined;
+    },
     async *getExportData() {
       yield /** @type {const} */ ([nameA, idA]);
     },
@@ -119,7 +122,11 @@ test('b0 import', async t => {
     },
     close: async () => undefined,
   };
-  const { kernelStorage } = await importSwingStore(exporter);
+  const ss = await importSwingStore(exporter);
+  t.teardown(ss.hostStorage.close);
+  await ss.hostStorage.commit();
+  const serialized = ss.debug.serialize();
+  const { kernelStorage } = initSwingStore(null, { serialized });
   const { bundleStore } = kernelStorage;
   t.truthy(bundleStore.hasBundle(idA));
   t.deepEqual(bundleStore.getBundle(idA), b0A);
@@ -131,6 +138,9 @@ test('b0 bad import', async t => {
   const idA = makeB0ID(b0A);
   const nameA = `bundle.${idA}`;
   const exporter = {
+    getHostKV(_key) {
+      return undefined;
+    },
     async *getExportData() {
       yield /** @type {const} */ ([nameA, idA]);
     },

package/test/test-export.js

@@ -36,6 +36,8 @@ const exportTest = test.macro(async (t, mode) => {
   const ss1 = initSwingStore(dbDir, options);
   const ks = ss1.kernelStorage;
 
+  ss1.hostStorage.kvStore.set('host.h1', 'hostvalue1');
+
   // build a DB with four spans (one in an old incarnation, two
   // historical but current incarnation, only one inUse) and two
   // snapshots (only one inUSe)
@@ -88,6 +90,13 @@ const exportTest = test.macro(async (t, mode) => {
   }
   const exporter = makeSwingStoreExporter(dbDir, { artifactMode });
 
+  // hostKV
+  t.is(exporter.getHostKV('host.h1'), 'hostvalue1');
+  t.is(exporter.getHostKV('host.hmissing'), undefined);
+  t.throws(() => exporter.getHostKV('nonhost'), {
+    message: 'getHostKV requires host keys',
+  });
+
   // exportData
   {
     const exportData = new Map();

package/test/test-exportImport.js

@@ -328,6 +328,7 @@ async function testExportImport(
   }
   t.is(failureMode, 'none');
   const ssIn = await doImport();
+  t.teardown(ssIn.hostStorage.close);
   await ssIn.hostStorage.commit();
   let dumpsShouldMatch = true;
   if (runMode === 'operational') {

package/test/test-import.js

@@ -54,6 +54,9 @@ function convert(orig) {
  */
 export function makeExporter(exportData, artifacts) {
   return {
+    getHostKV(_key) {
+      return undefined;
+    },
     async *getExportData() {
       for (const [key, value] of exportData.entries()) {
         /** @type { KVPair } */
@@ -81,6 +84,7 @@ test('import empty', async t => {
   t.teardown(cleanup);
   const exporter = makeExporter(new Map(), new Map());
   const ss = await importSwingStore(exporter, dbDir);
+  t.teardown(ss.hostStorage.close);
   await ss.hostStorage.commit();
   const data = convert(ss.debug.dump());
   t.deepEqual(data, {
@@ -164,6 +168,7 @@ const importTest = test.macro(async (t, mode) => {
 
   // now import
   const ss = await importSwingStore(exporter, dbDir, { artifactMode });
+  t.teardown(ss.hostStorage.close);
   await ss.hostStorage.commit();
   const data = convert(ss.debug.dump());
 

package/test/test-repair-metadata.js

@@ -6,7 +6,7 @@ import path from 'path';
 import test from 'ava';
 import sqlite3 from 'better-sqlite3';
 
-import { importSwingStore } from '../src/index.js';
+import { importSwingStore, openSwingStore } from '../src/index.js';
 
 import { makeExporter, buildData } from './test-import.js';
 import { tmpDir } from './util.js';
@@ -21,8 +21,9 @@ test('repair metadata', async t => {
   // then manually deleting the historical metadata entries from the
   // DB
   const exporter = makeExporter(exportData, artifacts);
-  const ss = await importSwingStore(exporter, dbDir);
-  await ss.hostStorage.commit();
+  const ssi = await importSwingStore(exporter, dbDir);
+  await ssi.hostStorage.commit();
+  await ssi.hostStorage.close();
 
   const filePath = path.join(dbDir, 'swingstore.sqlite');
   const db = sqlite3(filePath);
@@ -53,6 +54,8 @@ test('repair metadata', async t => {
   t.deepEqual(ss2, [7]);
 
   // now fix it
+  const ss = openSwingStore(dbDir);
+  t.teardown(ss.hostStorage.close);
   await ss.hostStorage.repairMetadata(exporter);
   await ss.hostStorage.commit();
 
@@ -64,6 +67,7 @@ test('repair metadata', async t => {
 
   // repair should be idempotent
   await ss.hostStorage.repairMetadata(exporter);
+  await ss.hostStorage.commit();
 
   const ts4 = getTS.all('v1');
   t.deepEqual(ts4, [0, 2, 5, 8]); // still there
@@ -78,11 +82,15 @@ test('repair metadata ignores kvStore entries', async t => {
   const { exportData, artifacts } = buildData();
 
   const exporter = makeExporter(exportData, artifacts);
-  const ss = await importSwingStore(exporter, dbDir);
-  await ss.hostStorage.commit();
+  const ssi = await importSwingStore(exporter, dbDir);
+  await ssi.hostStorage.commit();
+  await ssi.hostStorage.close();
 
   // perform the repair with spurious kv entries
   exportData.set('kv.key2', 'value2');
+
+  const ss = openSwingStore(dbDir);
+  t.teardown(ss.hostStorage.close);
   await ss.hostStorage.repairMetadata(exporter);
   await ss.hostStorage.commit();
 
@@ -97,14 +105,17 @@ test('repair metadata rejects mismatched snapshot entries', async t => {
   const { exportData, artifacts } = buildData();
 
   const exporter = makeExporter(exportData, artifacts);
-  const ss = await importSwingStore(exporter, dbDir);
-  await ss.hostStorage.commit();
+  const ssi = await importSwingStore(exporter, dbDir);
+  await ssi.hostStorage.commit();
+  await ssi.hostStorage.close();
 
   // perform the repair with mismatched snapshot entry
   const old = JSON.parse(exportData.get('snapshot.v1.4'));
   const wrong = { ...old, hash: 'wrong' };
   exportData.set('snapshot.v1.4', JSON.stringify(wrong));
 
+  const ss = openSwingStore(dbDir);
+  t.teardown(ss.hostStorage.close);
   await t.throwsAsync(async () => ss.hostStorage.repairMetadata(exporter), {
     message: /repairSnapshotRecord metadata mismatch/,
   });
@@ -117,14 +128,17 @@ test('repair metadata rejects mismatched transcript span', async t => {
   const { exportData, artifacts } = buildData();
 
   const exporter = makeExporter(exportData, artifacts);
-  const ss = await importSwingStore(exporter, dbDir);
-  await ss.hostStorage.commit();
+  const ssi = await importSwingStore(exporter, dbDir);
+  await ssi.hostStorage.commit();
+  await ssi.hostStorage.close();
 
   // perform the repair with mismatched transcript span entry
   const old = JSON.parse(exportData.get('transcript.v1.0'));
   const wrong = { ...old, hash: 'wrong' };
   exportData.set('transcript.v1.0', JSON.stringify(wrong));
 
+  const ss = openSwingStore(dbDir);
+  t.teardown(ss.hostStorage.close);
   await t.throwsAsync(async () => ss.hostStorage.repairMetadata(exporter), {
     message: /repairTranscriptSpanRecord metadata mismatch/,
   });

package/test/test-state.js

@@ -316,3 +316,37 @@ test('close will abort transaction', async t => {
   t.is(kvStore.get('key2'), undefined);
   t.falsy(kvStore.has('key2'));
 });
+
+test('savepoints', async t => {
+  const [dbDir, cleanup] = await tmpDir('testdb');
+  t.teardown(cleanup);
+  const ss1 = initSwingStore(dbDir);
+  ss1.kernelStorage.startCrank();
+  ss1.kernelStorage.kvStore.set('key', 'value1');
+  ss1.kernelStorage.establishCrankSavepoint('sp1');
+  ss1.kernelStorage.kvStore.set('key', 'value2');
+  ss1.kernelStorage.establishCrankSavepoint('sp2');
+  ss1.kernelStorage.kvStore.set('key', 'value3');
+  ss1.kernelStorage.rollbackCrank('sp1');
+  ss1.kernelStorage.endCrank();
+  await ss1.hostStorage.commit();
+  await ss1.hostStorage.close();
+
+  const ss2 = openSwingStore(dbDir);
+  t.is(ss2.kernelStorage.kvStore.get('key'), 'value1');
+});
+
+test('savepoints do not automatically commit', async t => {
+  const [dbDir, cleanup] = await tmpDir('testdb');
+  t.teardown(cleanup);
+  const ss1 = initSwingStore(dbDir);
+  ss1.kernelStorage.startCrank();
+  ss1.kernelStorage.establishCrankSavepoint('sp1');
+  ss1.kernelStorage.kvStore.set('key', 'value1');
+  // #8423 meant this .endCrank() accidentally did a commit()
+  ss1.kernelStorage.endCrank();
+  await ss1.hostStorage.close();
+
+  const ss2 = openSwingStore(dbDir);
+  t.false(ss2.kernelStorage.kvStore.has('key'));
+});