@agoric/swing-store 0.9.2-dev-2f092c3.0 → 0.9.2-dev-aa10ecd.0

package/docs/data-export.md

@@ -16,7 +16,7 @@ The SwingStore export protocol defines two stages (effectively two datasets). Th
 
 Each time a SwingStore API is used to modify the state somehow (e.g. adding/changing/deleting a `kvStore` entry, or pushing a new item on to a transcript), the contents of both datasets may change. New first-stage entries can be created, existing ones may be modified or deleted. And the set of second-stage artifacts may change.
 
-These export data/artifact changes can happen when calling into the kernel (e.g. invoking the external API of a device, causing the device code to change its own state or push messages onto the run-queue), or by normal kernel operations as it runs (any time `controller.run()` is executing). When the kernel is idle (after `controller.run()` has completed), the kernel will not make any changes to the SwingStore, and both datasets will be stable.
+These export data/artifact changes can happen when calling into the kernel (e.g. invoking the external API of a device, causing the device code to change its own state or push messages onto the run-queue), or by normal kernel operations as it runs (any time `controller.run()` is executing). When the kernel is idle (after `controller.run()` has completed), and `hostStorage.commit()` is called, the kernel will not make any changes to the SwingStore, and both datasets will be stable.
 
 Among other things, the SwingStore records a transcript of deliveries for each vat. The collection of all deliveries to a particular vat since its last heap snapshot was written is called the "current span". For each vat, the first-stage export data will record a single record that remembers the extent and the hash of the current span. This record then refers to a second-stage export artifact that contains the actual transcript contents.
 
@@ -83,6 +83,8 @@ So, to include SwingStore data in this state-sync snapshot, we need a way to get
 
 To support this, SwingStore has an "incremental export" mode. This is activated when the host application supplies an "export callback" option to the SwingStore instance constructor. Instead of retrieving the entire first-stage export data at the end of the block, the host application will be continuously notified about changes to this data as the kernel executes. The host application can then incorporate those entries into an existing hashed Merkle tree (e.g. the cosmos-sdk IAVL tree), whose root hash is included in the consensus block hash. Every time the callback is given `(key, value)`, the host should add a new (or modify some existing) IAVL entry, using an IAVL key within some range dedicated to the SwingStore first-stage export data. When the callback receives `(key, undefined)` or `(key, null)`, it should delete the entry. In this way, the IAVL tree maintains a "shadow copy" of the first-stage export data at all times, making the contents both covered by the consensus hash, and automatically included in the cosmos-sdk IAVL tree where it will become available to the new validator as it begins to reconstruct the SwingStore.
 
+The export callback must be established from the very beginning, so it includes all changes made during kernel initialization.
+
 All validator nodes use this export callback, even if they never perform the rest of the export process, to ensure that the consensus state includes the entire first-stage dataset. (Note that the first stage data is generally smaller than the full dataset, making this relatively inexpensive).
 
 Then, on the few occasions when the application needs to build a full state-sync snapshot, it can ask the SwingStore (after block commit) for the full set of artifacts that match the most recent commit.
@@ -177,18 +179,32 @@ As a result, for each active vat, the first-stage Export Data contains a record
 
 The `openSwingStore()` function has an option named `keepTranscripts` (which defaults to `true`), which causes the transcriptStore to retain the old transcript items. A second option named `keepSnapshots` (which defaults to `false`) causes the snapStore to retain the old heap snapshots. Opening the swingStore with a `false` option does not necessarily delete the old items immediately, but they'll probably get deleted the next time the kernel triggers a heap snapshot or transcript-span rollover. Validators who care about minimizing their disk usage will want to set both to `false`. In the future, we will arrange the SwingStore SQLite tables to provide easy `sqlite3` CLI commands that will delete the old data, so validators can also periodically use the CLI command to prune it.
 
-The `getArtifactNames()` API includes an option named `includeHistorical`. If `true`, all available historical artifacts will be included in the export (limited by what the `openSwingStore` options have deleted). If `false`, none will be included. Note that the "export data" is necessarily unaffected: if we *ever* want to validate this optional data, the hashes are mandatory. But the `getArtifactNames()` list will be smaller if you set `includeHistorical = false`. Also, re-exporting from a pruned copy will lack the old data, even if the re-export uses `includeHistorical = true`, because the second SwingStore cannot magically reconstruct the missing data.
+When exporting, the `makeSwingStoreExporter()` function takes an `exportMode=` argument. This serves to limit the set of artifacts that will be provided in the export. The defined values of `exportMode` are:
+* `current`: include only the current transcript span and current snapshot for each vat: just the minimum set necessary for current operations
+* `archival`: include all available transcript spans
+* `debug`: include all available transcript spans *and* all available snapshots. The old snapshots are never necessary for normal operations, nor are they likely to be usefor for extreme upgrade scenarios, but they might be useful for some unusual debugging operation
+
+Note that `exportMode` does not affect the Export Data generated by the exporter (if we *ever* want to validate this optional data, the hashes are mandatory). It only affects the names returned by `getArtifactNames()`: the list will be smaller for `current` than for `archival`. Re-exporting from a pruned copy will lack the old data, even if the re-export uses `archival`, because the second SwingStore cannot magically reconstruct the missing data.
 
 Note that when a vat is terminated, we delete all information about it, including transcript items and snapshots, both current and old. This will remove all the Export Data records, and well as the matching artifacts from `getArtifactNames`.
 
+When importing, the `importSwingStore()` function takes an options bag, which has property named `includeHistorical`. This property defaults to `false`, which makes the importer ignore any historical artifacts present in the export dataset. To import the historical transcript spans (and snapshots), you must set it to `true`.
+
+So, to convey historical transcript spans from one swingstore to another, you must set three options along the way:
+
+* the original swingstore must be opened with `{ includeHistorical: true }`, otherwise the old spans will be pruned immediately
+* the export must use `makeSwingStoreExporter(dirpath, 'archival')`, otherwise the export will omit the old spans
+* the import must use `importSwingStore(exporter, dirPath, { includeHistorical: true })`, otherwide teh import will ignore the old spans
+
 ## Implementation Details
 
-SwingStore contains components to accomodate all the various kinds of state that the SwingSet kernel needs to store. This currently consists of three portions:
+SwingStore contains components to accommodate all the various kinds of state that the SwingSet kernel needs to store. This currently consists of four portions:
 
 * `kvStore`, a general-purpose string/string key-value table
 * `transcriptStore`: append-only vat deliveries, broken into "spans", delimited by heap snapshot events
 * `snapshotStore`: binary blobs containing JS engine heap state, to limit transcript replay depth
+* `bundleStore`: code bundles that can be imported with `@endo/import-bundle`
 
-Currently, the SwingStore treats transcript spans and heap snapshots as export artifacts, with hashes recorded in the export data for validation (and to remember exactly which artifacts are necessary). The `kvStore` is copied one-to-one into the export data (i.e. we keep a full shadow copy in IAVL), because that is the fastest way to ensure the `kvStore` data is fully available and validated.
+Currently, the SwingStore treats transcript spans, heap snapshots, and bundles as export artifacts, with hashes recorded in the export data for validation (and to remember exactly which artifacts are necessary). The `kvStore` is copied one-to-one into the export data (i.e. we keep a full shadow copy in IAVL), because that is the fastest way to ensure the `kvStore` data is fully available and validated.
 
 If some day we implement an IAVL-like Merkle tree inside SwingStore, and use it to automatically generate a root hash for the `kvStore` at the end of each block, we will replace this (large) shadow copy with a single `kvStoreRootHash` entry, and add a new export artifact to contain the full contents of the kvStore. This reduce the size of the IAVL tree, as well as the rate of IAVL updates during block execution, at the cost of increased CPU and complexity within SwingStore.

package/docs/swingstore.md

@@ -0,0 +1,52 @@
+# SwingStore Data Model
+
+The "SwingStore" provides a database to hold SwingSet kernel state, with an API crafted to help both the kernel and the host application mutate, commit, export, and import this state.
+
+The state is broken up into several pieces, or "stores":
+
+* `bundleStore`: a string-keyed Bundle-value table, holding source bundles which can be evaluated by `importBundle` to create vats, or new Compartments within a vat
+* `transcriptStore`: records a linear sequence of deliveries and syscalls (with results), collectively known as "transcript entries", for each vat
+* `snapStore`: records one or more XS heap snapshots for each vat, to rebuild a worker more efficiently than replaying all transcript entries from the beginning
+* `kvStore`: a string-keyed string-valued table, which holds everything else. Currently, this holds each vat's c-list and vatstore data, as well as the kernel-wide object and promise tables, and run-queues.
+
+## Incarnations, Spans, Snapshots
+
+The kernel tracks the state of one or more vats. Each vat's execution is split into "incarnations", which are separated by a "vat upgrade" (a call to `E(vatAdminFacet).upgrade(newBundleCap, options)`, see https://github.com/Agoric/agoric-sdk/blob/master/packages/SwingSet/docs/vat-upgrade.md for details). Each incarnation gets a new worker, which erases the heap state and only retains durable vatstore data across the upgrade. Every active vat has a "current incarnation", and zero or more "historic incarnations". Only the current incarnation is instantiated.
+
+Within each incarnation, execution is broken into one or more "spans", with a "current span" and zero or more "historic spans". This breaks up the transcript into corresponding spans.
+
+Each historic span ends with a `save-snapshot` entry which records the creation and saving of an XS heap snapshot. The initial span starts with a `start-worker` entry, while all non-initial spans start with a `load-snapshot` entry. The final span of historic incarnations each end with a `shutdown-worker` entry.
+
+Each `save-snapshot` entry adds a new snapshot to the `snapStore`, so each vat has zero or more snapshots, of which the last one is called the "current" or "in-use" snapshot, and the earlier ones are called "historical snapshots".
+
+(note: the `deliveryNum` counter is scoped to the vat and does not reset at incarnation or span boundaries)
+
+## Artifacts
+
+The import/export process (using `makeSwingStoreExporter` and `importSwingStore`) defines some number of "artifacts" to contain much of the SwingStore data. Each bundle is a separate artifact, as is each heap snapshot. Each transcript span is a separate artifact (an aggregate of the individual transcript entries comprising that span).
+
+During export, the `getArtifactNames()` method provides a list of all available artifacts, while `getArtifact(name)` is used to retrieve the actual data. The import function processes each artifact separately.
+
+## Populated vs Pruned
+
+For normal operation, the kernel does not require historical incarnations, spans, or snapshots. It only needs the ability to reconstruct a worker for the current incarnation of each vat, which means loading the current snapshot (if any), and replaying the contents of the current transcript span.
+
+For this reason, the swingstore must always contain the current transcript span, and the current snapshot (if any), for every vat.
+
+However, to save space, historical spans/snapshots might be pruned, by deleting their contents from the database (but retaining the metadata, which includes a hash of the contents for later validation). Historical snapshots are pruned by default (unless `openSwingStore()` is given an options bag with `keepSnapshots: true`). Historical spans are not currently pruned (the `keepTranscripts` option defaults to `true`), but that may change.
+
+In addition, `importSwingStore()` can be used to create a SwingStore from data exported out of some other SwingStore. The export-then-import process might result in a pruned DB in one of three ways:
+
+* the import-time options might instruct the import process to ignore some of the available data
+* the export-time options might have done the same
+* the original DB was itself already pruned, so the data was not available in the first place
+
+In the future, a separate SwingStore API will exist to allow previously-pruned artifacts to be repopulated. Every artifact has a metadata record which *is* included in the export (in the `exportData` section, but separate from the kvStore shadow table entries, see [data-export.md](./data-export.md)), regardless of pruning modes, to ensure that this API can check the integrity of these repopulated artifacts. This reduces the reliance set and trust burden of the repopulation process (we can safely use untrusted artifact providers).
+
+When a snapshot is pruned, the `snapshots` SQL table row is modified, replacing its `compressedSnapshot` BLOB with a NULL. The other columns are left alone, especially the `hash` column, which retains the integrity-checking metadata to support a future repopulation.
+
+When a transcript span is pruned, the `transcriptSpans` row is left alone, but the collection of `transcriptItems` rows are deleted. Any span for which all the `transcriptItems` rows are present is said to be "populated", while any span that is missing one or more `transcriptItems` rows is said to be "pruned". (There is no good reason for a span to be only partially pruned, but until we compress historical spans into a single row, in some new table, there remains the possibility of partial pruning).
+
+During import, we create the metadata first (as the export-data is parsed), then later, we fill in the details as the artifacts are read.
+
+Bundles are never pruned, however during import, the `bundles` table will temporarily contain rows whose `bundle` BLOB is NULL.

package/package.json

@@ -1,9 +1,12 @@
 {
   "name": "@agoric/swing-store",
-  "version": "0.9.2-dev-2f092c3.0+2f092c3",
+  "version": "0.9.2-dev-aa10ecd.0+aa10ecd",
   "description": "Persistent storage for SwingSet",
   "type": "module",
-  "main": "src/swingStore.js",
+  "main": "./src/index.js",
+  "exports": {
+    ".": "./src/index.js"
+  },
   "repository": "https://github.com/Agoric/agoric-sdk",
   "author": "Agoric",
   "license": "Apache-2.0",
@@ -18,8 +21,8 @@
     "lint:eslint": "eslint ."
   },
   "dependencies": {
-    "@agoric/assert": "0.6.1-dev-2f092c3.0+2f092c3",
-    "@agoric/internal": "0.3.3-dev-2f092c3.0+2f092c3",
+    "@agoric/assert": "0.6.1-dev-aa10ecd.0+aa10ecd",
+    "@agoric/internal": "0.3.3-dev-aa10ecd.0+aa10ecd",
     "@endo/base64": "^0.2.32",
     "@endo/bundle-source": "^2.5.2",
     "@endo/check-bundle": "^0.2.19",
@@ -42,5 +45,5 @@
     ],
     "timeout": "2m"
   },
-  "gitHead": "2f092c3a20ac275c3402645d3d984a3657cf20d5"
+  "gitHead": "aa10ecdb73b23c9a11077b118fa562bff1708c74"
 }

package/src/assertComplete.js

@@ -0,0 +1,22 @@
+/**
+ * @param {import('./internal.js').SwingStoreInternal} internal
+ * @param {'operational'} level
+ * @returns {void}
+ */
+export function assertComplete(internal, level) {
+  assert.equal(level, 'operational'); // only option for now
+  // every bundle must be populated
+  internal.bundleStore.assertComplete(level);
+
+  // every 'isCurrent' transcript span must have all items
+  // TODO: every vat with any data must have a isCurrent transcript
+  // span
+  internal.transcriptStore.assertComplete(level);
+
+  // every 'inUse' snapshot must be populated
+  internal.snapStore.assertComplete(level);
+
+  // TODO: every isCurrent span that starts with load-snapshot has a
+  // matching snapshot (counter-argument: swing-store should not know
+  // those details about transcript entries)
+}

package/src/bundleStore.js

@@ -7,7 +7,6 @@ import { checkBundle } from '@endo/check-bundle/lite.js';
 import { Nat } from '@endo/nat';
 import { Fail, q } from '@agoric/assert';
 import { createSHA256 } from './hasher.js';
-import { buffer } from './util.js';
 
 /**
  * @typedef { { moduleFormat: 'getExport', source: string, sourceMap?: string } } GetExportBundle
@@ -16,7 +15,7 @@ import { buffer } from './util.js';
  * @typedef { EndoZipBase64Bundle | GetExportBundle | NestedEvaluateBundle } Bundle
  */
 /**
- * @typedef { import('./swingStore').SwingStoreExporter } SwingStoreExporter
+ * @typedef { import('./exporter').SwingStoreExporter } SwingStoreExporter
  *
  * @typedef {{
  *   addBundle: (bundleID: string, bundle: Bundle) => void;
@@ -27,7 +26,10 @@ import { buffer } from './util.js';
  *
  * @typedef {{
  *   exportBundle: (name: string) => AsyncIterableIterator<Uint8Array>,
- *   importBundle: (artifactName: string, exporter: SwingStoreExporter, bundleID: string) => void,
+ *   repairBundleRecord: (key: string, value: string) => void,
+ *   importBundleRecord: (key: string, value: string) => void,
+ *   importBundle: (name: string, dataProvider: () => Promise<Buffer>) => Promise<void>,
+ *   assertComplete: (level: 'operational') => void,
  *   getExportRecords: () => IterableIterator<readonly [key: string, value: string]>,
  *   getArtifactNames: () => AsyncIterableIterator<string>,
  *   getBundleIDs: () => IterableIterator<string>,
@@ -39,6 +41,18 @@ import { buffer } from './util.js';
  *
  */
 
+function bundleIDFromName(name) {
+  typeof name === 'string' || Fail`artifact name must be a string`;
+  const [tag, ...pieces] = name.split('.');
+  if (tag !== 'bundle' || pieces.length !== 1) {
+    Fail`expected artifact name of the form 'bundle.{bundleID}', saw ${q(
+      name,
+    )}`;
+  }
+  const bundleID = pieces[0];
+  return bundleID;
+}
+
 /**
  * @param {*} db
  * @param {() => void} ensureTxn
@@ -54,6 +68,9 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
     )
   `);
 
+  // A populated record contains both bundleID and bundle, while a
+  // pruned record has a bundle of NULL.
+
   function bundleArtifactName(bundleID) {
     return `bundle.${bundleID}`;
   }
@@ -62,20 +79,36 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
     return `b${Nat(version)}-${hash}`;
   }
 
-  const sqlAddBundle = db.prepare(`
-    INSERT OR REPLACE INTO bundles
-      (bundleID, bundle)
-    VALUES (?, ?)
+  // the PRIMARY KEY constraint requires the bundleID not already
+  // exist
+  const sqlAddBundleRecord = db.prepare(`
+    INSERT INTO bundles (bundleID, bundle) VALUES (?, NULL)
   `);
 
-  /**
-   * Store a bundle.  Here the bundle itself is presumed valid.
-   *
-   * @param {string} bundleID
-   * @param {Bundle} bundle
-   */
-  function addBundle(bundleID, bundle) {
+  // this sees both populated and pruned (not-yet-populated) records
+  const sqlHasBundleRecord = db.prepare(`
+    SELECT count(*)
+    FROM bundles
+    WHERE bundleID = ?
+  `);
+  sqlHasBundleRecord.pluck();
+
+  const sqlPopulateBundleRecord = db.prepare(`
+    UPDATE bundles SET bundle = $serialized WHERE bundleID = $bundleID
+  `);
+
+  function addBundleRecord(bundleID) {
+    ensureTxn();
+    sqlAddBundleRecord.run(bundleID);
+  }
+
+  function populateBundle(bundleID, serialized) {
     ensureTxn();
+    sqlHasBundleRecord.get(bundleID) || Fail`missing ${bundleID}`;
+    sqlPopulateBundleRecord.run({ bundleID, serialized });
+  }
+
+  function serializeBundle(bundleID, bundle) {
     const { moduleFormat } = bundle;
     let serialized;
     if (bundleID.startsWith('b0-')) {
@@ -98,19 +131,55 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
     } else {
       throw Fail`unsupported BundleID ${bundleID}`;
     }
-    sqlAddBundle.run(bundleID, serialized);
+    return serialized;
+  }
+
+  /**
+   * Store a complete bundle in a single operation, used by runtime
+   * (i.e. not an import). We rely upon the caller to provide a
+   * correct bundle (e.g. no unexpected properties), but we still
+   * check the ID against the contents.
+   *
+   * @param {string} bundleID
+   * @param {Bundle} bundle
+   */
+  function addBundle(bundleID, bundle) {
+    const serialized = serializeBundle(bundleID, bundle);
+    addBundleRecord(bundleID);
+    populateBundle(bundleID, serialized);
     noteExport(bundleArtifactName(bundleID), bundleID);
   }
 
-  const sqlHasBundle = db.prepare(`
+  const sqlGetPrunedBundles = db.prepare(`
+    SELECT bundleID
+    FROM bundles
+    WHERE bundle IS NULL
+    ORDER BY bundleID
+  `);
+  sqlGetPrunedBundles.pluck();
+
+  function getPrunedBundles() {
+    return sqlGetPrunedBundles.all();
+  }
+
+  function assertComplete(level) {
+    assert.equal(level, 'operational'); // for now
+    const pruned = getPrunedBundles();
+    if (pruned.length) {
+      throw Fail`missing bundles for: ${pruned.join(',')}`;
+    }
+  }
+
+  const sqlHasPopulatedBundle = db.prepare(`
     SELECT count(*)
     FROM bundles
     WHERE bundleID = ?
+    AND bundle IS NOT NULL
   `);
-  sqlHasBundle.pluck(true);
+  sqlHasPopulatedBundle.pluck(true);
 
   function hasBundle(bundleID) {
-    const count = sqlHasBundle.get(bundleID);
+    const count = sqlHasPopulatedBundle.get(bundleID);
     return count !== 0;
   }
 
@@ -119,15 +188,15 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
     FROM bundles
     WHERE bundleID = ?
   `);
-  sqlGetBundle.pluck(true);
 
   /**
    * @param {string} bundleID
    * @returns {Bundle}
    */
   function getBundle(bundleID) {
-    const rawBundle = sqlGetBundle.get(bundleID);
-    rawBundle || Fail`bundle ${q(bundleID)} not found`;
+    const row =
+      sqlGetBundle.get(bundleID) || Fail`bundle ${q(bundleID)} not found`;
+    const rawBundle = row.bundle || Fail`bundle ${q(bundleID)} pruned`;
     if (bundleID.startsWith('b0-')) {
       return harden(JSON.parse(rawBundle));
     } else if (bundleID.startsWith('b1-')) {
@@ -153,6 +222,31 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
     }
   }
 
+  // take an export-data record (id/hash but not bundle contents) and
+  // insert something in the DB
+  function importBundleRecord(key, value) {
+    const bundleID = bundleIDFromName(key);
+    assert.equal(bundleID, value);
+    addBundleRecord(bundleID);
+  }
+
+  function repairBundleRecord(key, value) {
+    // Bundle records have no metadata, and all bundles must be
+    // present (there's no notion of "historical bundle"). So there's
+    // no "repair", and if the repair process supplies a bundle record
+    // that isn't already present, we throw an error. The repair
+    // process doesn't get artifacts, so adding a new record here
+    // would fail the subsequent completeness check anyways.
+
+    const bundleID = bundleIDFromName(key);
+    assert.equal(bundleID, value);
+    if (sqlHasBundleRecord.get(bundleID)) {
+      // record is present, there's no metadata to mismatch, so ignore quietly
+      return;
+    }
+    throw Fail`unexpected new bundle record for ${bundleID} during repair`;
+  }
+
   /**
    * Read a bundle and return it as a stream of data suitable for export to
    * another store.
@@ -166,14 +260,10 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
    * @returns {AsyncIterableIterator<Uint8Array>}
    */
   async function* exportBundle(name) {
-    typeof name === 'string' || Fail`artifact name must be a string`;
-    const parts = name.split('.');
-    const [type, bundleID] = parts;
-    // prettier-ignore
-    (parts.length === 2 && type === 'bundle') ||
-      Fail`expected artifact name of the form 'bundle.{bundleID}', saw ${q(name)}`;
-    const rawBundle = sqlGetBundle.get(bundleID);
-    rawBundle || Fail`bundle ${q(name)} not available`;
+    const bundleID = bundleIDFromName(name);
+    const row =
+      sqlGetBundle.get(bundleID) || Fail`bundle ${q(bundleID)} not found`;
+    const rawBundle = row.bundle || Fail`bundle ${q(bundleID)} pruned`;
     yield* Readable.from(Buffer.from(rawBundle));
   }
 
@@ -209,23 +299,17 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
   }
 
   /**
-   * @param {string} name  Artifact name of the bundle
-   * @param {SwingStoreExporter} exporter  Whence to get the bits
-   * @param {string} bundleID  Bundle ID of the bundle
+   * Call addBundleRecord() first, then this importBundle() will
+   * populate the record.
+   *
+   * @param {string} name  Artifact name, `bundle.${bundleID}`
+   * @param {() => Promise<Buffer>} dataProvider  Function to get bundle bytes
    * @returns {Promise<void>}
    */
-  async function importBundle(name, exporter, bundleID) {
+  async function importBundle(name, dataProvider) {
     await 0; // no synchronous prefix
-    const parts = name.split('.');
-    const [type, bundleIDkey] = parts;
-    // prettier-ignore
-    parts.length === 2 && type === 'bundle' ||
-      Fail`expected artifact name of the form 'bundle.{bundleID}', saw '${q(name)}'`;
-    bundleIDkey === bundleID ||
-      Fail`bundle artifact name ${name} doesn't match bundleID ${bundleID}`;
-    const artifactChunks = exporter.getArtifact(name);
-    const inStream = Readable.from(artifactChunks);
-    const data = await buffer(inStream);
+    const bundleID = bundleIDFromName(name);
+    const data = await dataProvider();
     if (bundleID.startsWith('b0-')) {
       // we dissect and reassemble the bundle, to exclude unexpected properties
       const { moduleFormat, source, sourceMap } = JSON.parse(data.toString());
@@ -234,7 +318,7 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
       const serialized = JSON.stringify(bundle);
       bundleID === bundleIdFromHash(0, createSHA256(serialized).finish()) ||
         Fail`bundleID ${q(bundleID)} does not match bundle artifact`;
-      addBundle(bundleID, bundle);
+      populateBundle(bundleID, serialized);
     } else if (bundleID.startsWith('b1-')) {
       /** @type {EndoZipBase64Bundle} */
       const bundle = harden({
@@ -244,7 +328,7 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
       });
       // Assert that the bundle contents match the ID and hash
       await checkBundle(bundle, computeSha512, bundleID);
-      addBundle(bundleID, bundle);
+      populateBundle(bundleID, serializeBundle(bundleID, bundle));
     } else {
       Fail`unsupported BundleID ${q(bundleID)}`;
     }
@@ -264,7 +348,7 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
     const dump = {};
     for (const row of sql.iterate()) {
       const { bundleID, bundle } = row;
-      dump[bundleID] = encodeBase64(bundle);
+      dump[bundleID] = encodeBase64(Buffer.from(bundle, 'utf-8'));
     }
     return dump;
   }
@@ -281,15 +365,20 @@ export function makeBundleStore(db, ensureTxn, noteExport = () => {}) {
   }
 
   return harden({
+    importBundleRecord,
+    importBundle,
+    assertComplete,
+
     addBundle,
     hasBundle,
     getBundle,
     deleteBundle,
+
     getExportRecords,
     getArtifactNames,
     exportBundle,
-    importBundle,
     getBundleIDs,
+    repairBundleRecord,
 
     dumpBundles,
   });

package/src/exporter.js

@@ -0,0 +1,175 @@
+import sqlite3 from 'better-sqlite3';
+
+import { Fail, q } from '@agoric/assert';
+
+import { dbFileInDirectory } from './util.js';
+import { getKeyType } from './kvStore.js';
+import { makeBundleStore } from './bundleStore.js';
+import { makeSnapStore } from './snapStore.js';
+import { makeSnapStoreIO } from './snapStoreIO.js';
+import { makeTranscriptStore } from './transcriptStore.js';
+
+/**
+ * @template T
+ *  @typedef  { Iterable<T> | AsyncIterable<T> } AnyIterable<T>
+ */
+/**
+ * @template T
+ *  @typedef  { IterableIterator<T> | AsyncIterableIterator<T> } AnyIterableIterator<T>
+ */
+
+/**
+ *
+ * @typedef {readonly [
+ *   key: string,
+ *   value?: string | null | undefined,
+ * ]} KVPair
+ *
+ * @typedef {object} SwingStoreExporter
+ *
+ * Allows export of data from a swingStore as a fixed view onto the content as
+ * of the most recent commit point at the time the exporter was created.  The
+ * exporter may be used while another SwingStore instance is active for the same
+ * DB, possibly in another thread or process.  It guarantees that regardless of
+ * the concurrent activity of other swingStore instances, the data representing
+ * the commit point will stay consistent and available.
+ *
+ * @property {() => AnyIterableIterator<KVPair>} getExportData
+ *
+ * Get a full copy of the first-stage export data (key-value pairs) from the
+ * swingStore. This represents both the contents of the KVStore (excluding host
+ * and local prefixes), as well as any data needed to validate all artifacts,
+ * both current and historical. As such it represents the root of trust for the
+ * application.
+ *
+ * Content of validation data (with supporting entries for indexing):
+ * - kv.${key} = ${value}  // ordinary kvStore data entry
+ * - snapshot.${vatID}.${snapPos} = ${{ vatID, snapPos, hash }};
+ * - snapshot.${vatID}.current = `snapshot.${vatID}.${snapPos}`
+ * - transcript.${vatID}.${startPos} = ${{ vatID, startPos, endPos, hash }}
+ * - transcript.${vatID}.current = ${{ vatID, startPos, endPos, hash }}
+ *
+ * @property {() => AnyIterableIterator<string>} getArtifactNames
+ *
+ * Get a list of name of artifacts available from the swingStore.  A name
+ * returned by this method guarantees that a call to `getArtifact` on the same
+ * exporter instance will succeed.  The `exportMode` option to
+ * `makeSwingStoreExporter` controls the filtering of the artifact names
+ * yielded.
+ *
+ * Artifact names:
+ * - transcript.${vatID}.${startPos}.${endPos}
+ * - snapshot.${vatID}.${snapPos}
+ * - bundle.${bundleID}
+ *
+ * @property {(name: string) => AnyIterableIterator<Uint8Array>} getArtifact
+ *
+ * Retrieve an artifact by name as a sequence of binary chunks.  May throw if
+ * the artifact is not available, which can occur if the artifact is historical
+ * and wasn't preserved.
+ *
+ * @property {() => Promise<void>} close
+ *
+ * Dispose of all resources held by this exporter. Any further operation on this
+ * exporter or its outstanding iterators will fail.
+ */
+
+/**
+ * @typedef {'current' | 'archival' | 'debug'} ExportMode
+ */
+
+/**
+ * @param {string} dirPath
+ * @param { ExportMode } exportMode
+ * @returns {SwingStoreExporter}
+ */
+export function makeSwingStoreExporter(dirPath, exportMode = 'current') {
+  typeof dirPath === 'string' || Fail`dirPath must be a string`;
+  exportMode === 'current' ||
+    exportMode === 'archival' ||
+    exportMode === 'debug' ||
+    Fail`invalid exportMode ${q(exportMode)}`;
+  const exportHistoricalSnapshots = exportMode === 'debug';
+  const exportHistoricalTranscripts = exportMode !== 'current';
+  const filePath = dbFileInDirectory(dirPath);
+  const db = sqlite3(filePath);
+
+  // Execute the data export in a (read) transaction, to ensure that we are
+  // capturing the state of the database at a single point in time. Our close()
+  // will ROLLBACK the txn just in case some bug tried to change the DB.
+  const sqlBeginTransaction = db.prepare('BEGIN TRANSACTION');
+  sqlBeginTransaction.run();
+
+  // ensureTxn can be a dummy, we just started one
+  const ensureTxn = () => {};
+  const snapStore = makeSnapStore(db, ensureTxn, makeSnapStoreIO());
+  const bundleStore = makeBundleStore(db, ensureTxn);
+  const transcriptStore = makeTranscriptStore(db, ensureTxn, () => {});
+
+  const sqlGetAllKVData = db.prepare(`
+    SELECT key, value
+    FROM kvStore
+    ORDER BY key
+  `);
+
+  /**
+   * @returns {AsyncIterableIterator<KVPair>}
+   * @yields {KVPair}
+   */
+  async function* getExportData() {
+    for (const { key, value } of sqlGetAllKVData.iterate()) {
+      if (getKeyType(key) === 'consensus') {
+        yield [`kv.${key}`, value];
+      }
+    }
+    yield* snapStore.getExportRecords(true);
+    yield* transcriptStore.getExportRecords(true);
+    yield* bundleStore.getExportRecords();
+  }
+
+  /**
+   * @returns {AsyncIterableIterator<string>}
+   * @yields {string}
+   */
+  async function* getArtifactNames() {
+    yield* snapStore.getArtifactNames(exportHistoricalSnapshots);
+    yield* transcriptStore.getArtifactNames(exportHistoricalTranscripts);
+    yield* bundleStore.getArtifactNames();
+  }
+
+  /**
+   * @param {string} name
+   * @returns {AsyncIterableIterator<Uint8Array>}
+   */
+  function getArtifact(name) {
+    typeof name === 'string' || Fail`artifact name must be a string`;
+    const [type] = name.split('.', 1);
+
+    if (type === 'snapshot') {
+      return snapStore.exportSnapshot(name);
+    } else if (type === 'transcript') {
+      return transcriptStore.exportSpan(name);
+    } else if (type === 'bundle') {
+      return bundleStore.exportBundle(name);
+    } else {
+      throw Fail`invalid type in artifact name ${q(name)}`;
+    }
+  }
+
+  const sqlAbort = db.prepare('ROLLBACK');
+
+  async function close() {
+    // After all the data has been extracted, always abort the export
+    // transaction to ensure that the export was read-only (i.e., that no bugs
+    // inadvertantly modified the database).
+    sqlAbort.run();
+    db.close();
+  }
+
+  return harden({
+    getExportData,
+    getArtifactNames,
+    getArtifact,
+    close,
+  });
+}