@agoric/swing-store 0.9.1 → 0.9.2-calypso-dev-84eb287.0

package/docs/data-export.md

@@ -16,7 +16,7 @@ The SwingStore export protocol defines two stages (effectively two datasets). Th
 
 Each time a SwingStore API is used to modify the state somehow (e.g. adding/changing/deleting a `kvStore` entry, or pushing a new item on to a transcript), the contents of both datasets may change. New first-stage entries can be created, existing ones may be modified or deleted. And the set of second-stage artifacts may change.
 
-These export data/artifact changes can happen when calling into the kernel (e.g. invoking the external API of a device, causing the device code to change its own state or push messages onto the run-queue), or by normal kernel operations as it runs (any time `controller.run()` is executing). When the kernel is idle (after `controller.run()` has completed), the kernel will not make any changes to the SwingStore, and both datasets will be stable.
+These export data/artifact changes can happen when calling into the kernel (e.g. invoking the external API of a device, causing the device code to change its own state or push messages onto the run-queue), or by normal kernel operations as it runs (any time `controller.run()` is executing). When the kernel is idle (after `controller.run()` has completed), and `hostStorage.commit()` is called, the kernel will not make any changes to the SwingStore, and both datasets will be stable.
 
 Among other things, the SwingStore records a transcript of deliveries for each vat. The collection of all deliveries to a particular vat since its last heap snapshot was written is called the "current span". For each vat, the first-stage export data will record a single record that remembers the extent and the hash of the current span. This record then refers to a second-stage export artifact that contains the actual transcript contents.
 
@@ -39,30 +39,41 @@ The exporter is created by calling `makeSwingStoreExporter(dirpath)`, passing it
 After calling `hostStorage.commit()`, the host application can extract the first-stage export data, and then the second-stage export artifacts:
 
 ```js
-   import { buffer } from 'node:stream/consumers';
-
-   const dirPath = '.../swing-store';
-   const swingStore = openSwingStore(dirPath);
-   ...
-   await controller.run();
-   hostStorage.commit();
-   // spawn a child process
-
-  // child process does:
-  const exporter = makeSwingStoreExporter(dirPath);
-  // exporter now has a txn, parent process is free to proceed forward
-  const exportData = new Map();
-  for (const [key, value] of exporter.getExportData()) {
-    exportData.set(key, value);
-  }
-  const exportArtifacts = new Map();
-  for (const name of exporter.getArtifactNames()) {
-    const reader = exporter.getArtifact(name);
-    // reader is an async iterable of Uint8Array, e.g. a stream
-    const data = await buffer(reader);
-    exportArtifacts.set(name, data);
+const dirPath = '.../swing-store';
+const swingStore = openSwingStore(dirPath);
+...
+await controller.run();
+hostStorage.commit();
+// spawn a child process and wait for it to open a transaction
+const started = makePromiseKit();
+const child = fork(path, args);
+child.on('error', started.reject);
+child.on('exit', started.reject);
+child.on('message', msg => {
+  if (msg?.type === 'started') {
+    started.resolve();
   }
-  // export is 'exportData' and 'exportArtifacts'
+});
+await started.promise;
+...
+
+// child process does:
+import { buffer } from 'node:stream/consumers';
+const exporter = makeSwingStoreExporter(dirPath);
+// exporter now has a txn, so parent process is free to proceed forward
+process.send({ type: 'started' });
+const exportData = new Map();
+for (const [key, value] of exporter.getExportData()) {
+  exportData.set(key, value);
+}
+const exportArtifacts = new Map();
+for (const name of exporter.getArtifactNames()) {
+  const reader = exporter.getArtifact(name);
+  // reader is an async iterable of Uint8Array, e.g. a stream
+  const data = await buffer(reader);
+  exportArtifacts.set(name, data);
+}
+// export is the combination of 'exportData' and 'exportArtifacts'
 ```
 
 ![image 3](./images/data-export-3.jpg)
@@ -83,6 +94,8 @@ So, to include SwingStore data in this state-sync snapshot, we need a way to get
 
 To support this, SwingStore has an "incremental export" mode. This is activated when the host application supplies an "export callback" option to the SwingStore instance constructor. Instead of retrieving the entire first-stage export data at the end of the block, the host application will be continuously notified about changes to this data as the kernel executes. The host application can then incorporate those entries into an existing hashed Merkle tree (e.g. the cosmos-sdk IAVL tree), whose root hash is included in the consensus block hash. Every time the callback is given `(key, value)`, the host should add a new (or modify some existing) IAVL entry, using an IAVL key within some range dedicated to the SwingStore first-stage export data. When the callback receives `(key, undefined)` or `(key, null)`, it should delete the entry. In this way, the IAVL tree maintains a "shadow copy" of the first-stage export data at all times, making the contents both covered by the consensus hash, and automatically included in the cosmos-sdk IAVL tree where it will become available to the new validator as it begins to reconstruct the SwingStore.
 
+The export callback must be established from the very beginning, so it includes all changes made during kernel initialization.
+
 All validator nodes use this export callback, even if they never perform the rest of the export process, to ensure that the consensus state includes the entire first-stage dataset. (Note that the first stage data is generally smaller than the full dataset, making this relatively inexpensive).
 
 Then, on the few occasions when the application needs to build a full state-sync snapshot, it can ask the SwingStore (after block commit) for the full set of artifacts that match the most recent commit.
@@ -90,54 +103,58 @@ Then, on the few occasions when the application needs to build a full state-sync
 ![image 4](./images/data-export-4.jpg)
 
 ```js
-   const dirPath = '.../swing-store';
-   const iavl = ...;
-   function exportCallback(key, value) {
-     const iavlKey = `ssed.${key}`; // 'ssed' is short for SwingStoreExportData
-     if (value) {
-       iavl.set(iavlKey, value);
-     } else {
-       iavl.delete(iavlKey); // value===undefined means delete
-     }
-   }
-   const swingStore = openSwingStore(dirPath, { exportCallback });
-   ...
-   await controller.run();
-   hostStorage.commit();
-
-   // now, if the validator is configured to publish state-sync snapshots,
-   // and if this block height is one of the publishing points,
-   // do the following:
-
-  // spawn a child process
-
-  // child process does:
-  const exporter = makeSwingStoreExporter(dirPath);
-  // note: no exporter.getExportData(), the first-stage data is already in IAVL
-  const artifacts = new Map();
-  for (const name of exporter.getArtifactNames()) {
-    artifacts.set(name, exporter.getArtifact(name));
+const dirPath = '.../swing-store';
+const iavl = ...;
+function exportCallback(key, value) {
+  const iavlKey = `ssed.${key}`; // 'ssed' is short for SwingStoreExportData
+  if (value === undefined) {
+    iavl.delete(iavlKey);
+  } else {
+    iavl.set(iavlKey, value);
+  }
+}
+const swingStore = openSwingStore(dirPath, { exportCallback });
+...
+await controller.run();
+hostStorage.commit();
+
+// now, if the validator is configured to publish state-sync snapshots,
+// and if this block height is one of the publishing points,
+// do the following:
+
+// spawn a child process and wait for it to open a transaction
+const started = makePromiseKit();
+const child = fork(path, args);
+child.on('error', started.reject);
+child.on('exit', started.reject);
+child.on('message', msg => {
+  if (msg?.type === 'started') {
+    started.resolve();
   }
-  // instruct cosmos-sdk to include 'artifacts' in the state-sync snapshot
+});
+try {
+  await started.promise;
+} catch (err) {
+  ...
+}
+...
+
+// child process does:
+const exporter = makeSwingStoreExporter(dirPath);
+process.send({ type: 'started' });
+// note: no exporter.getExportData(), the first-stage data is already in IAVL
+const artifacts = new Map();
+for (const name of exporter.getArtifactNames()) {
+  artifacts.set(name, exporter.getArtifact(name));
+}
+// instruct cosmos-sdk to include 'artifacts' in the state-sync snapshot
 ```
 
 ## Import
 
-On other end of the export process is an importer. This is a new host application, which wants to start from the contents of the export, rather than initializing a brand new (empty) kernel state.
+On the other end of the export process is an importer. This is used to restore kernel state, so that a new host application can simply continue mostly as if it had been previously executing. The expectation is that the import and the execution are 2 independent events, and the execution doesn't need to be aware it was imported.
 
-When starting a brand new instance, host applications would normally call `openSwingStore(dirPath)` to create a new (empty) SwingStore, then call SwingSet's `initializeSwingset(config, .., kernelStorage)` to let the kernel initialize the DB with a config-dependent starting state:
-
-```js
-// this is done only the first time an instance is created:
-
-import { openSwingStore } from '@agoric/swing-store';
-import { initializeSwingset } from '@agoric/swingset-vat';
-const dirPath = './swing-store';
-const { hostStorage, kernelStorage } = openSwingStore(dirPath);
-await initializeSwingset(config, argv, kernelStorage);
-```
-
-Once the initial state is created, each time the application is launched, it will build a controller around the existing state:
+For reference, after the initial state is created, each time the application is launched, it builds a controller around the existing state:
 
 ```js
 import { openSwingStore } from '@agoric/swing-store';
@@ -148,7 +165,7 @@ const controller = await makeSwingsetController(kernelStorage);
 // ... now do things like controller.run(), etc
 ```
 
-When cloning an existing kernel, the initialization step is replaced with `importSwingStore`. The host application should feed the importer with the export data and artifacts, by passing an object that has the same API as the SwingStore's exporter:
+When cloning an existing kernel, the host application first imports and commits the restored state using `importSwingStore`. The host application should feed the importer with the export data and artifacts, by passing an object that has the same API as the SwingStore's exporter:
 
 ```js
 import { importSwingStore } from '@agoric/swing-store';
@@ -159,11 +176,13 @@ const exporter = {
   getArtifact(name) { // return blob of artifact data },
 };
 const { hostStorage } = importSwingStore(exporter, dirPath);
-hostStorage.commit();
-// now the swingstore is fully populated
+// Update any hostStorage as needed
+await hostStorage.commit();
+await hostStorage.close();
+// now the populated swingstore can be re-opened using `openSwingStore``
 ```
 
-Once the new SwingStore is fully populated with the previously-exported data, the host application can use `makeSwingsetController()` to build a kernel that will start from the exported state.
+Once the new SwingStore is fully populated with the previously-exported data, the host application can update any host specific state before committing and closing the SwingStore. `importSwingStore` returns only the host facet of the SwingStore instance, as it is not suitable for immediate execution.
 
 ## Optional / Historical Data
 
@@ -177,18 +196,41 @@ As a result, for each active vat, the first-stage Export Data contains a record
 
 The `openSwingStore()` function has an option named `keepTranscripts` (which defaults to `true`), which causes the transcriptStore to retain the old transcript items. A second option named `keepSnapshots` (which defaults to `false`) causes the snapStore to retain the old heap snapshots. Opening the swingStore with a `false` option does not necessarily delete the old items immediately, but they'll probably get deleted the next time the kernel triggers a heap snapshot or transcript-span rollover. Validators who care about minimizing their disk usage will want to set both to `false`. In the future, we will arrange the SwingStore SQLite tables to provide easy `sqlite3` CLI commands that will delete the old data, so validators can also periodically use the CLI command to prune it.
 
-The `getArtifactNames()` API includes an option named `includeHistorical`. If `true`, all available historical artifacts will be included in the export (limited by what the `openSwingStore` options have deleted). If `false`, none will be included. Note that the "export data" is necessarily unaffected: if we *ever* want to validate this optional data, the hashes are mandatory. But the `getArtifactNames()` list will be smaller if you set `includeHistorical = false`. Also, re-exporting from a pruned copy will lack the old data, even if the re-export uses `includeHistorical = true`, because the second SwingStore cannot magically reconstruct the missing data.
+When exporting, the `makeSwingStoreExporter()` function takes an `artifactMode` option (in an options bag). This serves to both limit, and provide some minimal guarantees about, the set of artifacts that will be provided in the export. The defined values of `artifactMode` each build upon the previous one:
+
+* `operational`: include only the current transcript span and current snapshot for each vat: just the minimum set necessary for current operations
+* `replay`: add all transcript spans for the current incarnation
+* `archival`: add all available transcript spans, even for old incarnations
+* `debug`: add all available snapshots, giving you everything. The old snapshots are never necessary for normal operations, nor are they likely to be useful for extreme upgrade scenarios, but they might be useful for some unusual debugging operations or investigations
+
+For each mode, the export will fail if the data necessary for those artifacts is not available (e.g. it was previously pruned). For example, an export with `artifactMode: 'replay'` will fail unless every vat has all transcript entries for each one's current incarnation. The `archival` mode will fail to export unless every vat has *every* transcript entry, back to the very first incarnation.
+
+However the `debug` export mode will never fail: it merely dumps everything in the swingstore, without limits or completeness checks.
+
+Note that `artifactMode` does not affect the Export Data generated by the exporter (because if we *ever* want to validate this optional data, the hashes are mandatory). It only affects the names returned by `getArtifactNames()`: `operational` returns a subset of `replay`, which returns a subset of `archival`. And re-exporting from a previously-pruned copy under `archival` mode will fail, because the second SwingStore cannot magically reconstruct the missing data.
+
+Also note that when a vat is terminated, we delete all information about it, including transcript items and snapshots, both current and old. This will remove all the Export Data records, and well as the matching artifacts from `getArtifactNames`.
+
+When importing, the `importSwingStore()` function's options bag takes a property named `artifactMode`, with the same meanings as for export. Importing with the `operational` mode will ignore any artifacts other than those needed for current operations, and will fail unless all such artifacts were available. Importing with `replay` will ignore spans from old incarnations, but will fail unless all spans from current incarnations are present. Importing with `archival` will fail unless all spans from all incarnations are present. There is no `debug` option during import.
+
+While `importSwingStore()`'s options bag accepts the same options as `openSwingStore()`, since it returns only the host facet of a SwingStore, some of these options might not be meaningful, such as `keepTranscripts`.
+
+So, to avoid pruning current-incarnation historical transcript spans when exporting from one swingstore to another, you must set (or avoid overriding) the following options along the way:
 
-Note that when a vat is terminated, we delete all information about it, including transcript items and snapshots, both current and old. This will remove all the Export Data records, and well as the matching artifacts from `getArtifactNames`.
+* the original swingstore must not be opened with `{ keepTranscripts: false }`, otherwise the old spans will be pruned immediately
+* the export must use `makeSwingStoreExporter(dirpath, { artifactMode: 'replay'})`, otherwise the export will omit the old spans
+* the import must use `importSwingStore(exporter, dirPath, { artifactMode: 'replay'})`, otherwise the import will ignore the old spans
+  * subsequent `openSwingStore` calls must not use `keepTranscripts: false`, otherwise the new swingstore will prune historical spans as new ones are created (during `rolloverSpan`).
 
 ## Implementation Details
 
-SwingStore contains components to accomodate all the various kinds of state that the SwingSet kernel needs to store. This currently consists of three portions:
+SwingStore contains components to accommodate all the various kinds of state that the SwingSet kernel needs to store. This currently consists of four portions:
 
 * `kvStore`, a general-purpose string/string key-value table
 * `transcriptStore`: append-only vat deliveries, broken into "spans", delimited by heap snapshot events
 * `snapshotStore`: binary blobs containing JS engine heap state, to limit transcript replay depth
+* `bundleStore`: code bundles that can be imported with [@endo/import-bundle](https://www.npmjs.com/package/@endo/import-bundle)
 
-Currently, the SwingStore treats transcript spans and heap snapshots as export artifacts, with hashes recorded in the export data for validation (and to remember exactly which artifacts are necessary). The `kvStore` is copied one-to-one into the export data (i.e. we keep a full shadow copy in IAVL), because that is the fastest way to ensure the `kvStore` data is fully available and validated.
+Currently, the SwingStore treats transcript spans, heap snapshots, and bundles as export artifacts, with hashes recorded in the export data for validation (and to remember exactly which artifacts are necessary). The `kvStore` is copied one-to-one into the export data (i.e. we keep a full shadow copy in IAVL), because that is the fastest way to ensure the `kvStore` data is fully available and validated.
 
 If some day we implement an IAVL-like Merkle tree inside SwingStore, and use it to automatically generate a root hash for the `kvStore` at the end of each block, we will replace this (large) shadow copy with a single `kvStoreRootHash` entry, and add a new export artifact to contain the full contents of the kvStore. This reduce the size of the IAVL tree, as well as the rate of IAVL updates during block execution, at the cost of increased CPU and complexity within SwingStore.

package/docs/swingstore.md

@@ -0,0 +1,52 @@
+# SwingStore Data Model
+
+The "SwingStore" provides a database to hold SwingSet kernel state, with an API crafted to help both the kernel and the host application mutate, commit, export, and import this state.
+
+The state is broken up into several pieces, or "stores":
+
+* `bundleStore`: a string-keyed Bundle-value table, holding source bundles which can be evaluated by `importBundle` to create vats, or new Compartments within a vat
+* `transcriptStore`: records a linear sequence of deliveries and syscalls (with results), collectively known as "transcript entries", for each vat
+* `snapStore`: records one or more XS heap snapshots for each vat, to rebuild a worker more efficiently than replaying all transcript entries from the beginning
+* `kvStore`: a string-keyed string-valued table, which holds everything else. Currently, this holds each vat's c-list and vatstore data, as well as the kernel-wide object and promise tables, and run-queues.
+
+## Incarnations, Spans, Snapshots
+
+The kernel tracks the state of one or more vats. Each vat's execution is split into "incarnations", which are separated by a "vat upgrade" (a call to `E(vatAdminFacet).upgrade(newBundleCap, options)`, see https://github.com/Agoric/agoric-sdk/blob/master/packages/SwingSet/docs/vat-upgrade.md for details). Each incarnation gets a new worker, which erases the heap state and only retains durable vatstore data across the upgrade. Every active vat has a "current incarnation", and zero or more "historic incarnations". Only the current incarnation is instantiated.
+
+Within each incarnation, execution is broken into one or more "spans", with a "current span" and zero or more "historic spans". This breaks up the transcript into corresponding spans.
+
+Each historic span ends with a `save-snapshot` entry which records the creation and saving of an XS heap snapshot. The initial span starts with a `start-worker` entry, while all non-initial spans start with a `load-snapshot` entry. The final span of historic incarnations each end with a `shutdown-worker` entry.
+
+Each `save-snapshot` entry adds a new snapshot to the `snapStore`, so each vat has zero or more snapshots, of which the last one is called the "current" or "in-use" snapshot, and the earlier ones are called "historical snapshots".
+
+(note: the `deliveryNum` counter is scoped to the vat and does not reset at incarnation or span boundaries)
+
+## Artifacts
+
+The import/export process (using `makeSwingStoreExporter` and `importSwingStore`) defines some number of "artifacts" to contain much of the SwingStore data. Each bundle is a separate artifact, as is each heap snapshot. Each transcript span is a separate artifact (an aggregate of the individual transcript entries comprising that span).
+
+During export, the `getArtifactNames()` method provides a list of all available artifacts, while `getArtifact(name)` is used to retrieve the actual data. The import function processes each artifact separately.
+
+## Populated vs Pruned
+
+For normal operation, the kernel does not require historical incarnations, spans, or snapshots. It only needs the ability to reconstruct a worker for the current incarnation of each vat, which means loading the current snapshot (if any), and replaying the contents of the current transcript span.
+
+For this reason, the swingstore must always contain the current transcript span, and the current snapshot (if any), for every vat.
+
+However, to save space, historical spans/snapshots might be pruned, by deleting their contents from the database (but retaining the metadata, which includes a hash of the contents for later validation). Historical snapshots are pruned by default (unless `openSwingStore()` is given an options bag with `keepSnapshots: true`). Historical spans are not currently pruned (the `keepTranscripts` option defaults to `true`), but that may change.
+
+In addition, `importSwingStore()` can be used to create a SwingStore from data exported out of some other SwingStore. The export-then-import process might result in a pruned DB in one of three ways:
+
+* the import-time options might instruct the import process to ignore some of the available data
+* the export-time options might have done the same
+* the original DB was itself already pruned, so the data was not available in the first place
+
+In the future, a separate SwingStore API will exist to allow previously-pruned artifacts to be repopulated. Every artifact has a metadata record which *is* included in the export (in the `exportData` section, but separate from the kvStore shadow table entries, see [data-export.md](./data-export.md)), regardless of pruning modes, to ensure that this API can check the integrity of these repopulated artifacts. This reduces the reliance set and trust burden of the repopulation process (we can safely use untrusted artifact providers).
+
+When a snapshot is pruned, the `snapshots` SQL table row is modified, replacing its `compressedSnapshot` BLOB with a NULL. The other columns are left alone, especially the `hash` column, which retains the integrity-checking metadata to support a future repopulation.
+
+When a transcript span is pruned, the `transcriptSpans` row is left alone, but the collection of `transcriptItems` rows are deleted. Any span for which all the `transcriptItems` rows are present is said to be "populated", while any span that is missing one or more `transcriptItems` rows is said to be "pruned". (There is no good reason for a span to be only partially pruned, but until we compress historical spans into a single row, in some new table, there remains the possibility of partial pruning).
+
+During import, we create the metadata first (as the export-data is parsed), then later, we fill in the details as the artifacts are read.
+
+Bundles are never pruned, however during import, the `bundles` table will temporarily contain rows whose `bundle` BLOB is NULL.

package/package.json

@@ -1,9 +1,12 @@
 {
   "name": "@agoric/swing-store",
-  "version": "0.9.1",
+  "version": "0.9.2-calypso-dev-84eb287.0+84eb287",
   "description": "Persistent storage for SwingSet",
   "type": "module",
-  "main": "src/swingStore.js",
+  "main": "./src/index.js",
+  "exports": {
+    ".": "./src/index.js"
+  },
   "repository": "https://github.com/Agoric/agoric-sdk",
   "author": "Agoric",
   "license": "Apache-2.0",
@@ -14,23 +17,23 @@
     "test:xs": "exit 0",
     "lint-fix": "yarn lint:eslint --fix",
     "lint": "run-s --continue-on-error lint:*",
-    "lint:types": "tsc -p jsconfig.json",
+    "lint:types": "tsc",
     "lint:eslint": "eslint ."
   },
   "dependencies": {
-    "@agoric/assert": "^0.6.0",
-    "@agoric/internal": "^0.3.2",
-    "@endo/base64": "^0.2.31",
-    "@endo/bundle-source": "^2.5.1",
-    "@endo/check-bundle": "^0.2.18",
-    "@endo/nat": "^4.1.27",
-    "better-sqlite3": "^8.2.0"
+    "@agoric/assert": "0.6.1-calypso-dev-84eb287.0+84eb287",
+    "@agoric/internal": "0.3.3-calypso-dev-84eb287.0+84eb287",
+    "@endo/base64": "^1.0.5",
+    "@endo/bundle-source": "^3.2.3",
+    "@endo/check-bundle": "^1.0.7",
+    "@endo/nat": "^5.0.7",
+    "better-sqlite3": "^9.1.1"
   },
   "devDependencies": {
-    "@endo/init": "^0.5.56",
-    "@types/better-sqlite3": "^7.5.0",
-    "ava": "^5.2.0",
-    "c8": "^7.13.0",
+    "@endo/init": "^1.1.2",
+    "@types/better-sqlite3": "^7.6.9",
+    "ava": "^5.3.0",
+    "c8": "^9.1.0",
     "tmp": "^0.2.1"
   },
   "publishConfig": {
@@ -38,9 +41,15 @@
   },
   "ava": {
     "files": [
-      "test/**/test-*.js"
+      "test/**/*.test.*"
+    ],
+    "require": [
+      "@endo/init/debug.js"
     ],
     "timeout": "2m"
   },
-  "gitHead": "b66bf7c881ae462fcb617cd6e8d41b5c40ec58ef"
+  "typeCoverage": {
+    "atLeast": 76.31
+  },
+  "gitHead": "84eb287915c8255d68e845a9b6e10c790f0d54cb"
 }

package/src/assertComplete.js

@@ -0,0 +1,21 @@
+/**
+ * @param {import('./internal.js').SwingStoreInternal} internal
+ * @param {Omit<import('./internal.js').ArtifactMode, 'debug'>} checkMode
+ * @returns {void}
+ */
+export function assertComplete(internal, checkMode) {
+  // every bundle must be populated
+  internal.bundleStore.assertComplete(checkMode);
+
+  // every 'isCurrent' transcript span must have all items
+  // TODO: every vat with any data must have a isCurrent transcript
+  // span
+  internal.transcriptStore.assertComplete(checkMode);
+
+  // every 'inUse' snapshot must be populated
+  internal.snapStore.assertComplete(checkMode);
+
+  // TODO: every isCurrent span that starts with load-snapshot has a
+  // matching snapshot (counter-argument: swing-store should not know
+  // those details about transcript entries)
+}