@agoric/swingset-vat 0.32.3-dev-0b910ea.0.0b910ea → 0.32.3-dev-df9efa9.0.df9efa9

package/README.md

@@ -2,32 +2,28 @@
 
 [![License][license-image]][license-url]
 
-This repository contains another proof-of-concept Vat host, like
-PlaygroundVat. This one is modeled after KeyKOS "Domains": all Vats run on
-top of a "kernel" as if they were userspace processes in an operating system.
+This repository implements an architecture in which Vats run on top of a
+"kernel" as if they were userspace processes in an operating system.
 Each Vat gets access to a "syscall" object, through which it can send
 messages into the kernel. Vats receive message from the kernel via a
 "dispatch" function which they register at startup.
 
-Our goal is to experiment with different serialization/queueing mechanisms.
-One such mechanism is implemented so far, named "live slots", but we know
-this is insufficient to provide persistence across restarts.
+See [docs](./docs) for more information.
 
-More docs are in the works. For now, try:
+[SwingSet Runner](../swingset-runner) can be used to explore a SwingSet
+instance, as can the simple `vat` utility in this package.
+
+## `vat` CLI
 
 ```console
-$ npm install
-$ npm test
-$ bin/vat run demo/encouragementBot
+$ yarn install
+$ bin/vat run --config demo/encouragementBot/swingset.json
 ```
 
-This repository is still in early development: APIs and features are not
-expected to stabilize for a while.
-
-## REPL Shell
+### REPL Shell
 
 ```console
-$ bin/vat shell demo/encouragementBot
+$ bin/vat shell --config demo/encouragementBot/swingset.json
 vat>
 ```
 
@@ -36,13 +32,13 @@ arguments. All vats are loaded, and three additional commands are added to
 the environment:
 
 * `dump()`: display the kernel tables, including the run queue
-* `step()`: execute the next action on the run queue
-* `run()`: keep stepping until the run queue is empty
+* `await step()`: execute the next action on the run queue
+* `await run()`: keep stepping until the run queue is empty
 
-## Vat Basedirs
+### Vat Basedirs
 
-The main argument to `bin/vat` is a "basedir", which contains sources for all
-the Vats that should be loaded into the container.
+`vat` must be called with either a SwingSet configuration file (as above) or a
+"basedir" argument containing sources for all the Vats that should be loaded.
 
 Every file named `vat-*.js` (e.g. `vat-foo.js` and `vat-bar-none.js`) will
 create a new Vat (with names like `foo` and `bar-none`). Each directory named
@@ -53,7 +49,7 @@ In addition, a file named `bootstrap.js` must be present. This will contain
 the source for the "bootstrap Vat", which behaves like a regular Vat except:
 
 * At startup, its `bootstrap` method will be invoked, as `bootstrap(argv, vats)`
-* The `argv` value will be an array of strings, from the command line. So
+* The `argv` value will be an array of strings from the command line. So
   running `bin/vat BASEDIR -- x1 x2 x3` will set `argv = ['x1', 'x2', 'x3']`.
 * The `vats` value will be an object with keys named after the other Vats
   that were created, and values which are each a Presence for that Vat's root
@@ -69,18 +65,23 @@ anything else.
 ## Vat Sources
 
 Each Vat source file (like `vat-foo.js` or `vat-bar.js`) is treated as a
-starting point for the `rollup` tool, which converts the Vat's source tree
+starting point for a bundling process that converts the Vat's source tree
 into a single string (so it can be evaluated in a SES realm). This starting
 point can use `import` to reference shared local files. No
 non-local imports are allowed yet.
 
-The source file is expected to contain a single default export function named
-`setup`. This low-level function is invoked with a `syscall` object, and is
-expected to return a `dispatch` object.
+The source file is expected to export a `buildRootObject` function that returns
+a Vat "root object" managed by [Liveslots](../swingset-liveslots) with which
+other vats can interact. It is also possible to bypass the "Live Slots" layer
+when the vat options include `enableSetup: true` and are associated with a
+`managerType` that supports such bypass (currently limited to "local"), by
+exporting a default function named "setup" to be invoked with a `syscall` object
+and expected to return a `dispatch` object (see
+[BaseVatOptions](./src/types-external.js)).
 
 The "Live Slots" layer provides a function to build `dispatch` out of
 `syscall`, as well as a way to register the root object. This requires a few
-lines of boilerplate in the `setup()` function.
+lines of boilerplate in the `setup` function.
 
 ```js
 function buildRootObject(E) {
@@ -101,143 +102,79 @@ export default function setup(syscall, state, helpers) {
 }
 ```
 
-## Exposed (pass-by-presence) Objects
-
-The Live Slots system enables delivery of messages to remote "Callable
-Objects" objects, as long as those objects are of a particular form. All
-Callable Objects must follow these rules:
-
-* all enumerable properties must be functions
-* all properties, and the object itself, must be `harden()`ed
-
-The system can pass-by-copy "Data Objects" with similar rules:
+### Exposed (pass-by-presence) Objects
 
-* all enumerable properties must be non-functions
-* the object's prototype must be Array or Object (or null)
-* all properties, and the object itself, must be `harden()`ed
+The Live Slots system enables delivery of inbound messages to local
+"[Remotable](https://docs.agoric.com/guides/js-programming/far#pass-styles-and-harden)"
+objects.
 
-## Root Objects
+### Root Objects
 
-The "Root Object" is a callable object returned by `buildRootObject()`. It
+The "Root Object" is a Remotable object returned by `buildRootObject()`. It
 will be made available to the Bootstrap Vat.
 
-## Sending Messages with Presences
+### Sending Messages with Presences
 
-When a Callable Object is sent to another Vat, it arrives as a Presence. This
-is a special (empty) object that represents the Callable Object, and can be
-used to send it messages.  If you are running SwingSet under SES (the default),
-then the so-called "wavy dot syntax" ([proposed for future ECMAScript](https://github.com/Agoric/proposal-wavy-dot))
-can be used to invoke eventual send methods.
+Each Remotable object in the data of inbound messages arrives locally as a
+Presence. This is a special (empty) object that represents a non-local
+Remotable (i.e., a Remotable hosted by a remote Vat), and can be used to send it
+messages via the special `E()` wrapper.
 
 Suppose Vat "bob" defines a Root Object with a method named `bar`. The
-bootstrap receives this as `vats.bob`, and can send a message like this:
-
-```js
-function bootstrap(argv, vats) {
-  vats.bob~.bar('hello bob');
-}
-```
-
-The `~.` operator (pronounced "wavy dot") has the same left-to-right precedence as
-the `.` "dot" operator, so that example is equivalent to
-`HandledPromise.applyMethod(vats.bob, 'bar', ['hello bob'])`.
-
-If you are not running under SES and your Javascript environment does not yet support "wavy dot syntax" (i.e. running `"abc"~.[2]` results in a syntax error, not a Promise), then the special `E()` wrapper can be used to get a proxy from which methods can be invoked, which looks like:
+bootstrap vat receives this as `vats.bob`, and can send a message like so:
 
 ```js
 function bootstrap(argv, vats) {
-  E(vats.bob).bar('hello bob');
+  void E(vats.bob).bar('hello bob');
 }
 ```
 
-## Other uses for wavy dot syntax
-
-The main purpose of the wavy dot syntax (and E wrapper) is to provide an
-"eventual send" operator, in which the message is always delivered on some
-later turn of the event loop. This happens regardless of whether the target
-is local or in some other Vat:
-
-```js
-const t1 = {
-  foo() { console.log('foo called'); },
-};
-t1~.foo()
-console.log('wavy dot called');
-```
-
-will print:
-
-```
-wavy dot called
-foo called
-```
-
-This is equivalent to:
-
-```js
-HandledPromise.applyMethod(t1, 'foo', [])
-```
-
-or
-
-```js
-E(t1).foo()
-```
-
-## Return Values
+### Return Values
 
 Eventual-sends return a Promise for their eventual result:
 
 ```js
-const fooP = bob~.foo();
-fooP.then(resolution => console.log('foo said', resolution),
-          rejection => console.log('foo errored with', rejection));
+const fooP = E(bob).foo();
+fooP.then(
+  fulfillment => console.log('foo said', fulfillment),
+  rejection => console.log('foo errored with', rejection),
+);
 ```
 
-## Sending Messages to Promises
+### Sending Messages to Promises
 
-Wavy dot syntax also accepts Promises, just like `Promise.resolve`. The
-method will be invoked (on some future turn) on whatever the Promise resolves
-to.
+Eventual-sends can target a not-yet-fulfilled Promise, deferring invocation of
+the method until the Promise fulfills to an object.
 
-If wavy dot syntax is used on a Promise which rejects, the method is not invoked, and
-the return promise's `rejection` function is called instead:
+If the target Promise rejects, or fulfills to something other than a Remotable
+or Presence which supports the method, the method is not invoked and the
+eventual-send promise is rejected.
 
 ```js
 const badP = Promise.reject(Error());
-const p2 = badP~.foo();
+const p2 = E(badP).foo();
 p2.then(undefined, rej => console.log('rejected', rej));
 // prints 'rejected'
 ```
 
-If the Promise resolves to something which does not support the method,
-the method delivery will reject with a `TypeError`.
+### Promise Pipelining
 
-## Promise Pipelining
-
-In `fooP = bob~.foo()`, `fooP` represents the (eventual) return value of
-whatever `foo()` executes. If that return value is also a Callable Object, it
+In `fooP = E(bob).foo()`, `fooP` represents the (eventual) return value of
+whatever `foo()` executes. If that return value is also a Remotable, it
 is possible to queue messages to be delivered to that future target. The
-Promise returned by an eventual-send can be used by wavy dot syntax too, and
+Promise returned by an eventual-send can be used as a target itself, and
 the method invoked will be turned into a queued message that won't be
 delivered until the first promise resolves:
 
 ```js
-const db = databaseServer~.openDB();
-const row = db~.select(criteria)
-const success = row~.modify(newValue);
+const db = E(databaseServer).openDB();
+const row = E(db).select(criteria)
+const success = E(row).modify(newValue);
 success.then(res => console.log('row modified'));
 ```
 
 If you don't care about them, the intermediate values can be discarded:
 
-```js
-databaseServer~.openDB()~.select(criteria)~.modify(newValue)
-  .then(res => console.log('row modified'));
-```
-
-This can be done outside of SES in legacy Javascript environments with the `E()` wrapper:
-
 ```js
 E(E(E(databaseServer).openDB()).select(criteria)).modify(newValue)
   .then(res => console.log('row modified'));
@@ -246,24 +183,26 @@ E(E(E(databaseServer).openDB()).select(criteria)).modify(newValue)
 This sequence could be expressed with plain `then()` clauses, but by chaining
 them together without `then`, the kernel has enough information to
 speculatively deliver the later messages to the Vat in charge of answering
-the earlier messages. This avoids unnecessary roundtrips, by sending the
-later messages during "downtime" while the target Vat thinks about the answer
-to the first one.
+the earlier messages (_if the deciding Vat is configured with
+`enablePipelining: true`, which is not currently the case for any
+[Liveslots](../swingset-liveslots) Vats_). This avoids unnecessary roundtrips,
+by sending the later messages during "downtime" while the target Vat thinks
+about the answer to the first one.
 
 This drastic reduction in latency is significant when the Vats are far away
 from each other, and the inter-Vat communication delay is large. The SwingSet
 container does not yet provide complete facilities for off-host messaging,
 but once that is implemented, promise pipelining will make a big difference.
 
-## Presence Identity Comparison
+### Presence Identity Comparison
 
 Presences preserve identity as they move from one Vat to another:
 
-* Sending the same Callable Object multiple times will deliver the same
-  Presence on the receiving Vat
+* Sending the same Remotable multiple times will deliver the same Presence on
+  the receiving Vat
 * Sending a Presence back to its "home Vat" will arrive as the original
-  Callable Object
-* Sending a Callable Object to two different Vats will result in Presences
+  Remotable
+* Sending a Remotable to two different Vats will result in Presences
   that cannot be compared directly, because those two Vats can only
   communicate with messages. But if those two Vats both send those Presences
   to a third Vat, they will arrive as the same Presence object

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agoric/swingset-vat",
-  "version": "0.32.3-dev-0b910ea.0.0b910ea",
+  "version": "0.32.3-dev-df9efa9.0.df9efa9",
   "description": "Vat/Container Launcher",
   "type": "module",
   "main": "src/index.js",
@@ -22,7 +22,7 @@
     "lint:eslint": "yarn run -T eslint ."
   },
   "devDependencies": {
-    "@agoric/xsnap": "0.14.3-dev-0b910ea.0.0b910ea",
+    "@agoric/xsnap": "0.14.3-dev-df9efa9.0.df9efa9",
     "@types/better-sqlite3": "^7.6.13",
     "@types/microtime": "^2.1.0",
     "@types/tmp": "^0.2.0",
@@ -30,15 +30,15 @@
     "ava": "^5.3.0"
   },
   "dependencies": {
-    "@agoric/internal": "0.3.3-dev-0b910ea.0.0b910ea",
-    "@agoric/kmarshal": "0.1.1-dev-0b910ea.0.0b910ea",
-    "@agoric/store": "0.9.3-dev-0b910ea.0.0b910ea",
-    "@agoric/swing-store": "0.9.2-dev-0b910ea.0.0b910ea",
-    "@agoric/swingset-liveslots": "0.10.3-dev-0b910ea.0.0b910ea",
-    "@agoric/swingset-xsnap-supervisor": "0.10.3-dev-0b910ea.0.0b910ea",
-    "@agoric/time": "0.3.3-dev-0b910ea.0.0b910ea",
-    "@agoric/vat-data": "0.5.3-dev-0b910ea.0.0b910ea",
-    "@agoric/xsnap-lockdown": "0.14.1-dev-0b910ea.0.0b910ea",
+    "@agoric/internal": "0.3.3-dev-df9efa9.0.df9efa9",
+    "@agoric/kmarshal": "0.1.1-dev-df9efa9.0.df9efa9",
+    "@agoric/store": "0.9.3-dev-df9efa9.0.df9efa9",
+    "@agoric/swing-store": "0.9.2-dev-df9efa9.0.df9efa9",
+    "@agoric/swingset-liveslots": "0.10.3-dev-df9efa9.0.df9efa9",
+    "@agoric/swingset-xsnap-supervisor": "0.10.3-dev-df9efa9.0.df9efa9",
+    "@agoric/time": "0.3.3-dev-df9efa9.0.df9efa9",
+    "@agoric/vat-data": "0.5.3-dev-df9efa9.0.df9efa9",
+    "@agoric/xsnap-lockdown": "0.14.1-dev-df9efa9.0.df9efa9",
     "@endo/base64": "^1.0.12",
     "@endo/bundle-source": "^4.1.2",
     "@endo/captp": "^4.4.8",
@@ -105,5 +105,5 @@
   "typeCoverage": {
     "atLeast": 76.59
   },
-  "gitHead": "0b910eab0eaa75ad6e59126116767a3a6d8f4854"
+  "gitHead": "df9efa9cc6831e761d94e3cc74dfb7a9f886cdb1"
 }

package/src/controller/controller.js

@@ -401,6 +401,10 @@ export async function makeSwingsetController(
 
       slogDuration,
 
+      dumpLog(idx) {
+        return deepCopyJsonable(kernel.dumpLog(idx));
+      },
+
       dump() {
         return deepCopyJsonable(kernel.dump());
       },
@@ -617,7 +621,7 @@ export async function makeSwingsetController(
  * the two stages; this can happen, for example, in some debugging cases.
  *
  * @param {SwingSetConfig} config
- * @param {string[]} argv
+ * @param {string[]} bootstrapArgs
  * @param {{
  *   kernelStorage?: SwingStoreKernelStorage;
  *   env?: Record<string, string>;
@@ -634,7 +638,7 @@ export async function makeSwingsetController(
  */
 export async function buildVatController(
   config,
-  argv = [],
+  bootstrapArgs = [],
   runtimeOptions = {},
   deviceEndowments = {},
 ) {
@@ -670,7 +674,7 @@ export async function buildVatController(
   if (!swingsetIsInitialized(kernelStorage)) {
     bootstrapResult = await initializeSwingset(
       config,
-      argv,
+      bootstrapArgs,
       kernelStorage,
       initializationOptions,
       runtimeOptions,

package/src/kernel/kernel.js

@@ -191,8 +191,8 @@ export default function buildKernel(
 
   // This is a low-level output-only string logger used by old unit tests to
   // see whether vats made progress or not. The array it appends to is
-  // available as c.dump().log . New unit tests should instead use the
-  // 'result' value returned by c.queueToKref()
+  // available from controller methods `dumpLog` and `dump`. New unit tests
+  // should instead use the 'result' value returned by c.queueToKref().
   function testLog(...args) {
     const rendered = args.map(arg =>
       typeof arg === 'string' ? arg : JSON.stringify(arg, abbreviateReplacer),
@@ -2310,14 +2310,18 @@ export default function buildKernel(
       };
     },
 
+    // Note: Logs are not deterministic, since log() does not go through the
+    // syscall interface (and we replay transcripts one vat at a time, so any
+    // log() calls that were interleaved during their original execution will be
+    // sorted by vat in the replay). They are therefore not kept in the
+    // persistent state and remain ephemeral.
+    dumpLog(idx = 0) {
+      return ephemeral.log.slice(idx);
+    },
     dump() {
-      // note: dump().log is not deterministic, since log() does not go
-      // through the syscall interface (and we replay transcripts one vat at
-      // a time, so any log() calls that were interleaved during their
-      // original execution will be sorted by vat in the replay). Logs are
-      // not kept in the persistent state, only in ephemeral state.
       return { log: ephemeral.log, ...kernelKeeper.dump() };
     },
+
     kdebugEnable,
 
     addImport,

package/tools/vat.js

@@ -1,43 +1,98 @@
 #!/usr/bin/env node
 
+import { dirname } from 'node:path';
+import yargsParser from 'yargs-parser';
 import '@endo/init/pre-bundle-source.js';
 import '@endo/init';
 import process from 'process';
 import repl from 'repl';
 import util from 'util';
-import { loadBasedir, buildVatController } from '../src/index.js';
+import {
+  loadSwingsetConfigFile,
+  loadBasedir,
+  buildVatController,
+} from '../src/index.js';
 import { buildLoopbox } from '../src/devices/loopbox/loopbox.js';
 
+const USAGE = `
+Usage: ${process.argv[1]} \\
+  {run|shell} [{-c|--config} <config file>] [<base dir>] [--] [<bootstrap arg>]...
+`.trim();
+
+const showUsage = (message, exitCode = 64) => {
+  const log = message ? console.error : console.log;
+  if (message) log(`${message}\n`);
+  log(USAGE);
+  return exitCode;
+};
+
 function deepLog(item) {
   console.log(util.inspect(item, false, null, true));
 }
 
 async function main() {
-  const argv = process.argv.splice(2);
-  let withSES = true;
-  if (argv[0] === '--no-ses') {
-    withSES = false;
-    argv.shift();
-  }
+  const {
+    _: argv,
+    '--': extraArgs = [],
+    ...options
+  } = yargsParser(process.argv.slice(2), {
+    configuration: { 'populate--': true, 'unknown-options-as-args': true },
+    string: ['config'],
+    alias: { config: ['c'] },
+  });
   const command = argv.shift();
-  if (command !== 'run' && command !== 'shell') {
-    throw Error(`use 'vat run' or 'vat shell', not 'vat ${command}'`);
+  if (command === undefined) {
+    if (argv.includes('--help')) return showUsage();
+    return showUsage('Missing command');
+  } else if (command !== 'run' && command !== 'shell') {
+    return showUsage(
+      command === 'help' ? undefined : `Unrecognized command: ${command}`,
+    );
+  }
+  const configPath = options.config;
+  let basedir = /** @type {string | undefined} */ (argv.shift());
+  if (basedir === undefined) {
+    basedir = configPath ? dirname(configPath) : '.';
   }
-  const basedir =
-    argv[0] === '--' || argv[0] === undefined ? '.' : argv.shift();
-  const vatArgv = argv[0] === '--' ? argv.slice(1) : argv;
+  const vatArgv = /** @type {string[]} */ ([...argv, ...extraArgs]);
 
-  assert(basedir);
-  const config = await loadBasedir(basedir);
-  const { loopboxSrcPath, loopboxEndowments } = buildLoopbox('immediate');
-  config.devices = [['loopbox', loopboxSrcPath, loopboxEndowments]];
+  const config = await (options.config
+    ? loadSwingsetConfigFile(options.config)
+    : loadBasedir(basedir));
+  assert(config);
+  config.devices ||= {};
+  const deviceEndowments = /** @type {Record<string, unknown>} */ ({});
+  // @ts-expect-error TS2339: Property 'loopboxSenders' does not exist on type 'SwingSetConfig'.
+  const { loopboxSenders } = config;
+  // @ts-expect-error
+  delete config.loopboxSenders;
+  if (loopboxSenders) {
+    const { loopboxSrcPath, loopboxEndowments } = buildLoopbox('immediate');
+    config.devices.loopbox = {
+      sourceSpec: loopboxSrcPath,
+      parameters: { senders: loopboxSenders },
+    };
+    deviceEndowments.loopbox = { ...loopboxEndowments };
+  }
 
-  // @ts-expect-error expects string[], not boolean, in second position
-  const controller = await buildVatController(config, withSES, vatArgv);
+  const controller = await buildVatController(
+    config,
+    vatArgv,
+    {},
+    deviceEndowments,
+  );
   if (command === 'run') {
     await controller.run();
+    for (const entry of controller.dumpLog()) console.log(entry);
     console.log('= vat finished');
   } else if (command === 'shell') {
+    let logIndex = 0;
+    const dumpLog = () => {
+      for (const entry of controller.dumpLog(logIndex)) {
+        logIndex += 1;
+        console.log(entry);
+      }
+    };
     const r = repl.start({ prompt: 'vat> ', replMode: repl.REPL_MODE_STRICT });
     r.context.dump = () => {
       const d = controller.dump();
@@ -51,15 +106,27 @@ async function main() {
       deepLog(d.acceptanceQueue);
     };
     r.context.dump2 = () => controller.dump();
-    r.context.run = () => {
+    r.context.run = async () => {
       console.log('run!');
-      return controller.run();
+      const result = await controller.run();
+      dumpLog();
+      return result;
     };
-    r.context.step = () => {
+    r.context.step = async () => {
       console.log('step!');
-      return controller.step();
+      const result = await controller.step();
+      dumpLog();
+      return result;
     };
   }
 }
 
-void main();
+main().then(
+  exitCode => {
+    if (exitCode !== undefined) process.exitCode ||= exitCode;
+  },
+  err => {
+    console.error(err);
+    process.exitCode ||= 1;
+  },
+);