@xyo-network/xl1-cli 2.0.18 → 3.0.0

package/README.md

@@ -127,7 +127,8 @@ Create a `xyo.config.json` (or .js, .yaml) file in the working directory to defi
 |---|---|
 | `api` | JSON-RPC gateway. Serves block / balance / transaction queries and accepts transfers. |
 | `producer` | Block producer. Drives heartbeat blocks and packages mempool transactions. |
-| `finalizer` | Selects a chain head from competing producer candidates and writes finalizations. |
+| `finalizer` | Selects a chain head from competing producer candidates and writes finalizations. Optionally publishes finalized blocks + the chain-state head to S3 (`publishOnFinalize`). |
+| `indexer` | Publishes the chain index (step-summary families) to the S3 `index` bucket for HTTP/CDN reads. |
 | `mempool` | Holds pending transactions until the producer pulls them. |
 | `bridge` | EVM bridge — relays deposits / withdrawals between XL1 and an EVM chain. |
 | `rewardRedemption` | Claims block rewards for the producer's account. |
@@ -140,7 +141,7 @@ You make four independent choices when deploying:
 
 1. **Process layout** — mono (everything in one process), split (one actor per process), or grouped (custom).
 2. **Storage backing** — `memory` (single-process only, ephemeral), `lmdb` (single-host, persistent, multi-process safe via shared root), or `mongodb` (multi-host, persistent, multi-process safe).
-3. **Provider tier** — `simple` (in-process tier 1, requires the corresponding backing), or `jsonrpc` (tier 3 proxy to another `xl1` process). The CLI picks `jsonrpc` automatically when the global `remote.rpc` is set; otherwise it picks `simple` when the backing is available.
+3. **Provider tier** — `simple` (in-process tier 1, requires the corresponding backing), or `jsonrpc` (tier 3 proxy to another `xl1` process). The CLI picks `jsonrpc` automatically when a global `default-rpc` connection is configured; otherwise it picks `simple` when the backing is available.
 4. **Config layout** — single file shared across all processes (each `xl1 start <subset>` filters which actors it activates), one file per process, or pure environment variables.
 
 ### Recipes
@@ -172,13 +173,13 @@ The `api + mempool + finalizer` subset is the canary combo: the CLI builds a sin
 
 #### Single process, LMDB (single-host persistent)
 
-Persistent state on one machine. LMDB writes are multi-process safe via the LMDB lockfile, so you can later split actors across processes against the same `storage.root`.
+Persistent state on one machine. LMDB writes are multi-process safe via the LMDB lockfile, so you can later split actors across processes against the same `local-store` connection.
 
 `xyo.json`:
 ```json
 {
   "xl1": {
-    "storage": { "root": "/var/xl1/data" },
+    "connections": { "local-store": { "type": "lmdb", "root": "/var/xl1/data" } },
     "actors": [
       { "name": "api", "host": "localhost", "port": 8080 },
       { "name": "producer", "heartbeatInterval": 1000 },
@@ -201,8 +202,9 @@ Same as the LMDB variant but with a MongoDB-backed archivist. Use this when stat
 ```json
 {
   "xl1": {
-    "storage": {
-      "mongo": {
+    "connections": {
+      "chain-mongo": {
+        "type": "mongo",
         "connectionString": "mongodb://user:pass@db.internal:27017",
         "database": "xl1",
         "domain": "internal"
@@ -230,7 +232,7 @@ Each actor runs as its own OS process. Useful when you want to restart one role
 ```json
 {
   "xl1": {
-    "storage": { "root": "/var/xl1/data" },
+    "connections": { "local-store": { "type": "lmdb", "root": "/var/xl1/data" } },
     "actors": [
       { "name": "api", "host": "localhost", "port": 8080 },
       { "name": "producer", "heartbeatInterval": 1000 },
@@ -257,8 +259,9 @@ Same as above but with MongoDB as the shared store. Now each process can run on
 ```json
 {
   "xl1": {
-    "storage": {
-      "mongo": {
+    "connections": {
+      "chain-mongo": {
+        "type": "mongo",
         "connectionString": "mongodb://user:pass@db.internal:27017",
         "database": "xl1",
         "domain": "internal"
@@ -287,13 +290,15 @@ xl1 start finalizer mempool
 
 #### Stateless API client (read-only proxy to a remote chain)
 
-The API actor in `stateless: true` mode owns no local store. All viewer providers resolve to JsonRpc proxies pointed at `remote.rpc`. Run several stateless API instances behind a load balancer to scale read traffic without duplicating state.
+The API actor in `stateless: true` mode owns no local store. All viewer providers resolve to JsonRpc proxies pointed at the `default-rpc` connection. Run several stateless API instances behind a load balancer to scale read traffic without duplicating state.
 
 `xyo.json`:
 ```json
 {
   "xl1": {
-    "remote": { "rpc": { "protocol": "https", "url": "https://chain.example.com/rpc" } },
+    "connections": {
+      "default-rpc": { "type": "rpc", "protocol": "https", "url": "https://chain.example.com/rpc" }
+    },
     "actors": [
       { "name": "api", "host": "0.0.0.0", "port": 8080, "stateless": true }
     ]
@@ -310,7 +315,8 @@ xl1 start api
 Every config key has a `XL1_*` env var equivalent. Use double-underscore for nested keys and numeric indices for array entries — useful in container orchestration where you'd rather not mount a file.
 
 ```sh
-export XL1_STORAGE__ROOT=/var/xl1/data
+export XL1_CONNECTIONS__LOCAL_STORE__TYPE=lmdb
+export XL1_CONNECTIONS__LOCAL_STORE__ROOT=/var/xl1/data
 export XL1_ACTORS__0__NAME=api
 export XL1_ACTORS__0__HOST=localhost
 export XL1_ACTORS__0__PORT=8080
@@ -326,12 +332,12 @@ When per-process config needs to live in different config-management systems (Ku
 
 `xyo.api.json`:
 ```json
-{ "xl1": { "storage": { "root": "/var/xl1/data" }, "actors": [{ "name": "api", "host": "localhost", "port": 8080 }] } }
+{ "xl1": { "connections": { "local-store": { "type": "lmdb", "root": "/var/xl1/data" } }, "actors": [{ "name": "api", "host": "localhost", "port": 8080 }] } }
 ```
 
 `xyo.producer.json`:
 ```json
-{ "xl1": { "storage": { "root": "/var/xl1/data" }, "actors": [{ "name": "producer", "heartbeatInterval": 1000 }] } }
+{ "xl1": { "connections": { "local-store": { "type": "lmdb", "root": "/var/xl1/data" } }, "actors": [{ "name": "producer", "heartbeatInterval": 1000 }] } }
 ```
 
 ```sh
@@ -339,6 +345,72 @@ xl1 -c ./xyo.api.json      start api
 xl1 -c ./xyo.producer.json start producer
 ```
 
+### S3 object storage (finalized / chain-state / index publishing)
+
+Beyond the authority store (`lmdb` / `mongodb`), the chain can publish a static, HTTP-servable copy of itself to S3-compatible object storage (e.g. Cloudflare R2). This is what serves cheap, CDN-cached reads to [`@xyo-network/xl1-rest-block-viewer`](https://www.npmjs.com/package/@xyo-network/xl1-rest-block-viewer) and the wallet / explorer clients — no RPC server in the read path. It layers on top of any backing; it does not replace it.
+
+**One writer per bucket:**
+
+| Bucket | Written by | Holds |
+|---|---|---|
+| `finalized` | `finalizer` (when `publishOnFinalize` is set) | finalized blocks, payloads, `manifest.json` |
+| `chainState` | `finalizer` (when `publishOnFinalize` is set) | the mutable head pointer (`head.json`) |
+| `index` | `indexer` | the step-summary families (`blocks`, `balances`, `schemas`, `transfers`), `manifest.json`, `head.json` watermark |
+
+The `indexer` is gated at the finalized head the `finalizer` has published, so `publishOnFinalize` must be enabled on the finalizer for the index to advance.
+
+Configure the buckets as named `connections` (`s3-finalized`, `s3-chain-state`, `s3-index`). Credentials and `prefix` can be set per connection; each bucket sets its own `bucket` name and public `readUrl` — the CDN base URL anonymous reads go through, so they hit the CDN rather than the (per-operation billed) S3 API.
+
+`xyo.json`:
+```json
+{
+  "xl1": {
+    "connections": {
+      "chain-mongo": { "type": "mongo", "connectionString": "mongodb://user:pass@db.internal:27017", "database": "xl1", "domain": "internal" },
+      "s3-finalized": { "type": "s3", "accountId": "…", "accessKeyId": "…", "secretAccessKey": "…", "prefix": "", "bucket": "xl1-chain", "readUrl": "https://chain.xl1.example" },
+      "s3-chain-state": { "type": "s3", "bucket": "xl1-chain-state", "readUrl": "https://chain-state.xl1.example" },
+      "s3-index": { "type": "s3", "bucket": "xl1-index", "readUrl": "https://index.xl1.example" }
+    },
+    "actors": [
+      { "name": "producer", "heartbeatInterval": 1000 },
+      { "name": "finalizer", "publishOnFinalize": true, "publishContentEncoding": "br" },
+      { "name": "indexer", "publishSyncInterval": 5000, "publishContentEncoding": "br" }
+    ]
+  }
+}
+```
+
+Per-bucket credentials override the shared defaults — e.g. give `s3-chain-state` its own `accessKeyId` / `secretAccessKey` on that connection entry.
+
+**Actor settings that drive publishing:**
+
+| Setting | Actor(s) | Default | Meaning |
+|---|---|---|---|
+| `publishOnFinalize` | `finalizer` | `false` | Push newly finalized blocks (head pointer last) to the `finalized` + `chainState` buckets immediately after each finalization pass. |
+| `publishSyncInterval` | `indexer` | `5000` | Milliseconds between index sync passes; each pass builds every family's frames up to the published finalized head, then writes the index manifest + watermark. |
+| `publishContentEncoding` | `finalizer`, `indexer` | unset | Wire/storage encoding for published files: `br`, `gzip`, or `none`. |
+| `publishProgressInterval` | `finalizer`, `indexer` | `100` | Publishes between progress log lines during a sync pass (runner default when unset). |
+
+**Environment-variable form** — `__` nests, and each screaming-snake segment maps to camelCase (`CHAIN_STATE` → `chainState`, `READ_URL` → `readUrl`):
+
+```sh
+export XL1_CONNECTIONS__S3_FINALIZED__TYPE=s3
+export XL1_CONNECTIONS__S3_FINALIZED__ACCOUNT_ID=…
+export XL1_CONNECTIONS__S3_FINALIZED__ACCESS_KEY_ID=…
+export XL1_CONNECTIONS__S3_FINALIZED__SECRET_ACCESS_KEY=…
+export XL1_CONNECTIONS__S3_FINALIZED__BUCKET=xl1-chain
+export XL1_CONNECTIONS__S3_FINALIZED__READ_URL=https://chain.xl1.example
+export XL1_CONNECTIONS__S3_CHAIN_STATE__TYPE=s3
+export XL1_CONNECTIONS__S3_CHAIN_STATE__BUCKET=xl1-chain-state
+export XL1_CONNECTIONS__S3_CHAIN_STATE__READ_URL=https://chain-state.xl1.example
+export XL1_CONNECTIONS__S3_INDEX__TYPE=s3
+export XL1_CONNECTIONS__S3_INDEX__BUCKET=xl1-index
+export XL1_CONNECTIONS__S3_INDEX__READ_URL=https://index.xl1.example
+# actor publish settings are array entries, e.g. if finalizer/indexer are actors 1 and 2:
+export XL1_ACTORS__1__PUBLISH_ON_FINALIZE=true
+export XL1_ACTORS__2__PUBLISH_SYNC_INTERVAL=5000
+```
+
 ### Mnemonics & wallets
 
 The root `mnemonic` field (or `XL1_MNEMONIC`) defines the wallet from which every actor derives an account at `accountPath`. Per-actor `mnemonic` fields are rejected — use `accountPath` to give each actor a distinct derivation.