@underpostnet/cyberia 3.3.73 → 3.3.77

package/CLI-HELP.md

@@ -1,6 +1,6 @@
 ## Underpost CLI
 
-> underpost ci/cd cli v3.3.73
+> underpost ci/cd cli v3.3.77
 
 **Usage:** `underpost [options] [command]`
 
@@ -37,7 +37,7 @@
 | [`db`](#underpost-db) | Manages database operations with support for MariaDB and MongoDB, including import/export, multi-pod targeting, and Git integration. |
 | [`metadata`](#underpost-metadata) | Manages cluster metadata operations, including import and export. |
 | [`cron`](#underpost-cron) | Manages cron jobs: execute jobs directly or generate and apply K8s CronJob manifests. |
-| [`fs`](#underpost-fs) | Manages file storage, defaulting to file upload operations. |
+| [`fs`](#underpost-fs) | Uploads, pulls, or deletes Cloudinary assets and synchronizes the selected storage manifest. |
 | [`monitor`](#underpost-monitor) | Manages health server monitoring, the cluster observability stack, and host dashboards. |
 | [`event`](#underpost-event) | Dispatches operational events and provisions the monitoring rules that trigger them. |
 | [`ssh`](#underpost-ssh) | Manages cluster scoped SSH credentials and sessions for remote access to cluster nodes or services. Users are registered in engine-private/deploy/conf.users.json and keys are stored in engine-private/deploy/users/<user>. |
@@ -387,6 +387,7 @@ Manages Kubernetes clusters, defaulting to Kind cluster initialization.
 | `--service-host <host>` | Set custom host/IP for exposed MongoDB and Valkey clients. |
 | `--postgresql` | Initializes the cluster with a PostgreSQL statefulset. |
 | `--mongodb4` | Initializes the cluster with a MongoDB 4.4 service. |
+| `--mongo-express` | Deploys the mongo-express web client for inspecting the MongoDB statefulset databases and collections. Reuses the mongodb-secret credentials, and drops them when the deployed mongod runs without --auth. Combine with --node-port to expose it on the node network. |
 | `--valkey` | Initializes the cluster with a Valkey service. |
 | `--ipfs` | Initializes the cluster with an ipfs-cluster statefulset. |
 | `--contour` | Initializes the cluster with Project Contour base HTTPProxy and Envoy. |
@@ -747,7 +748,7 @@ Manages cron jobs: execute jobs directly or generate and apply K8s CronJob manif
 
 ### underpost fs
 
-Manages file storage, defaulting to file upload operations.
+Uploads, pulls, or deletes Cloudinary assets and synchronizes the selected storage manifest.
 
 **Usage:** `underpost fs [options] [path]`
 
@@ -755,20 +756,25 @@ Manages file storage, defaulting to file upload operations.
 
 | Argument | Description |
 | --- | --- |
-| `path` | The absolute or relative directory path for file operations. |
+| `path` | Selects one file or all files below a directory. Required unless --tracked is set. |
 
 #### Options
 
 | Option | Description |
 | --- | --- |
-| `--rm` | Removes the specified file. |
-| `--git` | Displays current Git changes related to file storage. |
-| `--recursive` | Uploads files recursively from the specified path. |
-| `--deploy-id <deploy-id>` | Specifies the deployment configuration ID for file operations. |
-| `--pull` | Downloads the specified file. |
+| `--rm` | Deletes selected remote assets and their manifest entries. |
+| `--git` | Restricts filesystem selection to Git-tracked files. Does not change Git state. |
+| `--recursive` | Compatibility option. Directories always select files recursively. |
+| `--tracked` | Selects only manifest entries. An optional path limits their scope. |
+| `--deploy-id <deploy-id>` | Selects the deployment configuration. |
+| `--pull` | Downloads selected manifest assets. Use --tracked to restore missing local files. |
 | `--omit-unzip` | With --pull, keeps the downloaded .zip file and skips extraction. |
-| `--force` | Forces the action, overriding any warnings or conflicts. |
-| `--storage-file-path <storage-file-path>` | Specifies a custom file storage path. |
+| `--force` | Overwrites remote uploads or local downloads. Does not change selection. |
+| `--storage-id <sub-id>` | Selects storage.<sub-id>.json. The default is storage.json. |
+| `--key <path>` | Selects one exact manifest key within the path scope and filters. |
+| `--from-key <path>` | Starts an inclusive range in manifest key order. |
+| `--to-key <path>` | Ends an inclusive range. The default start is the first manifest key. |
+| `--key-regex <pattern>` | Filters manifest keys by a JavaScript regular expression. |
 | `-h, --help` | display help for command |
 
 ---
@@ -853,7 +859,7 @@ Dispatches operational events and provisions the monitoring rules that trigger t
 | `--service-status` | Reports whether the underpost-event unit is active and enabled. |
 | `--list` | Lists the registered events with their resolved probe targets. |
 | `--port <port>` | Listening port for --serve and the generated unit (default: 39099). |
-| `--cooldown-ms <ms>` | Minimum interval between two dispatches of one event in --serve (default: 300000). |
+| `--cooldown-ms <ms>` | Minimum interval between two dispatches of one event in --serve (default: 10000). |
 | `--spoke <spoke-id>` | Spoke to remediate when dispatching wireguard-spoke-down by hand; a webhook takes it from the alert labels. |
 | `--nodes <node-names>` | Comma-separated node documents to act on — hubs for wireguard-server-down, peers for wireguard-spoke-down. Empty covers every registered hub, or every peer of this node hub. |
 | `--namespace <namespace>` | Kubernetes namespace holding the observability stack. Defaults to "default". |
@@ -1124,6 +1130,7 @@ Runs specified scripts using various runners.
 | `--deploy-id <deploy-id>` | Sets deploy id context for the runner execution. |
 | `--user <user>` | Sets user context for the runner execution. |
 | `--hosts <hosts>` | Comma-separated list of hosts for the runner execution. |
+| `--split <mb>` | Zip part size in MB for push-bundle, or "none" to upload a single zip. Defaults to 8. |
 | `--instance-id <instance-id>` | Sets instance id context for the runner execution. |
 | `--pid <process-id>` | Sets process id context for the runner execution. |
 | `--timeout-response <duration>` | Sets HTTPProxy per-route response timeout (e.g., "1s", "300ms", "infinity"). |
@@ -1166,7 +1173,7 @@ Runs the test tiers locally, inside deployment pods, or as a cluster Job with Al
 
 | Argument | Description |
 | --- | --- |
-| `suite` | A comma-separated list of suites or tiers to run. Suites: unit, infra, app, cyberia, contracts, all. Tiers: unit, infra:1-security, infra:2-network, infra:3-cluster, infra:4-ingress, infra:5-observability, app, cyberia, contracts. Defaults to every tier, in tier order. (default: "") |
+| `suite` | A comma-separated list of suites or tiers to run. Suites: unit, cyberia, infra, app, contracts, all. Tiers: unit, cyberia:unit, infra:1-security, infra:2-network, infra:3-cluster, infra:4-ingress, infra:5-observability, app, cyberia:app, contracts. Defaults to every tier, in tier order. (default: "") |
 
 #### Options
 

package/Dockerfile

@@ -6,7 +6,7 @@
 # Stage 1 — builder: clone the private deploy, build it, then scrub secrets.
 # ---------------------------------------------------------------------------
 FROM rockylinux/rockylinux:9 AS builder
-ARG UNDERPOST_VERSION=3.3.73
+ARG UNDERPOST_VERSION=3.3.77
 # Pin Node to an exact patch: dnf's nodejs:24 module lags (24.14.1) while
 # underpost's dependencies require >=24.15.0, so install the official binary.
 ARG NODE_VERSION=24.15.0

package/Dockerfile.dev

@@ -6,7 +6,7 @@
 # Stage 1 — builder: clone the private deploy, build it, then scrub secrets.
 # ---------------------------------------------------------------------------
 FROM rockylinux/rockylinux:9 AS builder
-ARG UNDERPOST_VERSION=3.3.73
+ARG UNDERPOST_VERSION=3.3.77
 # Pin Node to an exact patch: dnf's nodejs:24 module lags (24.14.1) while
 # underpost's dependencies require >=24.15.0, so install the official binary.
 ARG NODE_VERSION=24.15.0

package/Dockerfile.test

@@ -6,7 +6,7 @@
 # Stage 1 — builder: clone the private deploy, build it, then scrub secrets.
 # ---------------------------------------------------------------------------
 FROM rockylinux/rockylinux:9 AS builder
-ARG UNDERPOST_VERSION=3.3.73
+ARG UNDERPOST_VERSION=3.3.77
 # Pin Node to an exact patch: dnf's nodejs:24 module lags (24.14.1) while
 # underpost's dependencies require >=24.15.0, so install the official binary.
 ARG NODE_VERSION=24.15.0
@@ -72,7 +72,7 @@ RUN --mount=type=secret,id=github_token \
     # --- install deps + env, then replay build-safe itc provisioning ----------
     npm install; \
     node bin app load --env development --args deploy-id=dd-cyberia; \
-    node bin fs --pull --recursive --deploy-id dd-cyberia --storage-file-path './engine-private/conf/dd-cyberia/storage.engine-cyberia.json'; \
+    node bin fs --pull --tracked --deploy-id dd-cyberia --storage-id engine-cyberia; \
     ( cd /home/dd/engine/hardhat && npm install --include=dev ); \
     ( cd /home/dd/engine && underpost clone underpostnet/cyberia-server ); \
     \

package/README.md

@@ -20,7 +20,7 @@ _Stackable Rendering Layers as a Unified Tokenized Reality_
 <!-- CI/CD and health -->
 <div align="center">
 
-[![Node.js CI](https://img.shields.io/github/actions/workflow/status/underpostnet/engine-cyberia/docker-image.engine-cyberia.ci.yml?branch=main&label=Node.js%20CI)](https://github.com/underpostnet/engine-cyberia/actions/workflows/docker-image.engine-cyberia.ci.yml) [![Test](https://img.shields.io/github/actions/workflow/status/underpostnet/engine-cyberia/coverall.cyberia.ci.yml?branch=main&label=Test)](https://github.com/underpostnet/engine-cyberia/actions/workflows/coverall.cyberia.ci.yml) [![Coverage Status](https://coveralls.io/repos/github/underpostnet/engine-cyberia/badge.svg?branch=main)](https://coveralls.io/github/underpostnet/engine-cyberia?branch=main) [![Socket Badge](https://socket.dev/api/badge/npm/package/cyberia/3.3.73)](https://socket.dev/npm/package/cyberia/overview/3.3.73)
+[![Node.js CI](https://img.shields.io/github/actions/workflow/status/underpostnet/engine-cyberia/docker-image.engine-cyberia.ci.yml?branch=main&label=Node.js%20CI)](https://github.com/underpostnet/engine-cyberia/actions/workflows/docker-image.engine-cyberia.ci.yml) [![Test](https://img.shields.io/github/actions/workflow/status/underpostnet/engine-cyberia/coverall.cyberia.ci.yml?branch=main&label=Test)](https://github.com/underpostnet/engine-cyberia/actions/workflows/coverall.cyberia.ci.yml) [![Coverage Status](https://coveralls.io/repos/github/underpostnet/engine-cyberia/badge.svg?branch=main)](https://coveralls.io/github/underpostnet/engine-cyberia?branch=main) [![Socket Badge](https://socket.dev/api/badge/npm/package/cyberia/3.3.77)](https://socket.dev/npm/package/cyberia/overview/3.3.77)
 
 </div>
 
@@ -45,26 +45,26 @@ Go server and the WASM client with data — it does not run the simulation or th
 
 ```text
                          ┌───────────────────────────────────────────────┐
-                         │              UNDERPOST PLATFORM                 │
-                         │  toolchain · deploy · PWA build · static        │
-                         │  delivery · cluster/image/db · monitoring       │
-                         │                                                 │
-                         │   underpost CLI  ──▶  builds & deploys all      │
+                         │              UNDERPOST PLATFORM               │
+                         │  toolchain · deploy · PWA build · static      │
+                         │  delivery · cluster/image/db · monitoring     │
+                         │                                               │
+                         │   underpost CLI  ──▶  builds & deploys all    │
                          └───────────────────────────────────────────────┘
                                             │  hosts / delivers
                                             ▼
-   ╔═══════════════════════════════ CYBERIA MMO ══════════════════════════════════╗
+   ╔═══════════════════════════════ CYBERIA MMO ═══════════════════════════════════╗
    ║                                                                               ║
-   ║   ┌──────────────────┐   gRPC    ┌──────────────────┐   WebSocket            ║
-   ║   │  engine-cyberia  │ ────────▶ │  cyberia-server  │ ───────────┐           ║
-   ║   │     (Node.js)    │           │       (Go)       │            │           ║
-   ║   │                  │           │                  │            ▼           ║
-   ║   │ content          │           │ authoritative    │   ┌──────────────────┐ ║
-   ║   │ validation       │           │ simulation       │   │  cyberia-client  │ ║
-   ║   │ persistence      │           │ world tick       │   │    (C / WASM)    │ ║
-   ║   │ gRPC/REST data   │           │ AOI replication  │   │                  │ ║
-   ║   │ asset metadata   │           │                  │   │ rendering        │ ║
-   ║   └──────────────────┘           └──────────────────┘   │ input            │ ║
+   ║   ┌──────────────────┐   gRPC    ┌──────────────────┐   WebSocket             ║
+   ║   │  engine-cyberia  │ ────────▶ │  cyberia-server  │ ───────────┐            ║
+   ║   │     (Node.js)    │           │       (Go)       │            │            ║
+   ║   │                  │           │                  │            ▼            ║
+   ║   │ content          │           │ authoritative    │    ┌──────────────────┐ ║
+   ║   │ validation       │           │ simulation       │    │  cyberia-client  │ ║
+   ║   │ persistence      │           │ world tick       │    │    (C / WASM)    │ ║
+   ║   │ gRPC/REST data   │           │ AOI replication  │    │                  │ ║
+   ║   │ asset metadata   │           │                  │    │ rendering        │ ║
+   ║   └──────────────────┘           └──────────────────┘    │ input            │ ║
    ║            │                                             │ prediction       │ ║
    ║            └──────────────── REST (content/assets) ─────▶│ presentation     │ ║
    ║                                                          └──────────────────┘ ║
@@ -157,17 +157,17 @@ Three supervised processes run in parallel, each with its own monitor and reconn
 
 ```text
         ┌──────────────────────────────────────────────────────────────┐
-        │                    SUPERVISION (parallel)                      │
-        │                                                                │
-        │   engine-cyberia        cyberia-server        cyberia-client   │
-        │   ┌────────────┐        ┌────────────┐        ┌────────────┐   │
-        │   │  monitor   │        │  monitor   │        │  monitor   │   │
-        │   │ +reconnect │        │ +reconnect │        │ +reconnect │   │
-        │   └─────┬──────┘        └─────┬──────┘        └─────┬──────┘   │
-        │         │                     │                     │          │
-        │         └──────────┬──────────┴──────────┬──────────┘          │
-        │                    ▼                     ▼                     │
-        │            all three up & connected?                          │
+        │                    SUPERVISION (parallel)                    │
+        │                                                              │
+        │   engine-cyberia        cyberia-server        cyberia-client │
+        │   ┌────────────┐        ┌────────────┐        ┌────────────┐ │
+        │   │  monitor   │        │  monitor   │        │  monitor   │ │
+        │   │ +reconnect │        │ +reconnect │        │ +reconnect │ │
+        │   └─────┬──────┘        └─────┬──────┘        └─────┬──────┘ │
+        │         │                     │                     │        │
+        │         └──────────┬──────────┴──────────┬──────────┘        │
+        │                    ▼                     ▼                   │
+        │            all three up & connected?                         │
         └────────────────────────────┬─────────────────────────────────┘
                                       │
                 ┌─────────────────────┼─────────────────────┐

package/bin/build.js

@@ -8,13 +8,23 @@ import {
   getPathsSSR,
   syncPrivateConf,
   syncDeployIdSources,
+  syncDeployIdSourcesBack,
   buildTemplate,
   updatePrivateEngineTestRepo,
 } from '../src/server/runtime/conf.js';
 import { resolveDeployList } from '../src/server/network/router.js';
 import { loadDeployCatalog } from '../src/server/build/catalog.js';
-import { buildProductPackageJson, productPackageOptionsFactory } from '../src/server/build/package.js';
-import { COVERAGE_BUNDLE_DIRECTORY, bundleCoverageReport } from '../src/server/build/coverage.js';
+import {
+  buildProductPackageJson,
+  installDeployDependencies,
+  productPackageOptionsFactory,
+} from '../src/server/build/package.js';
+import {
+  COVERAGE_BUNDLE_DIRECTORY,
+  bundleCoverageReports,
+  coverageReportCommand,
+  deployCoverageReports,
+} from '../src/server/build/coverage.js';
 import { shellExec } from '../src/server/runtime/process.js';
 import Underpost from '../src/index.js';
 
@@ -33,13 +43,16 @@ const basePath = '../pwa-microservices-template';
  * SSR assets, manifests, and packaging declared by its conf into the template repo.
  * @param {string} confName - A concrete deploy id (e.g. `dd-prototype`).
  */
-const buildDeployTemplate = async (confName) => {
+const buildDeployTemplate = async (confName, { force = false } = {}) => {
   const repoName = Underpost.repo.engineRepoFactory(confName);
   const catalog = await loadDeployCatalog(confName);
 
   if (catalog.sourceMoves.length) {
     Underpost.repo.sparseCheckoutDirectory(`conf/${confName}`);
-    if (catalog.sourceMoves.some(([src]) => !fs.existsSync(src))) Underpost.repo.pullSourceRepo(repoName);
+    // Only a path neither tree holds needs the repo: one the engine holds is mirrored out.
+    // --force pulls regardless, to bring the repo up to date before the mirror.
+    if (force || catalog.sourceMoves.some(([src, dest]) => !fs.existsSync(src) && !fs.existsSync(dest)))
+      Underpost.repo.pullSourceRepo(repoName);
   }
   syncDeployIdSources(catalog.sourceMoves);
 
@@ -125,7 +138,7 @@ const buildDeployTemplate = async (confName) => {
 
   // The manifest a product publishes is its catalog's declaration, resolved the same way for
   // every product: only the files a product places in the template are its own business.
-  const packageOptions = productPackageOptionsFactory({ catalog, underpostVersion: Underpost.version });
+  const packageOptions = productPackageOptionsFactory({ catalog, engineDependencies: sourcePackageJson.dependencies });
 
   switch (confName) {
     case 'dd-cyberia': {
@@ -221,16 +234,49 @@ const buildDeployTemplate = async (confName) => {
 };
 
 /**
- * Carries this build stage's coverage HTML report into the assembled template, so every
- * container started from the published source serves the report instead of generating one.
+ * The coverage reports a deploy id's conf declares, as {@link deployCoverageReports} reads them.
+ * The conf is fetched when this checkout does not carry it yet: a CI runner builds from a bare
+ * engine tree, and a deploy without source moves never pulls its conf through the assembly.
+ * @param {string} deployId - A concrete deploy id.
+ * @returns {Array<{id: string, suite?: string, path?: string}>}
  */
-const bundleDeployCoverage = () => {
-  const { bundled, from, to } = bundleCoverageReport('.', basePath);
-  if (bundled) return void logger.info('Build coverage artifact', { from, to });
-  logger.warn('No coverage report to bundle; the deploy will publish the unavailable page', {
-    hint: 'node bin/build <deploy-id> --coverage',
-    to,
-  });
+const deployReports = (deployId) => {
+  Underpost.repo.sparseCheckoutDirectory(`conf/${deployId}`);
+  return deployCoverageReports(
+    JSON.parse(fs.readFileSync(`./engine-private/conf/${deployId}/conf.server.json`, 'utf8')),
+  );
+};
+
+/**
+ * Runs, once each, the test suites the deploy ids' coverage reports name. A product suite
+ * imports the modules its catalog pins, so those dependencies are installed into this
+ * checkout first — the same install a product deploy performs before it builds.
+ * @param {string[]} deployIds - Concrete deploy ids.
+ */
+const runDeployCoverage = async (deployIds) => {
+  const suites = new Set();
+  for (const deployId of deployIds) {
+    await installDeployDependencies(deployId);
+    for (const { suite } of deployReports(deployId)) if (suite) suites.add(suite);
+  }
+  for (const suite of suites) shellExec(coverageReportCommand({ suite }));
+};
+
+/**
+ * Carries this build stage's coverage HTML reports into the assembled template, so every
+ * container started from the published source serves them instead of generating one.
+ * @param {string} deployId - A concrete deploy id.
+ */
+const bundleDeployCoverage = (deployId) => {
+  for (const { id, bundled, from, to } of bundleCoverageReports(deployReports(deployId), basePath)) {
+    if (bundled) logger.info('Build coverage artifact', { id, from, to });
+    else
+      logger.warn('No coverage report to bundle; the deploy will publish the unavailable page', {
+        id,
+        hint: 'node bin/build <deploy-id> --coverage',
+        to,
+      });
+  }
 };
 
 const program = new Command();
@@ -247,7 +293,7 @@ program
   )
   .option(
     '--coverage',
-    `Run the test suite to refresh ./coverage before assembly, so the artifact carries a current ${COVERAGE_BUNDLE_DIRECTORY} report.`,
+    `Run the test suites the deploy ids' coverage reports name before assembly, so the artifact carries current ${COVERAGE_BUNDLE_DIRECTORY} reports.`,
     false,
   )
   .option(
@@ -255,6 +301,16 @@ program
     'After assembling each deploy id, publish it to its private test source repo (underpostnet/engine-test-<id>) for isolated test deploys.',
     false,
   )
+  .option(
+    '--force',
+    'Always pull each deploy id source repo (e.g. ../engine-prototype) before syncing its sourceMoves, instead of only when a path is missing from both trees.',
+    false,
+  )
+  .option(
+    '--sync-sources',
+    'Copy each deploy id catalog sourceMoves path from this tree back to its source repo (e.g. ../engine-prototype) and exit (no template assembly). The engine ignores these paths, so the change is reviewed and committed there.',
+    false,
+  )
   .action(async (confName, env, options) => {
     const deployList = resolveDeployList(confName);
     logger.info('Build repository', { confName, basePath, deployList, conf: !!options.conf });
@@ -267,18 +323,37 @@ program
       return;
     }
 
-    // Tests run here, in the build stage, and exactly once: the artifact carries the report so
-    // no container ever has to produce its own. Refreshing is opt-in because a template
-    // assembly is not otherwise a test run.
-    if (options.coverage) shellExec(`npm run test:coverage`);
+    if (options.syncSources) {
+      for (const deployId of deployList) {
+        const { sourceMoves } = await loadDeployCatalog(deployId);
+        if (!sourceMoves.length) {
+          logger.warn('No sourceMoves declared; nothing to sync back', { deployId });
+          continue;
+        }
+        const mirrored = syncDeployIdSourcesBack(sourceMoves);
+        for (const src of mirrored) logger.info('Build sync source', src);
+        if (mirrored.length !== sourceMoves.length)
+          logger.warn('Some declared sources are not in this tree and were left as they are in the source repo', {
+            deployId,
+            hint: `node bin/build ${deployId}`,
+            skipped: sourceMoves.length - mirrored.length,
+          });
+      }
+      return;
+    }
+
+    // Tests run here, in the build stage, and once per declared suite: the artifact carries
+    // each report so no container ever has to produce its own. Refreshing is opt-in because
+    // a template assembly is not otherwise a test run.
+    if (options.coverage) await runDeployCoverage(deployList);
 
     for (const deployId of deployList) {
       // Reconstruct the base template from 0 before each deploy id so neither a previous
       // build run nor the deploy id assembled before this one leaks into it. Opt out with
       // --no-template-rebuild.
       if (options.templateRebuild) await buildTemplate({ toPath: basePath });
-      await buildDeployTemplate(deployId);
-      bundleDeployCoverage();
+      await buildDeployTemplate(deployId, { force: options.force });
+      bundleDeployCoverage(deployId);
       // Publish the just-assembled tree to the deploy id's private test repo so a
       // pod started with `--private-test-repo` clones this work-in-progress source.
       if (options.updatePrivate) await updatePrivateEngineTestRepo(deployId);