@_linked/server 3.5.0 → 3.6.1

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # @\_linked/server
 
+## 3.6.1
+
+### Patch Changes
+
+- [#200](https://github.com/linked-fw/server/pull/200) [`935d0ca`](https://github.com/linked-fw/server/commit/935d0caf014cae8c71ae3b40963305ac8d43792c) Thanks [@flyon](https://github.com/flyon)! - `POST /admin/sync-shapes` now answers which packages it reloaded (`reloaded`) and the node shapes registered afterwards (`shapes`), refuses (400) package names that are not valid npm names, `@_linked/*` packages, or neither the app nor one of its direct dependencies, and times out a hung sync (`LINKED_SHAPE_SYNC_TIMEOUT_MS`, default 120 s) or source reload (`LINKED_SOURCE_RELOAD_TIMEOUT_MS`, default 60 s) so the queue is released. The README documents the control channel and the side effects of a reload and a restart.
+
+## 3.6.0
+
+### Minor Changes
+
+- [#198](https://github.com/linked-fw/server/pull/198) [`8ff798a`](https://github.com/linked-fw/server/commit/8ff798a59104a1283c5a5ec9312a29e56f103c03) Thanks [@flyon](https://github.com/flyon)! - Add `POST /admin/sync-shapes` to the host control channel (`CN_APP_ADMIN_SECRET`). It re-syncs the app's registered shapes into its app data exactly as the boot does (core `syncShapes`, writing only shapes whose stored hash differs) and answers `{written, orphanScope}`; a failed sync answers 500 `{error}`. The optional body `{packages: [...]}` names packages whose source changed on disk: in dev their modules are invalidated in Vite and their backends reloaded first, so a shape file the host has just written is registered before the sync. Source reloads (`onSourceChange`) now run one at a time, and `LinkedServer.resyncShapes()` is available to code in the process.
+
 ## 3.5.0
 
 ### Minor Changes

package/lib/esm/shapes/LinkedServer.d.ts

@@ -15,6 +15,7 @@ import { Shape } from '@_linked/core/shapes/Shape';
  * needs; a plain function, an arrow function or a non-function value is not.
  */
 export declare function isProviderClass(value: unknown): boolean;
+import { type ShapeSyncResult } from '../utils/adminChannel.js';
 /**
  * Serve the local upload folder at `/uploads` (where `LocalFileStore` files are
  * reachable), with the headers for untrusted files.
@@ -104,21 +105,56 @@ export declare class LinkedServer extends Shape {
     /**
      * Materialize this server's registered shapes into the RDF store as SHACL
      * (core `syncShapes` → pure `sh:NodeShape` + property shapes), so a running
-     * app's app-data holds the shapes its instances validate/query against
-     * (plan-010 T1e.2). Whatever shapes the app package registers (published
-     * re-exports + generated) get written on boot (and re-run on shape-file HMR).
+     * app's app-data holds the shapes its instances validate/query against.
+     * Whatever shapes the app package registers (published re-exports +
+     * generated) are written on boot, and again when the host asks for it over
+     * the control channel ({@link resyncShapes}). A source reload (HMR) does not
+     * sync shapes by itself.
+     *
+     * Core writes only the shapes whose stored hash differs, so a sync of
+     * unchanged shapes writes nothing.
      *
      * Additive only: store shapes this process did not register are left alone
      * unless the operator sets `LINKED_SYNC_SHAPES_PRUNE_ORPHANS=true` — see
      * `bootOrphanScope` in utils/syncShapes for why.
      *
-     * Gated to servers whose DEFAULT dataset is a **concrete** materializable store
-     * (detected via `rawQuery`) — i.e. an app pointing at its app-data FusekiStore.
-     * CN's default is the context-routing `AppDataRouter` (no `rawQuery`), so CN is
-     * skipped here — it materializes its own pinned native shapes in its storage
-     * config.
+     * Opt-out via `linked.config` `syncShapesOnBoot` (default true); env
+     * LINKED_SYNC_SHAPES_ON_BOOT overrides for deployment. CN sets
+     * `syncShapesOnBoot: false` in its own linked.config — its default dataset is
+     * a context-router that can't materialize context-free, and it syncs its own
+     * pinned shapes separately (linked.backend.storage.js).
      */
     private materializeShapesIntoStore;
+    /** One shape sync into the default dataset; throws when it fails. */
+    private syncRegisteredShapes;
+    /** Tail of the {@link resyncShapes} queue: one re-sync at a time. */
+    private shapeSyncQueue;
+    /**
+     * Re-sync the registered shapes into app data, as the boot does, and report
+     * what was written. The host calls it (`POST /admin/sync-shapes`) after it
+     * changed the app's shape code, so the app — the only process that loads that
+     * code — writes its own shapes and their hashes.
+     *
+     * `packages` names the packages whose source changed: the app itself or its
+     * direct dependencies, never `@_linked/*` ({@link assertReloadablePackages}). In
+     * dev (Vite) their modules are invalidated and their backends reloaded first
+     * ({@link onSourceChange}), so a shape added since the boot is registered before
+     * the sync; without Vite (a built app) nothing is reloaded. The answer names the
+     * packages reloaded and the node shapes registered after it, so the host can check
+     * that the shape it wrote is there. Runs as the system, never as the caller of a
+     * request, one at a time; a run that takes longer than
+     * `LINKED_SHAPE_SYNC_TIMEOUT_MS` (default 120 s) fails and releases the queue.
+     */
+    resyncShapes({ packages }?: {
+        packages?: string[];
+    }): Promise<ShapeSyncResult>;
+    /** The packages a host may ask to reload: the app itself and its direct dependencies. */
+    private reloadablePackages;
+    /**
+     * Re-read one package's source from disk (dev only); see {@link resyncShapes}.
+     * Resolves whether it reloaded anything.
+     */
+    private reloadPackageFromDisk;
     /**
      * Boot the server and listen. Rejects — and makes the process exit non-zero —
      * when a provider throws a fatal error while booting (see utils/fatalError).
@@ -214,24 +250,30 @@ export declare class LinkedServer extends Shape {
         shapeProviders: any[];
     }>;
     /**
-     * Plan-011 phase 2 — HMR re-index entry point.
+     * Install the client-shell catch-all as the last ordinary layer on the
+     * router stack. See utils/spaFallback.ts for why ordering has to be
+     * re-asserted rather than assumed.
+     */
+    private installSpaFallback;
+    /** Tail of the {@link onSourceChange} queue: reloads never interleave. */
+    private sourceReloadQueue;
+    /**
+     * Reload one package's backend providers from source.
      *
      * Called by the CLI orchestrator's Vite watcher whenever a `.ts`/`.tsx`
-     * file inside a workspace package changes. Disposes the package's
-     * existing providers (so routes, listeners, timers don't accumulate),
-     * drops them from the registry, and re-runs indexPackageBackendProviders
-     * to pick up the freshly-imported module.
+     * file inside a workspace package changes, and by {@link resyncShapes}.
+     * Disposes the package's existing providers (so routes, listeners, timers
+     * don't accumulate), drops them from the registry, and re-runs
+     * indexPackageBackendProviders to pick up the freshly-imported module.
+     * Reloads run one at a time: each re-registers routes on the shared express
+     * stack. One that takes longer than `LINKED_SOURCE_RELOAD_TIMEOUT_MS` (default
+     * 60 s) rejects and releases the queue.
      *
      * Dispose calls have a 5 s soft timeout. A hanging dispose logs a
      * warning and is abandoned so HMR doesn't stall the whole dev loop.
      */
-    /**
-     * Install the client-shell catch-all as the last ordinary layer on the
-     * router stack. See utils/spaFallback.ts for why ordering has to be
-     * re-asserted rather than assumed.
-     */
-    private installSpaFallback;
     onSourceChange(pkg: string): Promise<void>;
+    private reloadPackageSource;
     processBackendMethodCall(request: any, response: any): Promise<any>;
     noCache(response: any): void;
     processAPICall(request: any, response: any, method: 'get' | 'post' | 'put' | 'delete'): Promise<void>;

package/lib/esm/shapes/LinkedServer.js

@@ -8,7 +8,6 @@ var __decorate = (this && this.__decorate) || function (decorators, target, key,
 var __metadata = (this && this.__metadata) || function (k, v) {
     if (typeof Reflect === "object" && typeof Reflect.metadata === "function") return Reflect.metadata(k, v);
 };
-import { timingSafeEqual } from 'crypto';
 import events from 'events';
 import express from 'express';
 import fetchCookie from 'fetch-cookie';
@@ -38,7 +37,7 @@ import { nativeImport } from '../utils/nativeImport.js';
 import { createJsonBodyParser } from '../utils/jsonBodyParser.js';
 import { LinkedStorage } from '@_linked/core/utils/LinkedStorage';
 import { autoLoadOntologyData } from '@_linked/core/utils/Package';
-import { getShapeClass, getSuperShapesClasses, } from '@_linked/core/utils/ShapeClass';
+import { getAllShapeClasses, getShapeClass, getSuperShapesClasses, } from '@_linked/core/utils/ShapeClass';
 import path from 'path';
 import { pathToFileURL } from 'node:url';
 import { isFatalError } from '../utils/fatalError.js';
@@ -140,6 +139,9 @@ import { resolveBootstrapEntry, resolveViteMainEntry, resolveViteServingMode, }
 import { resolveRouteAssets } from '../utils/routeAssets.js';
 import { resolveRouteComponent, ssrCssInlineUrls, ssrCssScope, } from '../utils/ssrCss.js';
 import { materializeShapes } from '../utils/syncShapes.js';
+import { AdminRequestError, assertReloadablePackages, registerAdminRoutes, } from '../utils/adminChannel.js';
+import { envMs, withTimeout } from '../utils/withTimeout.js';
+import { invalidateSourceUnder } from '../utils/reloadSource.js';
 import { installSpaFallback, repinSpaFallback, } from '../utils/spaFallback.js';
 import { LincdAPI } from './LincdAPI.js';
 import { getRpcExposureMode, isGenericRpcDisabled, logSafe, resolveCallable, setRpcExposureMode, warnUndeclaredCall, } from '../utils/rpcExposure.js';
@@ -292,6 +294,10 @@ let LinkedServer = class LinkedServer extends Shape {
          * error is logged and contained.
          */
         this.booting = false;
+        /** Tail of the {@link resyncShapes} queue: one re-sync at a time. */
+        this.shapeSyncQueue = Promise.resolve();
+        /** Tail of the {@link onSourceChange} queue: reloads never interleave. */
+        this.sourceReloadQueue = Promise.resolve();
         if (config && typeof config !== 'string' && !('id' in config)) {
             // The absence of `id` is the runtime discriminator for a LinkedConfig;
             // the widened `{id?: string}` member of the union keeps TS from narrowing
@@ -360,44 +366,119 @@ let LinkedServer = class LinkedServer extends Shape {
     /**
      * Materialize this server's registered shapes into the RDF store as SHACL
      * (core `syncShapes` → pure `sh:NodeShape` + property shapes), so a running
-     * app's app-data holds the shapes its instances validate/query against
-     * (plan-010 T1e.2). Whatever shapes the app package registers (published
-     * re-exports + generated) get written on boot (and re-run on shape-file HMR).
+     * app's app-data holds the shapes its instances validate/query against.
+     * Whatever shapes the app package registers (published re-exports +
+     * generated) are written on boot, and again when the host asks for it over
+     * the control channel ({@link resyncShapes}). A source reload (HMR) does not
+     * sync shapes by itself.
+     *
+     * Core writes only the shapes whose stored hash differs, so a sync of
+     * unchanged shapes writes nothing.
      *
      * Additive only: store shapes this process did not register are left alone
      * unless the operator sets `LINKED_SYNC_SHAPES_PRUNE_ORPHANS=true` — see
      * `bootOrphanScope` in utils/syncShapes for why.
      *
-     * Gated to servers whose DEFAULT dataset is a **concrete** materializable store
-     * (detected via `rawQuery`) — i.e. an app pointing at its app-data FusekiStore.
-     * CN's default is the context-routing `AppDataRouter` (no `rawQuery`), so CN is
-     * skipped here — it materializes its own pinned native shapes in its storage
-     * config.
+     * Opt-out via `linked.config` `syncShapesOnBoot` (default true); env
+     * LINKED_SYNC_SHAPES_ON_BOOT overrides for deployment. CN sets
+     * `syncShapesOnBoot: false` in its own linked.config — its default dataset is
+     * a context-router that can't materialize context-free, and it syncs its own
+     * pinned shapes separately (linked.backend.storage.js).
      */
     async materializeShapesIntoStore() {
+        try {
+            await this.syncRegisteredShapes();
+        }
+        catch (err) {
+            console.warn('[LinkedServer] shape materialization failed (non-fatal):', err);
+        }
+    }
+    /** One shape sync into the default dataset; throws when it fails. */
+    async syncRegisteredShapes() {
         var _a;
-        // Opt-out via `linked.config` `syncShapesOnBoot` (default true); env
-        // LINKED_SYNC_SHAPES_ON_BOOT overrides for deployment. CN sets
-        // `syncShapesOnBoot: false` in its own linked.config — its default dataset is
-        // a context-router that can't materialize context-free, and it syncs its own
-        // pinned shapes separately (linked.backend.storage.js).
         const configFlag = (_a = this.config) === null || _a === void 0 ? void 0 : _a.syncShapesOnBoot;
         const envFlag = process.env.LINKED_SYNC_SHAPES_ON_BOOT;
         if (configFlag === false || envFlag === 'false') {
-            return;
+            return { written: 0, skipped: 'disabled' };
         }
         const appData = LinkedStorage.getDefaultDataset();
         if (!appData)
-            return;
+            return { written: 0, skipped: 'no-default-dataset' };
+        const { syncShapes } = await import('@_linked/core');
+        // Explicit target: materialize EVERY registered shape into the app's own
+        // data store regardless of per-shape routing/pins.
+        const { count, orphanScope } = await materializeShapes(appData, syncShapes);
+        return { written: count, orphanScope };
+    }
+    /**
+     * Re-sync the registered shapes into app data, as the boot does, and report
+     * what was written. The host calls it (`POST /admin/sync-shapes`) after it
+     * changed the app's shape code, so the app — the only process that loads that
+     * code — writes its own shapes and their hashes.
+     *
+     * `packages` names the packages whose source changed: the app itself or its
+     * direct dependencies, never `@_linked/*` ({@link assertReloadablePackages}). In
+     * dev (Vite) their modules are invalidated and their backends reloaded first
+     * ({@link onSourceChange}), so a shape added since the boot is registered before
+     * the sync; without Vite (a built app) nothing is reloaded. The answer names the
+     * packages reloaded and the node shapes registered after it, so the host can check
+     * that the shape it wrote is there. Runs as the system, never as the caller of a
+     * request, one at a time; a run that takes longer than
+     * `LINKED_SHAPE_SYNC_TIMEOUT_MS` (default 120 s) fails and releases the queue.
+     */
+    async resyncShapes({ packages = [] } = {}) {
+        assertReloadablePackages(packages, this.reloadablePackages());
+        const timeoutMs = envMs('LINKED_SHAPE_SYNC_TIMEOUT_MS', 120000);
+        const run = this.shapeSyncQueue.then(() => withTimeout(runAsSystem(async () => {
+            const reloaded = [];
+            for (const pkg of packages) {
+                if (await this.reloadPackageFromDisk(pkg))
+                    reloaded.push(pkg);
+            }
+            const result = await this.syncRegisteredShapes();
+            return { ...result, reloaded, shapes: [...getAllShapeClasses().keys()].sort() };
+        }, 'shape sync requested by the host'), timeoutMs, `the shape sync did not finish within ${timeoutMs} ms`));
+        this.shapeSyncQueue = run.catch(() => undefined);
+        return run;
+    }
+    /** The packages a host may ask to reload: the app itself and its direct dependencies. */
+    reloadablePackages() {
+        var _a, _b;
+        const names = new Set();
+        if ((_a = this.package) === null || _a === void 0 ? void 0 : _a.name)
+            names.add(this.package.name);
         try {
-            const { syncShapes } = await import('@_linked/core');
-            // Explicit target: materialize EVERY registered shape into the app's own
-            // data store regardless of per-shape routing/pins.
-            await materializeShapes(appData, syncShapes);
+            const json = JSON.parse(fsNative.readFileSync(path.join(process.cwd(), 'package.json'), 'utf-8'));
+            if (json.name)
+                names.add(json.name);
+            for (const field of ['dependencies', 'devDependencies']) {
+                for (const name of Object.keys((_b = json[field]) !== null && _b !== void 0 ? _b : {}))
+                    names.add(name);
+            }
         }
-        catch (err) {
-            console.warn('[LinkedServer] shape materialization failed (non-fatal):', err);
+        catch (_c) { }
+        return names;
+    }
+    /**
+     * Re-read one package's source from disk (dev only); see {@link resyncShapes}.
+     * Resolves whether it reloaded anything.
+     */
+    async reloadPackageFromDisk(pkg) {
+        var _a, _b, _c;
+        const root = pkg === ((_a = this.package) === null || _a === void 0 ? void 0 : _a.name) ? process.cwd() : this.packageDirFor(pkg);
+        if (!root)
+            throw new AdminRequestError(`unknown package: ${pkg}`);
+        const graph = (_c = (_b = this.config.server) === null || _b === void 0 ? void 0 : _b.vite) === null || _c === void 0 ? void 0 : _c.moduleGraph;
+        if (!graph)
+            return false; // a built app: there is no source to reload
+        const roots = [root];
+        try {
+            roots.push(fsNative.realpathSync(root));
         }
+        catch (_d) { }
+        invalidateSourceUnder(graph, roots);
+        await this.onSourceChange(pkg);
+        return true;
     }
     // async serveData(req,res) {
     //   let nodeURI = `${req.protocol}://${req.get('host')}${req.originalUrl}`;
@@ -608,42 +689,12 @@ let LinkedServer = class LinkedServer extends Shape {
         this.server.head('__health', (_req, res) => {
             res.sendStatus(200);
         });
-        // CN control channel (plan-010 T1b.3). Present only when CN spawned this
-        // process with a shared secret (CN_APP_ADMIN_SECRET); a standalone app run
-        // without the secret leaves the channel closed (404). Auth is the
-        // x-cn-admin-secret header matching that env value.
-        const adminSecret = process.env.CN_APP_ADMIN_SECRET;
-        // Constant-time compare so the secret can't be recovered by timing the 401
-        // response. timingSafeEqual requires equal-length buffers, so guard length.
-        const secretMatches = (provided) => {
-            if (!adminSecret || !provided)
-                return false;
-            const a = Buffer.from(provided);
-            const b = Buffer.from(adminSecret);
-            return a.length === b.length && timingSafeEqual(a, b);
-        };
-        const requireAdmin = (req, res) => {
-            if (!adminSecret) {
-                res.sendStatus(404); // no control channel in this mode
-                return false;
-            }
-            if (!secretMatches(req.get('x-cn-admin-secret'))) {
-                res.sendStatus(401);
-                return false;
-            }
-            return true;
-        };
-        this.server.get('/admin/health', (req, res) => {
-            if (!requireAdmin(req, res))
-                return;
-            res.status(200).json({ status: 'ok' });
-        });
-        this.server.post('/admin/restart', (req, res) => {
-            if (!requireAdmin(req, res))
-                return;
-            res.sendStatus(202);
-            // CN's provisioner respawns on exit (see ChildProcessProvisioner).
-            setTimeout(() => process.exit(0), 50);
+        // The host's control channel (/admin/*), open only when the host started
+        // this process with CN_APP_ADMIN_SECRET; see utils/adminChannel.
+        registerAdminRoutes(this.server, process.env.CN_APP_ADMIN_SECRET, {
+            // The host's provisioner starts the app again after it exits.
+            exit: () => process.exit(0),
+            syncShapes: (request) => this.resyncShapes(request),
         });
         // after controller
         await this.callGenericBackendProvidersMethod('setupAfterControllers');
@@ -1377,18 +1428,6 @@ let LinkedServer = class LinkedServer extends Shape {
         this.shapeProviders.set(pkg, shapeProviders);
         return { backendProviderExports, shapeProviders };
     }
-    /**
-     * Plan-011 phase 2 — HMR re-index entry point.
-     *
-     * Called by the CLI orchestrator's Vite watcher whenever a `.ts`/`.tsx`
-     * file inside a workspace package changes. Disposes the package's
-     * existing providers (so routes, listeners, timers don't accumulate),
-     * drops them from the registry, and re-runs indexPackageBackendProviders
-     * to pick up the freshly-imported module.
-     *
-     * Dispose calls have a 5 s soft timeout. A hanging dispose logs a
-     * warning and is abandoned so HMR doesn't stall the whole dev loop.
-     */
     /**
      * Install the client-shell catch-all as the last ordinary layer on the
      * router stack. See utils/spaFallback.ts for why ordering has to be
@@ -1401,7 +1440,28 @@ let LinkedServer = class LinkedServer extends Shape {
             this.render(req, res);
         }));
     }
+    /**
+     * Reload one package's backend providers from source.
+     *
+     * Called by the CLI orchestrator's Vite watcher whenever a `.ts`/`.tsx`
+     * file inside a workspace package changes, and by {@link resyncShapes}.
+     * Disposes the package's existing providers (so routes, listeners, timers
+     * don't accumulate), drops them from the registry, and re-runs
+     * indexPackageBackendProviders to pick up the freshly-imported module.
+     * Reloads run one at a time: each re-registers routes on the shared express
+     * stack. One that takes longer than `LINKED_SOURCE_RELOAD_TIMEOUT_MS` (default
+     * 60 s) rejects and releases the queue.
+     *
+     * Dispose calls have a 5 s soft timeout. A hanging dispose logs a
+     * warning and is abandoned so HMR doesn't stall the whole dev loop.
+     */
     async onSourceChange(pkg) {
+        const timeoutMs = envMs('LINKED_SOURCE_RELOAD_TIMEOUT_MS', 60000);
+        const run = this.sourceReloadQueue.then(() => withTimeout(this.reloadPackageSource(pkg), timeoutMs, `the reload of ${pkg} did not finish within ${timeoutMs} ms`));
+        this.sourceReloadQueue = run.catch(() => undefined);
+        return run;
+    }
+    async reloadPackageSource(pkg) {
         var _a, _b, _c, _d, _e;
         const disposeWithTimeout = async (p, label) => {
             if (!(p === null || p === void 0 ? void 0 : p.dispose))