@_linked/server 2.15.0 → 2.16.1

package/CHANGELOG.md

@@ -1,5 +1,29 @@
 # @\_linked/server
 
+## 2.16.1
+
+### Patch Changes
+
+- [#119](https://github.com/linked-fw/server/pull/119) [`04d3708`](https://github.com/linked-fw/server/commit/04d3708c41a2b15a032b1b2cfa0bf9687a2d26bf) Thanks [@renovate](https://github.com/apps/renovate)! - Drop the unused direct `body-parser` dependency. The server has always parsed request bodies with Express 4's built-in `express.json({ limit: '50mb' })` (body-parser 1.x bundled inside Express), so nothing changes at runtime. The parser now lives in `createJsonBodyParser()` and its behaviour — `/call` arguments, the 50mb limit, 413/400 errors, multipart and urlencoded bodies left unparsed — is covered by tests.
+
+## 2.16.0
+
+### Minor Changes
+
+- [#122](https://github.com/linked-fw/server/pull/122) [`b6c2c9a`](https://github.com/linked-fw/server/commit/b6c2c9ac689c614966f97f39cf18c0bab7683e16) Thanks [@flyon](https://github.com/flyon)! - Load the backend providers of installed linked packages at boot, not only those of local workspace packages.
+
+  Every package in the app's dependency tree whose package.json declares `"linkedPackage": true` — registry install, localized checkout or workspace member — now has its `/backend` providers indexed at startup, each after the linked packages it depends on. Before, a registry-installed package's provider was indexed only on the first `/call/<pkg>/...`, so its `initRequest` / `supplyDataForRequest` hooks were silent until then and its `setupBeforeControllers` / `setupAfterControllers` never ran at all. With `@_linked/auth` installed from the registry, that meant a signed-in SSR page after a restart shipped an empty request-json and failed to hydrate.
+
+  The `"linkedPackage": true` flag is the only test, whatever the package's scope: an `@_linked/*` package without it (such as the `@_linked/localize` CLI tool) is not loaded, and a package in any other scope with it is. Legacy `"lincd": true` packages are still not loaded at boot.
+
+  Lazy indexing on `/call` stays as a fallback and shares one in-flight indexing per package with boot, so a provider is never constructed twice.
+
+  A provider whose hook throws no longer takes the server down with it. Boot lifecycle hooks (`setupBeforeControllers`, `setupBeforeCatchAllControllers`, `setupAfterControllers`) and per-request hooks (`initRequest`, `supplyDataForRequest`) now run per provider: an error is logged as `[linked] <package> <hook> failed: …` and the remaining providers still run. Before, a throwing boot hook aborted startup, and a synchronous throw in `supplyDataForRequest` failed every page request without naming the provider. This applies to local workspace providers as well as installed ones.
+
+  In dev, a package's backend is now loaded by the loader that owns the package: `vite.ssrLoadModule` for what the SSR runner bundles (`ssr.noExternal` — workspaces and `SSR_ENTRY_PACKAGES`), Node's `import()` for what it leaves external. Loading an external package's `/backend` through `ssrLoadModule` evaluated that package a second time, apart from the copy the app's own imports use.
+
+  Apps should be on `@_linked/sentry` 1.1.3 or later: before that, its `setupBeforeControllers` called `require` in ESM and threw `ReferenceError: require is not defined` whenever `SENTRY_DSN` and `SITE_ROOT` are set outside development. Now that that hook runs at boot, an older version logs the error instead of aborting startup, but Sentry stays inactive.
+
 ## 2.15.0
 
 ### Minor Changes

package/lib/esm/shapes/LinkedServer.d.ts

@@ -38,6 +38,7 @@ export declare class LinkedServer extends Shape {
     private analyse;
     private shapeProviders;
     private genericProviders;
+    private providerIndexing;
     private resizePathsMap;
     private api;
     /**
@@ -76,6 +77,15 @@ export declare class LinkedServer extends Shape {
      * @todo Check if default file store is set, if not, set it
      */
     initStores(): Promise<any[]>;
+    /**
+     * Run a lifecycle hook (setupBeforeControllers, setupBeforeCatchAllControllers,
+     * setupAfterControllers) on every generic provider, in indexing order.
+     *
+     * Each provider is isolated: a hook that throws (or rejects) is logged with
+     * the package and hook name and the next provider still runs, so one broken
+     * provider does not abort boot. That provider's routes or middleware from
+     * that hook may be missing, but the rest of the app starts.
+     */
     callGenericBackendProvidersMethod(method: string, ...args: any[]): Promise<void>;
     /**
      * Removes temporary nodes from memory if they've been seen twice
@@ -84,6 +94,24 @@ export declare class LinkedServer extends Shape {
      */
     initGarbageCollection(): void;
     initBackendProviders(): Promise<void>;
+    /**
+     * Index the backend providers of every linked package in the app's
+     * dependency tree, dependencies first. Workspace members (`localPackages`)
+     * are also imported whole, as before; for anything else only its `/backend`
+     * entry is loaded, and a package without one costs a failed resolve.
+     */
+    private indexLinkedDependencyProviders;
+    /**
+     * Index a package's backend providers once. Shared by boot and the lazy
+     * path in callBackendMethod / callShapeMethod. `onSourceChange` bypasses it
+     * on purpose: an HMR reload must re-index.
+     */
+    ensurePackageBackendProviders(pkg: string, warnIfNotFound?: boolean): Promise<void>;
+    /**
+     * Node's `import()`, as a method so a test can resolve a fixture app's
+     * node_modules (a test runner resolves bare specifiers from this file).
+     */
+    protected importModule(specifier: string): Promise<any>;
     /**
      * Filters local workspace packages to only those reachable from the app's
      * dependency tree. Mirrors the logic used by `buildAll` in lincd-cli.

package/lib/esm/shapes/LinkedServer.js

@@ -31,11 +31,48 @@ import { Shape } from '@_linked/core/shapes/Shape';
 import { LinkedErrorLogging } from '@_linked/core/utils/LinkedErrorLogging';
 import { FileStorePurposes, LinkedFileStorage, } from '@_linked/core/utils/LinkedFileStorage';
 import { getResizedImagesStore } from '../utils/resizedImagesPurpose.js';
+import { createJsonBodyParser } from '../utils/jsonBodyParser.js';
 import { LinkedStorage } from '@_linked/core/utils/LinkedStorage';
 import { autoLoadOntologyData } from '@_linked/core/utils/Package';
 import { getShapeClass, getSuperShapesClasses, } from '@_linked/core/utils/ShapeClass';
 import path from 'path';
 import { pathToFileURL } from 'node:url';
+import { discoverLinkedDependencies, isBundledBySsr, ssrNoExternalOf, } from '../utils/linkedDependencies.js';
+/**
+ * `instanceof ShapeProvider`, tolerating a second copy of server-utils.
+ *
+ * A registry-installed package's backend is loaded by Node (see
+ * isBundledBySsr), so in a dev runner that BUNDLES server-utils — the
+ * standalone case — its providers extend Node's copy of ShapeProvider, not the
+ * one this module imported. Fall back to the class name along the prototype
+ * chain. Production has one loader and one copy, so instanceof decides there.
+ */
+function isShapeProvider(provider) {
+    var _a;
+    if (provider instanceof ShapeProvider)
+        return true;
+    for (let proto = provider ? Object.getPrototypeOf(provider) : null; proto && proto !== Object.prototype; proto = Object.getPrototypeOf(proto)) {
+        if (((_a = proto.constructor) === null || _a === void 0 ? void 0 : _a.name) === 'ShapeProvider')
+            return true;
+    }
+    return false;
+}
+/**
+ * Call one provider's hook and contain its failure: a synchronous throw or a
+ * rejection is logged, naming the package and the hook, and swallowed. Used
+ * for hooks that run for every provider (boot lifecycle hooks and per-request
+ * hooks), where one provider's bug must not take down boot or every request
+ * without saying which provider it was.
+ */
+async function runProviderHook(pkg, hook, call) {
+    var _a;
+    try {
+        await call();
+    }
+    catch (err) {
+        console.error(chalk.red(`[linked] ${pkg} ${hook} failed: ${(_a = err === null || err === void 0 ? void 0 : err.message) !== null && _a !== void 0 ? _a : err}`), (err === null || err === void 0 ? void 0 : err.stack) ? `\n${err.stack}` : '');
+    }
+}
 import process from 'process';
 import * as React from 'react';
 import { renderToPipeableStream, renderToStaticMarkup } from 'react-dom/server';
@@ -125,6 +162,10 @@ let LinkedServer = class LinkedServer extends Shape {
         this.analyse = false;
         this.shapeProviders = new Map();
         this.genericProviders = new Map();
+        // One indexing per package: boot and the lazy /call path share it, so two
+        // concurrent first calls (or a call racing boot) never construct a
+        // package's providers twice.
+        this.providerIndexing = new Map();
         //from resizedFileName to full resized path (CDN or similar)
         this.resizePathsMap = new Map();
         if (config && typeof config !== 'string' && !('id' in config)) {
@@ -335,9 +376,8 @@ let LinkedServer = class LinkedServer extends Shape {
         //   origin: ['http://localhost:4001', 'https://www.mynd.site'],
         //   optionsSuccessStatus: 200, // some legacy browsers (IE11, various SmartTVs) choke on 204
         // };
-        // //accept JSON bodies
-        // this.server.use(bodyParser.json({limit: '50mb'}));
-        this.server.use(express.json({ limit: '50mb' }));
+        //accept JSON bodies
+        this.server.use(createJsonBodyParser());
         // this.server.use(cors(corsOptions));
         //
         // //compress server output with gzip,
@@ -499,15 +539,21 @@ let LinkedServer = class LinkedServer extends Shape {
             return store.init ? store.init() : Promise.resolve();
         }));
     }
-    // get generic backend providers for handle controller methods
+    /**
+     * Run a lifecycle hook (setupBeforeControllers, setupBeforeCatchAllControllers,
+     * setupAfterControllers) on every generic provider, in indexing order.
+     *
+     * Each provider is isolated: a hook that throws (or rejects) is logged with
+     * the package and hook name and the next provider still runs, so one broken
+     * provider does not abort boot. That provider's routes or middleware from
+     * that hook may be missing, but the rest of the app starts.
+     */
     async callGenericBackendProvidersMethod(method, ...args) {
-        for (let genericProvider of this.genericProviders.values()) {
-            if (!genericProvider || !genericProvider[method]) {
+        for (let [pkg, genericProvider] of this.genericProviders.entries()) {
+            if (!genericProvider || typeof genericProvider[method] !== 'function') {
                 continue;
             }
-            if (typeof genericProvider[method] == 'function') {
-                await Promise.resolve(genericProvider[method](...args));
-            }
+            await runProviderHook(pkg, method, () => genericProvider[method](...args));
         }
     }
     /**
@@ -535,16 +581,21 @@ let LinkedServer = class LinkedServer extends Shape {
         // // }, 1000 * 60 * 60); //every hour
     }
     async initBackendProviders() {
-        //get all local workspace lincd packages, then filter to only those
-        //in this app's dependency tree. This avoids loading npm-installed
-        //legacy packages that may use an old version of the core.
+        // Local workspace packages in this app's dependency tree. These are also
+        // imported whole at boot (indexLincdPackage), not only their /backend.
         const allLocalPackages = getLincdPackages();
         const localPackageMap = new Map(allLocalPackages.map((pkg) => [pkg.packageName, pkg]));
         const relevantPackages = this.filterPackagesByDependencyTree(localPackageMap, this.package);
-        for (let [pkgName, pkg] of relevantPackages) {
-            await this.indexPackageBackendProviders(pkgName);
-            await this.indexLincdPackage(pkgName);
-        }
+        // Every linked package in the dependency tree, installed or local, each
+        // after the linked packages it depends on. Registry installs and
+        // localized checkouts are not workspace members, and used to be indexed
+        // only lazily, on the first /call/<pkg>/... A provider can hook every
+        // request (initRequest, supplyDataForRequest, setupBeforeControllers
+        // routes), so those hooks stayed silent until some RPC to the package
+        // happened — an installed @_linked/auth did not put the session into a
+        // page's request data until then, and its setupBeforeControllers
+        // middleware never registered at all.
+        await this.indexLinkedDependencyProviders(relevantPackages);
         try {
             await fs
                 .readFile(path.join(process.cwd(), 'package.json'), 'utf-8')
@@ -569,6 +620,67 @@ let LinkedServer = class LinkedServer extends Shape {
             console.warn(err);
         }
     }
+    /**
+     * Index the backend providers of every linked package in the app's
+     * dependency tree, dependencies first. Workspace members (`localPackages`)
+     * are also imported whole, as before; for anything else only its `/backend`
+     * entry is loaded, and a package without one costs a failed resolve.
+     */
+    async indexLinkedDependencyProviders(localPackages) {
+        var _a;
+        const started = Date.now();
+        const installed = [];
+        const ordered = discoverLinkedDependencies(process.cwd(), this.package);
+        // A workspace member the walk cannot reach through node_modules still
+        // loads, after the rest, as it always did.
+        const remaining = new Set(localPackages.keys());
+        for (const { packageName } of ordered) {
+            const isLocal = remaining.delete(packageName);
+            await this.ensurePackageBackendProviders(packageName);
+            if (isLocal) {
+                await this.indexLincdPackage(packageName);
+            }
+            else if (this.genericProviders.get(packageName) ||
+                ((_a = this.shapeProviders.get(packageName)) === null || _a === void 0 ? void 0 : _a.length)) {
+                installed.push(packageName);
+            }
+        }
+        for (const packageName of remaining) {
+            await this.ensurePackageBackendProviders(packageName);
+            await this.indexLincdPackage(packageName);
+        }
+        if (installed.length > 0) {
+            console.log(chalk.gray(`[linked] indexed backend providers of ${ordered.length} linked package(s) in ${Date.now() - started}ms; installed ones with providers: ${installed.join(', ')}`));
+        }
+    }
+    /**
+     * Index a package's backend providers once. Shared by boot and the lazy
+     * path in callBackendMethod / callShapeMethod. `onSourceChange` bypasses it
+     * on purpose: an HMR reload must re-index.
+     */
+    async ensurePackageBackendProviders(pkg, warnIfNotFound = false) {
+        if (this.genericProviders.has(pkg) && this.shapeProviders.has(pkg)) {
+            return;
+        }
+        let indexing = this.providerIndexing.get(pkg);
+        if (!indexing) {
+            indexing = this.indexPackageBackendProviders(pkg, warnIfNotFound).then(() => undefined);
+            this.providerIndexing.set(pkg, indexing);
+        }
+        try {
+            await indexing;
+        }
+        finally {
+            this.providerIndexing.delete(pkg);
+        }
+    }
+    /**
+     * Node's `import()`, as a method so a test can resolve a fixture app's
+     * node_modules (a test runner resolves bare specifiers from this file).
+     */
+    importModule(specifier) {
+        return import(/* @vite-ignore */ specifier);
+    }
     /**
      * Filters local workspace packages to only those reachable from the app's
      * dependency tree. Mirrors the logic used by `buildAll` in lincd-cli.
@@ -897,7 +1009,7 @@ let LinkedServer = class LinkedServer extends Shape {
         }
     }
     async indexPackageBackendProviders(pkg, warnIfNotFound = false, backendIndexFilePath) {
-        var _a;
+        var _a, _b;
         if (!backendIndexFilePath) {
             backendIndexFilePath = `${pkg}/backend`;
         }
@@ -910,9 +1022,19 @@ let LinkedServer = class LinkedServer extends Shape {
         // @semantu/create-now/backend), which Node's ESM resolver can't resolve
         // from inside an installed package like @_linked/server.
         const vite = (_a = this.config.server) === null || _a === void 0 ? void 0 : _a.vite;
+        // In dev, load a package's backend with the loader that owns the package
+        // (see isBundledBySsr): Vite for the app itself and for what the SSR
+        // runner bundles (workspaces, SSR_ENTRY_PACKAGES), Node for what it
+        // leaves external (registry installs). ssrLoadModule of an EXTERNAL
+        // package's /backend evaluated that package a second time, apart from the
+        // copy the app's own imports reach.
+        const viteOwnsPackage = !!vite &&
+            typeof vite.ssrLoadModule === 'function' &&
+            (pkg === ((_b = this.package) === null || _b === void 0 ? void 0 : _b.name) ||
+                isBundledBySsr(pkg, ssrNoExternalOf(vite)));
         const loadModule = async (specifier) => {
             var _a, _b, _c;
-            if (vite && typeof vite.ssrLoadModule === 'function') {
+            if (viteOwnsPackage) {
                 // For the user app's own backend, resolve to ./src/backend.ts directly.
                 // Vite reads the app's package.json exports.development condition.
                 if (specifier === `${this.package.name}/backend`) {
@@ -952,7 +1074,8 @@ let LinkedServer = class LinkedServer extends Shape {
                 }
                 return await vite.ssrLoadModule(specifier);
             }
-            // No-vite path (e.g. production runtime).
+            // Node path: production runtime, or a package the dev SSR runner
+            // leaves external.
             //
             // The app's own backend cannot be imported by name here. A self-reference
             // resolves only from inside the package that declares it, and this code
@@ -967,7 +1090,7 @@ let LinkedServer = class LinkedServer extends Shape {
                 return await import(/* @vite-ignore */ pathToFileURL(target).href);
             }
             // Fall back to Node's resolver. The dynamic specifier is intentional.
-            return await import(/* @vite-ignore */ specifier);
+            return await this.importModule(specifier);
         };
         await loadModule(backendIndexFilePath)
             .then((backendProviderExports) => {
@@ -977,7 +1100,7 @@ let LinkedServer = class LinkedServer extends Shape {
                 //always send an instance of the express server
                 //TODO: do not create an instance, just save the class and instantiate it when needed
                 let provider = new providerClass(this.server, this);
-                if (provider instanceof ShapeProvider) {
+                if (isShapeProvider(provider)) {
                     shapeProviders.push(provider);
                     if (!Object.getOwnPropertyNames(provider).includes('shape')) {
                         console.warn(chalk.red(`${Object.getPrototypeOf(provider).constructor.name} in package ${pkg}
@@ -1192,7 +1315,7 @@ let LinkedServer = class LinkedServer extends Shape {
         var _a;
         //- index module providers if not done yet
         if (!this.shapeProviders.has(pkg)) {
-            await this.indexPackageBackendProviders(pkg, true);
+            await this.ensurePackageBackendProviders(pkg, true);
         }
         let packageShapeProviders = this.shapeProviders.get(pkg);
         let findProviderForShape = (nodeShapeId) => {
@@ -1200,7 +1323,7 @@ let LinkedServer = class LinkedServer extends Shape {
                 var _a, _b;
                 //access the static shape (which is a linkedShape() / Shape class)
                 //then access the SHACL NodeShape of that shape class, and its id
-                return (provider instanceof ShapeProvider &&
+                return (isShapeProvider(provider) &&
                     ((_b = (_a = provider.shape) === null || _a === void 0 ? void 0 : _a.shape) === null || _b === void 0 ? void 0 : _b.id) === nodeShapeId);
             });
         };
@@ -1649,32 +1772,29 @@ let LinkedServer = class LinkedServer extends Shape {
         res.json(jsonObject);
     }
     async initRequest(request, response) {
-        // initialise the request for all providers, do it synchroniously, one after the other
-        let p = Promise.resolve();
-        [...this.genericProviders.values()]
-            .filter(Boolean)
-            .forEach((backendProvider) => {
-            p = p
-                .then(() => {
-                return backendProvider.initRequest(request, response);
-            })
-                .catch((err) => {
-                console.warn(`Error during initRequest for provider ${Object.getPrototypeOf(backendProvider).constructor.name}: `, err);
-            });
-        });
-        return p;
+        // initialise the request for all providers, one after the other (order
+        // matters: later providers may read what earlier ones put on the request).
+        // A provider that throws is logged by package name and skipped.
+        for (const [pkg, backendProvider] of this.genericProviders.entries()) {
+            if (!backendProvider || typeof backendProvider.initRequest !== 'function') {
+                continue;
+            }
+            await runProviderHook(pkg, 'initRequest', () => backendProvider.initRequest(request, response));
+        }
     }
     async getRequestData(request, response) {
-        // Phase 1: providers return plain JSON data via supplyDataForRequest.
-        // requestLD is kept as empty string for now — SSR data seeding will be
-        // reworked in Phase 2/3 to inject query results instead of graph data.
+        // Providers return plain JSON data via supplyDataForRequest.
+        // requestLD is kept as empty string for now; SSR data seeding will
+        // inject query results instead of graph data.
+        // A provider that throws (synchronously or not) is logged by package
+        // name; the page still renders with the other providers' data.
         let requestData = {};
-        await Promise.all([...this.genericProviders.values()].map((backendProvider) => {
-            if (backendProvider) {
-                return Promise.resolve(backendProvider.supplyDataForRequest(request, response, requestData)).catch((err) => {
-                    console.warn(`Error requesting page-request data from ${Object.getPrototypeOf(backendProvider).constructor.name}: `, err);
-                });
+        await Promise.all([...this.genericProviders.entries()].map(([pkg, backendProvider]) => {
+            if (!backendProvider ||
+                typeof backendProvider.supplyDataForRequest !== 'function') {
+                return;
             }
+            return runProviderHook(pkg, 'supplyDataForRequest', () => backendProvider.supplyDataForRequest(request, response, requestData));
         }));
         let requestLD = '';
         let requestObject = JSONWriter.stringify(request['frontendData']);
@@ -1682,7 +1802,7 @@ let LinkedServer = class LinkedServer extends Shape {
     }
     async callBackendMethod(pkg, method, args, request, response) {
         if (!this.genericProviders.has(pkg)) {
-            await this.indexPackageBackendProviders(pkg, true);
+            await this.ensurePackageBackendProviders(pkg, true);
         }
         //retrieve the indexed provider class and create a new instance for this request
         let genericBackendProvider = this.genericProviders.get(pkg);