@malloy-publisher/server 0.9.0 → 0.9.1

package/README.docker.md

@@ -80,12 +80,26 @@ Every application file is root-owned, so the server cannot modify one in place.
 
 - `/publisher/`, the server root, where it creates `publisher.db`. Because it owns this directory, the server can also rename aside and replace any of its top-level entries: `package.json`, `bun.lock`, and the `packages/` and `node_modules/` directories. Such a change lasts as long as the container. Some storage drivers, Docker's default overlayfs among them, refuse to rename a directory that comes from an image layer, but that is the driver's limit, not the image's.
 - `/publisher/publisher_data/`.
+- `/publisher/ducklake_data/`, when a DuckLake storage destination's `bucketUrl` points there.
 - `/home/bun/.duckdb/extensions/`, for an extension the image did not bake.
 
-What you mount has to be writable by uid 1000 too:
+What you mount has to be writable by uid 1000 too. Per kind of mount:
 
-- **A new named volume on `/publisher/publisher_data`** works as-is. Docker seeds an empty named volume from the image's directory, ownership included, and that is the only writable mount point the image prepares.
-- **A new named volume anywhere else** starts owned by root, because the image has no directory there to copy ownership from. That includes a DuckLake storage destination whose `bucketUrl` is a local path. Prepare it yourself: build a derived image with `RUN mkdir -p /path && chown 1000:1000 /path` (new named volumes there are then seeded correctly), chown the volume once with `--user 0`, or bind-mount a host directory owned by uid 1000. DuckDB reports the unprepared case as `No such file or directory` (for example `Failed to create directory "/data/lake/main/daily_orders"`), not as `EACCES`. The one-time chown names the mount path twice, as the mount target and as chown's argument:
+| Mount | The server | What to do |
+| --- | --- | --- |
+| A new named volume on `/publisher/publisher_data` or `/publisher/ducklake_data` | writes | Nothing: Docker seeds it from the image, owned by uid 1000. |
+| A new named volume anywhere else | writes | `docker run --rm --user 0 --entrypoint chown -v <volume>:/path ms2data/malloy-publisher -R 1000:1000 /path`, once. |
+| A volume an older, root-run image wrote to | writes | The same `chown`, once, before starting the new image. With Compose, `docker compose run --rm --no-deps --user 0 --entrypoint chown publisher -R 1000:1000 /publisher/publisher_data`. |
+| A bind of a host directory the server writes | writes | `sudo chown -R 1000:1000 <dir>`, or keep your ownership and add an ACL, which needs no `sudo` on a directory you own: `setfacl -R -m u:1000:rwX -m d:u:1000:rwX <dir>`. |
+| A bind of a host directory the server only reads (a package `location`, a directory of zips) | reads | Readable by others, `chmod -R o+rX <dir>`, or group-readable with `sudo chgrp -R 1000 <dir> && chmod -R g+rX <dir>`. Mount it `:ro`. |
+| A single bound file (the config, a key file) | reads | Readable by others or by gid 1000; a `0600` file is not. See the key-file recipes below. Mount it `:ro`. |
+| A Kubernetes PersistentVolume | writes | `fsGroup: 1000` and `fsGroupChangePolicy: OnRootMismatch` in the pod's `securityContext`. |
+| A Kubernetes Secret or projected token | reads | `fsGroup: 1000` makes a `0600`-mounted file readable by the group. |
+
+Before starting the server, the probe at the end of this section confirms a mount is usable as uid 1000. The detail behind each row:
+
+- **A new named volume on `/publisher/publisher_data` or `/publisher/ducklake_data`** works as-is. Docker seeds an empty named volume from the image's directory, ownership included, and those are the two writable mount points the image prepares: `publisher_data` for packages and per-package sandboxes, `ducklake_data` for a DuckLake storage destination whose `bucketUrl` is a local path.
+- **A new named volume anywhere else** starts owned by root, because the image has no directory there to copy ownership from. Prepare it yourself: build a derived image with `RUN mkdir -p /path && chown 1000:1000 /path` (new named volumes there are then seeded correctly), chown the volume once with `--user 0`, or bind-mount a host directory owned by uid 1000. DuckDB reports the unprepared case as `No such file or directory` (for example `Failed to create directory "/data/lake/main/daily_orders"`), not as `EACCES`. The one-time chown names the mount path twice, as the mount target and as chown's argument:
 
   ```bash
   docker run --rm --user 0 --entrypoint chown \
@@ -110,8 +124,35 @@ What you mount has to be writable by uid 1000 too:
 
   `docker volume ls` shows the volume's full name if you would rather use the `docker run` form.
 
-- **A bind mount** keeps the host directory's ownership. On Linux, `chown -R 1000:1000` the host directory. Docker Desktop on macOS and Windows maps ownership for you. Running the container as some other uid to match the host is not a substitute: `/home/bun` is private to uid 1000, so that uid cannot read the baked DuckDB extensions.
-- **A read-only mount**, such as the config file, only needs to be readable.
+- **A bind mount** keeps the host directory's ownership. On Linux, `sudo chown -R 1000:1000` the host directory, or keep it yours and grant uid 1000 through a POSIX ACL, which also covers files created later and, unlike `chown`, needs no `sudo` on a directory you own:
+
+  ```bash
+  setfacl -R -m u:1000:rwX -m d:u:1000:rwX <dir>
+  ```
+
+  Docker Desktop on macOS and Windows maps ownership for you. Running the container as some other uid to match the host is not a substitute: `/home/bun` is private to uid 1000, so that uid cannot read the baked DuckDB extensions.
+- **A Kubernetes PersistentVolume** is not seeded from the image, so even a new one on `/publisher/publisher_data` starts owned by root (an `emptyDir` is world-writable and needs nothing). Set `fsGroup: 1000` in the pod's `securityContext`, which makes the volume group-writable by gid 1000, and `fsGroupChangePolicy: OnRootMismatch` so the kubelet does not re-chown a large volume on every start. `fsGroup` is also what makes a mounted Secret key file, or a projected service-account token, readable by uid 1000 where the platform mounts them `0600`.
+- **A read-only mount** only needs to be readable by uid 1000. That covers the config file, a package `location`, a directory of package zips (a `.zip` location is extracted into `publisher_data/`, never beside the archive), and the key file `GOOGLE_APPLICATION_CREDENTIALS` names. A key file bound from a host path that does not exist arrives as a directory, and the server says so. The usual trap is the key file itself: a `gcloud` application-default credentials file is `0600` and owned by you, so uid 1000 cannot read it through a bind on Linux, unless your own uid is 1000 (the first user on most Ubuntu installs), in which case nothing is needed. Otherwise bind a copy it can read, or grant the group. Both need root, because only root can hand a file to another uid or to a group you are not in:
+
+  ```bash
+  sudo install -o 1000 -g 1000 -m 0400 ~/.config/gcloud/application_default_credentials.json ./secrets/key.json
+  # or, keeping the original in place:
+  sudo chgrp 1000 key.json && chmod 0640 key.json
+  ```
+
+  For a directory you own, `chmod -R o+rX <dir>` needs no root; the `chgrp 1000` / `g+rX` equivalent does.
+- **Ownership is right and the server still reports `EACCES`:** on SELinux hosts (Fedora, RHEL) a bind mount needs the `:z` (shared) or `:Z` (private) option, or every access is refused whatever the owner. Under rootless Docker, Podman, or `userns-remap`, uid 1000 in the container is a subordinate uid on the host, so a host `chown 1000` names the wrong owner; use `podman unshare chown -R 1000:1000 <dir>`, or the remapped uid.
+
+To check a mount before starting the server, run the probe as the image's user:
+
+```bash
+docker run --rm --entrypoint sh -v <mount> ms2data/malloy-publisher \
+  -c 'id; touch <path>/.probe && rm <path>/.probe && echo writable'
+```
+
+For a read-only mount, the same with `cat <file> >/dev/null && echo readable`.
+
+When the server cannot make a write it needs, the response names the errno (`EACCES: permission denied, mkdir '…'`) rather than answering a bare `Internal server error.`. A package that cannot be mounted at boot, or added at runtime, is listed under `loadErrors` in `GET /api/v0/status` with that message, until it is added successfully or deleted. If the server cannot start at all, for example on a read-only root filesystem or with a config file it cannot read, `/api/v0/status` stays `initializing` and `initError` says why.
 
 If you cannot change the ownership yet, `--user 0` runs the server as root, as earlier images did. The image sets `HOME=/home/bun`, so a root run still finds the baked DuckDB extensions.
 

package/dist/app/api-doc.yaml

@@ -2876,11 +2876,27 @@ components:
             environments list as intended. Absent once any environment exists, including one created
             at runtime through the API, and absent when a config was found but
             lists no environments.
+        initError:
+          type: string
+          readOnly: true
+          description:
+            Present only when initialization failed, and says why. Such a
+            server stays up but never becomes ready, so operationalState stays
+            initializing; without this field that reads as a server still
+            starting. Typical causes are a config file the server cannot read
+            or parse, and a server root it cannot write (where it creates
+            publisher.db), such as a read-only root filesystem. A package or
+            environment that fails to load is reported under loadErrors
+            instead. Fix the cause and restart.
         loadErrors:
           type: array
           readOnly: true
           description:
-            Configured packages and environments that did not load, and why. A
+            Configured packages and environments that did not load, and why,
+            including a package add through the API that failed on the
+            server's side, carrying the same message the add was answered
+            with (an entry cleared by a later successful add of that package,
+            or by deleting it). A
             load failure is not fatal to the server, so operationalState stays
             serving and the affected packages are absent from environments,
             unless the entry carries stale = true, in which case the package is

package/dist/package_load_worker.mjs

@@ -13058,9 +13058,54 @@ function logInternalFailure(summary, error, level = "error") {
     }
   });
 }
-function internalErrorToHttpError(error) {
+function filesystemAccessFailure(error) {
+  let current = error;
+  for (let depth = 0;depth < 8 && current instanceof Error; depth++) {
+    const errno = current;
+    if (typeof errno.code === "string" && errno.code in FILESYSTEM_ACCESS_DESCRIPTIONS && typeof errno.syscall === "string") {
+      return errno;
+    }
+    current = errno.cause;
+  }
+  return;
+}
+function describeFilesystemAccessFailure(errno) {
+  const code = errno.code ?? "";
+  const target = errno.path ? ` '${errno.path}'` : "";
+  return `${code}: ${FILESYSTEM_ACCESS_DESCRIPTIONS[code] ?? code}, ${errno.syscall}${target}`;
+}
+function messageWithFilesystemCause(error) {
+  const message = error instanceof Error ? error.message : String(error);
+  const access = filesystemAccessFailure(error);
+  if (!access || message.includes(access.code ?? ""))
+    return message;
+  return `${message}: ${describeFilesystemAccessFailure(access)}`;
+}
+function errnoWireFields(error) {
+  const { code, syscall, path } = error;
+  if (typeof code !== "string")
+    return;
+  return {
+    code,
+    ...typeof syscall === "string" ? { syscall } : {},
+    ...typeof path === "string" ? { path } : {}
+  };
+}
+function internalErrorToHttpError(error, { log = true } = {}) {
+  const logInternal = (...args) => {
+    if (log)
+      logInternalFailure(...args);
+  };
+  const access = filesystemAccessFailure(error);
+  if (access) {
+    logInternal("Filesystem access refused", error, "warn");
+    return httpError(500, `The server cannot access a path it needs (${describeFilesystemAccessFailure(access)}). ` + `Give the user the server runs as access to it.`);
+  }
   if (error instanceof BadRequestError) {
     return httpError(400, error.message);
+  } else if (error instanceof ServerConfigurationError) {
+    logInternal("Server configuration error", error, "warn");
+    return httpError(500, error.message);
   } else if (error instanceof FrozenConfigError) {
     return httpError(403, error.message);
   } else if (error instanceof AccessDeniedError) {
@@ -13106,7 +13151,7 @@ function internalErrorToHttpError(error) {
     if (error.callerSafe) {
       return httpError(502, error.message);
     }
-    logInternalFailure("Upstream connection error", error, "warn");
+    logInternal("Upstream connection error", error, "warn");
     return httpError(502, GENERIC_UPSTREAM_MESSAGE);
   } else if (error instanceof MaterializationNotFoundError) {
     return httpError(404, error.message);
@@ -13117,7 +13162,7 @@ function internalErrorToHttpError(error) {
   } else if (error instanceof WriteConflictError) {
     return httpError(409, error.message);
   } else if (error instanceof WriteRolledBackError) {
-    logInternalFailure("Dashboard write rolled back", error, "warn");
+    logInternal("Dashboard write rolled back", error, "warn");
     return httpError(500, error.message);
   } else if (error instanceof ServiceUnavailableError) {
     return httpError(503, error.message);
@@ -13128,7 +13173,7 @@ function internalErrorToHttpError(error) {
   } else if (error instanceof NotImplementedError) {
     return httpError(501, error.message);
   } else {
-    logInternalFailure("Unhandled internal error", error);
+    logInternal("Unhandled internal error", error);
     return httpError(500, GENERIC_INTERNAL_MESSAGE);
   }
 }
@@ -13150,10 +13195,15 @@ function toQueryTextProblem(problem) {
     ...problem.at ? { at: { range: problem.at.range } } : {}
   };
 }
-var GENERIC_INTERNAL_MESSAGE = "Internal server error.", GENERIC_UPSTREAM_MESSAGE = "Upstream connection error.", MAX_LOGGED_DETAIL_CHARS = 2000, NotImplementedError, BadRequestError, InvalidArgumentError, CompileRefusedError, EnvironmentNotFoundError, PackageNotFoundError, ModelNotFoundError, DashboardNotFoundError, ConnectionNotFoundError, TableNotFoundError, ConnectionError, DestinationNotFoundError, ConnectionAuthError, UnsupportedCatalogFormatError, ModelCompilationError, PackageManifestError, MaterializationEligibilityError, PublisherConfigError, FrozenConfigError, AccessDeniedError, QueryCompileError, NotQueryableError, OffSurfaceError, MaterializationNotFoundError, MaterializationConflictError, WriteConflictError, WriteRolledBackError, InvalidStateTransitionError, ServiceUnavailableError, PayloadTooLargeError, ResponseUnserializableError, QueryTimeoutError;
+var GENERIC_INTERNAL_MESSAGE = "Internal server error.", GENERIC_UPSTREAM_MESSAGE = "Upstream connection error.", MAX_LOGGED_DETAIL_CHARS = 2000, FILESYSTEM_ACCESS_DESCRIPTIONS, NotImplementedError, BadRequestError, InvalidArgumentError, CompileRefusedError, EnvironmentNotFoundError, PackageNotFoundError, ModelNotFoundError, DashboardNotFoundError, ConnectionNotFoundError, TableNotFoundError, ConnectionError, DestinationNotFoundError, ConnectionAuthError, UnsupportedCatalogFormatError, ModelCompilationError, PackageManifestError, MaterializationEligibilityError, PublisherConfigError, ServerConfigurationError, FrozenConfigError, AccessDeniedError, QueryCompileError, NotQueryableError, OffSurfaceError, MaterializationNotFoundError, MaterializationConflictError, WriteConflictError, WriteRolledBackError, InvalidStateTransitionError, ServiceUnavailableError, PayloadTooLargeError, ResponseUnserializableError, QueryTimeoutError;
 var init_errors = __esm(() => {
   init_constants();
   init_logger();
+  FILESYSTEM_ACCESS_DESCRIPTIONS = {
+    EACCES: "permission denied",
+    EPERM: "operation not permitted",
+    EROFS: "read-only file system"
+  };
   NotImplementedError = class NotImplementedError extends Error {
     constructor(message) {
       super(message);
@@ -13174,8 +13224,8 @@ var init_errors = __esm(() => {
     }
   };
   PackageNotFoundError = class PackageNotFoundError extends Error {
-    constructor(message) {
-      super(message);
+    constructor(message, options) {
+      super(message, options);
     }
   };
   ModelNotFoundError = class ModelNotFoundError extends Error {
@@ -13246,6 +13296,12 @@ var init_errors = __esm(() => {
       this.cause = cause;
     }
   };
+  ServerConfigurationError = class ServerConfigurationError extends Error {
+    constructor(message) {
+      super(message);
+      this.name = "ServerConfigurationError";
+    }
+  };
   FrozenConfigError = class FrozenConfigError extends Error {
     constructor(message = `Publisher config can't be updated when ${PUBLISHER_CONFIG_NAME} has { "frozenConfig": true }`) {
       super(message);
@@ -13303,8 +13359,8 @@ var init_errors = __esm(() => {
     }
   };
   ServiceUnavailableError = class ServiceUnavailableError extends Error {
-    constructor(message) {
-      super(message);
+    constructor(message, options) {
+      super(message, options);
     }
   };
   PayloadTooLargeError = class PayloadTooLargeError extends Error {
@@ -13723,6 +13779,76 @@ var init_data_styles = __esm(() => {
   init_logger();
 });
 
+// src/package_load/error_wire.ts
+import { MalloyError as MalloyError2 } from "@malloydata/malloy";
+function serializeError(error) {
+  const serialized = serializeErrorShape(error);
+  const errno = error instanceof Error ? errnoWireFields(error) : undefined;
+  return errno ? { ...serialized, errno } : serialized;
+}
+function serializeErrorShape(error) {
+  if (error instanceof MalloyError2) {
+    return {
+      name: error.name,
+      message: error.message,
+      stack: error.stack,
+      malloyProblems: error.problems,
+      isCompilationError: true
+    };
+  }
+  if (error instanceof ModelCompilationError) {
+    return {
+      name: error.name,
+      message: error.message,
+      stack: error.stack,
+      isCompilationError: true
+    };
+  }
+  if (error instanceof PackageManifestError) {
+    return {
+      name: error.name,
+      message: error.message,
+      stack: error.stack,
+      isManifestError: true
+    };
+  }
+  if (error instanceof Error) {
+    return {
+      name: error.name,
+      message: error.message,
+      stack: error.stack
+    };
+  }
+  return { name: "Error", message: String(error) };
+}
+function deserializeError(serialized) {
+  const err = new Error(serialized.message);
+  err.name = serialized.name;
+  if (serialized.stack)
+    err.stack = serialized.stack;
+  if (serialized.errno)
+    Object.assign(err, serialized.errno);
+  if (serialized.malloyProblems) {
+    err.problems = serialized.malloyProblems;
+  }
+  if (serialized.isCompilationError) {
+    const wrapped = new ModelCompilationError(err);
+    if (serialized.stack)
+      wrapped.stack = serialized.stack;
+    return wrapped;
+  }
+  if (serialized.isManifestError) {
+    const manifestError = new PackageManifestError(serialized.message);
+    if (serialized.stack)
+      manifestError.stack = serialized.stack;
+    return manifestError;
+  }
+  return err;
+}
+var init_error_wire = __esm(() => {
+  init_errors();
+});
+
 // src/service/annotations.ts
 import { Annotations } from "@malloydata/malloy";
 function isReservedRoute(route) {
@@ -23935,6 +24061,7 @@ init_constants();
 init_authorize_metrics();
 init_data_styles();
 init_errors();
+init_error_wire();
 init_authorize();
 init_gate_classification();
 init_gate_dimension();
@@ -23943,7 +24070,6 @@ import {
   contextOverlay,
   isSourceDef as isSourceDef7,
   MalloyConfig,
-  MalloyError as MalloyError2,
   modelDefToModelInfo as modelDefToModelInfo2,
   Runtime
 } from "@malloydata/malloy";
@@ -24722,48 +24848,6 @@ async function loadPackage(job) {
     }
   };
 }
-function serializeError(error) {
-  if (error instanceof MalloyError2) {
-    return {
-      name: error.name,
-      message: error.message,
-      stack: error.stack,
-      malloyProblems: error.problems,
-      isCompilationError: true
-    };
-  }
-  if (error instanceof ModelCompilationError) {
-    return {
-      name: error.name,
-      message: error.message,
-      stack: error.stack,
-      isCompilationError: true
-    };
-  }
-  if (error instanceof PackageManifestError) {
-    return {
-      name: error.name,
-      message: error.message,
-      stack: error.stack,
-      isManifestError: true
-    };
-  }
-  if (error instanceof Error) {
-    return {
-      name: error.name,
-      message: error.message,
-      stack: error.stack
-    };
-  }
-  return { name: "Error", message: String(error) };
-}
-function deserializeError(serialized) {
-  const err = new Error(serialized.message);
-  err.name = serialized.name;
-  if (serialized.stack)
-    err.stack = serialized.stack;
-  return err;
-}
 var shuttingDown = false;
 var inFlightJobs = new Set;
 port.on("message", (message) => {