@malloy-publisher/server 0.0.382-dev → 0.2.0

package/README.docker.md

@@ -1,32 +1,157 @@
-# Creating a Publisher Docker Image
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
 
-There are 2 docker files in the repo. A base `production.docker` which creates the base image and `malloy-samples.docker` which is layered on top and copies the malloy samples into the Docker image so they can be served by the publisher.
+# Publisher in Docker
 
-To create a Docker image for your own packages, follow the example of `docker/malloy-samples.docker` which copies in the publisher config and malloy files.
+The canonical build is the root [`Dockerfile`](../../Dockerfile) and the CI smoke test (`docker_smoke_test` in `.github/workflows/build.yml`) builds and runs that exact image. The two-port REST + MCP server, the Snowflake ADBC driver, the DuckDB CLI, and the production app bundle all ship in it.
 
-## To create & run the malloy-samples Docker image:
+A short Docker section in the [deployment guide](../../docs/deployment.md) covers the canonical build + run; this doc goes deeper on runtime layout, environment variables, persistent storage, and credentials.
 
-1. Clone the Publisher repository + malloy-sample submodule as outline in the README.
+## Build and run
 
-2. Build the Docker imagee using the provided Dockerfiles:
+```bash
+docker build -t malloy-publisher .
+docker run -d \
+  --name malloy-publisher \
+  -p 4000:4000 -p 4040:4040 \
+  -v $(pwd)/publisher.config.json:/publisher/publisher.config.json:ro \
+  malloy-publisher
+```
+
+Once `/api/v0/status` reports `operationalState: "serving"`, the REST API is at `http://localhost:4000` and MCP at `http://localhost:4040/mcp`.
+
+If you don't have a config of your own yet, copy [`packages/server/publisher.config.example.duckdb.json`](./publisher.config.example.duckdb.json) (DuckDB-only samples, no credentials required) and mount that. There's also a [`publisher.config.example.bigquery.json`](./publisher.config.example.bigquery.json) sibling for the BigQuery samples.
+
+## Pre-built image
+
+If you don't want to build the image yourself, the official pre-built image is published to Docker Hub under the **`ms2data/`** namespace (not `malloydata/`):
+
+```bash
+docker pull ms2data/malloy-publisher
+docker run -d \
+  --name malloy-publisher \
+  -p 4000:4000 -p 4040:4040 \
+  -v $(pwd)/publisher.config.json:/publisher/publisher.config.json:ro \
+  ms2data/malloy-publisher
+```
+
+See the [Docker Hub tags page](https://hub.docker.com/r/ms2data/malloy-publisher/tags) for available versions. Tag-scheme guidance (`:latest`, `:X.Y.Z`, `:next`) lives in the [deployment guide](../../docs/deployment.md).
+
+## Runtime layout
+
+| Path inside container | What's there |
+|---|---|
+| `/publisher/` | `WORKDIR`. The server reads `<WORKDIR>/publisher.config.json` by default — that's the file you mount. |
+| `/publisher/packages/server/dist/` | The bundled server (built by `bun run build` in CI). |
+| `/publisher/packages/app/dist/` | The static SPA the server serves. |
+| `/publisher/publisher_data/` | Per-environment package clones, DuckDB extension cache, and per-package sandbox DBs. Created at runtime; **persist this as a named volume if you want first-run sample clones to survive a container restart.** |
+| `/root/.duckdb/` | DuckDB CLI + extension install dir. Bundled into the image. |
+
+To keep `publisher_data/` across restarts:
+
+```bash
+docker run -d \
+  --name malloy-publisher \
+  -p 4000:4000 -p 4040:4040 \
+  -v $(pwd)/publisher.config.json:/publisher/publisher.config.json:ro \
+  -v publisher_data:/publisher/publisher_data \
+  malloy-publisher
+```
+
+The first request after a fresh start clones sample packages from GitHub — a named volume turns that one-time cost into a one-time cost across all container lifecycles.
 
-   ```bash
-   docker build -t malloy-publisher:latest -f docker/production.docker .
-   docker build -t malloy-samples:latest -f docker/malloy-samples.docker .
-   ```
+For the same pattern as a complete Compose file (with a healthcheck against `/api/v0/status` and both ports mapped), see [`docker-compose.example.yml`](../../docker-compose.example.yml) at the repo root.
 
-3. Run the Docker container.
-   The Publisher server runs at port 4000, the MCP server at port 4040.
+## Configuration via environment variables
 
-   ```bash
-   docker run -p 4000:4000 -p 4040:4040 malloy-samples:latest
-   ```
+All flags exposed by `bin/malloy-publisher --help` have an equivalent env var, so they're easy to set from `docker run -e` or compose:
 
-4. (Optional) For BigQuery access, mount your GCP credentials.
-   Note that you cannot use personal credentials- only service account credentials can be used in the docker image.
+| Env var | Equivalent flag | Default | Purpose |
+|---|---|---|---|
+| `PUBLISHER_PORT` | `--port <n>` | `4000` | REST API port. |
+| `PUBLISHER_HOST` | `--host <h>` | `0.0.0.0` | Bind address. |
+| `MCP_PORT` | `--mcp_port <n>` | `4040` | MCP API port. |
+| `PUBLISHER_NO_MCP_CONFIG` | `--no-mcp-config` | `1` **in this image** | Suppresses the `.mcp.json` the server otherwise writes into its working directory on startup. That file exists so an AI agent opened in that directory finds the server; nothing starts an agent session inside the container, and the git-working-tree guard that would normally cover `/publisher` cannot fire because `.dockerignore` excludes `.git`. Left on, every boot would create a root-owned file, which matters if you bind-mount a project directory at `/publisher`. Pass `-e PUBLISHER_NO_MCP_CONFIG=` to turn it back on. Note this is the one env var the image sets for you: `docker run -e PUBLISHER_NO_MCP_CONFIG` (no `=`) and a Compose `environment:` entry with no value both *delete* it when the host does not have it set, which re-enables the write. |
+| `SERVER_ROOT` | `--server_root <path>` | `.` (cwd) at the server level; overridden to `/publisher` by the bundled CMD | Directory the server treats as its working dir. The image's CMD passes `--server_root /publisher` explicitly so the zero-arg `npx` bundled-default trigger doesn't fire inside the container. If you override CMD with your own entrypoint, set `SERVER_ROOT` yourself to keep this behaviour. |
+| `PUBLISHER_CONFIG_PATH` | `--config <path>` | unset | Absolute path to a `publisher.config.json`. Wins over `<SERVER_ROOT>/publisher.config.json`. Use this if you want to mount your config somewhere other than `/publisher/`. |
+| `INITIALIZE_STORAGE` | `--init` | `false` | Wipes `publisher_data/` and re-syncs it from the config on boot. A first boot with empty storage loads the config automatically, so set this only to reset state or resync after the on-disk config has drifted from `publisher_data/`. Re-initializing discards any state there that isn't reproducible from the config. See [configuration.md](../../docs/configuration.md#environment-variables--cli-flags). |
+| `SHUTDOWN_DRAIN_DURATION_SECONDS` | `--shutdown_drain_duration_seconds <s>` | `0` | On SIGTERM, how long to keep serving requests (readiness flips to not-ready immediately) before closing server sockets. Set this to your typical request duration to avoid 502s from K8s rolling deploys. |
+| `SHUTDOWN_GRACEFUL_CLOSE_TIMEOUT_SECONDS` | `--shutdown_graceful_close_timeout_seconds <s>` | `0` | Additional grace period after server close before `process.exit`. |
+| `GOOGLE_APPLICATION_CREDENTIALS` | — | unset | Path inside the container to a GCP service-account JSON. Required for BigQuery-backed environments. Personal user credentials don't work inside the container — use a service account. |
+| `PUBLISHER_MAX_MEMORY_BYTES` | — | unset (disabled) | Resident-set-size (RSS) cap in bytes. When set, the in-process **memory governor** polls RSS on `PUBLISHER_MEMORY_CHECK_INTERVAL_MS` and rejects new package loads and new queries with **HTTP 503** once RSS crosses the high-water mark. Designed to keep the pod under its k8s `resources.limits.memory` instead of getting OOM-killed. Set this to roughly `0.7 × resources.limits.memory` so the back-pressure band has headroom for traffic spikes and per-request DuckDB scratch. |
+| `PUBLISHER_MEMORY_HIGH_WATER_FRACTION` | — | `0.8` | Fraction of `PUBLISHER_MAX_MEMORY_BYTES` at which back-pressure activates. Must be in `(0, 1)` and strictly greater than the low-water fraction. |
+| `PUBLISHER_MEMORY_LOW_WATER_FRACTION` | — | `0.7` | Fraction at which back-pressure clears. The gap between low and high gives hysteresis so the governor doesn't flap on every GC cycle. |
+| `PUBLISHER_MEMORY_CHECK_INTERVAL_MS` | — | `5000` | How often the governor samples RSS. Minimum `100`. Smaller values catch spikes faster but burn a few extra microseconds per tick. |
+| `PUBLISHER_MEMORY_BACKPRESSURE` | — | `true` | When `false`, the governor still samples RSS and emits metrics but never flips the back-pressure flag. Useful for a monitoring-only rollout before enabling the 503 behaviour. |
+| `EMBEDDING_API_KEY` | — | _unset_ | Enables semantic ranking for `malloy_getContext` question retrieval; sent as a bearer token to the embedding endpoint. Unset keeps lexical retrieval, unchanged. Entity names, annotation text, and query strings are sent to the endpoint when enabled; see "Semantic retrieval" in `docs/configuration.md`. |
+| `EMBEDDING_MODEL` | — | `text-embedding-3-small` | Embedding model name. |
+| `EMBEDDING_API_BASE` | — | `https://api.openai.com/v1` | Base URL of an OpenAI-compatible embeddings API. |
+| `EMBEDDING_DIMENSIONS` | — | _unset_ | Optional `dimensions` request parameter; omitted when unset. |
+
+### Memory governor
+
+When `PUBLISHER_MAX_MEMORY_BYTES` is unset, the governor is **disabled** and the server's behaviour is identical to prior versions. When it's set, the governor:
+
+- Periodically samples `process.memoryUsage().rss`.
+- Once RSS crosses the high-water mark, **any code path that would allocate a new package into memory returns HTTP 503**, and new queries are rejected the same way. The package gate sits at the single choke point inside `Environment.getPackage` / `Environment.addPackage`, so it covers every controller that touches a not-yet-loaded package — including lazy loads on cache miss from `ModelController`, `ConnectionController`, `QueryController`, `DatabaseController`, etc. — not just the explicit `POST /packages` and `?reload=true` paths.
+- Already-loaded packages remain fully serviceable so dashboards keep rendering under pressure.
+- Once RSS drops back to the low-water mark, back-pressure clears automatically.
+- Recovery happens naturally as in-flight traffic completes and the kernel reclaims pages — the governor does **not** evict, unload, or interrupt loaded packages.
+- A documented `{ allowAdmission: true }` opt-out exists on `Environment.getPackage` / `addPackage` for future internal callers (e.g. warmup / health probes) that genuinely cannot tolerate 503s. No public REST endpoint sets it today.
+
+Metrics exposed on the existing `/metrics` Prometheus endpoint:
+
+| Metric | Type | Notes |
+|---|---|---|
+| `publisher_process_rss_bytes` | gauge | Sampled RSS. |
+| `publisher_memory_backpressure_active` | gauge | `1` when rejecting new loads, `0` otherwise. |
+| `publisher_memory_backpressure_activations_total` | counter | Increments on every `false → true` transition; alert on a non-trivial rate to catch flapping pods. |
+| `publisher_memory_max_bytes`, `publisher_memory_high_water_bytes`, `publisher_memory_low_water_bytes` | gauges | Static configured thresholds — useful for plotting the band alongside the RSS series. |
+
+#### Recommended k8s sizing
+
+A reasonable starting point (tune for your workload):
+
+```yaml
+resources:
+  requests:
+    memory: 2Gi
+  limits:
+    memory: 4Gi
+env:
+  - name: PUBLISHER_MAX_MEMORY_BYTES
+    # 2.8Gi — back-pressure activates at ~2.24Gi, clears at ~1.96Gi,
+    # leaving ~1.2Gi of headroom under the 4Gi k8s hard limit for
+    # in-flight DuckDB scratch and JS heap spikes.
+    value: "3006477107"
+  - name: PUBLISHER_MEMORY_BACKPRESSURE
+    value: "true"
+```
+
+If you want a soft-launch where the governor reports but doesn't act, deploy first with `PUBLISHER_MEMORY_BACKPRESSURE=false`, watch the `publisher_process_rss_bytes` series for a week, then enable.
+
+## BigQuery credentials
+
+To enable BigQuery samples or your own BigQuery connections, mount a service-account key and point `GOOGLE_APPLICATION_CREDENTIALS` at it:
 
 ```bash
-docker run -p 4000:4000 -p 4040:4040 \
-  -v /path/to/your/service_credentials.json:/app/gcp-credentials/key.json \
-  malloy-samples:latest
+docker run -d \
+  --name malloy-publisher \
+  -p 4000:4000 -p 4040:4040 \
+  -v $(pwd)/publisher.config.json:/publisher/publisher.config.json:ro \
+  -v $(pwd)/gcp-sa.json:/etc/publisher/gcp-sa.json:ro \
+  -e GOOGLE_APPLICATION_CREDENTIALS=/etc/publisher/gcp-sa.json \
+  malloy-publisher
 ```
+
+The Dockerfile creates `/etc/publisher/` as an empty directory outside the application tree at `/publisher/`. By the convention this doc establishes, mount credential material there to keep it separated from the app — but any writable path inside the container works.
+
+## The CI Dockerfile (`docker/Dockerfile.ci`)
+
+`docker/Dockerfile.ci` exists for the CI integration-test path (referenced from the repo's `docker-compose.yml`). It is **not** the production image and should not be used for deployment. Production users build the root [`Dockerfile`](../../Dockerfile).
+
+## Deprecated build paths
+
+`docker/production.docker` and `docker/malloy-samples.docker` are leftover from a previous Docker layout. They are not built by CI, are not referenced by any current workflow, and produce a different image than what is deployed. Don't use them — build the root [`Dockerfile`](../../Dockerfile) instead. They will be removed in a follow-up cleanup PR.

package/README.md

@@ -1,10 +1,72 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
 # Malloy Publisher Server
 
 The Malloy Publisher Server is an Express.js server that provides an API for managing and accessing Malloy data models, packages, and queries
 
+## Quick start
+
+This section is self-contained: this README ships in the npm tarball, where most of the repository does not, so nothing here needs a file you do not have. Later sections are written for a clone and do point into the repository.
+
+**New workspace?** One command scaffolds a package, the server config, the MCP wiring, and the agent skills:
+
+```bash
+npm create @malloy-publisher/malloy-package@latest sales
+npm start
+```
+
+Run bare like that, the package comes with a small sample dataset, so there is something to query immediately. To start from a file of your own instead, add `-- --data ./orders.csv`, naming a delimited or Parquet/Excel file you actually have (the `--` is required, and the path is relative to where you run the command). Keep the `@latest`: `npm create` resolves through the npx cache, and an unversioned name silently reuses whatever old copy is on the machine.
+
+**Existing directory of models?** Write a `publisher.config.json` beside a package directory that holds a `publisher.json`, then run the server pointing at it:
+
+```bash
+npx @malloy-publisher/server --server_root . --watch-env local
+```
+
+```json
+{
+  "frozenConfig": false,
+  "environments": [
+    {
+      "name": "local",
+      "packages": [{ "name": "sales", "location": "./sales" }],
+      "connections": []
+    }
+  ]
+}
+```
+
+The config is read from `<server_root>/publisher.config.json` by default; `--config <path>` overrides it.
+
+Three facts that are easy to get wrong:
+
+- **A flat-file (CSV/Parquet/XLSX) package needs no `connections` entry at all.** Every loaded package automatically gets its own DuckDB sandbox connection named `duckdb`, which is what `duckdb.table('data/file.csv')` resolves against. That name is reserved: an environment-level connection named `duckdb` fails the whole environment at init (call an env-level DuckDB connection `shared_duckdb` or similar).
+- **A package `location` is local only when it starts with `./`, `../`, `~/`, or `/`.** Write `./sales`, not `sales`: a bare name is read as a remote URI, and the package is skipped with `Invalid package path: "sales". Must be an absolute mounted path or a GCS/S3/GitHub URI.` in `loadErrors`. A `./` or `../` path resolves against the config file's directory; `~/` against your home directory.
+- **Local authoring means `--watch-env <env>`.** Without it the server copies each local package into `publisher_data/` at boot and serves the copy, so edits to your source directory are never read — a reload recompiles the copy and still answers 200. Adding the flag to a later boot does not undo a copy already made: pass `--watch-env <env> --init` once to re-mount (it wipes `publisher_data/` and re-syncs from config), then plain `--watch-env` boots keep watching. `publisher_data/<env>/<pkg>` tells you which you got — a symlink is mounted, a real directory is a copy.
+
+Poll `curl -s http://localhost:4000/api/v0/status` until `operationalState` is `"serving"`, then check `loadErrors` (absent when everything loaded). The MCP endpoint for agents is `http://localhost:4040/mcp`.
+
+## Configuration
+
+Two example configs ship in the npm package, beside this README: [`publisher.config.example.duckdb.json`](./publisher.config.example.duckdb.json) (the GitHub-hosted sample packages, no connection block) and [`publisher.config.example.bigquery.json`](./publisher.config.example.bigquery.json) (adds a BigQuery connection). Copy either one to `publisher.config.json` and point `--server_root` at its directory.
+
+In a clone, the live `publisher.config.json` lives in this directory, and the repository root contains a symlink (`/publisher.config.json` → `./packages/server/publisher.config.json`) so that running the server from either location picks up the same config. Edit one and you've edited both.
+
+### Remaining deprecation warnings
+
+Removing the unused `trino` CLI direct dep (it pulled in `@google-cloud/translate@0.7.x` → `request@2.x` → `har-validator` → `hawk` → `cryptiles`) cleaned the worst chain. About 25 `npm warn deprecated` lines remain on `npx @malloy-publisher/server` install, all upstream-owned:
+
+- **npm CLI tooling**: `npmlog`, `gauge`, `are-we-there-yet`, `glob@7/8/10`, `rimraf@3`, `tar@6.2.1`, `inflight`, `@npmcli/move-file`, `node-domexception`, `querystring` — pulled in by npm itself and by `node-pre-gyp`/`node-gyp`. Not actionable from this repo.
+- **`uuid@8.x` / `uuid@9.x`**: surfaced across multiple transitives (Malloy, AWS SDKs, others). Resolves when each upstream bumps to `uuid@11`.
+- **`q@1.5.1`**: pulled in via `thrift` → `@databricks/sql` → `@malloydata/db-databricks`. Resolves when Databricks upgrades `@databricks/sql` past the thrift dep, or when we replace the Databricks driver.
+- **`aws-sdk@2.1693.0`**: no longer a direct dep (removed from `packages/server/package.json`); anything still surfacing it is transitive. The actual S3 consumer is `@aws-sdk/client-s3` v3.
+
 ## K6 Test Presets
 
-The Malloy Publisher Server includes several K6 test presets to help you test its performance and stability.
+The Malloy Publisher Server includes several K6 test presets to help you test its performance and stability. These live in the repository, not the npm package, so run them from a clone.
 
 Below is a list of the available test presets:
 
@@ -109,14 +171,7 @@ For more information on how to configure OpenTelemetry collectors, please refer
 
 ## MCP Prompt Capability
 
-Publisher's MCP interface exposes **LLM-ready prompts** for explaining, generating, translating and summarising Malloy code.
-
-| Prompt ID                                      | Purpose                                                           |
-| ---------------------------------------------- | ----------------------------------------------------------------- |
-| `explain-malloy-query@1.0.0`                   | Explain a Malloy query (sources, transformations, output).        |
-| `generate-malloy-query-from-description@1.0.0` | Create Malloy based on a natural-language goal and model context. |
-| `translate-sql-to-malloy@1.0.0`                | Convert SQL into Malloy using a model for schema reference.       |
-| `summarize-malloy-model@1.0.0`                 | Summarise a Malloy model (purpose, entities, joins).              |
+Publisher's MCP interface exposes the bundled agent **skills** as **LLM-ready prompts**, so hosts that ingest MCP but do not load skill files can pull the same guidance. The skills are published separately as [`@malloy-publisher/skills`](https://www.npmjs.com/package/@malloy-publisher/skills); their source is [`skills/`](https://github.com/malloydata/publisher/tree/main/skills), and authoring guidance is in [`docs/agent-skills/`](https://github.com/malloydata/publisher/tree/main/docs/agent-skills).
 
 List prompts:
 
@@ -127,9 +182,7 @@ mcp-client prompts/list
 Get a prompt:
 
 ```bash
-mcp-client prompts/get \
-  --name explain-malloy-query@1.0.0 \
-  --arguments '{"query_code":"from flights"}'
+mcp-client prompts/get --name malloy-analysis
 ```
 
 These calls return `messages` ready for your LLM chat completion.

package/dist/app/api-doc.html

@@ -1,4 +1,9 @@
 <!doctype html>
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
 <html lang="en">
    <head>
       <meta charset="utf-8" />