@stelstone/server 0.26.0

package/README.md

@@ -0,0 +1,275 @@
+# @stelstone/server
+
+Express-based CMS server with pluggable adapters for content storage,
+media, auth, and build-status. Designed to be mounted from any Node
+process or — via [`stelstone`](../astro-cms) — directly into
+`astro dev`.
+
+The server is **glue code**: routes, JSON wiring, error handling. All
+behavior comes from adapter factories you instantiate from your
+`cms.config.mjs`.
+
+## Install
+
+```sh
+npm i @stelstone/server
+# Optional, only needed for the dev-mode admin UI middleware:
+npm i -D vite
+```
+
+## Usage
+
+```js
+// scripts/admin-server.mjs
+import path from "path";
+import { fileURLToPath } from "url";
+import { startCmsServer } from "@stelstone/server";
+import cmsConfig, { publicConfig } from "../cms.config.mjs";
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const ROOT_DIR = path.join(__dirname, "..");
+
+await startCmsServer({
+  config: cmsConfig,
+  publicConfig,
+  rootDir: ROOT_DIR,
+  realm: "My Site Admin",
+  // Optional. Left out, the server resolves the admin UI itself — source when
+  // it is present, the built bundle otherwise (see `resolveAdminUiOptions`).
+  adminUi: {
+    mode: "auto", // vite-dev when NODE_ENV !== "production", static otherwise
+    distDir: path.join(ROOT_DIR, "node_modules/@stelstone/admin-ui/dist"),
+    sourceDir: path.join(ROOT_DIR, "node_modules/@stelstone/admin-ui"),
+  },
+});
+```
+
+Most sites need none of this: `npx stelstone` reads `cms.config.mjs`, finds
+the admin UI and serves it. Reach for the API when you are embedding the CMS
+in a server you already have.
+
+## API
+
+### `createCmsServer({ config, rootDir, publicConfig?, realm? })`
+Returns `{ handle, middleware, adapters }` — `handle` is a Fetch handler
+(`Request` → `Response | null`), `middleware` the same thing adapted to Node's
+`(req, res, next)`. Nothing listens; you mount it where you like.
+
+### `resolveAdminUi(options)`
+Turns an admin UI choice into something mountable. Modes:
+
+- `{ mode: "static", dir }` — serves a prebuilt SPA bundle. Returns `{ fetchHandler }`.
+- `{ mode: "vite-dev", root, base? }` — dev middleware with HMR (requires `vite`,
+  and the admin UI's *source*, which a published tarball does not ship).
+  Returns `{ nodeMiddleware }`.
+- `{ mode: "auto", distDir, sourceDir }` — `vite-dev` when `NODE_ENV !== "production"`, `static` otherwise.
+
+Either mode answers `/admin` with a 301 to `/admin/`. The SPA links its bundle
+relatively, so without the trailing slash a browser resolves `./assets/…`
+against `/admin`, gets a 404, and shows a blank page with nothing in the
+console to explain it.
+
+### `resolveAdminUiOptions({ dev?, previewThemeCss?, onWarn? })`
+The source-then-dist decision, shared so callers cannot drift: the admin UI's
+source when it is there and `dev` is set, the built bundle otherwise, `null`
+when neither exists. `startCmsServer` and `stelstone` both use it.
+
+### `startCmsServer(opts)`
+Convenience: builds the handler, mounts the admin UI, listens. Returns
+`{ server, handle, adapters, stopScheduler }`.
+
+## Subpath exports
+
+- `@stelstone/server/media-url` — pure `createMediaUrl(mediaConfig)`
+  factory for building CDN URLs in your site code. No `express`
+  dependency, safe to import from Astro components.
+- `@stelstone/server/adapters` — direct access to each adapter
+  factory if you want to compose your own server.
+
+## Routes
+
+| Method  | Path                                   | Purpose                  |
+| ------- | -------------------------------------- | ------------------------ |
+| GET     | `/api/config`                          | sanitized `publicConfig` (allowlisted before auth) |
+| GET     | `/api/collections`                     | collection summaries     |
+| GET     | `/api/collections/:c`                  | list entries             |
+| GET     | `/api/collections/:c/:file`            | read entry               |
+| PUT     | `/api/collections/:c/:file`            | update entry             |
+| POST    | `/api/collections/:c`                  | create entry             |
+| DELETE  | `/api/collections/:c/:file`            | delete entry             |
+| POST    | `/api/collections/:c/:file/publish`    | publish ONE record — only its file is committed (501 on backends that publish on save) |
+| GET     | `/api/links`                           | every page's public path, for the `link` field's picker |
+| GET     | `/api/assets`                          | grouped local-assets list (allowlisted before auth) |
+| GET     | `/api/assets/:folder`                  | files in one local folder |
+| GET     | `/api/media/folders`                   | CDN folders (proxy)      |
+| GET     | `/api/media/folder/:folder`            | CDN files (proxy)        |
+| POST    | `/api/media/upload`                    | upload to CDN (proxy)    |
+| POST    | `/api/publish`                         | git commit + push (everything under publishPaths) |
+| GET     | `/api/publish/status`                  | pending changes, per file, plus `perEntryPublish` capability |
+| GET     | `/api/deploy/status`                   | Netlify deploy status    |
+
+## Adapters
+
+All under `src/adapters/`. Each is a pure factory; no module-scoped state.
+
+| Factory                  | Implements        | Notes                                  |
+| ------------------------ | ----------------- | -------------------------------------- |
+| `createFsJsonContent`    | ContentAdapter    | JSON files on disk + `git push`        |
+| `createLocalAssetsMedia` | MediaAdapter      | legacy `src/assets/` file picker       |
+| `createCdnProxyMedia`    | MediaAdapter      | proxies CloudFront/S3 listing + upload |
+| `createBasicAuth`        | AuthAdapter       | HTTP Basic + HMAC-SHA256 JWT for media |
+| `createNetlifyBuild`     | BuildAdapter      | Netlify deploy status                  |
+| `createMediaUrl`         | (URL helper)      | `cdnBase` + resize-prefix → URLs       |
+
+JSDoc contracts in `src/adapters/types.mjs`. Swap in your own
+implementation by passing different adapter instances; the server
+glue is agnostic.
+
+## Collection listing & the content index
+
+### The problem
+
+Listing a collection traditionally fetches every entry's full JSON via GraphQL or a fallback REST approach (one request per file). On Cloudflare Workers this exceeds the 50-subrequest limit. On GitHub's GraphQL, large collections return 502 errors. The solution: **a lightweight per-collection `_index.json` manifest** containing only entry metadata (id, slug, lang, collection, title, and brief meta). The server reads one manifest per collection instead of fetching every entry.
+
+### How `_index.json` works
+
+Each collection maintains a manifest at `<pagesDir>/<collection>/_index.json`:
+
+```json
+{
+  "entries": [
+    {
+      "id": "entry-1",
+      "slug": "my-first-post",
+      "lang": "en",
+      "collection": "blog",
+      "title": "My First Post",
+      "file": "entry-1.json",
+      "meta": { /* custom fields from metaFields */ }
+    }
+  ]
+}
+```
+
+The index is maintained **incrementally**: `createPage`, `writePage`, and `deletePage` upsert or remove entries. On batch operations, `writeBatch` regenerates the `_index.json` in the same commit. Normal CMS edits keep the index in sync automatically — no rebuild needed.
+
+### Forms — `config.mail` + `config.forms`
+
+The replacement for Netlify Forms, served by the same handler on both
+runtimes (`POST /api/forms/:name`, public). Delivery is Resend; the sender
+domain must be verified there (SPF/DKIM) or mail will not arrive.
+
+```js
+mail: { from: "Site <forms@site.com>" },   // keyEnv: "RESEND_API_KEY" is the default
+forms: {
+  iletisim: {
+    to: "info@site.com",
+    subject: (fields) => `Mesaj — ${fields["Ad ve Soyad"]}`,  // optional
+    replyTo: "E-Posta",          // reply-to comes from this field
+    redirect: "/tesekkurler/",   // plain HTML posts get a 303 here
+    honeypot: "bot-field",       // default
+    turnstile: false,            // true → verify cf-turnstile-response (TURNSTILE_SECRET)
+  },
+},
+```
+
+```html
+<form method="POST" action="/api/forms/iletisim">
+  <p hidden><input name="bot-field" tabindex="-1" autocomplete="off" /></p>
+  <input name="Ad ve Soyad" required />
+  <input name="E-Posta" type="email" required />
+  <textarea name="Mesaj" required></textarea>
+  <button>Gönder</button>
+</form>
+```
+
+Gates, in order: unknown form → 404 · filled honeypot → silent 200, nothing
+delivered (an error would teach the bot which field to skip) · size limits
+(30 fields, 5000 chars/field) → 400 · per-IP rate limit (5/min; in-memory on
+Node, KV-backed on Workers) → 429 · unconfigured mail → 503 · delivery
+failure → 502. JS-driven forms get JSON; plain HTML posts get the redirect.
+
+### Serving the Worker admin from the site's own address
+
+A deployed CMS Worker answers at `<name>.workers.dev`, which is neither an
+address to hand a client nor same-origin with the site. Pick by where the
+site is hosted; the first two make `siteadi.com/admin` literal, which also
+ends CORS configuration — auth headers never travel cross-origin:
+
+| Site hosting | Setup |
+| --- | --- |
+| Cloudflare | Attach a route to the Worker: `siteadi.com/admin*` (and `/api/*`). True same-origin. |
+| Netlify | Proxy in `_redirects`: `/admin/* https://<name>.workers.dev/admin/:splat 200` (same for `/api/*`). Netlify proxies server-side; the browser sees one origin. |
+| Elsewhere | Give the Worker a custom domain (`admin.siteadi.com`). Not same-origin, but presentable; keep `cors.origin` pinned to the site. |
+
+`workers.dev` is a fallback for testing, not an address to ship.
+
+### Configuration: `content.draftBranch` — saving without publishing
+
+Without it, this backend commits every save straight to `content.branch`;
+when CI deploys that branch, **saving a live page publishes it instantly** —
+there is no "work on it, publish when ready". Set a draft branch and the
+model becomes the fs backend's: saves land on the draft branch, the site
+keeps serving the published version, and Publish (the whole site or one
+record) moves files across with a single commit.
+
+```js
+content: {
+  provider: "github",
+  branch: "main",             // what CI deploys
+  draftBranch: "cms-drafts",  // where saves go; created automatically
+  // ...
+}
+```
+
+Two things to know: the `_index.json` manifests live on the draft branch and
+are never published (the site build ignores `_`-prefixed files), and content
+edits pushed to `main` outside the CMS will show up in `pendingChanges` as a
+diff — with a draft branch configured, content should change through the CMS.
+Config validation warns when the github backend runs without a draft branch.
+
+### Configuration: the `content.list` block
+
+In `cms.config.mjs`, configure listing behavior under `content.list`:
+
+```js
+content: {
+  provider: "github",
+  // ...
+  list: {
+    strategy: "index",        // default; reads _index.json
+    rebuild: "build",         // "build" (default) | "lazy"
+    indexFile: "_index.json", // manifest filename
+    resolve: undefined,       // optional: custom listing function
+  },
+}
+```
+
+| Option    | Values                  | Description |
+| --------- | ----------------------- | ----------- |
+| `strategy` | `"index"` | Built-in strategy: reads the `_index.json` manifest. Only option currently shipped. |
+| `rebuild` | `"build"` (default), `"lazy"` | **`"build"`**: server never cold-rebuilds an index at request time. If missing, returns empty and logs a warning to run the CLI. Safest for serverless (avoids subrequest blowups). **`"lazy"`**: if an index is missing, the server bootstraps it with one GraphQL request and persists it. Convenient for small/self-hosted setups, but risky on very large collections (GraphQL may fail). |
+| `indexFile` | string | Override the manifest filename (default: `_index.json`). |
+| `resolve` | `async (collection, { sortConfig }) => entries[] \| null` | Optional: bring your own listing logic. Completely replaces the built-in strategy. Return `null` for unknown collections. Plug in D1, KV, Algolia, Pagefind, or any external index here — this is the scale/search extension point. |
+
+### Out-of-band rebuild: the `build-index` CLI
+
+Regenerate all `_index.json` manifests locally:
+
+```sh
+npx stelstone build-index --config ./cms.config.mjs
+```
+
+Walks the local `pagesDir`, regenerates `_index.json` for every collection, and prints per-collection entry counts. Commit the result.
+
+**When to run:**
+- Once when adopting the index on an existing repo.
+- After bulk imports or migrations.
+- After content changed outside the CMS (e.g., direct git edits).
+- **Recommended in CI/deployment for serverless**: build and commit the index locally or in your CI pipeline, so the Worker only ever reads it (pairs with `rebuild: "build"`).
+
+### Quick decision guide
+
+- **Small site or self-hosted**: `rebuild: "lazy"` keeps setup simple.
+- **Serverless (Cloudflare Workers) or large collections (default, recommended)**: Use `rebuild: "build"` and run `build-index` in your build/deploy pipeline. Commit the index and the Worker reads it once per request — zero surprise subrequests.
+- **Real full-text search, faceting, or huge scale**: Provide a `resolve` hook backed by D1, KV, Algolia, Pagefind, or another external index.

package/bin/stelstone.mjs

@@ -0,0 +1,181 @@
+#!/usr/bin/env node
+/**
+ * stelstone CLI
+ *
+ * Usage:
+ *   stelstone [start] [--port 4001] [--config ./cms.config.mjs] [--dev]
+ *
+ * --dev serves the admin UI via Vite dev middleware (HMR, no rebuild step).
+ *   Requires a linked/monorepo admin-ui source; falls back to static dist.
+ *
+ * Reads cms.config.mjs from the current working directory (or --config path),
+ * auto-discovers the admin-ui dist, and starts the CMS server.
+ *
+ * In your package.json:
+ *   "scripts": {
+ *     "admin": "node --env-file=.env node_modules/.bin/stelstone"
+ *   }
+ */
+
+import { pathToFileURL } from "url";
+import path from "path";
+import fs from "fs";
+import { startCmsServer } from "../src/index.mjs";
+import { buildListEntry } from "../src/adapters/_shared.mjs";
+
+const args = process.argv.slice(2).filter((a) => a !== "start");
+
+// Detect subcommand (first non-flag positional arg)
+const subcommand = process.argv.slice(2).find((a) => !a.startsWith("-"));
+
+function flag(name) {
+  const i = args.indexOf(name);
+  return i !== -1 ? args[i + 1] : null;
+}
+
+function boolFlag(name) {
+  return args.includes(name);
+}
+
+// Load config helper
+async function loadConfig(cfgPath) {
+  try {
+    const mod = await import(pathToFileURL(cfgPath).href);
+    const config = mod.default;
+    const publicConfig = mod.publicConfig;
+    if (!config) throw new Error("cms.config.mjs must have a default export");
+    return { config, publicConfig };
+  } catch (err) {
+    if (err.code === "ERR_MODULE_NOT_FOUND" || err.code === "ERR_LOAD_URL") {
+      console.error(`[stelstone] Cannot find config file: ${cfgPath}`);
+      console.error("  Create cms.config.mjs in your project root, or use --config <path>.");
+    } else {
+      console.error(`[stelstone] Failed to load ${cfgPath}:\n  ${err.message}`);
+    }
+    process.exit(1);
+  }
+}
+
+const dev = boolFlag("--dev");
+
+const cwd      = process.cwd();
+const cfgPath  = path.resolve(cwd, flag("--config") || "cms.config.mjs");
+const port     = parseInt(flag("--port") || process.env.ADMIN_PORT || "4001", 10);
+const realm    = flag("--realm") || "CMS Admin";
+
+// Handle build-index subcommand
+if (subcommand === "build-index") {
+  const { config } = await loadConfig(cfgPath);
+
+  const pagesDir = config.content?.pagesDir;
+  if (!pagesDir) {
+    console.error("[stelstone] config.content.pagesDir is not set");
+    process.exit(1);
+  }
+
+  const pagesDirAbs = path.resolve(cwd, pagesDir);
+  if (!fs.existsSync(pagesDirAbs)) {
+    console.error(`[stelstone] Pages directory does not exist: ${pagesDirAbs}`);
+    process.exit(1);
+  }
+
+  const indexFileName = config.content?.list?.indexFile || "_index.json";
+
+  try {
+    const collections = fs.readdirSync(pagesDirAbs);
+    let totalEntries = 0;
+
+    // Pre-pass: build slug→displayName lookup tables for every collection so
+    // relation fields (e.g. a blog's `author` combobox) can be resolved to the
+    // referenced entry's name instead of storing a raw slug.
+    const lookups = {};
+    for (const collectionName of collections) {
+      if (collectionName.startsWith(".")) continue;
+      const collectionPath = path.join(pagesDirAbs, collectionName);
+      let stat;
+      try {
+        stat = fs.statSync(collectionPath);
+      } catch {
+        continue;
+      }
+      if (!stat.isDirectory()) continue;
+
+      const table = {};
+      for (const fileName of fs.readdirSync(collectionPath)) {
+        if (fileName === indexFileName || fileName.startsWith(".")) continue;
+        if (!fileName.endsWith(".json")) continue;
+        try {
+          const data = JSON.parse(fs.readFileSync(path.join(collectionPath, fileName), "utf-8"));
+          if (data.slug) table[data.slug] = data.meta?.title || data.meta?.name || data.slug;
+        } catch {
+          // Skip files that fail to parse
+        }
+      }
+      lookups[collectionName] = table;
+    }
+
+    for (const collectionName of collections) {
+      // Skip hidden names and non-directories
+      if (collectionName.startsWith(".")) continue;
+
+      const collectionPath = path.join(pagesDirAbs, collectionName);
+      let stat;
+      try {
+        stat = fs.statSync(collectionPath);
+      } catch {
+        continue;
+      }
+      if (!stat.isDirectory()) continue;
+
+      // Read JSON files in the collection
+      const entries = [];
+      const files = fs.readdirSync(collectionPath);
+
+      for (const fileName of files) {
+        // Skip _index.json and hidden files
+        if (fileName === indexFileName || fileName.startsWith(".")) continue;
+        if (!fileName.endsWith(".json")) continue;
+
+        const filePath = path.join(collectionPath, fileName);
+        let data;
+        try {
+          const content = fs.readFileSync(filePath, "utf-8");
+          data = JSON.parse(content);
+        } catch {
+          // Skip files that fail to parse
+          continue;
+        }
+
+        entries.push(buildListEntry(config.collections?.[collectionName], collectionName, fileName, data, lookups));
+      }
+
+      // Write the index file
+      const indexPath = path.join(collectionPath, indexFileName);
+      fs.writeFileSync(indexPath, JSON.stringify({ entries }, null, 2));
+
+      console.log(`${collectionName}: ${entries.length} entries`);
+      totalEntries += entries.length;
+    }
+
+    console.log(`\n[stelstone] Indexed ${totalEntries} entries across all collections`);
+    process.exit(0);
+  } catch (err) {
+    console.error(`[stelstone] Failed to build index:\n  ${err.message}`);
+    process.exit(1);
+  }
+}
+
+// Load config for server start
+let config, publicConfig;
+const { config: loadedConfig, publicConfig: loadedPublicConfig } = await loadConfig(cfgPath);
+config = loadedConfig;
+publicConfig = loadedPublicConfig;
+
+await startCmsServer({
+  config,
+  publicConfig,   // undefined is fine — server uses defaultPublicConfig(config)
+  rootDir: cwd,
+  realm,
+  port,
+  dev,
+});

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "@stelstone/server",
+  "version": "0.26.0",
+  "description": "Runtime-agnostic CMS server built on the Web Fetch API, with pluggable adapters for content, media, auth, and build.",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/natilon/stelstone.git",
+    "directory": "packages/server"
+  },
+  "keywords": [
+    "cms",
+    "headless-cms",
+    "git-based",
+    "astro",
+    "cloudflare-workers"
+  ],
+  "type": "module",
+  "bin": {
+    "stelstone": "./bin/stelstone.mjs"
+  },
+  "scripts": {
+    "test": "node --test 'test/**/*.test.mjs'"
+  },
+  "exports": {
+    ".": "./src/index.mjs",
+    "./media-url": "./src/adapters/media-url.mjs",
+    "./adapters": "./src/adapters/index.mjs",
+    "./public-config": "./src/default-public-config.mjs",
+    "./routes": "./src/routes.mjs",
+    "./handler": "./src/core/handler.mjs",
+    "./node-adapter": "./src/core/node-adapter.mjs",
+    "./static-files": "./src/core/static-files.mjs",
+    "./config-schema": "./src/core/config-schema.mjs",
+    "./adapter-options": "./src/core/adapter-options.mjs"
+  },
+  "files": [
+    "src",
+    "bin"
+  ],
+  "peerDependencies": {
+    "vite": "^5.0.0",
+    "@stelstone/admin-ui": ">=0.12.0"
+  },
+  "peerDependenciesMeta": {
+    "vite": {
+      "optional": true
+    },
+    "@stelstone/admin-ui": {
+      "optional": true
+    }
+  }
+}