@technomoron/mail-magic 1.0.23 → 1.0.33

package/CHANGES

@@ -1,3 +1,53 @@
+Version 1.0.33 (2026-02-09)
+
+- Route all env-derived config through `mailStore.vars` with explicit overrides for tests/examples.
+- Move shared helpers into `util/` modules (rate limiting, email parsing, path safety, uploads).
+- Switch template preprocessing to Unyuck’s asset collection while preserving existing URL and CID behavior.
+
+Version 1.0.32 (2026-02-08)
+
+- Add regression coverage for legacy plaintext API token migration to `token_hmac`.
+- Publish `docs/swagger/openapi.json` and install a local swagger handler that is stable regardless of `process.cwd()`.
+- Refactor public form submissions to use explicit `_mm_*` control fields (`_mm_form_key`, `_mm_locale`,
+  `_mm_recipients`) and stricter attachment field naming (`_mm_file*`).
+- Centralize captcha verification in `util/captcha` and add contract-style test coverage for the public form endpoint.
+
+Version 1.0.31 (2026-02-08)
+
+- Add regression coverage for form lookup ambiguity handling and secret-gated recipient overrides.
+
+Version 1.0.30 (2026-02-08)
+
+- Add regression coverage for Nunjucks HTML autoescape behavior and the `|safe` filter.
+
+Version 1.0.29 (2026-02-08)
+
+- Add regression coverage for CAPTCHA verification flows on the public form endpoint.
+
+Version 1.0.28 (2026-02-08)
+
+- Add regression coverage for public form anti-abuse controls (rate limiting, attachment limits, and upload cleanup).
+
+Version 1.0.27 (2026-02-08)
+
+- Refresh README/TUTORIAL with current API routes, config layout, and security-related configuration (API token pepper,
+  form_key, recipient allowlist, and form anti-abuse options).
+- Update `config-example/` to match the current init-data schema and domain-rooted config directory layout.
+
+Version 1.0.26 (2026-02-07)
+
+- Fix env/docs polish and remove unused code flagged by ESLint.
+
+Version 1.0.25 (2026-02-07)
+
+- Default `DB_AUTO_RELOAD` to false to avoid unexpected config reloads in production.
+- Added `DB_SYNC_ALTER` to control Sequelize `sync({ alter: ... })` on startup.
+
+Version 1.0.24 (2026-02-07)
+
+- Validate imported domain names against a safe pattern to prevent config path
+  traversal via malformed domain identifiers.
+
 Version 1.0.23 (2026-02-07)
 
 - Added a form recipient allowlist table plus `POST /api/v1/form/recipient` to map a

package/README.md

@@ -1,97 +1,312 @@
 # @technomoron/mail-magic
 
-Mail Magic is a TypeScript service for managing, templating, and delivering transactional emails. It exposes a small
-REST API built on `@technomoron/api-server-base`, persists data with Sequelize/SQLite, and renders outbound messages
-with Nunjucks templates.
+Mail Magic is a small TypeScript HTTP service that:
 
-## Features
+- stores transactional email templates and sends transactional messages (authenticated API)
+- stores “contact form” templates and accepts public form submissions (unauthenticated endpoint)
+- renders mail with Nunjucks, and delivers via Nodemailer
 
-- Upload, store, and send templated email content through a JSON API
-- Preprocess template assets with `@technomoron/unyuck` before persisting
-- Nodemailer transport configuration driven by environment variables
-- SQLite-backed data models for domains, users, forms, and templates
-- Type-safe configuration loader powered by `@technomoron/env-loader`
-- Bundled admin UI (placeholder) served at the root path `/`
+This README is intended to be “enough to run and operate the service”. For exact request/response shapes and every
+endpoint, use the OpenAPI spec described in **Swagger / OpenAPI** below.
 
-## Getting Started
+## Contents
 
-1. Clone the repository: `git clone git@github.com:technomoron/mail-magic.git`
-2. Install dependencies: `npm install`
-3. Create your environment file: copy `.env-dist` to `.env` and adjust values
-4. Populate the config directory (defaults to `./data/`; see `config-example/` for a reference layout). You can point
-   `CONFIG_PATH` at `./config` to use the bundled sample data.
-5. Build the project: `npm run build`
-6. Start the API server: `npm run start`
+- What You Get
+- Install
+- Quick Start
+- Concepts
+- Configuration
+- Swagger / OpenAPI
+- API Usage (Examples)
+- Public Form Endpoint Contract
+- Assets
+- Security Notes
+- Development (Repo)
 
-During development you can run `npm run dev` for a watch mode that recompiles on change and restarts via `nodemon`.
+## What You Get
+
+- REST API built on `@technomoron/api-server-base`
+- SQLite + Sequelize persistence by default (configurable)
+- Nunjucks templating with optional HTML autoescape (`AUTOESCAPE_HTML=true` by default)
+- Config tree on disk (`CONFIG_PATH`) for per-domain templates and assets
+- Recipient allowlist so public forms can route to named recipients without exposing emails
+- Optional anti-abuse controls on the public form endpoint (rate limiting, attachment limits, CAPTCHA)
+
+## Install
+
+```bash
+npm install @technomoron/mail-magic
+```
+
+The package ships a `mail-magic` CLI that loads a `.env` file and starts the server.
+
+## Quick Start
+
+### 1. Create a `.env`
+
+Start from the repo’s `.env-dist` and set at least:
+
+```ini
+# REQUIRED. Keep stable: used to HMAC API tokens before DB lookup.
+API_TOKEN_PEPPER=change-me-please-use-a-long-random-string
+
+CONFIG_PATH=./data
+
+API_HOST=127.0.0.1
+API_PORT=3776
+API_BASE_PATH=/api
+API_URL=http://127.0.0.1:3776
+
+SMTP_HOST=127.0.0.1
+SMTP_PORT=1025
+SMTP_SECURE=false
+SMTP_TLS_REJECT=false
+```
+
+### 2. Create a minimal config directory
+
+`CONFIG_PATH` points at a directory containing `init-data.json` plus per-domain subfolders:
+
+```text
+data/
+  init-data.json
+  example.test/
+    assets/
+      images/logo.png
+    tx-template/
+      welcome.njk
+    form-template/
+      contact.njk
+```
+
+Assets referenced via `asset('...')` must live under:
+
+`<CONFIG_PATH>/<domain>/assets/...`
+
+### 3. Start the server
+
+```bash
+mail-magic --env ./.env
+```
+
+## Concepts
+
+### Domain-first config layout
+
+Mail Magic treats each domain as a root for templates and assets on disk:
+
+- Transactional templates: `<CONFIG_PATH>/<domain>/tx-template/...`
+- Form templates: `<CONFIG_PATH>/<domain>/form-template/...`
+- Public assets: `<CONFIG_PATH>/<domain>/assets/...`
+
+### Transactional vs form messages
+
+- Transactional:
+    - authenticated endpoints
+    - you choose recipient email(s) in the send request
+    - templates can use `_vars_` and `_rcpt_email_`
+- Form:
+    - authenticated endpoint to store/update the form template and configuration
+    - unauthenticated endpoint to submit the form
+    - public submissions are identified by a random `form_key`
+
+### `form_key` (public identifier)
+
+`POST /api/v1/form/template` returns a stable random `form_key`. Public submissions use that key as `_mm_form_key`.
+
+Treat `form_key` as sensitive: anyone who has it can submit to that form.
+
+### Recipient allowlist
+
+For “choose a recipient” forms, do not accept user-supplied emails. Instead:
+
+1. Configure recipients server-side with `POST /api/v1/form/recipient` (authenticated).
+2. In the public request, pass `_mm_recipients` containing recipient `idname`s.
+
+The server resolves those `idname`s to real email addresses using the allowlist.
 
 ## Configuration
 
-- **Environment variables** are defined in `src/store/envloader.ts`. Important settings include SMTP credentials, API
-  host/port, the config directory path, database options, and `ADMIN_ENABLED`/`ADMIN_APP_PATH` to control the admin
-  UI/API.
-- **Config directory** (`CONFIG_PATH`) contains JSON seed data (`init-data.json`), optional API key files, and template
-  assets. Each domain now lives directly under the config root (for example `data/example.com/form-template/…`). Use an
-  absolute path or a relative one like `../data` when you want the config outside the repo. Review `config-example/` for
-  the recommended layout, in particular the `form-template/` and `tx-template/` folders used for compiled Nunjucks
-  templates.
-- **Database** defaults to SQLite (`maildata.db`). You can switch dialects by updating the environment options if your
-  deployment requires another database.
-- **Uploads** default to `<CONFIG_PATH>/<domain>/uploads` via `UPLOAD_PATH=./{domain}/uploads`. Set a fixed path if you
-  prefer a shared upload directory.
+Mail Magic is configured via environment variables plus the `CONFIG_PATH` directory.
+
+The full set of environment variables is documented in the repository’s `.env-dist`.
+
+Commonly used variables:
+
+- `API_HOST`, `API_PORT`, `API_URL`
+- `API_BASE_PATH` (default `/api`)
+- `CONFIG_PATH` (default `./data/`)
+- `ASSET_ROUTE` (default `/asset`)
+- `ASSET_PUBLIC_BASE` (optional public base URL for assets)
+- `AUTOESCAPE_HTML` (default `true`)
+- `UPLOAD_PATH`, `UPLOAD_MAX` (multipart uploads)
+- Public form anti-abuse:
+    - `FORM_RATE_LIMIT_WINDOW_SEC`, `FORM_RATE_LIMIT_MAX`
+    - `FORM_MAX_ATTACHMENTS`, `FORM_KEEP_UPLOADS`
+    - `FORM_CAPTCHA_PROVIDER`, `FORM_CAPTCHA_SECRET`, `FORM_CAPTCHA_REQUIRED`
+- Swagger/OpenAPI:
+    - `SWAGGER_ENABLED`, `SWAGGER_PATH`
+
+## Swagger / OpenAPI
+
+Mail Magic ships an OpenAPI JSON spec and can expose it at runtime.
+
+Packaged spec (on disk):
+
+- `node_modules/@technomoron/mail-magic/docs/swagger/openapi.json`
+
+Runtime spec endpoint:
+
+- set `SWAGGER_ENABLED=true`
+- optionally set `SWAGGER_PATH` (defaults to `<API_BASE_PATH>/swagger`, typically `/api/swagger`)
+- fetch the JSON from that endpoint and feed it to Swagger UI / Postman / Insomnia
+
+This spec is the canonical reference for:
+
+- the exact route list
+- request/response bodies
+- status codes and error shapes
+
+## API Usage (Examples)
+
+All authenticated routes require:
+
+```text
+Authorization: Bearer apikey-<user_token>
+```
+
+Tokens are stored as `HMAC-SHA256(token, API_TOKEN_PEPPER)` in the DB. You can seed a plaintext `token` in
+`init-data.json`; it will be HMACed on import and the plaintext cleared.
+
+### Transactional: store template (authenticated)
+
+```bash
+curl -X POST http://localhost:3776/api/v1/tx/template \
+  -H "Authorization: Bearer apikey-<token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "domain": "example.test",
+    "name": "welcome",
+    "sender": "Example <noreply@example.test>",
+    "subject": "Welcome",
+    "locale": "en",
+    "template": "<p>Hi {{ _vars_.first_name }}</p>"
+  }'
+```
+
+### Transactional: send message (authenticated)
+
+```bash
+curl -X POST http://localhost:3776/api/v1/tx/message \
+  -H "Authorization: Bearer apikey-<token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "domain": "example.test",
+    "name": "welcome",
+    "locale": "en",
+    "rcpt": ["person@example.test"],
+    "vars": { "first_name": "Ada" }
+  }'
+```
+
+### Forms: store form template (authenticated)
+
+This returns `data.form_key` which is used by the public endpoint.
+
+```bash
+curl -X POST http://localhost:3776/api/v1/form/template \
+  -H "Authorization: Bearer apikey-<token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "domain": "example.test",
+    "idname": "contact",
+    "sender": "Example Forms <forms@example.test>",
+    "recipient": "owner@example.test",
+    "subject": "New contact form submission",
+    "locale": "en",
+    "template": "<p>Contact from {{ _fields_.name }} ({{ _fields_.email }})</p>"
+  }'
+```
+
+### Forms: submit (public endpoint, no auth)
+
+```bash
+curl -X POST http://localhost:3776/api/v1/form/message \
+  -H "Content-Type: application/json" \
+  -d '{
+    "_mm_form_key": "<form_key from the template response>",
+    "name": "Kai",
+    "email": "kai@example.test",
+    "message": "Hello"
+  }'
+```
+
+## Public Form Endpoint Contract
+
+Endpoint:
+
+- `POST /api/v1/form/message` (no auth)
+
+Required system fields:
+
+- `_mm_form_key` (string)
+
+Optional system fields:
+
+- `_mm_locale` (string)
+- `_mm_recipients` (string[] or comma-separated string)
+
+CAPTCHA token fields (accepted as-is, provider-native):
+
+- `cf-turnstile-response`
+- `h-captcha-response`
+- `g-recaptcha-response`
+- `captcha`
+
+Attachments (multipart):
 
-When `DB_AUTO_RELOAD` is enabled the service watches `init-data.json` and refreshes templates and forms without a
-restart.
+- attachment field names must start with `_mm_file` (example: `_mm_file1`, `_mm_file2`)
+- the server enforces `FORM_MAX_ATTACHMENTS` (and `UPLOAD_MAX` per file)
 
-### Admin UI
+All other non-`_mm_*` fields are treated as user fields and are exposed to templates as `_fields_` (optionally filtered
+by the form template’s `allowed_fields` setting).
 
-The server mounts the admin UI at `/` only when `ADMIN_ENABLED` is true and the `@technomoron/mail-magic-admin` package
-is installed. You can point `ADMIN_APP_PATH` at a dist folder (or its parent) to override the package-provided build.
-The admin API module is loaded from the admin package as well. This is a placeholder Vue app today, but it is already
-wired so future admin features can live there without changing the server routing.
+## Assets
 
-### Template assets and inline resources
+Templates may reference assets with `asset('path')`:
 
-- Keep any non-inline files (images, attachments, etc.) under `<CONFIG_PATH>/<domain>/assets`. Mail Magic rewrites
-  `asset('logo.png')` using `ASSET_ROUTE` (default `/asset`) and a base URL from `ASSET_PUBLIC_BASE` (or `API_URL` if
-  unset). The default output looks like `http://localhost:3776/asset/<domain>/logo.png`.
-- Pass `true` as the second argument when you want to embed a file as an inline CID attachment:
-  `asset('logo.png', true)` stores the file in Nodemailer and rewrites the HTML to reference `cid:logo.png`.
-- Avoid mixing template-type folders for assets; everything that should be linked externally belongs in the shared
-  `<domain>/assets` tree so it can be served for both form and transactional templates.
+- `asset('images/logo.png')` rewrites to a public URL under `ASSET_ROUTE` (default `/asset`)
+- `asset('images/logo.png', true)` embeds as a CID attachment
 
-## API Overview
+All assets must live under:
 
-| Method | Path                | Description                                   |
-| ------ | ------------------- | --------------------------------------------- |
-| POST   | `/v1/tx/template`   | Store or update a transactional mail template |
-| POST   | `/v1/tx/message`    | Render and send a stored transactional mail   |
-| POST   | `/v1/form/template` | Store or update a form submission template    |
-| POST   | `/v1/form/message`  | Submit a form payload and deliver the email   |
+`<CONFIG_PATH>/<domain>/assets/...`
 
-All routes are mounted under `API_BASE_PATH` (default `/api`), so the full path is typically `/api/v1/...`.
+## Security Notes
 
-All authenticated routes expect an API token associated with a configured user. Attachments can be uploaded alongside
-the `/v1/tx/message` request and are forwarded by Nodemailer.
+If you expose `POST /api/v1/form/message` publicly, read:
 
-## Available Scripts
+- `docs/form-security.md` (in this package) for the contract and operational hardening guidance
 
-- `npm run dev` – Start the API server in watch mode
-- `npm run build` – Compile TypeScript to the `dist/` directory
-- `npm run start` – Launch the compiled server from `dist/`
-- `npm run lint` – Lint the project with ESLint
-- `npm run format` – Apply ESLint autofixes followed by Prettier formatting
-- `npm run cleanbuild` – Clean, lint, format, and rebuild the project
+At a minimum:
 
-## Repository & Support
+- treat `form_key` as sensitive
+- keep recipient routing server-side (`_mm_recipients` idnames only)
+- set conservative `UPLOAD_MAX` and `FORM_MAX_ATTACHMENTS` if you enable uploads
+- use a real edge rate limiter/WAF in front of the public endpoint
 
-- Repository: https://github.com/technomoron/mail-magic
-- Issues: https://github.com/technomoron/mail-magic/issues
+## Development (Repo)
 
-## License
+In this repository, `pnpm` is the preferred package manager:
 
-This project is released under the MIT License. See the [LICENSE](LICENSE) file for details.
+```bash
+pnpm install
+pnpm -w --filter @technomoron/mail-magic dev
+pnpm -w --filter @technomoron/mail-magic test
+pnpm -w --filter @technomoron/mail-magic cleanbuild
+```
 
-## Copyright
+Documentation:
 
-Copyright (c) 2025 Bjørn Erik Jacobsen. All rights reserved.
+- `packages/mail-magic/docs/tutorial.md` is a hands-on config walkthrough.
+- `packages/mail-magic/docs/form-security.md` covers the public form endpoint contract and recommended mitigations.

package/dist/api/assets.js

@@ -3,64 +3,17 @@ import path from 'path';
 import { ApiError, ApiModule } from '@technomoron/api-server-base';
 import { api_form } from '../models/form.js';
 import { api_txmail } from '../models/txmail.js';
-import { decodeComponent, sendFileAsync } from '../util.js';
+import { SEGMENT_PATTERN, normalizeSubdir } from '../util/paths.js';
+import { moveUploadedFiles } from '../util/uploads.js';
+import { decodeComponent, getBodyValue, sendFileAsync } from '../util.js';
 import { assert_domain_and_user } from './auth.js';
 const DOMAIN_PATTERN = /^[a-z0-9][a-z0-9._-]*$/i;
-const SEGMENT_PATTERN = /^[a-zA-Z0-9._-]+$/;
 export class AssetAPI extends ApiModule {
-    getBodyValue(body, ...keys) {
-        for (const key of keys) {
-            const value = body[key];
-            if (Array.isArray(value) && value.length > 0) {
-                return String(value[0]);
-            }
-            if (value !== undefined && value !== null) {
-                return String(value);
-            }
-        }
-        return '';
-    }
-    normalizeSubdir(value) {
-        if (!value) {
-            return '';
-        }
-        const cleaned = value.replace(/\\/g, '/').replace(/^\/+/, '').replace(/\/+$/, '');
-        if (!cleaned) {
-            return '';
-        }
-        const segments = cleaned.split('/').filter(Boolean);
-        for (const segment of segments) {
-            if (!SEGMENT_PATTERN.test(segment)) {
-                throw new ApiError({ code: 400, message: `Invalid path segment "${segment}"` });
-            }
-        }
-        return path.join(...segments);
-    }
-    async moveUploadedFiles(files, targetDir) {
-        await fs.promises.mkdir(targetDir, { recursive: true });
-        for (const file of files) {
-            const filename = path.basename(file.originalname || '');
-            if (!filename || !SEGMENT_PATTERN.test(filename)) {
-                throw new ApiError({ code: 400, message: `Invalid filename "${file.originalname}"` });
-            }
-            const destination = path.join(targetDir, filename);
-            if (destination === file.path) {
-                continue;
-            }
-            try {
-                await fs.promises.rename(file.path, destination);
-            }
-            catch {
-                await fs.promises.copyFile(file.path, destination);
-                await fs.promises.unlink(file.path);
-            }
-        }
-    }
     async resolveTemplateDir(apireq) {
         const body = apireq.req.body ?? {};
-        const templateTypeRaw = this.getBodyValue(body, 'templateType', 'template_type', 'type');
-        const templateName = this.getBodyValue(body, 'template', 'name', 'idname', 'formid');
-        const locale = this.getBodyValue(body, 'locale');
+        const templateTypeRaw = getBodyValue(body, 'templateType', 'template_type', 'type');
+        const templateName = getBodyValue(body, 'template', 'name', 'idname', 'formid');
+        const locale = getBodyValue(body, 'locale');
         if (!templateTypeRaw) {
             throw new ApiError({ code: 400, message: 'Missing templateType for template asset upload' });
         }
@@ -115,8 +68,8 @@ export class AssetAPI extends ApiModule {
             throw new ApiError({ code: 400, message: 'No files uploaded' });
         }
         const body = apireq.req.body ?? {};
-        const subdir = this.normalizeSubdir(this.getBodyValue(body, 'path', 'dir'));
-        const templateType = this.getBodyValue(body, 'templateType', 'template_type', 'type');
+        const subdir = normalizeSubdir(getBodyValue(body, 'path', 'dir'));
+        const templateType = getBodyValue(body, 'templateType', 'template_type', 'type');
         let targetRoot;
         if (templateType) {
             targetRoot = await this.resolveTemplateDir(apireq);
@@ -129,7 +82,7 @@ export class AssetAPI extends ApiModule {
         if (candidate !== targetRoot && !candidate.startsWith(normalizedRoot)) {
             throw new ApiError({ code: 400, message: 'Invalid asset target path' });
         }
-        await this.moveUploadedFiles(rawFiles, candidate);
+        await moveUploadedFiles(rawFiles, candidate);
         return [200, { Status: 'OK' }];
     }
     defineRoutes() {

package/dist/api/auth.js

@@ -1,18 +1,7 @@
 import { ApiError } from '@technomoron/api-server-base';
 import { api_domain } from '../models/domain.js';
 import { api_user } from '../models/user.js';
-function getBodyValue(body, ...keys) {
-    for (const key of keys) {
-        const value = body[key];
-        if (Array.isArray(value) && value.length > 0) {
-            return String(value[0]);
-        }
-        if (value !== undefined && value !== null) {
-            return String(value);
-        }
-    }
-    return '';
-}
+import { getBodyValue } from '../util.js';
 export async function assert_domain_and_user(apireq) {
     const body = apireq.req.body ?? {};
     const domain = getBodyValue(body, 'domain');

package/dist/api/form-replyto.js

@@ -0,0 +1,44 @@
+import emailAddresses from 'email-addresses';
+function getFirstStringField(body, key) {
+    const value = body[key];
+    if (Array.isArray(value) && value.length > 0) {
+        return String(value[0] ?? '');
+    }
+    if (value !== undefined && value !== null) {
+        return String(value);
+    }
+    return '';
+}
+function sanitizeHeaderValue(value, maxLen) {
+    const trimmed = String(value ?? '').trim();
+    if (!trimmed) {
+        return '';
+    }
+    // Prevent header injection.
+    if (/[\r\n]/.test(trimmed)) {
+        return '';
+    }
+    return trimmed.slice(0, maxLen);
+}
+export function extractReplyToFromSubmission(body) {
+    const emailRaw = sanitizeHeaderValue(getFirstStringField(body, 'email'), 320);
+    if (!emailRaw) {
+        return undefined;
+    }
+    const parsed = emailAddresses.parseOneAddress(emailRaw);
+    if (!parsed) {
+        return undefined;
+    }
+    const address = sanitizeHeaderValue(parsed?.address, 320);
+    if (!address) {
+        return undefined;
+    }
+    // Prefer a single "name" field, otherwise compose from first_name/last_name.
+    let name = sanitizeHeaderValue(getFirstStringField(body, 'name'), 200);
+    if (!name) {
+        const first = sanitizeHeaderValue(getFirstStringField(body, 'first_name'), 100);
+        const last = sanitizeHeaderValue(getFirstStringField(body, 'last_name'), 100);
+        name = sanitizeHeaderValue(`${first}${first && last ? ' ' : ''}${last}`, 200);
+    }
+    return name ? { name, address } : address;
+}