scrapme-flow 0.0.0-stage → 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 lastseeds
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,127 @@
-# Temporary Holding Version
+# scrapme-flow
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Scraping flows for Node apps. This repository has two separate things: the **`scrapme-flow` npm library** (root,
+`src/`) and the **monitor** app ([`monitor/`](monitor/README.md)) that shows the runs — deployable on CapRover.
+
+A flow is a **`*.flow.ts` file** exporting `new Flow({...})`: a list of steps, each running a small command class (scrape a page, parse it, save it, mail it…). Every command has its own **retries and
+timeout**. `scrapme.flow(flow, input)` puts a run on a **pg-boss queue** (Postgres) and returns at once; a worker runs
+it, and a failed run is run again by pg-boss. The progress of every run is kept in **CouchDB** and shown by a
+separate **monitor** app.
+
+```ts
+import { Container } from 'inversify';
+import { Scrapme } from 'scrapme-flow';
+import { helloFlow } from './hello.flow.js';
+
+const scrapme = Scrapme.create({
+  container: appContainer,                       // your inversify container: commands get your services from it
+  postgres: { connectionString: 'postgres://scrapme:scrapme@localhost:5434/scrapme' },
+  couch: { url: 'http://localhost:5984', user: 'admin', password: secret },
+});
+await scrapme.start();                           // connects pg-boss
+
+const runId = await scrapme.flow(helloFlow, { name: 'Ada' });   // queues a run; this process runs it
+const run = await scrapme.waitFor(runId);                        // optional: wait for the end
+console.log(run.status, run.steps[0]?.output);
+await scrapme.stop();
+```
+
+scrapme is a library: it reads **no environment variables** — the app passes the connection data of everything it
+needs. `postgres` and `couch` are required; `scrapAnything`, `mail` and `lighty` only when a flow uses them
+(without them `ScrapePageCommand` / `SendMailCommand` fail with `NOT_CONFIGURED`, errors are only logged).
+Flows are not registered anywhere: you pass the flow to `flow()`.
+
+## Install
+
+```bash
+npm install scrapme-flow inversify reflect-metadata
+```
+
+Node 26+, ESM. `inversify` and `reflect-metadata` are peer dependencies (one copy for your app and scrapme).
+
+## Hello world
+
+```ts
+// print-message.command.ts
+@injectable()
+export class PrintMessageCommand implements Command<{ text: string }, string> {
+  public constructor(@inject(ConsolePrinter) private readonly printer: ConsolePrinter) {}
+  public async execute(input: { text: string }) {
+    this.printer.print(input.text);
+    return input.text;
+  }
+}
+
+// hello.flow.ts
+export const helloFlow = new Flow({
+  name: 'hello',
+  params: { name: { default: 'World' } },
+  steps: [{ command: PrintMessageCommand, input: { text: (s) => 'Hello, ' + s.name + '!' } }],
+});
+```
+
+- **State:** each step's output is stored under its `key` (default: the command's class name). `input` picks from it
+  (`{ from: 'page.html' }`), adds literals (`{ value }`) or computes in plain JS (`(s) => 'doc:' + s.url`);
+  `inputFrom` passes one value; neither = the whole state. Format: [docs/FLOWS.md](docs/FLOWS.md).
+- **Commands:** `@injectable()` classes, nothing to bind — scrapme's container is a child of yours (`autobind`), so
+  `@inject(YourService)` just works. Built in: `ScrapePageCommand` (scrap-anything), `SaveDocumentCommand` (CouchDB),
+  `SendMailCommand` (SMTP).
+
+## Retries and timeouts
+
+| level | what | settings | default |
+|---|---|---|---|
+| command | attempts of one step inside a flow attempt, each with a timeout (`ctx.signal` aborts) | step `retries`, `timeoutMs`, `retryDelayMs` > command `options` > `defaults` | 3 retries, 60 s |
+| flow | the pg-boss job: run the whole flow again after a delay (backoff), expire a too long attempt | flow `retries`, `retryDelaySeconds`, `retryBackoff`, `timeoutSeconds` > `queue` | 3 retries, 30 s, backoff, 1 h |
+
+When a step runs out of attempts, the flow attempt fails and pg-boss runs the flow again from its first step.
+`PermanentError` (or a missing input value) is not retried. A run that fails for good is reported once to
+LightyErrors (`lighty.dsn`).
+
+## Monitor
+
+The library has no UI. The runs are documents in CouchDB (`couch.runsDb`, `run:<id>`), and the **scrapme monitor**
+— a separate app in [`monitor/`](monitor/README.md) — shows them: runs with status counts and a flow filter, each
+run's steps (which one is running, which failed and why), every flow attempt and command attempt. Run it locally
+(`./scripts/monitor.sh`) or deploy it to CapRover. Apps that only read runs can use `scrapme-flow/models` (the
+document types and status constants, no Node code).
+
+## API
+
+| | |
+|---|---|
+| `Scrapme.create(options)` | `container?`, `postgres`, `couch`, `queue?`, `scrapAnything?`, `mail?`, `lighty?`, `defaults?`, `logLevel?` |
+| `start({ worker? })` | connect pg-boss. `worker: false` = this process only queues runs |
+| `flow(flow, input, { runId?, startAfter?, priority? })` | queue a run → run id (UUID); starts the flow's worker here on first use |
+| `work(flow)` | run this flow's queued runs in this process without queueing one (a worker-only process) |
+| `getRun(id)`, `listRuns(limit)` | run documents: status, steps with outputs, attempts, errors |
+| `waitFor(id, { timeoutMs? })` | until succeeded / failed (`RunWaitTimeoutError`) |
+| `stop()` | stop the workers and the queue connection |
+
+## This repository
+
+```bash
+nvm use && npm install
+./scripts/up.sh                      # everything at once: restarts scrap-anything, sentry-clone, the containers, the
+                                     # monitor (background) and runs hello; ./scripts/up.sh down stops it all
+./scripts/deps.sh                    # own containers: Postgres :5434, CouchDB :5985, Mailpit :1026/:8026; checks scrap-anything
+./scripts/monitor.sh                 # terminal 1, leave it open: the monitor on http://localhost:3100
+./scripts/example.sh hello Ada       # terminal 2: the hello flow, then look at it in the monitor
+./scripts/example.sh biznesradar DOM-DEVELOPMENT   # scrape → cheerio → CouchDB → e-mail (needs scrap-anything)
+```
+
+The examples read `.env` (see `.env.example`) in `examples/example-services.ts` and pass the values to
+`Scrapme.create`. Examples have no tests. [`example-external/`](example-external/README.md) is the hello example as a
+standalone app that installs `scrapme-flow` from npm.
+
+```bash
+npm test                    # every src/**/*.test.ts — no docker / internet / .env (fake queue, CouchDB, SMTP, Lighty)
+npm run build               # dist/ (tsc)
+cd monitor && npm test      # the monitor's tests
+```
+
+Code layout and rules: `AGENTS.md`. Decisions: `docs/adr/`.
+
+## License
+
+MIT

package/dist/src/commands/commands.models.d.ts

@@ -0,0 +1,41 @@
+/**
+ * Input of ScrapePageCommand. scrap-anything decides how long to wait (it always waits until the page —
+ * SPA or not — has rendered) and retries on its own, so there is nothing to tune per page.
+ */
+export interface ScrapePageInput {
+    url: string;
+    /** Extra request headers for the target (e.g. Accept-Language). */
+    headers?: Record<string, string>;
+    /** 0–100, higher runs first on the server. */
+    priority?: number;
+    /** Accept HTTP 4xx answers of the target as a result (default: PermanentError HTTP_STATUS). */
+    allowHttpErrors?: boolean;
+}
+/** Output of ScrapePageCommand. */
+export interface ScrapedPage {
+    jobId: string;
+    url: string;
+    finalUrl: string;
+    httpStatus: number;
+    title: string | null;
+    contentType: string;
+    html: string;
+    durationMs: number;
+    fetchedAt: string;
+}
+/** Input of SaveDocumentCommand. */
+export interface SaveDocumentInput {
+    /** Document id (same id = new revision of the same document). */
+    id: string;
+    document: Record<string, unknown>;
+    /** Default couch.db. */
+    db?: string;
+}
+/** Input of SendMailCommand. */
+export interface SendMailInput {
+    /** Default MAIL_TO. */
+    to?: string | string[];
+    subject: string;
+    text: string;
+    html?: string;
+}

package/dist/src/commands/commands.models.js

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=commands.models.js.map

package/dist/src/commands/commands.models.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"commands.models.js","sourceRoot":"","sources":["../../../src/commands/commands.models.ts"],"names":[],"mappings":""}

package/dist/src/commands/save-document.command.d.ts

@@ -0,0 +1,9 @@
+import { type Command, type CommandContext } from '../flow/flow.models.js';
+import type { DocumentStore, SavedDocument } from '../storage/storage.models.js';
+import type { SaveDocumentInput } from './commands.models.js';
+/** Upserts a document into CouchDB (couch.db or input.db). Idempotent: a retry writes the same id again. */
+export declare class SaveDocumentCommand implements Command<SaveDocumentInput, SavedDocument> {
+    private readonly store;
+    constructor(store: DocumentStore);
+    execute(input: SaveDocumentInput, ctx: CommandContext): Promise<SavedDocument>;
+}

package/dist/src/commands/save-document.command.js

@@ -0,0 +1,33 @@
+var __decorate = (this && this.__decorate) || function (decorators, target, key, desc) {
+    var c = arguments.length, r = c < 3 ? target : desc === null ? desc = Object.getOwnPropertyDescriptor(target, key) : desc, d;
+    if (typeof Reflect === "object" && typeof Reflect.decorate === "function") r = Reflect.decorate(decorators, target, key, desc);
+    else for (var i = decorators.length - 1; i >= 0; i--) if (d = decorators[i]) r = (c < 3 ? d(r) : c > 3 ? d(target, key, r) : d(target, key)) || r;
+    return c > 3 && r && Object.defineProperty(target, key, r), r;
+};
+var __param = (this && this.__param) || function (paramIndex, decorator) {
+    return function (target, key) { decorator(target, key, paramIndex); }
+};
+import { inject, injectable } from 'inversify';
+import { TYPES } from '../di/types.js';
+import { PermanentError } from '../flow/flow-errors.js';
+import { ERROR_CODE } from '../flow/flow.models.js';
+/** Upserts a document into CouchDB (couch.db or input.db). Idempotent: a retry writes the same id again. */
+let SaveDocumentCommand = class SaveDocumentCommand {
+    store;
+    constructor(store) {
+        this.store = store;
+    }
+    async execute(input, ctx) {
+        if (!input?.id || !input.document)
+            throw new PermanentError('SaveDocumentCommand needs { id, document }', { code: ERROR_CODE.BAD_INPUT });
+        const saved = await this.store.save(input.id, input.document, input.db ? { db: input.db } : {});
+        ctx.logger.info(saved.created ? 'document created' : 'document updated', { db: saved.db, id: saved.id, rev: saved.rev, url: saved.url });
+        return saved;
+    }
+};
+SaveDocumentCommand = __decorate([
+    injectable(),
+    __param(0, inject(TYPES.DocumentStore))
+], SaveDocumentCommand);
+export { SaveDocumentCommand };
+//# sourceMappingURL=save-document.command.js.map

package/dist/src/commands/save-document.command.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"save-document.command.js","sourceRoot":"","sources":["../../../src/commands/save-document.command.ts"],"names":[],"mappings":";;;;;;;;;AAAA,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAC/C,OAAO,EAAE,KAAK,EAAE,MAAM,gBAAgB,CAAC;AACvC,OAAO,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AACxD,OAAO,EAAE,UAAU,EAAqC,MAAM,wBAAwB,CAAC;AAIvF,4GAA4G;AAErG,IAAM,mBAAmB,GAAzB,MAAM,mBAAmB;IACmC;IAAjE,YAAiE,KAAoB;QAApB,UAAK,GAAL,KAAK,CAAe;IAAG,CAAC;IAElF,KAAK,CAAC,OAAO,CAAC,KAAwB,EAAE,GAAmB;QAChE,IAAI,CAAC,KAAK,EAAE,EAAE,IAAI,CAAC,KAAK,CAAC,QAAQ;YAAE,MAAM,IAAI,cAAc,CAAC,4CAA4C,EAAE,EAAE,IAAI,EAAE,UAAU,CAAC,SAAS,EAAE,CAAC,CAAC;QAC1I,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,EAAE,KAAK,CAAC,QAAQ,EAAE,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;QAChG,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,kBAAkB,CAAC,CAAC,CAAC,kBAAkB,EAAE,EAAE,EAAE,EAAE,KAAK,CAAC,EAAE,EAAE,EAAE,EAAE,KAAK,CAAC,EAAE,EAAE,GAAG,EAAE,KAAK,CAAC,GAAG,EAAE,GAAG,EAAE,KAAK,CAAC,GAAG,EAAE,CAAC,CAAC;QACzI,OAAO,KAAK,CAAC;IACf,CAAC;CACF,CAAA;AATY,mBAAmB;IAD/B,UAAU,EAAE;IAES,WAAA,MAAM,CAAC,KAAK,CAAC,aAAa,CAAC,CAAA;GADpC,mBAAmB,CAS/B"}

package/dist/src/commands/scrape-page.command.d.ts

@@ -0,0 +1,20 @@
+import { type ScrapAnythingClient } from 'scrap-anything-client';
+import type { ScrapmeConfig } from '../config/config.models.js';
+import { type Command, type CommandContext, type CommandOptions } from '../flow/flow.models.js';
+import type { ScrapedPage, ScrapePageInput } from './commands.models.js';
+/**
+ * Renders a page with scrap-anything: schedules a job (createJob), waits for it (collect — the API is
+ * asynchronous, a queue with its own retries behind it) and returns the HTML. One attempt = one server job.
+ * When the attempt is aborted (step timeout, cancelled run) the server job is cancelled too. Retries follow the
+ * client's `retryable` (TLS / invalid target / 4xx from the API = permanent).
+ */
+export declare class ScrapePageCommand implements Command<ScrapePageInput, ScrapedPage> {
+    private readonly client;
+    private static readonly FIRST_CLIENT_ERROR;
+    readonly options: CommandOptions;
+    constructor(client: ScrapAnythingClient | undefined, config: ScrapmeConfig);
+    execute(input: ScrapePageInput, ctx: CommandContext): Promise<ScrapedPage>;
+    isRetryable(error: unknown): boolean;
+    /** Wait for the job; an abort of the attempt cancels it on the server (the next poll then ends the wait). */
+    private collect;
+}

package/dist/src/commands/scrape-page.command.js

@@ -0,0 +1,93 @@
+var __decorate = (this && this.__decorate) || function (decorators, target, key, desc) {
+    var c = arguments.length, r = c < 3 ? target : desc === null ? desc = Object.getOwnPropertyDescriptor(target, key) : desc, d;
+    if (typeof Reflect === "object" && typeof Reflect.decorate === "function") r = Reflect.decorate(decorators, target, key, desc);
+    else for (var i = decorators.length - 1; i >= 0; i--) if (d = decorators[i]) r = (c < 3 ? d(r) : c > 3 ? d(target, key, r) : d(target, key)) || r;
+    return c > 3 && r && Object.defineProperty(target, key, r), r;
+};
+var __param = (this && this.__param) || function (paramIndex, decorator) {
+    return function (target, key) { decorator(target, key, paramIndex); }
+};
+var ScrapePageCommand_1;
+import { inject, injectable, optional } from 'inversify';
+import { RESULT_KIND, ScrapAnythingError } from 'scrap-anything-client';
+import { TYPES } from '../di/types.js';
+import { FlowErrors, NotConfiguredError, PermanentError } from '../flow/flow-errors.js';
+import { ERROR_CODE } from '../flow/flow.models.js';
+/**
+ * Renders a page with scrap-anything: schedules a job (createJob), waits for it (collect — the API is
+ * asynchronous, a queue with its own retries behind it) and returns the HTML. One attempt = one server job.
+ * When the attempt is aborted (step timeout, cancelled run) the server job is cancelled too. Retries follow the
+ * client's `retryable` (TLS / invalid target / 4xx from the API = permanent).
+ */
+let ScrapePageCommand = class ScrapePageCommand {
+    static { ScrapePageCommand_1 = this; }
+    client;
+    static FIRST_CLIENT_ERROR = 400;
+    options;
+    constructor(client, config) {
+        this.client = client;
+        // waiting for the server job is the slow part: the attempt must outlive scrapAnything.waitTimeoutMs
+        const s = config.scrapAnything;
+        this.options = s ? { timeoutMs: s.stepTimeoutMs, retries: s.stepRetries } : { retries: 0 };
+    }
+    async execute(input, ctx) {
+        if (!this.client)
+            throw new NotConfiguredError('scrapAnything', 'ScrapePageCommand');
+        if (!input?.url)
+            throw new PermanentError('ScrapePageCommand needs { url }', { code: ERROR_CODE.BAD_INPUT });
+        const client = this.client;
+        const created = await client.createJob({
+            url: input.url,
+            ...(input.headers ? { headers: input.headers } : {}),
+            ...(input.priority !== undefined ? { priority: input.priority } : {}),
+        });
+        ctx.logger.info('scrape job scheduled', { jobId: created.jobId, url: input.url, status: created.status });
+        const res = await this.collect(client, created.jobId, ctx);
+        ctx.logger.info('scrape job done', { jobId: res.jobId, httpStatus: res.result.httpStatus, attempts: res.job.attempts, ms: res.result.durationMs });
+        if (res.result.kind !== RESULT_KIND.HTML || res.html === undefined) {
+            throw new PermanentError(`${input.url} is not an HTML page (${res.result.contentType})`, { code: ERROR_CODE.NOT_HTML });
+        }
+        if (res.result.httpStatus >= ScrapePageCommand_1.FIRST_CLIENT_ERROR && !input.allowHttpErrors) {
+            throw new PermanentError(`${input.url} answered HTTP ${res.result.httpStatus}`, { code: ERROR_CODE.HTTP_STATUS });
+        }
+        return {
+            jobId: res.jobId,
+            url: input.url,
+            finalUrl: res.result.finalUrl,
+            httpStatus: res.result.httpStatus,
+            title: res.result.title,
+            contentType: res.result.contentType,
+            html: res.html,
+            durationMs: res.result.durationMs,
+            fetchedAt: new Date().toISOString(),
+        };
+    }
+    isRetryable(error) {
+        return error instanceof ScrapAnythingError ? error.retryable : FlowErrors.isRetryable(error);
+    }
+    /** Wait for the job; an abort of the attempt cancels it on the server (the next poll then ends the wait). */
+    async collect(client, jobId, ctx) {
+        const cancel = () => {
+            ctx.logger.warn('attempt aborted, cancelling the scrape job', { jobId });
+            client.cancelJob(jobId).catch((error) => ctx.logger.warn('could not cancel the scrape job', { jobId, error }));
+        };
+        if (ctx.signal.aborted)
+            cancel();
+        else
+            ctx.signal.addEventListener('abort', cancel, { once: true });
+        try {
+            return await client.collect(jobId);
+        }
+        finally {
+            ctx.signal.removeEventListener('abort', cancel);
+        }
+    }
+};
+ScrapePageCommand = ScrapePageCommand_1 = __decorate([
+    injectable(),
+    __param(0, inject(TYPES.ScrapAnythingClient)),
+    __param(0, optional()),
+    __param(1, inject(TYPES.Config))
+], ScrapePageCommand);
+export { ScrapePageCommand };
+//# sourceMappingURL=scrape-page.command.js.map

package/dist/src/commands/scrape-page.command.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"scrape-page.command.js","sourceRoot":"","sources":["../../../src/commands/scrape-page.command.ts"],"names":[],"mappings":";;;;;;;;;;AAAA,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AACzD,OAAO,EAAE,WAAW,EAAE,kBAAkB,EAA+C,MAAM,uBAAuB,CAAC;AAErH,OAAO,EAAE,KAAK,EAAE,MAAM,gBAAgB,CAAC;AACvC,OAAO,EAAE,UAAU,EAAE,kBAAkB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AACxF,OAAO,EAAE,UAAU,EAA0D,MAAM,wBAAwB,CAAC;AAG5G;;;;;GAKG;AAEI,IAAM,iBAAiB,GAAvB,MAAM,iBAAiB;;IAKsC;IAJ1D,MAAM,CAAU,kBAAkB,GAAG,GAAG,CAAC;IACjC,OAAO,CAAiB;IAExC,YACkE,MAAuC,EACjF,MAAqB;QADqB,WAAM,GAAN,MAAM,CAAiC;QAGvG,oGAAoG;QACpG,MAAM,CAAC,GAAG,MAAM,CAAC,aAAa,CAAC;QAC/B,IAAI,CAAC,OAAO,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,CAAC,CAAC,aAAa,EAAE,OAAO,EAAE,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,CAAC;IAC7F,CAAC;IAEM,KAAK,CAAC,OAAO,CAAC,KAAsB,EAAE,GAAmB;QAC9D,IAAI,CAAC,IAAI,CAAC,MAAM;YAAE,MAAM,IAAI,kBAAkB,CAAC,eAAe,EAAE,mBAAmB,CAAC,CAAC;QACrF,IAAI,CAAC,KAAK,EAAE,GAAG;YAAE,MAAM,IAAI,cAAc,CAAC,iCAAiC,EAAE,EAAE,IAAI,EAAE,UAAU,CAAC,SAAS,EAAE,CAAC,CAAC;QAC7G,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;QAC3B,MAAM,OAAO,GAAG,MAAM,MAAM,CAAC,SAAS,CAAC;YACrC,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,KAAK,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACpD,GAAG,CAAC,KAAK,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SACtE,CAAC,CAAC;QACH,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,sBAAsB,EAAE,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,GAAG,EAAE,KAAK,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;QAC1G,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;QAC3D,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,iBAAiB,EAAE,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,UAAU,EAAE,GAAG,CAAC,MAAM,CAAC,UAAU,EAAE,QAAQ,EAAE,GAAG,CAAC,GAAG,CAAC,QAAQ,EAAE,EAAE,EAAE,GAAG,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC;QAEnJ,IAAI,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,WAAW,CAAC,IAAI,IAAI,GAAG,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;YACnE,MAAM,IAAI,cAAc,CAAC,GAAG,KAAK,CAAC,GAAG,yBAAyB,GAAG,CAAC,MAAM,CAAC,WAAW,GAAG,EAAE,EAAE,IAAI,EAAE,UAAU,CAAC,QAAQ,EAAE,CAAC,CAAC;QAC1H,CAAC;QACD,IAAI,GAAG,CAAC,MAAM,CAAC,UAAU,IAAI,mBAAiB,CAAC,kBAAkB,IAAI,CAAC,KAAK,CAAC,eAAe,EAAE,CAAC;YAC5F,MAAM,IAAI,cAAc,CAAC,GAAG,KAAK,CAAC,GAAG,kBAAkB,GAAG,CAAC,MAAM,CAAC,UAAU,EAAE,EAAE,EAAE,IAAI,EAAE,UAAU,CAAC,WAAW,EAAE,CAAC,CAAC;QACpH,CAAC;QACD,OAAO;YACL,KAAK,EAAE,GAAG,CAAC,KAAK;YAChB,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,QAAQ;YAC7B,UAAU,EAAE,GAAG,CAAC,MAAM,CAAC,UAAU;YACjC,KAAK,EAAE,GAAG,CAAC,MAAM,CAAC,KAAK;YACvB,WAAW,EAAE,GAAG,CAAC,MAAM,CAAC,WAAW;YACnC,IAAI,EAAE,GAAG,CAAC,IAAI;YACd,UAAU,EAAE,GAAG,CAAC,MAAM,CAAC,UAAU;YACjC,SAAS,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE;SACpC,CAAC;IACJ,CAAC;IAEM,WAAW,CAAC,KAAc;QAC/B,OAAO,KAAK,YAAY,kBAAkB,CAAC,CAAC,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,WAAW,CAAC,KAAK,CAAC,CAAC;IAC/F,CAAC;IAED,6GAA6G;IACrG,KAAK,CAAC,OAAO,CAAC,MAA2B,EAAE,KAAa,EAAE,GAAmB;QACnF,MAAM,MAAM,GAAG,GAAS,EAAE;YACxB,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,4CAA4C,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;YACzE,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,KAAc,EAAE,EAAE,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,iCAAiC,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QAC1H,CAAC,CAAC;QACF,IAAI,GAAG,CAAC,MAAM,CAAC,OAAO;YAAE,MAAM,EAAE,CAAC;;YAC5B,GAAG,CAAC,MAAM,CAAC,gBAAgB,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;QAClE,IAAI,CAAC;YACH,OAAO,MAAM,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;QACrC,CAAC;gBAAS,CAAC;YACT,GAAG,CAAC,MAAM,CAAC,mBAAmB,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;;AA9DU,iBAAiB;IAD7B,UAAU,EAAE;IAMR,WAAA,MAAM,CAAC,KAAK,CAAC,mBAAmB,CAAC,CAAA;IAAE,WAAA,QAAQ,EAAE,CAAA;IAC7C,WAAA,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAA;GANZ,iBAAiB,CA+D7B"}

package/dist/src/commands/send-mail.command.d.ts

@@ -0,0 +1,11 @@
+import type { ScrapmeConfig } from '../config/config.models.js';
+import { type Command, type CommandContext } from '../flow/flow.models.js';
+import type { Mailer, SentMail } from '../mail/mail.models.js';
+import type { SendMailInput } from './commands.models.js';
+/** Sends an e-mail (SMTP from Scrapme.create({ mail })). Recipients default to mail.to. Rejected addresses are not retried. */
+export declare class SendMailCommand implements Command<SendMailInput, SentMail> {
+    private readonly mailer;
+    private readonly config;
+    constructor(mailer: Mailer, config: ScrapmeConfig);
+    execute(input: SendMailInput, ctx: CommandContext): Promise<SentMail>;
+}

package/dist/src/commands/send-mail.command.js

@@ -0,0 +1,39 @@
+var __decorate = (this && this.__decorate) || function (decorators, target, key, desc) {
+    var c = arguments.length, r = c < 3 ? target : desc === null ? desc = Object.getOwnPropertyDescriptor(target, key) : desc, d;
+    if (typeof Reflect === "object" && typeof Reflect.decorate === "function") r = Reflect.decorate(decorators, target, key, desc);
+    else for (var i = decorators.length - 1; i >= 0; i--) if (d = decorators[i]) r = (c < 3 ? d(r) : c > 3 ? d(target, key, r) : d(target, key)) || r;
+    return c > 3 && r && Object.defineProperty(target, key, r), r;
+};
+var __param = (this && this.__param) || function (paramIndex, decorator) {
+    return function (target, key) { decorator(target, key, paramIndex); }
+};
+import { inject, injectable } from 'inversify';
+import { TYPES } from '../di/types.js';
+import { PermanentError } from '../flow/flow-errors.js';
+import { ERROR_CODE } from '../flow/flow.models.js';
+/** Sends an e-mail (SMTP from Scrapme.create({ mail })). Recipients default to mail.to. Rejected addresses are not retried. */
+let SendMailCommand = class SendMailCommand {
+    mailer;
+    config;
+    constructor(mailer, config) {
+        this.mailer = mailer;
+        this.config = config;
+    }
+    async execute(input, ctx) {
+        if (!input?.subject || !input.text)
+            throw new PermanentError('SendMailCommand needs { subject, text }', { code: ERROR_CODE.BAD_INPUT });
+        const to = input.to === undefined ? [...(this.config.mail?.to ?? [])] : Array.isArray(input.to) ? input.to : [input.to];
+        if (to.length === 0)
+            throw new PermanentError('no recipients (input.to or mail.to in Scrapme.create)', { code: ERROR_CODE.BAD_INPUT });
+        const sent = await this.mailer.send({ to, subject: input.subject, text: input.text, ...(input.html ? { html: input.html } : {}) });
+        ctx.logger.info('mail sent', { to, subject: input.subject, messageId: sent.messageId });
+        return sent;
+    }
+};
+SendMailCommand = __decorate([
+    injectable(),
+    __param(0, inject(TYPES.Mailer)),
+    __param(1, inject(TYPES.Config))
+], SendMailCommand);
+export { SendMailCommand };
+//# sourceMappingURL=send-mail.command.js.map

package/dist/src/commands/send-mail.command.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"send-mail.command.js","sourceRoot":"","sources":["../../../src/commands/send-mail.command.ts"],"names":[],"mappings":";;;;;;;;;AAAA,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAE/C,OAAO,EAAE,KAAK,EAAE,MAAM,gBAAgB,CAAC;AACvC,OAAO,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AACxD,OAAO,EAAE,UAAU,EAAqC,MAAM,wBAAwB,CAAC;AAIvF,+HAA+H;AAExH,IAAM,eAAe,GAArB,MAAM,eAAe;IAEe;IACA;IAFzC,YACyC,MAAc,EACd,MAAqB;QADrB,WAAM,GAAN,MAAM,CAAQ;QACd,WAAM,GAAN,MAAM,CAAe;IAC3D,CAAC;IAEG,KAAK,CAAC,OAAO,CAAC,KAAoB,EAAE,GAAmB;QAC5D,IAAI,CAAC,KAAK,EAAE,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI;YAAE,MAAM,IAAI,cAAc,CAAC,yCAAyC,EAAE,EAAE,IAAI,EAAE,UAAU,CAAC,SAAS,EAAE,CAAC,CAAC;QACxI,MAAM,EAAE,GAAG,KAAK,CAAC,EAAE,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACxH,IAAI,EAAE,CAAC,MAAM,KAAK,CAAC;YAAE,MAAM,IAAI,cAAc,CAAC,uDAAuD,EAAE,EAAE,IAAI,EAAE,UAAU,CAAC,SAAS,EAAE,CAAC,CAAC;QACvI,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,OAAO,EAAE,KAAK,CAAC,OAAO,EAAE,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC;QACnI,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,WAAW,EAAE,EAAE,EAAE,EAAE,OAAO,EAAE,KAAK,CAAC,OAAO,EAAE,SAAS,EAAE,IAAI,CAAC,SAAS,EAAE,CAAC,CAAC;QACxF,OAAO,IAAI,CAAC;IACd,CAAC;CACF,CAAA;AAdY,eAAe;IAD3B,UAAU,EAAE;IAGR,WAAA,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAA;IACpB,WAAA,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAA;GAHZ,eAAe,CAc3B"}

package/dist/src/config/config-builder.d.ts

@@ -0,0 +1,24 @@
+import type { ScrapmeConfig, ServiceOptions } from './config.models.js';
+/** The services given to Scrapme.create are invalid (thrown by Scrapme.create). */
+export declare class ConfigError extends Error {
+    constructor(message: string);
+}
+/** Checks the services an app gives to Scrapme.create (zod), fills in defaults and freezes the result. */
+export declare class ConfigBuilder {
+    /**
+     * CouchDB's own rule for database names (docs.couchdb.org, "PUT /{db}"). It is a rule about characters, so a
+     * regex is the clearest way to state it.
+     */
+    private static readonly COUCH_DB_NAME;
+    private static readonly STEP_TIMEOUT_MARGIN_MS;
+    private static readonly SMTP_PORT;
+    private static readonly SMTPS_PORT;
+    /** pg-boss schema names go into SQL as identifiers. */
+    private static readonly SQL_IDENTIFIER;
+    private static readonly QUEUE_PREFIX;
+    private static readonly ms;
+    private static readonly dbName;
+    private static readonly schema;
+    static build(options: ServiceOptions): ScrapmeConfig;
+    private static deepFreeze;
+}

package/dist/src/config/config-builder.js

@@ -0,0 +1,129 @@
+import { z } from 'zod';
+import { LOG_LEVEL, LOG_LEVELS } from '../logging/logger.models.js';
+import { Urls } from '../util/url.js';
+import { DsnCheck } from './dsn-check.js';
+/** The services given to Scrapme.create are invalid (thrown by Scrapme.create). */
+export class ConfigError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'ConfigError';
+    }
+}
+/** Checks the services an app gives to Scrapme.create (zod), fills in defaults and freezes the result. */
+export class ConfigBuilder {
+    /**
+     * CouchDB's own rule for database names (docs.couchdb.org, "PUT /{db}"). It is a rule about characters, so a
+     * regex is the clearest way to state it.
+     */
+    static COUCH_DB_NAME = /^[a-z][a-z0-9_$()+/-]*$/;
+    static STEP_TIMEOUT_MARGIN_MS = 60_000;
+    static SMTP_PORT = 587;
+    static SMTPS_PORT = 465;
+    /** pg-boss schema names go into SQL as identifiers. */
+    static SQL_IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]*$/;
+    static QUEUE_PREFIX = 'scrapme_';
+    static ms = (min = 0) => z.number().int().min(min);
+    static dbName = z.string().regex(ConfigBuilder.COUCH_DB_NAME, 'CouchDB names: lowercase letters, digits and _$()+/-, starting with a letter');
+    static schema = z.strictObject({
+        logLevel: z.enum(LOG_LEVELS).default(LOG_LEVEL.INFO),
+        defaults: z
+            .strictObject({
+            retries: ConfigBuilder.ms().default(3),
+            timeoutMs: ConfigBuilder.ms(1).default(60_000),
+            retryDelayMs: ConfigBuilder.ms().default(1_000),
+            maxRetryDelayMs: ConfigBuilder.ms().default(30_000),
+            maxStoredChars: ConfigBuilder.ms(100).default(2_000),
+        })
+            .default({ retries: 3, timeoutMs: 60_000, retryDelayMs: 1_000, maxRetryDelayMs: 30_000, maxStoredChars: 2_000 })
+            .refine((d) => d.maxRetryDelayMs >= d.retryDelayMs, { message: 'maxRetryDelayMs must be >= retryDelayMs' }),
+        postgres: z.strictObject({
+            connectionString: z.string().refine((v) => URL.canParse(v), 'must be a postgres:// URL'),
+            schema: z.string().regex(ConfigBuilder.SQL_IDENTIFIER, 'letters, digits and _ only').default('scrapme'),
+        }),
+        queue: z
+            .strictObject({
+            prefix: z.string().regex(ConfigBuilder.SQL_IDENTIFIER, 'letters, digits and _ only').default(ConfigBuilder.QUEUE_PREFIX),
+            retries: ConfigBuilder.ms().default(3),
+            retryDelaySeconds: ConfigBuilder.ms().default(30),
+            retryBackoff: z.boolean().default(true),
+            timeoutSeconds: ConfigBuilder.ms(1).default(3_600),
+            concurrency: ConfigBuilder.ms(1).default(1),
+            // pg-boss: at least 0.5 s
+            pollingIntervalSeconds: z.number().min(0.5).default(2),
+            retentionDays: ConfigBuilder.ms(1).default(7),
+            shutdownTimeoutMs: ConfigBuilder.ms(0).default(30_000),
+        })
+            .default({ prefix: ConfigBuilder.QUEUE_PREFIX, retries: 3, retryDelaySeconds: 30, retryBackoff: true, timeoutSeconds: 3_600, concurrency: 1, pollingIntervalSeconds: 2, retentionDays: 7, shutdownTimeoutMs: 30_000 }),
+        scrapAnything: z
+            .strictObject({
+            url: z.url(),
+            apiSecret: z.string().min(1, 'apiSecret is required (API_SECRET of scrap-anything)'),
+            requestTimeoutMs: ConfigBuilder.ms(1).default(30_000),
+            pollIntervalMs: ConfigBuilder.ms(1).default(2_000),
+            waitTimeoutMs: ConfigBuilder.ms(1).default(600_000),
+            stepTimeoutMs: ConfigBuilder.ms(1).optional(),
+            stepRetries: ConfigBuilder.ms().default(1),
+        })
+            .transform((s) => ({ ...s, url: Urls.withoutTrailingSlash(s.url), stepTimeoutMs: s.stepTimeoutMs ?? s.waitTimeoutMs + ConfigBuilder.STEP_TIMEOUT_MARGIN_MS }))
+            .refine((s) => s.stepTimeoutMs > s.waitTimeoutMs, { message: 'stepTimeoutMs must be greater than waitTimeoutMs (the step waits for the server job)' })
+            .optional(),
+        couch: z
+            .strictObject({
+            url: z.url(),
+            user: z.string(),
+            password: z.string(),
+            db: ConfigBuilder.dbName.default('scrapme'),
+            runsDb: ConfigBuilder.dbName.default('scrapme_runs'),
+            publicUrl: z.url().optional(),
+            timeoutMs: ConfigBuilder.ms(1).default(15_000),
+            retryLimit: ConfigBuilder.ms().default(2),
+        })
+            .transform((c) => ({ ...c, url: Urls.withoutTrailingSlash(c.url), publicUrl: Urls.withoutTrailingSlash(c.publicUrl ?? c.url) })),
+        mail: z
+            .strictObject({
+            host: z.string().min(1),
+            port: ConfigBuilder.ms(1).optional(),
+            secure: z.boolean().default(false),
+            user: z.string().optional(),
+            password: z.string().optional(),
+            timeoutMs: ConfigBuilder.ms(1).default(10_000),
+            from: z.string().min(1),
+            to: z.array(z.string().min(1)).default([]),
+        })
+            .transform((m) => ({ ...m, port: m.port ?? (m.secure ? ConfigBuilder.SMTPS_PORT : ConfigBuilder.SMTP_PORT), user: m.user ?? null, password: m.password ?? null }))
+            .optional(),
+        lighty: z
+            .strictObject({
+            dsn: z.string().refine((dsn) => DsnCheck.isValid(dsn), 'must look like https://<project-key>@<host>/<project>'),
+            environment: z.string().default('production'),
+            release: z.string().default('scrapme'),
+            flushTimeoutMs: ConfigBuilder.ms(1).default(5_000),
+        })
+            .optional(),
+    });
+    static build(options) {
+        const parsed = ConfigBuilder.schema.safeParse(options);
+        if (!parsed.success)
+            throw new ConfigError(`Invalid scrapme options:\n${z.prettifyError(parsed.error)}`);
+        const o = parsed.data;
+        return ConfigBuilder.deepFreeze({
+            logLevel: o.logLevel,
+            flow: o.defaults,
+            postgres: o.postgres,
+            queue: o.queue,
+            scrapAnything: o.scrapAnything ?? null,
+            couch: o.couch,
+            mail: o.mail ?? null,
+            lighty: o.lighty ?? null,
+        });
+    }
+    static deepFreeze(value) {
+        if (value && typeof value === 'object') {
+            for (const v of Object.values(value))
+                ConfigBuilder.deepFreeze(v);
+            Object.freeze(value);
+        }
+        return value;
+    }
+}
+//# sourceMappingURL=config-builder.js.map

package/dist/src/config/config-builder.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"config-builder.js","sourceRoot":"","sources":["../../../src/config/config-builder.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AACxB,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,6BAA6B,CAAC;AACpE,OAAO,EAAE,IAAI,EAAE,MAAM,gBAAgB,CAAC;AAEtC,OAAO,EAAE,QAAQ,EAAE,MAAM,gBAAgB,CAAC;AAE1C,mFAAmF;AACnF,MAAM,OAAO,WAAY,SAAQ,KAAK;IACpC,YAAmB,OAAe;QAChC,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,aAAa,CAAC;IAC5B,CAAC;CACF;AAED,0GAA0G;AAC1G,MAAM,OAAO,aAAa;IACxB;;;OAGG;IACK,MAAM,CAAU,aAAa,GAAG,yBAAyB,CAAC;IAC1D,MAAM,CAAU,sBAAsB,GAAG,MAAM,CAAC;IAChD,MAAM,CAAU,SAAS,GAAG,GAAG,CAAC;IAChC,MAAM,CAAU,UAAU,GAAG,GAAG,CAAC;IACzC,uDAAuD;IAC/C,MAAM,CAAU,cAAc,GAAG,0BAA0B,CAAC;IAC5D,MAAM,CAAU,YAAY,GAAG,UAAU,CAAC;IAE1C,MAAM,CAAU,EAAE,GAAG,CAAC,GAAG,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IAC5D,MAAM,CAAU,MAAM,GAAG,CAAC,CAAC,MAAM,EAAE,CAAC,KAAK,CAAC,aAAa,CAAC,aAAa,EAAE,8EAA8E,CAAC,CAAC;IAEvJ,MAAM,CAAU,MAAM,GAAG,CAAC,CAAC,YAAY,CAAC;QAC9C,QAAQ,EAAE,CAAC,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC;QACpD,QAAQ,EAAE,CAAC;aACR,YAAY,CAAC;YACZ,OAAO,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC;YACtC,SAAS,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;YAC9C,YAAY,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,CAAC;YAC/C,eAAe,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,MAAM,CAAC;YACnD,cAAc,EAAE,aAAa,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;SACrD,CAAC;aACD,OAAO,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,SAAS,EAAE,MAAM,EAAE,YAAY,EAAE,KAAK,EAAE,eAAe,EAAE,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,CAAC;aAC/G,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,eAAe,IAAI,CAAC,CAAC,YAAY,EAAE,EAAE,OAAO,EAAE,yCAAyC,EAAE,CAAC;QAC7G,QAAQ,EAAE,CAAC,CAAC,YAAY,CAAC;YACvB,gBAAgB,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,2BAA2B,CAAC;YACxF,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,KAAK,CAAC,aAAa,CAAC,cAAc,EAAE,4BAA4B,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC;SACxG,CAAC;QACF,KAAK,EAAE,CAAC;aACL,YAAY,CAAC;YACZ,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,KAAK,CAAC,aAAa,CAAC,cAAc,EAAE,4BAA4B,CAAC,CAAC,OAAO,CAAC,aAAa,CAAC,YAAY,CAAC;YACxH,OAAO,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC;YACtC,iBAAiB,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,EAAE,CAAC;YACjD,YAAY,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC;YACvC,cAAc,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;YAClD,WAAW,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;YAC3C,0BAA0B;YAC1B,sBAAsB,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;YACtD,aAAa,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;YAC7C,iBAAiB,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;SACvD,CAAC;aACD,OAAO,CAAC,EAAE,MAAM,EAAE,aAAa,CAAC,YAAY,EAAE,OAAO,EAAE,CAAC,EAAE,iBAAiB,EAAE,EAAE,EAAE,YAAY,EAAE,IAAI,EAAE,cAAc,EAAE,KAAK,EAAE,WAAW,EAAE,CAAC,EAAE,sBAAsB,EAAE,CAAC,EAAE,aAAa,EAAE,CAAC,EAAE,iBAAiB,EAAE,MAAM,EAAE,CAAC;QACxN,aAAa,EAAE,CAAC;aACb,YAAY,CAAC;YACZ,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE;YACZ,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,EAAE,sDAAsD,CAAC;YACpF,gBAAgB,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;YACrD,cAAc,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;YAClD,aAAa,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC;YACnD,aAAa,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,QAAQ,EAAE;YAC7C,WAAW,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC;SAC3C,CAAC;aACD,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,GAAG,EAAE,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,aAAa,EAAE,CAAC,CAAC,aAAa,IAAI,CAAC,CAAC,aAAa,GAAG,aAAa,CAAC,sBAAsB,EAAE,CAAC,CAAC;aAC7J,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,aAAa,GAAG,CAAC,CAAC,aAAa,EAAE,EAAE,OAAO,EAAE,sFAAsF,EAAE,CAAC;aACrJ,QAAQ,EAAE;QACb,KAAK,EAAE,CAAC;aACL,YAAY,CAAC;YACZ,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE;YACZ,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE;YAChB,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE;YACpB,EAAE,EAAE,aAAa,CAAC,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC;YAC3C,MAAM,EAAE,aAAa,CAAC,MAAM,CAAC,OAAO,CAAC,cAAc,CAAC;YACpD,SAAS,EAAE,CAAC,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE;YAC7B,SAAS,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;YAC9C,UAAU,EAAE,aAAa,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC;SAC1C,CAAC;aACD,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,GAAG,EAAE,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,SAAS,EAAE,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QAClI,IAAI,EAAE,CAAC;aACJ,YAAY,CAAC;YACZ,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;YACvB,IAAI,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,QAAQ,EAAE;YACpC,MAAM,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,OAAO,CAAC,KAAK,CAAC;YAClC,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE;YAC3B,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE;YAC/B,SAAS,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;YAC9C,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;YACvB,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;SAC3C,CAAC;aACD,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,aAAa,CAAC,UAAU,CAAC,CAAC,CAAC,aAAa,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,IAAI,IAAI,EAAE,QAAQ,EAAE,CAAC,CAAC,QAAQ,IAAI,IAAI,EAAE,CAAC,CAAC;aACjK,QAAQ,EAAE;QACb,MAAM,EAAE,CAAC;aACN,YAAY,CAAC;YACZ,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,uDAAuD,CAAC;YAC/G,WAAW,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,OAAO,CAAC,YAAY,CAAC;YAC7C,OAAO,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC;YACtC,cAAc,EAAE,aAAa,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;SACnD,CAAC;aACD,QAAQ,EAAE;KACd,CAAC,CAAC;IAEI,MAAM,CAAC,KAAK,CAAC,OAAuB;QACzC,MAAM,MAAM,GAAG,aAAa,CAAC,MAAM,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC;QACvD,IAAI,CAAC,MAAM,CAAC,OAAO;YAAE,MAAM,IAAI,WAAW,CAAC,6BAA6B,CAAC,CAAC,aAAa,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACzG,MAAM,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC;QACtB,OAAO,aAAa,CAAC,UAAU,CAAC;YAC9B,QAAQ,EAAE,CAAC,CAAC,QAAQ;YACpB,IAAI,EAAE,CAAC,CAAC,QAAQ;YAChB,QAAQ,EAAE,CAAC,CAAC,QAAQ;YACpB,KAAK,EAAE,CAAC,CAAC,KAAK;YACd,aAAa,EAAE,CAAC,CAAC,aAAa,IAAI,IAAI;YACtC,KAAK,EAAE,CAAC,CAAC,KAAK;YACd,IAAI,EAAE,CAAC,CAAC,IAAI,IAAI,IAAI;YACpB,MAAM,EAAE,CAAC,CAAC,MAAM,IAAI,IAAI;SACzB,CAAC,CAAC;IACL,CAAC;IAEO,MAAM,CAAC,UAAU,CAAI,KAAQ;QACnC,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;YACvC,KAAK,MAAM,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC;gBAAE,aAAa,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;YAClE,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACvB,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC"}

package/dist/src/config/config-loader.d.ts

@@ -0,0 +1,10 @@
+import type { ScrapmeConfig } from './config.models.js';
+export declare class ConfigError extends Error {
+    constructor(message: string);
+}
+/** The only place that reads environment variables. Empty strings count as "not set" (default applies). */
+export declare class ConfigLoader {
+    static load(env?: Record<string, string | undefined>): ScrapmeConfig;
+    private static toConfig;
+    private static deepFreeze;
+}