@mmerterden/multi-agent-pipeline 20.2.1 → 20.3.0

package/pipeline/skills/shared/external/nodejs-backend-patterns/references/runtime-patterns.md

@@ -0,0 +1,247 @@
+# Node.js runtime patterns
+
+Worked code for the project layout, configuration, errors, shutdown, request context, timeouts and streams summarised in SKILL.md.
+
+## Contents
+
+- Project shape
+- Configuration validated at startup
+- Errors
+- Graceful shutdown
+- Request context
+- Timeouts and cancellation
+- Streams and backpressure
+
+## Project shape
+
+Group by feature, keep transport at the edge, and keep the domain unaware of
+HTTP.
+
+```text
+src/
+  main.ts              # process entry: load config, build app, listen, handle signals
+  app.ts               # builds the server; no side effects at import time
+  config.ts            # parse and validate environment once
+  shared/
+    errors.ts          # AppError hierarchy and problem-details mapping
+    logger.ts
+    context.ts         # AsyncLocalStorage request context
+  orders/
+    orders.routes.ts   # HTTP: parse input, call service, shape output
+    orders.service.ts  # business rules; takes plain values, returns plain values
+    orders.repo.ts     # persistence only
+    orders.schema.ts   # input and output schemas
+    orders.test.ts
+```
+
+Rules that keep it testable:
+
+- `app.ts` exports a factory (`buildApp(deps)`), so tests create an app with
+  fakes and no port.
+- Services receive dependencies as arguments or constructor parameters; avoid
+  importing a database singleton deep inside business code.
+- No work at module top level beyond declarations. Connecting to a database on
+  import makes every test and script pay for it.
+
+`package.json` essentials:
+
+```json
+{
+  "type": "module",
+  "engines": { "node": ">=22" },
+  "scripts": {
+    "dev": "node --watch --env-file-if-exists=.env src/main.ts",
+    "test": "node --test",
+    "start": "node dist/main.js"
+  }
+}
+```
+
+Recent Node releases run TypeScript files with erasable-only syntax directly
+(type stripping). For production, most teams still compile with `tsc` or bundle
+with `tsup`/`esbuild` so startup does no extra work and `enum` or parameter
+properties are available.
+
+## Configuration validated at startup
+
+Read the environment once, validate it, freeze it, and fail fast.
+
+```ts
+// config.ts
+import { z } from "zod";
+
+const Env = z.object({
+  NODE_ENV: z.enum(["development", "test", "production"]).default("development"),
+  PORT: z.coerce.number().int().default(3000),
+  DATABASE_URL: z.string().url(),
+  REDIS_URL: z.string().url().optional(),
+  LOG_LEVEL: z.enum(["debug", "info", "warn", "error"]).default("info"),
+  SHUTDOWN_GRACE_MS: z.coerce.number().int().default(10_000),
+});
+
+export type Config = Readonly<z.infer<typeof Env>>;
+
+export function loadConfig(env = process.env): Config {
+  const parsed = Env.safeParse(env);
+  if (!parsed.success) {
+    console.error("Invalid configuration", parsed.error.flatten().fieldErrors);
+    process.exit(1);
+  }
+  return Object.freeze(parsed.data);
+}
+```
+
+Pass `config` into `buildApp`; do not read `process.env` elsewhere. Secrets come
+from the platform's secret store as environment variables or mounted files,
+never from a committed `.env`.
+
+## Errors
+
+Two kinds of error, handled differently:
+
+- **Expected** (not found, validation failed, conflict): throw a typed
+  `AppError` with a stable code and HTTP status.
+- **Unexpected** (bug, dependency down): let it propagate to one handler that
+  logs the full error with the request id and returns a generic 500.
+
+```ts
+// shared/errors.ts
+export class AppError extends Error {
+  constructor(
+    readonly code: string,
+    readonly status: number,
+    message: string,
+    readonly details?: unknown,
+  ) {
+    super(message);
+    this.name = "AppError";
+  }
+}
+export const notFound = (what: string) => new AppError("not-found", 404, `${what} not found`);
+```
+
+Fastify error handler mapping to RFC 9457 problem details:
+
+```ts
+app.setErrorHandler((err, req, reply) => {
+  if (err instanceof AppError) {
+    return reply.code(err.status).type("application/problem+json").send({
+      type: `https://api.example.com/problems/${err.code}`,
+      title: err.message,
+      status: err.status,
+      instance: req.id,
+      ...(err.details ? { errors: err.details } : {}),
+    });
+  }
+  req.log.error({ err }, "unhandled error");
+  return reply.code(500).type("application/problem+json").send({
+    type: "about:blank", title: "Internal error", status: 500, instance: req.id,
+  });
+});
+```
+
+Process-level safety nets: log and exit on `uncaughtException`, and treat
+`unhandledRejection` the same way (Node's default since v15 is to crash, which
+is correct). Let the orchestrator restart the process; do not try to continue
+in an unknown state.
+
+Lint for the root causes: `@typescript-eslint/no-floating-promises` and
+`no-misused-promises` catch most lost errors before runtime.
+
+## Graceful shutdown
+
+On `SIGTERM` (sent by Kubernetes, ECS, systemd), stop accepting work, finish
+what is already running, close resources, then exit.
+
+```ts
+// main.ts
+const config = loadConfig();
+const deps = await createDeps(config);
+const app = await buildApp(config, deps);
+await app.listen({ port: config.PORT, host: "0.0.0.0" });
+
+let closing = false;
+for (const signal of ["SIGTERM", "SIGINT"] as const) {
+  process.once(signal, async () => {
+    if (closing) return;
+    closing = true;
+    app.log.info({ signal }, "shutting down");
+    const force = setTimeout(() => process.exit(1), config.SHUTDOWN_GRACE_MS).unref();
+    try {
+      await app.close();          // stops listening, waits for open requests
+      await deps.queue.close();   // stop taking jobs, let current ones finish
+      await deps.db.end();
+      clearTimeout(force);
+      process.exit(0);
+    } catch (err) {
+      app.log.error({ err }, "shutdown failed");
+      process.exit(1);
+    }
+  });
+}
+```
+
+Pair it with a readiness probe that starts failing as soon as `closing` is set,
+so the load balancer drains traffic before connections close. Keep liveness
+separate and cheap; a liveness check that queries the database turns a database
+blip into a restart storm.
+
+## Request context
+
+`AsyncLocalStorage` carries per-request data (request id, principal, tenant)
+through every `await` without threading parameters.
+
+```ts
+// shared/context.ts
+import { AsyncLocalStorage } from "node:async_hooks";
+
+type Ctx = { requestId: string; accountId?: string };
+export const context = new AsyncLocalStorage<Ctx>();
+export const currentCtx = () => context.getStore();
+
+// in app.ts
+app.addHook("onRequest", (req, _reply, done) => {
+  context.run({ requestId: req.id }, done);
+});
+```
+
+Use it for logging and tracing correlation. Do not hide business inputs in it;
+a service that needs the account id should take it as a parameter so its
+contract stays visible. OpenTelemetry's Node SDK uses the same mechanism for
+trace propagation.
+
+## Timeouts and cancellation
+
+Every outbound call needs a deadline.
+
+```ts
+const res = await fetch(`${inventoryUrl}/products/${id}`, {
+  signal: AbortSignal.any([req.signal, AbortSignal.timeout(2_000)]),
+});
+```
+
+- `AbortSignal.timeout` gives a deadline; `AbortSignal.any` also cancels when
+  the client disconnects.
+- Set database statement timeouts and pool acquire timeouts.
+- Set server timeouts (`requestTimeout`, `keepAliveTimeout` above the load
+  balancer's idle timeout to avoid 502s on reused connections).
+
+## Streams and backpressure
+
+Stream anything large instead of buffering it.
+
+```ts
+import { pipeline } from "node:stream/promises";
+import { createGzip } from "node:zlib";
+
+app.get("/exports/orders.csv.gz", async (req, reply) => {
+  reply.header("content-type", "text/csv").header("content-encoding", "gzip");
+  const rows = orderRepo.streamCsv(req.auth.accountId); // a Readable
+  await pipeline(rows, createGzip(), reply.raw);
+  return reply;
+});
+```
+
+`pipeline` propagates errors and destroys every stage on failure; manual
+`.pipe()` chains leak file handles when a stage errors. Async iteration
+(`for await (const chunk of stream)`) respects backpressure automatically.

package/pipeline/skills/shared/external/nodejs-backend-patterns/references/testing-and-frameworks.md

@@ -0,0 +1,57 @@
+# Testing, Frameworks and Logging
+
+Companion to the Node.js backend skill: how to test a service built from an app factory, and how the common frameworks differ.
+
+## Testing with node:test
+
+```ts
+// orders/orders.test.ts
+import { describe, it, before, after } from "node:test";
+import assert from "node:assert/strict";
+import { buildApp } from "../app.ts";
+import { fakeDeps } from "../test/fakes.ts";
+
+describe("POST /orders", () => {
+  let app: Awaited<ReturnType<typeof buildApp>>;
+  before(async () => { app = await buildApp(testConfig, fakeDeps()); });
+  after(() => app.close());
+
+  it("rejects a zero quantity", async () => {
+    const res = await app.inject({
+      method: "POST", url: "/orders",
+      payload: { productId: "p_812abc", quantity: 0 },
+    });
+    assert.equal(res.statusCode, 400);
+    assert.equal(res.json().type.endsWith("/validation"), true);
+  });
+});
+```
+
+- `node --test` discovers `*.test.*` files; `--test-concurrency`, `--watch`,
+  mocking via `t.mock.fn()` and `t.mock.timers`, and coverage via
+  `--experimental-test-coverage` are built in.
+- Use `app.inject()` (Fastify) or `supertest` (Express) for HTTP tests without
+  opening a port.
+- Integration tests against a real Postgres or Redis in Testcontainers catch
+  what fakes cannot; keep them in a separate script.
+
+## Framework notes
+
+- **Fastify 5:** schema-first routes with JSON Schema or a type provider (Zod,
+  TypeBox), encapsulated plugins, fast serialisation. A strong default.
+- **Express 5:** now forwards rejected promises from async handlers to the error
+  middleware; still bring your own validation and structure.
+- **NestJS:** modules, DI and decorators suit large teams that want imposed
+  structure; keep business logic in plain services so it is not tied to the
+  framework.
+- **Hono:** small and runtime-portable (Node, Bun, Deno, edge workers).
+
+## Logging and observability
+
+- Structured JSON logs via `pino` (Fastify's default). Include `requestId`,
+  route, status, duration; redact `authorization`, cookies and secrets with
+  `redact` paths.
+- Log levels mean something: `error` pages someone, `warn` is looked at daily,
+  `info` records state changes, `debug` is off in production.
+- OpenTelemetry auto-instrumentation for HTTP, database and queue clients gives
+  traces with little code; export RED metrics (rate, errors, duration) per route.