vibecodeagent 0.0.0-stage → 0.1.65

package/dist/resources/skills/builtin/backend-engineering/references/stack-node.md

@@ -0,0 +1,274 @@
+# Node and TypeScript
+
+This file owns what is **true of the Node runtime and would be wrong in Python or Go**: the event loop, promise semantics, `AbortSignal`, streams and backpressure, the driver and ORM landscape, and the fact that TypeScript types do not exist at runtime. Everything stack-neutral stays with its owner and is restated here only where the Node mechanics differ. The parse-don't-validate doctrine, unknown-field policy, and body-size limits are [validation-and-errors.md](validation-and-errors.md) — the zod section below is only how that doctrine is spelled in this stack. Retry counts, backoff formula, DLQ policy, and idempotent-processor design are [concurrency-and-jobs.md](concurrency-and-jobs.md); BullMQ here is only the mapping onto that policy. Log fields, redaction targets, and alert thresholds are [observability.md](observability.md); pino here is only the transport mechanics. Timeout values, pool sizing, and graceful shutdown are [deploy-and-config.md](deploy-and-config.md) and [testing-and-perf.md](testing-and-perf.md) — every number quoted from them is restated verbatim, never re-derived. N+1 diagnosis is [data-modelling.md](data-modelling.md); the per-ORM N+1 *behaviour* is here.
+
+`kerja-agentik` owns running the suite and verification discipline; `dasar-vibecode` owns shell and git. This file adds only the Node-specific diagnostics those two cannot know about.
+
+## Operating Posture
+
+Three failure modes, ranked, and the first two are silent:
+
+1. **A promise nobody awaited.** The handler returns `200`, the write never happened, and the rejection surfaces as a process-wide crash minutes later in an unrelated request — or, under Express 4, as a socket that hangs until the client times out. No stack points at the line that dropped it.
+2. **One synchronous line eating the event loop.** A `JSON.stringify` over 80k rows, a `scryptSync`, a `readFileSync` in a hot path. Every concurrent request queues behind it. The symptom reads as "the database got slow", because every timing you have is measured from inside the same blocked loop.
+3. **A pool multiplied by a process count nobody counted.** Cluster workers × replicas × `pool.max` against a Postgres `max_connections` of 100. Loud, at least.
+
+**Never invent a Node default.** Not a `highWaterMark`, not an undici timeout, not a Prisma pool size. Every number below is a real default you can print with `node -p`, or a config value from the project. Writing `timeout: 30000` because it looks familiar is the same defect class as inventing a bcrypt cost.
+
+**Producing nothing is often right.** If the project is on Fastify with `@fastify/type-provider-typebox`, do not add zod beside it. If it already has a `pg` `Pool` and hand-written SQL that works, adding Prisma is a migration, not an improvement. Say so and stop.
+
+## 1. Framework Choice
+
+Pick on the constraint that actually decides it, not on ergonomics — all four are fine at writing a route.
+
+| The deciding constraint | Framework |
+| --- | --- |
+| Team already has one, and it works | **The one they have.** Not a rewrite. |
+| Throughput matters, JSON in / JSON out, you control the code | **Fastify** — schema-compiled serialization (`fast-json-stringify`) and validation (`ajv`) are the speed, not the router |
+| Deploying to a Workers / Deno / Bun / edge runtime, or you want the same code in a test with no server | **Hono** — Web Standard `Request`/`Response`, so `app.fetch(new Request(...))` *is* the integration test |
+| Large team, many modules, DI and a fixed layout are worth the ceremony | **NestJS** — and run it on the Fastify adapter, not the default Express one |
+| Middleware you must have exists only for Express, or the app is ten routes | **Express 5** — never a new Express 4 project |
+| Deciding between Fastify and Hono on "which is faster" | Neither. Decide on the runtime target; the difference is noise next to one N+1 |
+
+**Express 4 is disqualifying for new work for one reason: a rejected promise from an async handler is not routed to the error middleware.** It becomes an unhandled rejection and the request hangs. Express 5 awaits handler return values and forwards rejections to `next(err)`. Fastify and Hono have always done this. If you are stuck on Express 4, every async handler is wrapped or the wrapper is the bug:
+
+```ts
+const h = (fn) => (req, res, next) => Promise.resolve(fn(req, res, next)).catch(next);
+app.post("/orders", h(createOrder));   // unwrapped === hangs on rejection
+```
+
+Serialization is where Fastify's number comes from: declare a `response` schema and it compiles a serializer that emits **only the declared fields**. That is simultaneously the response-hygiene rule from [validation-and-errors.md](validation-and-errors.md) enforced by the framework — an added `password_hash` column cannot leak through a declared response schema.
+
+## 2. zod at the Edge
+
+The doctrine is [validation-and-errors.md](validation-and-errors.md). These are the zod specifics that bite in Node:
+
+```ts
+const CreateOrder = z.object({
+  customerId:  z.string().uuid(),
+  currency:    z.enum(["IDR", "USD"]),
+  amountMinor: z.number().int().positive().max(100_000_000),
+}).strict();                              // v4: z.strictObject({...})
+
+type CreateOrder = z.infer<typeof CreateOrder>;   // never a parallel interface
+```
+
+| Trap | What actually happens | Do |
+| --- | --- | --- |
+| Bare `z.object()` on a request body | Unknown keys are **stripped silently**, not rejected — the request looks accepted | `.strict()` / `z.strictObject` |
+| `z.coerce.number()` on a query param | It is `Number(v)`: `""` → `0`, `null` → `0`, `" 3 "` → `3` | `z.string().regex(/^\d+$/).transform(Number)`, or `.pipe(z.number().int())` |
+| `z.coerce.boolean()` on `"false"` | It is `Boolean(v)`: **`"false"` → `true`** | `z.enum(["true","false"]).transform(v => v === "true")` |
+| `.parse()` inside a handler | Throws `ZodError`; on Express 4 an async throw is the hang above | `.safeParse()` and branch, or parse in middleware with a real error handler |
+| `schema.parse(x)` in the service layer too | Two validators drift; the looser one is the one that gets found | Parse **once**, at the edge. Downstream takes `CreateOrder`, not `unknown` |
+
+**Re-validation stops at the process boundary, and starts again at every re-entry.** Inside one process a value that was parsed is trusted. A job payload pulled off Redis, a webhook body, a row of `jsonb`, and a cache hit are all **untrusted again** — they crossed a boundary and a previous deploy may have written them. Parse job payloads in the processor with the schema, not with `as JobData`.
+
+## 3. ORM and Query Builder
+
+| | **Prisma** | **Drizzle** | **Kysely** | **raw `pg`** |
+| --- | --- | --- | --- | --- |
+| Relations / N+1 | `include` fetches relations; without `relationJoins` these are separate queries stitched in JS — a loop over results with a lazy access is still N+1 | `db.query.x.findMany({ with: {...} })` emits one statement with lateral joins | **None.** You write the `JOIN`, so N+1 is a thing you typed | None. Same |
+| Transactions | `$transaction([...])` batches sequentially; `$transaction(async tx => …)` is interactive with **`maxWait` 2000 ms, `timeout` 5000 ms** defaults — exceed it and you get `P2028` | `db.transaction(async tx => …)`; use `tx`, not `db`, inside | `db.transaction().execute(async trx => …)` | `client.query('BEGIN')` on a **checked-out client** |
+| Migrations | `prisma migrate` owns the schema file; generated SQL is editable before apply | `drizzle-kit generate` diffs the TS schema, you review the SQL | Hand-written up/down through its `Migrator` | Whatever you bring (`node-pg-migrate`, plain SQL) |
+| Type safety is | Generated from the DB schema — verified | Generated from your TS schema — verified against what you declared | **Asserted.** The `Database` interface is a claim; regenerate with `kysely-codegen` or it silently lies | `any` rows. Parse them |
+| Pool caveat | Engine pool defaults to `num_physical_cpus * 2 + 1`, set via `?connection_limit=` in the URL — **it is not your `pg` pool** | Your `pg`/`postgres.js` pool, sized by you | Same | Same |
+
+Whatever the answer, every migration goes through [migrations.md](migrations.md) before it touches a live table, and index design through [data-modelling.md](data-modelling.md).
+
+**The per-request client is the trap that unites all four.** `new PrismaClient()` or `new Pool()` inside a handler, a factory, or a module that a serverless bundler re-evaluates opens a fresh pool per request. Under any load the database hits `max_connections` and starts refusing *everything*, including the health check. In Next.js dev, HMR re-evaluates the module on every save and reproduces this on a laptop.
+
+```ts
+// db.ts — module scope, one instance for the process lifetime.
+// globalThis guard survives HMR/module re-evaluation in dev; it is not a prod hack.
+export const db = globalThis.__db ?? (globalThis.__db = new Pool({
+  max: 10,                     // pg default; the ceiling formula is testing-and-perf.md §8
+  idleTimeoutMillis: 10_000,   // pg default
+  connectionTimeoutMillis: 2_000, // pg default is 0 = wait forever; 2s = deploy-and-config.md pool acquire
+}));
+```
+
+`pool.query()` checks out a connection per call and returns it. **A transaction must therefore hold one client**: `BEGIN` via `pool.query()` and `COMMIT` via `pool.query()` can land on two different connections, leaving an open transaction to be reaped by `idle_in_transaction_session_timeout`. Use `const c = await pool.connect()` … `finally { c.release() }`.
+
+## 4. Promises Without Holes
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| Process exits with `ERR_UNHANDLED_REJECTION`, stack points nowhere useful | **Since Node 15 the default is `--unhandled-rejections=throw`** — an unhandled rejection terminates the process | Await it, or `.catch()` it at the site that created it |
+| Handler returns `200`, the side effect never happened, log is clean | Floating promise: `sendEmail(x)` with no `await`. The `try/catch` around it catches nothing — the frame is gone before the rejection | `await`, or `void p.catch(err => log.error({err}))` — `void` alone is not enough |
+| One slow branch fails and an unrelated rejection crashes the process later | `Promise.all` rejects on the **first** failure but does **not cancel the siblings**; a second failure lands with no handler | `allSettled` when partial success is meaningful, or an `AbortController` shared by all branches |
+| Latency is the sum of the calls, not the max | Sequential `await` in a loop over independent work | `Promise.all` (bounded — see below) |
+
+```ts
+// Two literals with reasons:
+await Promise.all(ids.map(fetchOne));   // unbounded: 5k ids = 5k sockets = EMFILE
+                                        // and it drains the pool testing-and-perf.md sized
+// Bound it. 10 matches the pg pool default above, so DB work can never out-issue connections.
+const limit = pLimit(10);
+await Promise.all(ids.map(id => limit(() => fetchOne(id))));
+```
+
+Turn this on and treat it as an error, not a warning — it is the only thing that finds a floating promise mechanically: `@typescript-eslint/no-floating-promises` plus `no-misused-promises` (which catches an `async` function passed where a `void` callback is expected, e.g. `app.get('/x', asyncFn)` on Express 4). Both require type-aware linting (`parserOptions.project`); without it they are silently inert.
+
+**`process.on('unhandledRejection')` is a logging hook, not a fix.** Log the error with the trace id, then let the process die and be restarted ([deploy-and-config.md](deploy-and-config.md)). Swallowing it keeps a process running with unknown state, which is worse than a restart.
+
+## 5. AbortSignal, End to End
+
+**`fetch` in Node has no total-request timeout.** Restated from [deploy-and-config.md](deploy-and-config.md): undici's `connectTimeout` is 10s, `headersTimeout` and `bodyTimeout` are 300s, and none of them cap the whole call. A hung upstream holds your handler, its worker, and anything it checked out until the socket dies.
+
+```ts
+// AbortSignal.any: Node >= 20.3. Whichever fires first wins.
+// 5000 = the service -> third-party budget from deploy-and-config.md, not a guess.
+const res = await fetch(url, {
+  signal: AbortSignal.any([req.signal, AbortSignal.timeout(5000)]),
+});
+```
+
+| Where the signal comes from | Framework |
+| --- | --- |
+| `c.req.raw.signal` | Hono (Web Standard `Request`) |
+| `request.raw.on("close", …)` → your own `AbortController` | Fastify, Express |
+| `signal` on the job's context, or your own per-job controller | BullMQ (no built-in signal) |
+
+Two consequences worth naming:
+
+- **Classify the abort or you will report a client disconnect as your own outage.** `AbortSignal.timeout` rejects with a `TimeoutError` `DOMException`; `controller.abort()` gives an `AbortError`. `TimeoutError` maps to the gateway-timeout row in [api-contracts.md](api-contracts.md). `AbortError` from a client disconnect maps to **no response at all** — the socket is gone; log it at INFO and stop. Do not invent a status code for it.
+- **Aborting the fetch does not abort the query.** `pg`, Prisma, and Drizzle do not cancel an in-flight statement on an `AbortSignal`. The database-side stop is `statement_timeout` — **5s**, restated from [deploy-and-config.md](deploy-and-config.md). A request-scoped signal that stops HTTP work but leaves a 40-second sequential scan running is a pool leak wearing a cancellation costume.
+
+## 6. Streams and Backpressure
+
+**`res.write()` returns `false` when the socket's buffer is past its `highWaterMark` (16 KiB; 64 KiB since Node 22). Ignoring that return value is how a large response becomes an OOM kill.** The loop keeps producing, Node keeps buffering in heap, RSS climbs, and the container is `OOMKilled` with exit code 137 — with no exception, no stack, and a log that ends mid-request. A slow client makes it happen at a tenth of the data.
+
+```ts
+import { pipeline } from "node:stream/promises";
+
+// pipeline propagates backpressure AND destroys every stream on error or client abort.
+await pipeline(queryStream, jsonEncoder, res, { signal: req.signal });
+```
+
+Never `res.json(await db.query(...))` for an unbounded result set. Two separate cliffs: the array is fully materialised in heap, and `JSON.stringify` on it is a single synchronous call that blocks the loop (§7) and throws `RangeError: Invalid string length` past **V8's maximum string length of 536,870,888 characters** on 64-bit. The fix is a cursor plus a streaming encoder — or, better, the cursor pagination the hub already mandates.
+
+`.pipe()` is not `pipeline()`: on an error it leaves the source stream open and the file descriptor leaking. Use `pipeline` for every chain, including `fs` → `res` file downloads.
+
+## 7. The Event Loop
+
+One loop per process. Anything synchronous stops **every** concurrent request, and every latency number you collect is measured from inside the stall, so the blame lands on whatever was awaiting when the loop froze.
+
+| Blocks the loop | Instead |
+| --- | --- |
+| `JSON.parse` / `JSON.stringify` on multi-MB payloads | Stream it; cap the body at the limit in [validation-and-errors.md](validation-and-errors.md) |
+| `crypto.pbkdf2Sync`, `scryptSync`, `bcrypt.hashSync` | The async form — it runs on the libuv threadpool |
+| `fs.readFileSync`, `zlib.gzipSync` in a request path | `fs/promises`, `zlib` async, or read once at boot |
+| A catastrophic-backtracking regex over user input | Cap the string length **before** the regex ([validation-and-errors.md](validation-and-errors.md)) |
+| Sorting or mapping 10⁶ rows in JS | `ORDER BY` in the database, which is what it is for |
+
+The async crypto/`fs`/`dns.lookup` forms are not free either: they share the libuv threadpool, **default `UV_THREADPOOL_SIZE=4`** (max 1024). Five concurrent `scrypt` calls serialize into two rounds *and* stall unrelated file reads. Raise it deliberately against core count, or move the work out.
+
+```ts
+import { monitorEventLoopDelay } from "node:perf_hooks";
+const h = monitorEventLoopDelay({ resolution: 10 });  // ms sample interval
+h.enable();
+// h.percentile(99) is in NANOSECONDS. Export as a gauge — observability.md owns the alert.
+setInterval(() => metrics.eventLoopDelayP99.set(h.percentile(99) / 1e6), 10_000);
+```
+
+libuv's timer granularity is ~1ms, so a p99 delay in the **hundreds of milliseconds is your code, never scheduling noise** — that is a diagnostic tripwire, not an SLO. On Fastify, `@fastify/under-pressure` turns the same measurement into load shedding: over `maxEventLoopDelay` it returns `503` with `Retry-After` instead of accepting work it cannot do.
+
+**`worker_threads` is the escape hatch for CPU-bound work** — image resizing, a large parse, a report render. Use a pool (`piscina`) rather than spawning per request; thread startup is tens of milliseconds and unbounded spawning is the original problem with extra memory. `child_process` is for running a *different program*, not for offloading JS. Neither is a fix for a slow query.
+
+## 8. BullMQ
+
+Policy — attempt counts, the backoff formula, DLQ ownership, "cron only enqueues" — is [concurrency-and-jobs.md](concurrency-and-jobs.md). This is the mapping.
+
+```ts
+new Worker("payments", async (job) => {
+  // Idempotent by construction: the job id IS the idempotency key, and the DB
+  // constraint is what enforces it. "Check if it ran already" is a race, not a guard.
+  await db.charge({ idempotencyKey: job.id, ...Payload.parse(job.data) });
+}, {
+  connection: new IORedis(url, { maxRetriesPerRequest: null }), // BullMQ requires null
+  concurrency: 5,          // PER WORKER INSTANCE. 4 replicas x 5 = 20 concurrent jobs
+});                        //   against a pool sized in testing-and-perf.md §8
+```
+
+| BullMQ reality | Consequence |
+| --- | --- |
+| `backoff: { type: "exponential", delay: 1000 }` is `delay * 2^(attempts-1)` — **no jitter** | It violates the jitter rule in [concurrency-and-jobs.md](concurrency-and-jobs.md). Register a custom `settings.backoffStrategy` implementing `random(0, min(300s, 1s * 2**n))` |
+| A `Worker` holds a **blocking** Redis connection plus a regular one, and cannot share the blocking one | Connections ≈ (workers × 2) + queues + `QueueEvents`, × replicas. On a managed Redis with a low `maxclients`, ten workers per pod is an outage, not a config choice |
+| `attempts` defaults to 1 | A transient failure is a permanent failure unless you set it |
+| Completed and failed jobs are retained in Redis | Set `removeOnComplete: { age: 3600, count: 1000 }` and `removeOnFail: { age: 86_400 }` explicitly. Unbounded retention is a Redis memory incident that arrives as an eviction |
+| `job.data` is JSON from Redis, written by a previous deploy | Parse it with a schema (§2). `job.data as Payload` is a lie the compiler cannot catch |
+
+Shut down with `await worker.close()` inside the `SIGTERM` handler so in-flight jobs finish — the drain deadline is [deploy-and-config.md](deploy-and-config.md).
+
+## 9. pino, and Why `console.log` Is a Performance Bug
+
+Field names, required keys, redaction targets, and sampling are [observability.md](observability.md). The Node-specific part is that **`console.log` is synchronous on POSIX when stdout is a file or a TTY** (async only when it is a pipe). Every line is a blocking `write(2)` on the event loop, in the hot path, from every request. It is simultaneously an observability bug: `console.log(obj)` runs `util.inspect` with `depth: 2`, so nested fields render as `[Object]` and multi-line output breaks one event into many unparseable log lines.
+
+```ts
+export const log = pino({
+  level: process.env.LOG_LEVEL ?? "info",
+  timestamp: pino.stdTimeFunctions.isoTime,   // default is epoch ms
+  redact: { paths: [/* observability.md owns this list */], censor: "[REDACTED]" },
+});
+```
+
+Formatting and transports (`pino-pretty`, file, HTTP shippers) run in a **worker thread**, off the request path. Corollary: `pino-pretty` must never be enabled in production — you pay serialization plus prettification to produce something no log pipeline can parse. Pretty in dev, JSON to stdout everywhere else, and the platform ships it.
+
+## 10. Cluster, PM2, and Replicas
+
+**Prefer one Node process per container and let the orchestrator run N containers.** `cluster` and PM2 cluster mode hide the process count from the platform: the scheduler sees one pod, the database sees `workers × replicas` pools, restarts are managed by two systems that disagree, and per-process memory limits stop matching the container's.
+
+```
+total connections = replicas × processes_per_replica × pool.max
+                  ≤ (max_connections − superuser_reserved_connections − other_consumers)
+Postgres defaults: max_connections = 100, superuser_reserved_connections = 3
+```
+
+That is the ceiling from [testing-and-perf.md](testing-and-perf.md) §8 with the Node multiplier made explicit. Four replicas × four cluster workers × `pool.max: 10` is 160 connections against a ceiling of 97 — and the failure is not gradual: the pool that loses the race gets `too many clients already` on **every** query, including `/readyz`.
+
+**Never size workers from `os.cpus().length`.** It reports the host's CPUs, not the cgroup quota — a container limited to `cpu: "500m"` on a 64-core node forks 64 workers. `os.availableParallelism()` honours the CPU affinity mask but still not a CFS quota. Take the worker count from validated config ([deploy-and-config.md](deploy-and-config.md)), and divide `pool.max` by it.
+
+## 11. TypeScript Facts That Matter at Runtime
+
+**Types are erased. They validate nothing at any boundary.** A `.d.ts` is a claim about data you did not check.
+
+| Never | Instead | Because |
+| --- | --- | --- |
+| `JSON.parse(body) as CreateOrder` | `CreateOrder.parse(JSON.parse(body))` | `JSON.parse` returns `any`; the cast is a comment with syntax highlighting |
+| `as` to make an error go away | `satisfies` (TS 4.9+) | `satisfies` checks the value against the type **and** keeps the narrow inferred literal types; `as` silences the check |
+| `x as unknown as T` | Parse it | The double cast is the compiler telling you the claim is false |
+| A hand-written `interface` beside a zod schema | `z.infer<typeof S>` | Two sources of truth drift silently — [validation-and-errors.md](validation-and-errors.md) |
+| `function isUser(x): x is User { return !!x }` | A schema | A type predicate is an unchecked promise, not a check |
+| `process.env.FOO!` | The validated config object | `!` asserts non-null on a value that is `undefined` in the environment that matters |
+
+Compiler settings that change what ships: `strict: true` is the floor; `noUncheckedIndexedAccess` makes `rows[0]` `T | undefined`, which is what a query actually returns; `exactOptionalPropertyTypes` stops `{ note: undefined }` from satisfying `note?: string` and writing a `null` you did not intend. If the project runs `node --experimental-strip-types` (22.6+; unflagged in 23.6), set `erasableSyntaxOnly` — `enum`, `namespace`, and constructor parameter properties emit runtime code and will not strip. Prefer `as const` objects over `enum` regardless.
+
+## Node-Specific Diagnostics
+
+Verification method is `kerja-agentik`'s. These are the flags it cannot know:
+
+```bash
+node --trace-warnings app.js          # turns a bare "Warning:" into a stack — finds the leaked listener
+node --enable-source-maps app.js      # otherwise every prod stack points at dist/ line 1
+node --cpu-prof --cpu-prof-dir=/tmp   # writes a .cpuprofile on exit; open in Chrome DevTools
+node --trace-uncaught app.js          # stack for a thrown non-Error
+NODE_DEBUG=undici,net node app.js     # what the HTTP client actually did
+```
+
+## Never Ship
+
+| Never | Instead |
+|---|---|
+| Express 4 with an unwrapped async handler | Express 5, or a `.catch(next)` wrapper on every route |
+| A promise created and not awaited or `.catch`ed | `await`, or `void p.catch(log)`; enable `no-floating-promises` |
+| `fetch()` with no `signal` | `AbortSignal.any([req.signal, AbortSignal.timeout(5000)])` |
+| `new PrismaClient()` / `new Pool()` per request or per module reload | One module-scoped instance behind a `globalThis` guard |
+| `pool.query('BEGIN')` | `pool.connect()`, transact on the client, `release()` in `finally` |
+| `res.write()` in a loop with the return value ignored | `pipeline()` from `node:stream/promises` |
+| `res.json(await allRows)` on an unbounded query | Cursor + streaming encoder |
+| A `*Sync` crypto/fs/zlib call in a request path | The async form; CPU work to `worker_threads` |
+| `console.log` in a server | pino to stdout, pretty only in dev |
+| Unbounded `Promise.all` over user-sized input | A concurrency limit tied to the pool size |
+| BullMQ built-in `exponential` backoff | A custom strategy with full jitter |
+| Worker count or pool size from `os.cpus().length` | Validated config, divided by processes per container |
+| `as` on anything parsed from JSON, Redis, or `jsonb` | Schema `.parse()` at every re-entry |
+| A hand-written type beside a zod schema | `z.infer` |

package/dist/resources/skills/builtin/backend-engineering/references/stack-python.md

@@ -0,0 +1,296 @@
+# Python
+
+This file owns only what is **true of Python** and false elsewhere: the async/sync boundary in a FastAPI worker, pydantic v2 as the edge schema, SQLAlchemy 2.0 session lifecycle, Alembic autogenerate, ASGI process topology, the GIL, Celery's Python-specific footguns, and the virtualenv/lockfile rules the harness enforces. Everything stack-neutral belongs to a sibling and is not re-derived here.
+
+Neighbours, so you don't half-solve a problem here: what to validate and what an error body may contain is [validation-and-errors.md](validation-and-errors.md) — this file only says how pydantic v2 expresses it. Index design, N+1 detection, and the query-count budget are [data-modelling.md](data-modelling.md); the `selectinload`/`joinedload` choice here is that file's to-one/to-many table in SQLAlchemy syntax. Lock modes and expand/contract are [migrations.md](migrations.md); Alembic below only decides what the generator got wrong before you run it. Retry policy, backoff, visibility timeouts, and idempotent-consumer design are [concurrency-and-jobs.md](concurrency-and-jobs.md) — Celery below is that doctrine's config surface, never a second policy. Timeout values, pool sizing, and graceful shutdown are [deploy-and-config.md](deploy-and-config.md) and [testing-and-perf.md](testing-and-perf.md); the worker topology here is what multiplies those numbers.
+
+## Operating Posture
+
+Three failure modes, ranked. The first is worse because nothing is red.
+
+1. **A blocking call inside `async def`.** No exception, no traceback, no CPU saturation. Throughput stops scaling with concurrency and *every* endpoint on that worker slows together, including `/livez`. Reviewers read the handler, see correct-looking `await`s, and move on. Section 2 exists entirely for this.
+2. **The ORM object that outlives its session.** `DetachedInstanceError`, `MissingGreenlet`, or a serialiser quietly emitting a column you added to the table last week. Loud when it crashes, dangerous when it doesn't.
+3. **An environment nobody can reproduce.** Unpinned deps, `pip install` into a system interpreter, "works on my machine". Cheap to prevent, expensive to debug at 02:00.
+
+**Never invent a Python default.** Not the threadpool size, not Celery's visibility timeout, not gunicorn's worker formula. Every number below is a real default or a documented recommendation and is checkable in the source. `worker_prefetch_multiplier` guessed at from memory reads exactly like one that was measured.
+
+## 1. Framework Choice
+
+Keyed on what actually decides it, not on taste:
+
+| What decides it | Answer |
+| --- | --- |
+| The app is CRUD over a relational schema with an admin UI, auth, and forms | **Django** — you get the admin, the ORM, migrations, sessions, and CSRF for free. Rebuilding those in FastAPI is a quarter you will not get back. |
+| The service is an API in front of models, streaming, or async I/O fan-out (any LLM/AI-adjacent work) | **FastAPI** — pydantic-typed contracts, native async, OpenAPI generated from the signatures |
+| Genuinely tiny: one webhook receiver, a health shim, a sidecar | **Flask** (or FastAPI anyway — the cost is one dependency) |
+| Existing Django app needs one async endpoint | **Stay in Django.** Django 4.1+ has async views and an async ORM interface. Do not add a second framework to one process. |
+| "We should modernise it to FastAPI" | **No.** Port when you can name the limit you hit — request concurrency, WebSockets, streaming. "It's more modern" is not a limit. |
+
+**Producing no framework change is the common correct answer.** A working Django service is not a defect.
+
+## 2. The async/sync Trap
+
+This costs more real throughput than every other item in this file combined, and it is invisible in review.
+
+**The two paths, exactly:**
+
+- A path operation declared **`def`** is run by Starlette via `anyio.to_thread.run_sync`, in a threadpool bounded by anyio's default capacity limiter: **40 threads per process**. Request 41 queues. The event loop stays free.
+- A path operation declared **`async def`** runs **directly on the event loop**. There is one loop per worker process. Anything in it that does not yield stalls every other request on that worker, plus the health endpoint, plus the metrics scrape.
+
+Dependencies obey the same rule independently: an `async def` dependency runs on the loop even when the path operation is `def`. So does a `BackgroundTask`, and so does a `def` dependency (threadpool) attached to an `async def` route.
+
+### The decision rule
+
+| What the handler body actually does | Declare it |
+| --- | --- |
+| Only awaits async I/O — `httpx.AsyncClient`, `asyncpg`, `AsyncSession`, `redis.asyncio` | `async def` |
+| Contains any blocking call — `requests`, sync `psycopg`, `boto3`, `time.sleep`, `open().read()` on a large file, `subprocess.run`, PIL, pandas | `def` |
+| Mostly async with one unavoidable blocking call | `async def`, and wrap that call: `await anyio.to_thread.run_sync(fn, arg)` |
+| Pure-Python CPU work over ~50 ms (hashing, parsing, image work, tokenising) | Neither — section 6. A threadpool does not help; the GIL is the limit. |
+
+```python
+# WRONG — one loop, every request on this worker waits for this socket.
+@app.get("/quote")
+async def quote():
+    return requests.get(UPSTREAM, timeout=(2, 5)).json()   # blocking, on the loop
+
+# RIGHT if you must keep the sync client — the threadpool absorbs it (40 concurrent).
+@app.get("/quote")
+def quote():
+    return requests.get(UPSTREAM, timeout=(2, 5)).json()
+
+# BEST — async client, async handler, no thread hop.
+@app.get("/quote")
+async def quote(client: httpx.AsyncClient = Depends(get_client)):
+    r = await client.get(UPSTREAM, timeout=5.0)
+    return r.json()
+```
+
+The reverse mistake is quieter: a handler that only `await`s, declared `def`, is capped at 40 concurrent for no reason and pays a thread hop per request. Both directions are wrong; only one takes the service down.
+
+### The symptom, so you recognise it before you profile
+
+- Throughput flattens as concurrency rises while **CPU sits well under saturation** — the classic reading is 25% CPU and a queue.
+- p99 climbs on endpoints that share nothing with the slow one, including `/livez`, because they share a loop.
+- Under gunicorn's async worker: `[CRITICAL] WORKER TIMEOUT`. Async workers keep a heartbeat file that the loop must refresh; a blocked loop stops the heartbeat, so this message means *you blocked the event loop*, not *the request was slow*. Raising `--timeout` hides it.
+
+**Detection, not inspection.** Reading the handler is how this survives three refactors:
+
+```python
+# Any callback holding the loop longer than this is logged with its duration.
+loop = asyncio.get_running_loop()
+loop.set_debug(True)
+loop.slow_callback_duration = 0.1   # asyncio's own default is 0.1s; keep it, don't raise it
+# -> "Executing <Handle ...> took 0.512 seconds"
+```
+
+Run it in dev and in load tests. Export event-loop lag as a gauge in production ([observability.md](observability.md)).
+
+**Forbidden by name:** never add `async` in front of a function to "make it faster". `async def` around `requests.get` is strictly worse than the `def` it replaced. Never call `asyncio.run()` inside a running loop (`RuntimeError: asyncio.run() cannot be called from a running event loop`), and never `loop.run_until_complete` from inside a handler. Never raise the anyio thread limiter above 40 to fix a stall — a bigger threadpool against a fixed pool of database connections just moves the queue ([testing-and-perf.md](testing-and-perf.md) §8).
+
+## 3. pydantic v2 at the Edge
+
+The schema is the boundary; per [validation-and-errors.md](validation-and-errors.md) it is the only thing that touches the raw body, and unknown keys are rejected.
+
+```python
+class OrderCreate(BaseModel):
+    model_config = ConfigDict(extra="forbid", strict=False)
+    customer_id: UUID
+    currency: Literal["IDR", "USD"]          # closed set, never str
+    amount_minor: int = Field(gt=0, le=100_000_000)   # minor units, per data-modelling.md
+
+    @field_validator("currency", mode="after")
+    @classmethod
+    def upper(cls, v: str) -> str: return v.upper()
+
+    @model_validator(mode="after")
+    def cross_field(self) -> "OrderCreate":   # mode="after" gets the model, and must return it
+        ...
+        return self
+```
+
+**Three models per resource, not one.** `XCreate` (input), `XUpdate` (every field optional), `XPublic` (output, a hand-written field list). **Never return the ORM instance and never derive the response model from the table.** Two failures, both real:
+
+- **Silent disclosure.** A response model built by mirroring the ORM columns exposes every column added later. The day someone adds `password_hash` or `internal_notes`, the API starts returning it and no diff touched the router.
+- **Serialisation-time I/O.** FastAPI serialises after the request-scoped session has closed. A lazy relationship touched during serialisation raises `DetachedInstanceError` (sync) or `sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called` (async) — an error that points at the serialiser and is caused by the loader config in section 4.
+
+Facts worth carrying:
+
+- v2 is **lax by default**: `"123"` validates into `int`, and `bool` accepts `"yes"`, `"on"`, `"t"`, `1`. Where a coercion would hide a client bug, set `strict=True` on the field or `ConfigDict(strict=True)` — and decide it, don't inherit it.
+- `from_attributes=True` (v1's `orm_mode`) lets a model read a SQLAlchemy object. It reads whatever attributes it names — so name them.
+- `Field(exclude=True)` is a serialisation preference, not a security control. If a value must never leave, it is not on the output model at all.
+- v1→v2 renames that produce confusing errors: `@validator`→`@field_validator`, `@root_validator`→`@model_validator`, `.dict()`→`.model_dump()`, `.parse_obj()`→`.model_validate()`, `class Config`→`model_config = ConfigDict(...)`, `allow_population_by_field_name`→`populate_by_name`.
+- Validation runs in Rust (`pydantic-core`); a `mode="before"` validator drops back into Python per field. On a hot list endpoint, that is where the serialisation cost went.
+
+## 4. SQLAlchemy 2.0
+
+### Session scope
+
+One session per request, opened and closed by the framework, never a module-level global:
+
+```python
+engine = create_async_engine(DSN, pool_size=9, max_overflow=0)   # size per testing-and-perf.md §8
+SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
+
+async def get_session() -> AsyncIterator[AsyncSession]:
+    async with SessionLocal() as session:
+        yield session          # commit in the service layer, not here
+```
+
+### expire_on_commit — the surprise
+
+`Session(expire_on_commit=True)` is the **default**. After `commit()`, every attribute on every instance in the identity map is expired; the next attribute access emits a fresh `SELECT`. Symptoms, in order of how often they appear:
+
+- `MissingGreenlet` on an async session, because that refresh is implicit I/O outside a greenlet context.
+- `DetachedInstanceError: Instance <Order> is not bound to a Session` when the session already closed — typically at response-serialisation time.
+- No error at all: one extra `SELECT` per committed object, so a bulk endpoint doubles its query count invisibly.
+
+The fix is `expire_on_commit=False` (the documented recommendation for asyncio) plus building the response model **before** commit, or an explicit `await session.refresh(obj, ["status"])` when you genuinely need the database's value back.
+
+### Lazy loading is the N+1 engine
+
+Relationships default to `lazy="select"`: touching `order.items` in a loop emits one query per order. [data-modelling.md](data-modelling.md) owns detection and the query-count budget; here is the SQLAlchemy expression of its fix table.
+
+| Relationship shape | Loader | What it emits |
+| --- | --- | --- |
+| to-many collection (`order.items`) | `selectinload(Order.items)` | second `SELECT ... WHERE parent_id IN (...)`, batched 500 ids at a time, no row multiplication |
+| many-to-one / one-to-one (`order.customer`) | `joinedload(Order.customer)` | one `LEFT OUTER JOIN` |
+| to-many **with `LIMIT` on the parent** | `selectinload` only | `joinedload` multiplies rows and forces a subquery wrap |
+| you must **filter** on the related table | `.join(Order.items).options(contains_eager(Order.items))` | one query, filtered collection |
+
+`joinedload` does **not** filter. `.options(joinedload(Order.items)).where(Item.sku == x)` returns every item of the matching orders while looking like it filtered — a wrong-results bug, not a slow one. Use `contains_eager` when the join is also the filter.
+
+**Make an accidental lazy load loud** rather than slow: `relationship(..., lazy="raise_on_sql")` raises `InvalidRequestError` when the loader would issue SQL, so the N+1 fails in CI instead of degrading in production. Async already does this via `MissingGreenlet`; sync needs the flag.
+
+### A session is not thread safe and not task safe
+
+One `Session`/`AsyncSession` is a single connection and a single unit of work. Two coroutines using it concurrently — the seductive `asyncio.gather(fetch_a(s), fetch_b(s))` — produce `IllegalStateChangeError` / `InvalidRequestError: this session is provisioning a new connection; concurrent operations are not permitted`, or asyncpg's `InterfaceError: cannot perform operation: another operation is in progress`. Under load these arrive intermittently and get filed as "flaky database".
+
+**Concurrency means a session each**, from the same engine and pool — and remember each one occupies a pool slot, so `gather` over 20 sessions against a pool of 9 is a queue by construction. For WSGI/threads use `scoped_session`; for asyncio prefer per-request injection over `async_scoped_session`.
+
+Also: 2.0 style is `select()` + `session.execute()` / `session.scalars()`. `session.query()` is legacy and its loader defaults differ; do not mix the two in one codebase. Use `session.get(Order, id)` for primary-key lookups — it checks the identity map first.
+
+## 5. Alembic
+
+**`--autogenerate` produces a draft, never a migration.** Read the generated file, then print the SQL it will run — `alembic upgrade <rev> --sql` — before it touches anything. [migrations.md](migrations.md) owns the lock analysis; this is the list of what the generator does not see.
+
+| Blind spot | What actually happens |
+| --- | --- |
+| **Renames** | Emitted as `drop_column` + `add_column`. Silent data loss, and it looks like a normal diff. Same for renamed tables and indexes. |
+| **Server defaults** | Not compared unless `compare_server_default=True`, and even then comparison across types is unreliable. `compare_type` defaults to `True` since Alembic 1.12; `compare_server_default` still does not. |
+| **Postgres enums** | Adding a value to a native enum is never generated. Neither is `CREATE TYPE` for a new enum used by an `ALTER`. You write `op.execute("ALTER TYPE ... ADD VALUE ...")` yourself. |
+| **Unmodelled objects** | Tables, views, extensions, and PostGIS's `spatial_ref_sys` that `target_metadata` does not know about are diffed as *drops*. Filter with `include_object` / `include_name` or autogenerate will offer to delete your database. |
+| **CHECK constraints, partial indexes, expression indexes** | Generally not reflected or compared. Hand-write them. |
+| **Concurrent DDL** | `CREATE INDEX CONCURRENTLY` cannot run inside the runner's transaction — SQLSTATE `25001`. Alembic's escape hatch is an autocommit block, and nothing else goes in that file. |
+
+Two heads (`alembic heads` returning more than one) is a merge, not a rerun: `alembic merge`. And per [migrations.md](migrations.md), a migration that has run anywhere — CI included — is never edited.
+
+## 6. ASGI, WSGI, and Worker Topology
+
+WSGI is one synchronous callable per request: no WebSockets, no streaming request bodies, no lifespan. ASGI adds all three and lets one thread hold thousands of open connections. FastAPI/Starlette are ASGI-only; Django serves either and only its ASGI entrypoint runs async views concurrently.
+
+| Situation | Run it as |
+| --- | --- |
+| Container orchestrated (Kubernetes, ECS), one process per container | `uvicorn app:app --host 0.0.0.0 --port 8000` — let the orchestrator scale, and keep pool arithmetic honest |
+| One VM, need process supervision, worker recycling, `--max-requests` for a leak | `gunicorn -k uvicorn_worker.UvicornWorker -w N app:app` (the `uvicorn.workers` module was deprecated in uvicorn 0.30 and moved to the `uvicorn-worker` package) |
+| Legacy WSGI app (Flask, Django sync) | `gunicorn -w N app:app`, sync workers, `2 * cores + 1` — gunicorn's own documented formula, and it applies **only** to sync workers |
+| Async workers | 1 per core at most. One async worker already multiplexes; more workers only multiply the DB pool. |
+
+**Workers multiply everything.** `-w 4` with `pool_size=9` is 36 connections from one container, before replicas. That is the arithmetic in [testing-and-perf.md](testing-and-perf.md) §8, and the reason [deploy-and-config.md](deploy-and-config.md) calls the process-model concurrency factor aged. Never fork after opening the pool: create the engine in a lifespan handler or per worker, or children inherit a socket and you get `InterfaceError` on the second request.
+
+## 7. The GIL, Honestly
+
+One thread executes Python bytecode at a time per interpreter. What it does **not** stop is the case that matters most here:
+
+- **Released around blocking I/O** — sockets, files, `time.sleep`. This is why FastAPI's 40-thread pool genuinely serves concurrent sync database calls.
+- **Released by C extensions that ask** (`Py_BEGIN_ALLOW_THREADS`): numpy on large arrays, `hashlib` on big buffers, zlib, most DB drivers while waiting on the wire.
+- **Not released for pure-Python CPU work.** Four threads doing Python arithmetic on four cores are *slower* than one, because the interpreter hands the GIL around every `sys.setswitchinterval` — 5 ms by default — and each handoff is pure overhead.
+
+| Work | Answer |
+| --- | --- |
+| Blocking I/O | Threads. `def` handler, or `anyio.to_thread.run_sync`. |
+| Pure-Python CPU, > ~50 ms, small arguments and results | `ProcessPoolExecutor` — but everything crosses via `pickle`, so a 200 MB DataFrame argument costs more than the compute |
+| Pure-Python CPU, large data or long-running | Not in the request path at all. Queue it ([concurrency-and-jobs.md](concurrency-and-jobs.md)). |
+| Numeric work | A library that releases the GIL (numpy, polars). Threads then actually scale. |
+
+The GIL is **not** a lock you can borrow: `counter += 1` and `d[k] += 1` are multiple bytecodes and interleave. Use `threading.Lock`, or don't share the object.
+
+**Free-threaded CPython (PEP 703, `python3.13t`, officially supported from 3.14) is not the fix for a CPU problem today.** It is not the default build, single-threaded performance is lower, and a large share of C-extension wheels still assume the GIL. Do not put a service on it to avoid a process pool.
+
+## 8. Celery
+
+The policy — at-least-once, idempotent consumers, backoff with jitter, DLQ ownership — is [concurrency-and-jobs.md](concurrency-and-jobs.md) and is not restated. These are the Celery-specific ways to get it wrong.
+
+```python
+@app.task(
+    bind=True,
+    acks_late=True,                    # ack AFTER the task completes; without it a worker
+    task_reject_on_worker_lost=True,   # crash loses the task silently, with nothing in the log
+    autoretry_for=(TransientError,),
+    retry_backoff=True, retry_jitter=True, max_retries=5,
+    soft_time_limit=55, time_limit=60, # both default to None = a task may run forever
+)
+def charge(self, payment_id: str) -> None:   # an id, never an ORM object or a dict of one
+    ...
+```
+
+`acks_late=True` makes delivery at-least-once by construction, which is why the sibling's idempotency rule is not optional. Then:
+
+- **Redis broker has no real ack.** Celery emulates one with `broker_transport_options={"visibility_timeout": N}`, **default 3600 seconds**. A task that runs longer than that is handed to a second worker while the first is still running — the duplicate-execution mechanism the sibling's "3× p99" rule exists to prevent. It also breaks `countdown`/`eta`: a task scheduled 2 hours out on Redis is redelivered and re-executed every hour until it fires. Use a broker with real acks (RabbitMQ, SQS) for delayed work, or raise the timeout above the longest ETA and know you did.
+- **`worker_prefetch_multiplier` defaults to 4.** Each worker reserves 4 × concurrency messages up front, so short tasks queue behind a long one on a worker while other workers idle. For long or uneven tasks, set it to `1` — Celery's own documented advice.
+- **Result backend is a write per task.** Unless something reads it, set `task_ignore_result=True`. A million-task fan-out otherwise writes a million Redis keys that expire at `result_expires`, default 86400 s. And `AsyncResult.get()` inside a task is a deadlock: Celery raises `RuntimeError: Never call result.get() within a task!` — chain or chord instead.
+- **Pool choice**: prefork (default) for CPU-ish tasks, `--pool=gevent`/`eventlet` for I/O-bound fan-out, `--pool=solo` only to debug. Prefork workers each hold their own database pool; count them in the arithmetic from section 6.
+
+For a service already async end to end, `arq` or `dramatiq` are smaller and speak asyncio natively. Celery's argument is operational maturity, not ergonomics.
+
+## 9. Packaging and the Environment
+
+**A virtualenv, always. `pip install` outside one is the failure mode the harness escalates on** — it mutates the machine's interpreter, and on macOS and Debian PEP 668 blocks it outright with `error: externally-managed-environment`. Where it is *not* blocked is worse: you have quietly changed the OS's Python.
+
+```bash
+uv venv && uv pip sync requirements.txt   # exact: also REMOVES anything not in the file
+# or
+python3 -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
+```
+
+- **A lockfile, resolved and committed** — `uv.lock`, `poetry.lock`, or pip-tools output with `--generate-hashes`. An unpinned `fastapi` in `requirements.txt` means a different build every day; the symptom is "it worked yesterday" and a major-version import error nobody's diff caused. CI runs the frozen form (`uv sync --frozen`) so a stale lock fails the build instead of resolving around it.
+- **Pin the interpreter too**: `requires-python` in `pyproject.toml`, and a base image of `python:3.12-slim-bookworm`, never `python:latest`.
+- **Never `pip install` something to see if it fixes the bug** and leave it out of the lock. The next container does not have it.
+- **`sys.path.append` and `PYTHONPATH` juggling** mean the project is not installed. `pip install -e .` with a real `[project]` table, and import by package name.
+- Secrets reach the process through the environment and are parsed at boot ([deploy-and-config.md](deploy-and-config.md)); `pydantic-settings` failing at import is the correct behaviour, not an inconvenience.
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| A blocking call inside `async def` | `def` handler, or `await anyio.to_thread.run_sync(...)` |
+| `async` added to a sync function "for speed" | Leave it `def`; the threadpool is the mechanism |
+| Raising the anyio thread limiter to fix a stall | Find the blocked loop; the pool is downstream |
+| Returning an ORM instance from a handler | An explicit `XPublic` model with a hand-written field list |
+| A pydantic model without `extra="forbid"` | `model_config = ConfigDict(extra="forbid")` |
+| A response model mirroring the table's columns | Name the fields; new columns must not auto-publish |
+| Default `expire_on_commit=True` on an async session | `expire_on_commit=False` + build the response before commit |
+| Relationship left at `lazy="select"` on a list endpoint | `selectinload` / `joinedload` per the table in §4 |
+| `joinedload` used to filter a collection | `.join(...)` + `contains_eager(...)` |
+| One session shared across `asyncio.gather` | A session per task, counted against the pool |
+| An autogenerated Alembic migration run unread | Read it, then `alembic upgrade <rev> --sql` |
+| `target_metadata` with no `include_object` filter | Filter, or autogenerate proposes dropping unmodelled tables |
+| Async workers sized with `2 * cores + 1` | ≤1 per core; that formula is for sync workers only |
+| Threads for pure-Python CPU work | `ProcessPoolExecutor`, or a queue |
+| Celery without `acks_late` | `acks_late=True` + `task_reject_on_worker_lost=True` + idempotent task |
+| `visibility_timeout` left at Redis's 3600s under longer ETAs | A broker with real acks, or a timeout above the longest ETA |
+| `task_time_limit` / `soft_time_limit` unset | Both set, soft strictly below hard |
+| `result.get()` inside a task | A chain or chord |
+| `pip install` outside a virtualenv | `uv venv` / `python3 -m venv .venv` first |
+| Unpinned dependencies, or an unlocked ad-hoc install | A committed lockfile, installed frozen in CI |
+
+## No Guessing
+
+- **Never claim a handler is non-blocking because it is declared `async def`.** Enable `loop.set_debug(True)` with `slow_callback_duration = 0.1` and read what it logs, or measure throughput at two concurrency levels and watch CPU.
+- **Never state a query count from reading the mapping.** Count them — [data-modelling.md](data-modelling.md) owns the budget and the two-page-size assertion.
+- **Never trust an autogenerated migration's SQL.** Print it.
+- **Never invent a Celery, uvicorn, gunicorn, or anyio default.** They are in this file or in the project's config; a plausible number here reads identical to a measured one.
+
+## Output
+
+Write the code. Then, briefly: for every path operation you touched, whether it is `async def` or `def` and the one blocking call that decided it; the session scope and the loader options on any relationship the response reads; what the Alembic diff got wrong when you read it; and the verification — the request you actually made, the status code, and the query count or loop-lag reading if the change was about either. If the honest answer was "leave it on Django", "this doesn't need to be async", or "no migration", say it in one line and write nothing further.