vibecodeagent 0.0.0-stage → 0.1.64

package/dist/resources/skills/builtin/backend-engineering/references/testing-and-perf.md

@@ -0,0 +1,298 @@
+# Testing and Performance
+
+This file owns two claims: *this service works* and *this service is fast enough*. That means the test shape for a backend, what may be faked, how test data is built, and the latency/throughput playbook — triage order, pool sizing, caching, load testing. Schema, index design, and the N+1 *fixes* are [data-modelling.md](data-modelling.md); this file owns finding the problem and proving the fix. Locks, transactions, and queue mechanics are [concurrency-and-jobs.md](concurrency-and-jobs.md) — the concurrency *test* is here. What a slow request emits, the histogram it lands in, and the alert on it are [observability.md](observability.md). Timeout values, env config, and graceful shutdown are [deploy-and-config.md](deploy-and-config.md). Assertions on parsed input and error bodies are [validation-and-errors.md](validation-and-errors.md).
+
+`kerja-agentik` owns running the suite, reading the failure, and the rule that you never claim a test passes without executing it. Nothing here repeats that. This file only decides **what to assert and what to measure**.
+
+## Operating Posture
+
+Two failure modes, and the first is worse:
+
+1. **A green suite against something that is not your production database.** A mocked repository, a fake in-memory store, SQLite standing in for Postgres. The tests pass forever, including on the commit that ships the bug, because the constraint, the lock, and the type coercion under test do not exist in the thing you tested against. This defect is invisible in review — the diff looks like a well-tested feature.
+2. **Optimising by guess.** Rewriting the serialiser while the N+1 sits untouched, or raising the pool size to fix a stall that a bigger pool makes worse. Costs a week, moves p99 by nothing, and leaves the real cause in place with a fresh commit on top of it.
+
+**The right number of new tests is sometimes zero, and the right cache is often no cache.** A test that asserts the framework routes a request, a second test at a different page size that encodes the same property, a cache in front of a query that is missing an index — all defects. Say "this needs an index, not a cache" and add nothing.
+
+## 1. The Test Shape for a Service
+
+Not the pyramid. A service is thick in the middle, because a backend's logic is mostly SQL, constraints, transaction boundaries, and authorization filters, and none of those are pure functions.
+
+| Layer | Share | What lives there | Touches |
+| --- | --- | --- | --- |
+| **Unit** | thin | Genuinely pure logic: money arithmetic, a state machine, a parser, a permission predicate over already-loaded data | Nothing |
+| **Integration** | **thick — the bulk** | A handler end-to-end in-process: real router, real parsing, real queries, real transactions, real constraints, real authorization | Real database, stubbed third-party HTTP |
+| **End-to-end** | a few | The deployed process over the network: boot, config, migrations, health, one happy path per critical flow, one auth failure | Everything real, sandbox providers |
+
+**The integration layer is where backend bugs actually are.** The bug is a missing `WHERE tenant_id`, a `UNIQUE` that fires under a race, a `NOT NULL` the code path never sets, a migration that has not run, a rollback that leaves a row. A unit test cannot see any of them, because in a unit test none of them exist.
+
+| The change | The test that proves it |
+| --- | --- |
+| New endpoint | Integration: happy path, one `4xx` per validation class, one authorization denial from a *different* tenant |
+| A query got a filter | Integration: seed a row that must be excluded, assert it is absent |
+| A migration | Integration suite runs against a database built by the migration chain |
+| Money/time arithmetic | Unit, exhaustively — rounding, negative, zero, DST boundary |
+| A write that can collide | Concurrency test (§5) — not a unit test |
+| A serialiser field rename | Integration: assert the response body keys, not the model |
+
+## 2. Test Against a Real Database
+
+Not negotiable. The bugs live in the SQL, the constraints, the transaction semantics, and the migration; a mock contains none of those, and neither does another engine.
+
+```
+Ephemeral schema per test  — fastest, one Postgres, CREATE SCHEMA t_<uuid>; search_path
+Template database          — CREATE DATABASE test_x TEMPLATE tmpl (no session may be
+                             connected to tmpl), migrations run once per suite
+Testcontainers             — one container per suite run; use for CI parity and when the
+                             suite needs extensions, roles, or a replica
+```
+
+**Build the test database by running the migrations, never by loading a hand-written `schema.sql`.** Otherwise the migration chain is the one thing in the repo with no test, and it breaks in production ([migrations.md](migrations.md)).
+
+Between tests, prefer `TRUNCATE <tables> RESTART IDENTITY CASCADE` over wrapping each test in a rolled-back transaction. The rollback trick is faster and breaks the moment the code under test opens its own transaction, commits, tests `SERIALIZABLE` retry, or asserts on an `AFTER COMMIT` hook — the symptom is a test that passes in isolation and reports "no rows" when the handler commits into a transaction that never lands.
+
+**Named forbidden substitutes.** These are the plausible-looking guesses; refuse each by name:
+
+| Substitute | Why it produces a false green |
+| --- | --- |
+| SQLite for Postgres | Type affinity: `'abc'` into an `INTEGER` column stores the string, silently. `PRAGMA foreign_keys` is **OFF** by default, so FK violations do not raise. No `FOR UPDATE`, no `SKIP LOCKED`, no advisory locks — every locking test is a no-op. No `jsonb`, `timestamptz`, arrays, `ILIKE`, `DISTINCT ON`. Database-wide single writer, so no test can observe a row-level race. |
+| `pg-mem` / H2 / an in-memory dialect | Reimplements a subset. The planner, the constraint timing, and the isolation semantics you are testing are the subset it skipped. |
+| A mocked repository / stubbed ORM session | Asserts that your mock returns what you told it to. Tests the test. |
+| Mocking your own service/module | Same. If it is your code, run it. |
+
+## 3. What to Fake, and How
+
+| Dependency | Fake it? | How |
+| --- | --- | --- |
+| Your database | **Never** | §2 |
+| Your own modules | **Never** | Call them |
+| Third-party HTTP | **Yes**, at the HTTP layer | `nock`, `responses`/`respx`, `httpx.MockTransport`, WireMock — not by stubbing the client class |
+| The clock | **Yes**, injected | A `now()` dependency passed in |
+| Randomness, uuid, token generation | **Yes**, injected | A seeded generator passed in |
+| Queue / broker | **Yes** for producers | Assert the enqueued message; test the handler by calling it directly |
+| Payments, email, SMS | Stub in integration; sandbox in one e2e | Provider sandbox keys, never live |
+
+**Stub at the transport, not at the client object.** A stubbed `PaymentClient.charge()` can only return what you imagined. An HTTP-level stub can return `503`, a truncated body, a `200` with the wrong content type, and a socket that never answers — which is how the dependency actually fails, and the only way to test the timeout and retry path ([concurrency-and-jobs.md](concurrency-and-jobs.md)).
+
+**Inject time and randomness; do not patch them globally.** `freezegun`, `jest.useFakeTimers()`, and monkeypatched `time.time` freeze the whole process — including the connection pool's keepalive, the driver's statement timeout, and the test framework's own timers. The symptom is unmistakable and gets misdiagnosed for days: a test that passes alone and **hangs** in the full suite, or a pool that never reaps because its timer is pinned to 2021.
+
+```python
+# Injected, not patched. The handler takes its clock; the test hands it one.
+def expire_session(session, *, now: Callable[[], datetime]) -> bool:
+    return session.expires_at <= now()
+
+expire_session(s, now=lambda: FIXED)     # deterministic, process-wide side effects: none
+```
+
+## 4. Test Data: Factories, Never a Shared Seed
+
+A factory builds exactly the row the test asserts on and fills every other required column with a unique generated default. A shared `seed.sql` or a session-scoped fixture set couples every test to every other test.
+
+The symptom: **the failing test is one nobody touched.** Someone adds a row to the seed file for their feature, and a count assertion, an ordering assertion, or a "first result" assertion in an unrelated module goes red. The team's next move is to make the assertions looser, and from then on the suite proves less every month.
+
+- Every value the test asserts on appears **in the test body**. If you have to open another file to know why the expected number is 3, the test is unreadable and will be deleted rather than fixed.
+- Unique-by-default for anything with a `UNIQUE` constraint — `f"user-{uuid4()}@example.test"`. Shared literals make two tests collide only when run in the same transaction, i.e. only in CI.
+- Build the minimum. A factory that creates an org, a team, a project, and three users for a test about email formatting is a fixture wearing a factory's clothes.
+- One exception: genuinely static reference data (currencies, country codes) is fine as a seed, because no test ever mutates it.
+
+## 5. Testing Concurrency for Real
+
+The only test that proves an idempotency key, a unique constraint, or a `SELECT ... FOR UPDATE` does anything. Fire the same request in parallel and assert the **database state**, not the responses.
+
+```ts
+const N = 8;                                   // ≥8: two requests can be serialised by luck
+const results = await Promise.allSettled(
+  Array.from({ length: N }, () =>
+    POST('/api/orders', body, { 'Idempotency-Key': 'k-1' })),
+);
+
+const { rows } = await db.query(
+  'SELECT count(*)::int AS n FROM orders WHERE idempotency_key = $1', ['k-1']);
+expect(rows[0].n).toBe(1);                     // the real assertion: one row
+expect(results.filter(r => status(r) === 201)).toHaveLength(1);
+expect(results.filter(r => [200, 409].includes(status(r)))).toHaveLength(N - 1);
+```
+
+- **Two sequential-ish requests are not a race.** Use a barrier (`asyncio.gather`, a `threading.Barrier`, `Promise.all` over pre-built requests) and at least 8 in flight. A test where request two left 40 ms after request one passes against code with no protection at all.
+- **Assert the row count, never only the status codes.** A handler can return one `201` and one `409` while having written two rows, if the `409` came from a second check after the insert.
+- **Then assert the constraint exists**, because the database is what actually wins the race: `SELECT conname FROM pg_constraint WHERE conrelid = 'orders'::regclass`. Application-level "check then insert" loses under concurrency by construction.
+- Test the retryable path too: with `SERIALIZABLE`, some of those requests fail with SQLSTATE `40001` and must be retried, not returned as `500`.
+
+## 6. p50 / p95 / p99 — an Average Describes Nobody
+
+Mean latency is the one number that belongs to no user. A service at 40 ms mean with a 2 s p99 is a service where the most engaged users — the ones with the most rows — are the ones hitting the tail.
+
+| Percentile | Who it is |
+| --- | --- |
+| p50 | The typical request. Moves when the common path changes. |
+| p95 | The user with real data. This is where missing indexes appear first. |
+| p99 | The page a user actually notices and reports. |
+| p99.9 | Pool waits, GC pauses, lock waits, cold caches. The incident, before it is one. |
+
+**Tail latency compounds across fan-out.** A page that makes 20 backend calls has a `1 − 0.99²⁰ ≈ 18%` chance of touching at least one p99 call. A p99 the size of a timeout is therefore a p82 page-load problem.
+
+- **Never average a percentile** across instances, pods, or time buckets — the arithmetic is meaningless. Aggregate the histogram: `histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))`. Instrumentation is [observability.md](observability.md).
+- **Measure from the edge, not from inside the handler.** A timer that starts after the framework hands you the request excludes pool-acquire wait, event-loop queueing, and body read. The symptom is a well-known argument: "our metrics say 12 ms, users say 3 s." The users are right; your timer starts too late.
+- Report p50/p95/p99 together, always with the request count. A p99 over 40 requests is one request.
+
+## 7. Performance Triage, In This Order
+
+Work down. Each row states how to **confirm** it, because the whole point is not to guess. Change one thing, re-measure, then move on.
+
+| # | Suspect | Symptom | Confirm it |
+| --- | --- | --- | --- |
+| 1 | **Query count (N+1)** | Slow request, every query fast; latency scales with page size | Count queries per request; assert the count is constant across two page sizes ([data-modelling.md](data-modelling.md)) |
+| 2 | **Missing index** | One slow query; p95 worse than p50 by 10× | `EXPLAIN (ANALYZE, BUFFERS)` shows `Seq Scan` on a large table, or `Rows Removed by Filter` in the thousands |
+| 3 | **Payload size** | Time scales with bytes, not rows; mobile worse than desktop | `curl -w '%{size_download} %{time_total}\n'`; look for `SELECT *`, an uncapped `limit`, an embedded blob |
+| 4 | **Serialisation cost** | Query time flat, total time rising with page size | Flamegraph / `py-spy` / `--cpu-prof`: time spent after the last query returns |
+| 5 | **Pool exhaustion** | Bimodal latency; unrelated endpoints slow together; SQLSTATE `53300` | Pool wait-time metric > 0; `SELECT count(*), state FROM pg_stat_activity GROUP BY state` |
+| 6 | **Chatty service calls** | Latency = sum of hops; tail dominated by one dependency | Trace span count per request; a request with 30 spans to one service is the answer |
+| 7 | **Sync work that should be a job** | One endpoint slow, CPU or third-party bound, caller does not need the result | Time the handler section; if it is an email, a webhook, or a PDF, it is a job ([concurrency-and-jobs.md](concurrency-and-jobs.md)) |
+
+```sql
+-- Where the database's time actually goes. Requires the pg_stat_statements extension.
+SELECT calls, round(total_exec_time)   AS total_ms,
+       round(mean_exec_time::numeric,2) AS mean_ms, rows, query
+FROM pg_stat_statements
+ORDER BY total_exec_time DESC
+LIMIT 20;
+-- Read total_ms first, never mean_ms. The 0.4 ms query called 200 times per request
+-- is the bug; the 90 ms report nobody runs is not.
+```
+
+**One change per measurement.** Adding an index and rewriting the serialiser in one commit means you learn nothing, and the next person reverts both.
+
+## 8. Connection Pool Sizing
+
+**A bigger pool is usually slower.** The database has a fixed number of cores and disks; connections beyond that number do not add throughput, they add context switches, lock contention, and per-backend memory. Postgres forks a process per connection — 400 connections on an 8-core box is 400 processes fighting over 8 cores while each holds its own work memory.
+
+```
+per-instance pool  =  (cpu_cores * 2) + effective_spindle_count     # HikariCP / PG wiki
+hard ceiling:  pool_size <= (max_connections - superuser_reserved_connections
+                             - other_consumers) / instance_count
+```
+
+Postgres defaults: `max_connections = 100`, `superuser_reserved_connections = 3`. Worked, for a 4-core database:
+
+```
+formula          : (4 * 2) + 1 (SSD)          = 9 per instance
+budget           : 100 - 3 = 97 usable
+other consumers  : 8 job workers x 4  = 32
+                   migration runner    =  2
+                   metrics exporter    =  2
+                   humans in psql      =  5   -> 41 reserved
+app instances    : 6
+ceiling          : (97 - 41) / 6 = 9.3        -> pool_size = 9
+```
+
+Cross-check with Little's Law: a request that holds a connection for 5 ms means one connection serves 200 rps, so a pool of 9 covers 1,800 rps of database work. If your traffic is 300 rps, a pool of 40 was never doing anything except making the stall worse.
+
+- **Count every pool.** A read-replica pool, a worker pool, and the migration runner are separate consumers. The classic incident: "we scaled to 20 instances and everything broke" — 20 × 20 connections against `max_connections = 100`, so most instances got `53300` at boot.
+- **Acquire timeout must be shorter than the request timeout**, so exhaustion fails fast with a `503` instead of queueing until every request times out at once. The concrete timeout values and their ordering belong to [deploy-and-config.md](deploy-and-config.md).
+- **Beyond the ceiling, add PgBouncer in `transaction` mode, not connections.** Then respect what transaction pooling breaks: session state, `SET` (use `SET LOCAL` — see [auth-and-secrets.md](auth-and-secrets.md)), server-side prepared statements without protocol support, `LISTEN/NOTIFY`, and advisory locks held across statements.
+- **Pool wait time is a first-class metric.** Without it, exhaustion is indistinguishable from a slow database, and the two fixes are opposites.
+
+## 9. Caching
+
+**Fix the query first.** A cache in front of a missing index hides the defect until the first cold start, then delivers it all at once. Cache to avoid *unavoidable* work, never to avoid work you have not looked at.
+
+### Key design — the key contains everything that varies the answer
+
+```
+cache:v3:t:{tenant_id}:u:{user_id}:invoice:{id}:proj:{fields}:s:{schema_version}
+      │   │            │           │           │              └ bust the whole space on deploy
+      │   │            │           │           └ two projections are two answers
+      │   │            │           └ the entity
+      │   │            └ ANY per-principal filtering must be in the key, or it leaks
+      │   └ tenant scope, always
+      └ key format version — the only safe way to change key shape without a flush
+```
+
+**Never cache an authorization-dependent answer under a key that omits the principal.** The symptom is not slow, it is user B opening user A's dashboard — that is Boundary Checklist #1 wearing a cache costume, and it will be reported as a security incident, correctly. If the response varies by who is asking and you cannot put the asker in the key, do not cache it. The HTTP-layer equivalent: a response that varies by credentials needs `Cache-Control: private` and `Vary: Authorization` (RFC 9110 §12.5.5), or a shared proxy will do the leaking for you.
+
+Never cached, at all: anything with `Set-Cookie`, a token, a "things I can see" list keyed by query alone, or a permission decision keyed by resource alone.
+
+### TTL vs explicit invalidation
+
+| The data | Strategy |
+| --- | --- |
+| Expensive, staleness tolerable, no clean write event (aggregates, third-party lookups) | TTL only, jittered |
+| Single known writer that can emit an event (an entity you own) | **Delete on write, plus a TTL as the safety net** |
+| Correctness-critical (permissions, balances, flags gating money) | Don't cache, or ≤5 s TTL with an explicit bust |
+| Derived from many writers across services | TTL only — invalidation you cannot enumerate is invalidation you will miss |
+
+**Invalidation without a TTL is permanent staleness waiting for one missed code path.** Always both. And **delete the key on write, never update it** — an update races with an in-flight recompute and can write a value older than the one it replaced.
+
+### Stampede — the failure that only appears under load
+
+A hot key expires; the N requests in flight all miss and all run the expensive query simultaneously. The symptom is a **latency spike at exactly the TTL interval**, worsening as traffic grows, and it looks like a database problem because the database is what falls over.
+
+```python
+ttl = base_ttl * random.uniform(0.9, 1.1)   # jitter: 10k keys warmed by one deploy must
+                                            # not all expire in the same second
+```
+
+```ts
+// Single-flight: one computation per key per process; everyone else awaits it.
+const inflight = new Map<string, Promise<T>>();
+async function single(key: string, compute: () => Promise<T>) {
+  const running = inflight.get(key);
+  if (running) return running;                    // no second query, no second miss
+  const p = compute().finally(() => inflight.delete(key));
+  inflight.set(key, p);
+  return p;
+}
+```
+
+Per-process single-flight covers most of it. Across processes, take a short Redis lock — `SET lock:{key} {token} NX PX 5000` — where the loser serves the stale value or waits; Go has `x/sync/singleflight` built for exactly this. For a genuinely hot key, probabilistic early recomputation (XFetch, VLDB 2015) recomputes before expiry with rising probability, so the key is never cold.
+
+### Negative caching
+
+Cache the miss, briefly and distinctly — a `NOT_FOUND` sentinel with a 30 s TTL, never an empty string that a later read cannot tell from a miss. Without it, every lookup for a nonexistent id is a full uncached database hit, and an enumeration script becomes a denial of service that never touches a cached key. Two rules: **delete the negative key on insert**, or a just-created resource 404s for its whole TTL; and never negatively cache an *authorization* denial keyed without the principal.
+
+**Hit rate is not the metric — p99 with the cache cold is.** A 98% hit rate at 200 rps still sends 4 rps down the miss path, and after a deploy it briefly sends all 200. Report the cold number or you have not designed the cache, only the happy path.
+
+## 10. Load Testing That Means Something
+
+| Rule | Why |
+| --- | --- |
+| **Model the real mix** | Take the read/write ratio and endpoint distribution from the access log. A guessed 50/50 mix tests a service you do not run. |
+| **Real key distribution** | Uniform random ids over 10M rows gives a ~0% cache hit rate and a perfectly cold buffer pool. Real traffic is Zipf: a few hot keys. Uniform tests neither the cache nor the hot-row contention. |
+| **Warm first, then measure** | Discard the first 60 s: cold pool, cold cache, cold plans, un-JITted code. A benchmark of your cold start is a benchmark of nothing you serve. |
+| **Run ≥10 minutes** | Below that you never see GC pauses, connection recycling, TTL expiry waves, autovacuum, or the slow leak that ends the shift. |
+| **Open model, constant arrival rate** | Closed-loop VUs wait for their own response, so a slowing service receives less load and the numbers look fine. That is **coordinated omission**, and it hides queueing collapse entirely. Use k6 `constant-arrival-rate`, wrk2, or Gatling's open model. |
+| **Production-shaped data volume** | An index decision tested against 1,000 rows is untested; a seq scan on 1,000 rows is fast. |
+| **Report the knee, not the max** | Ramp until p99 crosses the SLO and report the rps at that point. Max rps is the number where the service is already failing. |
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| A mocked database, or SQLite standing in for Postgres | Testcontainers / ephemeral schema per test |
+| A test suite built from `schema.sql` | Build it by running the migration chain |
+| A shared seed dataset the tests assert against | Factories, unique by default |
+| Mocking your own code | Call it |
+| A stubbed client class for a third party | An HTTP-level stub that can time out and return `503` |
+| `freezegun` / global fake timers | An injected clock |
+| A "race" test with two sequential requests | Barrier, ≥8 in flight, assert the row count |
+| An average latency in a performance claim | p50 / p95 / p99 with the request count |
+| An averaged p99 across instances | `histogram_quantile` over summed buckets |
+| A perf fix with no before/after number | One change, measured with the command named |
+| Raising the pool size to fix latency | Measure pool wait time; the fix is usually a smaller pool |
+| Total pool across instances > `max_connections` | The formula and ceiling in §8 |
+| A cache key without the principal on a per-user answer | Principal in the key, or no cache |
+| A TTL with no jitter, or a hot key with no single-flight | Jitter + single-flight |
+| Invalidation with no TTL safety net | Delete on write **and** a TTL |
+| Caching before reading the query plan | Index first, cache second |
+| A 30-second closed-loop load test on uniform random ids | §10 |
+
+## No Guessing
+
+- **Never state a latency, hit rate, or throughput you did not measure.** Name the command and the run length beside the number.
+- **Never claim a cache will handle it** without stating the cold-cache p99 and the miss rate at target rps.
+- **Never name a pool size** without `max_connections`, the instance count, and every other consumer.
+- **Never declare a fix from one run.** Tail latency needs volume; compare distributions before and after, not two single requests.
+- **Never assert an index is used, or a query is the slow one, from reading code.** `EXPLAIN (ANALYZE, BUFFERS)` and `pg_stat_statements` are the sources.
+
+## Output
+
+Write the tests, or the fix. Then, briefly: which layer the test sits in and what real dependency it runs against; for a perf change, the metric before and after with the exact command that produced both, and the **one** thing you changed; for a cache, the key template, the TTL, the invalidation trigger, and the stampede protection. If the honest answer was "no new test — this is the framework's behaviour" or "don't cache this, it needs an index", say that in one line and add nothing.

package/dist/resources/skills/builtin/backend-engineering/references/validation-and-errors.md

@@ -0,0 +1,266 @@
+# Validation and Errors
+
+This file owns two boundaries: turning untrusted bytes into typed values, and turning a failure into a response. The **wire-level status code table and the response envelope** live in [api-contracts.md](api-contracts.md) — when a status code appears here it is restated from there, never re-derived. Injection, SSRF, CORS, and rate-limit policy are [security.md](security.md). Credential-shaped failures (which of `401`/`403`, token expiry, session revocation) are [auth-and-secrets.md](auth-and-secrets.md). What a failure writes to the log, and the trace id that ties it to a request, are [observability.md](observability.md). Retry *execution* — backoff, jitter, dead letters — is [concurrency-and-jobs.md](concurrency-and-jobs.md); this file only decides whether a failure is retryable and how that decision gets on the wire.
+
+`dasar-vibecode` §8 already establishes the philosophy: fail closed for money, permissions, and data integrity, and classify a retry on a **code**, never on a message string. Everything below is the backend implementation of that rule — the taxonomy those codes come from, and the response hygiene that keeps the code the only thing a client can branch on.
+
+## Operating Posture
+
+Two failure modes, and the first is worse:
+
+1. **A value that was never parsed reaches business logic.** It looks like working code, passes every test written against valid input, and fails on the day someone sends `{"amount": "10", "role": "admin"}`. The whole first half of this file exists to make that state unrepresentable rather than merely unlikely.
+2. **A failure loses its identity on the way out.** A `409` caught by a bare `except` and re-raised as a `500`, an upstream's error text pasted into your body, a retryable timeout that the client cannot tell from a permanent rejection. The second half exists to stop that.
+
+The most common validation bug is not a missing check. It is a value that *was* checked in the handler and then re-read from the raw request body three lines later — `req.body.email` after `parsed.email` was already produced. The check was real; it just wasn't load-bearing.
+
+## Parse, Do Not Validate
+
+One schema per endpoint, at the edge. It is the *only* thing that touches the raw body, and the application's types are **derived from it** so that an unparsed value has no type to travel in.
+
+```ts
+// The schema is the source of truth. The type is downstream of it, never beside it.
+const CreateOrder = z.object({
+  customerId: z.string().uuid(),
+  currency:   z.enum(["IDR", "USD"]),          // closed set, not z.string()
+  amountMinor: z.number().int().positive().max(100_000_000),
+  note:       z.string().max(2_000).optional(),
+}).strict();                                    // unknown key => reject, see below
+
+type CreateOrder = z.infer<typeof CreateOrder>; // NOT a hand-written interface
+
+// The handler receives the parsed value and never sees the request again.
+async function createOrder(input: CreateOrder, ctx: Ctx) { /* ... */ }
+```
+
+```python
+class CreateOrder(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=False)
+    customer_id: UUID
+    currency: Literal["IDR", "USD"]
+    amount_minor: int = Field(gt=0, le=100_000_000)
+```
+
+Three rules that make this real rather than decorative:
+
+- **`z.infer` / `model_dump`, never a parallel `interface`.** A hand-written type beside a schema drifts, and the drift is silent: the type says the field is required, the schema says optional, and production disagrees with the compiler.
+- **The raw request object does not leave the edge function.** If `req`, `request.json()`, or `c.Request` is reachable from a service function, the parse is advisory. Pass the parsed value; pass nothing else.
+- **Narrow past the primitives.** A `string` that is really a tenant-scoped id should be branded (`type OrderId = string & {__brand:"OrderId"}`, `NewType("OrderId", str)`) so an author id cannot be passed where an order id belongs. Money follows the hub's rule — integer minor units plus a currency, never a float, never a bare `number` called `amount`.
+
+**Permission to write nothing:** if the framework already parses into a typed model and rejects unknown fields (FastAPI + pydantic with `extra="forbid"`, a Go handler decoding with `DisallowUnknownFields`), add no second layer. Two validators eventually disagree, and the looser one is the one an attacker finds. Extend the existing schema; do not hand-roll a checker beside it.
+
+## Reject Unknown Fields by Default
+
+`.strict()` in zod, `extra="forbid"` in pydantic, `dec.DisallowUnknownFields()` in Go, `ActiveModel` strong parameters in Rails. Default is reject; allowing an unknown key is a per-endpoint decision with a reason next to it.
+
+Permissive parsing is how you get **mass assignment**. The mechanism is not exotic: the parser keeps the extra key, some layer below spreads the object into an update, and the client has just written a column it was never offered. In March 2012 Egor Homakov used exactly this against GitHub's Rails app and added his own key to the `rails/rails` repository. The bug was not in the update; it was in a parser that kept a field nobody had declared.
+
+Two related traps:
+
+- **Never `Object.assign(record, req.body)` or `Model(**data)` from untrusted input**, even after validating. Assign field by field from the parsed value. A schema that permits `role` on a different endpoint is one refactor away from permitting it here.
+- **Response models are separate types from request models.** Reusing one model for both is how internal columns (`password_hash`, `stripe_customer_id`, `internal_notes`) become fields the client can read *and* set. See the response hygiene section.
+
+## Hard Limits Every Endpoint Needs
+
+Every one of these is a denial-of-service gate, and each stops a different attack. Take the numbers from here or from the project's own config — **never invent a limit** and never write "a sensible maximum".
+
+```
+max request body (JSON)          256 KiB   # nginx client_max_body_size defaults to 1m;
+                                           # Express body-parser defaults to 100kb. 256 KiB
+                                           # covers real JSON payloads and is enforced at the
+                                           # proxy AND in-process, because only one of them
+                                           # is in front of you in every deployment.
+max request body (upload)          25 MiB  # streamed to disk/object storage, never buffered
+                                           # in memory; multipart parsers buffer by default.
+max array length per field           100   # matches the page-size cap in api-contracts.md.
+                                           # Stops "one request, 50,000 IN(...) parameters"
+                                           # and the O(n^2) validator behind it.
+max string length (scalar field)   1 KiB   # ids, emails, names, enum-ish values
+max string length (free text)     10 KiB   # notes, descriptions, comments
+max JSON nesting depth                 8   # Python's json raises RecursionError past
+                                           # sys.getrecursionlimit() (default 1000): a
+                                           # 5,000-deep array is a stack crash, not a 400.
+max query parameters                 100   # hash-flooding (CVE-2011-4885 and siblings hit
+                                           # PHP, Ruby, Java, Python at once). Note qs's
+                                           # arrayLimit defaults to 20 and parameterLimit
+                                           # to 1000 — a[100000]=x is a 100k-element array.
+request handler timeout               10s  # shorter than the caller's; see deploy-and-config.md
+db statement_timeout                   5s  # a query that outlives the request is pure waste
+```
+
+Two more that are not size limits but belong to the same gate:
+
+- **Cap the input length *before* any regex touches it.** A catastrophic-backtracking pattern is only exploitable if the attacker controls the length. `max(320)` on an email field turns a ReDoS into arithmetic.
+- **Decompression is a limit too.** If you accept `Content-Encoding: gzip`, cap the *decompressed* size, not the compressed one. A 1 MiB gzip bomb expands past 1 GiB.
+
+**Never write an email regex from memory.** RFC 5322's grammar does not fit in a pattern you can review, and every hand-rolled version rejects real addresses. Check for a single `@`, a non-empty local part, a domain containing a dot, cap the length at 320 bytes, then prove the address by sending mail to it. Deliverability is the only real validation.
+
+## Normalise Before You Validate — and Know Where That Is the Bug
+
+Order is fixed, because each step depends on the last:
+
+```
+1. size check on the RAW bytes        # NFKC can expand input; check before you grow it
+2. decode UTF-8, reject invalid       # not U+FFFD replacement — reject, an invalid encoding
+                                      # is the first half of every filter-bypass
+3. Unicode normalise (NFC for text, NFKC for identifiers)
+4. trim leading/trailing whitespace, collapse internal runs where the field is a name
+5. validate against the schema
+6. store the ORIGINAL form; index a derived normalised column for comparison
+```
+
+**Here is where normalisation itself becomes the vulnerability.** Case-folding and compatibility normalisation are many-to-one: they map distinct inputs onto one value. If that value is an *identity*, you have merged two accounts. Spotify's 2013 postmortem ("Creative usernames and Spotify account hijacking") is the canonical writeup: a username canonicalised at one layer and not another let an attacker recover someone else's account. The Turkish dotless `ı`, `İ` (U+0130), the Kelvin sign `K` folding to `k`, and `fi` (U+FB01) folding to `fi` are all live cases in any username field.
+
+The rule that follows:
+
+- **Normalisation for comparison is not normalisation for storage.** Store what the user typed. Compute a canonical form into a separate column, and put the `UNIQUE` constraint on *that* column (the hub's rule: constraints belong in the database).
+- **A canonical collision is a rejection, not a merge.** If the normalised form already exists, return `409` at signup. Silently attaching the login to the existing account is the hijack.
+- **Never case-fold a password, a token, a signature, or a code.** Fold identifiers; leave secrets byte-exact. Compare secrets with a constant-time function ([auth-and-secrets.md](auth-and-secrets.md)).
+- **Never trim a password.** Trimming changes the credential and quietly breaks a user whose manager stored a trailing space.
+
+## The Error Taxonomy
+
+Every failure in the service resolves to exactly one of these classes before it reaches the response layer. The class determines all four downstream columns; the handler decides none of them.
+
+| Class | Status | Log | Retryable | Detail safe to return |
+|---|---|---|---|---|
+| **Malformed input** — unparseable body, bad content type, bad JSON | `400` (`415` on media type) | `info` | No | Yes — the parse error position, never the parsed value |
+| **Semantic validation** — well-formed, wrong: unknown enum, negative amount, failing `CHECK` (Postgres `23514`) | `422` | `info` | No | Yes — field path + rule, **never the value** (may be a secret) |
+| **Payload too large** — over any limit above | `413` | `info` | No | Yes — the limit and the unit |
+| **Unauthenticated** — no/expired/invalid credential | `401` | `info` | No (until a new credential) | Generic only. Never "no such user" vs "wrong password" |
+| **Unauthorised** — authenticated, not permitted | `403`, or `404` where existence itself is private | `warn` | No | Generic. Name the required permission only if the caller could already know it exists |
+| **Not found** — absent, or present and not yours | `404` | `info` | No | Yes — resource type and the id *they sent* |
+| **Conflict** — unique violation (`23505`), state machine says no | `409` | `info` | No — the same request conflicts again | Yes — which constraint in business terms ("email already registered"), never the index name |
+| **Precondition failed** — `If-Match`/version mismatch | `412` | `info` | No — client must re-read first | Yes — current version/ETag |
+| **Serialization failure / deadlock** — Postgres `40001`, `40P01` | `409` (or `503` if you retry server-side) | `warn` | **Yes**, immediately, with jitter | Generic — "concurrent update, retry" |
+| **Rate limited** | `429` | `info` | **Yes**, after `Retry-After` | Yes — limit, window, reset |
+| **Dependency failure** — upstream `5xx`, connection refused, pool exhausted (`53300`) | `502` | `error` | **Yes**, capped | Generic — **never the upstream's own error text** |
+| **Timeout** — upstream slow, statement cancelled (`57014`) | `504` | `error` | **Yes if the operation is idempotent**, otherwise No | Generic — "upstream timed out" |
+| **Shedding / draining** — overload, shutdown in progress | `503` + `Retry-After` | `warn` | **Yes** | Generic |
+| **Bug** — anything unhandled | `500` | `error` + trace id | **No.** Never advertise retryable on a failure you do not understand | Trace id and nothing else |
+
+Two notes that carry most of the value:
+
+- **The most common error-handling bug is not the missing `try`/`catch` — it is the `catch` that turns a `409` into a `500`.** A broad handler that wraps a typed domain error into a generic one destroys the class, and with it the status, the log level, and the client's ability to retry. Catch *narrowly*, or re-raise anything that already carries a class.
+- **Postgres SQLSTATEs are your classifier, not the message.** Branch on `23505`, `23503`, `23514`, `40001`, `40P01`, `57014`, `53300` — never on `err.message.includes("duplicate")`, which is locale- and version-dependent. This is the hub's "classify on a code, not a message" made concrete.
+
+## Retryability Is Signalled, Never Inferred
+
+A client that guesses retryability from the status code is guessing. `500` is sometimes transient and sometimes a permanent bug; `409` is permanent for a unique violation and transient for a serialization failure. So say it, in the body and in the headers, using the RFC 9457 envelope the hub mandates plus two extension members:
+
+```json
+{
+  "type": "https://api.example.com/problems/serialization-failure",
+  "title": "Concurrent update",
+  "status": 409,
+  "detail": "Another request modified this order. Re-read it and reapply your change.",
+  "instance": "/orders/9f2c…/confirm",
+  "code": "ORDER_CONCURRENT_UPDATE",
+  "retryable": true,
+  "retry_after_ms": 250
+}
+```
+
+```
+Retry-After: 1        # RFC 9110 §10.2.3: delay-seconds or an HTTP-date. Send it on
+                      # every 429 and every 503. Derive the number from the limiter's
+                      # window or the drain deadline — never pick one because it looks calm.
+```
+
+- `code` is a **stable enum defined in one file**, screaming snake case, never constructed at the call site. **Never invent a code inline.** A typo'd code is a client branch that will never fire again, and you will not find out.
+- `retryable` is a boolean the *server* computes from the class table. It is not "did this look transient".
+- Retries are only safe on idempotent operations. If the endpoint is not idempotent, `retryable` is `false` regardless of class until an idempotency key makes it safe ([concurrency-and-jobs.md](concurrency-and-jobs.md)).
+- **Never invent a status code that "feels right".** `499` is nginx's internal client-closed-request and is not a real HTTP status; `509` is a cPanel extension; `418` is a joke from RFC 2324. If it is not in the hub's table or [api-contracts.md](api-contracts.md), it does not go on the wire.
+
+## What Never Appears in a Response Body
+
+Stack traces · SQL text or query plans · internal primary keys or row ids · upstream error text verbatim · file paths · library or framework versions · hostnames and internal service names · environment variable names · index and constraint identifiers · the value that failed validation.
+
+Every one of those is reconnaissance. The upstream text is the sharpest: it frequently carries *its* internal ids, *its* stack, and occasionally a credential fragment, and forwarding it makes your service a proxy for someone else's leak.
+
+Raw, and what it hands an attacker — schema names, ORM, file layout, and the fact that this table has a `users_email_key`:
+
+```json
+{ "error": "IntegrityError: duplicate key value violates unique constraint \"users_email_key\"\n  DETAIL: Key (lower(email))=(a@b.com) already exists.\n  at /srv/app/services/users.py:214 in create_user\n  SQLAlchemy 2.0.30 / psycopg 3.1.18" }
+```
+
+Sanitised — same information for the caller, none for the attacker:
+
+```json
+{
+  "type": "https://api.example.com/problems/email-taken",
+  "title": "Email already registered",
+  "status": 409,
+  "detail": "An account already exists for this email address. Sign in, or reset the password.",
+  "code": "USER_EMAIL_TAKEN",
+  "retryable": false,
+  "instance": "/users",
+  "trace_id": "01J8ZP4K9M7QX2"
+}
+```
+
+The raw version goes to the log, keyed by that same `trace_id`, with secrets redacted before writing ([observability.md](observability.md)). **The default response for an unclassified exception is `500` + `code` + `trace_id`, with the message discarded** — a framework that echoes `str(exc)` by default is a leak waiting for the first unusual exception, and `DEBUG=true` in production is that leak on purpose.
+
+Log hygiene has one trap of its own: do not log the raw body of a *validation* failure. The field that failed is safe; the body containing `password`, `card_number`, or a token is not.
+
+## Partial Failure
+
+A bulk endpoint has to decide this before it is written, because it is a contract, not an implementation detail.
+
+| Situation | Answer |
+|---|---|
+| Items share an invariant (a transfer's debit and credit; a batch that must not half-apply) | **All-or-nothing.** One transaction, one status. First failure aborts. |
+| Items are independent and the client can act per-item (import 500 contacts, send 100 notifications) | **Per-item results**, `207 Multi-Status` (RFC 4918 §13) or `200` with a `results` array |
+| Items are independent but the caller is a dumb client that only reads the status line | **All-or-nothing.** A partial success it cannot inspect is a silent data-loss bug |
+| The batch is large or slow (> ~10s of work) | Neither — accept it, return `202` with a job id, report per-item outcomes on the job resource |
+
+When you return per-item results, three non-negotiables: every item carries the **client's own id** (echo the request-supplied key; never a positional index, which breaks the instant a client reorders), every failed item carries the **full problem object** with its `code` and `retryable`, and the endpoint is **idempotent per item** so the client can resubmit only the failures. A partial-success response the client cannot safely re-drive is worse than a hard failure.
+
+**Never return `200` with a body that quietly reports failures** and no per-item structure. That is the hub's "never `200` with an error body" in its most dangerous form: monitoring sees success, and the missing rows surface a month later.
+
+## Messages That Tell the Consumer What To Do
+
+`dasar-vibecode` requires an error to say what failed and why. For an API, extend it by one clause, because **your consumer cannot read your logs**: what they should do next.
+
+| Bad | Good |
+|---|---|
+| `"Invalid request"` | `"currency must be one of IDR, USD; received JPY"` |
+| `"Validation failed"` | `"items must contain at most 100 entries; received 512. Split the request."` |
+| `"Unauthorized"` | `"Access token expired. Refresh at POST /auth/refresh and retry."` |
+| `"Conflict"` | `"Order is already confirmed and cannot be confirmed again."` |
+| `"Internal server error"` | `"Unexpected error. Retry is not safe; quote trace 01J8ZP4K9M7QX2 to support."` |
+
+Field errors ride in an `errors` array with a **JSON Pointer** path (RFC 6901) so a client can attach a message to an input without string-matching:
+
+```json
+"errors": [
+  { "pointer": "/items/3/quantity", "code": "OUT_OF_RANGE", "detail": "must be between 1 and 99" }
+]
+```
+
+Echo the field path and the rule. **Never echo the offending value** — validation failures are exactly where secrets get typed into the wrong box.
+
+**Permission to add nothing:** most endpoints need no new error code. Reach for an existing one first; a taxonomy with forty codes is a taxonomy no client branches on. Adding `ORDER_ITEM_QUANTITY_TOO_HIGH` beside a working `OUT_OF_RANGE` is a defect, not thoroughness.
+
+## Never Ship
+
+| Never | Instead |
+|---|---|
+| A hand-written type beside a schema | `z.infer` / the model class as the only type |
+| Reading `req.body` after parsing | Pass the parsed value; the raw request stops at the edge |
+| A permissive parser (`extra="allow"`, no `.strict()`) | Reject unknown fields; allow one only with a written reason |
+| `Object.assign(row, input)` / `Model(**data)` | Field-by-field assignment from the parsed value |
+| One model for request and response | Separate types; response models omit internal columns |
+| An endpoint with no body-size or array-length cap | The numbers in the limits block |
+| A regex over an uncapped string | Cap the length first |
+| Case-folding two identities into one | Canonical column + `UNIQUE` + `409` on collision |
+| Branching on `err.message` | SQLSTATE / typed error class / `code` |
+| A broad `catch` that re-raises as `500` | Catch narrowly; re-raise anything already classified |
+| `retryable` inferred by the client | Server-computed flag + `Retry-After` |
+| A status code not in the hub's table (`499`, `509`, `418`) | The class table above |
+| Upstream error text, SQL, stack, or path in a body | `code` + `trace_id`; the detail goes to the log |
+| `200` with hidden per-item failures | `207` / per-item results, or all-or-nothing |
+| A new error code for a case an existing one covers | The existing code |
+
+## Output
+
+Write the schema and the error mapping. Then, briefly: the **parse boundary** (which schema, and that unknown fields are rejected), the **limits** you applied and where they are enforced, the **classes** this endpoint can produce with their status codes, and one line on **what a client sees on the worst failure** — the exact sanitised body. If you removed a redundant validator or declined to add an error code, say so in one line. The code is the deliverable; do not expand this into a report.