vibecodeagent 0.0.0-stage → 0.1.64

package/dist/resources/skills/builtin/backend-engineering/references/concurrency-and-jobs.md

@@ -0,0 +1,386 @@
+# Concurrency and Deferred Work
+
+This file owns the mechanics of two things happening at once and of work that happens later: idempotency storage, locks, transaction boundaries, isolation levels, the outbox, queues, retries, and scheduled jobs. The idempotency **contract** — the `Idempotency-Key` header, which endpoints require it, the status code the client sees — belongs to [api-contracts.md](api-contracts.md); this file is what happens behind it. Queue-depth alerts and consumer-lag SLOs are [observability.md](observability.md). Process lifecycle for the HTTP server, pool sizes, and timeout config are [deploy-and-config.md](deploy-and-config.md). Locking behaviour of DDL against a live table is [migrations.md](migrations.md).
+
+## Operating Posture
+
+Every write is a concurrency question. Answer it before you write the handler, not after the incident.
+
+Two failure modes, and the first is worse:
+
+1. **The duplicate nobody sees.** The job ran twice, the card was charged twice, the email went out twice. Nothing in the logs is red. You learn about it from a customer, days later, and the money already moved.
+2. **The stall.** A connection held across a third-party timeout drains the pool, and endpoints that never touched your code start returning 500. Loud, ugly, and fixed by a restart.
+
+**Never invent a retry policy, a backoff schedule, a visibility timeout, or a TTL.** Take them from the tables below or from the project's own config. Never write `time.sleep(2 ** attempt)` with no cap and no jitter because it looks familiar — that line *is* the retry storm, written out.
+
+### Gate: does this need to be asynchronous at all?
+
+| Situation | Answer |
+| --- | --- |
+| Work is <100ms and the caller needs the result | Inline. Write no job. |
+| Work is slow but the caller needs the result | Inline, with a timeout; if that's unacceptable, the endpoint returns `202` + a status resource ([api-contracts.md](api-contracts.md)) |
+| Work must survive the request dying (charge, email, webhook fan-out) | Queue |
+| Work is triggered by time, not by a request | Scheduler that **enqueues** — see *Cron is not a queue* |
+| "It feels cleaner as a job" | Inline. Say so and move on. |
+
+A queue you added for tidiness costs a consumer process, a DLQ, an alert, a second idempotency story, and a new class of bug where the row is committed but the job hasn't run yet. Producing zero background jobs is a valid outcome.
+
+## 1. Idempotency Keys — the mechanics
+
+### Storage shape
+
+One row per key, written before the work starts.
+
+```sql
+CREATE TABLE idempotency_keys (
+  tenant_id     uuid        NOT NULL,          -- scope: see the leak below
+  key           text        NOT NULL,          -- client-supplied, opaque
+  fingerprint   bytea       NOT NULL,          -- sha256(method | path | raw body)
+  state         text        NOT NULL,          -- 'in_progress' | 'completed'
+  status_code   int,                           -- replayed verbatim
+  response_body jsonb,                         -- replayed verbatim
+  created_at    timestamptz NOT NULL DEFAULT now(),
+  expires_at    timestamptz NOT NULL,          -- created_at + 24 hours
+  PRIMARY KEY (tenant_id, key)
+);
+CREATE INDEX ON idempotency_keys (expires_at);
+```
+
+**The key is scoped to the tenant, never global.** A global key space means tenant B sends `Idempotency-Key: abc` and gets tenant A's stored response body back. That is not a race condition, it is an authorization leak with a concurrency costume on.
+
+**Fingerprint the raw bytes, before parsing.** Hash the body as received. If you hash the parsed struct after defaults are applied, the day you change a default the same retry produces a different fingerprint and every in-flight client retry starts failing with a mismatch — a bug that appears at deploy time and disappears when you roll back, which is the hardest kind to read.
+
+**TTL is 24 hours** (the same window Stripe keeps results for). Long enough that a client's retry queue drains, short enough that the table stays small. Delete expired rows on a schedule; do not rely on the index alone.
+
+### The algorithm
+
+The insert *is* the lock. There is no `SELECT` first — that is the classic check-then-act race.
+
+```sql
+INSERT INTO idempotency_keys (tenant_id, key, fingerprint, state, expires_at)
+VALUES ($1, $2, $3, 'in_progress', now() + interval '24 hours')
+ON CONFLICT (tenant_id, key) DO NOTHING
+RETURNING key;
+```
+
+1. **1 row returned** → you own this key. Execute the work. In the *same* transaction as the business write, `UPDATE` the row to `state='completed'` with `status_code` and `response_body`. Commit. Return the response.
+2. **0 rows returned** → someone got there first. `SELECT` the existing row and branch:
+
+| Existing row | Response |
+| --- | --- |
+| `state='completed'`, fingerprint matches | Replay `status_code` + `response_body` verbatim. Add `Idempotency-Replayed: true`. |
+| `state='in_progress'`, fingerprint matches | `409` with `Retry-After: 1`. The first request is still running; do not queue behind it, do not execute. |
+| Fingerprint differs, any state | `422`, code `idempotency_key_reuse`. The key was reused for a different request — that is a client bug, and executing it would silently corrupt their retry model. |
+
+The `409`-while-in-progress and `422`-on-mismatch mapping is the one in the IETF `Idempotency-Key` header draft; keep it identical to whatever [api-contracts.md](api-contracts.md) states.
+
+3. **On failure**, delete the `in_progress` row (or mark it failed) so a genuine retry can proceed. A crashed request that leaves `in_progress` behind forever turns a retryable failure into a permanent `409`; expiry is the backstop, and 24 hours of `409` is not a backstop anyone enjoys.
+
+**The stored response must be written in the same transaction as the effect it describes.** Two transactions means a window where the charge exists and the key row says `in_progress`, and the retry re-charges.
+
+## 2. Optimistic Locking — a version column
+
+For rows a human edits, where conflicts are rare and a retry is acceptable.
+
+```sql
+UPDATE documents
+   SET title = $1, body = $2, version = version + 1, updated_at = now()
+ WHERE id = $3 AND version = $4;
+```
+
+**Check the affected row count. Always.** `rowcount = 1` succeeded; `rowcount = 0` means someone else committed between the read and the write. Map that to `409` with a code the client can branch on (`version_conflict`), and include the current version in the problem detail so a UI can offer a merge.
+
+`rowcount = 0` is ambiguous — the row may have been deleted rather than updated. If the difference matters to the client, follow with `SELECT 1 FROM documents WHERE id = $3`: absent → `404`, present → `409`.
+
+The symptom of a missing version check is not an error. It is the last write silently winning: two editors save, the second overwrites the first, both see success, and the lost edit is discovered a week later with no log line anywhere.
+
+- `updated_at` is **not** a version. Two writes inside the same clock tick compare equal, and clock skew across app servers makes it worse.
+- A monotonic `integer` is enough. Don't reach for a hash of the row.
+- Use optimistic locking when conflicts are rare; use pessimistic locking when they are the normal case (counters, inventory, queue claims).
+
+## 3. Pessimistic Locking
+
+`SELECT ... FOR UPDATE` inside a transaction takes a row lock until commit. It is correct for read-modify-write where you cannot tolerate a retry loop.
+
+```sql
+BEGIN;
+SELECT balance FROM accounts WHERE id = $1 FOR UPDATE;   -- blocks other writers
+UPDATE accounts SET balance = balance - $2 WHERE id = $1;
+COMMIT;
+```
+
+| Clause | Use it when |
+| --- | --- |
+| `FOR UPDATE` | You will modify the row or its key columns |
+| `FOR NO KEY UPDATE` | You will only touch non-key columns — it does not block the `FOR KEY SHARE` lock a foreign-key insert takes, so child inserts keep flowing |
+| `FOR SHARE` | You need the row to stay unchanged for the rest of the transaction but won't write it |
+| `FOR UPDATE SKIP LOCKED` | Queue-like table: hand each worker a different row |
+| `FOR UPDATE NOWAIT` | Fail immediately rather than wait — raises `55P03`, map to `409` |
+
+### Queue-like tables: SKIP LOCKED
+
+Never `SELECT ... FOR UPDATE` the head of a jobs table from N workers — they serialize into a single-file line. `SKIP LOCKED` makes each worker step over rows already claimed:
+
+```sql
+UPDATE jobs SET state = 'running', started_at = now(), attempts = attempts + 1
+WHERE id IN (
+  SELECT id FROM jobs
+   WHERE state = 'queued' AND run_after <= now()
+   ORDER BY run_after
+   FOR UPDATE SKIP LOCKED
+   LIMIT 20
+)
+RETURNING *;
+```
+
+`LIMIT 20` is a batch, not a magic number: bigger batches amortize round-trips but extend the transaction and widen the redelivery window on a crash. Keep the whole claim under 50ms.
+
+### Deadlock, worked
+
+Two transfers run at once. T1 moves money from account 1 to 2; T2 moves from 2 to 1.
+
+```
+T1: BEGIN; SELECT ... WHERE id = 1 FOR UPDATE;   -- holds 1
+T2: BEGIN; SELECT ... WHERE id = 2 FOR UPDATE;   -- holds 2
+T1:        SELECT ... WHERE id = 2 FOR UPDATE;   -- waits on T2
+T2:        SELECT ... WHERE id = 1 FOR UPDATE;   -- waits on T1  → cycle
+```
+
+Postgres runs deadlock detection after `deadlock_timeout` (default `1s`), then kills one transaction with SQLSTATE `40P01`. So the symptom is not a hang: it is a 1-second latency spike plus a random failure under load, and it is invisible in staging where only one request runs at a time.
+
+**The fix is a consistent acquisition order, not a retry.** Sort the ids and lock ascending, always:
+
+```sql
+SELECT id, balance FROM accounts
+ WHERE id = ANY($1) ORDER BY id
+   FOR UPDATE;                -- $1 = [least(from,to), greatest(from,to)]
+```
+
+The same rule generalizes: if two code paths lock `orders` and `inventory`, every path locks `orders` first. Write the order down in a comment at both call sites.
+
+### Locking something that isn't a row
+
+Advisory locks. `pg_try_advisory_xact_lock(hashtext('import:tenant-42'))` returns `false` instead of waiting and releases at commit — correct for "only one importer per tenant". Session-scoped `pg_try_advisory_lock` must be released explicitly and leaks on a crashed connection until the backend dies.
+
+## 4. Isolation Levels
+
+| Level | Still permits (in Postgres) | Use for |
+| --- | --- | --- |
+| `READ COMMITTED` (default) | Non-repeatable reads, phantoms, **lost updates via read-modify-write in application code** | Almost everything |
+| `REPEATABLE READ` | Write skew (two transactions each read a consistent snapshot and write a combination neither would allow) | Multi-statement reports that must see one snapshot |
+| `SERIALIZABLE` | Nothing — SSI aborts one of the offenders | Invariants across rows: "at least one doctor on call", balance checks spanning accounts |
+
+Postgres permits no dirty reads at any level, so `READ UNCOMMITTED` is a synonym for `READ COMMITTED` there. `REPEATABLE READ` in Postgres is snapshot isolation and already prevents phantom reads — stricter than the SQL standard requires, and still not enough for write skew.
+
+**The trap that bites everyone:** `READ COMMITTED` does not prevent a lost update when the read-modify-write happens in your process.
+
+```python
+row = db.query("SELECT balance FROM accounts WHERE id = 1")   # 100
+new = row.balance - 10                                        # 90 in Python
+db.exec("UPDATE accounts SET balance = %s WHERE id = 1", new) # writes 90
+```
+
+Two concurrent runs both read 100 and both write 90. One withdrawal vanished. There is no error, and the row count is 1 both times. Three real fixes: do the arithmetic in SQL (`SET balance = balance - 10`, which Postgres re-evaluates against the updated row at `READ COMMITTED`), take `FOR UPDATE` on the read, or use a version column. Raising the isolation level to `REPEATABLE READ` turns the silent loss into a `40001` error — which is progress only if you retry it.
+
+### Serialization failures must be retried
+
+`40001` (serialization_failure) and `40P01` (deadlock_detected) are not bugs to log; they are the database telling you to run it again.
+
+```
+Retry only on SQLSTATE 40001 and 40P01. Max 3 attempts.
+Backoff: 25ms, 50ms, 100ms, each × random(0.5, 1.5).
+```
+
+**The retry lives outside the transaction, wrapping it.** A retry inside the transaction is impossible — the transaction is already aborted, and every statement after the failure returns `25P02`. Which means the retried function must be safely re-runnable end to end: it may contain database work and pure computation, and nothing else. If the function sends an email, the retry sends a second email. See the next section.
+
+## 5. Never Do Network I/O Inside a Transaction
+
+Not an HTTP call, not a queue publish, not an S3 upload, not a cache write, not a remote log flush, not a `sleep`. This is its own section because it is the single most expensive concurrency mistake, and it never shows up in code review as a concurrency mistake — it looks like tidy code that does everything in one place.
+
+The failure, in order:
+
+1. `BEGIN` checks a connection out of the pool. The pool has 10.
+2. Inside the transaction you call a payment provider. Node's `fetch` and Python's `requests` have **no default timeout** — the call can hang until the OS gives up, minutes later.
+3. That connection is pinned for the whole hang. Ten concurrent requests to this endpoint and the pool is empty.
+4. Every *other* endpoint now waits on connection acquisition and fails. Your dashboard shows `/users` and `/health` timing out while the actual dependency that broke was a payment provider nothing on that path touches.
+
+Plus two slower harms that survive the incident: row locks held for the duration turn a 40ms endpoint into a lock queue, and a long-open transaction pins the `xmin` horizon so `VACUUM` cannot reclaim *any* dead row newer than it — the table bloats for as long as the call hangs.
+
+**Rules:**
+
+- A transaction contains database statements and pure computation. Nothing else.
+- Do the network call **before** the transaction (and make the call itself idempotent with a key), or **after** it via the outbox below.
+- Need to react to the provider's answer? Two transactions and a state machine: `pending` → call → `confirmed`/`failed`. The intermediate state is visible and recoverable; a held connection is neither.
+- Set the guardrails at the connection level so a mistake is loud rather than silent:
+
+```sql
+SET statement_timeout = '5s';                     -- default 0 = unlimited
+SET idle_in_transaction_session_timeout = '10s';  -- default 0 = forever
+```
+
+`idle_in_transaction_session_timeout` is the one that catches this class directly: a transaction sitting idle while your code awaits a socket gets killed instead of holding the connection. Budget transaction wall time under 100ms and treat anything over 1s as a defect to explain.
+
+## 6. The Outbox Pattern
+
+The answer to "write to the database and publish an event". You cannot do both atomically across two systems, so stop trying.
+
+The dual-write failures, both real:
+
+- **Commit, then publish**, and the process dies in between → the row exists and the event never happens. Nothing is logged. The downstream service is permanently out of sync and nobody finds out until a reconciliation nobody wrote.
+- **Publish, then commit**, and the commit rolls back → consumers receive an event about a state that does not exist. They read the database, find nothing, and the bug reads as a race in the *consumer*, which is where the next three days of debugging go.
+
+The outbox makes the event part of the transaction:
+
+```sql
+CREATE TABLE outbox (
+  id             bigserial PRIMARY KEY,   -- monotonic: publish order
+  aggregate_type text NOT NULL,
+  aggregate_id   text NOT NULL,           -- partition key at the broker
+  event_type     text NOT NULL,
+  payload        jsonb NOT NULL,
+  created_at     timestamptz NOT NULL DEFAULT now(),
+  published_at   timestamptz
+);
+CREATE INDEX ON outbox (id) WHERE published_at IS NULL;   -- partial: stays tiny
+```
+
+1. In the business transaction: `INSERT INTO orders ...` and `INSERT INTO outbox ...`. One commit, both or neither.
+2. A relay polls every **500ms**, claims a batch with `SKIP LOCKED`, publishes, marks published:
+
+```sql
+SELECT * FROM outbox WHERE published_at IS NULL
+ ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED;
+-- publish each (outside the claim transaction if the broker is slow)
+UPDATE outbox SET published_at = now() WHERE id = ANY($1);
+```
+
+3. Crash after publishing and before the `UPDATE` → the event is published twice. **That is the design, not a leak.** The outbox is at-least-once; the consumer is idempotent (section 7).
+4. Delete or partition-drop rows older than 7 days. An outbox nobody prunes becomes the largest table in the database.
+
+Per-aggregate ordering holds only if one relay publishes a given `aggregate_id` at a time and the broker partitions on it — a parallel relay sharded by `hash(aggregate_id)` preserves this; a relay sharded by `id % N` does not. If polling latency is unacceptable, replace the relay with logical decoding (Debezium reading the WAL) — same table, same guarantees, no poll interval.
+
+## 7. Queues, Retries, and Dead Letters
+
+**At-least-once is the default, therefore every consumer must be idempotent.** Exactly-once *delivery* across a network does not exist; what vendors sell under that name is at-least-once plus deduplication with a window (SQS FIFO dedupes over 5 minutes) or transactional scope (Kafka EOS holds only inside Kafka). Neither covers your side effect. Assume every message arrives twice and make that boring.
+
+| Operation | How to make it idempotent |
+| --- | --- |
+| Insert a row | `INSERT ... ON CONFLICT DO NOTHING` on a natural or message-derived key |
+| Update a field | Set an absolute value, never `x = x + 1` |
+| Increment a counter | Insert a ledger row keyed by `message_id`, then aggregate |
+| Charge a card | Pass the message id as the provider's idempotency key |
+| Send an email | Unique constraint on `(user_id, template, dedupe_window)` before sending |
+| Call a non-idempotent third party | A `processed_messages(message_id)` table with a unique constraint, written in the same transaction as the effect |
+
+**Send an id, not the entity.** Payloads go stale between publish and consume, and brokers cap size (SQS 256 KiB, Kafka's `message.max.bytes` ~1 MiB by default). The consumer re-reads current state — from the **primary**, not a read replica: a consumer fast enough to beat replication finds no row and fails with "not found", which reads as a queue bug and is actually replica lag.
+
+### Backoff with jitter
+
+```
+delay = random_between(0, min(cap, base * 2 ** attempt))
+base = 1s      cap = 300s      max_attempts = 5
+```
+
+| Attempt | Delay drawn from |
+| --- | --- |
+| 1 | 0–1s |
+| 2 | 0–2s |
+| 3 | 0–4s |
+| 4 | 0–8s |
+| 5 | 0–16s → then DLQ |
+
+**Jitter is the part people delete because it looks unnecessary.** Without it, every consumer that failed during a downstream outage retries at the identical instant — a synchronized wall of traffic that re-kills the dependency in the same second it recovers, then re-synchronizes on the next failure. The dependency looks like it is flapping; it is being flapped. AWS's own backoff-and-jitter measurements put full jitter ahead of both plain exponential and "exponential plus a little jitter" on total work and completion time. Use the formula above; do not soften it into `delay * random(0.9, 1.1)`.
+
+### Non-retryable errors go straight to the DLQ
+
+Retrying a validation error five times produces five guaranteed failures and delays the DLQ alert by four minutes.
+
+| Error class | Retry? |
+| --- | --- |
+| Network timeout, `5xx`, `429`, connection reset | Yes, with backoff |
+| Serialization failure / deadlock (`40001`, `40P01`) | Yes, immediately (section 4) |
+| Malformed payload, schema mismatch, unknown event type | No — DLQ on attempt 1 |
+| `4xx` from a downstream that is not `429` | No — DLQ on attempt 1 |
+| Referenced entity permanently deleted | No — ack and drop, with a log line |
+
+### DLQ and poison messages
+
+A dead-letter queue is a queue with an owner, not a graveyard. It needs: an alert on `depth > 0 sustained 15 minutes` ([observability.md](observability.md)), enough context stored to diagnose (original payload, last error, attempt count), and a documented replay path. **Never auto-replay a DLQ into its source queue** — a poison message loops forever and the loop is invisible because each individual delivery looks like a first attempt.
+
+**Visibility timeout must exceed p99 handler time — use 3×.** If the handler takes longer than the timeout, the broker redelivers while the first worker is still running: two workers on the same row, duplicate side effects, and a duplicate rate that rises with load, which is exactly when nobody has time to read it. Handlers that can legitimately exceed the timeout must heartbeat/extend, not raise the timeout for everyone.
+
+## 8. Cron Is Not a Queue
+
+Cron has no retries, no visibility, no backpressure, no dead-letter path, and its failure output is an email to an address nobody configured. **A scheduled job's only job is to enqueue.** Put the actual work behind the queue, where retries, DLQ, and metrics already exist.
+
+**Two instances, one schedule.** This is the default state of the world, not an edge case: Kubernetes documents `CronJob` as at-least-once ("may create two Jobs, or none"), `concurrencyPolicy: Forbid` only stops overlap the controller knows about, and any pair of app instances running an in-process scheduler will both fire. So every scheduled job takes a lock before doing anything:
+
+```sql
+-- transaction-scoped, auto-released at COMMIT/ROLLBACK; returns false instead of waiting
+SELECT pg_try_advisory_xact_lock(hashtext('job:nightly-invoices'));
+```
+
+For a job too long to sit inside one transaction (which is most of them — see section 5), use a lease instead:
+
+```sql
+UPDATE job_leases
+   SET owner = $1, expires_at = now() + interval '5 minutes'
+ WHERE name = 'nightly-invoices'
+   AND (owner IS NULL OR expires_at < now())
+RETURNING name;                  -- 0 rows = someone else holds it, exit 0 quietly
+```
+
+Heartbeat the lease every 60s while working, and make the expiry longer than the heartbeat interval by at least 3×. A lease that expires mid-run hands a second worker the same job — the exact thing the lock exists to prevent.
+
+Three more decisions to make explicitly rather than by default:
+
+- **Schedule in UTC.** A job at 02:30 local time runs twice on the fall-back day and zero times on the spring-forward day. Billing that runs twice is an incident.
+- **Missed runs.** If the box was down at 03:00, does the 03:20 recovery run it? The silent default is "no". For invoicing and expiry sweeps that is wrong; decide, and encode the decision as a query over what still needs doing rather than a wall-clock trigger.
+- **Overlap.** If run N is still going when N+1 fires, the lock skips N+1 for free — emit a counter for it. A skip counter that climbs is the earliest warning that the job no longer fits its interval.
+
+## 9. Graceful Consumer Shutdown
+
+Ordered, and the order is the whole point. On `SIGTERM`:
+
+1. **Stop accepting.** Break the poll loop / pause the subscription. Do not fetch another message.
+2. **Finish in-flight**, with a deadline.
+3. **Ack** what completed. Nack or simply stop extending visibility on what didn't, so it redelivers cleanly.
+4. **Flush** metrics and traces, then close the DB pool.
+5. `exit 0`.
+
+```
+drain_deadline = 25s        # strictly under Kubernetes' 30s default
+                            # terminationGracePeriodSeconds
+```
+
+Exceed the grace period and the runtime sends `SIGKILL` mid-handler: the message redelivers and the work runs twice, so a sloppy shutdown converts every deploy into a duplicate-work event. If handlers can genuinely run longer than the grace period, either raise `terminationGracePeriodSeconds` deliberately and keep the drain deadline under it, or chunk the handler so it can checkpoint and resume.
+
+**Never ack before doing the work.** Ack after the transaction commits. Ack-first plus a crash means the work is lost, the queue is empty, and there is no artifact anywhere saying it should have happened — the one queue failure mode with no evidence at all. Related: release locks and leases in a path that runs on the failure branch only if that is what you meant; a `finally` that nacks on success is a duplicate generator.
+
+HTTP-server drain, readiness-probe ordering, and the deploy sequencing around it are [deploy-and-config.md](deploy-and-config.md).
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| `SELECT` then `INSERT` to enforce uniqueness | Unique constraint + `ON CONFLICT` |
+| An `UPDATE` whose affected-row count you don't check | Check it; 0 rows is a `409` or a `404` |
+| Read-modify-write in application code at `READ COMMITTED` | Arithmetic in SQL, `FOR UPDATE`, or a version column |
+| Retrying `40001` inside the aborted transaction | Retry the whole transaction from outside, max 3 |
+| Any network call between `BEGIN` and `COMMIT` | Before the transaction, or after it via the outbox |
+| Commit-then-publish (or publish-then-commit) | Outbox table in the same transaction |
+| A consumer that assumes one delivery | Idempotent by construction; at-least-once is the contract |
+| Retries without jitter, or without a cap | `random(0, min(300s, 1s * 2**n))`, 5 attempts |
+| Retrying a `4xx` or a parse error | Straight to DLQ on attempt 1 |
+| A DLQ with no alert and no replay path | Alert on depth > 0 for 15 min; document the replay |
+| Auto-replaying a DLQ into its source queue | Manual replay after a fix |
+| Visibility timeout ≈ p99 handler time | 3× p99, or heartbeat/extend |
+| Two code paths locking the same tables in different orders | One documented order, ascending by id |
+| A scheduled job with no lock | Advisory lock or a heartbeated lease |
+| Cron doing the work | Cron enqueues; the consumer works |
+| Ack before commit | Ack after commit |
+| Idempotency keys in a global namespace | Scoped to the tenant |
+
+## Output
+
+Write the code. Then, in a few lines: the concurrency answer for this change ("two of these at once → the unique constraint rejects the second, client gets 409"), the exact retry/backoff/TTL values you used and where they came from, and what you verified — the two-concurrent-request check, not the single-request one. If you declined to add a queue, a lock, or an idempotency key, say which and why in one line; "this stays synchronous" is a complete answer. Deeper conflict scenarios and load-time failure hunting are [review-audit.md](review-audit.md).