@zigrivers/scaffold 3.16.0 → 3.18.0

package/content/knowledge/backend/backend-fintech-order-lifecycle.md

@@ -0,0 +1,213 @@
+---
+name: backend-fintech-order-lifecycle
+description: Order state machine; fills, partial fills, cancellation; event-driven order tracking; idempotency; handling "unknown" states.
+topics: [backend, fintech, orders, state-machine, fills, partial-fills, event-driven, webhooks]
+---
+
+Orders in a trading system are long-lived, asynchronous, externally mutated objects — the exact shape of problem a disciplined state machine is built for. This doc covers the canonical states and transitions, how fills and partial fills land, why "unknown" is a real state you cannot wish away, and the reconciliation posture that keeps internal bookkeeping aligned with the broker of record.
+
+## Summary
+
+An order is a state machine, not a row that gets mutated ad-hoc. The canonical states are `new → submitted → partially-filled → filled | cancelled | rejected | expired`, with two terminal-branching transitions that can fire from any state: `error` (integration failure, policy violation, unhandled exception) and `cancelled` (when a cancel request arrives before terminal). Only terminal states (`filled`, `cancelled`, `rejected`, `expired`, `error`) are final; everything else is transitional and must have a forward path. Illegal transitions (say, `filled → submitted`) must be rejected at the persistence layer, not just filtered out by a `switch` statement in application code.
+
+Fills arrive asynchronously. The broker does not call your submit endpoint and return a filled order — submission returns an acknowledgement (or doesn't, see "unknown" below) and fills trickle in over seconds, minutes, or days via webhook, streaming API, or polling. A single order produces one or many fill events; partial fills are the norm for any non-trivial size. Each fill event carries its own broker-assigned execution id, a timestamp, a price, a quantity, and usually a running cumulative fill quantity for the order.
+
+Idempotency is not optional at any layer that accepts broker events. Webhooks are delivered at-least-once — sometimes many-times-once during broker incidents — and a duplicate fill event absolutely must not create a duplicate fill row or double-book an execution. Every fill-ingest path goes through a dedupe-on-insert against a unique `(broker, execution_id)` key with multi-month retention, and every downstream posting into the ledger (`backend-fintech-ledger.md`) carries its own idempotency key derived from the fill.
+
+"Unknown" is a first-class state, not an edge case. When the submit call to the broker times out, you do not know whether the order reached the exchange. Assuming failure invites duplicate submissions; assuming success invites phantom positions. The correct posture: mark the order `submit-unknown`, do not retry blind, and reconcile by querying the broker's order list with your client-order-id as the key. Only once reconciliation confirms presence or absence do you transition forward.
+
+Every state transition produces a row in an immutable `order_events` audit table and — when money or position moves — a journal entry in the ledger. The order table itself records the current state as a projection of those events. Cross-ref: `backend-fintech-ledger.md` for the journal mechanics, `backend-fintech-broker-integration.md` for the wire-level quirks of specific broker APIs, and `backend-fintech-risk-management.md` for pre-trade checks that run before `new → submitted`.
+
+## Deep Guidance
+
+### State Diagram and Transition Matrix
+
+The diagram below is the minimum viable state machine for equities and futures orders. Add states (`held-for-review`, `pending-cancel`, `pending-replace`) only when the broker's protocol actually distinguishes them.
+
+```
+                     ┌────── reject ──────┐
+                     │                    ▼
+   new ── submit ──► submitted ────► rejected (terminal)
+                     │    │
+                     │    ├── fill (partial) ──► partially-filled
+                     │    │                       │    │
+                     │    │                       │    ├── fill (partial) ──► partially-filled
+                     │    │                       │    ├── fill (final)  ──► filled (terminal)
+                     │    │                       │    ├── cancel        ──► cancelled (terminal)
+                     │    │                       │    └── expire (TIF)  ──► expired  (terminal)
+                     │    ├── fill (full)    ──► filled (terminal)
+                     │    ├── cancel-request ──► pending-cancel ──► cancelled (terminal)
+                     │    └── expire         ──► expired  (terminal)
+                     │
+                     └── submit-timeout ──► submit-unknown ──► (reconcile) ──► submitted | rejected
+
+   any non-terminal ── integration failure ──► error (terminal, requires manual review)
+```
+
+Transition preconditions are non-trivial. `submit` requires a pre-trade risk check pass. `cancel-request` is only legal from `submitted`, `partially-filled`, or `submit-unknown` and itself creates a transient `pending-cancel` until the broker confirms. A `fill` event for an order in `cancelled` or `expired` is possible — it's a race, not a bug — and must cause a controlled transition to `filled` or `partially-filled` with a flagged event for human review, never a silent drop.
+
+### Order Types and Lifecycle Variants
+
+Order type determines which transitions are possible and when `expired` fires.
+
+- **Market**: fills immediately at best available price; rarely lives long enough to cancel; no limit price. Usually `submitted → filled` within milliseconds, but partial fills still happen in illiquid names.
+- **Limit**: buy at-or-below / sell at-or-above a price. Can rest on the book for the full Time-In-Force window. Partial fills are common.
+- **Stop**: becomes a market order when the stop price is touched. Two-phase lifecycle: `submitted` (resting, no fills possible) → `triggered` (internal-only state) → market-order behavior. Store the trigger event explicitly.
+- **Stop-Limit**: triggers a limit order at the trigger; can sit unfilled if the limit is never reached after trigger.
+- **Trailing-Stop**: stop price moves with the market by an offset or percentage. The broker tracks the trailing stop; your system should record the initial parameters and the final trigger price when it fires.
+- **OCO (One-Cancels-Other)**: two orders linked; fill on one cancels the other. Model as a parent order-group with two child orders, both in the state machine, linked by `group_id` and a `cancels_on_fill` flag.
+- **Bracket**: parent entry order with two OCO children (profit-take and stop-loss). Children remain `new` until parent fills, then auto-submit. Failure to enable the children on parent fill is a common defect — exercise in integration tests.
+
+**Time-In-Force (TIF)** codes drive the `expired` transition: `DAY` (expires at session close), `GTC` (good-till-cancelled, broker-specific max lifetime — IBKR caps at 90 days), `IOC` (immediate-or-cancel, partial fill OK, remainder cancelled instantly), `FOK` (fill-or-kill, all-or-nothing, no partial), `GTD` (good-till-date), `OPG` (at-the-open), `CLS` (at-the-close). The broker enforces TIF; your system must record it and expect `expired` fills to arrive on the next session boundary, not at the millisecond TIF ended.
+
+### Partial-Fill Handling
+
+A partial fill is one execution report against an open order; cumulative fill quantity equals the sum of all prior execution quantities for that order. Two tracking strategies, both valid, only one allowed at a time:
+
+1. **Internal aggregation**: sum your own stored fill rows; ignore the broker's `cum_qty` field except as a cross-check.
+2. **Broker-reported cumulative**: trust the broker's `cum_qty` and `avg_px` in each execution report; don't sum locally.
+
+Mixing produces silent drift. Pick one per integration, document it, and reconcile the other as a monitor. The internal-aggregation path is more resilient to out-of-order webhook delivery; the broker-reported path is simpler but requires correct event ordering.
+
+**Average fill price** is volume-weighted, not arithmetic. For fills `(qty_i, price_i)`:
+
+```
+avg_fill_price = Σ(qty_i × price_i) / Σ(qty_i)
+```
+
+Compute in fixed-scale decimals, not float. Store the avg on the order row as a cached projection, recomputed on every fill insert within the same transaction.
+
+**When to mark an order "done"** is subtle. An IOC or FOK that partially fills transitions to `filled` for the fill quantity and `cancelled` for the remainder — some systems model this as two terminal transitions on the same order; others as one order that ends in `partially-filled` with a `cancel_reason: 'ioc_remainder'`. Pick a convention and enforce it. A DAY order that partially fills and then reaches session close transitions `partially-filled → expired` on the session boundary.
+
+### Cancellation Semantics
+
+A cancel is a *request*, not an *action*. The lifecycle is `cancel_requested → pending_cancel → (broker confirms) → cancelled` OR `(fill arrives first) → partially-filled or filled`. Brokers do not guarantee that a cancel will beat an incoming fill; the exchange decides. The cancel-replace pattern (modify price/qty of a working order) is especially fraught — most brokers do not guarantee atomicity, meaning the original can fill, the replacement can fill, or both can fill in sequence. Defensive design:
+
+- Never treat "cancel requested" as "cancelled" in UI state — use a visually distinct `pending-cancel` badge.
+- Always support a **cancel-confirmed fill race**: accept a fill event on a `pending-cancel` order, transition to `filled`/`partially-filled`, and flag the race in audit.
+- For cancel-replace, prefer a cancel-then-new sequence over a broker-side `replace` when the broker doesn't guarantee atomicity (Alpaca, Tradier); use native replace only where atomicity is documented (Interactive Brokers with `OrderModify`).
+
+### Webhook Delivery Guarantees and Dedupe
+
+Every major broker's webhook system is at-least-once: retries on 5xx, retries on timeout, retries on TCP reset. Many are also out-of-order under load. Idempotency is enforced by a dedupe table with the broker's execution id as the key:
+
+```sql
+CREATE TABLE broker_events (
+  broker         TEXT NOT NULL,          -- 'alpaca' | 'ibkr' | 'tradier' | ...
+  event_id       TEXT NOT NULL,          -- broker's execution_id or event_id
+  event_type     TEXT NOT NULL,          -- 'fill' | 'cancel_confirmed' | ...
+  order_id       UUID REFERENCES orders(id),
+  received_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
+  payload        JSONB NOT NULL,
+  PRIMARY KEY (broker, event_id)
+);
+-- Retention: 90 days minimum, 13 months for regulated flows (SEC 17a-4)
+```
+
+Retention must exceed the broker's maximum retry horizon plus a generous margin. Alpaca retries for ~24 hours; IBKR TWS replays on reconnect for the trading day; some crypto exchanges have no documented horizon. Ninety days is a defensible minimum; thirteen months aligns with SEC 17a-4 (`backend-fintech-compliance.md`). Purging too early lets a late duplicate slip in as "new."
+
+### Dedupe-on-Insert Fill Handler
+
+```typescript
+type FillEvent = {
+  broker: string;
+  executionId: string;              // broker-assigned, globally unique per broker
+  brokerOrderId: string;
+  clientOrderId: string;            // your id, echoed by the broker
+  side: 'buy' | 'sell';
+  quantity: Decimal;
+  price: Decimal;
+  executedAt: string;               // ISO 8601, broker's clock
+  cumQty: Decimal;
+  avgPx: Decimal;
+  receivedAt: string;               // your ingestion clock
+};
+
+async function ingestFill(evt: FillEvent): Promise<void> {
+  await db.tx(async (tx) => {
+    // 1. Dedupe — INSERT ... ON CONFLICT DO NOTHING on (broker, event_id)
+    const inserted = await tx.query(
+      `INSERT INTO broker_events (broker, event_id, event_type, order_id, payload)
+       VALUES ($1, $2, 'fill',
+               (SELECT id FROM orders WHERE client_order_id = $3), $4)
+       ON CONFLICT (broker, event_id) DO NOTHING
+       RETURNING event_id`,
+      [evt.broker, evt.executionId, evt.clientOrderId, evt]
+    );
+    if (inserted.rowCount === 0) return;          // duplicate — silent success
+
+    // 2. Insert fill row; transition order state; recompute avg price
+    await tx.query(
+      `INSERT INTO fills (order_id, broker_execution_id, qty_minor, price_minor,
+                          executed_at, received_at, broker_ts, storage_ts)
+         SELECT o.id, $1, $2, $3, $4, $5, $4, now()
+           FROM orders o WHERE o.client_order_id = $6`,
+      [evt.executionId, evt.quantity, evt.price,
+       evt.executedAt, evt.receivedAt, evt.clientOrderId]
+    );
+    await transitionOrder(tx, evt.clientOrderId, 'fill', evt);
+
+    // 3. Post ledger entry (idempotent on executionId)
+    await postFillToLedger(tx, evt);
+  });
+}
+```
+
+Dedupe happens inside the transaction — not in a prior `SELECT` — so two concurrent webhook deliveries from the broker's retry queue cannot both win the race.
+
+### Reconciliation Query
+
+On every process startup and on a scheduled cadence (every 5 minutes during market hours is typical), query the broker for all open orders and diff against the internal state machine.
+
+```typescript
+async function reconcileOpenOrders(broker: BrokerClient): Promise<Mismatch[]> {
+  const [external, internal] = await Promise.all([
+    broker.listOrders({ status: 'open' }),           // broker says these are live
+    db.query(
+      `SELECT client_order_id, state, broker_order_id
+         FROM orders
+         WHERE state IN ('submitted','partially-filled','pending-cancel','submit-unknown')`
+    ),
+  ]);
+
+  const extById = new Map(external.map(o => [o.clientOrderId, o]));
+  const intById = new Map(internal.rows.map(o => [o.client_order_id, o]));
+  const mismatches: Mismatch[] = [];
+
+  for (const [cid, ext] of extById) {
+    const int = intById.get(cid);
+    if (!int) mismatches.push({ kind: 'external-only', cid, ext });
+    else if (int.state === 'submit-unknown') mismatches.push({ kind: 'resolved-unknown', cid, ext });
+    else if (int.broker_order_id !== ext.id) mismatches.push({ kind: 'id-mismatch', cid });
+  }
+  for (const [cid, int] of intById) {
+    if (!extById.has(cid)) mismatches.push({ kind: 'internal-only-open', cid, int });
+  }
+  return mismatches;
+}
+```
+
+Mismatch workflows: `resolved-unknown` and `id-mismatch` auto-reconcile by pulling the broker's order detail and replaying events; `external-only` (broker has an order we never recorded) and `internal-only-open` (we think it's live, broker doesn't) go to a manual-review queue with SLA. Never auto-cancel an `external-only` order without operator approval — it may be the survivor of a different deploy.
+
+### Clock Drift and Timestamps
+
+Three distinct timestamps must be recorded on every fill and every state transition, because any one alone will lie:
+
+- **Broker timestamp** (`broker_ts`): when the broker says the event occurred. Authoritative for regulatory reporting and sequencing with exchange data. Subject to the broker's clock sync.
+- **Ingestion timestamp** (`received_at`): when your webhook endpoint received the payload. Bounds network + broker-queue latency. Critical for SLA alerting.
+- **Storage timestamp** (`storage_ts`): when the row was committed to your database. Authoritative for ordering within your system.
+
+Storing only one collapses three distinct failure modes (broker clock skew, webhook queue delay, your DB commit delay) into an un-debuggable blob. When reconstructing a timeline for a trade dispute, all three are demanded. NTP-sync every host, and alarm on drift > 500ms.
+
+### Common Pitfalls
+
+- **Losing fills on 5xx responses.** If your webhook endpoint returns 500 during a database outage, the broker retries — for a while. Alpaca gives up after ~24 hours; IBKR replays on TWS reconnect but not indefinitely. After the retry horizon, fills are permanently lost to the event stream and only reconciliation finds them. Never return 5xx on business-logic failures; ack with 200 and queue the event internally for retry.
+- **Double-counting when switching between webhook and polling.** Running both ingest paths simultaneously without a shared dedupe table duplicates every fill. The dedupe table (`broker_events` keyed on `(broker, event_id)`) is a hard dependency of either path and both.
+- **State stored as strings with no enum.** `order.status = "Filled"` vs `"filled"` vs `"FILLED"` — the bug reports write themselves. Use a database enum or a `CHECK (state IN (...))` constraint and a typed enum in code; make illegal transitions impossible at the persistence layer.
+- **Treating `submit-unknown` as `rejected`.** A developer sees a submit timeout, retries the order, and the broker happily accepts both — the customer is now long 200 shares instead of 100. Always reconcile, never retry blind.
+- **Ignoring out-of-order webhook delivery.** Two partial fills arrive, but delivery order is reversed; the naive handler marks `cum_qty` going backwards. Use the broker's own sequence number or `cum_qty` as the source of truth for ordering within an order's fill stream; reject inserts that would decrease `cum_qty` without flagging.
+- **Forgetting cancel-fill races.** The UI shows "cancelled" but a fill arrives 200ms later. Allow the transition, flag for review, do not silently drop. Dropping is fraud-adjacent — the customer has the position and you've hidden it.
+- **Not reconciling on reconnect.** A process restart while the broker's websocket was disconnected loses every event that arrived in the gap. Every reconnect triggers a full reconciliation pass for open orders, not just a resume.
+- **TIF mis-modeling on overnight sessions.** A DAY order in US equities expires at 4pm ET; a DAY order in futures expires at the session close of the specific contract (5pm CT for most CME). A GTC for IBKR auto-cancels at 90 days even if you meant "forever." Store the broker's exact TIF semantics, don't translate to your own enum.
+
+See also `backend-fintech-ledger.md` for the journal entries every fill emits, `backend-fintech-broker-integration.md` for per-broker protocol quirks, and `backend-fintech-risk-management.md` for the pre-trade checks gating the `new → submitted` transition.

package/content/knowledge/backend/backend-fintech-risk-management.md

@@ -0,0 +1,150 @@
+---
+name: backend-fintech-risk-management
+description: Position limits, drawdown caps, circuit breakers, kill switches; pre-trade and post-trade risk checks; operational risk controls.
+topics: [backend, fintech, risk, position-limits, drawdown, circuit-breakers, kill-switch, pre-trade-checks]
+---
+
+Risk management in a trading system is the set of controls that stops a bad day from becoming a catastrophic one. It lives in two places: *before* an order goes to the broker (pre-trade checks that block) and *after* fills land (post-trade monitoring that alerts, throttles, or halts). Neither half is optional, and both are exercised continuously, not just during incidents.
+
+## Summary
+
+There are two classes of controls and both are mandatory. Pre-trade checks run synchronously in the order-submission path and block an order before it ever reaches the broker wire — max order size, max position per symbol, max portfolio leverage, margin/buying-power available, restricted-symbol list (hard blocks for sanctions and compliance), and fat-finger price sanity (reject a limit order more than, say, 10% away from last trade). Post-trade checks run continuously on the position and P&L stream and throttle or halt new orders when thresholds are breached — realized vs unrealized P&L, rolling max drawdown, velocity of losses (dollars lost per minute), and position concentration (no more than X% of NAV in any single name).
+
+Pre-trade checks live on the hot path and must be fast (single-digit milliseconds); post-trade checks run out of band on a stream of fill events and ledger updates. Pre-trade is the *hard* layer — if a check fails, the order does not go. Post-trade is the *soft* layer that can escalate through severity tiers (warning → throttle → halt → kill) as conditions degrade. Both layers feed the same observability and audit surface (`backend-fintech-observability.md`).
+
+Every risk control is parameterized per account, per strategy, and per account segment. Retail accounts have different leverage limits than institutional; a market-making strategy has different velocity caps than a buy-and-hold rebalancer. Hardcoded limits are a defect — configuration lives in a versioned, auditable store (Postgres, Consul, LaunchDarkly) with blast-radius-aware deploys.
+
+A kill switch is a single boolean read on the order-submission path that, when set, halts all new orders instantly. It must be triggerable both manually (one click, one operator, logged and alerted) and automatically on hard threshold breach (e.g., drawdown exceeds 5% of NAV intraday). Deactivation is dual-control: two operators must sign off. The kill switch has a defined "safe state" — does activation merely halt new orders, or does it also flatten open positions? Different products pick different answers; the choice must be explicit, documented, and tested.
+
+Operational risk — the risk from the system itself, not the market — is controlled via canary accounts (new strategy changes go live on a throwaway sub-account first), staged rollouts (ramp from 1% → 10% → 100% of size over hours), and dry-run/shadow modes where the new logic runs on live data but never submits orders. Cross-ref: `backend-fintech-order-lifecycle.md` for where pre-trade checks fire in the order flow, `backend-fintech-compliance.md` for regulatory-driven blocks (Reg SHO, sanctions), `backend-fintech-broker-integration.md` for how broker-enforced limits interact with ours, and `backend-fintech-testing.md` for chaos and shadow-mode test patterns.
+
+## Deep Guidance
+
+### Pre-Trade Check Pipeline
+
+Pre-trade checks run as an ordered pipeline, fail-fast, with every outcome (pass, block, bypass) emitting a structured audit event. Ordering matters: cheap, definitive checks first; expensive, I/O-bound checks last, so that a restricted-symbol block never waits on a margin calculation.
+
+```python
+# pre_trade.py
+CHECKS = [
+    check_kill_switch,           # 1. global halt (microseconds, in-memory)
+    check_restricted_symbol,     # 2. sanctions / hard list (cache lookup)
+    check_max_order_size,        # 3. notional and quantity caps
+    check_fat_finger_price,      # 4. limit price vs last trade (< 10% band)
+    check_position_limit,        # 5. would this breach max position?
+    check_portfolio_leverage,    # 6. requires full portfolio read
+    check_buying_power,          # 7. requires margin engine call
+]
+
+def validate(order, account, market) -> CheckResult:
+    for check in CHECKS:
+        result = check(order, account, market)
+        audit.record(order.client_order_id, check.__name__, result)
+        if result.action == BLOCK:
+            return result  # fail fast
+        if result.action == BYPASS:
+            audit.alert("pre_trade_bypass", order, check.__name__, result.reason)
+    return CheckResult(action=PASS)
+```
+
+Manual overrides ("bypass this check for this order") exist for operational reasons (e.g., a portfolio manager closing a position that would otherwise violate the concentration rule). They must be explicitly logged with operator identity, ticket reference, and the specific check bypassed — and they must page on-call, not silently drop.
+
+### Per-Symbol vs Per-Account vs Per-Segment Limits
+
+Limits compose from three axes and the most restrictive wins. Per-symbol limits cap exposure to idiosyncratic risk ("no more than 50k shares of AAPL"). Per-account limits cap total exposure for a single client ("no more than $10M notional long"). Per-segment limits apply policy to a class of accounts ("retail accounts capped at 2x leverage under Reg T; institutional prime-brokered accounts can run portfolio margin up to 6–7x effective"). An order that would fit the account and segment limits but breach the per-symbol cap still rejects.
+
+Store limits in a versioned config table, not code. Every change goes through a PR-like workflow with an approver, and activation is timestamped so post-mortems can answer "what were the limits at 14:32:07 UTC?"
+
+### Margin and Buying Power
+
+Under Reg T (US retail equities), initial margin is 50% — a $100k long position requires $50k equity. Maintenance margin is 25% (FINRA floor; most brokers set 30%). Pattern-day-trader (PDT) accounts with over $25k equity get 4x intraday buying power on marginable securities; non-PDT accounts are capped at 2x. Portfolio margin (for accounts over $100k–$150k and approved) replaces the flat percentage with a risk-array calculation — typically 10–15% haircut on diversified portfolios, which yields roughly 6–7x gross leverage on balanced books.
+
+Buying-power math must update in real time as fills land and marks move. A common and costly bug: computing buying power from the position snapshot but ignoring working orders (resting limits that could fill at any moment). The correct denominator is `cash + market_value - initial_margin_of_open_positions - initial_margin_of_working_orders`. Re-run the calculation on every fill, every cancel, and every mark-to-market tick on volatile books.
+
+### Drawdown Tracking
+
+Drawdown is the peak-to-trough decline of equity over a time window. Track at least two windows: intraday (high-water mark resets at session open) and rolling N-day (e.g., 5-day and 30-day high-water marks). Intraday drawdown is the tripwire for kill-switch automation; multi-day drawdown drives strategy-level throttles.
+
+```python
+# drawdown.py
+from collections import deque
+
+class DrawdownTracker:
+    def __init__(self, window_sec: int):
+        self.window_sec = window_sec
+        self.samples = deque()  # (ts, equity)
+        self.peak = None
+
+    def update(self, ts: float, equity: float) -> float:
+        self.samples.append((ts, equity))
+        cutoff = ts - self.window_sec
+        while self.samples and self.samples[0][0] < cutoff:
+            self.samples.popleft()
+        self.peak = max(s[1] for s in self.samples)
+        return (equity - self.peak) / self.peak  # negative = drawdown
+```
+
+Intraday peak resets at the session open boundary, not on a rolling clock — reset on the *event* (market open) rather than a sliding window, so the 15:59 peak does not suppress the drawdown calculation at 09:31 the next morning.
+
+### Circuit Breakers: Tiered Severity
+
+A flat "halt on any bad thing" is too blunt. Use four tiers, each with explicit trigger, action, and notification:
+
+- **Warning** (drawdown > 1%): log, dashboard badge, no trading impact. Analyst acknowledges.
+- **Throttle** (drawdown > 2.5%, loss velocity > $5k/min): reduce max order size by 50%, pause strategies tagged "high-turnover". Page on-call desk.
+- **Halt** (drawdown > 4%): block all new opening orders. Closing orders still allowed (to let risk come off). Page desk + engineering on-call.
+- **Kill** (drawdown > 5%, or catastrophic event): set global kill switch. Block all new orders including closes. Page desk + engineering + exec on-call.
+
+Each tier's transition (up or down) emits an event and requires explicit operator action to step *down* in severity. Automatic de-escalation is tempting and dangerous; it hides the fact that you tripped at all.
+
+### Kill-Switch Implementation
+
+The kill switch is a single atomic boolean in a low-latency store (Redis, an in-memory feature-flag service like LaunchDarkly, or a sidecar like Consul). Every order-submission path reads it before sending to the broker. Read latency budget: sub-millisecond. Cache locally in each process with a short TTL (e.g., 500ms) and a pub/sub invalidation channel so activation propagates in well under a second.
+
+```python
+# kill_switch.py
+STATES = ("disabled", "armed", "active", "recovering")
+# disabled: killswitch feature off (testing environments)
+# armed:    normal operation, switch is off
+# active:   halt in effect, new orders rejected
+# recovering: manual thaw; new orders allowed but flagged + rate-limited
+
+TRANSITIONS = {
+    ("armed",      "active"):     ["manual_trigger", "auto_drawdown", "auto_loss_velocity"],
+    ("active",     "recovering"): ["dual_control_approval"],  # two operators
+    ("recovering", "armed"):      ["dual_control_approval", "min_cooldown_elapsed"],
+}
+```
+
+Activation is one-click by any authorized operator. Deactivation requires two distinct operator identities approving within a short window (a classic dual-control pattern). Every transition writes to an append-only audit log with operator, timestamp, trigger reason, and current risk state snapshot.
+
+The "safe state" must be defined: does activation merely halt new orders, or does it also issue flatten-all market orders? Neither is universally right. For directional strategies in liquid markets, flatten is sensible. For market-making books, flattening into a wide spread can be worse than holding. Document the choice per strategy and encode it in the kill-switch config.
+
+### Testing Risk Controls
+
+Risk controls that have never been tripped in production are indistinguishable from risk controls that do not work. Test them continuously:
+
+- **Chaos tests**: on a schedule, in a staging environment with live-like data, force a breach of each tier. Verify the expected action fires and pages the expected team. Treat a silent failure here the same as a production incident.
+- **Shadow mode**: when rolling out a new check or tightening a limit, run it in shadow — the check evaluates and emits an event if it *would* have blocked, but does not actually block. Bake for days, compare the shadow block rate to expectations, then cut over.
+- **Simulated bad fills**: inject fabricated fill events (through the test harness described in `backend-fintech-testing.md`) that would push position past limits, and verify the post-trade pipeline catches and escalates.
+
+### Operational Risk Controls
+
+Beyond market risk, the system itself is a source of loss. A new strategy with a sign error can lose a day's P&L in minutes; a misconfigured limit can let a fat finger through. Three controls blunt this:
+
+- **Canary accounts**: every new strategy or material change first runs on a small, clearly-labeled sub-account with tight notional caps. If the canary burns, the loss is bounded. Promote to full size only after a defined soak period and a review checkpoint.
+- **Staged rollouts**: when scaling a strategy's size or rolling out a risk-check change, step through defined tiers (1% → 10% → 50% → 100% of target notional) with bake time at each. Automated rollback on deviation from expected P&L, fill rate, or error rate.
+- **Dry-run / shadow modes**: the strategy runtime supports a mode where it runs on live market data, generates orders, and logs them — but the submission layer drops them on the floor. Diff the shadow's intended orders against the production strategy's actual orders; any deviation is a finding to investigate before promotion.
+
+All three of these are first-class features in the order-submission path, not ad-hoc scripts. The feature flag determining shadow vs live is read at the same place as the kill switch, and its state is audited alongside every order.
+
+### Common Pitfalls
+
+- **Lagging risk state**: pre-trade checks read position from a cache or replica that is seconds behind the true ledger. A burst of orders can all pass individually while collectively breaching. Mitigate with a per-account in-process reservation (increment the projected position at check time, decrement on reject or terminal).
+- **Admin-path bypasses**: internal tools, reconciliation jobs, or "just this once" scripts that bypass the pre-trade pipeline. Every order-submitting path — human UI, strategy runtime, ops tooling, test harnesses in prod — goes through the same checks. No exceptions; enforce at the broker-integration layer (`backend-fintech-broker-integration.md`).
+- **Hardcoded limits**: "max 1000 shares" embedded in a constants file. When a legitimate larger trade needs to go, someone edits the constant, deploys, and forgets to revert. Limits belong in config, per-account, auditable.
+- **Kill switch with no safe-state definition**: activated during an incident, then the team argues in Slack about whether to flatten. Decide in advance; codify it.
+- **Single-operator kill-switch deactivation**: a compromised or distracted operator can re-enable trading prematurely. Dual control is cheap to implement and catches real mistakes.
+- **No drill cadence**: the kill switch is tested once at build time and never again. Run a kill-switch drill at least quarterly; measure activation-to-halt latency and deactivation-to-trading latency.
+- **Missing working-order margin**: buying-power math that ignores resting limit orders produces over-allocation when several of those limits fill in the same second. Always include working-order initial margin in the denominator.
+- **Automatic de-escalation**: circuit breakers that silently step down as conditions improve hide the fact that a tier was tripped. Require explicit operator action to re-arm, even if the underlying condition has cleared.

package/content/knowledge/backend/backend-fintech-testing.md

@@ -0,0 +1,197 @@
+---
+name: backend-fintech-testing
+description: Deterministic backtests; financial-accuracy tests; broker sandbox testing; regulatory edge-case coverage.
+topics: [backend, fintech, testing, determinism, backtesting, sandbox, accuracy, property-based]
+---
+
+Fintech tests have unusual requirements: bit-exact numeric accuracy, full determinism across runs and hosts, rich regulatory edge-case coverage, and realistic multi-session flows that span market-hours boundaries. A flakey fintech test is worse than no test — it hides the exact race conditions that cause real money to move incorrectly. This doc covers the patterns that keep backtests reproducible, numeric tests honest, broker integrations verifiable, and regulatory behavior exercised before it becomes an incident.
+
+## Summary
+
+The single largest failure mode in fintech test suites is **non-determinism**. A test that calls `new Date()`, reads `Math.random()`, or pulls the system clock inside the code under test will pass on a developer laptop at 10:00 PT and fail in CI at 03:00 UTC — or worse, pass 999 times and fail on the thousandth when a tick happens to straddle a market-open boundary. Every fintech test must inject time, randomness, UUID generation, and any external clock. There are no exceptions to this rule: if the production code reads the clock directly, the production code is wrong, not the test.
+
+**Numeric accuracy** is the second-largest source of bugs. Fintech tests should assert exact decimal equality, not `toBeCloseTo`. If you cannot predict the exact output to the last representable digit, you don't understand the computation — figure it out with a spreadsheet and pin the value. Test rounding modes explicitly (`ROUND_HALF_EVEN` vs `ROUND_HALF_UP` differ on 0.125 and similar ties) and test at boundary scales: sub-cent fees, multi-million-dollar positions, negative balances from wash-sale loss disallowance. See `backend-fintech-ledger.md` for the decimal and ledger invariants the tests are protecting.
+
+**Broker integration** testing is a tiered strategy: unit tests against a mock layer, integration tests against the broker's sandbox (Alpaca paper, IBKR demo, Tradier sandbox, Tradovate demo), and contract tests that hit the real sandbox on a schedule to catch upstream breaking changes. Use record/replay (`nock`, `vcrpy`, `polly-js`, `msw` with fixtures) so CI runs are fast and hermetic, then rotate recordings on a schedule so they don't drift from the live API. See `backend-fintech-broker-integration.md` for the integration surface being tested.
+
+**Regulatory edges** must be covered by explicit fixtures and properties: PDT-rule thresholds (Reg T, $25,000 minimum, 4th day trade within 5 business days), T+1 settlement (since May 28, 2024, US equities settle T+1, not T+2 — update old fixtures), halt/resume with opening-auction prices, corporate actions (dividends, splits, spin-offs, mergers) adjusting cost basis, partial fills and cancel-replace races, and market-closed order rejection. Property-based tests (`fast-check`, `hypothesis`, `jqwik`) are unusually well-suited here because financial invariants are algebraic ("sum of debits equals sum of credits across any window," "position after N operations equals sum of signed fills"). See `backend-fintech-compliance.md` for the rules these tests enforce and `backend-fintech-order-lifecycle.md` for the state machines they exercise.
+
+## Deep Guidance
+
+### Determinism Patterns
+
+The non-negotiable rule: **production code never calls `new Date()`, `Date.now()`, `time.time()`, `uuid.uuid4()`, `random.random()`, or `crypto.randomUUID()` directly**. Every such call is injected via a `Clock`, `IdGenerator`, `Randomness`, or similar seam. In tests the seam is replaced by a `TestClock` (advances on command), a deterministic UUID generator (`uuidv7` seeded, or a counter), and a seeded PRNG (`seedrandom`, `numpy.random.default_rng(42)`, `java.util.Random(seed)`).
+
+Sorting is the second silent source of flakes. Any list returned from a DB query, a broker response, or a cache must be sorted by a **deterministic tie-breaker** before being compared in tests. Prefer composite keys: `ORDER BY timestamp ASC, id ASC`. Relying on insertion order, hash order, or "whatever the DB returned" will fail on Postgres after an `ANALYZE`, on a parallel query plan, or in a different locale.
+
+Timezone handling is the third. Every stored timestamp is UTC. Every test that involves market-hours logic pins `TZ=America/New_York` in the test harness (or uses a zone-aware library like `luxon`, `pendulum`, `java.time.ZoneId`). Run the full suite once with `TZ=UTC` and once with `TZ=Asia/Tokyo` in CI; if anything differs you have a latent bug.
+
+### Property-Based Testing for Financial Invariants
+
+Property-based testing (`fast-check` in TS, `hypothesis` in Python, `jqwik` in Java, `proptest` in Rust) generates thousands of randomized inputs and asserts an invariant holds. The invariants in fintech are unusually strong, which makes properties cheap to write and high-value:
+
+- **Ledger balance**: for any sequence of journal entries, `sum(debits) == sum(credits)`.
+- **Position conservation**: `starting_position + sum(signed_fills) == ending_position` across any time window.
+- **Idempotency**: posting the same event twice leaves the ledger in the same state as posting it once.
+- **Commutativity where it should hold**: two non-overlapping transfers in either order produce the same final balances.
+- **Non-commutativity where it should not**: a partial fill followed by a cancel differs from a cancel followed by a partial fill; test that the race is rejected, not silently reordered.
+- **Rounding monotonicity**: `round(a) + round(b) <= round(a + b) + 1 ulp` for `ROUND_HALF_EVEN`.
+
+Shrinking is the killer feature: when a property fails, the framework shrinks to a minimal counterexample. A 4-line failing test with `amount=0.125, mode=HALF_EVEN` is worth a hundred anecdotal bug reports.
+
+### Numeric Test Patterns
+
+Never use floating-point for money, and never use `toBeCloseTo` / `assertAlmostEqual` in money tests — those are noise suppressors that hide off-by-one-cent bugs. Use `Decimal` (`decimal.js`, `big.js`, Python `decimal.Decimal`, Java `BigDecimal`, `rust_decimal`) and assert exact string equality on the serialized form: `expect(result.toFixed(4)).toBe("1234.5678")`.
+
+Test rounding **modes** explicitly, not just results. `ROUND_HALF_EVEN` (banker's rounding, IEEE 754 default, required by many regulators) rounds 0.5 to the nearest even: 0.5 → 0, 1.5 → 2, 2.5 → 2. `ROUND_HALF_UP` rounds 0.5 away from zero: 0.5 → 1, 1.5 → 2, 2.5 → 3. A test that exercises 0.125 and 0.375 at 2-decimal scale will catch a mode swap.
+
+Pin the expected values by hand calculation, not by "run it and snapshot." A snapshot just records whatever the buggy code produced the first time. Work the expected output on paper (or in a spreadsheet you check in alongside the test fixture), then assert exact match.
+
+### Regulatory Scenario Fixtures
+
+Build a fixture library that exhaustively exercises the rules:
+
+- **PDT threshold**: account at $24,999.99 attempts a 4th day trade within a 5-business-day window → rejected with `PDT_VIOLATION`. Account at $25,000.00 → allowed. Account flagged PDT then rising above threshold on the next close → still restricted until equity holds for 5 business days.
+- **T+1 settlement**: a sale on Monday settles Tuesday. Friday sale → Monday settlement. Friday sale before a Monday holiday → Tuesday settlement. Generate a calendar for the current and prior year with holidays and test the arithmetic against a known-good source (NYSE calendar).
+- **Corporate actions**: 2-for-1 split of a 100-share position at $50 cost basis → 200 shares at $25. Cash dividend of $0.50/share → cash credit, no position change. Spin-off with cost-basis allocation ratio → both positions adjusted. Merger with cash-and-stock consideration → realized gain on cash portion, carryover basis on stock portion.
+- **Halt / resume**: order placed during LULD halt → rejected or queued per venue rules. Opening auction after resume → fills at the auction print, not the last pre-halt trade. Test that mark-to-market during a halt uses the last trade, not zero.
+- **After-hours and pre-market**: market-on-close order entered at 16:15 ET → rejected. Limit order marked `extended_hours=true` entered at 07:30 ET → accepted. Order from a user whose account is not enabled for extended hours → rejected with `EXT_HOURS_NOT_ENABLED`.
+
+### Broker Sandbox Testing
+
+Every major broker offers a sandbox — Alpaca Paper, IBKR Paper Trading, Tradier Sandbox, Tradovate Demo, Schwab (former TDA) Developer Sandbox. Sandboxes differ from production in important ways: instant fills regardless of liquidity, synthetic market data that may not match real quotes, relaxed rate limits (or stricter — Alpaca sandbox caps at 200 req/min vs 10,000/min in live), and simplified corporate-action handling. Write a table of documented sandbox-vs-prod behavioral differences and pin it to the integration-test README so nobody is surprised.
+
+Auth flows also differ. Most sandboxes use a fixed long-lived token; production uses OAuth with refresh. Tests must cover both paths — fake the OAuth refresh in unit tests, exercise the real refresh in a nightly contract test.
+
+### Record and Replay
+
+For CI speed and hermeticity, capture real sandbox responses once with `nock.recorder.rec()` (Node), `vcrpy` (Python), `WireMock` (Java), or `msw`'s `setupServer` with pre-baked fixtures, then replay them in CI. The recordings live next to the test file and are checked in.
+
+**Rotation strategy** is what makes this durable: a scheduled job (weekly or monthly via GitHub Actions cron) re-runs the recording step against the live sandbox and diffs the new captures against the committed ones. A non-trivial diff opens a PR for human review — either the API changed (update expectations and code) or the sandbox changed (update fixtures only). Without rotation, fixtures silently drift and CI becomes a liar.
+
+### Contract Tests Against Live Brokers
+
+Separate from replay-based tests, run a small **contract test** suite against the real sandbox on a schedule (weekly is typical, nightly for active integrations). These tests assert the shape of critical responses — field names, required fields, enum values, error codes — without asserting specific business outcomes. Failures here are early warning of broker API changes, often before the broker's own changelog is published. Tools: `pact` for consumer-driven contracts, or a hand-rolled `zod`/`pydantic`/`jsonschema` validator run against live responses.
+
+### Common Pitfalls
+
+- **Tests pass locally, fail in CI**: almost always timezone (`TZ` differs) or locale (`LANG`, `LC_ALL` affect number parsing). Pin both in the test harness and in CI config.
+- **Flakey market-hours tests**: the test reads the real clock and asserts "market is open." Fix by injecting a `Clock` and freezing it at a known instant inside the trading session.
+- **Sandbox instant fills**: code assumes fills are instant because sandbox makes them so; production has a partial-fill / queued-order path that was never tested. Write a mock broker that delays and partially fills, separate from the sandbox.
+- **Fixtures drift from schemas**: recorded responses reference fields the API no longer returns, and production code silently handles the absence incorrectly. Rotation (above) plus a strict schema validator on every response catches this.
+- **UUID comparisons**: tests that assert full UUID equality on generated ids are brittle; either inject the generator or assert shape (`expect(id).toMatch(/^[0-9a-f-]{36}$/)`). Prefer injection for determinism.
+- **Decimal serialization mismatches**: `Decimal("1.10")` and `Decimal("1.1")` compare equal but serialize differently; pin the canonical form in the test and assert on the serialized output.
+- **Shared state between tests**: a test that uses the real file system, a singleton clock, or a module-level cache breaks parallel execution. Every fixture is constructed per-test.
+
+### Code Examples
+
+Clock-injection wrapper (TypeScript, vitest):
+
+```typescript
+// src/infra/clock.ts
+export interface Clock { now(): Date; }
+export const SystemClock: Clock = { now: () => new Date() };
+export class TestClock implements Clock {
+  constructor(private current: Date) {}
+  now() { return new Date(this.current); }
+  advance(ms: number) { this.current = new Date(this.current.getTime() + ms); }
+  setTo(iso: string) { this.current = new Date(iso); }
+}
+
+// src/orders/submit.ts
+export function submitOrder(clock: Clock, order: Order) {
+  if (!isMarketOpen(clock.now())) throw new Error("MARKET_CLOSED");
+  return { ...order, submittedAt: clock.now().toISOString() };
+}
+
+// tests/orders/submit.test.ts
+import { describe, it, expect } from "vitest";
+import { TestClock } from "../../src/infra/clock";
+import { submitOrder } from "../../src/orders/submit";
+
+describe("submitOrder", () => {
+  it("rejects before market open", () => {
+    const clock = new TestClock(new Date("2026-04-15T13:29:59Z")); // 09:29:59 ET
+    expect(() => submitOrder(clock, baseOrder)).toThrow("MARKET_CLOSED");
+  });
+  it("accepts exactly at market open", () => {
+    const clock = new TestClock(new Date("2026-04-15T13:30:00Z")); // 09:30:00 ET
+    expect(submitOrder(clock, baseOrder).submittedAt).toBe("2026-04-15T13:30:00.000Z");
+  });
+});
+```
+
+Property-based invariant (Python, hypothesis):
+
+```python
+# tests/ledger/test_invariants.py
+from decimal import Decimal
+from hypothesis import given, strategies as st
+from app.ledger import post_entry, account_balance, Ledger
+
+amounts = st.decimals(min_value=Decimal("0.01"), max_value=Decimal("100000"), places=2)
+accounts = st.sampled_from(["cash", "customer:alice", "customer:bob", "fees"])
+
+@given(st.lists(st.tuples(accounts, accounts, amounts), min_size=0, max_size=50))
+def test_double_entry_sums_to_zero(transfers):
+    ledger = Ledger()
+    for src, dst, amt in transfers:
+        if src == dst:
+            continue
+        post_entry(ledger, src=src, dst=dst, amount=amt)
+    total = sum(account_balance(ledger, a) for a in ledger.accounts)
+    assert total == Decimal("0"), f"ledger imbalanced by {total}"
+
+@given(st.lists(st.tuples(accounts, accounts, amounts), min_size=1, max_size=20))
+def test_posting_is_idempotent(transfers):
+    l1, l2 = Ledger(), Ledger()
+    for i, (src, dst, amt) in enumerate(transfers):
+        if src == dst:
+            continue
+        post_entry(l1, src=src, dst=dst, amount=amt, idempotency_key=f"k-{i}")
+        post_entry(l2, src=src, dst=dst, amount=amt, idempotency_key=f"k-{i}")
+        post_entry(l2, src=src, dst=dst, amount=amt, idempotency_key=f"k-{i}")  # replay
+    for acct in l1.accounts:
+        assert account_balance(l1, acct) == account_balance(l2, acct)
+```
+
+Decimal-precision test with exact-match assertions (TypeScript, vitest + `decimal.js`):
+
+```typescript
+import { Decimal } from "decimal.js";
+import { describe, it, expect } from "vitest";
+import { computeCommission } from "../src/fees";
+
+describe("computeCommission", () => {
+  // Rate 0.00125 on notional $10,000.00 = $12.50 exactly
+  it("computes exact commission at representable scale", () => {
+    const notional = new Decimal("10000.00");
+    const rate = new Decimal("0.00125");
+    expect(computeCommission(notional, rate).toFixed(4)).toBe("12.5000");
+  });
+
+  // ROUND_HALF_EVEN (banker's) on 0.125 at 2dp -> 0.12 (even), not 0.13
+  it("uses ROUND_HALF_EVEN on tie", () => {
+    Decimal.set({ rounding: Decimal.ROUND_HALF_EVEN });
+    const notional = new Decimal("100.00");
+    const rate = new Decimal("0.00125"); // = $0.125
+    expect(computeCommission(notional, rate).toDecimalPlaces(2).toFixed(2)).toBe("0.12");
+  });
+
+  // Contrast: ROUND_HALF_UP would give 0.13
+  it("differs from ROUND_HALF_UP on tie", () => {
+    Decimal.set({ rounding: Decimal.ROUND_HALF_UP });
+    const notional = new Decimal("100.00");
+    const rate = new Decimal("0.00125");
+    expect(computeCommission(notional, rate).toDecimalPlaces(2).toFixed(2)).toBe("0.13");
+  });
+});
+```
+
+### Cross-References
+
+- `backend-fintech-ledger.md` — the invariants these tests protect (double-entry, idempotency, reconciliation).
+- `backend-fintech-order-lifecycle.md` — state machines exercised by order-flow tests.
+- `backend-fintech-data-modeling.md` — schemas whose contracts the fixtures match.
+- `backend-fintech-broker-integration.md` — the integration surface being sandboxed, recorded, and replayed.
+- `backend-testing.md` — general backend testing conventions layered beneath the fintech-specific patterns above.

package/content/knowledge/core/automated-review-tooling.md

@@ -193,3 +193,13 @@ When external CLIs are unavailable, the degraded-mode behavior defined in the Su
 6. Never silently drop unavailable channels — always record the channel status and compensating coverage label in the review output.
 
 **Claude CLI channel:** Claude CLI handles its own auth and is generally always available. The compensating-pass mechanism applies to external CLIs (Codex, Gemini) that have an installation/auth gate. When Codex or Gemini are unavailable, compensating passes are dispatched via `claude -p` with focused prompts targeting the missing channel's strength area.
+
+### Auth Recovery Paths
+
+Each external CLI has a distinct auth recovery path. Agents should surface these directly to the user rather than silently downgrading to a compensating pass:
+
+- **Codex:** `codex login` — opens an interactive OAuth flow. After success, `codex login status` should return cleanly.
+- **Gemini:** `gemini -p "hello"` — refreshes the token if expired; `NO_BROWSER=true` is required in headless environments.
+- **Claude:** `claude auth login` if `claude -p` returns auth errors; rare in practice because Claude CLI tokens are long-lived.
+
+If the user cannot complete auth recovery within the review session, treat the channel as unavailable and document the compensating pass. Never attempt to work around auth failures by embedding credentials in review prompts or by piping through alternative providers that weren't explicitly requested.