@zigrivers/scaffold 3.16.0 → 3.18.0

package/content/knowledge/backend/backend-fintech-data-modeling.md

@@ -0,0 +1,210 @@
+---
+name: backend-fintech-data-modeling
+description: Financial data models; currency handling; decimal precision; positions, trades, prices; time-series designs.
+topics: [backend, fintech, data-modeling, decimal, currency, time-series, positions, trades]
+---
+
+Financial data modeling is where most fintech bugs are born: a float creeps into a money field, a currency is implied instead of stored, a tick table grows unbounded, or a `current_position` column drifts from the journal. This doc covers the non-negotiable shapes of money, quantity, and price data, and the time-series and derived-view patterns that keep a trading or banking system honest at scale.
+
+## Summary
+
+**Money is never a float.** Every stack has a correct type and it is never IEEE-754 binary floating point. Use `NUMERIC(precision, scale)` or integer minor units in Postgres; `Decimal` with an explicit context in Python; `BigDecimal` with `setScale` in Java; `decimal.js`, `big.js`, or `bignumber.js` in JavaScript/TypeScript (with `string` on the wire, not `number`); `rust_decimal` in Rust; `shopspring/decimal` in Go. A `double` anywhere in the money path is a latent bug, not a performance optimization.
+
+**Currencies are not interchangeable.** Every money field is a `(amount, currency)` tuple, not a bare scalar. Store the currency as an ISO 4217 alpha-3 code (`USD`, `EUR`, `JPY`, `BTC`, `USDT`) alongside the amount, and reject any arithmetic that mixes currencies without an explicit FX conversion. Aggregations (`SUM(amount)`) across mixed currencies are always wrong — group by currency or convert at a snapshotted rate first.
+
+**Quantities have their own precision, separate from price.** Equity shares are typically integers (fractional-share brokers use 6–9 decimals); Bitcoin quantities need 8 decimals (satoshis); Ethereum and most ERC-20 tokens need 18 decimals (wei); stablecoins like USDT and USDC use 6 decimals; FX quantities are usually 2 decimals for major pairs but pip size varies per pair. Store the instrument's quantity scale in the instrument master and validate every fill against it.
+
+**Prices change constantly and the storage design must account for it.** Tick data (every quote, every trade) at scale means billions of rows per instrument per year — OLTP Postgres will not hold it. Use a columnar or purpose-built time-series store (TimescaleDB hypertables, ClickHouse with `ReplacingMergeTree`, InfluxDB, or Parquet-on-object-storage queried via DuckDB) with explicit retention policies. Bars (1-minute, 1-hour, 1-day OHLCV) live alongside ticks and are the common read path for charts and analytics.
+
+**Positions are a derived view, not a primary table.** The authoritative record is the journal of fills (and, for cash, the ledger — see `backend-fintech-ledger.md`). A position is `SUM(signed_quantity) GROUP BY account, instrument`, optionally as of a point in time. Maintain a materialized view or cache for read performance, but always be able to rebuild it from the journal — any system that cannot do this has lost its audit story.
+
+**Corporate actions require keeping both raw and adjusted prices.** Splits, dividends, and mergers retroactively change historical price series. Keep raw prices (what actually printed on the tape) immutable, and maintain a parallel `adjusted_price` column or adjustment-factor table so charting and backtesting get continuous series without losing the original record.
+
+## Deep Guidance
+
+### Decimal Types Across the Stack
+
+Floats fail on money because `0.1 + 0.2 !== 0.3` in every IEEE-754 language, and because the error compounds across additions in a way you cannot bound. The canonical replacements:
+
+- **Postgres**: `NUMERIC(precision, scale)` — e.g., `NUMERIC(20, 8)` for crypto amounts, `NUMERIC(18, 2)` for fiat. Arbitrary precision, exact arithmetic, slower than `BIGINT`. For high-throughput hot paths, store integer **minor units** (cents, satoshis, wei) in `BIGINT` or `NUMERIC(38, 0)` and keep the scale in the instrument master.
+- **Python**: `decimal.Decimal` with `getcontext().prec = 28` and an explicit rounding mode set at application startup. Never mix `Decimal` and `float` in arithmetic — Python will not error, it will coerce and you will silently lose precision.
+- **Java / Kotlin**: `BigDecimal`, always with `setScale(n, RoundingMode.HALF_EVEN)` or `HALF_UP` explicitly chosen per context. Never call `new BigDecimal(double)` — use `BigDecimal.valueOf(double)` or, better, pass strings.
+- **JavaScript / TypeScript**: JavaScript `Number` is a `double`; `BigInt` handles integer minor units but not fractional amounts. Use `decimal.js`, `big.js`, or `bignumber.js` for fractional money; serialize as strings on the wire (`"123.45"`, never `123.45`). GraphQL and JSON Schema both support string-encoded decimals.
+- **Rust**: `rust_decimal::Decimal` for fixed-scale, `num-bigint::BigInt` + scale for arbitrary precision.
+- **Go**: `shopspring/decimal` is the de facto standard.
+
+On the wire: JSON numbers are IEEE-754 doubles in most parsers, so serialize money as strings. Protobuf has no decimal type — use a message like `{ string value; int32 scale; string currency; }` or send the minor-unit integer with an out-of-band scale from the instrument master.
+
+### Currency Representation
+
+Use ISO 4217 alpha-3 codes: `USD`, `EUR`, `JPY`, `CHF`, `GBP`, plus crypto conventions that extend the space: `BTC`, `ETH`, `USDT`, `USDC`, `SOL`. Keep a reference `currencies` table with `code`, `name`, `minor_unit_exponent` (2 for USD, 0 for JPY, 8 for BTC, 18 for ETH, 6 for USDT), `is_fiat`, `is_active`.
+
+The "store money as cents" advice is only correct for fiat with 2 decimal places. JPY has no sub-unit (exponent 0); KWD has 3; BTC has 8; ETH has 18; USDT has 6. Hard-coding `*100` anywhere in the codebase is a bug waiting for a non-USD customer. Always read the scale from the currencies table or instrument master, or use a decimal type that carries scale explicitly.
+
+```sql
+CREATE TABLE currencies (
+  code                 CHAR(3) PRIMARY KEY,           -- or VARCHAR(10) for longer crypto tickers
+  name                 TEXT NOT NULL,
+  minor_unit_exponent  SMALLINT NOT NULL,             -- 2=USD, 0=JPY, 8=BTC, 18=ETH
+  is_fiat              BOOLEAN NOT NULL,
+  is_active            BOOLEAN NOT NULL DEFAULT true
+);
+
+CREATE TABLE cash_movements (
+  id               BIGSERIAL PRIMARY KEY,
+  account_id       UUID NOT NULL,
+  amount           NUMERIC(38, 18) NOT NULL,          -- scale wide enough for ETH
+  currency         CHAR(3) NOT NULL REFERENCES currencies(code),
+  occurred_at      TIMESTAMPTZ NOT NULL,
+  CHECK (amount <> 0)
+);
+```
+
+### Rounding Rules
+
+Rounding must be explicit, contextual, and documented. Defaults:
+
+- **Banker's rounding (HALF_EVEN)** for settlement and regulatory reporting — statistically unbiased across many rounding events.
+- **HALF_UP** for consumer-facing invoice totals where "round half away from zero" matches user expectations.
+- **Truncation (ROUND_DOWN)** for display of quantities where you must not overstate (e.g., a balance-available readout must never round up).
+- **HALF_UP or ceiling** for fees charged to the customer in your favor — so you do not under-collect — with regulatory review.
+
+The footgun case: `1.005` rendered to 2 decimals. In binary float it is `1.00499999...`, which HALF_UP rounds to `1.00`, not `1.01`. Use a decimal library and explicit context, and write a test for exactly this case.
+
+### Multi-Currency Positions
+
+Two patterns, pick one and be explicit:
+
+1. **Native-currency storage + periodic revaluation.** Each position and each cash balance is stored in its native currency. At report time, convert to the reporting currency using a snapshotted FX rate (end-of-day close, or an intraday snapshot for intraday P&L). Preserves accuracy and audit trail, simplifies settlement.
+2. **Functional-currency storage + memo native amounts.** Everything is denominated in the reporting currency (e.g., USD); foreign-currency fills are converted at execution time. Simpler to aggregate but loses information — revaluing an FX exposure later requires the memo column.
+
+For broker/exchange integration, pattern 1 is nearly always correct. For internal accounting that feeds a single GAAP/IFRS reporting currency, pattern 2 is common. See `backend-fintech-ledger.md` for the journal-entry mechanics of FX conversion.
+
+### Time-Series Storage: Ticks, Bars, and Retention
+
+Decide early what granularity you need to keep, and for how long. A naive "store every tick forever in Postgres" plan breaks at the first liquid instrument. Options:
+
+- **TimescaleDB hypertables** — Postgres extension, partitions by time transparently, supports continuous aggregates (materialized bars), native compression (10–20x) and retention policies. Good choice when you already have Postgres expertise.
+- **ClickHouse `ReplacingMergeTree` / `AggregatingMergeTree`** — columnar, extreme compression, fast time-range scans, scales to trillions of rows. Best for high-volume tick capture and analytics.
+- **InfluxDB** — time-series native, good for metrics-shaped data, less common for financial tick capture.
+- **Parquet on object storage + DuckDB / Athena / BigQuery** — effectively free cold storage, pay per query; excellent for long-term archive and backtesting over years of history.
+
+Tick schema: `(instrument_id, exchange_id, ts, price, size, side, seq)`. Always store the exchange timestamp *and* your ingest timestamp — clock skew matters. Bar schema: `(instrument_id, resolution, open_ts, open, high, low, close, volume, vwap, tick_count)`.
+
+```sql
+-- TimescaleDB: hypertable + retention + continuous aggregate
+CREATE TABLE ticks (
+  instrument_id  BIGINT NOT NULL,
+  exchange_id    SMALLINT NOT NULL,
+  ts             TIMESTAMPTZ NOT NULL,
+  price          NUMERIC(20, 8) NOT NULL,
+  size           NUMERIC(28, 8) NOT NULL,
+  side           CHAR(1) NOT NULL CHECK (side IN ('B','S','U')),
+  seq            BIGINT NOT NULL
+);
+SELECT create_hypertable('ticks', 'ts', chunk_time_interval => INTERVAL '1 day');
+
+-- Keep raw ticks for 30 days, then drop; bars are retained separately
+SELECT add_retention_policy('ticks', INTERVAL '30 days');
+
+-- 1-minute bars as a continuous aggregate, retained 10 years
+CREATE MATERIALIZED VIEW bars_1m
+WITH (timescaledb.continuous) AS
+SELECT instrument_id,
+       time_bucket('1 minute', ts) AS bucket,
+       first(price, ts) AS open,
+       max(price)       AS high,
+       min(price)       AS low,
+       last(price, ts)  AS close,
+       sum(size)        AS volume
+  FROM ticks
+  GROUP BY instrument_id, bucket;
+SELECT add_continuous_aggregate_policy('bars_1m',
+  start_offset => INTERVAL '2 days',
+  end_offset   => INTERVAL '1 minute',
+  schedule_interval => INTERVAL '1 minute');
+```
+
+### Corporate Actions and Price Adjustment
+
+A 2:1 stock split doubles the share count and halves the price. Historical charts must be adjusted so the price series is continuous, but *settlement* still used the unadjusted prices. Keep both:
+
+- `prices_raw(instrument_id, ts, price)` — immutable, what actually printed.
+- `corporate_actions(instrument_id, effective_date, action_type, ratio_numerator, ratio_denominator, cash_amount, currency)` — authoritative record of splits, dividends, mergers.
+- `adjusted_price` — either a materialized column computed from raw + the product of all adjustment factors effective after the tick date, or computed on read via a view.
+
+Never overwrite the raw price. If a data vendor re-emits history post-split and you overwrite, you cannot reconcile past trade tickets. The same principle applies to reference data generally: store the vendor's snapshot with a `received_at`, and derive any "current" projection.
+
+### Position Model
+
+A position is `SUM(signed_quantity) GROUP BY (account_id, instrument_id)` over the fills journal, where `signed_quantity` is positive for buys and negative for sells (and the reverse for shorts). Cost basis is `SUM(signed_quantity * price) / SUM(signed_quantity)` under average-cost, or a FIFO lot walk for tax-lot accounting.
+
+Implementation options:
+
+- **Read-time aggregation** — correct and simple, fine up to ~10M fills per account; use an index on `(account_id, instrument_id, executed_at)`.
+- **Materialized view, refreshed nightly** — for dashboards; accept T-1 staleness in exchange for cheap reads.
+- **Rolling snapshot table** — `positions(account_id, instrument_id, as_of_ts, quantity, avg_cost)` updated by trigger on fill insert. Fastest reads, most complex invariants; regenerate from the fills journal on any suspicion of drift.
+- **Event-sourced with periodic checkpoints** — for point-in-time queries ("what was my position at 14:32:17 on 2026-03-14?"), keep fills + periodic snapshots and replay from nearest snapshot.
+
+Whichever you pick, maintain a `rebuild_positions(account_id)` job that recomputes from the fills journal; run it nightly against a sample of accounts and alarm on any drift.
+
+### Trade Identifiers: Internal, Broker, Client
+
+Every executed trade has at least three identifiers, and you should store all of them:
+
+- **Internal trade id (UUID)** — your primary key; generated at ingest; never changes; used in all downstream references (ledger `external_id`, position rebuilds, tax lots).
+- **Broker execution id** — the exchange or broker's identifier for the fill. Used for reconciliation against the broker's clearing feed (see `backend-fintech-ledger.md` reconciliation). May arrive after the ack — treat as nullable initially and backfill.
+- **Client order id (`clOrdID`)** — generated by your order-management system before placing the order. Round-trips through the broker; used to match acks and fills back to the originating intent. Must be globally unique per client-session per the FIX spec (and per broker rules — some require monotonically increasing).
+
+Store venue, symbol (broker's symbology *and* your canonical instrument id), side, quantity, price, fees in their own currency, liquidity flag (maker/taker), and timestamps (client-submitted, broker-ack, exchange-executed, ingest). See `backend-fintech-order-lifecycle.md` for the state-machine that connects these.
+
+### Cross-Currency Arithmetic: Reject at the Boundary
+
+The single highest-leverage fintech habit: refuse to add two money values in different currencies. Write it as a typed function; make the compiler or runtime enforce it.
+
+```typescript
+import Decimal from 'decimal.js';
+
+type Money = { amount: Decimal; currency: string };
+
+function add(a: Money, b: Money): Money {
+  if (a.currency !== b.currency) {
+    throw new Error(
+      `cannot add ${a.currency} and ${b.currency} without explicit conversion`,
+    );
+  }
+  return { amount: a.amount.plus(b.amount), currency: a.currency };
+}
+
+function convert(m: Money, toCurrency: string, rate: Decimal, rateAsOf: Date): Money {
+  // rate is units of toCurrency per unit of m.currency, snapshotted at rateAsOf
+  return { amount: m.amount.times(rate), currency: toCurrency };
+}
+
+// Aggregations over mixed currencies must group explicitly
+function sumByCurrency(movements: Money[]): Map<string, Decimal> {
+  const totals = new Map<string, Decimal>();
+  for (const m of movements) {
+    const prev = totals.get(m.currency) ?? new Decimal(0);
+    totals.set(m.currency, prev.plus(m.amount));
+  }
+  return totals;
+}
+```
+
+In Python, subclass `Decimal` or wrap in a `Money` dataclass with `__add__` raising on currency mismatch; same pattern in Kotlin with a `Money` value class. Every SQL aggregation over money must `GROUP BY currency` or restrict to a single currency in the `WHERE` clause.
+
+### Common Pitfalls
+
+- **`0.1 + 0.2`-class bugs in money math.** JavaScript `Number`, Python `float`, `double` in JVM/.NET, `REAL`/`DOUBLE PRECISION` in Postgres — all unsafe for money. Use decimal types end to end, including the JSON wire format (strings, not numbers).
+- **Banker's rounding vs HALF_UP confusion.** `Decimal('0.5').quantize(Decimal('1'))` in Python rounds to 0 by default (banker's), not 1. Explicitly pass the rounding mode matching the business rule per call site.
+- **Equality comparison on floats or even decimals.** `amount == 0` is fine on integer minor units or exact decimals; `amount < epsilon` is a code smell that usually means a float slipped in. Money is exact.
+- **Mixing currencies in `SUM()`.** Running `SELECT SUM(amount) FROM cash_movements` without grouping by currency yields meaningless numbers; forbid it at the query-review layer.
+- **Hardcoding `*100` for cents conversion.** Works for USD, breaks for JPY (no sub-unit), BTC (8 decimals), ETH (18), USDT (6). Always read the minor-unit exponent from the currencies table.
+- **Unbounded tick retention in OLTP.** Every liquid instrument floods the table. Pick a columnar or TSDB backend, set a retention policy, materialize bars, and move old raw ticks to cold storage.
+- **Overwriting raw prices after corporate actions.** Split-adjust historical bars at read time, not by mutating the raw series. Keep the corporate-actions table as the authoritative record.
+- **`current_position` columns in user tables.** Drift from the fills journal, get clobbered by races, silently wrong. Derive, materialize, and nightly-reconcile.
+- **Mixing transaction time and system time.** Exchange timestamp, broker ack timestamp, and your ingest timestamp are three different clocks. Store all three on fills and ticks; alarm on skew above threshold.
+- **Implicit currency from account context.** "This account is a USD account, so amounts are in USD" — until it isn't, or until a crypto product ships. Every money field carries its currency.
+
+See also `backend-fintech-ledger.md` for double-entry posting over these same primitives, `backend-fintech-testing.md` for property-based money-math tests and time-series fixture patterns, and `backend-fintech-compliance.md` for the audit-trail and retention constraints that shape historical data storage.

package/content/knowledge/backend/backend-fintech-ledger.md

@@ -0,0 +1,226 @@
+---
+name: backend-fintech-ledger
+description: Double-entry accounting for fintech ledgers; journal vs ledger tables; idempotent posting; reconciliation patterns; balance invariants.
+topics: [backend, fintech, ledger, double-entry, accounting, reconciliation, idempotency, invariants]
+---
+
+A fintech ledger is the authoritative record of money movement; if it is wrong, nothing else in the system can be trusted. The discipline is borrowed intact from 700 years of double-entry bookkeeping — not a new invention, and not negotiable. This doc covers the invariants, schema shape, idempotent posting mechanics, and reconciliation patterns that keep a ledger survivable at production scale.
+
+## Summary
+
+The single most important invariant in any fintech system: **for every credit there is an equal-amount debit, and the sum of debits across a journal entry equals the sum of credits**. This is double-entry accounting. Enforce it in the database, not just the application — a balanced journal entry is the atomic unit of money movement. A single journal entry may have two lines (simple transfer) or dozens (payroll run, fee split, multi-leg trade settlement), but the sum-to-zero constraint holds for every entry.
+
+You **cannot** derive balances from `current_balance` columns updated in application tables. Those columns drift, get clobbered by race conditions, and are unauditable. The balance of an account at any point in time is defined as the sum of all posted ledger lines against that account up to that instant. The journal is the system of record; balances are a projection.
+
+The canonical schema is three layers: **journal** (immutable events — one row per business event, with an external idempotency key and a posting timestamp), **ledger lines** (the double-entry rows — two or more per journal entry, each referencing an account and signed amount), and **account balances** (either a materialized view refreshed periodically, a rolling aggregation maintained by trigger, or computed on demand for low-volume accounts). Write-once to journal and ledger-lines; balances are derived and can always be rebuilt from the journal.
+
+Every journal insert MUST carry an external idempotency key — usually a UUID derived from the upstream event (webhook event ID, Stripe payment intent ID, broker execution ID, client-supplied `Idempotency-Key` header). Retries are a fact of life: payment processors retry webhooks, queues redeliver, operators click twice. The key is enforced by a unique index on `(source, external_id)` so the second attempt fails cleanly and the caller receives the original posting. Libraries like `ledgers.db` (Mercury's pattern), `tigerbeetle`, Square's `subzero`, and `medici` (Node) codify this. Most fintech outages can be traced to a missing or mis-scoped idempotency key.
+
+Reconciliation is a daily, non-optional process. For each counterparty (bank, broker, card processor, payment rail), the ledger's expected cash movement is matched against the counterparty's settlement file or API feed. Matched items clear; unmatched items go into a "breaks queue" with aging (1 day, 3 days, 7 days, escalate). A break is either a timing difference that will resolve on the next feed, a fee the counterparty applied that you didn't book, or a bug. Break quarantine prevents a single unmatched item from blocking close; root cause must be found and booked before period close. See `backend-fintech-compliance.md` for audit-trail requirements and `backend-fintech-order-lifecycle.md` for trade-specific settlement flows.
+
+## Deep Guidance
+
+### Chart of Accounts Design
+
+The chart of accounts (COA) is the taxonomy every ledger line must pick from. Classic classes: **assets** (cash, receivables, inventory), **liabilities** (customer deposits, payables), **equity** (retained earnings, owner's capital), **revenue** (fees, interest earned), **expenses** (processing fees, bad debt). Customer deposits in a consumer fintech are a *liability* on your balance sheet — the money belongs to the customer, you're holding it.
+
+Granularity matters more than people expect. Per-customer wallet balances require a distinct account per customer (or per customer-currency pair) — millions of rows of accounts is fine, this is what modern ledgers are designed for. Operational accounts (cash-at-bank, processing-fee-expense, interchange-revenue) are few and coarse. A typical pattern: customer accounts identified by `customer_id`, operational accounts identified by a stable account code (`1000-CASH-USD`, `4000-FEE-REVENUE`, `5000-PROCESSOR-COST`).
+
+Avoid "merge" accounts that combine classes — "customer cash and fees" is wrong; split them. Every account has exactly one normal balance (debit for assets/expenses, credit for liabilities/equity/revenue) and one class.
+
+```sql
+CREATE TABLE accounts (
+  id            UUID PRIMARY KEY,
+  code          TEXT UNIQUE,           -- '1000-CASH-USD' or NULL for customer accounts
+  name          TEXT NOT NULL,
+  class         TEXT NOT NULL CHECK (class IN ('asset','liability','equity','revenue','expense')),
+  normal_side   TEXT NOT NULL CHECK (normal_side IN ('debit','credit')),
+  currency      CHAR(3) NOT NULL,      -- ISO 4217
+  customer_id   UUID REFERENCES customers(id),
+  is_active     BOOLEAN NOT NULL DEFAULT true,
+  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+```
+
+### Journal Entry and Ledger Line Structure
+
+A journal entry represents one business event. Ledger lines represent the debits and credits that settle that event. Every ledger line belongs to exactly one journal entry. Every journal entry must sum to zero per currency.
+
+```sql
+CREATE TABLE journal_entries (
+  id               UUID PRIMARY KEY,
+  source           TEXT NOT NULL,        -- 'stripe' | 'broker' | 'manual' | 'internal'
+  external_id      TEXT NOT NULL,        -- idempotency key (e.g. Stripe event id)
+  counterparty_id  UUID REFERENCES counterparties(id),
+  memo             TEXT,
+  transaction_date DATE NOT NULL,        -- when economically occurred
+  posting_date     DATE NOT NULL,        -- when booked to the ledger
+  posted_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
+  posted_by        UUID NOT NULL,        -- actor id
+  UNIQUE (source, external_id)
+);
+
+CREATE TABLE ledger_lines (
+  id                  BIGSERIAL PRIMARY KEY,
+  journal_entry_id    UUID NOT NULL REFERENCES journal_entries(id),
+  account_id          UUID NOT NULL REFERENCES accounts(id),
+  direction           TEXT NOT NULL CHECK (direction IN ('debit','credit')),
+  amount_minor_units  BIGINT NOT NULL CHECK (amount_minor_units > 0),
+  currency            CHAR(3) NOT NULL,
+  created_at          TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX ON ledger_lines (account_id, created_at);
+CREATE INDEX ON ledger_lines (journal_entry_id);
+```
+
+Distinguish **transaction date** (when the economic event occurred — the customer's card was charged at 23:59 Dec 31) from **posting date** (when you booked it — Jan 2, after the batch settled). Financial statements group by posting date; period-end cutoffs use transaction date for accruals. Both are needed.
+
+### Double-Entry Invariants in the Database
+
+The sum-to-zero rule must be enforced by the database, not just by application code. Application bugs happen; a CHECK constraint or deferred trigger does not regress silently.
+
+```sql
+-- Deferred constraint: check at end of transaction, after all lines inserted
+CREATE OR REPLACE FUNCTION assert_journal_balanced()
+RETURNS trigger LANGUAGE plpgsql AS $$
+DECLARE imbalance RECORD;
+BEGIN
+  FOR imbalance IN
+    SELECT currency,
+           SUM(CASE WHEN direction = 'debit'  THEN amount_minor_units ELSE 0 END)
+         - SUM(CASE WHEN direction = 'credit' THEN amount_minor_units ELSE 0 END) AS delta
+    FROM ledger_lines
+    WHERE journal_entry_id = NEW.journal_entry_id
+    GROUP BY currency
+    HAVING SUM(CASE WHEN direction='debit' THEN amount_minor_units ELSE 0 END)
+         <> SUM(CASE WHEN direction='credit' THEN amount_minor_units ELSE 0 END)
+  LOOP
+    RAISE EXCEPTION 'journal entry % unbalanced in %: delta=%',
+      NEW.journal_entry_id, imbalance.currency, imbalance.delta;
+  END LOOP;
+  RETURN NEW;
+END;
+$$;
+
+CREATE CONSTRAINT TRIGGER journal_balance_check
+  AFTER INSERT ON ledger_lines
+  DEFERRABLE INITIALLY DEFERRED
+  FOR EACH ROW EXECUTE FUNCTION assert_journal_balanced();
+```
+
+Also block UPDATE and DELETE on `ledger_lines` and `journal_entries` via triggers — corrections are *reversing* journal entries, not edits. This is the same append-only posture described in `backend-fintech-compliance.md`.
+
+### Idempotent Posting
+
+Every write path into the ledger goes through a single posting function. The function takes a journal-entry spec plus an idempotency key; on conflict it returns the existing journal-entry id without side effects.
+
+```typescript
+type PostingLine = {
+  accountId: string;
+  direction: 'debit' | 'credit';
+  amountMinorUnits: bigint;
+  currency: string;
+};
+
+type PostingRequest = {
+  source: string;             // 'stripe' | 'broker' | ...
+  externalId: string;         // caller-supplied idempotency key
+  transactionDate: string;    // YYYY-MM-DD
+  postingDate: string;
+  counterpartyId?: string;
+  memo?: string;
+  lines: PostingLine[];       // must sum to zero per currency, length >= 2
+  postedBy: string;
+};
+
+// Returns existing journal_entry_id on duplicate (source, externalId) without
+// reinserting lines. Raises on imbalanced entry, unknown account, inactive
+// account, negative amounts, or currency mismatch between line and account.
+async function postJournalEntry(req: PostingRequest): Promise<string>;
+```
+
+Implementation notes: start a transaction, `INSERT ... ON CONFLICT (source, external_id) DO NOTHING RETURNING id` on `journal_entries`; if nothing was returned, `SELECT id` the existing row and short-circuit. Otherwise insert all ledger lines in the same transaction. The deferred constraint trigger fires on commit.
+
+For admin corrections, *require* an idempotency key — it is the common audit-gap in fintech. Admins produce a key like `correction-2026-04-14-ticket-12345` or the UUID of a ticketing-system record.
+
+### Balance Queries and Materialization
+
+Balances are derived. The simplest correct query:
+
+```sql
+-- Balance of an account at a point in time (pseudo-SQL)
+SELECT account_id,
+       currency,
+       COALESCE(SUM(CASE WHEN direction = 'debit'  THEN amount_minor_units ELSE 0 END)
+              - SUM(CASE WHEN direction = 'credit' THEN amount_minor_units ELSE 0 END), 0)
+       AS debit_minus_credit_minor_units
+  FROM ledger_lines
+  WHERE account_id = $1
+    AND created_at <= $2   -- 'as-of' timestamp
+  GROUP BY account_id, currency;
+```
+
+Interpret the sign by the account's `normal_side`: for an asset account the balance is debits minus credits; for a liability the balance is credits minus debits. Always compute in minor units (cents, satoshis) and format for display only — never use floating point. Use `BIGINT` in Postgres, `bigint` / `BigInt` in TS, `decimal.Decimal` in Python, never `double`.
+
+For high-volume accounts (customer wallets), maintain rolling balances via a trigger or via TigerBeetle's built-in running balances. Refresh materialized views nightly for reporting; read-time aggregation is fine up to roughly 10M lines per account.
+
+### Multi-Currency and FX
+
+A multi-currency ledger must either (a) denominate every account in a single currency and use separate FX accounts for conversions, or (b) denominate lines in their native currency and revalue to a reporting currency at close. Mixing currencies within a single account is an anti-pattern.
+
+Every FX conversion is a three-leg journal entry: debit source-currency account, credit source-currency FX clearing, debit target-currency FX clearing, credit target-currency account — with the rate snapshot stored alongside the entry. Snapshot the rate at booking (use a feed like OpenExchangeRates, ECB, or Fixer with a timestamp); do not re-read it at settlement, or you introduce drift. At period end, revalue open foreign-currency balances to the reporting currency using the closing rate and book the realized/unrealized FX gain or loss as a separate journal entry.
+
+Precision: never `double`. Minor units (ints) for fiat. For crypto, use a fixed-scale decimal library (`decimal.js`, Go `shopspring/decimal`, Python `decimal.Decimal`) and store the scale explicitly; satoshi-level BIGINTs work for Bitcoin but not for many Ethereum tokens.
+
+### Reconciliation Patterns
+
+Reconciliation matches internal ledger entries against external sources of truth: bank statements (BAI2, MT940, CAMT.053), card processor settlement files (Visa Base II, Stripe balance transactions), broker clearing files (DTCC CNS). For each counterparty, on each business day, every external line must match an internal journal entry (or vice versa).
+
+Event sourcing from counterparty feeds works best: ingest the feed into a `counterparty_events` table, then run a match job that joins against journal entries via a natural key (broker execution id, card network auth code, ACH trace number). Unmatched rows on either side go to a **breaks queue**.
+
+```sql
+-- Unmatched broker executions (external) with no matching journal entry (internal)
+SELECT bx.execution_id,
+       bx.trade_date,
+       bx.symbol,
+       bx.amount_minor_units,
+       bx.received_at
+  FROM broker_executions bx
+  LEFT JOIN journal_entries je
+    ON je.source = 'broker'
+   AND je.external_id = bx.execution_id
+  WHERE je.id IS NULL
+    AND bx.trade_date >= current_date - INTERVAL '7 days'
+  ORDER BY bx.trade_date, bx.received_at;
+```
+
+Break items age with SLAs: T+1 auto-retry, T+3 operator review, T+7 escalation. Auto-matching should use counterparty id plus amount plus date within a small tolerance; never match on amount alone. Manual-review UI exposes side-by-side diffs and lets an operator post a reversing or adjusting entry — with mandatory idempotency key and reason code.
+
+### Period-Close
+
+At month-end, quarter-end, and year-end, the books are **closed**: a cutoff timestamp is recorded, no journal entries with a posting date on or before the cutoff may be inserted, and an opening balance sheet is materialized for the next period. Late-arriving events (a broker settlement that arrives 3 days after trade date) post to the *next* period with the prior period's transaction date — so accruals remain correct but the closed period's reported balances do not move.
+
+Implementation: a `periods` table with `status IN ('open','closing','closed')`; a trigger on `journal_entries` that rejects inserts into closed periods; a nightly job that materializes the opening balance sheet into an `account_period_balances` table. Auditors will ask for the close runbook; keep it in-repo alongside the code.
+
+### Performance, Partitioning, Archiving
+
+Journal and ledger-line tables grow forever — a mid-size consumer fintech posts tens of millions of lines per year. Strategies:
+
+- **Partition by posting month** (native Postgres declarative partitioning, or `pg_partman`). Old partitions become read-mostly and can be moved to cheaper storage.
+- **Archive closed periods** to columnar storage (ClickHouse, Snowflake, S3+Parquet) for analytics; keep a queryable summary in the primary DB.
+- **Index carefully:** `(account_id, created_at)` is the hot path for balance queries; `(journal_entry_id)` for entry lookup; `(source, external_id)` unique index for idempotency. Do not over-index — every index costs on every insert and the write path is hot.
+- **Purpose-built ledger DBs:** TigerBeetle is designed specifically for double-entry workloads with built-in idempotency and running balances; consider it for greenfield high-volume systems.
+
+### Common Pitfalls
+
+- **Negative balances silently accepted.** A wallet debit that overdraws should either be rejected at the posting function or post to an overdraft-receivable account — never silently go negative in a customer wallet. Enforce with a CHECK on the rolling balance trigger or pre-flight check in the posting function.
+- **FX rate drift between booking and settlement.** If you read the FX rate at posting time and again at settlement, small differences accumulate. Snapshot once at booking and book any settlement-day delta as a realized FX gain/loss entry.
+- **Double-posting from webhook retries.** Stripe and similar providers retry webhooks aggressively. Without a unique index on `(source, external_id)`, the second delivery books a duplicate journal entry. Always key on the provider event id, never on your own generated UUID at receive time.
+- **Missing idempotency keys on manual admin corrections.** A support engineer clicks "post adjustment" twice and double-credits the customer. The admin UI must require and persist an idempotency key on every posting action.
+- **`current_balance` columns in application tables.** These drift, lie about history, and cannot be audited. The ledger is the source; balances are derived. If a dashboard needs a fast balance read, materialize it from the ledger, not alongside it.
+- **Using floats anywhere in the money path.** JavaScript `Number`, Python `float`, Java `double` — all unsafe. Minor-unit integers or fixed-scale decimals only, end to end, including JSON wire formats (send as strings).
+- **Mixing posting-date and transaction-date reports.** A balance-sheet-as-of report uses posting date; an accrual-based P&L uses transaction date. Label every report and use the right column.
+- **Correcting entries by UPDATE.** Never. A correction is a *reversing* journal entry plus a corrected entry, both with idempotency keys and memos linking to the original.
+
+See also `backend-fintech-data-modeling.md` for the broader schema-design patterns, `backend-fintech-compliance.md` for audit-trail and retention requirements, and `backend-fintech-order-lifecycle.md` for trade-settlement flows that terminate in ledger postings.

package/content/knowledge/backend/backend-fintech-observability.md

@@ -0,0 +1,151 @@
+---
+name: backend-fintech-observability
+description: Trade event correlation; market-hours aware scheduling; SLOs for fintech systems; compliance logging; alerting strategy.
+topics: [backend, fintech, observability, tracing, slos, alerting, correlation-id, market-hours]
+---
+
+Observability for a trading system is not generic APM with a finance skin — it is the ability to reconstruct any single order, end to end, across six or more services, on demand, years later, with timezone-correct timestamps and a stable correlation identifier. It is also the early-warning system that catches a sudden drop in fill rate at 09:31 ET before the desk calls. Done well it overlaps with — but does not replace — the immutable audit trail (`backend-fintech-compliance.md`).
+
+## Summary
+
+Every trade flow is distributed. A single order crosses the wizard (UI), order-management service, risk-check service, broker adapter, fill-processing worker, ledger, and balance projection — typically six to ten hops across HTTP and message queues. A correlation ID must be minted at the edge, propagated across every hop (HTTP header, MQ header, database row, log line, span attribute), and indexed in both logs and tracing. Without this, a "my order is stuck" ticket becomes an archaeology project.
+
+SLOs for fintech are tighter than typical SaaS and are defined per flow, not per service. Order-submission (client click → broker ack) runs in the low hundreds of milliseconds at p99. Fill-processing (broker fill event → ledger write) targets single-digit seconds. Ledger-to-balance propagation (fill persisted → UI-visible balance update) targets under a minute — users refresh aggressively after a trade. Error budgets are monthly, tracked per flow, and breached budgets freeze risky deploys.
+
+Market-hours-aware alerting is non-negotiable. A stale-price alert at 03:00 ET on a US equity feed is noise; at 09:35 ET it is a P1. A prolonged broker outage at any hour is a P1 — after-hours orders still queue, and overnight index futures trade nearly 24 hours. The alerting layer must consult a trading-calendar service (per venue, per asset class, with half-days and holidays) and route accordingly. Maintenance windows are encoded as first-class suppression rules, not tribal knowledge.
+
+Regulatory logging is additive to operational observability. The audit trail (WORM storage, 7-year retention for most US regimes) is written in parallel with — never replaced by — ops logs. Ops logs rotate at 30–90 days. Conflating the two produces either cripplingly expensive storage or a compliance breach; keep the pipelines separate with different retention classes.
+
+Alert on anomalies, not just errors. A zero-error service that suddenly stops filling orders is worse than one throwing 500s. Track fill-rate baselines (by symbol, venue, time-of-day bucket), risk-check reject rates, P&L velocity, and queue depths; alert on deviation from baseline, not just on thresholds. Cross-ref: `backend-fintech-order-lifecycle.md` for the flow being instrumented, `backend-fintech-risk-management.md` for risk-specific signals, `backend-fintech-broker-integration.md` for per-broker health, `backend-fintech-compliance.md` for the audit-log boundary.
+
+## Deep Guidance
+
+### Correlation IDs Across Every Hop
+
+Use a two-tier identifier scheme. The outer tier is the `client_order_id` — minted in the UI (or API client), echoed on every user-facing artifact (order ticket, confirmation email, support ticket), and stable for the lifetime of the order. The inner tier is a W3C Trace Context `trace-id` per service interaction, standard across OpenTelemetry instrumentations. The `client_order_id` is the noun a human searches for ("why is order ABC-123 stuck?"); the `trace-id` is the edge-traversal graph a tool renders.
+
+Propagation rules are absolute. HTTP boundaries carry `traceparent` and `tracestate` per W3C; add an `X-Client-Order-Id` header for the outer ID. Message queues (SQS, Kafka, RabbitMQ) put both in message attributes/headers — never only in the payload, because DLQ tools and retry wrappers often strip payload context. Database rows for orders, fills, and ledger entries store the `client_order_id` column indexed. Every structured log line and every span carries both. If a queue hop drops the correlation ID, that is a P2 bug to be fixed, not tolerated.
+
+### Structured Logging Schema
+
+Logs are JSON, single-line, with a fixed minimum set of fields plus event-type-specific extensions. Emit through a shared library so the schema is enforced, not hoped for.
+
+```json
+{
+  "timestamp": "2026-04-15T13:31:04.128Z",
+  "timestamp_local": "2026-04-15T09:31:04.128-04:00",
+  "level": "info",
+  "service": "order-management",
+  "event_type": "order.submitted",
+  "client_order_id": "ABC-123",
+  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
+  "span_id": "00f067aa0ba902b7",
+  "account_id": "acct_9f3c",
+  "symbol": "AAPL",
+  "side": "buy",
+  "quantity": 100,
+  "limit_price": { "amount": "182.50", "currency": "USD" },
+  "broker_id": "alpaca",
+  "venue": "NASDAQ",
+  "latency_ms": 42
+}
+```
+
+Monetary amounts are objects with `amount` (string decimal, never float) and `currency`. Timestamps are ISO-8601 with explicit UTC *and* the exchange-local time for operator sanity — "09:31 ET" is how a trader thinks, "13:31 UTC" is how storage indexes. `event_type` uses dotted nouns (`order.submitted`, `fill.received`, `risk.blocked`, `ledger.posted`) so alerting and dashboards can route on prefix.
+
+### SLOs Per Flow
+
+Define SLOs on user-visible flows, not on internal microservice latency. Typical targets:
+
+| Flow | Metric | Target | Window |
+|------|--------|--------|--------|
+| Order submission (click → broker ack) | p99 latency | 400 ms | 30 days |
+| Fill processing (broker fill → ledger) | p99 latency | 5 s | 30 days |
+| Ledger-to-balance propagation | p95 lag | 45 s | 30 days |
+| Balance query (API read) | p99 latency | 200 ms | 30 days |
+| Order-submission availability | success rate | 99.9% | 30 days during market hours |
+
+Error budgets are consumed only against market-hours traffic; off-hours degradation is tracked separately. When a budget is below 25% remaining, ship freeze applies to risk-touching services until the window resets or a postmortem authorizes deploy.
+
+```yaml
+# slo/order-submission.yaml
+slo:
+  name: order-submission-latency
+  service: order-management
+  flow: order.submitted -> order.broker_ack
+  objective: 99.0
+  indicator:
+    type: latency
+    percentile: 99
+    threshold_ms: 400
+  window:
+    duration: 30d
+    market_hours_only: true
+    calendar: us-equities
+  alerting:
+    burn_rate:
+      - window: 1h
+        factor: 14.4   # fast burn
+        severity: page
+      - window: 6h
+        factor: 6
+        severity: ticket
+```
+
+### Market-Hours-Aware Alerting
+
+A trading-calendar service (backed by `pandas-market-calendars`, `iex-cloud`, or Polygon reference data) exposes `is_market_open(venue, timestamp)` and `next_open/close(venue)`. Alert routing rules consult it. Stale-price alerts are suppressed outside regular session; broker-outage alerts escalate regardless; ledger-lag alerts during after-hours page only if open positions exist. Encode the routing in the alert platform (PagerDuty event rules, Grafana OnCall) rather than in per-alert bespoke logic, so maintenance is tractable.
+
+Half-days (1 pm ET close around Thanksgiving and July 3rd) and early-close holidays are where this fails in practice — test the calendar service against a full year of real sessions, not a regex on weekdays.
+
+### Trade Anomaly Detection
+
+Error rates alone miss the most dangerous incidents. Track baselines and deviation. Fill rate per minute per symbol, compared against a trailing 20-session median for the same minute-bucket, alerting on >3σ deviation. Risk-check reject rate per account, alerting on sudden jumps (often a config deploy gone wrong). P&L velocity (dollars-per-minute realized loss) with tiered alerts — warn, page, and auto-trigger the kill switch at escalating thresholds. Queue depth on the fill-processing worker, alerting when backlog grows faster than drain rate.
+
+Honeycomb BubbleUp or Datadog Watchdog handle the baseline math adequately for most teams; rolling homegrown detectors is rarely worth it until scale forces it.
+
+### Multi-Broker Observability
+
+Every broker integration (`backend-fintech-broker-integration.md`) exports the same metric taxonomy — `broker.submit.latency`, `broker.submit.errors`, `broker.fill.lag`, `broker.reconnect.count` — tagged with `broker_id`. Dashboards show aggregate health on top (all brokers combined) with per-broker drill-downs below. When one broker degrades, failover logic (if implemented) must emit its own events (`broker.failover.triggered`) with both old and new broker IDs and the failover reason.
+
+### Distributed Tracing with OpenTelemetry
+
+OpenTelemetry SDKs in every service, exporter to a backend (Honeycomb, Datadog APM, Grafana Tempo, or a Grafana LGTM stack). Span attributes carry financial context — `trading.symbol`, `trading.side`, `trading.quantity`, `trading.account_id`, `trading.broker_id`, `trading.client_order_id`. Sample tail-based: keep 100% of traces that contain an error or a latency outlier, throttle successes to 5–10%. Head-based sampling (decide at root) loses the ability to keep all errors and is the wrong default for fintech.
+
+```typescript
+// correlation-middleware.ts  (Express + OpenTelemetry)
+import { trace, context } from '@opentelemetry/api';
+import { randomUUID } from 'node:crypto';
+import type { Request, Response, NextFunction } from 'express';
+
+export function correlationMiddleware(req: Request, res: Response, next: NextFunction) {
+  const clientOrderId = req.header('x-client-order-id') ?? `srv-${randomUUID()}`;
+  const span = trace.getActiveSpan();
+  span?.setAttribute('trading.client_order_id', clientOrderId);
+
+  res.setHeader('x-client-order-id', clientOrderId);
+  (req as any).clientOrderId = clientOrderId;
+
+  // Attach to async context so downstream logger picks it up.
+  const ctx = context.active().setValue(Symbol.for('client_order_id'), clientOrderId);
+  context.with(ctx, () => next());
+}
+```
+
+### Log Retention vs Audit Retention
+
+Two pipelines, two retention classes. Ops logs ship to Loki or a hot Elasticsearch tier with 30–90 day retention, indexed for high-cardinality ad-hoc querying. Audit events ship to append-only WORM storage (S3 Object Lock, Glacier with compliance-mode retention, or a dedicated vendor like DataBP) for the regulatory retention period — typically 7 years for SEC 17a-4 style records, sometimes longer. The two pipelines share schema where possible but diverge in transport, storage, and access controls. Never satisfy audit requirements by pointing at your Datadog log archive — it is not WORM unless explicitly configured, and even then access controls are wrong.
+
+### Common Pitfalls
+
+Correlation IDs dropped at queue hops because a retry wrapper reconstructed the message from payload only. Fix: make propagation a library concern, test explicitly.
+
+Timezones not logged. A 14:30 timestamp is ambiguous; a 14:30Z plus 09:30-05:00 is not. Log both.
+
+Alerts firing during scheduled broker maintenance windows because the suppression was a shared Google Doc. Fix: encode maintenance in the alerting platform with a defined owner and a review cadence.
+
+No dashboard for the first-minute-of-open latency spike. The open is where queues are fullest, baselines are least representative, and incidents cluster. A dedicated "09:30–09:35 ET" dashboard saves incidents.
+
+Tracing without financial attributes. Spans that show HTTP method and path but not symbol, side, or account are useless at 2 am. Every span on a trade path must carry the trading tuple.
+
+Sampling that throws away errors. Head-based 1% sampling loses 99% of failure traces. Use tail-based sampling or keep 100% of errors regardless of sample rate.