shadowjoin 0.0.0-stage → 0.3.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Harsh N. Chandak
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,325 @@
-# Temporary Holding Version
+# shadowjoin
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+**Measure the matches your exact join is silently dropping.**
+
+Your pipeline joins two sources on a name, an email, a SKU, a company. It matches
+what matches. Everything else just... isn't there. No error, no exception, no
+alert — the query returns, the dashboard renders, and a quarter of the rows that
+should have matched quietly don't.
+
+shadowjoin runs a shadow matcher alongside your real join key and tells you how
+many matches you are losing, which normalization rule would recover each one, and
+— if you have any ground truth at all — proves the recoveries are real.
+
+```
+npx shadowjoin scan \
+  --left  customers.csv   --left-key  company_name \
+  --right crm_export.csv  --right-key account_name
+```
+
+[![shadowjoin measuring a country-name join: 205 of 249 matched exactly, which looks fine, while five real matches were being dropped including Vietnam, spelled Viet Nam by the UN](assets/brag.gif)](https://github.com/harsh-chandak/shadowjoin/blob/main/assets/brag.mp4)
+
+<sub>Fourteen seconds, no sound. Every figure in it is verbatim CLI output from
+the committed example below — [higher-resolution
+version](https://github.com/harsh-chandak/shadowjoin/blob/main/assets/brag.mp4).</sub>
+
+The progress output above is real: shadowjoin shows what it is doing while it
+works. All of it is suppressed automatically when stdout is not a terminal, so
+piping to a file, to `jq`, or running in CI produces byte-identical output to
+before — and there is **no postinstall script**, because a tool you point at a
+production database has no business running code at install time.
+
+## What it found on public data
+
+Joining the **S&P 500 constituent list** against **NYSE + Nasdaq listings** on
+company name. Both files carry ticker symbols, so every recovered match can be
+checked rather than believed:
+
+```
+  left rows 503   right rows 6,709
+
+  exact join matches                1   0.2%
+  recovered by shadow join        428   85.09%
+  still unexplained                74
+
+  LOWER BOUND ON LOST MATCHES: 85.09% of left rows
+  verified against ground truth: 419 correct, 1 wrong, 8 ambiguous  (precision 99.76%)
+
+  rules worth adopting, in order (greedy by marginal recovery):
+  1.  strip security/listing boilerplate (common stock, ADS, ...)   +68       68
+  2.  strip corporate suffixes (inc, corp, ltd, plc, gmbh ...)     +319      387
+  3.  strip punctuation                                             +33      420
+  4.  sort tokens (order-insensitive match)                          +4      424
+  5.  remove all whitespace                                          +1      425
+```
+
+The exact join found **one** match. There were **four hundred and twenty-eight**,
+and 419 of them are confirmed correct by ticker.
+
+Reproduce it yourself — the data ships with the repo:
+
+```bash
+git clone https://github.com/harsh-chandak/shadowjoin && cd shadowjoin
+npm install && npm run build
+node dist/cli.js scan \
+  --left  examples/sp500.csv    --left-key  Security --left-id Symbol --left-truth Symbol \
+  --right examples/listings.json --right-key name    --right-id symbol --right-truth symbol
+```
+
+### A join you would actually ship
+
+The example above is deliberately unflattering — a short index name against a
+full security description. Here is the realistic case.
+
+**Unicode CLDR country names** (what your UI displays) joined to **UN official
+country names** (what your reference data uses), on the country name, with ISO
+3166 alpha-3 as ground truth. That is a join people write every week.
+
+```
+  exact join matches              205 / 249   82.33%
+  recovered by shadow join          5          2.01%   precision 100%
+  still unexplained                39
+
+  rules worth adopting:
+  1.  & -> and                 +4
+  2.  remove all whitespace    +1
+```
+
+It looks healthy. 82% matched, no errors, nobody investigating. And it is
+silently dropping `Antigua & Barbuda`, `Trinidad & Tobago`,
+`São Tomé & Príncipe`, `Turks & Caicos Islands` and `Vietnam` — the last one
+because the UN spells it `Viet Nam`.
+
+Two things this example shows that the first one does not:
+
+**Applying every rule finds nothing.** The full rule set recovers zero here; the
+two-rule subset recovers five. Over-normalization collapses keys that should stay
+distinct, and shadowjoin names the better subset instead of letting you assume
+more rules is better.
+
+**It separates mechanical misses from semantic ones.** The 39 it cannot explain
+are not normalization problems at all:
+
+```
+  Vatican City    vs  Holy See
+  North Korea     vs  Democratic People's Republic of Korea
+  Cape Verde      vs  Cabo Verde
+  Bosnia          vs  Bosnia and Herzegovina
+  Hong Kong       vs  China, Hong Kong Special Administrative Region
+```
+
+No string transform recovers those. They need an alias table or a real entity
+resolution library. Knowing *which* of your misses are mechanical and which are
+semantic is the difference between a one-line fix and a project.
+
+```bash
+node dist/cli.js scan \
+  --left  examples/countries_ui.csv        --left-key  country --left-truth  iso3 \
+  --right examples/countries_reference.csv --right-key country --right-truth iso3
+```
+
+> That particular join is deliberately unflattering — a short index name against a
+> full security description. It is a worked example of the *measurement*, not a
+> claim that anyone ships that join. Point it at your own data to get a figure
+> that means something about you.
+
+## The number is a floor, not an estimate
+
+You cannot compute recall without labels. The entity-resolution literature calls
+the metric **pair completeness**, and it assumes an oracle you do not have in
+production.
+
+So shadowjoin reports a **lower bound**: matches the shadow matcher found that
+exact matching missed. The shadow matcher has unknown recall of its own, so the
+true loss is *at least* this large and may be larger. Every figure ships with
+sample pairs so a human can verify them by hand, and when ground truth is
+available on both sides, the report carries a precision figure too.
+
+This is deliberately the weaker claim. It is also the one that survives being
+questioned.
+
+## Why not just ask an LLM to check?
+
+For a one-off look at a small table, do that — it is faster and this tool is
+redundant. It stops working when:
+
+- **You need the same number twice.** An agent writes different SQL every run, so
+  you cannot tell whether the pipeline changed or the query did.
+- **You need rules nobody thought of.** Asked "is this join lossy," an agent tries
+  `lower()` and `trim()`. It does not systematically try NFKD folding, diacritics,
+  leading zeros, corporate suffixes, token reordering or listing boilerplate. The
+  exhaustive rule set *is* the product.
+- **You need it in CI.** `--max-loss` exits non-zero. "Ask Claude" has no exit code.
+- **Exploratory SQL on production is expensive.** This does hash joins on
+  precomputed keys, under a statement timeout, in a read-only transaction.
+
+## Install
+
+```bash
+npm install shadowjoin      # or run it with npx, no install
+```
+
+Zero runtime dependencies. `pg` and `@modelcontextprotocol/sdk` are optional
+peers, pulled in only if you use the Postgres adapter or the MCP server.
+
+## Use it
+
+### CLI
+
+```bash
+# CSV or JSON, no database involved
+npx shadowjoin scan --left a.csv --left-key name --right b.json --right-key title
+
+# with ground truth on both sides, so recoveries are verified
+npx shadowjoin scan \
+  --left  a.csv --left-key  name  --left-truth  customer_id \
+  --right b.csv --right-key title --right-truth customer_id
+
+# straight at Postgres
+npx shadowjoin scan \
+  --left  'postgres://ro@host/db' --left-table  orders    --left-key  customer_name \
+  --right 'postgres://ro@host/db' --right-table customers --right-key name
+
+# in CI: fail the build if the floor climbs above 2%
+npx shadowjoin scan --left a.csv --left-key name --right b.csv --right-key name --max-loss 2
+
+# MySQL and SQLite work the same way
+npx shadowjoin scan \
+  --left  'mysql://ro@host/db' --left-table  accounts --left-key  company_name \
+  --right ./reference.sqlite   --right-table vendors  --right-key name
+
+npx shadowjoin rules     # list the normalization rules
+```
+
+**Sources:** CSV, JSON, Postgres, MySQL and SQLite. SQLite uses Node's built-in
+`node:sqlite` (Node 22.5+), so it needs nothing installed; Postgres and MySQL
+need `pg` or `mysql2`, which are optional peers.
+
+**Anything else** works through the library in three lines, because the core
+takes two arrays rather than a connection:
+
+```js
+import { scan } from "shadowjoin";
+const [rows] = await anyDriver.query("SELECT id, company_name FROM accounts");
+const left = rows.map((r) => ({ id: String(r.id), key: r.company_name }));
+scan(left, right);
+```
+
+### Library
+
+```js
+import { scan, normalize, rulesByIds, toSql } from "shadowjoin";
+
+const report = scan(leftRows, rightRows);        // rows are { id, key, truth? }
+console.log(report.lowerBoundLossPct, report.stack);
+
+// adopt the recommended rules in your own pipeline
+const rules = rulesByIds(report.stack.map((s) => s.rule));
+const key = (value) => normalize(value, rules);
+toSql("company_name", rules);                     // the same thing as Postgres SQL
+```
+
+### MCP server
+
+Lets an agent measure a join, look at real examples, propose a fix, and then
+**prove the fix worked** — a loop it cannot fake, because the measurement is
+not something it writes.
+
+```json
+{
+  "mcpServers": {
+    "shadowjoin": {
+      "command": "npx",
+      "args": ["-y", "-p", "shadowjoin", "shadowjoin-mcp"],
+      "env": { "SHADOWJOIN_CONFIG": "/abs/path/to/shadowjoin.config.json" }
+    }
+  }
+}
+```
+
+```json
+{
+  "sources": {
+    "prod":  { "type": "postgres", "connectionString": "postgres://shadowjoin_ro@host/db" },
+    "crm":   { "type": "csv",      "path": "./crm_export.csv" }
+  }
+}
+```
+
+Tools: `scan_join`, `explain_misses`, `suggest_normalization`, `compare_runs`.
+
+**Two things this MCP server deliberately does not have:**
+
+1. **No tool that executes arbitrary SQL.** An agent can name a configured source,
+   a table and a column. That is the entire surface.
+2. **No tool that accepts a connection string.** Credentials live in your config
+   file and never enter a tool call, because tool calls end up in transcripts.
+
+## Pointing it at a real database
+
+shadowjoin opens a `READ ONLY` transaction with `statement_timeout`,
+`lock_timeout` and `idle_in_transaction_session_timeout` set, and it **refuses to
+run** if the role it connected with can write to the table it is about to read.
+
+Give it a role that makes trust unnecessary:
+
+```sql
+CREATE ROLE shadowjoin_ro LOGIN PASSWORD 'xxx';
+GRANT CONNECT ON DATABASE yourdb TO shadowjoin_ro;
+GRANT USAGE  ON SCHEMA public    TO shadowjoin_ro;
+GRANT SELECT ON public.orders, public.customers TO shadowjoin_ro;
+```
+
+Point it at a replica where you can.
+
+**It makes no outbound network connection except to the source you give it.** No
+telemetry, not even opt-out. Check the imports — there are three dependencies and
+none of them is an HTTP client.
+
+## Limits
+
+Read these before quoting a number from it.
+
+- **The figure is a floor.** The shadow matcher's own recall is unknown. True loss
+  is at least this, possibly more.
+- **Recoveries can be wrong.** In the bundled example, 1 of 428 was contradicted
+  by ground truth and 8 were ambiguous — mostly share classes, where stripping
+  "Class A" merges genuinely different securities. Ambiguous matches need a
+  tie-breaker, not a normalization rule. Always read `samplePairs`.
+- **Per-rule counts overlap and will not sum to the total.** Each is measured
+  independently against the exact baseline, deliberately, so the numbers are not
+  an artifact of the order rules happen to run in. Use `stack` — the greedy
+  ordering — to decide what to actually adopt.
+- **More rules is not better.** Applying every rule can recover *fewer* matches
+  than the best subset, because over-normalization merges keys that should stay
+  apart. shadowjoin warns when this happens and names the better subset.
+- **It reads your join keys**, so those values are in memory while it runs.
+  `--redact` hashes them in sample output and server-side normalization keeps
+  whole rows in the database, but neither makes the keys invisible. For regulated
+  data, run it on a replica inside your own environment.
+- **It does not write, fix or migrate anything.** It measures.
+- **`unaccent` in emitted SQL** needs the Postgres extension of the same name.
+
+## Prior art
+
+Blocking recall — *pair completeness* — is well established in the
+entity-resolution literature, along with its companions pairs quality and
+reduction ratio:
+
+- [A Survey of Blocking and Filtering Techniques for Entity Resolution](https://arxiv.org/pdf/1905.06167)
+- [Comparative Analysis of Approximate Blocking Techniques for Entity Resolution](http://www.vldb.org/pvldb/vol9/p684-papadakis.pdf)
+- [Supervised Meta-blocking](http://www.vldb.org/pvldb/vol7/p1929-papadakis.pdf)
+
+Those papers assume a labelled gold standard. shadowjoin's contribution is not the
+metric — it is computing a defensible version of it against a live pipeline with
+no labels at all, and reporting it as a bound rather than a point estimate.
+
+For *doing* the matching rather than measuring what yours misses, use
+[Splink](https://github.com/moj-analytical-services/splink),
+[Dedupe](https://github.com/dedupeio/dedupe) or
+[recordlinkage](https://github.com/J535D165/recordlinkage). shadowjoin is
+complementary: it tells you whether you need them.
+
+## License
+
+MIT © Harsh N. Chandak

package/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: shadowjoin
+description: Measure how many matches a join or matching pipeline is silently dropping, and fix it. Use when records "aren't matching", duplicates slipped through, a join returns fewer rows than expected, a dedupe step looks wrong, or before and after editing any matching, joining, dedupe or entity-resolution logic. Requires the shadowjoin MCP server.
+---
+
+# shadowjoin
+
+A join that drops rows raises no error. The query returns, the dashboard renders,
+and the missing matches are invisible. This skill measures that loss and proves
+whether a fix worked.
+
+## When to use it
+
+- The user says records aren't matching, rows are missing from a join, duplicates
+  got through, or a dedupe/merge produced the wrong count.
+- **Before** you change any matching, joining, normalization or dedupe logic —
+  take a baseline first.
+- **After** you change it — you cannot otherwise tell whether your edit helped.
+
+## The loop
+
+1. **`scan_join`** — baseline. Note the `runId`, the `exactMatchRate`, and the
+   `lowerBoundLossPct`.
+2. **`explain_misses`** — pull real pairs. Show the user actual values, not a
+   percentage. A reviewer believes `"Altria" -> "Altria Group Inc."`; nobody
+   believes "85%".
+3. **`suggest_normalization`** — get the ranked rules with pasteable SQL and JS.
+   Apply them to the user's pipeline.
+4. **`scan_join`** again, then **`compare_runs`** — prove the change landed.
+
+## Rules for reporting this
+
+**The number is a floor, never an estimate.** Say "at least 23% of matches are
+being lost", never "23% of matches are lost". The shadow matcher's own recall is
+unknown, so the true figure can only be larger.
+
+**Never claim an improvement without `compare_runs`.** If you changed matching
+logic and `exactMatchRate` did not move, your change did not take effect — say
+that plainly rather than reporting success.
+
+**Always surface contradicted and ambiguous pairs.** `contradicted` means the
+matcher merged two records that ground truth says are different. `ambiguous`
+means several candidates collapsed onto one key — typically share classes,
+product variants or franchise locations. Those need a tie-breaker, not another
+normalization rule. Never recommend adopting a rule without showing these counts.
+
+**More rules is not better.** If the report warns that the full rule set recovers
+fewer matches than a subset, recommend the subset it names. Over-normalization
+merges records that should stay apart.
+
+**Adopting a rule widens matching.** Say so. Recovering 400 matches at the cost
+of 12 wrong merges may be a bad trade in a billing system and a fine one in
+search. That judgement belongs to the user — give them both numbers.
+
+## Privacy
+
+`explain_misses` returns real key values. If the data looks sensitive — people's
+names, emails, account numbers — tell the user before dumping examples into the
+conversation, and prefer summarising the *rule* and the count over pasting rows.
+
+## What it will not do
+
+It does not write, fix, migrate or dedupe anything. It measures, and it hands you
+expressions to apply yourself. If the user asks it to "fix the matching", the fix
+is you editing their pipeline with `suggest_normalization`'s output and then
+proving it with `compare_runs`.

package/dist/adapters/csv.d.ts

@@ -0,0 +1,13 @@
+import type { Row } from "../types.js";
+/** Minimal RFC-4180 parser: quoted fields, embedded commas, doubled quotes. */
+export declare function parseCsv(text: string): string[][];
+export interface CsvOptions {
+    /** Column holding the join key. */
+    keyColumn: string;
+    /** Column holding a stable id. Defaults to the row number. */
+    idColumn?: string;
+    /** Column holding ground truth, if you have one. */
+    truthColumn?: string;
+}
+export declare function rowsFromCsvText(text: string, options: CsvOptions): Row[];
+export declare function rowsFromCsv(path: string, options: CsvOptions): Promise<Row[]>;

package/dist/adapters/csv.js

@@ -0,0 +1,73 @@
+import { readFile } from "node:fs/promises";
+/** Minimal RFC-4180 parser: quoted fields, embedded commas, doubled quotes. */
+export function parseCsv(text) {
+    const rows = [];
+    let field = "", row = [], quoted = false;
+    const src = text.replace(/^/, "");
+    for (let i = 0; i < src.length; i++) {
+        const c = src[i];
+        if (quoted) {
+            if (c === '"') {
+                if (src[i + 1] === '"') {
+                    field += '"';
+                    i++;
+                }
+                else
+                    quoted = false;
+            }
+            else
+                field += c;
+            continue;
+        }
+        if (c === '"')
+            quoted = true;
+        else if (c === ",") {
+            row.push(field);
+            field = "";
+        }
+        else if (c === "\n") {
+            row.push(field);
+            rows.push(row);
+            row = [];
+            field = "";
+        }
+        else if (c !== "\r")
+            field += c;
+    }
+    if (field.length > 0 || row.length > 0) {
+        row.push(field);
+        rows.push(row);
+    }
+    return rows.filter((r) => r.length > 1 || (r[0] ?? "").trim() !== "");
+}
+export function rowsFromCsvText(text, options) {
+    const table = parseCsv(text);
+    if (table.length === 0)
+        return [];
+    const header = table[0].map((h) => h.trim());
+    const at = (name) => {
+        const i = header.findIndex((h) => h.toLowerCase() === name.toLowerCase());
+        if (i === -1)
+            throw new Error(`Column "${name}" not found. Columns: ${header.join(", ")}`);
+        return i;
+    };
+    const k = at(options.keyColumn);
+    const idx = options.idColumn ? at(options.idColumn) : -1;
+    const t = options.truthColumn ? at(options.truthColumn) : -1;
+    const rows = [];
+    for (let i = 1; i < table.length; i++) {
+        const rec = table[i];
+        const key = (rec[k] ?? "").trim();
+        if (!key)
+            continue;
+        rows.push({
+            id: idx === -1 ? String(i) : (rec[idx] ?? String(i)).trim(),
+            key,
+            ...(t === -1 ? {} : { truth: (rec[t] ?? "").trim() || undefined }),
+        });
+    }
+    return rows;
+}
+export async function rowsFromCsv(path, options) {
+    return rowsFromCsvText(await readFile(path, "utf8"), options);
+}

package/dist/adapters/json.d.ts

@@ -0,0 +1,4 @@
+import type { Row } from "../types.js";
+import type { CsvOptions } from "./csv.js";
+export declare function rowsFromJsonText(text: string, options: CsvOptions): Row[];
+export declare function rowsFromJson(path: string, options: CsvOptions): Promise<Row[]>;

package/dist/adapters/json.js

@@ -0,0 +1,25 @@
+import { readFile } from "node:fs/promises";
+export function rowsFromJsonText(text, options) {
+    const parsed = JSON.parse(text);
+    if (!Array.isArray(parsed))
+        throw new Error("Expected the JSON file to contain an array of objects.");
+    const rows = [];
+    parsed.forEach((rec, i) => {
+        if (typeof rec !== "object" || rec === null)
+            return;
+        const obj = rec;
+        const key = String(obj[options.keyColumn] ?? "").trim();
+        if (!key)
+            return;
+        const truth = options.truthColumn ? String(obj[options.truthColumn] ?? "").trim() : "";
+        rows.push({
+            id: options.idColumn ? String(obj[options.idColumn] ?? i) : String(i),
+            key,
+            ...(truth ? { truth } : {}),
+        });
+    });
+    return rows;
+}
+export async function rowsFromJson(path, options) {
+    return rowsFromJsonText(await readFile(path, "utf8"), options);
+}

package/dist/adapters/mysql.d.ts

@@ -0,0 +1,21 @@
+import type { Row } from "../types.js";
+export interface MysqlSource {
+    /** mysql:// connection string. Prefer a replica and a read-only user. */
+    connectionString: string;
+    table: string;
+    keyColumn: string;
+    idColumn?: string;
+    truthColumn?: string;
+    limit?: number;
+    /** Per-statement cap in ms. Default 30000. */
+    statementTimeoutMs?: number;
+    /** shadowjoin refuses a user that can write to the table it is about to read. */
+    allowWritableRole?: boolean;
+}
+/**
+ * Read join keys out of MySQL inside a READ ONLY transaction with a statement
+ * timeout, and refuse a user that holds write privileges on the target.
+ *
+ * Requires the optional peer dependency `mysql2`.
+ */
+export declare function rowsFromMysql(source: MysqlSource): Promise<Row[]>;

package/dist/adapters/mysql.js

@@ -0,0 +1,72 @@
+const IDENT = /^[A-Za-z_][A-Za-z0-9_$]*$/;
+function quoteIdent(name) {
+    const parts = name.split(".");
+    for (const part of parts) {
+        if (!IDENT.test(part)) {
+            throw new Error(`Refusing identifier "${name}". Use a plain table or column name; shadowjoin does not accept arbitrary SQL.`);
+        }
+    }
+    return parts.map((p) => `\`${p}\``).join(".");
+}
+/**
+ * Read join keys out of MySQL inside a READ ONLY transaction with a statement
+ * timeout, and refuse a user that holds write privileges on the target.
+ *
+ * Requires the optional peer dependency `mysql2`.
+ */
+export async function rowsFromMysql(source) {
+    let mysql;
+    try {
+        mysql = await import("mysql2/promise");
+    }
+    catch {
+        throw new Error("The MySQL adapter needs the optional peer dependency `mysql2`. Install it with: npm i mysql2");
+    }
+    const table = quoteIdent(source.table);
+    const key = quoteIdent(source.keyColumn);
+    const id = source.idColumn ? quoteIdent(source.idColumn) : null;
+    const truth = source.truthColumn ? quoteIdent(source.truthColumn) : null;
+    const limit = Math.max(1, Math.min(source.limit ?? 100_000, 5_000_000));
+    const timeout = source.statementTimeoutMs ?? 30_000;
+    const conn = await mysql.createConnection(source.connectionString);
+    try {
+        if (!source.allowWritableRole) {
+            const [grants] = await conn.query("SHOW GRANTS FOR CURRENT_USER()");
+            const writable = grants.some((g) => {
+                const text = String(Object.values(g)[0] ?? "").toUpperCase();
+                if (!/\bON\b/.test(text))
+                    return false;
+                return /\b(ALL PRIVILEGES|INSERT|UPDATE|DELETE)\b/.test(text);
+            });
+            if (writable) {
+                throw new Error(`The connected MySQL user holds write privileges. shadowjoin only reads — use a SELECT-only user (see the README) or pass allowWritableRole: true to override.`);
+            }
+        }
+        await conn.query(`SET SESSION max_execution_time = ${Number(timeout)}`);
+        await conn.query("START TRANSACTION READ ONLY");
+        const select = [
+            `${key} AS \`key\``,
+            id ? `CAST(${id} AS CHAR) AS \`id\`` : `CAST(ROW_NUMBER() OVER () AS CHAR) AS \`id\``,
+            truth ? `CAST(${truth} AS CHAR) AS \`truth\`` : `NULL AS \`truth\``,
+        ].join(", ");
+        const [rows] = await conn.query(`SELECT ${select} FROM ${table} WHERE ${key} IS NOT NULL LIMIT ${limit}`);
+        await conn.query("COMMIT");
+        return rows
+            .filter((r) => String(r.key ?? "").trim() !== "")
+            .map((r) => ({
+            id: String(r.id),
+            key: String(r.key).trim(),
+            ...(r.truth ? { truth: String(r.truth).trim() } : {}),
+        }));
+    }
+    catch (err) {
+        try {
+            await conn.query("ROLLBACK");
+        }
+        catch { /* connection may already be gone */ }
+        throw err;
+    }
+    finally {
+        await conn.end();
+    }
+}

package/dist/adapters/postgres.d.ts

@@ -0,0 +1,28 @@
+import type { Row } from "../types.js";
+export interface PostgresSource {
+    /** Postgres connection string. Prefer a replica and a read-only role. */
+    connectionString: string;
+    /** Table, optionally schema-qualified. */
+    table: string;
+    /** Column holding the join key. */
+    keyColumn: string;
+    idColumn?: string;
+    truthColumn?: string;
+    /** Row cap. Default 100000. */
+    limit?: number;
+    /** Per-statement timeout in ms. Default 30000. */
+    statementTimeoutMs?: number;
+    /**
+     * shadowjoin refuses a role that can write to the table it is about to read.
+     * A measurement tool has no business holding write capability.
+     */
+    allowWritableRole?: boolean;
+}
+/**
+ * Read join keys out of Postgres. Opens a READ ONLY transaction with a
+ * statement and lock timeout, so it cannot write, cannot block a writer, and
+ * cannot run away with your CPU.
+ *
+ * Requires the optional peer dependency `pg`.
+ */
+export declare function rowsFromPostgres(source: PostgresSource): Promise<Row[]>;