shadowjoin 0.1.0 → 0.3.1

package/README.md

@@ -17,6 +17,18 @@ npx shadowjoin scan \
   --right crm_export.csv  --right-key account_name
 ```
 
+[![shadowjoin measuring a country-name join: 205 of 249 matched exactly, which looks fine, while five real matches were being dropped including Vietnam, spelled Viet Nam by the UN](assets/brag.gif)](https://github.com/harsh-chandak/shadowjoin/blob/main/assets/brag.mp4)
+
+<sub>Fourteen seconds, no sound. Every figure in it is verbatim CLI output from
+the committed example below — [higher-resolution
+version](https://github.com/harsh-chandak/shadowjoin/blob/main/assets/brag.mp4).</sub>
+
+The progress output above is real: shadowjoin shows what it is doing while it
+works. All of it is suppressed automatically when stdout is not a terminal, so
+piping to a file, to `jq`, or running in CI produces byte-identical output to
+before — and there is **no postinstall script**, because a tool you point at a
+production database has no business running code at install time.
+
 ## What it found on public data
 
 Joining the **S&P 500 constituent list** against **NYSE + Nasdaq listings** on
@@ -54,6 +66,58 @@ node dist/cli.js scan \
   --right examples/listings.json --right-key name    --right-id symbol --right-truth symbol
 ```
 
+### A join you would actually ship
+
+The example above is deliberately unflattering — a short index name against a
+full security description. Here is the realistic case.
+
+**Unicode CLDR country names** (what your UI displays) joined to **UN official
+country names** (what your reference data uses), on the country name, with ISO
+3166 alpha-3 as ground truth. That is a join people write every week.
+
+```
+  exact join matches              205 / 249   82.33%
+  recovered by shadow join          5          2.01%   precision 100%
+  still unexplained                39
+
+  rules worth adopting:
+  1.  & -> and                 +4
+  2.  remove all whitespace    +1
+```
+
+It looks healthy. 82% matched, no errors, nobody investigating. And it is
+silently dropping `Antigua & Barbuda`, `Trinidad & Tobago`,
+`São Tomé & Príncipe`, `Turks & Caicos Islands` and `Vietnam` — the last one
+because the UN spells it `Viet Nam`.
+
+Two things this example shows that the first one does not:
+
+**Applying every rule finds nothing.** The full rule set recovers zero here; the
+two-rule subset recovers five. Over-normalization collapses keys that should stay
+distinct, and shadowjoin names the better subset instead of letting you assume
+more rules is better.
+
+**It separates mechanical misses from semantic ones.** The 39 it cannot explain
+are not normalization problems at all:
+
+```
+  Vatican City    vs  Holy See
+  North Korea     vs  Democratic People's Republic of Korea
+  Cape Verde      vs  Cabo Verde
+  Bosnia          vs  Bosnia and Herzegovina
+  Hong Kong       vs  China, Hong Kong Special Administrative Region
+```
+
+No string transform recovers those. They need an alias table or a real entity
+resolution library. Knowing *which* of your misses are mechanical and which are
+semantic is the difference between a one-line fix and a project.
+
+```bash
+node dist/cli.js scan \
+  --left  examples/countries_ui.csv        --left-key  country --left-truth  iso3 \
+  --right examples/countries_reference.csv --right-key country --right-truth iso3
+```
+
 > That particular join is deliberately unflattering — a short index name against a
 > full security description. It is a worked example of the *measurement*, not a
 > claim that anyone ships that join. Point it at your own data to get a figure
@@ -119,9 +183,28 @@ npx shadowjoin scan \
 # in CI: fail the build if the floor climbs above 2%
 npx shadowjoin scan --left a.csv --left-key name --right b.csv --right-key name --max-loss 2
 
+# MySQL and SQLite work the same way
+npx shadowjoin scan \
+  --left  'mysql://ro@host/db' --left-table  accounts --left-key  company_name \
+  --right ./reference.sqlite   --right-table vendors  --right-key name
+
 npx shadowjoin rules     # list the normalization rules
 ```
 
+**Sources:** CSV, JSON, Postgres, MySQL and SQLite. SQLite uses Node's built-in
+`node:sqlite` (Node 22.5+), so it needs nothing installed; Postgres and MySQL
+need `pg` or `mysql2`, which are optional peers.
+
+**Anything else** works through the library in three lines, because the core
+takes two arrays rather than a connection:
+
+```js
+import { scan } from "shadowjoin";
+const [rows] = await anyDriver.query("SELECT id, company_name FROM accounts");
+const left = rows.map((r) => ({ id: String(r.id), key: r.company_name }));
+scan(left, right);
+```
+
 ### Library
 
 ```js
@@ -136,6 +219,19 @@ const key = (value) => normalize(value, rules);
 toSql("company_name", rules);                     // the same thing as Postgres SQL
 ```
 
+### Claude Code plugin (easiest)
+
+One command installs the skill and the MCP server together, and prompts for your
+config path:
+
+```bash
+claude plugin marketplace add harsh-chandak/shadowjoin
+claude plugin install shadowjoin@harsh-chandak
+```
+
+Also listed in the [official MCP registry](https://registry.modelcontextprotocol.io)
+as `io.github.harsh-chandak/shadowjoin`.
+
 ### MCP server
 
 Lets an agent measure a join, look at real examples, propose a fix, and then
@@ -147,7 +243,7 @@ not something it writes.
   "mcpServers": {
     "shadowjoin": {
       "command": "npx",
-      "args": ["-y", "-p", "shadowjoin", "shadowjoin-mcp"],
+      "args": ["-y", "-p", "shadowjoin", "-p", "@modelcontextprotocol/sdk", "shadowjoin-mcp"],
       "env": { "SHADOWJOIN_CONFIG": "/abs/path/to/shadowjoin.config.json" }
     }
   }

package/dist/adapters/mysql.d.ts

@@ -0,0 +1,21 @@
+import type { Row } from "../types.js";
+export interface MysqlSource {
+    /** mysql:// connection string. Prefer a replica and a read-only user. */
+    connectionString: string;
+    table: string;
+    keyColumn: string;
+    idColumn?: string;
+    truthColumn?: string;
+    limit?: number;
+    /** Per-statement cap in ms. Default 30000. */
+    statementTimeoutMs?: number;
+    /** shadowjoin refuses a user that can write to the table it is about to read. */
+    allowWritableRole?: boolean;
+}
+/**
+ * Read join keys out of MySQL inside a READ ONLY transaction with a statement
+ * timeout, and refuse a user that holds write privileges on the target.
+ *
+ * Requires the optional peer dependency `mysql2`.
+ */
+export declare function rowsFromMysql(source: MysqlSource): Promise<Row[]>;

package/dist/adapters/mysql.js

@@ -0,0 +1,72 @@
+const IDENT = /^[A-Za-z_][A-Za-z0-9_$]*$/;
+function quoteIdent(name) {
+    const parts = name.split(".");
+    for (const part of parts) {
+        if (!IDENT.test(part)) {
+            throw new Error(`Refusing identifier "${name}". Use a plain table or column name; shadowjoin does not accept arbitrary SQL.`);
+        }
+    }
+    return parts.map((p) => `\`${p}\``).join(".");
+}
+/**
+ * Read join keys out of MySQL inside a READ ONLY transaction with a statement
+ * timeout, and refuse a user that holds write privileges on the target.
+ *
+ * Requires the optional peer dependency `mysql2`.
+ */
+export async function rowsFromMysql(source) {
+    let mysql;
+    try {
+        mysql = await import("mysql2/promise");
+    }
+    catch {
+        throw new Error("The MySQL adapter needs the optional peer dependency `mysql2`. Install it with: npm i mysql2");
+    }
+    const table = quoteIdent(source.table);
+    const key = quoteIdent(source.keyColumn);
+    const id = source.idColumn ? quoteIdent(source.idColumn) : null;
+    const truth = source.truthColumn ? quoteIdent(source.truthColumn) : null;
+    const limit = Math.max(1, Math.min(source.limit ?? 100_000, 5_000_000));
+    const timeout = source.statementTimeoutMs ?? 30_000;
+    const conn = await mysql.createConnection(source.connectionString);
+    try {
+        if (!source.allowWritableRole) {
+            const [grants] = await conn.query("SHOW GRANTS FOR CURRENT_USER()");
+            const writable = grants.some((g) => {
+                const text = String(Object.values(g)[0] ?? "").toUpperCase();
+                if (!/\bON\b/.test(text))
+                    return false;
+                return /\b(ALL PRIVILEGES|INSERT|UPDATE|DELETE)\b/.test(text);
+            });
+            if (writable) {
+                throw new Error(`The connected MySQL user holds write privileges. shadowjoin only reads — use a SELECT-only user (see the README) or pass allowWritableRole: true to override.`);
+            }
+        }
+        await conn.query(`SET SESSION max_execution_time = ${Number(timeout)}`);
+        await conn.query("START TRANSACTION READ ONLY");
+        const select = [
+            `${key} AS \`key\``,
+            id ? `CAST(${id} AS CHAR) AS \`id\`` : `CAST(ROW_NUMBER() OVER () AS CHAR) AS \`id\``,
+            truth ? `CAST(${truth} AS CHAR) AS \`truth\`` : `NULL AS \`truth\``,
+        ].join(", ");
+        const [rows] = await conn.query(`SELECT ${select} FROM ${table} WHERE ${key} IS NOT NULL LIMIT ${limit}`);
+        await conn.query("COMMIT");
+        return rows
+            .filter((r) => String(r.key ?? "").trim() !== "")
+            .map((r) => ({
+            id: String(r.id),
+            key: String(r.key).trim(),
+            ...(r.truth ? { truth: String(r.truth).trim() } : {}),
+        }));
+    }
+    catch (err) {
+        try {
+            await conn.query("ROLLBACK");
+        }
+        catch { /* connection may already be gone */ }
+        throw err;
+    }
+    finally {
+        await conn.end();
+    }
+}

package/dist/adapters/sqlite.d.ts

@@ -0,0 +1,18 @@
+import type { Row } from "../types.js";
+export interface SqliteSource {
+    /** Path to the database file. */
+    file: string;
+    table: string;
+    keyColumn: string;
+    idColumn?: string;
+    truthColumn?: string;
+    limit?: number;
+}
+/**
+ * Read join keys out of a SQLite file. Opens the database READ ONLY.
+ *
+ * Uses Node's built-in `node:sqlite`, so there is nothing to install — but that
+ * module landed in Node 22.5, so older runtimes get a clear error rather than a
+ * confusing one.
+ */
+export declare function rowsFromSqlite(source: SqliteSource): Promise<Row[]>;

package/dist/adapters/sqlite.js

@@ -0,0 +1,49 @@
+const IDENT = /^[A-Za-z_][A-Za-z0-9_$]*$/;
+function quoteIdent(name) {
+    if (!IDENT.test(name)) {
+        throw new Error(`Refusing identifier "${name}". Use a plain table or column name; shadowjoin does not accept arbitrary SQL.`);
+    }
+    return `"${name}"`;
+}
+/**
+ * Read join keys out of a SQLite file. Opens the database READ ONLY.
+ *
+ * Uses Node's built-in `node:sqlite`, so there is nothing to install — but that
+ * module landed in Node 22.5, so older runtimes get a clear error rather than a
+ * confusing one.
+ */
+export async function rowsFromSqlite(source) {
+    let DatabaseSync;
+    try {
+        ({ DatabaseSync } = await import("node:sqlite"));
+    }
+    catch {
+        throw new Error(`The SQLite adapter needs Node's built-in node:sqlite, added in Node 22.5. This process is ${process.version}. Upgrade Node, or export the table to CSV and point shadowjoin at that.`);
+    }
+    const table = quoteIdent(source.table);
+    const key = quoteIdent(source.keyColumn);
+    const id = source.idColumn ? quoteIdent(source.idColumn) : null;
+    const truth = source.truthColumn ? quoteIdent(source.truthColumn) : null;
+    const limit = Math.max(1, Math.min(source.limit ?? 100_000, 5_000_000));
+    const db = new DatabaseSync(source.file, { readOnly: true });
+    try {
+        const select = [
+            `${key} AS key`,
+            id ? `CAST(${id} AS TEXT) AS id` : `CAST(rowid AS TEXT) AS id`,
+            truth ? `CAST(${truth} AS TEXT) AS truth` : `NULL AS truth`,
+        ].join(", ");
+        const rows = db
+            .prepare(`SELECT ${select} FROM ${table} WHERE ${key} IS NOT NULL LIMIT ${limit}`)
+            .all();
+        return rows
+            .filter((r) => String(r.key ?? "").trim() !== "")
+            .map((r) => ({
+            id: String(r.id),
+            key: String(r.key).trim(),
+            ...(r.truth ? { truth: String(r.truth).trim() } : {}),
+        }));
+    }
+    finally {
+        db.close();
+    }
+}

package/dist/cli.js

@@ -6,20 +6,23 @@ import { formatReport } from "./format.js";
 import { ALL_RULE_IDS, OPTIONAL_RULES } from "./normalize.js";
 import { rowsFromCsv } from "./adapters/csv.js";
 import { rowsFromJson } from "./adapters/json.js";
+import { banner, footer, isTty, Progress } from "./ui.js";
+import { createRequire } from "node:module";
+const VERSION = createRequire(import.meta.url)("../package.json").version;
 const USAGE = `
 shadowjoin — measure the matches your exact join is silently dropping.
 
   npx shadowjoin scan --left <src> --left-key <col> --right <src> --right-key <col>
 
-Sources are .csv, .json (array of objects), or a postgres:// connection string
-(which also needs --left-table / --right-table).
+Sources are .csv, .json (array of objects), .db/.sqlite, or a postgres:// or
+mysql:// connection string. Database sources also need --left-table/--right-table.
 
-  --left, --right              file path or postgres:// connection string
+  --left, --right              file path, or postgres:// / mysql:// URL
   --left-key, --right-key      column holding the join key          (required)
   --left-id, --right-id        column holding a stable id
   --left-truth, --right-truth  column holding ground truth; supply it on BOTH
                                sides and every recovery is verified, not guessed
-  --left-table, --right-table  table name (postgres sources only)
+  --left-table, --right-table  table name (database sources only)
 
   --rules <a,b,c>   limit to these rules (default: all; see \`shadowjoin rules\`)
   --samples <n>     example recoveries to print            (default 20)
@@ -38,19 +41,30 @@ async function load(src, kind, v, limit) {
     if (!keyColumn)
         throw new Error(`--${kind}-key is required`);
     const opts = { keyColumn, idColumn: v[`${kind}-id`], truthColumn: v[`${kind}-truth`] };
-    if (src.startsWith("postgres://") || src.startsWith("postgresql://")) {
-        const table = v[`${kind}-table`];
+    const table = v[`${kind}-table`];
+    const needsTable = () => {
         if (!table)
-            throw new Error(`--${kind}-table is required for a postgres source`);
+            throw new Error(`--${kind}-table is required for a database source`);
+        return table;
+    };
+    if (src.startsWith("postgres://") || src.startsWith("postgresql://")) {
         const { rowsFromPostgres } = await import("./adapters/postgres.js");
-        return rowsFromPostgres({ connectionString: src, table, limit, ...opts });
+        return rowsFromPostgres({ connectionString: src, table: needsTable(), limit, ...opts });
+    }
+    if (src.startsWith("mysql://")) {
+        const { rowsFromMysql } = await import("./adapters/mysql.js");
+        return rowsFromMysql({ connectionString: src, table: needsTable(), limit, ...opts });
     }
     const ext = extname(src).toLowerCase();
+    if (ext === ".db" || ext === ".sqlite" || ext === ".sqlite3") {
+        const { rowsFromSqlite } = await import("./adapters/sqlite.js");
+        return rowsFromSqlite({ file: src, table: needsTable(), limit, ...opts });
+    }
     if (ext === ".json")
         return rowsFromJson(src, opts);
     if (ext === ".csv" || ext === ".tsv" || ext === "")
         return rowsFromCsv(src, opts);
-    throw new Error(`Unsupported source "${src}". Use .csv, .json, or a postgres:// connection string.`);
+    throw new Error(`Unsupported source "${src}". Use .csv, .json, .db/.sqlite, or a postgres:// or mysql:// connection string.`);
 }
 async function main() {
     const { values, positionals } = parseArgs({
@@ -67,7 +81,9 @@ async function main() {
     });
     const command = positionals[0] ?? (values.left ? "scan" : "help");
     if (values.help || command === "help") {
+        process.stdout.write(banner(VERSION));
         console.log(USAGE);
+        process.stdout.write(footer(VERSION));
         return 0;
     }
     if (command === "rules") {
@@ -89,16 +105,34 @@ async function main() {
     }
     const v = values;
     const limit = values.limit ? Number(values.limit) : 100_000;
-    const [left, right] = await Promise.all([
-        load(values.left, "left", v, limit),
-        load(values.right, "right", v, limit),
-    ]);
+    const quiet = Boolean(values.json) || !isTty();
+    if (!quiet)
+        process.stdout.write(banner(VERSION));
+    const progress = new Progress();
+    if (!quiet)
+        progress.start(`reading ${values.left}`);
+    let left, right;
+    try {
+        left = await load(values.left, "left", v, limit);
+        if (!quiet)
+            progress.step(`reading ${values.right}`);
+        right = await load(values.right, "right", v, limit);
+    }
+    catch (err) {
+        progress.abort();
+        throw err;
+    }
     const report = scan(left, right, {
+        onPhase: quiet ? undefined : (label) => progress.step(label),
         rules: values.rules ? values.rules.split(",").map((s) => s.trim()).filter(Boolean) : ALL_RULE_IDS,
         maxSamplePairs: values.samples ? Number(values.samples) : 20,
         redactSamples: Boolean(values.redact),
     });
+    if (!quiet)
+        progress.stop();
     console.log(values.json ? JSON.stringify(report, null, 2) : formatReport(report));
+    if (!quiet)
+        process.stdout.write(footer(VERSION));
     if (values["max-loss"] !== undefined) {
         const threshold = Number(values["max-loss"]);
         if (report.lowerBoundLossPct > threshold) {

package/dist/format.js

@@ -44,6 +44,17 @@ export function formatReport(report) {
         }
         out.push("");
     }
+    if (report.nearMisses.length) {
+        out.push("  closest neighbours among rows NO RULE explains (suggestions, not counted):");
+        const mark = { verified: "ok ", contradicted: "BAD", ambiguous: "?? ", unverifiable: "   " };
+        for (const nm of report.nearMisses.slice(0, 10)) {
+            out.push(`    ${mark[nm.verification]} ${pad(nm.leftKey, 34)} ~  ${pad(nm.rightKey, 38)} ${nm.similarity}`);
+        }
+        if (report.noCandidate > 0) {
+            out.push(`    ${report.noCandidate} more have no string similarity at all — those need an alias table.`);
+        }
+        out.push("");
+    }
     for (const w of report.warnings)
         out.push(`  ! ${w}`);
     if (report.warnings.length)

package/dist/index.d.ts

@@ -4,4 +4,5 @@ export { BASE_RULES, OPTIONAL_RULES, ALL_RULE_IDS, normalize, rulesByIds, toSql
 export { formatReport } from "./format.js";
 export { parseCsv, rowsFromCsv, rowsFromCsvText } from "./adapters/csv.js";
 export type { CsvOptions } from "./adapters/csv.js";
-export type { Report, Row, Rule, RuleContribution, SamplePair, Verification } from "./types.js";
+export { trigrams, dice, TrigramIndex } from "./similarity.js";
+export type { NearMiss, Report, Row, Rule, RuleContribution, SamplePair, StackStep, Verification } from "./types.js";

package/dist/index.js

@@ -2,3 +2,4 @@ export { scan } from "./scan.js";
 export { BASE_RULES, OPTIONAL_RULES, ALL_RULE_IDS, normalize, rulesByIds, toSql } from "./normalize.js";
 export { formatReport } from "./format.js";
 export { parseCsv, rowsFromCsv, rowsFromCsvText } from "./adapters/csv.js";
+export { trigrams, dice, TrigramIndex } from "./similarity.js";

package/dist/mcp.js

@@ -11,13 +11,36 @@
 import { readFile } from "node:fs/promises";
 import { resolve } from "node:path";
 import { randomUUID } from "node:crypto";
-import { Server } from "@modelcontextprotocol/sdk/server/index.js";
-import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
-import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
+import { createRequire } from "node:module";
 import { scan } from "./scan.js";
 import { ALL_RULE_IDS, rulesByIds, toSql } from "./normalize.js";
 import { rowsFromCsv } from "./adapters/csv.js";
 import { rowsFromJson } from "./adapters/json.js";
+const VERSION = createRequire(import.meta.url)("../package.json").version;
+/**
+ * The SDK is an optional peer, so the core library and CLI stay dependency-free.
+ * Import it dynamically to turn a missing install into an instruction rather
+ * than a module-resolution stack trace, which is what an MCP client shows the
+ * user as nothing more than "Connection closed".
+ */
+async function loadSdk() {
+    try {
+        const [{ Server }, { StdioServerTransport }, types] = await Promise.all([
+            import("@modelcontextprotocol/sdk/server/index.js"),
+            import("@modelcontextprotocol/sdk/server/stdio.js"),
+            import("@modelcontextprotocol/sdk/types.js"),
+        ]);
+        return { Server, StdioServerTransport, ...types };
+    }
+    catch {
+        process.stderr.write("shadowjoin-mcp needs @modelcontextprotocol/sdk, which is an optional peer dependency.\n" +
+            "Run it with the SDK provided:\n" +
+            "  npx -y -p shadowjoin -p @modelcontextprotocol/sdk shadowjoin-mcp\n" +
+            "or install it alongside shadowjoin:\n" +
+            "  npm i shadowjoin @modelcontextprotocol/sdk\n");
+        process.exit(1);
+    }
+}
 const CONFIG_PATH = resolve(process.env.SHADOWJOIN_CONFIG ?? "shadowjoin.config.json");
 async function loadConfig() {
     try {
@@ -38,11 +61,19 @@ async function loadSide(side, config, limit) {
         throw new Error(`Unknown source "${side.source}". Configured: ${Object.keys(config.sources).join(", ") || "(none)"}`);
     }
     const opts = { keyColumn: side.column, idColumn: side.idColumn, truthColumn: side.truthColumn };
-    if (def.type === "postgres") {
+    if (def.type === "postgres" || def.type === "mysql" || def.type === "sqlite") {
         if (!side.table)
             throw new Error(`Source "${side.source}" is a database; "table" is required.`);
-        const { rowsFromPostgres } = await import("./adapters/postgres.js");
-        return rowsFromPostgres({ connectionString: def.connectionString, table: side.table, limit, ...opts });
+        if (def.type === "postgres") {
+            const { rowsFromPostgres } = await import("./adapters/postgres.js");
+            return rowsFromPostgres({ connectionString: def.connectionString, table: side.table, limit, ...opts });
+        }
+        if (def.type === "mysql") {
+            const { rowsFromMysql } = await import("./adapters/mysql.js");
+            return rowsFromMysql({ connectionString: def.connectionString, table: side.table, limit, ...opts });
+        }
+        const { rowsFromSqlite } = await import("./adapters/sqlite.js");
+        return rowsFromSqlite({ file: def.path, table: side.table, limit, ...opts });
     }
     return def.type === "json" ? rowsFromJson(def.path, opts) : rowsFromCsv(def.path, opts);
 }
@@ -125,7 +156,8 @@ const TOOLS = [
         },
     },
 ];
-const server = new Server({ name: "shadowjoin", version: "0.1.0" }, { capabilities: { tools: {} } });
+const { Server, StdioServerTransport, CallToolRequestSchema, ListToolsRequestSchema } = await loadSdk();
+const server = new Server({ name: "shadowjoin", version: VERSION }, { capabilities: { tools: {} } });
 server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: TOOLS }));
 server.setRequestHandler(CallToolRequestSchema, async (request) => {
     const { name } = request.params;

package/dist/scan.d.ts

@@ -6,6 +6,14 @@ export interface ScanOptions {
     maxSamplePairs?: number;
     /** Replace key values in samples with a short hash. Counts are unaffected. */
     redactSamples?: boolean;
+    /** Look for trigram neighbours among rows no rule explains. Default true. */
+    nearMisses?: boolean;
+    /** Minimum Dice score for a near miss. Default 0.45. */
+    nearMissThreshold?: number;
+    /** Cap on near misses reported. Default 25. */
+    maxNearMisses?: number;
+    /** Called as each phase begins, so a CLI can show progress. */
+    onPhase?: (label: string) => void;
 }
 /**
  * Compare an exact join against a shadow matcher and report what the exact

package/dist/scan.js

@@ -1,4 +1,5 @@
 import { ALL_RULE_IDS, normalize, rulesByIds } from "./normalize.js";
+import { TrigramIndex } from "./similarity.js";
 /** Tickers, SKUs and ids differ cosmetically across sources; compare them loosely. */
 function canonicalTruth(value) {
     return value.replace(/[^\p{L}\p{N}]+/gu, "").toUpperCase();
@@ -54,7 +55,9 @@ export function scan(left, right, options = {}) {
     if (left.length === 0 || right.length === 0) {
         warnings.push("One side of the join is empty; there is nothing to measure.");
     }
+    const phase = options.onPhase ?? (() => { });
     // 1. The baseline: the join exactly as it runs today.
+    phase("measuring the join as it runs today");
     const exactIndex = buildIndex(right, null);
     const missed = [];
     let exactMatches = 0;
@@ -66,6 +69,7 @@ export function scan(left, right, options = {}) {
     }
     // 2. Each rule measured independently against that baseline, so the number
     //    beside a rule is what the rule alone buys and not a product of ordering.
+    phase(`testing ${rules.length} rules against that baseline`);
     const attribution = new Map();
     const byRule = rules.map((rule) => {
         const index = buildIndex(right, [rule]);
@@ -90,6 +94,7 @@ export function scan(left, right, options = {}) {
     // 3. Greedy forward selection: which rules, in which order, actually pay.
     //    Most recoveries need a combination, so per-rule figures alone are not
     //    enough to act on.
+    phase("selecting the rules worth adopting");
     const stack = [];
     const covered = new Set();
     const chosen = [];
@@ -176,6 +181,35 @@ export function scan(left, right, options = {}) {
     }
     const recovered = attribution.size;
     const unexplained = missed.length - recovered;
+    // 6. The residue. No normalization rule explains these, so look for the
+    //    closest trigram neighbour instead. Reported as advice, never counted:
+    //    a suggestion that has not cleared the rule bar must not move the floor.
+    const nearMisses = [];
+    let noCandidate = 0;
+    if (options.nearMisses !== false && unexplained > 0) {
+        phase(`analysing the ${unexplained} rows no rule explains`);
+        const index = new TrigramIndex(right);
+        const maxNear = options.maxNearMisses ?? 25;
+        for (const row of missed) {
+            if (attribution.has(row.id))
+                continue;
+            const hit = index.best(row.key, { threshold: options.nearMissThreshold ?? 0.45 });
+            if (!hit) {
+                noCandidate++;
+                continue;
+            }
+            if (nearMisses.length < maxNear) {
+                nearMisses.push({
+                    leftId: row.id,
+                    leftKey: options.redactSamples ? redact(row.key) : row.key,
+                    rightId: hit.row.id,
+                    rightKey: options.redactSamples ? redact(hit.row.key) : hit.row.key,
+                    similarity: hit.similarity,
+                    verification: verify(row, [hit.row]),
+                });
+            }
+        }
+    }
     const pct = (n) => (left.length === 0 ? 0 : Number(((n / left.length) * 100).toFixed(2)));
     if (rules.length > 1) {
         warnings.push("Per-rule counts are measured independently against the exact baseline, so they overlap and will not sum to the total.");
@@ -183,6 +217,12 @@ export function scan(left, right, options = {}) {
     if (chosen.length > 0 && chosen.length < rules.length && fullStackRecovered < covered.size) {
         warnings.push(`Applying every rule recovers ${covered.size - fullStackRecovered} fewer match(es) than the best subset. Over-normalization is costing you matches — rerun with --rules ${chosen.map((r) => r.id).join(",")}`);
     }
+    if (nearMisses.length > 0) {
+        warnings.push(`${nearMisses.length} unexplained row(s) have a close trigram neighbour. These are suggestions, NOT counted in the floor — confirm them before adding an alias.`);
+    }
+    if (noCandidate > 0) {
+        warnings.push(`${noCandidate} unexplained row(s) have no string similarity to anything on the right (e.g. "Vatican City" vs "Holy See"). No normalization rule will ever recover those; they need an alias table.`);
+    }
     if (contradicted > 0) {
         warnings.push(`${contradicted} recovered match(es) are contradicted by ground truth — the shadow matcher is over-merging. Review samplePairs before acting.`);
     }
@@ -212,6 +252,8 @@ export function scan(left, right, options = {}) {
             ? null
             : Number(((verified / (verified + contradicted)) * 100).toFixed(2)),
         samplePairs,
+        nearMisses,
+        noCandidate,
         warnings,
         rulesApplied: ruleIds,
     };

package/dist/similarity.d.ts

@@ -0,0 +1,34 @@
+import type { Row } from "./types.js";
+/**
+ * Trigram similarity, for the rows no normalization rule explains.
+ *
+ * Implemented in-process rather than delegating to Postgres `pg_trgm`, so the
+ * same residue analysis runs on CSVs, on JSON and on databases that have no
+ * such extension.
+ */
+/** Postgres-style padding so short strings and word starts are weighted sensibly. */
+export declare function trigrams(value: string): Set<string>;
+/** Sørensen–Dice: 2|A∩B| / (|A|+|B|). 1 is identical, 0 shares nothing. */
+export declare function dice(a: Set<string>, b: Set<string>): number;
+export interface NearMissOptions {
+    /** Minimum Dice score to report. Default 0.45. */
+    threshold?: number;
+    /** Cap on candidates scored per unexplained row. Default 200. */
+    maxCandidates?: number;
+}
+export interface NearMissHit {
+    row: Row;
+    similarity: number;
+}
+/**
+ * An inverted trigram index over the right-hand side. Built once, queried per
+ * unexplained row, so cost grows with the residue rather than with every pair.
+ */
+export declare class TrigramIndex {
+    private readonly rows;
+    private readonly postings;
+    private readonly grams;
+    constructor(rows: Row[]);
+    /** Best scoring right-hand row for `key`, or null if nothing clears the bar. */
+    best(key: string, options?: NearMissOptions): NearMissHit | null;
+}

package/dist/similarity.js

@@ -0,0 +1,81 @@
+/**
+ * Trigram similarity, for the rows no normalization rule explains.
+ *
+ * Implemented in-process rather than delegating to Postgres `pg_trgm`, so the
+ * same residue analysis runs on CSVs, on JSON and on databases that have no
+ * such extension.
+ */
+/** Postgres-style padding so short strings and word starts are weighted sensibly. */
+export function trigrams(value) {
+    const padded = `  ${value.trim().toLowerCase()} `;
+    const out = new Set();
+    for (let i = 0; i + 3 <= padded.length; i++)
+        out.add(padded.slice(i, i + 3));
+    return out;
+}
+/** Sørensen–Dice: 2|A∩B| / (|A|+|B|). 1 is identical, 0 shares nothing. */
+export function dice(a, b) {
+    if (a.size === 0 || b.size === 0)
+        return 0;
+    let shared = 0;
+    const [small, large] = a.size <= b.size ? [a, b] : [b, a];
+    for (const g of small)
+        if (large.has(g))
+            shared++;
+    return (2 * shared) / (a.size + b.size);
+}
+/**
+ * An inverted trigram index over the right-hand side. Built once, queried per
+ * unexplained row, so cost grows with the residue rather than with every pair.
+ */
+export class TrigramIndex {
+    rows;
+    postings = new Map();
+    grams = [];
+    constructor(rows) {
+        this.rows = rows;
+        rows.forEach((row, i) => {
+            const g = trigrams(row.key);
+            this.grams.push(g);
+            for (const gram of g) {
+                const bucket = this.postings.get(gram);
+                if (bucket)
+                    bucket.push(i);
+                else
+                    this.postings.set(gram, [i]);
+            }
+        });
+    }
+    /** Best scoring right-hand row for `key`, or null if nothing clears the bar. */
+    best(key, options = {}) {
+        const threshold = options.threshold ?? 0.45;
+        const maxCandidates = options.maxCandidates ?? 200;
+        const query = trigrams(key);
+        // Skipping near-universal trigrams keeps large indexes fast, but on a small
+        // right-hand side every bucket looks universal and the filter would discard
+        // everything. Only apply it once there are enough rows for it to mean something.
+        const universal = this.rows.length >= 20 ? this.rows.length / 2 : Infinity;
+        const overlap = new Map();
+        for (const gram of query) {
+            const bucket = this.postings.get(gram);
+            if (!bucket || bucket.length > universal)
+                continue;
+            for (const i of bucket)
+                overlap.set(i, (overlap.get(i) ?? 0) + 1);
+        }
+        if (overlap.size === 0)
+            return null;
+        // Score the most promising candidates only; shared-gram count bounds Dice.
+        const ranked = [...overlap.entries()]
+            .sort((a, b) => b[1] - a[1])
+            .slice(0, maxCandidates);
+        let best = null;
+        for (const [i] of ranked) {
+            const score = dice(query, this.grams[i]);
+            if (score >= threshold && (best === null || score > best.similarity)) {
+                best = { row: this.rows[i], similarity: Number(score.toFixed(3)) };
+            }
+        }
+        return best;
+    }
+}

package/dist/types.d.ts

@@ -49,6 +49,22 @@ export interface SamplePair {
     candidates: number;
     verification: Verification;
 }
+/**
+ * A row no rule explains, paired with its closest trigram neighbour.
+ *
+ * Deliberately NOT counted in `recovered` or `lowerBoundLossPct`: these are
+ * unverified suggestions below the confidence bar, and letting them inflate the
+ * headline would turn a defensible floor into a guess.
+ */
+export interface NearMiss {
+    leftId: string;
+    leftKey: string;
+    rightId: string;
+    rightKey: string;
+    /** Sørensen-Dice over character trigrams. 1 is identical. */
+    similarity: number;
+    verification: Verification;
+}
 export interface Report {
     generatedAt: string;
     leftRows: number;
@@ -82,6 +98,10 @@ export interface Report {
     /** verified / (verified + contradicted), or null when no ground truth was supplied. */
     precision: number | null;
     samplePairs: SamplePair[];
+    /** Closest trigram neighbours for rows no rule explains. Advisory, not counted. */
+    nearMisses: NearMiss[];
+    /** Unexplained rows with no trigram neighbour at all — these need an alias table. */
+    noCandidate: number;
     warnings: string[];
     rulesApplied: string[];
 }

package/dist/ui.d.ts

@@ -0,0 +1,19 @@
+export declare const isTty: () => boolean;
+export declare function banner(version: string): string;
+export declare function footer(version: string): string;
+/** A spinner that degrades to nothing when it isn't being watched by a human. */
+export declare class Progress {
+    private timer;
+    private frame;
+    private label;
+    private readonly started;
+    private readonly done;
+    start(label: string): void;
+    /** Mark the current step finished and begin the next one. */
+    step(label: string): void;
+    stop(): void;
+    /** Clear the spinner without claiming the step succeeded. */
+    abort(): void;
+    private clear;
+    private paint;
+}

package/dist/ui.js

@@ -0,0 +1,80 @@
+/**
+ * Terminal presentation. Everything here is silent unless stdout is a TTY, so
+ * piping to a file, to jq, or running in CI produces exactly the same bytes it
+ * always did.
+ */
+const FRAMES = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
+const DIM = "\x1b[2m", RESET = "\x1b[0m", BOLD = "\x1b[1m";
+const BLUE = "\x1b[38;5;75m", GREY = "\x1b[38;5;244m", AMBER = "\x1b[38;5;214m";
+export const isTty = () => Boolean(process.stdout.isTTY) && !process.env.NO_COLOR;
+export function banner(version) {
+    if (!isTty())
+        return "";
+    return [
+        "",
+        `  ${BLUE}▌${RESET}${BOLD} shadowjoin ${RESET}${GREY}v${version}${RESET}`,
+        `  ${BLUE}▌${RESET}${DIM} the matches your join never told you it was losing${RESET}`,
+        "",
+    ].join("\n");
+}
+export function footer(version) {
+    if (!isTty())
+        return "";
+    return `  ${GREY}shadowjoin v${version}  ·  MIT  ·  Harsh N. Chandak  ·  harsh-chandak.com${RESET}\n`;
+}
+/** A spinner that degrades to nothing when it isn't being watched by a human. */
+export class Progress {
+    timer = null;
+    frame = 0;
+    label = "";
+    started = Date.now();
+    done = [];
+    start(label) {
+        this.label = label;
+        if (!isTty())
+            return;
+        if (this.timer)
+            return;
+        this.timer = setInterval(() => this.paint(), 80);
+        this.timer.unref?.();
+        this.paint();
+    }
+    /** Mark the current step finished and begin the next one. */
+    step(label) {
+        if (isTty()) {
+            this.clear();
+            process.stdout.write(`  ${BLUE}✓${RESET} ${DIM}${this.label}${RESET}\n`);
+        }
+        this.done.push(this.label);
+        this.label = label;
+        if (isTty())
+            this.paint();
+    }
+    stop() {
+        if (this.timer) {
+            clearInterval(this.timer);
+            this.timer = null;
+        }
+        if (!isTty())
+            return;
+        this.clear();
+        const ms = Date.now() - this.started;
+        process.stdout.write(`  ${BLUE}✓${RESET} ${DIM}${this.label}${RESET}  ${GREY}${ms}ms${RESET}\n`);
+    }
+    /** Clear the spinner without claiming the step succeeded. */
+    abort() {
+        if (this.timer) {
+            clearInterval(this.timer);
+            this.timer = null;
+        }
+        if (isTty())
+            this.clear();
+    }
+    clear() {
+        process.stdout.write("\r\x1b[2K");
+    }
+    paint() {
+        const f = FRAMES[this.frame++ % FRAMES.length];
+        process.stdout.write(`\r\x1b[2K  ${AMBER}${f}${RESET} ${this.label}`);
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "shadowjoin",
-  "version": "0.1.0",
+  "version": "0.3.1",
   "description": "Measure the matches your exact join is silently dropping. Runs a shadow matcher alongside your real join key and reports a verified floor on lost matches, plus the normalization rule that would recover each one.",
   "keywords": [
     "entity-resolution",
@@ -14,7 +14,9 @@
     "csv",
     "mcp",
     "data-validation",
-    "recall"
+    "recall",
+    "mysql",
+    "sqlite"
   ],
   "license": "MIT",
   "type": "module",
@@ -30,7 +32,9 @@
   "exports": {
     ".": "./dist/index.js",
     "./mcp": "./dist/mcp.js",
-    "./adapters/postgres": "./dist/adapters/postgres.js"
+    "./adapters/postgres": "./dist/adapters/postgres.js",
+    "./adapters/mysql": "./dist/adapters/mysql.js",
+    "./adapters/sqlite": "./dist/adapters/sqlite.js"
   },
   "files": [
     "dist",
@@ -40,11 +44,12 @@
   ],
   "scripts": {
     "build": "tsc",
-    "test": "node --test test/",
+    "test": "node --test test/*.test.js",
     "prepublishOnly": "npm run build"
   },
   "peerDependencies": {
     "@modelcontextprotocol/sdk": ">=1",
+    "mysql2": ">=3",
     "pg": ">=8"
   },
   "peerDependenciesMeta": {
@@ -53,12 +58,16 @@
     },
     "@modelcontextprotocol/sdk": {
       "optional": true
+    },
+    "mysql2": {
+      "optional": true
     }
   },
   "devDependencies": {
     "@modelcontextprotocol/sdk": "^1.32.1",
     "@types/node": "^22.0.0",
     "@types/pg": "^8.23.1",
+    "mysql2": "^3.24.5",
     "typescript": "^5.6.0"
   },
   "repository": {
@@ -69,5 +78,6 @@
   "author": "Harsh N. Chandak",
   "bugs": {
     "url": "https://github.com/harsh-chandak/shadowjoin/issues"
-  }
+  },
+  "mcpName": "io.github.harsh-chandak/shadowjoin"
 }