ligatura 0.0.1 → 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bret Perry
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,105 @@
 # ligatura
 
-Ligature tool. Placeholder release; the real package is in progress.
+Find where one text quotes, paraphrases or alludes to another.
+
+```
+npx ligatura link imitation.jsonl gospels.jsonl > links.jsonl
+```
+
+ligatura reads two texts as units (verses, sentences, paragraphs). For each unit of the first, it
+finds the passages in the second that the unit echoes, and writes them as suggestions with their
+evidence. A person confirms each one: ligatura finds candidates, it doesn't decide.
+
+**The default is free and local.** Every model runs on your machine, downloads once, and needs no
+account or API key.
+
+## How it works
+
+1. **Embed** both texts (nomic-embed-text-v1.5) and score every target unit by meaning.
+2. **Overlap:** score every target unit by shared rare words. This catches quotation, which
+   embeddings blur. Archaic forms (followeth, walketh) fold to their stems.
+3. **Fuse** the two rankings (reciprocal rank fusion).
+4. **Rerank (opt-in):** with `--rerank`, a reranker model reads the top 30 for each unit and
+   scores each pair. It's off by default: the in-process Qwen3-Reranker-0.6B scored below plain
+   retrieval on the test fixture (see Results). A reranker server or a chat model as judge may do
+   better; measure it with `eval` first.
+
+## Install
+
+```
+npm install -g ligatura
+```
+
+Or run it with `npx ligatura`. Needs Node 20 or later.
+
+## Inputs
+
+- **JSONL**, one unit per line: `{"ref": "MAT.5.3", "text": "Blessed are the poor in spirit..."}`.
+  `ref` is any stable id: a verse, a Stephanus number, `book.chapter.sentence`.
+- **Plain text**, split into sentences, or paragraphs or lines with `--split`. Refs are the unit
+  numbers.
+
+## Output
+
+One JSON object per suggested link:
+
+```json
+{"source":"4.14.5","target":"LUK.24.35","rank":1,"score":0.792,"dense":0.792,"lex":0.295,
+ "shared":["breaking","bread","way"],"status":"suggested","method":{"embed":"nomic-embed-text-v1.5","rerank":"none"},"date":"2026-11-08"}
+```
+
+`score` is the reranker's P(yes) when a reranker ran, else the embedding cosine. `--top` (default
+3) and `--min-score` decide how many links reach review.
+
+## Models and servers
+
+| Stage | In process | Or a server you run |
+|---|---|---|
+| Embed | `nomic-embed-text-v1.5`, `bge-small-en-v1.5`, `bge-large-en-v1.5` | `--embed openai:<base-url>#<model>`: any OpenAI-compatible `/embeddings` (Ollama, LM Studio, llama.cpp, vLLM) |
+| Rerank | off by default; `--rerank qwen3-reranker-0.6b` | `--rerank api:<url>#<model>`: a Cohere/Jina-shaped `/rerank` endpoint (llama.cpp `llama-server`, vLLM, TEI, Infinity) |
+| Judge | – | `--rerank chat:<base-url>#<model>`: a chat model picks the links (Ollama, LM Studio, `mlx_lm.server`) |
+
+- **Speed:** the in-process reranker runs on the CPU at 0.5 to 0.75 s per pair. A 2,400-sentence
+  book at `--pool 30` takes many hours. A reranker server on a GPU is much faster: the same model
+  under MLX on an Apple M3 took 0.17 s per pair. Lowering `--pool` trades recall for time.
+- **Keys:** a server that needs one gets `LIGATURA_API_KEY` as a bearer token.
+- **Paid APIs bill your API account, not a chat subscription** such as Claude Pro or ChatGPT Plus.
+  ligatura has no paid backend built in. If you point it at one, you pay that provider per call.
+
+## Measure before you trust it
+
+```
+ligatura eval fixtures/kempis-gospels --neighbours 1
+```
+
+`eval` runs the pipeline on a fixture (`source.jsonl`, `target.jsonl`, and `gold.jsonl` of known
+links) and prints recall at 1, 3, 5 and 10. Try a model change here before using it on new texts.
+
+The fixture in this repo is The Imitation of Christ against the four Gospels, scored against the
+edition's own footnotes (49 links; see `fixtures/kempis-gospels/README.md`):
+
+| Stage | R@1 | R@3 | R@5 | R@10 |
+|---|---|---|---|---|
+| Retrieval (embedding + overlap) | 35 | 40 | 42 | 44 |
+| Reranked, `--rerank qwen3-reranker-0.6b` (ONNX q8, generic task) | 28 | 38 | 43 | 44 |
+| Reranked, q8, `--task` naming the two texts | 29 | 41 | 43 | 44 |
+
+Recall counts a link found when the gold verse or its neighbour is in the top k (`--neighbours 1`),
+of 49. Runs from 2026-11-08 on an Apple M3 CPU; the q8 reranker took 710 to 980 s for the 1,320 pairs.
+
+Reranking costs R@1 here, so it's off by default. The cause is the q8 quantization of the only
+ONNX export: an fp32 export of the same model, built locally, scores pairs within 0.006 of the
+Python (MLX) run but still reached only 32 / 43 / 44 / 45.
+
+## Limits
+
+- **English first.** The overlap scorer and the default embedder are English. Greek and Latin
+  need a multilingual embedder (`--embed openai:...`).
+- **Allusions to a whole story** (the prodigal son), or sayings with no shared words, rarely reach
+  the top 30, so no reranker sees them.
+- **Suggestions, not findings.** Check every link against the texts.
+
+## License
+
+MIT. The fixture texts are public domain. Models download from Hugging Face under their own
+licenses; check each model card before redistributing a model.

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,68 @@
+#!/usr/bin/env node
+import { parseArgs } from 'node:util';
+import { readFileSync } from 'node:fs';
+import { LOCAL_EMBEDDERS } from './embed.js';
+import { evaluate, formatEval } from './eval.js';
+import { link, toLinks, DEFAULTS } from './link.js';
+import { loadUnits } from './units.js';
+import { setQuiet } from './util.js';
+const HELP = `ligatura: find where one text quotes, paraphrases or alludes to another.
+
+Usage
+  ligatura link <source> <target> [options] > links.jsonl
+  ligatura eval <fixture-dir> [options]
+
+Inputs are JSONL ({"ref", "text"} per line) or plain text, split into sentences.
+Every link is a suggestion for a person to confirm.
+
+Options
+  --embed <model>     ${Object.keys(LOCAL_EMBEDDERS).join(', ')} (in process; default ${DEFAULTS.embed}),
+                      or openai:<base-url>#<model> for any OpenAI-compatible server
+  --rerank <model>    none (default), qwen3-reranker-0.6b (in process), api:<url>#<model> for a
+                      /v1/rerank server, or chat:<base-url>#<model> for a chat model as judge
+  --pool <n>          candidates the reranker reads per unit (default ${DEFAULTS.pool})
+  --top <n>           links kept per source unit (default 3)
+  --min-score <x>     keep only links scoring at least x (reranker P(yes), or embedding cosine)
+  --task <text>       the instruction the reranker reads
+  --split <mode>      for plain text: sentence (default), paragraph or line
+  --neighbours <n>    eval: also count a target n units away from the gold one (default 0)
+  --json              eval: print JSON
+  --quiet             no progress on stderr
+
+Defaults run locally and need no account. Models download on first use and are cached.
+API keys for remote servers come from LIGATURA_API_KEY.`;
+const { values, positionals } = parseArgs({
+    allowPositionals: true,
+    options: {
+        embed: { type: 'string' }, rerank: { type: 'string' }, pool: { type: 'string' }, top: { type: 'string' },
+        'min-score': { type: 'string' }, task: { type: 'string' }, split: { type: 'string' }, neighbours: { type: 'string' },
+        json: { type: 'boolean' }, quiet: { type: 'boolean' }, help: { type: 'boolean', short: 'h' }, version: { type: 'boolean', short: 'v' },
+    },
+});
+const num = (s) => (s === undefined ? undefined : Number(s));
+const [cmd, ...args] = positionals;
+setQuiet(!!values.quiet);
+const opts = { embed: values.embed, rerank: values.rerank, pool: num(values.pool), task: values.task };
+if (values.version) {
+    console.log(JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version);
+}
+else if (values.help || !cmd) {
+    console.log(HELP);
+}
+else if (cmd === 'link' && args.length === 2) {
+    const split = (values.split ?? 'auto');
+    const source = loadUnits(args[0], split);
+    const target = loadUnits(args[1], split);
+    for await (const r of link(source, target, opts)) {
+        for (const l of toLinks(r, target, { ...opts, top: num(values.top) ?? 3, minScore: num(values['min-score']) }))
+            process.stdout.write(`${JSON.stringify(l)}\n`);
+    }
+}
+else if (cmd === 'eval' && args.length === 1) {
+    const e = await evaluate(args[0], { ...opts, neighbours: num(values.neighbours) });
+    console.log(values.json ? JSON.stringify(e, null, 1) : formatEval(e));
+}
+else {
+    console.error(HELP);
+    process.exit(2);
+}

package/dist/embed.d.ts

@@ -0,0 +1,15 @@
+export interface Embedder {
+    name: string;
+    embed(texts: string[], role: 'query' | 'doc'): Promise<Float32Array[]>;
+}
+interface LocalModel {
+    id: string;
+    pooling: 'cls' | 'mean' | 'last_token';
+    query?: string;
+    doc?: string;
+}
+export declare const LOCAL_EMBEDDERS: Record<string, LocalModel>;
+export declare const DEFAULT_EMBEDDER = "nomic-embed-text-v1.5";
+export declare function makeEmbedder(spec: string): Embedder;
+export declare const auth: () => Record<string, string>;
+export {};

package/dist/embed.js

@@ -0,0 +1,84 @@
+// Embedding backends. Local models run in process (transformers.js, ONNX); `openai:` calls any
+// OpenAI-compatible /v1/embeddings server (Ollama, LM Studio, llama.cpp, vLLM).
+import { createHash } from 'node:crypto';
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { cacheDir, log } from './util.js';
+// Query and document prefixes follow each model card.
+export const LOCAL_EMBEDDERS = {
+    'nomic-embed-text-v1.5': { id: 'nomic-ai/nomic-embed-text-v1.5', pooling: 'mean', query: 'search_query: ', doc: 'search_document: ' },
+    'bge-small-en-v1.5': { id: 'Xenova/bge-small-en-v1.5', pooling: 'cls' },
+    'bge-large-en-v1.5': { id: 'Xenova/bge-large-en-v1.5', pooling: 'cls', query: 'Represent this sentence for searching relevant passages: ' },
+};
+export const DEFAULT_EMBEDDER = 'nomic-embed-text-v1.5';
+export function makeEmbedder(spec) {
+    const inner = spec.startsWith('openai:') ? openaiEmbedder(spec) : localEmbedder(spec);
+    return { name: inner.name, embed: (texts, role) => cached(inner, texts, role) };
+}
+// Vectors are cached per model, role and exact input, so a rerun embeds nothing.
+async function cached(e, texts, role) {
+    const key = createHash('sha1').update(`${e.name}\0${role}\0${texts.join('\0')}`).digest('hex');
+    const dir = join(cacheDir(), 'vectors');
+    const file = join(dir, `${key}.json`);
+    if (existsSync(file))
+        return JSON.parse(readFileSync(file, 'utf8')).map((v) => Float32Array.from(v));
+    const out = await e.embed(texts, role);
+    mkdirSync(dir, { recursive: true });
+    writeFileSync(file, JSON.stringify(out.map((v) => Array.from(v))));
+    return out;
+}
+function localEmbedder(name) {
+    const cfg = LOCAL_EMBEDDERS[name];
+    if (!cfg)
+        throw new Error(`unknown embedder "${name}"; local models: ${Object.keys(LOCAL_EMBEDDERS).join(', ')}; or openai:<base-url>#<model>`);
+    let pipe;
+    return {
+        name,
+        async embed(texts, role) {
+            if (!pipe) {
+                const { pipeline } = await import('@huggingface/transformers');
+                pipe = (await pipeline('feature-extraction', cfg.id, { dtype: 'q8' }));
+            }
+            const prefix = cfg[role] ?? '';
+            const out = [];
+            for (let i = 0; i < texts.length; i += 32) {
+                const t = await pipe(texts.slice(i, i + 32).map((x) => prefix + x), { pooling: cfg.pooling, normalize: true });
+                out.push(...t.tolist().map((v) => Float32Array.from(v)));
+                log(`embed ${name} ${role}: ${Math.min(i + 32, texts.length)}/${texts.length}`, true);
+            }
+            return out;
+        },
+    };
+}
+// openai:http://localhost:11434/v1#nomic-embed-text sends {model, input} to <base>/embeddings.
+// A key, if the server needs one, comes from LIGATURA_API_KEY.
+function openaiEmbedder(spec) {
+    const [base, model] = spec.slice('openai:'.length).split('#');
+    if (!base || !model)
+        throw new Error('openai embedder needs openai:<base-url>#<model>');
+    return {
+        name: spec,
+        async embed(texts) {
+            const out = [];
+            for (let i = 0; i < texts.length; i += 64) {
+                const res = await fetch(`${base.replace(/\/$/, '')}/embeddings`, {
+                    method: 'POST',
+                    headers: { 'content-type': 'application/json', ...auth() },
+                    body: JSON.stringify({ model, input: texts.slice(i, i + 64) }),
+                });
+                if (!res.ok)
+                    throw new Error(`${base}/embeddings: ${res.status} ${await res.text()}`);
+                const body = (await res.json());
+                for (const d of body.data.sort((a, b) => a.index - b.index))
+                    out.push(normalize(Float32Array.from(d.embedding)));
+                log(`embed ${model}: ${Math.min(i + 64, texts.length)}/${texts.length}`, true);
+            }
+            return out;
+        },
+    };
+}
+export const auth = () => (process.env.LIGATURA_API_KEY ? { authorization: `Bearer ${process.env.LIGATURA_API_KEY}` } : {});
+function normalize(v) {
+    const n = Math.hypot(...v) || 1;
+    return v.map((x) => x / n);
+}

package/dist/eval.d.ts

@@ -0,0 +1,28 @@
+import { type LinkOptions } from './link.js';
+export interface EvalOptions extends LinkOptions {
+    neighbours?: number;
+}
+export declare function evaluate(dir: string, opts?: EvalOptions): Promise<{
+    gold: number;
+    scored: number;
+    missingTargets: string[];
+    fused: {
+        [k: string]: number;
+    };
+    final: {
+        [k: string]: number;
+    };
+    rows: {
+        source: string;
+        target: string;
+        fused: number;
+        final: number;
+    }[];
+    method: {
+        embed: string;
+        rerank: string;
+        pool: number;
+        neighbours: number;
+    };
+}>;
+export declare function formatEval(e: Awaited<ReturnType<typeof evaluate>>): string;

package/dist/eval.js

@@ -0,0 +1,59 @@
+// Recall against a gold set: known links between two texts. A fixture directory holds
+// source.jsonl and target.jsonl (units) and gold.jsonl ({"source", "target"} per line).
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { link, DEFAULTS } from './link.js';
+import { loadUnits } from './units.js';
+const KS = [1, 3, 5, 10];
+export async function evaluate(dir, opts = {}) {
+    const source = loadUnits(join(dir, 'source.jsonl'));
+    const target = loadUnits(join(dir, 'target.jsonl'));
+    const tIndex = new Map(target.map((u, i) => [u.ref, i]));
+    const gold = readFileSync(join(dir, 'gold.jsonl'), 'utf8').split('\n').filter((l) => l.trim()).map((l) => JSON.parse(l));
+    const missing = gold.filter((g) => !tIndex.has(g.target));
+    const usable = gold.filter((g) => tIndex.has(g.target));
+    const wanted = new Set(usable.map((g) => g.source));
+    const units = source.filter((u) => wanted.has(u.ref));
+    const near = opts.neighbours ?? 0;
+    const rows = [];
+    for await (const r of link(units, target, opts)) {
+        const finalRank = new Map(r.ranked.map((c, k) => [c.index, k + 1]));
+        for (const g of usable.filter((x) => x.source === r.source.ref)) {
+            const t = tIndex.get(g.target);
+            const hits = [...Array(2 * near + 1).keys()].map((d) => t + d - near).filter((j) => j >= 0 && j < target.length);
+            rows.push({
+                source: g.source,
+                target: g.target,
+                fused: Math.min(...hits.map((j) => r.fusedRank[j])),
+                // Outside the pool, a target keeps its fused rank.
+                final: Math.min(...hits.map((j) => finalRank.get(j) ?? r.fusedRank[j])),
+            });
+        }
+    }
+    const recall = (key, k) => rows.filter((x) => x[key] <= k).length;
+    return {
+        gold: gold.length,
+        scored: rows.length,
+        missingTargets: missing.map((g) => g.target),
+        fused: Object.fromEntries(KS.map((k) => [k, recall('fused', k)])),
+        final: Object.fromEntries(KS.map((k) => [k, recall('final', k)])),
+        rows,
+        method: { embed: opts.embed ?? DEFAULTS.embed, rerank: opts.rerank ?? DEFAULTS.rerank, pool: opts.pool ?? DEFAULTS.pool, neighbours: near },
+    };
+}
+export function formatEval(e) {
+    const head = `| Stage | ${KS.map((k) => `R@${k}`).join(' | ')} |`;
+    const line = (name, r) => `| ${name} | ${KS.map((k) => r[k]).join(' | ')} |`;
+    const misses = e.rows.filter((r) => r.final > 10).sort((a, b) => a.final - b.final);
+    return [
+        `Gold links: ${e.scored} scored of ${e.gold}${e.missingTargets.length ? ` (${e.missingTargets.length} point at targets not in target.jsonl)` : ''}. Neighbours counted: ${e.method.neighbours}.`,
+        `Embed ${e.method.embed}, rerank ${e.method.rerank} on the top ${e.method.pool}.`,
+        '',
+        head,
+        `|---|${KS.map(() => '---|').join('')}`,
+        line('Retrieval (embedding + overlap)', e.fused),
+        ...(e.method.rerank === 'none' ? [] : [line('Reranked', e.final)]),
+        '',
+        ...(misses.length ? [`Outside the top 10: ${misses.map((m) => `${m.source} → ${m.target} (${m.final})`).join('; ')}`] : []),
+    ].join('\n');
+}

package/dist/index.d.ts

@@ -0,0 +1,6 @@
+export { link, toLinks, DEFAULTS, type Link, type LinkOptions, type UnitResult } from './link.js';
+export { evaluate, formatEval } from './eval.js';
+export { loadUnits, splitText, type Unit } from './units.js';
+export { makeEmbedder, LOCAL_EMBEDDERS } from './embed.js';
+export { makeReranker } from './rerank.js';
+export { lexicalScorer, sharedWords } from './lexical.js';

package/dist/index.js

@@ -0,0 +1,6 @@
+export { link, toLinks, DEFAULTS } from './link.js';
+export { evaluate, formatEval } from './eval.js';
+export { loadUnits, splitText } from './units.js';
+export { makeEmbedder, LOCAL_EMBEDDERS } from './embed.js';
+export { makeReranker } from './rerank.js';
+export { lexicalScorer, sharedWords } from './lexical.js';

package/dist/lexical.d.ts

@@ -0,0 +1,5 @@
+export declare function words(s: string): Set<string>;
+export declare function sharedWords(a: string, b: string): string[];
+export declare function lexicalScorer(docs: string[]): {
+    scores: (query: string) => Float32Array<ArrayBuffer>;
+};

package/dist/lexical.js

@@ -0,0 +1,34 @@
+// Word overlap weighted by rarity (tf-idf cosine). Catches quotation, which embeddings blur.
+// The stop list and stem fold archaic English (thou, hath, followeth), common in older texts.
+const STOP = new Set('the and of to that in a is he for it not be his they all which with as but i you thou thee thy shall unto this them me my him are have will by from or so what was on no who when if there their we then do hath doth'.split(' '));
+export function words(s) {
+    return new Set(s.toLowerCase().match(/[a-z]+/g)?.filter((w) => w.length > 2 && !STOP.has(w)) ?? []);
+}
+const stem = (w) => w.replace(/(eth|est|ed|ing|s)$/, '');
+const stems = (s) => [...words(s)].map(stem).filter((w) => w.length > 2);
+export function sharedWords(a, b) {
+    const wb = words(b);
+    return [...words(a)].filter((w) => wb.has(w));
+}
+// Document frequencies come from the target texts; scores are cosines in [0, 1].
+export function lexicalScorer(docs) {
+    const df = new Map();
+    for (const d of docs)
+        for (const w of new Set(stems(d)))
+            df.set(w, (df.get(w) ?? 0) + 1);
+    const idf = (w) => Math.log((docs.length + 1) / ((df.get(w) ?? 0) + 1));
+    const vec = (s) => {
+        const v = new Map();
+        for (const w of stems(s))
+            v.set(w, (v.get(w) ?? 0) + idf(w));
+        return { v, norm: Math.hypot(...v.values()) || 1 };
+    };
+    const cos = (a, b) => {
+        let sum = 0;
+        for (const [w, x] of a.v)
+            sum += x * (b.v.get(w) ?? 0);
+        return sum / (a.norm * b.norm);
+    };
+    const targets = docs.map(vec);
+    return { scores: (query) => { const q = vec(query); return Float32Array.from(targets, (t) => cos(q, t)); } };
+}

package/dist/link.d.ts

@@ -0,0 +1,45 @@
+import type { Unit } from './units.js';
+export interface LinkOptions {
+    embed?: string;
+    rerank?: string;
+    task?: string;
+    pool?: number;
+}
+export interface Scored {
+    index: number;
+    dense: number;
+    lex: number;
+    fusedRank: number;
+    rerank?: number;
+}
+export interface UnitResult {
+    source: Unit;
+    ranked: Scored[];
+    fusedRank: Int32Array;
+}
+export declare const DEFAULTS: {
+    embed: string;
+    rerank: string;
+    pool: number;
+};
+export declare function link(source: Unit[], target: Unit[], opts?: LinkOptions): AsyncGenerator<UnitResult>;
+export interface Link {
+    source: string;
+    target: string;
+    rank: number;
+    score: number;
+    dense: number;
+    lex: number;
+    rerank?: number;
+    shared: string[];
+    status: 'suggested';
+    method: {
+        embed: string;
+        rerank: string;
+    };
+    date: string;
+}
+export declare function toLinks(r: UnitResult, target: Unit[], opts: LinkOptions & {
+    top: number;
+    minScore?: number;
+}): Link[];

package/dist/link.js

@@ -0,0 +1,60 @@
+// The pipeline: embed both texts, score every target unit two ways (embedding similarity and word
+// overlap), fuse the two rankings, then let a reranker read the top of the fused list.
+import { makeEmbedder, DEFAULT_EMBEDDER } from './embed.js';
+import { lexicalScorer, sharedWords } from './lexical.js';
+import { makeReranker, DEFAULT_RERANKER } from './rerank.js';
+import { dot, fuse, log, ranks, topK } from './util.js';
+export const DEFAULTS = { embed: DEFAULT_EMBEDDER, rerank: DEFAULT_RERANKER, pool: 30 };
+export async function* link(source, target, opts = {}) {
+    const embedder = makeEmbedder(opts.embed ?? DEFAULTS.embed);
+    const reranker = makeReranker(opts.rerank ?? DEFAULTS.rerank, opts.task);
+    const pool = opts.pool ?? DEFAULTS.pool;
+    log(`embedding ${target.length} target and ${source.length} source units with ${embedder.name}`);
+    const tVec = await embedder.embed(target.map((u) => u.text), 'doc');
+    const sVec = await embedder.embed(source.map((u) => u.text), 'query');
+    const lex = lexicalScorer(target.map((u) => u.text));
+    const started = Date.now();
+    for (const [i, s] of source.entries()) {
+        const dense = Float32Array.from(tVec, (t) => dot(sVec[i], t));
+        const lexical = lex.scores(s.text);
+        const fused = fuse(dense, lexical);
+        const fusedRank = ranks(fused);
+        const top = topK(fused, pool);
+        const ranked = top.map((j) => ({ index: j, dense: dense[j], lex: lexical[j], fusedRank: fusedRank[j] }));
+        if (reranker) {
+            // Candidates go to the reranker in target order, so the fused rank can't lead it.
+            const ordered = [...ranked].sort((a, b) => a.index - b.index);
+            const scores = await reranker.score(s, ordered.map((c) => target[c.index]));
+            ordered.forEach((c, k) => { c.rerank = scores[k]; });
+            ranked.sort((a, b) => b.rerank - a.rerank || a.fusedRank - b.fusedRank);
+        }
+        if (reranker) {
+            const each = (Date.now() - started) / (i + 1);
+            log(`rerank ${reranker.name}: ${i + 1}/${source.length}, about ${Math.round((each * (source.length - i - 1)) / 60000)} min left`, true);
+        }
+        yield { source: s, ranked, fusedRank };
+    }
+    if (reranker)
+        log(`reranked ${source.length} units in ${Math.round((Date.now() - started) / 1000)} s`);
+}
+// The links a person reviews: the top n of each pool, at or above a score if one is set.
+export function toLinks(r, target, opts) {
+    const date = new Date().toISOString().slice(0, 10);
+    const round = (x) => Math.round(x * 1000) / 1000;
+    return r.ranked
+        .slice(0, opts.top)
+        .filter((c) => opts.minScore === undefined || (c.rerank ?? c.dense) >= opts.minScore)
+        .map((c, k) => ({
+        source: r.source.ref,
+        target: target[c.index].ref,
+        rank: k + 1,
+        score: round(c.rerank ?? c.dense),
+        dense: round(c.dense),
+        lex: round(c.lex),
+        ...(c.rerank === undefined ? {} : { rerank: round(c.rerank) }),
+        shared: sharedWords(r.source.text, target[c.index].text),
+        status: 'suggested',
+        method: { embed: opts.embed ?? DEFAULTS.embed, rerank: opts.rerank ?? DEFAULTS.rerank },
+        date,
+    }));
+}

package/dist/rerank.d.ts

@@ -0,0 +1,8 @@
+import type { Unit } from './units.js';
+export interface Reranker {
+    name: string;
+    score(query: Unit, candidates: Unit[]): Promise<number[]>;
+}
+export declare const DEFAULT_RERANKER = "none";
+export declare const DEFAULT_TASK = "Given a passage from one text, decide whether it quotes, paraphrases or alludes to this passage from another text";
+export declare function makeReranker(spec: string, task?: string): Reranker | undefined;

package/dist/rerank.js

@@ -0,0 +1,122 @@
+// Rerank backends read each source unit with its candidate pool and score every candidate.
+//   qwen3-reranker-0.6b       in process (transformers.js, ONNX q8); no server, slowest
+//   api:<url>#<model>         a Cohere/Jina-shaped rerank endpoint: llama.cpp llama-server, vLLM, TEI, Infinity
+//   chat:<base-url>#<model>   an OpenAI-compatible chat model as judge: Ollama, LM Studio, mlx_lm.server
+//   none                      keep the retrieval order (default)
+import { auth } from './embed.js';
+import { log } from './util.js';
+// The q8 ONNX reranker scored below plain retrieval on the Kempis fixture (README, Results), so
+// reranking is opt-in until a model beats retrieval there.
+export const DEFAULT_RERANKER = 'none';
+export const DEFAULT_TASK = 'Given a passage from one text, decide whether it quotes, paraphrases or alludes to this passage from another text';
+export function makeReranker(spec, task = DEFAULT_TASK) {
+    if (spec === 'none')
+        return undefined;
+    if (spec === 'qwen3-reranker-0.6b')
+        return qwenLocal('onnx-community/Qwen3-Reranker-0.6B-ONNX', spec, task);
+    if (spec.startsWith('api:'))
+        return rerankApi(spec, task);
+    if (spec.startsWith('chat:'))
+        return chatJudge(spec, task);
+    throw new Error(`unknown reranker "${spec}"; use qwen3-reranker-0.6b, api:<url>#<model>, chat:<base-url>#<model> or none`);
+}
+// Qwen3-Reranker scores a pair as P("yes") for the next token after its fixed prompt.
+const QWEN_PREFIX = '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n';
+const QWEN_SUFFIX = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n';
+function qwenLocal(id, name, task) {
+    let tok;
+    let model;
+    const batch = Number(process.env.LIGATURA_BATCH ?? 4);
+    return {
+        name,
+        async score(query, candidates) {
+            if (!tok || !model) {
+                const t = await import('@huggingface/transformers');
+                tok = (await t.AutoTokenizer.from_pretrained(id));
+                model = (await t.AutoModelForCausalLM.from_pretrained(id, { dtype: 'q8' }));
+            }
+            const [yes] = tok.encode('yes', { add_special_tokens: false });
+            const [no] = tok.encode('no', { add_special_tokens: false });
+            const texts = candidates.map((c) => `${QWEN_PREFIX}<Instruct>: ${task}\n<Query>: ${query.text}\n<Document>: ${c.text}${QWEN_SUFFIX}`);
+            const out = [];
+            for (let i = 0; i < texts.length; i += batch) {
+                const enc = tok(texts.slice(i, i + batch), { padding: true });
+                const { logits } = await model(enc);
+                const [rows, len, vocab] = logits.dims;
+                const mask = enc.attention_mask;
+                for (let r = 0; r < rows; r++) {
+                    // The last real token of each row, whichever side the tokenizer pads.
+                    let last = len - 1;
+                    while (last > 0 && Number(mask.data[r * len + last]) === 0)
+                        last--;
+                    const base = (r * len + last) * vocab;
+                    out.push(1 / (1 + Math.exp(logits.data[base + no] - logits.data[base + yes])));
+                }
+            }
+            return out;
+        },
+    };
+}
+// api:http://localhost:8080/v1/rerank#qwen3-reranker sends {model, query, documents} and reads
+// results[].{index, relevance_score}, the shape Cohere, Jina, llama.cpp and vLLM share.
+function rerankApi(spec, task) {
+    const [url, model] = spec.slice('api:'.length).split('#');
+    if (!url)
+        throw new Error('api reranker needs api:<url>#<model>');
+    return {
+        name: spec,
+        async score(query, candidates) {
+            const res = await fetch(url, {
+                method: 'POST',
+                headers: { 'content-type': 'application/json', ...auth() },
+                body: JSON.stringify({ model, query: `${task}\n${query.text}`, documents: candidates.map((c) => c.text) }),
+            });
+            if (!res.ok)
+                throw new Error(`${url}: ${res.status} ${await res.text()}`);
+            const body = (await res.json());
+            const out = new Array(candidates.length).fill(0);
+            for (const r of body.results)
+                out[r.index] = r.relevance_score;
+            return out;
+        },
+    };
+}
+const JUDGE = (task, query, candidates) => `${task}.
+
+Below is a passage and candidate passages, listed in their own order, not by relevance. Pick the candidates the passage quotes, paraphrases, or alludes to: it takes their words, image, story or teaching. A shared common word or a general theme is NOT enough. Judge only from the texts given.
+
+Return up to 5 refs, strongest first, or an empty list. Only use refs from the candidates. Reply with JSON only: {"refs": ["<ref>"], "why": "one short line per ref, ';'-separated, naming quote/paraphrase/allusion"}
+
+Passage: ${query.text}
+
+Candidates:
+${candidates.map((c) => `${c.ref}: ${c.text}`).join('\n')}`;
+// chat:http://localhost:11434/v1#qwen3:8b asks the model to pick; picks score 1, 0.99, ...; the
+// rest 0. Candidates go in their target order so retrieval rank doesn't lead the model.
+function chatJudge(spec, task) {
+    const [base, model] = spec.slice('chat:'.length).split('#');
+    if (!base || !model)
+        throw new Error('chat judge needs chat:<base-url>#<model>');
+    return {
+        name: spec,
+        async score(query, candidates) {
+            const res = await fetch(`${base.replace(/\/$/, '')}/chat/completions`, {
+                method: 'POST',
+                headers: { 'content-type': 'application/json', ...auth() },
+                body: JSON.stringify({ model, temperature: 0, messages: [{ role: 'user', content: JUDGE(task, query, candidates) }] }),
+            });
+            if (!res.ok)
+                throw new Error(`${base}/chat/completions: ${res.status} ${await res.text()}`);
+            const body = (await res.json());
+            const text = body.choices[0]?.message.content.replace(/<think>[\s\S]*?<\/think>/, '') ?? '';
+            let refs = [];
+            try {
+                refs = JSON.parse(text.match(/\{[\s\S]*\}/)?.[0] ?? '{}').refs ?? [];
+            }
+            catch {
+                log(`judge: unreadable reply for ${query.ref}`);
+            }
+            return candidates.map((c) => { const k = refs.indexOf(c.ref); return k < 0 ? 0 : 1 - k / 100; });
+        },
+    };
+}

package/dist/units.d.ts

@@ -0,0 +1,7 @@
+export interface Unit {
+    ref: string;
+    text: string;
+}
+export type Split = 'auto' | 'sentence' | 'paragraph' | 'line';
+export declare function loadUnits(path: string, split?: Split): Unit[];
+export declare function splitText(raw: string, split?: Split): string[];

package/dist/units.js

@@ -0,0 +1,27 @@
+// Units are the pieces a text is linked by: a verse, a sentence, a paragraph. Each has a stable ref.
+import { readFileSync } from 'node:fs';
+// JSONL ({"ref", "text"} per line) loads as is. Plain text splits into units numbered from 1:
+// "auto" means sentences.
+export function loadUnits(path, split = 'auto') {
+    const raw = readFileSync(path, 'utf8');
+    if (path.endsWith('.jsonl')) {
+        return raw.split('\n').filter((l) => l.trim()).map((l, i) => {
+            const o = JSON.parse(l);
+            const ref = o.ref ?? o.id;
+            if (ref === undefined || typeof o.text !== 'string')
+                throw new Error(`${path}:${i + 1}: each line needs "ref" and "text"`);
+            return { ref: String(ref), text: o.text };
+        });
+    }
+    return splitText(raw, split).map((text, i) => ({ ref: String(i + 1), text }));
+}
+export function splitText(raw, split = 'auto') {
+    const flat = (s) => s.replace(/\s+/g, ' ').trim();
+    if (split === 'line')
+        return raw.split('\n').map(flat).filter(Boolean);
+    const paras = raw.split(/\n\s*\n/).map(flat).filter(Boolean);
+    if (split === 'paragraph')
+        return paras;
+    // A sentence ends at . ; : ? or ! followed by a space and a capital or an opening quote.
+    return paras.flatMap((p) => p.split(/(?<=[.;:?!])\s+(?=[A-Z“"'‘])/)).map(flat).filter((s) => s.split(' ').length >= 3);
+}

package/dist/util.d.ts

@@ -0,0 +1,8 @@
+export declare const cacheDir: () => string;
+export declare const setQuiet: (q: boolean) => void;
+export declare function log(msg: string, progress?: boolean): void;
+export declare const dot: (a: Float32Array, b: Float32Array) => number;
+export declare function ranks(scores: Float32Array): Int32Array;
+export declare const RRF_K = 60;
+export declare function fuse(...lists: Float32Array[]): Float32Array;
+export declare const topK: (scores: Float32Array, k: number) => number[];

package/dist/util.js

@@ -0,0 +1,34 @@
+import { homedir } from 'node:os';
+import { join } from 'node:path';
+export const cacheDir = () => process.env.LIGATURA_CACHE ?? join(process.env.XDG_CACHE_HOME ?? join(homedir(), '.cache'), 'ligatura');
+let quiet = false;
+export const setQuiet = (q) => { quiet = q; };
+// Progress goes to stderr so stdout stays clean JSONL. Progress lines overwrite each other on a TTY.
+export function log(msg, progress = false) {
+    if (quiet)
+        return;
+    if (progress && process.stderr.isTTY)
+        process.stderr.write(`\r\x1b[K${msg}`);
+    else if (!progress)
+        process.stderr.write(`${process.stderr.isTTY ? '\r\x1b[K' : ''}${msg}\n`);
+}
+export const dot = (a, b) => {
+    let s = 0;
+    for (let i = 0; i < a.length; i++)
+        s += a[i] * b[i];
+    return s;
+};
+// Ranks from scores: 1 is best. Ties keep input order.
+export function ranks(scores) {
+    const order = [...scores.keys()].sort((a, b) => scores[b] - scores[a]);
+    const r = new Int32Array(scores.length);
+    order.forEach((j, i) => { r[j] = i + 1; });
+    return r;
+}
+export const RRF_K = 60;
+// Reciprocal rank fusion: each list adds 1 / (60 + rank).
+export function fuse(...lists) {
+    const rs = lists.map(ranks);
+    return Float32Array.from(rs[0], (_, j) => rs.reduce((s, r) => s + 1 / (RRF_K + r[j]), 0));
+}
+export const topK = (scores, k) => [...scores.keys()].sort((a, b) => scores[b] - scores[a]).slice(0, k);

package/package.json

@@ -1,8 +1,30 @@
 {
   "name": "ligatura",
-  "version": "0.0.1",
-  "description": "Ligature tool. Name reserved; first release coming.",
+  "version": "0.1.0",
+  "description": "Find where one text quotes, paraphrases or alludes to another. Local models by default; every link is a suggestion for a person to confirm.",
+  "keywords": ["intertextuality", "allusion", "text-reuse", "digital-humanities", "embeddings", "reranker", "bible", "cli"],
   "author": "Bret Perry",
   "license": "MIT",
-  "files": ["README.md"]
+  "repository": { "type": "git", "url": "git+https://github.com/bretperry/ligatura.git" },
+  "homepage": "https://github.com/bretperry/ligatura#readme",
+  "bugs": "https://github.com/bretperry/ligatura/issues",
+  "type": "module",
+  "bin": { "ligatura": "dist/cli.js" },
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": { ".": "./dist/index.js" },
+  "files": ["dist", "README.md", "LICENSE"],
+  "engines": { "node": ">=20" },
+  "scripts": {
+    "build": "tsc",
+    "test": "tsc && node --test 'test/*.test.js'",
+    "prepublishOnly": "npm test"
+  },
+  "dependencies": {
+    "@huggingface/transformers": "^3.8.1"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "typescript": "^5.9.0"
+  }
 }