@concordance-wiki/inference 0.1.0

package/README.md

@@ -0,0 +1,15 @@
+# @concordance-wiki/inference
+
+Link production, confidence combination, relation typing, bounded neighbourhood, term candidates.
+
+Today: explicit links (`explicitLinks`): every markdown link written in a note that resolves to another note gives a link at the `explicit_link` confidence of the profile, with the file, the line, the link text and the anchor as provenance; a link to a non-markdown file gives a `documents` link from the resource to the note; a missing target is `E-LINK-BROKEN`; a target in another source, written `<source>:<path>` or as a relative path climbing into a sibling source, resolves when `inference.cross_source_links` allows it and is `W-LINK-CROSS-SOURCE` otherwise. Links from one note to the same target are merged with every provenance kept; a link between two notes is `related` until the relation typing step names it. Bounded neighbourhood (`accumulateCooccurrences`, `neighbourhoodOptions`): two entities named in the same paragraph are neighbours, counted per paragraph; every node keeps its K best neighbours by count then by identifier (`inference.neighbours.k`, 50 by default), its row trimmed whenever it grows past 2K so that the full matrix is never held, and a paragraph naming more than 200 entities pairs only the first 200. `cooccurrenceLinks` turns every pair into one `related` link at the `cooccurrence` confidence with the number of shared paragraphs as provenance; `neighbourhoodToModel` writes the `neighbours` block of `model.json`. Confidence combination (`combineLinks`, `combineOptions`): one link per source, target, relation and attributes, at `1 − Π(1 − cᵢ)` over its methods (`combineConfidences`, clamped to [0, 1] and rounded to four decimals), the glossary occurrences of a group counting as one method at the base of the first plus `per_occurrence` per additional occurrence up to `cap` (`glossaryConfidence`); every provenance is kept in canonical order, an exact duplicate once, and the function is pure and idempotent. Displayed neighbourhood (`displayedNeighbourhood`, `displayOptions`): the one-hop neighbours of every entity through the links of the model in either direction, typed entities and noteless keyword pages alike, each flagged by its `kind`, carrying the largest confidence and the relation of the most confident link between the two nodes, its `direction` seen from the entity and its `rank`, the position of its type in the `display.neighbours_order` the profile declares for the entity's type (types not listed, keyword pages included, share the rank after the last one); sorted by rank, then by decreasing confidence, then by identifier, and truncated after that to `site.neighbourhood.size` (6 by default, never more than `MAX_DISPLAYED_NEIGHBOURS`, 12), so that the nodes shown are the best of the priority order; a type without a declaration sorts by confidence alone; an entity without any neighbour has an empty list. `displayedNeighbourhoodToModel` writes the `displayed_neighbourhood` block of `model.json`, precomputed at build so that the browser never computes it. Confidence combination (`combineLinks`, `combineOptions`): one link per source, target, relation and attributes, at `1 − Π(1 − cᵢ)` over its methods (`combineConfidences`, clamped to [0, 1] and rounded to four decimals), the glossary occurrences of a group counting as one method at the base of the strongest plus `per_occurrence` per additional occurrence up to `cap` (`glossaryConfidence`); every provenance is kept in canonical order, an exact duplicate once, and the function is pure and idempotent. Displayed neighbourhood (`displayedNeighbourhood`, `displayOptions`): the one-hop neighbours of every entity through the links of the model in either direction, typed entities and noteless keyword pages alike, each flagged by its `kind`, carrying the largest confidence and the relation of the most confident link between the two nodes and its `direction` seen from the entity; sorted by decreasing confidence then by identifier and truncated to `site.neighbourhood.size` (6 by default, never more than `MAX_DISPLAYED_NEIGHBOURS`, 12); an entity without any neighbour has an empty list. `displayedNeighbourhoodToModel` writes the `displayed_neighbourhood` block of `model.json`, precomputed at build so that the browser never computes it.
+
+Frontmatter references (`frontmatterLinks`): every reference-typed attribute of the profile that carries a relation (`reads`, `writes`, `rules`, `roles`, `consumers`, `affects`, `broader`, `business_object`, ...) turns each of its values into a link of that relation at the `frontmatter_ref` confidence, with the attributes the profile declares (`accesses` in `read` mode for `reads`), reversed when the attribute is `inverse`, and the file, line 1 and the attribute name as provenance. A value is resolved by identifier (full, or relative to the source of the note), then by source-relative path, then by exact title (`resolveReference` over an `indexEntities` index); a value that matches nothing, a title shared by several notes or a value that is not a string is `W-REF-UNRESOLVED` and gives no link; a note of a type the attribute does not accept gives its link, which the relation typing step drops with `E-META-REL` when the matrix forbids the relation. Several values naming the same target are merged with every provenance kept.
+
+Relation typing (`typeRelations`): once every producer has spoken and their links are combined, names the relation of every link from the profile on four rungs, mapped section, typed frontmatter attribute, type pair admitting a single relation, `related`. The first two stand as produced and are only checked against the profile matrix: a relation the profile does not allow between the two types, or does not declare, drops the link with `E-META-REL` on the file, line and attribute of its provenance. A `related` link whose ordered type pair admits exactly one relation besides the wildcards takes it, marked `relation_origin: pair` in its attributes, turned around when only the reverse pair admits one; otherwise it stays `related`, capped at `relations.related.cap` (0.60) after the combination, and yields one `I-REL-AMBIGUOUS` finding on its first provenance that names a file; a `related` link known from co-occurrences alone yields none, since a co-occurrence is unnamed by nature and locates nothing. Links that then say the same thing are combined again, a pair-named link joining a declared link of the same triple and attributes and losing its marker; undirected relations go from the smaller identifier; a link with an endpoint the entity list does not type is kept as produced. `relationLabel(profile, relation, locale, { inverse })` gives the label the site displays, `label` or `inverse_label` of the profile, never a string of the code. Both are pure, deterministic and, for the step, idempotent.
+
+Mentions (`mentionLinks`): every occurrence of a note's title or alias read in another note gives a link. Under an H2 section that the profile maps for the type of the note (`mappedSection`: the heading is compared with the section key and with its label in every locale, ignoring case, accents and whitespace), the link carries the relation the section produces, its attributes and its direction (`inverse` sections link the mentioned entity to the note), at the `section_mention` confidence with the section key and the line as provenance; when the type pair does not admit that relation, or outside a mapped section, the mention gives a `related` link at the confidence the scan gave the occurrence (the base `glossary_occurrence` confidence, plus the type prefix bonus, halved for a homonym; the base alone when the occurrence carries none) with the line as provenance, oriented from the smaller identifier since `related` is undirected. When the occurrence carries its context, the provenance keeps it under `occurrences` (line, position, context, heading) with the matched text as written under `text`, so that the mentions panel of the site can show the passage and mark the words naming the entity. Self-mentions are dropped; mentions of one target in one relation merge into one link with a provenance per mention. The package declares the occurrence shape it reads structurally and does not depend on the scan. Twin resources (`resolveDuplicateResources`, `duplicateOptions`): every pair of resources is scored by adding its signals, capped at 1 (frontmatter declaration 1.0, same base name 0.7 in the same folder or 0.5 elsewhere, close base names at Jaro-Winkler 0.9 times 0.8, property title equal to the heading 0.6, similar text 0.7 from a Jaccard index of 0.8 or 0.4 from 0.6, same commit 0.3, folder proximity up to 0.2); above `merge_above` the pair merges into one group with every representation and the criterion that grouped it, from `candidate_above` it yields `W-DUP-CANDIDATE`. Text similarity runs on the words the caller provides through `normalizeText` (the comparison form of the language pack, stopwords removed): 5-word shingles, MinHash of 128 functions under a fixed seed, LSH banding so that only the pairs sharing a band are compared, exact Jaccard recomputed according to `inference.duplicates.mode`; the lock file's `merged` and `separated` pairs override the score; `DuplicateStats` counts the pairs brought together by the banding, the pairs scored, the exact verifications and the time spent, and `formatDuplicateStats` gives the build summary its lines.
+
+Operation notes (`attachOperations`): every hand-written `endpoint` note is matched to an operation imported from the contract of its API, on the `operation_id` of its frontmatter, then on its `method` and `path` (or `port` and title for a SOAP operation), then on its title in comparison form with spaces and punctuation removed, against the operation title or identifier; the note names its API through the reference attribute of the profile (`api`), through a link already recorded with the API note, or is a candidate for every API of its source that has imported operations. A note stops at the first rung that matches, and an operation matched at a stronger rung is no longer offered to the weaker ones. A matched pair merges into the note, which keeps its identifier, markdown and frontmatter, takes the contract attributes it does not set, the operation's aliases and summary when it has none, lists the contract as a representation `{ kind: "contract", path, operation }` next to its own file (after any twin resources already merged) and names the rung in `grouped_by`; the imported operation disappears and every link that named it, the `exposes` link of the API first of all, now names the note. A note matching several operations, or an operation claimed by several notes, yields `W-OPERATION-AMBIGUOUS` naming every candidate and attaches nothing; a note that names an API with an imported contract and matches none of its operations yields `W-OPERATION-UNMATCHED`, saying that the operation left the contract or that the note is ahead of it, while a note that names no API is left alone. The step is pure and deterministic and returns sorted entities, links and findings; in the pipeline it runs after the sources step, which imports the contracts, and before the mention scan, so that the merged note is the one the dictionary and the sections link.
+
+Part of [Concordance](../../README.md).

package/dist/combine/confidence.d.ts

@@ -0,0 +1,16 @@
+/** How an extra glossary occurrence raises the confidence, and where it stops. */
+export interface GlossaryCombination {
+    bonus: number;
+    cap: number;
+}
+/**
+ * The confidence that at least one of several independent methods is right: `1 − Π(1 − cᵢ)`,
+ * clamped to [0, 1] and rounded to four decimals. No method gives no confidence.
+ */
+export declare function combineConfidences(values: readonly number[]): number;
+/**
+ * The confidence of a term mentioned `occurrences` times (at least once) in one document: the base
+ * plus one bonus per additional occurrence, never above the cap, rounded to four decimals.
+ */
+export declare function glossaryConfidence(base: number, occurrences: number, options: GlossaryCombination): number;
+//# sourceMappingURL=confidence.d.ts.map

package/dist/combine/confidence.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"confidence.d.ts","sourceRoot":"","sources":["../../src/combine/confidence.ts"],"names":[],"mappings":"AAAA,kFAAkF;AAClF,MAAM,WAAW,mBAAmB;IAClC,KAAK,EAAE,MAAM,CAAC;IACd,GAAG,EAAE,MAAM,CAAC;CACb;AAUD;;;GAGG;AACH,wBAAgB,kBAAkB,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,MAAM,CAIpE;AAED;;;GAGG;AACH,wBAAgB,kBAAkB,CAChC,IAAI,EAAE,MAAM,EACZ,WAAW,EAAE,MAAM,EACnB,OAAO,EAAE,mBAAmB,GAC3B,MAAM,CAER"}

package/dist/combine/confidence.js

@@ -0,0 +1,24 @@
+// Four decimals: `model.json` serialises the number and a build must not differ by a floating-point
+// tail such as `0.96` written `0.9600000000000001`.
+const SCALE = 10_000;
+function round(value) {
+    return Math.round(value * SCALE) / SCALE;
+}
+/**
+ * The confidence that at least one of several independent methods is right: `1 − Π(1 − cᵢ)`,
+ * clamped to [0, 1] and rounded to four decimals. No method gives no confidence.
+ */
+export function combineConfidences(values) {
+    let miss = 1;
+    for (const value of values)
+        miss *= 1 - value;
+    return round(Math.min(1, Math.max(0, 1 - miss)));
+}
+/**
+ * The confidence of a term mentioned `occurrences` times (at least once) in one document: the base
+ * plus one bonus per additional occurrence, never above the cap, rounded to four decimals.
+ */
+export function glossaryConfidence(base, occurrences, options) {
+    return round(Math.min(base + options.bonus * (occurrences - 1), options.cap));
+}
+//# sourceMappingURL=confidence.js.map

package/dist/combine/confidence.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"confidence.js","sourceRoot":"","sources":["../../src/combine/confidence.ts"],"names":[],"mappings":"AAMA,oGAAoG;AACpG,oDAAoD;AACpD,MAAM,KAAK,GAAG,MAAM,CAAC;AAErB,SAAS,KAAK,CAAC,KAAa;IAC1B,OAAO,IAAI,CAAC,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC,GAAG,KAAK,CAAC;AAC3C,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,kBAAkB,CAAC,MAAyB;IAC1D,IAAI,IAAI,GAAG,CAAC,CAAC;IACb,KAAK,MAAM,KAAK,IAAI,MAAM;QAAE,IAAI,IAAI,CAAC,GAAG,KAAK,CAAC;IAC9C,OAAO,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;AACnD,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,kBAAkB,CAChC,IAAY,EACZ,WAAmB,EACnB,OAA4B;IAE5B,OAAO,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,GAAG,OAAO,CAAC,KAAK,GAAG,CAAC,WAAW,GAAG,CAAC,CAAC,EAAE,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC;AAChF,CAAC"}

package/dist/combine/links.d.ts

@@ -0,0 +1,15 @@
+import { type Link } from "@concordance-wiki/core";
+import type { Profile } from "@concordance-wiki/profile";
+import { type GlossaryCombination } from "./confidence.js";
+export interface CombineOptions {
+    glossary: GlossaryCombination;
+}
+/** The bonus and cap of `confidence.glossary_occurrence` in the profile, 0.05 and 0.80 when unset. */
+export declare function combineOptions(profile: Profile): CombineOptions;
+/**
+ * One link per source-target-relation-attributes group with the confidence that at least one of
+ * its methods is right and every provenance it received, exact duplicates listed once. The input
+ * is left untouched; feeding the output back gives the same output.
+ */
+export declare function combineLinks(links: readonly Link[], options: CombineOptions): Link[];
+//# sourceMappingURL=links.d.ts.map

package/dist/combine/links.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"links.d.ts","sourceRoot":"","sources":["../../src/combine/links.ts"],"names":[],"mappings":"AAAA,OAAO,EAGL,KAAK,IAAI,EAEV,MAAM,wBAAwB,CAAC;AAChC,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,2BAA2B,CAAC;AAGzD,OAAO,EAA0C,KAAK,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AAEnG,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,mBAAmB,CAAC;CAC/B;AAUD,sGAAsG;AACtG,wBAAgB,cAAc,CAAC,OAAO,EAAE,OAAO,GAAG,cAAc,CAM/D;AAsED;;;;GAIG;AACH,wBAAgB,YAAY,CAAC,KAAK,EAAE,SAAS,IAAI,EAAE,EAAE,OAAO,EAAE,cAAc,GAAG,IAAI,EAAE,CAiBpF"}

package/dist/combine/links.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"links.js","sourceRoot":"","sources":["../../src/combine/links.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,YAAY,EACZ,kBAAkB,GAGnB,MAAM,wBAAwB,CAAC;AAGhC,OAAO,EAAE,UAAU,EAAE,MAAM,2BAA2B,CAAC;AACvD,OAAO,EAAE,kBAAkB,EAAE,kBAAkB,EAA4B,MAAM,iBAAiB,CAAC;AAYnG,MAAM,gBAAgB,GAAwB,EAAE,KAAK,EAAE,IAAI,EAAE,GAAG,EAAE,GAAG,EAAE,CAAC;AAExE,sGAAsG;AACtG,MAAM,UAAU,cAAc,CAAC,OAAgB;IAC7C,MAAM,KAAK,GAAG,OAAO,CAAC,UAAU,CAAC,mBAAmB,CAAC;IACrD,OAAO;QACL,QAAQ,EACN,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,gBAAgB,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,KAAK,CAAC,cAAc,EAAE,GAAG,EAAE,KAAK,CAAC,GAAG,EAAE;KAC3F,CAAC;AACJ,CAAC;AAED,yGAAyG;AACzG,SAAS,mBAAmB,CAAC,UAA+C;IAC1E,MAAM,OAAO,GAAG,MAAM,CAAC,OAAO,CAAC,UAAU,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IACtF,OAAO,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,WAAW,CAAC,OAAO,CAAC,CAAC,CAAC;AACrD,CAAC;AAED,SAAS,QAAQ,CAAC,IAAU;IAC1B,OAAO,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,EAAE,EAAE,IAAI,CAAC,QAAQ,EAAE,mBAAmB,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAC7F,CAAC;AAED,+EAA+E;AAC/E,SAAS,aAAa,CAAC,UAAsB;IAC3C,OAAO;QACL,UAAU,CAAC,MAAM;QACjB,UAAU,CAAC,IAAI,IAAI,EAAE;QACrB,MAAM,CAAC,UAAU,CAAC,IAAI,IAAI,EAAE,CAAC;QAC7B,UAAU,CAAC,OAAO,IAAI,EAAE;KACzB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AACd,CAAC;AAED,mGAAmG;AACnG,kCAAkC;AAClC,SAAS,YAAY,CAAC,CAAa,EAAE,CAAa;IAChD,OAAO,kBAAkB,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,UAAU,CAAC,CAAC,CAAC,OAAO,IAAI,EAAE,EAAE,CAAC,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC;AAClF,CAAC;AAED,sGAAsG;AACtG,SAAS,aAAa,CAAC,UAAsB;IAC3C,MAAM,MAAM,GAAG,UAAU,CAAC,WAAW,EAAE,MAAM,IAAI,CAAC,CAAC;IACnD,OAAO,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC;AACnC,CAAC;AAED;;;;;GAKG;AACH,SAAS,iBAAiB,CAAC,WAAkC,EAAE,OAAuB;IACpF,MAAM,MAAM,GAAa,EAAE,CAAC;IAC5B,IAAI,YAAgC,CAAC;IACrC,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,KAAK,MAAM,UAAU,IAAI,WAAW,EAAE,CAAC;QACrC,IAAI,UAAU,CAAC,MAAM,KAAK,qBAAqB,EAAE,CAAC;YAChD,MAAM,CAAC,IAAI,CAAC,UAAU,CAAC,UAAU,CAAC,CAAC;YACnC,SAAS;QACX,CAAC;QACD,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,YAAY,IAAI,CAAC,EAAE,UAAU,CAAC,UAAU,CAAC,CAAC;QAClE,WAAW,IAAI,aAAa,CAAC,UAAU,CAAC,CAAC;IAC3C,CAAC;IACD,IAAI,YAAY,KAAK,SAAS,EAAE,CAAC;QAC/B,MAAM,CAAC,IAAI,CAAC,kBAAkB,CAAC,YAAY,EAAE,WAAW,EAAE,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC;IAC/E,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,YAAY,CAAC,KAAY,EAAE,OAAuB;IACzD,MAAM,UAAU,GAAG,CAAC,GAAG,KAAK,CAAC,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;IACtE,OAAO;QACL,IAAI,EAAE,KAAK,CAAC,KAAK,CAAC,IAAI;QACtB,EAAE,EAAE,KAAK,CAAC,KAAK,CAAC,EAAE;QAClB,QAAQ,EAAE,KAAK,CAAC,KAAK,CAAC,QAAQ;QAC9B,UAAU,EAAE,KAAK,CAAC,KAAK,CAAC,UAAU,IAAI,EAAE;QACxC,UAAU,EAAE,kBAAkB,CAAC,iBAAiB,CAAC,UAAU,EAAE,OAAO,CAAC,CAAC;QACtE,UAAU;KACX,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,YAAY,CAAC,KAAsB,EAAE,OAAuB;IAC1E,MAAM,MAAM,GAAG,IAAI,GAAG,EAAiB,CAAC;IACxC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC;QAC3B,IAAI,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QAC5B,IAAI,KAAK,KAAK,SAAS,EAAE,CAAC;YACxB,KAAK,GAAG,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,EAAE,WAAW,EAAE,IAAI,GAAG,EAAE,EAAE,CAAC;YACrD,MAAM,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;QACzB,CAAC;QACD,KAAK,MAAM,UAAU,IAAI,IAAI,CAAC,UAAU,EAAE,CAAC;YACzC,MAAM,EAAE,GAAG,aAAa,CAAC,UAAU,CAAC,CAAC;YACrC,IAAI,CAAC,KAAK,CAAC,WAAW,CAAC,GAAG,CAAC,EAAE,CAAC;gBAAE,KAAK,CAAC,WAAW,CAAC,GAAG,CAAC,EAAE,EAAE,EAAE,GAAG,UAAU,EAAE,CAAC,CAAC;QAC/E,CAAC;IACH,CAAC;IACD,OAAO,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC;SACxB,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,YAAY,CAAC,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,KAAK,CAAC,IAAI,UAAU,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC;SAC1E,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,YAAY,CAAC,KAAK,EAAE,OAAO,CAAC,CAAC,CAAC;AAClD,CAAC"}

package/dist/display/neighbourhood.d.ts

@@ -0,0 +1,15 @@
+import type { DisplayedNeighbourhood, DisplayedNeighbourhoodInput } from "./types.js";
+/** The mini-map stays legible at twelve labelled nodes; the configuration schema stops there too. */
+export declare const MAX_DISPLAYED_NEIGHBOURS = 12;
+/**
+ * The one-hop neighbours of every entity through the links in either direction. A neighbour
+ * reached by several links shows the relation of the most confident one; the merge is
+ * commutative, so the result depends on the set of links alone. A link whose end is not an
+ * entity of the model, or which loops on its node, shows nothing. The neighbours of a page come
+ * grouped by the priority of their type for the page's type (`display.neighbours_order` of the
+ * profile), best first within a group, and the truncation applies after the grouping: the nodes
+ * shown are the best of the priority order, not the most confident overall. A type without a
+ * declaration lists its neighbours by decreasing confidence alone.
+ */
+export declare function displayedNeighbourhood(input: DisplayedNeighbourhoodInput): DisplayedNeighbourhood;
+//# sourceMappingURL=neighbourhood.d.ts.map

package/dist/display/neighbourhood.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"neighbourhood.d.ts","sourceRoot":"","sources":["../../src/display/neighbourhood.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAGV,sBAAsB,EACtB,2BAA2B,EAE5B,MAAM,YAAY,CAAC;AAEpB,qGAAqG;AACrG,eAAO,MAAM,wBAAwB,KAAK,CAAC;AA6D3C;;;;;;;;;GASG;AACH,wBAAgB,sBAAsB,CAAC,KAAK,EAAE,2BAA2B,GAAG,sBAAsB,CA8BjG"}

package/dist/display/neighbourhood.js

@@ -0,0 +1,83 @@
+import { neighbourOrder } from "@concordance-wiki/profile";
+import { byCodeUnit } from "../neighbourhood/order.js";
+/** The mini-map stays legible at twelve labelled nodes; the configuration schema stops there too. */
+export const MAX_DISPLAYED_NEIGHBOURS = 12;
+/** Best first: the lower rank, then the larger confidence, then the lower identifier. */
+function compareDisplayed(a, b) {
+    return a.rank - b.rank || b.confidence - a.confidence || byCodeUnit(a.id, b.id);
+}
+/** The rank of every neighbour type for a page of the given type; unlisted types share the last rank. */
+function rankOf(order) {
+    return (type) => {
+        const index = order.indexOf(type);
+        return index === -1 ? order.length : index;
+    };
+}
+function merge(row, entity, link, direction) {
+    const existing = row.get(entity.id);
+    if (existing === undefined) {
+        row.set(entity.id, { entity, relation: link.relation, direction, confidence: link.confidence });
+        return;
+    }
+    if (link.confidence > existing.confidence ||
+        (link.confidence === existing.confidence && link.relation < existing.relation)) {
+        existing.relation = link.relation;
+        existing.confidence = link.confidence;
+    }
+    if (existing.direction !== direction) {
+        existing.direction = "both";
+    }
+}
+function toDisplayed({ entity, relation, direction, confidence }, rank) {
+    return {
+        id: entity.id,
+        title: entity.title,
+        type: entity.type,
+        kind: entity.keyword === true ? "keyword" : "entity",
+        relation,
+        direction,
+        confidence,
+        rank: rank(entity.type),
+    };
+}
+/**
+ * The one-hop neighbours of every entity through the links in either direction. A neighbour
+ * reached by several links shows the relation of the most confident one; the merge is
+ * commutative, so the result depends on the set of links alone. A link whose end is not an
+ * entity of the model, or which loops on its node, shows nothing. The neighbours of a page come
+ * grouped by the priority of their type for the page's type (`display.neighbours_order` of the
+ * profile), best first within a group, and the truncation applies after the grouping: the nodes
+ * shown are the best of the priority order, not the most confident overall. A type without a
+ * declaration lists its neighbours by decreasing confidence alone.
+ */
+export function displayedNeighbourhood(input) {
+    const size = Math.min(input.size, MAX_DISPLAYED_NEIGHBOURS);
+    const entities = new Map(input.entities.map((entity) => [entity.id, entity]));
+    const rows = new Map();
+    const rowOf = (id) => {
+        let row = rows.get(id);
+        if (row === undefined) {
+            row = new Map();
+            rows.set(id, row);
+        }
+        return row;
+    };
+    for (const link of input.links) {
+        const from = entities.get(link.from);
+        const to = entities.get(link.to);
+        if (from === undefined || to === undefined || from === to) {
+            continue;
+        }
+        merge(rowOf(from.id), to, link, "out");
+        merge(rowOf(to.id), from, link, "in");
+    }
+    const nodes = new Map();
+    for (const [id, entity] of [...entities].sort(([a], [b]) => byCodeUnit(a, b))) {
+        const rank = rankOf(neighbourOrder(input.profile, entity.type));
+        const row = rows.get(id);
+        const neighbours = row === undefined ? [] : [...row.values()].map((neighbour) => toDisplayed(neighbour, rank));
+        nodes.set(id, neighbours.toSorted(compareDisplayed).slice(0, size));
+    }
+    return nodes;
+}
+//# sourceMappingURL=neighbourhood.js.map

package/dist/display/neighbourhood.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"neighbourhood.js","sourceRoot":"","sources":["../../src/display/neighbourhood.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAE3D,OAAO,EAAE,UAAU,EAAE,MAAM,2BAA2B,CAAC;AASvD,qGAAqG;AACrG,MAAM,CAAC,MAAM,wBAAwB,GAAG,EAAE,CAAC;AAS3C,yFAAyF;AACzF,SAAS,gBAAgB,CAAC,CAAqB,EAAE,CAAqB;IACpE,OAAO,CAAC,CAAC,IAAI,GAAG,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,UAAU,GAAG,CAAC,CAAC,UAAU,IAAI,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;AAClF,CAAC;AAED,yGAAyG;AACzG,SAAS,MAAM,CAAC,KAAwB;IACtC,OAAO,CAAC,IAAI,EAAE,EAAE;QACd,MAAM,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;QAClC,OAAO,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC;IAC7C,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,KAAK,CACZ,GAA6B,EAC7B,MAAyB,EACzB,IAAU,EACV,SAA6B;IAE7B,MAAM,QAAQ,GAAG,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IACpC,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;QAC3B,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,SAAS,EAAE,UAAU,EAAE,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;QAChG,OAAO;IACT,CAAC;IACD,IACE,IAAI,CAAC,UAAU,GAAG,QAAQ,CAAC,UAAU;QACrC,CAAC,IAAI,CAAC,UAAU,KAAK,QAAQ,CAAC,UAAU,IAAI,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC,QAAQ,CAAC,EAC9E,CAAC;QACD,QAAQ,CAAC,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC;QAClC,QAAQ,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;IACxC,CAAC;IACD,IAAI,QAAQ,CAAC,SAAS,KAAK,SAAS,EAAE,CAAC;QACrC,QAAQ,CAAC,SAAS,GAAG,MAAM,CAAC;IAC9B,CAAC;AACH,CAAC;AAED,SAAS,WAAW,CAClB,EAAE,MAAM,EAAE,QAAQ,EAAE,SAAS,EAAE,UAAU,EAAe,EACxD,IAA8B;IAE9B,OAAO;QACL,EAAE,EAAE,MAAM,CAAC,EAAE;QACb,KAAK,EAAE,MAAM,CAAC,KAAK;QACnB,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,IAAI,EAAE,MAAM,CAAC,OAAO,KAAK,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ;QACpD,QAAQ;QACR,SAAS;QACT,UAAU;QACV,IAAI,EAAE,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC;KACxB,CAAC;AACJ,CAAC;AAED;;;;;;;;;GASG;AACH,MAAM,UAAU,sBAAsB,CAAC,KAAkC;IACvE,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,EAAE,wBAAwB,CAAC,CAAC;IAC5D,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,MAAM,CAAC,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC;IAC9E,MAAM,IAAI,GAAG,IAAI,GAAG,EAAoC,CAAC;IACzD,MAAM,KAAK,GAAG,CAAC,EAAU,EAA4B,EAAE;QACrD,IAAI,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACvB,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YACtB,GAAG,GAAG,IAAI,GAAG,EAAE,CAAC;YAChB,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;QACpB,CAAC;QACD,OAAO,GAAG,CAAC;IACb,CAAC,CAAC;IACF,KAAK,MAAM,IAAI,IAAI,KAAK,CAAC,KAAK,EAAE,CAAC;QAC/B,MAAM,IAAI,GAAG,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACrC,MAAM,EAAE,GAAG,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QACjC,IAAI,IAAI,KAAK,SAAS,IAAI,EAAE,KAAK,SAAS,IAAI,IAAI,KAAK,EAAE,EAAE,CAAC;YAC1D,SAAS;QACX,CAAC;QACD,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,CAAC,CAAC;QACvC,KAAK,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;IACxC,CAAC;IACD,MAAM,KAAK,GAAG,IAAI,GAAG,EAAgC,CAAC;IACtD,KAAK,MAAM,CAAC,EAAE,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC9E,MAAM,IAAI,GAAG,MAAM,CAAC,cAAc,CAAC,KAAK,CAAC,OAAO,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;QAChE,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACzB,MAAM,UAAU,GACd,GAAG,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,WAAW,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC,CAAC;QAC9F,KAAK,CAAC,GAAG,CAAC,EAAE,EAAE,UAAU,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC;IACtE,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}

package/dist/display/options.d.ts

@@ -0,0 +1,5 @@
+import type { SiteConfig } from "@concordance-wiki/core";
+import type { DisplayOptions } from "./types.js";
+/** `site.neighbourhood.size` of the configuration, 6 when unset. */
+export declare function displayOptions(site?: SiteConfig): DisplayOptions;
+//# sourceMappingURL=options.d.ts.map

package/dist/display/options.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"options.d.ts","sourceRoot":"","sources":["../../src/display/options.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,wBAAwB,CAAC;AAEzD,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAIjD,oEAAoE;AACpE,wBAAgB,cAAc,CAAC,IAAI,CAAC,EAAE,UAAU,GAAG,cAAc,CAEhE"}

package/dist/display/options.js

@@ -0,0 +1,6 @@
+const DEFAULT_SIZE = 6;
+/** `site.neighbourhood.size` of the configuration, 6 when unset. */
+export function displayOptions(site) {
+    return { size: site?.neighbourhood?.size ?? DEFAULT_SIZE };
+}
+//# sourceMappingURL=options.js.map

package/dist/display/options.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"options.js","sourceRoot":"","sources":["../../src/display/options.ts"],"names":[],"mappings":"AAIA,MAAM,YAAY,GAAG,CAAC,CAAC;AAEvB,oEAAoE;AACpE,MAAM,UAAU,cAAc,CAAC,IAAiB;IAC9C,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,aAAa,EAAE,IAAI,IAAI,YAAY,EAAE,CAAC;AAC7D,CAAC"}

package/dist/display/serialize.d.ts

@@ -0,0 +1,7 @@
+import type { DisplayedNeighbour, DisplayedNeighbourhood } from "./types.js";
+/**
+ * The `displayed_neighbourhood` block of `model.json`: keys in identifier order, each list in
+ * priority order with the `rank` of every neighbour, so that a panel can separate the groups.
+ */
+export declare function displayedNeighbourhoodToModel(neighbourhood: DisplayedNeighbourhood): Record<string, DisplayedNeighbour[]>;
+//# sourceMappingURL=serialize.d.ts.map

package/dist/display/serialize.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"serialize.d.ts","sourceRoot":"","sources":["../../src/display/serialize.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,kBAAkB,EAAE,sBAAsB,EAAE,MAAM,YAAY,CAAC;AAE7E;;;GAGG;AACH,wBAAgB,6BAA6B,CAC3C,aAAa,EAAE,sBAAsB,GACpC,MAAM,CAAC,MAAM,EAAE,kBAAkB,EAAE,CAAC,CAMtC"}

package/dist/display/serialize.js

@@ -0,0 +1,13 @@
+import { byCodeUnit } from "../neighbourhood/order.js";
+/**
+ * The `displayed_neighbourhood` block of `model.json`: keys in identifier order, each list in
+ * priority order with the `rank` of every neighbour, so that a panel can separate the groups.
+ */
+export function displayedNeighbourhoodToModel(neighbourhood) {
+    const block = {};
+    for (const [id, neighbours] of [...neighbourhood].sort(([a], [b]) => byCodeUnit(a, b))) {
+        block[id] = neighbours.map((neighbour) => ({ ...neighbour }));
+    }
+    return block;
+}
+//# sourceMappingURL=serialize.js.map

package/dist/display/serialize.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"serialize.js","sourceRoot":"","sources":["../../src/display/serialize.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,2BAA2B,CAAC;AAGvD;;;GAGG;AACH,MAAM,UAAU,6BAA6B,CAC3C,aAAqC;IAErC,MAAM,KAAK,GAAyC,EAAE,CAAC;IACvD,KAAK,MAAM,CAAC,EAAE,EAAE,UAAU,CAAC,IAAI,CAAC,GAAG,aAAa,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvF,KAAK,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,EAAE,GAAG,SAAS,EAAE,CAAC,CAAC,CAAC;IAChE,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}

package/dist/display/types.d.ts

@@ -0,0 +1,44 @@
+import type { Link } from "@concordance-wiki/core";
+import type { Profile } from "@concordance-wiki/profile";
+/** What the displayed neighbourhood needs of a node: a keyword page is a noteless word. */
+export interface DisplayableEntity {
+    id: string;
+    type: string;
+    title: string;
+    keyword?: boolean;
+}
+export type NeighbourKind = "entity" | "keyword";
+/** Where the links between the node and its neighbour point, seen from the node. */
+export type NeighbourDirection = "out" | "in" | "both";
+/** One node of the mini-map, as serialised under `displayed_neighbourhood` in `model.json`. */
+export interface DisplayedNeighbour {
+    id: string;
+    title: string;
+    type: string;
+    kind: NeighbourKind;
+    /** The relation of the most confident link, the first in code-unit order on a tie. */
+    relation: string;
+    direction: NeighbourDirection;
+    /** The largest confidence over the links between the two nodes. */
+    confidence: number;
+    /**
+     * The position of the neighbour's type in the `display.neighbours_order` of the page's type;
+     * a type not listed, a keyword page included, ranks after every listed one, at the length of
+     * the list. Neighbours are grouped by rank, so a change of rank separates two priority groups.
+     */
+    rank: number;
+}
+export interface DisplayedNeighbourhoodInput {
+    entities: readonly DisplayableEntity[];
+    links: readonly Link[];
+    /** Where the priority order of neighbour types comes from, per type of the page. */
+    profile: Profile;
+    /** Nodes shown per page; the computation never shows more than `MAX_DISPLAYED_NEIGHBOURS`. */
+    size: number;
+}
+/** Every entity in identifier order, each with its neighbours in priority order, empty when it has none. */
+export type DisplayedNeighbourhood = ReadonlyMap<string, DisplayedNeighbour[]>;
+export interface DisplayOptions {
+    size: number;
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/display/types.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/display/types.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,wBAAwB,CAAC;AACnD,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,2BAA2B,CAAC;AAEzD,2FAA2F;AAC3F,MAAM,WAAW,iBAAiB;IAChC,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,MAAM,CAAC;IACb,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED,MAAM,MAAM,aAAa,GAAG,QAAQ,GAAG,SAAS,CAAC;AAEjD,oFAAoF;AACpF,MAAM,MAAM,kBAAkB,GAAG,KAAK,GAAG,IAAI,GAAG,MAAM,CAAC;AAEvD,+FAA+F;AAC/F,MAAM,WAAW,kBAAkB;IACjC,EAAE,EAAE,MAAM,CAAC;IACX,KAAK,EAAE,MAAM,CAAC;IACd,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,aAAa,CAAC;IACpB,sFAAsF;IACtF,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,EAAE,kBAAkB,CAAC;IAC9B,mEAAmE;IACnE,UAAU,EAAE,MAAM,CAAC;IACnB;;;;OAIG;IACH,IAAI,EAAE,MAAM,CAAC;CACd;AAED,MAAM,WAAW,2BAA2B;IAC1C,QAAQ,EAAE,SAAS,iBAAiB,EAAE,CAAC;IACvC,KAAK,EAAE,SAAS,IAAI,EAAE,CAAC;IACvB,oFAAoF;IACpF,OAAO,EAAE,OAAO,CAAC;IACjB,8FAA8F;IAC9F,IAAI,EAAE,MAAM,CAAC;CACd;AAED,4GAA4G;AAC5G,MAAM,MAAM,sBAAsB,GAAG,WAAW,CAAC,MAAM,EAAE,kBAAkB,EAAE,CAAC,CAAC;AAE/E,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;CACd"}

package/dist/display/types.js

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=types.js.map

package/dist/display/types.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.js","sourceRoot":"","sources":["../../src/display/types.ts"],"names":[],"mappings":""}

package/dist/duplicates/content.d.ts

@@ -0,0 +1,29 @@
+import type { DuplicateOptions, DuplicateResource } from "./types.js";
+/** How the texts of two resources compare, from the shingle sets, never from the binary files. */
+export interface ContentSimilarity {
+    jaccard: number;
+    /** Whether `jaccard` was recomputed on the full shingle sets. */
+    exact: boolean;
+    /** Words of the shorter text over words of the longer one. */
+    sizeRatio: number;
+    /** Distinct lines in common over the distinct lines of both; present with an exact verification. */
+    sharedLines?: number;
+}
+export interface ContentPair {
+    /** The earlier resource. */
+    a: DuplicateResource;
+    b: DuplicateResource;
+    similarity: ContentSimilarity;
+}
+export interface ContentResult {
+    /** In resource order, the only pairs whose content was compared. */
+    pairs: ContentPair[];
+    exactVerifications: number;
+}
+/**
+ * Estimates the similarity of every pair the LSH banding brings together, and recomputes it
+ * exactly according to the mode: on every pair in `exact`, never in `estimate`, from
+ * `exactAbove` in `auto`.
+ */
+export declare function contentSimilarities(resources: readonly DuplicateResource[], normalizeText: (text: string) => string[], options: DuplicateOptions): ContentResult;
+//# sourceMappingURL=content.d.ts.map

package/dist/duplicates/content.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"content.d.ts","sourceRoot":"","sources":["../../src/duplicates/content.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAEtE,kGAAkG;AAClG,MAAM,WAAW,iBAAiB;IAChC,OAAO,EAAE,MAAM,CAAC;IAChB,iEAAiE;IACjE,KAAK,EAAE,OAAO,CAAC;IACf,8DAA8D;IAC9D,SAAS,EAAE,MAAM,CAAC;IAClB,oGAAoG;IACpG,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,MAAM,WAAW,WAAW;IAC1B,4BAA4B;IAC5B,CAAC,EAAE,iBAAiB,CAAC;IACrB,CAAC,EAAE,iBAAiB,CAAC;IACrB,UAAU,EAAE,iBAAiB,CAAC;CAC/B;AAED,MAAM,WAAW,aAAa;IAC5B,oEAAoE;IACpE,KAAK,EAAE,WAAW,EAAE,CAAC;IACrB,kBAAkB,EAAE,MAAM,CAAC;CAC5B;AAcD;;;;GAIG;AACH,wBAAgB,mBAAmB,CACjC,SAAS,EAAE,SAAS,iBAAiB,EAAE,EACvC,aAAa,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,MAAM,EAAE,EACzC,OAAO,EAAE,gBAAgB,GACxB,aAAa,CA6Bf"}

package/dist/duplicates/content.js

@@ -0,0 +1,42 @@
+import { candidatePairs } from "./lsh.js";
+import { createMinHash, estimatedJaccard } from "./minhash.js";
+import { exactJaccard, normaliseText, sharedLines, shingleSet } from "./shingles.js";
+// Only shingled texts are compared, so neither count is zero.
+function sizeRatio(a, b) {
+    return Math.min(a, b) / Math.max(a, b);
+}
+/**
+ * Estimates the similarity of every pair the LSH banding brings together, and recomputes it
+ * exactly according to the mode: on every pair in `exact`, never in `estimate`, from
+ * `exactAbove` in `auto`.
+ */
+export function contentSimilarities(resources, normalizeText, options) {
+    const minHash = createMinHash(options.minhashFunctions);
+    const prepared = resources.map((resource) => {
+        const text = normaliseText(resource.text, normalizeText);
+        const shingles = shingleSet(text.words, options.shingleSize);
+        return { resource, text, shingles, signature: minHash.signature(shingles) };
+    });
+    // A text shorter than a shingle has nothing to compare; its empty signature must not bucket with every other.
+    const shingled = prepared.filter((item) => item.shingles.size > 0);
+    const pairs = [];
+    let exactVerifications = 0;
+    for (const [a, b] of candidatePairs(shingled, (item) => item.signature)) {
+        const estimate = estimatedJaccard(a.signature, b.signature);
+        const verify = options.mode === "exact" || (options.mode === "auto" && estimate >= options.exactAbove);
+        const ratio = sizeRatio(a.text.words.length, b.text.words.length);
+        let similarity = { jaccard: estimate, exact: false, sizeRatio: ratio };
+        if (verify) {
+            exactVerifications++;
+            similarity = {
+                jaccard: exactJaccard(a.shingles, b.shingles),
+                exact: true,
+                sizeRatio: ratio,
+                sharedLines: sharedLines(a.text.lines, b.text.lines),
+            };
+        }
+        pairs.push({ a: a.resource, b: b.resource, similarity });
+    }
+    return { pairs, exactVerifications };
+}
+//# sourceMappingURL=content.js.map

package/dist/duplicates/content.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"content.js","sourceRoot":"","sources":["../../src/duplicates/content.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,UAAU,CAAC;AAC1C,OAAO,EAAE,aAAa,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AAC/D,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,WAAW,EAAE,UAAU,EAAE,MAAM,eAAe,CAAC;AAmCrF,8DAA8D;AAC9D,SAAS,SAAS,CAAC,CAAS,EAAE,CAAS;IACrC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;AACzC,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,mBAAmB,CACjC,SAAuC,EACvC,aAAyC,EACzC,OAAyB;IAEzB,MAAM,OAAO,GAAG,aAAa,CAAC,OAAO,CAAC,gBAAgB,CAAC,CAAC;IACxD,MAAM,QAAQ,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAY,EAAE;QACpD,MAAM,IAAI,GAAG,aAAa,CAAC,QAAQ,CAAC,IAAI,EAAE,aAAa,CAAC,CAAC;QACzD,MAAM,QAAQ,GAAG,UAAU,CAAC,IAAI,CAAC,KAAK,EAAE,OAAO,CAAC,WAAW,CAAC,CAAC;QAC7D,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,QAAQ,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,CAAC,QAAQ,CAAC,EAAE,CAAC;IAC9E,CAAC,CAAC,CAAC;IACH,8GAA8G;IAC9G,MAAM,QAAQ,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,CAAC,CAAC;IACnE,MAAM,KAAK,GAAkB,EAAE,CAAC;IAChC,IAAI,kBAAkB,GAAG,CAAC,CAAC;IAC3B,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,cAAc,CAAC,QAAQ,EAAE,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC;QACxE,MAAM,QAAQ,GAAG,gBAAgB,CAAC,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,SAAS,CAAC,CAAC;QAC5D,MAAM,MAAM,GACV,OAAO,CAAC,IAAI,KAAK,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,KAAK,MAAM,IAAI,QAAQ,IAAI,OAAO,CAAC,UAAU,CAAC,CAAC;QAC1F,MAAM,KAAK,GAAG,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;QAClE,IAAI,UAAU,GAAsB,EAAE,OAAO,EAAE,QAAQ,EAAE,KAAK,EAAE,KAAK,EAAE,SAAS,EAAE,KAAK,EAAE,CAAC;QAC1F,IAAI,MAAM,EAAE,CAAC;YACX,kBAAkB,EAAE,CAAC;YACrB,UAAU,GAAG;gBACX,OAAO,EAAE,YAAY,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,QAAQ,CAAC;gBAC7C,KAAK,EAAE,IAAI;gBACX,SAAS,EAAE,KAAK;gBAChB,WAAW,EAAE,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC;aACrD,CAAC;QACJ,CAAC;QACD,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,UAAU,EAAE,CAAC,CAAC;IAC3D,CAAC;IACD,OAAO,EAAE,KAAK,EAAE,kBAAkB,EAAE,CAAC;AACvC,CAAC"}

package/dist/duplicates/lsh.d.ts

@@ -0,0 +1,8 @@
+/** Rows per band: with 128 functions, 32 bands catch a pair at 0.6 similarity with probability 0.99. */
+export declare const LSH_ROWS = 4;
+/**
+ * The pairs of items whose signatures agree on at least one band, each pair once with the
+ * earlier item first, in item order. Two items that share no band are never compared.
+ */
+export declare function candidatePairs<T>(items: readonly T[], signatureOf: (item: T) => Uint32Array, rows?: number): [T, T][];
+//# sourceMappingURL=lsh.d.ts.map

package/dist/duplicates/lsh.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"lsh.d.ts","sourceRoot":"","sources":["../../src/duplicates/lsh.ts"],"names":[],"mappings":"AAAA,wGAAwG;AACxG,eAAO,MAAM,QAAQ,IAAI,CAAC;AAO1B;;;GAGG;AACH,wBAAgB,cAAc,CAAC,CAAC,EAC9B,KAAK,EAAE,SAAS,CAAC,EAAE,EACnB,WAAW,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,WAAW,EACrC,IAAI,GAAE,MAAiB,GACtB,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,CA6BV"}

package/dist/duplicates/lsh.js

@@ -0,0 +1,38 @@
+/** Rows per band: with 128 functions, 32 bands catch a pair at 0.6 similarity with probability 0.99. */
+export const LSH_ROWS = 4;
+/**
+ * The pairs of items whose signatures agree on at least one band, each pair once with the
+ * earlier item first, in item order. Two items that share no band are never compared.
+ */
+export function candidatePairs(items, signatureOf, rows = LSH_ROWS) {
+    const placed = items.map((item, index) => ({ index, item }));
+    const first = items[0];
+    const functions = first === undefined ? 0 : signatureOf(first).length;
+    const bands = Math.floor(functions / rows);
+    const pairs = new Map();
+    const width = items.length;
+    for (let band = 0; band < bands; band++) {
+        const buckets = new Map();
+        for (const entry of placed) {
+            const key = signatureOf(entry.item)
+                .subarray(band * rows, (band + 1) * rows)
+                .join(",");
+            const bucket = buckets.get(key);
+            if (bucket === undefined) {
+                buckets.set(key, [entry]);
+            }
+            else {
+                bucket.push(entry);
+            }
+        }
+        for (const bucket of buckets.values()) {
+            bucket.forEach((first, position) => {
+                for (const second of bucket.slice(position + 1)) {
+                    pairs.set(first.index * width + second.index, [first.item, second.item]);
+                }
+            });
+        }
+    }
+    return [...pairs].sort(([a], [b]) => a - b).map(([, pair]) => pair);
+}
+//# sourceMappingURL=lsh.js.map

package/dist/duplicates/lsh.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"lsh.js","sourceRoot":"","sources":["../../src/duplicates/lsh.ts"],"names":[],"mappings":"AAAA,wGAAwG;AACxG,MAAM,CAAC,MAAM,QAAQ,GAAG,CAAC,CAAC;AAO1B;;;GAGG;AACH,MAAM,UAAU,cAAc,CAC5B,KAAmB,EACnB,WAAqC,EACrC,OAAe,QAAQ;IAEvB,MAAM,MAAM,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,KAAK,EAAa,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;IACxE,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;IACvB,MAAM,SAAS,GAAG,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC;IACtE,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,SAAS,GAAG,IAAI,CAAC,CAAC;IAC3C,MAAM,KAAK,GAAG,IAAI,GAAG,EAAkB,CAAC;IACxC,MAAM,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC;IAC3B,KAAK,IAAI,IAAI,GAAG,CAAC,EAAE,IAAI,GAAG,KAAK,EAAE,IAAI,EAAE,EAAE,CAAC;QACxC,MAAM,OAAO,GAAG,IAAI,GAAG,EAAuB,CAAC;QAC/C,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;YAC3B,MAAM,GAAG,GAAG,WAAW,CAAC,KAAK,CAAC,IAAI,CAAC;iBAChC,QAAQ,CAAC,IAAI,GAAG,IAAI,EAAE,CAAC,IAAI,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC;iBACxC,IAAI,CAAC,GAAG,CAAC,CAAC;YACb,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;YAChC,IAAI,MAAM,KAAK,SAAS,EAAE,CAAC;gBACzB,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,KAAK,CAAC,CAAC,CAAC;YAC5B,CAAC;iBAAM,CAAC;gBACN,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACrB,CAAC;QACH,CAAC;QACD,KAAK,MAAM,MAAM,IAAI,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;YACtC,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,QAAQ,EAAE,EAAE;gBACjC,KAAK,MAAM,MAAM,IAAI,MAAM,CAAC,KAAK,CAAC,QAAQ,GAAG,CAAC,CAAC,EAAE,CAAC;oBAChD,KAAK,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,GAAG,KAAK,GAAG,MAAM,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;gBAC3E,CAAC;YACH,CAAC,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IACD,OAAO,CAAC,GAAG,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,CAAC;AACtE,CAAC"}

package/dist/duplicates/minhash.d.ts

@@ -0,0 +1,11 @@
+/** The seed every build uses: signatures never depend on the run. */
+export declare const MINHASH_SEED = 1597463007;
+export interface MinHash {
+    functions: number;
+    /** The minimum of every hash function over the shingles; all ones for an empty set. */
+    signature(shingles: Iterable<string>): Uint32Array;
+}
+export declare function createMinHash(functions: number, seed?: number): MinHash;
+/** The share of equal positions, an unbiased estimate of the Jaccard index of the two sets. */
+export declare function estimatedJaccard(a: Uint32Array, b: Uint32Array): number;
+//# sourceMappingURL=minhash.d.ts.map

package/dist/duplicates/minhash.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"minhash.d.ts","sourceRoot":"","sources":["../../src/duplicates/minhash.ts"],"names":[],"mappings":"AAAA,qEAAqE;AACrE,eAAO,MAAM,YAAY,aAAa,CAAC;AAqCvC,MAAM,WAAW,OAAO;IACtB,SAAS,EAAE,MAAM,CAAC;IAClB,uFAAuF;IACvF,SAAS,CAAC,QAAQ,EAAE,QAAQ,CAAC,MAAM,CAAC,GAAG,WAAW,CAAC;CACpD;AAED,wBAAgB,aAAa,CAAC,SAAS,EAAE,MAAM,EAAE,IAAI,SAAe,GAAG,OAAO,CAgB7E;AAED,+FAA+F;AAC/F,wBAAgB,gBAAgB,CAAC,CAAC,EAAE,WAAW,EAAE,CAAC,EAAE,WAAW,GAAG,MAAM,CAMvE"}

package/dist/duplicates/minhash.js

@@ -0,0 +1,51 @@
+/** The seed every build uses: signatures never depend on the run. */
+export const MINHASH_SEED = 0x5f3759df;
+// FNV-1a over the UTF-16 code units: cheap, and every shingle gets one base hash.
+function baseHash(shingle) {
+    let hash = 0x811c9dc5;
+    for (let index = 0; index < shingle.length; index++) {
+        hash = Math.imul(hash ^ shingle.charCodeAt(index), 0x01000193);
+    }
+    return hash >>> 0;
+}
+// The finaliser of murmur3, for the second base hash of a shingle.
+function mix(value) {
+    let hash = value;
+    hash ^= hash >>> 16;
+    hash = Math.imul(hash, 0x85ebca6b);
+    hash ^= hash >>> 13;
+    hash = Math.imul(hash, 0xc2b2ae35);
+    hash ^= hash >>> 16;
+    return hash >>> 0;
+}
+function hashed(shingle, seed) {
+    const first = (baseHash(shingle) ^ seed) >>> 0;
+    return { first, second: mix(first) | 1 };
+}
+export function createMinHash(functions, seed = MINHASH_SEED) {
+    return {
+        functions,
+        signature(shingles) {
+            const bases = Array.from(shingles, (shingle) => hashed(shingle, seed));
+            const signature = new Uint32Array(functions);
+            for (let index = 0; index < functions; index++) {
+                let minimum = 0xffffffff;
+                // One expression per shingle: this loop runs once per shingle and per function.
+                for (const base of bases)
+                    minimum = Math.min(minimum, (base.first + Math.imul(index, base.second)) >>> 0);
+                signature[index] = minimum;
+            }
+            return signature;
+        },
+    };
+}
+/** The share of equal positions, an unbiased estimate of the Jaccard index of the two sets. */
+export function estimatedJaccard(a, b) {
+    let equal = 0;
+    a.forEach((value, index) => {
+        if (value === b[index])
+            equal++;
+    });
+    return equal / a.length;
+}
+//# sourceMappingURL=minhash.js.map

package/dist/duplicates/minhash.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"minhash.js","sourceRoot":"","sources":["../../src/duplicates/minhash.ts"],"names":[],"mappings":"AAAA,qEAAqE;AACrE,MAAM,CAAC,MAAM,YAAY,GAAG,UAAU,CAAC;AAEvC,kFAAkF;AAClF,SAAS,QAAQ,CAAC,OAAe;IAC/B,IAAI,IAAI,GAAG,UAAU,CAAC;IACtB,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,OAAO,CAAC,MAAM,EAAE,KAAK,EAAE,EAAE,CAAC;QACpD,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,GAAG,OAAO,CAAC,UAAU,CAAC,KAAK,CAAC,EAAE,UAAU,CAAC,CAAC;IACjE,CAAC;IACD,OAAO,IAAI,KAAK,CAAC,CAAC;AACpB,CAAC;AAED,mEAAmE;AACnE,SAAS,GAAG,CAAC,KAAa;IACxB,IAAI,IAAI,GAAG,KAAK,CAAC;IACjB,IAAI,IAAI,IAAI,KAAK,EAAE,CAAC;IACpB,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;IACnC,IAAI,IAAI,IAAI,KAAK,EAAE,CAAC;IACpB,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;IACnC,IAAI,IAAI,IAAI,KAAK,EAAE,CAAC;IACpB,OAAO,IAAI,KAAK,CAAC,CAAC;AACpB,CAAC;AAYD,SAAS,MAAM,CAAC,OAAe,EAAE,IAAY;IAC3C,MAAM,KAAK,GAAG,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;IAC/C,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;AAC3C,CAAC;AAQD,MAAM,UAAU,aAAa,CAAC,SAAiB,EAAE,IAAI,GAAG,YAAY;IAClE,OAAO;QACL,SAAS;QACT,SAAS,CAAC,QAAQ;YAChB,MAAM,KAAK,GAAG,KAAK,CAAC,IAAI,CAAC,QAAQ,EAAE,CAAC,OAAO,EAAE,EAAE,CAAC,MAAM,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC,CAAC;YACvE,MAAM,SAAS,GAAG,IAAI,WAAW,CAAC,SAAS,CAAC,CAAC;YAC7C,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,SAAS,EAAE,KAAK,EAAE,EAAE,CAAC;gBAC/C,IAAI,OAAO,GAAG,UAAU,CAAC;gBACzB,gFAAgF;gBAChF,KAAK,MAAM,IAAI,IAAI,KAAK;oBACtB,OAAO,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,MAAM,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC;gBAClF,SAAS,CAAC,KAAK,CAAC,GAAG,OAAO,CAAC;YAC7B,CAAC;YACD,OAAO,SAAS,CAAC;QACnB,CAAC;KACF,CAAC;AACJ,CAAC;AAED,+FAA+F;AAC/F,MAAM,UAAU,gBAAgB,CAAC,CAAc,EAAE,CAAc;IAC7D,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,CAAC,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,EAAE;QACzB,IAAI,KAAK,KAAK,CAAC,CAAC,KAAK,CAAC;YAAE,KAAK,EAAE,CAAC;IAClC,CAAC,CAAC,CAAC;IACH,OAAO,KAAK,GAAG,CAAC,CAAC,MAAM,CAAC;AAC1B,CAAC"}

package/dist/duplicates/options.d.ts

@@ -0,0 +1,6 @@
+import type { InferenceConfig } from "@concordance-wiki/core";
+import type { DuplicateOptions } from "./types.js";
+export declare const DUPLICATE_DEFAULTS: DuplicateOptions;
+/** `inference.duplicates` of the configuration over the defaults. */
+export declare function duplicateOptions(inference?: InferenceConfig): DuplicateOptions;
+//# sourceMappingURL=options.d.ts.map

package/dist/duplicates/options.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"options.d.ts","sourceRoot":"","sources":["../../src/duplicates/options.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AAE9D,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAEnD,eAAO,MAAM,kBAAkB,EAAE,gBAQhC,CAAC;AAEF,qEAAqE;AACrE,wBAAgB,gBAAgB,CAAC,SAAS,CAAC,EAAE,eAAe,GAAG,gBAAgB,CAW9E"}

package/dist/duplicates/options.js

@@ -0,0 +1,23 @@
+export const DUPLICATE_DEFAULTS = {
+    mode: "auto",
+    exactAbove: 0.5,
+    sizeRatioMin: 0.5,
+    shingleSize: 5,
+    minhashFunctions: 128,
+    mergeAbove: 0.9,
+    candidateAbove: 0.5,
+};
+/** `inference.duplicates` of the configuration over the defaults. */
+export function duplicateOptions(inference) {
+    const duplicates = inference?.duplicates;
+    return {
+        mode: duplicates?.mode ?? DUPLICATE_DEFAULTS.mode,
+        exactAbove: duplicates?.exact_above ?? DUPLICATE_DEFAULTS.exactAbove,
+        sizeRatioMin: duplicates?.size_ratio_min ?? DUPLICATE_DEFAULTS.sizeRatioMin,
+        shingleSize: duplicates?.shingle_size ?? DUPLICATE_DEFAULTS.shingleSize,
+        minhashFunctions: duplicates?.minhash_functions ?? DUPLICATE_DEFAULTS.minhashFunctions,
+        mergeAbove: duplicates?.merge_above ?? DUPLICATE_DEFAULTS.mergeAbove,
+        candidateAbove: duplicates?.candidate_above ?? DUPLICATE_DEFAULTS.candidateAbove,
+    };
+}
+//# sourceMappingURL=options.js.map

package/dist/duplicates/options.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"options.js","sourceRoot":"","sources":["../../src/duplicates/options.ts"],"names":[],"mappings":"AAIA,MAAM,CAAC,MAAM,kBAAkB,GAAqB;IAClD,IAAI,EAAE,MAAM;IACZ,UAAU,EAAE,GAAG;IACf,YAAY,EAAE,GAAG;IACjB,WAAW,EAAE,CAAC;IACd,gBAAgB,EAAE,GAAG;IACrB,UAAU,EAAE,GAAG;IACf,cAAc,EAAE,GAAG;CACpB,CAAC;AAEF,qEAAqE;AACrE,MAAM,UAAU,gBAAgB,CAAC,SAA2B;IAC1D,MAAM,UAAU,GAAG,SAAS,EAAE,UAAU,CAAC;IACzC,OAAO;QACL,IAAI,EAAE,UAAU,EAAE,IAAI,IAAI,kBAAkB,CAAC,IAAI;QACjD,UAAU,EAAE,UAAU,EAAE,WAAW,IAAI,kBAAkB,CAAC,UAAU;QACpE,YAAY,EAAE,UAAU,EAAE,cAAc,IAAI,kBAAkB,CAAC,YAAY;QAC3E,WAAW,EAAE,UAAU,EAAE,YAAY,IAAI,kBAAkB,CAAC,WAAW;QACvE,gBAAgB,EAAE,UAAU,EAAE,iBAAiB,IAAI,kBAAkB,CAAC,gBAAgB;QACtF,UAAU,EAAE,UAAU,EAAE,WAAW,IAAI,kBAAkB,CAAC,UAAU;QACpE,cAAc,EAAE,UAAU,EAAE,eAAe,IAAI,kBAAkB,CAAC,cAAc;KACjF,CAAC;AACJ,CAAC"}