@memberjunction/ai-vector-dupe 5.42.0 → 5.44.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +86 -7
- package/dist/duplicateRecordDetector.d.ts +167 -2
- package/dist/duplicateRecordDetector.d.ts.map +1 -1
- package/dist/duplicateRecordDetector.js +397 -57
- package/dist/duplicateRecordDetector.js.map +1 -1
- package/dist/index.d.ts +4 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +5 -0
- package/dist/index.js.map +1 -1
- package/dist/reasoning/DuplicateReasoningProvider.d.ts +84 -0
- package/dist/reasoning/DuplicateReasoningProvider.d.ts.map +1 -0
- package/dist/reasoning/DuplicateReasoningProvider.js +215 -0
- package/dist/reasoning/DuplicateReasoningProvider.js.map +1 -0
- package/dist/reasoning/DuplicateReasoningTypes.d.ts +159 -0
- package/dist/reasoning/DuplicateReasoningTypes.d.ts.map +1 -0
- package/dist/reasoning/DuplicateReasoningTypes.js +14 -0
- package/dist/reasoning/DuplicateReasoningTypes.js.map +1 -0
- package/dist/reasoning/MatchedSetDeltaBuilder.d.ts +66 -0
- package/dist/reasoning/MatchedSetDeltaBuilder.d.ts.map +1 -0
- package/dist/reasoning/MatchedSetDeltaBuilder.js +95 -0
- package/dist/reasoning/MatchedSetDeltaBuilder.js.map +1 -0
- package/dist/reasoning/PromptReasoningProvider.d.ts +32 -0
- package/dist/reasoning/PromptReasoningProvider.d.ts.map +1 -0
- package/dist/reasoning/PromptReasoningProvider.js +95 -0
- package/dist/reasoning/PromptReasoningProvider.js.map +1 -0
- package/package.json +13 -10
package/README.md
CHANGED
|
@@ -4,7 +4,9 @@
|
|
|
4
4
|
<!-- [](https://www.npmjs.com/package/@memberjunction/ai-vector-dupe) -->
|
|
5
5
|
<!-- [](https://github.com/MemberJunction/MJ/actions) -->
|
|
6
6
|
|
|
7
|
-
**AI-powered duplicate record detection for MemberJunction entities** -- finds, scores, tracks, and optionally auto-merges duplicate records using vector similarity, hybrid search (RRF), and optional reranking.
|
|
7
|
+
**AI-powered duplicate record detection for MemberJunction entities** -- finds, scores, tracks, and optionally auto-merges duplicate records using vector similarity, hybrid search (RRF), and optional reranking, with an optional **LLM reasoning layer** that judges borderline matches per-candidate before they ever reach a human.
|
|
8
|
+
|
|
9
|
+
> **Two-stage by design.** Vector/hybrid search is the cheap, fast recall stage; the LLM reasoning layer is an *opt-in, threshold-gated* precision stage. Reasoning is **off by default** and, when enabled, only fires for matched sets whose top vector score clears a per-entity `ReasoningThreshold` -- so you pay for the LLM only on the matches that are actually ambiguous. See [LLM Reasoning Layer](#llm-reasoning-layer).
|
|
8
10
|
|
|
9
11
|
---
|
|
10
12
|
|
|
@@ -35,8 +37,10 @@
|
|
|
35
37
|
| (hybrid if supported)|
|
|
36
38
|
| 5. Filter self-matches |
|
|
37
39
|
| 6. Apply thresholds |
|
|
38
|
-
| 7.
|
|
39
|
-
|
|
|
40
|
+
| 7. LLM reasoning (opt-in,|
|
|
41
|
+
| threshold-gated) |
|
|
42
|
+
| 8. Persist match results|
|
|
43
|
+
| 9. Auto-merge (optional)|
|
|
40
44
|
+-------------------------+
|
|
41
45
|
|
|
|
42
46
|
+------------------+------------------+
|
|
@@ -56,9 +60,13 @@
|
|
|
56
60
|
| `@memberjunction/ai-vectordb` | Vector database abstraction (query, hybrid search) |
|
|
57
61
|
| `@memberjunction/ai-vectors` | `VectorBase` base class with metadata and RunView helpers |
|
|
58
62
|
| `@memberjunction/ai-vector-sync` | `EntityVectorSyncer` for record vectorization, template parsing |
|
|
63
|
+
| `@memberjunction/ai-prompts` | Runs the reasoning prompt in `'Prompt'` mode (`PromptReasoningProvider`) |
|
|
64
|
+
| `@memberjunction/record-comparison` | Computes the field-level deltas across a matched set that the reasoner judges |
|
|
59
65
|
| `@memberjunction/core` | Core types: `PotentialDuplicateRequest`, `DuplicateDetectionOptions`, etc. |
|
|
60
|
-
| `@memberjunction/core-entities` | Generated entity classes for Duplicate Runs, Lists, Entity Documents |
|
|
61
|
-
| `@memberjunction/global` | `MJGlobal` class factory, `UUIDsEqual` |
|
|
66
|
+
| `@memberjunction/core-entities` | Generated entity classes for Duplicate Runs, Lists, Entity Documents (incl. the `*Reasoning*` / `AutomationLevel` columns) |
|
|
67
|
+
| `@memberjunction/global` | `MJGlobal` class factory (resolves the reasoning provider by mode), `UUIDsEqual` |
|
|
68
|
+
|
|
69
|
+
> The `'Agent'`-mode provider (`DuplicateReasoningAgentProvider`) is **not** a dependency of this package -- it lives in `@memberjunction/ai-agents` and registers against the reasoning seam at runtime. See [LLM Reasoning Layer](#llm-reasoning-layer).
|
|
62
70
|
|
|
63
71
|
---
|
|
64
72
|
|
|
@@ -222,6 +230,77 @@ See the [Duplicate Detection Guide](docs/DUPLICATE_DETECTION_GUIDE.md#reranking-
|
|
|
222
230
|
|
|
223
231
|
---
|
|
224
232
|
|
|
233
|
+
## LLM Reasoning Layer
|
|
234
|
+
|
|
235
|
+
Vector and hybrid search are good at *recall* (surfacing candidates) but not great at the final *is-this-actually-the-same-entity?* judgment -- two records can be semantically near without being duplicates (a parent company vs. its subsidiary), and two true duplicates can look lexically different (typos, abbreviations, stale addresses). The reasoning layer lets a (possibly small/cheap) LLM make that judgment **per candidate**, on top of the vector scores.
|
|
236
|
+
|
|
237
|
+
It is **additive and off by default** -- when `EnableLLMReasoning` is false on the Entity Document, the pipeline behaves exactly as the vector-only path documented above.
|
|
238
|
+
|
|
239
|
+
### Cost control: the reasoning gate
|
|
240
|
+
|
|
241
|
+
The whole point is *not* to run an LLM on every record. Reasoning runs **once per source record's matched set**, and only when the gate is open. The gate (`DuplicateRecordDetector.IsReasoningGateOpen`) opens when:
|
|
242
|
+
|
|
243
|
+
1. `EntityDocument.EnableLLMReasoning` is `true`, **and**
|
|
244
|
+
2. the set has at least one candidate, **and**
|
|
245
|
+
3. the set's **top** vector `MatchProbability` is `>= EntityDocument.ReasoningThreshold` (a `null` threshold means "reason over any non-empty set").
|
|
246
|
+
|
|
247
|
+
So configuring `ReasoningThreshold = 0.85` means "only spend an LLM call when vector search is already fairly confident" -- you tune the recall/precision/cost tradeoff per entity.
|
|
248
|
+
|
|
249
|
+
### Configuration (on the Entity Document)
|
|
250
|
+
|
|
251
|
+
All reasoning configuration lives on the `MJ: Entity Documents` record, per entity:
|
|
252
|
+
|
|
253
|
+
| Field | Type | Default | Purpose |
|
|
254
|
+
|---|---|---|---|
|
|
255
|
+
| `EnableLLMReasoning` | `boolean` | `false` | Master switch. Off = vector-only behavior, unchanged. |
|
|
256
|
+
| `ReasoningMode` | `'Prompt' \| 'Agent'` | `'Prompt'` | Which provider runs (see below). |
|
|
257
|
+
| `ReasoningThreshold` | `number \| null` | `null` | Vector-score gate (0–1). LLM runs only when the set's top score clears this. `null` = reason over any non-empty set. |
|
|
258
|
+
| `ReasoningPromptID` | `string \| null` | -- | The AI Prompt to use in `'Prompt'` mode. Falls back to the seeded "Duplicate Resolution" prompt. |
|
|
259
|
+
| `ReasoningAgentID` | `string \| null` | -- | The AI Agent to use in `'Agent'` mode. Falls back to the seeded "Duplicate Resolution Agent". |
|
|
260
|
+
| `AutomationLevel` | `'ReviewAll' \| 'LLMGated' \| 'AutoMergeAboveAbsolute'` | `'ReviewAll'` | How far automation goes after reasoning (review everything / let the LLM gate review / auto-merge above the absolute threshold). |
|
|
261
|
+
|
|
262
|
+
### The pluggable provider seam
|
|
263
|
+
|
|
264
|
+
Reasoning is delegated through an abstract `DuplicateReasoningProvider`, resolved at runtime via the MJ class factory by `ReasoningMode`. Two providers ship; both emit the **identical** `DuplicateReasoningOutput`, so promoting an entity from `Prompt` to `Agent` is a config change, not a rewrite.
|
|
265
|
+
|
|
266
|
+
| Provider | `@RegisterClass` key | Package | Path |
|
|
267
|
+
|---|---|---|---|
|
|
268
|
+
| `PromptReasoningProvider` | `PROMPT_REASONING_PROVIDER_KEY` (`'Prompt'`) | `@memberjunction/ai-vector-dupe` | Single-shot AI Prompt. Persists `AIPromptRunID`. |
|
|
269
|
+
| `DuplicateReasoningAgentProvider` | `AGENT_REASONING_PROVIDER_KEY` (`'Agent'`) | `@memberjunction/ai-agents` | Orchestrated agent run (unlocks memory-note injection + future context tools). Persists `AIAgentRunID`. |
|
|
270
|
+
|
|
271
|
+
> The Agent provider lives in `@memberjunction/ai-agents`, **not** this package, because `ai-agents` depends on `ai-vector-dupe` -- importing `AgentRunner` here would create a build cycle. It registers against the seam under the `'Agent'` key, so the detector resolves it via the class factory with no static import back into the pipeline. Registration is handled by the class-registration manifest (no `Load*()` helper needed).
|
|
272
|
+
|
|
273
|
+
To add a custom reasoning strategy, subclass `DuplicateReasoningProvider`, implement `Reason(input, context)`, and register it under a new mode key:
|
|
274
|
+
|
|
275
|
+
```typescript
|
|
276
|
+
import { RegisterClass } from '@memberjunction/global';
|
|
277
|
+
import {
|
|
278
|
+
DuplicateReasoningProvider,
|
|
279
|
+
DuplicateReasoningInput,
|
|
280
|
+
DuplicateReasoningOutput,
|
|
281
|
+
DuplicateReasoningContext,
|
|
282
|
+
} from '@memberjunction/ai-vector-dupe';
|
|
283
|
+
|
|
284
|
+
@RegisterClass(DuplicateReasoningProvider, 'MyMode')
|
|
285
|
+
export class MyReasoningProvider extends DuplicateReasoningProvider {
|
|
286
|
+
public async Reason(
|
|
287
|
+
input: DuplicateReasoningInput,
|
|
288
|
+
context: DuplicateReasoningContext,
|
|
289
|
+
): Promise<DuplicateReasoningOutput> {
|
|
290
|
+
// ...inspect input.SourceRecord, input.Candidates, input.FieldDeltas...
|
|
291
|
+
// return a structured verdict with per-candidate CandidateVerdicts
|
|
292
|
+
}
|
|
293
|
+
}
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
### Per-candidate verdicts
|
|
297
|
+
|
|
298
|
+
A matched set is the top-K neighbors of one source record, so it routinely mixes true duplicates with false positives. The reasoner therefore returns a verdict **per candidate** (`DuplicateReasoningOutput.CandidateVerdicts`), each judged independently against the source -- a false-positive candidate reads `NotDuplicate` even when another candidate in the same set is a confident `Merge`. The detector stamps each candidate's own verdict onto its match row; the set-level `Recommendation`/`Confidence` are *derived* values used only for the group's dominant display and the auto-merge gate.
|
|
299
|
+
|
|
300
|
+
The output also carries a proposed `SurvivorRecordID` and per-field survivor choices (`FieldChoices`) that feed `Metadata.MergeRecords` at merge time. A `null` `Confidence` means the model returned no usable confidence and must be rendered/stored as "unknown" -- never conflated with a real `0` (which reads as "confidently NOT a duplicate"). See `DuplicateReasoningTypes` for the full contract.
|
|
301
|
+
|
|
302
|
+
---
|
|
303
|
+
|
|
225
304
|
## Progress Reporting
|
|
226
305
|
|
|
227
306
|
The `OnProgress` callback fires at each phase of the pipeline:
|
|
@@ -309,11 +388,11 @@ The package reads from and writes to these MJ entities:
|
|
|
309
388
|
|
|
310
389
|
| Entity | Purpose |
|
|
311
390
|
|---|---|
|
|
312
|
-
| `MJ: Entity Documents` | Configuration: template, AI model, vector DB, thresholds |
|
|
391
|
+
| `MJ: Entity Documents` | Configuration: template, AI model, vector DB, thresholds, **and the reasoning config** (`EnableLLMReasoning`, `ReasoningMode`, `ReasoningThreshold`, `ReasoningPromptID`, `ReasoningAgentID`, `AutomationLevel`) |
|
|
313
392
|
| `MJ: Lists` / `MJ: List Details` | Source records to check for duplicates |
|
|
314
393
|
| `MJ: Duplicate Runs` | Tracks each detection run (status, timing) |
|
|
315
394
|
| `MJ: Duplicate Run Details` | Per-record tracking within a run; includes `RecordMetadata` (vector DB metadata snapshot) |
|
|
316
|
-
| `MJ: Duplicate Run Detail Matches` | Individual match results with probability scores; includes `RecordMetadata`
|
|
395
|
+
| `MJ: Duplicate Run Detail Matches` | Individual match results with probability scores; includes `RecordMetadata` and, when reasoning ran, the per-candidate LLM verdict columns (recommendation, confidence, reasoning, and `AIPromptRunID`/`AIAgentRunID`) |
|
|
317
396
|
|
|
318
397
|
---
|
|
319
398
|
|
|
@@ -20,6 +20,8 @@ import { MJDuplicateRunDetailEntity, MJDuplicateRunDetailMatchEntity, MJDuplicat
|
|
|
20
20
|
import { VectorBase } from "@memberjunction/ai-vectors";
|
|
21
21
|
import { EntityDocumentTemplateParser } from "@memberjunction/ai-vector-sync";
|
|
22
22
|
import type { MJTemplateEntityExtended } from "@memberjunction/core-entities";
|
|
23
|
+
import { DuplicateReasoningProvider } from "./reasoning/DuplicateReasoningProvider.js";
|
|
24
|
+
import { DuplicateReasoningInput, DuplicateReasoningOutput, ReasoningFieldDelta } from "./reasoning/DuplicateReasoningTypes.js";
|
|
23
25
|
/**
|
|
24
26
|
* Internal result from querying duplicates for a single source record.
|
|
25
27
|
*/
|
|
@@ -28,6 +30,21 @@ interface RecordQueryResult {
|
|
|
28
30
|
TemplateText: string;
|
|
29
31
|
Duplicates: PotentialDuplicateResult;
|
|
30
32
|
}
|
|
33
|
+
/**
|
|
34
|
+
* A matched set's reasoning verdict together with the detector-resolved literal survivor
|
|
35
|
+
* field map. The provider returns the verdict (with per-field {@link DuplicateReasoningFieldChoice}
|
|
36
|
+
* choices addressed by record id); the detector resolves those choices to literal
|
|
37
|
+
* `{FieldName, Value}` entries (ready for {@link RecordMergeRequest.FieldMap}) by reading the
|
|
38
|
+
* chosen record's value out of the matched-set deltas it already loaded for the prompt.
|
|
39
|
+
*/
|
|
40
|
+
interface SetReasoning {
|
|
41
|
+
Output: DuplicateReasoningOutput;
|
|
42
|
+
/** FieldChoices resolved to literal values; empty when the reasoner proposed none. */
|
|
43
|
+
FieldMap: {
|
|
44
|
+
FieldName: string;
|
|
45
|
+
Value: unknown;
|
|
46
|
+
}[];
|
|
47
|
+
}
|
|
31
48
|
/**
|
|
32
49
|
* Modernized duplicate record detection engine.
|
|
33
50
|
*
|
|
@@ -42,8 +59,20 @@ interface RecordQueryResult {
|
|
|
42
59
|
export declare class DuplicateRecordDetector extends VectorBase {
|
|
43
60
|
private vectorDB;
|
|
44
61
|
private embedding;
|
|
62
|
+
/**
|
|
63
|
+
* The embedding model's API identifier (e.g. 'Xenova/gte-small', 'text-embedding-3-small'),
|
|
64
|
+
* resolved from the entity document's AIModel. Passed to `EmbedTexts` — required by local
|
|
65
|
+
* providers (which load the named ONNX pipeline) and honored by cloud providers.
|
|
66
|
+
*/
|
|
67
|
+
private embeddingModelAPIName;
|
|
45
68
|
/** The Pinecone/pgvector/Qdrant index name resolved from the entity document's VectorIndex */
|
|
46
69
|
private indexName;
|
|
70
|
+
/**
|
|
71
|
+
* The EntityDocumentID for this run. Passed as the vector query `id` for providers that key
|
|
72
|
+
* by EntityDocumentID (the in-process SimpleVectorServiceProvider — see
|
|
73
|
+
* {@link VectorDBBase.QueryKeyIsEntityDocumentID}) instead of the logical index name.
|
|
74
|
+
*/
|
|
75
|
+
private entityDocumentID;
|
|
47
76
|
/**
|
|
48
77
|
* Tracks already-seen source↔match pairs across the entire run to suppress inverse duplicates.
|
|
49
78
|
* If A→B is persisted, B→A is skipped. Key format: "smallerID::largerID" for consistent ordering.
|
|
@@ -217,6 +246,27 @@ export declare class DuplicateRecordDetector extends VectorBase {
|
|
|
217
246
|
* Parse raw vector DB matches into a PotentialDuplicateResult.
|
|
218
247
|
*/
|
|
219
248
|
ParseVectorMatches(queryResponse: BaseResponse, sourceKey?: CompositeKey): PotentialDuplicateResult;
|
|
249
|
+
/**
|
|
250
|
+
* True when two composite keys identify the same record. Compares the normalized
|
|
251
|
+
* (trimmed, lower-cased) URL-segment representation, so it is robust to UUID-casing
|
|
252
|
+
* differences across platforms (SQL Server upper- vs PostgreSQL lower-case) and to a
|
|
253
|
+
* bare-PK vs concatenated key shape.
|
|
254
|
+
*/
|
|
255
|
+
protected isSameRecord(a: CompositeKey, b: CompositeKey): boolean;
|
|
256
|
+
/**
|
|
257
|
+
* Populate a candidate's composite key from a vector match's RecordID, tolerant of the two
|
|
258
|
+
* formats different vector providers emit:
|
|
259
|
+
* - **Concatenated** ("ID|<guid>", or "F1|v1||F2|v2" for composite PKs) — e.g. providers that
|
|
260
|
+
* stored the full key. Parsed via {@link CompositeKey.LoadFromConcatenatedString}.
|
|
261
|
+
* - **Bare primary-key value** ("<guid>") — what the in-process SimpleVectorServiceProvider
|
|
262
|
+
* returns, since it stores `MJ: Entity Record Documents.RecordID` as the raw PK value. For
|
|
263
|
+
* this form we rebuild the key using the source record's PK field name(s).
|
|
264
|
+
*
|
|
265
|
+
* Getting this wrong leaves `KeyValuePairs` empty (the bare value has no field delimiter, so
|
|
266
|
+
* `LoadFromConcatenatedString` no-ops), which silently breaks self-match filtering, candidate
|
|
267
|
+
* field-delta loading (the reasoner sees an empty candidate), and `MatchRecordID` persistence.
|
|
268
|
+
*/
|
|
269
|
+
protected loadCandidateKey(candidate: PotentialDuplicate, recordID: string, sourceKey?: CompositeKey): void;
|
|
220
270
|
/**
|
|
221
271
|
* Filter out self-matches where the candidate is the same record as the source.
|
|
222
272
|
*/
|
|
@@ -255,16 +305,131 @@ export declare class DuplicateRecordDetector extends VectorBase {
|
|
|
255
305
|
protected CreateRunDetailRecords(recordIDs: string[], duplicateRunID: string, entityInfo: EntityInfo, metadataMap?: Map<string, string>): Promise<MJDuplicateRunDetailEntity[]>;
|
|
256
306
|
/**
|
|
257
307
|
* Persist match results and update run detail records.
|
|
308
|
+
*
|
|
309
|
+
* When LLM reasoning is enabled on the entity document AND a matched set clears the
|
|
310
|
+
* reasoning gate (top vector score >= ReasoningThreshold), reasoning runs ONCE for the
|
|
311
|
+
* whole set; the verdict is written onto every match row's LLM* columns and carried on
|
|
312
|
+
* the result for the auto-merge step. When reasoning is disabled the loop is unchanged.
|
|
313
|
+
*/
|
|
314
|
+
protected PersistMatchResults(queryResults: RecordQueryResult[], duplicateRunDetails: MJDuplicateRunDetailEntity[], entityInfo: EntityInfo, entityDocument: MJEntityDocumentEntity, options: DuplicateDetectionOptions, startTime: number, contextUser?: UserInfo): Promise<PotentialDuplicateResult[]>;
|
|
315
|
+
/**
|
|
316
|
+
* Drop inverse duplicates in place: if A→B was already persisted this run, skip B→A.
|
|
317
|
+
*/
|
|
318
|
+
protected suppressInverseDuplicates(qr: RecordQueryResult): void;
|
|
319
|
+
/**
|
|
320
|
+
* Locate the matching run-detail row, create the per-candidate match records (carrying
|
|
321
|
+
* the set's LLM verdict when present), mark the detail complete, and stamp the created
|
|
322
|
+
* match-record IDs back onto the result (kept in lockstep with Duplicates for auto-merge).
|
|
258
323
|
*/
|
|
259
|
-
protected
|
|
324
|
+
protected persistDetailMatches(qr: RecordQueryResult, duplicateRunDetails: MJDuplicateRunDetailEntity[], reasoning?: DuplicateReasoningOutput): Promise<void>;
|
|
260
325
|
/**
|
|
261
326
|
* Create match records for a single run detail, saving in parallel batches.
|
|
327
|
+
*
|
|
328
|
+
* When a per-set {@link DuplicateReasoningOutput} is supplied, each match row gets THIS
|
|
329
|
+
* candidate's own verdict (recommendation / confidence / reasoning), while the set-level
|
|
330
|
+
* proposed survivor / field map / run id ride along on every row.
|
|
331
|
+
*/
|
|
332
|
+
protected CreateMatchRecordsForDetail(duplicateRunDetailID: string, duplicateResult: PotentialDuplicateResult, reasoning?: DuplicateReasoningOutput): Promise<MJDuplicateRunDetailMatchEntity[]>;
|
|
333
|
+
/**
|
|
334
|
+
* Write the LLM verdict + run id onto a single match row's generated columns. The
|
|
335
|
+
* recommendation / confidence / reasoning are taken from THIS candidate's own verdict (judged
|
|
336
|
+
* independently), so a false-positive candidate reads NotDuplicate even when another candidate
|
|
337
|
+
* in the same set is a confident Merge. Falls back to the set-level summary only when the
|
|
338
|
+
* reasoner returned no per-candidate verdict for this record. The proposed survivor + field
|
|
339
|
+
* map stay set-level (they describe the merge of the set's true duplicates).
|
|
340
|
+
*
|
|
341
|
+
* @param candidateRecordID this candidate's record id (matches the input candidate RecordID)
|
|
342
|
+
*/
|
|
343
|
+
protected applyReasoningToMatch(match: MJDuplicateRunDetailMatchEntity, reasoning: DuplicateReasoningOutput, candidateRecordID: string): void;
|
|
344
|
+
/**
|
|
345
|
+
* Run LLM reasoning for one source record's matched set, ONCE, only when:
|
|
346
|
+
* 1. the entity document has EnableLLMReasoning = true, AND
|
|
347
|
+
* 2. the set is non-empty, AND
|
|
348
|
+
* 3. the set's top MatchProbability >= the (non-null) ReasoningThreshold.
|
|
349
|
+
*
|
|
350
|
+
* Returns `undefined` in every other case — including when reasoning is disabled — so
|
|
351
|
+
* the surrounding persist/merge path stays byte-for-byte identical to the vector-only
|
|
352
|
+
* behavior. A failed reasoning call returns an output with Success = false (never throws),
|
|
353
|
+
* so one bad set never aborts the run.
|
|
354
|
+
*/
|
|
355
|
+
protected RunReasoningForSet(qr: RecordQueryResult, entityInfo: EntityInfo, entityDocument: MJEntityDocumentEntity, contextUser?: UserInfo): Promise<SetReasoning | undefined>;
|
|
356
|
+
/**
|
|
357
|
+
* Null out a hallucinated survivor: the reasoner can return a SurvivorRecordID that isn't
|
|
358
|
+
* any record in the set (a made-up GUID, or a candidate that was filtered out). Persisting
|
|
359
|
+
* it would carry a phantom id to the UI and auto-merge. Validate against the set's record
|
|
360
|
+
* ids (source + candidates) and drop it if it doesn't match.
|
|
361
|
+
*/
|
|
362
|
+
protected validateSurvivor(output: DuplicateReasoningOutput, input: DuplicateReasoningInput): void;
|
|
363
|
+
/**
|
|
364
|
+
* Resolve the reasoner's per-field survivor choices ({FieldName, SourceRecordID}) into
|
|
365
|
+
* literal `{FieldName, Value}` entries for {@link RecordMergeRequest.FieldMap}, by reading
|
|
366
|
+
* the chosen record's value out of the matched-set deltas. Choices whose field or record
|
|
367
|
+
* can't be located in the deltas are skipped (the survivor's existing value then stands),
|
|
368
|
+
* so a partial/garbled choice can never inject an undefined override into the merge.
|
|
369
|
+
*/
|
|
370
|
+
protected resolveReasoningFieldMap(output: DuplicateReasoningOutput, fieldDeltas: ReasoningFieldDelta[]): {
|
|
371
|
+
FieldName: string;
|
|
372
|
+
Value: unknown;
|
|
373
|
+
}[];
|
|
374
|
+
/** Case-insensitive, trimmed equality of two record-id (URL-segment) strings. */
|
|
375
|
+
protected recordIdMatches(a: string, b: string): boolean;
|
|
376
|
+
/**
|
|
377
|
+
* Gate predicate: reasoning enabled + non-empty set + top score clears the threshold.
|
|
378
|
+
* A null ReasoningThreshold means "no gate" — reasoning runs for any non-empty set.
|
|
379
|
+
*/
|
|
380
|
+
protected IsReasoningGateOpen(qr: RecordQueryResult, entityDocument: MJEntityDocumentEntity): boolean;
|
|
381
|
+
/**
|
|
382
|
+
* Resolve the reasoning provider for this entity document's ReasoningMode via the
|
|
383
|
+
* ClassFactory. Returns null when ClassFactory falls back to the abstract base (no
|
|
384
|
+
* implementation registered for the mode) so the caller can skip gracefully.
|
|
385
|
+
*/
|
|
386
|
+
protected ResolveReasoningProvider(entityDocument: MJEntityDocumentEntity): DuplicateReasoningProvider | null;
|
|
387
|
+
/**
|
|
388
|
+
* Assemble the reasoning input for a matched set: source description, candidate
|
|
389
|
+
* descriptions, and the differing-field deltas loaded for the whole set.
|
|
262
390
|
*/
|
|
263
|
-
protected
|
|
391
|
+
protected BuildReasoningInput(qr: RecordQueryResult, entityInfo: EntityInfo, entityDocument: MJEntityDocumentEntity, contextUser?: UserInfo): Promise<DuplicateReasoningInput>;
|
|
392
|
+
/**
|
|
393
|
+
* Resolve a record's display label: prefer the name loaded from the record (via the delta
|
|
394
|
+
* builder), then a secondary fallback (e.g. a candidate's vector-metadata name), then the
|
|
395
|
+
* raw record-id key as a last resort. Guards against the loaded label itself being the
|
|
396
|
+
* record key (the engine's own fallback) so we still try the secondary source.
|
|
397
|
+
*/
|
|
398
|
+
protected resolveRecordLabel(labels: Map<string, string>, recordID: string, fallback: string | null): string;
|
|
399
|
+
/** Extract a display label from a candidate's vector metadata snapshot, if present. */
|
|
400
|
+
protected reasoningLabel(metadata?: Record<string, string>): string | null;
|
|
401
|
+
/**
|
|
402
|
+
* Carry the set-level verdict + resolved survivor field map onto the result so the
|
|
403
|
+
* auto-merge step (AutoMergeAboveAbsolute) can consult the recommendation and apply the
|
|
404
|
+
* literal {FieldName, Value} overrides via {@link RecordMergeRequest.FieldMap}. The UI
|
|
405
|
+
* still reads the persisted per-row {@link MJDuplicateRunDetailMatchEntity.LLMProposedFieldMap}
|
|
406
|
+
* (the raw choices) and lets the reviewer override before a manual merge.
|
|
407
|
+
*/
|
|
408
|
+
protected applyReasoningToResult(result: PotentialDuplicateResult, output: DuplicateReasoningOutput, fieldMap: {
|
|
409
|
+
FieldName: string;
|
|
410
|
+
Value: unknown;
|
|
411
|
+
}[]): void;
|
|
264
412
|
/**
|
|
265
413
|
* Automatically merge records that meet the absolute match threshold.
|
|
266
414
|
*/
|
|
267
415
|
protected ProcessAutoMerges(response: PotentialDuplicateResponse, entityDocument: MJEntityDocumentEntity, options?: DuplicateDetectionOptions): Promise<void>;
|
|
416
|
+
/**
|
|
417
|
+
* Decide whether a single candidate is eligible for automatic merge.
|
|
418
|
+
*
|
|
419
|
+
* Back-compat path (EnableLLMReasoning = false): eligible iff the vector score meets the
|
|
420
|
+
* absolute threshold — byte-for-byte the original behavior. `AutomationLevel` is ignored.
|
|
421
|
+
*
|
|
422
|
+
* Reasoning path (EnableLLMReasoning = true): `AutomationLevel` governs.
|
|
423
|
+
* - ReviewAll / LLMGated → never auto-merge (everything goes to human review).
|
|
424
|
+
* - AutoMergeAboveAbsolute → at/above the absolute threshold AND the set's LLM
|
|
425
|
+
* recommendation is 'Merge'.
|
|
426
|
+
*/
|
|
427
|
+
protected IsAutoMergeEligible(dupe: PotentialDuplicate, dupeResult: PotentialDuplicateResult, entityDocument: MJEntityDocumentEntity, absoluteThreshold: number): boolean;
|
|
428
|
+
/**
|
|
429
|
+
* Execute one guarded auto-merge for an eligible candidate, applying the LLM's resolved
|
|
430
|
+
* field map when present, and stamp the match record on success.
|
|
431
|
+
*/
|
|
432
|
+
protected executeAutoMerge(dupe: PotentialDuplicate, dupeResult: PotentialDuplicateResult, entityDocument: MJEntityDocumentEntity, index: number): Promise<void>;
|
|
268
433
|
/**
|
|
269
434
|
* Update a match record's status after a successful merge.
|
|
270
435
|
*/
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"duplicateRecordDetector.d.ts","sourceRoot":"","sources":["../src/duplicateRecordDetector.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;GAeG;AAGH,OAAO,EACH,yBAAyB,EACzB,0BAA0B,EAC1B,YAAY,EACZ,QAAQ,EACR,UAAU,EACV,wBAAwB,EAIxB,UAAU,EACV,kBAAkB,EAClB,yBAAyB,
|
|
1
|
+
{"version":3,"file":"duplicateRecordDetector.d.ts","sourceRoot":"","sources":["../src/duplicateRecordDetector.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;GAeG;AAGH,OAAO,EACH,yBAAyB,EACzB,0BAA0B,EAC1B,YAAY,EACZ,QAAQ,EACR,UAAU,EACV,wBAAwB,EAIxB,UAAU,EACV,kBAAkB,EAClB,yBAAyB,EAE5B,MAAM,sBAAsB,CAAC;AAC9B,OAAO,EAAE,YAAY,EAA6C,MAAM,6BAA6B,CAAC;AAEtG,OAAO,EACH,0BAA0B,EAC1B,+BAA+B,EAC/B,oBAAoB,EACpB,sBAAsB,EACtB,YAAY,EAEf,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,UAAU,EAAE,MAAM,4BAA4B,CAAC;AACxD,OAAO,EAAE,4BAA4B,EAA6C,MAAM,gCAAgC,CAAC;AAEzH,OAAO,KAAK,EAAE,wBAAwB,EAA2B,MAAM,+BAA+B,CAAC;AACvG,OAAO,EAAE,0BAA0B,EAAE,MAAM,wCAAwC,CAAC;AACpF,OAAO,EACH,uBAAuB,EACvB,wBAAwB,EAExB,mBAAmB,EACtB,MAAM,qCAAqC,CAAC;AAuC7C;;GAEG;AACH,UAAU,iBAAiB;IACvB,SAAS,EAAE,YAAY,CAAC;IACxB,YAAY,EAAE,MAAM,CAAC;IACrB,UAAU,EAAE,wBAAwB,CAAC;CACxC;AAED;;;;;;GAMG;AACH,UAAU,YAAY;IAClB,MAAM,EAAE,wBAAwB,CAAC;IACjC,sFAAsF;IACtF,QAAQ,EAAE;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,OAAO,CAAA;KAAE,EAAE,CAAC;CACrD;AAED;;;;;;;;;;GAUG;AACH,qBAAa,uBAAwB,SAAQ,UAAU;IACnD,OAAO,CAAC,QAAQ,CAAe;IAC/B,OAAO,CAAC,SAAS,CAAiB;IAClC;;;;OAIG;IACH,OAAO,CAAC,qBAAqB,CAAuB;IACpD,8FAA8F;IAC9F,OAAO,CAAC,SAAS,CAAS;IAC1B;;;;OAIG;IACH,OAAO,CAAC,gBAAgB,CAAS;IACjC;;;OAGG;IACH,OAAO,CAAC,UAAU,CAAqB;IAEvC;;;;;;;;;;;;;;;;;;;;;;;;;OAyBG;IACU,mBAAmB,CAAC,MAAM,EAAE,yBAAyB,EAAE,WAAW,CAAC,EAAE,QAAQ,GAAG,OAAO,CAAC,0BAA0B,CAAC;IA0GhI;;;OAGG;IACU,iBAAiB,CAC1B,gBAAgB,EAAE,MAAM,EACxB,QAAQ,EAAE,YAAY,EACtB,OAAO,CAAC,EAAE,yBAAyB,EACnC,WAAW,CAAC,EAAE,QAAQ,GACvB,OAAO,CAAC,wBAAwB,CAAC;IA0DpC;;;;;;;;;;;;;;;;;;;;;;;;;OAyBG;YACW,YAAY;IAwE1B;;;OAGG;cACa,oBAAoB,CAAC,MAAM,EAAE,yBAAyB,EAAE,UAAU,EAAE,UAAU,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;IAWlH;;OAEG;cACa,qBAAqB,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;IAexE;;OAEG;cACa,qBAAqB,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;IAyBhG;;;OAGG;cACa,uBAAuB,CAAC,UAAU,EAAE,UAAU,EAAE,WAAW,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;IAmBxG;;OAEG;IACH;;;OAGG;cACa,sBAAsB,CAAC,gBAAgB,EAAE,MAAM,GAAG,OAAO,CAAC,sBAAsB,GAAG,IAAI,CAAC;IAOxG;;;;OAIG;cACa,mBAAmB,CAAC,cAAc,EAAE,sBAAsB,GAAG,OAAO,CAAC,IAAI,CAAC;IA6D1F;;OAEG;cACa,sBAAsB,CAAC,cAAc,EAAE,sBAAsB,EAAE,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAkBpH;;;OAGG;IACH,SAAS,CAAC,mBAAmB,CAAC,cAAc,EAAE,sBAAsB,GAAG,MAAM;IAmB7E;;;OAGG;cACa,2BAA2B,CACvC,MAAM,EAAE,yBAAyB,EACjC,cAAc,EAAE,sBAAsB,EACtC,OAAO,EAAE,yBAAyB,GACnC,OAAO,CAAC,oBAAoB,CAAC;IAkBhC;;OAEG;cACa,kBAAkB,CAC9B,cAAc,EAAE,sBAAsB,EACtC,MAAM,CAAC,EAAE,MAAM,GAChB,OAAO,CAAC,oBAAoB,CAAC;cAmBhB,gBAAgB,CAAC,cAAc,EAAE,MAAM,GAAG,OAAO,CAAC,oBAAoB,CAAC;IAUvF;;OAEG;cACa,wBAAwB,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,oBAAoB,GAAG,IAAI,CAAC;IAU9F;;OAEG;cACa,iBAAiB,CAAC,aAAa,EAAE,YAAY,EAAE,EAAE,UAAU,EAAE,UAAU,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;IAiB/G;;;OAGG;cACa,mBAAmB,CAAC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;cAmB5E,cAAc,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,YAAY,CAAC;IAcrE;;;;;OAKG;cACa,qBAAqB,CACjC,cAAc,EAAE,UAAU,CAAC,OAAO,4BAA4B,CAAC,cAAc,CAAC,EAC9E,cAAc,EAAE,sBAAsB,EACtC,OAAO,EAAE,UAAU,EAAE,EACrB,WAAW,EAAE,QAAQ,GACtB,OAAO,CAAC,MAAM,EAAE,CAAC;IA0BpB;;OAEG;IACH,SAAS,CAAC,YAAY,CAAC,cAAc,EAAE,sBAAsB,GAAG,wBAAwB;IAkBxF;;;;;;;;;;;;;;;;;;;;;;;OAuBG;cACa,yBAAyB,CACrC,OAAO,EAAE,UAAU,EAAE,EACrB,OAAO,EAAE,MAAM,EAAE,EAAE,EACnB,aAAa,EAAE,MAAM,EAAE,EACvB,cAAc,EAAE,sBAAsB,EACtC,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,yBAAyB,EAClC,WAAW,EAAE,MAAM,GACpB,OAAO,CAAC,iBAAiB,EAAE,CAAC;IA+B/B;;OAEG;cACa,kBAAkB,CAC9B,MAAM,EAAE,MAAM,EAAE,EAChB,YAAY,EAAE,MAAM,EACpB,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,yBAAyB,GACnC,OAAO,CAAC,YAAY,CAAC;IA2BxB;;OAEG;IACI,kBAAkB,CAAC,aAAa,EAAE,YAAY,EAAE,SAAS,CAAC,EAAE,YAAY,GAAG,wBAAwB;IAgC1G;;;;;OAKG;IACH,SAAS,CAAC,YAAY,CAAC,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,YAAY,GAAG,OAAO;IAIjE;;;;;;;;;;;;OAYG;IACH,SAAS,CAAC,gBAAgB,CAAC,SAAS,EAAE,kBAAkB,EAAE,QAAQ,EAAE,MAAM,EAAE,SAAS,CAAC,EAAE,YAAY,GAAG,IAAI;IAe3G;;OAEG;IACH;;;OAGG;IACH,SAAS,CAAC,sBAAsB,CAAC,OAAO,EAAE,UAAU,EAAE,EAAE,UAAU,EAAE,UAAU,GAAG,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC;IA4DpG,SAAS,CAAC,iBAAiB,CAAC,UAAU,EAAE,kBAAkB,EAAE,EAAE,SAAS,EAAE,YAAY,GAAG,kBAAkB,EAAE;IAS5G;;;;;;;;;;;;;;;OAeG;cACa,wBAAwB,CAAC,YAAY,EAAE,iBAAiB,EAAE,EAAE,UAAU,EAAE,UAAU,GAAG,OAAO,CAAC,IAAI,CAAC;IAgClH;;;;OAIG;cACa,qBAAqB,CAAC,UAAU,EAAE,UAAU,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;IA8BnH;;OAEG;cACa,sBAAsB,CAClC,SAAS,EAAE,MAAM,EAAE,EACnB,cAAc,EAAE,MAAM,EACtB,UAAU,EAAE,UAAU,EACtB,WAAW,CAAC,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,GAClC,OAAO,CAAC,0BAA0B,EAAE,CAAC;IA+BxC;;;;;;;OAOG;cACa,mBAAmB,CAC/B,YAAY,EAAE,iBAAiB,EAAE,EACjC,mBAAmB,EAAE,0BAA0B,EAAE,EACjD,UAAU,EAAE,UAAU,EACtB,cAAc,EAAE,sBAAsB,EACtC,OAAO,EAAE,yBAAyB,EAClC,SAAS,EAAE,MAAM,EACjB,WAAW,CAAC,EAAE,QAAQ,GACvB,OAAO,CAAC,wBAAwB,EAAE,CAAC;IAyBtC;;OAEG;IACH,SAAS,CAAC,yBAAyB,CAAC,EAAE,EAAE,iBAAiB,GAAG,IAAI;IAahE;;;;OAIG;cACa,oBAAoB,CAChC,EAAE,EAAE,iBAAiB,EACrB,mBAAmB,EAAE,0BAA0B,EAAE,EACjD,SAAS,CAAC,EAAE,wBAAwB,GACrC,OAAO,CAAC,IAAI,CAAC;IAsBhB;;;;;;OAMG;cACa,2BAA2B,CACvC,oBAAoB,EAAE,MAAM,EAC5B,eAAe,EAAE,wBAAwB,EACzC,SAAS,CAAC,EAAE,wBAAwB,GACrC,OAAO,CAAC,+BAA+B,EAAE,CAAC;IAiC7C;;;;;;;;;OASG;IACH,SAAS,CAAC,qBAAqB,CAC3B,KAAK,EAAE,+BAA+B,EACtC,SAAS,EAAE,wBAAwB,EACnC,iBAAiB,EAAE,MAAM,GAC1B,IAAI;IAiBP;;;;;;;;;;OAUG;cACa,kBAAkB,CAC9B,EAAE,EAAE,iBAAiB,EACrB,UAAU,EAAE,UAAU,EACtB,cAAc,EAAE,sBAAsB,EACtC,WAAW,CAAC,EAAE,QAAQ,GACvB,OAAO,CAAC,YAAY,GAAG,SAAS,CAAC;IA0BpC;;;;;OAKG;IACH,SAAS,CAAC,gBAAgB,CAAC,MAAM,EAAE,wBAAwB,EAAE,KAAK,EAAE,uBAAuB,GAAG,IAAI;IAYlG;;;;;;OAMG;IACH,SAAS,CAAC,wBAAwB,CAC9B,MAAM,EAAE,wBAAwB,EAChC,WAAW,EAAE,mBAAmB,EAAE,GACnC;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,OAAO,CAAA;KAAE,EAAE;IAe1C,iFAAiF;IACjF,SAAS,CAAC,eAAe,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,MAAM,GAAG,OAAO;IAIxD;;;OAGG;IACH,SAAS,CAAC,mBAAmB,CAAC,EAAE,EAAE,iBAAiB,EAAE,cAAc,EAAE,sBAAsB,GAAG,OAAO;IAgBrG;;;;OAIG;IACH,SAAS,CAAC,wBAAwB,CAAC,cAAc,EAAE,sBAAsB,GAAG,0BAA0B,GAAG,IAAI;IAc7G;;;OAGG;cACa,mBAAmB,CAC/B,EAAE,EAAE,iBAAiB,EACrB,UAAU,EAAE,UAAU,EACtB,cAAc,EAAE,sBAAsB,EACtC,WAAW,CAAC,EAAE,QAAQ,GACvB,OAAO,CAAC,uBAAuB,CAAC;IAgCnC;;;;;OAKG;IACH,SAAS,CAAC,kBAAkB,CAAC,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,GAAG,IAAI,GAAG,MAAM;IAW5G,uFAAuF;IACvF,SAAS,CAAC,cAAc,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM,GAAG,IAAI;IAO1E;;;;;;OAMG;IACH,SAAS,CAAC,sBAAsB,CAC5B,MAAM,EAAE,wBAAwB,EAChC,MAAM,EAAE,wBAAwB,EAChC,QAAQ,EAAE;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,OAAO,CAAA;KAAE,EAAE,GAClD,IAAI;IAaP;;OAEG;cACa,iBAAiB,CAC7B,QAAQ,EAAE,0BAA0B,EACpC,cAAc,EAAE,sBAAsB,EACtC,OAAO,GAAE,yBAA8B,GACxC,OAAO,CAAC,IAAI,CAAC;IAsBhB;;;;;;;;;;OAUG;IACH,SAAS,CAAC,mBAAmB,CACzB,IAAI,EAAE,kBAAkB,EACxB,UAAU,EAAE,wBAAwB,EACpC,cAAc,EAAE,sBAAsB,EACtC,iBAAiB,EAAE,MAAM,GAC1B,OAAO;IAWV;;;OAGG;cACa,gBAAgB,CAC5B,IAAI,EAAE,kBAAkB,EACxB,UAAU,EAAE,wBAAwB,EACpC,cAAc,EAAE,sBAAsB,EACtC,KAAK,EAAE,MAAM,GACd,OAAO,CAAC,IAAI,CAAC;IAuBhB;;OAEG;YACW,2BAA2B;IAwBzC,OAAO,CAAC,cAAc;CAoBzB"}
|