@futux/whitebox-dedup 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +103 -0
- package/package.json +1 -1
package/README.md
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# @futux/whitebox-dedup (Interpretable Entity Deduplication in TypeScript)
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/@futux/whitebox-dedup)
|
|
4
|
+
[](https://www.npmjs.com/package/@futux/whitebox-dedup)
|
|
5
|
+
[](https://opensource.org/licenses/MIT)
|
|
6
|
+
[](https://futuxagriconsult.github.io/whitebox-dedup-ml/)
|
|
7
|
+
|
|
8
|
+
> **Auditable, transparent entity deduplication and record linkage with Multivariate Adaptive Regression Splines (MARS) and Inductive Conformal Prediction (ICP).** High-performance, zero-native-dependency TypeScript library for Node.js, Next.js, and browser Web Workers.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
### 🌐 Try the Interactive Web Playground
|
|
13
|
+
|
|
14
|
+
Test `@futux/whitebox-dedup` live directly in your browser:
|
|
15
|
+
👉 **[Launch Live Web Demo (https://futuxagriconsult.github.io/whitebox-dedup-ml/)](https://futuxagriconsult.github.io/whitebox-dedup-ml/)**
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## 🚀 Key Features
|
|
20
|
+
|
|
21
|
+
- **Auditable MARS Scoring:** No black-box neural networks. Outputs clear hinge-function math explaining precisely why two entities matched.
|
|
22
|
+
- **Three Conformal Statistical Tiers:**
|
|
23
|
+
- **Tier 1 (Auto-Merge, $\ge 0.85$):** Conformal $p$-value $\ge 0.95$; statistically safe for automated merging.
|
|
24
|
+
- **Tier 2 (Circuit Breaker / Review, $0.60 - 0.84$):** Ambiguous boundary cases routed to human adjudicators with exact contributing features.
|
|
25
|
+
- **Tier 3 (Auto-Reject, $< 0.60$):** Distinct records; kept separate.
|
|
26
|
+
- **Client & Server Compatible:** Runs in Node.js, Bun, Deno, Next.js API routes, and client-side Web Workers.
|
|
27
|
+
- **Zero Heavy Native Dependencies:** Fast cold-start, pure TypeScript, instant browser compilation.
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## 📦 Installation
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
# npm
|
|
35
|
+
npm install @futux/whitebox-dedup
|
|
36
|
+
|
|
37
|
+
# yarn
|
|
38
|
+
yarn add @futux/whitebox-dedup
|
|
39
|
+
|
|
40
|
+
# pnpm
|
|
41
|
+
pnpm add @futux/whitebox-dedup
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## 💡 Quick Start
|
|
47
|
+
|
|
48
|
+
```typescript
|
|
49
|
+
import { DeduplicationPipeline } from "@futux/whitebox-dedup";
|
|
50
|
+
|
|
51
|
+
const records = [
|
|
52
|
+
{ id: "1", name: "Ibrahim Musa", phone: "+254712345678" },
|
|
53
|
+
{ id: "2", name: "Ibraheem Mussa", phone: "+254712345678" },
|
|
54
|
+
{ id: "3", name: "Sarah Cherono", phone: "+254787654321" }
|
|
55
|
+
];
|
|
56
|
+
|
|
57
|
+
const pipeline = new DeduplicationPipeline({
|
|
58
|
+
model: "text_phone_v1",
|
|
59
|
+
confidence: 0.95
|
|
60
|
+
});
|
|
61
|
+
|
|
62
|
+
const results = pipeline.run(records);
|
|
63
|
+
|
|
64
|
+
results.matches.forEach(match => {
|
|
65
|
+
console.log(`Pair: ${match.recordA.name} <-> ${match.recordB.name}`);
|
|
66
|
+
console.log(`Score: ${(match.score * 100).toFixed(1)}% | Tier: ${match.tier}`);
|
|
67
|
+
console.log(`Auditable Basis:`, match.explanation);
|
|
68
|
+
});
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## 🔬 How the Math Works
|
|
74
|
+
|
|
75
|
+
The score is calculated via transparent piecewise linear splines:
|
|
76
|
+
|
|
77
|
+
```text
|
|
78
|
+
Score = 1.0 / (1.0 + exp(-(
|
|
79
|
+
-3.8421
|
|
80
|
+
+ 5.1204 * max(0, jaro_winkler - 0.7200)
|
|
81
|
+
- 2.8910 * max(0, 0.7200 - jaro_winkler)
|
|
82
|
+
+ 1.8432 * max(0, token_sort_ratio - 0.6500)
|
|
83
|
+
+ 0.9420 * exact_phone_match
|
|
84
|
+
)))
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Every prediction can be audited, reviewed, and justified in regulatory or administrative hearings.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## 🐍 Also Available in Python
|
|
92
|
+
|
|
93
|
+
For Python data pipelines and pandas integration, install the companion package:
|
|
94
|
+
```bash
|
|
95
|
+
pip install whitebox-dedup
|
|
96
|
+
```
|
|
97
|
+
See [GitHub Repository](https://github.com/futuxagriconsult/whitebox-dedup-ml) for CLI and calibration tools.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## 📄 License
|
|
102
|
+
|
|
103
|
+
MIT. Developed and maintained by **FutuX Agri-consult Limited** ([info@futuxconsult.com](mailto:info@futuxconsult.com)).
|
package/package.json
CHANGED