@futux/dqi 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +123 -0
- package/package.json +1 -1
package/README.md
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# @futux/dqi (Data Quality Index for JavaScript & TypeScript)
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/@futux/dqi)
|
|
4
|
+
[](https://www.npmjs.com/package/@futux/dqi)
|
|
5
|
+
[](https://opensource.org/licenses/Apache-2.0)
|
|
6
|
+
[](https://futuxagriconsult.github.io/open-dqi-core/)
|
|
7
|
+
|
|
8
|
+
> **Lightweight, schema-agnostic Data Quality Index (DQI) library for Node.js, Next.js, and modern browser runtimes.** Score any tabular dataset (CSV, JSON, array of objects) across four weighted dimensions: **Completeness**, **Validity**, **Uniqueness**, and **Outliers** in milliseconds.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
### 🌐 Try the Interactive Web Playground
|
|
13
|
+
|
|
14
|
+
Test `@futux/dqi` directly in your browser:
|
|
15
|
+
👉 **[Launch Live Web Demo (https://futuxagriconsult.github.io/open-dqi-core/)](https://futuxagriconsult.github.io/open-dqi-core/)**
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## 🚀 Key Highlights
|
|
20
|
+
|
|
21
|
+
- **Zero-Config Auto-Inference:** Evaluates arbitrary JSON/CSV records out-of-the-box without requiring predefined schemas.
|
|
22
|
+
- **Universal Single Metric:** Outputs an intuitive 0–100 score and academic letter grade (A–F) for easy communication with non-technical stakeholders.
|
|
23
|
+
- **Client & Server Ready:** Runs natively in Node.js, Bun, Deno, Web Workers, and modern browser single-page applications.
|
|
24
|
+
- **Pure JavaScript / Zero Heavy Native Addons:** Extremely small bundle footprint, ultra-fast cold start.
|
|
25
|
+
- **Auditable Defect Breakdown:** Pinpoints exact row numbers, defect types, and violating values for instant remediation.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## 📦 Installation
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
# npm
|
|
33
|
+
npm install @futux/dqi
|
|
34
|
+
|
|
35
|
+
# yarn
|
|
36
|
+
yarn add @futux/dqi
|
|
37
|
+
|
|
38
|
+
# pnpm
|
|
39
|
+
pnpm add @futux/dqi
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## 💡 Quick Start
|
|
45
|
+
|
|
46
|
+
### 1. Zero-Config Evaluation
|
|
47
|
+
|
|
48
|
+
Pass any array of tabular objects. `@futux/dqi` automatically infers data types, detects sentinel nulls, checks formats, and flags anomalies:
|
|
49
|
+
|
|
50
|
+
```typescript
|
|
51
|
+
import { DQIEngine } from "@futux/dqi";
|
|
52
|
+
|
|
53
|
+
const dataset = [
|
|
54
|
+
{ id: "REC-101", name: "Ibrahim Musa", phone: "+254712345678", yield_kg: 42.5 },
|
|
55
|
+
{ id: "REC-102", name: "Sarah Cherono", phone: "+254787654321", yield_kg: 38.0 },
|
|
56
|
+
{ id: "REC-103", name: "N/A", phone: "invalid-phone", yield_kg: 9999.0 },
|
|
57
|
+
{ id: "REC-101", name: "Ibrahim Musa", phone: "+254712345678", yield_kg: 42.5 } // Duplicate
|
|
58
|
+
];
|
|
59
|
+
|
|
60
|
+
const engine = new DQIEngine();
|
|
61
|
+
const report = engine.evaluate(dataset);
|
|
62
|
+
|
|
63
|
+
console.log(`Overall DQI: ${report.overall.toFixed(2)}/100 (Grade: ${report.grade})`);
|
|
64
|
+
console.log(`- Completeness: ${report.dimensions.completeness.score.toFixed(1)}%`);
|
|
65
|
+
console.log(`- Validity: ${report.dimensions.validity.score.toFixed(1)}%`);
|
|
66
|
+
console.log(`- Uniqueness: ${report.dimensions.uniqueness.score.toFixed(1)}%`);
|
|
67
|
+
console.log(`- Outliers: ${report.dimensions.outliers.score.toFixed(1)}%`);
|
|
68
|
+
|
|
69
|
+
// List flagged defects
|
|
70
|
+
console.log("Flagged Defects:", report.defects);
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
### 2. Custom Schema & Dimension Weights
|
|
74
|
+
|
|
75
|
+
Configure custom validation rules and weighting:
|
|
76
|
+
|
|
77
|
+
```typescript
|
|
78
|
+
import { DQIEngine, Schema } from "@futux/dqi";
|
|
79
|
+
|
|
80
|
+
const schema = new Schema({
|
|
81
|
+
name: "farmer_registry",
|
|
82
|
+
weights: {
|
|
83
|
+
completeness: 0.35,
|
|
84
|
+
validity: 0.35,
|
|
85
|
+
uniqueness: 0.15,
|
|
86
|
+
outliers: 0.15
|
|
87
|
+
},
|
|
88
|
+
fields: {
|
|
89
|
+
id: { type: "id", required: true, unique: true },
|
|
90
|
+
name: { type: "text", required: true },
|
|
91
|
+
phone: { type: "phone", required: true },
|
|
92
|
+
yield_kg: { type: "numeric", min: 1.0, max: 500.0 }
|
|
93
|
+
}
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
const engine = new DQIEngine(schema);
|
|
97
|
+
const report = engine.evaluate(dataset);
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
---
|
|
101
|
+
|
|
102
|
+
## 🔬 Four Quality Dimensions
|
|
103
|
+
|
|
104
|
+
1. **Completeness (35% default):** Measures missing, null, empty, whitespace-only, and sentinel placeholder strings (`N/A`, `NULL`, `none`, `-`, `?`).
|
|
105
|
+
2. **Validity (35% default):** Audits regex conformance, ISO-8601 dates, E.164 phone formats, GPS coordinate boundaries, and enum lists.
|
|
106
|
+
3. **Uniqueness (15% default):** Identifies exact duplicate rows and primary key collisions.
|
|
107
|
+
4. **Outlier Detection (15% default):** Detects statistical extreme anomalies using Interquartile Range ($1.5 \times \text{IQR}$).
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## 🌐 Also Available in Python
|
|
112
|
+
|
|
113
|
+
For Python data pipelines and pandas integration, install the companion package:
|
|
114
|
+
```bash
|
|
115
|
+
pip install open-dqi-core
|
|
116
|
+
```
|
|
117
|
+
See [GitHub Repository](https://github.com/futuxagriconsult/open-dqi-core) for Python CLI and offline HTML reporting details.
|
|
118
|
+
|
|
119
|
+
---
|
|
120
|
+
|
|
121
|
+
## 📄 License
|
|
122
|
+
|
|
123
|
+
Apache-2.0. Developed and maintained by **FutuX Agri-consult Limited** ([info@futuxconsult.com](mailto:info@futuxconsult.com)).
|